- Zig 98.1%
- Python 1.8%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
glibc converts EUC-CN to BIG5, EUC-CN to GBK and GBK to EUC-CN directly rather than through UCS-4, and gets answers of its own. convertGlibcAlloc goes through Unicode for every pair, so say so rather than claiming it always gives glibc's answer. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01P9FTtWx5YQhHu7irRjDsKM |
||
| .forgejo/workflows | ||
| data | ||
| docs/adr | ||
| LICENSES | ||
| src | ||
| tests | ||
| tools | ||
| .gitignore | ||
| build.zig | ||
| build.zig.zon | ||
| CONTRIBUTING.md | ||
| flake.lock | ||
| flake.nix | ||
| GLOSSARY.md | ||
| README.md | ||
| REUSE.toml | ||
zig-charset
Converting between UTF-8 and the legacy character encodings — windows-1252, Shift_JIS, Big5, gb18030 and the rest — as the WHATWG Encoding Standard defines them.
The API documentation is generated from the doc comments and published at
https://jeff.jcollie.page/zig-charset/; zig build docs-serve reads the
same thing locally.
const charset = @import("charset");
// A label off a Content-Type header, if it names an encoding at all.
const encoding = charset.Encoding.fromLabel("Shift-JIS") orelse return error.UnknownCharset;
const utf8 = try charset.decodeAlloc(gpa, encoding, bytes, .{});
defer gpa.free(utf8);
What is in it
The forty encodings the standard defines, and no others:
| UTF-8 | UTF-8 |
| Single-byte | IBM866, ISO-8859-2 through -16, ISO-8859-8-I, KOI8-R, KOI8-U, macintosh, windows-874, windows-1250 through -1258, x-mac-cyrillic |
| Chinese | GBK, gb18030, Big5 |
| Japanese | EUC-JP, ISO-2022-JP, Shift_JIS |
| Korean | EUC-KR |
| Other | replacement, UTF-16BE, UTF-16LE, x-user-defined |
Plus EBCDIC, which the standard does not define, as a separate type:
cp037 (US/Canada), cp500 (International), cp875 (Greek) and cp1026
(Turkish). See EBCDIC below for why it is kept apart.
And two Korean encodings the standard leaves out, as another separate type: Wansung, which is KS X 1001 in EUC form without code page 949's additions, and Johab, KS X 1001 Annex 3 (code page 1361). See Korean outside the standard.
And, as a third separate type, charsets as glibc 2.44's iconv defines
them, matched byte for byte, for a program that has to convert the way
something built on glibc does. See glibc's charsets.
That the list is closed is the standard's doing and is deliberate. A label the
server and the client resolve differently is an attack rather than a feature,
which is also why the labels for ISO-2022-KR, ISO-2022-CN and HZ-GB-2312 all
name replacement, an encoding whose entire output is one U+FFFD.
Several of these are not what their name says. iso-8859-1, latin1,
ascii and us-ascii are all labels for windows-1252, because content
labelled that way is windows-1252 often enough that decoding it as Latin-1
gets the wrong answer more often than the right one. EUC-KR is really
Windows code page 949, Big5 includes the Hong Kong Supplementary Character
Set, and Shift_JIS is code page 932. The standard documents each of these,
and this library follows it rather than the registered definitions; see
differences from other implementations.
Using it
zig fetch --save git+https://git.jcollie.dev/jeff/zig-charset.git
// build.zig
const charset = b.dependency("charset", .{});
exe.root_module.addImport("charset", charset.module("charset"));
It needs Zig 0.17.0 and has no dependencies of its own. The last release for
Zig 0.16.0 is the v0.1.0 tag, and fixes for it go on the zig-0.16 branch.
One-shot
const utf8 = try charset.decodeAlloc(gpa, .shift_jis, bytes, .{});
const bytes = try charset.encodeAlloc(gpa, .windows_1252, "café", .{});
charset.decodeBomAlloc is the same but lets a leading byte order mark
override the encoding you pass, which is what HTML and XML both want; the mark
itself is not part of the result.
Streaming
Text off a socket does not arrive on character boundaries. Decoder and
Encoder keep the state a character split across two chunks needs — a lead
byte, a half-read escape sequence, a leading surrogate — so that it decodes as
itself rather than as two replacement characters.
var decoder: charset.Decoder = .init(.euc_jp, .{});
while (try readChunk()) |chunk| try decoder.decode(chunk, writer);
try decoder.finish(writer);
finish is not optional. A sequence left half-read at the end of the text is
an error, and nothing else can tell that apart from a sequence whose remaining
bytes have not arrived yet. For the encoder it is what returns ISO-2022-JP to
ASCII mode before the text ends.
When the bytes do not mean anything
Decoding does not fail by default: a byte sequence that means nothing becomes
U+FFFD, which is what leaves a document with one bad byte in it still a
document. Pass .{ .error_mode = .fatal } when the input is a checked format
rather than prose, and you get error.InvalidByteSequence instead.
Encoding fails by default, with error.UnmappableCodePoint and the offending
scalar value in Encoder.unmappable — most of these encodings can write only
a few thousand characters, and losing the rest silently is worse than
stopping. .{ .error_mode = .html } is the other choice the standard defines:
it writes 中 and carries on, which HTML form submission requires and
which cannot be told apart from those characters typed literally. Use UTF-8
and the question does not arise.
Three encodings have no encoder at all — replacement, UTF-16BE and
UTF-16LE — so Encoder.init returns error.NoEncoder for them.
Encoding.outputEncoding maps those three to UTF-8 and is what URL parsing
and form submission are meant to use; Encoder.initOutput applies it for you
and cannot fail.
EBCDIC
EBCDIC is not in the Encoding Standard, and it is not a member of Encoding.
It has a type of its own:
const utf8 = try charset.decodeEbcdicAlloc(gpa, .cp037, bytes, .{});
const bytes = try charset.encodeEbcdicAlloc(gpa, .cp037, "HELLO", .{});
// Or streaming, through the same Decoder and Encoder as everything else.
var decoder: charset.Decoder = .initEbcdic(.cp500, .{});
Keeping it out of Encoding is the point rather than an inconvenience.
Encoding.fromLabel refuses everything outside the standard, and that is what
stops a Content-Type header choosing an encoding nobody vetted; there is no
fromLabel for EBCDIC at all, so it can only be reached by naming a code page
in code. Everything below the type is shared — the same decoder and encoder,
the same error modes, the same streaming behaviour, the same table generator.
Two things about EBCDIC catch people out. It shares nothing with ASCII, not
even the letters: A–I, J–R and S–Z sit in three runs with gaps
between them, a hole punched by the card codes it grew out of, so code that
arranges bytes by arithmetic gets it wrong. And every one of the 256 positions
is assigned in these four pages, so decoding cannot fail — Microsoft's tables
spell an unassigned position as U+001A SUBSTITUTE rather than leaving it out.
Adding a code page is a mapping file in data/ and a line in the generator,
provided the mapping comes from somewhere citable.
Korean outside the standard
The standard's EUC-KR is Windows code page 949: KS X 1001 plus the Unified
Hangul Code, which fits the 8,822 syllables KS X 1001 left out into byte
ranges EUC leaves free. That is what the web sends. The two older ways of
writing Korean have a type of their own, charset.Korean, kept out of
Encoding for the same reason EBCDIC is:
const utf8 = try charset.decodeKoreanAlloc(gpa, .johab, bytes, .{});
const bytes = try charset.encodeKoreanAlloc(gpa, .wansung, "한글", .{});
var decoder: charset.Decoder = .initKorean(.johab, .{});
- Wansung (완성형, "precomposed") is KS X 1001 in EUC form and nothing
else: both bytes
A1toFE. It has codes for 2,350 Hangul syllables. A syllable outside them, such as 똠, is written as eight bytes, the way KS X 1001 defines: the Hangul fillerA4 D4, then the initial consonant, vowel and final consonant from row 4, with the filler again for no final. So 똠 isA4 D4 A4 A8 A4 C7 A4 B1. A code page 949 pair decodes as an error. - Johab (조합형, "combining") spells Hangul out rather than looking it up: a one bit, then five bits each for the initial consonant, the vowel and the final consonant. So all 11,172 syllables have a code. KS X 1001's symbols and Hanja are moved into the remaining byte ranges by arithmetic.
Neither needs a table of its own. KS X 1001 is the part of the standard's
euc-kr index whose bytes are both A1 to FE, including the euro and
registered signs the 1998 revision added, and both encodings read it from
there.
Johab's jamo deserve a note. Unicode's Compatibility Jamo has one code point
per letter, but Johab has a code per position: ㄱ can be written as an
initial or as a final. This follows Unicode.org's JOHAB.TXT, which gives
each letter one code. A consonant that can begin a syllable is an initial,
and the eleven clusters that cannot (ㄳ, ㄵ, ㄶ, ㄺ to ㅀ, ㅄ) are finals. The
other spellings decode as errors, so whatever decodes also encodes back to
the same bytes.
glibc's charsets
The same name often means different bytes to the Encoding Standard and to
glibc. The standard's iso-8859-1 label is windows-1252, and glibc's
ISO-8859-1 is Latin-1. The standard's EUC-KR is code page 949, and
glibc's is KS X 1001 alone. A program that has to give the answers something
built on glibc gives needs glibc's meaning: zig-git, for example, has to show
a commit the way git does, and git converts with glibc's iconv. That is
charset.Glibc, a type of its own like EBCDIC and the Korean encodings:
const from = switch (charset.Glibc.lookup(header_value)) {
.supported => |c| c,
.unsupported => return error.NotReproducible, // glibc knows it; this library does not
.unknown => return showUnconverted(), // glibc would refuse the name
};
const out = try charset.convertGlibcAlloc(gpa, from, .utf_8, bytes);
defer gpa.free(out);
Glibc.lookup resolves a name the way glibc 2.44's iconv_open does, and
knows all 1,180 of glibc's names, so it can tell a name glibc accepts from one
it would refuse even for the charsets this library does not have. It reads
names as glibc does:
- Case does not matter.
- Anything other than a letter, a digit or
_ - . , : /is ignored wherever it is, soutf 8isUTF8. - Trailing slashes, commas and white space are ignored.
- After a second slash come suffixes, separated by
/or,.
A name that turns on //IGNORE or //TRANSLIT is .unsupported. So is a
name that comes down to nothing, which glibc takes to mean the charset of the
caller's locale. Names are resolved only here: Encoding.fromLabel never
sees one.
convertGlibcAlloc converts all of the input or none of it. It returns the
bytes glibc would have written, or fails on exactly the inputs glibc fails on
(InvalidByteSequence or UnmappableCodePoint). Running out of memory is a
separate error. decodeGlibcAlloc, encodeGlibcAlloc, Decoder.initGlibc
and Encoder.initGlibc are there too, and with .fatal they agree with
glibc. With .replacement or .html they behave as the rest of the library
does, which is not what glibc does: glibc never writes U+FFFD.
These are the charsets, by glibc's names for them. Each answers to all of glibc's aliases too:
| Unicode | UTF-8, UTF-16, UTF-16LE, UTF-16BE, UTF-32, UTF-32LE, UTF-32BE, UCS-4LE |
| UCS-2 and UCS-4 | ISO-10646/UCS2 (also UCS-2), UNICODE, UNICODELITTLE (also UCS-2LE), UNICODEBIG (also UCS-2BE), ISO-10646/UCS4 (also UCS-4 and UCS-4BE) |
| ASCII and ISO 8859 | ANSI_X3.4-1968, ISO-8859-1 through -11 and -13 through -16, TIS-620 |
| Windows | CP1250 to CP1258, IBM874 (code page 874) |
| Cyrillic and Mac | KOI8-R, KOI8-U, KOI8-RU, IBM866, MACINTOSH, MAC-UK (also MAC-CYRILLIC), CP10007 |
| EBCDIC | IBM037, IBM500, IBM875, IBM1026 |
| Korean | UHC (code page 949), EUC-KR, JOHAB |
| Japanese | CP932 (also WINDOWS-31J), EUC-JP (also UJIS), ISO-2022-JP, SJIS (also SHIFT_JIS) |
| Chinese | GBK (also CP936), EUC-CN (also GB2312), GB18030, BIG5 (also CP950) |
A charset is on that list only if tools/crosscheck_glibc.py finds no
difference from glibc at all, over every byte, every pair of bytes where the
charset has pairs, and every Unicode scalar value, and over texts of several
characters encoded whole. The charsets in units of two or four bytes also get
inputs made to find their edges: byte order marks, surrogates, values either
side of every limit, and units cut short. ISO-2022-JP gets every escape
sequence of three bytes, in ASCII mode and in JIS X 0208 mode, EUC-JP gets
8F followed by every pair of bytes, GB18030 gets every four-byte sequence
below the supplementary planes, and CP1255 and
CP1258 get every byte followed by every two of their combining marks. The rest
are .unsupported until they get there. That includes glibc's
BIG5HKSCS, EUC-TW, ISO-2022-JP-2, ISO-2022-JP-3, EUC-JISX0213 and
WCHAR_T.
Matching byte for byte means matching glibc's oddities too:
- Every one of these encoders except those for UTF-8, UTF-16, UTF-32, UCS-4 and ISO-2022-JP writes nothing for the tag characters U+E0000 to U+E007F, because glibc's do. ISO-2022-JP fails on them.
UTF-16,UTF-32andUNICODEwrite a byte order mark before the first character, and nothing at all for an empty text. Reading them, a mark as the first unit, in either byte order, chooses the order and is dropped; a second mark is the character U+FEFF. The other charsets in this family never treat a mark as anything but that character.- Some of them are in the machine's byte order, as glibc's are: the
unmarked texts
UTF-16,UTF-32andUNICODEread and what they write after the mark, andISO-10646/UCS2throughout. On the usual x86-64 and AArch64 machines that is little-endian, soUCS-2isUCS-2LEthere, whileUCS-4is always big-endian. - An unpaired surrogate in UTF-16, any surrogate in UCS-2, and a unit cut short at the end are all errors. UCS-2 cannot write anything above U+FFFF.
There is one known difference, text that is outside Unicode. glibc's
UTF-8 decoder still reads the original UTF-8 of RFC 2279: values up to
0x7FFFFFFF, in sequences of up to six bytes. Its UCS-4 decoders read the same
values, and surrogates too. Converting such a value into most of these
charsets fails in glibc as well, but into UTF-8 or UCS-4 it can succeed, and
this library cannot represent it. So convertGlibcAlloc reports it as
error.OutsideUnicode, separately from bytes that are invalid everywhere.
A UCS-4 surrogate is the exception: glibc writes one only as UCS-4, so into
any other charset, UTF-8 included, it is error.InvalidByteSequence, as it is
an error in glibc.
There is a second difference, between some pairs of Chinese charsets.
convertGlibcAlloc converts every pair through Unicode. glibc does too,
except for three pairs it converts directly, and for those it gives
different answers:
EUC-CNtoBIG5turns simplified characters into traditional ones, so 张 comes out as the Big5 for 張.EUC-CNtoGBKfails when the text ends in a two-byte character.GBKtoEUC-CNcopiesA1A4andA1AAthrough unchanged, thoughEUC-CNreads them as other characters, and fails onA844.
For these pairs, this library gives the answer that glibc gives when the
text goes through UTF-8 in two separate conversions. glibc's direct
conversion of the same text answers differently. Matching
EUC-CN to BIG5 would take glibc's own simplified-to-traditional table,
and no published table matches it.
Glibc.utf_8 is what every one of glibc's names for UTF-8 resolves to, so a
caller can tell whether the target was UTF-8.
Differences from other implementations
These are the standard's departures from the registered definitions, not this library's. Each is there because deployed content needed it.
- windows-125x and windows-874 fill the positions Microsoft never assigned with the C1 control of the same number, rather than rejecting them. There are a handful in each of the windows-125x pages and two dozen in windows-874.
- KOI8-U takes ten positions from KOI8-R where RFC 2319 takes
eight. The eight are the Ukrainian letters є і ї ґ and their capitals; the
extra two are
AEandBE, which the standard reads as ў and Ў — KOI8-RU's Belarusian short U — and RFC 2319 leaves as the box-drawing characters ╝ and ╬. CPython'skoi8_ufollows the RFC, so those two positions are the whole of the difference. - Shift_JIS decodes pointers 8836 to 10715 into the private use area (the
Windows end-user defined characters) and refuses to encode them again. Its
encoder skips pointers 8272 to 8835, so U+2170 comes out as
FA 40rather than theEE EFthat some implementations produce. - Big5 decodes the Hong Kong extensions but will not write them, and takes the last pointer for six code points where everything else takes the first. Four pointers decode to two code points each, a letter plus a combining mark.
- GBK writes the euro sign as
80and will not use the four-byte form; gb18030 does the opposite. gb18030's encoder keeps eighteen private use characters where GB18030-2005 had them, so those do not round trip, and U+E5E5 cannot be encoded at all. - Shift_JIS, EUC-JP and ISO-2022-JP write U+00A5 as
5Cand U+203E as7E, which is why a path written on a Japanese system comes back with a yen sign where the backslash was. - EUC-JP reads JIS X 0212 and will not write it.
The rest are not the standard's doing, because these encodings are not in it.
- Johab follows
JOHAB.TXTexcept at5C, which is the backslash rather than the won sign (the file's own notes say the backslash "might be a better idea", and Windows and CPython both use it), and atD9 E6andD9 E7, the euro and registered signs, which the file predates. Against CPython the encoders agree on every scalar value. CPython's decoder also accepts the sixteen spellings of a jamo as a final thatJOHAB.TXTleaves out (84 42for ㄱ, and so on), and reads84 41, all three fields filled, as U+3000. Both are errors here. - Wansung agrees with CPython's
euc_kron every scalar value, and on which eight-byte sequences are syllables and which syllable each one is. The two part company overA4 D4that does not begin a syllable. Here it is U+3164 HANGUL FILLER, which is what KS X 1001 has at that position, and whatever followed it is decoded afresh. CPython reports an error for theA4alone and carries on from theD4, which puts every pair after it out of step:A4 D4 A4 A3 A4 BF A4 D4, a sequence that fails because ㄳ cannot begin a syllable, is filler, ㄳ, ㅏ, filler here and<EFBFBD>渡$엘<EFBFBD>there. - CP875 encodes U+001A to
3Fwhere CPython givesFD. Microsoft's table spells six unassigned positions as SUBSTITUTE alongside the real one, and3Fis the position every other EBCDIC page keeps SUBSTITUTE in — including, in CPython,cp037,cp500andcp1026.
The html encoder error mode also departs from the letter of the standard for
EBCDIC alone. The standard says to emit the bytes 0x26 0x23 … 0x3B, which
works because every encoding it defines spells ASCII as ASCII; EBCDIC does
not, so what is written is those characters through the encoder. For the
standard's own encodings the two are the same bytes.
glibc's charsets are each built from a table this library already has, by a rule or a short list of exceptions observed from glibc. These are the places where glibc departs from the source it starts from:
CP1250toCP1258andIBM874leave unassigned the positions Microsoft never assigned, where the standard fills them with C1 controls.CP1255leavesCAunassigned too.CP1255andCP1258compose. Reading, a character followed by one of the code page's combining marks is read as the single character Unicode has for the two, if there is one: in CP1258AandCCare À, and in CP1255 ב and dagesh are U+FB31. The test is that the character's full canonical decomposition and the mark hold the same characters as the result's, in any order, so Ó and a tilde make Ṍ although Unicode spells that with the tilde first. In CP1255 a further mark goes on composing; in CP1258 only a character read from a byte of its own takes a mark. Writing, a character with no byte of its own is spelt as a base and a mark, the shortest way it can be, so Ṍ isD3 DE. CP1258 also writes U+0340 and U+0341 as the grave and acute accents, and U+1FEE asA8 EC, which reads back as U+0385.ISO-8859-9andISO-8859-11are windows-1254 and windows-874 with80to9Fas the C1 controls.TIS-620is ISO-8859-11 with nothing at80toA0.KOI8-Ufollows RFC 2319 atAEandBE(╝ and ╬).KOI8-RUis the standard's KOI8-U with nine of93to9Fchanged to typographic symbols:93“,96”,97—,98№,99™,9B»,9C®,9D« and9F¤.MACINTOSHreadsC6as Δ U+0394 rather than ∆ U+2206, and the Apple logo atF0as U+E01E rather than U+F8FF.MAC-UKhas ¤ atFFwhere the standard has €, andCP10007has that and ¢ atA2.IBM875has ∇ at74and · atDD, and leaves6Aand the six extra SUBSTITUTE positions unassigned.IBM1026has ˛ at9Dand — atBC.CP932refuses80. It writes the end-user defined characters U+E000 to U+E757 back toF040toF9FC, and writes six characters one way: ¢, £ and ¬ as their fullwidth forms, and U+2014, U+2016 and U+301C as815C,8161and8160.EUC-KRis KS X 1001 alone, without Wansung's eight-byte syllables. It reads80to9Fas the C1 controls, has KS X 1001:2002's postal code mark ㉾ atA2E8, and writes U+20A9 WON SIGN asA3DC, the fullwidth won sign.JOHABreads5Cas the won sign, asJOHAB.TXTdoes, so the backslash cannot be written. It also has the postal code mark, atD9E8.ISO-2022-JPis RFC 1468's. Its JIS X 0208 has none of code page 932's rows 13 and 89 to 92, and reads2141,2142and215Das 〜 U+301C, ‖ U+2016 and − U+2212, and2171,2172and224Cas ¢, £ and ¬, where the standard has the fullwidth forms. It has no halfwidth katakana, soESC ( Iis not a designation and halfwidth katakana cannot be written. An ESC that does not begin one of the four designations is the character U+001B, a control byte in JIS X 0208 mode is that ASCII character, and designating the mode already in force is not an error. Writing, it stays in JIS X 0201 Roman for the printable characters Roman shares with ASCII, and leaves JIS X 0208 for anything else, a space or a line break included.EUC-JPhas the same JIS X 0208 asISO-2022-JP, and writes JIS X 0212 as well as reading it, where the standard only reads it.80to9F, apart from8Eand8F, are the C1 controls. ¥ and ‾ are written as5Cand7E, which read back as\and~.SJISis Shift_JIS as JIS X 0208 defines it, not code page 932: its single bytes are JIS X 0201, so5Cand7Eread as ¥ and ‾, and its two-byte characters are the same JIS X 0208 asEUC-JP's, with nothing fromF0up. The backslash and the tilde are still written as5Cand7E, and the fullwidth ¢, £ and ¬ as the cells for ¢, £ and ¬.GBKis the standard's gb18030 two-byte table without what GB 18030 added to GBK and without the Private Use Area. Every cell the standard maps to U+E000–F8FF is unassigned, and so are the euro sign atA2E3,A3A0, the vertical forms in rowA6, ḿ and ǹ atA8BCandA8BF,A989toA995, andFE50toFEA0. The euro sign is the single byte80, and there are no four-byte sequences.EUC-CN(GB2312) is that GBK cut down to GB 2312's square,A1A1toFEFE, with no single bytes above7F. It also drops the cells GBK added inside the square — the small roman numerals atA2A1toA2AA,A6D9toA6F5andA8BBtoA8C0— and readsA1A4as U+30FB andA1AAas U+2015, where GBK has U+00B7 and U+2014.GB18030is the standard's gb18030 with three differences, each both ways, so that every scalar value reads back as itself.80is an error rather than the euro sign.A3A0is U+E5E5 rather than U+3000. And the eighteen private use code points GB18030-2022 moved are four-byte sequences —82359037to82359134and84318236to84318335— where the standard's encoder writes them as two bytes and its decoder reads those four-byte sequences as U+9FB4–9FBB and U+FE10–FE19.BIG5is Big5 itself, not the standard's Big5-HKSCS: lead bytesA1toF9only, nothing atA3C0toA3E0, the ETEN extensions atC6A1toC8FEas the Private Use Area from U+F6B1,F9FEas ▓ U+2593 rather than U+FFED, and80as U+0080. A character it has twice is written as its first cell, except 十 and 卅, which go toA451andA4CA.
tools/crosscheck.py walks every input of every encoding against CPython's
codecs module and prints what differs; the list above is what it finds, and
anything else appearing there would be a bug.
It also prints what it skipped, which is utf-16le, utf-16be and
replacement asked to encode. The standard makes those three decode-only —
Encoding.canEncode is this library's answer about which — so there is
nothing to compare, and the run says so rather than failing.
The tables
src/tables/ is generated from the index files the standard publishes — and,
for EBCDIC, from Microsoft's mapping tables as Unicode.org publishes them, and
for composing, from the canonical decompositions in the Unicode Character
Database's UnicodeData.txt — which are committed in data/ so that a build needs no network and so that a
change upstream shows up in review as a diff to its input rather than as an
unexplained change to a mapping.
nix develop -c zig build gen-tables # then read `git diff`
The workflow regenerates and checks for a dirty tree, so an index that was updated without the tables being remade fails the build.
data/glibc-names.txt is the exception: it is not published anywhere, so
tools/glibc_names.py writes it. It takes every name iconv -l lists, groups
them by the alias and module lines of glibc's gconv-modules files, and has
glibc confirm the groups of the few names it builds in. It also writes
tests/glibc-name-cases.txt, several thousand ways of writing those names,
each with the verdict glibc gives.
nix develop -c python3 tools/glibc_names.py # rewrite both
nix develop -c python3 tools/glibc_names.py --check # what CI runs
Testing
nix develop -c zig build test # unit and conformance tests
nix develop -c python3 tools/crosscheck.py # against CPython's codecs
nix develop -c python3 tools/crosscheck_glibc.py # against glibc's iconv
nix develop -c zig build fuzz-run # our own fuzzing loop
nix develop -c zig build fuzz --fuzz # Zig's fuzzer, until interrupted
The conformance tests are spot checks taken from the standard's worked examples and from arithmetic done by hand against the algorithms — a test whose expected value was read out of the same table the code reads proves nothing.
The fuzz targets check three properties: decoding cannot crash and cannot
produce invalid UTF-8; where a chunk boundary falls does not change the
answer; and text that has already been through an encoding once comes out of
it unchanged the second time. zig build fuzz --fuzz hands them to Zig's
fuzzer, with coverage feedback, until it is interrupted; zig build fuzz-run
is a loop of our own over a corpus that stops after a fixed number of inputs,
which is what CI runs. See tools/fuzz.zig.
The glibc tools run only on glibc 2.44, which is what the devshell provides,
and refuse any other version. A nixpkgs update that moves glibc therefore
fails CI. Moving to a new glibc means running both tools again, putting the
new version in the README, and taking out of Glibc any charset that no
longer matches.
Where this lives
- Forgejo: https://git.jcollie.dev/jeff/zig-charset
- Tangled: https://tangled.org/jcollie.dev/zig-charset
- Radicle:
rad:z2SapCVyHHcNssHjcrBvpA2XXuQnW
git clone https://git.jcollie.dev/jeff/zig-charset.git
rad clone rad:z2SapCVyHHcNssHjcrBvpA2XXuQnW
A Radicle repository is findable only by its ID, so that string is the one thing a reader needs in order to seed or clone it.
Report issues and send pull requests on the Tangled mirror,
https://tangled.org/jcollie.dev/zig-charset. CONTRIBUTING.md covers what
a change needs to pass.
References cited
- van Kesteren, A. (2026). Encoding Standard (Living Standard, 21 May 2026). WHATWG. https://encoding.spec.whatwg.org/
- Unicode Consortium. EBCDIC code page mapping tables
(
MAPPINGS/VENDORS/MICSFT/EBCDIC). https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/ - IBM. EBCDIC to ASCII conversion table (InfoSphere Information Server 11.3.0). https://www.ibm.com/docs/en/iis/11.3.0?topic=tables-ebcdic-ascii
- Shin, J. (2011). Johab to Unicode table (
JOHAB.TXT, version 1.1). Unicode Consortium. https://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/KSC/JOHAB.TXT - Microsoft. Windows best fit table for code page 1361
(
bestfit1361.txt). Unicode Consortium. https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/bestfit1361.txt - Choi, U., Chon, K., & Park, H. (1993). Korean Character Encoding for Internet Messages (RFC 1557). RFC Editor. https://www.rfc-editor.org/info/rfc1557
- Yergeau, F. (1998). UTF-8, a transformation format of ISO 10646 (RFC 2279). RFC Editor. https://www.rfc-editor.org/info/rfc2279
- KOI8-U Working Group. (1998). Ukrainian Character Set KOI8-U (RFC 2319). RFC Editor. https://www.rfc-editor.org/info/rfc2319
- Murai, J., Crispin, M., & van der Poel, E. (1993). Japanese Character Encoding for Internet Messages (RFC 1468). RFC Editor. https://www.rfc-editor.org/info/rfc1468
- Unicode Consortium. (2026). Unicode Character Database (Version 18.0.0;
UnicodeData.txt). https://www.unicode.org/Public/18.0.0/ucd/ - Whistler, K. (Ed.). (2026). Unicode Normalization Forms (Unicode Standard Annex #15, Revision 58). Unicode Consortium. https://www.unicode.org/reports/tr15/tr15-58.html
- Free Software Foundation. (2026). The GNU C Library (Version 2.44). https://www.gnu.org/software/libc/
The first is the whole specification this implements: the encodings and their
labels, the decoder and encoder algorithm for each, and the index files that
data/ is a copy of. The second is where the four EBCDIC tables come from.
The third was consulted and deliberately not used. It does not say which code
page it is, it states that the translation is not bidirectional, and it
collapses everything unmappable to 0x1A, so nothing built on it could round
trip.
JOHAB.TXT is what Johab is checked against; see differences from other
implementations for the three
positions where this library departs from it. Windows' own table for code page
1361 was consulted and not followed. It decodes a standalone jamo to the
conjoining Hangul Jamo block rather than the Compatibility Jamo that KS X 1001
itself uses, and a few of its entries disagree with the arithmetic Annex 3
defines. RFC 1557 is the EUC-KR that Wansung is: KS X 1001 in EUC form, with
nothing added.
RFC 1468 is the ISO-2022-JP that glibc's is. The Unicode Character Database's
canonical decompositions, which data/UnicodeData.txt is a copy of, are what
CP1255 and CP1258 compose by, and UAX #15 defines canonical decomposition.
RFC 2279 is the original UTF-8, which glibc's decoder still reads and which
is where outside Unicode comes from. RFC 2319 is the
KOI8-U that glibc's is. glibc 2.44 is the implementation Glibc reproduces.
It is a test oracle and the source of the list of names, and never the source
of a mapping: see docs/adr/0001-glibc-charsets.md.
Licensing
MIT, and the project follows REUSE —
nix develop -c reuse lint checks it.
The index files under data/ are the WHATWG's, under the CC-BY 4.0 that
covers the Encoding Standard. The tables generated from them are
MIT AND BSD-3-Clause, the standard's own terms for portions of it
incorporated into source code. The EBCDIC mapping files and the table
generated from them carry the Unicode licence. data/glibc-names.txt and the
table generated from it are a list of glibc's names and which charset each
belongs to. That is facts about glibc rather than any part of it, and they
are MIT like the rest.