Converting between UTF-8 and the legacy character encodings, as the WHATWG Encoding Standard defines them.
  • Zig 98.1%
  • Python 1.8%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Jeffrey C. Ollie d266c5516a
All checks were successful
test / test (push) Successful in 28m52s
test / docs (push) Successful in 4m59s
Document the charset pairs glibc converts directly
glibc converts EUC-CN to BIG5, EUC-CN to GBK and GBK to EUC-CN directly
rather than through UCS-4, and gets answers of its own. convertGlibcAlloc
goes through Unicode for every pair, so say so rather than claiming it
always gives glibc's answer.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01P9FTtWx5YQhHu7irRjDsKM
2026-11-03 20:20:52 -06:00
.forgejo/workflows Add glibc's charsets, matched byte for byte; version 0.4.0 2026-11-02 22:34:43 -06:00
data Add glibc's CP1255 and CP1258, which compose; version 0.7.0 2026-11-03 13:46:29 -06:00
docs/adr Add glibc's UTF-16, UTF-32, UCS-2 and UCS-4; version 0.5.0 2026-11-03 10:56:28 -06:00
LICENSES Add EBCDIC, as a type of its own rather than an Encoding 2026-09-15 19:34:53 -05:00
src Document the charset pairs glibc converts directly 2026-11-03 20:20:52 -06:00
tests Add glibc's GB2312, BIG5 and GB18030; version 0.11.0 2026-11-03 15:44:57 -06:00
tools Add glibc's GB2312, BIG5 and GB18030; version 0.11.0 2026-11-03 15:44:57 -06:00
.gitignore Remove the Nix package 2026-10-03 01:22:15 -05:00
build.zig Move to Zig 0.17.0 2026-10-03 01:20:10 -05:00
build.zig.zon Add glibc's GB2312, BIG5 and GB18030; version 0.11.0 2026-11-03 15:44:57 -06:00
CONTRIBUTING.md Add glibc's charsets, matched byte for byte; version 0.4.0 2026-11-02 22:34:43 -06:00
flake.lock Add glibc's charsets, matched byte for byte; version 0.4.0 2026-11-02 22:34:43 -06:00
flake.nix Add glibc's charsets, matched byte for byte; version 0.4.0 2026-11-02 22:34:43 -06:00
GLOSSARY.md Refuse a UCS-4 surrogate going to UTF-8, as glibc does; version 0.5.1 2026-11-03 12:24:32 -06:00
README.md Document the charset pairs glibc converts directly 2026-11-03 20:20:52 -06:00
REUSE.toml Add glibc's CP1255 and CP1258, which compose; version 0.7.0 2026-11-03 13:46:29 -06:00

zig-charset

Converting between UTF-8 and the legacy character encodings — windows-1252, Shift_JIS, Big5, gb18030 and the rest — as the WHATWG Encoding Standard defines them.

The API documentation is generated from the doc comments and published at https://jeff.jcollie.page/zig-charset/; zig build docs-serve reads the same thing locally.

const charset = @import("charset");

// A label off a Content-Type header, if it names an encoding at all.
const encoding = charset.Encoding.fromLabel("Shift-JIS") orelse return error.UnknownCharset;

const utf8 = try charset.decodeAlloc(gpa, encoding, bytes, .{});
defer gpa.free(utf8);

What is in it

The forty encodings the standard defines, and no others:

UTF-8 UTF-8
Single-byte IBM866, ISO-8859-2 through -16, ISO-8859-8-I, KOI8-R, KOI8-U, macintosh, windows-874, windows-1250 through -1258, x-mac-cyrillic
Chinese GBK, gb18030, Big5
Japanese EUC-JP, ISO-2022-JP, Shift_JIS
Korean EUC-KR
Other replacement, UTF-16BE, UTF-16LE, x-user-defined

Plus EBCDIC, which the standard does not define, as a separate type: cp037 (US/Canada), cp500 (International), cp875 (Greek) and cp1026 (Turkish). See EBCDIC below for why it is kept apart.

And two Korean encodings the standard leaves out, as another separate type: Wansung, which is KS X 1001 in EUC form without code page 949's additions, and Johab, KS X 1001 Annex 3 (code page 1361). See Korean outside the standard.

And, as a third separate type, charsets as glibc 2.44's iconv defines them, matched byte for byte, for a program that has to convert the way something built on glibc does. See glibc's charsets.

That the list is closed is the standard's doing and is deliberate. A label the server and the client resolve differently is an attack rather than a feature, which is also why the labels for ISO-2022-KR, ISO-2022-CN and HZ-GB-2312 all name replacement, an encoding whose entire output is one U+FFFD.

Several of these are not what their name says. iso-8859-1, latin1, ascii and us-ascii are all labels for windows-1252, because content labelled that way is windows-1252 often enough that decoding it as Latin-1 gets the wrong answer more often than the right one. EUC-KR is really Windows code page 949, Big5 includes the Hong Kong Supplementary Character Set, and Shift_JIS is code page 932. The standard documents each of these, and this library follows it rather than the registered definitions; see differences from other implementations.

Using it

zig fetch --save git+https://git.jcollie.dev/jeff/zig-charset.git
// build.zig
const charset = b.dependency("charset", .{});
exe.root_module.addImport("charset", charset.module("charset"));

It needs Zig 0.17.0 and has no dependencies of its own. The last release for Zig 0.16.0 is the v0.1.0 tag, and fixes for it go on the zig-0.16 branch.

One-shot

const utf8 = try charset.decodeAlloc(gpa, .shift_jis, bytes, .{});
const bytes = try charset.encodeAlloc(gpa, .windows_1252, "café", .{});

charset.decodeBomAlloc is the same but lets a leading byte order mark override the encoding you pass, which is what HTML and XML both want; the mark itself is not part of the result.

Streaming

Text off a socket does not arrive on character boundaries. Decoder and Encoder keep the state a character split across two chunks needs — a lead byte, a half-read escape sequence, a leading surrogate — so that it decodes as itself rather than as two replacement characters.

var decoder: charset.Decoder = .init(.euc_jp, .{});
while (try readChunk()) |chunk| try decoder.decode(chunk, writer);
try decoder.finish(writer);

finish is not optional. A sequence left half-read at the end of the text is an error, and nothing else can tell that apart from a sequence whose remaining bytes have not arrived yet. For the encoder it is what returns ISO-2022-JP to ASCII mode before the text ends.

When the bytes do not mean anything

Decoding does not fail by default: a byte sequence that means nothing becomes U+FFFD, which is what leaves a document with one bad byte in it still a document. Pass .{ .error_mode = .fatal } when the input is a checked format rather than prose, and you get error.InvalidByteSequence instead.

Encoding fails by default, with error.UnmappableCodePoint and the offending scalar value in Encoder.unmappable — most of these encodings can write only a few thousand characters, and losing the rest silently is worse than stopping. .{ .error_mode = .html } is the other choice the standard defines: it writes &#20013; and carries on, which HTML form submission requires and which cannot be told apart from those characters typed literally. Use UTF-8 and the question does not arise.

Three encodings have no encoder at all — replacement, UTF-16BE and UTF-16LE — so Encoder.init returns error.NoEncoder for them. Encoding.outputEncoding maps those three to UTF-8 and is what URL parsing and form submission are meant to use; Encoder.initOutput applies it for you and cannot fail.

EBCDIC

EBCDIC is not in the Encoding Standard, and it is not a member of Encoding. It has a type of its own:

const utf8 = try charset.decodeEbcdicAlloc(gpa, .cp037, bytes, .{});
const bytes = try charset.encodeEbcdicAlloc(gpa, .cp037, "HELLO", .{});

// Or streaming, through the same Decoder and Encoder as everything else.
var decoder: charset.Decoder = .initEbcdic(.cp500, .{});

Keeping it out of Encoding is the point rather than an inconvenience. Encoding.fromLabel refuses everything outside the standard, and that is what stops a Content-Type header choosing an encoding nobody vetted; there is no fromLabel for EBCDIC at all, so it can only be reached by naming a code page in code. Everything below the type is shared — the same decoder and encoder, the same error modes, the same streaming behaviour, the same table generator.

Two things about EBCDIC catch people out. It shares nothing with ASCII, not even the letters: A–I, J–R and S–Z sit in three runs with gaps between them, a hole punched by the card codes it grew out of, so code that arranges bytes by arithmetic gets it wrong. And every one of the 256 positions is assigned in these four pages, so decoding cannot fail — Microsoft's tables spell an unassigned position as U+001A SUBSTITUTE rather than leaving it out.

Adding a code page is a mapping file in data/ and a line in the generator, provided the mapping comes from somewhere citable.

Korean outside the standard

The standard's EUC-KR is Windows code page 949: KS X 1001 plus the Unified Hangul Code, which fits the 8,822 syllables KS X 1001 left out into byte ranges EUC leaves free. That is what the web sends. The two older ways of writing Korean have a type of their own, charset.Korean, kept out of Encoding for the same reason EBCDIC is:

const utf8 = try charset.decodeKoreanAlloc(gpa, .johab, bytes, .{});
const bytes = try charset.encodeKoreanAlloc(gpa, .wansung, "한글", .{});

var decoder: charset.Decoder = .initKorean(.johab, .{});
  • Wansung (완성형, "precomposed") is KS X 1001 in EUC form and nothing else: both bytes A1 to FE. It has codes for 2,350 Hangul syllables. A syllable outside them, such as 똠, is written as eight bytes, the way KS X 1001 defines: the Hangul filler A4 D4, then the initial consonant, vowel and final consonant from row 4, with the filler again for no final. So 똠 is A4 D4 A4 A8 A4 C7 A4 B1. A code page 949 pair decodes as an error.
  • Johab (조합형, "combining") spells Hangul out rather than looking it up: a one bit, then five bits each for the initial consonant, the vowel and the final consonant. So all 11,172 syllables have a code. KS X 1001's symbols and Hanja are moved into the remaining byte ranges by arithmetic.

Neither needs a table of its own. KS X 1001 is the part of the standard's euc-kr index whose bytes are both A1 to FE, including the euro and registered signs the 1998 revision added, and both encodings read it from there.

Johab's jamo deserve a note. Unicode's Compatibility Jamo has one code point per letter, but Johab has a code per position: ㄱ can be written as an initial or as a final. This follows Unicode.org's JOHAB.TXT, which gives each letter one code. A consonant that can begin a syllable is an initial, and the eleven clusters that cannot (ㄳ, ㄵ, ㄶ, ㄺ to ㅀ, ㅄ) are finals. The other spellings decode as errors, so whatever decodes also encodes back to the same bytes.

glibc's charsets

The same name often means different bytes to the Encoding Standard and to glibc. The standard's iso-8859-1 label is windows-1252, and glibc's ISO-8859-1 is Latin-1. The standard's EUC-KR is code page 949, and glibc's is KS X 1001 alone. A program that has to give the answers something built on glibc gives needs glibc's meaning: zig-git, for example, has to show a commit the way git does, and git converts with glibc's iconv. That is charset.Glibc, a type of its own like EBCDIC and the Korean encodings:

const from = switch (charset.Glibc.lookup(header_value)) {
    .supported => |c| c,
    .unsupported => return error.NotReproducible, // glibc knows it; this library does not
    .unknown => return showUnconverted(),         // glibc would refuse the name
};
const out = try charset.convertGlibcAlloc(gpa, from, .utf_8, bytes);
defer gpa.free(out);

Glibc.lookup resolves a name the way glibc 2.44's iconv_open does, and knows all 1,180 of glibc's names, so it can tell a name glibc accepts from one it would refuse even for the charsets this library does not have. It reads names as glibc does:

  • Case does not matter.
  • Anything other than a letter, a digit or _ - . , : / is ignored wherever it is, so utf 8 is UTF8.
  • Trailing slashes, commas and white space are ignored.
  • After a second slash come suffixes, separated by / or ,.

A name that turns on //IGNORE or //TRANSLIT is .unsupported. So is a name that comes down to nothing, which glibc takes to mean the charset of the caller's locale. Names are resolved only here: Encoding.fromLabel never sees one.

convertGlibcAlloc converts all of the input or none of it. It returns the bytes glibc would have written, or fails on exactly the inputs glibc fails on (InvalidByteSequence or UnmappableCodePoint). Running out of memory is a separate error. decodeGlibcAlloc, encodeGlibcAlloc, Decoder.initGlibc and Encoder.initGlibc are there too, and with .fatal they agree with glibc. With .replacement or .html they behave as the rest of the library does, which is not what glibc does: glibc never writes U+FFFD.

These are the charsets, by glibc's names for them. Each answers to all of glibc's aliases too:

Unicode UTF-8, UTF-16, UTF-16LE, UTF-16BE, UTF-32, UTF-32LE, UTF-32BE, UCS-4LE
UCS-2 and UCS-4 ISO-10646/UCS2 (also UCS-2), UNICODE, UNICODELITTLE (also UCS-2LE), UNICODEBIG (also UCS-2BE), ISO-10646/UCS4 (also UCS-4 and UCS-4BE)
ASCII and ISO 8859 ANSI_X3.4-1968, ISO-8859-1 through -11 and -13 through -16, TIS-620
Windows CP1250 to CP1258, IBM874 (code page 874)
Cyrillic and Mac KOI8-R, KOI8-U, KOI8-RU, IBM866, MACINTOSH, MAC-UK (also MAC-CYRILLIC), CP10007
EBCDIC IBM037, IBM500, IBM875, IBM1026
Korean UHC (code page 949), EUC-KR, JOHAB
Japanese CP932 (also WINDOWS-31J), EUC-JP (also UJIS), ISO-2022-JP, SJIS (also SHIFT_JIS)
Chinese GBK (also CP936), EUC-CN (also GB2312), GB18030, BIG5 (also CP950)

A charset is on that list only if tools/crosscheck_glibc.py finds no difference from glibc at all, over every byte, every pair of bytes where the charset has pairs, and every Unicode scalar value, and over texts of several characters encoded whole. The charsets in units of two or four bytes also get inputs made to find their edges: byte order marks, surrogates, values either side of every limit, and units cut short. ISO-2022-JP gets every escape sequence of three bytes, in ASCII mode and in JIS X 0208 mode, EUC-JP gets 8F followed by every pair of bytes, GB18030 gets every four-byte sequence below the supplementary planes, and CP1255 and CP1258 get every byte followed by every two of their combining marks. The rest are .unsupported until they get there. That includes glibc's BIG5HKSCS, EUC-TW, ISO-2022-JP-2, ISO-2022-JP-3, EUC-JISX0213 and WCHAR_T.

Matching byte for byte means matching glibc's oddities too:

  • Every one of these encoders except those for UTF-8, UTF-16, UTF-32, UCS-4 and ISO-2022-JP writes nothing for the tag characters U+E0000 to U+E007F, because glibc's do. ISO-2022-JP fails on them.
  • UTF-16, UTF-32 and UNICODE write a byte order mark before the first character, and nothing at all for an empty text. Reading them, a mark as the first unit, in either byte order, chooses the order and is dropped; a second mark is the character U+FEFF. The other charsets in this family never treat a mark as anything but that character.
  • Some of them are in the machine's byte order, as glibc's are: the unmarked texts UTF-16, UTF-32 and UNICODE read and what they write after the mark, and ISO-10646/UCS2 throughout. On the usual x86-64 and AArch64 machines that is little-endian, so UCS-2 is UCS-2LE there, while UCS-4 is always big-endian.
  • An unpaired surrogate in UTF-16, any surrogate in UCS-2, and a unit cut short at the end are all errors. UCS-2 cannot write anything above U+FFFF.

There is one known difference, text that is outside Unicode. glibc's UTF-8 decoder still reads the original UTF-8 of RFC 2279: values up to 0x7FFFFFFF, in sequences of up to six bytes. Its UCS-4 decoders read the same values, and surrogates too. Converting such a value into most of these charsets fails in glibc as well, but into UTF-8 or UCS-4 it can succeed, and this library cannot represent it. So convertGlibcAlloc reports it as error.OutsideUnicode, separately from bytes that are invalid everywhere. A UCS-4 surrogate is the exception: glibc writes one only as UCS-4, so into any other charset, UTF-8 included, it is error.InvalidByteSequence, as it is an error in glibc.

There is a second difference, between some pairs of Chinese charsets. convertGlibcAlloc converts every pair through Unicode. glibc does too, except for three pairs it converts directly, and for those it gives different answers:

  • EUC-CN to BIG5 turns simplified characters into traditional ones, so 张 comes out as the Big5 for 張.
  • EUC-CN to GBK fails when the text ends in a two-byte character.
  • GBK to EUC-CN copies A1A4 and A1AA through unchanged, though EUC-CN reads them as other characters, and fails on A844.

For these pairs, this library gives the answer that glibc gives when the text goes through UTF-8 in two separate conversions. glibc's direct conversion of the same text answers differently. Matching EUC-CN to BIG5 would take glibc's own simplified-to-traditional table, and no published table matches it.

Glibc.utf_8 is what every one of glibc's names for UTF-8 resolves to, so a caller can tell whether the target was UTF-8.

Differences from other implementations

These are the standard's departures from the registered definitions, not this library's. Each is there because deployed content needed it.

  • windows-125x and windows-874 fill the positions Microsoft never assigned with the C1 control of the same number, rather than rejecting them. There are a handful in each of the windows-125x pages and two dozen in windows-874.
  • KOI8-U takes ten positions from KOI8-R where RFC 2319 takes eight. The eight are the Ukrainian letters є і ї ґ and their capitals; the extra two are AE and BE, which the standard reads as ў and Ў — KOI8-RU's Belarusian short U — and RFC 2319 leaves as the box-drawing characters ╝ and ╬. CPython's koi8_u follows the RFC, so those two positions are the whole of the difference.
  • Shift_JIS decodes pointers 8836 to 10715 into the private use area (the Windows end-user defined characters) and refuses to encode them again. Its encoder skips pointers 8272 to 8835, so U+2170 comes out as FA 40 rather than the EE EF that some implementations produce.
  • Big5 decodes the Hong Kong extensions but will not write them, and takes the last pointer for six code points where everything else takes the first. Four pointers decode to two code points each, a letter plus a combining mark.
  • GBK writes the euro sign as 80 and will not use the four-byte form; gb18030 does the opposite. gb18030's encoder keeps eighteen private use characters where GB18030-2005 had them, so those do not round trip, and U+E5E5 cannot be encoded at all.
  • Shift_JIS, EUC-JP and ISO-2022-JP write U+00A5 as 5C and U+203E as 7E, which is why a path written on a Japanese system comes back with a yen sign where the backslash was.
  • EUC-JP reads JIS X 0212 and will not write it.

The rest are not the standard's doing, because these encodings are not in it.

  • Johab follows JOHAB.TXT except at 5C, which is the backslash rather than the won sign (the file's own notes say the backslash "might be a better idea", and Windows and CPython both use it), and at D9 E6 and D9 E7, the euro and registered signs, which the file predates. Against CPython the encoders agree on every scalar value. CPython's decoder also accepts the sixteen spellings of a jamo as a final that JOHAB.TXT leaves out (84 42 for ㄱ, and so on), and reads 84 41, all three fields filled, as U+3000. Both are errors here.
  • Wansung agrees with CPython's euc_kr on every scalar value, and on which eight-byte sequences are syllables and which syllable each one is. The two part company over A4 D4 that does not begin a syllable. Here it is U+3164 HANGUL FILLER, which is what KS X 1001 has at that position, and whatever followed it is decoded afresh. CPython reports an error for the A4 alone and carries on from the D4, which puts every pair after it out of step: A4 D4 A4 A3 A4 BF A4 D4, a sequence that fails because ㄳ cannot begin a syllable, is filler, ㄳ, ㅏ, filler here and <EFBFBD>渡$엘<EFBFBD> there.
  • CP875 encodes U+001A to 3F where CPython gives FD. Microsoft's table spells six unassigned positions as SUBSTITUTE alongside the real one, and 3F is the position every other EBCDIC page keeps SUBSTITUTE in — including, in CPython, cp037, cp500 and cp1026.

The html encoder error mode also departs from the letter of the standard for EBCDIC alone. The standard says to emit the bytes 0x26 0x23 … 0x3B, which works because every encoding it defines spells ASCII as ASCII; EBCDIC does not, so what is written is those characters through the encoder. For the standard's own encodings the two are the same bytes.

glibc's charsets are each built from a table this library already has, by a rule or a short list of exceptions observed from glibc. These are the places where glibc departs from the source it starts from:

  • CP1250 to CP1258 and IBM874 leave unassigned the positions Microsoft never assigned, where the standard fills them with C1 controls. CP1255 leaves CA unassigned too.
  • CP1255 and CP1258 compose. Reading, a character followed by one of the code page's combining marks is read as the single character Unicode has for the two, if there is one: in CP1258 A and CC are À, and in CP1255 ב and dagesh are U+FB31. The test is that the character's full canonical decomposition and the mark hold the same characters as the result's, in any order, so Ó and a tilde make Ṍ although Unicode spells that with the tilde first. In CP1255 a further mark goes on composing; in CP1258 only a character read from a byte of its own takes a mark. Writing, a character with no byte of its own is spelt as a base and a mark, the shortest way it can be, so Ṍ is D3 DE. CP1258 also writes U+0340 and U+0341 as the grave and acute accents, and U+1FEE as A8 EC, which reads back as U+0385.
  • ISO-8859-9 and ISO-8859-11 are windows-1254 and windows-874 with 80 to 9F as the C1 controls. TIS-620 is ISO-8859-11 with nothing at 80 to A0.
  • KOI8-U follows RFC 2319 at AE and BE (╝ and ╬). KOI8-RU is the standard's KOI8-U with nine of 93 to 9F changed to typographic symbols: 93 “, 96 ”, 97 —, 98 №, 99 ™, 9B », 9C ®, 9D « and 9F ¤.
  • MACINTOSH reads C6 as Δ U+0394 rather than ∆ U+2206, and the Apple logo at F0 as U+E01E rather than U+F8FF. MAC-UK has ¤ at FF where the standard has €, and CP10007 has that and ¢ at A2.
  • IBM875 has ∇ at 74 and · at DD, and leaves 6A and the six extra SUBSTITUTE positions unassigned. IBM1026 has ˛ at 9D and — at BC.
  • CP932 refuses 80. It writes the end-user defined characters U+E000 to U+E757 back to F040 to F9FC, and writes six characters one way: ¢, £ and ¬ as their fullwidth forms, and U+2014, U+2016 and U+301C as 815C, 8161 and 8160.
  • EUC-KR is KS X 1001 alone, without Wansung's eight-byte syllables. It reads 80 to 9F as the C1 controls, has KS X 1001:2002's postal code mark ㉾ at A2E8, and writes U+20A9 WON SIGN as A3DC, the fullwidth won sign.
  • JOHAB reads 5C as the won sign, as JOHAB.TXT does, so the backslash cannot be written. It also has the postal code mark, at D9E8.
  • ISO-2022-JP is RFC 1468's. Its JIS X 0208 has none of code page 932's rows 13 and 89 to 92, and reads 2141, 2142 and 215D as 〜 U+301C, ‖ U+2016 and − U+2212, and 2171, 2172 and 224C as ¢, £ and ¬, where the standard has the fullwidth forms. It has no halfwidth katakana, so ESC ( I is not a designation and halfwidth katakana cannot be written. An ESC that does not begin one of the four designations is the character U+001B, a control byte in JIS X 0208 mode is that ASCII character, and designating the mode already in force is not an error. Writing, it stays in JIS X 0201 Roman for the printable characters Roman shares with ASCII, and leaves JIS X 0208 for anything else, a space or a line break included.
  • EUC-JP has the same JIS X 0208 as ISO-2022-JP, and writes JIS X 0212 as well as reading it, where the standard only reads it. 80 to 9F, apart from 8E and 8F, are the C1 controls. ¥ and ‾ are written as 5C and 7E, which read back as \ and ~.
  • SJIS is Shift_JIS as JIS X 0208 defines it, not code page 932: its single bytes are JIS X 0201, so 5C and 7E read as ¥ and ‾, and its two-byte characters are the same JIS X 0208 as EUC-JP's, with nothing from F0 up. The backslash and the tilde are still written as 5C and 7E, and the fullwidth ¢, £ and ¬ as the cells for ¢, £ and ¬.
  • GBK is the standard's gb18030 two-byte table without what GB 18030 added to GBK and without the Private Use Area. Every cell the standard maps to U+E000–F8FF is unassigned, and so are the euro sign at A2E3, A3A0, the vertical forms in row A6, ḿ and ǹ at A8BC and A8BF, A989 to A995, and FE50 to FEA0. The euro sign is the single byte 80, and there are no four-byte sequences.
  • EUC-CN (GB2312) is that GBK cut down to GB 2312's square, A1A1 to FEFE, with no single bytes above 7F. It also drops the cells GBK added inside the square — the small roman numerals at A2A1 to A2AA, A6D9 to A6F5 and A8BB to A8C0 — and reads A1A4 as U+30FB and A1AA as U+2015, where GBK has U+00B7 and U+2014.
  • GB18030 is the standard's gb18030 with three differences, each both ways, so that every scalar value reads back as itself. 80 is an error rather than the euro sign. A3A0 is U+E5E5 rather than U+3000. And the eighteen private use code points GB18030-2022 moved are four-byte sequences — 82359037 to 82359134 and 84318236 to 84318335 — where the standard's encoder writes them as two bytes and its decoder reads those four-byte sequences as U+9FB4–9FBB and U+FE10–FE19.
  • BIG5 is Big5 itself, not the standard's Big5-HKSCS: lead bytes A1 to F9 only, nothing at A3C0 to A3E0, the ETEN extensions at C6A1 to C8FE as the Private Use Area from U+F6B1, F9FE as ▓ U+2593 rather than U+FFED, and 80 as U+0080. A character it has twice is written as its first cell, except 十 and 卅, which go to A451 and A4CA.

tools/crosscheck.py walks every input of every encoding against CPython's codecs module and prints what differs; the list above is what it finds, and anything else appearing there would be a bug.

It also prints what it skipped, which is utf-16le, utf-16be and replacement asked to encode. The standard makes those three decode-only — Encoding.canEncode is this library's answer about which — so there is nothing to compare, and the run says so rather than failing.

The tables

src/tables/ is generated from the index files the standard publishes — and, for EBCDIC, from Microsoft's mapping tables as Unicode.org publishes them, and for composing, from the canonical decompositions in the Unicode Character Database's UnicodeData.txt — which are committed in data/ so that a build needs no network and so that a change upstream shows up in review as a diff to its input rather than as an unexplained change to a mapping.

nix develop -c zig build gen-tables    # then read `git diff`

The workflow regenerates and checks for a dirty tree, so an index that was updated without the tables being remade fails the build.

data/glibc-names.txt is the exception: it is not published anywhere, so tools/glibc_names.py writes it. It takes every name iconv -l lists, groups them by the alias and module lines of glibc's gconv-modules files, and has glibc confirm the groups of the few names it builds in. It also writes tests/glibc-name-cases.txt, several thousand ways of writing those names, each with the verdict glibc gives.

nix develop -c python3 tools/glibc_names.py           # rewrite both
nix develop -c python3 tools/glibc_names.py --check   # what CI runs

Testing

nix develop -c zig build test            # unit and conformance tests
nix develop -c python3 tools/crosscheck.py   # against CPython's codecs
nix develop -c python3 tools/crosscheck_glibc.py   # against glibc's iconv
nix develop -c zig build fuzz-run        # our own fuzzing loop
nix develop -c zig build fuzz --fuzz     # Zig's fuzzer, until interrupted

The conformance tests are spot checks taken from the standard's worked examples and from arithmetic done by hand against the algorithms — a test whose expected value was read out of the same table the code reads proves nothing.

The fuzz targets check three properties: decoding cannot crash and cannot produce invalid UTF-8; where a chunk boundary falls does not change the answer; and text that has already been through an encoding once comes out of it unchanged the second time. zig build fuzz --fuzz hands them to Zig's fuzzer, with coverage feedback, until it is interrupted; zig build fuzz-run is a loop of our own over a corpus that stops after a fixed number of inputs, which is what CI runs. See tools/fuzz.zig.

The glibc tools run only on glibc 2.44, which is what the devshell provides, and refuse any other version. A nixpkgs update that moves glibc therefore fails CI. Moving to a new glibc means running both tools again, putting the new version in the README, and taking out of Glibc any charset that no longer matches.

Where this lives

git clone https://git.jcollie.dev/jeff/zig-charset.git
rad clone rad:z2SapCVyHHcNssHjcrBvpA2XXuQnW

A Radicle repository is findable only by its ID, so that string is the one thing a reader needs in order to seed or clone it.

Report issues and send pull requests on the Tangled mirror, https://tangled.org/jcollie.dev/zig-charset. CONTRIBUTING.md covers what a change needs to pass.

References cited

The first is the whole specification this implements: the encodings and their labels, the decoder and encoder algorithm for each, and the index files that data/ is a copy of. The second is where the four EBCDIC tables come from.

The third was consulted and deliberately not used. It does not say which code page it is, it states that the translation is not bidirectional, and it collapses everything unmappable to 0x1A, so nothing built on it could round trip.

JOHAB.TXT is what Johab is checked against; see differences from other implementations for the three positions where this library departs from it. Windows' own table for code page 1361 was consulted and not followed. It decodes a standalone jamo to the conjoining Hangul Jamo block rather than the Compatibility Jamo that KS X 1001 itself uses, and a few of its entries disagree with the arithmetic Annex 3 defines. RFC 1557 is the EUC-KR that Wansung is: KS X 1001 in EUC form, with nothing added.

RFC 1468 is the ISO-2022-JP that glibc's is. The Unicode Character Database's canonical decompositions, which data/UnicodeData.txt is a copy of, are what CP1255 and CP1258 compose by, and UAX #15 defines canonical decomposition.

RFC 2279 is the original UTF-8, which glibc's decoder still reads and which is where outside Unicode comes from. RFC 2319 is the KOI8-U that glibc's is. glibc 2.44 is the implementation Glibc reproduces. It is a test oracle and the source of the list of names, and never the source of a mapping: see docs/adr/0001-glibc-charsets.md.

Licensing

MIT, and the project follows REUSE — nix develop -c reuse lint checks it.

The index files under data/ are the WHATWG's, under the CC-BY 4.0 that covers the Encoding Standard. The tables generated from them are MIT AND BSD-3-Clause, the standard's own terms for portions of it incorporated into source code. The EBCDIC mapping files and the table generated from them carry the Unicode licence. data/glibc-names.txt and the table generated from it are a list of glibc's names and which charset each belongs to. That is facts about glibc rather than any part of it, and they are MIT like the rest.