Mojibake is a low-level Unicode 17 text-processing library written in C11 and compatible
with C++17. It is released under the MIT License.
Usage
You don't need to install anything. There are two files (mojibake.c, mojibake.h) to add to your
C/C++ project. Download it here mojibake-amalgamation-027.zip
Examples of normalization, characters count and NFKC casefold.
This output:
Mojibake aims to be:
Small
Easy to use
Fast
Self-contained
Mojibake do:
Run in all modern OSes (Linux, macOS, FreeBSD, OpenBSD, NetBSD, Windows 10/11)
Pass the official Unicode test suites for supported algorithms
Implement all Unicode standard algorithms
Satisfy all Unicode Conformance Requirements
Unicode Conformance Requirements
Feature highlights
All the C files, together with the Unicode data tables, are concatenated into a single large file
and header: mojibake.c and mojibake.h. Zero dependencies.
Text transformation
Normalization: NFC/NFD/NFKC/NFKD (mjb_normalize), identifier-oriented NFKC case folding
(mjb_nfkc_casefold), plus a fast quick-check
(mjb_string_is_normalized) (UAX #15, Unicode 17.0.0)
Case conversion: uppercase, lowercase, titlecase, and case folding with full special-casing
and conditional mappings (mjb_case)
Filtering: strip controls, spaces, or numeric characters while normalizing
(mjb_string_filter)
Text analysis
Character database: every Unicode Character Database property: category, script and
Script_Extensions, block, plane, numeric value, name (mjb_codepoint_character,
mjb_codepoint_script_extensions)
Segmentation: grapheme clusters, words, sentences, and line-break opportunities
(UAX #29, Unicode 17.0.0,
UAX #14, Unicode 17.0.0)
Bidirectional text: full Unicode Bidirectional Algorithm: paragraph resolution, line
reordering, runs (UAX #9, Unicode 17.0.0)
Emoji: codepoint properties, sequence analysis, RGI emoji detection
Display width: East Asian width and terminal display width, with width-aware truncation
(mjb_display_width, mjb_truncate_width)
Sorting and comparison
Collation: Unicode Collation Algorithm string comparison and sort keys, in shifted and
non-ignorable modes (mjb_string_compare, mjb_collation_key,
UTS #10, Unicode 17.0.0)
Security
Confusable detection: generate reusable skeletons and check if strings are visually
confusable (mjb_confusable_skeleton, mjb_string_is_confusable,
UTS #39, Unicode 17.0.0)
Identifier validation: XID/ID checks for parser and compiler authors
(mjb_string_is_identifier, UAX #31, Unicode 17.0.0)
Integration
Encodings: the API accepts and outputs UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE
strings, with encoding detection and conversion (mjb_string_encoding,
mjb_string_convert_encoding)
Parsing and string functions: character-by-character iteration (mjb_next_character) and
standard C string.h-style helpers (mjb_string_length, and others)
Locales: strict BCP 47 language tag parsing (mjb_locale_parse)
Embeddable: custom allocators (mjb_set_memory_functions), build-time feature flags to trim
table size, a C++17 wrapper (src/cpp/mojibake.hpp), a CLI tool (src/shell), and a
WASM + TypeScript API (src/api)
Tested: Mojibake uses Attractor as test suite and run
1.5M+ assertions including the
official Unicode conformance suites for supported algorithms
Fuzz Mojibake is fuzzed with libFuzzer over untrusted
byte input
AddressSanitizer and UBSan clean
Build-time features
Mojibake can compile out optional feature tables to reduce binary size. Feature macros default to
enabled.
#define MJB_FEATURE_CHARACTER_NAMES controls the Unicode character-name tables used by
mjb_codepoint_character(...) to fill mjb_character.name. When disabled, the tables are not
compiled and mjb_character.name is reported as Codepoint U+XXXX. This will redude the output
of ~30%.
With CMake:
With the provided Makefile:
API documentation
See API.md or the site for the detailed
documentation.
CLI
The src/shell directory builds the mojibake CLI used to test the library. Example usage:
Building from source and contributing
See CONTRIBUTING.md for instructions.
Licenses
Mojibake is released under the MIT License (see LICENSE).
Legalese
Here you can find the very detailed and boring informations needed to have this library conformant
to the Unicode standard, or at least what I got, at
CONFORMANCE_REQUIREMENTS.md
Thanks
Mojibake is built using the work of extraordinary individuals and teams.
Unicode Character Database - Copyright © 1991-2026 Unicode, Inc.
(see license.txt)
Unicode CLDR Project - Copyright © 2004-2026 Unicode, Inc.
(see LICENSE)