tc-lib-unicode

Technical overview and integration notes for tc-lib-unicode

Updated:

Overview

tc-lib-unicode provides Unicode conversion utilities and bidirectional algorithm support for multilingual text processing.

Directionality is where most of the work is. A paragraph mixing Hebrew or Arabic with Latin text and numerals has to be reordered before it can be drawn, and this package implements UAX #9 to do it.

Repository and API Docs

Project Metadata

ItemValue
Namespace\Com\Tecnick\Unicode
LicenseGNU LGPL v3

Installation

composer require tecnickcom/tc-lib-unicode

Where It Fits

Any text that is not plain left-to-right Latin: right-to-left scripts, Arabic shaping, Devanagari and Hangul reordering, or mixed-direction runs on one line.

Features

Unicode Utilities

  • Conversions between UTF-8 strings, character arrays and code point arrays
  • Latin1, UTF-16BE and hexadecimal string conversions
  • Integration-ready conversion methods for document engines

Bidirectional Support

  • Unicode Bidirectional Algorithm (UAX #9) implementation verified against the official BidiCharacterTest.txt and BidiTest.txt conformance suites
  • Right-to-left and mixed-direction text processing
  • Arabic shaping driven by the Joining_Type property, with the lam-alef and allah ligatures written into the slot of the first character so combining marks keep following their base

Character Substitution

  • Context-sensitive codepoint-level substitution via Substitution::replaceChars(), a pure codepoint transform with no font or PDF dependency
  • Devanagari (U+0900–U+097F): left-positional matras (U+093F, U+094E) are moved before their base consonant cluster, including conjuncts joined by Virama (U+094D) and a trailing Nukta (U+093C); the additional consonant range U+0978–U+097F is recognized
  • Hangul Jamo (U+1100–U+11FF, U+A960–U+A97F, U+D7B0–U+D7FF): composed into precomposed syllables (U+AC00–U+D7A3) per section 3.12 of the Unicode standard
  • Thai (U+0E00–U+0E7F): returned unchanged, because Thai preposed vowels are already stored in visual order

Codepoints belonging to unsupported scripts are passed through unchanged.

Version 3.0 Changes

Release 3.0 is a breaking change driven by UAX #9 conformance work and the Unicode 17 data update:

  • The bidi implementation now conforms to UAX #9 and is validated against the official conformance suite.
  • Arabic shaping is driven by the Joining_Type property instead of a hand-maintained table.
  • The tables come from tc-lib-unicode-data, which is generated from UCD 17.0.0; Composer resolves the matching release.
  • UTF-8 codepoint and character conversions are vectorized through a single bulk mb_convert_encoding() call per invocation instead of per-character ord()/chr() loops.

Typed Enums

\Com\Tecnick\Unicode\TextDirection is a backed enum for the forced paragraph direction: Auto (''), Rtl ('R'), and Ltr ('L'). The Bidi constructor and the tc-lib-pdf text methods accept either the enum case or the plain string.

Known Limitations

  • The paragraph separator is dropped during processing and appended again at the end of the paragraph output rather than being reset by rule L1 and reversed by L2. A strict UAX #9 implementation would place it at the visual left edge of a right-to-left paragraph; keeping it at the end preserves line splitting for consumers.
  • Rule L3 (combining marks displayed in a different order) is not implemented.
  • Shaping is Arabic only; the other cursive scripts (Syriac, N’Ko, Mandaic, Adlam) are returned unshaped.
  • Bidi and Convert require valid UTF-8: malformed byte sequences raise an exception, and codepoints that cannot be encoded (out of range or surrogate) are replaced with ?.

Integration Notes

Normalize at the point text enters the system. The same visible string can be encoded several ways, and two of them will not compare equal, will not hash equal, and may not shape the same.

Punctuation and numerals are where bidirectional layout goes wrong, not the letters. Put brackets, quotes, dates, and currency amounts into your mixed-direction test cases; the neutral characters are the ones the algorithm has to resolve from context.

Requirements

  • PHP 8.2 or later
  • Extensions: ctype, mbstring, pcre
  • Package dependency: tecnickcom/tc-lib-unicode-data
  • Composer

Example

<?php

require_once __DIR__ . '/vendor/autoload.php';

use Com\Tecnick\Unicode\Bidi;
use Com\Tecnick\Unicode\TextDirection;

$bidi = new Bidi('hello ', null, null, TextDirection::Rtl, false);
echo $bidi->getString();

// Script-specific codepoint substitution.
$sub = new \Com\Tecnick\Unicode\Substitution();

// Devanagari: the left matra is repositioned before its base consonant cluster.
$result = $sub->replaceChars([0x0915, 0x093F]);  // [0x093F, 0x0915]

// Hangul: Jamo composed into a precomposed syllable.
$result = $sub->replaceChars([0x1100, 0x1161, 0x11A8]);  // [0xAC01]

Development and Packaging

  • QA and local checks: make deps, make help, make qa
  • Coverage report: make qa-coverage
  • Local example server: make server (or make server PORT=8080)
  • Packaging: make rpm, make deb

Support and Contribution