valknar/llmx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: enhance toCodePoints to prevent potential unicode 14 errors (#615)

Browse Source 
## Description

`Array.from` may fail when handling certain characters newly added in
Unicode 14. Where possible, it seems better to use `Intl.Segmenter` for
more reliable processing.


![image](https://github.com/user-attachments/assets/2cbd779d-69d3-448e-b76a-d793cb639d96)
This commit is contained in:
Luci
2025-04-25 01:49:18 +08:00
committed by GitHub
parent 1008e1b9a0
commit 6d68a90064
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
codex-cli/src/text-buffer.ts
View File

@@ -34,6 +34,10 @@ function clamp(v: number, min: number, max: number): number {
* ---------------------------------------------------------------------- */ * ---------------------------------------------------------------------- */
function toCodePoints(str: string): Array<string> { function toCodePoints(str: string): Array<string> {
if (typeof Intl !== "undefined" && "Segmenter" in Intl) {
const seg = new Intl.Segmenter();
return [...seg.segment(str)].map((seg) => seg.segment);
}
// [...str] or Array.from both iterate by UTF‑32 code point, handling // [...str] or Array.from both iterate by UTF‑32 code point, handling
// surrogate pairs correctly. // surrogate pairs correctly.
return Array.from(str); return Array.from(str);