[Jprogramming] utf8 and latin-1 code plane

[Elijah Stone]

I wonder what happens when you create two files with distinct names, and 
then unicode changes such that they are the same when 
normalised/casefolded/..

Probably nothing good.

On Tue, 22 Mar 2022, Eric Iverson wrote:

> My favorite unicode story is from macos filenames.
>
> They decompose filenames and only track the decomposed form (letter
> separate from the overstrike).
>
> The following accented chars look the same, but have different values.
>
>   c=: 195 164{a. NB. composed
>   c
> ä
>   d=: 97 204 136{a. NB. decomposed
>   d
> ä
>   c-:d
> 0
>   'abc'fwrite c
>   fread c NB. fails on macos as the system has the decomposed form as the
> name it looks for
>
> Torvald has a wonderful rant about this that is a fun read.
>
> On Tue, Mar 22, 2022 at 7:02 PM Raul Miller <rauldmiller at gmail.com> wrote:
>
>> I ran into a situation, today (dealing with files), where most of the
>> files were utf-8 encoded but some represented the latin-1 "code plane"
>> with 8 bit characters.
>>
>> To cope with this issue, I coded up a mechanism to test whether the
>> file contained only valid utf-8 sequences, and used {{ ": 10 u: y }}
>> for the files which failed this test.
>>
>> In other words:
>>
>> cclass=: (i.9) (48+i.9)} 256#9
>> cstates=: 0 10#:10* ".;._2{{)n
>>   0    7.3  2    3    4    5    6    7.3  7.3  7.1 NB. 0: start char
>> sequence
>>   0    7.3  2    3    4    5    6    7.3  7.3  7.1 NB. 1: finish char
>> sequence, start next
>>   7.3  1    7.3  7.3  7.3  7.3  7.3  7.3  7.3  7.3 NB. 2: need one
>> more character
>>   7.3  2    7.3  7.3  7.3  7.3  7.3  7.3  7.3  7.3 NB. 3: need two
>> more characters
>>   7.3  3    7.3  7.3  7.3  7.3  7.3  7.3  7.3  7.3 NB. 4: need three
>> more characters
>>   7.3  4    7.3  7.3  7.3  7.3  7.3  7.3  7.3  7.3 NB. 5: need four
>> more characters
>>   7.3  5    7.3  7.3  7.3  7.3  7.3  7.3  7.3  7.3 NB. 6: need five
>> more characters
>>   7.3  7.3  7.3  7.3  7.3  7.3  7.3  7.3  7.3  7.2 NB. 7: end
>> }}
>>
>> utf8lenb=: <:2#.>1 #each~1+i.8
>> utf8ok=: {{
>>   try.
>>     (1;cstates;cclass) ;: '.',~'012345678_'{~ utf8lenb I. 3 u: y
>>     1
>>   catch.
>>     0
>>   end.
>> }}
>>
>> NB. most content is utf-8 -- assume non-utf-8 sequences are ascii+latin-1
>> latin2utf8=: {{
>>   if.utf8ok y do. y else. ":10 u: y end.
>> }}
>>
>> I don't know if this approach would be useful to anyone else here,
>> but... just in case...
>>
>> FYI,
>>
>> --
>> Raul
>> ----------------------------------------------------------------------
>> For information about J forums see http://www.jsoftware.com/forums.htm
>>
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm