[Jprogramming] Unicode puzzle
Björn Helgason
gosinn at gmail.com
Thu Oct 12 22:19:28 HKT 2006
I did a similar test for a few German letters "ößüäÖÜÄ"
require'files'
a=:fread'c:\temp\ger.txt'
a. i. a
246 223 252 228 214 220 196
'ößüäÖÜÄ'fwrite'c:\temp\geruni.txt'
14
b=:fread'c:\temp\geruni.txt'
a. i. b
195 182 195 159 195 188 195 164 195 150 195 156 195 132
c=:fread'c:\temp\geruninote.txt'
a. i. c
255 254 246 0 223 0 252 0 228 0 214 0 220 0 196 0
d=:fread'c:\temp\gerunibignote.txt'
a. i. d
254 255 0 246 0 223 0 252 0 228 0 214 0 220 0 196
e=:fread'c:\temp\gerutfnote.txt'
a. i. e
239 187 191 195 182 195 159 195 188 195 164 195 150 195 156 195 132
b
ößüäÖÜÄ
e
ößüäÖÜÄ
a
b
ößüäÖÜÄ
c
d
e
ößüäÖÜÄ
Same results a,c and d do not display in J
Even if all the files look the same in Notepad and e and b look the same in
J they are not the same
e=b
|length error
| e =b
|[-0]
e-:b
0
2006/10/12, Björn Helgason <gosinn at gmail.com>:
>
> I have exactly the same problem as you have with my Íslandish letters
>
> As an example I creeate a file with Notepad with "þæö"
>
> It can be using codepages or it can be in Unicode
>
> As it happens with Notepad you can store the file as Unicode, Unicode Big
> Endian and Utf-8 as well as ANSI (using codepages)
> With ANSI and codepages there are a number of places "þæö" and the rest of
> the Íslandish chars can be
>
> Let me create a file with only "þæö" in it and save it as different
> options of Notepad allows
>
> require'files'
>
> a=:fread'c:\temp\thorn.txt'
>
> a. i. a
> 254 230 246
>
> 'þæö'fwrite'c:\temp\thornuni.txt'
> 6
>
> b=:fread'c:\temp\thornuni.txt'
>
> a. i. b
> 195 190 195 166 195 182
>
> c=:fread'c:\temp\thornuninote.txt'
>
> a. i. c
> 255 254 254 0 230 0 246 0
>
> d=:fread'c:\temp\thornunibignote.txt'
>
> a. i. d
> 254 255 0 254 0 230 0 246
>
> e=:fread'c:\temp\thornutfnote.txt'
>
> a. i. e
> 239 187 191 195 190 195 166 195 182
>
> In Notepad they all look the same
> In J only b and e look the same but the a. i. display is not the same
> b is created from J as you saw from above the rest from different
> encodings by Notepad
>
> a
>
>
> b
> þæö
>
> c
>
>
> d
>
>
> e
> þæö
>
> I am sure you may be even more confused by the above
>
> How this issue is to be resolved - wellllll.....
>
>
> 2006/10/12, Ronan Reilly <ronan at cs.nuim.ie>:
> >
> > Thanks Chris, Björn and Bill.
> >
> > That clarifies things. However, I'm finding that the text strings I
> > read
> > from the file are not getting converted by applying utf8. It seems that
> > they have to be datatype unicode for this to work, but they are read in
> > and
> > stored as literals. Is there any way of coercing datatypes in J to get
> > around this?
> >
> >
> > Thanks again,
> >
> > Ronan
> >
> > PS: I'm running J on a Mac if that's relevant
> >
> >
> >
> > On 11/10/2006 23:59, "Chris Burke" < cburke at jsoftware.com> wrote:
> >
> > > Ronan Reilly wrote:
> > >> I'm trying to display some German text using plot (J601). The text
> > has been
> > >> read in from a file using fread, decoded using ". , and stored in a
> > table.
> > >>
> > >> When I extract a word with an umlaut from the table (e.g.,
> > Aktualität),
> > >> assign it to W1, and plot it using
> > >>
> > >> pd 'reset;text 0 _1x ',W1,';show'
> > >>
> > >> the letter with an umlaut does not display correctly.
> > >>
> > >> However, if I evaluate W1 in the jwd, like so:
> > >>
> > >> W1
> > >> Aktualität
> > >>
> > >> and then edit and evaluate it like so
> > >>
> > >> W2 =: Aktualität
> > >>
> > >> and then display W2 using
> > >>
> > >> pd 'reset;text 0 _1x ',W2,';show'
> > >>
> > >> W2 displays correctly.
> > >>
> > >> Also
> > >>
> > >> $W1
> > >> 10
> > >> $W2
> > >> 11
> > >>
> > >> What is going on here? How can I display the unicode characters
> > using plot?
> > >
> > > In general, J assumes incoming text is in utf8 format. J also supports
> > a
> > > "unicode" data type, which is 2-byte unicode, see the help for u: .
> > >
> > > Text as either utf8 or unicode will display correctly in the session,
> > > but only utf8 will work in plot.
> > >
> > > In this example, W2 is in utf8 format, and W1 in 2-byte unicode. You
> > > need to convert W1 to utf8, using the utf8 verb.
> > >
> > > Here is what is happening:
> > >
> > > #W2=: 'Aktualität'
> > > 11
> > > #W1=: ucp W2
> > > 10
> > >
> > > W2 -: utf8 W1
> > >
> > > datatype W2
> > > literal
> > > datatype W1
> > > unicode
> > >
> > > a.i.W2
> > > 65 107 116 117 97 108 105 116 195 164 116
> > > a.i.W1
> > > 65 107 116 117 97 108 105 116 228 116
> > >
> > > ----------------------------------------------------------------------
> > > For information about J forums see http://www.jsoftware.com/forums.htm
> > >
> >
> > --
> > Professor Ronan Reilly
> > Head of Department
> > Department of Computer Science
> > NUI Maynooth
> > Maynooth
> > Co. Kildare
> > IRELAND
> >
> > t: +353-1-7083847
> > e: Ronan.Reilly at nuim.ie
> > w: http://www.cs.nuim.ie; http://cortex.cs.nuim.ie
> >
> >
> >
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
> >
>
>
>
> --
> Björn Helgason, Verkfræðingur
> Fugl&Fiskur ehf, Þerneyjarsund 23, Box 127
> 801 Grímsnes ,t-póst: gosinn at gmail.com
> Skype: gosiminn, gsm: +3546985532
> Landslags og skrúðgarðagerð, gröfuþjónusta
> http://groups.google.com/group/J-Programming
>
>
> Tæknikunnátta höndlar hið flókna, sköpunargáfa er meistari einfaldleikans
>
> góður kennari getur stigið á tær án þess að glansinn fari af skónum
> /|_ .-----------------------------------.
> ,' .\ / | Með léttri lund verður |
> ,--' _,' | Dagurinn í dag |
> / / | Enn betri en gærdagurinn |
> ( -. | `-----------------------------------'
> | ) | (\_ _/)
> (`-. '--.) (='.'=)
> `. )----' (")_(")
>
--
Björn Helgason, Verkfræðingur
Fugl&Fiskur ehf, Þerneyjarsund 23, Box 127
801 Grímsnes ,t-póst: gosinn at gmail.com
Skype: gosiminn, gsm: +3546985532
Landslags og skrúðgarðagerð, gröfuþjónusta
http://groups.google.com/group/J-Programming
Tæknikunnátta höndlar hið flókna, sköpunargáfa er meistari einfaldleikans
góður kennari getur stigið á tær án þess að glansinn fari af skónum
/|_ .-----------------------------------.
,' .\ / | Með léttri lund verður |
,--' _,' | Dagurinn í dag |
/ / | Enn betri en gærdagurinn |
( -. | `-----------------------------------'
| ) | (\_ _/)
(`-. '--.) (='.'=)
`. )----' (")_(")
More information about the Programming
mailing list