[Jprogramming] How to speed up XML parsing of a large XML file

[Dan Farmer]

Hi again,

So I am trying to parse some large (2-9 GB) XML files for an idea I
had for using JDB. My plan was to use XSLT to flatten these things out
(they are deeply nested structures), but figured I'd do a quick and
easy test to make sure I had a reasonable grip on J's facilities
before diving in.

Unfortunately with the code I came up with it is so slow that I don't
think it's even worth attempting, can anyone provide some tips on how
I can maybe speed this up? I read up on the J performance monitor and
clocked it, it said 75% of the time was spent in cdcallback (which
makes me think there's nothing I can do short of writing something in
C/C++, but maybe I'm wrong). Here's the code (loosely adapted from
Oleg & John Baker's examples for sax).

For the record, I created two smaller test files (500KB and 6MB) and
the code below works correctly on both of those. I've also written
Python code using lxml's element tree module and it can process the 2
GB file in about 60 seconds, I let this code run for 30 minutes and
then killed it.

Any ideas?

Thanks,
Dan

require 'jmf'
require 'files dir'
require 'xml/sax'

saxclass ‘xp’

startDocument=: 3 : 0
ids=: ''
)


startElement=: 4 : 0
if. y-:,’Node’ do.
  ids=: ids,< x getAttribute '_Id'
end.
)


endDocument=: 3 : 0
s: ids
)

NB. =========================================================
cocurrent 'base'

fn=: 'c:/data/test/2GBfile.xml’'

unmap_jmf_ 'xfile' NB. Hokey, but for debugging

JCHAR map_jmf_ 'xfile';fn

process_xp_ xfile