[Jprogramming] tables/dsv extreme memory usage

[Aaron Ash]

Hi,

I've noticed that the tables/dsv addon seems to have an extremely high
memory growth factor when processing csv data:

load 'tables/csv'dat=: (34;'45';'hello';_5.34),:
12;'32';'goodbye';1.23d=: makecsv dat# d
NB. 45 chars longtimespacex 'fixcsv d'NB. 2.28e_5 48644864 % 45 NB.
108.089 factor of memory growth

This makes loading many datasets effectively impossible even on
reasonably specced machines.

A 1GB csv file would require 108GB of memory to load which seems
fairly extreme to the point where I would consider this a bug.

Someone on irc mentioned that generally larger datasets should be
loaded in to jd and that's fair enough but I still would expect to be
able to load csv data reasonably quickly and memory efficiently.

Is this a bug? Is there a better library to use for csv data?

Cheers,

Aaron.