[Jprogramming] Insert zeroes into large data file

[Randy MacDonald]

Hello Matthew;

Okay, one minute (60 seconds) will suffice then.

The conjunction idea is essentially correct. A 'filterFile' conjunction 
could be written like so:

   TargetFile ChunkSize filterFile FilterVerb SourceFile

This would be the modern form of a flat APL utility I recall developing, 
back when 10MB and 512KB were the hard drive and RAM sizes 
respectively.  There will always be a bigger data source.  I recall 
manually adjusting the chunk size as time passed, benchmarking all the 
way.  FilterVerb was a direct definition string, a pre-dynamic function 
if you will.

This tool was also be tweaked at some point to chunk on newline 
boundaries.  Generalizing to a 'big cut' looked feasible enough that it 
never got implemented.


Matthew Brand wrote:
> it can be done in 43 seconds using C#  for comparison:
>
>           DateTime t = DateTime.Now;
>
>             StreamReader sr = new StreamReader(@"C:\\input.csv");
>
>             StreamWriter sw = new StreamWriter(@"C:\\output.csv");
>
>             string s = "";
>
>             string[] slt=null;
>
>             string news="";
>
>
>
>             while(!sr.EndOfStream)
>
>             {
>
>                 s=sr.ReadLine();
>
>                 news = "";
>
>                 slt = s.Split(',');
>
>                 foreach(string p in slt)
>
>                 {
>
>                     if (p == "")
>
>                         news += "0";
>
>                     else
>
>                         news += p;
>
>                     news += ",";
>
>                 }
>
>                 news = news.Remove(news.Length-1);
>
>                 sw.WriteLine(news);
>
>             }
>
>             sw.Close();
>
>             sr.Close();
>
>             MessageBox.Show("Took " + DateTime.Now.Subtract(t).Seconds + "
> seconds");
>
> I think the main problem in J is running out of ram during the process. I
> tried the (#!.'0'~ 1 j. ',,' E. ]) solution on a 4Gb 64-bit XP machine and
> it was very slow, it started to use the page file in task manager - at that
> point I stopped the process. I think that given enough ram J would do it
> quickly with (#!.'0'~ 1 j. ',,' E. ]) ... but what about a 1Gb file, or a
> 10Gb file?
>
> I don't think that 138MB is considered very large these days. Intraday
> trading data or output of global climate models can easily be larger than
> this.
>
> I was wondering whether it is possible to write an adverb that auto
> splits whatever is coming in to it on the basis of availible memory or to
> make some kind of chunkifiaction happen automatically... but don't think it
> would be easy to do. One might input a set of asserts to guide the splitting
> process and use info about the ram size to determine wether it is neccesary
> and how many splits to do.
>
> The chunkify method takes 86 seconds, which is good enough for me at the
> moment...
>
>    require'jmf'        NB. map file utilities loaded in jmf
>    require'files dir'
>    textfile =. 'C:\input.csv'
>    JCHAR map_jmf_ 'bigtext';textfile
>    chunk_idx     =.  (i.@:<.&.(%&chunk_size <i.@:%3C.&.(%25&chunk_size> =:
> 10000))@:#
>    chunkify_mask =.  (($@:[ $ 0"_) (1"_)`]`[} _1 , ] + ',,' -:"1 ({~ (,.
>   
>> :))) chunk_idx
>>     
>    null2zero     =.  #!.'0'~ 1 j.',,' E. ]
>    ts=: 6!:2, 7!:2@]
>    ts ' ((;@:(<@:null2zero;.2)~ chunkify_mask) bigtext ) 1!:2
> <''C:\output.csv'' '
> 85.6266 5.38447e8
>
>  It would be nice if this chunkification just happened somehow in the
> background though and all you needed to write is:
>
>   require'jmf'        NB. map file utilities loaded in jmf
>    require'files dir'
>    textfile =. 'C:\input.csv'
>    JCHAR map_jmf_ 'bigtext';textfile
>
>  (  #!.'0'~ 1 j.',,' E. ]) MEMHANDLE (assert1`assert2`...) bigtext
>
> Where MEMHANDLE is a conjunction which manages the splitting and assert1,...
> are a list of things that have to hold true in each split.
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
>   

-- 
------------------------------------------------------------------------
|\/| Randy A MacDonald       | APL: If you can say it, it's done.. (ram)
|/\| ramacd <at> nbnet.nb.ca |
|\ |                         | The only real problem with APL is that
BSc(Math) UNBF'83            | it is "still ahead of its time."
Sapere Aude                  |     - Morten Kromberg
Natural Born APL'er          |
-----------------------------------------------------(INTP)----{ gnat }-