Handling huge file with tcl - scripting

Question

can anyone tell me how I can update the following procedure to handle big files please (size <= 10 G):
proc read_text_file { file } {
set fp [open ${file} r]
set return_data ""
while { [gets $fp each_line] != -1 } {
lappend return_data ${each_line}
}
close $fp
return ${return_data}
}
my objective is to read a huge file line by line in a better runtime
Thanks

Answer 1

When you have a very large file, you categorically want to avoid bringing it all into memory at once. (Also, Tcl 8.* has a memory chunk allocation limit that makes bringing in 50GB of data intensely exciting. That's a long-standing API bug that's fixed in 9.0 — in alpha — but you'll have to put up with it for now.)
If you can, do a pass over the file to identify where the interesting sub-chunks of it are. For the sake of argument, let's assume that those are the lines that match a pattern; here's an example that finds where procedures are in a Tcl script (under some simple assumptions).
proc buildIndices {filename} {
set f [open $filename]
set indices {}
try {
while {![eof $f]} {
set idx [tell $f]
set line [gets $f]
if {[regexp {^proc (\w+)} $line -> name]} {
dict set indices $name $idx
}
}
return $indices
} finally {
close $f
}
}
Now you have the indices, you can then pull in a procedure from the file like this:
proc obtainProcedure {filename procName indices} {
set f [open $filename]
try {
seek $f [dict get $indices $procName]
set procedureDefinition ""
while {[gets $f line] >= 0} {
append procedureDefinition $line "\n"
if {[info complete $procedureDefinition]} {
# We're done; evaluate the script in the caller's context
tailcall eval $procedureDefinition
}
}
} finally {
close $f
}
}
You'd use that like this:
# Once (possibly even save this to its own file)
set indices [buildIndices somefile.tcl]
# Then, to use
obtainProcedure somefile.tcl foobar $indices
If you're doing this a lot, convert your code to use a database; they're a lot more efficient in the long run. The index building is equivalent to building the database and the other procedure is equivalent to doing a DB query.