Speedup VBA: Large tables from Word to Excel - vba

Question

I have large tables in RTF format, from 20-150 Mb in size. I first tried to export the RTF -> HTML -> Import to excel. It took about 35 minutes for a 60 Mb file. Next, I tried copying the table directly from Word -> excel. It always fails midway (everything gets pasted, by data is not in the right cell).
I tried a few more ways (importing all cells into memory before transferring to excel, and other permutations, and methods detailed in this and other sources) before settling on the .ConvertToText method.
This method is relatively faster, taking about 25 minutes for the same 60 Mb file (this is without displaying Word, setting repagination, events, dispayupdate and tableautofit to false).
Considering that these files can be entirely loaded into RAM memory in less than 10 seconds, I wonder why does it take 25 minutes to read data off a 60 Mb file. I understand that the table engine in Word is slow because of the change to HTML format, but reading a table cell by cell is atrociously slow. First few cells are super fast, Last cells are slower - I'm sure manual reading is faster than that. It defeats the whole purpose of automation. However, I do not have a choice.
The code is:
Dim oWord As Word.Application
Dim RTF As Word.Document
Set oWord = CreateObject("Word.Application")
Set RTF = oWord.Documents.Open(filename:=Fname, ConfirmConversions:=False, ReadOnly:=False) ', ReadOnly:=True)
Application.StatusBar = vbNullString
Application.EnableEvents = False
Application.ScreenUpdating = False
Application.DisplayAlerts = False
With oWord
Options.Pagination = False
Options.AllowReadingMode = False
Application.AutoRecover.Enabled = False
Options.SaveInterval = 0
Options.CheckGrammarAsYouType = False
Options.CheckGrammarWithSpelling = False
End With
With RTF
Options.Pagination = False
Options.AllowReadingMode = False
Application.AutoRecover.Enabled = False
Options.SaveInterval = 0
Options.CheckGrammarAsYouType = False
Options.CheckGrammarWithSpelling = False
End With
Dim AAF As Table
For Each AAF In RTF.Tables
AAF.AllowAutoFit = False
Next
oWord.Visible = False
Dim rng As Word.Range
Dim sData As String
Dim aData1() As String
Dim aData2() As String
Dim aDataAll() As String
Dim nrRecs As Long
Dim nrFields As Long
Dim lRecs As Long
Dim lFields As Long
Dim CTbl As Table 'Data Table
Dim oCell As Cell
'I'm not displaying the code which replaces all ^p with a spl character to maintain the table structure - it is staright forward, and does the job
Set rng = CTbl.ConvertToText(Separator:="$", NestedTables:=False)
sData = rng.Text 'This contains the entire table, delimited by vbCr and $...
Application.StatusBar = "Closing open files..."
RTF.Close (wdDoNotSaveChanges) 'All data has been extracted, hence quit word
oWord.Quit
Set oWord = Nothing
sData = Mid(sData, 1, Len(sData) - 1)
aData1() = Split(sData, vbCr)
nrRecs = UBound(aData1())
If Dbg Then MsgBox "The table contained " & nrRecs + 1 & " rows"
For lRecs = LBound(aData1()) To nrRecs 'Cycle through all rows
aData2() = Split(aData1(lRecs), "$") 'Split rows into arrays
Debug.Print aData1(lRecs)
nrFields = UBound(aData2()) 'Find out the number of columns
If lRecs = LBound(aData1()) Then 'If this is the first row/cycle,
ReDim Preserve aDataAll(nrRecs, 9) 'nrFields) 'Resize the array - currently I'm using a fixed size for the column since the first row of my table contains merged rows
End If
For lFields = LBound(aData2()) To nrFields 'Cycle through all columns
aDataAll(lRecs, lFields) = aData2(lFields) 'Collate the data in a single array
'If MsgBox(aDataAll(lRecs, lFields), vbYesNo, "Continue?") = vbNo Then Exit For
Next
Next 'All of this was slapped together from MS code samples and stackoverflow examples
Any suggestions to improve performance?

Answer 1

The conversion will go a lot faster if you first split the table (I assume there's one very large table) into smaller tables and then convert each table to text.
I tried this on a table with 10000 rows and 10 columns. The time to convert to text went from ~280 seconds to ~70 seconds (i.e. 4X faster).
I ran the code below directly from the document with the 10000 row table (as opposed to running from Excel) for simplicity.
Splt then convert:
Sub SplitThenConvert()
Dim t As Table
Set t = ActiveDocument.Tables.Item(1)
Dim rowCount As Integer
Dim index As Integer
Dim numSteps As Integer
Dim splitRow As Integer
Dim increment As Integer
Dim start_time, end_time
start_time = Now()
Application.ScreenUpdating = False
rowCount = t.Rows.Count
numSteps = 10
increment = rowCount / numSteps
splitRow = rowCount - increment
For index = 1 To numSteps
Debug.Print "Split #" + CStr(index)
ActiveDocument.Tables(1).Rows(splitRow).Select
Selection.SplitTable
splitRow = splitRow - increment
If splitRow < increment Then
Exit For
End If
Next index
index = ActiveDocument.Tables.Count
While index > 0
Debug.Print "Convert #" + CStr(index)
ActiveDocument.Tables(index).ConvertToText ","
index = index - 1
Wend
end_time = Now()
Application.ScreenUpdating = True
MsgBox (DateDiff("s", start_time, end_time))
End Sub
Convert entire table without splitting:
Sub ConvertAll()
Dim start_time, end_time
Application.ScreenUpdating = False
start_time = Now()
ActiveDocument.Tables(1).ConvertToText ","
end_time = Now()
Application.ScreenUpdating = True
MsgBox (DateDiff("s", start_time, end_time))
End Sub

Answer 2

I do agree with #KazJaw: reading/writing from/to MS Office programs (including .rtf because is treated as Word) is very computational expensive, better relying on other means as much as possible (just converting the .rtf file reading into a simple .txt file reading would improve the speed a lot). I have recently answered a post on these lines.
The other proposal I have is reducing the number of "live Office variables" as much as possible. Instead creating the RTF and the oDoc variables at the same time, better doing it one after the other (the same for Excel). What should be done only under exceptional circumstamces (because of being too computational expensive) is copying/pasting in real time between two different instances (for example, two different Word documents).
Thus, use the connection to Office programs for what it is intended, that is, top-level access to a file storing information in a pretty complex way: populate values, change formatting, perform complex actions (e.g., searching through the whole document); but intend to reduce the iterative behaviour (e.g., copying from one cell and pasting into another one over and over) as much as possible. See it in this way: copying/pasting in a .txt file involves just inspecting the input value/the target location and performing the action; doing it in Word involves the same than in the .txt file plus accounting for the huge amount of variables analysed while considering each record (formatting, references to other elements, special actions, etc.).