Reading Excel file with SQL returns null due to empty column - sql

Question

I am attempting to read in an Excel File from a third party but it falls over and returns null due to empty column.
The file has 50 columns, a blank column and a further 48 columns. I only want the first 50 columns. If I delete or populate the empty column in Excel it works but I need to read 50+ files one after the other, regularly, so opening/deleting/saving isn't an option.
Function read_excel_file(StrPath, StrFile, ByRef TotalFileArray() As Variant)
Dim ReadFileArray() As Variant
Dim connectionString As String
connectionString = _
"Provider=Microsoft.ACE.OLEDB.12.0;" & _
"Data Source=""" & StrPath & StrFile & """;" & _
"Extended Properties=""Excel 12.0;HDR=Yes;"""
'This assumes the Excel file contains column headers -- HDR=Yes
'Routine to get unknown sheet name
Set conn = CreateObject("ADODB.Connection")
conn.connectionString = "Provider=Microsoft.ACE.OLEDB.12.0;" & _
"Data Source=""" & StrPath & StrFile & """;" & _
"Extended Properties=""Excel 12.0;HDR=No"""
conn.Open
Set bs = conn.OpenSchema(20) ' 20 = adSchemaTables
Do Until bs.EOF = True
'Debug.Print bs.Fields!Table_Name.Value
SheetName = bs.Fields!Table_Name.Value
bs.MoveNext
Loop
bs.Close: conn.Close
Set bs = Nothing
Set conn = Nothing
Dim sql As String
'sql = "SELECT Meter Id, Date, 00:30, 01:00, 01:30, 02:00, 02:30, 03:00, 03:30, 04:00, 04:30, 05:00, 05:30, 06:00, 06:30, 07:00, 07:30, 08:00, 08:30, 09:00, 09:30, 10:00, 10:30, 11:00, 11:30, 12:00, 12:30, 13:00, 13:30, 14:00, 14:30, 15:00, 15:30, 16:00, 16:30, 17:00, 17:30, 18:00, 18:30, 19:00, 19:30, 20:00, 20:30, 21:00, 21:30, 22:00, 22:30, 23:00, 23:30, 24:00:00 FROM [" + SheetName + "]" '
sql = "SELECT * FROM [" + SheetName + "]" '
'Dim rs As Object 'New adodb.Recordset
'Set rs = CreateObject("ADODB.Connection")
Dim rs As New ADODB.Recordset
rs.Open sql, connectionString
ReadFileArray() = rs.GetRows 'Puts the data from the recordset into an array
rs.Close
Set rs = Nothing
I did try using specific requests for the columns I wanted but this still returned null. I attempted a single column but again returned null.
Is there a way around the empty column? Even if reading the file in a different way. (I nearly always read in csv files which are a breeze to get into an array and deal with.)
Picture of source data headers

Answer 1

Given that you have no choice but to work with the files in the format provided, the best approach/workaround (rather than manually opening/deleting/saving/closing the files individually) would appear to be to pre-process them using VBA to delete the offending column, before then running your query.

Answer 2

If Excel worksheet contains columns in first row and you specify HDR=Yes, then your first SQL statement would work if you properly enclosed the column names with brackets since they contain special characters like spaces and start with numbers.
' OPEN CONNECTION WITH HEADERS INDICATED
connectionString = _
"Provider=Microsoft.ACE.OLEDB.12.0;" & _
"Data Source=""" & StrPath & StrFile & """;" & _
"Extended Properties=""Excel 12.0;HDR=Yes;"""
conn.Open connectionString
' RUN SQL WITH COLUMNS SPECIFIED
sql = "SELECT [Meter Id], [Date]" _
& " , [00:30], [01:00], ..., [23:30], [24:00:00]" _
& " FROM [" + SheetName + "]"
rs.Open sql, connectionString
...
In fact, your original attempt should have raised an error in attempting to run. Also, the second SQL with SELECT * FROM should raise an error with empty column since one of the column names is zero length which is an invalid identifier. Therefore, you need to explicitly select columns.
After deeper review, it appears your time columns may be in Custom format as time values which really are formatted Excel decimal values. Hence, the SQL engine cannot find [00:30] column. To fix, convert time value columns into text format.
Consider below subroutine to automate the conversion with Text(). You can then call this VBA sub or function from other main method on each worksheet before any SQL query. Do note there is no such time value as 24:00:00 so this may be already in text format.
Sub Clean_Time_Columns()
Dim i As Integer, lastcol As Integer
Dim tmp_time As Variant
With ThisWorkbook.Worksheets("DATA")
For i = 3 To 49 ' LEAVE OUT 24:00:00
tmp_time = .Cells(1, i).Value
.Cells(1, i).NumberFormat = "#"
.Cells(1, i).Value = Application.WorksheetFunction.Text(tmp_time, "HH:MM")
Next i
End With
End Sub