Import csv with quoted newline using QueryTables in Excel - vba

I have written a visual basic macro to load a csv file into Excel that I use quite frequently.
Unfortunately, if the csv file contains quoted newlines, the result is different from what you would get if you opened the csv file directly with excel. Unlike the usual import facility, QueryTables.add() assumes any newline it runs into, whether quoted or not, is the end of the row.
Is there a way around this? I'd prefer a solution that did not involve pre-modifying the incoming csv files to remove the newlines, but I'm open to suggestions on that front as well. I do want to have newlines in the resulting excel file cells, though.
The relevant part of my macro:
Sub LoadMyFile()
' Query the table of interest
With ActiveSheet.QueryTables.Add(Connection:="TEXT;" _
& ThisWorkbook.Path & "\" & Range("A1").Value & ".csv", _
Destination:=Range("$A$2"))
.Name = ActiveSheet.Name
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.RefreshStyle = xlOverwriteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.TextFilePromptOnRefresh = False
.TextFilePlatform = 437
.TextFileStartRow = 1
.TextFileParseType = xlDelimited
.TextFileTextQualifier = xlTextQualifierDoubleQuote
.TextFileConsecutiveDelimiter = False
.TextFileTabDelimiter = False
.TextFileSemicolonDelimiter = False
.TextFileCommaDelimiter = True
.TextFileSpaceDelimiter = False
.TextFileColumnDataTypes = Array(1, 1, 1, 1, 1)
.TextFileTrailingMinusNumbers = True
.Refresh BackgroundQuery:=False
End With
End Sub
Here's an example csv file with quoted newlines
"firstCol","secondCol"
"name1","data
one"
"name
2","data two"
The macro reads the file name (minus the .csv extension) from cell A1 and assumes the csv file is in the same directory as the excel file containing the macro.
I'm using 32 bit Office Professional 2010 on a windows 7 machine.

the import of such CSV files (newlines in data-points) works only with Workbooks.Open and only with CSVs in the locale format (delimiter, text-delimiter), the Excel is used.
Set wb = Workbooks.Open(Filename:="C:\Users\axel\Desktop\test.csv", Local:=True)
aData = wb.Worksheets(1).UsedRange.Value
lRows = UBound(aData, 1)
lCols = UBound(aData, 2)
With ActiveSheet
.Range(.Cells(1, 1), .Cells(lRows, lCols)).Value = aData
End With
wb.Close
Greetings
Axel

Edit: the code previously provided was actually designed with the specific example you provided in mind, with 2 columns and a relatively small number of data in the source CSV. I have reviewed the code below to fit other possible scenarios - also including a number of optimizations for runtime efficiency as well.
Note I am not used to using the seeking facilities relating to the Open method that I am relying on here, and I still have a couple misgivings re the way they actually work in some contexts tbh, but after running a couple tests the code looks to work just fine.
Sub csvImportbis()
Dim s As String
Dim i As Long
Dim j As Long
Dim a() As String
myfile = FreeFile
i = 1
j = 1
'ENTER YOUR PATH/FILE NAME HERE
Open "YOUR_PATH/FILENAME" For Input As #myfile
Do Until EOF(myfile)
Do
Input #myfile, s
cur = Seek(myfile)
Seek myfile, cur - 1
i = i + 1
Loop While input(1, #myfile) <> vbLf
ReDim a(1 To i - 1, 1 To 10000)
i = 1
Seek #myfile, 1
Do Until EOF(myfile)
Input #myfile, a(i, j)
i = i + 1
If i > UBound(a, 1) Then
i = 1
j = j + 1
End If
If j > UBound(a, 2) Then
ReDim Preserve a(1 To UBound(a, 1), 1 To UBound(a, 2) + 10000)
End If
Loop
Loop
Close #myfile
sup = j
ReDim Preserve a(1 To UBound(a, 1), 1 To sup)
'QUALIFY THE RANGE WITH YOUR WORKBOOK & WORKSHEET REFERENCES
Range("A1").Resize(sup, UBound(a, 1)) = WorksheetFunction.Transpose(a)
End Sub

Related

Proceed to next empty cell if a condition is met

I have a sheet with Item names in the firs row.
I am using a Forloop to pass trough the cells in row 1 - i.
I use the value content of each cell to import a column from a .CSV file in the corresponding cell below it in row 2, by using j for that.
However, I have some .CSV files that are missing and I need to move on to the next cell in row 2, while moving on to the next cell in row 1. Basically skipping a column.
What I have so far is:
Dim FSO As Object
Dim Folder As Object
Dim File As String
Set FSO = CreateObject("Scripting.FileSystemObject")
Set Folder = FSO.GetFolder("C:\Users\Betty\AppData\Roaming\MetaQuotes\Terminal\B4D9BCD10BE9B5248AFCB2BE2411BA10\MQL4\Files")
For i = 2 To HCP.Cells(1, HCP.Columns.Count).End(xlToLeft).Column
Item = HCP.Cells(1, i).Value
FilePath = Folder & "\" & Item & "1440.CSV"
If Item = "" Or Dir(FilePath) = "" Then GoTo Continue
j = HCP.Cells(2, HCP.Columns.Count).End(xlToLeft).Column
With HCP.QueryTables.Add(Connection:="TEXT;" & FilePath, Destination:=HCP.Cells(2, j + 1))
.TextFileParseType = xlDelimited
.TextFileCommaDelimiter = True
.TextFileSpaceDelimiter = True
.TextFileColumnDataTypes = Array(9, 9, 9, 9, 9, 1, 9, 9, 9, 9)
.Refresh BackgroundQuery:=False
End With
Continue:
Next
I need the column index of j to be corresponding to the column index of i at all times.
I would avoid using GoTo Continue. Just check the negative of your statements before entering the loop. You are also missing some End If statement in both your question and solution.
I left comments showing where the code will skip to if either Item or Dir are blank. Same result, just cleaner code.
For i = 2 To HCP.Cells(1, HCP.Columns.Count).End(xlToLeft).Column
Item = HCP.Cells(1, i).Value
FilePath = Folder & "\" & Item & "1440.CSV"
If Item <> "" Or Dir(FilePath) <> "" Then 'Test Here
j = HCP.Cells(2, HCP.Columns.Count).End(xlToLeft).Column
If j <> i Then j = i
With HCP.QueryTables.Add(Connection:="TEXT;" & FilePath, Destination:=HCP.Cells(2, j))
.TextFileParseType = xlDelimited
.TextFileCommaDelimiter = True
.TextFileSpaceDelimiter = True
.TextFileColumnDataTypes = Array(9, 9, 9, 9, 9, 1, 9, 9, 9, 9)
.Refresh BackgroundQuery:=False
End With
End If 'Skips to here if either are blank.
Next i
I figured it out. This is what I am using now.
For i = 2 To HCP.Cells(1, HCP.Columns.Count).End(xlToLeft).Column
Item = HCP.Cells(1, i).Value
FilePath = Folder & "\" & Item & "1440.CSV"
If Item = "" Or Dir(FilePath) = "" Then GoTo Continue
j = HCP.Cells(2, HCP.Columns.Count).End(xlToLeft).Column
If j <> i Then j = i
With HCP.QueryTables.Add(Connection:="TEXT;" & FilePath, Destination:=HCP.Cells(2, j))
.TextFileParseType = xlDelimited
.TextFileCommaDelimiter = True
.TextFileSpaceDelimiter = True
.TextFileColumnDataTypes = Array(9, 9, 9, 9, 9, 1, 9, 9, 9, 9)
.Refresh BackgroundQuery:=False
End With
Continue:
Next
And this is the result:
Please, feel free to make any other suggestions.
Solution 3: Nested ForLoop with Nested IfStatement
For i = 1 To BS.Cells(1, BS.Columns.Count).End(xlToLeft).Column
For j = 1 To BS.Cells(2, BS.Columns.Count - 1).End(xlToLeft).Column
Item = BS.Cells(1, i).Value
FilePath = Folder & "\" & Item & "1440.CSV"
If ((Item <> "") Or (Dir(FilePath) <> "") And (i = j)) Then
With BS.QueryTables.Add(Connection:="TEXT;" & FilePath, Destination:=BS.Cells(2, j + 1))
.TextFileParseType = xlDelimited
.TextFileCommaDelimiter = True
.TextFileSpaceDelimiter = True
.TextFileColumnDataTypes = Array(9, 9, 9, 9, 9, 1, 9, 9, 9, 9)
.Refresh BackgroundQuery:=False
End With
End If
Next j
Next i
Solution 2: Nested Do Loop
Avoiding the "Continue" command, as it is not a VBA command.
For i = 2 To BS.Cells(1, BS.Columns.Count).End(xlToLeft).Column: Do
Item = BS.Cells(1, i).Value
FilePath = Folder & "\" & Item & "1440.CSV"
If Item = "" Or Dir(FilePath) = "" Then Exit Do
j = BS.Cells(2, BS.Columns.Count).End(xlToLeft).Column
If j <> i Then j = i
With BS.QueryTables.Add(Connection:="TEXT;" & FilePath, Destination:=BS.Cells(2, j))
.TextFileParseType = xlDelimited
.TextFileCommaDelimiter = True
.TextFileSpaceDelimiter = True
.TextFileColumnDataTypes = Array(9, 9, 9, 9, 9, 1, 9, 9, 9, 9)
.Refresh BackgroundQuery:=False
End With
Loop While False: Next i
Please, Notice the : Do at the end of the end of the For i.
If one of the following conditions, i.e. Item = "" or Dir(FilePath) = "" is False , then the Do loop is exited. True if the Loop While False: Next i condition for the Do loop is stated.
The two conditions may also be presented as:
For i = 2 To BS.Cells(1, BS.Columns.Count).End(xlToLeft).Column: Do
If Item <> "" Or Dir(FilePath) <> "" Then
'Do something...
Else: Exit Do
End If
Loop While True: Next i
Having the Or condition in the If Item = "" Or Dir(FilePath) = "" Then Exit Do is mandatory as i may be Value <> " ", but the FilePath to the file may be non-existent i.e. Dir(FilePath) = " ", which will spit out an error as the previous I had.
The If j <> i Then j = i in this case is mandatory, as the For i is stated as =2 To, meaning that the loop starts from column 2.
This can be avoided by stating the For i loop as For i = 1 To. However this was the initial loop to get the job done.
Further j can be stated as j = BS.Cells(2, i), obtaining the value of i for a column index.
However, the If j <> i Then j = i statement is advisable for further assurance purposes.
In further search more solutions emerged.
See Solution 3: Nested For Loop with Nested If Statement

VBA loop to pull cell addresses and sum them in a cell in another master spreadsheetl

I have the following code but it doesn't get to me to quite where I want:
Public Sub populateFile()
Dim wbk As Workbook
Dim fileName As String
Dim path As String
Dim pulledFormula As String
Dim pulledPath As String
Application.ScreenUpdating = False
Application.DisplayAlerts = False
Application.EnableEvents = False
Application.Calculation = xlCalculationManual
path = "C:\Users\Bob\Desktop\Source Files\"
fileName = Dir(path & "*.xlsx*")
Do While fileName <> ""
Set wbk = Workbooks.Open(path & fileName, UpdateLinks:=False)
j = 24
For i = 8 To 16
With Workbooks("MasterFile.xlsm").Sheets("Sheet1")
If Application.WorksheetFunction.CountA(Workbooks(fileName).Sheets("SummaryTab").Range(Cells(i - 1, j - 21), Cells(i - 1, j - 13))) > 0 Then
.Cells(i, j - 10).Value = fileName & vbNewLine & .Cells(i, j - 10).Value
For j = 15 To 23
pulledFormula = "+" & Application.WorksheetFunction.Index(Workbooks(fileName).Sheets("SummaryTab").Range("C6:K164"), _
Application.WorksheetFunction.Match(.Cells(i, 1), Workbooks(fileName).Sheets("SummaryTab").Range("A6:A164"), 0), _
Application.WorksheetFunction.Match(.Cells(6, j), Workbooks(fileName).Sheets("SummaryTab").Range("C5:K5"), 0)).Address(External:=True)
.Cells(i, j).Value = pulledFormula & .Cells(i, j).Formula
Next j
End If
End With
Next i
wbk.Close
fileName = Dir
Loop
Application.ScreenUpdating = True
Application.DisplayAlerts = True
Application.EnableEvents = True
Application.Calculation = xlCalculationAutomatic
End Sub
So I am trying to pull relevant (based on the index/match) cell addresses from a bunch of files in a directory. The next step is to iteratively add the addresses together with the + sign in the hope that in the end the master spreadsheet can have the sum of all the relevant cells from all the opened files in one cell (and to do this for a bunch of cells as well). The exact problem is that Excel refuses to evaluate the resulting concatenated formula. Would really appreciate any ideas on how to improve this code!
Thanks!
Note: I need to keep the cell addresses in the master file cells so other people can follow those addresses to the feeder files, so I can't use the Evaluate formula.
This sounds like a recalculation, or non-recalcualtion, issue.
You may use this simple macro to apply F2 + Enter on the currently selected range:
Sub ApplyF2()
Selection.Value = Selection.FormulaR1C1
End Sub
Can you modify that technique to suit your needs?

I am trying to write a script that will extract hourly values from a web page table into excel

The table is in 15 minute intervals but I only need the values that are every Hour. Is there a way to tell the vba to only read the values that are timestamped to HH:00:00?
Sub GetWaterLevels()
Dim URL As String
Dim qt As QueryTable
Dim ws As Worksheet
Set ws = ActiveSheet
'Clears previously loaded data
ActiveSheet.Range("C4:D85").ClearContents
URL = "http://waterdata.quinteconservation.ca/KiWIS/KiWIS?service=kisters&type=queryServices&request=getTimeseriesValues&datasource=0&format=html&ts_id=3641042&metadata=true&md_returnfields=station_name,ts_name,ts_unitname&&period=PT10H&width=600&height=400"
'Downloads the table into excel
Set qt = ws.QueryTables.Add( _
Connection:="URL;" & URL, _
Destination:=Range("C4"))
With qt
.RefreshOnFileOpen = False
.Name = "WaterLevels"
.FieldNames = True
.WebSelectionType = xlAllTables
.Refresh BackgroundQuery:=False
End With
End Sub
I learn something new everyday. The way you get that table is amazingly fast. My skills are no where near as good as yours, but with the dilemma you have the below is the only answer I can come up with. I don't know if you can select certain rows with your method or not on the import side. I looked at webtables property but don't know enough about it. What I would do is cull the table once it is on my sheet with the following code. When I use your code to import the table it puts the active data starting on row 10. You might have to adjust that to suit your needs.
Sub sortTable()
Dim lastRow As Long, firstRow As Long, myArray1, myArray2, myCounter As Long, i As Long
' When I ran your getWaterLevels script it put the data starting on row 10
firstRow = 10
lastRow = ActiveSheet.Range("C65536").End(xlUp).Row
myCounter = 1
ReDim myArray1(1 To 1)
ReDim myArray2(1 To 1)
Application.ScreenUpdating = False
For i = firstRow To lastRow
If Range("C" & i).Value <> "" And InStr(1, Range("C" & i).Value, ":00:00.") > 0 Then
ReDim Preserve myArray1(1 To myCounter)
ReDim Preserve myArray2(1 To myCounter)
myArray1(myCounter) = Range("C" & i)
myArray2(myCounter) = Range("D" & i)
myCounter = myCounter + 1
End If
Next i
Cells.Select
With Selection
.Font.Name = "Arial"
.Font.Size = 11
.Interior.Pattern = xlNone
.Interior.TintAndShade = 0
.Interior.PatternTintAndShade = 0
End With
Range("C10:D" & lastRow).Select
Selection.ClearContents
Range("A1").Select
myCounter = 10
For i = LBound(myArray1) To UBound(myArray1)
Range("C" & myCounter).Value = myArray1(i)
Range("D" & myCounter).Value = myArray2(i)
myCounter = myCounter + 1
Next i
Application.ScreenUpdating = True
End Sub

Workbook split hangs up here

The project: It deals with very sensitive HR/performance data, and I need to send 1000s of employees' data to their individual managers (about 100 managers who can only see their team's data, and no one else's), so I need about 100 files split (1 for each manager).
The file: - Many different tabs, separated by role. - First column is a unique identifier made by concatenating the Manager's name with the job title ex. John Stevens_Office Manager
The task: John Stevens will have team members in many different job roles, and needs all that data in one file, separated into tabs by job role.
Based on that sample data, the ideal macro would give me 3 files with 3 worksheets in each, and 1 row of data in each worksheet. However, I will settle for the worksheet being split into multiple files.
Here is my code.
Sub SplitWB()
Application.DisplayAlerts = False
Application.ScreenUpdating = False
ActiveWorkbook.Save
Dim OutputFolderName As String
OutputFolderName = ""
Set myDlg = Application.FileDialog(msoFileDialogFolderPicker)
myDlg.AllowMultiSelect = False
myDlg.Title = "Select Output Folder for Touchstone Files:"
If myDlg.Show = -1 Then OutputFolderName = myDlg.SelectedItems(1) & "\" Else Exit Sub
Set myDlg = Nothing
Application.CutCopyMode = False
'''''''''''''''''''''''''''''
'''''''''''''''''''''''''''''
Dim d As Object, c As Range, k, tmp As String, unique(500)
i = 0
With ActiveSheet
lastRow = .Cells(.Rows.Count, "A").End(xlUp).Row
End With
Set d = CreateObject("scripting.dictionary")
For Each c In Range(Cells(1, 1), Cells(lastRow, 1))
tmp = Trim(c.Value)
If Len(tmp) > 0 Then d(tmp) = d(tmp) + 1
Next c
For Each k In d.keys
Debug.Print k, d(k)
i = i + 1
unique(i) = k
Next k
UniqueCount = i
'start deleting
For i = 1 To UniqueCount
'Actions for new workbook
wpath = Application.ActiveWorkbook.FullName
wbook = ActiveWorkbook.Name
wsheet = ActiveSheet.Name
ActiveWorkbook.SaveAs Filename:=OutputFolderName & unique(i), FileFormat:=xlOpenXMLWorkbook, CreateBackup:=False
For j = 1 To lastRow
If Range("A" & j) <> "" And Range("A" & j) <> unique(i) Then
Rows(j).Delete
j = j - 1
End If
Next
'hide helper columns
' If HideC = False And DeleteC = True Then
Columns("A:D").Hidden = True
' End If
'
Range("E8").Select
'Select Instructions tab
'Worksheets("Guidelines").Activate
'Save new workbook
ActiveWorkbook.Close SaveChanges:=True
Workbooks.Open (wpath)
'ActiveWorkbook.Close False
Workbooks(wbook).Activate
Next
Application.DisplayAlerts = True
Application.ScreenUpdating = True
MsgBox ("Macro has completed successfully!" & vbNewLine & vbNewLine & "Generated files can be found in the following directory:" & vbNewLine & OutputFolderName)
End Sub
The code hangs at "If Range("A" & j) <> "" And Range("A" & j) <> unique(i) Then"
It is located about half-way into the code and the chunk starts with "For j = 1 To lastRow"
Please help. It will literally save me a day's work or more. Thanks!!!
Try
For j = lastRow to 1 step -1
If Range("A" & j) <> "" And Range("A" & j) <> unique(i) Then
Rows(j).Delete
End If
Next
instead. This allows you to loop backwards so when you delete a row it won't lose track of where you want j to be in the loop.

VBA code not working for a large number of iterations

For some reason this macro freezes EXCEL when I set the for home = X to XX loop to more than 10 iterations.
This code downloads a webpage into excel, extracts cells that contain either 'overall' or 'carried' and copy them into another sheet in the same workbook.
Thank you
Sub Macro1()
'
' Macro1 Macro
'
'
Dim home As Integer
Dim Calc_sheet As Worksheet
Dim score_count As Integer
Dim inspection_count As Integer
Dim output_rows As Integer
Dim output_columns As Integer
Dim date_columns As Integer
'Counting variables
score_count = 3
inspection_count = 8
'Output rows and columns starting values
output_rows = 3
output_columns = 3
date_columns = 8
For home = 20 To 23
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;http://www.XXXXXXXX.org.uk/directory/" & Sheets("Output").Cells(home, 1), Destination:=Range("$A$1") _
)
'.CommandType = 0
.Name = "Homes"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
For x = 20 To 250
Select Case Left(Cells(x, 1), 7)
'Is it a score?
Case Is = "Overall"
Cells(x, 1).Copy
Sheets("Output").Select
Cells(output_rows, output_columns).Select
ActiveSheet.Paste
output_columns = output_columns + 1
'Is it a date?
Case Is = "Carried"
Cells(x, 1).Copy
Sheets("Output").Select
Cells(output_rows, date_columns).Select
ActiveSheet.Paste
date_columns = date_columns + 1
Case Else
End Select
Sheets("Calc_sheet").Activate
Cells(x, 1).Activate
Next x
'Clean sheet
ActiveSheet.Cells.Delete
'Go back to top
Range("A1").Select
'Reset column count
output_columns = 3
date_columns = 8
output_rows = output_rows + 1
Next home
End Sub
I updated the code, try it again!
Try replacing your inner-loop with this one :
Dim wsC As Worksheet
Dim wsO As Worksheet
Set wsC = Worksheets("Calc_sheet")
Set wsO = Worksheets("Output")
For x = 20 To 250
yourContent = wsC.Cells(x, 1)
yourCase = Left(yourContent, 7)
Select Case yourCase
'Is it a score?
Case Is = "Overall"
wsO.Cells(output_rows, output_columns) = yourContent
output_columns = output_columns + 1
'Is it a date?
Case Is = "Carried"
wsO.Cells(output_rows, date_columns) = yourContent
date_columns = date_columns + 1
Case Else
End Select
Next x