How to import datatable into Excel without Loop using VB - vb.net

I am trying to export a SQL table result into Excel using blue prism. Currently it is done by exporting SQL results to collection and then collection to Excel. This is taking a longer time (10 mins) to export ~20K records. This is because the VBO uses for each loop.
I am trying to use some customized VB code to load the collection into excel using bulk update.
Can anyone help me on this?
I tried the below code but its not working
' Get to the cell
Dim ws As Object = GetWorksheet(handle, workbookname, worksheetname)
Dim origin As Object = ws.Range(cellref, cellref)
Dim cell As Object = origin
Dim colInd As Integer = 0, rowCount As Integer, rowInd As Integer = 0 ' Offsets from the origin cell
' Deal with the column names first
If includecolnames Then
For Each col As DataColumn In Collection.Columns
Try
cell = origin.Offset(rowInd, colInd)
Catch ex As Exception ' Hit the edge.
Exit For
End Try
SetProperty(cell, "Value", col.ColumnName)
colInd += 1
Next
rowInd += 1
End If
rowCount = Collection.Rows.Count
xlRange = ws.Range(cellref & ":H" & rowCount)
xlRange.Value = Collection
Error that I am getting is: Member not found. (Exception from HRESULT: 0x80020003 (DISP_E_MEMBERNOTFOUND))

I think that you're looking for this function of WorkSheet:
ws.SetRangeValues(RowIndex, ColumnIndex, Array(, ))
Just build two-dimensional array in memory based on your SQL data and call this once. It will fill the data from selected RowIndex and ColumnIndex all at once much faster then iterative way.

the way youre approaching this seems not very efficient, grabbing results from one place to put into another and then into another. here are two other alternatives you can try?
Automate command through CMD
sqlcmd -S . -d AzureDemo50 -E -s, -W -Q "SELECT * FROM dbo04.ExcelTest" > ExcelTest.csv
something like this more info (https://www.excel-sql-server.com/sql-server-export-to-excel-using-bcp-sqlcmd-csv.htm)
here you can run a command through cmd line with blue prism and then push the value where you want it to go without having to put the memory in blue prism at all.
Use a db object like OLEDB to parse a command straight into the DB to push the values into excel.
in both of these options the data never gets into Blueprism so of note, you dont get the opportunity to save, alter etc on the data in blueprism however it is much faster to operate.
test: made a db with 20k rows 5 columns data export with above cmd took 1.7 seconds to execute pluse extra 1/2 seconds for blueprism running it so 3-5 seconds total with smaller memory allocation that your current implementation.
is this something like you were looking for?

Thanks for your answers. I have added a new page called Write Collection - Fast and added code stage in Blue Prism MS Excel VBO. Pasted the below code
Dim ws As Object = GetWorksheet(handle, workbookname, worksheetname)
Dim sqlCon As New ADODB.Connection
Dim recordSet As New ADODB.Recordset
Dim iCol As Integer
sqlCon = New ADODB.Connection
sqlCon.ConnectionString = "driver={SQL Server};server=xxx\SQLEXPRESS;uid=zzzz;pwd=yyyy;database=testData"
sqlCon.ConnectionTimeout = 30
sqlCon.Open
recordSet.Open (SQL, sqlCon)
For iCol = 0 To recordSet.Fields.Count - 1
ws.Cells(1, iCol + 1).Value = recordSet.Fields(iCol).Name
Next
ws.Range("A2").CopyFromRecordset(recordSet)
recordSet.Close
sqlCon.Close

Related

How can I change the command text of an SQL connected table in Excel using VBA? [duplicate]

I have an Excel document that has a macro which when run will modify a CommandText of that connection to pass in parameters from the Excel spreadsheet, like so:
Sub RefreshData()
ActiveWorkbook.Connections("Job_Cost_Code_Transaction_Summary")
.OLEDBConnection.CommandText = "Job_Cost_Code_Transaction_Summary_Percentage_Pending #monthEndDate='" & Worksheets("Cost to Complete").Range("MonthEndDate").Value & "', #job ='" & Worksheets("Cost to Complete").Range("Job").Value & "'"
ActiveWorkbook.Connections("Job_Cost_Code_Transaction_Summary").Refresh
End Sub
I would like the refresh to not only modify the connection command but also modify the connection as I would like to use it with a different database also:
Just like the macro replaces the command parameters with values from the spreadsheet I would like it to also replace the database server name and database name from values from the spreadsheet.
A complete implementation is not required, just the code to modify the connection with values from the sheet will be sufficient, I should be able to get it working from there.
I tried to do something like this:
ActiveWorkbook
.Connections("Job_Cost_Code_Transaction_Summary")
.OLEDBConnection.Connection = "new connection string"
but that does not work. Thanks.
The answer to my question is below.
All of the other answers are mostly correct and focus on modifying the current connection, but I want just wanting to know how to set the connection string on the connection.
The bug came down to this. If you look at my screenshot you will see that the connection string was:
Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=True;Initial Catalog=ADCData_Doric;Data Source=doric-server5;Use Procedure for Prepare=1;Auto Translate=True;Packet Size=4096;Workstation ID=LHOLDER-VM;Use Encryption for Data=False;Tag with column collation when possible=False
I was trying to set that string with ActiveWorkbook.Connections("Job_Cost_Code_Transaction_Summary").OLEDBConnection.Connection = "connection string"
I was getting an error when i was simply trying to assign the full string to the Connection. I was able to MsgBox the current connection string with that property but not set the connection string back without getting the error.
I have since found that the connection string needs to have OLEDB; prepended to the string.
so this now works!!!
ActiveWorkbook.Connections("Job_Cost_Code_Transaction_Summary").OLEDBConnection.Connection = "OLEDB;Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=True;Initial Catalog=ADCData_Doric;Data Source=doric-server5;Use Procedure for Prepare=1;Auto Translate=True;Packet Size=4096;Workstation ID=LHOLDER-VM;Use Encryption for Data=False;Tag with column collation when possible=False"
very subtle but that was the bug!
I think you are so close to achieve what you want.
I was able to change for ODBCConnection. Sorry that I couldn't setup OLEDBConnection to test, you can change occurrences of ODBCConnection to OLEDBConnection in your case.
Try add this 2 subs with modification, and throw in what you need to replace in the CommandText and Connection String. Note I put .Refresh to update the connection, you may not need until actual data refresh is needed.
You can change other fields using the same idea of breaking things up then Join it later:
Private Sub ChangeConnectionString(sInitialCatalog As String, sDataSource As String)
Dim sCon As String, oTmp As Variant, i As Long
With ThisWorkbook.Connections("Job_Cost_Code_Transaction_Summary").ODBCConnection
sCon = .Connection
oTmp = Split(sCon, ";")
For i = 0 To UBound(oTmp) - 1
' Look for Initial Catalog
If InStr(1, oTmp(i), "Initial Catalog", vbTextCompare) = 1 Then
oTmp(i) = "Initial Catalog=" & sInitialCatalog
' Look for Data Source
ElseIf InStr(1, oTmp(i), "Data Source", vbTextCompare) = 1 Then
oTmp(i) = "Data Source=" & sDataSource
End If
Next
sCon = Join(oTmp, ";")
.Connection = sCon
.Refresh
End With
End Sub
Private Sub ChangeCommanText(sCMD As String)
With ThisWorkbook.Connections("Job_Cost_Code_Transaction_Summary").ODBCConnection
.CommandText = sCMD
.Refresh
End With
End Sub
You could use a function that takes the OLEDBConnection and the parameters to be updated as inputs, and returns the new connection string. It's similar to Jzz's answer but allows some flexibility without having to edit the connection string within the VBA code each time you want to change it - at worst you'd have to add new parameters to the functions.
Function NewConnectionString(conTarget As OLEDBConnection, strCatalog As String, strDataSource As String) As String
NewConnectionString = conTarget.Connection
NewConnectionString = ReplaceParameter("Initial Catalog", strCatalog)
NewConnectionString = ReplaceParameter("Data Source", strDataSource)
End Function
Function ReplaceParameter(strConnection As String, strParamName As String, strParamValue As String) As String
'Find the start and end points of the parameter
Dim intParamStart As Integer
Dim intParamEnd As Integer
intParamStart = InStr(1, strConnection, strParamName & "=")
intParamEnd = InStr(intParamStart + 1, strConnection, ";")
'Replace the parameter value
Dim strConStart As String
Dim strConEnd As String
strConStart = Left(strConnection, intParamStart + Len(strParamName & "=") - 1)
strConEnd = Right(strConnection, Len(strConnection) - intParamEnd + 1)
ReplaceParameter = strConStart & strParamValue & strConEnd
End Function
Note that I have modified this from existing code that I have used for a particular application, so it's partly tested and might need some tweaking before it totally meets your needs.
Note as well that it'll need some kind of calling code as well, which would be (assuming that the new catalog and data source are stored in worksheet cells):
Sub UpdateConnection(strConnection As String, rngNewCatalog As Range, rngNewSource As Range)
Dim conTarget As OLEDBConnection
Set conTarget = ThisWorkbook.Connections.OLEDBConnection(strConnection)
conTarget.Connection = NewConnectionString(conTarget, rngNewCatalog.Value, rngNewSource.Value)
conTarget.Refresh
End Sub
I would like to give my small contribute here to this old topic.
If you have many connections in your Excel file, and you want to change the DB name and DB server for all of them, you can use the following code as well:
It iterates through all connections and extracts the connection string
Each connection string is split into an array of strings
It iterates through the array searching for the right connection values to modify, the others are not touched
The it recompose the array into the string and commit the change
This way you don't need to use replace and to know the previous value, and the rest of the string will remain intact.
Also, we can refer to a cell name, so you can have names in your Excel file
I hope it can help
Sub RelinkConnections()
Dim currConnValues() As String
For Each currConnection In ThisWorkbook.Connections
currConnValues = Split(currConnection.OLEDBConnection.Connection, ";")
For i = 0 To UBound(currConnValues)
If (InStr(currConnValues(i), "Initial Catalog") <> 0) Then
currConnValues(i) = "Initial Catalog=" + Range("DBName").value
ElseIf (InStr(currConnValues(i), "Data Source") <> 0) Then
currConnValues(i) = "Data Source=" + Range("DBServer").value
End If
Next
currConnection.OLEDBConnection.Connection = Join(currConnValues, ";")
currConnection.Refresh
Next
End Sub
This should do the trick:
Sub jzz()
Dim conn As Variant
Dim connectString As String
For Each conn In ActiveWorkbook.Connections
connectString = conn.ODBCConnection.Connection
connectString = Replace(connectString, "Catalog=ADCData_Doric", "Catalog=Whatever")
connectString = Replace(connectString, "Data Source=doric-server5", "Data Source=Whatever")
conn.ODBCConnection.Connection = connectString
Next conn
End Sub
It loops every connection in your workbook and change the Connection String (in the 2 replace statements).
So to modify your example:
ActiveWorkbook.Connections("Job_Cost_Code_Transaction_Summary").ODBCConnection.Connection = "new connection string"
I assume it is necessary for your to keep the same connection-name? Otherwise, it would be simplest to ignore it and create a new Connection.
You might rename the connection, and create a new one using the name:
ActiveWorkbook.Connections("Job_Cost_Code_Transaction_Summary").Name = "temp"
'or, more drastic:
'ActiveWorkbook.Connections("Job_Cost_Code_Transaction_Summary").Delete
ActiveWorkbook.Connections.Add "Job_Cost_Code_Transaction_Summary", _
"a description", "new connection string", "command text" '+ ,command type
Afterwards, Delete this connection and reinstate the old connection/name. (I am unable to test this myself currently, so tread carefully.)
Alternatively, you might change the current connections SourceConnectionFile:
ActiveWorkbook.Connections("Job_Cost_Code_Transaction_Summary").OLEDBConnection.SourceConnectionFile = "..file location.."
This typically references an .odc file (Office Data Connection) saved on your system that contains the connection details. You can create this file from the Window's Control Panel.
You haven't specified, but an .odc file may be what your current connection is using.
Again, I am unable to test these suggestions, so you should investigate further and take some precautions - so that you won't risk losing the current connection details.

Multi criteria filter in VBA (Not equal to)

I am using the below code in Blue prism for filtering in excel for multi criteria.
But i am not able to filter multi criteria for Not equal to scenario.
Dim wb As Object
Dim excel as Object
Dim range as Object
Try
wb = GetWorkbook(Handle, Workbook)
excel = wb.Application
range = excel.Range(FRange)
Dim listOfValues as Array
listOfValues = Split(FCriteria,";")
wb.worksheets(Worksheet).select
range.select
range.Autofilter(FCol,listOfValues,7)
Success = True
Catch e As Exception
Success = False
Message = e.Message
Finally
wb = Nothing
End Try
Please help me tweaking the script
I'm almost sure that there is no filter option to set a "negative list". You can specify either a (positive) list of values (this is what your code does so far, for this you have to set the 7 as third parameter), or you can give a maximum of 2 individual criteria (in Excel, choose "Custom Filter" to set them.
You should play with the filter directly in Excel and try to set it like you want. Once you are satisfied with it, clear the filter, record a macro and repeat the filtering. Go to the VBA editor and see what's in there. It is straightforward to translate this into C# code.
But:
It's not possible to set any filtering by code (neither C# nor VBA) that you cannot set via the Excel GUI
I would question what you are trying to do. Since you are using Blue Prism, you should be trying to access the underlying data in a BP Collection(VB DataTable), rather than applying a filter, which is a visual tool for humans to further play with the interface. The robot will still have to do something with the filtered data, and it far easier to write code to proceed with data during the loop.
Otherwise use the Filter Collection Page of the 'Utilities - Collection Manipulation' VBO to get a filtered collection.
Also you are using VBA Split function, when you should use Split in VB as a method of the String.
Try this for a new page in the 'Utilities - Collection Manipulation' VBO(untested):
Dim NewRow As DataRow
Collection_Out = Collection_In.Clone
Dim Select_Concat As String
Select_Concat = "NOT(" & fieldName & " = '" & [String].Join("' OR " & fieldName & " = '", FCriteria.Split(";"c)) & "')"
For Each parentRow As DataRow In Collection_In.Select(Select_Concat)
NewRow = Collection_Out.NewRow
For Each c As DataColumn In NewRow.Table.Columns
NewRow(c.ColumnName) = parentRow(c.ColumnName)
Next c
Collection_Out.Rows.Add(NewRow)
Next parentRow
NewRow = Nothing
Collection_In = Nothing
Inputs: Collection_In(Collection), fieldName(Text), FCriteria(Text)
Outputs: Collection_Out(Collection)
You first need to get the entire range into an unfiltered Collection(which will be your Collection_In to this page, and then get the filtered Collection out....

Update a table linked to a PowerPivot datamodel using EPPlus and it corrupts the datamodel

Using EPPlus I read an XLSX file.
I replace the data in a table and set the table range.
When I open the resulting spreadsheet I get an error:
"We found a problem with some content in 'MySpreadsheet.xlsx'. Do you want us to try to recover as much as we can?" -- I click Yes and I get another error:
"Excel was able to open the file by repairing or removing the unreadable content. Removed Part: Data store"
The error only happens after I add this table to a PowerPivot data model.
[EDIT] - I created a win forms app that reproduces this problem. You
can download it at here
I found the problem but don't know how
to fix it.
rename the xlsx to zip
Open the zip and browse to the xl\workbook.xml file
Look for the node collection.
Notice how EPPlus changes the <definedNames> collection to use absolute cell addresses.
Excel: <definedName name="_xlcn.LinkedTable_MyDate" hidden="1">MyDate[]</definedName>
EPPlus: <definedName name="_xlcn.LinkedTable_MyDate" hidden="1">'MyDate'!$A$2:$A$5</definedName>
If I modify this line after EPPlus is done saving then I can pull it
up in Excel without corrupting the Data Model.
I tried changing the WorkbookXml but it is happening when the
ExcelPackage.Save method runs.
For Each node In pck.Workbook.WorkbookXml.GetElementsByTagName("definedNames")(0).ChildNodes
node.innerText = "MyDate[]"
Next
Any ideas?
Try this first: create a spreadsheet with one table in it. Name the worksheet and table "DateList". Save it and run the below code on it -- it will work.
Then do this: open the same spreadsheet and add the DateList table to a pivottable data model. Save it and run the below code on it -- it will fail.
Here's some code from my MVC Controller -- only the relevant bits:
Public Class ScorecardProgressReportDatesVM
Public Property WeekRange As Date
End Class
Public Function GetScorecardProgressReport(id As Integer) As ActionResult
Dim contentType As String = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
Dim DateList As New List(Of ScorecardProgressReportDatesVM)
DateList.Add(New ScorecardProgressReportDatesVM With {.WeekRange = CDate("Jan 1, 2015")})
DateList.Add(New ScorecardProgressReportDatesVM With {.WeekRange = CDate("Jan 1, 2015")})
Dim templateFile As New IO.FileInfo("c:\test.xlsx")
Dim ms As New IO.MemoryStream
Using pck As New ExcelPackage(templateFile)
ExtendTable(pck, "DateList", DateList)
pck.SaveAs(ms)
ms.Position = 0
End Using
Dim fsr = New FileStreamResult(ms, contentType)
fsr.FileDownloadName = "StipProgress.xlsx"
Return fsr
End Function
Private Sub ExtendTable(package As ExcelPackage, tableName As String, newList As Object)
Dim ws As OfficeOpenXml.ExcelWorksheet
ws = package.Workbook.Worksheets(tableName)
Dim OutRange = ws.Cells("A1").LoadFromCollection(newList, True)
Dim t = ws.Tables(tableName)
Dim te = t.TableXml.DocumentElement
Dim newRange = String.Format("{0}:{1}", t.Address.Start.Address, OutRange.End.Address)
te.Attributes("ref").Value = newRange
te("autoFilter").Attributes("ref").Value = newRange
End Sub
In order to fix this we had to change the EPPlus v4.04 source code.
ExcelWorkbook.cs # line 975 was changed
//elem.InnerText = name.FullAddressAbsolute; This was causing issues with power pivot
to
elem.InnerText = name.FullAddress; //Changed to full address... so far everything is working

Excel drop down values from a SQL Server source

I am trying to get a cell drop-down values in Excel from a SQL Server. I don't want to use the method of putting all the data to another sheet and the use data validation to control the drop down values. That always give my a bunch of empty lines towards the end since I want to make sure I have room for any addition in the DB.
Is there a way to retrieve the drop-down values directly from SQL Server? Using a statement something like:
Select name from employees
Thanks for your help...
Use ADODB to retrieve the values you want, and use the retrieved values to populate a dropdown shape in Excel which you can create dynamically.
In a similar situation, since the source data was basically static, I populated a global array from an ADODB recordset when the application started and used that array when populating the items in the dropdown. Here's a snippet of that code:
Dim InstrumentIDs() As String
Dim InstrumentIDReader As Integer
Dim InstrumentIDCount As Integer
Public PositionRange As String
Public Sub GetInstrumentIDs()
'
'Populate InstrumentIDs array from current contents of Instrument table in EMS database
'
Dim conn As New ADODB.Connection
Dim rs As New ADODB.Recordset
Dim sql As String
Dim loader As Integer, sn As String
InstrumentIDReader = 0
On Error GoTo GetInstrumentError
conn.ConnectionString = "Provider=sqloledb; Data Source=myServer; Initial Catalog=myDatabase; User ID=myUser;Password=myPassword"
conn.Open
sql = "Select Count([SerialNo]) As [Number] From [Instrument]"
rs.Open sql, conn, adOpenStatic
InstrumentIDCount = CInt(rs![Number])
ReDim InstrumentIDs(InstrumentIDCount - 1)
rs.Close
sql = "Select [SerialNo] From [Instrument] Order By [SerialNo]"
rs.Open sql, conn, adOpenForwardOnly
loader = 0
rs.MoveFirst
Do While Not rs.EOF
sn = CStr(rs![SerialNo])
InstrumentIDs(loader) = sn
loader = loader + 1
rs.MoveNext
Loop
rs.Close
conn.Close
Set rs = Nothing
Set conn = Nothing
Exit Sub
GetInstrumentError:
MsgBox "Error loading instruments: " & Err.Description
End Sub
You must set a reference to Microsoft ActiveX Data Objects m.n Library (latest version on my computer is 2.8) from Tools > References in VBA editor.
See article
http://www.thespreadsheetguru.com/blog/2014/5/14/vba-for-excels-form-control-combo-boxes for tips on how to manage dropdown boxes in Excel.
You can use the MS Query Wizard in Excel to store a query and use it's data any time.
This this link for details http://www.techrepublic.com/article/use-excels-ms-query-wizard-to-query-access-databases/

Runtime COMException Unhandeled

I'm working on an app that writes to excel. The following piece f code is working properly ( it fills the requested cell) but generating a run time exception that I can't get rid of.
For i = 1 To 1000 Step 1
If Not (cPart.Range("A" & i).Value = Nothing) Then
If (cPart.Range("L" & i).Value = Nothing) Then
cPart.Range("L" & i).Interior.ColorIndex = 3
End If
i = i + 1
End If
Next
the exception is: COMException was unhandled :Exception from HRESULT: 0x800A01A8
any help?
That HRESULT means Object Required. So it seems like one or more of the objects you try to operate on don't exist but as the code is written at the moment, it's difficult to be sure which it is. An immediate concern though is that you're comparing values to Nothing, in VB.Net you're supposed to use Is Nothing to check for that. Also, you've already set up the For loop to go from 1 to 1000, with a step of 1 (which you don't need to include since it's the default) but you're then doing i = i + 1 which looks like a mistake?
So fixing that and splitting it up into it's parts it might give you a better idea to what's not working:
For i = 1 To 1000
Dim aRange As Object = cPart.Range("A" & i)
If aRange IsNot Nothing AndAlso aRange.Value IsNot Nothing Then
Dim lRange As Object = cPart.Range("L" & i)
If lRange IsNot Nothing AndAlso lRange.Value Is Nothing Then
Dim interior As Object = lRange.Interior
If interior IsNot Nothing Then
interior.ColorIndex = 3
End If
End If
End If
Next
I've declared the new objects as Object which might need to be changed to the correct data types (depending on your project settings).
Hopefully you should now be able to run through the code without error and you should also be able to step through the code and find that one of the new objects (aRange, lRange and interior) is Nothing at some point when it shouldn't be which will show you why it threw that error before.
Another advantage to splitting up the code like this is that you'll now be able to dispose of the Excel objects properly so that the Excel instance can shut down cleanly. See this Q&A for info: Excel.Range object not disposing so not Closing the Excel process