I just installed Postgres and am trying some simple examples. I used ADO in the past.
My table would be
create table zonk (nameone varchar primary key, dataone varchar not null);
I have a single row in the table where the expected length of the dataone field is around 500.
My VBA test program is the following
Option Explicit
Sub TestPostgres()
Dim objDb_con As ADODB.Connection
Dim rsdatatype As ADODB.Recordset
Dim sql As String
Dim str As String
Dim connString As String
Set objDb_con = New ADODB.Connection
Set rsdatatype = New ADODB.Recordset
connString = "Driver={PostgreSQL UNICODE};Server=123.456.789.010;Database=cowbell;UID=charlotte;Pwd=spider;" ' original
connString = connString & "Extended Properties=""text;HDR=NO;FMT=Delimited;"";" ' some advice
objDb_con.Open connString
sql = "select dataone from zonk where nameone = 'new.forecast';"
rsdatatype.Open sql, objDb_con, adOpenKeyset, adLockpessimistic
If rsdatatype.EOF = False Then
Debug.Print rsdatatype("dataone")
rsdatatype.Close
End If
objDb_con.Close
End Sub
The problem is that whatever value comes out is truncated, ADO reports data type 202 and returned length is 320. There was another similar problem elsewhere where the advice was to have HDR=NO
connString = connString & "Extended Properties=""text;HDR=NO;FMT=Delimited;"";"
in the connection string. (No joy)
Thanks in advance for any suggestions.
It's all in the Connection String.
First, connection string extended properties as mentioned in the question have to do with reading Excel tables with ADO and are not relevant for Postgres.
However, additional parameters for longer text field sizes are available.
connString = "Driver={PostgreSQL UNICODE};Server=123.456.789.010;Database=cowbell;UID=charlotte;Pwd=spider;MaxVarcharSize=1000000;"
Fields in the sample table are all varchar.
More info here: https://github.com/ClickHouse/ClickHouse/issues/9363
Related
I have some very large .csv files that I want to query and pull entries out. I've set this up using ADO in Excel 2016 successfully, or so it seemed, and remarkably fast. I am looking to query a field ID using;
SELECT * FROM <file> WHERE ID = #.
This has worked fine up until the ID numbers which contain letters, i.e. 960545H4. These appear much further down the file, around ~400k rows in.
I believe the problem is that excel is assuming this field (under the header ID) is numeric based one the first x number of entries. I want this to be set to text.
Is there anyway to set the datatype of the field/column so that I can query successfully?
Sub testSQL()
Dim xlcon as ADOB.Connection
Dim xlrs as ADOB.RecordSet
Dim nextRow as Integer
Dim datafilepath as String
Dim datafilename as String
Set xlcon = New ADOB.Connection
Set xlrs = New ADOB.RecordSet
datafilepath = "U:\Common\"
datafilename = "test_file"
xlcon.Provider = "Microsoft.Jet.OLEDB.4.0"
xlcon.ConnectionString = "Data Source=" & datafilepath & ";" & "Extended Properties=""text;HDR=Yes;FMT=Delimited,"""
xlcon.Open
xlrs.Open "SELECT * FROM [" & datafilename & ".csv] WHERE ID = '023487562HH'", xlcon
'the rest of the code...
I was hoping something slong the lines of;
SELECT * FROM [file] WHERE CStr(ID) = 34897562FD
but this doesn't seem to work.
Any help is greatly appreciated!!
Thanks to Storax's recommendation on using a schema file I have got this working.
I created a schema file in the same location as the file. The file looked like;
[test_file.csv]
Format=CSVDelimited
ColNameHeader=True
MaxScanRows=0
I can now return the IDs I am looking for! Many thanks to Storax for this solution!
One could take Killuminati's solution and add also the field names as needed. The parameter MaxScanRows just makes sure that the OLEDB driver will scan the whole file as stated also in the documentation
The data types of the fields can also be determined. Use the
MaxScanRows option to indicate how many rows should be scanned when
determining the column types. If you set MaxScanRows to 0, the whole
file is scanned. The MaxScanRows setting in Schema.ini overrides the
setting in the Windows Registry, file by file.
In order to declare the data type for each field one could use a schema.ini like that
[test_file.csv]
Format=Delimited(;)
DecimalSymbol=,
ColNameHeader=False
Col1=Field1 Text
Col2=Field2 Text
Col3=Field3 Text
Col4=StartDate Date
Col5=EndDate Date
Col6=Price Currency
The entry after the equal sign is name of the field resp. column. If the file contains a header it can be the same but does not have to but it is required.
Code could look like that
Sub ADO()
Dim rs As New ADODB.Recordset
Dim conn As New ADODB.Connection
Dim myPath As String
myPath = ThisWorkbook.Path & "\TextFiles\"
conn.Open "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & myPath & ";Extended Properties=""text;HDR=No;FMT=Delimited()"";"
With rs
.ActiveConnection = conn
.CursorType = adOpenKeyset
.LockType = adLockOptimistic
.Open "SELECT * FROM [test_file.csv]"
.AddNew
.Fields("Field1") = "Doe"
.Fields("Field2") = "John"
.Fields("Field3") = "123456"
.Fields("StartDate") = Date
.Fields("EndDate") = "05.10.2018"
.Fields("Price") = 1234.56
.Update
End With
conn.Close
End Sub
Hi I am working in a excel file with 46 pivot tables. The code below changes the connection of each pivot table to a new existing connection.
Sub changeConnection()
Dim pTable As Variant
Dim sheet As Variant
Dim workBookName As String
workBookName = "filename.xlsm"
For Each sheet In Workbooks(workBookName).Worksheets
For Each pTable In sheet.PivotTables
pTable.changeConnection Workbooks(workBookName ).Connections("connection name")
Next pTable
Next sheet
End Sub
I want everything to stay the same for my pivot tables but I want a password on the file that I am connected to. Since excel can not do this I used ADO to access a password protected excel file.
Public Function readFile()
Dim xl As Object
Dim conn As New ADODB.connection
Dim recSet As ADODB.Recordset
Dim conString As String
Dim wkbName As String
Dim SQL As String
Dim DBPath As String
'Path to excel file
DBPath = "path\to\file.xlsm"
Set xl = GetObject(DBPath)
'Name of table
wkbName = "[IS$]"
conString = "Provider=MSDASQL.1;DSN=Excel Files;DBQ=" & DBPath & ";HDR=Yes';"
'Query
SQL = "select * from " & wkbName
'Open connection
conn.Open conString
'Itterate over record set
Set recSet = New ADODB.Recordset
recSet.Open SQL, conn
'Print out col1 from table
Do Until recSet.EOF
'process your data here
Debug.Print recSet!ISData
recSet.MoveNext
Loop
End Function
The code above will access a table inside of the password protected workbook stored externally. Then using a record set print out in debug all the items.
I want to essentially use my workaround in the second snippet of code so I can replace all my pivot table connections so my data source can have a password on it. All my pivot tables point to the same connection so using the same connection won't cause issues.
Thank in advance and please comment if I should clarify anything.
IIR there isn't a data provider that can do this. That driver will give an error to the effect of “could not decrypt file” even if you attempt to store the password in the connection string.
The second bit of code is basically a hack to get around this and it relies on Excel to manage the credential prompt from the user. It does not solve the fact that you can't supply a password in your connection string - it is a work-around. Given that you can't supply a connection string that works to ADO, you're not going to be able to supply it to the stored connection string either.
I would suggest using an actual database for the back end instead of an Excel file. This will give you much more flexibility in managing user access.
When I try to do an import from an Excel document the comments get truncated. I have checked the usually issue that the Table would be limited but is set as:
Comments ... nvarchar(MAX)
Sample of the code, please note even running the code in Debug mode I can see the parameter is truncated before it even goes to the stored procedure.
Dim excelConnectionString As String = (Convert.ToString("Provider=Microsoft.ACE.OLEDB.12.0; Data Source=") & vFileNameFolder) + "; Extended Properties='Excel 12.0;HDR=YES;IMEX=1;';"
'#### Upload, Rename and save file
'#### Open Excel to Parse here
Dim ds As New DataSet
Dim oleda As New OleDbDataAdapter()
Dim cmdExcel As New OleDbCommand()
'#### End - Open Excel to Parse here
Dim vActionRef As String = ""
Try
Dim excelConnection As New OleDbConnection(excelConnectionString)
With cmdExcel
.CommandText = "Select * from [Portal$A1:BB9999]" 'Names we want to select and the name of the sheet
.CommandType = CommandType.Text
.Connection = excelConnection
End With
excelConnection.Open()
oleda = New OleDbDataAdapter(cmdExcel)
oleda.Fill(ds, "dataExcel")
If ds.Tables("dataExcel").Rows.Count > 0 Then
'#### Stored procedure details
Dim connection As SqlConnection
Dim commandSQL As New SqlCommand
Dim FRAUPRN As String = ""
Dim ConnectionString As String = System.Configuration.ConfigurationManager.ConnectionStrings("SQLLocal").ToString()
'########### End - Stored procedure details
'Set date once
Dim vDate As Date
vDate = DateTime.Now.AddDays(0)
connection = New SqlConnection(ConnectionString)
connection.Open()
'Dims for error handling and checking for invalid characters
Dim iImported As Integer
For j As Integer = 0 To ds.Tables("dataExcel").Rows.Count - 1 ' counted rows so loop through, ignores first row with names in
If (IsDBNull(ds.Tables("dataExcel").Rows(j)("UPRN"))) Then
'skip
Else
iImported = iImported + 1
'Bring the data across, the rows(i)("xxx") must match a name on the Excel sheet but DOES NOT have to be in order
With commandSQL
.Parameters.Clear()
.Connection = connection
.CommandText = "spAddCSVDataLine" 'Stored procedure here
If Trim(ds.Tables("dataExcel").Rows(j)("Comments")) = "0" Then
.Parameters.AddWithValue("Comments", " ")
Else
' .Parameters.AddWithValue("Comments", If(IsDBNull(ds.Tables("dataExcel").Rows(j)("Comments")), "", Trim(ds.Tables("dataExcel").Rows(j)("Comments"))))
Dim vComments As String
vComments = ds.Tables("dataExcel").Rows(j)("Comments")
.Parameters.AddWithValue("Comments", vComments)
Session.Item("Comments") = Session.Item("Comments").ToString & "//" & vComments
End If
I have looked at similar questions such as ADO is truncating Excel data which talks about numerical issues but am struggling to find the reason why I am losing data before I export the data. 'Common sense' says excel is not passing over more than 255 characters but then this is programming!
I've had all sorts of problems with the JET/Ace DB engine truncating and doing other sorry-ass guesses at data type. Check out this Microsoft article that talks a bit about how JET uses only the first 8 records to determine field length (http://support.microsoft.com/kb/189897/en-us). You can edit a registry setting to change how many records it will scan to determine field length, but the results still seem to be hit or miss for folks.
You might also find some luck in creating a dummy record at the top of the excel sheet that contains a comment with the maximum number of characters of any of your comments. Then just delete that one record after it comes through. Again... results seem to be mixed here.
I'm getting the email address in encoded format like "annie#h ꇟ|（ƓƓⲘ" and i'm catching it in a string then not able to store it in server database. So how do i decode it to normal email address. or not getting which type of Encoding is it(base64/ascii/ect..).and the column type is long varchar,
machine i'm using it windows xp. I'm pulling my hair out.
Please help..
i caught the answer but i'm not sure is this the right way to do.
Now I read the record from ADODB.Recordset instead of Dataset.
is this the right way to read data.
I don't know why Dataset give me the wired email address,But using Record set i solve the issue
Here is the code sample that i use now
Dim str_query = "select * from table"
Dim objRS= New ADODB.Recordset
objRS= Cn.Execute(str_query )
Do While Not objRS.EOF
For k = 0 To objRS.Fields.Count - 1
Debug.Print objRS(k).Name & ": " & objRS(k).Value
Next
Debug.Print "_____"
objRS.MoveNext
Loop
And previously I used code this
Dim str_query = "select * from table"
Dim objRS= New ADODB.Recordset
objRS= Cn.Execute(str_query )
Dim ds As DataSet = New DataSet()
Dim da As OleDbDataAdapter = New OleDbDataAdapter
da.Fill(ds, objRS, "my_table")
For Each dr As DataRow In ds.Tables("my_table").Rows
Next
I am using the following code to select from one database and need to insert into another database.. Please suggest me the code:
Code :
'Connection for Original database from where i have to import
Dim constrOrg As String = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + strPath
Dim conn_OrgDB As New OleDb.OleDbConnection(constrOrg)
'Connection for my database to where i have to import
Dim App_Path = System.AppDomain.CurrentDomain.BaseDirectory()
Dim constr As String = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + App_Path + "Mydb.accdb"
Dim cnnOLEDB As New OleDb.OleDbConnection(constr)
Dim strSelInv_one As String = ("SELECT * FROM INV_ONE where (DOCU_DT>=#stdt AND DOCU<=#enddt) ")
Dim comm_inv_one As OleDb.OleDbCommand = New OleDb.OleDbCommand(strSelInv_one, conn_OrgDB )
comm_inv_one.Parameters.AddWithValue("#stdt", StTime)
comm_inv_one.Parameters.AddWithValue("#enddt", Endtime)
dr = comm_inv_one.ExecuteReader
Do While dr.Read = True
Dim strInsInv1 As String = "INSERT INTO INVOICE_ONE(CODE_NO,LAY,..) SELECT CODE_NO,LAY,... FROM INV_ONE where (CODE_NO=#code)"
Dim comm_Insinv1 As OleDb.OleDbCommand = New OleDb.OleDbCommand(strInsInv1, cnnOLEDB)
comm_Insinv1.Parameters.AddWithValue("#code", Code_no)
comm_Insinv1.ExecuteNonQuery()
Loop
'Here INVOICE_ONE table belongs cnnOLEDB connection obj of one database and INV_ONE table belongs to conn_OrgDB connection object of another database..
' How to use the 2 connection object ? If i use only one connection object i.e. cnnOLEDB it gives the following Error:
" The Microsoft Office Access database engine cannot find the input table or query 'INV_ONE'. Make sure it exists and that its name is spelled correctly. "
Please suggest me the code..
Thank you
try this code, obviously don't make it as long as his ( basically wack out stuff you don't need and imput your code, now ....
in addition to :
Dim connectionStr = Constants.Input.MDB.CONNECTION_STRING & _
"Data Source=" & dbFullPath & ";"
Dim connection As New System.Data.OleDb.OleDbConnection(connectionStr)
in addition use
Dim connectionStr2 = "Other conection string"
Dim connection2 As New System.Data.OleDb.OleDbConnection(connectionStr)
and import it into his code.
link:
Selecting data in MS Access with vb.net is very slow. Am I doing it right?
I am sorry if this won't help, since i don't have an access db i cannot further test this.
Reguards.
My suggestion would be use to Access to create Linked Tables in the target database that point to the corresponding tables in the source (production) database. Then you can at least manipulate both sets of tables using the same connection, thereby giving you the option of doing
INSERT INTO INVOICE_1 (codeno, lchlndt, ...)
SELECT codeno, lchlndt, ... FROM INV_ONE ...