I am new to VB.net and I would like to convert and display UnixStamp data in a new column.
Variant DatGridView = datasource is Dataset.
I can create empty columns within an SQL query (DataGridView_Dataset), unfortunately I can't use direct data conversion into new columns using my own function. Error see.SQL Error Code
The function works independently. see. Working UnixStampFunction
I got the result of 56,000 sentences in 7 seconds, without getting date and time values from UnixTimeStamp
Is there a solution for using udf in SQL Statement?
DataGridView variant - odbcExecuteRead Solution
Using the given code is not a problem to display eg 10 sentences (10 sentences result), but if the records are more than about 100 (around 50 thousand by month), an error like this will be displayed (Managed Debug Helper ContextSwitchDeadlock: The module CLR could not go out of context COM 0xb45680 to context 0xb455c8 for 60 seconds.).
Unchecking the ContextSwitchDeadlock option by
Debug > Windows > Exception Settings in VS 2019
I got the result of 56 000 record in awfull 228 seconds.
Is it possible to optimize the code or is it possible to use another solution?
Code:
Public Class Form1
Public strDateTime, strDate, strTime As String
Public x As Integer =0
Public y As Integer =0
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Try
Dim dgvDT As New DataGridView
Dim odbcConn As OdbcConnection
Dim odbcComm As OdbcCommand
Dim odbcAdapter As New Odbc.OdbcDataAdapter
Dim odbcDataset As New DataSet
Dim odbcDataTable As DataTable
Dim strConn, strSQL, strSQL1 As String
dgvDT.Location = New Point(382, 2)
dgvDT.Width = 360
dgvDT.Height = 600
Me.Controls.Add(dgvDT)
strConn = "Driver=Firebird/InterBase(r) driver;User=;Password=;DataSource=...." '
odbcConn = New OdbcConnection(strConn)
odbcConn.Open()
strSQL = "SELECT TEST.UNIXTIMESTAMP, " & "'dd.mm.yyyy" & "'" & "AS Date_ , " & "'hh:mm:ss" & "'" & "AS Time_ " _
& "From TEST " _
& "Where TEST.UNIXTIMESTAMP > 1646092800 " _ '1.3.2022
& "Order By TEST.ID "
strSQL1 = "SELECT TEST.UNIXTIMESTAMP, UnixTimestampToDateOrTime(TEST.UNIXTIMESTAMP,1) As Date_, " & "'hh:mm:ss" & "'" & "AS Time_ " _
& "From TEST " _
& "Where TEST.UNIXTIMESTAMP > 1646092800 " _ '1.3.2022
& "Order By TEST.ID "
odbcComm = New OdbcCommand(strSQL, odbcConn)
'odbcComm = New OdbcCommand(strSQL1, odbcConn)
odbcAdapter.SelectCommand() = odbcComm
odbcAdapter.Fill(odbcDataset, "TEST")
odbcDataTable = odbcDataset.Tables("TEST")
dgvDT.DataSource = odbcDataTable
dgvDT.Columns(0).HeaderText = "UnixTimeStamp"
dgvDT.Columns(1).HeaderText = "Date"
dgvDT.Columns(2).HeaderText = "Time"
dgvDT.Visible = True
Catch ex As Exception
MessageBox.Show("Error: " & ex.Message, "Error")
End Try
End Sub
Private Sub Button3_Click(sender As Object, e As EventArgs) Handles Button3.Click
Try
Dim dgvDT1 As New DataGridView
Dim odbcConn1 As OdbcConnection
Dim odbcComm1 As OdbcCommand
Dim odbcDR As OdbcDataReader
Dim x As Integer = 0
Dim y As Integer = 0
Dim strConn1, strSQL, strSQL2 As String
dgvDT1.Location = New Point(382, 2)
dgvDT1.Width = 360
dgvDT1.Height = 600
For i As Integer = 0 To 2
Dim dgvNC As New DataGridViewTextBoxColumn
dgvNC.Name = "Column" & i.ToString
dgvDT1.Columns.Add(dgvNC)
Next
dgvDT1.Columns(0).HeaderText = "UnixTimeStamp"
dgvDT1.Columns(1).HeaderText = "Date"
dgvDT1.Columns(2).HeaderText = "Time"
dgvDT1.ReadOnly = True
dgvDT1.AllowUserToAddRows = False
dgvDT1.AllowUserToDeleteRows = False
strSQL2 = "SELECT TEST.UNIXTIMESTAMP " _
& "From TEST " _
& "Where TEST.UNIXTIMESTAMP > 1646092800 " _
& "Order By TEST.ID "
strConn1 = "Driver=Firebird/InterBase(r) driver;User=;Password=;DataSource="
odbcConn1 = New OdbcConnection(strConn1)
odbcConn1.Open()
odbcComm1 = New OdbcCommand(strSQL2, odbcConn1)
odbcDR = odbcComm1.ExecuteReader()
While (odbcDR.Read()) 'And y <= 10
dgvDT1.Rows.Add()
dgvDT1.Rows(y).Cells("Column0").Value = (odbcDR.GetValue(0).ToString())
dgvDT1.Rows(y).Cells("Column1").Value = (UnixTimestampToDateOrTime(odbcDR.GetValue(0), 1))
dgvDT1.Rows(y).Cells("Column2").Value = (UnixTimestampToDateOrTime(odbcDR.GetValue(0), 2))
y = y + 1
End While
Me.Controls.Add(dgvDT1)
dgvDT1.Visible = True
Catch ex As Exception
MessageBox.Show("Error: " & ex.Message, "Error")
End Try
End Sub
Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
MsgBox(UnixTimestampToDateOrTime(1646092800, 1) & vbNewLine & UnixTimestampToDateOrTime(1646092800, 2))
End Sub
Public Function UnixTimestampToDateOrTime(ByVal _UnixTimeStamp As Long, ByRef _Parameter As Integer) As String
strDateTime = New DateTime(1970, 1, 1, 0, 0, 0).AddSeconds(_UnixTimeStamp).ToString()
strDate = strDateTime.Substring(0, strDateTime.IndexOf(" "))
strTime = strDateTime.Substring(strDateTime.IndexOf(" "), strDateTime.Length - strDateTime.IndexOf(" "))
If _Parameter = 1 Then
Return (strDate)
Else
Return (strTime)
End If
End Function
End Class
You write:
I would like to convert and display UnixStamp data in a new column.
Yes you can. But i do not believe that would make your applicatio nany faster.
However I would outline the strategy i personally would had used getting data conversion without UDF.
Removing UDF, i believe, would enhance your database in general. But would not fix performance problems. Due to amount of information i could not format it as mere comment.
So, about removing UDF from the query.
Link to Firebird 2.5 documentation: https://www.firebirdsql.org/file/documentation/html/en/refdocs/fblangref25/firebird-25-language-reference.html#fblangref25-psql-triggers
Link to Firebird 3 documentation:
https://www.firebirdsql.org/file/documentation/html/en/refdocs/fblangref30/firebird-30-language-reference.html
The key difference would be that FB3+ has "stored functions" and even DETERMINISTIC ones, and FB2 - only "stored procedures".
I would avoid using UDFs here: they are declared obsolete, they make deployment more complex, and they would reduce your flexibiltiy (for example, you can not use Firebird for Linux/ARM if you only have UDF DLL for Win64)
So, the approach i speculate, you can use.
You would have to find an algorythm, how to "parse" UNIX date/time into separate values for day, month, etc.
You would have to implement that algorythm in Procedural SQL stored function (in FB3, make it DETERMINISTIC) or stored procedure (FB2, make it selectable by having SUSPEND command, after you set value for the output parameter and before exit) - see the Ch. 7 of the manual.
For assembling values of DATE or TIME or DATETIME types you would have to use integer to string (VARCHAR) to resulting type coersion, see Ch. 3.8 of the manual.
Example:
select
cast( 2010 || '-' || 2 || '-' || 11 AS DATE ),
cast( 23 || ':' || 2 || ':' || 11 AS TIME )
from rdb$database
CAST | CAST
:--------- | :-------
2010-02-11 | 23:02:11
db<>fiddle here
You would have add the converted columns to your table. Those columns qwould be read-only, but they would change "unix time" to Firebird native date and/or type.
Read Ch. 5.4.2. of the manual, about ALTER TABLE <name> ADD <column> command.
Read 5.4.1. CREATE TABLE of the manual about Calculated fields ( columns, declared using COMPUTED BY ( <expression> ) instead of actual datatype. Those columns you would have to creat, and here is where difference between FB3 and FB2 kicks in.
in FB3 i believe you would be able to directly use your PSQL Function as the expression.
in FB2 i believe you would have to use a rather specific trampoline, to coerce a stored procedure into expression:
ALTER TABLE {tablename}
ADD {columnname} COMPUTED BY
(
(
SELECT {sp_output_param_name} FROM {stored_proc_name}( {table_unixtime_column} )
)
)
In you Visual Basic application you would read those read-only converted columns instead of original value columns
Now, this would address the perfomance problem. Again, writing this aas an answer, because it is too large to fit as a comment.
No, while UDF has a number of problems - those are not about performance, but about being "old-school" and prone to low-level problems, such as memory leaks. UDFs can trigger problems if used in other places in query, by prohibiting SQL optimizer to use fast, indexed codepaths, but it is not your case. You only use your UDF in the result columns list of SELECT - and here should be no prerformance troubles.
Your application/database therefore should have other bottlenecks.
first of all, you use Where TEST.UNIXTIMESTAMP > 1646092800 and Order By TEST.ID in your query. The obvious question is if your table does have index on those columns. If not, you force the database server to do full natural scan to apply where condition, and then use external, temporary file sorting. THAT can be really slow, and can scale badly as the table grows.
Use your database design tool of choice to check query plan of your select.
Does Firebird use indexed or non indexed access paths.
There are articles online how to read Firebird's query. I don't instantly know English language ones though. Also, it would be specific to your database design tool, how to get it.
Sorry, there is no silver bullet. That is where learning about databases is required.
Read Data Definition Language / Create Index chapter of the documentation.
Link to Firebird 2.5 documentation: https://www.firebirdsql.org/file/documentation/html/en/refdocs/fblangref25/firebird-25-language-reference.html#fblangref25-psql-triggers
Link to Firebird 3 documentation: https://www.firebirdsql.org/file/documentation/html/en/refdocs/fblangref30/firebird-30-language-reference.html
Check your database structure, if UNIXTIMESTAMP and ID have index on them or not. In general, indices speed some (not all) reading queries and slow down (slightly) all writing queries.
You may decide you want to add those indices if they do not exist yet.
Again, it would be dependent upon your database design tool, how to check for existing of the indexes. It also would depend on your data and your applications which kind of indexes is needed or not. That is not what someone else can decide for you.
i also have a lot of suspicion about odbcAdapter.Fill(odbcDataset, "TEST") command. Basically, you try to read all the data in one go. And you do it via ODBC connection, that is not natural for C#.
Usually desktop application only read first 100 or so rows. People would rarely actually read anything after first page or two. Humans are not machines.
Try to somehow connect your visual grid to the select query without reading ALL the table. There should be way.
Additionally, there is free Firebird .Net Provider - this should work natively with VB.Net and should be your first choice, not ODBC.
There also is commercial IBProvider, based on native OLE DB technology it should be worse choice than .Net Provider, but it can work too and has some code examples for VB.Net, and i suppose it is still better mantained than ODBC driver.
you may also change Firebird configuration and allow it using more RAM for cache. This may somewhat relax problems of index-less selecting and sorting. But only somewhat relax and offset, not solve. You can find articles about it on www.ib-aid.com
Related
There are three mdb files in folders on a network drive that may hold the required record(s). How do I determine which db holds the record(s), ideally without data transfer/linking/etc.? Then a single SQL or DAO select can get the data from the correct db. Note: I'm trying to use Access as a front end to SQL using existing Access data spread all around the network drives.
My current solution of configuring 3 DAO objects and checking for no results, in succession until found, seems to load the remote tables to the local recordset and takes too long.
Is there a way to use IF EXISTS in this scenario?
This code throws "Invalid SQL statement; expected DELETE,INSERT,PROCEDURE,SELECT,OR UPDATE" error but is generally what I'd like to do :
Dim strSQL As String
Dim strSku As String
Dim intDbToSearch As Integer
strSku = DLookup("SKUNo", "tblCurrentItem") 'Note: this returns valid SKU#
strSQL = "IF EXISTS(SELECT xxTable.SKUNo "
strSQL = strSQL & "FROM [S:\Our Inventory\Cust Sleeves.mdb].[xxTable] "
strSQL = strSQL & "Where xxTable.SKUNo = " & "'" & strSku & "') Then intDbToSearch = 1"
DoCmd.RunSQL strSQL
This is one of three IF Exists that would run if SKUNo not found in db 1 or 2.
Ultimately intDbToSearch should point to db 1,2,or 3 if SKUNo found or 0 if not.
Thanks
In the end, I pushed usage rules for the 3 databases upstream and can now predetermine which database to search. Thanks again for your input.
Will the sought for SKU always occur in only 1 of the tables?
If you don't want to set table links or use VBA recordsets, only other approach I can see is a query object with a dynamic parameter that references a form control for the SKU input. No idea if this will be faster and will need a query for each remote table.
SELECT SKUNo FROM xxTable IN "S:\Our Inventory\Cust Sleeves.mdb" WHERE SKUNo = Forms!formname!cbxSKU
Then just do DCount on the query.
Dim intDbToSearch As Integer
If DCount("*", "xxQuery") > 0 Then
intDbToSearch = 1
End If
Could UNION the SELECT statements so would have only 1 query object to work with.
SELECT "x1" AS Source, SKUNo FROM xxTable IN "S:\Our Inventory 1\Cust Sleeves.mdb" WHERE SKUNo = Forms!formname!cbxSKU
UNION SELECT "x2", SKUNo FROM xxTable IN "S:\Our Inventory 2\Cust Sleeves.mdb" WHERE SKUNo = Forms!formname!cbxSKU
UNION SELECT "x3", SKUNo FROM xxTable IN "S:\Our Inventory 3\Cust Sleeves.mdb" WHERE SKUNo = Forms!formname!cbxSKU;
How about a simple Function to check if exists by passing the table name and value?
Something like this:
Public Function ExistInTable(Byval TableName As String, ByVal Value As String) As Boolean
ExistInTable = (DCount("*", TableName, "[SKUNo]='" & Value & "'" > 0)
End Function
To call it:
Sub Test()
If ExistInTable("T1", "Whatever") Then 'Exists in T1
If ExistInTable("T2", "Whatever") Then 'Exists in T2
'....
End Sub
This almost works. I get an error at the last line that looks like it's complaining about the C1 reference. Is there a simple way around this? There is nothing wrong with the query or connection.
Dim CmdString As String
Dim con As New SqlConnection
Try
con.ConnectionString = PubConn
CmdString = "select * from " & PubDB & ".dbo.Suppliers as S " & _
" join " & PubDB & ".dbo.Address as A" & _
" on S.Supplier_Address_Code = A.Address_IDX" & _
" join " & PubDB & ".dbo.Contacts as C1" & _
" on S.Supplier_Contact1 = C1.Contact_IDX" &
" join " & PubDB & ".dbo.Contacts as C2" & _
" on S.Supplier_Contact2 = C2.Contact_IDX" &
" WHERE S.Supplier_IDX = " & LookupIDX
Dim cmd As New SqlCommand(CmdString)
cmd.Connection = con
con.Open()
Dim DAdapt As New SqlClient.SqlDataAdapter(cmd)
Dim Dset As New DataSet
DAdapt.Fill(Dset)
con.Close()
With Dset.Tables(0).Rows(0)
txtAddress1.Text = .Item("Address1").ToString
txtAddress2.Text = .Item("Address2").ToString
txtSupplierName.Text = .Item("Address_Title").ToString
txtAttn.Text = .Item("Attn").ToString
txtBusinessPhone1.Text = .Item("C1.Contact_Business_Phone").ToString
You would not include the "C1" table alias as part of your column name. It will be returned from your query as Contact_Business_Phone.
For accessing multiple rows you could use the indexer as you are in the example above "Rows(0)" by placing your With block into a For loop and accessing the "Rows(i)" with your loop variable. However, this would not help much as your are assigning this to individual text boxes, so you'd only see the last value on your page/screen.
The alias C1 is used by SQL Server and is not persisted to the result set. Have you taken this query into SQL Management Studio to see the results?
Since you requested all columns (*) and joined to the Contacts table twice, you'll end up with duplicate column names in the result. For example, if the Contacts table has a LastName field, you'll end up with TWO LastName columns in your result.
I haven't tried to duplicate this in my local environment, but I can't imagine the data adapter is going to like having duplicate column names.
I recommend specifically including the columns you want to return instead of using the *. That's where you'll use the alias of C1, then you can rename the duplicate columns using the AS keyword:
SELECT C1.LastName AS [Supplier1_LastName],
C2.LastName AS [Supplier2_LastName],
...
This should solve your problem.
Good Luck!
You should only be pulling back the columns that you're in fact interested in, as opposed to *. It's sort of hard to tell exactly what data exists in which tables since you're pulling the full set, but at a quick guess, you'll want in your select statement to pull back A.Address1, A.Address2, A.AddressTitle, ?.Attn (not sure which table this actually derives from) and C1.Contact_Business_Phone. Unless you actually NEED the other fields, you're much better off specifying the individual fields in your query, besides having the possible duplicate field issue that you're running into here, it can also be a significant performance hit pulling everything in. After you clean up the query and only pull in the results you want, you can safely just reference them the way you are for the other fields, without needing a table alias (which as others have pointed out, isn't persisted to the result set anyways).
I have a perplexing problem that I have been grappling with for many hours to no avail.
Our (general audience) application compares queries in the latest version of our MS-Access database with end-users currently installed versions and and updates/syncs. the DB accordingly. Although this functionality has been working, it is now broken. I have resorted to the simplest test to demonstrate the behavior. OLEDB is NOT returning the correct number of queries.
After opening the Access 2007 database, I display the location and correct number of queries:
?CurrentDB.Name
C:\Users\Ron\Documents\Database4.accdb
?CurrentDB.QueryDefs.Count
1
The following simple form calls the OleDbSchemaTable method, but returns the wrong number of rows / queries (=0):
Public Class Form1
Private ConnStr As String = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Users\Ron\Documents\Database4.accdb"
Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
Dim dbo As DataTable = GetSchemaInfo()
Debug.Print("Number of queries=" & dbo.Rows.Count)
End Sub
Private Function GetSchemaInfo()
Try
Dim db As DataTable
Using conn = New OleDb.OleDbConnection(ConnStr)
conn.Open()
db = conn.GetOleDbSchemaTable(System.Data.OleDb.OleDbSchemaGuid.Procedures, Nothing)
End Using
Return db
Catch ex As Exception
MsgBox(ex.Message)
Return Nothing
End Try
End Function
End Class
Immediate Window:
Number of queries=0
I have tried everything I can think of (Compact/Repair, running on a different machine). Unfortunately, I have only one copy of Access 2007 to test with. Could this be a virus?
Any and all ideas are appreciated.
There are 2 types of saved queries.
Simple SELECT queries which could be created with CREATE VIEW DDL statements.
All other queries. These can not be created with CREATE VIEW but could be with CREATE PROCEDURE. Included within this category are: "action queries" (insert, update, delete, "make table"); parameter queries; simple SELECT queries which include an ORDER BY clause; and perhaps more I can't recall at the moment.
Schema views include the first type. Schema procedures include the second type. QueryDefs.Count gives you the count of all saved queries, which includes both types.
Try the VBA procedure below in your database. With my database, I get this output in the Immediate window:
QueryDefs.Count: 66
Views: 34
Procedures: 32
Unfortunately I don't know how to translate this VBA to Dot.Net. Perhaps that doesn't matter. I just want to emphasize that QueryDefs.Count should be the count of views plus the count of procedures. And it looks to me like your code asks for procedures only.
Public Sub CountQueries()
Const adSchemaProcedures = 16
Const adSchemaViews = 23
Dim cn As Object
Dim rs As Object
Dim i As Long
Debug.Print "QueryDefs.Count: " & CurrentDb.QueryDefs.Count
Set cn = CurrentProject.Connection
Set rs = cn.OpenSchema(adSchemaViews)
i = 0
With rs
Do While Not .EOF
i = i + 1
'Debug.Print !TABLE_NAME '
.MoveNext
Loop
.Close
End With
Debug.Print "Views: " & CStr(i)
Set rs = cn.OpenSchema(adSchemaProcedures)
i = 0
With rs
Do While Not .EOF
i = i + 1
'Debug.Print !PROCEDURE_NAME '
.MoveNext
Loop
.Close
End With
Debug.Print "Procedures: " & CStr(i)
Set rs = Nothing
Set cn = Nothing
End Sub
There are two types of schemas for MS Access, schema views and schema procedures. Procedures are action queries, views are select queries. So you need
OleDbSchemaGuid.Views
Note that the OleDbSchemaGuid returns the wrong data type for memo fields.
I'm trying to query old Access database tables and compare them with SQL Server tables.
They often don't have primary keys, or they have extra fields that had some purpose in the nineties, etc., or the new tables have new fields, etc.
I need to find records - based on a set of fields specified at runtime - that are in one table but not another.
So, I do this kind of query all the time in SQL, when I'm comparing data in different tables:
dim fields_i_care_about as string = "field1, field2, field3"
'This kind of thing gets set by a caller, can be any number of fields, depends on the
'table
dim s as string= ""
dim flds = fields_i_care_about.split(",")
for i as integer = 0 to ubound(flds)
if s > "" then s += " AND "
s += " dysfunctional_database_table." & flds(i) & "=current_database_table." & flds(i)
next
s = "SELECT * from dysfunctional_database_table where not exists (SELECT * from current_database_table WHERE " & s & ")"
====
I'm trying to do this using Linq because it seems like some of the datatype problems with two different database types become less of a headache,
but I'm new to Linq and totally stuck.
I got as far as this:
Put old and new tables into datatables as dt1 and dt2
Dim new_records = _
From new_recs In dt2.AsEnumerable
Where Not ( _
From old_recs In dt1.AsEnumerable Where old_recs(field1) = new_recs(field1) AndAlso old_recs(field2) = new_recs(field2)).Any
Select new_recs
But I can't figure out how to put this part in on the fly -
old_recs(field1) = new_recs(field1) AndAlso old_recs(field2) = new_recs(field2)
So far I've tried:
putting the fields I want to compare and making them a string and just putting that string in as a variable ( I thought I was probably cheating, and I guess I was)
dim str = old_recs(field1) = new_recs(field1) AndAlso old_recs(field2) = new_recs(field2)
From new_recs In dt2.AsEnumerable
Where Not ( _
From old_recs In dt1.AsEnumerable Where str).Any
Select new_recs
It tells me it can't convert a Boolean -
Is there any way to do this without Linq expressions? They seem far more complex than what I'm trying to do here, and they take a lot of code, and also I can't seem to find examples of Expressions where we're comparing two fields in a subquery.
Is there a simpler way? I know I could do the usual EXISTS query using JOIN or IN - in this case I don't need the query to be super fast or anything. And I don't need to use a DataTable or DataSet - I can put the data in some other kind of object.
So I found a lot of sample code that used MethodInfo and reflection and things like that, but I couldn't get any of it to work - these Datarows have a Field method but it requires that you add an (of object) argument before the field name argument and that's tricky to do.
So I'm not sure if this solution is the most efficient way, but at least it works. I'd be interested in finding out whether this way of doing it is efficient and why or why not. It seemed like most people used reflection to do this kind of thing, but I couldn't get that working properly and anyway what I'm trying to do is pretty simple while those methods were pretty complex. I suppose I'm doing Linq with a SQL mindset, but anyway it works.
Dim f As Func(Of DataRow, DataRow, String, Boolean) = Function(d1 As DataRow, d2 As DataRow, s As String)
Dim fields = Split(s, ",")
Dim results As Boolean = True
For k As Integer = 0 To UBound(fields)
Dim obj = DataRowExtensions.Field(Of Object)(d1, fields(k))
Dim obj2 = DataRowExtensions.Field(Of Object)(d2, fields(k))
If obj <> obj2 Then results = False : Exit For
Next
Return results
End Function
Dim new_records = _
From new_recs In dt2.AsEnumerable.AsQueryable()
Where Not ( _
From old_recs In dt1.AsEnumerable.AsQueryable Where f(old_recs, new_recs, id_key)).Any
Select new_recs
Try
Return new_records.CopyToDataTable
Catch ex As Exception
Stop
End Try
I'm a desktop developer writing for internal users, so I'm not worried about malicious hackers, but I would like to know if there's anything they could enter when updating a value that would execute sql on the server.
The business defines their content schema and I have a CRUD application for them that doesn't have to be changed when their schema changes because the validation details are table-driven and the updates are with dynamic SQL. I have to support single quotes in their data entry, so when they enter them, I double them before the SQL is executed on the server. From what I've read, however, this shouldn't be enough to stop an injection.
So my question is, what text could they enter in a free-form text field that could change something on the server instead of being stored as a literal value?
Basically, I'm building an SQL statement at runtime that follows the pattern:
update table set field = value where pkField = pkVal
with this VB.NET code:
Friend Function updateVal(ByVal newVal As String) As Integer
Dim params As Collection
Dim SQL As String
Dim ret As Integer
SQL = _updateSQL(newVal)
params = New Collection
params.Add(SQLClientAccess.instance.sqlParam("#SQL", DbType.String, 0, SQL))
Try
ret = SQLClientAccess.instance.execSP("usp_execSQL", params)
Catch ex As Exception
Throw New Exception(ex.Message)
End Try
Return ret
End Function
Private Function _updateSQL(ByVal newVal As String) As String
Dim SQL As String
Dim useDelimiter As Boolean = (_formatType = DisplaySet.formatTypes.text)
Dim position As Integer = InStr(newVal, "'")
Do Until position = 0
newVal = Left(newVal, position) + Mid(newVal, position) ' double embedded single quotes '
position = InStr(position + 2, newVal, "'")
Loop
If _formatType = DisplaySet.formatTypes.memo Then
SQL = "declare #ptrval binary(16)"
SQL = SQL & " select #ptrval = textptr(" & _fieldName & ")"
SQL = SQL & " from " & _updateTableName & _PKWhereClauses
SQL = SQL & " updatetext " & _updateTableName & "." & _fieldName & " #ptrval 0 null '" & newVal & "'"
Else
SQL = "Update " & _updateTableName & " set " & _fieldName & " = "
If useDelimiter Then
SQL = SQL & "'"
End If
SQL = SQL & newVal
If useDelimiter Then
SQL = SQL & "'"
End If
SQL = SQL & _PKWhereClauses
End If
Return SQL
End Function
when I update a text field to the value
Redmond'; drop table OrdersTable--
it generates:
Update caseFile set notes = 'Redmond''; drop table OrdersTable--' where guardianshipID = '001168-3'
and updates the value to the literal value they entered.
What else could they enter that would inject SQL?
Again, I'm not worried that someone wants to hack the server at their job, but would like to know how if they could accidentally paste text from somewhere else and break something.
Thanks.
Regardless of how you cleanse the user input increasing the attack surface is the real problem with what you're doing. If you look back at the history of SQL Injection you'll notice that new and even more creative ways to wreak havoc via them have emerged over time. While you may have avoided the known it's always what's lurking just around the corner that makes this type of code difficult to productionize. You'd be better to simply use a different approach.
You can also evaluate an alternative solution. Dynamic generation of SQL with parameters. Something like this:
// snippet just for get the idea
var parameters = new Dictionary<string, object>();
GetParametersFromUI(parameters);
if (parameters.ContainsKey("#id")) {
whereBuilder.Append(" AND id = #id");
cmd.Parameters.AddWithValue("#id", parameters["#id"]);
}
...
Assuming you escape string literals (which from what you said you are doing), you should be safe. The only other thing I can think of is if you use a unicode-based character set to communicate with the database, make sure the strings you send are valid in that encoding.
As ugly as your doubling up code is (:p - Try String.Replace instead.) I'm pretty sure that will do the job.
The only safe assumption is that if you're not using parameterized queries (and you're not, exclusively, here, because you're concatenating the input string into your sql), then you're not safe.
You never never ever never want to build a SQL statement using user input that will be then directly executed. This leads to SQL injection attacks, as you've found. It would be trivial for someone to drop a table in your database, as you've described.
You want to use parameterized queries, where you build an SQL string using placeholders for the values, then pass the values in for those parameters.
Using VB you'd do something like:
'Define our sql query'
Dim sSQL As String = "SELECT FirstName, LastName, Title " & _
"FROM Employees " & _
"WHERE ((EmployeeID > ? AND HireDate > ?) AND Country = ?)"
'Populate Command Object'
Dim oCmd As New OledbCommand(sSQL, oCnn)
'Add up the parameter, associated it with its value'
oCmd.Parameters.Add("EmployeeID", sEmpId)
oCmd.Parameters.Add("HireDate", sHireDate)
oCmd.Parameters.Add("Country", sCountry)
(example taken from here) (also not I'm not a VB programmer so this might not be proper syntax, but it gets the point across)