I'm currently working with a database of over 50 million records, where I read a file which a person wants to search the database for etc. I have noticed my data reader part is running particularly slow, where as the query seems almost instant (database is indexed). I was just wondering does anyone know as to why it seems to be running slow?
con.Open()
Using sw As New StreamWriter("G:\USER\SEARCH-RESULTS.txt")
Try
For Each word As String In result
Using com As New SqlCommand("select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd, t.LnePos from (Select SmeNbr, MAX(FilDte) as MaxDate from Test_Table where SmeNbr = #word group by SmeNbr)r inner join Test_Table t on t.SmeNbr = r.SmeNbr and t.FilDte = R.MaxDate", con)
com.Parameters.AddWithValue("#word", word)
Using RDR = com.ExecuteReader
If RDR.HasRows Then
Do While RDR.Read
MyFilePath = RDR.Item("FilPth").ToString()
linePos = RDR.Item("LnePos").ToString()
Using sr As New StreamReader(MyFilePath)
sr.BaseStream.Seek(4096 * (linePos - 1), SeekOrigin.Begin)
FoundWords.Add(sr.ReadLine)
For Each item As String In FoundWords
sw.WriteLine(item)
Next
FoundWords.Clear()
End Using
Loop
Else
Continue For
End If
End Using
End Using
Next
Catch ex As Exception
MessageBox.Show("Couldn't process search")
Finally
con.Close()
End Try
End Using
MsgBox("Complete!")
So it works perfect, as in it gets the records and bits of info I want very quickly through the query and all and even the writing reults to a new file is near instant, I used breakpoints and like I said it seems to take ages between the "Using RDR = com.ExecuteReader" and "If RDR.HasRows Then"
Any help or ideas would be greatly appreciated.
com.Parameters.AddWithValue("#word", word)
AddWithValue infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n) parameter with the length of the actual value. I see from your comments that the actual column data type is char(13) so it would be best to explicitly specify that as the parameter data type:
com.Parameters.Add("#word", SqlDbType.Char, 13).Value = word
The implications with AddWithValue are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.
Related
I'm creating a simple relational database system, to ensure no repeated data is saved in. I have created a chunk of code to check if the value of Album is already present in the selected table:
If Album IsNot Nothing Then 'checks if ALBUM exists
ALBUM_CHECK = New SqlCommand("SELECT ALBUM_ID FROM ALBUM_DB WHERE NAME=(#NAME)", SQLcon)
ALBUM_CHECK.Parameters.AddWithValue("#NAME", Album)
If ALBUM_CHECK.ExecuteScalar IsNot Nothing Then
album_Exist = True
Else
album_Exist = False
End If
End If
However this returns the error:
System.Data.SqlClient.SqlException: 'The data types text and nvarchar are incompatible in the equal to operator.'
Any ideas on how to get round this?
I believe it's not allowing me to read if the value returned is null. All help appreciated!
From the documentation of SQL Server 2017 :
IMPORTANT! ntext, text, and image data types will be removed in a future version of SQL Server. Avoid using these data types in new development work, and plan to modify applications that currently use them. Use nvarchar(max), varchar(max), and varbinary(max) instead.
I would suggest to change the datatype of this column to varchar(max), which has the same capacity in terms of storage and is properly supported by this RDBMS.
Text is a very poor choice for an Album. Comments and explanations in line.
Private Sub OPCode()
Dim Album As String = ""
Dim albumExist As Boolean
If String.IsNullOrEmpty(Album) Then
Return
End If
'Using blocks ensure that your database objects are not only closed but disposed.
'The disposal is important because they can contain unmanaged objects.
Using SQLcon As New SqlConnection("Your connection string")
'If Exists is a good method for large tables because it stops as soon
'as it finds a match.
Using ALBUM_CHECK As New SqlCommand("If Exists (SELECT 1 FROM ALBUM_DB WHERE NAME = #NAME) Select 1 Else Select 0;", SQLcon)
'Using .Add instead of .AddWithValue tells the database what kind of
'data we are sending. Without this we have no control over what ADO
'decides to sent to the database.
ALBUM_CHECK.Parameters.Add("#NAME", SqlDbType.NVarChar).Value = Album
'The query will only return a 0 or a 1.
SQLcon.Open
albumExist = CBool(ALBUM_CHECK.ExecuteScalar())
End Using
End Using
End Sub
I have some simple code, an if statement checking the HasRows of a data reader and for some reason when I run my code in Visual Studio 2017 it takes forever to evaluate and return the answer (while writing this, my code has been running for 4 minutes). Any suggestions?
Dim cmd As OdbcCommand
Dim rdr As OdbcDataReader
cmd = New OdbcCommand("select GLPN,GLFY,GLDCT,GLDOC,GLCO,GLDGJ,GLANI,GLSBL,GLLT,GLCRCD,GLAA,GLU,GLGLC,GLEXA,GLICUT,GLR2,GLR1,GLSFX,GLOKCO" _
& ",GLEXR,GLODOC,GLPKCO,GLPDCT,GLCN,GLDKJ,GLVINV,GLIVD,GLPO,GLDCTO,GLLNID,GLTORG,GLAN8,GLICU,GLOPSQ,GLJBCD" _
& ",GLACR,GLABR2,GLABR1,GLDGJ,GLLT,GLCRCD,GLEXA,GLICUT,GLEXR,GLDKJ,GLIVD,GLAN8,GLICU,GLACR,GLKCO,GLSBLT,GLOBJ,GLSUB,GLJELN,GLEXTL,GLCRR,GLBCRC" _
& " from " _
& "PRODDTA.F0911 where GLPOST = 'P' and GLDGJ >= ? and GLDGJ <= ? and (GLLT = 'AA' or GLLT = 'CA') and GLDOC = 206940", cnn)
cmd.Parameters.Add("?GLUPMJs", OdbcType.Int).Value = todaysdate - 14
cmd.Parameters.Add("?GLUPMJe", OdbcType.Int).Value = todaysdate
cnn.Open()
cmd.CommandTimeout = 300
rdr = cmd.ExecuteReader
If rdr.HasRows() Then
'Do a bunch of stuff
End if
Edit1: Still getting the funny issue but it I have noticed it's only in one spot, I have the "HasRows()" Check in multiple spots and it is working fast 3ms and such. it's only on the one query.
Edit2: The query I referenced above runs on SQL developer very fast total of 1.202 seconds last time I tried, it also returns no messages.
Edit3: I am wondering if it has something to do with the amount of fields I am returning, the other queries that run fast on this line are returning much smaller field counts.
This may not speed it up any but have you tried??
While rdr.Read()
'Do a bunch of stuff
End While
EDIT:
Dim dt As DataTable = New DataTable
dt.Load(rdr)
If dt.Rows.Count >= 0 Then
'Do a bunch of stuff
End If
Maybe the problem is that the query is taking a lot to be executed. If you are only interested in checking the existence of records you could perform a query with "SELECT TOP 1 1" and use ExecuteScalar instead of ExecuteReader.
It seems that you are not the only one having trouble with the HasRows method. In this post, the OP stated a problem with that method, although it is SQL-SERVER. But I think the solution might be useful, since both DataReaders inherit from the same class and they are closely related.
If you read the post that they reference in the question and the most voted answer, you'll see that they concluded that HasRows showed a buggy behavior when the SQL query execution returned not only some data results, but also a message. They ended up using an alternative way to check if there were any data on the data reader based on the Read() method, which proved to be more reliable.
In their case, the problem was that HasRows property was set to false when the reader actually contained data, but maybe it is related to your performance problem. You should definitely give a try to the Read method just to be sure that your query is not the problem.
P.S: another interesting link with a person reporting the same problem.
The reason for the delay on HasRows() is because the ExecuteReader didn't actually execute the query, it Defers the execution until it's needed. HasRows is the one executing it. thanks to Dwillis for pointing it out.
I tested this by putting a line of code ahead of it and this is where the delay sat and when hasRows ran it was fast.
Interestingly the same sql run in Sql Management Studio runes in 1.5 seconds returning all rows, yet in the Code it takes for ever which is a different issue but also may be due to ODBC settings.
Okay, I'm creating an application which requires the user to input there Gamer-Tag during the registration process. Only 'Members' of a group may have access to the program and so I need the application to refuse anyone who's Gamer-Tag is not in the SQL Server database.
I adapted a function I created to log-in with your details and changed it to check if the Gamer-Tag is on the database, but I'm receiving a strange error:
The column 'Gamer_Tag' is not a Member of 'Members_Details' Table.
Please note, that's an error and not what I am wanting it to do. It's saying the column 'Gamer_Tag' isn't in the table, but I know it is! I've checked spelling and spacing and everything's as it should be, so I'm assuming it's something wrong with the code.
The (relevant) register button code:
(Note, RegUserName.Text is the textbox that contains the users Gamer-Tag.
Dim dbManager As New DatabaseManager()
If dbManager.CheckGamerTagisMember(RegUserName.Text) Then
MsgBox("Gamer-Tag Not A Member.")
My.Settings.RegisterCount = 1
My.Settings.Save()
RegisterBTN.Enabled = True
MsgBox("Registration Failed.")
GoTo Ender
Else
MsgBox("Gamer-Tag is A Member.")
GoTo Ender
End If
And the function this is calling:
Public Function CheckGamerTagisMember(ByVal gamertag As String) As Boolean
Connection = New SqlConnection("Data Source =" & My.Settings.ServerIP & ";Initial Catalog=Members_Details;Integrated Security=False;User=" & My.Settings.UserName & ";Password=********;")
Connection.Open()
Dim gamertagDatSet As New DataSet()
usersDataAdapter.FillSchema(gamertagDatSet, SchemaType.Source, "Members_Details")
usersDataAdapter.Fill(gamertagDatSet, "Members_Details")
Dim table As DataTable = gamertagDatSet.Tables("Members_Details")
For i As Integer = 0 To table.Rows.Count - 1
Dim currentUsergt As String = table.Rows(i)("Gamer_Tag").ToString.Trim()
If (currentUsergt = gamertag) Then
gamertagDatSet.Dispose()
Connection.Close()
Return True
End If
Next
gamertagDatSet.Dispose()
Connection.Close()
Return False
End Function
And to prove the column is in the table, here's the table's columns;
I'm new to structuring my code into classes, so I could be 'passing' things around wrongly.
Any help would be much appreciated!
I haven't a clue what the issue is with your code (there's probably something else going on which is not evident in your question).
But I will say that the approach you are taking will not scale well. With each registration, you are downloading the entire table to the program and then scanning through the rows, one by one, looking for an ID.
It would be much easier to write a simple stored procedure to do this for you. You would also get the advantage of using indexes. For example:
CREATE PROCEDURE spGetMembers_ID
#Gamer_Tag varchar(255)
AS
SET NOCOUNT ON;
SELECT Members_ID
FROM Members_Details
WHERE Gamer_Tag = #Gamer_Tag
GO
You can call this stored proc this way:
Dim getMembers_IDCommand As New SqlCommand("TestProcedure", Connection)
getMembers_IDCommand.CommandType = CommandType.StoredProcedure
Dim gamerTagParam As SqlParameter = getMembers_IDCommand.Parameters.Add("#Gamer_Tag", SqlDbType.VarChar, 255)
gamerTagParam.Value = gamertag
SqlDataReader result = getMembers_IDCommand.ExecuteReader();
Dim exists As Boolean = myReader.Read()
result.Close()
Return exists
There, of course, many ways to execute a stored procedure and get the results. Several are documented here. But the point is that a stored procedure is faster, more scalable and generally faster.
I created this simple test for reading data from a stored procedure. I must be overlooking something obvious, because I can't figure out why I'm continuing to get no data.
myReader.HasRows is always true.
I noticed that it is returning the correct number of rows, but I am having trouble getting at any data that might be in the rows.
Stored Procedure
ALTER PROCEDURE [dbo].[testProc]
-- Add the parameters for the stored procedure here
#carID uniqueidentifier
AS
BEGIN
SELECT * FROM carTable WHERE carID=#carID
END
VB.Net
Dim cmd As New SqlClient.SqlCommand("testProc", _conn)
cmd.CommandType = CommandType.StoredProcedure
cmd.Parameters.AddWithValue("carID", carID)
_conn.Open()
Dim myReader As SqlDataReader
myReader = cmd.ExecuteReader()
If myReader.HasRows = True Then
While (myReader.Read())
If Not IsDBNull(myReader.GetString(0)) Then
' do stuff here
End If
End While
End If
From MSDN about SqlDataReader.GetString Method method:
No conversions are performed; therefore, the data retrieved must
already be a string.
Call IsDBNull to check for null values before
calling this method.
My guess is IsDBNull.myReader.GetString(0) returns False because CarId is not a String column (it's uniqueidentifier).
Instead, to test is the value is null, try to use the SqlDataReader.IsDBNull Method, which apply to any datatype.
If Not myReader.IsDBNull(0) Then
First, try running the query in your management studio with the supplied car ID to see what results it is passing back. Secondly I would try to change the if statement as follows and see if you still have issues:
If Not myReader.IsDBNull(0) Then
Additionally, all subsequent fields that allow NULL values must be checked for IsDBNull before they are used. This may also cause some issues.
I would like to fill a datatable with results from a SQL select statment but using a transaction. The reason that I am using a transaction is because I have a list of names (as a datatable), and I want to iterate through the list of names and select the database rows where the name = the name on the list. There are 500,000 names in the database and I only want to retreive the relevant rows. I have the code for the procedure as I think it should look like (untested) BUT I dont know HOW to place the data into a datatable .... so Im missing something where I declare the datatable and the 'fill' of that table , could someone help with this ? Or suggest how else I can get the information out of the batabase without looking up each name individually.
Using connection As New SQLite.SQLiteConnection(R2WconectionString)
connection.Open()
Dim sqliteTran As SQLite.SQLiteTransaction = connection.BeginTransaction()
Try
oMainQueryR = "SELECT NameID, Address, Ocupation FROM Employees Where Name= :Name"
Dim cmdSQLite As SQLite.SQLiteCommand = connection.CreateCommand()
With cmdSQLite
.CommandType = CommandType.Text
.CommandText = oMainQueryR
.Parameters.Add(":Name", SqlDbType.VarChar)
End With
'Prevent duplicate selects by using a dictionary
Dim NameInalready As New Dictionary(Of String, String)
For Each row As DataRow In TheLIST.Rows
If NameInalready.ContainsKey(row.Item("Name")) Then
Else
NameInalready.Add(row.Item("Name"), "")
cmdSQLite.Parameters(":Name").Value = row.Item("Name")
cmdSQLite.ExecuteNonQuery()
End If
Next
sqliteTran.Commit()
Catch ex As Exception
End Try
End Using
First, you don't need a transaction because you aren't updating the database.
Second, depending on the possible number of Names in TheLIST, it might be worthwhile for you to change the name selector to IN (i.e. SELECT * FROM Employees WHERE Name IN ('name1', 'name2'). However, if you expect more than about 10, this is probably not worth trouble.
Finally, you need to create a new DataTable to hold the results. Then you need to create a DataAdapter passing cmdSqlLite as the constructor parameter. And finally, replace your ExecuteNonQuery with DataAdapter.Fill(DataTable).
For example (after Dim cmdSQLite):
Dim oDataTable As New DataTable("Employees")
Dim oAdapter As New SqliteDataAdapter(cmdSQLite)
and replacing the ExecuteNonQuery line with:
oAdapter.Fill(oDataTable)
I will qualify this code by saying it may need some tweaks. I only work with class objects and collections, so my preference would have actually been to load a collection of Employee class instances.
I would have done that by replacing ExecuteNonQuery with ExecuteReader and then the loading the read data into a new class instance. This type of approach resolves various issues with serializing the data across service boundaries (i.e. Xml for web services) and also lets you embed business logic, if needed, into the classes.