I'm sure this is a comparatively simple question, but after several hours, I haven't been able to find the answer, and I'm still somewhat new to the wonderful world of databasing.
I'm starting to use Webmatrix and I'm trying to import a database with 13 columns and a little over 200 rows (just enough so I don't want to type it all out again).
All my work remains offline and won't go online for at least another few months. So far I have worked on a Mac (php, mysql,etc) and just switched to PC to try Webmatrix. Since I couldn't find a simple enough feature to import a database in Webmatrix, I figured the easiest way might be to export an SQL file from phpMyAdmin on my Mac and execute it once via a cshtml page (coding razor).
With a few adjustments I recreated my table in this way and I'm also able to insert values in individual rows, however, I can't seem to be able to insert more than one row at a time.
My intended code is basically:
var db = Database.Open("Database");
var insertQuery = "INSERT INTO table (field1, field2, field3) VALUES ('val1', 'val2', 'val3'),('val4','val5','val6'),(etc)";
db.Execute(insertQuery);
Could anyone shed some light on what might be going wrong here?
I also looked at other methods of importing databases, I read about MySQL benchmark, but admit that's going a bit over my head.
Thanks in advance.
Your options depend on the type of database you are using. If you are using the default WebMatrix database (SQL Compact Edition 4.0), you can save the PHP data to a csv file, and then in WebMatrix, read that using File.ReadAllLines. That will give you an array of strings each containing comma separated values representing a row of data. You can use string.Split to create an array of individual values and insert that within a foreach loop:
var rows = File.ReadAllLines(path_to_csv_file);
foreach(var row in rows){
var vals = row.Split(new[]{','});
var sql = "INSERT INTO table(f1, f2, f3) VALUES (#0, #1, #2)";
db.Execute(sql, data[0], data[1], data[2]);
}
If you are using SQL Server, you can use BULK INSERT for a file. There are also a couple of other approaches for SQL CE too: Bulk Insert In SQL Server CE
Related
I have a large (25,000) list of ID values that, at times, I'd like to just script a query to use instead of running against the full data set (100's of millions or rows) and the hard limit of 8,000 characters in the OLE DC Source SQL Command Text field won't allow them all. The client's server is locked up and I can't run BULK INSERT and can only use Temp Tables.
It would be something like this:
Select ID, FieldA, FieldB, FieldC, ...
From TableX
Where ID in (list of ID's from flat file)
I was hoping there would be some way to refer to a flat file with all the ID's in it either in the WHERE clause or via somekind of ForEach Loop. I've setup other ForEach loops but am unsure of A) IF it can even be done and B) how to go about it.
I've poked around the web but not getting any direct hits. Any direction to go and research would be appreciated. Thanks in advance!
I am working on developing an application for my company. From the beginning we were planning on having a split DB with an access front end, and storing the back end data on our shared server. However, after doing some research we realized that storing the data in a back end access DB on a shared drive isn’t the best idea for many reasons (vpn is so slow to shared drive from remote offices, access might not be the best with millions of records, etc.). Anyways, we decided to still use the access front end, but host the data on our SQL server.
I have a couple questions about storing data on our SQL server. Right now when I insert a record I do it with something like this:
Private Sub addButton_Click()
Dim rsToRun As DAO.Recordset
Set rsToRun = CurrentDb.OpenRecordset("SELECT * FROM ToRun")
rsToRun.AddNew
rsToRun("MemNum").Value = memNumTextEntry.Value
rsToRun.Update
memNumTextEntry.Value = Null
End Sub
It seems like it is inefficient to have to use a sql statement like SELECT * FROM ToRun and then make a recordset, add to the recordset, and update it. If there are millions of records in ToRun will this take forever to run? Would it be more efficient just to use an insert statement? If so, how do you do it? Our program is still young in development so we can easily make pretty substantial changes. Nobody on my team is an access or SQL expert so any help is really appreciated.
If you're working with SQL Server, use ADO. It handles server access much better than DAO.
If you are inserting data into a SQL Server table, an INSERT statement can have (in SQL 2008) up to 1000 comma-separated VALUES groups. You therefore need only one INSERT for each 1000 records. You can just append additional inserts after the first, and do your entire data transfer through one string:
INSERT INTO ToRun (MemNum) VALUES ('abc'),('def'),...,('xyz');
INSERT INTO ToRun (MemNum) VALUES ('abcd'),('efgh'),...,('wxyz');
...
You can assemble this in a string, then use an ADO Connection.Execute to do the work. It is frequently faster than multiple DAO or ADO .AddNew/.Update pairs. You just need to remember to requery your recordset afterwards if you need it to be populated with your newly-inserted data.
There are actually two questions in your post:
Will OpenRecordset("SELECT * FROM ToRun") immediately load all recordsets?
No. By default, DAO's OpenRecordset opens a server-side cursor, so the data is not retrieved until you actually start to move around the recordset. Still, it's bad practice to select lots of rows if you don't need to. This leads to the next question:
How should I add records in an attached SQL Server database?
There are a few ways to do that (in order of preference):
Use an INSERT statment. That's the most elegant and direct solution: You want to insert something, so you execute INSERT, not SELECT and AddNew. As Monty Wild explained in his answer, ADO is prefered. In particular, ADO allows you to use parameterized commands, which means that you don't have to put-into-quotes-and-escape your strings and correctly format your dates, which is not so easy to do right.
(DAO also allows you to execute INSERT statements (via CurrentDb.Execute), but it does not allow you to use parameters.)
That said, ADO also supports the AddNew syntax familiar to you. This is a bit less elegant but requires less changes to your existing code.
And, finally, your old DAO code will still work. As always: If you think you have a performance problem, measure if you really have one. Clean code is great, but refactoring has a cost and it makes sense to optimize those places first where it really matters. Test, measure... then optimize.
It seems like it is inefficient to have to use a sql statement like SELECT * FROM ToRun and then make a recordset, add to the recordset, and update it. If there are millions of records in ToRun will this take forever to run?
Yes, you do need to load something from the table in order to get your Recordset, but you don't have to load any actual data.
Just add a WHERE clause to the query that doesn't return anything, like this:
Set rsToRun = CurrentDb.OpenRecordset("SELECT * FROM ToRun WHERE 1=0")
Both INSERT statements and Recordsets have their pros and cons.
With INSERTs, you can insert many records with relatively little code, as shown in Monty Wild's answer.
On the other hand, INSERTs in the basic form shown there are prone to SQL Injection and you need to take care of "illegal" characters like ' inside your values, ideally by using parameters.
With a Recordset, you obviously need to type more code to insert a record, as shown in your question.
But in exchange, a Recordset does some of the work for you:
For example, in the line rsToRun("MemNum").Value = memNumTextEntry.Value you don't have to care about:
characters like ' in the input, which would break an INSERT query unless you use parameters
SQL Injection
getting the date format right when inserting date/time values
I can INSERT a row into SQL like this:
INSERT INTO MyTable VALUES (0, 'This is some text...');
However, what if I wanted This is some text... to be the contents of C:\SomeFile.txt. Is there a method in Oracle that makes this possible?
I dug through the Docs a bit and found the LOAD FILE method, however it appears to be for bulk loading data. For example, it wants a FIELDS TERMINATED BY parameter and what not. I want to simply INSERT a single row and set a single column to by the contents of a file on the local disk.
You should never be reading files from the DATABASE SERVER's file system to insert into the db.
Why do you want to do this? You really should read the file in your application code, then insert the string or binary data through standard SQL.
If you really must do this you will need to use the oracle UTL_FILE function. and write some PL/SQL to store in a variable, then insert.
My first time answering a question, so forgive me.
If you're using PHP (which you may well not be, but this is what I know), then you could do something like this:
File: "importantfile.php"
$variable 1 = "0";
$variable 2 = "This is some text...";
File that inserts text: "index.php"
require "importantfile.php";
$query = mysql_query("INSERT INTO MyTable VALUES ('$variable1', '$variable2')");
I hope this helps you in some way :)
To do this I think you would have to use PL/SQL. This is technically an "advanced SQL".
search up a bit about PL/SQL. You can copy your SQL directly into PL/SQL and use the same database.
Got about a 400 MB .txt file here that is delimited by '|'. Using a Windows Form with C#, I'm inserting each row of the .txt file into a table in my SQL server database.
What I'm doing is simply this (shortened by "..." for brevity):
while ((line = file.ReadLine()) != null)
{
string[] split = line.Split(new Char[] { '|' });
SqlCommand cmd = new SqlCommand("INSERT INTO NEW_AnnualData VALUES (#YR1984, #YR1985, ..., #YR2012)", myconn);
cmd.Parameters.AddWithValue("#YR1984", split[0]);
cmd.Parameters.AddWithValue("#YR1985", split[1]);
...
cmd.Parameters.AddWithValue("#YR2012", split[28]);
cmd.ExecuteNonQuery();
}
Now, this is working, but it is taking awhile. This is my first time to do anything with a huge amount of data, so I need to make sure that A) I'm doing this in an efficient manner, and that B) my expectations aren't too high.
Using a SELECT COUNT() while the loop is going, I can watch the number go up and up over time. So I used a clock and some basic math to figure out the speed that things are working. In 60 seconds, there were 73881 inserts. That's 1231 inserts per second. The question is, is this an average speed, or am I getting poor performance? If the latter, what can I do to improve the performance?
I did read something about SSIS being efficient for this purpose exactly. However, I need this action to come from clicking a button in a Windows Form, not going through SISS.
Oooh - that approach is going to give you appalling performance. Try using BULK INSERT, as follows:
BULK INSERT MyTable
FROM 'e:\orders\lineitem.tbl'
WITH
(
FIELDTERMINATOR ='|',
ROWTERMINATOR ='\n'
)
This is the best solution in terms of performance. There is a drawback, in that the file must be present on the database server. There are two workarounds for this that I've used in the past, if you don't access to the server's file system from where you're running the process. One is to install an instance of SQL Express on the workstation, add the main server as a linked server to the workstation instance, and then run "BULK INSERT MyServer.MyDatabase.dbo.MyTable...". The other option is to reformat the CSV file as XML, which can be processed very quickly, and then passing the XML to query and processing it using OPENXML. Both BULK INSERT and OPENXML are well documented on MSDN, and you'd do well to read through the examples.
Have a look at SqlBulkCopy on MSDN, or the nice blog post here. For me that goes up to tens of thousands of inserts per second.
I'd have to agree with Andomar. I really quite like SqlBulkCopy. It is really fast (you need to play around with BatchSizes to make sure you find one that suits your situation.)
For a really in depth article discussing the various options, check out Microsoft's "Data Loading Performance Guide";
http://msdn.microsoft.com/en-us/library/dd425070(v=sql.100).aspx
Also, take a look at the C# example with SqlBulkCopy of CSV Reader. It isn't free, but if you can write a fast and accurate parser in less time, then go for it. At least, it'll give you some ideas.
I have fonud SSIS to be much faster than this type of method but there are a bunch of variables that can affect performence.
If you want to experiment with SSIS, use the Import and Export wizard in Management Studio to generate a SSIS package that will import a pipe delimited file. You can save out the package and run it from a .NET application
See this article: http://blogs.msdn.com/b/michen/archive/2007/03/22/running-ssis-package-programmatically.aspx for info on how to run an SSIS package programatically. It includes options on how to run from the client, from the server, or wherever.
Also, take a look at this article for additional ways you can improve bulk insert performance in general. http://msdn.microsoft.com/en-us/library/ms190421.aspx
I need to upload a lot of data to a MySQL db. For most models I use django's ORM, but one of my models will have billions (!) of instances and I would like to optimize its insert operation.
I can't seem to find a way to make executemany() work, and after googling it seems there are almost no examples out there.
I'm looking for the correct sql syntax + correct command syntax + correct values data structure to support an executemany command for the following sql statement:
INSERT INTO `some_table` (`int_column1`, `float_column2`, `string_column3`, `datetime_column4`) VALUES (%d, %f, %s, %s)
Yes, I'm explicitly stating the id (int_column1) for efficiency.
A short example code would be great
Here's a solution that actually uses executemany() !
Basically the idea in the example here will work.
But note that in Django, you need to use the %s placeholder rather than the question mark.
Also, you will want to manage your transactions. I'll not get into that here as there is plenty of documentation available.
from django.db import connection,transaction
cursor = connection.cursor()
query = ''' INSERT INTO table_name
(var1,var2,var3)
VALUES (%s,%s,%s) '''
query_list = build_query_list()
# here build_query_list() represents some function to populate
# the list with multiple records
# in the tuple format (value1, value2, value3).
cursor.executemany(query, query_list)
transaction.commit()
are you serisouly suggesting loading billions of rows (sorry instances) of data via some ORM data access layer - how long do you have ?
bulk load if possible - http://dev.mysql.com/doc/refman/5.1/en/load-data.html
If you need to modify the data, bulk load with load data into a temporary table as is. Then apply modifications with an insert into select command. IME, this is by far the fastest way to get a lot of data into a table.
I'm not sure how to use the executemany() command, but you can use a single SQL INSERT statement to insert multiple records