I've an SSIS package that runs a stored proc for exporting to an excel file. Everything worked like a champ until I needed to a do a bit of rewriting on the stored proc. The proc now takes about 1 minute to run and the exported columns are different, so my problems are the following;
1) SSIS complains when I hit the preview button "No column information returned by command"
2) It times out after about 30 seconds.
What I've done.
Tried to clean up/optimize the query. That helped a bit, but it still is doing some major calculations and it runs just fine in SSMS.
Changed the timeout values to 90 seconds. Didn't seem to help. Maybe someone here can?
Thanks,
Found this little tidbit which helped immensely.
No Column Names
Basically all you need to do is add the following to your SQL query text in SSIS.
SET FMTONLY OFF
SET NOCOUNT ON
Only problem now is it runs slow as molasses :-(
EDIT: It's running just too damn slow.
Changed from using #tempTable to tempTable. Adding in appropriate drop statements. argh...
Although it appears you may have answered part of your own question, you are probably getting the "No column information returned by command" error because the table doesn't exist at the time it tries to validate the metadata. Creating the tables as non-temporary tables resolves this issue.
If you insist on using temporary tables, you can create the temporary tables in the step preceeding the data flow. You would need to create it as a ## table and turn off connection sharing for the connection for this to work, but it is an alternative to creating permanent tables.
A shot in the dark based on something obscure I hit years ago: When you modified the procedure, did you add a call to a second procedure? This might mess up SSIS's ability to determine the returned data set.
As for (2), does the procedure take 30+ or 90+ seconds to run in SSMS? If not, do you know that the query is actually getting into SQL from SSIS? Might be worth firing up SQL Profiler to see what's actually being sent to SQL Server. [Which was the way I found out my obscure factoid.]
Related
I have an excel file that will select roughly 1100 rows with 5 columns of data. Most columns are 5 digits long and are integers. I am using a macro to connect to a SQL server database and insert these rows into one maybe two tables. This is all its doing and then it closes the connection. So the user opens an excel file that has the rows, clicks a button and it executes the macro.
My question is, should the query be written in Excel since its simple and merely inserts the data into a few tables. Or is it more efficient calling a stored procedure and passing all of the values in the stored procedure and have it allocate where the values go in the different tables. When I mean efficient, i mean which is the quickest? I know this will probably take a few seconds to complete. I just feel going to a stored procedure is an extra point along the path that the data has to get to before it reaches the tables. Am I wrong? Any thoughts?
There are some advantages to using stored procedures in SQL Server. One is that SQL Server precompiles and saves the query execution plan, which increases performance. With your current method, SQL Server will generally need to generate the execution plan each time. Stored procedures can also reduce client/server network traffic.
So, even though it may seem like an extra point along the path, it actually can be faster.
In addition to #mark d.'s answer, another reason for using a stored procedure is security.
Your comment says that a customer is entering the data into Excel, so if you are putting direct SQL into your spreadsheet, then there is a risk that someone will open your spreadsheet and find out information about your database. But if you use a stored procedure then there is far less that can be learned.
Either way, make sure that you aren't hardcoding any connection string/account credentials into the spreadsheet.
I have a weird issue I've never ran into before. I have a stored procedure that joins a bunch of data. When I do an update on a table that is found in one of the joins it doesnt update, until say 20-30 seconds later, or not at all. I see the value updated in the actual table, but the stored procedure has the old value. I didnt think stored procedures could cache like this, or delay like this. Where should I look to fix this?
Try aliasing all of your tables and using the correct alias in the SELECT portion of your query.ff
Unfortunately there is some caching possibly of the old query plan for the previous version of the stored procedure. It seems like a bug in SQL Server. After a short period of time, it seems to update it but yes - pretty sure it's a bug.
Source: using SSMS on SQL Server 2012 -> 2017 for large data management
I use an application connected with an sql database. I found using the profiler that the application runs an update query with a syntax error. I don't have access to the application's source code. The result is that the record is not updated. Is there a way to modify the query every time it is executed with something like trigger? I can't use INSTEAD OF because there ism't any record updated or inserted.
This answer
https://stackoverflow.com/a/3319031/1359088
suggests a way to log to a text file all the errors. You could write a little utility and schedule it to run every hour or whatever, which could read through this log, find the erroneous sql statements, fix them, then run them itself.
I'm providing maintenance support for some SSIS packages. The packages have some data flow sources with complex embedded SQL scripts that need to be modified from time to time. I'm thinking about moving those SQL scripts into stored procedures and call them from SSIS, so that they are easier to modify, test, and deploy. I'm just wondering if there is any negative impact for the new approach. Can anyone give me a hint?
Yes there are issues with using stored procs as data sources (not in using them in Execute SQL tasks though in the control flow)
You might want to read this:
http://www.jasonstrate.com/2011/01/31-days-of-ssis-no-more-procedures-2031/
Basically the problem is that SSIS cannot always figure out the result set and thus the columns from a stored proc. I personally have run into this if you write a stored proc that uses a temp table.
I don't know that I would go as far as the author of the article and not use procs at all, but be careful that you are not trying to do too much with them and if you have to do something complicated, do it in an execute sql task before the dataflow.
I can honestly see nothing but improvements. Stored procedures will offer better security, the possibility for better performance due to cached execution plans, and easier maintenance, like you pointed out.
Refactor away!
You will not face issues using only simple stored procedures as data source. If procedure is using temp tables and CTE - there is no guarantee you will not face issues. Even when you can preview results in design time - you may get errors in a run time.
My experience has been that trying to get a sproc to function as a data source is just not worth the headache. Maybe some simple sprocs are fine, and in some cases TVFs will work well instead, but if you need to do some complex operations there's no alternative to a sproc.
The best workaround I've found is to create an output table for each sproc you need to use in SSIS.
Modify the sproc to truncate the new output table at start, and to write its output to this instead of (or in addition to) ending with a SELECT statement.
Call the sproc with an Exec SQL task before your data flow.
Have your data flow read from the output table - a much simpler task.
If you want to save space, truncate the output table again with another Exec SQL. I prefer to leave it, as it lets me examine the data later and lets me rerun the output data flow if it fails without calling the sproc again.
This is certainly less elegant than reading directly from a sproc's output, but it works. FWIW, this pattern follows the philosophy (obligate in Oracle) that a sproc should not try to be a parameterized view.
Of course, all this assumes that you have privs to adjust the sproc in question. If necessary, you could write a new wrapper sproc which truncates the output table, then calls the old sproc and redirects its output to the new table.
How do I make SQL Server commit inserts in chunks? I need to copy a large amount of rows from an old db into a new table, and it has several problems:
it takes ages to finish, and I don't see any rows in my table until the entire transaction is finished.
my log file is growing like crazy and it will probably run out of space.
if something breaks in the middle, I have to repeat everything.
If I add SET ROWCOUNT 500, I can limit the number of rows, but I don't know how to continue with the last inserted ID. I might query the new table to see what got inserted last, but I am not sure if that's the right thing to do. And it's a bit difficult because my where clause does not use the ID column, so I am not sure how to know exactly where to continue.
What's the best approach for this? Is there a "for loop" or something which would allow me to commit every once in a while?
I am using SSMS for SQL Server 2008 R2.
Even if TomTom's answer is sarcastic, it contains two basic options, which may help you:
You can write a loop in T-SQL (see for example while) and use TOP and OFFSET to select chunks (you need an order by). You can minimize looging according to Microsoft. And if you just worry about restarting without redoing everything this should be fine, though I don't expect it to be fast.
You can export your selection to a file and use the bulk insert to load it.
Some more options you may find here (About Bulk Import and Bulk Export Operations) and here (INSERT Section Best Practices).
How do I make SQL Server commit inserts in chunks?
You program code that inserts it in chunks, quite easy.
1: Yes, that is how transactions work, you know.
2: Yes, that is how transactions work, you know.
3: Yes, guess what - THAT IS HOW TRANSACTIONS WORK, you know ;)
but I don't know how to continue with the last inserted ID
Called programming. Generate ID's on client side. Remember the last one you generated.
In general, I would advice not to INSERT in code - depending how your code works, this sounds like a data copy operation, and there are other mechanisms for that (bulk copy interface).
I am using SSMS for SQL Server 2008 R2
Basically it behaves as you program it. You can easily put in some loop in the SSMS side, or export to a file, then bulk insert the file. Wont help you with item 2 though... unless you go to simple backup model and do not care about restoring.
Also 3 is complex then - how do you properly restart? Lots of additional code.
The best approach depends on the circumstances, and sadly you are way too vague to make more than a blind guess.