Visual Studio Writes Partial Data to Excel File - sql

I have a visual studio package that does not write all the data needed to a blank excel file.
More specifically, the package goes through these steps:
Copies a template excel file to overwrite a shell file.
Connects to a SQL DB.
Runs a Select Statement.
Converts one column to unicode.
Pastes to shell file.
There are a few more steps afterward (like emailing the excel file) but those work fine.
The issue comes up for step 4. when Visual Studio or SSIS runs the package, I pull about 1400 rows. When I just run the select statement in SQL Server Management Studio or as a connection in Excel I pull about 2800 rows. 2800 is the right number.
I've tried building the process from scratch (excel files, connection files, etc.) but that rebuild elicits the same result. It's like Visual Studio just doesn't like the select statement. Double checked the mappings - all good. The data is pasting and being delivered fine, just not enough. No errors on visual studio either - it gives me that lovely (albeit confusing) check mark.
This was running as an automated package for about a year before this happened and I have no explanation. Seriously a headscratcher.
The only other clue I have is that when I pull the data manually with the select statement, there are no null values in a particular column, but when I run the package with that exact same select statement the output contains a null in the referenced column - almost as if the select statement in Visual Studio is pulling slightly different data than the manual pull, but the statements are exactly the same, so I don't know why that would be.
Any ideas?

I've seen this issue before. The timeout was set to an low value on the connection and caused it to only pull part of data before the timeout hit and killed the connection. Make sure you are not swallowing any exceptions and double check your timeouts.

Thanks for replying folks!
In the end, I solved the issue by completely remaking the package. While trying your solutions above, I was using the same file but building the connections and queries from scratch. Once I started from a new file it ran without error.
I guess to all those new folks to Visual Studio - always consider remaking the file from nothing!

Related

Cannot open a Script Task, but only one specific SSIS package, any suggestions please?

For only one particular SSIS package (v. 2005) I am getting the following error when trying to open the script task...
TITLE: Microsoft Visual Studio
Cannot show the editor for this task.
ADDITIONAL INFORMATION:
The operation could not be completed. (Microsoft.VisualBasic.Vsa.DT)
BUTTONS:
OK
I need to get into this to be able to edit it, also I am in the process of upgrading it to 2014. Once upgraded I can get in, but there is no code, so assume the upgrade is not working as it=self cannot see the code within.
I have tried other machines - same problem.
I have tried other packages - they work fine - even in the same solution.
I have tried a few resets found on the net/re-installs - same problem.
Clearly its something to do with this specific package only, but I am stumped.
I would expect to be able to open the Script task like any other, and be able to edit it. I would also expect the upgrade to work and contain the code.
If you want to read the Script Task code, open the package file (.dtsx) using a text editor (Notepad++), and search for the Script Task code, copy the Script Task code and recreate the script and paste the code within the Script editor.
If you have a problem with Script Tasks in visual studio 2005, then copy the code to an external file, upgrade the package to 2014 then paste the code inside the Script Task (since it will be empty after upgrade)

SSIS Error: VS_NEEDSNEWMETADATA

I'm currently updating all of our ETLs using Visual Studio 2015 (made in BIDS 2008) and redeploying them to a new reporting server running on SQL Server 2016 (originally 2008R2).
While updating one of the ETLs and trying to run on the new server I got this error:
The package execution failed. The step failed.
Sometimes it also produces this error:
Source: Load Fact Table SSIS.Pipeline Description: "Copy To Fact
Table" failed validation and returned validation status
"VS_NEEDSNEWMETADATA".
I've tried deleting and re-adding the OLEDB Destination, connection strings and opened up the column mappings to refresh the meta data. I also recreated the whole data flow task but I'm still getting the same error.
The package runs fine on my local machine.
UPDATE:
I started taking the package apart and running only pieces of it to try and narrow down which part was failing. It seemed to be failing on loading into the staging table but I couldn't find out why.
I eventually decided to just try and re-create the whole thing. After re-creating the entire package, still no luck. The picture below is from the event viewer on the server itself but it didn't give me any new information.
Package error from event viewer
I have tried all the solutions provided above and the other sites. Nothing worked.
I got a suggestion from my friend Which worked for me.
Here are the steps:
Right click on the Source/Target Data flow component.
Go to Advanced Editor -> Component Properties
Find ValidateExternalMetadata and set it to False.
Try your luck. This is a pathetic issue and left me clueless for 2 days.
I finally found the issue and here's how I did it.
Because the error messages I was getting from SSMS weren't very insightful I first opened up my remote desktop and logged into the server. Then I went to Administrative Tools>Event Viewer and then Windows Logs>Application to see if the failed event would provide greater detail.
It didn't give me much still.
The next step I took was to run the package from the command line because the messages should be more verbose. Opened up cmd, changed directory to the one my package was in and then...
DTEXEC /FILE YourPackageName.dtsx
Finally, the error message here showed a missing column in the tables the package was trying to write to. I added those columns and voila!
As stated in comments,
if it runs ok in your development environment, then the problem isn't with the package, it's with the scheduled job on the server. Try recreating that.
If that doesn't work,
It seems like the server has a cached instance of the package it's using instead of the updated one. Try renaming your package and creating a new job with the new package name and see if that works.
If that doesn't work,
all I can recommend at that point is to cut the package down until it succeeds, then add the next step that fails.
Sounds like from your solution the development environment is more forgiving of schema updates than the deployed solution. Glad you were able to resolve, eliminating clutter helps.
I had the same problem and my issue was a difference between two environments, the same field in the same table once was written with a capital and once not. So the name was the same, but with this small difference (e.g. isActive vs IsActive).
This came from a refactoring effort, where we used VS database publish that did not updated the field name.
Have you tried deleting and re-creating the source? When I get this I can generally modify OK any object that has the error but have to delete and rebuild the paths between them, however sometimes I have to delete everything in the data flow and re-create it.
A Proxy for SSIS Package Execution should be created under the SQL Server Agent. You should then change your job step (or steps) to Run As the Proxy you've created.
I had your same problem some time ago and the proxy fixed it.
Forgive me if you've already tried this.
It is very common to get that message when 2 columns in the source file are being inserted into the same field of the table.
i.e.
My text file has twice "neighborhood" (same label for different columns) and my table has "neighborhood" and "neighborhoodb" (notice the "b" at the end). The import will try to import both text columns into the field "neighborhood" and ignore the "neighborhoodb" field, it will fail with the "VS_NEEDSNEWMETADATA" error.
Re-creating the job worked for me. Some cached version of the job may have been causing the VS_NEEDSNEWMETADATA error. The package was executing correctly but it was failing, when it was executed by an agent job.
This ended up being a permissions issue for me. The OLE DB Source was using a stored procedure that selected from a SQL view. This view joined to a table in another database and unfortunately the proxy account the SQL Agent job step was running the package under did not have SELECT permission to the table in that database. This is why the package ran fine in Visual Studio but not from a job when deployed to the server. I found the root cause of the error by taking the SELECT statement out of the stored procedure and putting it directly in the Source Query box of the OLE DB Source control which caused it to finally return the 'SELECT permission denied' error message. This error was apparently hidden from SSIS since the proxy account DID have execute permission on the stored procedure.
It works for me after changing the ValidateExternalMetadata to false. I was transferring the data from MSSQL database to MySQL database. Changed "ADO NET Destination".
You may need to strongly type your Source Query.
Example:
If your DestinationDB has a FullName field Nvarchar(255)
and in your source query you have
select firstname + lastname as FullName from...
Try this:
Select CONVERT(NVARCHAR(255),firstname + lastname) as Fullname from...
So if you are going from db to db and both are nvarchar(255) I don't have this issue, but if you are concatenating fields in your query specify the data type and length.
This error can also occur when an entire SSIS project needs to be redeployed rather than just one of the packages (for VS versions that allow deployment of a single package in a multi-package project), particularly when a project connection has been changed or added. For example, if you've added or removed columns from a flat-file project connection. In that case, you need to deploy the entire project to push out the updated project connection properties. This can be true even if the project only has one package in it. In VS Solution Explorer, rather than click on the package name to deploy, select the bolded project name at the top, and then click deploy.

SQL Sever 2012 - generating scripts - Save to File = Not Run

I am creating scripts of a SQL Server 2012 database because I cannot backup the database to a local drive. I understand how to create the script but at the end of the process the application seems to get stuck at the Save to file = Not Run.
The database is a huge database, but it appears that not much data is being written to the drive.
This appears to be a bug in SQL Server 11.0.6020 tools.
There is no trace in event log or server log, the script generator wizard just stops at the last step (which is writing the script to the destination, which can be a file or a new script window - either remains in status "not run" forever, with "Cancel" as the only possible user action).
Some experimenting showed that it indeed depends on script size.
I was not able to reproduce the problem on any lower or newer version of Microsoft SQL Server.
The solution is annoying: click yourself through the wizard multiple times, first scripting only database definition in parts:
datatypes, functions and tables
then only views, and
then only procedures
You can later concatenate the three resulting files if that is a requirement.
This approach will not help if any of the three parts alone is bigger than the (unknown) treshold. Eventually, script the data, selecting only smaller sets of tables for each run.
As mentioned in one of the comments on the original post, this is still an issue with SQL Server Management Studio v17.9.1 To work around it, I was able to use the "Single file per object" option. I definitely would have preferred a single file, but at least it worked this way. There was still a bit of a delay between the time that the status for all database objects showed Completed and the time that the "Save to file" line item changed from "Not Run" to Completed.
I'm using SQL Server Management Studio version 18.6. I wanted to export to a query in a new window:
But it resulted in it saying "Not run":
And a few seconds later, it said "Error":
Instead, what worked, was to save the result to a script file:
And here is the successful result:

Automating Monthly SQL JOB/Query

I frequent this site a lot but have never posted but here goes! I am fairly new only about a month in on the job but have had some experience with SQL before.
I have a simple query that runs monthly which counts the number of active members and notifications sent per Organization for the month.
I have created a one-step job with SQL SERVER AGENT that runs the query on the 5th of the month and records the information for the previous month.
I have the output of the job going to a file named MonthlyReport.txt. This .txt file is then mailed to the client.
The client opens the file with Excel as default... this removes all the formatting. I recommended opening Excel and importing the document and this has temporarily resolved the issue.
However, there are TWO very big issues:
1) asking the client to import the file for formatting is very inconvenient. Importing the file myself will create a lot of overhead as there are several of these reports for multiple databases.
2) the .txt file includes the lines "MonthlyReport' : Step 1, 'Collect Data' : Began Executing 2015-03-17 12:39:58"------ This really messes up the formatting of the column headers
I am looking for other ways to resolve this task by exporting directly to EXCEL or a formatted .txt file
I have tried saving the output as MonthlyReport.csv but the problems still remain and this requires importing into excel
FYI-- My company is running Windows Server 2012 which has SSIS functionality but we also are running a few legacy Windows Server 2008 R2 and I need the solution to work on both servers so SSIS packages are not compatible with Windows Server 2008.
I am sorry for the longwinded response and appreciate all the time and help that the community is able to provide.

Visual Foxpro SQL Server - Can't find the Call to SQL server in Foxpro

My DBA's are saying my foxpro application or .DBC (Database container) are hitting SQL server but searching all the code can't find the SQL call (FMTONLY ON/OFF).
This is the SQL command being sent:
FMTONLY ON/OFF
Getting called 16260 times every few minuets?
Any ideas how to find this or what could be causing it, maybe my DBC file?
If you can't find it embedded in the .DBC, but not entirely sure its NOT in there, you can use a VFP tool to dump its contents to a .prg file... GENDBC which is in your installation folder of {VFP}\Tools\GenDBC\GenDBC.prg
Open your database, then run that program, it will cycle through all the tables, indexes, relations, connections, etc and generate the code corresponding to everything in it... You could then look at the output .prg file and see if something in there might be triggering what you can't see otherwise.