I'm currently updating all of our ETLs using Visual Studio 2015 (made in BIDS 2008) and redeploying them to a new reporting server running on SQL Server 2016 (originally 2008R2).
While updating one of the ETLs and trying to run on the new server I got this error:
The package execution failed. The step failed.
Sometimes it also produces this error:
Source: Load Fact Table SSIS.Pipeline Description: "Copy To Fact
Table" failed validation and returned validation status
"VS_NEEDSNEWMETADATA".
I've tried deleting and re-adding the OLEDB Destination, connection strings and opened up the column mappings to refresh the meta data. I also recreated the whole data flow task but I'm still getting the same error.
The package runs fine on my local machine.
UPDATE:
I started taking the package apart and running only pieces of it to try and narrow down which part was failing. It seemed to be failing on loading into the staging table but I couldn't find out why.
I eventually decided to just try and re-create the whole thing. After re-creating the entire package, still no luck. The picture below is from the event viewer on the server itself but it didn't give me any new information.
Package error from event viewer
I have tried all the solutions provided above and the other sites. Nothing worked.
I got a suggestion from my friend Which worked for me.
Here are the steps:
Right click on the Source/Target Data flow component.
Go to Advanced Editor -> Component Properties
Find ValidateExternalMetadata and set it to False.
Try your luck. This is a pathetic issue and left me clueless for 2 days.
I finally found the issue and here's how I did it.
Because the error messages I was getting from SSMS weren't very insightful I first opened up my remote desktop and logged into the server. Then I went to Administrative Tools>Event Viewer and then Windows Logs>Application to see if the failed event would provide greater detail.
It didn't give me much still.
The next step I took was to run the package from the command line because the messages should be more verbose. Opened up cmd, changed directory to the one my package was in and then...
DTEXEC /FILE YourPackageName.dtsx
Finally, the error message here showed a missing column in the tables the package was trying to write to. I added those columns and voila!
As stated in comments,
if it runs ok in your development environment, then the problem isn't with the package, it's with the scheduled job on the server. Try recreating that.
If that doesn't work,
It seems like the server has a cached instance of the package it's using instead of the updated one. Try renaming your package and creating a new job with the new package name and see if that works.
If that doesn't work,
all I can recommend at that point is to cut the package down until it succeeds, then add the next step that fails.
Sounds like from your solution the development environment is more forgiving of schema updates than the deployed solution. Glad you were able to resolve, eliminating clutter helps.
I had the same problem and my issue was a difference between two environments, the same field in the same table once was written with a capital and once not. So the name was the same, but with this small difference (e.g. isActive vs IsActive).
This came from a refactoring effort, where we used VS database publish that did not updated the field name.
Have you tried deleting and re-creating the source? When I get this I can generally modify OK any object that has the error but have to delete and rebuild the paths between them, however sometimes I have to delete everything in the data flow and re-create it.
A Proxy for SSIS Package Execution should be created under the SQL Server Agent. You should then change your job step (or steps) to Run As the Proxy you've created.
I had your same problem some time ago and the proxy fixed it.
Forgive me if you've already tried this.
It is very common to get that message when 2 columns in the source file are being inserted into the same field of the table.
i.e.
My text file has twice "neighborhood" (same label for different columns) and my table has "neighborhood" and "neighborhoodb" (notice the "b" at the end). The import will try to import both text columns into the field "neighborhood" and ignore the "neighborhoodb" field, it will fail with the "VS_NEEDSNEWMETADATA" error.
Re-creating the job worked for me. Some cached version of the job may have been causing the VS_NEEDSNEWMETADATA error. The package was executing correctly but it was failing, when it was executed by an agent job.
This ended up being a permissions issue for me. The OLE DB Source was using a stored procedure that selected from a SQL view. This view joined to a table in another database and unfortunately the proxy account the SQL Agent job step was running the package under did not have SELECT permission to the table in that database. This is why the package ran fine in Visual Studio but not from a job when deployed to the server. I found the root cause of the error by taking the SELECT statement out of the stored procedure and putting it directly in the Source Query box of the OLE DB Source control which caused it to finally return the 'SELECT permission denied' error message. This error was apparently hidden from SSIS since the proxy account DID have execute permission on the stored procedure.
It works for me after changing the ValidateExternalMetadata to false. I was transferring the data from MSSQL database to MySQL database. Changed "ADO NET Destination".
You may need to strongly type your Source Query.
Example:
If your DestinationDB has a FullName field Nvarchar(255)
and in your source query you have
select firstname + lastname as FullName from...
Try this:
Select CONVERT(NVARCHAR(255),firstname + lastname) as Fullname from...
So if you are going from db to db and both are nvarchar(255) I don't have this issue, but if you are concatenating fields in your query specify the data type and length.
This error can also occur when an entire SSIS project needs to be redeployed rather than just one of the packages (for VS versions that allow deployment of a single package in a multi-package project), particularly when a project connection has been changed or added. For example, if you've added or removed columns from a flat-file project connection. In that case, you need to deploy the entire project to push out the updated project connection properties. This can be true even if the project only has one package in it. In VS Solution Explorer, rather than click on the package name to deploy, select the bolded project name at the top, and then click deploy.
Similar to this post
I have an SSIS Package with a Script Task that creates an Excel file on disk and populates it with data from a SQL Stored Procedure (using Microsoft.Office.Interop.Excel). This works great when testing and when running the deployed package manually through the SSIS Catalog, but when I schedule the task to run automatically through SQL Server Agent, the Package fails in the Script Task step. I have the Job running as a Proxy account that is the same as the account I'm logged into the server with when testing (and the same as the account that works when manually running the packages).
My understanding is that even though the job is running using a Proxy, any desktop interaction occurs within the Profile context of the SQL Server Agent login. Since that profile isn't actively logged in, the interaction fails. Digging in more, there is a bool System Variable in the package called "InteractiveMode" that is set to "False". I have a feeling that if I could switch that to True, everything would be hunky dorey. Trouble is, that variable is only accessible to my Script Task as "ReadOnly"...
Is there any way to set the System:InteractiveMode Variable in an SSIS package manually or programatically at runtime? Please help! I'm having to run these scheduled jobs manually for now, which is a big pain.
Thanks.
I had this problem a few months ago and it turned out that the execution options needed to be set to use 32 bit runtime. If you're using SQL Server 2008 R2, you can open your job and double click on the step. It's under the Execution Options tab.
If you continue to have errors, you may want to consider changing the package so that it uses a file system task to create/rename the excel document and then a Data Flow Task to move the data from your stored procedure to your excel document. Depending on your data, you may need to add a Data Conversion step in between. Here's a good article on the topic: http://www.mssqltips.com/sqlservertip/3046/sql-server-integration-services-data-type-conversion-testing/
Edit:
I haven't used SQL Server 2012 yet, but according to MSDN, it looks like the option is under the Configuration tab. Here's their article: http://msdn.microsoft.com/en-us/library/gg471507(v=sql.110).aspx
I have generated a script for the SQL Server Agent job that I created.
However, when I run the script even though the job is created successfully (able to verify it through sysjobs_view system view) but it does not get listed under SQL Server Agent jobs list.
Does anyone know what the issue could be?
Thanks!
If dbrosier's suggestion to refresh the list doesn't work, I would check to see if you have a filter set on your jobs list. I would also wonder if it's a permissions issue, but in that case, you shouldn't be able to see any of the jobs and should get an error.
I got an error message in my Log Reader Agent:
The process could not execute 'sp_replcmds' on 'Database Name'.
I created another agent profile with a large query timeout and a min value to batch, but it still doesn't work.
Can someone help me?
I'm using SQL Server 2008 and I'm trying to do a replication between databases on different servers.
It could be possible that owner of the database could be someone other than what you have permissions for. Below there's a simple command to change ownership...if you have the rights to do so.
--TSQL Code--
USE PublishedDatabase
GO
EXEC sp_changedbowner 'sa'
GO
There are a lot of things that can cause this error (which include, but is not limited to):
The database has been publication disabled
The account trying to run the log reader agent doesn't have the ability connect to the publisher server
The account trying to run the log reader agent doesn't have permission to run sp_replcmds
In my experience, there's a little more to the error in the replication monitor. Is this the case for you?
This could be due to Owner is not set for the database.
You can check by right clicking on database then choose Property and go to File Table and the Owner selection should be there.
I want to take backup of jobs.I know all of these are stored in MSDB database,so i had taken a backup of the msdb database but i'm not able to restore that database as i'm getting a error while restoring on another system that the process terminated abnormally.So,what should i do to restore that.
Thanks Nick Kavadias
But the problem is that when i script my job,there r four steps.I dont get these four steps in detail. There is a condition which is used in those four steps which automatically generates an exception.But i'm not able to find that condition in the script
probably belongs on serverfault. But i think i know what Tarun is talking about.
IMHO the best way to take a copy of all your SQL Server agent jobs is to script them out using management studio. You can do this by clicking on jobs in object explorer, then object explorer details window. All jobs in this instance will be listed here, Select all, and pick script jobs from the right click context window. This will generate a t-sql script with all the jobs on the server.
If you do wish to restore the msdb database ( to keep job history ect.) then you will need to stop the SQL Server Agent service first.
Recently I got a chance to transferred the jobs from SQL Server 2000 to SQL Server 2005 using SSIS (BIDS)
In SSIS there Jobs Transfer Task, you needs to select and give the name and select the source and destination servers, you can transfer all the jobs or specific jobs.
Please try this option, I think it will solve your problem
Cheers
Venkat