I developed a simple job that loads data from multiple excel file to a mysql database in pentaho Kettle, I'm using Kitchen.bat from cmd line to run the job. If i need to move the job to other production serverthe application size is more than a GB, is there any way i could deploy the job without moving all the libraries to the production server.
Related
I have an SSIS package that uses the Execute Process Task to run an application. Running on my local visual studio has no issues. Running it on the server as a job, from the logs, I can see that it just hangs on the EPT step. It never errors or stops.
The old version of the application runs from the same directory as part of a job on a server with no issue. I'm simply just running a new version of the application. It does pass arguments slightly different though.
I log into the server as the SQL Agent service account and run the executable commands from CMD the same as the SSIS package runs it, and it runs fine.
The task is configured with default configurations except for the window is hidden, the same as the old version that works.
I've seen many people have this issue and have tried everything..
I'm assuming that it's prompting me something when it runs on the server. Is there any way to see what it's prompting if I am logged into the server running the job?
We have a database created on SQL Server 2017 (Express edition) on one of our servers and we are trying to move this database to our another server which has SQL Server 2014 (licensed version) installed on it. We have tried to restore the backup, detach/attach database files and also tried generating the script and running them on new server.
But unfortunately we are not able to restore the database. For backup/restore, we are getting following error message.
For generating scripts and running them on new server; the problem is that the file size of the scripts is around 3.88 GB. We are not able to edit this file before executing it because file size is too large. We have also tried to first generate scripts without data and then with data but the file size with only Data comes up to 3.88 GB (there is only very little difference when only Data is selected).
What are the options we have?
Since you probably already have the schema and data scripts.
You could use VSCode with SQL Server extension to execute the scripts against a previous SQL server version. Sublime is also able to handle large files. See this question for all editors with large file support.
But only if you generated the scripts using the correct compatibility level.
Be aware that with the script not everything is scripted by default like triggers. So you might want to change other options.
I am using a pentaho community edition 7. I want to schedule a JOB with two sub transformations in it.
I want to schedule it to run on every monday. Can anyone please guide me in saving the files using the correct filepaths and schedule them from BI server.
You can find a guidance for your question in an accepted answer here:
How to deploy scheduled Kettle jobs on Pentaho BI server v6 CE
Basically:
Add job file and xaction file (triggers job file) to Pentaho server
Add transformation files to server's filesystem
Schedule xaction file in Pentaho using "Schedule..." file action (Weekly, Monday, preferred time can be set up in pop-up dialog)
We have a task where we need to automatically convert an excel file to a csv to prep it for loading into a SQL database. The developers built this process into a SSIS package. For the conversion, they initially tried to have a task in the SSIS package execute a VBscript to convert the file. When they were running this on there local machines, this worked correctly. When they ran the package manually through VS on the server, it ran correctly. When they ran the package manually via the Integration Catalog it ran correctly. We did this both as our accounts and as the service account and got the same results. However, when we scheduled it as a job it would hang on the part of the process that executed the VBScript. No errors, it would just hang until you killed the job.
The job was executing as the service account which has full admin access on the server, explicit full access to the share where the files are stored and converted (which is on the same server) and full admin access to SQL. The job owner is set to sa which uses the service account. And all the job does is execute the package from the integration catalog which works if you run it independently of a job. When we compared the ssisdb execution report for the manual run in the integration catalog using the service account to the job run they looked the same except the job hung on the conversion task and the other did not.
After spending some time trying to figure this out, the developers tried a different solution. They changed the conversion script from using VBScript to using C#. They ran the package from there local machine and once again the package worked. This time when they ran it manually on the server it failed. When we ran it from the integration catalog it failed and when we ran it from a job it failed.
The error we keep getting is "Create CSV file: Error: Exception has been thrown by the target of an invocation" After spending several hours looking into this error nothing suggested seems to be working.
We also tried these same solutions on a newly built server to make sure we weren't dealing with an odd configuration setting that could have been changed (It is a Dev server) and it still failed there.
At this point, we are pretty lost at what is happening. The first solution worked, but for some reason would never work as a job. The 2nd solution works everywhere except when ran on the server.
We are looking at some other solutions to try to get around this. The next thing may be trying to using powershell to convert the file, but not sure if that will bring us back to the same issue. Any suggestion you guys have will be greatly appreciated
Also, we are using SQL Server 2012 dev edition, VS 2012 pro, Windows Server 2012 R2
This might be because of a bug that Excel has when trying to run jobs (that use Excel) and no user is logged on a specific machine. This might affect also the excel library. The solution is to create the following 2 folders:
C:\Windows\SysWOW64\config\systemprofile\Desktop
C:\Windows\System32\config\systemprofile\Desktop
and then restart the machine. :)
(Edited to show that a restart is needed. thanks for Dustin for checking this)
our team currently updates our production databases via the dacpacs/bacpacs since we do not have any client data. This will not be the case in the near future and I'm looking to change the process of only modifying the database via SQL scripts via build automation.
Is managing these SQL scripts in Team Foundation Server and executing them in the build do-able? And If so how have people approached this?
Thanks!
You should probably not be deploying DACPACs during the build process; it's more appropriate to push changes as part of a release/deployment workflow that occurs after a build.
That said, you can publish a DACPAC without dropping the database and recreating it; there are numerous options for controlling the database publishing behavior that can be set at the project level and overridden at the command line if necessary.
There are two main ways of managing SQL data base changes when deploying to production.
DACPAC
Sequential SQL Scripts
Both have there own issue and bonus when trying to deploy. If you control your entire DevOps pipeline then Dacpac is a good option. If you have to deal with corporate dba's then SQL scripts can also be done.