I would like to run ETL transformation during my deployment - from current database to transformed and then swap transformed db to current, for ex. when I have to change my documents structure.
Is there any api to do this from the code?
https://ravendb.net/docs/article-page/3.5/csharp/studio/overview/settings/etl
Thanks
The ETL configuration is saved as a replication destination.
You can save the document on the new db from the old one, and it will work
Related
I have been using AWS Cloudformation and Terraform to manage cloud infrastructure as code (IAC). The benefits are obvious.
1) Template file to concisely describe your infrastructure
2) Versioning
3) Rollbacks
I also have a PostgreSQL DB where I can dump the schema into a single file. Now, it would be amazing if I could edit a dumped SQL file like I do a (IFC) template. I could then validate my new SQL template and apply changes to my DB with the same workflow as Cloudformation or Terraform.
Does anyone know if a tool like this exists for any of the various SQL providers?
Have you given Flyway a try?
It supports versioning database migrations as well as rolling back and undoing migrations when needed. It also keeps a schema table in the database that tracks which migrations have been applied to that database, so that you can continuously deploy new scripts and changes to an existing application that is using Flyway.
I'm new with Pentaho and I'm trying to set up an automatic deployment process for the pentaho business analytics platform repository, but I'm having troubles to find out how to proceed with the data sources.
I would like to do export/import all the data sources, the same that here is explained with the repository (Reporting, Analyzer, Dashboards, Solution Files...) but with the data connections, mondrian files, schemas....
I know there's way to backup and restore the entire repository (explained here), but that's not the way I want to proceed, since the entire repository could contain undesired changes for production.
This would need to be with command line or rest system or some other thing that be triggered by Jenkins.
Did you try import-export with the -ds(DataSource) qualifier ? This will include the data connection, mondrian schema and metadata models.
Otherwise, you can import everything, unzip, filter according a certain logic (to be defined by the guy in charge of the deployment), zip again and export it to prod. A half day project with the Pentaho Data Integrator.
I'm working on a BI project and I want to retrieve data from ServiceNow and load it to Pentaho Data Integration so I can record it in my data warehouse, and I want to do this regulary, in other words I want to retrieve the new records regulary from servicenow , only the new ones that haven't been loaded yet to the data warehouse, someone knows how can I acheive my goal? Help me please
The question is too vague.
You need to set up an ETL job that incrementally loads data. That will require you to define a timestamp or incremental key to identify which records are more recent than the ones already loaded.
You will need to schedule that job, e.g., using crontab and calling kitchen from the command line.
Your question pretty much translates to "please develop my ETL project". Too wide in scope.
I would like to migrate few tables from one database server to other database server (both are SQL server). How can I do this? I have heard about SSIS package but never done this. I would like to understand this process in detail.
Source database is refreshed daily. what can I do to refresh my destination database to reflect the source database all the time.
Please help me, I would like to understand this process from beginning till end because I have never done this before.
Thanks,
Here are the high level steps.
Create an SSIS project. Use a data flow task for each table. Depending on your requirements you might be able to just clear the tables and just reload them.
Create a new sql agent job on the server to schedule the job. Make sure you schedule after the source database is completely loaded.
I'm looking for an easy way to transfer a database schema I have developed inside visual studio as a strongly typed dataset (xsd file) into a corresponding sql server database. Silly me I assumed the process would be forthright, but I can't find out how to do it. I assume I could duplicate the tables column by column, but that seems so error prone. Does anyone know of a way to perform the schema transfer like this? Maybe a tool to translate the xsd file into a corresponding sql server ddl file?
Final thought once I have the schema transferred moving data around between the two data stores will be straight forward, its just getting the schemas synced that has me stumped...
Thanks,
Keith
Why didn't you implement your data model directly in SQL Server ?! It is more common and engineered and I think this is why Microsoft has not provided any wizard or tool for this case. As well you can make your data model as scripts or .sql files and they can be managed via SVN and whenever you need the model implementation you can sue them.