I am using Toad for Oracle 14.2 database it
I have an sql script which checks duplicates and not nulls in various columns (and tables) and returns a count of rows if nulls or duplicates exist.
This data integrity test script is run every two weeks. The script is written in a way that the results of the query are spooled into a text file in a shared folder.
I wish to automate it and schedule its run every two weeks.
I know this can be done in a crontab using Linux.
But I would like to know how could it be achieved using POWER BI with a scheduled refresh
Thank you
This is super-easy in Power BI, once you get the Oracle drivers installed on both your dev environment and on the On-Prem gateway you set up.
This would simply be an import mode Dataset with a custom SQL Statements and a scheduled refresh.
You may have to break the script up into multiple queries in Power Query. If they all return the same columns you can merge them into a single table in the Power BI model.
Related
I have a pipeline which runs sql counts everyday as part of regression testing, I would like to be able to export the results of each build into an excel document where the results can be stored and compared, is this possible?
Thanks in advance
I am using a LinkedServer in SQL 2012 and refreshing a table from Oracle 9G using below procedure on daily basis. The current records in the table is 15M and it is increasing every day by 2-3K new records and the old records are also deleting and updating randomly. It takes 7-8 hours to complete this job overnight.Considering the table is already optimized on index level at Oracle side, What can be the most efficient way to attempt this?
My current process is below :
Truncate table SQLTable
Select * into SQLTable from openquery (LinkedServerName,'Select * from OracleTable')
It doesn't make sense to truncate 15M rows just for 3000-8000 rows changes.
I would consider using an ETL tool like https://sourceforge.net/projects/pentaho/. You can start with a free community edition.
This tool provides a Spoon tool that basically provides graphical interface to create a workflow. With the Pan tool you can execute the file you create using spoon tool. Basically create a batch file with Pan command and provide .ktr file as an argument. Now, this batch file you can schedule using windows task manager or Unix CRON Job.
With this, you can create a workflow, which can look for changes and only insert or update changes.
I have 2 tables. The source table being from a linked server and destination table being from the other server.
I want my data load to happen in the following manner:
Everyday at night I have scheduled a job to do a full dump i.e. truncate the table and load all the data from the source to the destination.
Every 15 minutes to do incremental load as data gets ingested into the source on second basis. I need to replicate the same on the destination too.
For incremental load as of now I have created scripts which are stored in a stored procedure but for future purposes we would like to implement SSIS for this case.
The scripts run in the below manner:
I have an Inserted_Date column, on the basis of this column I take the max of that column and delete all the rows that are greater than or equal to the Max(Inserted_Date) and insert all the similar values from the source to the destination. This job runs evert 15 minutes.
How to implement similar scenario in SSIS?
I have worked on SSIS using the lookup and conditional split using ID columns, but these tables I am working with have a lot of rows so lookup takes up a lot of the time and this is not the right solution to be implemented for my scenario.
Is there any way I can get Max(Inserted_Date) logic into SSIS solution too. My end goal is to remove the approach using scripts and replicate the same approach using SSIS.
Here is the general Control Flow:
There's plenty to go on here, but you may need to learn how to set variables from an Execute SQL and so on.
I am trying to run a process that rolls over several time periods using data that sits in a Sql Server 2008 R2 Database. If it were smaller I'd pull it all in to R and just subset based on date. However the data is ~15GB and I need to be able to generate csv files directly from Sql Server (I've found the R-Sql connectors to be too slow to move large amounts of data) via the R-SQL connector (RODBC, etc.) and then read them into R.
It seems that sqlcmd or bcp are the only options but I wanted to check before going in that direction.
Any suggestions would be appreciated.
I use the package RODBC a lot it works like a charm, large query using sqlQuery can take some times though. Once the query is done you will get a dataframe that you can easily export in csv via write.csv. It will work without any issues.
How do I copy data from multiple tables within one database to another database residing on a different server?
Is this possible through a BTEQ Script in Teradata?
If so, provide a sample.
If not, are there other options to do this other than using a flat-file?
This is not possible using BTEQ since you have mentioned both the databases are residing in different servers.
There are two solutions for this.
Arcmain - You need to use Arcmain Backup first, which creates files containing data from your tables. Then you need to use Arcmain restore which restores the data from the files
TPT - Teradata Parallel Transporter. This is a very advanced tool. This does not create any files like Arcmain. It directly moves the data between two teradata servers.(Wikipedia)
If I am understanding your question, you want to move a set of tables from one DB to another.
You can use the following syntax in a BTEQ Script to copy the tables and data:
CREATE TABLE <NewDB>.<NewTable> AS <OldDB>.<OldTable> WITH DATA AND STATS;
Or just the table structures:
CREATE TABLE <NewDB>.<NewTable> AS <OldDB>.<OldTable> WITH NO DATA AND NO STATS;
If you get real savvy you can create a BTEQ script that dynamically builds the above statement in a SELECT statement, exports the results, then in turn runs the newly exported file all within a single BTEQ script.
There are a bunch of other options that you can do with CREATE TABLE <...> AS <...>;. You would be best served reviewing the Teradata Manuals for more details.
There are a few more options which will allow you to copy from one table to another.
Possibly the simplest way would be to write a smallish program which uses one of their communication layers (ODBC, .NET Data Provider, JDBC, cli, etc.) and use that to take a select statement and an insert statement. This would require some work, but it would have less overhead than trying to learn how to write TPT scripts. You would not need any 'DBA' permissions to write your own.
Teradata also sells other applications which hides the complexity of some of the tools. Teradata Data Mover handles provides an abstraction layer between tools like arcmain and tpt. Access to this tool is most likely restricted to DBA types.
If you want to move data from one server to another server then
We can do this with the flat file.
First we have fetch data from source table to flat file through any utility such as bteq or fastexport.
then we can load this data into target table with the help of mload,fastload or bteq scripts.