I'm new to Redshift and quite a beginner in AWS. I have a Redshift Cluster, and I need to execute a bash script- that has some SQLs running inside of it.
Is there any way I can execute my Bash script on my Redshift Cluster? I want to be able to connect to the Redshift Cluster, execute the Bash Script and all the SQLs inside on the cluster.
Can I do this through Lambda? A little detail will be appreciated.
Amazon Redshift is a database. You can connect to it via JDBC or via the Redshift Data API and then run SQL commands against the database. However, it is not a compute platform.
While Amazon Redshift does have the ability to run stored procedures, bash scripts cannot be run.
You would need to run your script on a compute platform (eg an Amazon EC2 instance or your own computer), and it can 'call' the Amazon Redshift cluster to run the SQL.
AWS Lambda can run bash scripts, although you would need to do some setup to support it. You would be better-off rewriting your code to work in a natively-supported language.
This is possible through lambda if you wanna use the bash script. But if you wanna just execute sqls, wrap those sqls in a .sql script and use Redshift query scheduler to schedule these via either manual schedule run or cron job schedule run
Related
i'm learn about airlow
I wrote dag about hive operator task.
I don't know what happen when apache hive execute done, the hql is error and how hive interact with airflow?
thanks a lot.
I have a MySQL DB on AWS.
I want to run a few simple SQL statements that select data from MySQL and insert to Azure DB.
Something like
select *
into Azure_Table
from
MySQL_Table
I also want to schedule this on a daily basis.
How can I do it directly from Azure SQL without having to use Data Factory / SSIS
Thank you
You can use Data Ingestion in ADF.
You can select the source and sink. Then schedule as per your need.
Note: Since you have the Source as MySQL on AWS i.e. outside of Azure Cloud, you would have to setup Self-hosted integration runtime for the linked service at source. Follow official MS doc for Setting up a self-hosted integration runtime using UI.
You can Migrate Amazon RDS for MySQL to Azure Database for MySQL using MySQL Workbench.
You can refer to below official documentation where you can get step by step explanation:
Migrate Amazon for MySQL to Azure Database for MySQL using MySQL Workbench.
Workaround – There is no direct way to query third-party database from Azure. But, you can migrate it to Azure and then perform operations.
If I have an Azure Managed Instance, what is the difference between scheduling jobs that contain T-SQL and/or packages within the MI on SSMS using SQL Agent and Azure Data Factory? Does the scheduled job run on the VM/Managed Instance instead of the IR in Data Factory? If so how does this handle SSIS Packages? I assume it will just fail?
I thought you could only schedule the jobs with SSIS packages inside Data Factory but I guess I was wrong as it has let me create a schedule (albeit it has not run yet).
Some of these jobs have been created from an offline sql server DB and recreated using a T-SQL script.
EDIT: I am aware of the differences between Data Factory and SQL Agent. I suppose the questions I am asking are:
Can the SQL Agent Job Scheduler run Jobs that have both a T-SQL step and a SSIS package step just like a normal sql server or do the packages strictly have to be scheduled to run in Data Factory with SSIS IR? The official Microsoft Documentation is confusing
If so, what is the best way to run a job that uses both T-SQL scripts and SSIS Packages in a Managed Instance?
For your first question "Can the SQL Agent Job Scheduler run Jobs that have an SSIS package in a step or do the packages strictly have to be scheduled to run strictly in Data Factory with SSIS IR?
Yes, there many tutorials talked about this, you could ref this official document:
Microsoft: SQL Server Agent Jobs for Packages: You can automate and schedule the execution of SQL Server Integration Services packages by using SQL Server Agent. You can schedule packages that are deployed to the Integration Services server, and are stored in SQL Server, the SSIS Package Store, and the file system.
Azure SQL managed instance support SQL Agent job and can schedule run the SSIS packages, Data Factory also could achieve that with SSIS-IR.
Second question, what is the best way to run a job that uses both T-SQL scripts and SSIS Packages in a Managed Instance?
Azure SQL managed instance has almost the same feature with on-premise SQL Server. So you could create the SQL Agent job to execute the T-SQL scripts or SSIS Packages like on-premise.
HTH.
I need to migrate data from SQL Server to S3, I would like to do that with GLUE.
Basically, I have SQL Server in place in my VPC and I would like to copy it to S3. I searched for several tutorials on the web and did not find, I would like to know how do I connect to the data source's SQL Server and run a SQL query with a dynamic condition. Would have any tutorial or command how can I do this with Glue?
I'm trying to work out how to upload a 15gb database to my Amazon AMI on EC2. It keeps dropping. My ISP gives 1.5mb/s upload. Also my download speed is goes to neraly nothing when I am uploading. Is there a way to send the database over quickly using EC2?
Thanks
You can use any database GUI manager to upload the database. Connect to your EC2 database using your database GUI manager and upload the data from your local system. If you are using MySQL as your database I will suggest SQLYog MySQL GUI manager, Connect to your EC2 using ssh tunneling and upload your data.
Consider using rsync with options like:
rsync -PazSHAX LOCALDIRECTORY/ REMOTEHOST:/REMOTEDIRECTORY/
Keep running it until it completes successfully as it will resume where it left off.
You can consider Amazon Import/Export for that. It costs some money, though.