We are trying configure AWS Databricks Runtime to use the AWS Glue Data Catalog as its metastore. In this environment ,Azure ADLS is one of the source system.In that case,Can AWS Glue catalog point to a data location in Azure ADLS?
AWS glue catalog can speak JDBC, so if you can configure Azure ADLS to speak JDBC, which it seems like you can, you should be able to do this.
glue catalog documentation
Related
I need information about how to load data to S3 from Oracle with informatica powercenter 10.2, I start creating mapping but I don't know how to create a target file for amazon S3 and configure the connection to S3 buckets.
I found that I should create amazon S3 data objects in developer tools which need powerexchange Amazon S3.
I worked in another requirement in order to load data into Redshift but here we used ODBC connection.
There is anyone who can tell me more structured details about how to create target for S3 and configure connection and the file size?
Is there any simple tool to help me copy data from Amazon S3 to Azure Blob or Azure Data Lake Gen2?
Azure Data Factory team recently built a storage explorer extension, which is used to copy data from Amazon s3 to Azure Blob or Azure Data Lake Gen2 with simple drag and drop.
Check it here:
https://github.com/Azure/Azure-DataFactory/blob/main/StorageExplorerExtension/storage-explorer-plugin.md
Demo: https://www.youtube.com/watch?reload=9&v=GacGa5T0flk
How can we migrate on-premise data to Amazon s3.
1) MaprDB binary table data migration to AWS Hbase.
2) Hive data migration to AWS Hive tables.
Should we use S3DistCp or AWS DataSync Or there are some other way for the same.
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-3x-s3distcp.html
the best option as of now from AWS side to explore more on DataSync if you are doing online data migration.
want to have central hive meta store to consume from databrick, spectrum etc ..
Is it possible to setup w/o installing hadoop
Yes, Hive metastore installation does not require Hadoop.
Querying data from the Hive metastore requires a Hive client (within Spark) and a Hadoop compatible filesystem (such as S3)
AWS Glue Data Catalog is the recommended system nowadays, not RDS
I want to connect to Amazon S3 from Oracle Data Integrator.
I could not find an api for Oracle Data Integrator.
How can I connect to S3 and retrieve data? Can I connect via Pl/Sql?
from ODI we do not have any direct technology defined in the topology to connect to Amazon s3, however if i am not wrong they must be able to generate WSDL web services through which data can be accessed into ODI.
As mentioned by Adrian in this Oracle Community thread:
ODI do not natively integrate with Amazon.
The only way to integrate with Amazon S3 is using CLI commands