Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 days ago.
Improve this question
I'm looking for a lib or a command line tool that can help me to do data migration from Redis to Amazon DynamoDB. Does anyone know a tool or lib that can do the job?
Thanks!
I would suggest you have a look at redis-rdb-tools to extract data from Redis.
This package can dump the content of the Redis database as a JSON file. You can then use any loader tool provided by Amazon to feed their database (or write your own).
For instance the AWS command line interface support feeding the DynamoDB with JSON:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.CLI.html
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/index.html
You may have to transform the JSON file in order to use the AWS CLI commands though.
Amazon recommended way to bulk load data into DynamoDB is Amazon EMR (i.e. map/reduce jobs). http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EMRforDynamoDB.html
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 days ago.
Improve this question
I want to create a java utility to read S3 bucket information.
We can connect to s3 via native s3 APIs and the Hadoop filesystem approach.
Approach 1: Using S3 APIs
AmazonS3 s3client = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withRegion(Regions.valueOf(region))
.build();
Approach 1: Using Hadoop Filesystem:
configuration.set("fs.s3a.access.key","XXXXXXXXXXX");
configuration.set("fs.s3a.secret.key","XXXXXXXXXXX");
configuration.set("fs.s3a.impl","org.apache.hadoop.fs.s3a.S3AFileSystem");
configuration.set("fs.s3a.endpoint","http://127.0.0.1:8080");
UserGroupInformation.setConfiguration(configuration);
fileSystem = new Path("s3a://"+ bucketName).getFileSystem(configuration);
Do we know when we use which approach? Which approach is more efficient to read data?
In my observation, the filesystem route is slower. But I have not found any documentation supporting the performance difference.
Performance shouldn't be the only factor. If you want higher performance, or at least better file operation consistency guarantees, look into S3Guard.
But if you have to create a Java client that will only ever talk to S3, and never needs to integrate with Hadoop ecosystem, or use other Hadoop compatible filesystems (HDFS, GCS, ADLS, etc), then you should use plain AWS SDK.
If you're trying to run some mocked S3 service (or MinIO) on 127.0.0.1, then that's not a proper benchmark to a real S3 service
Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 months ago.
Improve this question
I have been using rclone to back up google drive data to AWS S3 cloud storage. I have multiple google drive accounts whose backup happens on AWS S3. All those google drives have different numbers of documents.
I want to compress those documents into a single zip file and then it needs to be copied on S3.
Is there any way to achieve the same?
I referred to the link below, but it doesn't have complete steps to accomplish the task.
https://rclone.org/compress/
Any suggestion would be appreciated.
Rclone can't compress the files, but you can instead use a simple code to zip or rar the files and then use rclone to back them up to AWS.
If this is OK, I can explain the details here.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
We would like to watermark images automatically, after its uploaded to S3. How do we achieve this using a java based library?
The flow should be as follows to watermark images whenever it is uploaded:
Raw image (without watermark) is uploaded to S3 bucket
Use S3 events (such as s3:ObjectCreated:* based on your need) to trigger Lambda. Hence, you need to configure S3 event destination as Lambda.
Lambda shall apply watermark and upload watermarked image
NOTE: If you don't use Lambda, other way is via SQS/SNS as event destination, and your worker shall take request from it and apply watermarking.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I need to generate reports based on a dataset pulled from a third party API, but I can't store the data on disk. (If not storing the data were not a requirement, I would be storing the data in a relational database, and writing a query to join several tables to generate the export as a CSV, for instance.) I've been reading about Redis and I wanted to know if that is a potential solution here as a temporary datastore? Or would I have a hard time putting the tables in the dataset together? If not Redis, what is the recommended way to cache data for reporting purposes in an Azure environment?
I'm filling in a lot of the gaps with assumptions, but to answer your question, yes.
Azure Redis Cache could be used to run your reports "in memory" generally speaking.
For the solution, "it depends" on:
the type of data
how you ingest the data
the type of reports you are trying to run
You have a platform that can run reports with Azure Redis Cache, but you still need to model the data properly to build your reports. Redis is not a relational database. Without more details, you should start here: https://redis.io/topics/data-types-intro
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I am creating a web-based RESTful service and want to cloud-enable it for scalability.
I don't want to get locked into one cloud provider though. I'd like to be able to switched between Go Grid or Amazon EC2, etc. as pricing and needs evolve.
Is there a common API to control the launch, monitoring and shutdown of cloud resources?
I've seen Right Scale, but their pricing is just from another planet.
Similarly, is there a common API for cloud storage?
Take a look at libcloud
If you are working with scala or java, you can also check jclouds(http://groups.google.com/group/jclouds)
For Java there is:
typica
jclouds
Dasein
AWS SDK for Java
What platform are you looking an API for? Our SecureBlackbox product offers CloudBlackbox package of components for uniform access to various cloud storages. Currently supported are S3 and Azure and Google (API for other services is possible on demand).
We don't have uniform API for computational functionality (at least at the moment).