Jmeter-How to copy files from one AWS S3 bucket to another bucket? - amazon-s3

I have tar.zip files placed in newbucket of AWS S3 location. I have script which will cut the file and place it in another S3 bucket. Every time I need to upload the files from local to newbucket as JSSR preprocessor to upload the files from local. Can I do copy paste of file in S3 from one bucket to another bucket ?

I think the "official" way is to use AWS CLI in general and aws s3 sync command in particular:
aws s3 sync s3://DOC-EXAMPLE-BUCKET-SOURCE s3://DOC-EXAMPLE-BUCKET-TARGET
The command can be kicked off either from JSR223 Sampler or from the OS Process Sampler
If you prefer doing this programmatically - check out Copy an Object Using the AWS SDK for Java article, the code snippet just in case:
import com.amazonaws.AmazonServiceException;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.CopyObjectRequest;
import java.io.IOException;
public class CopyObjectSingleOperation {
public static void main(String[] args) throws IOException {
Regions clientRegion = Regions.DEFAULT_REGION;
String bucketName = "*** Bucket name ***";
String sourceKey = "*** Source object key *** ";
String destinationKey = "*** Destination object key ***";
try {
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withCredentials(new ProfileCredentialsProvider())
.withRegion(clientRegion)
.build();
// Copy the object into a new object in the same bucket.
CopyObjectRequest copyObjRequest = new CopyObjectRequest(bucketName, sourceKey, bucketName, destinationKey);
s3Client.copyObject(copyObjRequest);
} catch (AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
} catch (SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}
}
}

Related

Creating a bucket, folder structure and uploading a file in S3MockContainer

hi i have created a testcontainer and want to set up a bucket and folder, and upload a file into the container so i can test my request call.
How do i do this.. i have set up the container
import com.adobe.testing.s3mock.testcontainers.S3MockContainer;
#Testcontainers
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class TestContainerSetUp {
protected static final String BUCKET_NAME = "testbucket";
#Container
public static final S3MockContainer s3MockContainer = new S3MockContainer(DockerImageName.parse("adobe/s3mock:2.4.13"))
.withInitialBuckets(BUCKET_NAME);
I managed to get this to work, by autowiring a S3 client and using it put the object into the bucket before starting my tests

How to save and get plain text data in amazon aws s3 bucket using asp.net mvc?

I am trying to save plain text data at AWS S3 bucket using ASP.NET MVC can you help to achieve this ??
Save and GET data in aws s3 bucket in asp.net mvc :-
To save plain text data at amazon s3 bucket.
1.First you need a bucket created on aws than
2.You need your aws credentials like a)aws key b) aws secretkey c) region
// code to save data at aws // Note you can get access denied error. to remove this please check AWS account and give //read and write rights
Name space need to add from NuGet package
using Amazon;
using Amazon.S3;
using Amazon.S3.Model;
var credentials = new Amazon.Runtime.BasicAWSCredentials(awsKey, awsSecretKey);
try`
{
AmazonS3Client client = new AmazonS3Client(credentials, RegionEndpoint.APSouth1);
// simple object put
PutObjectRequest request = new PutObjectRequest()
{
ContentBody = "put your plain text here",
ContentType = "text/plain",
BucketName = "put your bucket name here",
Key = "1"
//put unique key to uniquly idenitify your data
// you can pass here any data with unique id like primary key
//in db
};
PutObjectResponse response = client.PutObject(request);
}
catch(exception ex)
{
//
}
Now go to your AWS account and check the bucket you can get data with "1" Name in the AWS s3 bucket. Note:- if you get any other issue please ask me a question here will try to resolve it.
To get data from AWS s3 bucket:-
try
{
var credentials = new Amazon.Runtime.BasicAWSCredentials(awsKey, awsSecretKey);
AmazonS3Client client = new AmazonS3Client(credentials, RegionEndpoint.APSouth1);
GetObjectRequest request = new GetObjectRequest()
{
BucketName = bucketName,
Key = "1"// because we pass 1 as unique key while save
//data at the s3 bucket
};
using (GetObjectResponse response = client.GetObject(request))
{
StreamReader reader = new
StreamReader(response.ResponseStream);
vccEncryptedData = reader.ReadToEnd();
}
}
catch (AmazonS3Exception)
{
throw;
}

Java AWS SDK v1 - S3 API - Not able to upload multiple files parallely using multipart api

I have 5 files of size 200 MB each. I am uploading these files parallel y using Executor service and
using TransferManager with multipart threashold = 50 MB and waiting for upload to finish by using blocking method call to upload.waitForCompletion() (which says current thread suspends until
upload succeeds or throws error).
Please find below code excerpts:
private static final ExecutorService executor = Executors.newFixedThreadPool(10);
executor.execute(() -> upload("bucketName", new File(fullFilePath)));
public static void upload(final String bucketName, final File filePath) {
AmazonS3 amazonS3 = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(getCredentals()))
.withRegion(Regions.DEFAULT_REGION).build();
TransferManager tm = TransferManagerBuilder.standard().withS3Client(amazonS3)
.withMultipartUploadThreshold((long) (50 * 1024 * 1025)).build();
final String fileName = filePath.getName();
try {
Upload upload = tm.upload(s3Bucket, fileName, filePath);
upload.waitForCompletion();
log.info("Successfully uploaded file = " + fileName);
} catch (Exception e) {
log.info("Upload failed for file = " + fileName);
log.error(e);
}
}
Main thread does not exits until "Successfully uploaded" is there for all 5 files.
Now this program does not throw any error and prints success for all 5 files but when i open the bucket in aws console, nothing is there.
Can anyone suggest what might be happening here or how to further debug it ?
Code is working fine i was looking into the wrong bucket.

How to download a file using from s3 private bucket without AWS cli

Is it possible to download a file from AWS s3 without AWS cli? In my production server I would need to download a config file which is in S3 bucket.
I was thinking of having Amazon Systems Manger run a script that would download the config (YAML files) from the S3. But we do not want to install AWS cli on the production machines. How can I go about this?
You would need some sort of program to call the Amazon S3 API to retrieve the object. For example, a PowerShell script (using AWS Tools for Windows PowerShell) or a Python script that uses the AWS SDK.
You could alternatively generate an Amazon S3 pre-signed URL, which would allow a private object to be downloaded from Amazon S3 via a normal HTTPS call (eg curl). This can be done easily using the AWS SDK for Python, or you could code it yourself without using libraries (it's a bit more complex).
In all examples above, you would need to provide the script/program with a set of IAM Credentials for authenticating with AWS.
Just adding notes for any C# code lovers to solve problem with .Net
Firstly write(C#) code to download private file as string
public string DownloadPrivateFileS3(string fileKey)
{
string accessKey = "YOURVALUE";
string accessSecret = "YOURVALUE";;
string bucket = "YOURVALUE";;
using (s3Client = new AmazonS3Client(accessKey, accessSecret, "YOURVALUE"))
{
var folderPath = "AppData/Websites/Cases";
var fileTransferUtility = new TransferUtility(s3Client);
Stream stream = fileTransferUtility.OpenStream(bucket, folderPath + "/" + fileKey);
using (var memoryStream = new MemoryStream())
{
stream.CopyTo(memoryStream);
var response = memoryStream.ToArray();
return Convert.ToBase64String(response);
}
return "";
}
}
Second Write JQuery Code to download string as Base64
function downloadPrivateFile() {
$.ajax({url: 'DownloadPrivateFileS3?fileName=' + fileName, success: function(result){
var link = this.document.createElement('a');
link.download = fileName;
link.href = "data:application/octet-stream;base64," + result;
this.document.body.appendChild(link);
link.click();
this.document.body.removeChild(link);
}});
}
Call downloadPrivateFile method from anywhere of HTML/C#/JQuery -
Enjoy Happy Coding and Solutions of Complex Problems

AWS S3 HTTPS API request(URL) signed with temporary security credentials to access object

How to generate the HTTPS API request(URL) signed with temporary security credentials to access AWS S3 object.I am able to access object using amazon java sdk but I would like to generate the complete url with temporary security credential like pre signed url.
package com.siriusxm.repo.test;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import com.amazonaws.auth.BasicSessionCredentials;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Region;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient;
import com.amazonaws.services.securitytoken.model.Credentials;
import com.amazonaws.services.securitytoken.model.GetSessionTokenRequest;
import com.amazonaws.services.securitytoken.model.GetSessionTokenResult;
import com.siriusxm.repo.DownloadServiceImpl;
public class TemporaryCredential {
private static String bucketName = "myrepo";
private static String key = "test.pdf";
public static void main(String[] args) {
System.out.println("");
AWSSecurityTokenServiceClient stsClient = new AWSSecurityTokenServiceClient(
new ProfileCredentialsProvider());
// stsClient.setRegion(regionName);sts.us-west-2.amazonaws.com
//
// Start a session.
GetSessionTokenRequest getSessionTokenRequest = new GetSessionTokenRequest();
GetSessionTokenResult sessionTokenResult = stsClient
.getSessionToken(getSessionTokenRequest);
Credentials sessionCredentials = sessionTokenResult.getCredentials();
System.out.println("Session Credentials: "
+ sessionCredentials.toString());
// Package the session credentials as a BasicSessionCredentials
// object for an S3 client object to use.
BasicSessionCredentials basicSessionCredentials = new BasicSessionCredentials(
sessionCredentials.getAccessKeyId(),
sessionCredentials.getSecretAccessKey(),
sessionCredentials.getSessionToken());
AmazonS3Client s3object = new AmazonS3Client(basicSessionCredentials);
// Test. For example, get object keys for a given bucket.
ObjectListing objects = s3object.listObjects(bucketName);
s3object.getObject( new GetObjectRequest(bucketName, key));
System.out.println("No. of Objects = "
+ objects.getObjectSummaries().size());
}
}
This code generate the dynamically access key,secret key and security token.Now i need to generate url with authorization header with signature so that i can access S3 object directly.Is there way?
From this code i want to generate url using x-amz-security-token
if you want to do this in java, you have to use AmazonS3.generatePresignedUrl
AmazonS3 s3client = new AmazonS3Client(new ProfileCredentialsProvider());
java.util.Date expiration = new java.util.Date();
long msec = expiration.getTime();
msec += 1000 * 60 * 60; // 1 hour.
expiration.setTime(msec);
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(bucketName, objectKey);
generatePresignedUrlRequest.setMethod(HttpMethod.GET); // Default.
generatePresignedUrlRequest.setExpiration(expiration);
URL s = s3client.generatePresignedUrl(generatePresignedUrlRequest);
if you want to do this from the console, go to you s3 bucket, click download on the object. This displays a box where you can click on "download". If you right click on this link and copy the address link, you get a pre-signed url for this object