store batch data from aws sqs to aws s3 using aws lambda - amazon-s3

I'm trying to get polled messages (iot data) in aws sqs from aws lambda (nodejs runtime) and trying to store it in aws s3 bucket. But I'm getting error in storing it.
Here is my payload:-
{ "device_name" : "xxx", "temp":23.44, "humidity":33.44, "timestamp" : epoch timestamp }
Here is my code:-
import { S3Client, PutObjectCommand } from "#aws-sdk/client-s3";
export const putInS3 = async (device_name,year,month,day) => {
let bucketName = "my-bucket-name";
const client = new S3Client({Bucket:bucketName,key:`${device_name}/${year}/${month}/${day}/`});
try {
const putObj = new PutObjectCommand(client);
const sendRequest = await client.send(putObj);
console.log(sendRequest);
} catch (err) {
if(err) console.log(err);
}
}
export const handler = async(event) => {
// TODO implement
// console.log(s3Bucket);
const date = new Date();
const year = date.getFullYear();
const month = date.getMonth() + 1;
const day = date.getDate();
event.Records.forEach(record => {
let { body } = record;
body = JSON.parse(body);
let device_name = body.device_name;
putInS3(device_name,year,month,day);
console.log(body + " ");
});
};
The above code gives following error :
Error: No value provided for input HTTP label: Bucket.
Any help would be greatly appreciated. Thanks

You are providing an incorrect value to PutObjectCommand. You are passing the client but you should be passing parameters.
This should work:
const { S3Client, PutObjectCommand } = require('#aws-sdk/client-s3');
const s3Client = new S3Client({'region': 'us-east-1'});
export const putInS3 = async (device_name,year,month,day, body) => {
let bucketName = "my-bucket-name";
const parameters = {
'Bucket': bucketName,
'Key': `${device_name}/${year}/${month}/${day}/`,
'Body': body
}
try {
const putObj = new PutObjectCommand(parameters);
const sendRequest = await s3Client.send(putObj);
console.log(sendRequest);
} catch (err) {
if(err) console.log(err);
}
}

Related

S3 to IPFS from Pinata

I am trying to upload a lot of files from S3 to IPFS via Pinata. I haven't found in Pinata documentation something like that.
This is my solution, using the form-data library. I haven't tested it yet (I will do it soon, I need to code some things).
Is it a correct approach? anyone who has done something similar?
async uploadImagesFolder(
items: ItemDocument[],
bucket?: string,
path?: string,
) {
try {
const form = new FormData();
for (const item of items) {
const file = getObjectStream(item.tokenURI, bucket, path);
form.append('file', file, {
filename: item.tokenURI,
});
}
console.log(`Uploading files to IPFS`);
const pinataOptions: PinataOptions = {
cidVersion: 1,
};
const result = await pinata.pinFileToIPFS(form, {
pinataOptions,
});
console.log(`PiƱata Response:`, JSON.stringify(result, null, 2));
return result.IpfsHash;
} catch (e) {
console.error(e);
}
}
I had the same problem
So, I have found this: https://medium.com/pinata/stream-files-from-aws-s3-to-ipfs-a0e23ffb7ae5
But in the article If am not wrong, is used a different version to the JavaScript AWS SDK v3 (nowadays the most recent: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/index.html).
This is for the Client side with TypeScript:
If you have this version, for me works this code snippet:
export const getStreamObjectInAwsS3 = async (data: YourParamsType) => {
try {
const BUCKET = data.bucketTarget
const KEY = data.key
const client = new S3Client({
region: 'your-region',
credentials: {
accessKeyId: 'your-access-key',
secretAccessKey: 'secret-key'
}
})
const resource = await client.send(new GetObjectCommand({
Bucket: BUCKET,
Key: KEY
}))
const response = resource.Body
if (response) {
return new Response(await response.transformToByteArray()).blob()
}
return null
} catch (error) {
return null
}
}
With the previous code, you can get the Blob Object for pass it to the File object with this method and get the URL resource using the API:
export const uploadFileToIPFS = async(file: Response) => {
const url = `https://api.pinata.cloud/pinning/pinFileToIPFS`
const data = new FormData()
data.append('file', file)
try {
const response = await axios.post(url, data, {
maxBodyLength: Infinity,
headers: {
pinata_api_key: 'your-api',
pinata_secret_api_key: 'your-secret'
},
data: data
})
return {
success: true,
pinataURL: `https://gateway.pinata.cloud/ipfs/${ response.data.IpfsHash }`
}
} catch (error) {
console.log(error)
return null
}
}
I have found this solution from this nice article and you can explore other implementations (including the Node.js side)

getdownloadurl from Firebase storage expo cli

I am trying to display the image from firebase storage. Below is the file location copied from firebase storage. It is a jpeg file
profile/2186uPKjgo4pMOQNm0Cm/profilepic
My following code returned error.
useEffect(() => {
function geturl(){
const filename = "profile/"+userid+"/profilepic.jpeg";
var ref = firebase.storage().ref(filename);
console.log(filename);
// This returns the exact file name
ref.getDownloadURL().then((url)=> {
console.log(url);
});
}
geturl();
}, []);
I got this error [object Object]. After that, I tried the following code async await
useEffect(() => {
async function geturl(){
const filename = "profile/"+userid+"/profilepic.jpeg";
var ref = firebase.storage().ref(filename);
console.log("inside geturl");
const downloadurl = await ref.getDownloadURL();
console.log(downloadurl);
}
geturl();
}, []);
Now Im getting the following error.
Possible Unhandled Promise Rejection (id: 29):
"code_": "storage/object-not-found",
"message_": "Firebase Storage: Object 'profile/2186uPKjgo4pMOQNm0Cm/profilepic.jpeg' does not exist.",
"name_": "FirebaseError",
"serverResponse_": "{
\"error\": {
\"code\": 404,
\"message\": \"Not Found. Could not get object\",
\"status\": \"GET_OBJECT\"
}
}",
}
Please let me know how I can get the url?
here you go you can use this function it uploads image to firebase storage and get the image uri at the same time
const uploadImage = async () => {
const response = await fetch(image);
const blob = await response.blob();
let filename = image.substring(image.lastIndexOf('/')+1);
const ext = filename.split('.').pop();
const name = filename.split('.').slice(0, -1).join('.');
filename = name + Date.now() + '.' + ext;
try {
var ref = firebase.storage().ref().child('post-images/'+filename);
await ref.put(blob)
.then(snapshot => {
return snapshot.ref.getDownloadURL();
})
.then(downloadURL => {
console.log(`Successfully uploaded file and got download link');
return downloadURL;
});
return null;
} catch (error) {
return null;
}
}

MERN and Amazon-s3 for file upload

How to post a file to Amazon S3 using node and react and save it path to mongoDB. with mongoose and formidable.
private async storeFile(file: { buffer: Buffer, fileId: string }): Promise<string> {
try {
const awsConfig = new AWS.Config(storageConfig);
const s3 = new AWS.S3(awsConfig);
let storageLink = undefined;
fs.readFile(file.buffer, (err, data) => {
if (err) {
throw err;
}
const params = {
Bucket:storageConfig.s3Bucket,
Key: `${storageConfig.s3Prefix}${file.fileId}`,
Body: data,
};
s3.upload(params, (s3Err: Error, s3Data: AWS.S3.ManagedUpload.SendData) => {
if (s3Err) {
throw s3Err;
}
storageLink = s3Data.Location;
});
});
return storageLink;
} catch (error) {
throw error;
}
}
In your Service file where you wanna call this function, update with record in collection
const storageLink = this.storeFile({ buffer, fileId });
const file = await file.updateOne({ _id: fileId }, {
status: fileStatus.UPLOADED, // just a flag
fileId: storageLink,
});

Cannot upload image to s3 using serverless framework but it work in offline (buffer issue)

I'm trying to deploy a lambda function allowing me to upload a picture to S3.
The lambda works well in offline but when I'm deploy it to AWS, the function doesn't work.
The first error I encountered was this one :
ERROR (node:7) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
So, I followed the recommendation to use Buffer.from() method instead. But it doesn't work too. The lambda run until the timeout.
Can someone tell me where I was wrong or suggest me another solution ?
Below my lambda function :
const AWS = require("aws-sdk");
const Busboy = require("busboy");
const uuidv4 = require("uuid/v4");
require("dotenv").config();
AWS.config.update({
accessKeyId: process.env.ACCESS_KEY_ID,
secretAccessKey: process.env.SECRET_ACCESS_KEY,
subregion: process.env.SUB_REGION
});
const s3 = new AWS.S3();
const getContentType = event => {
// see the second block of codes
};
const parser = event => {
// see the third block of codes
};
module.exports.main = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
const uuid = uuidv4();
const uploadFile = async (image, uuid) =>
new Promise(() => {
// const bitmap = new Buffer(image, "base64"); // <====== deprecated
const bitmap = Buffer.from(image, "base64"); // <======== problem here
const params = {
Bucket: "my_bucket",
Key: `${uuid}.jpeg`,
ACL: "public-read",
Body: bitmap,
ContentType: "image/jpeg"
};
s3.putObject(params, function(err, data) {
if (err) {
return callback(null, "ERROR");
}
return callback(null, "SUCCESS");
});
});
parser(event).then(() => {
uploadFile(event.body.file, uuid);
});
};
getContentType() :
const getContentType = event => {
const contentType = event.headers["content-type"];
if (!contentType) {
return event.headers["Content-Type"];
}
return contentType;
};
parser()
const parser = event =>
new Promise((resolve, reject) => {
const busboy = new Busboy({
headers: {
"content-type": getContentType(event)
}
});
const result = {};
busboy.on("file", (fieldname, file, filename, encoding, mimetype) => {
file.on("data", data => {
result.file = data;
});
file.on("end", () => {
result.filename = filename;
result.contentType = mimetype;
});
});
busboy.on("field", (fieldname, value) => {
result[fieldname] = value;
});
busboy.on("error", error => reject(error));
busboy.on("finish", () => {
event.body = result;
resolve(event);
});
busboy.write(event.body, event.isBase64Encoded ? "base64" : "binary");
busboy.end();
});
new Buffer(number) // Old
Buffer.alloc(number) // New
new Buffer(string) // Old
Buffer.from(string) // New
new Buffer(string, encoding) // Old
Buffer.from(string, encoding) // New
new Buffer(...arguments) // Old
Buffer.from(...arguments) // New
You are using callbackWaitsForEmptyEventLoop which basically let lambda function thinks that the work is not over yet. Also, you are wrapping it in promise but not resolving it. You can simplify this logic using following inbuilt promise function on aws-sdk
module.exports.main = async event => {
const uuid = uuidv4();
await parser(event); // not sure if this needs to be async or not. check
const bitmap = Buffer.from(event.body.file, "base64"); // <======== problem here
const params = {
Bucket: "my_bucket",
Key: `${uuid}.jpeg`,
ACL: "public-read",
Body: bitmap,
ContentType: "image/jpeg"
};
const response = await s3.putObject(params).promise();
return response;
};

How to upload image to firebase using react native

I need way to upload image to firebase
i tried to use react-native-fetch-blob library
but I think there is something wrong with installing the library
No need to use react-native-fetch-blob. Here is how I do it on my project.
Install both react-native-firebase and react-native-image-picker. Follow the installation steps from their documentation guide.
Then implement 2 small functions to do image pick and upload to firebase. Here is the sample code.
// 1. Import required library
import firebase from 'react-native-firebase';
import ImagePicker from 'react-native-image-picker';
// 2. Create a function to pick the image
const pickImage = () => {
return new Promise((resolve, reject) => {
ImagePicker.showImagePicker(pickerOptions, response => {
if (response.didCancel) return;
if (response.error) {
const message = `An error was occurred: ${response.error}`;
reject(new Error(message));
return;
}
const { path: uri } = response;
resolve(uri);
});
});
};
// 3. Create a function to upload to firebase
const uploadImage = async (fileName, uri) {
return new Promise(
(resolve, reject) => {
firebase
.storage()
.ref(`uploads/${filename}`)
.putFile(uri)
.then(resolve)
.catch(reject);
}
);
}
Then simply firing both function as you need, here is the sample to pick and immediately upload it.
const pickImageAndUpload = async () => {
const uri = await pickImage();
const fileName = 'someImage.jpg';
const { state, downloadURL } = await uploadImage(fileName, uri);
}
async function uploadImageAsync(itemImage, passedParameter, ItemName, ItemDesc, ItemPrice, ItemWeight) {
const response = await fetch(itemImage);
const blob = await response.blob();
console.log("uri of the elements ius", blob)
var storageRef = firebase.storage().ref();
var file = blob
var metadata = {
contentType: 'image/jpeg',
};
const timeStamp = Date.now();
var uploadTask = storageRef.child('CategoryDescription' + "/" + `${passedParameter}` + "/" + `${ItemName}`).put(file, metadata);
//For image pick
pickImage = async () => {
const { CAMERA, CAMERA_ROLL } = Permissions;
const permissions = {
[CAMERA]: await Permissions.askAsync(CAMERA),
[CAMERA_ROLL]: await Permissions.askAsync(CAMERA_ROLL),
};
if (permissions[CAMERA].status === 'granted' && permissions[CAMERA_ROLL].status === 'granted') {
let result = await ImagePicker.launchImageLibraryAsync({
allowsEditing: false,
aspect:[4,3],
quality: 0.5,
});
// console.log(result);
if (!result.cancelled) {
this.setState({ itemImage: result.uri });
}
}