How can i encode a string in base64 using meteor - amazon-s3

I am trying to use a form to upload files to a s3 bucket using Meteor. I am following this amazon article. At "Sign Your S3 POST Form", near the end, I need to encode a string to base64 but I've been unable to find a way to do this. Can anyone tell me how to do this? Notice that the string first needs to be encoded and then signed. This is how it's done in python:
import base64
import hmac, hashlib
policy = base64.b64encode(policy_document)
signature = base64.b64encode(hmac.new(AWS_SECRET_ACCESS_KEY, policy, hashlib.sha1).digest())

You can do this without the NodeJS crypto module, creating a package looked a bit like breaking a fly on the wheel to me so I figured out this:
if (Meteor.isServer) {
Meteor.methods({
'base64Encode':function(unencoded) {
return new Buffer(unencoded || '').toString('base64');
},
'base64Decode':function(encoded) {
return new Buffer(encoded || '', 'base64').toString('utf8');
},
'base64UrlEncode':function(unencoded) {
var encoded = Meteor.call('base64Encode',unencoded);
return encoded.replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
},
'base64UrlDecode':function(encoded) {
encoded = encoded.replace(/-/g, '+').replace(/_/g, '/');
while (encoded.length % 4)
encoded += '=';
return Meteor.call('base64Decode',encoded);
}
console.log(Meteor.call('base64Encode','abc'));
});
This is based on the base64.js by John Hurliman found at https://gist.github.com/jhurliman/1250118 Note that this will work like a charm on the server but for porting it to the client you have call the methods with a callback function that stores the result as a session variable.

You need NodeJS crypto module to perform these tasks.
First create a "packages" directory at the root of your meteor project, then create a "my-package" directory.
Inside it, you need two files : a "package.js" and "my-package.js".
package.js should look like :
Package.describe({
summary:"MyPackage doing amazing stuff with AWS."
});
Package.on_use(function(api){
// add your package file to the server app
api.add_files("my-package.js","server");
// what we export outside of the package
// (this is important : packages have their own scope !)
api.export("MyPackage","server");
});
my-package.js should look like :
var crypto=Npm.require("crypto");
MyPackage={
myFunction:function(arguments){
// here you can use crypto functions !
}
};
The function you will probably need is crypto.createHmac.
Here is an example code of how I encode a JSON security policy in base64 then use it to generate a security signature in my own app :
encodePolicy:function(jsonPolicy){
// stringify the policy, store it in a NodeJS Buffer object
var buffer=new Buffer(JSON.stringify(jsonPolicy));
// convert it to base64
var policy=buffer.toString("base64");
// replace "/" and "+" so that it is URL-safe.
return policy.replace(/\//g,"_").replace(/\+/g,"-");
},
encodeSignature:function(policy){
var hmac=crypto.createHmac("sha256",APP_SECRET);
hmac.update(policy);
return hmac.digest("hex");
}
This will allow you to call MyPackage.myFunction in the server-side of your Meteor app.
Last but not last, don't forget to "meteor add my-package" in order to use it !

You can use meteor-crypto-base64 package.
CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse('Hello, World!'));
//"SGVsbG8sIFdvcmxkIQ=="

Related

How to authenticate Shopware 6 <base-app-url> correctly

With the Admin SDK it's possible to further enrich the administration in Shopware 6. As in the installation guide for apps stated, an entry point (base-app-url) needs to be provided in the manifest file of an app.
Since every request needs to be authenticated properly, this GET request also needs authentication. However, I am not able to authenticate this one in the same way as I am successfully doing it with the GET request from modules.
The base-app-url request looks the following (in my case with some [custom] entity privileges):
http://localhost:3000/sdk?location-id=sw-main-hidden&privileges=%7B%22read%22%3A%5B%22language%22%2C%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22create%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22update%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22delete%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%7D&shop-id=sbzqJiPRrbHAlC2K&shop-url=http://localhost:8888&timestamp=1674045964&sw-version=6.4.18.0&sw-context-language=2fbb5fe2e29a4d70aa5854ce7ce3e20b&sw-user-language=de-DE&shopware-shop-signature=e7b20a46487046a515638f76c6fadab6b1c749ea4a8ac6e7653527e73ba18380
The shop has the following data
Shop {
_id: 'sbzqJiPRrbHAlC2K',
_url: 'http://localhost:8888',
_secret: '3c5a2f031006791f2aca40ffa22e8febbc8a53d8',
_apiKey: 'SWIAB2PVODCWSLZNDMC5ZM1XWA',
_secretKey: 'VnNwM0ZOMnN1Y05YdUlKazlPdlduWTdzOHhIdFpacjVCYkgzNEg'
}
I am currently authenticating my modules like the following (Node.js):
const SHOPWARE_SHOP_SIGNATURE = 'shopware-shop-signature';
export function authenticateGetRequest(req: Request, shop: Shop): void {
// e7b20a46487046a515638f76c6fadab6b1c749ea4a8ac6e7653527e73ba18380
const signature = getSignatureFromQuery(req);
verifySignature(shop.secret, removeParamsFromQuery(req), signature);
}
function getSignatureFromQuery(req: Request): string {
if (!req.query[SHOPWARE_SHOP_SIGNATURE]) {
throw new Error('Signature is not present in request!');
}
return req.query[SHOPWARE_SHOP_SIGNATURE] as string;
}
function removeParamsFromQuery(req: Request): string {
// Some code
// Returns following string - Does neither work for base-app-url nor for module GET requests:
// 'shop-id=sbzqJiPRrbHAlC2K&shop-url=http://localhost:8888&timestamp=1674045964'
// If the string follows this pattern, it works only for modules:
// shop-id={id}&shop-url={url}&timestamp={ts}&sw-version={v}&sw-context-language={cl}&sw-user-language={ul}
}
function verifySignature(secret: string, message: string, signature: string): void {
const hmac = crypto.createHmac('sha256', secret).update(message).digest('hex');
if (hmac !== signature) {
throw new Error('Signature could not be verified!');
}
}
However the base-app-url cannot be verified correctly and the "Signature could not be verified!" error is thrown.
What am I doing wrong here?
More info:
Additionally I added a GET request for a module where everything is working:
http://localhost:3000/faq?shop-id=sbzqJiPRrbHAlC2K&shop-url=http://localhost:8888&timestamp=1674045963&sw-version=6.4.18.0&sw-context-language=2fbb5fe2e29a4d70aa5854ce7ce3e20b&sw-user-language=de-DE&shopware-shop-signature=0f0889c9e8086c6c3553dc946a01f2ef27b34cd1c55b0c03901b6d8a6a9b6f53
The resulting string can be verified:
shop-id=sbzqJiPRrbHAlC2K&shop-url=http://localhost:8888&timestamp=1674045963&sw-version=6.4.18.0&sw-context-language=2fbb5fe2e29a4d70aa5854ce7ce3e20b&sw-user-language=de-DE
Try out following code in some php sandbox environment:
<?php
$message = 'shop-id=sbzqJiPRrbHAlC2K&shop-url=http://localhost:8888&timestamp=1674045963&sw-version=6.4.18.0&sw-context-language=2fbb5fe2e29a4d70aa5854ce7ce3e20b&sw-user-language=de-DE';
$secret = '3c5a2f031006791f2aca40ffa22e8febbc8a53d8';
$signature = '0f0889c9e8086c6c3553dc946a01f2ef27b34cd1c55b0c03901b6d8a6a9b6f53';
$hmac = hash_hmac('sha256', $message, $secret);
if (!hash_equals($hmac, $signature)) {
echo 'Signature not valid';
} else {
echo 'Signature valid';
}
SOLUTION:
Express decodes the query strings automatically with req.query depending on your express configuration. Keep in mind to validate the hmac with encoded query params as they are passed from shopware.
In my case the only difference where the decoded privileges and they looked like this:
&privileges={"read":["language","ce_atl_faq_group_faqs","ce_atl_faq_group","ce_atl_faq"],"create":["ce_atl_faq_group_faqs","ce_atl_faq_group","ce_atl_faq"],"update":["ce_atl_faq_group_faqs","ce_atl_faq_group","ce_atl_faq"],"delete":["ce_atl_faq_group_faqs","ce_atl_faq_group","ce_atl_faq"]}
But they need to look like this:
&privileges=%7B%22read%22%3A%5B%22language%22%2C%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22create%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22update%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22delete%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%7D
Looking at the QuerySigner, this is how the signature is generated on the side of Shopware with the actual arguments:
hash_hmac(
'sha256',
'location-id=sw-main-hidden&privileges=%7B%22read%22%3A%5B%22language%22%2C%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22create%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22update%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22delete%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%7D&shop-id=sbzqJiPRrbHAlC2K&shop-url=http://localhost:8888&timestamp=1674045964&sw-version=6.4.18.0&sw-context-language=2fbb5fe2e29a4d70aa5854ce7ce3e20b&sw-user-language=de-DE',
'VnNwM0ZOMnN1Y05YdUlKazlPdlduWTdzOHhIdFpacjVCYkgzNEg'
);
// 8034a13561b75623420b06fb7be01f20d97556441268939e9a5222ffec12215a
Given on your side you remove the shopware-shop-signature query param AND that the secrets are equal on both sides, you should be able to regenerate the matching signature.
const crypto = require('crypto');
const message = 'location-id=sw-main-hidden&privileges=%7B%22read%22%3A%5B%22language%22%2C%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22create%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22update%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%2C%22delete%22%3A%5B%22ce_atl_faq_group_faqs%22%2C%22ce_atl_faq_group%22%2C%22ce_atl_faq%22%5D%7D&shop-id=sbzqJiPRrbHAlC2K&shop-url=http://localhost:8888&timestamp=1674045964&sw-version=6.4.18.0&sw-context-language=2fbb5fe2e29a4d70aa5854ce7ce3e20b&sw-user-language=de-DE';
const hmac = crypto.createHmac('sha256', 'VnNwM0ZOMnN1Y05YdUlKazlPdlduWTdzOHhIdFpacjVCYkgzNEg').update(message).digest('hex');
// 8034a13561b75623420b06fb7be01f20d97556441268939e9a5222ffec12215a
So in theory your code looks fine. Verify that the query string matches exactly. Things to check:
Maybe your node server decodes the url entities unwantedly?
Does your node serve escape special characters in the query string?
Do the secrets match on both sides?
To consider additionally:
Consider to just point the base-app-url to a static page outside of the scope of your app server instead. As that page will be loaded inside an iframe, you can use client side javascript to read the query parameters and, only if necessary, make requests to your app server using the credentials from inside the iframe. Keep in mind you really only need the authentication if you need to handle personalized data, otherwise you might as well serve static assets without the need for authentication.

Windows Azure Storage Blobs to zip file with Express

I am trying to use this pluggin (express-zip). At the Azure Storage size we have getBlobToStream which give us the file into a specific Stream. What i do now is getting image from blob and saving it inside the server, and then res.zip it. Is somehow possible to create writeStream which will write inside readStream?
Edit: The question has been edited to ask about doing this in express from Node.js. I'm leaving the original answer below in case anyone was interested in a C# solution.
For Node, You could use a strategy similar to what express-zip uses, but instead of passing a file read stream in this line, pass in a blob read stream obtained using createReadStream.
Solution using C#:
If you don't mind caching everything locally while you build the zip, the way you are doing it is fine. You can use a tool such as AzCopy to rapidly download an entire container from storage.
To avoid caching locally, you could use the ZipArchive class, such as the following C# code:
internal static void ArchiveBlobs(CloudBlockBlob destinationBlob, IEnumerable<CloudBlob> sourceBlobs)
{
using (Stream blobWriteStream = destinationBlob.OpenWrite())
{
using (ZipArchive archive = new ZipArchive(blobWriteStream, ZipArchiveMode.Create))
{
foreach (CloudBlob sourceBlob in sourceBlobs)
{
ZipArchiveEntry archiveEntry = archive.CreateEntry(sourceBlob.Name);
using (Stream archiveWriteStream = archiveEntry.Open())
{
sourceBlob.DownloadToStream(archiveWriteStream);
}
}
}
}
}
This creates a zip archive in Azure storage that contains multiple blobs without writing anything to disk locally.
I'm the author of express-zip. What you are trying to do should be possible. If you look under the covers, you'll see I am in fact adding streams into the zip:
https://github.com/thrackle/express-zip/blob/master/lib/express-zip.js#L55
So something like this should work for you (prior to me adding support for this in the interface of the package itself):
var zip = zipstream(exports.options);
zip.pipe(express.response || http.ServerResponse.prototype); // res is a writable stream
var addFile = function(file, cb) {
zip.entry(getBlobToStream(), { name: file.name }, cb);
};
async.forEachSeries(files, addFile, function(err) {
if (err) return cb(err);
zip.finalize(function(bytesZipped) {
cb(null, bytesZipped);
});
});
Apologize if I've made horrible errors above; I haven't been on this for a bit.

Display result from server in IBM Worklight

I have implemented HTTP adapter in IBM Worklight. I want to display the result returned from server. I want to display HTML file. My code is
function getFeeds() {
var input = {
method : 'get',
returnedContentType : 'text',
path : "marketing/partners.html"
};
WL.Logger.debug("sdfsds");
return WL.Server.invokeHttp(input);
}
I want to receive(display) WL.Server.invokeHttp(input). After receiving it I want to parse the data.
Take a look at the Server-side Development Getting Started Modules. Inside the HTTP adapter – Communicating with HTTP back-end systems Module on Slide 15 - 'XSL Transformation Filtering' will show you how to filter data you get back from the backend. Further parsing and showing data has to be done on the client using onSuccess callback for WL.Client.invokeProcedure. There's a module for that too.
Here's an example of getting data and showing to a user:
var invocationData = {
adapter : 'adapter-name',
procedure : 'procedure-name',
parameters : []
};
var options = {};
options.onSuccess = function (response) {
//response is a JavaScript object
$("#id").html(response.invocationResponse.text);
}
options.onFailure = function (response) {
alert('Failed!'); //You probably want something more meaningful here.
}
WL.Client invokeProcedure(invocationData, options);
There are JavaScript libraries you can add to make searching for values inside the JSON response easier, such as: jspath and jquery-jspath. There's also XPath if you're working with XML.
If you retrieve it as plain text, once you got it back to your application, do something like
$("#container-id").html(response.invocationResponse.text);
This will inject the HTML you've retrieved to an element with id container-id.

How to send base64 encoded file to PlayFramework server?

I'd like to implement a FileUpload using the new FileReader API. From the client side, everything works well and I can send a PUT request to the server with the correct fields containing the file in Base64 encoded.
But in the server side, it's not going great, here are my results :
Logger.info(String.valueOf(request().body().asRaw())); // null
Logger.info(String.valueOf(request().body().asText())); // null
And most importantly :
Logger.info(String.valueOf(request().body().isMaxSizeExceeded())); // true !
What am I missing? How can I make it work?
I found the answer to my question !
For those who are looking for it, here's the answer :
You need to add a BodyParser as annotation for your method, and specify a higher maxLength value.
#BodyParser.Of(value = BodyParser.Json.class, maxLength = 1024 * 1024)
public static Result method() {
Logger.info(String.valueOf(request().body().asJson())); // Will not be empty!
}

when to check for file size/mimetype in node.js upload script?

I created an upload script in node.js using express/formidable. It basically works, but I am wondering where and when to check the uploaded file e. g. for the maximum file size or if the file´s mimetype is actually allowed.
My program looks like this:
app.post('/', function(req, res, next) {
req.form.on('progress', function(bytesReceived, bytesExpected) {
// ... do stuff
});
req.form.complete(function(err, fields, files) {
console.log('\nuploaded %s to %s', files.image.filename, files.image.path);
// ... do stuff
});
});
It seems to me that the only viable place for checking the mimetype/file size is the complete event where I can reliably use the filesystem functions to get the size of the uploaded file in /tmp/ – but that seems like a not so good idea because:
the possibly malicious/too large file is already uploaded on my server
the user experience is poor – you watch the upload progress just to be told that it didnt work afterwards
Whats the best practice for implementing this? I found quite a few examples for file uploads in node.js but none seemed to do the security checks I would need.
With help from some guys at the node IRC and the node mailing list, here is what I do:
I am using formidable to handle the file upload. Using the progress event I can check the maximum filesize like this:
form.on('progress', function(bytesReceived, bytesExpected) {
if (bytesReceived > MAX_UPLOAD_SIZE) {
console.log('### ERROR: FILE TOO LARGE');
}
});
Reliably checking the mimetype is much more difficult. The basic Idea is to use the progress event, then if enough of the file is uploaded use a file --mime-type call and check the output of that external command. Simplified it looks like this:
// contains the path of the uploaded file,
// is grabbed in the fileBegin event below
var tmpPath;
form.on('progress', function validateMimetype(bytesReceived, bytesExpected) {
var percent = (bytesReceived / bytesExpected * 100) | 0;
// pretty basic check if enough bytes of the file are written to disk,
// might be too naive if the file is small!
if (tmpPath && percent > 25) {
var child = exec('file --mime-type ' + tmpPath, function (err, stdout, stderr) {
var mimetype = stdout.substring(stdout.lastIndexOf(':') + 2, stdout.lastIndexOf('\n'));
console.log('### file CALL OUTPUT', err, stdout, stderr);
if (err || stderr) {
console.log('### ERROR: MIMETYPE COULD NOT BE DETECTED');
} else if (!ALLOWED_MIME_TYPES[mimetype]) {
console.log('### ERROR: INVALID MIMETYPE', mimetype);
} else {
console.log('### MIMETYPE VALIDATION COMPLETE');
}
});
form.removeListener('progress', validateMimetype);
}
});
form.on('fileBegin', function grabTmpPath(_, fileInfo) {
if (fileInfo.path) {
tmpPath = fileInfo.path;
form.removeListener('fileBegin', grabTmpPath);
}
});
The new version of Connect (2.x.) has this already baked into the bodyParser using the limit middleware: https://github.com/senchalabs/connect/blob/master/lib/middleware/multipart.js#L44-61
I think it's much better this way as you just kill the request when it exceeds the maximum limit instead of just stopping the formidable parser (and letting the request "go on").
More about the limit middleware: http://www.senchalabs.org/connect/limit.html