Is it possible to generate an S3 presigned URL in a Lambda function and return that URL to a client, so the client can use it to do an unauthenticated HTTP PUT?
I'm finding that S3 is unexpectedly closing my HTTPS connection when I try to PUT to the URL I get from the lambda function, and I don't know if it's because the server and client are different entities.
Can I do what I want to do? Is there a secret step I'm missing here?
EDIT: per Anon Coward's request, the server-side code is:
presigned_upload_parts = []
for part in range(num_parts):
resp = s3.generate_presigned_url(
ClientMethod = 'upload_part',
Params = {
'Bucket': os.environ['USER_UPLOADS_BUCKET'],
'Key': asset_id,
'UploadId': s3_upload_id,
'PartNumber': part
}
)
presigned_upload_parts.append({"part": part, "url": resp})
return custom_http_response_wrapper(presigned_upload_parts)
The client-side code is:
for idx, part in enumerate(urls):
startByte = idx * bytes_per_part
endByte = min(filesize, ((idx + 1) * bytes_per_part))
f.seek(startByte, 0)
bytesBuf = f.read(endByte - startByte)
print(f"Buffer is type {type(bytesBuf)} with length {len(bytesBuf):,}")
print(f"Part {str(idx)}: bytes {startByte:,} to {endByte:,} as {part['url']}")
#resp = requests.post(part['url'], data = bytesBuf, headers = self.get_standard_headers())
resp = requests.put(
url = part['url'],
data = bytesBuf
)
The error I'm getting is:
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
The presigned URL looks like:
https://my-bucket-name.s3.amazonaws.com/my/item/key?uploadId=yT2W....iuiggs-&partNumber=0&AWSAccessKeyId=ASIAR...MY&Signature=i6duc...Mmpc%3D&x-amz-security-token=IQoJ...%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F...SWHC&Expires=1657135314
There was a bug in my code somewhere. I ran the code under WSL as a test, and in the Linux environment got a more friendly error that helped me find and fix a minor bug, and now it's running as expected in the Windows environment. Whether that's because of the bugfix or some other environmental change I'll never know.
Related
I am trying to create a function that can call REST with the http socket lua.
And I tried to set the timeout this way. But, when I run this function, the timeout is not running. How should I set the timeout?
local http = require "socket.http"
local socket = require "socket"
local respbody = {}
http.request {
method = req_method,
url = req_url,
source = ltn12.source.string(req_body),
headers =
{
["Content-Type"] = req_content_type,
["content-length"] = string.len(req_body),
["Host"] = host,
},
sink = ltn12.sink.table(respbody),
create = function()
local req_sock = socket.tcp()
req_sock:settimeout(3, 't')
return req_sock
end,
}
You may want to check lua-http. I use it to call REST and works like a charm. I am not an expert but, as far as I can tell, it is a good LUA http implementation.
You can set a two seconds timeout as simple as:
local http_client = require "http.client"
local myconnection = http_client.connect {
host = "myrestserver.domain.com";
timeout = 2;
}
Full documentation in here.
if I implement the example with my requirements, will it be like this? cmiiw
local http_client = require "http.client"
local req_body = "key1=value1&key2=value2"
local myconnection = http_client.connect {
method = "POST";
url = "myrestserver.domain.com/api/example";
host = "myrestserver.domain.com";
source = req_body
headers = {
["Content-Type"] = "application/x-www-form-urlencoded",
["content-length"] = string.len(req_body),
},
timeout = 2;
}
LuaSocket implicitly set http.TIMEOUT to the socket object.
Also you have to remember that socket timeout is not the same as request timeout.
Socket timeout means timeout for each operation independently. For simple case you can wait connection up to timeout seconds and then each read operation can take up to timeout seconds. And because of HTTP client read response line by line you get timeout seconds for each header plus for each body chunk. Also, there may be redirecions where each redirection is a separate HTTP request/response. If you use TLS there also will be hendshake after connection which also took several send/receive operation.
I did not use lua-http module and do not know how timeout implemented there.
But I prefer use modules like cURL if I really need to restrict request timeout.
This is the message I get when I am trying to get the token data using web.contents query from "VMware vRealize Automation API":
There was an error when processing the data in the dataset.
Please try again later or contact support. If you contact support, please provide these details.
Data source error
{"error":{"code":"ModelRefresh_ShortMessage_ProcessingError","pbi.error":{"code":"ModelRefresh_ShortMessage_ProcessingError","parameters":
{},"details":[{"code":"Message","detail":{"type":1,"value":"Web.Contents failed to get contents from 'https://xxxxxxxxx.com/identity/api/tokens'
(404): Not Found"}}],"exceptionCulprit":1}}}
Table: GetToken.
The url passed to the first parameter of Web.Contents (authUrl = "https://xxxxxxxxx.com/identity/api/tokens") is accessible but always return the HTTP ERROR 405, probably
because this API uses a a JSON object in the request body parameter with the users credentials to obtain the Response.
API
My query
The main issues:
Your API uses HTTP POST verses GET, so you need to set Options[Content]
You can get refresh errors on the service unless you use Options[RelativePath]
You can "opt-in" to handling errors for specific HTTP Status codes, combined with Value.MetaData you get more detailed error messages.
Let it generate JSON for you from records and lists by using Query or Content parameters see: docs: Web.Contents
This is equivalent to your curl POST request
let
BaseUrl = "https://www.example.com",
Options = [
RelativePath = "/identity/api/tokens",
Headers = [
Accept="application/json"
],
Content = [
username = "username",
password = "password",
tenant = "tenant"
],
ManualStatusHandling = {400, 405}
],
// wrap 'Response' in 'Binary.Buffer' if you are using it multiple times
response = Web.Contents(BaseUrl, Options),
buffered = Binary.Buffer(response),
response_metadata = Value.Metadata(response),
status_code = response_metadata[Response.Status],
from_json = Json.Document(final_result)
in
from_json
I have related Web.Contents examples here, like Chaining Web.Contents requests: ninmonkeys.com/Power-Query-Custom-Functions-Cheat-Sheet
I am trying to obtain an authentication ticket using a POST request with 3 parameters(user,pass,realm) to access Proxmox API Server to be parsed for further queries.
As I am writing the code in Groovy Script for a parameter in a Jenkins job, I am not getting much help in terms of errors. I have tried the POST request on insomnia and it has no problems.
I am still very new to GroovyScript any pointers in the right direction is much appreciated.
def url = new URL("https://$HOST/api2/json/access/ticket")
def connection = url.openConnection()
connection.setDoOutput(true)
connection.setRequestMethod("POST")
connection.setRequestProperty("Content-Type", "application/json")
connection.setRequestProperty('Username', '$USER')
connection.setRequestProperty('Password', '$PASS')
connection.setRequestProperty('Realm', '$REALM')
def requestCode = connection.getResponseCode
Not need connection.setRequestProperty('Realm', '$REALM'), using Username#realm
try this, write to API:
def url = new URL("https://$HOST/api2/json/access/ticket")
def connection = url.openConnection()
connection.setDoOutput(true)
connection.setRequestMethod("POST")
connection.setRequestProperty("Content-Type", "application/json")
connection.setRequestProperty('Password', '$PASS')
connection.setRequestProperty('Username', '$USER' + '#' + '$REALM')
def requestCode = connection.getResponseCode
I created an API function to work with S3. I imported the template swagger. After deployment, I tested with a Node.js project by the npm module aws-api-gateway-client.
It works well with: get bucket lists, get bucket info, get one item, put a bucket, put a plain text object, however I am blocked with put a binary file.
firstly, I ensure ACL is allowed with all permissions on S3. secondly, binary support also added
image/gif
application/octet-stream
The code snippet is as below. The behaviors are:
1) after invokeAPI, the callback function is never hit, after sometime, the Node.js project did not respond. no any error message. The file size (such as an image) is very small.
2) with only two times, the uploading seemed to work, but the result file size is bigger (around 2M bigger) than the original file, so the file is corrupt.
Could you help me out? Thank you!
var filepathname = './items/';
var filename = 'image1.png';
fs.stat(filepathname+filename, function (err, stats) {
var fileSize = stats.size ;
fs.readFile(filepathname+filename,'binary',function(err,data){
var len = data.length;
console.log('file len' + len);
var pathTemplate = '/my-test-bucket/' +filename ;
var method = 'PUT';
var params = {
folder: '',
item:''
};
var additionalParams = {
headers: {
'Content-Type': 'application/octet-stream',
//'Content-Type': 'image/gif',
'Content-Length': len
}
};
var result1 = apigClient.invokeApi(params,pathTemplate,method,additionalParams,data)
.then(function(result){
//never hit :(
console.log(result);
}).catch( function(result){
//never hit :(
console.log(result);
});;
});
});
We encountered the same problem. API Gateway is meant for limited data (10MB as of now), limits shown here,
http://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html
Self Signed URL to S3:
Create an S3 self signed URL for POST from the lambda or the endpoint where you are trying to post.
How do I put object to amazon s3 using presigned url?
Now POST the image directly to S3.
Presigned POST:
Apart from posting the image if you want to post additional properties, you can post it in multi-form format as well.
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#createPresignedPost-property
If you want to process the file after delivering to S3, you can create a trigger from S3 upon creation and process with your Lambda or anypoint that need to process.
Hope it helps.
I am building a google contextual gadget in it i use the following code to load a page:
var params = {};
url = "http://example.com:2057/tasks/create";
params[gadgets.io.RequestParameters.CONTENT_TYPE] = gadgets.io.ContentType.JSON;
params[gadgets.io.RequestParameters.AUTHORIZATION] = gadgets.io.AuthorizationType.SIGNED;
params["OAUTH_SERVICE_NAME"] = "HMAC";
params[gadgets.io.RequestParameters.METHOD] = gadgets.io.MethodType.GET;
gadgets.io.makeRequest(url, function(response)
{
if (response.data && response.data.RedirectUrl)
HandleLogin(response.data.RedirectUrl);
else if(response.text)
{
showOneSection('main');
$('#main').append(response.text);
}
else
ShowDebug(response);
}, params);
The call does not reach my server. and when i try reaching the url in a browser it returns fast.
what can be the problem? how can i trouble shoot it?
Thanks
I finally found the problem.
when making a signed request you have to first obtain a consumer key + secret key.
see http://www.google.com/support/forum/p/apps-apis/thread?tid=31db71169fb6fc77&hl=en
you can do that here: https://www.google.com/gadgets/directory/verify
without the keys google is unable to sign the request (although one would expect a proper error message).