When I had my hard-coded access key and secret key present, I was able to generate authenticated urls for users to view private files on S3. This was done with the following code:
import hmac
import sha
import urllib
import time
filename = "".join([s3_url_prefix, filename])
expiry = str(int(time.time()) + 3600)
h = hmac.new(
current_app.config.get("S3_SECRET_KEY", None),
"".join(["GET\n\n\n", expiry, "\n", filename]),
sha
)
signature = urllib.quote_plus(base64.encodestring(h.digest()).strip())
return "".join([
"https://s3.amazonaws.com",
filename,
"?AWSAccessKeyId=",
current_app.config.get("S3_ACCESS_KEY", None),
"&Expires=",
expiry,
"&Signature=",
signature
])
Which gave me something to the effect of https://s3.amazonaws.com/bucket_name/path_to_file?AWSAccessKeyId=xxxxx&Expires=5555555555&Signature=eBFeN32eBb2MwxKk4nhGR1UPhk%3D. Unfortunately, I am unable to store keys in config files for security reasons. For this reason, I switched over to IAM roles. Now, I get my keys using:
_iam = boto.connect_iam()
S3_ACCESS_KEY = _iam.access_key
S3_SECRET_KEY = _iam.secret_key
However, this gives me the error "The AWS Access Key Id you provided does not exist in our records.". From my research, I understand that this is because my IAM keys aren't the actual keys, but instead used with a token. My question therefore, is twofold:
How do I get the token programatically? It doesn't seem that there is a simple iam property I can use.
How do I send the token in the signature? I believe my signature should end up looking something like "".join(["GET\n\n\n", expiry, "\n", token, filename]), but I can't figure out what format to use.
Any help would be greatly appreciated.
There was a change to the generate_url method in https://github.com/boto/boto/commit/99e14f3df039997f54a5377cb6aecc83f22c2670 (June 2012) that made it possible to sign a URL using session credentials. That means you would need to be using a boto version 2.6.0 or later. If you are, you should be able to just do this:
import boto
s3 = boto.connect_s3()
url = s3.generate_url(expires_in=3600, method='GET', bucket='<bucket_name>', key='<key_name>')
What version are you using?
Related
I wrote a python script that uploads files from a local folder into Google cloud storage.
I also created a service account with sufficient permission and tested it on my computer using that service account JSON key and it worked.
Now I send the code and JSON key to someone else to run but the authentication fails on her side.
Are we missing any authentication through GCP UI?
def config_gcloud():
subprocess.run(
[
shutil.which("gcloud"),
"auth",
"activate-service-account",
"--key-file",
CREDENTIALS_LOCATION,
]
)
storage_client = storage.Client.from_service_account_json(CREDENTIALS_LOCATION)
return storage_client
def file_upload(bucket, source, destination):
storage_client = config_gcloud()
...
The error happens in the config_cloud and it says it is expecting str, path, ... but gets NonType.
As I said, the code is fine and works on my computer. How anotehr person can use it using JSON key which I sent her?She stored Json locally and path to Json is in the code.
CREDENTIALS_LOCATION is None instead of the correct path, hence it complaining about it being NoneType instead of str|Path.
Also you don't need that gcloud call, that would only matter for gcloud/gsutil commands, not python client stuff.
And please post the actual stacktrace of the error next time, not just a misspelled interpretation of it.
Here is my code that I use to create a s3 client and generate a presigned url, which are some quite standard codes. They have been up running in the server for quite a while. I pulled the code out and ran it locally in a jupyter notebook
def get_s3_client():
return get_s3(create_session=False)
def get_s3(create_session=False):
session = boto3.session.Session() if create_session else boto3
S3_ENDPOINT = os.environ.get('AWS_S3_ENDPOINT')
if S3_ENDPOINT:
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
AWS_DEFAULT_REGION = os.environ["AWS_DEFAULT_REGION"]
s3 = session.client('s3',
endpoint_url=S3_ENDPOINT,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
region_name=AWS_DEFAULT_REGION)
else:
s3 = session.client('s3', region_name='us-east-2')
return s3
s3 = get_s3_client()
BUCKET=[my-bucket-name]
OBJECT_KEY=[my-object-name]
signed_url = s3.generate_presigned_url(
'get_object',
ExpiresIn=3600,
Params={
"Bucket": BUCKET,
"Key": OBJECT_KEY,
}
)
print(signed_url)
When I tried to download the file using the url in the browser, I got an error message and it says "The specified key does not exist." I noticed in the error message that my object key becomes "[my-bucket-name]/[my-object-name]" rather than just "[my-object-name]".
Then I used the same bucket/key combination to generate a presigned url using aws cli, which is working as expected. I found out that somehow the s3 client method (boto3) inserted [my-object-name] in front of [my-object-name] compared to the aws cli method. Here are the results
From s3.generate_presigned_url()
https://[my-bucket-name].s3.us-east-2.amazonaws.com/[my-bucket-name]/[my-object-name]?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAV17K253JHUDLKKHB%2F20210520%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210520T175014Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=5cdcc38e5933e92b5xed07b58e421e5418c16942cb9ac6ac6429ac65c9f87d64
From aws cli s3 presign
https://[my-bucket-name].s3.us-east-2.amazonaws.com/[my-object-name]?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYA7K15LJHUDAVKHB%2F20210520%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210520T155926Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=58208f91985bf3ce72ccf884ba804af30151d158d6ba410dd8fe9d2457369894
I've been working on this and searching for solutions for day and half and I couldn't find out what was wrong with my implementation. I guess it might be that I ignored some basic but important settings to create a s3 client using boto3 or something else. Thanks for the help!
Ok, myth is solved, I shouldn't provide the endpoint_url=S3_ENDPOINT param when I create the s3 client, boto3 will figure it out. After i removed it, everything works as expected.
I have an app where the user details are passed as a JWT containing information about the current user and it's roles.
Everytime the user is logged in (via a KeyCloak instance), the information from the JWT is parsed on my end in a function that updates the user object via SQLAlchemy. However, since there is no user object being passed back and forth in the backend, I have to parse the JWT for roles for every action that requires it. I also have a need for auditing, and due to the structure of the app, this module does not necessarily have access to the request objects at the time of logging.
I'm looking for a neat way to make something like flask_users current_user() functionality work by mapping JWT -> ORM user object, to be able to transparently get the current user. Is there any way to go about this? The user registration and so on is completely separate from the app, and Flask only knows which user it is based on tokens in the requests that are being sent.
TLDR; Is there a way to load a user from the DB based on an already issued JWT (which contains information to map to a user), and is there perhaps already a lib or extension to flask that supports this?
I use a decorator to parse the JWT token using pyjwt.
Then from the parsed token you can get the user and do the proper authorization.
If you don't want to add the decorator to all your functions that require authorization you can use Flasks before_request.
from functools import wraps
from flask import Response, current_app, request
from jwt import decode
from jwt.exceptions import (DecodeError, ExpiredSignatureError,
InvalidSignatureError)
def authorize(func):
#wraps(func)
def check_authorization(*args, **kwargs):
try:
jwt_token = request.cookies.get('auth_token') # get token from request
if jwt_token is None:
return Response(status=401, headers={'WWW-Authenticate': 'Bearer'})
token = decode(
jwt_token,
key='pub_key', # public key to validate key
algorithms=['RS256'], # list of algs the key could be signed
verify=True
)
# you can call another function to do check user roles here
# e.g authorize(token['sub'])
return func(*args, **kwargs)
except (InvalidSignatureError, DecodeError, ExpiredSignatureError):
return Response(
response='{ "error": "token_invalid"}',
status=401,
headers={'WWW-Authenticate': 'Bearer'})
return check_authorization
This is supported with flask-jwt-extended: https://flask-jwt-extended.readthedocs.io/en/stable/complex_objects_from_token/
I'm trying to figure out a concise way to get data from s3 via boto
my current code looks like this. s3 manager is simply a class that does all the s3 setup for my app.
log.debug("generating downloader")
downloader = s3_manager()
log.debug("accessing bucket")
bucket_archive = downloader.s3_buckets['#archive']
log.debug("getting key")
key = bucket_archive.get_key(archive_filename)
log.debug("getting key into string")
source = key.get_contents_as_string()
the problem is that , looking at my debug logs, i'm making two requests to amazon s3:
key = bucket_archive.get_key(archive_filename)
source = key.get_contents_as_string()
looking at the docs [ http://boto.readthedocs.org/en/latest/ref/s3.html ] , it seems that the call to get_key checks to see if it exists , while the second call gets the actual data. does anyone know of a method to do both at once ? a more concise way of doing this with one request is preferable for our app.
The get_key() method performs a HEAD request on the object to verify that it exists. If you are certain that the bucket and key exist and would prefer not to have the overhead of a HEAD request, you can simply create a Key object directly. Something like this would work:
import boto
s3 = boto.connect_s3()
bucket = s3.get_bucket('mybucket', validate=False)
key = bucket.new_key('myexistingkey')
contents = key.get_contents_as_string()
The validate=False on the call to get_bucket eliminates a GET request that also is intended to validate that the bucket exists.
I'm new to Oauth and I stack on getting oauth_access_token to work with Xero. Web Service authentication doesn't work for me.
Xero returns the following error message "oauth_problem=signature_invalid&oauth_problem_advice=Failed to validate signature".
The generated signature is incorrect, but what is right way to generate it?
Here is APEX code which generates Endpoint. What is wrong?
Http h = new Http();
String consumer_key='XXX';
Long tmp=(System.now().getTime()/1000);
Blob isItCorrect = Crypto.generateMac('HMacSHA1', Blob.valueOf('https://api.xero.com/api.xro/2.0'), Blob.valueOf(consumer_key));
String signature= EncodingUtil.urlEncode(EncodingUtil.base64Encode(isItCorrect), 'UTF-8');
// Try to get access token
HttpRequest req = new HttpRequest();
req.setEndpoint('https://api.xero.com/oauth/RequestToken?oauth_consumer_key='+consumer_key+
'&oauth_signature_method=RSA-SHA1'+
'&oauth_signature='+signature+
'&oauth_timestamp='+tmp+ '&oauth_nonce='+tmp+'&oauth_version=1.0&scope=https%3A%2F%2Fapi.xero.com%2Fapi.xro%2F2.0');
req.setMethod('GET');
// Send the request, and return a response
HttpResponse res = h.send(req);
System.debug('~~~ '+res.getBody());
It generates following Endpoint:
Endpoint=https://api.xero.com/oauth/RequestToken?oauth_consumer_key=ICSP7Y5K2TG7RIIC6Y7R7KLC1AHWYC&oauth_signature_method=RSA-SHA1&oauth_signature=gWP02y2EIatw4xilTvd5Iq3e0%2Fw%3D&oauth_timestamp=1372123781&oauth_nonce=1372123781&oauth_version=1.0&scope=https%3A%2F%2Fapi.xero.com%2Fapi.xro%2F2.0
Just as an aside: I've never worked with salesforce so I'm not sure if there's a better
way to leverage existing oauth work on the platform, it's very rare
now to have to write all the oauth signature stuff yourself and it's
easy to make a mistake but here goes]
I think your signature base string is incorrect.
As far as I can tell you're just performing HMAC-SHA1 over https://api.xero.com/api.xro/2.0
if you read the OAuth Spec here: http://oauth.net/core/1.0/#anchor14 you need to construct the following base string (based on the request above)
GET&https%3A%2F%2Fapi.xero.com%2Foauth%2Frequesttoken&oauth_consumer_key%3DCONSUMER_KEY%26oauth_nonce (etc etc, just append all your query parameters apart from oauth_consumer as url encoded key=value pairs, in alphabetical order)
and then you need to create the hash with the key CONSUMER_KEY&CONSUMER_SECRET (both CONSUMER_KEY and CONSUMER_SECRET should be parameter encoded as per the OAuth Spec)
That should give you a valid signature..
Edit: I found this library which might be of help: https://code.google.com/p/sfdc-oauth-playground/