I'm working on a Python 3 script designed to get S3 space utilization statistics from AWS CloudFront using the Boto3 library.
I started with the AWS CLI and found I could get what I'm after with a command like this:
aws cloudwatch get-metric-statistics --metric-name BucketSizeBytes --namespace AWS/S3 --start-time 2017-03-06T00:00:00Z --end-time 2017-03-07T00:00:00Z --statistics Average --unit Bytes --r
from datetime import datetime, timedelta
import boto3
seconds_in_one_day = 86400 # used for granularity
cloudwatch = boto3.client('cloudwatch')
response = cloudwatch.get_metric_statistics(
Namespace='AWS/S3',
Dimensions=[
{
'Name': 'BucketName',
'Value': 'foo-bar'
},
{
'Name': 'StorageType',
'Value': 'StandardStorage'
}
],
MetricName='BucketSizeBytes',
StartTime=datetime.now() - timedelta(days=7),
EndTime=datetime.now(),
Period=seconds_in_one_day,
Statistics=[
'Average'
],
Unit='Bytes'
)
print(response)
If I execute above code it is returning json output ,but I want to define function from cloudwatch on wards entire code ..and make it parameterized,but the problem is when i define a function
It returns. Error code saying that response variable not defined...
Pls suggest how to use that in function
Related
My setup is the following:
React-native app client -> AWS API Gateway -> AWS Lambda function -> AWS S3 -> AWS Transcribe -> AWS S3
I am successfully able to upload an audio file to an S3 bucket from the lambda, start the transcription and even access it manually in the S3 bucket. However when I try to access the json file with the transcription data using TranscriptFileUri I am getting 403 response.
On the s3 bucket with the transcriptions I have the following CORS configuration:
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"GET",
"PUT",
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": [
"ETag"
]
}
]
My lambda function code looks like this:
response = client.start_transcription_job(
TranscriptionJobName=jobName,
LanguageCode='en-US',
MediaFormat='mp4',
Media={
'MediaFileUri': s3Path
},
OutputBucketName = 'my-transcription-bucket',
OutputKey = str(user_id) + '/'
)
while True:
result = client.get_transcription_job(TranscriptionJobName=jobName)
if result['TranscriptionJob']['TranscriptionJobStatus'] in ['COMPLETED', 'FAILED']:
break
time.sleep(5)
if result['TranscriptionJob']['TranscriptionJobStatus'] == "COMPLETED":
data = result['TranscriptionJob']['Transcript']['TranscriptFileUri']
data = requests.get(data)
print(data)
In Cloudwatch I get the following: <Response [403]> when printing the response.
As far as I can tell, your code is invoking requests.get(data) where data is the TranscriptFileUri. What does that URI look like? Is it signed? If not, as I suspect, then you cannot use requests to get the file from S3 (it would have to be a signed URL or a public object for this to work).
You should use an authenticated mechanism such as get_object.
I have an application that needs to be deployed on EKS, and I'm having trouble setting up an ingress ALB.
I am using the following as a sample for how this should be set up.
https://github.com/aws-samples/nexus-oss-on-aws/blob/d3a092d72041b65ca1c09d174818b513594d3e11/src/lib/sonatype-nexus3-stack.ts#L207-L242
It's in TypeScript and I'm converting it to Python. My code is as below.
from aws_cdk import (
Stack,
aws_eks as eks,
aws_ec2 as ec2,
aws_iam as iam,
Duration
)
from constructs import Construct
class TestStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
vpc = ec2.Vpc(self, "test-vpc",
vpc_name="test-vpc",
cidr="10.0.0.0/16"
)
eks_role = iam.Role(
self, 'test-eks-role',
role_name = 'test-eks-role',
assumed_by=iam.CompositePrincipal(
iam.ServicePrincipal('eks.amazonaws.com')
),
managed_policies=[iam.ManagedPolicy.from_aws_managed_policy_name('AmazonEKSClusterPolicy')],
)
cluster = eks.Cluster(
self, "test-cluster",
cluster_name="test-cluster",
masters_role=eks_role,
version=eks.KubernetesVersion.V1_21,
vpc=vpc,
vpc_subnets=[ec2.SubnetSelection(subnet_type=ec2.SubnetType.PRIVATE_WITH_NAT)]
)
alb_service_account = cluster.add_service_account(
'test-cluster-service-account',
name='test-cluster-service-account'
)
import requests
alb_controller_url = 'https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json'
policy_json = requests.get(url=alb_controller_url).json()
for statement in policy_json['Statement']:
alb_service_account.add_to_principal_policy(iam.PolicyStatement.from_json(statement))
cluster.add_helm_chart(
'aws-load-balancer-controller-helm-chart',
chart='aws-load-balancer-controller',
repository='https://aws.github.io/eks-charts',
release='aws-load-balancer-controller',
version='1.4.1',
wait=True,
timeout=Duration.minutes(15),
values={
"clusterName": cluster.cluster_name,
"image": {
"repository": "602401143452.dkr.ecr.ap-southeast-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.4.1",
},
"serviceAccount": {
"create": False,
"name": alb_service_account.service_account_name,
},
},
)
Right now I'm getting the following cryptic error message.
Received response status [FAILED] from custom resource. Message returned: Error: b'Error: UPGRADE FAILED: another operation (i
nstall/upgrade/rollback) is in progress\n'
Any advice would be greatly appreciated!
There is an AlbController construct available in the CDK, you could try with that and see if that works for you.
I am actually using the construct myself but am facing the same error message. There is this GitHub issue regarding the Helm error itself, however the rollback solution mentioned there is not applicable for me, there appears to be no state of the Helm release despite the error. I have raised this as an issue on the CDK repo.
we are trying to migrate data from aws s3 to gcp storage. we tried transfer job in gcp and its working fine. So we wanted to achieve that programmatically with aws lambda since we have dependencies on aws.
When i tried importing google.cloud module I am getting this error
lambda cloudwatch logs
Here is my code:
import os
import logging
import boto3
#from StringIO import StringIO
from google.cloud import storage
#import google-cloud-storage
# Setup logging
LOG = logging.getLogger(__name__)
LOG.setLevel(os.environ.get('LOG_LEVEL', 'INFO'))
GCS_BUCKET_NAME=os.environ['GCS_BUCKET_NAME']
S3 = boto3.client('s3')
def lambda_handler(event, context):
try:
l_t_bucketKey = _getKeys(event)
# Create google client
storage_client = storage.Client()
gcs_bucket = storage_client.get_bucket(os.environ['GCS_BUCKET_NAME'])
LOG.debug('About to copy %d files', len(l_t_bucketKey))
for bucket, key in l_t_bucketKey:
try:
inFileObj = StringIO()
S3.download_fileobj(
Bucket=bucket,
Key=key,
Fileobj=inFileObj
)
blob = gcs_bucket.blob(key)
blob.upload_from_file(inFileObj, rewind=True) # seek(0) before reading file obj
LOG.info('Copied s3://%s/%s to gcs://%s/%s', bucket, key, GCS_BUCKET_NAME, key)
except:
LOG.exception('Error copying file: {k}'.format(k=key))
return 'SUCCESS'
except Exception as e:
LOG.exception("Lambda function failed:")
return 'ERROR'
def _getKeys(d_event):
"""
Extracts (bucket, key) from event
:param d_event: Event dict
:return: List of tuples (bucket, key)
"""
l_t_bucketKey = []
if d_event:
if 'Records' in d_event and d_event['Records']:
for d_record in d_event['Records']:
try:
bucket = d_record['s3']['bucket']['name']
key = d_record['s3']['object']['key']
l_t_bucketKey.append((bucket, key))
except:
LOG.warn('Error extracting bucket and key from event')
return l_t_bucketKey
And I downloaded google-cloud-storage module from pypi website and imported that in aws lambda layer. Please help in providing me the best link for downloading this module.
Google Storage Bucket can be used with S3 APIs, so you can just use it in your Lambda functions without any extra GCP libraries.
source_client = boto3.client(
's3',
endpoint_url='https://storage.googleapis.com',
aws_access_key_id=os.environ['GCP_KEY'],
aws_secret_access_key=os.environ['GCP_SECRET']
To get access_key and secret - go to the GS bucket settings -> Interoperability -> Access keys for your user account -> Create a key
We are not able to create neptune cluster using this python boto3 library, and boto3 function is given below
**Functions is:**
import boto3
client = boto3.client('neptune')
response = client.create_db_cluster(
AvailabilityZones=[
'us-west-2c', 'us-west-2b',
],
BackupRetentionPeriod=1,
DatabaseName='testdcluster',
DBClusterIdentifier='testdb',
DBClusterParameterGroupName='default.neptune1',
VpcSecurityGroupIds=[
'sg-xxxxxxxxx',
],
DBSubnetGroupName='profilex',
Engine='neptune',
EngineVersion='1.0.1.0',
Port=8182,
Tags=[
{
'Key': 'purpose',
'Value': 'test'
},
],
StorageEncrypted=False,
EnableIAMDatabaseAuthentication=False,
DeletionProtection=False,
SourceRegion='us-west-2'
)
Error message is also given below
**error message :**
when calling the CreateDBCluster operation: The parameter DatabaseName is not valid for engine: neptune
could you please help to fix this ?
Rather than using DatabaseName just use DBClusterIdentifier and that will become the name of your cluster. The DatabaseName parameter is not needed when creating a Neptune cluster.
I have a Chalice app that reads config data from a file in an S3 bucket. The file can change from time to time, and I want the app to immediately use the updated values, so I am using the on_s3_event decorator to reload the config file.
My code looks something like this (stripped way down for clarity):
CONFIG = {}
app = Chalice(app_name='foo')
#app.on_s3_event(bucket=S3_BUCKET, events=['s3:ObjectCreated:*'],
prefix='foo/')
def event_handler(event):
_load_config()
def _load_config():
# fetch json file from S3 bucket
CONFIG['foo'] = some item from the json file...
CONFIG['bar'] = some other item from the json file...
_load_config()
#app.route('/')
def home():
# refer to CONFIG values here
My problem is that for a short while (maybe 5-10 minutes) after uploading a new version of the config file, the app still uses the old config values.
Am I doing this wrong? Should I not be depending on global state in a Lambda function at all?
So your design here is flawed.
When you create an S3 Event in chalice it will create a separate Lambda function for that event. the CONFIG variable would get updated in the running instance of that Lambda function and all new instances of the Lambda function. However any other Lambdas in your Chalice app that are already running would just continue on with their current settings until they were cleaned up and restarted.
If you cannot live with a config that is only changeable when you deploy your Lambda functions you could use redis or some other in memory cache/db.
You should be using the .config/config.json file to store your variables for your chalice application. Those variables are stored in the os library and can be called:
URL = os.environ['MYVAR']
Your config.json file might look like this:
{
"version": "2.0",
"app_name": "MyApp",
"manage_iam_role": false,
"iam_role_arn": "arn:aws:iam::************:role/Chalice",
"lambda_timeout": 300,
"stages": {
"development": {
"environment_variables": {
"MYVAR": "foo"
}
},
"production": {
"environment_variables": {
"MYVAR": "bar"
}
}
},
"lambda_memory_size": 2048
}