Terraform data source of an existing S3 bucket fails plan stage attempting a GetBucketWebsite request which returns NoSuchWebsiteConfiguration - amazon-s3

I'm trying to use a data source of an existing S3 bucket like this:
data "aws_s3_bucket" "src-config-bucket" {
bucket = "single-word-name" }
And Terraform always fails the plan stage with the message:
Error: UnauthorizedOperation: You are not authorized to perform this operation.
status code: 403, request id: XXXXX
The requests failing can be viewed with the following info in the results:
{
"eventVersion": "1.08",
"userIdentity": {
​​
"type": "IAMUser",
"principalId": "ANONYMIZED",
"arn": "arn:aws:iam::1234567890:user/terraformops",
"accountId": "123456789012",
"accessKeyId": "XXXXXXXXXXXXXXXXXX",
"userName": "terraformops"
}​​,
"eventTime": "2021-02-02T18:12:19Z",
"eventSource": "s3.amazonaws.com",
"eventName": "GetBucketWebsite",
"awsRegion": "eu-west-1",
"sourceIPAddress": "X.Y.Z.W",
"userAgent": "[aws-sdk-go/1.36.28 (go1.15.5; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.14.4 (+https://www.terraform.io)]",
"errorCode": "NoSuchWebsiteConfiguration",
"errorMessage": "The specified bucket does not have a website configuration",
"requestParameters": {
​​
"bucketName": "s3-bucket-name",
"website": "",
"Host": "s3-bucket-name.s3.eu-west-1.amazonaws.com"
}
Why can't I use an existing S3 bucket as a data source within Terraform ? I don't treat it as a website anywhere in the terraform project so I don't know why it asks the server the GetBucketWebsite call and fail. Hope someone can help.
Thanks.

I don't know why it asks the server the GetBucketWebsite call and fail.
It asks GetBucketWebsite as data source aws_s3_bucket returns this information by providing website_endpoint and website_domain.
So you need to have permissions to call this action on the bucket. The error message suggests that the IAM user/role which you use for querying the bucket does not have all permissions to get needed information.

Related

Filepulse Connector error with S3 provider (Source Connector)

I am trying to poll csv files from S3 buckets using Filepulse source connector. When the task starts I get the following error. What additional libraries do I need to add to make this work from S3 bucket ? Config file below.
Where did I go wrong ?
Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:208)
java.nio.file.FileSystemNotFoundException: Provider "s3" not installed
at java.base/java.nio.file.Path.of(Path.java:212)
at java.base/java.nio.file.Paths.get(Paths.java:98)
at io.streamthoughts.kafka.connect.filepulse.fs.reader.LocalFileStorage.exists(LocalFileStorage.java:62)
Config file :
{
"name": "FilePulseConnector_3",
"config": {
"connector.class": "io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector",
"filters": "ParseCSVLine, Drop",
"filters.Drop.if": "{{ equals($value.artist, 'U2') }}",
"filters.Drop.invert": "true",
"filters.Drop.type": "io.streamthoughts.kafka.connect.filepulse.filter.DropFilter",
"filters.ParseCSVLine.extract.column.name": "headers",
"filters.ParseCSVLine.trim.column": "true",
"filters.ParseCSVLine.seperator": ";",
"filters.ParseCSVLine.type": "io.streamthoughts.kafka.connect.filepulse.filter.DelimitedRowFilter",
"fs.cleanup.policy.class": "io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy",
"fs.cleanup.policy.triggered.on":"COMMITTED",
"fs.listing.class": "io.streamthoughts.kafka.connect.filepulse.fs.AmazonS3FileSystemListing",
"fs.listing.filters":"io.streamthoughts.kafka.connect.filepulse.fs.filter.RegexFileListFilter",
"fs.listing.interval.ms": "10000",
"file.filter.regex.pattern":".*\\.csv$",
"offset.policy.class":"io.streamthoughts.kafka.connect.filepulse.offset.DefaultSourceOffsetPolicy",
"offset.attributes.string": "name",
"skip.headers": "1",
"topic": "connect-file-pulse-quickstart-csv",
"tasks.reader.class": "io.streamthoughts.kafka.connect.filepulse.fs.reader.LocalRowFileInputReader",
"tasks.file.status.storage.class": "io.streamthoughts.kafka.connect.filepulse.state.KafkaFileObjectStateBackingStore",
"tasks.file.status.storage.bootstrap.servers": "172.27.157.66:9092",
"tasks.file.status.storage.topic": "connect-file-pulse-status",
"tasks.file.status.storage.topic.partitions": 10,
"tasks.file.status.storage.topic.replication.factor": 1,
"tasks.max": 1,
"aws.access.key.id":"<<>>",
"aws.secret.access.key":"<<>>",
"aws.s3.bucket.name":"mytestbucketamtrak",
"aws.s3.region":"us-east-1"
}
}
What should I put in the libraries to make this work ? Note : The lenses connector sources from S3 bucket without issues. So its not a credentials issue.
As mentioned in comments by #OneCricketeer
Suggest you follow - github.com/streamthoughts/kafka-connect-file-pulse/issues/382 pointed to root cause.
Modifying the config file to use this property sourced the file:
"tasks.reader.class": "io.streamthoughts.kafka.connect.filepulse.fs.reader.AmazonS3RowFileInputReader"

Validation error in aws cloudwatch events rule?

I am triggering my codebuild using codebuild triggers feature with an cron expression cron(*/2 * * * ? *) which triggers for every 2 minutes . Unfortunately, this didn't run after 2 minutes when i checked the cloudwatch show metrics i can see that there were some failedinvocations. To know the cause the of the error i enabled the cloudtrail logs and i can see the error like this
{
"eventVersion": "1.04",
"userIdentity": {
"type": "IAMUser",
"principalId": "xx",
"arn": "arn:aws:iam::xx:user/xx",
"accountId": "xx",
"accessKeyId": "xx",
"userName": "xx",
"sessionContext": {
"attributes": {
"mfaAuthenticated": "true",
"creationDate": "2019-03-04T06:21:22Z"
}
},
"invokedBy": "signin.amazonaws.com"
},
"eventTime": "2019-03-04T09:04:56Z",
"eventSource": "monitoring.amazonaws.com",
"eventName": "DescribeAlarms",
"awsRegion": "ap-south-1",
"sourceIPAddress": "xxx",
"userAgent": "signin.amazonaws.com",
"errorCode": "ValidationException",
"errorMessage": "1 validation error detected: Value 'INVALID_FOR_SUMMARY' at 'stateValue' failed to satisfy constraint: Member must satisfy enum value set: [INSUFFICIENT_DATA, ALARM, OK]",
"requestParameters": {
"stateValue": "INVALID_FOR_SUMMARY"
},
"responseElements": null,
"requestID": "94f3a789-3e5c-11e9-92f8-xxx",
"eventID": "c9ecfca2-a650-4997-b707-xxx",
"eventType": "AwsApiCall",
"recipientAccountId": "xxx"
}
What is this exactly mean 1 validation error detected: Value 'INVALID_FOR_SUMMARY' at 'stateValue' failed to satisfy constraint: Member must satisfy enum value set: [INSUFFICIENT_DATA, ALARM, OK] ?
Does this error is the reason for not triggering my code build ?
Any help is appreciated
Thanks
Do it from amazon command line, it looks like a known issue in AWS. I managed to update mine via the regular CI job of the orchestration.

Not able to create Tag in Google Tag Manager using the Google Tag Manager API

I want to use the Google Tag Manager API to programmatically create Tags in a containers workspace from a JSON file.
In doing so I encountered the following error after sending of a create-Request:
{
"errors": {
"error": {
"errors": [
{
"domain": "global",
"reason": "invalidArgument",
"message": "Bad Request"
}
],
"code": 400,
"message": "Bad Request"
}
}
I then tried to create a minimal tag in the same workspace following this example: https://developers.google.com/tag-manager/api/v2/devguide with the same result.
Its the same for another newly created container and workspace with no other elements in it.
Does someone know a possible solution? Thanks in advance.

OpenShift Service Account Permissions to Read Pod and Deployment Status

I would like to use the OpenShift REST api to make queries from a separate portal. We first created a service account using the following steps (where my-id is an admin in the project):
C:\openshift>oc login
Authentication required for https://openshift-test.foo.com:8443 (openshift)
Username: my-id
Password:
Login successful.
You have access to the following projects and can switch between them with 'oc project <projectname>':
* datalake-replication-consumers
datalake-replication-demo
Using project "datalake-replication-consumers".
C:\openshift>oc create serviceaccount gmi-registry
serviceaccount "gmi-registry" created
C:\openshift>oc policy add-role-to-user admin system:serviceaccounts:datalake-replication-consumers:gmi-registry
role "admin" added: "system:serviceaccounts:datalake-replication-consumers:gmi-registry"
C:\openshift>oc serviceaccounts get-token gmi-registry
<token here>
I then pasted that token as a bearer token into Postman to make a few api calls. Since I added my service account to the admin role within my project, I assumed this would work, but instead we're getting back a 403.
GET pods:
https://openshift-test.foo.com:8443/api/v1/namespaces/datalake-replication-consumers/pods
Response:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "User \"system:serviceaccount:datalake-replication-consumers:gmi-registry\" cannot list pods in project \"datalake-replication-consumers\"",
"reason": "Forbidden",
"details": {
"kind": "pods"
},
"code": 403
}
GET specific deployment:
https://openshift-test.foo.com:8443/oapi/v1/namespaces/datalake-replication-consumers/deploymentconfigs/entity-65869977-9d56-49a5-afa2-4a547df82d5c
Response:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "User \"system:serviceaccount:datalake-replication-consumers:gmi-registry\" cannot get deploymentconfigs in project \"datalake-replication-consumers\"",
"reason": "Forbidden",
"details": {
"name": "entity-65869977-9d56-49a5-afa2-4a547df82d5c",
"kind": "deploymentconfigs"
},
"code": 403
}
What are we missing for service account permissions here?
UDPATE: I should also add that I pulled my own bearer token out of the CLI and used that for both calls. That worked.
Not a very exciting answer, but our problem was solved when we installed a v3.7 instance. My initial tests were on v1.5, which I think corresponds to 3.5 or 3.6 in the enterprise offering?

AWS xray put trace segment command return error

I am trying to send segment doc manually using the CLI with example on this page: https://docs.aws.amazon.com/xray/latest/devguide/xray-api-sendingdata.html#xray-api-segments
I created my own Trace ID and also start and end time.
The command i used are:
> DOC='{"trace_id": "'$TRACE_ID'", "id": "6226467e3f841234", "start_time": 1581596193, "end_time": 1581596198, "name": "test.com"}'
>echo $DOC
{"trace_id": "1-5e453c54-3dc3e03a3c86f97231d06c88", "id": "6226467e3f845502", "start_time": 1581596193, "end_time": 1581596198, "name": "test.com"}
> aws xray put-trace-segments --trace-segment-documents $DOC
{
"UnprocessedTraceSegments": [
{
"ErrorCode": "ParseError",
"Message": "Invalid segment. ErrorCode: ParseError"
},
{
"ErrorCode": "MissingId",
"Message": "Invalid segment. ErrorCode: MissingId"
},
{
"ErrorCode": "MissingId",
"Message": "Invalid segment. ErrorCode: MissingId"
},
.................
The put-trace-segment keep giving me error. The segment doc comply with the JSON schema too. Am i missing something else?
Thanks.
I need to enclose the JSON with "..". The command that works for me was: aws xray put-trace-segments --trace-segment-documents "$DOC"
This is probably due an error in the documentation or that the xray team was using another kind of shell.