How do I specify a FIPS endpoint url in a boto3 S3 client configuration? - amazon-s3
I am currently attempting to configure my boto3 client to connect using a custom endpoint URL for FIPS. https://aws.amazon.com/compliance/fips/
I have the following code that works with boto3 for EC2.
ec2_client.py
import boto3
from botocore.config import Config
config = Config(
retries = dict(
max_attempts = 1
)
)
boto3.set_stream_logger(name='botocore')
ec2_client = boto3.client(
service_name='ec2',
endpoint_url='https://ec2-fips.us-east-1.amazonaws.com',
config=config
)
ec2_client.describe_regions()
Output:
$ python ec2_client.py
2019-08-22 17:35:55,183 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-08-22 17:35:55,187 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2019-08-22 17:35:55,187 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-08-22 17:35:55,189 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-08-22 17:35:55,190 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-08-22 17:35:55,190 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-08-22 17:35:55,191 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-08-22 17:35:55,193 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-08-22 17:35:55,193 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-08-22 17:35:55,194 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-08-22 17:35:55,194 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-08-22 17:35:55,218 botocore.credentials [DEBUG] Looking for credentials via: env
2019-08-22 17:35:55,219 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2019-08-22 17:35:55,219 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2019-08-22 17:35:55,220 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2019-08-22 17:35:55,220 botocore.loaders [DEBUG] Loading JSON file: /Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/data/endpoints.json
2019-08-22 17:35:55,226 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x1056ae488>
2019-08-22 17:35:55,244 botocore.loaders [DEBUG] Loading JSON file: /Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/data/ec2/2016-11-15/service-2.json
2019-08-22 17:35:55,275 botocore.hooks [DEBUG] Event creating-client-class.ec2: calling handler <function add_generate_presigned_url at 0x10577de18>
2019-08-22 17:35:55,275 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
2019-08-22 17:35:55,280 botocore.endpoint [DEBUG] Setting ec2 timeout as (60, 60)
2019-08-22 17:35:55,281 botocore.loaders [DEBUG] Loading JSON file: /Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/data/_retry.json
2019-08-22 17:35:55,282 botocore.client [DEBUG] Registering retry handlers for service: ec2
2019-08-22 17:35:55,283 botocore.hooks [DEBUG] Event before-parameter-build.ec2.DescribeRegions: calling handler <bound method ParameterAlias.alias_parameter_in_call of <botocore.handlers.ParameterAlias object at 0x1057e6438>>
2019-08-22 17:35:55,283 botocore.hooks [DEBUG] Event before-parameter-build.ec2.DescribeRegions: calling handler <function generate_idempotent_uuid at 0x1057deea0>
2019-08-22 17:35:55,283 botocore.hooks [DEBUG] Event before-call.ec2.DescribeRegions: calling handler <function inject_api_version_header_if_needed at 0x1057e29d8>
2019-08-22 17:35:55,283 botocore.endpoint [DEBUG] Making request for OperationModel(name=DescribeRegions) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.9.6 Python/3.6.3 Darwin/18.7.0 Botocore/1.12.146'}, 'body': {'Action': 'DescribeRegions', 'Version': '2016-11-15'}, 'url': 'https://ec2-fips.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x106d3a4a8>, 'has_streaming_input': False, 'auth_type': None}}
2019-08-22 17:35:55,283 botocore.hooks [DEBUG] Event request-created.ec2.DescribeRegions: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x106d3a438>>
2019-08-22 17:35:55,284 botocore.hooks [DEBUG] Event choose-signer.ec2.DescribeRegions: calling handler <function set_operation_specific_signer at 0x1057ded90>
2019-08-22 17:35:55,284 botocore.auth [DEBUG] Calculating signature using v4 auth.
2019-08-22 17:35:55,285 botocore.auth [DEBUG] CanonicalRequest:
POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:ec2-fips.us-east-1.amazonaws.com
x-amz-date:20190822T213555Z
x-amz-security-token:__xxx__
content-type;host;x-amz-date;x-amz-security-token
__xxx__
2019-08-22 17:35:55,285 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20190822T213555Z
20190822/us-east-1/ec2/aws4_request
791b3e04eac140d25ccb2c00d0d2489c3bab1cccf619bfa5df7a8d22a5826d7f
2019-08-22 17:35:55,285 botocore.auth [DEBUG] Signature:
__xxx__
2019-08-22 17:35:55,285 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ec2-fips.us-east-1.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'Boto3/1.9.6 Python/3.6.3 Darwin/18.7.0 Botocore/1.12.146', 'X-Amz-Date': b'20190822T213555Z', 'X-Amz-Security-Token': b'__xxx__', 'Authorization': b'AWS4-HMAC-SHA256 Credential=__xxx__/20190822/us-east-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=__xxx__', 'Content-Length': '41'}>
2019-08-22 17:35:55,495 botocore.parsers [DEBUG] Response headers: {'Content-Type': 'text/xml;charset=UTF-8', 'Content-Length': '3655', 'vary': 'accept-encoding', 'Date': 'Thu, 22 Aug 2019 21:35:55 GMT', 'Server': 'AmazonEC2'}
2019-08-22 17:35:55,495 botocore.parsers [DEBUG] Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<DescribeRegionsResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">\n <requestId>1cf5ab2e-a72b-4cb7-bd77-1c9dbe1d89c1</requestId>\n <regionInfo>\n <item>\n <regionName>eu-north-1</regionName>\n <regionEndpoint>ec2.eu-north-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>ap-south-1</regionName>\n <regionEndpoint>ec2.ap-south-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>eu-west-3</regionName>\n <regionEndpoint>ec2.eu-west-3.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>eu-west-2</regionName>\n <regionEndpoint>ec2.eu-west-2.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>eu-west-1</regionName>\n <regionEndpoint>ec2.eu-west-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>ap-northeast-2</regionName>\n <regionEndpoint>ec2.ap-northeast-2.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>ap-northeast-1</regionName>\n <regionEndpoint>ec2.ap-northeast-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>sa-east-1</regionName>\n <regionEndpoint>ec2.sa-east-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>ca-central-1</regionName>\n <regionEndpoint>ec2.ca-central-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>ap-southeast-1</regionName>\n <regionEndpoint>ec2.ap-southeast-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>ap-southeast-2</regionName>\n <regionEndpoint>ec2.ap-southeast-2.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>eu-central-1</regionName>\n <regionEndpoint>ec2.eu-central-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>us-east-1</regionName>\n <regionEndpoint>ec2.us-east-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>us-east-2</regionName>\n <regionEndpoint>ec2.us-east-2.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>us-west-1</regionName>\n <regionEndpoint>ec2.us-west-1.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n <item>\n <regionName>us-west-2</regionName>\n <regionEndpoint>ec2.us-west-2.amazonaws.com</regionEndpoint>\n <optInStatus>opt-in-not-required</optInStatus>\n </item>\n </regionInfo>\n</DescribeRegionsResponse>'
2019-08-22 17:35:55,497 botocore.hooks [DEBUG] Event needs-retry.ec2.DescribeRegions: calling handler <botocore.retryhandler.RetryHandler object at 0x106d3a588>
2019-08-22 17:35:55,497 botocore.retryhandler [DEBUG] No retry needed.
I have the following code that does not work with boto3 for S3.
s3_client.py
import boto3
from botocore.config import Config
config = Config(
retries = dict(
max_attempts = 1
)
)
boto3.set_stream_logger(name='botocore')
s3_client = boto3.client(
service_name='s3',
endpoint_url='https://s3-fips.us-east-1.amazonaws.com',
config=config
)
s3_client.list_buckets()
Output: https://gist.github.com/brokenthumbs/a2e3f05c877582e92cb4d09cc9f05459
$ python s3_client.py
2019-08-22 17:41:54,834 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-08-22 17:41:54,837 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2019-08-22 17:41:54,838 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-08-22 17:41:54,840 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-08-22 17:41:54,840 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-08-22 17:41:54,840 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-08-22 17:41:54,841 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-08-22 17:41:54,843 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-08-22 17:41:54,843 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-08-22 17:41:54,843 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-08-22 17:41:54,843 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-08-22 17:41:54,864 botocore.credentials [DEBUG] Looking for credentials via: env
2019-08-22 17:41:54,865 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2019-08-22 17:41:54,865 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2019-08-22 17:41:54,866 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2019-08-22 17:41:54,867 botocore.loaders [DEBUG] Loading JSON file: /Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/data/endpoints.json
2019-08-22 17:41:54,872 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x10a268510>
2019-08-22 17:41:54,881 botocore.loaders [DEBUG] Loading JSON file: /Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/data/s3/2006-03-01/service-2.json
2019-08-22 17:41:54,891 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10a33b158>
2019-08-22 17:41:54,891 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x10a4020d0>
2019-08-22 17:41:54,909 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10a335ea0>
2019-08-22 17:41:54,909 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
2019-08-22 17:41:54,913 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2019-08-22 17:41:54,915 botocore.loaders [DEBUG] Loading JSON file: /Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/data/_retry.json
2019-08-22 17:41:54,916 botocore.client [DEBUG] Registering retry handlers for service: s3
2019-08-22 17:41:54,917 botocore.client [DEBUG] Using S3 path style addressing.
2019-08-22 17:41:54,918 botocore.hooks [DEBUG] Event before-parameter-build.s3.ListBuckets: calling handler <function validate_bucket_name at 0x10a39a378>
2019-08-22 17:41:54,918 botocore.hooks [DEBUG] Event before-parameter-build.s3.ListBuckets: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x10b1b6390>>
2019-08-22 17:41:54,918 botocore.hooks [DEBUG] Event before-parameter-build.s3.ListBuckets: calling handler <function generate_idempotent_uuid at 0x10a397f28>
2019-08-22 17:41:54,919 botocore.hooks [DEBUG] Event before-call.s3.ListBuckets: calling handler <function add_expect_header at 0x10a39a840>
2019-08-22 17:41:54,919 botocore.hooks [DEBUG] Event before-call.s3.ListBuckets: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x10b1b6390>>
2019-08-22 17:41:54,919 botocore.hooks [DEBUG] Event before-call.s3.ListBuckets: calling handler <function inject_api_version_header_if_needed at 0x10a39ba60>
2019-08-22 17:41:54,919 botocore.endpoint [DEBUG] Making request for OperationModel(name=ListBuckets) with params: {'url_path': '/', 'query_string': '', 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.9.6 Python/3.6.3 Darwin/18.7.0 Botocore/1.12.146'}, 'body': b'', 'url': 'https://s3-fips.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x10b0b19e8>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': None}}}
2019-08-22 17:41:54,919 botocore.hooks [DEBUG] Event request-created.s3.ListBuckets: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x10b0b14e0>>
2019-08-22 17:41:54,919 botocore.hooks [DEBUG] Event choose-signer.s3.ListBuckets: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x109e0c550>>
2019-08-22 17:41:54,919 botocore.hooks [DEBUG] Event choose-signer.s3.ListBuckets: calling handler <function set_operation_specific_signer at 0x10a397e18>
2019-08-22 17:41:54,920 botocore.auth [DEBUG] Calculating signature using v4 auth.
2019-08-22 17:41:54,920 botocore.auth [DEBUG] CanonicalRequest:
GET
/
host:s3-fips.us-east-1.amazonaws.com
x-amz-content-sha256:__xxx__
x-amz-date:20190822T214154Z
x-amz-security-token:__xxx__
host;x-amz-content-sha256;x-amz-date;x-amz-security-token
__xxx__
2019-08-22 17:41:54,935 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20190822T214154Z
20190822/us-east-1/s3/aws4_request
__xxx__
2019-08-22 17:41:54,935 botocore.auth [DEBUG] Signature:
__xxx__
2019-08-22 17:41:54,936 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3-fips.us-east-1.amazonaws.com/, headers={'User-Agent': b'Boto3/1.9.6 Python/3.6.3 Darwin/18.7.0 Botocore/1.12.146', 'X-Amz-Date': b'20190822T214154Z', 'X-Amz-Security-Token': b'__xxx__', 'X-Amz-Content-SHA256': b'__xxx__', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAX6MMDPJ5HINHNPVO/20190822/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=__xxx__'}>
2019-08-22 17:41:54,944 botocore.endpoint [DEBUG] Exception received when sending HTTP request.
Traceback (most recent call last):
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/httpsession.py", line 258, in send
decode_content=False,
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/util/retry.py", line 333, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
conn = self._new_conn()
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x10b1b6ba8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 17, in <module>
s3_client.list_buckets()
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/client.py", line 648, in _make_api_call
operation_model, request_dict, request_context)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/client.py", line 667, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/endpoint.py", line 231, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/endpoint.py", line 244, in _send
return self.http_session.send(request)
File "/Users/meme/.pyenv/versions/3.6.3/lib/python3.6/site-packages/botocore/httpsession.py", line 278, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://s3-fips.us-east-1.amazonaws.com/"
What do I need to do differently in order to get the boto3 s3 client to connect to a FIPS endpoint? I see that the documentation states:
Note: These Endpoints can only be used with Virtual Hosted-Style addressing. For example: https://bucket.s3-fips.us-east-2.amazonaws.com. Visit the Amazon S3 Documentation page for more information.
I am unsure how this can be applied to my boto3 s3 client configuration. Any ideas on how I can correct my configuration to use FIPS correctly for s3?
Here's what I found that will address your problem.
Modify your Config object as such:
config = Config(
retries = {'max_attempts': 1},
s3 = {'addressing_style': 'virtual'}
)
Related
Ingress NGINX client closed connection while SSL handshaking
we have ingress-nginx running for a while and about 10% of requests ending up with some SSL handshake problem. Here is an example of a failing connection: 2019/02/14 10:15:35 [debug] 237#237: *4612 accept: **.**.**.**:40928 fd:53 2019/02/14 10:15:35 [debug] 237#237: *4612 event timer add: 53: 60000:5527050245 2019/02/14 10:15:35 [debug] 237#237: *4612 reusable connection: 1 2019/02/14 10:15:35 [debug] 237#237: *4612 epoll add event: fd:53 op:1 ev:80002001 2019/02/14 10:15:45 [debug] 237#237: *4612 http check ssl handshake 2019/02/14 10:15:45 [debug] 237#237: *4612 http recv(): 0 2019/02/14 10:15:45 [info] 237#237: *4612 client closed connection while SSL handshaking, client: **.**.**.**, server: 0.0.0.0:443 2019/02/14 10:15:45 [debug] 237#237: *4612 close http connection: 53 2019/02/14 10:15:45 [debug] 237#237: *4612 event timer del: 53: 5527050245 2019/02/14 10:15:45 [debug] 237#237: *4612 reusable connection: 0 2019/02/14 10:15:45 [debug] 237#237: *4612 free: 00007F4CC5858E00, unused: 232 10% of failures seems to be quite a lot to expect. I really would appreciate any help in this!
How to find nested element in Android Appium testing?
I am new to Appium. I searched the web and tried several ways, but nothing worked. Please find the code below:
driver.findElementByXPath("//android.widget.FrameLayout[1]/android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.RelativeLayout[#resource-id=\"in.dmart:id/linear_guide_view_skip\"]/android.widget.android.widget.TextView").click();
And
driver.findElementByXPath("//android.widget.LinearLayout[1]/android.widget.RelativeLayout[1]/android.widget.android.widget.TextView[1]").click();
And
driver.findElementByAndroidUIAutomator("text(\"GOT IT\")").click();
And
List<WebElement>textView=driver.findElements(By.className("android.widget.android.widget.TextView"));
for(i=0;i<textView.size();i++){
if(textView.get(i).getText().equals("GOT IT"))
textView.get(i).click()
}
Please find UIAutomator Viewer screenshot attached:
Server logs :
[debug] [W3C] Calling AppiumDriver.findElements() with args: ["class name","android.widget.android.widget.TextView","e75769f1-53cb-44f7-9094-988b73fbc7be"]
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, class name, accessibility id, -android uiautomator
[debug] [BaseDriver] Waiting up to 0 ms for condition
[debug] [AndroidBootstrap] Sending command to android: {"cmd":"action","action":"find","params":{"strategy":"class name","selector":"android.widget.android.widget.TextView","context":"","multiple":true}}
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Got data from client: {"cmd":"action","action":"find","params":{"strategy":"class name","selector":"android.widget.android.widget.TextView","context":"","multiple":true}}
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Got command of type ACTION
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Got command action: find
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Finding 'android.widget.android.widget.TextView' using 'CLASS_NAME' with the contextId: '' multiple: true
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Using: UiSelector[CLASS=android.widget.android.widget.TextView]
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] getElements selector:UiSelector[CLASS=android.widget.android.widget.TextView]
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Element[] is null: (0)
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] getElements tmp selector:UiSelector[CLASS=android.widget.android.widget.TextView, INSTANCE=0]
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Failed to locate element. Clearing Accessibility cache and retrying.
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Finding 'android.widget.android.widget.TextView' using 'CLASS_NAME' with the contextId: '' multiple: true
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Using: UiSelector[CLASS=android.widget.android.widget.TextView]
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] getElements selector:UiSelector[CLASS=android.widget.android.widget.TextView]
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Element[] is null: (0)
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] getElements tmp selector:UiSelector[CLASS=android.widget.android.widget.TextView, INSTANCE=0]
[debug] [AndroidBootstrap] Received command result from bootstrap
[debug] [AndroidBootstrap] [BOOTSTRAP LOG] [debug] Returning result: {"status":0,"value":[]}
[debug] [W3C] Responding to client with driver.findElements() result: []
[HTTP] <-- POST /wd/hub/session/e75769f1-53cb-44f7-9094-988b73fbc7be/elements 200 13691 ms - 12
You can try like this. Please edit className value if it is different as per your app.
get all elements of class name
Click where text is "GOT IT"
List<WebElement>textView =
driver.findElements(By.className("android.widget.TextView"));
for(i=0;i<textView.size();i++){
if(textView.get(i).getText().equals("GOT IT"))
textView.get(i).click()
}
By looking at the view hierarchy, to locate the TextView you don't need to worry about nested views.
You can try with UiSelector() from UIAutomator as an example below. Refer to UiSelector documentation here.
driver.findElementByAndroidUIAutomator("new UiSelector().text(\""+GOT IT+"\")").click();
Update:
Try casting driver with AndroidDriver.
((AndroidDriver<MobileElement>)driver).findElementByAndroidUIAutomator("new UiSelector().text(\"GOT IT\")");
Nginx inputstream ssk offloading
WHen using NginX for ssl offloading things work fine, however when trying to do this for serialized stream over a socket it seems to work incorrectly, and the documentation over using streams http://nginx.org/en/docs/stream/ngx_stream_upstream_module.html makes no sense,
can anyone help me with a working foo bar example of how to ssl offload for a streaming input(made of serialized logging event objects in this case) with nginx
Edit:
we are trying to forward and strip SSL off of log4j socketappender output but cannot even get the forwarding working when the SSL is not enabled.
here is the config file without the ssl offloading portion(when retrieving non https the forwarding still does not work correctly)
events {
worker_connections 1024;
}
http {
server {
listen 4560 ;
server_name logstash.corelims.com;
location / {
proxy_pass_request_headers on;
proxy_pass http://localhost:4561/;
proxy_redirect http://localhost:4561/ http://logstash.corelims.com:4560/;
}
error_log C:/ELK/nginx-1.9.4/logs/debug.log debug;
}
}
which leads us to these debug logs:
2015/08/27 00:54:59 [info] 5052#3716: *45 WSARecv() failed (10054: An existing connection was forcibly closed by the remote host) while reading client request line, client: IP.ADDRESS.HIDDENFOR.POST, server: foo.bar.com, request: "’ "
2015/08/27 00:54:59 [debug] 5052#3716: *45 lingering read: -1
2015/08/27 00:54:59 [debug] 5052#3716: *45 http request count:1 blk:0
2015/08/27 00:54:59 [debug] 5052#3716: *45 http close request
2015/08/27 00:54:59 [debug] 5052#3716: *45 http log handler
2015/08/27 00:54:59 [debug] 5052#3716: *45 free: 008DE8E8, unused: 1991
2015/08/27 00:54:59 [debug] 5052#3716: *45 close http connection: 300
2015/08/27 00:54:59 [debug] 5052#3716: *45 event timer del: 300: 1822859279
2015/08/27 00:54:59 [debug] 5052#3716: *45 select del event fd:300 ev:0
2015/08/27 00:54:59 [debug] 5052#3716: *45 reusable connection: 0
2015/08/27 00:54:59 [debug] 5052#3716: *45 free: 023BCED0
2015/08/27 00:54:59 [debug] 5052#3716: *45 free: 023BEEA8, unused: 24
2015/08/27 12:27:45 [debug] 5052#3716: select del event fd:316 ev:0
Connection time out after request is read: http-incoming-3931
One of our WSO2 ESB (4.8.1) log shows this error intermittently, What actually causes this error?
TID: [0] [ESB] [2015-07-08 09:30:09,982] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-6409: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [0] [ESB] [2015-07-08 09:30:09
TID: [0] [ESB] [2015-07-09 12:08:10,018] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-3931 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,319] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-7634: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,319] WARN {org.apache.synapse.FaultHandler} - ERROR_CODE : 101507 {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,319] WARN {org.apache.synapse.FaultHandler} - ERROR_MESSAGE : Error in Sender {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.FaultHandler} - ERROR_DETAIL : Error in Sender {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.FaultHandler} - ERROR_EXCEPTION : null {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.FaultHandler} - FaultHandler : AnonymousEndpoint {org.apache.synapse.FaultHandler}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : AnonymousEndpoint will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2015-07-09 12:08:10,320] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : AnonymousEndpoint - current suspend duration is : 30000ms - Next retry after : Thu Jul 09 12:08:40 IST 2015 {org.apache.synapse.endpoints.EndpointContext}
WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-3931
The above log says that the connection between the client and the ESB got timeout before ESB sends the response to the client. By default this timeout is 60 seconds (the socket timeout of http listener). So ESB is taking more than 60 seconds to send a response to the client. Reason might be because of your slow backend. You can increase this socket timeout of the passthrough http transport by adding http.socket.timeout=120000 to passthru-http.properties file in $ESB_HOME/repository/conf/ directory. Here socket timeout is set to 120seconds
WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-6409: Connection time out while in state: REQUEST_DONE
The above log says that the connection between the ESB and the backend got timeout before ESB gets the response from the backend. By default this timeout is 60 seconds (the socket timeout of http sender). So your backend is taking more than 60 seconds to respond. You can increase the socket timeout of the passthrough http transport by adding http.socket.timeout=120000 to passthru-http.properties file in $ESB_HOME/repository/conf directory. Here socket timeout is set to 120seconds
Please follow this troubleshoot guide to configure timeout values correctly.
deploying war using tomcat7.maven plugin in eclipse but always throwing 401 unauthorized error
Need your help with this issue. I'm trying to deploy war file at localhost:8080 through eclipse using tomcat7.maven plugin version 2.1 scripts but always getting 401 unauthorized error. My PC OS is windows 7 64 bit.
Here is my POM:
<plugin>
<groupId>org.apache.tomcat.maven</groupId>
<artifactId>tomcat7-maven-plugin</artifactId>
<configuration>
<url>http://localhost:8080/manager/text</url>
<server>localhost</server>
<username>tomcat</username>
<password>tomcat</password>
<path>/${project.build.finalName}</path>
<warDirectory>${project.build.directory}/exploded/${project.build.finalName}.war</warDirectory>
<update>true</update>
</configuration>
<version>2.1</version>
</plugin>
Here is my tomcat-users.xml :
<tomcat-users>
<user name="admin" password="admin" roles="admin-gui,manager-gui"/>
<role rolename="manager-script"/>
<role rolename="manager"/>
<role rolename="tomcat"/>
<user username="tomcat" password="tomcat" roles="manager-script"/>
</tomcat-users>
Here is my settings.xml in /.M2 directory.
<settings>
<servers>
<server>
<id>localhost</id>
<username>tomcat</username>
<password>tomcat</password>
</server>
</servers>
</settings>
Here is my log file output:
[DEBUG] Configuring mojo org.apache.tomcat.maven:tomcat7-maven-plugin:2.1:deploy from plugin realm ClassRealm[plugin>org.apache.tomcat.maven:tomcat7-maven-plugin:2.1, parent: sun.misc.Launcher$AppClassLoader#5d4177f3]
[DEBUG] Configuring mojo 'org.apache.tomcat.maven:tomcat7-maven-plugin:2.1:deploy' with basic configurator -->
[DEBUG] (f) charset = ISO-8859-1
[DEBUG] (f) contextFile = C:\java\Spring3HibernateMaven\target\Spring3HibernateMaven-0.0.1-SNAPSHOT\META-INF\context.xml
[DEBUG] (f) ignorePackaging = false
[DEBUG] (f) mode = war
[DEBUG] (f) packaging = war
[DEBUG] (f) password = tomcat
[DEBUG] (f) path = /Spring3HibernateMaven-0.0.1-SNAPSHOT
[DEBUG] (f) server = localhost
[DEBUG] (f) update = true
[DEBUG] (f) url = http://localhost:8080/manager/text
[DEBUG] (f) username = tomcat
[DEBUG] (f) version = 2.1
[DEBUG] (f) warFile = C:\java\Spring3HibernateMaven\target\Spring3HibernateMaven-0.0.1-SNAPSHOT.war
[DEBUG] -- end configuration --
[INFO] Deploying war to http://localhost:8080/Spring3HibernateMaven-0.0.1-SNAPSHOT
[DEBUG] Connection request: [route: {}->http://localhost:8080][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 5]
[DEBUG] Connection leased: [id: 0][route: {}->http://localhost:8080][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 5]
[DEBUG] Connecting to localhost:8080
[DEBUG] CookieSpec selected: best-match
[DEBUG] Re-using cached 'basic' auth scheme for http://localhost:8080
[DEBUG] Target auth state: SUCCESS
[DEBUG] Proxy auth state: UNCHALLENGED
[DEBUG] Attempt 1 to execute request
[DEBUG] Sending request: PUT /manager/text/deploy?path=%2FSpring3HibernateMaven-0.0.1-SNAPSHOT&update=true HTTP/1.1
[DEBUG] >> "PUT /manager/text/deploy?path=%2FSpring3HibernateMaven-0.0.1-SNAPSHOT&update=true HTTP/1.1[\r][\n]"
[DEBUG] >> "User-Agent: Apache Tomcat Maven Plugin/2.1[\r][\n]"
[DEBUG] >> "Content-Length: 11860176[\r][\n]"
[DEBUG] >> "Host: localhost:8080[\r][\n]"
[DEBUG] >> "Connection: Keep-Alive[\r][\n]"
[DEBUG] >> "Authorization: Basic dG9tY2F0OnRvbWNhdA==[\r][\n]"
[DEBUG] >> "[\r][\n]"
[DEBUG] >> PUT /manager/text/deploy?path=%2FSpring3HibernateMaven-0.0.1-SNAPSHOT&update=true HTTP/1.1
[DEBUG] >> User-Agent: Apache Tomcat Maven Plugin/2.1
[DEBUG] >> Content-Length: 11860176
[DEBUG] >> Host: localhost:8080
[DEBUG] >> Connection: Keep-Alive
[DEBUG] >> Authorization: Basic dG9tY2F0OnRvbWNhdA==
Uploading: http://localhost:8080/manager/text/deploy?path=%2FSpring3HibernateMaven-0.0.1-SNAPSHOT&update=true
Uploaded: http://localhost:8080/manager/text/deploy?path=%2FSpring3HibernateMaven-0.0.1-SNAPSHOT&update=true (11583 KB at 1833.5 KB/sec)
[DEBUG] << "HTTP/1.1 401 Unauthorized[\r][\n]"
[DEBUG] << "Server: Apache-Coyote/1.1[\r][\n]"
[DEBUG] << "Cache-Control: private[\r][\n]"
[DEBUG] << "Expires: Wed, 31 Dec 1969 18:00:00 CST[\r][\n]"
[DEBUG] << "WWW-Authenticate: Basic realm="Tomcat Manager Application"[\r][\n]"
[DEBUG] << "Content-Type: text/html;charset=ISO-8859-1[\r][\n]"
[DEBUG] << "Transfer-Encoding: chunked[\r][\n]"
[DEBUG] << "Date: Fri, 20 Sep 2013 05:24:35 GMT[\r][\n]"
[DEBUG] << "[\r][\n]"
[DEBUG] Receiving response: HTTP/1.1 401 Unauthorized
[DEBUG] << HTTP/1.1 401 Unauthorized
[DEBUG] << Server: Apache-Coyote/1.1
[DEBUG] << Cache-Control: private
[DEBUG] << Expires: Wed, 31 Dec 1969 18:00:00 CST
[DEBUG] << WWW-Authenticate: Basic realm="Tomcat Manager Application"
[DEBUG] << Content-Type: text/html;charset=ISO-8859-1
[DEBUG] << Transfer-Encoding: chunked
[DEBUG] << Date: Fri, 20 Sep 2013 05:24:35 GMT
[DEBUG] Connection can be kept alive indefinitely
[DEBUG] localhost:8080 requested authentication
[DEBUG] Authorization challenge processed
[DEBUG] Authentication failed
[DEBUG] Removing from cache 'basic' auth scheme for http://localhost:8080
[DEBUG] << "9ea[\r][\n]"
[DEBUG] << "[\r][\n]"
[DEBUG] << "<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">[\r][\n]"
[DEBUG] << "<html>[\r][\n]"
[DEBUG] << " <head>[\r][\n]"
[DEBUG] << " <title>401 Unauthorized</title>[\r][\n]"
[DEBUG] << " <style type="text/css">[\r][\n]"
[DEBUG] << " <!--[\r][\n]"
[DEBUG] << " BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;font-size:12px;}[\r][\n]"
[DEBUG] << " H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}[\r][\n]"
[DEBUG] << " PRE, TT {border: 1px dotted #525D76}[\r][\n]"
[DEBUG] << " A {color : black;}A.name {color : black;}[\r][\n]"
[DEBUG] << " -->[\r][\n]"
[DEBUG] << " </style>[\r][\n]"
[DEBUG] << " </head>[\r][\n]"
[DEBUG] << " <body>[\r][\n]"
[DEBUG] << " <h1>401 Unauthorized</h1>[\r][\n]"
[DEBUG] << " <p>[\r][\n]"
[DEBUG] << " You are not authorized to view this page. If you have not changed[\r][\n]"
[DEBUG] << " any configuration files, please examine the file[\r][\n]"
[DEBUG] << " <tt>conf/tomcat-users.xml</tt> in your installation. That[\r][\n]"
[DEBUG] << " file must contain the credentials to let you use this webapp.[\r][\n]"
My Run Configuration in Eclipse:
Maven Runtime: 3.1.0 ( External )
Goal: -X -e tomcat7:deploy