Currently writing tests and trying to make use of the Stubber provided by botocore.
I'm trying:
client = boto3.client("s3")
response = {'Body': 'content'}
expected_params = {'Bucket': 'a_bucket_name', 'Key': 'a_path', 'Filename': 'a_target'}
with Stubber(client) as stubber:
stubber.add_response('download_file', response, expected_params)
download_file(client, "a_bucket_name", "a_path", "a_target")
Where that download file is my own function that just wraps the client download_file call. It works in practice.
However, the test fails on the stubber.add_response due to a 'OperationNotFound' error. I stepped through using the debugger, and the issue appears here in the stub API:
if not hasattr(self.client, method):
raise ValueError(
"Client %s does not have method: %s"
% (self.client.meta.service_model.service_name, method))
# Create a successful http response
http_response = AWSResponse(None, 200, {}, None)
operation_name = self.client.meta.method_to_api_mapping.get(method) <------- Error here
self._validate_response(operation_name, service_response)
There doesn't seem to be a mapping between the two in the dictionary, is this a failure of the stub API or am I missing something?
I've just found this issue, so looks like for once it really is the library and not me:
https://github.com/boto/botocore/issues/974
That's because download_file and upload_file are customizations which live in boto3. They call out to one or many requests under the hood. Right now there's not a great story for supporting customizations other than recording underlying commands they use and adding them to the stubber. There's an external library that can handle that for you, though we don't support it ourselves.
Related
We have a data pipeline built in Google Cloud Dataflow that consumes messages from a pubsub topic and streams them into BigQuery. In order to test that it works successfully we have some tests that run in a CI pipeline, these tests post messages onto the pubsub topic and verify that the messages are written to BigQuery successfully.
This is the code that posts to the pubsub topic:
from google.cloud import pubsub_v1
def post_messages(project_id, topic_id, rows)
futures = dict()
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(
project_id, topic_id
)
def get_callback(f, data):
def callback(f):
try:
futures.pop(data)
except:
print("Please handle {} for {}.".format(f.exception(), data))
return callback
for row in rows:
# When you publish a message, the client returns a future. Data must be a bytestring
# ...
# construct a message in var json_data
# ...
message = json.dumps(json_data).encode("utf-8")
future = publisher.publish(
topic_path,
message
)
futures_key = str(message)
futures[futures_key] = future
future.add_done_callback(get_callback(future, futures_key))
# Wait for all the publish futures to resolve before exiting.
while futures:
time.sleep(1)
When we run this test in our CI pipeline it has started failing intermittently with error
21:38:55: AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7f5247407220>" raised exception!
Traceback (most recent call last):
File "/opt/conda/envs/py3/lib/python3.8/site-packages/grpc/_plugin_wrapping.py", line 89, in __call__
self._metadata_plugin(
File "/opt/conda/envs/py3/lib/python3.8/site-packages/google/auth/transport/grpc.py", line 101, in __call__
callback(self._get_authorization_headers(context), None)
File "/opt/conda/envs/py3/lib/python3.8/site-packages/google/auth/transport/grpc.py", line 87, in _get_authorization_headers
self._credentials.before_request(
File "/opt/conda/envs/py3/lib/python3.8/site-packages/google/auth/credentials.py", line 134, in before_request
self.apply(headers)
File "/opt/conda/envs/py3/lib/python3.8/site-packages/google/auth/credentials.py", line 110, in apply
_helpers.from_bytes(token or self.token)
File "/opt/conda/envs/py3/lib/python3.8/site-packages/google/auth/_helpers.py", line 130, in from_bytes
raise ValueError("***0!r*** could not be converted to unicode".format(value))
ValueError: None could not be converted to unicode
Error: The operation was canceled.
Unfortunately this only fails in our CI pipeline, and even then it is failing intermittently (only fails on a small percentage of all CI pipeline runs). If I run the same test locally it succeeds every time. When running in the CI pipeline the code is authenticating as a service account whereas when I run it locally it is authenticating as myself
I know from the error message that it is failing on this code:
if isinstance(result, six.text_type):
return result
else:
raise ValueError("{0!r} could not be converted to unicode".format(value))
https://github.com/googleapis/google-auth-library-python/blob/3c3fbf40b07e090f2be7fac5b304dbf438b5cd6c/google/auth/_helpers.py#L127-L130
which is in a python library from google that we install using pip.
Clearly the expression:
isinstance(result, six.text_type)
is evaluating to False. I put a breakpoint on that code when I ran it locally and discovered that under normal circumstances (i.e. when it works) the value of result is something like this:
That looks like some sort of auth token.
Given the error message:
ValueError: None could not be converted to unicode
it seems that whatever action is being undertaken by the google authentication libraries it is passing None through to the code shown above.
I am at the bounds of my knowledge here. Given this is only failing in a CI pipeline I don't have the opportunity to put a breakpoint in my code and debug it. Given the call stack in the error message this is something to do with authentication.
I'm hoping someone can advise on a course of action.
Can anyone explain a means by which I can discover why None is being passed through to the code that is raising an error?
We had the same error. Finally solved it by using a JSON Web Token for authentication per Google's Quckstart. Like so:
import json
from google.cloud import pubsub_v1
from google.auth import jwt
def post_messages(credentials_path, topic, list_of_messages):
credentials_dict = json.load(open(credentials_path,'r'))
audience = "https://pubsub.googleapis.com/google.pubsub.v1.Publisher"
credentials_ob = jwt.Credentials.from_service_account_info(
credentials_dict, audience=audience
)
publisher = pubsub_v1.PublisherClient(credentials=credentials_ob)
for message_dict in list_of_message_dicts:
message = json.dumps(message_dict, default=str).encode("utf-8")
future = publisher.publish(topic, message)
We also updated our environment but it didn't fix the ValueError until we changed to jwt. Here's the environment in any case:
google-api-core==2.4.0
google-api-python-client==2.36.0
google-auth==2.3.2
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-core==2.1.0
google-cloud-pubsub==2.9.0
Tried the jwt solution above and though it solved the issue, it drastically degraded my write throughput.
Offering another work around that solved this issue for me.
My GOOGLE_APPLICATION_CREDENTIALS env var was set to the location of my key-file. Instead, unset that env variable and, at the start of your process, run
gcloud auth activate-service-account {account_name} --key-file {location_of_key_file}
This allows the google auth to bypass a key file and use the default service account set up (which is now the original, intended service account). Works with normal throughput and zero errors. :)
Recently my application started getting an error related to proxies
> in __init__
> raise ProxySchemeUnknown(proxy.scheme) urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None
I did not make any changes to the code or performed any updates to python3.8, which is what im using.
here is the function im using to fetch proxies from an api that pulls them from the DB
def get_proxy(self):
try:
req = self.session.post(url=self.script_function_url, headers=self.script_function_header, json={"action": "proxy"}, verify=False, timeout=20).json()
self.proxy = {"https": req['ipAddress']+":"+req['port']}
except Exception as e:
print(f'Proxy error: {e}')
exit()
any help would be greatly appreciated i am completely new to python.
I don't know what exact line is causing the error in your code and if you have a proxy yourselve, but I know that you need to specify a scheme to do API calls behind a proxy.
So in windows you would do:
set http_proxy=http://xxx.xxx.xxx.xxx:xxxx
set https_proxy=http://xxx.xxx.xxx.xxx:xxxx
Key point here is to add the http:// in front.
We already know that scenarios are run paralelly. But we had the case where we need to return variables from feature files (that are gonna be called from another feature file).
We had multiple scenarios in the feature file as below:
#mutation
Feature: Test GraphQL Create Item
Background:
Given url baseUrl
* configure headers = { Authorization: '#(token)' }
#negative
Scenario: Create item unauthorized
* configure headers = { Authorization: ""}
#Features calling function and others
And match response.errors[0].message == errorUnauthorized
Scenario: Create story authorized
#Features calling function and others
And def idItem = response.data.CreateItem.id
We are reusing the feature file above to obtain variable to be used on another feature file. However it seems that the other feature files fail intermittently complaining the variables obtained from the other feature file is null.
My assumption is that the returned variable is not returned properly since there are more than one scenarios on the feature file. We tried deleting the #negative scenario and have only exactly 1 scenario. Suddenly the intermittent failures gone.
Is there any way to avoid this intermittent failures while still retaining the ability to run scenarios paralelly?
Thanks
Can't say without seeing your code. But you can try using the #parallel=false annotation in the "calling" feature file: https://github.com/intuit/karate#parallelfalse
Otherwise this may be a bug in Karate - so please follow this process: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
I need to add some custom headers to every boto3 request that is sent out. Is there a way to manage the connection itself to add these headers?
For boto2, connection.AWSAuthConnection has a method build_base_http_request which has been helpful. I've yet to find an analogous function within the boto3 documentation though.
This is pretty dated but we encountered the same issue, so I'm posting our solution.
I wanted to add custom headers to boto3 for specific requests.
I found this: https://github.com/boto/boto3/issues/2251, and used the event system for adding the header
def _add_header(request, **kwargs):
request.headers.add_header('x-trace-id', 'trace-trace')
print(request.headers) # for debug
some_client = boto3.client(service_name=SERVICE_NAME)
event_system = some_client.meta.events
event_system.register_first('before-sign.EVENT_NAME.*', _add_header)
You can try using a wildcard for all requests:
event_system.register_first('before-sign.*.*', _add_header)
*SERVICE_NAME- you can find all available services here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/index.html
For more information about register a function to a specific event: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html
Answer from #May Yaari is pretty awesome. To the concern raised by #arainchi:
This works, there is no way to pass custom data to event handlers, currently we have to do it in a non-pythonic way using global variables/queues :( I have opened issue ticket with Boto3 developers for this exact case
Actually, we could leverage the python functional programming property: returning a function inside a function to get around:
In the case we want to add a custom value custom_variable to the header, we could do
some_client = boto3.client(service_name=SERVICE_NAME)
event_system = some_client.meta.events
event_system.register_first('before-sign.EVENT_NAME.*', _register_callback(custom_variable))
def _register_callback(custom_variable):
def _add_header(request, **kwargs):
request.headers.add_header('header_name_you_want', custom_variable)
return _add_header
Or a more pythonic way using lambda
some_client = boto3.client(service_name=SERVICE_NAME)
event_system = some_client.meta.events
event_system.register_first('before-sign.EVENT_NAME.*', lambda request, **kwargs: _add_header(request, custom_variable))
def _add_header(request, custom_variable):
request.headers.add_header('header_name_you_want', custom_variable)
What's the correct way to go about logging out information about tests using the velocity framework with Meteor?
I have some mocha tests that I'd like to output some values from, I guess it'd be good if the output could end up in the logs section of the velocity window... but there doesn't seem to be any documentation anywhere?
I haven't seen it documented either.
I don't know how to log messages into the Velocity window, though I don't like the idea of logging into the UI.
What I've done is created a simple Logger object that wraps all of my console.{{method}} calls and prevents logging if process.env.IS_MIRROR. That will only output test framework messages on the terminal. If I need to debug an specific test, I activate logging output for a while on Logger.
This is a terrible hack. It will expose an unprotected method that writes to your DB.
But it works.
I was really annoyed to lack this feature so I digged into the Velocity code to find out that they have a VelocityLogs collection that is globally accessible. But you need to access it from your production, not testing, instance to see it in the web reporter.
So it then took me a good while to get Meteor CORS enabled, but I finally managed - even for Firefox - to create a new route within IronRouter to POST log messages to. (CORS could be nicer with this suggestion - but you really shouldn't expose this anyway.)
You'll need to meteor add http for this.
Place outside of /tests:
if Meteor.isServer
Router.route 'log', ->
if #request.method is 'OPTIONS'
#response.setHeader 'Access-Control-Allow-Origin', '*'
#response.setHeader 'Access-Control-Allow-Methods', 'POST, OPTIONS'
#response.setHeader 'Access-Control-Max-Age', 1000
#response.setHeader 'Access-Control-Allow-Headers', 'origin, x-csrftoken, content-type, accept'
#response.end()
return
if #request.method is 'POST'
logEntry = #request.body
logEntry.level ?= 'unspecified'
logEntry.framework ?= 'log hack'
logEntry.timestamp ?= moment().format("HH:mm:ss.SSS")
_id = VelocityLogs.insert(logEntry)
#response.setHeader 'Access-Control-Allow-Origin', '*'
#response.end(_id)
return
, where: 'server'
Within tests/mocha/lib or similar, as a utility function:
#log = (message, framework, level) ->
HTTP.post "http://localhost:3000/log",
data: { message: message, framework: framework, level: level}
(error) -> console.dir error
For coffee haters: coffeescript.org > TRY NOW > Paste the code to convert > Get your good old JavaScript.