AWS Notebook Instance is working but Lambda is not accepting the input - tensorflow

I developed an ANN tool by using pycharm/tensorflow on my own computer. I uploaded the h5 and json files to Amazon Sagemaker by creating a Notebook Instance. I was finally able to successfully create an endpoint and make it work. The following code in Notebook Instance -Jupyter works:
import json
import boto3
import numpy as np
import io
import sagemaker
from sagemaker.tensorflow.model import TensorFlowModel
client = boto3.client('runtime.sagemaker')
data = np.random.randn(1,6).tolist()
endpoint_name = 'sagemaker-tensorflow-**********'
response = client.invoke_endpoint(EndpointName=endpoint_name, Body=json.dumps(data))
response_body = response['Body']
print(response_body.read())
However, the problem occurs when I created a lambda function and call the endpoint from there. The input should be a row of 6 features -that is a 1-by-6 vector. I enter the following input into lambda {"data":"1,1,1,1,1,1"} and it gives me the following error:
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 20, in lambda_handler
Body=payload)
File "/var/runtime/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 635, in _make_api_call
raise error_class(parsed_response, operation_name)
I think the problem is that the input needs to be 1-by-6 instead of 6-by-1 and I don't know how to do that.

I assume the content type you specified is text/csv, so try out:
{"data": ["1,1,1,1,1,1"]}

Related

Using scrapy 2.7 ModuleNotFoundError: No module named 'scrapy.squeue'

I run my scrapy as a standalone script like this
if __name__ == "__main__":
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
s = get_project_settings()
process = CrawlerProcess(s)
process.crawl(MySpider)
process.start()
my scraper was consuming huge memory so i thought of using these two custom settings.
SCHEDULER_DISK_QUEUE = "scrapy.squeue.PickleFifoDiskQueue"
SCHEDULER_MEMORY_QUEUE = "scrapy.squeue.FifoMemoryQueue"
But after adding these two custom settings and when i run my standalone spider I get error saying.
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/twisted/internet/defer.py", line 1696, in _inlineCallbacks
result = context.run(gen.send, result)
File "/usr/local/lib/python3.9/dist-packages/scrapy/crawler.py", line 118, in crawl
yield self.engine.open_spider(self.spider, start_requests)
ModuleNotFoundError: No module named 'scrapy.squeue'
Any ideas what's the issue with this ?
ModuleNotFoundError: No module named 'scrapy.squeue'
You have a typo:
SCHEDULER_DISK_QUEUE = "scrapy.squeues.PickleFifoDiskQueue"
SCHEDULER_MEMORY_QUEUE = "scrapy.squeues.FifoMemoryQueue"

Loading .txt file from Google Cloud Storage into a Pandas DF

I'm trying to load a .txt file from a GCS bucket into pandas df via pd.read_csv. When I run this code on my local machine (sourcing the .txt file from a local directory), it works perfectly. However, when I try and run the code in a cloud function , accessing the same .txt file but from a GCS bucket, I get a 'TypeError: cannot use a string pattern on a bytes-like object'
The only thing that's different is the fact that I'm accessing the .txt file via the GCS bucket so its a bucket object (Blob) instead of a normal file. Would I need to download the blob as a string or as a file-like object first before doing pd.read_csv? code is below
def stage1_cogs_vfc(data, context):
from google.cloud import storage
import pandas as pd
import dask.dataframe as dd
import io
import numpy as np
start_bucket = 'my_bucket'
storage_client = storage.Client()
source_bucket = storage_client.bucket(start_bucket)
df = pd.DataFrame()
file_path = 'gs://my_bucket/SCE_Var_Fact_Costs.txt'
df = pd.read_csv(file_path,skiprows=12, encoding ='utf-8', error_bad_lines= False, warn_bad_lines= False , header = None ,sep = '\s+|\^+',engine='python')
Traceback (most recent call last):
File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 383, in run_background_function _function_handler.invoke_user_function(event_object) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 217, in invoke_user_function return call_user_function(request_or_event) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 214, in call_user_function event_context.Context(**request_or_event.context)) File "/user_code/main.py", line 20, in stage1_cogs_vfc df = pd.read_csv(file_path,skiprows=12, encoding ='utf-8', error_bad_lines= False, warn_bad_lines= False , header = None ,sep = '\s+|\^+',engine='python') File "/env/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 702, in parser_f return _read(filepath_or_buffer, kwds) File "/env/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 429, in _read parser = TextFileReader(filepath_or_buffer, **kwds) File "/env/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 895, in __init__ self._make_engine(self.engine) File "/env/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1132, in _make_engine self._engine = klass(self.f, **self.options) File "/env/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 2238, in __init__ self.unnamed_cols) = self._infer_columns() File "/env/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 2614, in _infer_columns line = self._buffered_line() File "/env/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 2689, in _buffered_line return self._next_line() File "/env/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 2791, in _next_line next(self.data) File "/env/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 2379, in _read yield pat.split(line.strip()) TypeError: cannot use a string pattern on a bytes-like object
``|
I found a similar situation here.
I also noticed that on the line:
source_bucket = storage_client.bucket(source_bucket)
you are using "source_bucket" for both: your variable name and parameter. I would suggest to change one of those.
However, I think you'd like to see this doc for any further question related to the API itself: Storage Client - Google Cloud Storage API
Building on points from #K_immer is my updated code that includes reading into 'Dask' df...
def stage1_cogs_vfc(data, context):
from google.cloud import storage
import pandas as pd
import dask.dataframe as dd
import io
import numpy as np
import datetime as dt
start_bucket = 'my_bucket'
destination_path = 'gs://my_bucket/ddf-*_cogs_vfc.csv'
storage_client = storage.Client()
bucket = storage_client.get_bucket(start_bucket)
blob = bucket.get_blob('SCE_Var_Fact_Costs.txt')
df0 = pd.DataFrame()
file_path = 'gs://my_bucket/SCE_Var_Fact_Costs.txt'
df0 = dd.read_csv(file_path,skiprows=12, dtype=object ,encoding ='utf-8', error_bad_lines= False, warn_bad_lines= False , header = None ,sep = '\s+|\^+',engine='python')
df7 = df7.compute() # converts dask df to pandas df
# then do your heavy ETL stuff here using pandas...

Keras flask API not giving me output

I am very new to flask. I developed a document classification model using CNN model in Keras in Python3. Below is the code i am using for app.py file in windows machine.
I got the code example from here and improvised it to suit my needs
import os
from flask import jsonify
from flask import request
from flask import Flask
import numpy as np
from keras.models import model_from_json
from keras.models import load_model
from keras.preprocessing.text import Tokenizer, text_to_word_sequence
from keras.preprocessing.sequence import pad_sequences
#star Flask application
app = Flask(__name__)
path = 'C:/Users/user/Model/'
json_file = open(path+'/model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
keras_model_loaded = model_from_json(loaded_model_json)
keras_model_loaded.load_weights(path+'/model.h5')
print('Model loaded...')
def preprocess_text(text,num_max = 1000,max_review_length = 100):
tok = Tokenizer(num_words=num_max)
tok.fit_on_texts(texts)
cnn_texts_seq = tok.texts_to_sequences(texts)
cnn_texts_mat = sequence.pad_sequences(cnn_texts_seq,maxlen=max_review_length)
return cnn_texts_mat
# URL that we'll use to make predictions using get and post
#app.route('/predict',methods=['GET','POST'])
def predict():
try:
text = request.args.get('text')
x = preprocess_text(text)
y = int(np.round(keras_model_loaded.predict(x)))
#print(y)
return jsonify({'prediction': str(y)})
except:
response = jsonify({'error': 'problem predicting'})
response.status_code = 400
return response
if __name__ == "__main__":
port = int(os.environ.get('PORT', 5000))
# Run locally
app.run(host='0.0.0.0', port=port)
In my windows machine i navigate to the path in the console where i have saved app.py file and execute the command py -3.6 app.py
When i go the url http://localhost:5000/predict and type in browser
http://localhost:5000/predict?text=I've had my Fire HD 8 two weeks now and I love it. This tablet is a great value. We are Prime Members and that is where this tablet SHINES.
it does not give me any class as output, but instead i get this as output {"error":"problem predicting"}.
Any help on how to fix this?
Edit: I removed the try except block in the predict function. Below is how predict function looks like
def predict():
text = request.args.get('text')
x = preprocess_text(text)
y = int(np.round(keras_model_loaded.predict(x)))
return jsonify({'prediction': str(y)})
Now i am getting exception. error message is
[2018-05-28 18:33:59,008] ERROR in app: Exception on /predict [GET]
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\_compat.py", line 35, in reraise
raise value
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "app.py", line 59, in predict
x = preprocess_text(text)
File "app.py", line 37, in preprocess_text
tok.fit_on_texts(texts)
NameError: name 'texts' is not defined
127.0.0.1 - - [28/May/2018 18:33:59] "GET /predict?text=I%27ve%20had%20my%20Fire%20HD%208%20two%20weeks%20now%20and%20I%20love%20it.%20This%20tablet%20is%20a%20great%20value.%20We%20are%20Prime%20Members%20and%20that%20is%20where%20this%20tablet%20SHINES. HTTP/1.1" 500 -
Edit2: I have edited code to
def preprocess_text(texts,num_max = 1000,max_review_length = 100):
tok = Tokenizer(num_words=num_max)
tok.fit_on_texts(texts)
cnn_texts_seq = tok.texts_to_sequences(texts)
cnn_texts_mat = pad_sequences(cnn_texts_seq,maxlen=max_review_length)
return cnn_texts_mat
# URL that we'll use to make predictions using get and post
#app.route('/predict',methods=['GET','POST'])
def predict():
text = request.args.get('text')
x = preprocess_text(text)
y = keras_model_loaded.predict(x)
return jsonify({'prediction': str(y)})
and now the error message is
packages\tensorflow\python\framework\ops.py", line 3402, in _as_graph_element_locked
raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("output/Sigmoid:0", shape=(?, 1), dtype=float32) is not an element of this graph.
127.0.0.1 - - [28/May/2018 19:39:11] "GET /predict?text=I%27ve%20had%20my%20Fire%20HD%208%20two%20weeks%20now%20and%20I%20love%20it.%20This%20tablet%20is%20a%20great%20value.%20We%20are%20Prime%20Members%20and%20that%20is%20where%20this%20tablet%20SHINES. HTTP/1.1" 500 -
I am unable to understand and debug this error. Not sure what this means. Can anyone help me understand this error and suggest a solution for this?
Also, i am unable to post the entire error message in stackoverflow as most of the chunk in my question appears to be code.
Thanks!!
Now it is what I guessed. There is a problem when using cross-threads with Flask and Tensorflow. Here is a fix for it:
import tensorflow as tf
# ...
graph = tf.get_default_graph()
def predict():
text = request.args.get('text')
x = preprocess_text(text)
with graph.as_default():
y = int(np.round(keras_model_loaded.predict(x)))
return jsonify({'prediction': str(y)})
by wrapping the prediction to forcefully use the default graph.

ImportError: No module named Image when importing ironpython dll

I have a python package called CoreCode which I have compiled using clr.CompileModules() in IronPython 2.7.5. This generated a file called CoreCode.dll. I then import this dll into my IronPython module by using clr.AddReference(). I know the dll works because I have successfully tested some of the classes as shown below. However, my problem lies with the Base_Slice_Previewer class. This class makes use of Image and ImageDraw from PIL in order to generate and save a bitmap file.
I know the problem doesn't lie with PIL because the package works perfectly well when run in Python 2.7. I'm assuming that this error is coming up because IronPython can't find PIL but I'm not sure how to work around this problem. Any help will be much appreciated.
Code to create the dll
import clr
clr.CompileModules("CoreCode.dll", "CoreCode\AdvancedFileHandlers\ScannerSliceWriter.py", "CoreCode\AdvancedFileHandlers\__init__.py", "CoreCode\MarcamFileHandlers\MTTExport.py", "CoreCode\MarcamFileHandlers\MTTImporter.py", "CoreCode\MarcamFileHandlers\__init__.py", "CoreCode\Visualizer\SlicePreviewMaker.py", "CoreCode\Visualizer\__init__.py", "CoreCode\Timer.py", "CoreCode\__init__.py")
Test for Timer.py
>>> import clr
>>> clr.AddReference('CoreCode.dll')
>>> from CoreCode.Timer import StopWatch
>>> stop_watch = StopWatch()
>>> print stop_watch.__str__()
0:00:00:00 0:00:00:00
>>>
Test for MTTExport.py
>>> from CoreCode.MarcamFileHandlers.MTTExport import MTT_Layer_Exporter
>>> mttlayer = MTT_Layer_Exporter()
>>> in_val = (2**20)+ (2**16) + 2
>>> bytes = mttlayer.write_lf_int(in_val, force_full_size=True)
>>> print "%s = %s" %(bytes, [hex(ord(x)) for x in bytes])
à ◄ ☻ = ['0xe0', '0x0', '0x0', '0x0', '0x0', '0x11', '0x0', '0x2']
>>>
Test for SlicePreviewMaker.py
>>> from CoreCode.Visualizer.SlicePreviewMaker import Base_Slice_Previewer
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "CoreCode\Visualizer\SlicePreviewMaker", line 1, in <module>
ImportError: No module named Image
>>>

py2exe setup.py not working

I have an small code that uses pandas and sqlalchemy and is declared in my main.py as:
import pandas as pd
from sqlalchemy import create_engine
this is my complete setup.py:
from distutils.core import setup
import py2exe
from glob import glob
data_files = [("Microsoft.VC90.CRT", glob(r'C:\Users\Flavio\Documents\Python_dll\*.*'))]
opts = {
"py2exe": {
"packages": ["pandas", "sqlalchemy"]
}
}
setup(
data_files=data_files,
options = opts,
console=['main.py']
)
And I'm using this command in terminal:
python setup.py py2exe
But when I run main.exe it's open the terminal start to execute code and suddenly close window.
when I run over terminal it's the error:
C:\Users\Flavio\Documents\python\python\untitled\dist>main.exe
Please add a valid tradefile date as yyyymmdd: 20150914
Traceback (most recent call last):
File "main.py", line 11, in <module>
File "C:\Users\Flavio\Anaconda3\lib\site-packages\sqlalchemy\engine\__init__.p
y", line 386, in create_engine
return strategy.create(*args, **kwargs)
File "C:\Users\Flavio\Anaconda3\lib\site-packages\sqlalchemy\engine\strategies
.py", line 75, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "C:\Users\Flavio\Anaconda3\lib\site-packages\sqlalchemy\connectors\pyodbc
.py", line 51, in dbapi
return __import__('pyodbc')
ImportError: No module named 'pyodbc'
without know what your program does
I would try the following 1st
open a command window and run your .exe from there
The window will not close and any error messages (if any) will be displayed