I have developed a python plugin which is capable of sending log file in json format
in mm code i have used requests.post(url, data={})
what will be the api structure that catch this data and will be available for
send anywhere with GET request
If you are fairly new to web programming I would suggest using a lightweight framework like Flask. With it you can define custom paths that your server accepts requests on as follows:
from flask import Flask
from flask import request, jsonify
app = Flask(__name__)
log_file = None
#app.route("/api/logfile", methods=['GET', 'POST'])
def post_logfile():
if request.method == 'GET':
if log_file is not None:
return "Log file not instantiated yet", 404
else:
return jsonify(log_file)
elif request.method == 'POST':
log_file = request.form
if log_file is not None:
# log_file variable will have all the information
# from the JSON log file
return "Ok"
else:
return "No data provided", 400
if __name__ == "__main__":
app.run(port=9000)
As you can see, we have a global variable log_file which will be used to store the JSON logfile data, and a function that accepts both POST and GET requests and acts accordingly. If a GET request is sent, it checks if log_file variable is assigned. If so, it returns the log file as a JSON file else it return a 404 error. If a POST request is sent it checks if it has the log file and stores in the log_file variable, making it useful for all subsequent GET requests.
The URL used are:
localhost:9000/api/logfile
And you only need to change the method of the request(e.g. POST or GET)
Related
I'm generating a PDF file using pdfkit. As I do it in memory, I use the following code:
result = pdfkit.from_string(html, False)
result is bytes type, then I want Flask to send it to the client to be downloaded:
response = make_response(result)
response.headers.set('Content-Type', 'application/pdf')
response.headers.set(
'Content-Disposition', 'attachment', filename= 'report.pdf')
return response
I take it on the client side (JavaScript, React) and try to save:
FileDownload(response.data, 'myfile.pdf')
But the file is always empty with weird title somewhere inside. I think the problem is with encoding but I can't figure out what exactly to do.
This worked for me and produced a valid pdf.
import pdfkit
app = Flask(__name__)
#app.route("/")
def index():
pdf = pdfkit.from_string('Hello!', False)
response = make_response(pdf)
response.headers.set('Content-Type', 'application/pdf')
response.headers.set('Content-Disposition', 'inline', filename='report.pdf')
return response
if __name__ == "__main__":
app.run(host="127.0.0.1", port=8080, debug=True)
I'm building a Flask test predictor using AllenNLP.
I'm passing 'passage' and 'question' from a .json file to the predictor.
However, when I pass the json file using curl, it doesn't return a response. Is there a special return in Flask to get it?
Code looks like:
from allennlp.predictors.predictor import Predictor as AllenNLPPredictor
from flask import Flask
from flask import request
app = Flask(__name__)
#app.route("/", methods=['GET','POST'])
def hello():
return "<h1>Test app!</h1>"
class PythonPredictor:
def __init__(self, config):
self.predictor = AllenNLPPredictor.from_path(
"https://storage.googleapis.com/allennlp-public-models/bidaf-elmo-model-2018.11.30-charpad.tar.gz"
)
def predict(self, payload):
if request.method == "POST":
prediction = self.predictor.predict(
passage=payload["passage"], question=payload["question"]
)
return prediction["best_span_str"]
Curl command looks like:
curl http://127.0.0.1:5000 -X POST -H "Content-Type: application/json" -d #sample.json
Unless I've misunderstood (I'm guessing you're asking how to obtain the JSON submission in your route, and return the result) it sounds like you need to do something like:
p = PythonPredictor()
#app.route("/", methods=['POST'])
def hello():
data = request.get_json()
result = p.predict(data)
return result
This effectively runs the data in your sample.json through your PythonPredictor.predict method, and returns that prediction to the client.
Notice this code creates the instance p outside the route function, so that the NLP model is only loaded when your flask app starts (not on every request). However it looks like this may re-download that file, unless AllenNLPPredictor.from_path does some caching, so it would probably be advisable to manually download that file to your own storage first, and load from there in the PythonPredictor.__init__ function.
Let me know if any of this needs clarification, or I've missunderstood.
I am fetching google photos from my account using Google Photo API. Now there is a need for me to execute that php file via terminal, but the problem is that I can't authenticate with Google API in doing so. Is there a way to do this, and if yes, then how shall it be done?
Yes, it is possible, you need an interactive login for the first authentication but then you can save the token and refresh it automatically as required.
I have implemented this class in Python to do just that.
from requests.adapters import HTTPAdapter
from requests_oauthlib import OAuth2Session
from pathlib import Path
from urllib3.util.retry import Retry
from typing import List, Optional
from json import load, dump, JSONDecodeError
import logging
log = logging.getLogger(__name__)
# OAuth endpoints given in the Google API documentation
authorization_base_url = "https://accounts.google.com/o/oauth2/v2/auth"
token_uri = "https://www.googleapis.com/oauth2/v4/token"
class Authorize:
def __init__(
self, scope: List[str], token_file: Path,
secrets_file: Path, max_retries: int = 5
):
""" A very simple class to handle Google API authorization flow
for the requests library. Includes saving the token and automatic
token refresh.
Args:
scope: list of the scopes for which permission will be granted
token_file: full path of a file in which the user token will be
placed. After first use the previous token will also be read in from
this file
secrets_file: full path of the client secrets file obtained from
Google Api Console
"""
self.max_retries = max_retries
self.scope: List[str] = scope
self.token_file: Path = token_file
self.session = None
self.token = None
try:
with secrets_file.open('r') as stream:
all_json = load(stream)
secrets = all_json['installed']
self.client_id = secrets['client_id']
self.client_secret = secrets['client_secret']
self.redirect_uri = secrets['redirect_uris'][0]
self.token_uri = secrets['token_uri']
self.extra = {
'client_id': self.client_id,
'client_secret': self.client_secret}
except (JSONDecodeError, IOError):
print('missing or bad secrets file: {}'.format(secrets_file))
exit(1)
def load_token(self) -> Optional[str]:
try:
with self.token_file.open('r') as stream:
token = load(stream)
except (JSONDecodeError, IOError):
return None
return token
def save_token(self, token: str):
with self.token_file.open('w') as stream:
dump(token, stream)
self.token_file.chmod(0o600)
def authorize(self):
""" Initiates OAuth2 authentication and authorization flow
"""
token = self.load_token()
if token:
self.session = OAuth2Session(self.client_id, token=token,
auto_refresh_url=self.token_uri,
auto_refresh_kwargs=self.extra,
token_updater=self.save_token)
else:
self.session = OAuth2Session(self.client_id, scope=self.scope,
redirect_uri=self.redirect_uri,
auto_refresh_url=self.token_uri,
auto_refresh_kwargs=self.extra,
token_updater=self.save_token)
# Redirect user to Google for authorization
authorization_url, _ = self.session.authorization_url(
authorization_base_url,
access_type="offline",
prompt="select_account")
print('Please go here and authorize,', authorization_url)
# Get the authorization verifier code from the callback url
response_code = input('Paste the response token here:')
# Fetch the access token
self.token = self.session.fetch_token(
self.token_uri, client_secret=self.client_secret,
code=response_code)
self.save_token(self.token)
# note we want retries on POST as well, need to review this once we
# start to do methods that write to Google Photos
retries = Retry(total=self.max_retries,
backoff_factor=0.1,
status_forcelist=[500, 502, 503, 504],
method_whitelist=frozenset(['GET', 'POST']),
raise_on_status=False)
self.session.mount('https://', HTTPAdapter(max_retries=retries))
In my flask app (python 2.7), I am trying to trigger the access token to refresh via the refresh token whenever it expires with the #jwt.expired_token_loader decorator. Both the access token and refresh token are stored in a cookie (JWT_TOKEN_LOCATION = 'cookies').
I am utilizing the same code provided in the documentation (https://flask-jwt-extended.readthedocs.io/en/latest/tokens_in_cookies.html) to do this and I am able to successfully generate a new access token. However, upon generation of a new access token the identity claim of the access token is equal to None (get_raw_jwt()). No matter what I do to the refresh token or the access token whenever I print the jwt_claims or attempt to grab the current user with current_user = get_jwt_identity() it returns the identity as None. It is important for me to know which user is submitting queries to the Neural Net so I can properly keep track of which queries were submitted by what users (one-to-many relationship).
I have tried troubleshooting (decoding) the refresh_token and I ran into a separate issue: When I try to decode the refresh_token, with decode_token(), I get a long traceback that ends in InvalidSignatureError: Signature verification failed. I took the refresh_token and ran it through https://jwt.io and it decodes the token. I can see in the decoded token that the "identity" claim is providing me the user's identity, but it tells me that the token is not verified. However, once I check the secret base64 encoded box on the screen the signature becomes verified and the signature portion of the jwt changed along with it. I attempt to decode this modified jwt that https://jwt.io provided me with the decode_token function and it still provides me the same error: InvalidSignatureError: Signature verification failed.
I have spent hours reading everything google provides me on flask-jet-extended and PyJWT and I cannot figure out how to fix it. I have tried modifying my JWT_SECRET_KEY configuration to different strings and even encoding it with base64 and none of this solves the issues. I have turned on and off the JWT_COOKIE_CSRF_PROTECT configuration. I have turned on and off JWT_ACCESS_COOKIE_PATH and JWT_REFRESH_COOKIE_PATH.I have tried to decode both the refresh_token_cookie and CSRF_refresh_token cookie. I have attempted to run decode_token() with the refresh_token_cookie and CSRF_refresh_token cookie while providing the csrf_value=request.cookies.get('csrftoken') argument. I have tried using the decode() function from jwt directly (
from jwt import decode).
I just don't know what else to do and cannot find any additional online resources. Any help is much appreciated!!
My next step is either to move my authentication system to flask-jet-simple or PyJWT. I really want to use JWT to authenticate my users. I do not know how to combine JWT with flask-login or if this is even possible. I can't manage to find any resources online where someone has utilized flask-login with JWT. I did find a fairly recent repo called flask-jwt-login that I might try to use if I can't figure this out. Ultimately I would like to stay with flask-jwt-extended. I have other parts of this web app I need to focus on and want to get his part squared away.
Anyways, here is my code, the workflow starts on the /login page. This will redirect you to the /NN page. Once the access token expires, if you try to reload the /NN page it will reroute itself to the /token/refresh page. Once it refreshes the token, it will return back to the /NN page.
Please let me know if I need to upload any additional files.
P.S. This is my first post on stack overflow so forgive me for any formatting issues.
application.py
from flask import url_for,render_template, redirect,request, jsonify,flash,\
make_response, session
from flask_jwt_extended import (create_access_token, create_refresh_token,
jwt_required, get_jwt_identity, get_jwt_claims,get_current_user,
set_access_cookies,set_refresh_cookies,
unset_jwt_cookies, get_raw_jwt, jwt_refresh_token_required,decode_token)
from jwt import decode
from forms import RegisterForm, LoginForm, NNForm
from models import Users
from website import app,db,jwt
#ToDo When the token expires I get an HTTP status code of 401 I can use expired_token_loader refresh token.
#app.route('/token/refresh', methods=['GET','POST'])
#jwt_refresh_token_required
#jwt.expired_token_loader
def refresh():
#Create the new access token
ref_token = request.cookies.get('refresh_token_cookie')
csrftoken = request.cookies.get('csrftoken')
decode_ref_token = decode_token(ref_token)
current_user = get_jwt_identity()
print('ref_token:', ref_token)
print('current_user:', current_user, get_raw_jwt())
access_token = create_access_token(identity=current_user)
#Set the JWT access cookie in the response
print('from refresh():', request.url)
response = make_response(redirect(request.url))
set_access_cookies(response,access_token)
#set_refresh_cookies()
return response
#app.route('/token/remove', methods=['POST'])
def logout():
#ToDo Still need to build the logout page.
response = make_response(redirect(url_for('logout_page')))
unset_jwt_cookies(response)
return response
#app.route('/register/', methods=['GET','POST'])
def register_page():
form = RegisterForm(request.form)
print( request.method, form.validate_on_submit())
if request.method == "POST" and form.validate_on_submit():
user = Users(form.first_name.data, form.last_name.data, \
form.email.data, form.password.data, form.organization.data)
user.save_to_db()
flash("Thanks for Registering. Please login")
return redirect(url_for("NN_page"))
return render_template('register.html',form=form)
#app.route('/login/', methods=['GET','POST'])
def login_page():
form = LoginForm(request.form)
print(request.method, request.form)
if request.method == "POST":
#This checks if the user is in the db and returns the user obj.
user = form.validate_on_submit()
if user:
access_token = create_access_token(identity=user.email, fresh=True)
refresh_token = create_refresh_token(identity=user.email)
response = make_response(redirect(url_for('NN_page')))
set_access_cookies(response, access_token)
set_refresh_cookies(response, refresh_token)
#response.headers['Authorization'] = 'Bearer {}'.format(access_token)
print(response)
return response
#return jsonify({'access_token':access_token})
#return redirect((url_for("NN_page")))
return render_template('login_page.html', form=form)
#jwt.invalid_token_loader #This allows me to stop people who have not logged in yet.
def missing_JWT_token(msg):
print('from missing_JWT_token:', msg)
return redirect(url_for('login_page'))
# return "The site being accessed requires a valid JWT to view." \
# "Error: {}".format(msg)
#app.route('/NN/', methods=['GET','POST'])
#jwt_required
def NN_page():
jwt_claims = get_raw_jwt()
print(jwt_claims)
print('cookie keys:', request.cookies.get('refresh_token_cookie'))
user = get_jwt_identity()
print('User:',user)
form = NNForm(request.form, headers=request.headers)
print(request.form, form.validate_on_submit())
if request.method == "POST" and form.validate_on_submit():
return redirect((url_for("success_NN_submission")))
return render_template('NN_page.html', form=form)
config.py
import os
from datetime import timedelta
from base64 import b64encode
secret_key = os.urandom(24)
jwt_secret_key = b64encode('I_love_my_smokes!')
class BaseConfig(object):
SECRET_KEY = secret_key
SQLALCHEMY_DATABASE_URI = 'sqlite:///Protein_NN.db'
SQLALCHEMY_TRACK_MODIFICATION = False
#JWT_SECRET_KEY = jwt_secret_key
JWT_ACCESS_TOKEN_EXPIRES = timedelta(minutes=10)
JWT_REFRESH_TOKEN_EXPIRES = timedelta(minutes=1)
JWT_TOKEN_LOCATION = 'cookies'
#JWT_ACCESS_COOKIE_PATH = '/NN/'
#JWT_REFRESH_COOKIE_PATH ='/token/refresh'
JWT_COOKIE_CSRF_PROTECT = False
SESSION_COOKIE_SECURE = True
class DevelopmentConfig(BaseConfig):
DEBUG = True
JWT_ACCESS_TOKEN_EXPIRES = timedelta(seconds=5)
SESSION_COOKIE_SECURE = False
#PROPOGATE_EXCEPTION = True
#EMAIL SETTINGS
MAIL_SERVER = 'smtp.gmail.com'
MAIL_PORT = 465
#MAIL_PORT = 587 # This is for TLS
MAIL_USE_TLS = False
MAIL_USE_SSL = True
#MAIL_USERNAME = os.environ['EMAIL_USER']
#MAIL_PASSWORD = os.environ['EMAIL_PASSWORD']
#BOOTSTRAP_SERVE_LOCAL = True
This is what get_raw_jwt() returns after the access token has been refreshed by the refresh token.
{'user_claims': {}, u'jti': u'9fb01b6c-619b-4fe6-91d3-73f8609f2f61',
u'exp': 1547022397, u'iat': 1547022392, u'fresh': False,
u'type': u'access', u'nbf': 1547022392, u'identity': None}
As you can see the identity claim is equal to None.
Here is the traceback I see:
Traceback (most recent call last):
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/flask/app.py", line 1719, in handle_user_exception
return handler(e)
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/flask_jwt_extended/jwt_manager.py", line 93, in handle_expired_error
return self._expired_token_callback()
File "/Users/Danny/Documents/Codes/Ellington/NN App/website/application.py", line 43, in refresh
print('current_user:', current_user, decode('eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiIzNzVlNWExMy1mNjRiLTQxNmItOTY0ZC0wMDg5ODI4NGY2NGQiLCJleHAiOjE1NDcwMTk5ODUsImlhdCI6MTU0NzAxOTkyNSwidHlwZSI6InJlZnJlc2giLCJuYmYiOjE1NDcwMTk5MjUsImlkZW50aXR5IjoiZGFubnlAbWUuY29tIn0.LVEj6As2Uh_xgTbjm94b0M6mJeD0YLkf9KpgNKTZJOw'))
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/jwt/api_jwt.py", line 92, in decode
jwt, key=key, algorithms=algorithms, options=options, **kwargs
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/jwt/api_jws.py", line 156, in decode
key, algorithms)
File "/Users/Danny/.virtualenvs/Raghav_NN_WebApp/lib/python2.7/site-packages/jwt/api_jws.py", line 223, in _verify_signature
raise InvalidSignatureError('Signature verification failed')
InvalidSignatureError: Signature verification failed
Having those two separate decorators on your refresh function won’t work the way you want it to. The expired loader decorator isn’t going to have a current user set because the jwt isn’t valid when that callback function is called.
Instead try breaking out the refresh code into a helper function used by both decorators independently:
def refresh_token(username):
# return flask response from here
#jwt.expired_token_loader
def handle_expired_token():
# get username here from raw jwt
username = 'todo'
return refresh_token(username)
#app.route(‘/refresh)
#jwt_refresh_token_required
def refresh_endpoint():
username = get_current_identity()
return refresh_token(username)
You could also use a custom decorator instead of the jwt_required decorator and achieve a similar thing. Some examples of that are discussed here: https://gitter.im/flask-jwt-extended/Lobby?at=5c1a9b37c35a3002474ddf3d
I want to write a UserAgentMiddleware for scrapy,
the docs says:
Middleware that allows spiders to override the default user agent.
In order for a spider to override the default user agent, its user_agent attribute must be set.
docs:
https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.useragent
But there is no a example,I have no ideas how to write it.
Any suggestions?
You look at it in install scrapy path
/Users/tarun.lalwani/.virtualenvs/project/lib/python3.6/site-packages/scrapy/downloadermiddlewares/useragent.py
"""Set User-Agent header per spider or use a default value from settings"""
from scrapy import signals
class UserAgentMiddleware(object):
"""This middleware allows spiders to override the user_agent"""
def __init__(self, user_agent='Scrapy'):
self.user_agent = user_agent
#classmethod
def from_crawler(cls, crawler):
o = cls(crawler.settings['USER_AGENT'])
crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
return o
def spider_opened(self, spider):
self.user_agent = getattr(spider, 'user_agent', self.user_agent)
def process_request(self, request, spider):
if self.user_agent:
request.headers.setdefault(b'User-Agent', self.user_agent)
You can see a below example for setting Random user agent
https://github.com/alecxe/scrapy-fake-useragent/blob/master/scrapy_fake_useragent/middleware.py
First visit some website and get some of the newest user agents. Then in your standard middleware do something like this. This is the same place you would setup your own proxy settings. Grab a random UA from the text file, and put it in the headers. This is sloppy to show an example you would want to import random at the top and also make sure to closer useragents.txt when you are done with it. I would probably just load them into a list at the top of the document.
class GdataDownloaderMiddleware(object):
#classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_request(self, request, spider):
# Called for each request that goes through the downloader
# middleware.
user_agents = open('useragents.txt', 'r')
user_agents = user_agents.readlines()
import random
user_agent = random.choice(user_agents)
request.headers.setdefault(b'User-Agent', user_agent)
# Must either:
# - return None: continue processing this request
# - or return a Response object
# - or return a Request object
# - or raise IgnoreRequest: process_exception() methods of
# installed downloader middleware will be called
return None
def process_response(self, request, response, spider):
# Called with the response returned from the downloader.
# Must either;
# - return a Response object
# - return a Request object
# - or raise IgnoreRequest
return response
def process_exception(self, request, exception, spider):
# Called when a download handler or a process_request()
# (from other downloader middleware) raises an exception.
# Must either:
# - return None: continue processing this exception
# - return a Response object: stops process_exception() chain
# - return a Request object: stops process_exception() chain
pass
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)