python 3 flask how to debug 400 Bad Request: The browser (or proxy) sent a request that this server could not understand - sql

New to Python 3.7 RESTFUl API. In the following POST endpoint, I keep getting the following error message in the POST endpoint, but the GET endpoint works fine.
Exception in _query: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand.
Wonder how can I debug this? There is no clear error message in SQL or Python. Thank you.
# all the imports
import os
import sqlite3
from flask import Flask, request, session, g, redirect, url_for, abort, \
render_template, flash
app = Flask(__name__) # create the application instance :)
app.config.from_object(__name__) # load config from this file , flaskr.py
def init_db():
db = get_db()
with app.open_resource('schema.sql', mode='r') as f:
db.cursor().executescript(f.read())
print('create database successfully')
db.commit()
# there are other codes in between being ignored
#app.route('/', methods=['GET'])
def show_entries():
try:
db = get_db()
# cur = db.execute('select * from mouse_tracking')
check_table="SELECT name FROM sqlite_master WHERE name = 'mouse_tracking' and type = 'table'; "
print(check_table)
cur = db.execute(check_table)
entries = cur.fetchall()
print(type(entries))
print(entries)
except sqlite3.Error as e:
print("Database error: %s" % e)
except Exception as e:
print("Exception in _query: %s" % e)
return render_template('show_entries.html', entries=entries)
#app.route('/add', methods=['POST'])
def add_entry():
try:
print('posting the content')
db = get_db()
db.execute('insert into mouse_tracking (user_id, date, x, y, click) values (?, ?, ?, ?, ?)',
[request.form['user_id'], request.form['date'], request.form['x'], request.form['y']
, request.form['click']])
db.commit()
flash('New entry was successfully posted')
except sqlite3.Error as e:
print("Database error: %s" % e)
except Exception as e:
print("Exception in _query: %s" % e)
return redirect(url_for('show_entries'))
Here is the show_entries.html
{% extends "layout.html" %}
{% block body %}
{% if session.logged_in %}
<form action="{{ url_for('add_entry') }}" method=post class=add-entry>
<dl>
<dt>Title:
<dd><input type=text size=30 name=title>
<dt>Text:
<dd><textarea name=text rows=5 cols=40></textarea>
<dd><input type=submit value=Share>
</dl>
</form>
{% endif %}
<ul class=entries>
{% for entry in entries %}
<li><h2>{{ entry.title }}</h2>{{ entry.text|safe }}
{% else %}
<li><em>Unbelievable. No entries here so far</em>
{% endfor %}
</ul>
{% endblock %}
here is the curl call
curl --location --request POST 'http://127.0.0.1:5000/add' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic cm9vdDpkZWZhdWx0' \
--data-raw '{
"user_id": "1",
"date": "2020-01-01",
"x": "720",
"y": "50",
"click": "0"
}'
Here is the schema.sql
create table if not exists mouse_tracking (
user_id integer not null,
date timestamp not null,
x integer not null,
y integer not null,
click integer not null
);

Related

404 error while activating user in djoser (django)

I created class based view to access uid and token . Here I create one web page which have one button of activate user.
I wish to activate user that created. I am receiving the activation link via email. When I click on the link an activation page is opened. this page has a button to activate user. On click the button a post request is executed to activate user but I am getting 404 error.
Activate_User_App/views.py
import json
from common_strings import ACTIVATION_BASE_ROUTE
from django.views import View
from django.shortcuts import render
from django.http import HttpResponse
import requests
class ActivationView(View):
def get (self, request, uid, token):
print('get called in activate')
return render(request, 'activate.html')
def post (self, request, uid, token):
print('UID : ', uid)
print('Token : ', token)
payload = json.dumps({'uid': uid, 'token': token})
print("payload : " , payload)
protocol = 'https://' if request.is_secure() else 'http://'
web_url = protocol + request.get_host() + '/'
post_url = web_url + ACTIVATION_BASE_ROUTE
print('post_url : ' + post_url)
response = requests.post(post_url, data = payload)
print("response : ", response)
return HttpResponse(response.text)
Activate_User_App/urls.py:
urlpatterns = [re_path(r'^(?P<uid>[\w-]+)/(?P<token>[\w-]+)/$',ActivationView.as_view()),]
settings.py
DJOSER = {
'PASSWORD_RESET_CONFIRM_URL': AUTHENTICATION_BASE_ROUTE + 'password/reset/confirm/{uid}/{token}',
'USERNAME_RESET_CONFIRM_URL': AUTHENTICATION_BASE_ROUTE + 'username/reset/confirm/{uid}/{token}',
'ACTIVATION_URL': ACTIVATION_BASE_ROUTE + '{uid}/{token}/',
'SEND_ACTIVATION_EMAIL': True,
'SEND_CONFIRMATION_EMAIL': True,
'PASSWORD_CHANGED_EMAIL_CONFIRMATION': True,
'USERNAME_CHANGED_EMAIL_CONFIRMATION': True,
'USER_CREATE_PASSWORD_RETYPE': True, #Designed to propote good programming practice
'SET_PASSWORD_RETYPE': True, #Designed to propote good programming practice
'PASSWORD_RESET_CONFIRM_RETYPE': True, #Designed to propote good programming practice
'LOGOUT_ON_PASSWORD_CHANGE' : True, #Note : Logout only works with token based authentication. djoser 2.10
'PASSWORD_RESET_SHOW_EMAIL_NOT_FOUND': False, #Please note that setting this to True will expose information whether an email is registered in the system
'USERNAME_RESET_SHOW_EMAIL_NOT_FOUND': False, #Please note that setting this to True will expose information whether an email is registered in the system
'HIDE_USERS': True,
'token': 'djoser.serializers.TokenSerializer',
'token_create': 'djoser.serializers.TokenCreateSerializer',
'LOGIN_FIELD': 'email', #Default: User.USERNAME_FIELD where User is the model set with Django’s setting AUTH_USER_MODEL.
'SERIALIZERS': {
'user': 'user_profile.serializer.UserSerializer',
},
project/urls.py
urlpatterns = [
path(ACTIVATION_BASE_ROUTE, include('Activate_User_App.urls')),
re_path(r'^api/authentication/', include('djoser.urls')),
re_path(r'^api/authentication/', include('djoser.urls.authtoken')),
common_strings
AUTHENTICATION_BASE_ROUTE = 'authentication/user/'
ACTIVATION_BASE_ROUTE = AUTHENTICATION_BASE_ROUTE + 'activate/'
activate.html
<form action="" method="post">
{% csrf_token %}
<td ><button type="submit">Click Here For Activate Account</button></td>
</form>

How to get the stock option train data with tda-api?

I have registered a tdameritrade developer account and got a api key.
curl -X GET --header "Authorization: " \
"https://api.tdameritrade.com/v1/marketdata/aapl/quotes?apikey=my_tda_api_key"
It can get aapl's data such as below:
{"AAPL": {"assetType":"EQUITY","assetMainType":"EQUITY","cusip":"037833100","symbol":"AAPL",
#omitted all other info
Now i want to get aapl's stock option train with curl and my api key:
curl -X GET --header "Authorization: " "https://api.tdameritrade.com/v1/marketdata/chains? \
apikey=my_tda_api_key& \
symbol=aapl&contractType=all&includeQuotes=false&strategy=single&interval=1&range=all& \
fromDate=2021-8-14&toDate=2023-8-14&expMonth=all&optionType=all"
I always get all HTTP/1.1 400 Bad Request ,tried many different arguments combination in the url ,still can't get normal response,how to fix then?
https://developer.tdameritrade.com/option-chains/apis/get/marketdata/chains#
import requests
import json
td_consumer_key = "xxxxxxxxxxxxxxxxxxxxxx"
base_url = 'https://api.tdameritrade.com/v1/marketdata/chains?\
&symbol={stock_ticker}&contractType={contract_type}\
&strike={strike}&fromDate={date}&toDate={date}'
endpoint = base_url.format(stock_ticker = 'AAPL',
contract_type = 'PUT',
strike = 125,
date='2021-08-27')
page = requests.get(url=endpoint,
params={'apikey' : td_consumer_key})
content = json.loads(page.content)

gcloud deployment-manager undefined properties: UndefinedError: 'properties' is undefined

There is sample code here that uses the following code snippet to handle undefined properties.
{% if properties["cidr"] is defined %}
{% set cidr = properties["cidr"] %}
{% else %}
{% set cidr = "10.10.0.0/16" %}
{% endif %}
github deployment-manager example
However, when I attempt to use similar code:
{% if properties['prod'] is defined %}
{% set machine_type = 'n1-highmem-8' %}
{% set num_nodes = 3 %}
{% else %}
{% set machine_type = 'g1-small' %}
{% set num_nodes = 1 %}
{% endif %}
and deploy as follows:
gcloud deployment-manager deployments create xxx --template kubernetes.jinja --automatic-rollback-on-error --preview
I get the following error:
- code: MANIFEST_EXPANSION_USER_ERROR location: /deployments/xxx/manifests/manifest-1597981115450 message: |-
Manifest expansion encountered the following errors: Exception in kubernetes.jinja
Traceback (most recent call last):
return template.render(resource)
return self.environment.handle_exception(exc_info, True)
reraise(exc_type, exc_value, tb)
File "<template>", line 19, in top-level template code
return obj[argument]
UndefinedError: 'properties' is undefined
Resource: kubernetes.jinja Resource: config
When I deploy with properties it works.
gcloud deployment-manager deployments create xxx --template kubernetes.jinja --automatic-rollback-on-error --preview --properties prod:false
How can I setup a jinja script to have default values and/or detect missing properties without throwing an exception?
UPDATE:
Adding ANY property, even a different one seems to be enough to have the script execute. It appears the properties attribute is COMPLETELY missing by default!
gcloud deployment-manager deployments create xxx --template kubernetes.jinja --automatic-rollback-on-error --preview --properties adsfsdfsdf:Asdfasdfasdf
You need to define properties either in the config that uses the template or when deploying from the command line.
See: https://cloud.google.com/deployment-manager/docs/configuration/templates/define-template-properties
E.g.
gcloud deployment-manager deployments create xxx \
--template=kubernetes.jinja \
--properties=prod:...

How to login to Flask App when using Locust

First time using Locust. I have a Flask App that requires user to login to access most routes.
I cant get Locust to successfully login to my Flask App.
Here is my Locust.py file:
from locust import HttpLocust, TaskSet, task
import re
class UserBehavior(TaskSet):
def on_start(self):
""" on_start is called when a Locust start before any task is scheduled """
self.client.verify = False
self.get_token()
self.login()
def on_stop(self):
""" on_stop is called when the TaskSet is stopping """
self.logout()
def get_token(self):
response = self.client.get("/login")
# Sample string from response:
# <input id="csrf_token" name="csrf_token" type="hidden" value="REDACTED">
self.csrftoken = re.search(' name="csrf_token" .* value="(.+?)"', response.text).group(1)
print(f"DEBUG: self.csrftoken = {self.csrftoken}")
def login(self):
response = self.client.post("/login",
{"email": "REDACTED", "password": "REDACTED"},
headers={"X-CSRFToken": self.csrftoken})
print(f"DEBUG: login response.status_code = {response.status_code}")
def logout(self):
self.client.get("/logout")
#task(5)
def list_domains(self):
response = self.client.get("/domains", headers={"X-CSRFToken": self.csrftoken})
print(f"DEBUG list: response.status_code = {response.status_code}")
class WebsiteUser(HttpLocust):
task_set = UserBehavior
min_wait = 5000
max_wait = 9000
Here is the login function of my Flask App: (with a few debug statements added)
#users.route('/login', methods=['GET', 'POST'])
def login():
if request.method == 'POST': ##DEBUG
logging.debug(f"debug0: inside login func with method == POST") ##DEBUG
if current_user.is_authenticated:
return redirect(url_for('main.home'))
form = LoginForm()
if form.validate_on_submit():
logging.debug(f"debug0.1: inside validate_on_submit") ##DEBUG
user = Users.query.filter_by(email=form.email.data).first()
if user and user.check_password(form.password.data):
login_user(user, remember=form.remember.data)
next_page = request.args.get('next')
if not next_page or url_parse(next_page).netloc != '':
next_page = url_for('main.home')
logging.debug(f"debug1: Login was successful") ##DEBUG
return redirect(next_page)
else:
logging.debug(f"debug2: Login failed") ##DEBUG
flash(f'Login unsuccessful. Please check email and password!', 'danger')
logging.debug(f"debug3: the end of login func") ##DEBUG
return render_template('login.html', title='Login', form=form)
When i run Locust, I get this output:
[2019-09-16 18:03:06,598] Mac-mini-3.local/INFO/locust.main: Starting web monitor at *:8089
[2019-09-16 18:03:06,598] Mac-mini-3.local/INFO/locust.main: Starting Locust 0.11.0
[2019-09-16 18:03:14,069] Mac-mini-3.local/INFO/locust.runners: Hatching and swarming 2 clients at the rate 1 clients/s...
[2019-09-16 18:03:14,138] Mac-mini-3.local/ERROR/stderr: /Users/myuser/.local/share/virtualenvs/locustio-gB1-mbqd/lib/python3.7/site-packages/urllib3/connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)
[2019-09-16 18:03:14,162] Mac-mini-3.local/INFO/stdout: DEBUG: self.csrftoken = REDACTED
[2019-09-16 18:03:14,183] Mac-mini-3.local/INFO/stdout: DEBUG: login response.status_code = 200
[2019-09-16 18:03:14,213] Mac-mini-3.local/INFO/stdout: DEBUG list: response.status_code = 200
[2019-09-16 18:03:15,112] Mac-mini-3.local/INFO/stdout: DEBUG: self.csrftoken = REDACTED
[2019-09-16 18:03:15,137] Mac-mini-3.local/INFO/stdout: DEBUG: login response.status_code = 200
I'm not concerned about the 'InsecureRequestWarning' as this is because I am using a self signed cert and i have disabled verification with 'self.client.verify = False'
The csrftoken looks correct.
From the Flask App itself, I get this output:
DEBUG:user:debug0: inside login func with method == POST
INFO:flask_wtf.csrf:The CSRF token is missing.
DEBUG:user:debug3: the end of login func
DEBUG:user:debug3: the end of login func
DEBUG:user:debug3: the end of login func
DEBUG:user:debug3: the end of login func
So, it's hitting the login function (proven by debug0) but it's not getting into the 'form.validate_on_submit()' conditional.
So far I have spent all day on this, reading articles and trying a lot of things, ie adding the X-CSRFToken headers.
I feel I am missing something fundamental, and would really appreciate some help.
thanks,
WJ
Ok, I solved it, so thought I would share the anwser for anyone else that comes across this. As suggested by #user10788336 its is not a Locust issue.
The issue was that when POSTing to the flask route, the form was not being validated (ie form.validate() was not getting set).
So, I made two changes.
1) changed the POST to have an additional form item which i called "test-mode" and I set the value to "locust-test"
here is the new Locust.py file:
from locust import HttpLocust, TaskSet, task
import re
class UserBehavior(TaskSet):
def on_start(self):
""" on_start is called when a Locust start before any task is scheduled """
self.client.verify = False
self.get_token()
self.login()
def on_stop(self):
""" on_stop is called when the TaskSet is stopping """
self.logout()
def get_token(self):
response = self.client.get("/login")
# Sample string from response:
# <input id="csrf_token" name="csrf_token" type="hidden" value="REDACTED">
self.csrftoken = re.search(' name="csrf_token" .* value="(.+?)"', response.text).group(1)
print(f"DEBUG: self.csrftoken = {self.csrftoken}")
def login(self):
response = self.client.post("/login",
{"email": "REDACTED",
"password": "REDACTED",
"test-mode": "locust-test"
},
headers={"X-CSRFToken": self.csrftoken})
print(f"DEBUG: login response.status_code = {response.status_code}")
def logout(self):
self.client.get("/logout")
#task(5)
def list_domains(self):
response = self.client.get("/domains", headers={"X-CSRFToken": self.csrftoken})
print(f"DEBUG list: response.status_code = {response.status_code}")
class WebsiteUser(HttpLocust):
task_set = UserBehavior
min_wait = 5000
max_wait = 9000
The diff between old and new is:
< {"email": "REDACTED", "password": "REDACTED"},
---
> {"email": "REDACTED",
> "password": "REDACTED",
> "test-mode": "locust-test"
> },
2) I changed my login function the Flask app:
The change is that I don't need the form to be validated, so I skip that ONLY when I detect that I am running in test-mode.
here is the new login function:
#users.route('/login', methods=['GET', 'POST'])
def login():
if current_user.is_authenticated:
return redirect(url_for('main.home'))
form = LoginForm()
# shortcut for Locust testing - need to avoid form.validate() (which is within form.validate_on_submit())
form_is_ok = False
if request.method == 'POST':
if request.form.get('test-mode') == 'locust-test':
form_is_ok = True
else:
form_is_ok = form.validate_on_submit()
if form_is_ok:
logging.debug(f"debug0.1: inside validate_on_submit") # DEBUG
user = Users.query.filter_by(email=form.email.data).first()
if user and user.check_password(form.password.data):
login_user(user, remember=form.remember.data)
next_page = request.args.get('next')
if not next_page or url_parse(next_page).netloc != '':
next_page = url_for('main.home')
logging.debug(f"debug1: Login was successful") # DEBUG
return redirect(next_page)
else:
logging.debug(f"debug2: Login failed") # DEBUG
flash(f'Login unsuccessful. Please check email and password!', 'danger')
logging.debug(f"debug3: the end of login func") # DEBUG
return render_template('login.html', title='Login', form=form)
The diff between old and new is:
< if form.validate_on_submit():
---
>
> # shortcut for Locust testing - need to avoid form.validate() (which is within form.validate_on_submit())
> form_is_ok = False
> if request.method == 'POST':
> if request.form.get('test-mode') == 'locust-test':
> form_is_ok = True
> else:
> form_is_ok = form.validate_on_submit()
>
> if form_is_ok:
I think this is still secure ... thoughts on that?
I might add a config variable, that disables/enables this functionality.
And it works!!!!
BTW, Locust is awesome!
Hope this helps.
cheers,
WJ
Can you share the code for what the form.validate_on_submit method is expecting/validating? Chances are there is a hidden field in the login form that is acting as a dynamic token and/or you're missing a required field. Can you also share the HTML source of the login form?
I'd also be curious to see you add an additional debug statement in your login method that outputs the value of the CSRF token, to make sure it is valid, e.g.
def login(self):
print(f"DEBUG: login csrftoken = {self.csrftoken}")
response = self.client.post("/login",
{"email": "REDACTED",
"password": "REDACTED",
"test-mode": "locust-test"
},
headers={"X-CSRFToken": self.csrftoken})
print(f"DEBUG: login response.status_code = {response.status_code}")
I'd also be curious to see if there are any error messages when you submit the form without a valid CSRF token. Does it return a 200 status code and report the invalid token?
My guess is that it's something to do with the CSRF token not being valid or handled correctly.
Is this an older or customized version of Flask login? I'm not seeing an X-CSRFToken header method in the latest version of Flask login docs at https://flask-login.readthedocs.io/en/latest/

Sylius : An exception has been thrown during the rendering of a template ("") in #SyliusShop\Product\Show\_inventory.html.twig at line 4

i have this exception :
An exception has been thrown during the rendering of a template ("") in #SyliusShop\Product\Show_inventory.html.twig at line 4.
500 Internal Server Error - Twig_Error_Runtime
1 linked Exception: NotFoundHttpException »
and this is the file that throws the exception :
ShopBundle/Resources/views/Product/Show/_inventory.html.twig:
{% if product.simple and not sylius_inventory_is_available(product.firstVariant) %}
{% include '#SyliusShop/Product/Show/_outOfStock.html.twig' %}
{% else %}
{% render(url('sylius_shop_cart_item_add', {'template': '#SyliusShop/Product/Show/_addToCart.html.twig', 'productId': product.id})) %}
{% endif %}
ShopBundle/Resources/config/routing/cart.yml:
sylius_shop_cart_item_add:
path: /add
methods: [GET, POST]
defaults:
_controller: sylius.controller.cart_item:addAction
_sylius:
factory:
method: createForProduct
arguments: [expr:service('sylius.repository.product').find($productId)]
form:
type: sylius_cart_item
options:
product: expr:service('sylius.repository.product').find($productId)
template: $template
redirect:
route: sylius_shop_cart_summary
parameters: {}
flash: sylius.cart.item_add_completed
i make sure that the product id is not null , what can be the problem ?