Python 3 basic auth with pinnaclesports API - api

i am trying to grab betting lines with python from pinnaclesports using their API http://www.pinnaclesports.com/api-xml/manual
which requires basic authentication (http://www.pinnaclesports.com/api-xml/manual#authentication):
Authentication
API use HTTP Basic access authentication . Always use HTTPS to access
the API. You need to send HTTP Request header like this:
Authorization: Basic
For example:
Authorization: Basic U03MyOT23YbzMDc6d3c3O1DQ1
import urllib.request, urllib.parse, urllib.error
import socket
import base64
url = 'https://api.pinnaclesports.com/v1//feed?sportid=12&leagueid=6164'
username = "abc"
password = "xyz"
base64 = "Basic: " + base64.b64encode('{}:{}'.format(username,password).encode('utf-8')).decode('ascii')
print (base64)
details = urllib.parse.urlencode({ 'Authorization' : base64 })
details = details.encode('UTF-8')
url = urllib.request.Request(url, details)
url.add_header("User-Agent","Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13")
responseData = urllib.request.urlopen(url).read().decode('utf8', 'ignore')
print (responseData)
Unfortunately i get a http 500 error. Which from my point means either my authentication isn't working properly or their API is not working.
Thanks in advance

As it happens, I don't seem to use the Python version you use, so this has not been tested using your code, but there is an extraneous colon after "Basic" in your base64 string. In my own code, adding this colon after "Basic" indeed yields a http 500 error.
Edit: Code example using Python 2.7 and urllib2:
import urllib2
import base64
def get_leagues():
url = 'https://api.pinnaclesports.com/v1/leagues?sportid=33'
username = "myusername"
password = "mypassword"
b64str = "Basic " + base64.b64encode('{}:{}'.format(username,password).encode('utf-8')).decode('ascii')
headers = {'Content-length' : '0',
'Content-type' : 'application/xml',
'Authorization' : b64str}
req = urllib2.Request(url, headers=headers)
responseData = urllib2.urlopen(req).read()
ofn = 'api_leagues.txt'
with open(ofn, 'w') as ofile:
ofile.write(responseData)

Related

Converting HTML to PDF from https requiring authentication

I've been trying to convert html to pdf from my company's https secured authentication required web.
I tried directly converting it with pdfkit first.
pdfkit.from_url("https://companywebsite.com", 'output.pdf')
However I'm receiving these errors
Error: Authentication Required
Error: Failed to load https://companywebsite.com,
with network status code 204 and http status code 401 - Host requires authentication
So I added options to argument
pdfkit.from_url("https://companywebsite.com", 'output.pdf', options=options)
options = {'username': username,
'password': password}
It's loading forever without any output
My second method was to try creating session with requests
def download(session,username,password):
session.get('https://companywebsite.com', auth=HTTPBasicAuth(username,password),verify=False)
ua = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'
session.headers = {'User-Agent': ua}
payload = {'UserName':username,
'Password':password,
'AuthMethod':'FormsAuthentication'}
session.post('https://companywebsite.com', data = payload, headers = session.headers)
my_html = session.get('https://companywebsite.com/thepageiwant')
my_pdf = open('myfile.html','wb+')
my_pdf.write(my_html.content)
my_pdf.close()
path_wkthmltopdf = 'C:\Program Files\wkhtmltopdf\bin\wkhtmltopdf.exe'
config = pdfkit.configuration(wkhtmltopdf=bytes(path_wkthmltopdf, 'utf8'))
pdfkit.from_file('myfile.html', 'out.pdf')
download(session,username,password)
Could someone help me, I am getting 200 from session.get so its definitely getting the session
Maybe try using selenium to access to that site and snap the screenshot

Login in to Amazon using BeautifulSoup

I am working on a script to scrape some information off Amazon's Prime Now grocery website. However, I am stumbling on the first step in which I am attempting to start a session and login to the page.
I am fairly positive that the issue is in building the 'data' object. There are 10 input's in the html but the data object I have constructed only has 9, with the missing one being the submit button. I am not entirely sure if it is relevant as this is my first time working with BeautifulSoup.
Any help would be greatly appreciated! All of my code is below, with the last if/else statement confirming that it has not worked when I run the code.
import requests
from bs4 import BeautifulSoup
# define URL where login form is located
site = 'https://primenow.amazon.com/ap/signin?clientContext=133-1292951-7489930&openid.return_to=https%3A%2F%2Fprimenow.amazon.com%2Fap-post-redirect%3FsiteState%3DclientContext%253D131-7694496-4754740%252CsourceUrl%253Dhttps%25253A%25252F%25252Fprimenow.amazon.com%25252Fhome%252Csignature%253DIFISh0byLJrJApqlChzLdkc2FCEj3D&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.assoc_handle=amzn_houdini_desktop_us&openid.mode=checkid_setup&marketPlaceId=A1IXFGJ6ITL7J4&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&pageId=amzn_pn_us&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.pape.max_auth_age=3600'
# initiate session
session = requests.Session()
# define session headers
session.headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.61 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Referer': site
}
# get login page
resp = session.get(site)
html = resp.text
# get BeautifulSoup object of the html of the login page
soup = BeautifulSoup(html , 'lxml')
# scrape login page to get all the needed inputs required for login
data = {}
form = soup.find('form')
for field in form.find_all('input'):
try:
data[field['name']] = field['value']
except:
pass
# add username and password to the data for post request
data['email'] = 'my email'
data['password'] = 'my password'
# submit post request with username / password and other needed info
post_resp = session.post(site, data = data)
post_soup = BeautifulSoup(post_resp.content , 'lxml')
if post_soup.find_all('title')[0].text == 'Your Account':
print('Login Successfull')
else:
print('Login Failed')

Cookie authentication error using Python requests

I am trying to POST a request to Kibana using the "/api/console/proxy" path.
I have 3 headers in my request:
es_headers = {
'kbn-version': "5.5.0",
'Content-Type': "application/json",
'Cookie': "session_2=eyJhbGciOi....(long string)"
}
I am using Python requests as following:
session = requests.Session()
r = session.post(url, timeout=15, data=json.dumps(body), headers=es_headers)
From "Postman" it works just fine, but from my Python script I get a [200] response but the content of the response is like this:
'Error encountered = Unable to decrypt session details from cookie. So
clearing it.'
I googled this response but couldn't find any info about it (which is weird ...)
Any help appreciated here
Thanks
Try including the cookies separately from the headers, like this:
import requests
es_headers = {
'kbn-version': "5.5.0",
'Content-Type': "application/json",
}
session = requests.Session()
session.cookies.update({'Cookie': "session_2=eyJhbGciOi....(long string)"})
r = session.post(url, timeout=15, data=json.dumps(body), headers=es_headers)
hope this helps

Binance API Keys

I have set up a read-only API key on Binance to access account information like currency balances but I can't see the JSON data. The string query I put into the URL returns the following error:
{"code":-2014,"msg":"API-key format invalid."}
The URL I am using is this: https://api.binance.com/api/v3/account?X-MBX-APIKEY=**key**&signature=**s-key**
The documentation for Binance API can be found here: https://www.binance.com/restapipub.html. What am I doing wrong ?
Binance's websocket API kinda tricky to use. Also there is no way to use a secret key.
Common usage
Send HTTP POST request with your secret API key as a X-MBX-APIKEY header to https://api.binance.com/api/v1/userDataStream
You will get listen key which should be used for websocket connection. It will be available 1 hour.
{"listenKey": "your listen key here"}
Use it when connecting to Binance's websocket
wss://stream.binance.com:9443/ws/{your listen key here}
Python example
import ssl
from websocket import create_connection
import requests
KEY = 'your-secret-key'
url = 'https://api.binance.com/api/v1/userDataStream'
listen_key = requests.post(url, headers={'X-MBX-APIKEY': KEY})['listenKey']
connection = create_connection('wss://stream.binance.com:9443/ws/{}'.format(KEY),
sslopt={'cert_reqs': ssl.CERT_NONE})
def get_listen_key_by_REST(binance_api_key):
url = 'https://api.binance.com/api/v1/userDataStream'
response = requests.post(url, headers={'X-MBX-APIKEY': binance_api_key}) # ['listenKey']
json = response.json()
return json['listenKey']
print(get_listen_key_by_REST(binance_api_key))
def get_all_orders(symbol, binance_api_key, binance_secret_key):
"""Get all account orders; active, canceled, or filled.
Args: symbol: Symbol name, e.g. `BTCUSDT`.
Returns:
"""
from datetime import datetime, timezone, timedelta
now = datetime.now(timezone.utc)
epoch = datetime(1970, 1, 1, tzinfo=timezone.utc) # use POSIX epoch
posix_timestamp_micros = (now - epoch) // timedelta(microseconds=1)
posix_timestamp_millis = posix_timestamp_micros // 1000 # or `/ 1e3` for float
import hmac, hashlib
queryString = "symbol=" + symbol + "&timestamp=" + str(
posix_timestamp_millis)
signature = hmac.new(binance_secret_key.encode(), queryString.encode(), hashlib.sha256).hexdigest()
url = "https://api.binance.com/api/v3/allOrders"
url = url + f"?{queryString}&signature={signature}"
response = requests.get(url, headers={'X-MBX-APIKEY': binance_api_key})
return response.json()
You put it in the header. Following is tested working PHP example borrowed from jaggedsoft binance PHP library, it's a signed request that will return the account status.
$api_key = "cool_key";
$secret = "awesome_secret";
$opt = [
"http" => [
"method" => "GET",
"header" => "User-Agent: Mozilla/4.0 (compatible; PHP Binance API)\r\nX-MBX-APIKEY: {$api_key}\r\n"
]
];
$context = stream_context_create($opt);
$params['timestamp'] = number_format(microtime(true)*1000,0,'.','');
$query = http_build_query($params, '', '&');
$signature = hash_hmac('sha256', $query, $secret);
$endpoint = "https://api.binance.com/wapi/v3/accountStatus.html?{$query}&signature={$signature}";
$res = json_decode(file_get_contents($endpoint, false, $context), true);
X-MBX-APIKEY should be set as a field in the HTTP header, and not as a HTTP parameter. See this page for more information on HTTP header fields.
However, I tried the same with Excel and could not get it running until now.
Another open question is how to use the secret key.
This worked for me:
base_url="https://api.binance.com"
account_info="/api/v3/account"
url="${base_url}${account_info}"
apikey="your_apikey"
secret="your_secret"
queryString="timestamp=$(date +%s)" #$(python3 binance_time.py) must sync
requestBody=""
signature="$(echo -n "${queryString}${requestBody}" | openssl dgst -sha256 -hmac $secret)"
signature="$(echo $signature | cut -f2 -d" ")"
req=$(curl -H "X-MBX-APIKEY: $apikey" -X GET "$url?$queryString&signature=$signature")
echo $req
You should set the API key in the request header, not as a parameter in the request url. Please provide more information on your request procedure (language, etc.).
If you are based in USA - make sure to switch your base url to https://api.binance.us
_httpClient.DefaultRequestHeaders.Add("X-MBX-APIKEY", "apikey");
_httpClient.DefaultRequestHeaders.Add("SecretKey", "secretkey");
curl -H "X-MBX-APIKEY:your_api_key" -X POST https://api.binance.com/api/v1/userDataStream

How to get github token using username and password

I am developing mobile apps using rhodes. I want to access private repo of github. I am having only username and password.
How to get token of given username and password.
Once you have only login and password you can use them using basic auth. First of all, check if this code shows you json data of desired repo. Username and password must be separated by a colon.
curl -u "user:pwd" https://api.github.com/repos/user/repo
If succeeded you should consider doing this request from code.
import urllib2
import json
from StringIO import StringIO
import base64
username = "user#example.com"
password = "naked_password"
req = urllib2.Request("https://api.github.com/repos/user/repo")
req.add_header("Authorization", "Basic " + base64.urlsafe_b64encode("%s:%s" % (username, password)))
req.add_header("Content-Type", "application/json")
req.add_header("Accept", "application/json")
res = urllib2.urlopen(req)
data = res.read()
repository = json.load(StringIO(data))
You should use oauth instead: http://developer.github.com/v3/oauth/
Github users can create Personal Access Tokens at their application settings. You can use this token as an alternative to username/password in basic http authentication to call the API or to access private repositories on the github website.
Simply use a client that supports basic http authentication. Set the username equal to the token, and the password equal to x-oauth-basic. For example with curl:
curl -u <token>:x-oauth-basic https://api.github.com/user
See also https://developer.github.com/v3/auth/.
Send A POST request to /authorizations
With headers
Content-Type: application/json
Accept: application/json
Authorization: Basic base64encode(<username>:<password>)
But remember to take Two factor Authentication in mind
https://developer.github.com/v3/auth/#working-with-two-factor-authentication
Here You will receive a token which can be used for further request
Follow this guide at help.github.com. It describes how to find your api-token (it's under "Account Settings" > "Account Admin") and configuring git so it uses the token.
Here is the code to use GitHub Basic Authentication in JavaScript
let username = "*******";
let password = "******";
let auth = "Basic " + new Buffer(username + ":" + password).toString("base64");
var options = {
host: 'api.github.com',
path: '/search/repositories?q=google%20maps%20api',
method: 'GET',
headers: {
'user-agent': 'node.js',
"Authorization": auth
}
};
var request = https.request(options, function (res) {
}));