HLS Stream OneTimePassword (GetHLSToken from 511nj.org) - api

For research purposes, we analyzed and extracted large-scale traffic data from 511nj cameras. Starting from 2020, it adds an OTP(one-time password) to the video link, which is requested by the website API: https://511nj.org/api/client/camera/getHlsToken?Id=2&rnd=202205281151, as observed from the network traffic. However, the website itself gets a response of 200 with a valid token, while accessing it by myself gets a response of 401, which indicates an unauthorized request. What makes me confused is why I can get the camera feed with the OTP through the website, but can't access the api. Is there a solution?
Failed attempt:
requests.get('https://511nj.org/api/client/camera/getHlsToken?Id=2&rnd='+datetime.datetime.now().strftime('%Y%m%d%H%M')).json()
20220601 Update: Modification has been made based on #Brad 's comment. However, the error message now turns into "An error has occurred". Please help!
def GetOTP():
import requests
import json
import datetime
Now = datetime.datetime.now()
DT = Now.strftime('%Y%m%d%I%M%S')
DT24 = Now.strftime('%Y%m%d%H%M')
Enigma = dict(zip(map(str,range(0,10)),('2','1','*','3','4','5','6','7','8','9')))
cookies = {
'ReloadVersion': '112',
'_ga': 'GA1.2.335422978.1653768841',
'_gid': 'GA1.2.721252567.1653768841',
}
headers = {
'authority': '511nj.org',
'accept': 'application/json, text/plain, */*',
'accept-language': 'en-US,en;q=0.9',
'referer': 'https://511nj.org/Scripts/Directives/HLSPlayer.html?v=V223',
'responsetype': "".join([Enigma[key] for key in DT[4:6]])+'$'.join([Enigma[key] for key in DT[6:8]])+'$'.join([Enigma[key] for key in DT[:4]])+'#'+Enigma[DT[9]]+"!".join([Enigma[key] for key in DT[10:12]])+"!"+DT[-2:]+"#PK",
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="102", "Google Chrome";v="102"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"macOS"',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36',
}
params = {
'Id': '2',
'rnd': DT24,
}
Response = requests.get('https://511nj.org/api/client/camera/getHlsToken?Id=2&rnd='+DT24, headers=headers,cookies=cookies,params=params).json()
print("".join([Enigma[key] for key in DT[4:6]])+'$'+''.join([Enigma[key] for key in DT[6:8]])+'$'+''.join([Enigma[key] for key in DT[:4]])+'#'+Enigma[DT[9]]+"!"+''.join([Enigma[key] for key in DT[10:12]])+"!"+DT[-2:]+"#PK")
print(params['rnd'])
print(Response)
return Response['Data']

Related

Express sameSite cookies not accessable to react dev server

I am using jwt to authorize users on a react app I am working on. As opposed to using localStorage or sessionStorage to store these tokens, I am sending httpOnly, sameSite: strict, secure cookies as shown.
res
.cookie("jwt-access", accessToken, {
httpOnly: true,
sameSite: "strict",
secure: true,
})
.cookie("jwt-refresh", refreshToken, {
httpOnly: true,
sameSite: "strict",
secure: true,
})
.json({ error: false, response: "Authorized" });
My app is successfully able to communicate with my api using the access and refresh tokens while hosted on the same domain as the api server (www.example.com/app, www.example.com/api).
If I understand correctly the react development server is hosted on an http server running on port 3000. Because this server is considered to be running on a different and unsecure domain, these secure cookies are not able to be properly communicated. I thought reconfiguring these cookies to secure: false and sameSite: "none" would have allowed these cookies to be correctly communicated but it hasn't.
Why is it that these updated cookies are unable to be sent/received with the development server and what can I do to properly configure them?
If its any use, here is the request header from each server:
Production server correctly receives cookies
[Symbol(kHeaders)]: {
host: 'localhost:446',
connection: 'close',
'sec-ch-ua': '"Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"',
accept: 'application/json, text/plain, */*',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
referer: 'https://zyae.net/',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
cookie: 'jwt-access=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjYzODJkOWNjYTMwMDM2N2NkOGZlZGZiZiIsImlhdCI6MTY2OTU5ODg4MiwiZXhwIjoxNjY5NTk5NDgyfQ.bYSN7jJwuL2UyLpu3rModl5HAQ7FQ8R7sQUDEOq7Z0g; jwt-refresh=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjYzODJkOWNjYTMwMDM2N2NkOGZlZGZiZiIsImlhdCI6MTY2OTU5ODg4Mn0.gv4BycPhBlEXOc6TLYj6w07diAQ0hOBQT03tTNfF9Hs',
'if-none-match': 'W/"9c-GSJl0bJ1H5bULPZaYoSPi1M3gMw"'
}
Development server
[Symbol(kHeaders)]: {
host: 'localhost:446',
connection: 'close',
pragma: 'no-cache',
'cache-control': 'no-cache',
'sec-ch-ua': '"Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"',
accept: 'application/json, text/plain, */*',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
'sec-ch-ua-platform': '"Windows"',
origin: 'http://zyae.net:3000',
'sec-fetch-site': 'cross-site',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
referer: 'http://zyae.net:3000/',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9'
}

Scraping angellist start-up data

I want to scrape data in a spreadsheet from this site Angel.co startup list i have tried many ways,but it shows an error. used IMPORTXML,IMPORTHTML in spreadsheet it's not working
format : startup name, location, category
Thanks in advance for help.
tried to used this below request method to scrape data however it shows no output.
import requests
URL = 'https://angel.co/social-network-2'
headers = {
"Host": "www.angel.co",
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux armv8l; rv:88.0)
Gecko/20100101 Firefox/88.0",
"Accept": "application/json, text/javascript, */*; q=0.01",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate",
"Referer": "https://angel.co/social-network-2",
"X-Requested-With": "XMLHttpRequest",
"via": "1.1 google"
}
datas = requests.get(URL, headers=headers).json()
import re
for i in datas['data']:
for j in re.findall('class="uni-link">(.*)</a>',i['title']):
print(j)
I am afraid that you will not be able to scrape this webpage.
The problem is that they use cloudflare protection that is specially designed to prevent such automatic bot scraping...
Only suggestion would be to accept this fact and not waste your time...

Scraping Lazada data

I have used Selenium to get data like item name, price, reviews and so on from the Lazada website. However, it will block me after the first scraping. My question is there any way to solve this? Could you guys give some solution in details. Thankyou
Lazada having high security, for getting data without blocking you must use proxy. you can even get the data using python request try below code
cookies = {
"user": "en"
}
req_headers = {
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"x-requested-with": "XMLHttpRequest",
}
proxies = {"https": "http://000.0.0.0:0000"}
response_data = requests.get(product_url, headers=req_headers, cookies=cookies, proxies=proxies, verify=False)
you can get the product data from response text.
for getting reviews you can use this url :
host = "lazada.sg" // you can use any region here
"https://my.{}/pdp/review/getReviewList?itemId={}&pageSize=100&filter=0&sort=1&pageNo={}".format(host,item_id,page_no)
if you want to use selenium you need to set proxy in selenium

Axios post request failing due to CORS but the same request using ajax is getting no issues

I am getting following error wile doing axios post request.
But when I use ajax request there is no issue:
request has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Ajax Request:
Axios request:
let payload = {
type: ["a", "b"],
category: ["a", "b"],
category: ["a", "b"],
accountNumber: "123"
};
var apiURL = this.$apiBaseURL + "/Product/ProductDetails";
$.ajax({
url: apiURL,
type: "POST",
data: { payload },
xhrFields: {
withCredentials: true
},
success: function (result) {
console.log(JSON.stringify(result));
}
});
this.$http.post(apiURL,payload,{withCredentials: true})
**UPDATE 1 **
I am still facing the same issue. Here I will share the request header in both ajax and axios request
AJAX Working code and request header :
{
var apiURL = this.$apiBaseURL + "/Request/MediaUpload";
$.ajax({
method: 'post',
processData: false,
contentType: false,
cache: false,
data: fileformData,
enctype: 'multipart/form-data',
url: apiURL,
xhrFields: {
withCredentials: true
}
});
}
Request header:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 7610
Content-Type: multipart/form-data; boundary=----
WebKitFormBoundaryAjc8HwVPaRtQ5Iby
Host: localhost:62148
Origin: http://localhost:8989
Referer: http://localhost:8989/
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36
AXIOS failing code and header
var apiURL = this.$apiBaseURL + "/Request/MediaUpload";
var self=this;
let config={
headers:{
'Content-Type': 'multipart/form-data'
}
}
this.$http.post(apiURL, { withCredentials: true },fileformData,
config)
Request Headers:
Provisional headers are shown
Accept: application/json, text/plain, */*
Content-Type: application/json;charset=UTF-8
Referer: http://localhost:8989/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36
Here is my web api config where I have enabled cors
string origin = "http://localhost:8989";
EnableCorsAttribute cors = new EnableCorsAttribute(origin, "*", "*", "*");
cors.SupportsCredentials = true;
config.EnableCors(cors);
// Web API routes
config.MapHttpAttributeRoutes();
config.Routes.MapHttpRoute(
name: "DefaultApi",
routeTemplate: "api/{controller}/{id}",
defaults: new { id = RouteParameter.Optional }
);
**UPDATE 2 **
The CORS configuration at server side is done correctly thats why I am able to make the call successfully via AJAX.
Is this a known AXIOS issue which occurs only when we enable the windows authentication?
This issue arises because jQuery sends the request as application/x-www-form-urlencoded by default and does not send preflight request whereas axios sends as application/json and sends a preflight request i.e. OPTIONS before sending actual request.
One way it could work is by setting Content-type header to 'Content-Type': 'application/x-www-form-urlencoded' in axios.
You will have to change the settings on server side to get around this issue.
Add this in your web.config file:
<system.webServer>
<handlers>
<remove name="ExtensionlessUrlHandler-Integrated-4.0" />
<remove name="OPTIONSVerbHandler" />
<add name="ExtensionlessUrlHandler-Integrated-4.0" path="*." verb="*" type="System.Web.Handlers.TransferRequestHandler" preCondition="integratedMode,runtimeVersionv4.0" />
</handlers>
</system.webServer>
Using default ajax won't send any OPTIONS request, but using axios does. This will enable accepting OPTIONS requests.

Error when connecting to wunderground API

I am trying to scrape wind speed data for different UK weather stations using the site wunderground. I assume they have an API, I just have a hard to connecting to it.
Here's the XHR link I use:
https://api.weather.com/v1/location/EGNV:9:GB/observations/historical.json?apiKey=6532d6454b8aa370768e63d6ba5a832e&units=e&startDate=20150101&endDate=20150131
This is the data I would like. The table in the bottom for wind speed:
https://www.wunderground.com/history/monthly/gb/darlington/EGNV/date/2015-1
My code, pretty simple: I first load the headers, my function get_data gets me the response in json format.
In my main I append the data to a dataframe and print it.
from bs4 import BeautifulSoup
import pandas as pd
import requests
import urllib
from urllib.request import urlopen
headers = {
':authority': 'api.weather.com',
#':path': '/v1/location/EGNV:9:GB/observations/historical.json?apiKey=6532d6454b8aa370768e63d6ba5a832e&units=e&startDate=20150101&endDate=20150131',
':scheme': 'https',
'accept': 'application/json, text/plain, */*',
'accept-encoding' : 'gzip, deflate, br',
'accept-language': 'en-GB,en;q=0.9,en-US;q=0.8,da;q=0.7',
'origin': 'https://www.wunderground.com',
#'apiKey': '6532d6454b8aa370768e63d6ba5a832e',
'referer': 'https://www.wunderground.com/history/monthly/gb/darlington/EGNV/date/2015-1',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'cross-site',
'user-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36'
}
def get_data(response):
df = response.json()
return df
if __name__ == "__main__":
date = pd.datetime.now().strftime("%d-%m-%Y")
api_key = "6532d6454b8aa370768e63d6ba5a832e"
start_date = "20150101"
end_date = "20150131"
urls = [
"https://api.weather.com/v1/location/EGNV:9:GB/observations/historical.json?apiKey="+ api_key +"&units=e&startDate="+start_date+"&endDate="+end_date+""
]
df = pd.DataFrame()
for url in urls:
res = requests.get(url, headers= headers)
data = get_data(res)
df = df.append(data)
print(df)
The error I get:
SSLError: HTTPSConnectionPool(host='api.weather.com', port=443): Max retries exceeded with url: /v1/location/EGNV:9:GB/observations/historical.json?apiKey=6532d6454b8aa370768e63d6ba5a832e&units=e&startDate=20150101&endDate=20150131
(Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
Update:
Even when without trying to connect to the API, but by scraping the page using BS4, I still get denied access. Not sure why, and how they can detect my scraper?
I solved it.
If I add verify = False in my requests.get() I manage to go around the error.