i have multiple files which is HTTP request in json file format and every file content as following .
{
"http://testphp.vulnweb.com/search.php": {
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US,en;q=0.5",
"Connection": "close",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0"
},
"method": "POST",
"params": [
"goButton",
"searchFor"
]
}
}%
desired output is raw request as following
POST /search.php HTTP/1.1
Host: testphp.vulnweb.com
Content-Length: 23
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36 Edg/95.0.1020.30
Connection: close
searchFor=a&goButton=go
and BTW the default char in the end of lines is \r\n as it the default format of HTTP raw request and there is extra \r\n in the line before parameters.
i really don`t know if awk could handle this or not but if it how hard it will be??
plz also recommend any online solution to do so if i have to go that way because i struggling in search for reformat from json to raw request.
Thanks
Related
I am using jwt to authorize users on a react app I am working on. As opposed to using localStorage or sessionStorage to store these tokens, I am sending httpOnly, sameSite: strict, secure cookies as shown.
res
.cookie("jwt-access", accessToken, {
httpOnly: true,
sameSite: "strict",
secure: true,
})
.cookie("jwt-refresh", refreshToken, {
httpOnly: true,
sameSite: "strict",
secure: true,
})
.json({ error: false, response: "Authorized" });
My app is successfully able to communicate with my api using the access and refresh tokens while hosted on the same domain as the api server (www.example.com/app, www.example.com/api).
If I understand correctly the react development server is hosted on an http server running on port 3000. Because this server is considered to be running on a different and unsecure domain, these secure cookies are not able to be properly communicated. I thought reconfiguring these cookies to secure: false and sameSite: "none" would have allowed these cookies to be correctly communicated but it hasn't.
Why is it that these updated cookies are unable to be sent/received with the development server and what can I do to properly configure them?
If its any use, here is the request header from each server:
Production server correctly receives cookies
[Symbol(kHeaders)]: {
host: 'localhost:446',
connection: 'close',
'sec-ch-ua': '"Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"',
accept: 'application/json, text/plain, */*',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
referer: 'https://zyae.net/',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
cookie: 'jwt-access=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjYzODJkOWNjYTMwMDM2N2NkOGZlZGZiZiIsImlhdCI6MTY2OTU5ODg4MiwiZXhwIjoxNjY5NTk5NDgyfQ.bYSN7jJwuL2UyLpu3rModl5HAQ7FQ8R7sQUDEOq7Z0g; jwt-refresh=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjYzODJkOWNjYTMwMDM2N2NkOGZlZGZiZiIsImlhdCI6MTY2OTU5ODg4Mn0.gv4BycPhBlEXOc6TLYj6w07diAQ0hOBQT03tTNfF9Hs',
'if-none-match': 'W/"9c-GSJl0bJ1H5bULPZaYoSPi1M3gMw"'
}
Development server
[Symbol(kHeaders)]: {
host: 'localhost:446',
connection: 'close',
pragma: 'no-cache',
'cache-control': 'no-cache',
'sec-ch-ua': '"Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"',
accept: 'application/json, text/plain, */*',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36',
'sec-ch-ua-platform': '"Windows"',
origin: 'http://zyae.net:3000',
'sec-fetch-site': 'cross-site',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
referer: 'http://zyae.net:3000/',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9'
}
Using Directus online (https://projectcode.directus.app install of local or server install). I can successfully create item but data I send is always null.
This is happening running my code
axios.post(
`${endpoint}/items/collection_name`,
{ data: { "field_name": "testthis" } },
{ headers: headers }
);
And using Postman. Result is always
date_created: "2022-05-05T13:55:47"
id: 14
field_name: null
I can get items, search and filter with no issue. What am I missing?
ETA: Postman works with graphql and query
mutation {
create_collection_name_item(data: { field_name: "Hello again!" }) {
field_name
}
}
So why is api getting null values?
ETA 2: Postman api headers
Request
Authorization: Bearer the_very_long_token
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Cache-Control: no-cache
Postman-Token: 45943ddb-5c60-4d45-8887-a05207f9469e
Host: k2g2xa7b.directus.app
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 13
Response
Content-Type: application/json; charset=utf-8
Content-Length: 135
Connection: keep-alive
Date: Mon, 09 May 2022 14:04:54 GMT
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: Content-Range
Cache-Control: no-cache
Content-Security-Policy: script-src 'self' 'unsafe-eval';worker-src 'self' blob:;child-src 'self' blob:;img-src 'self' data: blob: https://cdn.directus.io;media-src 'self' https://cdn.directus.io;connect-src 'self' https://*;default-src 'self';base-uri 'self';block-all-mixed-content;font-src 'self' https: data:;frame-ancestors 'self';object-src 'none';script-src-attr 'none';style-src 'self' https: 'unsafe-inline'
Etag: W/"87-n/ABsgX1O7F1qQ7x0xdJgMV7VDc"
Server: Caddy
Vary: Origin, Cache-Control
X-Powered-By: Directus
X-Cache: Miss from cloudfront
Via: 1.1 dcd16c430149132ea12a5783d54ff114.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: YTO50-P2
X-Amz-Cf-Id: Z19onMWqxvINuZ7iMQIJQeSRFrgW12a4gxZtcWPyubaWQCeaUnfLfA==
What headers do you provide? It seems that Content-Type is not determined automatically, so it may be as simple as including Content-Type: application/json in your headers.
in python:
import requests, json
token = 'your token'
headers = {
'Accept': 'application/json, text/plain, */*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.5',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36',
'Content-Type': 'application/json',
'Authorization': f"Bearer {token}",
}
payload = [
{
"title" : "my first item",
}
]
r= requests.post(
url = "https://cms.uat.<your subdomain>.com/items/<your collection>",
data = json.dumps(payload),
headers = headers
)
print(r.json())
I want to scrape data in a spreadsheet from this site Angel.co startup list i have tried many ways,but it shows an error. used IMPORTXML,IMPORTHTML in spreadsheet it's not working
format : startup name, location, category
Thanks in advance for help.
tried to used this below request method to scrape data however it shows no output.
import requests
URL = 'https://angel.co/social-network-2'
headers = {
"Host": "www.angel.co",
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux armv8l; rv:88.0)
Gecko/20100101 Firefox/88.0",
"Accept": "application/json, text/javascript, */*; q=0.01",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate",
"Referer": "https://angel.co/social-network-2",
"X-Requested-With": "XMLHttpRequest",
"via": "1.1 google"
}
datas = requests.get(URL, headers=headers).json()
import re
for i in datas['data']:
for j in re.findall('class="uni-link">(.*)</a>',i['title']):
print(j)
I am afraid that you will not be able to scrape this webpage.
The problem is that they use cloudflare protection that is specially designed to prevent such automatic bot scraping...
Only suggestion would be to accept this fact and not waste your time...
I am trying to log in to the pge.com website using Google Apps Scripts, with my username and password, so that I can retrieve my electricity consumption details. The need is similar to this thread
I have inspected network calls during login but I cannot figure out how my username and password are passed.
I can see two subsequent calls to the login service.
First request (doesn't seem to carry a payload):
**General**
Request URL: https://apigprd.cloud.pge.com/myaccount/v1/login?ts=1614544850626
Request Method: OPTIONS
Status Code: 200 OK
Remote Address: 130.19.47.120:443
Referrer Policy: strict-origin-when-cross-origin Response Headers
** Response Headers **
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: authorization
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Origin: https://www.pge.com
Content-Encoding: gzip
Content-Length: 59
Content-Type: text/plain;charset=UTF-8
Date: Sun, 28 Feb 2021 20:40:50 GMT
Set-Cookie: TS01ef76cb=015399e11<...>dc1f; Path=/; Domain=.apigprd.cloud.pge.com
X-Tracking-ID
X-Transaction-ID: 00000177b2ec4203-10baebc
** Request Headers **
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Access-Control-Request-Headers: authorization
Access-Control-Request-Method: GET
Connection: keep-alive
Host: apigprd.cloud.pge.com
Origin: https://www.pge.com
Referer: https://www.pge.com/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36
A subsequent request is then made:
** General **
Request URL: https://apigprd.cloud.pge.com/myaccount/v1/login?ts=1614544850626
Request Method: GET
Status Code: 200 OK
Remote Address: 130.19.47.120:443
Referrer Policy: strict-origin-when-cross-origin
** Response Headers **
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers
Access-Control-Allow-Origin: https://www.pge.com
Content-Encoding: gzip
Content-Length: 101
Content-Type: application/json;charset=UTF-8
Date: Sun, 28 Feb 2021 20:40:50 GMT
set-cookie: PF=cuLZ7Vh4y6keKdWEzXVaxD4kzuePT0wcOOyvxZ6V3u0K; Path=/; Domain=.cloud.pge.com; Secure; HttpOnly
set-cookie: PA.ToI.CustomerWeb=eyJhb<...>MifQ..8p6<...>iqw.1BewS--tpmP<...>3fo-rbtoh-<...>Egz-acEIN58H-dO<...>rWR-Q<...>w; Path=/; Domain=pge.com; Secure; HttpOnly
Set-Cookie: TS01ef76cb=015399e11<...>ec2d9; Path=/; Domain=.apigprd.cloud.pge.com
Set-Cookie: TS01d056a5=015399e11<...>2fc17; path=/; domain=pge.com
X-Tracking-ID
X-Transaction-ID: 0000017602681258-9359011
** Request Headers **
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization: Basic bWFy<...>wc2U=
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Cookie: optimizelyEndUserId=oeu1<...>2r0.2464<....>336; oo_inv_percent=0; _gcl_au=1.1.1250622131.1614535637; AMCVS_DF70BB6B55BA62677F000101%40AdobeOrg=1; s_cc=true; _ga=GA1.2.676<...>72.1614535638; _gid=GA1.2.127<...>54.1614535638; _fbp=fb.1.1614535638419.135544411; userLanguageSelected=English; com.silverpop.iMAWebCookie=654ee074-<...>-<...>-0784-4940b13c1dc8; CARE_LANG=English; BIGipServer~Prod~P-itiampingaccess.cloud.pge.com-pool=1796742666.54795.0000; com.silverpop.iMA.session=2764a7b2-1c77-<...>-<...>-b56816c03631; com.silverpop.iMA.page_visit=2144181430:; s_sq=%5B%5BB%5D%5D; PF=dyxdIf<...>wnCDfDTC4EeNVNzFtF3; care_hashed_acct_id=811067339<...>E559C253; PGE_EN=4nx9s6<...>NigxfgqMgNTc/Hs7mt6oL<...>M3oG2lGBv+FNpQhj4U1N8YVv<...>TEUP1x9NU5QJQPcImd/s3FVCFgLy<...>qhrhuv+HQ1J/OANiGah+0exaD2ScPRBd3MQ/+4EqtzOEbpsjjKDKN<...>qe+mOMjkjZ0otbEknx5I7Z1p8w/KxI0fvwT32RBc5U+92j6CxxIDoYwm3KlmCQ==; TS01d056a5=015399e11<...>e65810; TS01ef76cb=015399e11<...>73b336; ADRUM=s=1614544835448&r=https%3A%2F%2Fm.pge.com%2Findex.html%3F35; oo_OODynamicRewrite_weight=0; oo_inv_hit=3; AMCV_DF70BB6B55BA62677F000101%40AdobeOrg=-1303530583%7C<...>Bpb2zX5dvJdYQJzPXImdj0y%7CMCOPTOUT-1614552037s%7CNONE%7CMCAID%7CNONE%7CMCSYNCSOP%7C411-18694%7CvVersion%7C3.3.0; OptanonConsent=isIABGlobal=false&datestamp=Sun+Feb+28+2021+12%3A40%3A37+GMT-0800+(Pacific+Standard+Time)&version=6.1.0&consentId=749474e8-c692-470b-be09-77aa32646232&interactionCount=0&landingPath=NotLandingPage&groups=C0001%3A1%2CC0002%3A1%2CC0003%3A1%2CC0004%3A1%2CC0005%3A1&hosts=&legInt=&AwaitingReconsent=false; _dc_gtm_UA-64722139-1=1; _gat_UA-77056718-1=1; s_tp=2617; s_ppv=PG%2526E%252C%2520Pacific%2520Gas%2520and%2520Electric%2520-%2520Gas%2520and%2520power%2520company%2520for%2520California%2C35%2C35%2C914; s_plt=3.13; utag_main=v_id:0177e9d2<...>78001407000ac8$_sn:2$_ss:1$_st:1614546650462$vapi_domain:pge.com$dc_visit:8$_pn:1%3Bexp-session$ses_id:1614544836893%3Bexp-session$dc_event:2%3Bexp-session$dc_region:us-east-1%3Bexp-session
Host: apigprd.cloud.pge.com
Origin: https://www.pge.com
Referer: https://www.pge.com/
sec-ch-ua: "Chromium";v="88", "Google Chrome";v="88", ";Not A Brand";v="99"
sec-ch-ua-mobile: ?0
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36
How can I use the Google Apps Script to login?
PS I have tried the following:
function loginPge() {
var baseUrl = "https://apigprd.cloud.pge.com/myaccount/v1/login";
var now = Date.now();
var url = baseUrl + '?ts=' + now;
var payload =
{
"username" : "xxx",
"password" : "xxx",
};
var headers =
{
"Accept":"*/*",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
"Access-Control-Request-Headers": "authorization",
"Access-Control-Request-Method": "GET",
"Connection": "keep-alive",
"Origin": "https://www.pge.com",
"Referer": "https://www.pge.com/",
"Sec-Fetch-Dest": "empty",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Site": "same-site",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36"
};
var options =
{
"headers":headers,
"method" : "get",
"payload" : payload,
"followRedirects" : false
};
var login = UrlFetchApp.fetch(url, options);
return;
}
But I am getting an error message
Exception: Request failed for https://apigprd.cloud.pge.com returned
code 400. Truncated server response: {"user": "unknown",
"errorCode":"33"}
I am a little late, but maybe it helps someone else.
The username and password are sent in 'Authorization' header. I figured out the value for this header by inspecting the login page source where it appears as:
<set-header value="Basic base64($username:$password)" name="Authorization"></set-header>
Below is Python code to login and get user details:
import requests
import base64
username = 'test'
password = 'test'
session = requests.Session()
auth = ':'.join([username, password]).encode('utf8')
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:88.0) Gecko/20100101 Firefox/88.0',
'Accept': '*/*',
'Accept-Language': 'en-US,en;q=0.5',
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': 'Basic {}'.format(
base64.b64encode(auth).decode('utf8')
),
'Origin': 'https://www.pge.com',
'Connection': 'keep-alive',
'Referer': 'https://www.pge.com/',
}
resp = session.get(
'https://apigprd.cloud.pge.com/myaccount/v1/login',
headers=headers
)
print(resp.json()) # get the json response
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:88.0) Gecko/20100101 Firefox/88.0',
'Accept': 'application/json, text/javascript, */*; q=0.01',
'Accept-Language': 'en-US,en;q=0.5',
'Content-Type': 'application/json',
'X-RAS-API-USERKEY': 'pgecocmobile',
'cocGUID': 'null',
'Origin': 'https://m.pge.com',
'Connection': 'keep-alive',
'Referer': 'https://m.pge.com/',
}
params = (
('userId', username),
)
resp = session.get(
'https://apigprd.cloud.pge.com/myaccount/v1/cocaccount/secure/account/retrieveMyEnergyAccounts',
params=params, headers=headers
)
print(resp.json()) # get the json response with acc details
Access to XMLHttpRequest from has been blocked by CORS policy: Response to preflight request doesn't pass
access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
My CORS configuration:
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT",
"POST",
"GET",
"DELETE"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": [
"ETag",
"x-amz-server-side-encryption",
"x-amz-request-id",
"x-amz-id-2",
"Content-Length",
"Content-Type",
"Connection",
"Date",
"x-amz-version-id",
"Server"
],
"MaxAgeSeconds": 10
}
]
Request headers:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
Access-Control-Request-Headers: authorization,content-type,x-amz-content-sha256,x-amz-date,x-amz-
user-agent
Access-Control-Request-Method: POST
Cache-Control: no-cache
Connection: keep-alive
Host: s3.amazonaws.com
Origin: http://localhost:8080
Pragma: no-cache
Referer: http://localhost:8080/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/86.0.4240.111 Safari/537.36
It was trying to reach a wrong region as I didn't specify one