I'm having an annoying issue.
I want to send a request with Cypress, and it is like follows:
cy.request({
method: 'POST',
url: 'http://myUrl.com/a/nice/path',
body: {email: email},
});
And it fails:
The request we sent was:
Method: POST
URL: http://backend-openpay-pe.test.geopagos.com/api/registrations/send-code
Headers: {
"Connection": "keep-alive",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/106.0.5249.119 Safari/537.36",
"accept": "*/*",
"accept-encoding": "gzip, deflate",
"referer": "aNiceReferer"
}
Redirects: [
"308: http://myUrl.com/a/nice/path"
]
And the status code from the response is 404.
When I try to set the host header as follows:
cy.request({
method: 'POST',
headers: {
host: 'myUrl.com',
},
url: 'http://myUrl.com/a/nice/path',
})
The log from Cypress was:
The request we sent was:
Method: POST
URL: http://backend-openpay-pe.test.geopagos.com/api/registrations/send-code
Headers: {
"Connection": "keep-alive",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/106.0.5249.119 Safari/537.36",
"accept": "*/*",
"accept-encoding": "gzip, deflate",
"referer": "http://myUrl.com/a/nice/path"
}
Redirects: [
"308: http://myUrl.com/a/nice/path"
]
And again, a response with 404. What I can see in the previous response, is that Cypress is not sending the host header, even when I manually set it. I tried testing some other headers:
cy.request({
method: 'POST',
headers: {
host: 'myUrl.com',
banana: 'MonkeyLikesBananas',
},
url: 'http://myUrl.com/a/nice/path',
})
And the log was:
The request we sent was:
Method: POST
URL: http://backend-openpay-pe.test.geopagos.com/api/registrations/send-code
Headers: {
"Connection": "keep-alive",
"banana": "MonkeyLikesBananas",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/106.0.5249.119 Safari/537.36",
"accept": "*/*",
"accept-encoding": "gzip, deflate",
"referer": "http://myUrl.com/a/nice/path"
}
Redirects: [
"308: http://myUrl.com/a/nice/path"
]
Again, Cypress seems to correctly send any header I set, except for host, that for some reason is missing there.
Cypress version:
$ npx cypress -v
Cypress package version: 10.9.0
Cypress binary version: 10.9.0
Electron version: 19.0.8
Bundled Node version: 16.14.2
Any help will be welcome.
Thanks!
Related
I'm learning to build a scraper that scrapes search results but previously needs to log in. I read the documentation and this article here. Unfortunately, I'm still stuck. My spider reports the following <403 https://github.com/login>: HTTP status code is not handled or not allowed.
class GitHubSpider(CrawlSpider):
name = "github"
start_urls = [
"https://github.com/search?p=1&q=React+Django&type=Users",
]
rules = (
Rule(
LinkExtractor(restrict_css="a.mr-1"),
callback="parse_engineer",
),
Rule(LinkExtractor(restrict_css=".next_page")),
)
def start_requests(self):
return [
scrapy.FormRequest(
url="https://github.com/login",
formdata={
"login": "scrapy",
"password": "12345",
},
callback=self.parse,
)
]
def parse_engineer(self, response):
yield {
"username": response.css(".vcard-username::text").get().strip(),
}
Edit: Answering on #SuperUser's suggestion.
headers = {
[...]
}
def start_requests(self):
# Do I have access on response here?
token = response.xpath('//form/input[#name="authenticity_token"]/#value').get()
return [
scrapy.FormRequest(
url="https://github.com/login",
formdata={
"login": "scrapy",
"password": "12345",
"authenticity_token": token, # <-------------
},
headers=self.headers,
callback=self.parse,
)
]
Go to settings.py and set 'ROBOTSTXT_OBEY=False'
Replace the default user_agent with another one
Add the request headers from the requested page, you can get it with your browser's devtools.
Just know that they can block your IP, and also block your account.
I suggest you to use PyGithub instead.
Edit:
The request headers:
class GitHubSpider(CrawlSpider):
name = "github"
start_urls = [
"https://github.com/search?p=1&q=React+Django&type=Users",
]
rules = (
Rule(
LinkExtractor(restrict_css="a.mr-1"),
callback="parse_engineer",
),
Rule(LinkExtractor(restrict_css=".next_page")),
)
headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.5",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"DNT": "1",
"Host": "github.com",
"Pragma": "no-cache",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
"Sec-GPC": "1",
"TE": "trailers",
"Upgrade-Insecure-Requests": "1",
"USER_AGENT": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36",
}
def start_requests(self):
return [
scrapy.FormRequest(
url="https://github.com/login",
formdata={
"login": "scrapy",
"password": "12345",
},
headers=self.headers,
callback=self.parse,
)
]
def parse_engineer(self, response):
yield {
"username": response.css(".vcard-username::text").get().strip(),
}
Also notice that you need to get the csrf token:
token = response.xpath('//form/input[#name="authenticity_token"]/#value').get()
Pass the token with the username and password.
formdata={
"login":...,
"password":...,
"authenticity_token": token,
}
i will try to explain my problem clearly.
I have an API who writes something in DynamoDB with a lambda function written in Node.js. When i'm calling it within the AWS console, the API works as expected. I send a body like that:
{
"user-id":"4dz545zd",
"name":"Bush",
"firstname":"Gerard",
}
And that creates the entry within my dynamoDB table. But when i call the same API (freshly deployed) with Postman, i get this error:
{
"statusCode": "400",
"body": "One or more parameter values were invalid: An AttributeValue may not contain an empty string",
"headers": {
"Content-Type": "application/json"
}
}
When i check in cloudwatch why it fails, i see:
Method request body before transformations: [Binary Data]
This is weird, because i sent JSON with the two headers:
Content-Type:application/json
Accept:application/json
And then in cloudwatch, i see that being processed is:
{
"user-id":"",
"name":"",
"firstname":"",
}
Thats explains the error, but i don't understand why when i'm sending it with postman, being not empty, with the json format, it still sends it as "binary" data, and so not being processed by my mapping rule (And so lambda processing it with an empty json):
#set($inputRoot = $input.path('$'))
{
"httpMethod": "POST",
"body": {
"TableName": "user",
"Item": {
"user-id":"$inputRoot.get('user-id')",
"name":"$inputRoot.get('name')",
"firstname":"$inputRoot.get('firstname')",
}
}
}
Thank you in advance !
EDIT: I'm adding the lambda code function
'use strict';
console.log('Function Prep');
const doc = require('dynamodb-doc');
const dynamo = new doc.DynamoDB();
exports.handler = (event, context, callback) => {
const done = (err, res) => callback(null, {
statusCode: err ? '400' : '200',
body: err ? err.message : res,
headers: {
'Content-Type': 'application/json'
},
});
switch (event.httpMethod) {
case 'DELETE':
dynamo.deleteItem(event.body, done);
break;
case 'HEAD':
dynamo.getItem(event.body, done);
break;
case 'GET':
if (event.queryStringParameters !== undefined) {
dynamo.scan({ TableName: event.queryStringParameters.TableName }, done);
}
else {
dynamo.getItem(event.body, done);
}
break;
case 'POST':
dynamo.putItem(event.body, done);
break;
case 'PUT':
dynamo.putItem(event.body, done);
break;
default:
done(new Error(`Unsupported method "${event.httpMethod}"`));
}
};
That's because when testing from AWS Lambda's console, you're sending the JSON you actually expect. But when this is invoked from API Gateway, the event looks different.
You'll have to access the event.body object in order to get your JSON, however, the body is a Stringified JSON, meaning you'll have to first parse it.
You didn't specify what language you're coding in, but if you're using NodeJS you can parse the body like this:
JSON.parse(event.body).
If you're using Python, then you can do this:
json.loads(event["body"])
If you're using any other language, I suggest you look up how to parse a JSON from a given String
That gives what you need.
This is what an event from API Gateway looks like:
{
"path": "/test/hello",
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, lzma, sdch, br",
"Accept-Language": "en-US,en;q=0.8",
"CloudFront-Forwarded-Proto": "https",
"CloudFront-Is-Desktop-Viewer": "true",
"CloudFront-Is-Mobile-Viewer": "false",
"CloudFront-Is-SmartTV-Viewer": "false",
"CloudFront-Is-Tablet-Viewer": "false",
"CloudFront-Viewer-Country": "US",
"Host": "wt6mne2s9k.execute-api.us-west-2.amazonaws.com",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36 OPR/39.0.2256.48",
"Via": "1.1 fb7cca60f0ecd82ce07790c9c5eef16c.cloudfront.net (CloudFront)",
"X-Amz-Cf-Id": "nBsWBOrSHMgnaROZJK1wGCZ9PcRcSpq_oSXZNQwQ10OTZL4cimZo3g==",
"X-Forwarded-For": "192.168.100.1, 192.168.1.1",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https"
},
"pathParameters": {
"proxy": "hello"
},
"requestContext": {
"accountId": "123456789012",
"resourceId": "us4z18",
"stage": "test",
"requestId": "41b45ea3-70b5-11e6-b7bd-69b5aaebc7d9",
"identity": {
"cognitoIdentityPoolId": "",
"accountId": "",
"cognitoIdentityId": "",
"caller": "",
"apiKey": "",
"sourceIp": "192.168.100.1",
"cognitoAuthenticationType": "",
"cognitoAuthenticationProvider": "",
"userArn": "",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36 OPR/39.0.2256.48",
"user": ""
},
"resourcePath": "/{proxy+}",
"httpMethod": "GET",
"apiId": "wt6mne2s9k"
},
"resource": "/{proxy+}",
"httpMethod": "GET",
"queryStringParameters": {
"name": "me"
},
"stageVariables": {
"stageVarName": "stageVarValue"
},
"body": "'{\"user-id\":\"123\",\"name\":\"name\", \"firstname\":\"firstname\"}'"
}
EDIT
After further discussion in the comments, one more problem is that the you're using the DynamoDB API rather than the DocumentClient API. When using the DynamoDB API, you must specify the types of your objects. DocumentClient, on the other hands, abstracts this complexity away.
I have also refactored your code a little bit (only dealing with POST at the moment for the sake of simplicity), so you can make use of async/await
'use strict';
console.log('Function Prep');
const AWS = require('aws-sdk');
const dynamo = new AWS.DynamoDB.DocumentClient();
exports.handler = async (event) => {
switch (event.httpMethod) {
case 'POST':
await dynamo.put({TableName: 'users', Item: JSON.parse(event.body)}).promise();
break;
default:
throw new Error(`Unsupported method "${event.httpMethod}"`);
}
return {
statusCode: 200,
body: JSON.stringify({message: 'Success'})
}
};
Here's the Item in DynamoDB:
And this is my Postman request:
With proper headers:
When creating API Gateway, I checked the box Use Lambda Proxy integration. My API looks like this:
If you reproduce these steps it should just work.
I got the exact same problem, the solution for me was deploying the api to make my changes available through Postman !
Hope it helps, even one year later
you need to deploy your Amazon API Gateway!!! It took me forever to figure this out, than
Deploy API
I encountered the same problem while working with java and I fixed it by just checking the Use Lambda Proxy integration for POST method.
I'm trying to POST the following but I keep getting an error:
"http: error: argument REQUEST_ITEM: "with" is not a valid value"
http POST https://someurl.com fields:='{\"example-api-identifier\":\"String with spaces\"}' Token:randomnumbers
How do I escape these spaces? I'm assuming that's the issue here?
I don't personally know about powershell, but httpie should be fine with spaces without needing the := syntax
$ http POST http://httpbin.org/post example-api-identifier="String with spaces"
yields
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: close
Content-Length: 413
Content-Type: application/json
Date: Sat, 01 Feb 2020 00:25:41 GMT
Server: gunicorn/19.9.0
{
"args": {},
"data": "{\"example-api-identifier\": \"String with spaces\"}",
"files": {},
"form": {},
"headers": {
"Accept": "application/json, */*",
"Accept-Encoding": "gzip, deflate",
"Connection": "keep-alive",
"Content-Length": "48",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "HTTPie/1.0.0"
},
"json": {
"example-api-identifier": "String with spaces"
},
"origin": "127.0.0.1",
"url": "http://httpbin.org/post"
}
i'm trying to change some headers but nothing is working:
var casper = require('casper').create({ //
stepTimeout: 15000,
verbose: false,
logLevel: 'error',
pageSettings: {
loadImages: true,
loadPlugins: true,
userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.364',
customHeaders: {
Connection: 'keep-alive',
Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
} } });
I also tried:
phantom.page.customHeaders = {
"User-Agent" : "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate",
"Connection" : "keep-alive" };
And for a single connection:
this.open('http://localhost/post.php', {
method: 'post',
headers: { 'Accept': 'application/json' }
});
None of them are working or am i doing something wrong?
Thanks
I cannot reproduce your problem. It seems to work for me... Maybe you have an issue with a redirection somewhere, like discussed here.
May I suggest you to do like this guy and try the following code?
casper.on('started', function () {
this.page.customHeaders = {
"User-Agent" : "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate",
"Connection" : "keep-alive"
}
});
I am trying to pass a value to the PHP server side. My Store code is as follows;
Ext.define('MyApp.store.MyArrayStore', {
extend: 'Ext.data.Store',
requires: [
'MyApp.model.MyMOD'
],
config: {
autoLoad: true,
model: 'MyApp.model.MyMOD',
storeId: 'MyArrayStore',
proxy: {
type: 'ajax',
actionMethods: 'POST',
url: 'http://localhost/mm/app/php/res.php',
reader: {
type: 'json'
}
},
listeners: [
{
fn: 'onArraystoreBeforeLoad',
event: 'beforeload'
}
]
},
onArraystoreBeforeLoad: function(store, operation, eOpts) {
this.proxy.extraParams.VALUES1 = "pass some name here";
}
});
PHP Code
<?php
error_reporting(E_ERROR | E_PARSE);
require_once 'conn.php'; // contains the connection
$v = $_POST['VALUES1'];
echo json_encode($v);
?>
What gets returned is null, and not the value that i am passing from the store (which is pass some name here).
How can i correct this ?
UPDATE
Request URL:http://localhost/test/app/php/res.php?_dc=1373343459447
Request Method:POST
Status Code:200 OK
Request Headersview source
Accept:*/*
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:23
Content-Type:application/x-www-form-urlencoded; charset=UTF-8
Host:localhost
Origin:http://localhost
Referer:http://localhost/test/app.html
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36
X-Requested-With:XMLHttpRequest
Query String Parametersview sourceview URL encoded
_dc:1373343459447
Form Dataview sourceview URL encoded
page:1
start:0
limit:25
Response Headersview source
Connection:Keep-Alive
Content-Length:24
Content-Type:text/html
Date:Tue, 09 Jul 2013 04:17:39 GMT
Keep-Alive:timeout=5, max=96
Server:Apache/2.2.14 (Unix) DAV/2 mod_ssl/2.2.14 OpenSSL/0.9.8l PHP/5.3.1 mod_perl/2.0.4 Perl/v5.10.1
X-Powered-By:PHP/5.3.1
You need to change the way you setting extraParams.. in this case i will using
store.getProxy().setExtraParam('VALUES1','pass some name here');
If you need to send more than one parameter then use setExtraParams
var param = { VALUES1: 'param1', VALUES2 : 'param2'};
store.getProxy().setExtraParams(param);
So full Store code
Ext.define('MyApp.store.MyArrayStore', {
extend: 'Ext.data.Store',
requires: [
'MyApp.model.MyMOD'
],
config: {
autoLoad: true,
model: 'MyApp.model.MyMOD',
storeId: 'MyArrayStore',
proxy: {
type: 'ajax',
actionMethods: 'POST',
url: 'http://localhost/mm/app/php/res.php',
reader: {
type: 'json'
}
},
listeners: [
{
fn: 'onArraystoreBeforeLoad',
event: 'beforeload'
}
]
},
onArraystoreBeforeLoad: function(store, operation, eOpts) {
store.getProxy().setExtraParam('VALUES1 ','pass some name here');
}
});
Instead of this.proxy try this.getProxy().
I find the console very useful for this sort of thing. In my own app running Ext.getStore('MyStore').proxy; gets me undefined whereas Ext.getStore('MyStore').getProxy() gets me my proxy.
Use the console, for me it is the most valuable development tool next the API.
Good luck, Brad