I tried setting up Scrapy Playwright.
Without proxy, cookie applied correctly. But when I use proxy (brightdata), then the cookie is not applied. Did I miss anything?
class ScrapyTest(scrapy.Spider):
name = 'scrapy test'
def start_requests(self):
cookies = {
'cookieconsent_dismissed': 'yes'
}
url = 'https://example.com'
yield scrapy.Request(url, cookies=cookies, meta={"playwright": True}, callback=self.parse)
settings.py
PLAYWRIGHT_LAUNCH_OPTIONS = {
"proxy": {
'server': 'http://zproxy.lum-superproxy.io:22225',
'username': 'lum-customer-user',
'password': 'password'
}
}
COOKIES_ENABLED = True
Related
i want upload a file from local server to host with api and work with requests library but i don't know how do this with aiohttp
code with requests library
url = "https://example.com/upload"
payload={'Domain': 'yourdomain','path': '/'}
files=[
('file',('test.txt',open('your file path','rb'),'text/plain'))
]
headers = {
'Authorization': 'yourtoken'
}
response = requests.request("POST", url,
headers=headers, data=payload, files=files)
print(response.text)
aiohttp code
url = "https://example.com/files/upload"
headers = {
'Authorization': 'yourtoken'
}
fields = {'Domain': 'yourdomain',
'path': '/',
'file':('test.txt',open('your file path','rb'),'text/plain')
}
async with aiohttp.ClientSession() as session :
async with session.post(url,data=fields,headers=headers) as res :
print(res.text)
just return 200 [ok]
I have a local stub server which has a PATCH request. When I call this local endpoint inside cypress, it fails.
My cypress code
Cypress.Commands.add('callLocalAPI', (id: string = '1') => {
const options = {
method: 'PATCH',
url: `${Cypress.env().baseUrl}test/api`,
// failOnStatusCode: false,
retryOnStatusCodeFailure: true,
log: true,
body: {
id,
},
}
// we need cy to visit the current URL so it grabs cookie
// for our request
cy.visit('/')
cy.request(options).then((response) => {
if (!String(response.body.id)) {
console.error(response)
throw Error(`Fail to get response with ${id}`)
}
})
})
then calling this custom cypress command
cy.callLocalAPI('2')
When I access this endpoint using postman (http://localhost:8882/test/api), I get the expected response with a status code of 200 so there is no issues with the mock server.
But when I run the cypress tests, I always get a 404.
Status: 404 - Not Found
Headers: {
"server": "stubby/5.0.0 node/v12.19.0 (darwin x64)",
"date": "Fri, 03 Sep 2021 03:36:49 GMT",
"connection": "keep-alive",
"keep-alive": "timeout=5",
"transfer-encoding": "chunked"
}
Try
const options = {
method: 'PATCH',
url: 'test/api',
...
From Cypress docs - request
If you make a cy.request() after visiting a page, Cypress assumes the url used for the cy.visit() is the host.
cy.visit('http://localhost:8080/app')
cy.request('users/1.json') // url is http://localhost:8080/users/1.json
It may simply be that this line is missing a path separator
url: `${Cypress.env().baseUrl}test/api`
Cypress.env().baseUrl
Note that the value for this string is
taken from the file cypress.env.json (not cypress.json)
{
"baseUrl": "http://localhost:8882/"
}
or overridden in the command line
cypress open --env baseUrl=http://localhost:8882/
in my identityserver4 redirect callback i get cors error.
i enabled cors policy, and used identityserver4 defaultcors class with AllowAll to true.
and i have my domain name in AllowedCorsOrigins in client config.
i tried any way and all articles but it always give cors error.
but if i try to make ajax request in that callback page it dont give me cors error to same page.
i cant understand it.
i have http://ids.example.com
and i have http://example.com
i there anything i can do to solve it. and this configuration works perfect in localhost.
Ids Config:
services.AddSingleton<ICorsPolicyService>(container =>
{
var logger = container.GetRequiredService<ILogger<DefaultCorsPolicyService>>();
return new DefaultCorsPolicyService(logger)
{
AllowAll = true
};
});
Client Config:
new Client
{
ClientId = "js.admin.prod",
ClientName = "Instrumententafel",
RequireClientSecret = false,
RequireConsent = false,
AllowedGrantTypes = GrantTypes.Code,
AccessTokenType = AccessTokenType.Reference,
// AccessTokenLifetime = 299,
RequirePkce = true,
AllowPlainTextPkce = false,
AllowAccessTokensViaBrowser = true,
AllowOfflineAccess = true,
UpdateAccessTokenClaimsOnRefresh = true,
RedirectUris = new List<string>
{
"http://example.com",
"http://example.com/logincb.html",
"http://example.com/silent-renew.html"
},
PostLogoutRedirectUris = new List<string>
{
"http://example.com/",
"http://example.com"
},
AllowedCorsOrigins = new List<string>
{
"http://example.com"
},
AllowedScopes = new List<string>
{
"openid",
"profile",
"email",
"phone",
"role",
"api1.full"
}
}
logincb.html:
<script src="./oidc-client.js"></script>
<script crossorigin>
var mgr = new Oidc.UserManager({
response_mode: "query",
userStore: new Oidc.WebStorageStateStore({ store: window.localStorage })
})
.signinRedirectCallback()
.then(function(user) {
console.log("signin response success", user);
window.location.href = "/";
})
.catch(function(err) {
console.log(err);
});
</script>
and the error is:
Access to XMLHttpRequest at 'http://ids.example.com/connect/token' from origin 'http://example.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
logincb.html?code=396EA4E3D60DFF3620FAB600F3F0DB8AC7DC5CC7922341A3F8E7A3E973E10705&scope=openid profile email phone role offline_access api1.full&state=7fc1f62c72034eddab48a1cac1d261f2&session_state=rQ_HS6gus4BkM7tmL_xgJbWwLhm83nMbks1NCQYymBo.CC889AAAE84EAA20ABD2DFD643BB7797:22 Error: Network Error
at XMLHttpRequest.req.onerror (JsonService.js:179)
i want to get id token from my credentials, i have an url, header, and body from client:
i do code in jupyter like this but did not work, any idea?
import requests
import json
import os
url = "https://xxxxx"
querystring = {"page":"0","limit":"50","sort":"desc"}
payload = "{'key':'value'}"
headers = {
'Content-Type': "application/json"
}
body = {
'email': "{{email}}",
'password': "{{password}}"
}
resp = requests.request("POST", url, data=payload, headers=headers, body=body)
respText = json.loads(resp.text)
respText['data']
I need to keep some cookies from test to test, I'm using Nightwatch and Selenium, I don't know how to get them for the current session o where to store them.
I've tried to create and set an account, but that doesn't seem to work either
"selenium" : {
"start_process" : false,
"server_path" : "",
"log_path" : "",
"host" : "127.0.0.1",
"port" : 4444,
"cli_args" : {
"webdriver.chrome.driver" : "",
"webdriver.ie.driver" : "",
"webdriver.firefox.profile" : "webdriver"
}
},
Any help is apreciated. Thanks
As of v0.9.14, setCookie does not persist cookies between tests. It looks like the client is created anew for each test.
What I've done is create a custom command that sets and caches the cookie value. This works well for maintaining* authentication state between tests since the custom commands are singletons. The actual authentication only happens once, then subsequent calls use setCookie with the cached session value.
Keep in mind that setCookie won't work until the client has navigated to the relevant domain.
How can we set cookies in Nightwatch. I was tried in Supertest it's works.
How to do in nightwatch.
Below is the code to work with superTest
var args = {
data: {
email: "xxxx",
password: "xxxx"
},
headers: {
"Content-Type": "application/json"
}
};
service.post("localhost", args, function(data, response) {
global.Cookies = response.headers['set-cookie'].pop().split(';')[0]
this.meServiceUrl = superTest.agent("localhost/users").get('');
// Set cookie to get saved user session
this.meServiceUrl.cookies = Cookies;
this.meServiceUrl.set('Accept', 'application/json')
.expect('Content-Type', /json/)
.expect(200)
.end(function(err, res) {
usersApiResponse = res.body
usersServiceResponse.push(usersApiResponse)
console.log(usersServiceResponse)
console.log(usersApiResponse.result.length)
});
});