I want more sure, than my python selenium chrome will not detected.
So, how i can avoide failed of these tests:
https://bot.sannysoft.com/ CHR_MEMORY FAIL
https://bot.incolumitas.com/ "inconsistentWebWorkerNavigatorPropery": "FAIL",
"inconsistentServiceWorkerNavigatorPropery": "FAIL".
I read and used any helpfull recommendations like this,this or this but yet have no idea, how i can passed tests.
For CHR_MEMORY FAIL i used this:
driver.execute_script("Object.defineProperty(navigator, 'deviceMemory', {get: () => 8})")
or
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source": """
Object.defineProperty(navigator, 'deviceMemory', {
get: () => Number(2)
});
"""
})
But both not work, however Fp-collect info points to "deviceMemory": 8.
The rest of the tests my code passes successfully.
What mean this params from second tests set? WebWorkerNavigatorPropery ServiceWorkerNavigatorPropery How can i try to get around them too?
Any help is much appreciated
sannysoft results IMG
incolumitas results IMG
The whole problem lies in the user agent. try this sprit that i wrote below. Having registered a memory device and a user agent in one script. I struggled with this for a whole week until I tried this combination and it worked for me.
js.executeScript("Object.defineProperty(navigator, 'deviceMemory', {\n" +
" get: () => 8\n" +
" });\n" +
" Object.defineProperty(navigator, 'userAgent', {\n" +
" get: () => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'\n" +
" });");
Related
My Playwright UI tests (which use page, not request) work fine. Only the API tests are failing. For example const response = await request.get('https://reqres.in/api/users/3')
I found that it must be the corporate proxy by leaving the corporate wifi, joining my phone's hotspot, and seeing the tests pass. I've played with VSCode proxy settings but have not found one that fixes the timeout. It also fails when I run the test script directly from the root directory with npx playwright test --config=api.config.ts so I don't think it's VSCode.
The error message (containing my code) is below, the problem comes from the get method. The script times out there, and the code after it is never attempted. All API methods, such as get, post delete time out similarly.
1) [chromium] › api.spec.ts:6:10 › API Testing › Simple API Test - Assert Response Status ========
apiRequestContext.get: read ETIMEDOUT
=========================== logs ===========================
→ GET https://reqres.in/api/users/3
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.5195.19 Safari/537.36
accept: */*
accept-encoding: gzip,deflate,br
============================================================
5 |
6 | test.only('Simple API Test - Assert Response Status', async ({ request }) => {
> 7 | const response = await request.get('https://reqres.in/api/users/3')
| ^
8 | console.log(response)
9 | //expect(response.status()).toBe(200)
10 |
at /Users/USERNAME/playwright-framework/tests/api/api.spec.ts:7:40
Solved. The proxy is set within the Playwright config file, along with the rest of the more common configuration items. If required, username and password can be set here too.
For example:
use: {
headless: false,
viewport: { width: 1280, height: 720 },
actionTimeout: 10000,
screenshot: 'only-on-failure',
ignoreHTTPSErrors: true,
proxy: { server: 'http://yourproxy.com:8080/' },
trace: 'off',
video: 'off'
},
I have used Selenium to get data like item name, price, reviews and so on from the Lazada website. However, it will block me after the first scraping. My question is there any way to solve this? Could you guys give some solution in details. Thankyou
Lazada having high security, for getting data without blocking you must use proxy. you can even get the data using python request try below code
cookies = {
"user": "en"
}
req_headers = {
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"x-requested-with": "XMLHttpRequest",
}
proxies = {"https": "http://000.0.0.0:0000"}
response_data = requests.get(product_url, headers=req_headers, cookies=cookies, proxies=proxies, verify=False)
you can get the product data from response text.
for getting reviews you can use this url :
host = "lazada.sg" // you can use any region here
"https://my.{}/pdp/review/getReviewList?itemId={}&pageSize=100&filter=0&sort=1&pageNo={}".format(host,item_id,page_no)
if you want to use selenium you need to set proxy in selenium
Building an express app with morgan 1.9.0 and browser-refresh 1.7.2.
Morgan seems to be ignoring the skip option. My app is initialized with
const skipFcn = (req, res) => req.path.startsWith('/static/views')
app.use((req, res, next) => {
console.log('test:', skipFcn(req, res))
next()
})
app.use(morgan('combined', {
skip: skipFcn
}))
But in my logs I'm still getting hundreds of hits from browser-refresh:
test: true
::1 - - [14/Oct/2017:10:09:07 +0000] "GET /static/views/project$0.0.0/views/components/service-form.marko.jsHTTP/1.1" 200 3748 "http://localhost:4002/admin/services" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
I use the following code and it worked
app.use(morgan('combined', {skip: (req, res) => {return req.originalUrl.startsWith('/src')}));
You probably want to try it and see if it works on your end.
Just checking in here.
I've been having the same problem. I couldn't figure out what I was doing wrong, and after far longer than it should have been, I went into node_modules/morgan/index.js and on line 80, added the following:
console.log('SKIP FUNCTION:', skip.toString());
This immediately showed the secret location in my project where I was also configuring Morgan, which was clobbering the new place where I was trying to configure Morgan (again).
After deleting my legacy configuration, morgan skip function started working beautifully.
I have the following node route using selenium and chrome driver which is working correctly and returning expected html in the console:
app.get('/google', function (req, res) {
var driver = new webdriver
.Builder()
.forBrowser('chrome')
.build();
driver.get('https://www.google.com')
driver
.manage()
.window()
.setSize(1200, 1024);
driver.wait(webdriver.until.elementLocated({xpath: '//*[#id="lst-ib"]'}));
return driver
.findElement({xpath: '//*[#id="lst-ib"]'})
.sendKeys('stackoverflow' + webdriver.Key.RETURN)
.then((html) => {
return driver
.findElement({xpath: '//*[#id="rso"]/div[1]/div/div/div/div'})
.getAttribute("innerHTML")
})
.then((result) => {
console.log(result)
})
.then(() => {
res
.status(200)
.send('ok')
});
I have also installed the phantom js driver and tested that its working by returning the URL title - it works. When I use the above exact route and replace the chrome with phantomjs I get no results returned. There are no errors - just no print out in my console. The status and result are never sent to the browser so it doesn't appear to be stepping through promise chain.
Any suggestions?
The issue was that there was different html being rendered depending on the user agent. By forcing a user agent I was able to retrieve the results i needed.
Here is the code snippet replaced above to get this working.
.Builder()
// .forBrowser('phantomjs')
.withCapabilities(webdriver.Capabilities.phantomjs()
.set("phantomjs.page.settings.userAgent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"))
.build();
I'm using PhantomJs and Seleno to implement some UI tests. But the problem is whenever I'm trying to find an element on my page it throws an error. "Unknown Command"
code for initializing the servers are like this:
var projectPath =
new DirectoryInfo(Environment.CurrentDirectory).Parent.Parent.Parent.GetDirectories("Foo")
.First()
.FullName;
var loc = ProjectLocation.FromPath(projectPath);
var service =PhantomJSDriverService.CreateDefaultService();
service.Port= 123;
var phantomJs = new PhantomJSDriver(service);
Func<PhantomJSDriver> newFunc = () => phantomJs;
var app = new WebApplication(loc, 123);
Instance.Run(app, c => c
.UsingLoggerFactory(new ConsoleFactory())
.WithRemoteWebDriver(newFunc)
);
it opens the iisexpress using port 123 and PhantomGhost is pointing to the same port.
and showing this error:
Unknown Command - Request => {"headers":{"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8","Accept-Encoding":"gzip, deflate","Accept-Language":"en-US,en;q=0.5","Cache-Control":"max-age=0","Connection":"keep-alive","Cookie":"ASP.NET_SessionId=a2umglrwcaquccg2rar0vzqa; .ASPXAUTH=7CBEDA8FC6170B15E116E77016D2136D4F58C8B73B0B2D54149B96847FE8A26E8D8FA24E41E5F0F0AFFE336D896B53C4628AB5B67B1960CB34727C85B6EF9720F7FF2A792BF1B5ECEECE5429DE212D8B7BA948978F302EF9B3A1040F05902AE92280FF8047D380583465D6CE6C6B103E5286F6FE37E75CFE22910E271BE2BEB4B552124B","Host":"localhost:12346","User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0"},"httpVersion":"1.1","method":"GET","url":"/","urlParsed":{"anchor":"","query":"","file":"","directory":"/","path":"/","relative":"/","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/","queryKey":{},"chunks":[""]}}
I can browse to it using firefox as well it's showing the same error. and obviously selenium can't find the elements and it will show an error.
I find out if you upgrade the PhantomJS to the newest version it will solve the problem