I am trying to scrape https://www.mywebsite.com.sg but the following command returns 400 bad request error:
scrapy view https://www.mywebsite.com.sg
If I use:
data=requests.get("https://www.mywebsite.com.sg")
I can get the content of the webpage in data.text and data.content.
However all the xpath operations in my script dotn't work as data.xpath and data.content are both empty.
There seems to be no protection in the webpage as postman software can get result with a simple HTTP GET query.
How do I get the response object to be properly filled?
Related
I am performing a search request in jmeter. So my test plan flow is home then login then product catalogue and then search. I tried to make a post request for search but it failing all the time. I used a CSV file so each time the query is changed. But then I used a get request and used the query variable in the search path like this search?query=${search_input}and then it passed but when i checked the html it is not the correct page. In the html response I also see this
{{noSearchResults.query}}'. But if i put the url on the browser it works fine. Can you please help me with this?
Double check that your ${search_input} variable has the anticipated value using Debug Sampler and View Results Tree listener combination
It might be the case that your ${search_input} variable contains special characters which need to be URL-encoded so you might need to wrap the variable into __urlencode() function like:
search?query=${__urlencode(${search_input})}
JMeter automatically treats responses with status code below 400 as successful, if you need to add an extra layer of check for presence of the results or absence of {{noSearchResults.query}} - use Response Assertion
We have implemented an error handler for Yii 1. Also we have implemented the mail functionality with this as any error occurred an email will be send to us but the problem is we are not getting the current URL on which error is generating. Like one page controller/action can contain many images favicons etc. So if any image is missing then we are getting the image URL which showing 404 from:
$url = Yii::app()->createAbsoluteUrl(Yii::app()->request->url);
But we are not getting current URL not even in $error = Yii::app()->errorHandler->error.
So we are not getting the page in which image is absent. Please let me know if is there any way to get current page URL as I have tried many ways but all they are returning the missing images URL instead of main page URL for which images are missing.
createAbsoluteUrl() expects route as first argument - it may return random results if you provide URL instead of route (like in your code snippet).
If you want absolute URL of current request, you may use combination of getUrl() and getHostInfo():
$url = Yii::app()->request->getHostInfo() . Yii::app()->request->getUrl();
In case of error you can get current page url using Yii::app()->request->requestUri in Yii 1.
I have a GET Request that is returning an XML that contains a TicketName. I have setup the Regular Expression Extractor with Debug Sampler. It is picking up the TicketName as required and is displaying it in the View Results Tree, with the correct name variable name ticketID_g1.
However when I pass that variable to the next GET request the test plan fails with Non HTTP response message: Socket closed.
The thing is that the GET request looks find when I look at the request tab in the Results Tree.
I have changed my regular expression a number of times with each one extracting the TicketName properly but each time I apply it as a variable the GET request fails. However if I copy the request showing in the Results Tree Request Tab and paste it directly into my browser I get the desired result.
I have been through the manuals and on-line tutorials and it appears that I am doing everything right but obviously I am missing something.
The 1st GET Request returns an XML that contains name="2019-05-09-16-59-54cmrpip000613_EDASERVE" needsPrompt
I am using the following regular expression to extract the name for my variable ticketID
name="([^"]+)" needsPrompt - This works
The Results Tree is showing the following response from the Debug Sampler -
ticketID_g1=2019-05-09-16-59-54cmrpip000613_EDASERVE
When I pass the ticketID variable to the next GET request
//localhost:8080/ibi_apps/rs?IBIRS_action=getReport&IBIRS_ticketName=${ticketID_g1}cmrpip000589_EDASERVE&IBIRS_service=defer
The Response tab in the Results Tree for the second GET request is showing that the request is good but is failing.
GET http://localhost:8080/ibi_apps/rs?IBIRS_action=getReport&IBIRS_ticketName=2019-05-09-16-59-54cmrpip000613_EDASERVE&IBIRS_service=defer
What I am expecting is that this second GET will run with the variable and return a report but is throwing the Non HTTP response message: Socket closed error.
You have below variable which is capturing ticket id.
ticketID_g1=2019-05-09-16-59-54cmrpip000613_EDASERVE
But, in the below request you are passing the same which also have repeated content that is
"cmrpip000589_EDASERVE"
Request:-//localhost:8080/ibi_apps/rs?IBIRS_action=getReport&IBIRS_ticketName=${ticketID_g1}cmrpip000589_EDASERVE&IBIRS_service=defer
Please pass the ticketID variable correctly and hopefully it solves the issue. If I am correct you request should look like:-
Request:-//localhost:8080/ibi_apps/rs?IBIRS_action=getReport&IBIRS_ticketName=${ticketID_g1}&IBIRS_service=defer
Via the Fogbugz REST API I try to get all articles with a certain tag. I wrote some code in python to get it but I got "zero" as result. Here is my code:
import requests
...
some code to log in
...
req_params={"cmd": "search", "token": self.token,"q":"tag:\"my_cool_tag\""}
response = requests.get(req_url, data=req_body, headers=req_header, params=req_params, verify=False)
print (response.text)
as response I got:
...cases count="0"...
Is there a way to get all articles with a certain tag in a list via REST-API and how I can achieve this?
I am using FogBugz Version 8.8.49.0.
Try the search with curl or directly in your web browser to check that it works, then see if you can debug your Python.
In a browser I can successfully query FogBugz Online with something like:
https://<domain>.fogbugz.com/api.asp?token=<token>&cmd=search&q=tag:%22<my_tag>%22
Although I entered quotes around my tag, the browser url encoded them to %22. Obviously <domain>, <token> and <my_tag> should be replaced with your own values.
Your basic parameters look OK, but I haven't used Python so am not sure whether escaping the quotes around the tag translates well to the GET request? Maybe try url encoding "my_cool_tag".
I am new to jmeter and am working on putting together a test plan. The hurdle I've encountered is as follows:
First, a POST is made to processForm.aspx
Then, the user is redirected to pleaseWait.aspx
This page either redirects immediately to results.aspx OR loads, with a META REFRESH tag set to refresh in 5 seconds (and this step is repeated).
Now -- I can get this to execute by doing the following:
HTTP Sampler POST to processForm.aspx
Assert Response contains "<something on pleaseWait.aspx>"
While LAST
HTTP Sampler GET to pleaseWait.aspx
Assert Response contains "<something on results.aspx>"
However -- I don't care for this method, because it results in failed assertions (even though things are working as expected). I am sure there must be some other way to do this? Anyone more familiar with JMeter than I?
UPDATE:
Got it going by using Regular Expression Extractor instead of Assertions.
1) Add a User Defined Variables section at Test Plan Root
2) Add a variable to it "LoginWait" and "false"
HTTP Sampler POST to processForm.aspx
RegEx Extract Response Body contains "<something on pleaseWait.aspx>" into LoginWait
While ${LoginWait}
HTTP Sampler GET to pleaseWait.aspx
RegEx Extract Response Body contains "<something on pleaseWait.aspx>" into LoginWait
...
You could try using "follow redirects" on your HTTP Request. It would eliminate the logic you need, and still get you to the page you're going.