I can't extract the links with scrapy - scrapy

i need help for extract the links in the page: https://www.remax.pt/comprar-empreendimentos?searchQueryState={%22page%22:1,%22sort%22:{%22fieldToSort%22:%22PublishDate%22,%22order%22:1}}

You could shorten it, you don't have to target from the top element to your target. It's easier to debug then.
response.css('div.developments-search-details-component a::attr(href)').get()
You can change this to Xpath if you prefer that. But usually when you try to target an element and it returns null or empty list it's because of a typo or because that element is dynamically rendered after page load.
To debug I'll usually start at a higher element in the tree and see if that exists.
In this case you could try:
response.css('div.developments-search-details-component').get()
first and see if that works.

Related

How to find an element containing #nbsp; in text?

I've an element with html -
<h3>App-1 Playground Login</h3>
I want to identify it with entire text - App-1 Playground Login, but causing issues to identify it. Please help how this element can be identified.
Please use the below xpath. I have already tested that and it is working fine. In the second argument of the translate method you need to type "ALT+0160" and in the third argument you will have to put just a normal space.
//h3[contains(translate(text(),' ',' ' ), 'App-1 Playground Login')]
One of the way to select your title could be :
//h3[text()= concat('App-1 Playground',codepoints-to-string(160),'Login')]
Works fine on http://xpather.com/

How to write a XPath for the text one4

I want to use XPath to locate a link behind a text.
I want to use XPath to locate a link behind a text. For example, locate "one4" by "what10". You can only use the text message "what10", but you can't use it in any other way, because the information on this page will change. I want to get is the "one4" link node.
<body>
<p>
so
<br>what1 one
<br>what2two
<br>what11one4
<br>what3three
<br>what4one1
<br>what5two2
<br>what6three3
<br>what7one3
<br>what8two3
<br>what9three3
<br>what10one4
<br>just return
<br></p>
</body>
For some special reasons, what I want to pass is that the text of what10 is positioned to one4.
Please help me.
You can use below line
WebElement loginLink = driver.findElement(By.linkText("one4"));
Selenium doesn't supports xpath-2.0 but uses xpath-1.0
The element which you are trying to refer i.e. which contains the text what10 is a Text Node and Selenium can't use it as a reference. So finding the node with text as one4 with reference to the text what10 won't be possible. As an alternative if the desired node is always the last but one node you can use the following solution:
xpath:
driver.findElement(By.xpath("//body/p//a[position()=last()-1]"));
Update
As per #MosheSlavin counter question here is the snapshot to demonstrate that the XPath works perfecto:

Concise Xpath to simulate finding an element regardless of page structure? (selenium)

If you're visually looking at a webpage and there is something clickable and unique on the page, you'll just click it. Without thinking about the page structure.
I'm interested to see what the most concise xpath is that could be constructed to simulate this in a versatile manner.
For example, target the "I'm feeling Lucky" button on the Google homepage:
//*[contains(#*, 'Lucky')]
The above works. But would fail in the element contained Lucky as inner text, or if the wrong case was specified. As such, our xpath needs to cater for any sensitivity and also look for the given string matching inner-text as well.
How could the above xpath be expressed in the most concise yet encompassing structure?
There is nothing thats very generic and executing such xpaths could be costly also at times.
"//*[contains(#*, 'Lucky')] | //*[contains(text(), 'Lucky')]"
Above is one xpath you can combine to get some results. You start specifying which nodes you don't to examine or ones which you want to examine
"//*[contains(#*, 'Lucky')] | //*[contains(text(), 'Lucky')][not(self::script|self::td)]"
And you can keep improving it
It's not possible to create a versatile XPath to accurately/reliability locate an element by text.
Why?
Because the text evaluated by an XPath is not necessary rendered in the page.
Because there's a hight chance to end-up with multiple matches since each ancestor also contains the expected text.
But mainly because there's too many rules/specific cases to consider.
But if I had to create one, then I'd start with this one:
"(html/body//*[not(self::script or self::style)][contains(concat(#value, normalize-space()), 'MyText')])[last()]"
Get all the descendants of the <body>
html/body//*
except <script> and <style>
[not(self::script or self::style)]
where the value attribute or normalize html contains 'MyText'
[contains(concat(#value, normalize-space()), 'MyText')]
then returns the last and deepest match
[last()]

Finding text on page with Selenium 2

How can I check whether a given text string is present on the current page using Selenium?
The code is this:
def elem = driver.findElement(By.xpath("//*[contains(.,'search_text')]"));
if (elem == null) println("The text is not found on the page!");
If your searching the whole page for some text , then providing an xpath or selector to find an element is not necessary. The following code might help..
Assert.assertEquals(driver.getPageSource().contains("text_to_search"), true);
For some reason, certain elements don't seem to respond to the "generic" search listed in the other answer. At least not in Selenium2library under Robot Framework which is where I needed this incantation to find the particular element:
xpath=//script[contains(#src, 'super-sekret-url.example.com')]
A simpler (but probably less efficient) alternative to XPaths is to just get all the visible text in the page body like so:
def pageText = browser.findElement(By.tagName("body")).getText();
Then if you're using JUnit or something, you can use an assertion to check that the string you are searching for is contained in it.
assertThat("Text not found on page", pageText, containsString(searchText));
Using an XPath is perhaps more efficient, but this way is simpler to understand for those unfamiliar with it. Also, an AssertionError generated by assertThat will include the text that does exist on the page, which may be desirable for debugging as anybody looking at the logs can clearly see what text is on the page if what we are looking for isn't.

selenium getXpathCount

HI there
selenium.getXpathCount does not find element, any one hoas any idea ? Here is my code:
if (existArtist){
int result = selenium.getXpathCount("//*[#id='chugger-results']/div[1]/ul/li").intValue();
if (result>0){
//DO THIS
Either you have a broken DOM (Do a W3C Validation and see if you have any unclosed tags) or your XPath is looking for an element that doesn't exist.
We would need to see the entire HTML of the page to be able to answer your question (more visibility of your test code would be useful too)