I want to scrape facebook comments from fb live or post,..is there any way to get this? If anyone know please help me.
I am using this but it does not working.
Code:
comments = driver.find_elements_by_css_selector('.UFICommentBody span')
print("Comment found " +str(len(comments)))
for x in range(len(comments)):
print (comments[x])
Output:
<selenium.webdriver.remote.webelement.WebElement (session="0ea2c4e211c05d504536a1bef2259260", element="a0a4c59f-9c84-4a5c-855d-3ba51cea249a")>
<selenium.webdriver.remote.webelement.WebElement (session="0ea2c4e211c05d504536a1bef2259260", element="504c6ef2-e9fe-42b7-9f68-dcee5e7dbfde")>
<selenium.webdriver.remote.webelement.WebElement (session="0ea2c4e211c05d504536a1bef2259260", element="06d25f07-3a20-4783-98d7-f9c0ae01c230")>
<selenium.webdriver.remote.webelement.WebElement (session="0ea2c4e211c05d504536a1bef2259260", element="5e6b1e94-fee8-4636-9d9e-fd992c945c19")>
if you want to print text from your list then you can use for loop to retrieve each element through your list and then use .text with each object to print assoociated text.
print("Comment found " +str(len(comments)))
for x in range(len(comments)):
print (comments[x].text)
or
print("Comment found " +str(len(comments)))
for x in range(len(comments)):
print (x.text)
To get the text from the elements, you need to do this (assuming you are using python):
print (comments[x].text)
Related
I try to get the xpath for the following button on instagram making an automate unfollowing soft. enter image description here
I found it just like this:
driver.find_element_by_xpath('//div[#class="qF0y9 Igw0E rBNOH YBx95 ybXk5 _4EzTm soMvl "]').click()
But i want to itterate over all ,,Following" Buttons , but like this is stuck at the first one!
This is my Code:
fBody = driver.find_element_by_xpath("//div[#class='isgrP']")
for i in range(1, 1500):
driver.find_element_by_xpath('//div[#class=" qF0y9 Igw0E rBNOH YBx95 ybXk5 _4EzTm soMvl "]').click()
#driver.find_element_by_xpath("//*[text()='Following']").click()
print("Am apasat follow")
sleep(5)
driver.find_element_by_xpath('//button[#class="aOOlW -Cab_ "]').click()
sleep(5)
driver.execute_script('arguments[0].scrollTop = arguments[0].scrollTop + arguments[0].offsetHeight;', fBody)
print("Ma bag la somn 1 min")
sleep(2)
print("salut")
Selenium does Not "like" empty or white spaces in the attributes.
I suggest using a CSS selector and using *= in order to find text contains:
driver.find_element_by_CSS('//div[class*="qF0y9"][class*="Igw0E"]').click();
Avoid using white or empty spaces and, underscores (_) and hyphens (-) for the element's attributes.
I think the classes on the elements change as yours do not match with mine. Here is a more generic XPath that matches the "following" button.
//div//button[div[text()='Following']
When using this in a test I found it instantly failing unless I surrounded it with an explicit wait condition.
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("//div//button[div[text()='Following']"))).click();
Ill post my example when Instagram stops giving me connectivity issues.
I am still handling with a view problems. I am finally able to load the whole page with a view window.scrollBy(..) commands...
The problem I am facing now is that I would like to filter all headlines out of a text. The text I would like to screen is shown with this command:
[Code]
main = driver.find_element_by_id("mrt-node-quoteNewsStream-0-Stream")
print(main.text)
That works well and all the result is shown. Within these results I am now wanna filter, as mentioned above all headlines. This should work wit the following code:
articles = main.find_elements_by_tag_name("mrt-node-quoteNewsStream-0-Stream") # li
for mrt-node-quoteNewsStream-0-Stream in articles:
header = article.find_element_by_class_name("M(0)")
print(header.text)
Unfortunately it shows me the following syntax error message:
File "", line 7
for mrt-node-quoteNewsStream-0-Stream in articles:
^
SyntaxError: can't assign to operator
Line 7 is the one with the following one:
for mrt-node-quoteNewsStream-0-Stream in articles:
Any help is highly appreciated. Thanks
This line of code returns a list of elements:
articles = main.find_elements_by_tag_name("mrt-node-quoteNewsStream-0-Stream")
So articles is a list of WebElement and each of them looks like:
<selenium.webdriver.remote.webelement.WebElement (session="04a9fac269c3a9cb724cc72769aed4e0", element="1b8ee8d0-b26a-4c67-be10-615286a4d427")>
As per your second set of code trials, mrt-node-quoteNewsStream-0-Stream is the tag_name and you must not look for the tag_name within the element.
I have this simple code to click the first paper link Organoid Modeling of the Tumor Immune Microenvironment. at this link.
title_wait = WebDriverWait(driver,5).until(
EC.presence_of_element_located((By.CLASS_NAME, "docsum-title")))
print('found title '+str(title))
element = WebDriverWait(driver,5).until(EC.element_to_be_clickable((By.LINK_TEXT, str(title)))).click();
print('found link to click')
My code will sometimes work, but around 50% of the time it just skips right over the .click() and goes to the print below. Any help would be appreciated!
Actually, the problem is not with the wait time. U have made a small mistake in this line: element = WebDriverWait(driver,5).until(EC.element_to_be_clickable((By.LINK_TEXT, str(title)))).click();
First of all, title is a variable of type selenium.webdriver.remote.webelement.WebElement. When u convert this into a str, u get this: <selenium.webdriver.remote.webelement.WebElement (session="21a8944e81b4dce8386fdf91067a2ddd", element="17660b77-61f4-4b2d-b5e8-f845ce97ad1e")>. So this is not the right way to get the text.
The right way is to use .text. Replace str(title) with title.text. Ur code should work. Here is the final code:
title = WebDriverWait(driver,5).until(
EC.presence_of_element_located((By.CLASS_NAME, "docsum-title")))
print('found title '+ title.text)
element = WebDriverWait(driver,5).until(EC.element_to_be_clickable((By.LINK_TEXT , title.text))).click();
print('found link to click')
As Arundeep pointed out, u can increase the wait time to make ur code better. But this was the main problem in ur code.
I'm trying to print out all the titles of the products of this website using scrapy shell: 'https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas'
Once it is open I start fetching:
fetch('https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas')
And I try to print out the title of each product as a result nothing is selected:
>>> response.css('.shelfProductTile-descriptionLink::text')
output: []
Also tried:
>>> response.css('a')
output: []
How can I do ? Thanks
Your code is correct. What happens is that there is no a element in the HTML retrieved by scrapy. When you visit the page with your browser, the product list is populated with javascript, on the browser side. They are not in the HTML code.
In the doc you'll find techniques to pre-render javascript. Maybe you should try that.
How can I check whether a given text string is present on the current page using Selenium?
The code is this:
def elem = driver.findElement(By.xpath("//*[contains(.,'search_text')]"));
if (elem == null) println("The text is not found on the page!");
If your searching the whole page for some text , then providing an xpath or selector to find an element is not necessary. The following code might help..
Assert.assertEquals(driver.getPageSource().contains("text_to_search"), true);
For some reason, certain elements don't seem to respond to the "generic" search listed in the other answer. At least not in Selenium2library under Robot Framework which is where I needed this incantation to find the particular element:
xpath=//script[contains(#src, 'super-sekret-url.example.com')]
A simpler (but probably less efficient) alternative to XPaths is to just get all the visible text in the page body like so:
def pageText = browser.findElement(By.tagName("body")).getText();
Then if you're using JUnit or something, you can use an assertion to check that the string you are searching for is contained in it.
assertThat("Text not found on page", pageText, containsString(searchText));
Using an XPath is perhaps more efficient, but this way is simpler to understand for those unfamiliar with it. Also, an AssertionError generated by assertThat will include the text that does exist on the page, which may be desirable for debugging as anybody looking at the logs can clearly see what text is on the page if what we are looking for isn't.