Scraping data from CrowdTangle using API return expired image links - selenium

Question

I wanted to download images from CrowdTangle Dashboard. I wrote the code to fetch data using its API. However, historical posts scraped using the API return expired media links. While downloading the image, I got "URL expired" error. How to generate new links?

Answer 1

After talking with people, I figured out that I needed to scroll in the CrowdTangle dashboard to generate new image links. However, scrolling manually through thousands of posts will be a tedious task. Hence I decided to code a bot that scrolls. This solved my problem and I was able to generate new links.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
import time
options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])
browser = webdriver.Chrome(options=options)
link = {insert_link}
browser.get(link)
browser.maximize_window()
fb_button = browser.find_element(by=By.LINK_TEXT, value="click here.")
fb_button.click()
time.sleep(7)
phone = browser.find_element(by=By.ID,value="email")
password = browser.find_element(by=By.ID,value="pass")
submit = browser.find_element(by=By.ID,value="loginbutton")
phone.send_keys({phone number})
password.send_keys({password})
submit.click()
time.sleep(6)
element = browser.find_element(by=By.XPATH, value="/html/body/div[1]/div/div/div[3]/div")
while True:
browser.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", element)
time.sleep(3)
Go to the CrowdTangle dashboard, enter your filters, and query. Copy the link from the browser in the code. I would recommend running the scroll bot for each month. Sometimes more posts won't load. This is an issue with CrowdTangle. Just close the browser and move on to the next month.