Web Scraping/ Web Crawling - selenium

Question

Can somebody help me figure out how to scrape / crawl this website? https://www.arkansasonline.com/i/lrcrime/
I've downloaded the page source, with requests and parced with BeautifulSoup, but I can't figure out what's going on.
Here is what I have so far:
#####################################################
import requests
from bs4 import BeautifulSoup
url = 'https://www.arkansasonline.com/i/lrcrime/'
r = requests.get(url, headers = headers).text
soup = BeautifulSoup(r,'html.parser' )
data = [x.get_text() for x in soup.find_all('td')]
#####################################################
Typically this would do the trick and I'd get a list of all table data inputs..
But I'm getting
['SEARCH | DISPATCH LOG | STORIES | HOMICIDES | OLD MAP',
'\n\n\n\nClick here to load this Caspio Cloud Database\nCloud Database by Caspio\n']
Which is far from what I need....
Also, how do I craw the 3000 pages?
I also tried to do it with a macro and just record my keystrokes and save to google drive, but the page moves around as you go through the pages, so it makes that basically impossible. They are trying to hide the crime data In my opinion. I want to scrape it all into 1 database and release it to the public.

Answer 1

What happens?
Most of content is provided dynamically, so you won't get it with requests, only the first table with some navigation is provided static, thats why you get this as result.
There is also a "big" hint you should respect.
This Content is Only Available to Subscribers.
How to fix?
NOTE: Be kind and subscribe to consume the content - would be the best solution.
Assuming this is the case you should go with selenium that will render the page as a browser will do and also provide the table you are looking for.
You have to wait until the table is loaded:
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'table[data-cb-name="cbTable"]')))
Grab the page_source and load it with pandas as dataframe.
Example (selenium 4)
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
service = Service(executable_path='ENTER YOUR PATH TO CHROMEDRIVER')
driver = webdriver.Chrome(service=service)
driver.get('https://www.arkansasonline.com/i/lrcrime/')
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'table[data-cb-name="cbTable"]')))
pd.read_html(driver.page_source)[1]
Output
INCIDENT NUMBER
OFFENSE DESCRIPTION
Location/address
ZIP
INCIDENT DATE
2021-134824
ALL OTHER LARCENY
8109 W 34TH ST
72204
11/1/2021
2021-134815
BURGLARY/B&E
42 NANDINA LN
72210
11/1/2021
2021-134790
ROBBERY
1800 BROADWAY ST
72206
11/1/2021
2021-134788
AGGRAVATED ASSAULT
11810 PLEASANT RIDGE RD
72223
11/1/2021
2021-134778
THEFT FROM MOTOR VEHICLE
4 HANOVER DR
72209
11/1/2021
...
...
...
...
...