Unable to parse the links of different cases from next pages using requests
I’ve created a script to parse the links of different cases revealed upon selecting an option in dropdown from a webpage. This is the website link and this is the option Probate that should be chosen from the dropdown titled as Case Type located at the top right before hitting the search button. All the other options should be as they are.
The script can parse the links of different cases from the first page flawlessly. However, I can't make the script go on to the next pages to collect links from there as well.
This is how next pages are visible in there at the bottom: 
And the dropdown should look when the option is chosen: 
I’ve tried so far:
import requests from bs4 import BeautifulSoup link = "http://surrogateweb.co.ocean.nj.us/BluestoneWeb/Default.aspx" with requests.Session() as s: s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36' r = s.get(link) soup = BeautifulSoup(r.text,"lxml") payload = {i['name']:i.get('value','') for i in soup.select('input[name],select')} for k,v in payload.items(): if k.endswith('ComboBox_case_type'): payload[k] = "Probate" elif k.endswith('ComboBox_case_type_VI'): payload[k] = "WILL" elif k.endswith('ComboBox_case_type$DDD$L'): payload[k] = "WILL" elif k.endswith('ComboBox_town$DDD$L'): payload[k] = "%" r = s.post(link,data=payload) soup = BeautifulSoup(r.text,"lxml") for pk_id in soup.select("a.dxeHyperlink_Youthful[href*='Q_PK_ID']"): print(pk_id.get("href")) How can I collect the links of different cases from next pages using requests?
PS I’m not after any selenium related solution.
This codes works but use selenium instead of requests.
You need to install selenium python lib and download gecko driver. If you do not want to have geckodriver in c:/program you have to change executable_path= to the path you have geckodriver in. You maybe want to make the sleep time shorter to, but the site is loading so slow (for me) so i have to set long sleep times so the site loads correctly before trying to read from it.
from selenium import webdriver from bs4 import BeautifulSoup import time link = "http://surrogateweb.co.ocean.nj.us/BluestoneWeb/Default.aspx" driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe') driver.get(link) dropdown = driver.find_element_by_css_selector('#ContentPlaceHolder1_ASPxSplitter1_ASPxComboBox_case_type_B-1') dropdown.click() time.sleep(0.5) cases = driver.find_elements_by_css_selector('.dxeListBoxItem_Youthful') for case in cases: if case.text == 'Probate': time.sleep(5) case.click() time.sleep(5) search = driver.find_element_by_css_selector('#ContentPlaceHolder1_ASPxSplitter1_ASPxButton_search') search.click() while True: time.sleep(15) soup = BeautifulSoup(driver.page_source,"lxml") for pk_id in soup.select("a.dxeHyperlink_Youthful[href*='Q_PK_ID']"): print(pk_id.get("href")) next = driver.find_elements_by_css_selector('.dxWeb_pNext_Youthful') if len(next) > 0: next[0].click() else: break 
