Handling Pagination On Website Using Input Buttons

In trying to scrape this website using selenium. I have the code working but it currently only scrapes the first page. The page uses input buttons as a way to navigate through page

Solution 1:

Try with: find_elements_by_xpath instead of find_element_by_xpath which will return you the list.

PS: I didn't tried your code locally but the error you mentioned is the solution which I mentioned.

Solution 2:

As per your question to handle Pagination on the website http://www.boston.gov.uk/index.aspx?articleid=6207 you can use the following solution:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.chrome.options import Options
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.support.select import Select
  
  options = Options()
  options.add_argument("start-maximized")
  options.add_argument("disable-infobars")
  options.add_argument("--disable-extensions")
  driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\ChromeDriver\chromedriver_win32\chromedriver.exe')
  driver.get('http://www.boston.gov.uk/index.aspx?articleid=6207&ShowAdvancedSearch=true')
  mySelectElement = Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#DatePresets[name='DatePresets']"))))
  mySelectElement.select_by_visible_text('Last month')
  driver.find_element_by_css_selector("input.button[name='searchFilter']").click()
  numLinks = len(WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "input.pageNumberButton"))))
  print(numLinks)
  for i inrange(numLinks):
      print("Perform your scrapping here on page {}".format(str(i+1)))
      WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@class='pageNumberButton selected']//following::input[1]"))).click()
  driver.quit()
  

• Console Output:

  DevTools listening on ws://127.0.0.1:12115/devtools/browser/2ece3f6a-0431-4b74-9276-f61fcf70dd6d10
  Perform your scrapping here on page 1
  Perform your scrapping here on page 2
  Perform your scrapping here on page 3
  Perform your scrapping here on page 4
  Perform your scrapping here on page 5
  Perform your scrapping here on page 6
  Perform your scrapping here on page 7
  Perform your scrapping here on page 8
  Perform your scrapping here on page 9
  Perform your scrapping here on page 10