Issue Crawling Amazon, Element Cannot Be Scrolled Into View

I'm having an issue crawling pages on Amazon. I've tried using: Executing JS Script Action Chains Explicit Waits Nothing seems to work. Everything throws one exception or error o

Solution 1:

That's because you're trying to handle hidden link. Try below instead

next_button = ff.find_element_by_partial_link_text('Next')
next_button.click()

or

next_button= ff.find_element_by_link_text('Next→')

Note that find_element_by_partial_link_text/find_element_by_link_text searching for visible links only.

Also you might need to call

ff.implicitly_wait(10)

once in your script (somewhere after your WebDriver instance definition) or use ExplicitWait as below

from selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

next_button = WebDriverWait(ff, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, 'Next→')))

to be sure that required element will be find even with rendering delay