Cloudflare And Chromedriver - Cloudflare Distinguishes Between Chromedriver And Genuine Chrome?

I would like to use chromedriver to scrape some stories from fanfiction.net. I try the following: from selenium import webdriver import time path = 'D:\chromedriver\chromedriver.e

Solution 1:

This error message...

...implies that the Cloudflare have detected your requests to the website as an automated bot and subsequently denying you the access to the application.

---

Solution

In these cases the a potential solution would be to use the undetected-chromedriver to initialize the Chrome Browsing Context.

undetected-chromedriver is an optimized Selenium Chromedriver patch which does not trigger anti-bot services like Distill Network / Imperva / DataDome / Botprotect.io. It automatically downloads the driver binary and patches it.

• Code Block:

  import undetected_chromedriver as uc
  from selenium import webdriver
  import time
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  driver = uc.Chrome(options=options)
  url1 = 'https://www.fanfiction.net/s/8832472'
  url2 = 'https://www.fanfiction.net/s/5218118'
  driver.get(url1)
  time.sleep(5)
  driver.get(url2)
  

---

References

You can find a couple of relevant detailed discussions in:

• Selenium app redirect to Cloudflare page when hosted on Heroku
• How to bypass being rate limited ..HTML Error 1015 using Python