Website Navigates To No-access Page Using ChromeDriver And Chrome Through Selenium Probably Bot Protected

June 27, 2022 Post a Comment

My target site is https://www.nike.com/kr/ko_kr When using selenium driver.get to connect to this target using webdriver.Chrome().get, the connection is done. But if I click elemen

Solution 1:

You can try to set a timeout of a few seconds between your actions, to act more "human-like".

There is an implicit and an explicit way (source: Selenium Waits).

An explicit wait is a code you define to wait for a certain condition to occur before proceeding further in the code. The extreme case of this is time.sleep(), which sets the condition to an exact time period to wait

You are probably looking for the implicit way: An implicit wait tells WebDriver to poll the DOM for a certain amount of time when trying to find any element (or elements) not immediately available. The default setting is 0. Once set, the implicit wait is set for the life of the WebDriver object.

from selenium import webdriver

driver = webdriver.Firefox()
driver.implicitly_wait(10) # seconds
driver.get("http://somedomain/url_that_delays_loading")
myDynamicElement = driver.find_element_by_id("myDynamicElement")

Another way to wait for a few seconds works with the package time:

import time 
time.sleep(5) #wait 5 seconds

Solution 2:

I made some tweaks to your code and executed the test as follows:

Code Block:

from selenium import webdriver 

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.naver.com/")
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_script("Object.defineProperty(navigator, 'plugins', {get: function() {return[1, 2, 3, 4, 5]}})")
driver.execute_script("Object.defineProperty(navigator, 'languages', {get: function() {return['ko-KR', 'ko']}})")
driver.execute_script("const getParameter = WebGLRenderingContext.getParameter;WebGLRenderingContext.prototype.getParameter = function(parameter) {if (parameter === 37445) {return 'NVIDIA Corporation'} if (parameter === 37446) {return 'NVIDIA GeForce GTX 980 Ti OpenGL Engine';}return getParameter(parameter);};")
driver.get("https://www.nike.com/kr/ko_kr/")

Observation

Similar to your observation, I have hit the same roadblock being redirected to No Access page as follows:

Deep Dive

It seems Selenium driven ChromeDriver initiated Chrome Browsing Context is getting detected as a automated bot.

Meanwhile, while inspecting the DOM Tree of the webpage it was observed that some of the <script> tag contains the keyword akam. As an example:

<script type="text/javascript" src="https://www.nike.com/akam/11/43465b03" defer=""></script>
<noscript><img src="https://www.nike.com/akam/11/pixel_43465b03?a=dD1kMjkzYzhlOTA4OWVmZTlhOGZhMjg2MjBmNjk5YWVjZmM0Y2U2NWY5JmpzPW9mZg==" style="visibility: hidden; position: absolute; left: -999px; top: -999px;" /></noscript>
<link id="dnsprefetchlink" href="//gerxi63iifbfuxxtmreq-f-e92349eda-clientnsv4-s.akamaihd.net">

Which is a clear indication that the website is protected by Bot Manager an advanced bot detection service provided by Akamai and the response gets blocked.

Bot Manager

As per the article Bot Manager - Foundations:

akamai_detection

Conclusion

So it can be concluded that the request for the data is detected as being performed by Selenium driven WebDriver instance and the response is blocked.

References

A couple of documentations:

tl; dr

A couple of relevant discussions:

Python Guru