How To Fix Python, Urlopen Error [errno 8], Using Eventlet Green

Python novice here. I'm making a lot of asynchronous http requests using eventlet and urllib2. At the top of my file I have import eventlet import urllib from eventlet.green import

Solution 1:

The 'nodename not known' error means DNS resolution failed. Most likely cause is upstream DNS server rate limit. If you do web crawling seriously, I can recommend two approaches:

• easy: upon getting this error, just throttle down your concurrency limit, make fewer requests per minute. Treat first N occurrences of this error as temporary, repeat fetching of URL after a little delay. Setup local caching recursive DNS server (e.g. dnsmasq, unbound).
• hard: split DNS resolving and HTTP fetching. Have a separate queue of DNS names to resolve. Pass resolved IP address in URL http://1.2.3.4/path and Host: domain header to urlopen. This will allow to limit concurrency of DNS requests and actual HTTP requests separately. This will not help if you mostly fetch only one request per unique host. Find yourself many recursive DNS servers to distribute work, collect their response time stats, use faster ones more frequently.