Requests, Mechanize, Urllib Fails But Curl Works

Whilst attempting to access this site through requests, I receive: ('Connection aborted.', error(54, 'Connection reset by peer')) I have also tried to access the site through mech

Solution 1:

The server requires the use of SNI and just closes the connection if no SNI is used. Looks like curl uses SNI, while at least the version of the requests library you are using does not use SNI.

You can try this with OpenSSL. Without SNI you get an error:

$ openssl s_client -connect datamuster.marketdatasuite.com:443
CONNECTED(00000003)
write:errno=104

But if you use SNI (-servername ...) then it works:

CONNECTED(00000003)depth=1C=GB,ST=GreaterManchester,L=Salford,O=COMODOCALimited,CN=COMODOSSLCA...SSL-Session:Protocol  :TLSv1.2Cipher    :AES128-SHA256

According to the FAQ for request SNI is not support with Python 2, only with Python 3. See this resource for information on how to make SNI possible with Python 2.

Solution 2:

This process wasn't totally straightforward so I thought I'd post a new answer to make it easy to follow for others.

Following this thread, I needed to install these libraries get SNI to work with Python 2:

• pyOpenSSL
• ndg-httpsclient
• pyasn1

However, pyOpenSSL may cause problems when installed with pip install pyOpenSSL. I actually had to remove my existing openssl, since pyOpenSSL version 0.14 didn't seem to work:

pip uninstall pyOpenSSL

The following command installed all necessary dependencies:

pip install pyOpenSSL==0.13 ndg-httpsclient pyasn1

This should get requests to now work with SNI on python 2.

---

Keep reading for the issues with pyOpenSSL ver. 0.14...

When installing ver. 0.14 I get the following error:

Command /usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/private/var/folders/04/3f_y5fw166v03k7b51j1tsl80000gn/T/pip_build_alex/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/04/3f_y5fw166v03k7b51j1tsl80000gn/T/pip-7QR71B-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /private/var/folders/04/3f_y5fw166v03k7b51j1tsl80000gn/T/pip_build_alex/cryptography
Storing debug logfor failure in /Users/alex/.pip/pip.log

and pyOpenSSL installs as ver. 0.14 incompletely:

$pipshowpyOpenSSL---Name:pyOpenSSLVersion:0.14Location:/usr/local/lib/python2.7/site-packagesRequires:cryptography,six

as can be seen from the requests.get() attempt:

importrequestsresponse= requests.get("http://datamuster.marketdatasuite.com")

(...lots of errors...)
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(54, 'Connection reset by peer'))

The following commands revert to pyOpenSSL ver. 0.13 and correct the issue

pip uninstall pyOpenSSL
pip install pyOpenSSL==0.13

and then in python:

import requests
requests.get("http://datamuster.marketdatasuite.com")

<Response [200]>

Solution 3:

I got the same issues when using Python requests to send testing requests to WordPress (I installed WordPress on a dedicate server). I tried to update SSL- packages without success.

Then, I realised that the requests sent to server got delays in receiving response. The long delays requests were always "kicked-off" and caused ('Connection aborted.', error(54, 'Connection reset by peer')). It turned out that the web server (apache) resets the connection while the request is still waiting for response.

I increase the KeepAliveTimeout from 5 seconds to 20 seconds (in Apache web server) and never get this error again.

Improve code for Exceptions: Increasing KeepAliveTimeout works in most of the tests. However, in some tests, I still got the same error and program stops. I add the code that catches Exception and repeat the request it occurs.

import requests
...
while(1):
    requestOK = Truetry:
       r = session.get(requestURL, headers=headers, timeout=None)
    except requests.exceptions.ConnectionError: 
       print ("'Connection aborted.', error(54, 'Connection reset by peer')")
       print ("\tResend request...")
       requestOK = Falseif requestOK:
       break

Hope this will help!