urllib3.contrib package¶
These modules implement various extra features, that may not be ready for prime time or that require optional third-party dependencies.
urllib3.contrib.appengine module¶
This module provides a pool manager that uses Google App Engine’s URLFetch Service.
Example usage:
from urllib3 import PoolManager
from urllib3.contrib.appengine import AppEngineManager, is_appengine_sandbox
if is_appengine_sandbox():
# AppEngineManager uses AppEngine's URLFetch API behind the scenes
http = AppEngineManager()
else:
# PoolManager uses a socket-level API behind the scenes
http = PoolManager()
r = http.request('GET', 'https://google.com/')
There are limitations to the URLFetch service and it may not be the best choice for your application. There are three options for using urllib3 on Google App Engine:
You can use
AppEngineManager
with URLFetch. URLFetch is cost-effective in many circumstances as long as your usage is within the limitations.You can use a normal
PoolManager
by enabling sockets. Sockets also have limitations and restrictions and have a lower free quota than URLFetch. To use sockets, be sure to specify the following in yourapp.yaml
:env_variables: GAE_USE_SOCKETS_HTTPLIB : 'true'
3. If you are using App Engine Flexible, you can use the standard
PoolManager
without any configuration or special environment variables.
-
class
urllib3.contrib.appengine.
AppEngineManager
(headers=None, retries=None, validate_certificate=True, urlfetch_retries=True)¶ Bases:
urllib3.request.RequestMethods
Connection manager for Google App Engine sandbox applications.
This manager uses the URLFetch service directly instead of using the emulated httplib, and is subject to URLFetch limitations as described in the App Engine documentation here.
- Notably it will raise an
AppEnginePlatformError
if: - URLFetch is not available.
- If you attempt to use this on App Engine Flexible, as full socket support is available.
- If a request size is more than 10 megabytes.
- If a response size is more than 32 megabtyes.
- If you use an unsupported request method such as OPTIONS.
Beyond those cases, it will raise normal urllib3 errors.
-
urlopen
(method, url, body=None, headers=None, retries=None, redirect=True, timeout=<object object>, **response_kw)¶
- Notably it will raise an
-
exception
urllib3.contrib.appengine.
AppEnginePlatformError
¶ Bases:
urllib3.exceptions.HTTPError
-
exception
urllib3.contrib.appengine.
AppEnginePlatformWarning
¶
-
urllib3.contrib.appengine.
is_appengine
()¶
-
urllib3.contrib.appengine.
is_appengine_sandbox
()¶
-
urllib3.contrib.appengine.
is_local_appengine
()¶
-
urllib3.contrib.appengine.
is_prod_appengine
()¶
-
urllib3.contrib.appengine.
is_prod_appengine_mvms
()¶
urllib3.contrib.ntlmpool module¶
NTLM authenticating pool, contributed by erikcederstran
Issue #10, see: http://code.google.com/p/urllib3/issues/detail?id=10
-
class
urllib3.contrib.ntlmpool.
NTLMConnectionPool
(user, pw, authurl, *args, **kwargs)¶ Bases:
urllib3.connectionpool.HTTPSConnectionPool
Implements an NTLM authentication version of an urllib3 connection pool
-
scheme
= 'https'¶
-
urlopen
(method, url, body=None, headers=None, retries=3, redirect=True, assert_same_host=True)¶ Get a connection from the pool and perform an HTTP request. This is the lowest level call for making a request, so you’ll need to specify all the raw details.
Note
More commonly, it’s appropriate to use a convenience method provided by
RequestMethods
, such asrequest()
.Note
release_conn will only behave as expected if preload_content=False because we want to make preload_content=False the default behaviour someday soon without breaking backwards compatibility.
Parameters: - method – HTTP request method (such as GET, POST, PUT, etc.)
- body – Data to send in the request body (useful for creating POST requests, see HTTPConnectionPool.post_url for more convenience).
- headers – Dictionary of custom headers to send, such as User-Agent, If-None-Match, etc. If None, pool headers are used. If provided, these headers completely replace any pool-specific headers.
- retries (
Retry
, False, or an int.) –Configure the number of retries to allow before raising a
MaxRetryError
exception.Pass
None
to retry until you receive a response. Pass aRetry
object for fine-grained control over different types of retries. Pass an integer number to retry connection errors that many times, but no other types of errors. Pass zero to never retry.If
False
, then retries are disabled and any exception is raised immediately. Also, instead of raising a MaxRetryError on redirects, the redirect response will be returned. - redirect – If True, automatically handle redirects (status codes 301, 302, 303, 307, 308). Each redirect counts as a retry. Disabling retries will disable redirect, too.
- assert_same_host – If
True
, will make sure that the host of the pool requests is consistent else will raise HostChangedError. When False, you can use the pool on an HTTP proxy and request foreign hosts. - timeout – If specified, overrides the default timeout for this one
request. It may be a float (in seconds) or an instance of
urllib3.util.Timeout
. - pool_timeout – If set and the pool is set to block=True, then this method will
block for
pool_timeout
seconds and raise EmptyPoolError if no connection is available within the time period. - release_conn – If False, then the urlopen call will not release the connection
back into the pool once a response is received (but will release if
you read the entire contents of the response such as when
preload_content=True). This is useful if you’re not preloading
the response’s content immediately. You will need to call
r.release_conn()
on the responser
to return the connection back into the pool. If None, it takes the value ofresponse_kw.get('preload_content', True)
. - chunked – If True, urllib3 will send the body using chunked transfer encoding. Otherwise, urllib3 will send the body using the standard content-length form. Defaults to False.
- body_pos (int) – Position to seek to in file-like body in the event of a retry or redirect. Typically this won’t need to be set because urllib3 will auto-populate the value when needed.
- **response_kw – Additional parameters are passed to
urllib3.response.HTTPResponse.from_httplib()
-
urllib3.contrib.pyopenssl module¶
SSL with SNI-support for Python 2. Follow these instructions if you would like to verify SSL certificates in Python 2. Note, the default libraries do not do certificate checking; you need to do additional work to validate certificates yourself.
This needs the following packages installed:
- pyOpenSSL (tested with 16.0.0)
- cryptography (minimum 1.3.4, from pyopenssl)
- idna (minimum 2.0, from cryptography)
However, pyopenssl depends on cryptography, which depends on idna, so while we use all three directly here we end up having relatively few packages required.
You can install them with the following command:
pip install pyopenssl cryptography idna
To activate certificate checking, call
inject_into_urllib3()
from your Python code
before you begin making HTTP requests. This can be done in a sitecustomize
module, or at any other time before your application begins using urllib3
,
like this:
try:
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
except ImportError:
pass
Now you can use urllib3
as you normally would, and it will support SNI
when the required modules are installed.
Activating this module also has the positive side effect of disabling SSL/TLS compression in Python 2 (see CRIME attack).
If you want to configure the default list of supported cipher suites, you can
set the urllib3.contrib.pyopenssl.DEFAULT_SSL_CIPHER_LIST
variable.
-
urllib3.contrib.pyopenssl.
inject_into_urllib3
()¶ Monkey-patch urllib3 with PyOpenSSL-backed SSL-support.
-
urllib3.contrib.pyopenssl.
extract_from_urllib3
()¶ Undo monkey-patching by
inject_into_urllib3()
.
urllib3.contrib.socks module¶
This module contains provisional support for SOCKS proxies from within
urllib3. This module supports SOCKS4 (specifically the SOCKS4A variant) and
SOCKS5. To enable its functionality, either install PySocks or install this
module with the socks
extra.
The SOCKS implementation supports the full range of urllib3 features. It also supports the following SOCKS features:
- SOCKS4
- SOCKS4a
- SOCKS5
- Usernames and passwords for the SOCKS proxy
Known Limitations:
- Currently PySocks does not support contacting remote websites via literal IPv6 addresses. Any such connection attempt will fail. You must use a domain name.
- Currently PySocks does not support IPv6 connections to the SOCKS proxy. Any such connection attempt will fail.
-
class
urllib3.contrib.socks.
SOCKSConnection
(*args, **kwargs)¶ Bases:
urllib3.connection.HTTPConnection
A plain-text HTTP connection that connects via a SOCKS proxy.
-
class
urllib3.contrib.socks.
SOCKSHTTPConnectionPool
(host, port=None, strict=False, timeout=<object object>, maxsize=1, block=False, headers=None, retries=None, _proxy=None, _proxy_headers=None, **conn_kw)¶ Bases:
urllib3.connectionpool.HTTPConnectionPool
-
ConnectionCls
¶ alias of
SOCKSConnection
-
-
class
urllib3.contrib.socks.
SOCKSHTTPSConnection
(*args, **kwargs)¶ Bases:
urllib3.contrib.socks.SOCKSConnection
,urllib3.connection.VerifiedHTTPSConnection
-
class
urllib3.contrib.socks.
SOCKSHTTPSConnectionPool
(host, port=None, strict=False, timeout=<object object>, maxsize=1, block=False, headers=None, retries=None, _proxy=None, _proxy_headers=None, key_file=None, cert_file=None, cert_reqs=None, ca_certs=None, ssl_version=None, assert_hostname=None, assert_fingerprint=None, ca_cert_dir=None, **conn_kw)¶ Bases:
urllib3.connectionpool.HTTPSConnectionPool
-
ConnectionCls
¶ alias of
SOCKSHTTPSConnection
-
-
class
urllib3.contrib.socks.
SOCKSProxyManager
(proxy_url, username=None, password=None, num_pools=10, headers=None, **connection_pool_kw)¶ Bases:
urllib3.poolmanager.PoolManager
A version of the urllib3 ProxyManager that routes connections via the defined SOCKS proxy.
-
pool_classes_by_scheme
= {'http': <class 'urllib3.contrib.socks.SOCKSHTTPConnectionPool'>, 'https': <class 'urllib3.contrib.socks.SOCKSHTTPSConnectionPool'>}¶
-