.. _topics-download-handlers:

=================
Download handlers
=================

Download handlers are Scrapy :ref:`components <topics-components>` used to
download :ref:`requests <topics-request-response>` and produce responses from
them.

Using download handlers
=======================

The :setting:`DOWNLOAD_HANDLERS_BASE` and :setting:`DOWNLOAD_HANDLERS` settings
tell Scrapy which handler is responsible for a given URL scheme. Their values
are merged into a mapping from scheme names to handler classes. When Scrapy
initializes it creates instances of all configured download handlers (except
for :ref:`lazy ones <lazy-download-handlers>`) and stores them in a similar
mapping. When Scrapy needs to download a request it extracts the scheme from
its URL, finds the handler for this scheme, passes the request to it and gets a
response from it.  If there is no handler for the scheme, the request is not
downloaded and a :exc:`~scrapy.exceptions.NotSupported` exception is raised.

The :setting:`DOWNLOAD_HANDLERS_BASE` setting contains the default mapping of
handlers. You can use the :setting:`DOWNLOAD_HANDLERS` setting to add handlers
for additional schemes and to replace or disable default ones:

.. code-block:: python

    DOWNLOAD_HANDLERS = {
        # disable support for ftp:// requests
        "ftp": None,
        # replace the default one for http://
        "http": "my.download_handlers.HttpHandler",
        # http:// and https:// are different schemes,
        # even though they may use the same handler
        "https": "my.download_handlers.HttpHandler",
        # support for any custom scheme can be added
        "sftp": "my.download_handlers.SftpHandler",
    }

.. seealso:: :ref:`security-unencrypted-protocols` and
    :ref:`security-local-resources`, for the security implications of the
    default ``http``, ``ftp``, ``file`` and ``data`` handlers.

Replacing HTTP(S) download handlers
-----------------------------------

While Scrapy provides a default handler for ``http`` and ``https`` schemes,
users may want to use a different handler, provided by Scrapy or by some
3rd-party package. There are several considerations to keep in mind related to
this.

First of all, as ``http`` and ``https`` are separate schemes, they need
separate entries in the :setting:`DOWNLOAD_HANDLERS` setting, even though it's
likely that the same handler class will be used for both schemes.

Additionally, some of the Scrapy settings, like :setting:`DOWNLOAD_MAXSIZE`,
are honored by the default HTTP(S) handler but not necessarily by alternative
ones. The same may apply to other Scrapy features, e.g. the
:signal:`bytes_received` and :signal:`headers_received` signals.

.. _lazy-download-handlers:

Lazy instantiation of download handlers
---------------------------------------

A download handler can be marked as "lazy" by setting its ``lazy`` class
attribute to ``True``. Such handlers are only instantiated when they need to
download their first request. This may be useful when the instantiation is slow
or requires dependencies that are not always available, and the handler is not
needed on every spider run. For example, :class:`the built-in S3 handler
<.S3DownloadHandler>` is lazy.

Writing your own download handler
=================================

A download handler is a :ref:`component <topics-components>` that defines
the following API:

.. autoclass:: scrapy.core.downloader.handlers.DownloadHandlerProtocol
    :members:

An optional base class for custom handlers is provided:

.. autoclass:: scrapy.core.downloader.handlers.base.BaseDownloadHandler
    :members:
    :undoc-members:
    :exclude-members: close, download_request, lazy

.. _download-handlers-exceptions:

Exceptions raised by download handlers
======================================

.. versionadded:: 2.15.0

The built-in download handlers raise Scrapy-specific exceptions instead of
implementation-specific ones, so that code that handles these exceptions can be
written in a generic way. We recommend custom download handlers to also use
these exceptions.

.. autoexception:: scrapy.exceptions.CannotResolveHostError

.. autoexception:: scrapy.exceptions.DownloadCancelledError

.. autoexception:: scrapy.exceptions.DownloadConnectionRefusedError

.. autoexception:: scrapy.exceptions.DownloadFailedError

.. autoexception:: scrapy.exceptions.DownloadTimeoutError

.. autoexception:: scrapy.exceptions.ResponseDataLossError

.. autoexception:: scrapy.exceptions.UnsupportedURLSchemeError

.. _download-handlers-ref:

Built-in HTTP download handlers reference
=========================================

Scrapy ships several handlers for HTTP and HTTPS requests. While all of them
support basic features, they may differ in support of specific Scrapy features
and settings and HTTP protocol features. See the documentation of specific
handlers and specific settings for more information. Additionally, as the
underlying HTTP client implementations differ between handlers, the behavior of
specific websites may be different when doing the same Scrapy requests but
using different handlers.

Here is a comparison of some features of the built-in HTTP handlers, see the
individual handler docs for more differences:

=================== ================= ===================== ====================
Feature             H2DownloadHandler HTTP11DownloadHandler HttpxDownloadHandler
=================== ================= ===================== ====================
Requires asyncio    No                No                    Yes
Requires a reactor  Yes               Yes                   No
HTTP/1.1            No                Yes                   Yes
HTTP/2              Yes               No                    Yes
TLS implementation  ``cryptography``  ``cryptography``      Stdlib ``ssl``
HTTP proxies        No                Yes                   Yes
SOCKS proxies       No                No                    Yes
Bad header handling Not applicable    Skip bad              Fail
=================== ================= ===================== ====================

Bad header handling is what a handler does when a response has a bad header
line, e.g. one with no colon in it, which some servers send. Handlers that skip
bad header lines, like web browsers do, still parse the header lines that follow
them; other handlers also lose those, or cannot download such responses at all.

You can find additional HTTP download handlers in the
scrapy-download-handlers-incubator_ package. This package is made by the Scrapy
developers and contains experimental handlers that may be included in some
later Scrapy version but can already be used. Please refer to the documentation
of this package for more information.

.. _scrapy-download-handlers-incubator: https://github.com/scrapy-plugins/scrapy-download-handlers-incubator

.. _twisted-http2-handler:

H2DownloadHandler
-----------------

.. note:: Requires the :ref:`twisted-http2 <extras>` extra.

.. autoclass:: scrapy.core.downloader.handlers.http2.H2DownloadHandler

| Supported scheme: ``https``.
| :ref:`Lazy <lazy-download-handlers>`: yes.
| :ref:`Requires asyncio support <using-asyncio>`: no.
| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: yes.

This handler supports ``https://host/path`` URLs and uses the HTTP/2 protocol
for them.

It's implemented using :mod:`twisted.web.client` and the ``h2`` library.

If you want to use this handler you need to replace the default one for the
``https`` scheme:

.. code-block:: python

    DOWNLOAD_HANDLERS = {
        "https": "scrapy.core.downloader.handlers.http2.H2DownloadHandler",
    }

Features and limitations
^^^^^^^^^^^^^^^^^^^^^^^^

=========================== ================================================
HTTP proxies                No (not implemented)
SOCKS proxies               No (not supported by the library)
HTTP/2                      Yes
Bad header handling         Not applicable (HTTP/2 only)
``response.certificate``    :class:`twisted.internet.ssl.Certificate` object
Per-request ``bindaddress`` Yes
TLS implementation          ``pyOpenSSL``/``cryptography``
=========================== ================================================

Other limitations:

-   No support for HTTP/1.1.

-   IPv6 support requires setting :setting:`TWISTED_DNS_RESOLVER`
    to ``scrapy.resolver.CachingHostnameResolver``.

Known limitations of the HTTP/2 support:

-   No support for HTTP/2 Cleartext (h2c), since no major browser supports
    HTTP/2 unencrypted (refer `http2 faq`_).

-   No support for `server pushes`_, which are ignored.

.. _http2 faq: https://http2.github.io/faq/#does-http2-require-encryption
.. _server pushes: https://datatracker.ietf.org/doc/html/rfc7540#section-8.2

HTTP11DownloadHandler
---------------------

.. autoclass:: scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler

| Supported schemes: ``http``, ``https``.
| :ref:`Lazy <lazy-download-handlers>`: no.
| :ref:`Requires asyncio support <using-asyncio>`: no.
| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: yes.

This handler supports ``http://host/path`` and ``https://host/path`` URLs and
uses the HTTP/1.1 protocol for them.

It's implemented using :mod:`twisted.web.client`.

Features and limitations
^^^^^^^^^^^^^^^^^^^^^^^^

=========================== ================================================
HTTP proxies                Yes
SOCKS proxies               No (not supported by the library)
HTTP/2                      No (implemented as a separate handler)
Bad header handling         Skip bad, like web browsers do
``response.certificate``    :class:`twisted.internet.ssl.Certificate` object
Per-request ``bindaddress`` Yes
TLS implementation          ``pyOpenSSL``/``cryptography``
=========================== ================================================

.. versionchanged:: 2.18.0
   Bad header lines with no colon in them are now skipped, instead of making
   the whole response impossible to download.

Other limitations:

-   IPv6 support requires setting :setting:`TWISTED_DNS_RESOLVER`
    to ``scrapy.resolver.CachingHostnameResolver``.

-   HTTPS proxies to HTTPS destinations are not supported.

.. _httpx-handler:

HttpxDownloadHandler
--------------------

.. note:: Requires the :ref:`httpx <extras>` extra.

.. versionadded:: 2.15.0

.. autoclass:: scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler

| Supported schemes: ``http``, ``https``.
| :ref:`Lazy <lazy-download-handlers>`: no.
| :ref:`Requires asyncio support <using-asyncio>`: yes.
| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: no.

This handler supports ``http://host/path`` and ``https://host/path`` URLs and
uses the HTTP/1.1 or HTTP/2 protocol for them.

It's implemented using the httpx2_ library.

.. _httpx2: https://httpx2.pydantic.dev/

If you want to use this handler you need to replace the default ones for the
``http`` and ``https`` schemes:

.. code-block:: python

    DOWNLOAD_HANDLERS = {
        "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
        "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
    }

Features and limitations
^^^^^^^^^^^^^^^^^^^^^^^^

.. warning::

    This handler is experimental, and not yet recommended for production
    environments. Future Scrapy versions may introduce related changes without
    a deprecation period or warning or even remove it altogether.

=========================== =======================================
HTTP proxies                Yes
SOCKS proxies               Yes (SOCKS5)
HTTP/2                      Yes
Bad header handling         Fail (not supported by the library)
``response.certificate``    DER bytes
Per-request ``bindaddress`` No (not supported by the library)
TLS implementation          Standard library ``ssl``
=========================== =======================================

Other limitations:

-   The handler creates a separate connection pool for each proxy URL (due to
    limitations of ``httpx``) which may lead to higher resource usage when
    using proxy rotation.

.. setting:: HTTPX_HTTP2_ENABLED

HTTPX_HTTP2_ENABLED
^^^^^^^^^^^^^^^^^^^

.. versionadded:: 2.17.0

Default: ``False``

Whether to enable HTTP/2 support in this handler.

Built-in non-HTTP download handlers reference
=============================================

DataURIDownloadHandler
----------------------

.. autoclass:: scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler

| Supported scheme: ``data``.
| :ref:`Lazy <lazy-download-handlers>`: no.
| :ref:`Requires asyncio support <using-asyncio>`: no.
| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: no.

This handler supports RFC 2397 ``data:content/type;base64,`` data URIs.

FileDownloadHandler
-------------------

.. autoclass:: scrapy.core.downloader.handlers.file.FileDownloadHandler

| Supported scheme: ``file``.
| :ref:`Lazy <lazy-download-handlers>`: no.
| :ref:`Requires asyncio support <using-asyncio>`: no.
| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: no.

This handler supports ``file:///path`` local file URIs. It doesn't
support remote files.

FTPDownloadHandler
------------------

.. autoclass:: scrapy.core.downloader.handlers.ftp.FTPDownloadHandler

| Supported scheme: ``ftp``.
| :ref:`Lazy <lazy-download-handlers>`: no.
| :ref:`Requires asyncio support <using-asyncio>`: no.
| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: yes.

This handler supports ``ftp://host/path`` FTP URIs.

It's implemented using :mod:`twisted.protocols.ftp`.

.. _s3-handler:

S3DownloadHandler
-----------------

.. note:: Requires the :ref:`s3 <extras>` extra.

.. autoclass:: scrapy.core.downloader.handlers.s3.S3DownloadHandler

| Supported scheme: ``s3``.
| :ref:`Lazy <lazy-download-handlers>`: yes.
| :ref:`Requires asyncio support <using-asyncio>`: no.
| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: no.

This handler supports ``s3://bucket/path`` S3 URIs.

It's implemented using the botocore_ library.

.. _botocore: https://github.com/boto/botocore
