.. _topics-debug:

=================
Debugging Spiders
=================

This document explains the most common techniques for debugging spiders.
Consider the following Scrapy spider below:

.. skip: next
.. code-block:: python

    import scrapy
    from myproject.items import MyItem


    class MySpider(scrapy.Spider):
        name = "myspider"
        start_urls = (
            "http://example.com/page1",
            "http://example.com/page2",
        )

        def parse(self, response):
            # <processing code not shown>
            # collect `item_urls`
            for item_url in item_urls:
                yield scrapy.Request(item_url, self.parse_item)

        def parse_item(self, response):
            # <processing code not shown>
            item = MyItem()
            # populate `item` fields
            # and extract item_details_url
            yield scrapy.Request(
                item_details_url, self.parse_details, cb_kwargs={"item": item}
            )

        def parse_details(self, response, item):
            # populate more `item` fields
            return item

Basically this is a simple spider which parses two pages of items (the
start_urls). Items also have a details page with additional information, so we
use the ``cb_kwargs`` functionality of :class:`~scrapy.Request` to pass a
partially populated item.


Parse Command
=============

The most basic way of checking the output of your spider is to use the
:command:`parse` command. It allows to check the behaviour of different parts
of the spider at the method level. It has the advantage of being flexible and
simple to use, but does not allow debugging code inside a method.

.. highlight:: none

.. skip: start

In order to see the item scraped from a specific url::

    $ scrapy parse --spider=myspider -c parse_item -d 2 <item_url>
    [ ... scrapy log lines crawling example.com spider ... ]

    >>> STATUS DEPTH LEVEL 2 <<<
    # Scraped Items  ------------------------------------------------------------
    [{'url': <item_url>}]

    # Requests  -----------------------------------------------------------------
    []

Using the ``--verbose`` or ``-v`` option we can see the status at each depth level::

    $ scrapy parse --spider=myspider -c parse_item -d 2 -v <item_url>
    [ ... scrapy log lines crawling example.com spider ... ]

    >>> DEPTH LEVEL: 1 <<<
    # Scraped Items  ------------------------------------------------------------
    []

    # Requests  -----------------------------------------------------------------
    [<GET item_details_url>]


    >>> DEPTH LEVEL: 2 <<<
    # Scraped Items  ------------------------------------------------------------
    [{'url': <item_url>}]

    # Requests  -----------------------------------------------------------------
    []

Checking items scraped from a single start_url, can also be easily achieved
using::

    $ scrapy parse --spider=myspider -d 3 'http://example.com/page1'

.. skip: end


Scrapy Shell
============

While the :command:`parse` command is very useful for checking behaviour of a
spider, it is of little help to check what happens inside a callback, besides
showing the response received and the output. How to debug the situation when
``parse_details`` sometimes receives no item?

.. highlight:: python

Fortunately, the :command:`shell` is your bread and butter in this case (see
:ref:`topics-shell-inspect-response`):

.. code-block:: python

    from scrapy.shell import inspect_response


    def parse_details(self, response, item=None):
        if item:
            # populate more `item` fields
            return item
        else:
            inspect_response(response, self)

See also: :ref:`topics-shell-inspect-response`.


Open in browser
===============

Sometimes you just want to see how a certain response looks in a browser, you
can use the :func:`~scrapy.utils.response.open_in_browser` function for that:

.. autofunction:: scrapy.utils.response.open_in_browser


Logging
=======

Logging is another useful option for getting information about your spider run.
Although not as convenient, it comes with the advantage that the logs will be
available in all future runs should they be necessary again:

.. code-block:: python

    def parse_details(self, response, item=None):
        if item:
            # populate more `item` fields
            return item
        else:
            self.logger.warning("No item received for %s", response.url)

For more information, check the :ref:`topics-logging` section.

.. _debug-live-traffic:

Inspecting live traffic
=======================

Sometimes it's important to see what exactly was sent to the server or received
from it, such as header values, formatting and order (Scrapy cannot log this,
as underlying HTTP libraries produce the final values for request headers and
canonicalize response ones) or TLS handshake details. There are two ways to see
and log the real traffic of a running spider:

-   Capture the traffic with a tool such as Wireshark_. As your requests likely
    use TLS, you will need to decrypt the traffic (see the `Wireshark TLS
    documentation`_ for detailed instructions). You will need the encryption
    key which you can save as described in :ref:`debug-tls`. As this way of
    capturing traffic is passive, it cannot interfere with the spider.

-   Use mitmproxy_ between the spider and the server, as described below. This
    is easier to set up and in addition to inspecting the traffic allows
    modifying it, but it's not passive: there is now a connection between
    Scrapy and mitmproxy and another one between mitmproxy and the server
    instead of a direct connection between Scrapy and the server. Due to this,
    low-level connection behavior is different from normal crawls, which may
    change the server behavior, and you cannot easily use mitmproxy and regular
    proxies in the same crawl.

Using mitmdump
--------------

You should refer to the mitmproxy documentation for more details, additional
interception modes and advanced features but here is one simple way to use it.
First, run a ``mitmdump`` instance (it will use the port 8080 by default),
asking it to log the traffic details on the terminal (``--flow-detail 2`` will
log headers but not bodies):

.. code-block:: shell

    mitmdump --flow-detail 2

Then configure ``http://127.0.0.1:8080`` as a proxy in your spider using
:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`:

.. code-block:: shell

    https_proxy=http://127.0.0.1:8080 scrapy crawl myspider

To inspect only some requests, set their :reqmeta:`proxy` meta key instead.

.. _mitmproxy: https://mitmproxy.org/
.. _Wireshark: https://www.wireshark.org/
.. _Wireshark TLS documentation: https://wiki.wireshark.org/TLS

.. _debug-tls:

Decrypting TLS traffic
======================

Scrapy writes the session keys of its HTTPS connections to the file that the
``SSLKEYLOGFILE`` environment variable points to, using the `NSS key log
format`_ that traffic analysis tools such as Wireshark understand.

.. versionadded:: VERSION

.. code-block:: shell

    SSLKEYLOGFILE=/tmp/sslkeylog scrapy crawl myspider

.. _NSS key log format: https://firefox-source-docs.mozilla.org/security/nss/legacy/key_log_format/index.html

.. warning:: Anyone who can read the key log file can decrypt the traffic of
    the connections recorded in it, including any credentials that they carry.

.. _debug-vscode:

Visual Studio Code
==================

.. highlight:: json

To debug spiders with Visual Studio Code you can use the following ``launch.json``::

    {
        "version": "0.1.0",
        "configurations": [
            {
                "name": "Python: Launch Scrapy Spider",
                "type": "python",
                "request": "launch",
                "module": "scrapy",
                "args": [
                    "runspider",
                    "${file}"
                ],
                "console": "integratedTerminal"
            }
        ]
    }


Also, make sure you enable "User Uncaught Exceptions", to catch exceptions in
your Scrapy spider.
