# Project architecture

This document describes how the project is architectured, both regarding boilerplate and actual code. We start by giving an overview of the project's contents:

```python exec="1" session="filetree"
from fnmatch import fnmatch
from pathlib import Path

exclude = {"dist", "*cache*", ".devbox", ".hypothesis", ".pdm*", ".coverage*", "profile.*", ".gitpod*"}
no_recurse = {".venv*", "site", "htmlcov", ".git", "fixtures"}

descriptions = {
    ".github": "GitHub workflows, issue templates and other configuration.",
    ".venv": "The default virtual environment (git-ignored). See [`make setup`][command-setup] command.",
    ".venvs": "The virtual environments for all supported Python versions (git-ignored). See [`make setup`][command-setup] command.",
    ".vscode": "The configuration for VSCode (git-ignored). See [`make vscode`][command-vscode] command.",
    "docs": "Documentation sources (Markdown pages). See [`make docs`][task-docs] task.",
    "docs/.overrides": "Customization of [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/)' templates.",
    "docs/reference/api": "Python API reference, injected with [mkdocstrings](https://mkdocstrings.github.io/).",
    "config": "Contains our tooling configuration. See [Scripts, configuration](#scripts-configuration).",
    "htmlcov": "HTML report for Python code coverage (git-ignored), integrated in the [Coverage report](../coverage/) page. See [`make coverage`][task-coverage] task.",
    "scripts": "Our different scripts. See [Scripts, configuration](#scripts-configuration).",
    "site": "Documentation site, built with `make run mkdocs build` (git-ignored).",
    "src": "The source of our Python package(s). See [Sources](#sources) and [Program structure](#program-structure).",
    "src/griffe": "Our public API, exposed to users. See [Program structure](#program-structure).",
    "packages/griffelib/src/griffe/_internal": "Our internal API, hidden from users. See [Program structure](#program-structure).",
    "packages/griffecli/tests": "Our test suite. See [Tests](#tests).",
    "packages/griffelib/tests": "Our test suite. See [Tests](#tests).",
    ".copier-answers.yml": "The answers file generated by [Copier](https://copier.readthedocs.io/en/stable/). See [Boilerplate](#boilerplate).",
    "devdeps.txt": "Our development dependencies specification. See [`make setup`][command-setup] command.",
    "duties.py": "Our project tasks, written with [duty](https://pawamoy.github.io/duty). See [Tasks][tasks].",
    ".envrc": "The environment configuration, automatically sourced by [direnv](https://direnv.net/). See [commands](../commands/).",
    "Makefile": "A dummy makefile, only there for auto-completion. See [commands](../commands/).",
    "mkdocs.yml": "The build configuration for our docs. See [`make docs`][task-docs] task.",
    "pyproject.toml": "The project metadata and production dependencies.",
}

def exptree(path):
    files = []
    dirs = []
    for node in Path(path).iterdir():
        if any(fnmatch(node.name, pattern) for pattern in exclude):
            continue
        if node.is_dir():
            dirs.append(node)
        else:
            files.append(node)

    annotated = []
    recurse = set()
    print("```tree")
    for directory in sorted(dirs):
        if not any(fnmatch(directory.name, pattern) for pattern in no_recurse):
            recurse.add(directory)
            annotated.append(directory)
            print(f"{directory.name}/ # ({len(annotated)})!")
        elif str(directory) in descriptions:
            annotated.append(directory)
            print(f"{directory.name}/ # ({len(annotated)})!")
        else:
            print(f"{directory.name}/")
    for file in sorted(files):
        if str(file) in descriptions:
            annotated.append(file)
            print(f"{file.name} # ({len(annotated)})!")
        else:
            print(file.name)
    print("```\n")

    for index, node in enumerate(annotated, 1):
        print(f"{index}. {descriptions.get(str(node), '')}\n")
        if node.is_dir() and node in recurse:
            print('    ```python exec="1" session="filetree" idprefix=""')
            print(f'    exptree("{node.as_posix()}")')
            print("    ```\n")
```

```python exec="1" session="filetree" idprefix="" id="filetree"
exptree(".")
```

## Boilerplate

This project's skeleton (the file-tree shown above) is actually generated from a [Copier](https://copier.readthedocs.io/en/stable/) template called [copier-uv](https://pawamoy.github.io/copier-uv/). When generating the project, Copier asks a series of questions (configured by the template itself), and the answers are used to render the file and directory names, as well as the file contents. Copier also records answers in the `.copier-answers.yml` file, allowing to update the project with latest changes from the template while reusing previous answers.

To update the project (in order to apply latest changes from the template), we use the following command:

```bash
copier update --trust --skip-answered
```

## Scripts, configuration

We have a few scripts that let us manage the various maintenance aspects for this project. The entry-point is the `make` script located in the `scripts` folder. It doesn't need any dependency to be installed to run. See [Management commands](commands.md) for more information.

The `make` script can also invoke what we call "[tasks][]". Tasks need our development dependencies to be installed to run. These tasks are written in the `duties.py` file, and the development dependencies are listed in `devdeps.txt`.

The tools used in tasks have their configuration files stored in the `config` folder, to unclutter the root of the repository. The tasks take care of calling the tools with the right options to locate their respective configuration files.

## Sources

Sources are located in the `packages/` subfolders, following the [src-layout](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/).
We use [Hatch](https://hatch.pypa.io/latest/) to build source and wheel distributions, and configure it in `pyproject.toml`.

## Tests

Our test suite is located in the `tests` folder. It is located outside of the sources as to not pollute distributions (it would be very wrong to publish a `tests` package as part of our distributions, since this name is extremely common), or worse, the public API. The `tests` folder is however included in our source distributions (`.tar.gz`), alongside most of our metadata and configuration files. Check out our `pyproject.toml` files to get the full list of files included in our source distributions for every individual package within the `packages/` folder.

The test suite is based on [pytest](https://docs.pytest.org/en/8.2.x/). Test modules reflect our internal API structure, and except for a few test modules that test specific aspects of our API, each test module tests the logic from the corresponding module in the internal API. For example, `test_finder.py` tests code of the `griffe._internal.finder` internal module, while `test_functions` tests our ability to extract correct information from function signatures, statically. The general rule of thumb when writing new tests is to mirror the internal API. If a test touches to many aspects of the loading process, it can be added to the `test_loader` test module.

## Program structure

Griffe is split into two pieces: the library and the CLI.

Each of them has an internal API contained within an `_internal` folder:

- `packages/griffelib/src/griffe/_internal` for the library,
- `packages/griffecli/src/griffecli/_internal` for the CLI.

Griffe can be installed in library-only mode, which means that the CLI package from `packages/griffecli` is not present.
Library-only mode can be preferred if the user does not utilize the CLI functionality of Griffe and does not want to incorporate its dependencies.

The top-level `packages/griffelib/src/griffe/__init__.py` module exposes all the public API available: it always re-exports internal objects from various submodules of `griffe._internal` and, if the CLI is installed, it re-exports the public API of `griffecli` as well.

Users then import `griffe` directly, or import objects from it. If they don't have `griffecli` installed, they cannot import the CLI-related functionality,
such as [`griffecli.check`][].

We'll be honest: our code organization is not the most elegant, but it works :shrug: Have a look at the following module dependency graph, which will basically tell you nothing except that we have a lot of inter-module dependencies. Arrows read as "imports from". The code base is generally pleasant to work with though.

```python exec="true" html="true" id="pydeps-module-graph"
import os
if os.getenv("DEPLOY") == "true":
    from pydeps import cli, colors, dot, py2depgraph
    from pydeps.pydeps import depgraph_to_dotsrc
    from pydeps.target import Target

    cli.verbose = cli._not_verbose
    options = cli.parse_args(["packages/griffelib/src/griffe", "--noshow", "--reverse"])
    colors.START_COLOR = 128
    target = Target(options["fname"])
    with target.chdir_work():
        dep_graph = py2depgraph.py2dep(target, **options)
    dot_src = depgraph_to_dotsrc(target, dep_graph, **options)
    svg = dot.call_graphviz_dot(dot_src, "svg").decode()
    svg = "".join(svg.splitlines()[6:])
    svg = svg.replace('fill="white"', 'fill="transparent"')
    print(f'<div class="interactiveSVG pydeps">{svg}</div>')
```

<small><i>You can zoom and pan all diagrams on this page with mouse inputs.</i></small>

The following sections are generated automatically by iterating on the modules of our public and internal APIs respectively, and extracting the comment blocks at the top of each module. The comment blocks are addressed to readers of the code (maintainers, contributors), while module docstrings are addressed to users of the API. Module docstrings in our internal API are never written, because our [module layout][module-layout] is hidden, and therefore modules aren't part of the public API, so it doesn't make much sense to write "user documentation" in them.

```python exec="1" session="comment_blocks"
--8<-- "scripts/gen_structure_docs.py"
```

### CLI entrypoint

```python exec="1" idprefix="entrypoint-" session="comment_blocks" id="entrypoint"
render_entrypoint(heading_level=4)
```

### Public API

```python exec="1" idprefix="public-" session="comment_blocks" id="public-api"
render_public_api(heading_level=4)
```

### Internal API

```python exec="1" idprefix="internal-" session="comment_blocks" id="internal-api"
import os
if os.getenv("DEPLOY") == "true":
    render_internal_api(heading_level=4)
```

<style>
    .interactiveSVG svg {
        min-height: 200px;
    }
    .graph > polygon {
        fill-opacity: 0.0;
    }

    /* pydeps dependency graph. */
    [data-md-color-scheme="default"] .pydeps .edge > path,
    [data-md-color-scheme="default"] .pydeps .edge > polygon {
        stroke: black;
    }

    [data-md-color-scheme="slate"] .pydeps .edge > path,
    [data-md-color-scheme="slate"] .pydeps .edge > polygon {
        stroke: white;
    }


    /* Code2Flow call graphs. */
    [data-md-color-scheme="default"] .code2flow .cluster > polygon {
        stroke: black;
    }
    [data-md-color-scheme="default"] .code2flow .cluster > text {
        fill: black;
    }

    [data-md-color-scheme="slate"] .code2flow .cluster > polygon {
        stroke: white;
    }
    [data-md-color-scheme="slate"] .code2flow .cluster > text {
        fill: white;
    }
</style>

<script>
    document.addEventListener("DOMContentLoaded", function(){
        const divs = document.getElementsByClassName("interactiveSVG");
        for (let i = 0; i < divs.length; i++) {
            if (!divs[i].firstElementChild.id) {
                divs[i].firstElementChild.id = `interactiveSVG-${i}`
            }
            svgPanZoom(`#${divs[i].firstElementChild.id}`, {});
        }
    });
</script>
