# ast_serialize - Python Parser and AST Serializer

This is a fast Python extension for parsing Python files and serializing the AST using 
the native binary format used by mypy. This will eventually replace the current mypy parser,
which uses the Python stdlib `ast` module for parsing.

**This is work in progress.**

## Development

Prerequisites:

- Recent Rust toolchain
- Python 3.10+ (3.13t or 3.14t for free-threaded builds)
- [maturin](https://github.com/PyO3/maturin): `pip install maturin`
- Checkout of [mypy repo](https://github.com/python/mypy) for testing

Development build (fast compilation, unoptimized):
```bash
maturin develop
```

Optimized development build:
```bash
maturin develop --release
```

## Testing

**Rust unit tests:**
```bash
cargo test
```

**Python integration tests:** 
Run end-to-end parser and serialization/deserialization tests using mypy's test suite in 
the new-parser branch:
```bash
cd ~/src/mypy  [or wherever you have mypy]
pytest mypy -k NativeParser
```

Add new test cases to `test-data/unit/native-parser.test` (in the mypy repository). See
the main parser test cases in `parse.test` for the expected output format.

**Note:** Run `maturin develop` before testing if you've modified Rust code.

Use `TEST_NATIVE_PARSER=1 pytest mypy/test/testcheck.py` to run mypy type checking 
tests using ast_serialize. Note that some tests are still skipped.

## Creating PRs

You can create PRs in this repository, and/or in mypy repo. 
Contributions are welcome! Run mypy tests (see above) to get ideas about bugs and missing 
functionality. If your contributions needs changes in both mypy and ast_serialize, please
mention this in the PR summary.

## Building Wheels

```bash
# Build all wheels (stable ABI + free-threaded)
./build_wheels.sh

# Or manually:
# Stable ABI wheel (Python 3.9+)
maturin build --release

# Free-threaded wheel (Python 3.13t)
maturin build --release --no-default-features --features free-threaded --interpreter python3.13t
```

If you see `Python 3.13t not found`, you'll need to install the free-threaded build of CPython 3.13
in PATH.

## Making a release

1. Bump the version number in `pyproject.toml` and `Cargo.toml` in this repository.
2. Update `test_ast_serialize.py` (optional but recommended if the release includes major features).
3. Commit and push (pushing directly to master is fine).
4. Wait until all [builds](https://github.com/mypyc/ast_serialize/actions) complete successfully
   (no release is triggered yet).
5. Once builds are complete, tag the release (`git tag vX.Y.Z`; `git push origin vX.Y.Z`).
6. Go to the ["Actions" tab](https://github.com/mypyc/ast_serialize/actions) and click "Build wheels"
   on the left.
7. Click "Run workflow" and pick the newly created tag from the drop-down list. This will build
   *and upload* the wheels.
8. After the workflow completes, verify that `pip install -U ast-serialize` installs the new version
   from PyPI using a compiled wheel.
9. Create a PR to update the `ast-serialize` version in `pyproject.toml` in the mypy repository.

The process should take about 15 minutes.

## Vendored Ruff crates

To facilitate packaging/distributing `ast_serialize` we vendor Ruff crates that are necessary for
Ruff parser. They are located in `crates/` directory, and their shared config is in `[workspace]`
section of `Cargo.toml`. Note we include a minimal required subset of Ruff (and thus only a subset
of its third party dependencies). The process of updating the Ruff crates is following:

1. Clone [Ruff repo](https://github.com/astral-sh/ruff) and checkout the desired tag.
2. Copy all Ruff crates specified in our `Cargo.toml`.
3. Copy versions of all `[workspace.dependencies]` from Ruff.
4. Change the versions of the Ruff crates in their `Cargo.toml` files (eg. `crates/ruff_python_ast/Cargo.toml`) to this format: `<crate-ver>-ast-serialize.<ast-serialize-ver>`, eg. `0.0.0-ast-serialize.0.8.0`.
5. If new third party dependencies are added to Ruff, copy them until `cargo test` passes.

## Using Coding Agents

This project is designed to support coding agent assisted development (such as Claude Code, Codex
or OpenCode). Notes:
 * Ensure your coding agent has access to AGENTS.md (e.g. create a CLAUDE.md symbolic link).
 * For the best experience, clone this repository as `~/src/ast_serialize`, and clone mypy
   (with the new-parser branch checked out) as `~/src/mypy`.
 * Have your mypy virtualenv active when starting the coding agent, or place the virtualenv
   at `~/venv/mypy` (this is referred to in AGENTS.md).

## Acknowledgments

This is a wrapper around the [Ruff](https://github.com/astral-sh/ruff) parser. Credits to Ruff 
maintainers for developing a fast Python parser!

## License

MIT
