171 lines
5.5 KiB
Markdown
171 lines
5.5 KiB
Markdown
# Development
|
|
|
|
Here you'll find a contributing guide to get started with development.
|
|
|
|
## Environment
|
|
|
|
For local development, it is required to have Python 3.10 (or a later version) installed.
|
|
|
|
We use [uv](https://docs.astral.sh/uv/) for project management. Install it and set up your IDE accordingly.
|
|
|
|
We use [Poe the Poet](https://poethepoet.natn.io/) as a task runner, similar to npm scripts in `package.json`.
|
|
All tasks are defined in `pyproject.toml` under `[tool.poe.tasks]` and can be run with `uv run poe <task>`.
|
|
|
|
### Available tasks
|
|
|
|
| Task | Description |
|
|
| ---- | ----------- |
|
|
| `install-dev` | Install development dependencies |
|
|
| `check-code` | Run lint, type-check, and unit-tests |
|
|
| `lint` | Run linter |
|
|
| `format` | Fix lint issues and format code |
|
|
| `type-check` | Run type checker |
|
|
| `unit-tests` | Run unit tests |
|
|
| `unit-tests-cov` | Run unit tests with coverage |
|
|
| `e2e-templates-tests` | Run end-to-end template tests |
|
|
| `build-docs` | Build documentation website |
|
|
| `run-docs` | Run documentation website locally |
|
|
| `build` | Build package |
|
|
| `clean` | Remove build artifacts and clean caches |
|
|
|
|
## Dependencies
|
|
|
|
To install this package and its development dependencies, run:
|
|
|
|
```sh
|
|
uv run poe install-dev
|
|
```
|
|
|
|
## Code checking
|
|
|
|
To execute all code checking tools together, run:
|
|
|
|
```sh
|
|
uv run poe check-code
|
|
```
|
|
|
|
### Linting
|
|
|
|
We utilize [ruff](https://docs.astral.sh/ruff/) for linting, which analyzes code for potential issues and enforces consistent style. Refer to `pyproject.toml` for configuration details.
|
|
|
|
To run linting:
|
|
|
|
```sh
|
|
uv run poe lint
|
|
```
|
|
|
|
### Formatting
|
|
|
|
Our automated code formatting also leverages [ruff](https://docs.astral.sh/ruff/), ensuring uniform style and addressing fixable linting issues. Configuration specifics are outlined in `pyproject.toml`.
|
|
|
|
To run formatting:
|
|
|
|
```sh
|
|
uv run poe format
|
|
```
|
|
|
|
### Type checking
|
|
|
|
Type checking is handled by [ty](https://docs.astral.sh/ty/), verifying code against type annotations. Configuration settings can be found in `pyproject.toml`.
|
|
|
|
To run type checking:
|
|
|
|
```sh
|
|
uv run poe type-check
|
|
```
|
|
|
|
### Unit tests
|
|
|
|
We use [pytest](https://docs.pytest.org/) as a testing framework with many plugins. Check `pyproject.toml` for configuration details and installed plugins.
|
|
|
|
To run unit tests:
|
|
|
|
```sh
|
|
uv run poe unit-tests
|
|
```
|
|
|
|
To run unit tests with coverage report:
|
|
|
|
```sh
|
|
uv run poe unit-tests-cov
|
|
```
|
|
|
|
## End-to-end tests
|
|
|
|
Prerequisites:
|
|
|
|
- [apify-cli](https://docs.apify.com/cli/docs/installation) installed and available in `PATH`
|
|
- Set `APIFY_TEST_USER_API_TOKEN` to your [Apify API token](https://docs.apify.com/platform/integrations/api#api-token)
|
|
|
|
To run end-to-end tests:
|
|
|
|
```sh
|
|
uv run poe e2e-templates-tests
|
|
```
|
|
|
|
## Documentation
|
|
|
|
We follow the [Google docstring format](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) for code documentation. All user-facing classes and functions must be documented. Documentation standards are enforced using [Ruff](https://docs.astral.sh/ruff/).
|
|
|
|
Our API documentation is generated from these docstrings using [pydoc-markdown](https://pypi.org/project/pydoc-markdown/) with custom post-processing. Additional content is provided through markdown files in the `docs/` directory. The final documentation is rendered using [Docusaurus](https://docusaurus.io/) and published to GitHub Pages.
|
|
|
|
To run the documentation locally, ensure you have `Node.js` 20+ installed, then run:
|
|
|
|
```sh
|
|
uv run poe run-docs
|
|
```
|
|
|
|
## Commits
|
|
|
|
We use [Conventional Commits](https://www.conventionalcommits.org/) format for commit messages. This convention is used to automatically determine version bumps during the release process.
|
|
|
|
### Available commit types
|
|
|
|
| Type | Description |
|
|
| ---- | ----------- |
|
|
| `feat` | A new feature |
|
|
| `fix` | A bug fix |
|
|
| `docs` | Documentation only changes |
|
|
| `style` | Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc) |
|
|
| `refactor` | A code change that neither fixes a bug nor adds a feature |
|
|
| `perf` | A code change that improves performance |
|
|
| `test` | Adding missing tests or correcting existing tests |
|
|
| `build` | Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm) |
|
|
| `ci` | Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs) |
|
|
| `chore` | Other changes that don't modify src or test files |
|
|
| `revert` | Reverts a previous commit |
|
|
|
|
## Release process
|
|
|
|
Publishing new versions to [PyPI](https://pypi.org/project/crawlee) is automated through GitHub Actions.
|
|
|
|
- **Beta releases**: On each commit to the master branch, a new beta release is automatically published. The version number is determined based on the latest release and conventional commits. The beta version suffix is incremented by 1 from the last beta release on PyPI.
|
|
- **Stable releases**: A stable version release may be created by triggering the `release` GitHub Actions workflow. The version number is determined based on the latest release and conventional commits (`auto` release type), or it may be overridden using the `custom` release type.
|
|
|
|
### Publishing to PyPI manually
|
|
|
|
1. **Do not do this unless absolutely necessary.** In all conceivable scenarios, you should use the `release` workflow instead.
|
|
2. **Make sure you know what you're doing.**
|
|
|
|
3. Update the version number:
|
|
|
|
- Modify the `version` field under `project` in `pyproject.toml`.
|
|
|
|
```toml
|
|
[project]
|
|
name = "crawlee"
|
|
version = "x.z.y"
|
|
```
|
|
|
|
4. Build the package:
|
|
|
|
```sh
|
|
uv run poe build
|
|
```
|
|
|
|
5. Upload to PyPI:
|
|
|
|
```sh
|
|
uv publish --token YOUR_API_TOKEN
|
|
```
|