---
id: storage-clients
title: Storage clients
description: How to work with storage clients in Crawlee, including the built-in clients and how to create your own.
---

import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import CodeBlock from '@theme/CodeBlock';

import MemoryStorageClientBasicExample from '!!raw-loader!roa-loader!./code_examples/storage_clients/memory_storage_client_basic_example.py';
import FileSystemStorageClientBasicExample from '!!raw-loader!roa-loader!./code_examples/storage_clients/file_system_storage_client_basic_example.py';
import FileSystemStorageClientConfigurationExample from '!!raw-loader!roa-loader!./code_examples/storage_clients/file_system_storage_client_configuration_example.py';
import CustomStorageClientExample from '!!raw-loader!roa-loader!./code_examples/storage_clients/custom_storage_client_example.py';
import RegisteringStorageClientsExample from '!!raw-loader!roa-loader!./code_examples/storage_clients/registering_storage_clients_example.py';
import SQLStorageClientBasicExample from '!!raw-loader!roa-loader!./code_examples/storage_clients/sql_storage_client_basic_example.py';
import SQLStorageClientConfigurationExample from '!!raw-loader!./code_examples/storage_clients/sql_storage_client_configuration_example.py';
import RedisStorageClientBasicExample from '!!raw-loader!./code_examples/storage_clients/redis_storage_client_basic_example.py';
import RedisStorageClientConfigurationExample from '!!raw-loader!./code_examples/storage_clients/redis_storage_client_configuration_example.py';

Storage clients provide a unified interface for interacting with <ApiLink to="class/Dataset">`Dataset`</ApiLink>, <ApiLink to="class/KeyValueStore">`KeyValueStore`</ApiLink>, and <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink>, regardless of the underlying implementation. They handle operations like creating, reading, updating, and deleting storage instances, as well as managing data persistence and cleanup. This abstraction makes it easy to switch between different environments, such as local development and cloud production setups.

## Built-in storage clients

Crawlee provides three main storage client implementations:

- <ApiLink to="class/FileSystemStorageClient">`FileSystemStorageClient`</ApiLink> - Provides persistent file system storage with in-memory caching.
- <ApiLink to="class/MemoryStorageClient">`MemoryStorageClient`</ApiLink> - Stores data in memory with no persistence.
- <ApiLink to="class/SqlStorageClient">`SqlStorageClient`</ApiLink> - Provides persistent storage using a SQL database ([SQLite](https://sqlite.org/), [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/) or [MariaDB](https://mariadb.org/)). Requires installing the extra dependency: `crawlee[sql_sqlite]` for SQLite, `crawlee[sql_postgres]` for PostgreSQL or `crawlee[sql_mysql]` for MySQL and MariaDB.
- <ApiLink to="class/RedisStorageClient">`RedisStorageClient`</ApiLink> - Provides persistent storage using a [Redis](https://redis.io/) database v8.0+. Requires installing the extra dependency `crawlee[redis]`.
- [`ApifyStorageClient`](https://docs.apify.com/sdk/python/reference/class/ApifyStorageClient) - Manages storage on the [Apify platform](https://apify.com), implemented in the [Apify SDK](https://github.com/apify/apify-sdk-python).

```mermaid
---
config:
    class:
        hideEmptyMembersBox: true
---

classDiagram

%% ========================
%% Abstract classes
%% ========================

class StorageClient {
    <<abstract>>
}

%% ========================
%% Specific classes
%% ========================

class FileSystemStorageClient

class MemoryStorageClient

class SqlStorageClient

class RedisStorageClient

class ApifyStorageClient

%% ========================
%% Inheritance arrows
%% ========================

StorageClient --|> FileSystemStorageClient
StorageClient --|> MemoryStorageClient
StorageClient --|> SqlStorageClient
StorageClient --|> RedisStorageClient
StorageClient --|> ApifyStorageClient
```

### File system storage client

The <ApiLink to="class/FileSystemStorageClient">`FileSystemStorageClient`</ApiLink> provides persistent storage by writing data directly to the file system. It uses intelligent caching and batch processing for better performance while storing data in human-readable JSON format. This is the default storage client used by Crawlee when no other storage client is specified, making it ideal for large datasets and long-running operations where data persistence is required.

:::warning Concurrency limitation
The `FileSystemStorageClient` is not safe for concurrent access from multiple crawler processes. Use it only when running a single crawler process at a time.
:::

This storage client is ideal for large datasets, and long-running operations where data persistence is required. Data can be easily inspected and shared with other tools.

<RunnableCodeBlock className="language-python" language="python">
    {FileSystemStorageClientBasicExample}
</RunnableCodeBlock>

Configuration options for the <ApiLink to="class/FileSystemStorageClient">`FileSystemStorageClient`</ApiLink> can be set through environment variables or the <ApiLink to="class/Configuration">`Configuration`</ApiLink> class:

- **`storage_dir`** (env: `CRAWLEE_STORAGE_DIR`, default: `'./storage'`) - The root directory for all storage data.
- **`purge_on_start`** (env: `CRAWLEE_PURGE_ON_START`, default: `True`) - Whether to purge default storages on start.

Data is stored using the following directory structure:

```text
{CRAWLEE_STORAGE_DIR}/
├── datasets/
│   └── {DATASET_NAME}/
│       ├── __metadata__.json
│       ├── 000000001.json
│       └── 000000002.json
├── key_value_stores/
│   └── {KVS_NAME}/
│       ├── __metadata__.json
│       ├── key1.json
│       ├── key2.txt
│       └── key3.json
└── request_queues/
    └── {RQ_NAME}/
        ├── __metadata__.json
        ├── {REQUEST_ID_1}.json
        └── {REQUEST_ID_2}.json
```

Where:
- `{CRAWLEE_STORAGE_DIR}` - The root directory for local storage.
- `{DATASET_NAME}`, `{KVS_NAME}`, `{RQ_NAME}` - The unique names for each storage instance (defaults to `"default"`).
- Files are stored directly without additional metadata files for simpler structure.

Here is an example of how to configure the <ApiLink to="class/FileSystemStorageClient">`FileSystemStorageClient`</ApiLink>:

<RunnableCodeBlock className="language-python" language="python">
    {FileSystemStorageClientConfigurationExample}
</RunnableCodeBlock>

### Memory storage client

The <ApiLink to="class/MemoryStorageClient">`MemoryStorageClient`</ApiLink> stores all data in memory using Python data structures. It provides fast access but does not persist data between runs, meaning all data is lost when the program terminates. This storage client is primarily suitable for testing and development, and is usually not a good fit for production use. However, in some cases where speed is prioritized over persistence, it can make sense.

:::warning Persistence limitation
The `MemoryStorageClient` does not persist data between runs. All data is lost when the program terminates.
:::

<RunnableCodeBlock className="language-python" language="python">
    {MemoryStorageClientBasicExample}
</RunnableCodeBlock>

### SQL storage client

:::warning Experimental feature
The `SqlStorageClient` is experimental. Its API and behavior may change in future releases.
:::

The <ApiLink to="class/SqlStorageClient">`SqlStorageClient`</ApiLink> provides persistent storage using a SQL database (SQLite by default, or PostgreSQL, MySQL, MariaDB). It supports all Crawlee storage types and enables concurrent access from multiple independent clients or processes.

:::note dependencies
The <ApiLink to="class/SqlStorageClient">`SqlStorageClient`</ApiLink> is not included in the core Crawlee package.
To use it, you need to install Crawlee with the appropriate extra dependency:

- For SQLite support, run:
  <code>pip install 'crawlee[sql_sqlite]'</code>
- For PostgreSQL support, run:
  <code>pip install 'crawlee[sql_postgres]'</code>
- For MySQL or MariaDB support, run:
  <code>pip install 'crawlee[sql_mysql]'</code>
:::

By default, <ApiLink to="class/SqlStorageClient">SqlStorageClient</ApiLink> uses SQLite.
To use a different database, just provide the appropriate connection string via the `connection_string` parameter. No other code changes are needed—the same client works for all supported databases.

<RunnableCodeBlock className="language-python" language="python">
    {SQLStorageClientBasicExample}
</RunnableCodeBlock>

Data is organized in relational tables. Below are the main tables and columns used for each storage type:

```mermaid
---
config:
    class:
        hideEmptyMembersBox: true
---

classDiagram

%% ========================
%% Storage Clients
%% ========================

class SqlDatasetClient {
    <<Dataset>>
}

class SqlKeyValueStoreClient {
    <<Key-value store>>
}

%% ========================
%% Dataset Tables
%% ========================

class datasets {
    <<table>>
    + dataset_id (PK)
    + internal_name
    + name
    + accessed_at
    + created_at
    + modified_at
    + item_count
    + buffer_locked_until
}

class dataset_records {
    <<table>>
    + item_id (PK)
    + dataset_id (FK)
    + data
}

class dataset_metadata_buffer {
    <<table>>
    + id (PK)
    + accessed_at
    + modified_at
    + delta_item_count
}

%% ========================
%% Key-Value Store Tables
%% ========================

class key_value_stores {
    <<table>>
    + key_value_store_id (PK)
    + internal_name
    + name
    + accessed_at
    + created_at
    + modified_at
    + buffer_locked_until
}

class key_value_store_records {
    <<table>>
    + key_value_store_id (FK, PK)
    + key (PK)
    + value
    + content_type
    + size
}

class key_value_store_metadata_buffer {
    <<table>>
    + id (PK)
    + accessed_at
    + modified_at
}

%% ========================
%% Client to Table arrows
%% ========================

SqlDatasetClient --> datasets
SqlDatasetClient --> dataset_records
SqlDatasetClient --> dataset_metadata_buffer

SqlKeyValueStoreClient --> key_value_stores
SqlKeyValueStoreClient --> key_value_store_records
SqlKeyValueStoreClient --> key_value_store_metadata_buffer
```
```mermaid
---
config:
    class:
        hideEmptyMembersBox: true
---

classDiagram

%% ========================
%% Storage Clients
%% ========================

class SqlRequestQueueClient {
    <<Request queue>>
}

%% ========================
%% Request Queue Tables
%% ========================

class request_queues {
    <<table>>
    + request_queue_id (PK)
    + internal_name
    + name
    + accessed_at
    + created_at
    + modified_at
    + had_multiple_clients
    + handled_request_count
    + pending_request_count
    + total_request_count
    + buffer_locked_until
}

class request_queue_records {
    <<table>>
    + request_id (PK)
    + request_queue_id (FK, PK)
    + data
    + sequence_number
    + is_handled
    + time_blocked_until
    + client_key
}

class request_queue_state {
    <<table>>
    + request_queue_id (FK, PK)
    + sequence_counter
    + forefront_sequence_counter
}

class request_queue_metadata_buffer {
    <<table>>
    + id (PK)
    + accessed_at
    + modified_at
    + client_id
    + delta_handled_count
    + delta_pending_count
    + delta_total_count
    + need_recalc
}

%% ========================
%% Client to Table arrows
%% ========================

SqlRequestQueueClient --> request_queues
SqlRequestQueueClient --> request_queue_records
SqlRequestQueueClient --> request_queue_state
SqlRequestQueueClient --> request_queue_metadata_buffer
```

Configuration options for the <ApiLink to="class/SqlStorageClient">`SqlStorageClient`</ApiLink> can be set through environment variables or the <ApiLink to="class/Configuration">`Configuration`</ApiLink> class:

- **`storage_dir`** (env: `CRAWLEE_STORAGE_DIR`, default: `'./storage'`) - The root directory where the default SQLite database will be created if no connection string is provided.
- **`purge_on_start`** (env: `CRAWLEE_PURGE_ON_START`, default: `True`) - Whether to purge default storages on start.

Configuration options for the <ApiLink to="class/SqlStorageClient">`SqlStorageClient`</ApiLink> can be set via constructor arguments:

- **`connection_string`** (default: SQLite in <ApiLink to="class/Configuration">`Configuration`</ApiLink> storage dir) - SQLAlchemy connection string, e.g. `sqlite+aiosqlite:///my.db`, `postgresql+asyncpg://user:pass@host/db`, `mysql+aiomysql://user:pass@host/db` or `mariadb+aiomysql://user:pass@host/db`.
- **`engine`** - Pre-configured SQLAlchemy AsyncEngine (optional).

For advanced scenarios, you can configure <ApiLink to="class/SqlStorageClient">`SqlStorageClient`</ApiLink> with a custom SQLAlchemy engine and additional options via the <ApiLink to="class/Configuration">`Configuration`</ApiLink> class. This is useful, for example, when connecting to an external PostgreSQL database or customizing connection pooling.

:::warning
If you use MySQL or MariaDB, pass the `isolation_level='READ COMMITTED'` argument to `create_async_engine`. MySQL/MariaDB default to the `REPEATABLE READ` isolation level, which can cause unnecessary locking, deadlocks, or stale reads when multiple Crawlee workers access the same tables concurrently. Using `READ COMMITTED` ensures more predictable row-level locking and visibility semantics for `SqlStorageClient`.
:::

<CodeBlock className="language-python" language="python">
    {SQLStorageClientConfigurationExample}
</CodeBlock>

### Redis storage client

:::warning Experimental feature
The <ApiLink to="class/RedisStorageClient">`RedisStorageClient`</ApiLink> is experimental. Its API and behavior may change in future releases.
:::

The <ApiLink to="class/RedisStorageClient">`RedisStorageClient`</ApiLink> provides persistent storage using [Redis](https://redis.io/) database. It supports concurrent access from multiple independent clients or processes and uses Redis native data structures for efficient operations.

:::note dependencies
The <ApiLink to="class/RedisStorageClient">`RedisStorageClient`</ApiLink> is not included in the core Crawlee package.
To use it, you need to install Crawlee with the Redis extra dependency:

<code>pip install 'crawlee[redis]'</code>

Additionally, Redis version 8.0 or higher is required.
:::

:::note Redis persistence
Data persistence in Redis depends on your [database configuration](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/).
:::

The client requires either a Redis connection string or a pre-configured Redis client instance. Use a pre-configured client when you need custom Redis settings such as connection pooling, timeouts, or SSL/TLS encryption.

<CodeBlock className="language-python" language="python">
    {RedisStorageClientBasicExample}
</CodeBlock>

Data is organized using Redis key patterns. Below are the main data structures used for each storage type:

```mermaid
---
config:
    class:
        hideEmptyMembersBox: true
---

classDiagram

%% ========================
%% Storage Client
%% ========================

class RedisDatasetClient {
    <<Dataset>>
}

%% ========================
%% Dataset Keys
%% ========================

class DatasetKeys {
    datasets:[name]:items - JSON Array
    datasets:[name]:metadata - JSON Object
}

class DatasetsIndexes {
    datasets:id_to_name - Hash
    datasets:name_to_id - Hash
}

%% ========================
%% Client to Keys arrows
%% ========================

RedisDatasetClient --> DatasetKeys
RedisDatasetClient --> DatasetsIndexes
```

```mermaid
---
config:
    class:
        hideEmptyMembersBox: true
---

classDiagram

%% ========================
%% Storage Clients
%% ========================

class RedisKeyValueStoreClient {
    <<Key-value store>>
}

%% ========================
%% Key-Value Store Keys
%% ========================

class KeyValueStoreKeys {
    key_value_stores:[name]:items - Hash
    key_value_stores:[name]:metadata_items - Hash
    key_value_stores:[name]:metadata - JSON Object
}

class KeyValueStoresIndexes {
    key_value_stores:id_to_name - Hash
    key_value_stores:name_to_id - Hash
}

%% ========================
%% Client to Keys arrows
%% ========================

RedisKeyValueStoreClient --> KeyValueStoreKeys
RedisKeyValueStoreClient --> KeyValueStoresIndexes
```

```mermaid
---
config:
    class:
        hideEmptyMembersBox: true
---

classDiagram

%% ========================
%% Storage Clients
%% ========================

class RedisRequestQueueClient {
    <<Request queue>>
}

%% ========================
%% Request Queue Keys
%% ========================

class RequestQueueKeys{
    request_queues:[name]:queue - List
    request_queues:[name]:data - Hash
    request_queues:[name]:in_progress - Hash
    request_queues:[name]:added_bloom_filter - Bloom Filter | bloom queue_dedup_strategy
    request_queues:[name]:handled_bloom_filter - Bloom Filter | bloom queue_dedup_strategy
    request_queues:[name]:pending_set - Set | default queue_dedup_strategy
    request_queues:[name]:handled_set - Set | default queue_dedup_strategy
    request_queues:[name]:metadata - JSON Object
}

class RequestQueuesIndexes {
    request_queues:id_to_name - Hash
    request_queues:name_to_id - Hash
}

%% ========================
%% Client to Keys arrows
%% ========================

RedisRequestQueueClient --> RequestQueueKeys
RedisRequestQueueClient --> RequestQueuesIndexes
```

Configuration options for the <ApiLink to="class/RedisStorageClient">`RedisStorageClient`</ApiLink> can be set through environment variables or the <ApiLink to="class/Configuration">`Configuration`</ApiLink> class:

- **`purge_on_start`** (env: `CRAWLEE_PURGE_ON_START`, default: `True`) - Whether to purge default storages on start.

Configuration options for the <ApiLink to="class/RedisStorageClient">`RedisStorageClient`</ApiLink> can be set via constructor arguments:

- **`connection_string`** - Redis connection string, e.g. `redis://localhost:6379/0`.
- **`redis`** - Pre-configured Redis client instance (optional).

<CodeBlock className="language-python" language="python">
    {RedisStorageClientConfigurationExample}
</CodeBlock>

## Creating a custom storage client

A storage client consists of two parts: the storage client factory and individual storage type clients. The <ApiLink to="class/StorageClient">`StorageClient`</ApiLink> acts as a factory that creates specific clients (<ApiLink to="class/DatasetClient">`DatasetClient`</ApiLink>, <ApiLink to="class/KeyValueStoreClient">`KeyValueStoreClient`</ApiLink>, <ApiLink to="class/RequestQueueClient">`RequestQueueClient`</ApiLink>) where the actual storage logic is implemented.

Here is an example of a custom storage client that implements the <ApiLink to="class/StorageClient">`StorageClient`</ApiLink> interface:

<RunnableCodeBlock className="language-python" language="python">
    {CustomStorageClientExample}
</RunnableCodeBlock>

Custom storage clients can implement any storage logic, such as connecting to a database, using a cloud storage service, or integrating with other systems. They must implement the required methods for creating, reading, updating, and deleting data in the respective storages.

## Registering storage clients

Storage clients can be registered in multiple ways:
- **Globally** - Using the <ApiLink to="class/ServiceLocator">`ServiceLocator`</ApiLink> or passing directly to the crawler.
- **Per storage** - When opening a specific storage instance like <ApiLink to="class/Dataset">`Dataset`</ApiLink>, <ApiLink to="class/KeyValueStore">`KeyValueStore`</ApiLink>, or <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink>.

<RunnableCodeBlock className="language-python" language="python">
    {RegisteringStorageClientsExample}
</RunnableCodeBlock>

You can also register different storage clients for each storage instance, allowing you to use different backends for different storages. This is useful when you want to use a fast in-memory storage for <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> while persisting scraping results in <ApiLink to="class/Dataset">`Dataset`</ApiLink> or <ApiLink to="class/KeyValueStore">`KeyValueStore`</ApiLink>.

## Conclusion

Storage clients in Crawlee provide different backends for data storage. Use <ApiLink to="class/MemoryStorageClient">`MemoryStorageClient`</ApiLink> for testing and fast operations without persistence, or <ApiLink to="class/FileSystemStorageClient">`FileSystemStorageClient`</ApiLink> for environments where data needs to persist. You can also create custom storage clients for specialized backends by implementing the <ApiLink to="class/StorageClient">`StorageClient`</ApiLink> interface.

If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee-python) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!