102 lines
6.3 KiB
Plaintext
102 lines
6.3 KiB
Plaintext
---
|
|
id: deployment
|
|
title: Running your crawler in the Cloud
|
|
sidebar_label: Running in the Cloud
|
|
description: Deploying Crawlee-python projects to the Apify platform
|
|
---
|
|
|
|
import CodeBlock from '@theme/CodeBlock';
|
|
import MainExample from '!!raw-loader!./code_examples/09_apify_sdk.py';
|
|
|
|
## Apify platform
|
|
|
|
Crawlee is developed by [**Apify**](https://apify.com), the web scraping and automation platform. You could say it is the **home of Crawlee projects**. In this section you'll see how to deploy the crawler there with just a few simple steps. You can deploy a **Crawlee** project wherever you want, but using the [**Apify platform**](https://console.apify.com) will give you the best experience.
|
|
|
|
{/*In case you want to deploy your Crawlee project to other platforms, check out the [**Deployment**](../deployment) section.*/}
|
|
|
|
With a few simple steps, you can convert your Crawlee project into a so-called **Actor**. Actors are serverless micro-apps that are easy to develop, run, share, and integrate. The infra, proxies, and storages are ready to go. [Learn more about Actors](https://apify.com/actors).
|
|
|
|
{/*:::info Choosing between Crawlee CLI and Apify CLI for project setup
|
|
|
|
We started this guide by using the Crawlee CLI to bootstrap the project - it offers the basic Crawlee templates, including a ready-made `Dockerfile`. If you know you will be deploying your project to the Apify platform, you might want to start with the Apify CLI instead. It also offers several project templates, and those are all set up to be used on the Apify platform right ahead.
|
|
|
|
:::*/}
|
|
|
|
## Dependencies
|
|
|
|
Before we get started, you'll need to install two new dependencies:
|
|
|
|
- [**Apify SDK**](https://pypi.org/project/apify/), a toolkit for working with the Apify platform. This will allow us to wire the storages (e.g. [`RequestQueue`](https://docs.apify.com/sdk/python/reference/class/RequestQueue) and [`Dataset`](https://docs.apify.com/sdk/python/reference/class/Dataset)) to the Apify cloud products. The Apify SDK, like Crawlee itself, is available as a PyPI package and can be installed with any Python package manager. To install it using [pip](https://pip.pypa.io/), run:
|
|
|
|
```sh
|
|
pip install apify
|
|
```
|
|
|
|
- [**Apify CLI**](https://docs.apify.com/cli/), a command-line tool that will help us with authentication and deployment. It is a [Node.js](https://nodejs.org/) package, and can be installed using any Node.js package manager. In this guide, we will use [npm](https://npmjs.com/). We will install it globally, so you can use it across all your Crawlee and Apify projects. To install it using npm, run:
|
|
|
|
```sh
|
|
npm install -g apify-cli
|
|
```
|
|
|
|
## Logging in to the Apify platform
|
|
|
|
The next step will be [creating your Apify account](https://console.apify.com/sign-up). Don't worry, we have a **free tier**, so you can try things out before you buy in! Once you have that, it's time to log in with the just-installed [Apify CLI](https://docs.apify.com/cli/). You will need your personal access token, which you can find at https://console.apify.com/account#/integrations.
|
|
|
|
```sh
|
|
apify login
|
|
```
|
|
|
|
## Adjusting the code
|
|
|
|
Now that you have your account set up, you will need to adjust the code a tiny bit. We will use the [Apify SDK](https://docs.apify.com/sdk/python/), which will help us to wire the Crawlee storages (like the [`RequestQueue`](https://docs.apify.com/sdk/python/reference/class/RequestQueue)) to their Apify platform counterparts - otherwise Crawlee would keep things only in memory.
|
|
|
|
Open your `src/main.py` file, and wrap everything in your `main` function with the [`Actor`](https://docs.apify.com/sdk/python/reference/class/Actor) context manager. Your code should look like this:
|
|
|
|
<CodeBlock className="language-python" title="src/main.py">
|
|
{MainExample}
|
|
</CodeBlock>
|
|
|
|
The context manager will configure Crawlee to use the Apify API instead of its default memory storage interface. It also sets up few other things, like listening to the platform events via websockets. After the body is finished, it handles graceful shutdown.
|
|
|
|
:::info Understanding `async with Actor` behavior with environment variables
|
|
|
|
The [`Actor`](https://docs.apify.com/sdk/python/reference/class/Actor) context manager works conditionally based on the environment variables, namely based on the `APIFY_IS_AT_HOME` env var, which is set to `true` on the Apify platform. This means that your project will remain working the same locally, but will use the Apify API when deployed to the Apify platform.
|
|
|
|
:::
|
|
|
|
## Initializing the project
|
|
|
|
You will also need to initialize the project for Apify, to do that, use the Apify CLI again:
|
|
|
|
```sh
|
|
apify init
|
|
```
|
|
|
|
The CLI will check the project structure and guide you through the setup process. If prompted, follow the instructions and answer the questions to configure the project correctly. For more information follow the [Apify CLI documentation](https://docs.apify.com/cli/docs).
|
|
|
|
This will create a folder called `.actor`, and an `actor.json` file inside it - this file contains the configuration relevant to the Apify platform, namely the Actor name, version, build tag, and few other things. Check out the [relevant documentation](https://docs.apify.com/platform/actors/development/actor-definition/actor-json) to see all the different things you can set there up.
|
|
|
|
## Ship it!
|
|
|
|
And that's all, your project is now ready to be published on the Apify platform. You can use the Apify CLI once more to do that:
|
|
|
|
```sh
|
|
apify push
|
|
```
|
|
|
|
This command will create an archive from your project, upload it to the Apify platform and initiate a Docker build. Once finished, you will get a link to your new Actor on the platform.
|
|
|
|
## Learning more about web scraping
|
|
|
|
:::tip Explore Apify Academy Resources
|
|
|
|
If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
|
|
|
|
{/*If you want to do one more project, checkout our tutorial on building a [HackerNews scraper using Crawlee](https://blog.apify.com/crawlee-web-scraping-tutorial/).*/}
|
|
|
|
:::
|
|
|
|
## Thank you! 🎉
|
|
|
|
That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee-python) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
|