Getting started with Dagster Cloud#

Welcome to Dagster Cloud!

In this guide, we'll give you everything you need to get up and running with Dagster Cloud, including:

Creating your Dagster Cloud account
Selecting a deployment type
Deploying your code
Setting up CI/CD
Setting up environment variables and secrets

Let's get started!

Step 1: Create a Dagster Cloud account and organization#

If you don't already have a Dagster Cloud account, sign up for one before continuing.

You'll also be asked to create a Dagster Cloud organization. When your organization is created, a single deployment, named prod, will also be created. Refer to the Managing deployments guide for more information about deployments in Dagster Cloud.

Step 2: Select a deployment type#

In this step, you'll select the type of deployment you want to use: Serverless or Hybrid.

Type	How it works	May be a good fit if...
Serverless	In a Serverless deployment, Dagster manages the infrastructure needed to run Dagster jobs. This means you don't need to spin up or manage any infrastructure. Refer to the Serverless deployment documentation for more info.	You want to try Dagster Cloud without a lot of time investment You don't want to focus on infrastructure management Your Dagster jobs aren't computationally expensive or memory intensive
Hybrid	In a Hybrid deployment, your Dagster code is executed in your infrastructure. The agent runs in your infrastructure, executes your code, and streams metadata about code execution and assets and jobs (for visualization) over HTTPS back to Dagster Cloud. Refer to the Hybrid deployment documentation for more info. Dagster Cloud supports a variety of agents, each with its own prerequisites for setup. Refer to the Agent documentation for more info.	You want to use your own infrastructure You can create and configure an agent in your infrastructure You can use one of the agents currently supported for Dagster Cloud

Select the deployment type you want to use and proceed to the next step. Note: If you find your deployment type isn't right for you, it can be changed at any time.

Step 3: Deploy your code#

Now that you've selected a deployment type, the next step is to tell Dagster Cloud the location of your Dagster code. In addition to this, you'll set up Continuous Integration (CI) and Continuous Deployment (CD). Once completed, Branch Deployments and continuous code location deployments will be enabled for your Dagster project.

The steps for accomplishing this vary depending on the deployment type you selected in the previous step:

Click the tab for your deployment type - Serverless or Hybrid - to view what's next.

Serverless#

For Serverless deployments, there are two ways to deploy your code to Dagster Cloud:

Start from a template - Use one of our quickstart templates to get up and running. All templates come with CI/CD already configured and will be cloned to a new GitHub repository.
Import an existing project - Import an existing GitHub repository using our GitHub integration or the dagster-cloud CLI. Note: If using the GitHub integration, Dagster Cloud will automatically set up CI/CD for you.

With GitHub

Click Select to select a template.
Sign in to your GitHub account, if prompted.
In Dagster Cloud, define the following:
- Git scope - Select the organization or account to create the new repository in.
- Repository Name - Enter a name for the new repository.
- Check the Make git repository private box to make the repository private.
When finished, click Clone and deploy.

When finished, continue to the next step.

With Gitlab

Click Select to select a template.
Sign in to your Gitlab account, if prompted.
In Dagster Cloud, define the following:
- Namespace - Select the group or account to create the new project in.
- Project - Enter a name for the new project.
- Check the Make git project private box to make the project private.
When finished, click Clone and deploy.

When finished, continue to the next step.

To import existing Dagster code, you can use Dagster's GitHub / Gitlab app or the dagster-cloud CLI.

Using the GitHub integration

Using the GitHub integration to import an existing GitHub repository also sets up CI/CD for you.

Before you get started, note that the repository must have a dagster_cloud.yaml file in order for Dagster Cloud to deploy it. This file defines the code locations in your Dagster code.

If this file doesn't currently exist, create it in the root of your repository with the following code:


# dagster_cloud.yaml

locations:
  - location_name: my_location_name # the name of the code location
    code_source:
      package_name: hackernews # the name of the python package associated with the code location

After you've committed the file to the repository, come back to Dagster Cloud to complete the import process:

Click the Import an existing project tab.
Sign in to your GitHub account, if prompted.
In Dagster Cloud, define the following:
- Git scope - Select the organization or account that the repository is in.
- Repository - Select the repository.
Click Deploy.

When finished, continue to the next step.

Using the Gitlab integration

Using the Gitlab integration to import an existing Gitlab project also sets up CI/CD for you.

Before you get started, note that the project must have a dagster_cloud.yaml file in order for Dagster Cloud to deploy it. This file defines the code locations in your Dagster code.

If this file doesn't currently exist, create it in the root of your repository with the following code:


# dagster_cloud.yaml

locations:
  - location_name: my_location_name # the name of the code location
    code_source:
      package_name: hackernews # the name of the python package associated with the code location

After you've committed the file to the project, come back to Dagster Cloud to complete the import process:

Click the Import an existing project tab.
Sign in to your Gitlab account, if prompted.
In Dagster Cloud, define the following:
- Namespace - Select the group or account that the project is in.
- Project - Select the project.
Click Deploy.

When finished, continue to the next step.

Using the dagster-cloud CLI

Using the dagster-cloud CLI requires a recent version of Python 3 and Docker.

To complete this step using the CLI, you can use your own Dagster code or the Dagster starter kit. The starter kit is a template with everything you need to get started using Serverless in Dagster Cloud, including CI/CD configuration and the required dagster_cloud.yaml file.

If using the template, clone the repository to your local environment.
If not using the template, add dagster-cloud as a dependency in setup.py. [Click here for an example](https://github.com/dagster-io/quickstart-etl/blob/main/setup.py. This is already done for you if using the starter kit.

After you've finished setting up your local project, move on to deploying using the CLI:

To install the dagster-cloud CLI, run:


pip install dagster-cloud

Next, you'll need to authenticate to Dagster Cloud. Run the following command and follow the prompts to log in:
```
dagster-cloud config setup
```

After you've successfully authenticated, run the following to deploy your code to Dagster Cloud:


dagster-cloud serverless deploy \
      --location-name "<LOCATION_NAME>" \       # name of the code loation
      --package-name "<PACKAGE_NAME>" \           # name of the Python package associated with the code location

When finished, continue to the next step.

Hybrid#

To set up Hybrid deployment and deploy your code, you'll need to:

Set up an agent
Configure CI/CD for your project

For most Hybrid deployments - with the exception of those using a local agent - you'll need to create a Docker image containing your Dagster code and then add a code location to Dagster Cloud that references the image.

Step 1: Create a Hybrid agent

The dialog that displays will contain a pre-generated agent token and details about the agents currently supported by Dagster Cloud.
Note: Keep this token handy - you'll need it to set up CI/CD in the next step.
Follow the steps for setting up and deploying your agent:

The dialog will indicate when Dagster Cloud receives an agent heartbeat. Click Continue.

Step 2: Configure CI/CD for your project

Next, you'll configure CI/CD for your Dagster project. How this is accomplished depends on your CI/CD provider:

GitHub - If using GitHub, you can use our GitHub Action workflow to set up CI/CD for your project.
Another CI/CD provider - If you're not using GitHub, you can configure CI/CD using the dagster-cloud CLI.

To set up continuous integration using GitHub Actions, you can use your own Dagster code or the Dagster Cloud Hybrid Quickstart. The quickstart is a template with everything you need to get started using Hybrid deployment in Dagster Cloud.

If using the template, clone the repository.
If not using the template, copy the GitHub workflow files (.github/workflows) from the Hybrid quickstart repository and add them to your repository. This is already done for you if using the quickstart.

In the repository, set the necessary GitHub action secrets:
- DAGSTER_CLOUD_API_TOKEN - The Dagster Cloud agent token from the previous section. Refer to the agent tokens documentation for more info.
- ORGANIZATION_ID - The organization ID of your Dagster Cloud organization, found in your account's URL. For example, if your organization is found at https://dagster.cloud/pied-piper or https://pied-piper.dagster.cloud/, this would be pied-piper.
Refer to the GitHub docs for more info about GitHub Action secrets.
In both workflow files, uncomment the step that is relevant to your Docker container registry. For example, if using DockerHub, you'd uncomment the DockerHub step in these files.

When finished setting up the repository, navigate back to Dagster Cloud and move on to the next step.

The agent will attempt to load your code. After the code is loaded, the new code location will show a Loaded status and a sample Dagster job will display in Dagster Cloud.

When finished, continue to the next step.

Step 4: Set up environment variables and secrets#

Congrats! At this point, your Dagster Cloud deployment should be up and running. To ensure the external services you use in your Dagster project work correctly, start setting up your environment variables. Using environment variables, you can securely pass in sensitive info like passwords, API tokens, etc.

Next steps#

From here, you can: