Automating Development, Staging, and Production Workflows with GitHub Actions and GCP

In today’s fast-paced development environments, maintaining consistency and efficiency across development, staging, and production workflows is essential. By using GitHub Actions and Google Cloud Platform (GCP), you can create a robust CI/CD pipeline that builds, tests, and deploys your application while following the DRY (Don’t Repeat Yourself) principle. This post outlines how to automate workflows for building and publishing Python packages and Docker images across dev, staging, and production environments using independent GCP projects.

The Goal

The pipeline achieves the following:

1. Development Environment:
   • Builds and tests the code when a Pull Request (PR) is opened or updated.
   • Pushes the Docker image to a development GCP project registry.
2. Staging Environment:
   • Builds, tests, and deploys the application when code is merged into the main branch.
   • Pushes the Docker image to the staging GCP project registry.
3. Production Environment:
   • Manually promotes the staging image to production after approval.
   • Deploys the Docker image to the production environment.

Environment Setup

Development

• Purpose: Validate code changes early through automated testing and Docker builds.
• GCP Project: gcp-dev-project
• Registry: gcr.io/gcp-dev-project
• Trigger: PR opened or updated.

Staging

• Purpose: Simulate the production environment for final validation.
• GCP Project: gcp-stg-project
• Registry: gcr.io/gcp-stg-project
• Trigger: Merge into main branch.

Production

• Purpose: Serve end-users with the live application.
• GCP Project: gcp-prd-project
• Registry: gcr.io/gcp-prd-project
• Trigger: Manual approval and promotion.

Workflow Definitions

Development Workflow

The development workflow focuses on validating code changes by running tests and building Docker images.

Key Steps:

1. Checkout the code from the PR.
2. Set up the Python environment and install dependencies.
3. Run unit and integration tests.
4. Build a Docker image and tag it as dev-<commit-sha>.
5. Push the image to gcr.io/gcp-dev-project.

GitHub Actions Configuration:

name: Development Workflow

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  build-and-test:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v2

    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.8'

    - name: Install dependencies
      run: pip install -r requirements.txt

    - name: Run tests
      run: pytest

    - name: Build Docker image
      run: docker build -t gcr.io/$GCP_DEV_PROJECT/my-app:dev-${{ github.sha }} .

    - name: Push Docker image
      run: |
        echo ${{ secrets.GCP_DEV_SERVICE_ACCOUNT_KEY }} | docker login -u _json_key --password-stdin https://gcr.io
        docker push gcr.io/$GCP_DEV_PROJECT/my-app:dev-${{ github.sha }}

Staging Workflow

The staging workflow ensures that the application functions as expected in a production-like environment.

Key Steps:

1. Checkout the latest code from the main branch.
2. Run tests to confirm stability.
3. Build a Docker image and tag it as stg-<commit-sha>.
4. Push the image to gcr.io/gcp-stg-project.
5. Deploy the image to the staging environment.

GitHub Actions Configuration:

name: Staging Workflow

on:
  push:
    branches:
      - main

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v2

    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.8'

    - name: Install dependencies
      run: pip install -r requirements.txt

    - name: Run tests
      run: pytest

    - name: Build Docker image
      run: docker build -t gcr.io/$GCP_STG_PROJECT/my-app:stg-${{ github.sha }} .

    - name: Push Docker image
      run: |
        echo ${{ secrets.GCP_STG_SERVICE_ACCOUNT_KEY }} | docker login -u _json_key --password-stdin https://gcr.io
        docker push gcr.io/$GCP_STG_PROJECT/my-app:stg-${{ github.sha }}

    - name: Deploy to Staging
      run: |
        gcloud auth activate-service-account --key-file=${{ secrets.GCP_STG_SERVICE_ACCOUNT_KEY }}
        gcloud config set project gcp-stg-project
        gcloud run deploy my-app --image=gcr.io/$GCP_STG_PROJECT/my-app:stg-${{ github.sha }} --region=$GCP_REGION

Production Workflow

The production workflow promotes a validated staging image to production.

Key Steps:

1. Pull the Docker image from staging.
2. Tag the image as prd-<commit-sha>.
3. Push the image to gcr.io/gcp-prd-project.
4. Deploy the image to the production environment.

GitHub Actions Configuration:

name: Production Workflow

on:
  workflow_dispatch:

jobs:
  promote-and-deploy:
    runs-on: ubuntu-latest

    steps:
    - name: Pull Staging Image
      run: docker pull gcr.io/$GCP_STG_PROJECT/my-app:stg-${{ github.sha }}

    - name: Tag Production Image
      run: docker tag gcr.io/$GCP_STG_PROJECT/my-app:stg-${{ github.sha }} gcr.io/$GCP_PRD_PROJECT/my-app:prd-${{ github.sha }}

    - name: Push Production Image
      run: |
        echo ${{ secrets.GCP_PRD_SERVICE_ACCOUNT_KEY }} | docker login -u _json_key --password-stdin https://gcr.io
        docker push gcr.io/$GCP_PRD_PROJECT/my-app:prd-${{ github.sha }}

    - name: Deploy to Production
      run: |
        gcloud auth activate-service-account --key-file=${{ secrets.GCP_PRD_SERVICE_ACCOUNT_KEY }}
        gcloud config set project gcp-prd-project
        gcloud run deploy my-app --image=gcr.io/gcp-prd-project/my-app:prd-${{ github.sha }} --region=$GCP_REGION

Benefits of This Workflow

1. Environment Isolation: Separate GCP projects ensure development, staging, and production environments do not interfere with each other.
2. Automation with Control: Automated deployments for dev and staging reduce manual errors, while manual promotion to production ensures control over critical changes.
3. Reusability: Common steps, such as Docker builds and tests, follow the DRY principle to minimize repetition.
4. Flexibility: Workflows are designed to be extensible and customizable as requirements evolve.

With this setup, you can ensure a streamlined, reliable, and scalable CI/CD pipeline that keeps your development process efficient and your production environment stable.