Building Cross-Project GCP Artifact Registries with Terraform: A Modular Approach

Managing artifact registries efficiently is critical for software development, especially when dealing with multi-project setups. Google Cloud Artifact Registry simplifies artifact storage and access, and by combining Terraform’s modularity with GCP’s capabilities, you can achieve a streamlined, cross-project artifact registry system. This blog post will guide you through setting up both virtual and physical artifact registries in GCP, enabling access across multiple projects.

Why Use Virtual and Physical Artifact Registries?

Artifact registries come in two types in GCP:

• Physical Artifact Registries: Actual repositories where artifacts (e.g., Docker images, Python packages) are stored.
• Virtual Artifact Registries: Logical repositories that aggregate multiple physical registries, providing a unified view.

This separation offers flexibility, isolation, and improved organization. Virtual registries simplify access by consolidating artifacts from multiple physical repositories into a single endpoint. When multiple projects and environments (dev, staging, prod) are involved, virtual registries shine by streamlining artifact discovery and management.

Use Case: Cross-Project Access to Artifact Registries

Imagine a scenario where you have separate GCP projects for development, staging, and production. Each project has its own artifact registries. To improve efficiency, you want your development pipelines to access shared base images or Python packages stored in other projects. Here’s how to achieve this using Terraform.

Designing the Setup

1. Separate Artifact Registries for Each Environment: Each project (dev, staging, prod) has its own Docker and Python physical artifact registries.
2. Virtual Registries for Aggregation: Virtual registries aggregate physical registries within and across projects.
3. IAM Roles for Cross-Project Access: Permissions are set to allow cross-project access to artifact registries.
4. Modular Terraform Approach: Reusable modules manage the creation of networks, IAM roles, and artifact registries.

Terraform Configuration

Variables for Flexibility

Define input variables to capture essential details like project IDs, registry names, and external repositories.

variable "project_id" {}
variable "location" {}
variable "docker_repo_name" {}
variable "python_repo_name" {}
variable "docker_virtual_repo_name" {}
variable "python_virtual_repo_name" {}
variable "external_repositories" {
  type = list(object({
    project_id = string
    repo_name  = string
    repo_type  = string
  }))
}

Artifact Registry Module

Create a Terraform module to manage artifact registries and their permissions.

resource "google_artifact_registry_repository" "docker_repo" {
  name        = var.docker_repo_name
  format      = "DOCKER"
  location    = var.location
  project     = var.project_id
  description = "Docker artifact registry"
}

resource "google_artifact_registry_repository" "python_repo" {
  name        = var.python_repo_name
  format      = "PYTHON"
  location    = var.location
  project     = var.project_id
  description = "Python artifact registry"
}

resource "google_artifact_registry_repository_iam_member" "external_access" {
  for_each = { for repo in var.external_repositories : repo.repo_name => repo }

  repository = each.value.repo_name
  role       = "roles/artifactregistry.reader"
  member     = "serviceAccount:${google_service_account.artifact_registry_sa.email}"
}

resource "google_service_account" "artifact_registry_sa" {
  account_id   = "artifact-registry-sa"
  display_name = "Artifact Registry Service Account"
}

Environment Configuration

Define environments (development, stage, production) with separate artifact registries and shared access to external repositories.

module "dev_environment" {
  source                    = "./modules/artifact_registry"
  project_id                = var.dev_project_id
  docker_repo_name          = "dev-docker-repo"
  python_repo_name          = "dev-python-repo"
  location                  = "us-central1"
  external_repositories     = [
    { project_id = "external-project", repo_name = "external-docker-repo", repo_type = "DOCKER" },
    { project_id = "external-project", repo_name = "external-python-repo", repo_type = "PYTHON" }
  ]
}

module "stg_environment" {
  source                    = "./modules/artifact_registry"
  project_id                = var.stg_project_id
  docker_repo_name          = "stg-docker-repo"
  python_repo_name          = "stg-python-repo"
  location                  = "us-central1"
  external_repositories     = [
    { project_id = "external-project", repo_name = "external-docker-repo", repo_type = "DOCKER" },
    { project_id = "external-project", repo_name = "external-python-repo", repo_type = "PYTHON" }
  ]
}

Main Terraform File

Bring everything together in the main file to define your project structure.

provider "google" {
  project = var.project_id
  region  = "us-central1"
}

module "dev" {
  source      = "./modules/environment"
  project_id  = "dev-project-id"
  location    = "us-central1"
  docker_repo_name = "dev-docker-repo"
  python_repo_name = "dev-python-repo"
}

module "staging" {
  source      = "./modules/environment"
  project_id  = "staging-project-id"
  location    = "us-central1"
  docker_repo_name = "stg-docker-repo"
  python_repo_name = "stg-python-repo"
}

module "production" {
  source      = "./modules/environment"
  project_id  = "production-project-id"
  location    = "us-central1"
  docker_repo_name = "prod-docker-repo"
  python_repo_name = "prod-python-repo"
}

How Virtual and Physical Registries Work Together

• Physical Registries store artifacts for specific environments and projects.
• Virtual Registries aggregate these physical registries to provide a unified interface for artifact access.
• IAM Roles ensure secure, cross-project access to physical registries while maintaining environment isolation.

Benefits of This Setup

1. Scalability: Add new projects or registries without disrupting existing configurations.
2. Security: IAM roles and service accounts enforce secure, controlled access.
3. Flexibility: Virtual registries aggregate artifacts across projects, streamlining access.
4. Reusability: Modular Terraform code reduces redundancy and improves maintainability.

Conclusion

Using Terraform to manage GCP artifact registries across multiple projects provides a robust, scalable, and secure solution for modern CI/CD workflows. By leveraging virtual and physical artifact registries, you can simplify access, enhance security, and improve the overall efficiency of your artifact management processes.