8.9 KiB

Raw Permalink Blame History

AgentOps Deployment Infrastructure

Multi-tenant Kubernetes deployment system for CrewAI agents with ALB ingress routing.

Architecture Overview

Kubernetes Cluster: EKS cluster running agent deployments
Application Load Balancer: Shared ALB with host-based routing for multi-tenant isolation
Instance Files: ConfigMap-based deployment with content hashing
Container Registry: ECR for storing built agent images
Build System: Kaniko-based image building with S3 cache in dedicated builder namespace

Load Balancer Setup

The deployment system uses AWS Application Load Balancer (ALB) with the AWS Load Balancer Controller for internet-facing traffic routing.

Architecture

Internet → ALB (host-based routing) → Kubernetes Services → Pods

Each project gets its own hostname: {project-uuid}.deploy.agentops.ai
Shared ALB: All deployments use a single ALB instance (alb-deployments group) providing one IP address for wildcard DNS routing
ALB routes traffic based on hostname to the correct service using host-based routing
Services use ClusterIP type (not LoadBalancer) for cost efficiency
ALB Ingress Controller manages ALB lifecycle automatically
Deployments are tagged with deployment and hostname labels for routing

Prerequisites

EKS Cluster with appropriate IAM permissions
AWS CLI configured with ELB permissions
Domain configured for wildcard DNS (*.deploy.agentops.ai)
Subnets tagged with kubernetes.io/role/elb=1 for ALB creation
AWS credentials available via aws-credentials secret in kube-system namespace
Builder namespace configured for image building operations

Step 1: Install AWS Load Balancer Controller

Install the AWS Load Balancer Controller to manage ALBs:

# Install cert-manager (required dependency)
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml

# Wait for cert-manager to be ready
kubectl wait --for=condition=ready pod -l app=cert-manager -n cert-manager --timeout=60s

# Install required CRDs for ALB controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.2/config/crd/bases/elbv2.k8s.aws_targetgroupbindings.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.2/config/crd/bases/elbv2.k8s.aws_ingressclassparams.yaml

# Copy AWS credentials to kube-system namespace
kubectl get secret aws-credentials -n default -o yaml | sed 's/namespace: default/namespace: kube-system/' | kubectl apply -f -

# Apply the ALB controller and webhook configurations
kubectl apply -f configs/aws-load-balancer-controller.yaml
kubectl apply -f configs/aws-load-balancer-webhook.yaml

# Tag public subnets for ALB creation (replace with your subnet IDs)
aws ec2 create-tags --resources subnet-07bb9663fa1d8f572 subnet-0b0bc494208f4d020 --tags Key=kubernetes.io/role/elb,Value=1

Step 2: Configure Services

Services should use ClusterIP type to work with ALB ingress:

apiVersion: v1
kind: Service
metadata:
  name: svc-{project-uuid}
spec:
  type: ClusterIP  # Important: NOT LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: {project-uuid}

Step 3: Ingress Configuration

The deployment system automatically creates ingress resources with proper ALB annotations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ing-{project-uuid}
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/group.name: group-ing-{project-uuid}  # Ensures separate ALB per project
spec:
  rules:
  - host: {project-uuid}.deploy.agentops.ai
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: svc-{project-uuid}
            port:
              number: 80

Key Features

Project Isolation: Each project gets its own ALB group for complete isolation
Host-based Routing: Traffic routed by hostname to correct service
SSL Termination: ALB handles HTTPS with automatic redirects
Health Checks: ALB performs HTTP health checks on /health endpoint
Cost Effective: One ALB per project vs individual Classic ELBs

Instance Files Management

Instance files (server.py, providers/) are managed via Kubernetes ConfigMaps with content hashing.

How It Works

Content Hashing: Files are hashed to create unique ConfigMap names
Tar Archive: All instance files packaged into base64-encoded tar.gz
ConfigMap: Tar archive stored in ConfigMap with hash-based name
Build Process: ConfigMap mounted to builder and extracted into image
Immutable Deployments: Hash changes force new ConfigMap creation

Benefits

No Stale Files: Hash-based naming prevents stale file reuse
Fast Updates: Only changed files create new ConfigMaps
Version Control: Each deployment gets exactly the files it was built with
No External Dependencies: No EFS or external storage required

Deployment Process

The deployment system automatically handles ALB creation:

Service Creation: Creates ClusterIP service for each deployment
Ingress Creation: Creates ingress with unique group name for ALB isolation
ALB Provisioning: AWS Load Balancer Controller creates dedicated ALB
DNS Configuration: Update DNS to point hostname to ALB IP addresses

Verifying ALB Creation

# Check ALB controller status
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

# Check ingress status (should show ALB address when ready)
kubectl get ingress

# Check ingress events for troubleshooting
kubectl describe ingress ing-{project-uuid}

# List ALBs via AWS CLI
aws elbv2 describe-load-balancers --query 'LoadBalancers[*].[LoadBalancerName,DNSName,State.Code]' --output table

# Get ALB IP addresses for /etc/hosts
nslookup {alb-dns-name}

Troubleshooting

Common issues and solutions:

Subnet discovery failed: Ensure public subnets are tagged with kubernetes.io/role/elb=1
No credentials: Verify aws-credentials secret exists in kube-system namespace
Certificate errors: Use HTTP-only ingress or configure SSL certificates
Permission errors: Check ALB controller logs and ensure all CRDs are installed

Environment Configuration

Builder Namespace

The deployment system uses a dedicated namespace for image building operations to simplify management and logging.

Environment Variable: BUILDER_NAMESPACE (default: builder)

All Kaniko builder jobs run in this namespace, which allows for:

Centralized builder pod management
Simplified logging with jockey logs-builder
Cleaner separation between build and deployment concerns
Easy builder resource monitoring

Setup Builder Namespace

Use the built-in command to create the builder namespace and AWS credentials:

# Create builder namespace and AWS credentials secret
jockey setup-builder

# View builder logs (automatically uses builder namespace)
jockey logs-builder

# Or specify a different namespace
jockey logs-builder --namespace custom-builder

The builder namespace requires the aws-credentials secret to authenticate with ECR for image pushing.

Shared ALB Management

The deployment system uses a shared ALB (alb-deployments group) for all deployments to enable wildcard DNS routing with a single IP address. This allows *.deploy.agentops.ai to point to one ALB that handles all subdomain routing. All deployments automatically use the shared ALB configuration.

ALB Configuration

Group Name: alb-deployments (configurable via ALB_SHARED_GROUP_NAME environment variable)
Host-based Routing: Each deployment gets its own hostname for traffic routing
Tagging: Each ingress is tagged with deployment and hostname for identification
Health Checks: Configured to use /health endpoint with fast deregistration (30s)

Management Commands

# Check the status of the shared ALB
jockey alb-status

# List all ingresses using the shared ALB
jockey list-alb-ingresses --namespace default

# Validate ALB routing for a specific hostname
jockey validate-alb-routing yourdomain.deploy.agentops.ai

Environment Variables

ALB_SHARED_GROUP_NAME: Name of the shared ALB group (default: alb-deployments)
DEPLOYMENT_DOMAIN: Base domain for deployments (default: deploy.agentops.ai)

The shared ALB is automatically provisioned when the first ingress is created and will handle all subsequent deployments through host-based routing. This provides a single IP address that can be used for wildcard DNS records (*.deploy.agentops.ai), simplifying DNS management and enabling seamless subdomain routing.

8.9 KiB Raw Permalink Blame History