# GitOps in Practice: How to Manage Kubernetes State with ArgoCD and Helm

Kubernetes configuration drift is a silent killer of operational confidence. A developer applies a quick fix directly with `kubectl` during an incident. A CI pipeline pushes an update that partially applies before timing out. An engineer edits a ConfigMap in production to test a theory and forgets to update the repository. Six weeks later, nobody is sure what is actually running, the cluster state has diverged from the repository that supposedly defines it.

**GitOps** solves this by making Git the single source of truth for cluster state. Every change, application version, configuration, replica count, ingress rule, is expressed as a commit. The cluster continuously reconciles toward that committed state. Unauthorized changes are detected and can be automatically reverted. The audit trail is your Git log.

**ArgoCD** is the most widely adopted GitOps controller for Kubernetes. Combined with **Helm** for templated application packaging, it provides a complete GitOps platform, continuous reconciliation, multi-environment promotion, RBAC-controlled access, and a UI that shows the real-time gap between desired and actual cluster state.

This guide covers how to implement a production-grade GitOps workflow with ArgoCD and Helm, from installation and Application setup through the App of Apps pattern, sync policies, secrets management, and progressive delivery.

## The GitOps Principles in Practice
Before implementation, understand the four principles GitOps enforces:

**Declarative:** the entire system state is expressed declaratively in Git. Not scripts, not imperative commands, YAML manifests and Helm values that describe the desired end state.

**Versioned and immutable:** Git history is the audit trail. Every state change is a commit with author, timestamp, and diff. Rolling back is a revert commit, not a manual procedure.

**Pulled automatically:** the cluster pulls configuration from Git, rather than CI/CD pushing configuration to the cluster. This eliminates the need for CI systems to have cluster credentials, a significant security improvement.

**Continuously reconciled:** ArgoCD continuously compares desired state (Git) against actual state (cluster) and alerts on drift. With auto-sync enabled, it corrects drift automatically.

## Step 1 - Repository Structure for GitOps
A clear repository structure is the foundation of a maintainable GitOps setup. The most common pattern separates application code from deployment configuration:

```
/gitops-repo                    ← dedicated GitOps repository
  /apps                         ← ArgoCD Application manifests
    /root-app.yaml              ← App of Apps root
    /platform/
      platform-apps.yaml        ← platform-level applications
    /teams/
      team-alpha-apps.yaml
      team-beta-apps.yaml
  /charts                       ← Helm charts for your services
    /orders-service/
      Chart.yaml
      values.yaml               ← default values
      templates/
        deployment.yaml
        service.yaml
        ingress.yaml
        hpa.yaml
  /envs                         ← environment-specific values
    /staging/
      orders-service.yaml       ← staging overrides
    /production/
      orders-service.yaml       ← production overrides
  /infrastructure               ← cluster-level resources
    /cert-manager/
    /ingress-nginx/
    /monitoring/
```

**Key design decisions:**
Separate application charts (`/charts`) from environment values (`/envs`). The chart defines the template, the environment values define the configuration. Promoting from staging to production is a values change, not a chart change.

Keep all ArgoCD Application manifests in Git (`/apps`). ArgoCD manages itself, the App of Apps pattern means ArgoCD's own configuration is version-controlled and reconciled like everything else.

## Step 2 - Installing ArgoCD
```bash
# Create ArgoCD namespace and install
kubectl create namespace argocd
kubectl apply -n argocd -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all components to be ready
kubectl wait --for=condition=available --timeout=300s \
  deployment/argocd-server -n argocd

# Retrieve initial admin password
kubectl get secret argocd-initial-admin-secret \
  -n argocd \
  -o jsonpath='{.data.password}' | base64 -d

# Port-forward the UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
```

For production installations, manage ArgoCD itself via Helm and the ArgoCD Helm chart, enabling version-controlled ArgoCD configuration and upgrades:

```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --values argocd-values.yaml
```

## Step 3 - Creating Your First ArgoCD Application
An ArgoCD `Application` is the core resource, it links a Git source to a Kubernetes destination and defines how reconciliation should behave:

```yaml
# apps/orders-service-staging.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-service-staging
  namespace: argocd
  labels:
    environment: staging
    team: backend
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # cascade delete on app removal
spec:
  project: backend-team

  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: charts/orders-service
    helm:
      valueFiles:
        - ../../envs/staging/orders-service.yaml  # relative to chart path

  destination:
    server: https://kubernetes.default.svc
    namespace: orders-staging

  syncPolicy:
    automated:
      prune: true         # delete resources removed from Git
      selfHeal: true      # revert unauthorized manual changes
      allowEmpty: false   # never sync an empty resource set (safety guard)
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - RespectIgnoreDifferences=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
```

The `selfHeal: true` option is the GitOps enforcement mechanism, if someone runs `kubectl edit deployment orders-service` directly, ArgoCD detects the drift within 3 minutes and reverts it to match Git. This is what makes Git the actual source of truth rather than a suggestion.

## Step 4 - Helm Values for Multi-Environment Configuration
Structure Helm values for environment-specific overrides:

```yaml
# charts/orders-service/values.yaml — defaults
replicaCount: 1
image:
  repository: your-org/orders-service
  tag: latest
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80
  targetPort: 3000

ingress:
  enabled: false
  host: ""

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 70

env: {}
```

```yaml
# envs/staging/orders-service.yaml — staging overrides
replicaCount: 1

image:
  tag: "2.4.1-staging"    # pinned to staging image tag

ingress:
  enabled: true
  host: orders.staging.your-domain.com

resources:
  requests:
    cpu: 50m
    memory: 64Mi

env:
  NODE_ENV: staging
  DATABASE_URL: postgresql://staging-db:5432/orders
  LOG_LEVEL: debug
```

```yaml
# envs/production/orders-service.yaml — production overrides
replicaCount: 3

image:
  tag: "2.4.1"            # pinned to production image tag

ingress:
  enabled: true
  host: orders.your-domain.com

resources:
  requests:
    cpu: 100m
    memory: 224Mi
  limits:
    cpu: 500m
    memory: 360Mi

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20

env:
  NODE_ENV: production
  LOG_LEVEL: warn
```

Image tags are pinned, never `latest` and promotion from staging to production is a single-line diff in `envs/production/orders-service.yaml`. That diff is a pull request, code-reviewed, and merged. ArgoCD detects the merge and applies the change to the production cluster automatically.

## Step 5 - The App of Apps Pattern
Managing dozens of ArgoCD `Application` resources individually doesn't scale. The **App of Apps** pattern treats the collection of applications as a managed set, a root Application that points to a directory of Application manifests, and ArgoCD manages them all:

```yaml
# apps/root-app.yaml — the root of everything
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: apps            # ArgoCD watches this directory for Application manifests

  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
```

Apply this single manifest manually once:

```bash
kubectl apply -f apps/root-app.yaml
```

ArgoCD discovers every Application manifest in the `apps/` directory and creates/manages them all. Adding a new application is adding a new YAML file to the `apps/` directory and committing. Removing an application is removing its manifest, with `prune: true`, ArgoCD cleans up the cluster resources automatically.

This is the GitOps pattern in its purest form, the cluster configuration, including its own management configuration, is fully defined in and controlled by Git.

## Step 6 - Projects and RBAC for Multi-Team Environments
ArgoCD `AppProject` resources define boundaries between teams, controlling which repositories, clusters, and namespaces each team's Applications can target:

```yaml
# infrastructure/argocd/projects/backend-team.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: backend-team
  namespace: argocd
spec:
  description: Backend engineering team applications

  # Which Git repositories this project can deploy from
  sourceRepos:
    - https://github.com/your-org/gitops-repo

  # Which clusters and namespaces this project can deploy to
  destinations:
    - server: https://kubernetes.default.svc
      namespace: orders-*     # wildcard — any namespace starting with orders-
    - server: https://kubernetes.default.svc
      namespace: users-*

  # Which Kubernetes resource types this project can manage
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace

  namespaceResourceWhitelist:
    - group: 'apps'
      kind: Deployment
    - group: ''
      kind: Service
    - group: 'networking.k8s.io'
      kind: Ingress
    - group: 'autoscaling'
      kind: HorizontalPodAutoscaler

  # Deny deploying privileged containers
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota

  roles:
    - name: developer
      description: Read-only access plus sync trigger
      policies:
        - p, proj:backend-team:developer, applications, get, backend-team/*, allow
        - p, proj:backend-team:developer, applications, sync, backend-team/*, allow
      groups:
        - backend-developers     # maps to your IdP group

    - name: lead
      description: Full access to backend team applications
      policies:
        - p, proj:backend-team:lead, applications, *, backend-team/*, allow
      groups:
        - backend-leads
```

RBAC is enforced at the ArgoCD level, developers can trigger syncs and view application state, but cannot modify Application resources, change sync policies, or deploy to namespaces outside their project's boundaries.

## Step 7 - Secrets Management with ArgoCD
ArgoCD should never store secrets in Git, even encrypted. The two recommended approaches:

### Sealed Secrets
```bash
# Install Sealed Secrets controller
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

# Encrypt a secret for Git storage
kubectl create secret generic orders-db-credentials \
  --from-literal=DATABASE_URL=postgresql://user:password@db:5432/orders \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > charts/orders-service/templates/db-secret.sealed.yaml
```

The `SealedSecret` in Git can only be decrypted by the Sealed Secrets controller running in your specific cluster, safe to commit.

### External Secrets Operator with Vault
For teams already using HashiCorp Vault (as covered in our secrets management guide):

```yaml
# charts/orders-service/templates/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: orders-service-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: orders-service-env
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: secret/production/orders-service/database
        property: url
    - secretKey: STRIPE_API_KEY
      remoteRef:
        key: secret/production/orders-service/stripe
        property: api_key
```

ArgoCD deploys the `ExternalSecret` resource, the External Secrets Operator fetches the actual values from Vault and creates the Kubernetes `Secret`. Vault credentials never appear in Git.

## Step 8 - Progressive Delivery with ArgoCD Rollouts
For production deployments where blue/green or canary strategies are required, **Argo Rollouts** extends ArgoCD with progressive delivery capabilities:

```yaml
# charts/orders-service/templates/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: orders-service
spec:
  replicas: 5
  selector:
    matchLabels:
      app: orders-service
  template:
    spec:
      containers:
        - name: orders-service
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

  strategy:
    canary:
      steps:
        - setWeight: 10           # route 10% of traffic to new version
        - pause: { duration: 5m } # wait 5 minutes — observe error rate
        - setWeight: 25
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100          # full traffic if no issues detected

      # Automatically roll back if error rate exceeds threshold
      analysis:
        templates:
          - templateName: success-rate-check
        startingStep: 1
        args:
          - name: service-name
            value: orders-service
```

```yaml
# infrastructure/argocd/analysis-templates/success-rate.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-check
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.98
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus-server.monitoring.svc.cluster.local
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))
```

The canary automatically rolls back if the success rate drops below 98% for three consecutive minutes, combining GitOps promotion (the image tag change in Git) with data-driven delivery (the rollback decision from Prometheus).

## Step 9 - Handling Sync Waves and Dependencies
Some resources must exist before others, namespaces before deployments, CRDs before custom resources, Secrets before Deployments that mount them. ArgoCD sync waves control this ordering:

```yaml
# Apply this first (wave -1) — before all other resources
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
---
# Apply this second (wave 0, default)
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"
---
# Apply this last (wave 1) — after secrets and config are ready
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"
```

Use sync waves for:
- Database migrations (`wave: 0`) before the application deployment (`wave: 1`)
- CRD installation (`wave: -1`) before custom resources (`wave: 0`)
- Secrets (`wave: -1`) before Deployments that reference them (`wave: 0`)

## Common Pitfalls to Avoid
**Using `latest` image tags.** ArgoCD reconciles to the state in Git. If `latest` resolves to a different image without a Git commit, ArgoCD won't detect it as a change and won't reflect it in the UI. Always pin image tags and update them via Git commits.

**Auto-sync without `selfHeal` in production.** Auto-sync without `selfHeal` applies changes from Git but doesn't revert unauthorized manual changes. Enable both for full GitOps enforcement in production environments.

**Storing unencrypted secrets in Git.** Even in private repositories, plaintext secrets in Git are a security risk. Use Sealed Secrets or External Secrets Operator, there is no legitimate reason for plaintext credentials in a GitOps repository.

**No sync windows for production.** Configure sync windows to restrict when ArgoCD can apply changes to production clusters, preventing automated syncs during peak traffic hours or business-critical periods:

```yaml
syncPolicy:
  syncWindows:
    - kind: allow
      schedule: '0 2-6 * * 1-5'  # weekdays 2am-6am only
      duration: 4h
      applications: ['*']
```

**Ignoring ArgoCD health status.** ArgoCD tracks application health (Healthy, Progressing, Degraded) based on Kubernetes resource conditions. A `Degraded` health status on a synced application means the desired state was applied but the application isn't healthy, a deployment with crashing pods, for example. Treat health status with the same urgency as a failed deployment.

## Conclusion
GitOps with ArgoCD and Helm is not just a deployment mechanism, it is a complete operational model that eliminates an entire category of operational risk. Cluster drift, unclear ownership, undocumented changes, and failed rollbacks are structural problems that Git-driven reconciliation solves at the architecture level rather than the process level.

The App of Apps pattern scales the model across dozens of services and teams. Projects and RBAC enforce team boundaries without ticket-based access control. Sync policies enforce Git as the source of truth, not a guideline. Progressive delivery with Argo Rollouts brings data-driven safety to the final mile of deployment.

The investment is real, repository structure, RBAC design, secrets management, and sync policy tuning require deliberate engineering. But once the platform is in place, the operational discipline it enforces is automatic rather than cultural, which is the only kind of discipline that holds at scale.

*Running multiple Kubernetes clusters across environments or cloud providers? ArgoCD's multi-cluster support with ApplicationSets scales the App of Apps pattern to fleet-level management without per-cluster configuration duplication.*

#GitOps #ArgoCD #Helm #Kubernetes #DevOps