GitOps architectures

Key points:

• Use pull-based GitOps: an in-cluster agent applies Git, so no outside system holds cluster credentials and drift gets reverted.
• Keep deployment config in its own repo, with a folder per environment, not a branch per environment.
• Run one ArgoCD per cluster when clusters are separated for compliance; a hub holds admin credentials to every cluster.
• Never commit plain secrets - use External Secrets Operator, Sealed Secrets or SOPS.
• “Synced” isn’t “working” - keep checking from the outside.

Push vs pull

• Push: CI builds, then runs helm upgrade or kubectl apply. Simple, but CI holds cluster credentials and nothing notices manual changes.
• Pull (GitOps): CI builds and updates Git; an agent inside the cluster (ArgoCD or Flux) makes the cluster match. No outside system holds cluster credentials; drift is detected and reverted.

Git is the source of truth: rollback is git revert, audit log is git log, a change is a reviewed PR. Full comparison, including progressive delivery: Kubernetes CD: push, pull, or progressive.

flowchart LR
    DEV[Developer] -->|PR| APP[App repo]
    APP -->|CI: test, build, push image| REG[(Registry)]
    APP -->|CI: bump image tag| CFG[Config repo]
    CFG -->|watched by| ARGO[ArgoCD in cluster]
    ARGO -->|apply + self-heal| K8S[Cluster]
    REG -->|pull image| K8S

Where ArgoCD runs

	One ArgoCD per cluster	One hub ArgoCD managing many clusters
Credentials	Each instance only touches its own cluster	The hub holds admin credentials to every cluster
Blast radius	A compromise or bad change hits one cluster	A compromise reaches everything
Visibility	One UI per cluster	One pane of glass
Upgrades	Many instances to keep current	One to upgrade
Regulated or sovereign setups	Natural fit - nothing shared across the boundary	Hard to justify

• Default: many teams start with a hub for convenience.
• Use one per cluster when: clusters sit in separate regions for compliance - each stays self-contained.

Repository layouts

Separate app code from deployment config. CI committing image tags into the repo it builds from triggers a build per deploy. A config repo also gives deploys their own access control and history.

Use a folder per environment, not a branch per environment:

deploy/
├── base/                 # shared manifests or Helm values
└── envs/
    ├── dev/
    │   └── payments/     # kustomization.yaml or values.yaml
    ├── staging/
    │   └── payments/
    └── prod/
        └── payments/

Branches drift in ways nobody reviews and only hold up for small setups. With folders, diff envs/staging envs/prod shows exactly what promotion changes, and promotion is a PR copying a change into the next folder.

App-of-apps vs ApplicationSets

• App-of-apps: a parent Application points at a folder of child Application manifests. Explicit, but every new app needs a manifest.
• ApplicationSet: a template plus a generator producing Applications per folder, cluster or list entry. Add a folder, get an app.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: prod-apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/my-org/deploy.git
        revision: main
        directories:
          - path: envs/prod/*
  template:
    metadata:
      name: 'prod-{{.path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/my-org/deploy.git
        targetRevision: main
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Generators combine: a matrix of the cluster and git generators deploys every app folder to every matching cluster.

Secrets

Never commit plain secrets, even to a private repo. Use one of:

• External Secrets Operator - Git holds a reference; the secret lives in AWS Secrets Manager, Azure Key Vault, GCP Secret Manager or Vault. See getting CA certificates through secret managers intact.
• Sealed Secrets - encrypted with the cluster’s public key; only the in-cluster controller can decrypt.
• SOPS - files encrypted with KMS or age keys, decrypted at sync time.

Gotchas

• HPA and ArgoCD fight over replicas. ArgoCD resets the count the autoscaler set. Remove replicas from the manifest when an HPA manages it, or tell ArgoCD to ignore it:

  spec:
    ignoreDifferences:
      - group: apps
        kind: Deployment
        jsonPointers:
          - /spec/replicas
    syncPolicy:
      syncOptions:
        - RespectIgnoreDifferences=true

• prune: true deletes what left Git. Renaming a folder or Application can delete and recreate everything in it. For stateful resources, review prunes or set the argocd.argoproj.io/sync-options: Prune=false annotation.

• Self-heal reverts manual hotfixes. By design, it undoes hand-made incident fixes. Disable auto-sync first, or commit the fix to Git.

• Ordering matters. CRDs must precede resources using them, namespaces their contents. Use sync waves (argocd.argoproj.io/sync-wave).

• Large CRDs fail to apply. Client-side apply stores the whole object in an annotation with a size limit. Use ServerSideApply=true.

• “Synced” isn’t “working”. Synced = cluster matches Git; Healthy = resources report ready; neither means users are served. Keep checking from the outside.

• ArgoCD itself needs upgrading. It holds powerful credentials. Patch critical vulnerabilities within days, and upgrade one cluster first.