Module 5: Gitops With Argocd

8. Project: a Git change, reflected on its own

Next →

Description

This project closes the module with the proof every previous lesson prepared: a real change, pushed to Git with git push, reflected on andes-cargo-cluster with no one running kubectl apply for anything other than observing. This isn't a toy change — you're going to bump andes-cargo-status-api's replicas from 3 to 5, a real business change, the kind a real team would make on any given Tuesday — and you're going to measure, with real timestamps, how long it takes to converge. As a bonus, this project documents a real finding that showed up unplanned while writing this lesson: Module 3's HorizontalPodAutoscaler and the replicas Git declares end up competing for the same field — and the real production fix that resolves that tension, verified with literal evidence.

Connection to the module

This project uses, with no changes, the seven previous lessons: Gitea (lesson 3) as the repository, ArgoCD (lesson 4) as the operator, the already-synced Application (lessons 5-6), and the precise distinction between scaling and RollingUpdate (lesson 7) that makes it possible to describe, accurately, which mechanism you're seeing here.

Step 1 — Confirm the starting state

kubectl get pods -n andes-cargo -l app=andes-cargo-status-api
kubectl get deployment andes-cargo-status-api -n andes-cargo -o jsonpath='{.spec.replicas} desired, {.status.readyReplicas} ready{"\n"}'

What to expect (literal, executed — Pod names with a hash suffix are your variable value; the three-replica count is literal at this point of the module, inherited from lesson 6):

NAME                                      READY   STATUS    RESTARTS   AGE
andes-cargo-status-api-548966dd97-bpqjq   1/1     Running   0          76m
andes-cargo-status-api-548966dd97-ls2bc   1/1     Running   0          76m
andes-cargo-status-api-548966dd97-z7bpd   1/1     Running   0          41s

3 desired, 3 ready
grep -n "replicas:" deployment.yaml
10:  replicas: 3

Three replicas, on the cluster and in the repository — this project's exact starting point.

Step 2 — Change deployment.yaml, locally, and confirm the diff

sed -i '' 's/replicas: 3/replicas: 5/' deployment.yaml
git diff

What to expect (literal, executed):

diff --git a/deployment.yaml b/deployment.yaml
index 423cef4..9086693 100644
--- a/deployment.yaml
+++ b/deployment.yaml
@@ -7,7 +7,7 @@ metadata:
   labels:
     app: andes-cargo-status-api
 spec:
-  replicas: 3
+  replicas: 5
   strategy:
     type: RollingUpdate
     rollingUpdate:

One line. Nothing else changed — not the image, not strategy, not any other Pod template field. As lesson 7 confirmed, this is, precisely, a scaling event — the existing ReplicaSet is going to grow, with no new one created.

Step 3 — git commit, git push — and nothing else

git add deployment.yaml
git commit -m "Scale andes-cargo-status-api from 3 to 5 replicas"
git push origin main

What to expect (literal, executed — the commit hash is your variable value):

To http://localhost:3000/andes-cargo/andes-cargo-k8s.git
   820515f..cd0b5ca  main -> main

This is everything you're going to run against the cluster. From here on, no command in the following steps changes anything — they all just observe.

Step 4 — Watch the convergence, with real timestamps

for i in $(seq 1 18); do
  ts=$(date +%H:%M:%S)
  sync=$(kubectl get application andes-cargo-status-api -n argocd -o jsonpath='{.status.sync.status}')
  revision=$(kubectl get application andes-cargo-status-api -n argocd -o jsonpath='{.status.sync.revision}' | cut -c1-7)
  replicas=$(kubectl get deployment andes-cargo-status-api -n andes-cargo -o jsonpath='{.spec.replicas}')
  echo "$ts sync=$sync revision=$revision spec.replicas=$replicas"
  if [ "$replicas" = "5" ]; then break; fi
  sleep 10
done

What to expect (literal, executed — the exact timestamps are your variable value; the pattern — several cycles with no change, then convergence — is what confirms lesson 2's mechanism):

15:54:13 sync=Synced revision=820515f spec.replicas=3
15:54:23 sync=Synced revision=820515f spec.replicas=3
15:54:33 sync=Synced revision=820515f spec.replicas=3
15:54:43 sync=Synced revision=820515f spec.replicas=3
15:54:53 sync=Synced revision=820515f spec.replicas=3
15:55:03 sync=Synced revision=820515f spec.replicas=3
15:55:13 sync=Synced revision=820515f spec.replicas=3
15:55:24 sync=Synced revision=820515f spec.replicas=3
15:55:34 sync=Synced revision=820515f spec.replicas=3
15:55:44 sync=Synced revision=820515f spec.replicas=3
15:55:54 sync=Synced revision=cd0b5ca spec.replicas=5

About one hundred seconds between Step 3's git push and the detected convergence — within the range lesson 2 predicted (ArgoCD's polling interval, by default, a few minutes). Not a single kubectl apply, kubectl scale, or kubectl edit ran at any point during this wait — the number went from 3 to 5 because ArgoCD pushed it there, not because anyone forced it.

Step 5 — Confirm the Pods, at two moments

A few seconds after revision changed to cd0b5ca:

kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o wide

What to expect (literal, executed — still transitioning, one Pod finishing):

NAME                                      READY   STATUS        RESTARTS   AGE   IP            NODE
andes-cargo-status-api-548966dd97-8w8tx   1/1     Running       0          15s   10.244.1.17   andes-cargo-cluster-worker
andes-cargo-status-api-548966dd97-9nh96   1/1     Running       0          15s   10.244.1.18   andes-cargo-cluster-worker
andes-cargo-status-api-548966dd97-bjcqx   1/1     Terminating   0          91s   10.244.1.16   andes-cargo-cluster-worker
andes-cargo-status-api-548966dd97-bpqjq   1/1     Running       0          79m   10.244.1.2    andes-cargo-cluster-worker
andes-cargo-status-api-548966dd97-hqlbt   1/1     Running       0          15s   10.244.2.10   andes-cargo-cluster-worker2
andes-cargo-status-api-548966dd97-ls2bc   1/1     Running       0          79m   10.244.2.2    andes-cargo-cluster-worker2

And, a few seconds later, already stable:

kubectl get all -n andes-cargo

What to expect (literal, executed — Pod names are your variable value; the five-replica pattern is literal):

NAME                                          READY   STATUS    RESTARTS   AGE
pod/andes-cargo-status-api-548966dd97-br69f   1/1     Running   0          42s
pod/andes-cargo-status-api-548966dd97-g4xhv   1/1     Running   0          42s
pod/andes-cargo-status-api-548966dd97-hqlbt   1/1     Running   0          70s
pod/andes-cargo-status-api-548966dd97-ls2bc   1/1     Running   0          79m
pod/andes-cargo-status-api-548966dd97-wtv4s   1/1     Running   0          42s

NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/status-api-service   ClusterIP   10.96.239.125   <none>        80/TCP    79m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/andes-cargo-status-api   5/5     5            5           79m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/andes-cargo-status-api-548966dd97   5         5         5       79m

NAME                                                             REFERENCE                           TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/andes-cargo-status-api-hpa   Deployment/andes-cargo-status-api   cpu: 1%/50%   2         6         5          79m

5/5, a single ReplicaSet (548966dd97, the same hash as always — confirming, again, that this was scaling, not a version RollingUpdate, exactly as lesson 7 predicted). Confirm it from ArgoCD's angle too:

argocd app get andes-cargo-status-api

What to expect (literal, executed):

Sync Status:        Synced to main (cd0b5ca)
Health Status:      Healthy

GROUP              KIND                     NAMESPACE    NAME                            STATUS  HEALTH   HOOK  MESSAGE
                   Namespace                andes-cargo  andes-cargo                     Synced  Running        namespace/andes-cargo unchanged
networking.k8s.io  NetworkPolicy            andes-cargo  default-deny-ingress            Synced                 networkpolicy.networking.k8s.io/default-deny-ingress unchanged
networking.k8s.io  NetworkPolicy            andes-cargo  allow-from-ingress-nginx        Synced                 networkpolicy.networking.k8s.io/allow-from-ingress-nginx unchanged
                   Secret                   andes-cargo  andes-cargo-status-api-secrets  Synced                 secret/andes-cargo-status-api-secrets configured
                   ConfigMap                andes-cargo  andes-cargo-status-api-config   Synced                 configmap/andes-cargo-status-api-config unchanged
                   Service                  andes-cargo  status-api-service              Synced  Healthy        service/status-api-service unchanged
apps               Deployment               andes-cargo  andes-cargo-status-api          Synced  Healthy        deployment.apps/andes-cargo-status-api configured
autoscaling        HorizontalPodAutoscaler  andes-cargo  andes-cargo-status-api-hpa      Synced  Healthy        horizontalpodautoscaler.autoscaling/andes-cargo-status-api-hpa unchanged
networking.k8s.io  Ingress                  andes-cargo  status-api-ingress              Synced  Healthy        ingress.networking.k8s.io/status-api-ingress unchanged
argoproj.io        Application              argocd       andes-cargo-status-api          Synced                 application.argoproj.io/andes-cargo-status-api configured

Sync to cd0b5ca — exactly Step 3's commit. Synced, Healthy, five replicas, zero kubectl apply. This is the whole module's central proof.

                    WHAT JUST HAPPENED

  You                    Gitea                  ArgoCD              andes-cargo-cluster
   │                       │                       │                        │
   │──git push (cd0b5ca)──▶│                       │                        │
   │                       │◀── poll (~100s) ──────│                        │
   │                       │────commit cd0b5ca────▶│                        │
   │                       │                       │──apply replicas: 5────▶│
   │                       │                       │◀───5/5 Ready───────────│
   │                       │                       │                        │
   │  You never ran: kubectl apply, kubectl scale, kubectl edit             │

Step 6 — The real finding: the HPA and Git compete for the same field

Everything above is the proof this project promised. What follows is a real, unplanned discovery that showed up while writing this lesson — and it's worth documenting in full, because it's exactly the kind of problem a real team runs into the first week of using GitOps alongside autoscaling.

A few minutes after Step 5's convergence, with no one touching anything:

kubectl get deployment andes-cargo-status-api -n andes-cargo -o jsonpath='{.spec.replicas}{"\n"}'
kubectl get hpa -n andes-cargo
argocd app get andes-cargo-status-api --refresh | head -12

What to expect (literal, executed — a surprise: the number dropped again, and ArgoCD flags it):

2

NAME                         REFERENCE                           TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
andes-cargo-status-api-hpa   Deployment/andes-cargo-status-api   cpu: 1%/50%   2         6         2          94m

Sync Status:        OutOfSync from main (cd0b5ca)
Health Status:      Healthy

OutOfSync. Module 3's HorizontalPodAutoscaler (andes-cargo-status-api-hpa, with minReplicas: 2) did exactly what it was designed to do: with real CPU at 1%, well below the 50% target, it lowered replicas toward the allowed minimum — writing spec.replicas: 2 directly to the Deployment, bypassing Git. deployment.yaml, in Gitea, still says replicas: 5 — Git and the real cluster now say different things, and ArgoCD detects it precisely (that's exactly its job: comparing).

With selfHeal: true active since lesson 5, the obvious question is: why doesn't ArgoCD revert this to 5, the way it did with that same lesson's manual kubectl scale? The answer is in the controller's own logs:

kubectl logs -n argocd argocd-application-controller-0 --tail=200 | grep -i "skipping auto-sync"

What to expect (literal, executed):

{"msg":"Skipping auto-sync: already attempted sync to [cd0b5cadbb43cfdf8c6aaa9c46a7fcd0dd8a63eb] with timeout 0s (retrying in 3m41.370372326s)", ...}

ArgoCD did attempt an automatic fix as soon as it detected the difference — but, after that first attempt, it applies a cooldown period before trying again against the same revision. Without this mechanism, selfHeal and the HPA would enter an infinite fight: ArgoCD forces 5, the HPA lowers it back to 2 on its next cycle (every few seconds), ArgoCD forces it again, indefinitely — the exact kind of oscillation no production system should tolerate. The cooldown avoids that fight, at the cost of leaving the Application in OutOfSync indefinitely while the HPA stays active and CPU stays low.

              THE FIGHT selfHeal DOESN'T WAGE INDEFINITELY

  Git: replicas = 5          HPA controller: cpu 1%/50%,
  (unchanged)                  lowers to replicas = 2
        │                              │
        └──── ArgoCD compares, detects difference ────┘
                        │
                        ▼
        First attempt: forces replicas = 5 ✓
                        │
                        ▼
        HPA lowers it back to 2, seconds later
                        │
                        ▼
        ArgoCD: "already attempted sync against this revision,
                 retrying in ~3-4 minutes" — NOT an endless fight

Step 7 — The real fix: telling ArgoCD that field isn't its business

The real production solution, documented by the project itself, isn't disabling selfHeal or lowering the polling frequency — it's explicitly telling ArgoCD that that specific field isn't within its purview. ignoreDifferences does exactly that:

# application.yaml (added fragment)
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

ignoreDifferences excludes /spec/replicas from the Sync Status calculation — ArgoCD stops considering that field when deciding whether the Deployment is synced or not. RespectIgnoreDifferences=true, inside syncOptions, is the necessary complement: without this flag, a real sync would still apply Git's full value (including replicas: 5) every time it ran, overwriting whatever the HPA had decided — the flag tells the sync operation itself to respect the value already on the cluster for that field, not just to ignore it when comparing.

git add application.yaml
git commit -m "Ignore spec.replicas on the Deployment: the HorizontalPodAutoscaler owns it, not Git"
git push origin main
argocd app sync andes-cargo-status-api

What to expect (literal, executed — the hash is your variable value):

To http://localhost:3000/andes-cargo/andes-cargo-k8s.git
   cd0b5ca..db09bbe  main -> main

Sync Status:        Synced to main (db09bbe)
Health Status:      Healthy
...
apps  Deployment  andes-cargo  andes-cargo-status-api  Synced  Healthy  deployment.apps/andes-cargo-status-api unchanged

unchanged — even though the real Deployment still had 2 replicas (the value the HPA left) and Git still declares 5. ArgoCD no longer considers that a difference. Confirm it with the same continuity test Step 6 ran, this time for two full minutes:

for i in $(seq 1 10); do
  ts=$(date +%H:%M:%S)
  sync=$(kubectl get application andes-cargo-status-api -n argocd -o jsonpath='{.status.sync.status}')
  replicas=$(kubectl get deployment andes-cargo-status-api -n andes-cargo -o jsonpath='{.spec.replicas}')
  echo "$ts sync=$sync spec.replicas=$replicas"
  sleep 12
done

What to expect (literal, executed):

16:15:27 sync=Synced spec.replicas=2
16:15:39 sync=Synced spec.replicas=2
16:15:51 sync=Synced spec.replicas=2
16:16:03 sync=Synced spec.replicas=2
16:16:15 sync=Synced spec.replicas=2
16:16:27 sync=Synced spec.replicas=2
16:16:39 sync=Synced spec.replicas=2
16:16:51 sync=Synced spec.replicas=2
16:17:03 sync=Synced spec.replicas=2
16:17:15 sync=Synced spec.replicas=2

Synced, stably, with the HPA free to move replicas wherever real CPU calls for — 2 at this moment — with ArgoCD never flagging it as a difference again. This is the correct production state: Git remains the source of truth for what runs (the image, the configuration, the probes), and the HPA is the source of truth for how many replicas, in real time — two different authorities, each owning its own field, without fighting each other.

Module's final checklist

  • Gitea running inside the cluster, with the andes-cargo/andes-cargo-k8s repository as the single source of truth for nine manifests (lesson 3).
  • ArgoCD v3.5.1 running inside the cluster, automatically syncing that repository (lesson 4).
  • A real change (replicas: 3 → 5) pushed with git push and reflected on the cluster with not a single kubectl apply, measured with real timestamps (~100 seconds to converge) — this module's central proof (Steps 1-5 of this project).
  • selfHeal: true fixing a manual change in seconds (lesson 5) and respecting, via ignoreDifferences, the field that belongs to another controller (Steps 6-7 of this project) — both faces of the same GitOps property, applied with judgment.

Common mistakes

Concluding Step 6's finding means "GitOps and autoscaling are incompatible" (hasty generalization, this lesson's most important mistake). What happens: someone, seeing OutOfSync with no obvious reason, concludes ArgoCD and HorizontalPodAutoscaler shouldn't be used together. How to spot it: if your conclusion is "you have to choose between GitOps or autoscaling." How to fix it: Step 7 demonstrates the standard, officially documented solution (ignoreDifferences + RespectIgnoreDifferences=true) — it's a known pattern, not a flaw in either tool. Any team using ArgoCD alongside a HorizontalPodAutoscaler in real production configures this from the start.

Applying ignoreDifferences without RespectIgnoreDifferences=true and expecting the same result (incomplete YAML). What happens: someone adds only the ignoreDifferences block, without the complementary sync option. How to spot it: if a manual sync (argocd app sync) keeps overwriting the value the HPA had set. How to fix it: ignoreDifferences by itself only affects the diff calculation (what counts as OutOfSync); without RespectIgnoreDifferences=true, a real sync still applies Git's full manifest, including the "ignored" field. The two pieces work together, not one instead of the other.

Not understanding why selfHeal didn't fight indefinitely before the fix (mechanism, already explained with Step 6's literal log). What happens: someone expects to see ArgoCD forcing replicas: 5 every few seconds, in a visible fight against the HPA. How to spot it: if, checking the namespace's events during the OutOfSync period, you expected to see dozens of ScalingReplicaSet events alternating between 2 and 5. How to fix it: the controller's own Skipping auto-sync: already attempted sync ... retrying in Nm message confirms ArgoCD attempts the fix once, and then waits a cooldown period before retrying against the same revision — a protection mechanism against exactly this project's scenario, documented in the controller's own behavior, not in the higher-level documentation.

Exercises

Exercise 1 — Reconstruct the full sequence from memory. Without going back to this lesson, list, in order, the five events that happened between Step 3's git push and the final Synced/Healthy state with ignoreDifferences active.

See solution

(1) git push sends replicas: 5 to Gitea. (2) ArgoCD polls the repository (~100 seconds later) and detects the change. (3) ArgoCD applies replicas: 5 to the Deployment, with no manual kubectl apply — convergence confirmed. (4) Minutes later, the HorizontalPodAutoscaler, with real CPU low, lowers replicas to 2 directly on the cluster, bypassing Git — ArgoCD detects it as OutOfSync, but doesn't fight indefinitely (controller cooldown). (5) ignoreDifferences + RespectIgnoreDifferences=true get added to the Application, via another git push — Sync Status goes back to Synced stably, with the HPA free to move replicas without ArgoCD ever flagging it again.

Exercise 2 — Explain ignoreDifferences without using the word "ignore." In one sentence, without using the word "ignore" or "ignores," explain what ignoreDifferences: [{group: apps, kind: Deployment, jsonPointers: [/spec/replicas]}] does.

See solution

A reasonable explanation: "It tells ArgoCD that this particular Deployment's replicas field belongs to another controller (the HorizontalPodAutoscaler), so it shouldn't count it as a difference between Git and the real cluster, nor overwrite it during a sync."

Exercise 3 — Design proof that the fix works, without copying Step 7. Without looking at Step 7 again, describe an experiment — with concrete commands — that demonstrates ignoreDifferences is working correctly for this Application.

See solution

A reasonable answer: force the HPA to visibly change replicas (for example, by generating real load with kubectl run load-generator, as in Module 3, so it rises above 5, or by waiting for CPU to drop and the HPA to lower it naturally, as this project did) and, while that happens, run kubectl get application andes-cargo-status-api -n argocd -o jsonpath='{.status.sync.status}' in a loop for several minutes. If the result stays Synced throughout the experiment, no matter what value replicas has at any given moment, the fix works — exactly the two-minute test this lesson's Step 7 ran.

Summary and next step

This project demonstrated, with literal evidence and real timestamps, this whole module's central promise: a real business change (replicas: 3 → 5), pushed with git push, reflected on andes-cargo-cluster in about one hundred seconds, with not a single kubectl apply. It also documented, in full, a real unplanned finding — the tension between ArgoCD's selfHeal and Module 3's HorizontalPodAutoscaler over the same replicas field — and its real production fix (ignoreDifferences + RespectIgnoreDifferences=true), verified with a two-minute stability test. andes-cargo-status-api is, from today, a service whose what runs lives in Git (image, configuration, probes, NetworkPolicy) and whose how much runs lives in real time, in the HorizontalPodAutoscaler — two sources of truth, each owning its own field, coexisting without fighting.

Before moving on you should be able to: reproduce, on your own cluster, this project's git push with no kubectl apply; explain why selfHeal didn't fight indefinitely against the HPA; and write from memory the ignoreDifferences block that resolves that tension.

Next module: runtime security, admission control, and image scanning. Module 6 takes the same andes-cargo-k8s repository ArgoCD syncs and adds the layer that decides, before any object reaches etcd, whether it's allowed to exist — OPA Gatekeeper and Kyverno, two different policy engines, both actually run against andes-cargo-cluster.

Resources

  1. Argo CD — Diffing, Application-Level Configuration — official documentation for ignoreDifferences, the exact source for this project's fix.
  2. Argo CD — Sync Options, Respect Ignore Difference on Sync — documentation for RespectIgnoreDifferences=true.
  3. Kubernetes — Horizontal Pod Autoscaling — reference for the controller competing for spec.replicas in this project.
  4. kubernetes-and-eks-in-production-guide (NIEVA), Module 3, lessons 7-8 — the origin of the HorizontalPodAutoscaler this project picks back up with a new finding.
← Previous7. Deployment strategies: rolling, blue/green, and canaryNext →1. Module introduction: the gatekeeper before `etcd`