Interactive study guide

Learn the GitOps loop by building it yourself.

Work through one task at a time, copy commands when needed, and tick each checkbox directly here.

Merge→Build image→Update Git→Argo CD sync→CronJob runs

Study roadmap

Implementing GitOps Deployment for a Kubernetes CronJob with Argo CD

argo-cdkubernetescronjobgitopsgithub-actionsghcrkustomizeci-cdcontainer-imagesdeployment-automationstudy-plan

0 of 0 complete

Goal

Understand the complete path from merging code to automatically deploying a new CronJob image, while keeping Git as the deployment source of truth.

Interactive dashboard

Use the hosted study dashboard to tick tasks directly in the browser and keep progress without editing these Markdown files.

Intended structure

app/                    # Small program executed by the CronJob
Dockerfile              # Container image definition
deploy/base/            # Base Kubernetes CronJob manifest
deploy/overlays/local/  # Local Kustomize image configuration
argocd/                 # Argo CD Application manifest
.github/workflows/      # Image build and Git tag-update workflow
docs/                   # Study plan and project documentation

Phase 1 — Build the scheduled application

Create a small application that prints its execution time and Git commit version.
Containerize it and run the image manually.
Verify its output before introducing Kubernetes.

Milestone: The container runs locally and prints the expected version.

Phase 2 — Run it as a Kubernetes CronJob

Create a local Kind cluster.
Write a batch/v1 CronJob manifest that initially runs every two minutes.
Learn schedule, concurrencyPolicy, job history, retry behavior, and suspend.
Trigger a manual Job from the CronJob and inspect its logs.

Milestone: Kubernetes creates Jobs on schedule and the logs are understandable.

Phase 3 — Deploy through Argo CD

Install Argo CD in the local cluster.
Create a Kustomize base and local overlay.
Create an Argo CD Application that watches deploy/overlays/local on main.
Enable automatic sync, pruning, self-healing, and namespace creation.
Change the CronJob schedule in Git and observe Argo CD synchronize it.

Milestone: A Git commit changes the cluster without deploying through kubectl or CI.

Phase 4 — Build an image after every merge

Add a GitHub Actions workflow triggered by application changes merged into main.
Test and build the container.
Tag it as sha-<full-commit-sha> and push it to GHCR.
Use immutable tags instead of latest.

Milestone: Every application merge produces a traceable GHCR image.

Phase 5 — Update the desired image tag in Git

Let CI update only the Kustomize newTag value after publishing the image.
Commit and push that configuration change back to this repository.
Restrict workflow paths so the automated configuration commit does not start another image build.
Let Argo CD discover and deploy the new desired state.

Milestone: CI hands deployment responsibility to Git, and Argo CD performs the deployment.

Phase 6 — Verify the complete GitOps loop

For one merged pull request, verify:

GitHub Actions succeeds.
GHCR contains the merge commit's image tag.
Kustomize contains that exact tag.
Argo CD reports the application as synced.
The next scheduled Job prints the new version.

Phase 7 — Practice failures and rollback

Diagnose an invalid image tag and ImagePullBackOff.
Diagnose an application that exits unsuccessfully.
Test concurrencyPolicy: Forbid with a long-running Job.
Test Argo CD self-healing after a manual cluster change.
Revert an image-tag commit and verify that the following Job uses the previous image.

Milestone: Failures are diagnosed from evidence, and rollback is performed through Git.

Optional Phase 8 — Compare Argo CD Image Updater

Replace the CI tag-edit step with the CRD-based Argo CD Image Updater.
Track SHA tags with the newest-build strategy and Git write-back.
Compare asynchronous registry reconciliation with the deterministic CI-driven update used above.

Definition of done

A merge builds one immutable image tied to its commit SHA.
Git records the exact image selected for deployment.
Argo CD automatically synchronizes the cluster from Git.
The next CronJob execution uses the selected image.
A Git revert performs a successful rollback.

Primary references

Phase 1

Building and Containerizing the Scheduled Application

argo-cdkubernetescronjobdockercolimashell-scriptcontainer-imagesimplementationlocal-development

0 of 19 complete

Checklist

Step 1 — Open the repository so every following command runs from the project root.

cd /Users/davidleora/Projects/argocd-cronjob-gitops-lab
pwd

Step 2 — Install the missing local tools required by later phases.

brew install colima kind argocd kustomize

Step 2A — Start Colima with the Docker runtime and select its Docker context before running Docker commands.

colima start \
  --runtime docker \
  --cpu 4 \
  --memory 6

docker context use colima
docker info --format 'Docker daemon ready: {{.ServerVersion}}'

Step 3 — Confirm Colima is running and every required CLI is available.

colima status
docker version
git --version
gh --version
kind version
kubectl version --client
argocd version --client
kustomize version

Step 4 — Confirm GitHub CLI is authenticated to the account that owns the repository.

gh auth status

Step 5 — The application directory and repository ignore rules are already prepared. Step 6 — Create a shell application that prints its execution time and image version and supports later failure exercises.

cat > app/run.sh <<'EOF'
#!/bin/sh
set -eu

printf 'execution_time=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
printf 'app_version=%s\n' "${APP_VERSION:-unknown}"

sleep_seconds="${SLEEP_SECONDS:-0}"
if [ "$sleep_seconds" -gt 0 ]; then
  printf 'sleep_seconds=%s\n' "$sleep_seconds"
  sleep "$sleep_seconds"
fi

if [ "${FORCE_FAILURE:-false}" = "true" ]; then
  printf 'result=forced-failure\n' >&2
  exit 1
fi

printf 'result=success\n'
EOF

chmod +x app/run.sh

Step 7 — Validate and run the script without Docker.

sh -n app/run.sh
APP_VERSION=sha-local ./app/run.sh

Step 8 — Create a small non-root container image for the application.

cat > Dockerfile <<'EOF'
FROM alpine:3.22

ARG APP_VERSION=dev
ENV APP_VERSION=$APP_VERSION

COPY app/run.sh /usr/local/bin/run
RUN chmod 0555 /usr/local/bin/run

USER 65532:65532
ENTRYPOINT ["/usr/local/bin/run"]
EOF

Step 9 — The Docker build-context ignore rules are already prepared. Step 10 — Build the local image with a visible test version.

docker build \
  --build-arg APP_VERSION=sha-local \
  --tag argocd-cronjob-gitops-lab:local \
  .

Step 11 — Run the container and confirm all four output fields appear.

docker run --rm argocd-cronjob-gitops-lab:local

Step 12 — Run the failure mode and confirm the container exits with a nonzero status.

docker run --rm \
  --env FORCE_FAILURE=true \
  argocd-cronjob-gitops-lab:local

echo $?

Step 13 — Review exactly what will be committed.

git status --short
sed -n '1,240p' Dockerfile
sed -n '1,240p' app/run.sh

Step 14 — Commit and push the completed application container.

git add Dockerfile app/run.sh
git commit -m "Add scheduled application container"
git push

Completion check

Run every verification command before ticking the completion boxes.

sh -n app/run.sh
echo $?

docker run --rm argocd-cronjob-gitops-lab:local

docker run --rm \
  --env FORCE_FAILURE=true \
  argocd-cronjob-gitops-lab:local
echo $?

git status --short --branch

The script passes sh -n. The local container prints execution_time, app_version, and result. The forced-failure run returns a nonzero exit status. The main branch is clean and pushed.

Phase 2

Running the Container as a Kubernetes CronJob on Kind

kubernetescronjobkindkustomizejobcontainer-imagesimplementationlocal-cluster

0 of 22 complete

Checklist

Step 1 — Open the repository and confirm the Phase 1 image exists.

cd /Users/davidleora/Projects/argocd-cronjob-gitops-lab
docker image inspect argocd-cronjob-gitops-lab:local

Step 2 — Create the local Kind cluster.

kind create cluster --name argocd-study

Step 3 — Confirm kubectl is using the new cluster.

kubectl config use-context kind-argocd-study
kubectl cluster-info
kubectl get nodes

Step 4 — Load the local image directly into the Kind node.

kind load docker-image \
  argocd-cronjob-gitops-lab:local \
  --name argocd-study

Step 5 — Create the base manifest directory.

mkdir -p deploy/base

Step 6 — Create a CronJob that runs every two minutes and prevents overlapping executions.

cat > deploy/base/cronjob.yaml <<'EOF'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cronjob-lab
  labels:
    app.kubernetes.io/name: cronjob-lab
spec:
  schedule: "*/2 * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 60
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 2
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 90
      template:
        metadata:
          labels:
            app.kubernetes.io/name: cronjob-lab
        spec:
          restartPolicy: Never
          containers:
            - name: cronjob-lab
              image: argocd-cronjob-gitops-lab:local
              imagePullPolicy: IfNotPresent
EOF

Step 7 — Create the base Kustomize file.

cat > deploy/base/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cronjob.yaml
EOF

Step 8 — Render the manifests locally and inspect the result before applying it.

kustomize build deploy/base

Step 9 — Create the workload namespace.

kubectl create namespace cronjob-study

Step 10 — Apply the rendered CronJob to the namespace.

kubectl apply -k deploy/base --namespace cronjob-study
kubectl get cronjob --namespace cronjob-study

Step 11 — Create a uniquely named manual Job from the CronJob template.

manual_job_name="manual-$(date +%s)"
kubectl create job \
  --from=cronjob/cronjob-lab \
  "${manual_job_name}" \
  --namespace cronjob-study

Step 12 — Wait for the manual Job to finish and read its logs.

kubectl wait \
  --for=condition=complete \
  "job/${manual_job_name}" \
  --namespace cronjob-study \
  --timeout=120s

kubectl logs \
  "job/${manual_job_name}" \
  --namespace cronjob-study

Step 13 — Inspect the CronJob, Job, and Pod relationship.

kubectl get cronjobs,jobs,pods \
  --namespace cronjob-study \
  --show-labels

Step 14 — Wait at least two minutes and confirm the controller creates a scheduled Job.

kubectl get jobs \
  --namespace cronjob-study \
  --watch

Step 15 — Stop the watch with Control-C and inspect the newest Job manually.

kubectl get jobs \
  --namespace cronjob-study \
  --sort-by=.metadata.creationTimestamp

Step 16 — Capture the newest Job name and read its logs.

newest_job_name="$(kubectl get jobs \
  --namespace cronjob-study \
  --sort-by=.metadata.creationTimestamp \
  --output jsonpath='{.items[-1:].metadata.name}')"

kubectl logs \
  "job/${newest_job_name}" \
  --namespace cronjob-study

Step 17 — Commit and push the base Kubernetes configuration.

git add deploy/base
git commit -m "Add Kubernetes CronJob base"
git push

Completion check

Kind reports one Ready node. The CronJob exists in cronjob-study. A manually created Job completes successfully. A scheduled Job appears after the configured interval. Both Jobs print app_version=sha-local.

Phase 3

Deploying the CronJob from a Private Git Repository with Argo CD

argo-cdkubernetescronjobkindkustomizegitopsprivate-repositoryssh-deploy-keyautomated-syncimplementation

0 of 22 complete

Checklist

Step 1 — Open the repository and select the Kind cluster.

cd /Users/davidleora/Projects/argocd-cronjob-gitops-lab
kubectl config use-context kind-argocd-study

Step 2 — Create the Argo CD namespace and install the stable manifests.

kubectl create namespace argocd
kubectl apply \
  --namespace argocd \
  --server-side \
  --force-conflicts \
  --filename https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Step 3 — Wait for the Argo CD server and repository server to become available.

kubectl wait \
  --for=condition=Available \
  deployment/argocd-server \
  deployment/argocd-repo-server \
  --namespace argocd \
  --timeout=300s

Step 4 — Create a dedicated SSH deploy key for this private repository.

mkdir -p .local-secrets
ssh-keygen \
  -t ed25519 \
  -C "argocd-cronjob-gitops-lab" \
  -f .local-secrets/argocd-repository \
  -N ""

Step 5 — Add the public key to GitHub with write access reserved for the optional Image Updater phase.

gh repo deploy-key add \
  .local-secrets/argocd-repository.pub \
  --repo davidleora/argocd-cronjob-gitops-lab \
  --title "Argo CD local lab" \
  --allow-write

Step 6 — Start the Argo CD API port-forward in a separate terminal and leave it running.

kubectl port-forward \
  service/argocd-server \
  --namespace argocd \
  8080:443

Step 7 — Retrieve the initial password and log in from the original terminal.

argocd_admin_password="$(kubectl get secret argocd-initial-admin-secret \
  --namespace argocd \
  --output jsonpath='{.data.password}' | base64 -D)"

argocd login localhost:8080 \
  --username admin \
  --password "${argocd_admin_password}" \
  --insecure

Step 8 — Register the private Git repository with Argo CD.

argocd repo add \
  git@github.com:davidleora/argocd-cronjob-gitops-lab.git \
  --name argocd-cronjob-gitops-lab \
  --ssh-private-key-path .local-secrets/argocd-repository

argocd repo list

Step 9 — Create the local Kustomize overlay.

mkdir -p deploy/overlays/local
cat > deploy/overlays/local/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: cronjob-study
resources:
  - ../../base
images:
  - name: argocd-cronjob-gitops-lab
    newName: argocd-cronjob-gitops-lab
    newTag: local
EOF

Step 10 — Render the overlay and confirm the image remains the Kind-loaded local image.

kustomize build deploy/overlays/local

Step 11 — Create the Argo CD Application manifest with automated sync, pruning, and self-healing.

mkdir -p argocd
cat > argocd/application.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cronjob-lab
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@github.com:davidleora/argocd-cronjob-gitops-lab.git
    targetRevision: main
    path: deploy/overlays/local
  destination:
    server: https://kubernetes.default.svc
    namespace: cronjob-study
  syncPolicy:
    automated:
      enabled: true
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
EOF

Step 12 — Commit and push the overlay and Application before creating the Application resource.

git add deploy/overlays/local argocd/application.yaml
git commit -m "Add Argo CD managed local overlay"
git push

Step 13 — Create the Argo CD Application in the cluster.

kubectl apply --filename argocd/application.yaml

Step 14 — Wait for Argo CD to synchronize and report a healthy application.

argocd app wait cronjob-lab \
  --sync \
  --health \
  --timeout 300

argocd app get cronjob-lab

Step 15 — Confirm the live CronJob matches the Git overlay.

kubectl get cronjob cronjob-lab \
  --namespace cronjob-study \
  --output yaml

Step 16 — Change the Git schedule from every two minutes to every three minutes.

sed -i '' \
  's|schedule: "\*/2 \* \* \* \*"|schedule: "\*/3 \* \* \* \*"|' \
  deploy/base/cronjob.yaml

git diff -- deploy/base/cronjob.yaml

Step 17 — Commit and push the desired-state change.

git add deploy/base/cronjob.yaml
git commit -m "Change CronJob schedule through GitOps"
git push

Step 18 — Wait for Argo CD and confirm the live schedule becomes every three minutes.

argocd app wait cronjob-lab --sync --timeout 300

kubectl get cronjob cronjob-lab \
  --namespace cronjob-study \
  --output jsonpath='{.spec.schedule}{"\n"}'

Completion check

Argo CD reports the private repository as Successful. The cronjob-lab Application is Synced and Healthy. The live CronJob schedule changes only after the Git commit is pushed. The deploy key files remain ignored under .local-secrets.

Phase 4

Building and Publishing a Multi-Platform Image After Every Merge

github-actionsghcrdockercontainer-imagesmulti-platform-buildarm64amd64ciimmutable-tagsprivate-registryimplementation

0 of 25 complete

Checklist

Step 1 — Open the repository and create a feature branch for the workflow.

cd /Users/davidleora/Projects/argocd-cronjob-gitops-lab
git switch main
git pull --ff-only
git switch -c phase-4-publish-images

Step 2 — Create a GitHub Actions workflow that publishes commit-SHA images for AMD64 and ARM64.

mkdir -p .github/workflows
cat > .github/workflows/publish-image.yaml <<'EOF'
name: Publish container image

on:
  push:
    branches:
      - main
    paths:
      - app/**
      - Dockerfile
      - .dockerignore
      - .github/workflows/publish-image.yaml

concurrency:
  group: publish-image-main
  cancel-in-progress: true

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - name: Check out the repository
        uses: actions/checkout@v6

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push the immutable image
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          platforms: linux/amd64,linux/arm64
          build-args: |
            APP_VERSION=sha-${{ github.sha }}
          tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}
EOF

Step 3 — Review the workflow and confirm its YAML indentation.

sed -n '1,240p' .github/workflows/publish-image.yaml
git diff --check

Step 4 — Commit and push the workflow branch.

git add .github/workflows/publish-image.yaml
git commit -m "Add multi-platform GHCR publishing workflow"
git push -u origin phase-4-publish-images

Step 5 — Open a pull request for the workflow.

gh pr create \
  --base main \
  --head phase-4-publish-images \
  --title "Add GHCR image publishing" \
  --body "Build and publish an immutable multi-platform image after application changes reach main."

Step 6 — Record the pull request number and merge it into main.

pr_number="$(gh pr view --json number --jq '.number')"
gh pr merge "${pr_number}" --squash --delete-branch

Step 7 — Read the merge commit SHA that should become the image tag.

merge_sha="$(gh pr view "${pr_number}" \
  --json mergeCommit \
  --jq '.mergeCommit.oid')"

printf 'expected_tag=sha-%s\n' "${merge_sha}"

Step 8 — Find and watch the workflow run created by the merge.

run_id="$(gh run list \
  --workflow publish-image.yaml \
  --branch main \
  --limit 1 \
  --json databaseId \
  --jq '.[0].databaseId')"

gh run watch "${run_id}" --exit-status

Step 9 — Refresh local GitHub authentication with package-reading permission if GHCR denies the next login.

gh auth refresh \
  --hostname github.com \
  --scopes read:packages

Step 10 — Log Docker into the private GHCR package without placing the token in shell history.

gh auth token | docker login ghcr.io \
  --username davidleora \
  --password-stdin

Step 11 — Pull the ARM64 variant of the newly published private image.

docker pull \
  "ghcr.io/davidleora/argocd-cronjob-gitops-lab:sha-${merge_sha}"

Step 12 — Run the published image and confirm its embedded version matches the merge SHA.

docker run --rm \
  "ghcr.io/davidleora/argocd-cronjob-gitops-lab:sha-${merge_sha}"

Step 13 — Return the local checkout to the updated main branch.

git switch main
git pull --ff-only
git status --short --branch

Step 14 — Create a local Kubernetes pull secret for the private GHCR package.

kubectl create secret docker-registry ghcr-pull \
  --namespace cronjob-study \
  --docker-server=ghcr.io \
  --docker-username=davidleora \
  --docker-password="$(gh auth token)" \
  --dry-run=client \
  --output yaml | kubectl apply --filename -

Step 15 — Create an overlay patch that attaches the pull secret to every CronJob Pod.

cat > deploy/overlays/local/image-pull-secret-patch.yaml <<'EOF'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cronjob-lab
spec:
  jobTemplate:
    spec:
      template:
        spec:
          imagePullSecrets:
            - name: ghcr-pull
EOF

Step 16 — Add the image-pull patch to the local Kustomize overlay.

cat >> deploy/overlays/local/kustomization.yaml <<'EOF'
patches:
  - path: image-pull-secret-patch.yaml
EOF

Step 17 — Replace the local image with the first published GHCR tag.

cd deploy/overlays/local
kustomize edit set image \
  "argocd-cronjob-gitops-lab=ghcr.io/davidleora/argocd-cronjob-gitops-lab:sha-${merge_sha}"
cd ../../..

Step 18 — Render the overlay and confirm the private image and pull secret are present.

kustomize build deploy/overlays/local | \
  grep -E 'image:|imagePullSecrets:|- name: ghcr-pull'

Step 19 — Commit and push the first GHCR deployment configuration.

git add deploy/overlays/local
git commit -m "Deploy first GHCR image through Argo CD"
git push

Step 20 — Wait for Argo CD and run the CronJob image manually.

argocd app wait cronjob-lab --sync --timeout 300

manual_job_name="ghcr-$(date +%s)"
kubectl create job \
  --from=cronjob/cronjob-lab \
  "${manual_job_name}" \
  --namespace cronjob-study

kubectl wait \
  --for=condition=complete \
  "job/${manual_job_name}" \
  --namespace cronjob-study \
  --timeout=180s

kubectl logs \
  "job/${manual_job_name}" \
  --namespace cronjob-study

Completion check

The GitHub Actions run succeeds. GHCR contains sha-<merge-sha> for AMD64 and ARM64. The private image runs locally on ARM64. Kubernetes can pull the image with ghcr-pull. The manual Job prints the merge SHA.

Phase 5

Updating the Desired Image Tag in Git After Every Merge

github-actionsgitopsargo-cdkustomizeghcrimage-tagsgit-write-backdeployment-automationimplementation

0 of 17 complete

Checklist

Step 1 — Open the repository and create a branch for the Git write-back workflow.

cd /Users/davidleora/Projects/argocd-cronjob-gitops-lab
git switch main
git pull --ff-only
git switch -c phase-5-update-image-tag

Step 2 — Replace the publishing workflow with a version that also commits the new Kustomize tag.

cat > .github/workflows/publish-image.yaml <<'EOF'
name: Publish container image

on:
  push:
    branches:
      - main
    paths:
      - app/**
      - Dockerfile
      - .dockerignore
      - .github/workflows/publish-image.yaml

concurrency:
  group: publish-image-main
  cancel-in-progress: true

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      packages: write
    steps:
      - name: Check out the repository
        uses: actions/checkout@v6
        with:
          fetch-depth: 0

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push the immutable image
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          platforms: linux/amd64,linux/arm64
          build-args: |
            APP_VERSION=sha-${{ github.sha }}
          tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}

      - name: Update the desired image tag in Git
        env:
          IMAGE_TAG: sha-${{ github.sha }}
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git fetch origin main
          git checkout main
          git pull --ff-only origin main
          sed -i "s|^[[:space:]]*newTag:.*|    newTag: ${IMAGE_TAG}|" \
            deploy/overlays/local/kustomization.yaml
          grep "newTag:" deploy/overlays/local/kustomization.yaml
          if git diff --quiet -- deploy/overlays/local/kustomization.yaml; then
            exit 0
          fi
          git add deploy/overlays/local/kustomization.yaml
          git commit -m "chore: deploy ${IMAGE_TAG}"
          git push origin main
EOF

Step 3 — Confirm only application and workflow paths trigger image builds so bot commits cannot loop.

sed -n '1,260p' .github/workflows/publish-image.yaml
git diff --check

Step 4 — Commit and push the workflow branch.

git add .github/workflows/publish-image.yaml
git commit -m "Update desired image tag after publishing"
git push -u origin phase-5-update-image-tag

Step 5 — Open and merge the workflow pull request.

gh pr create \
  --base main \
  --head phase-5-update-image-tag \
  --title "Automate GitOps image tag updates" \
  --body "Publish the merge image, commit its immutable tag to Kustomize, and let Argo CD deploy it."

pr_number="$(gh pr view --json number --jq '.number')"
gh pr merge "${pr_number}" --squash --delete-branch

Step 6 — Record the merge SHA expected in both GHCR and Kustomize.

merge_sha="$(gh pr view "${pr_number}" \
  --json mergeCommit \
  --jq '.mergeCommit.oid')"

printf 'expected_tag=sha-%s\n' "${merge_sha}"

Step 7 — Watch the workflow through image publishing and Git write-back.

run_id="$(gh run list \
  --workflow publish-image.yaml \
  --branch main \
  --limit 1 \
  --json databaseId \
  --jq '.[0].databaseId')"

gh run watch "${run_id}" --exit-status

Step 8 — Pull the bot commit and inspect the last three commits.

git switch main
git pull --ff-only
git log --oneline -3

Step 9 — Confirm Kustomize contains the merge SHA rather than the bot commit SHA.

grep "newTag:" deploy/overlays/local/kustomization.yaml
test "$(sed -n 's/^[[:space:]]*newTag: //p' deploy/overlays/local/kustomization.yaml)" = "sha-${merge_sha}"

Step 10 — Force an Argo CD refresh and wait for synchronization.

argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --timeout 300

Step 11 — Confirm the live CronJob template uses the Git-recorded image.

kubectl get cronjob cronjob-lab \
  --namespace cronjob-study \
  --output jsonpath='{.spec.jobTemplate.spec.template.spec.containers[0].image}{"\n"}'

Step 12 — Create a manual Job and verify its embedded version.

manual_job_name="gitops-$(date +%s)"
kubectl create job \
  --from=cronjob/cronjob-lab \
  "${manual_job_name}" \
  --namespace cronjob-study

kubectl wait \
  --for=condition=complete \
  "job/${manual_job_name}" \
  --namespace cronjob-study \
  --timeout=180s

kubectl logs \
  "job/${manual_job_name}" \
  --namespace cronjob-study

Completion check

One merge produces one sha-<merge-sha> image. The workflow creates one bot commit that changes only newTag. The bot commit does not trigger another workflow. Argo CD deploys the tag recorded in Git. The next Job prints the merge SHA.

Phase 6

Verifying the Complete Merge-to-CronJob GitOps Loop

argo-cdgitopsgithub-actionsghcrkubernetescronjobkustomizeend-to-end-testingverification

0 of 23 complete

Checklist

Step 1 — Open the latest main branch and confirm the working tree is clean.

cd /Users/davidleora/Projects/argocd-cronjob-gitops-lab
git switch main
git pull --ff-only
git status --short --branch

Step 2 — Create a feature branch for a visible application change.

git switch -c verify-complete-gitops-loop

Step 3 — Add a message that will appear only in the newly built container.

cat >> app/run.sh <<'EOF'
printf 'message=complete-gitops-loop\n'
EOF

sh -n app/run.sh
APP_VERSION=sha-local ./app/run.sh

Step 4 — Build and run the changed image locally before opening the pull request.

docker build \
  --build-arg APP_VERSION=sha-local-e2e \
  --tag argocd-cronjob-gitops-lab:e2e \
  .

docker run --rm argocd-cronjob-gitops-lab:e2e

Step 5 — Commit and push the application change.

git add app/run.sh
git commit -m "Add GitOps verification message"
git push -u origin verify-complete-gitops-loop

Step 6 — Open the verification pull request.

gh pr create \
  --base main \
  --head verify-complete-gitops-loop \
  --title "Verify the complete GitOps loop" \
  --body "Use a visible application message to verify merge, image publishing, Git write-back, Argo CD sync, and CronJob execution."

Step 7 — Record the pull request number and merge it.

pr_number="$(gh pr view --json number --jq '.number')"
gh pr merge "${pr_number}" --squash --delete-branch

Step 8 — Record the exact merge SHA used by the publishing workflow.

merge_sha="$(gh pr view "${pr_number}" \
  --json mergeCommit \
  --jq '.mergeCommit.oid')"

expected_tag="sha-${merge_sha}"
printf 'expected_tag=%s\n' "${expected_tag}"

Step 9 — Watch the matching GitHub Actions run until all steps finish.

run_id="$(gh run list \
  --workflow publish-image.yaml \
  --branch main \
  --limit 1 \
  --json databaseId \
  --jq '.[0].databaseId')"

gh run watch "${run_id}" --exit-status

Step 10 — Pull main and confirm the bot committed the expected tag.

git switch main
git pull --ff-only
git log --oneline -3
grep "newTag:" deploy/overlays/local/kustomization.yaml
test "$(sed -n 's/^[[:space:]]*newTag: //p' deploy/overlays/local/kustomization.yaml)" = "${expected_tag}"

Step 11 — Pull the exact private image from GHCR to confirm it exists.

docker pull \
  "ghcr.io/davidleora/argocd-cronjob-gitops-lab:${expected_tag}"

Step 12 — Refresh Argo CD and wait for Git and the cluster to synchronize.

argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --health --timeout 300

Step 13 — Capture the image recorded in Git and the image deployed in Kubernetes.

git_tag="$(sed -n 's/^[[:space:]]*newTag: //p' deploy/overlays/local/kustomization.yaml)"
live_image="$(kubectl get cronjob cronjob-lab \
  --namespace cronjob-study \
  --output jsonpath='{.spec.jobTemplate.spec.template.spec.containers[0].image}')"

printf 'git_tag=%s\n' "${git_tag}"
printf 'live_image=%s\n' "${live_image}"

Step 14 — Assert that the live image ends with the expected tag.

case "${live_image}" in
  *":${expected_tag}") printf 'image_match=true\n' ;;
  *) printf 'image_match=false\n'; exit 1 ;;
esac

Step 15 — Create a manual Job from the synchronized CronJob.

manual_job_name="e2e-$(date +%s)"
kubectl create job \
  --from=cronjob/cronjob-lab \
  "${manual_job_name}" \
  --namespace cronjob-study

Step 16 — Wait for completion and save the Job output.

kubectl wait \
  --for=condition=complete \
  "job/${manual_job_name}" \
  --namespace cronjob-study \
  --timeout=180s

job_output="$(kubectl logs \
  "job/${manual_job_name}" \
  --namespace cronjob-study)"

printf '%s\n' "${job_output}"

Step 17 — Assert that the Job reports both the merge SHA and the verification message.

printf '%s\n' "${job_output}" | grep "app_version=${expected_tag}"
printf '%s\n' "${job_output}" | grep "message=complete-gitops-loop"

Step 18 — Confirm the repository and Argo CD application finish cleanly.

git status --short --branch
argocd app get cronjob-lab
kubectl get cronjobs,jobs,pods --namespace cronjob-study

Completion check

The pull request merge SHA equals the GHCR image tag. The Kustomize tag equals the GHCR image tag. The live CronJob image equals the Kustomize image. The Job logs contain the merge SHA. The Job logs contain message=complete-gitops-loop.

Phase 7

Troubleshooting CronJob Failures and Performing Git Rollbacks

argo-cdkubernetescronjobgitopstroubleshootingimagepullbackoffjob-failureconcurrency-policyself-healingrollbackincident-response

0 of 28 complete

Exercise 1 — Invalid image tag

Step 1 — Open a clean main branch and record the currently working image.

cd /Users/davidleora/Projects/argocd-cronjob-gitops-lab
git switch main
git pull --ff-only
git status --short --branch

kubectl get cronjob cronjob-lab \
  --namespace cronjob-study \
  --output jsonpath='{.spec.jobTemplate.spec.template.spec.containers[0].image}{"\n"}'

Step 2 — Change the desired image to a tag that does not exist.

cd deploy/overlays/local
kustomize edit set image \
  argocd-cronjob-gitops-lab=ghcr.io/davidleora/argocd-cronjob-gitops-lab:sha-does-not-exist
cd ../../..

git diff -- deploy/overlays/local/kustomization.yaml

Step 3 — Commit and push the intentionally broken desired state.

git add deploy/overlays/local/kustomization.yaml
git commit -m "test: deploy a nonexistent image tag"
broken_commit="$(git rev-parse HEAD)"
git push

Step 4 — Refresh Argo CD and confirm the CronJob template contains the broken tag.

argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --timeout 300

kubectl get cronjob cronjob-lab \
  --namespace cronjob-study \
  --output jsonpath='{.spec.jobTemplate.spec.template.spec.containers[0].image}{"\n"}'

Step 5 — Create a Job that will fail to pull the image.

broken_job_name="broken-image-$(date +%s)"
kubectl create job \
  --from=cronjob/cronjob-lab \
  "${broken_job_name}" \
  --namespace cronjob-study

Step 6 — Watch the Pod until ImagePullBackOff appears and then stop with Control-C.

kubectl get pods \
  --namespace cronjob-study \
  --selector "job-name=${broken_job_name}" \
  --watch

Step 7 — Inspect the Pod events that explain the image-pull failure.

broken_pod_name="$(kubectl get pods \
  --namespace cronjob-study \
  --selector "job-name=${broken_job_name}" \
  --output jsonpath='{.items[0].metadata.name}')"

kubectl describe pod "${broken_pod_name}" --namespace cronjob-study

Step 8 — Revert the broken Git commit and push the rollback.

git revert --no-edit "${broken_commit}"
git push

Step 9 — Wait for Argo CD and confirm the working image returns.

argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --timeout 300

kubectl get cronjob cronjob-lab \
  --namespace cronjob-study \
  --output jsonpath='{.spec.jobTemplate.spec.template.spec.containers[0].image}{"\n"}'

Exercise 2 — Application exits unsuccessfully

Step 10 — Create a Kustomize patch that forces the application to exit with status 1.

cat > deploy/overlays/local/forced-failure-patch.yaml <<'EOF'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cronjob-lab
spec:
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: cronjob-lab
              env:
                - name: FORCE_FAILURE
                  value: "true"
EOF

cat >> deploy/overlays/local/kustomization.yaml <<'EOF'
  - path: forced-failure-patch.yaml
EOF

Step 11 — Render the overlay and confirm FORCE_FAILURE is present.

kustomize build deploy/overlays/local | grep -A3 FORCE_FAILURE

Step 12 — Commit and push the failure configuration.

git add deploy/overlays/local
git commit -m "test: force the CronJob application to fail"
failure_commit="$(git rev-parse HEAD)"
git push

Step 13 — Wait for Argo CD and create a failing manual Job.

argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --timeout 300

failure_job_name="forced-failure-$(date +%s)"
kubectl create job \
  --from=cronjob/cronjob-lab \
  "${failure_job_name}" \
  --namespace cronjob-study

Step 14 — Wait for the Job to fail and inspect its logs and status.

kubectl wait \
  --for=condition=failed \
  "job/${failure_job_name}" \
  --namespace cronjob-study \
  --timeout=240s

kubectl logs \
  "job/${failure_job_name}" \
  --namespace cronjob-study

kubectl describe \
  "job/${failure_job_name}" \
  --namespace cronjob-study

Step 15 — Revert the forced-failure configuration and push the rollback.

git revert --no-edit "${failure_commit}"
git push
argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --timeout 300

Exercise 3 — Forbid concurrent executions

Step 16 — Create a patch that runs every minute but sleeps for three minutes.

cat > deploy/overlays/local/long-running-patch.yaml <<'EOF'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cronjob-lab
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      activeDeadlineSeconds: 240
      template:
        spec:
          containers:
            - name: cronjob-lab
              env:
                - name: SLEEP_SECONDS
                  value: "180"
EOF

cat >> deploy/overlays/local/kustomization.yaml <<'EOF'
  - path: long-running-patch.yaml
EOF

Step 17 — Commit and push the long-running configuration.

git add deploy/overlays/local
git commit -m "test: verify forbidden CronJob concurrency"
concurrency_commit="$(git rev-parse HEAD)"
git push

argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --timeout 300

Step 18 — Watch Jobs for several schedule boundaries and stop with Control-C.

kubectl get jobs \
  --namespace cronjob-study \
  --watch

Step 19 — Confirm only one Job is active and inspect skipped-schedule events.

kubectl get jobs --namespace cronjob-study
kubectl describe cronjob cronjob-lab --namespace cronjob-study

Step 20 — Revert the long-running configuration and push the rollback.

git revert --no-edit "${concurrency_commit}"
git push
argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --timeout 300

Exercise 4 — Argo CD self-healing

Step 21 — Manually suspend the live CronJob without changing Git.

kubectl patch cronjob cronjob-lab \
  --namespace cronjob-study \
  --type merge \
  --patch '{"spec":{"suspend":true}}'

Step 22 — Watch Argo CD restore the Git-declared value and stop with Control-C after it changes back.

kubectl get cronjob cronjob-lab \
  --namespace cronjob-study \
  --watch \
  --output custom-columns='NAME:.metadata.name,SUSPEND:.spec.suspend'

Step 23 — Confirm the application and repository finish cleanly.

argocd app get cronjob-lab
git status --short --branch
git log --oneline -8

Completion check

ImagePullBackOff was diagnosed from Pod events. A nonzero application exit was diagnosed from Job logs. concurrencyPolicy=Forbid prevented overlapping Jobs. Argo CD reversed a manual live-cluster change. Every persisted rollback was performed with git revert.

Optional phase 8

Comparing Argo CD Image Updater with CI-Driven Tag Updates

argo-cdargo-cd-image-updaterkubernetesgitopsghcrprivate-registrykustomizenewest-buildgit-write-backcomparisonoptional

0 of 22 complete

Checklist

Step 1 — Open the latest main branch and create the comparison branch.

cd /Users/davidleora/Projects/argocd-cronjob-gitops-lab
git switch main
git pull --ff-only
git switch -c phase-8-image-updater

Step 2 — Replace the workflow with a build-only version so CI stops editing Kustomize.

cat > .github/workflows/publish-image.yaml <<'EOF'
name: Publish container image

on:
  push:
    branches:
      - main
    paths:
      - app/**
      - Dockerfile
      - .dockerignore
      - .github/workflows/publish-image.yaml

concurrency:
  group: publish-image-main
  cancel-in-progress: true

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - name: Check out the repository
        uses: actions/checkout@v6

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push the immutable image
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          platforms: linux/amd64,linux/arm64
          build-args: |
            APP_VERSION=sha-${{ github.sha }}
          tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}
EOF

Step 3 — Commit, push, and merge the build-only workflow.

git add .github/workflows/publish-image.yaml
git commit -m "Delegate image tag updates to Argo CD Image Updater"
git push -u origin phase-8-image-updater

gh pr create \
  --base main \
  --head phase-8-image-updater \
  --title "Compare Argo CD Image Updater" \
  --body "Keep image publishing in CI and move Git tag write-back to Argo CD Image Updater."

pr_number="$(gh pr view --json number --jq '.number')"
gh pr merge "${pr_number}" --squash --delete-branch

Step 4 — Watch the workflow publish an image without creating a tag-update commit.

run_id="$(gh run list \
  --workflow publish-image.yaml \
  --branch main \
  --limit 1 \
  --json databaseId \
  --jq '.[0].databaseId')"

gh run watch "${run_id}" --exit-status

git switch main
git pull --ff-only
git log --oneline -3

Step 5 — Install Argo CD Image Updater into the Argo CD namespace.

kubectl apply \
  --namespace argocd \
  --filename https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/config/install.yaml

kubectl wait \
  --for=condition=Available \
  deployment/argocd-image-updater \
  --namespace argocd \
  --timeout=300s

Step 6 — Create a same-namespace pull secret so Image Updater can scan the private GHCR package.

kubectl create secret docker-registry ghcr-image-updater \
  --namespace argocd \
  --docker-server=ghcr.io \
  --docker-username=davidleora \
  --docker-password="$(gh auth token)" \
  --dry-run=client \
  --output yaml | kubectl apply --filename -

Step 7 — Confirm Argo CD still has the write-enabled SSH repository credential created in Phase 3.

argocd repo list
gh repo deploy-key list \
  --repo davidleora/argocd-cronjob-gitops-lab

Step 8 — Create a CRD-based ImageUpdater resource that scans only full commit-SHA tags and writes Kustomize.

cat > argocd/image-updater.yaml <<'EOF'
apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: cronjob-lab
  namespace: argocd
spec:
  writeBackConfig:
    method: git
    gitConfig:
      repository: git@github.com:davidleora/argocd-cronjob-gitops-lab.git
      branch: main
      writeBackTarget: kustomization
  applicationRefs:
    - namePattern: cronjob-lab
      images:
        - alias: cronjob
          imageName: ghcr.io/davidleora/argocd-cronjob-gitops-lab
          commonUpdateSettings:
            updateStrategy: newest-build
            allowTags: "regexp:^sha-[0-9a-f]{40}$"
            pullSecret: "pullsecret:argocd/ghcr-image-updater"
          manifestTargets:
            kustomize:
              name: argocd-cronjob-gitops-lab
EOF

Step 9 — Commit and push the ImageUpdater configuration without applying it through Argo CD.

git add argocd/image-updater.yaml
git commit -m "Add CRD-based Argo CD Image Updater configuration"
git push

Step 10 — Apply the ImageUpdater resource manually to the Argo CD namespace.

kubectl apply --filename argocd/image-updater.yaml
kubectl get imageupdater --namespace argocd

Step 11 — Watch the controller reconcile the newest GHCR tag and stop with Control-C after a successful update.

kubectl logs \
  deployment/argocd-image-updater \
  --namespace argocd \
  --follow

Step 12 — Inspect the ImageUpdater status and recent update information.

kubectl get imageupdater cronjob-lab \
  --namespace argocd \
  --output yaml

Step 13 — Pull the Image Updater commit and confirm it changed the Kustomize image.

git pull --ff-only
git log --oneline -5
grep -A3 '^images:' deploy/overlays/local/kustomization.yaml

Step 14 — Wait for Argo CD to deploy the Image Updater commit.

argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --timeout 300

kubectl get cronjob cronjob-lab \
  --namespace cronjob-study \
  --output jsonpath='{.spec.jobTemplate.spec.template.spec.containers[0].image}{"\n"}'

Step 15 — Make and merge one more application change without editing Kustomize.

git switch -c verify-image-updater
cat >> app/run.sh <<'EOF'
printf 'image_updater_verification=success\n'
EOF

git add app/run.sh
git commit -m "Add Image Updater verification output"
git push -u origin verify-image-updater

gh pr create \
  --base main \
  --head verify-image-updater \
  --title "Verify Argo CD Image Updater" \
  --body "Publish a new immutable image and let Image Updater select and commit its tag."

pr_number="$(gh pr view --json number --jq '.number')"
gh pr merge "${pr_number}" --squash --delete-branch

Step 16 — Watch the build, Image Updater commit, and Argo CD sync as three separate reconciliations.

run_id="$(gh run list \
  --workflow publish-image.yaml \
  --branch main \
  --limit 1 \
  --json databaseId \
  --jq '.[0].databaseId')"

gh run watch "${run_id}" --exit-status

kubectl logs \
  deployment/argocd-image-updater \
  --namespace argocd \
  --follow

Step 17 — Stop the log follow with Control-C, pull the updater commit, and verify the deployed Job.

git switch main
git pull --ff-only
git log --oneline -5

argocd app get cronjob-lab --hard-refresh
argocd app wait cronjob-lab --sync --timeout 300

manual_job_name="image-updater-$(date +%s)"
kubectl create job \
  --from=cronjob/cronjob-lab \
  "${manual_job_name}" \
  --namespace cronjob-study

kubectl wait \
  --for=condition=complete \
  "job/${manual_job_name}" \
  --namespace cronjob-study \
  --timeout=180s

kubectl logs \
  "job/${manual_job_name}" \
  --namespace cronjob-study

Completion check

CI builds and pushes images but does not edit Git. Image Updater detects only sha-<40-hex-character> tags. Image Updater commits the new Kustomize tag to main. Argo CD deploys the Image Updater commit. The Job prints image_updater_verification=success.