LearnComputing FoundationsGit, Shell, Linux for AI

⚙️EasyMLOps & Deployment

Git, Shell, Linux for AI

Build a reproducible AI repo with safe Git defaults, a clean-clone eval gate, shell diagnostics, and Linux process habits.

11 min read

Learning path

Step 1 of 177 in the full curriculum

Docker for Reproducible AI

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Before Python, Docker, or tests, you need a repo that survives a fresh clone. This baseline includes safe Git defaults, one reproducible eval command, shell checks that tell you what machine and dataset you're using, and Linux habits that keep long jobs alive.

One tiny access-request eval file runs through the whole lesson. Another machine should be able to clone the repo, run one command, and get the same 0.667 result instead of "it worked on my laptop." Git's snapshot model is built so a clean clone gets the same tracked files and history.^{[1]Reference 1Pro Git (2nd ed.)https://git-scm.com/book/en/v2} Identical eval behavior still needs an explicit environment contract (activation script, pinned deps, and later a container), not the Git tree alone.

Tracked vs local-only artifacts for access-rag: eval fixture and activate/repro scripts travel with a clean clone, while .venv, .env, and Git hooks stay machine-local; laptop and GPU box both print 2/3 correct and score 0.667. — Follow the three-row access-request eval.jsonl. On the laptop it becomes a committed, gated artifact. On the GPU box the same three rows produce the same 0.667 because the environment contract (gitignore, activation script, shell helpers) traveled with the repo.

Start with the smallest useful repo

Create a new directory and initialize Git as you would for any real AI project.

terminal

mkdir access-rag && cd access-rag
git init

The .git directory is the repository's memory. Everything that follows will be tracked or explicitly ignored.

The .gitignore that protects AI work

Create .gitignore with the patterns that real LLM projects need:

.gitignore

# Python
__pycache__/
*.py[cod]
*$py.class
.venv/
env/
ENV/

# Environment & secrets (do not commit these)
.env
.env.local
*.pem
secrets/

# Generated model and vector artifacts that do not travel with the repo
checkpoints/
models/local/
*.pt
*.pth
chroma/
faiss_index/
*.db
*.sqlite3

# OS and editor noise
.DS_Store
.idea/
.vscode/
*.swp

# Evaluation caches that should be regenerated
eval_cache/
runs/
wandb/
mlruns/

# Do not ignore *.gguf, *.safetensors, or *.bin here.
# .gitattributes routes those extensions through Git LFS when configured.

Track large files with Git LFS (Large File Storage) so the repo stays small while model weights travel with the project when needed. GitHub warns on files over 50 MiB and hard-blocks any single file over 100 MiB; LFS replaces the file in history with a small pointer and stores the bytes separately.^{[2]Reference 2About large files on GitHubhttps://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github} A path can't be both ignored and LFS-tracked, so the LFS-managed extensions stay out of .gitignore; generated checkpoints and local model directories remain ignored. Git LFS is a separate tool, so check for it first. If it isn't installed yet, don't add those model files. Without git lfs pull (or with GIT_LFS_SKIP_SMUDGE=1), clones keep tiny pointer files that look like valid paths until a weight load fails. Production datasets often outgrow LFS quotas; then use object-store manifests or tools such as DVC, and keep only the pointer/manifest in Git.

terminal-2

cat > .gitattributes << 'EOF'
# Install Git LFS before committing model weights:
# git lfs track "*.gguf" "*.safetensors" "*.bin"
EOF

if command -v git-lfs >/dev/null 2>&1; then
  git lfs install
  git lfs track "*.gguf" "*.safetensors" "*.bin"
else
  echo "Git LFS is not installed. Safe for now: do not commit model weights yet."
fi

git add .gitattributes

Commit the skeleton.

terminal-3

git add .gitignore .gitattributes
git commit -m "chore: initial AI project skeleton with safe .gitignore and LFS"

This repo can be cloned without leaking keys or placing a 7 GB model blob in ordinary Git history. When LFS-managed weights exist, document whether setup should fetch them immediately or only for workflows that need them.

The eval that must travel with the code

Place the three-row access-request evaluation file that the rest of the curriculum will reuse.

terminal-4

mkdir -p eval
cat > eval/access_requests.jsonl << 'EOF'
{"prompt": "Access request 101 status?", "expected": "approved", "prediction": "approved"}
{"prompt": "Access request 102 status?", "expected": "blocked", "prediction": "escalated"}
{"prompt": "Access request 103 status?", "expected": "restored", "prediction": "restored"}
EOF

This tiny file is the contract. The same three rows also live in assets/access_requests.jsonl so the runnable scorer below can open a real fixture without depending on your laptop path.

Add the provided scorer

The shell gate needs a small program that reads JSONL rows and computes exact-match accuracy. Don't treat this Git-and-shell chapter as a Python prerequisite: copy the standard-library scorer as a provided utility and focus on its command-line contract. The later Python lesson explains functions, exceptions, tests, and CI before asking you to extend it.

Diagram showing git clone, install hooks, source activate.sh, and ./repro.sh. — git clone, install hooks, source activate.sh, and ./repro.sh.

The utility has one job: validate the three required fields, print the exact-match receipt, and exit nonzero when the fixture is missing, malformed, empty, or below the 2/3 gate. Save it as scripts/score_access_requests.py; scripts/run_eval.sh will invoke it on every clone.

scripts/score_access_requests.py

import json
import sys
from pathlib import Path

REQUIRED = ("prompt", "expected", "prediction")
EVAL_FILE = Path("eval/access_requests.jsonl")
ASSET_FILE = Path("assets/access_requests.jsonl")

path = EVAL_FILE if EVAL_FILE.exists() else ASSET_FILE
if not path.exists():
    print(f"ERROR: {path} missing; commit the eval fixture", file=sys.stderr)
    raise SystemExit(1)

rows = []
for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1):
    if not line.strip():
        continue
    try:
        row = json.loads(line)
    except json.JSONDecodeError as error:
        print(f"ERROR: line {line_number}: {error.msg}", file=sys.stderr)
        raise SystemExit(1)
    missing = [field for field in REQUIRED if field not in row]
    if missing:
        print(f"ERROR: line {line_number} missing {missing}", file=sys.stderr)
        raise SystemExit(1)
    rows.append(row)

if not rows:
    print("ERROR: eval file is empty", file=sys.stderr)
    raise SystemExit(1)

def normalize(label: str) -> str:
    return str(label).strip().lower()

correct = sum(normalize(row["expected"]) == normalize(row["prediction"]) for row in rows)
total = len(rows)
score = correct / total
print(f"Eval rows: {total}")
print(f"Exact-match accuracy on tiny fixture: {score:.3f} ({correct}/{total})")
# Count gate: fail below 2/3 correct (same rule Docker and Python reuse).
if correct * 3 < total * 2:
    print("Gate failed: score regressed below 2/3", file=sys.stderr)
    raise SystemExit(1)
print("Gate passed. You may commit.")

Output

Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.

Normalization is part of the shared contract: labels are compared after strip().lower(), and the gate uses counts (correct * 3 < total * 2), not a float equality check against 0.667. Success prints 3, 0.667, and exits zero; bad input or a regression exits nonzero.

The pre-commit gate that protects the score

Create a tiny executable that the pre-commit hook and clean-clone reproduction command will run.

terminal-5

mkdir -p scripts
cat > scripts/run_eval.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

EVAL_FILE="eval/access_requests.jsonl"
if [[ ! -f "$EVAL_FILE" ]]; then
  echo "ERROR: $EVAL_FILE missing. Did you forget to commit the fixture or pull the latest repo?"
  exit 1
fi

PYTHON_BIN="${PYTHON_BIN:-python3}"
if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
  echo "ERROR: Python 3 is required by the provided scorer. Install it or set PYTHON_BIN."
  exit 1
fi

if [[ -f scripts/score_access_requests.py ]]; then
  "$PYTHON_BIN" scripts/score_access_requests.py
else
  echo "ERROR: scripts/score_access_requests.py missing. Add the Python scorer from this chapter."
  exit 1
fi
EOF
chmod +x scripts/run_eval.sh

Create a repo-local reproduction command. This is important: shell aliases and Git hooks are local machine state, but repro.sh travels with the repo.

terminal-6

cat > repro.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

./scripts/run_eval.sh
EOF
chmod +x repro.sh

Now wire the same gate as a pre-commit hook. Don't try to commit .git/hooks/pre-commit; files under .git/ are Git internals, not normal tracked project files. Commit the hook source under scripts/, then install it into .git/hooks/ on each clone.

terminal-7

cat > scripts/pre-commit-ai-eval.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

echo "Running AI eval gate before commit..."
./scripts/run_eval.sh
echo "Eval gate passed."
EOF

cat > scripts/install_hooks.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

mkdir -p .git/hooks
cp scripts/pre-commit-ai-eval.sh .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "Installed .git/hooks/pre-commit"
EOF

chmod +x scripts/pre-commit-ai-eval.sh scripts/install_hooks.sh
./scripts/install_hooks.sh

Test it.

terminal-8

./repro.sh
git add scripts/score_access_requests.py scripts/run_eval.sh scripts/pre-commit-ai-eval.sh scripts/install_hooks.sh repro.sh eval/access_requests.jsonl
git commit -m "feat: add three-row eval and pre-commit gate that protects 0.667"

Output

Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.
Running AI eval gate before commit...
Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.
Eval gate passed.

If the scorer reports a regression or the file disappears, the commit is rejected before the broken state enters history.

Shell one-liners that make the invisible visible

Add a few reusable functions to ~/.zshrc or ~/.bashrc.

For NVIDIA machines, nvidia-smi --query-gpu with --format=csv is the scriptable query interface you'll want in shell helpers.^{[3]Reference 3nvidia-smi documentationhttps://docs.nvidia.com/deploy/nvidia-smi/index.html}

terminal-9

# GPU snapshot (works on NVIDIA, falls back gracefully)
gpu() {
  if command -v nvidia-smi >/dev/null 2>&1; then
    nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu --format=csv,noheader
  else
    echo "No NVIDIA GPU or nvidia-smi not in PATH"
  fi
}

# Dataset size at a glance
ds() {
  du -sh "${1:-.}" 2>/dev/null | awk '{print $1 "  " $2}'
  echo "JSONL rows: $(find "${1:-.}" -name '*.jsonl' -exec wc -l {} + 2>/dev/null | tail -1 | awk '{print $1}')"
}

# One-command reproduction of the current eval
repro() {
  if [[ -x ./repro.sh ]]; then
    ./repro.sh
  elif [[ -x ./scripts/run_eval.sh ]]; then
    ./scripts/run_eval.sh
  elif [[ -x ./reproduce.sh ]]; then
    ./reproduce.sh
  else
    echo "No reproducible entrypoint found (looked for repro.sh, scripts/run_eval.sh, or reproduce.sh)"
    return 1
  fi
}

After sourcing, gpu, ds, and repro become muscle memory. You type one word and immediately know whether the machine has the resources the workload expects. In a clean clone, use ./repro.sh; aliases should make the common path faster, not hide the real entry point.

Inspect data and processes without crashing the box

Two shell checks cover common remote-machine failures: inspecting a dataset that's too large to open and finding a process that still holds GPU memory. For the first, never run cat train.jsonl on a multi-gigabyte file. It floods the terminal and stalls a remote host. Stream the file instead so each tool reads a little, passes it on, and keeps memory use flat.

terminal-inspect

head -n 1 eval/access_requests.jsonl    # peek at the schema of one row
wc -l eval/access_requests.jsonl        # count rows without loading the file
grep -c '"expected"' eval/access_requests.jsonl  # how many rows have the field

The pipe | chains these into one pass. grep '"restored"' eval/access_requests.jsonl | wc -l filters, then counts, without ever holding the whole file in memory.

The second reflex is reclaiming a GPU from a hung or detached job. A PyTorch process can remain alive after its controlling shell exits, and nvidia-smi reports currently active compute processes.^{[3]Reference 3nvidia-smi documentationhttps://docs.nvidia.com/deploy/nvidia-smi/index.html} When the process table shows a PID, inspect it, ask it to exit cleanly, and only force-kill if it refuses.

terminal-kill

nvidia-smi                       # read the PID in the bottom "Processes" table
kill 12345                       # SIGTERM (15): let the process flush and release VRAM
for _ in {1..10}; do             # give graceful shutdown up to ten seconds
  kill -0 12345 2>/dev/null || break
  sleep 1
done
kill -0 12345 2>/dev/null && kill -9 12345   # escalate only if it is still alive

If VRAM remains allocated but nvidia-smi shows no owning compute PID, there's no process ID to kill. Check container or PID-namespace visibility and other driver clients, then escalate to container-runtime or driver diagnosis instead of signaling a nonexistent process.

Start with SIGTERM. Signal 15 lets the process clean up, close files, and release CUDA memory. SIGKILL (signal 9) can't be caught or handled and risks leaving lock files or corrupt checkpoints behind, so reserve it for a process that refuses to exit.^{[4]Reference 4The Open Group Base Specifications: signal.hhttps://pubs.opengroup.org/onlinepubs/9699919799/basedefs/signal.h.html}

Linux fundamentals that keep long jobs alive

Training and evaluation jobs can run for hours, often through an SSH session. These commands keep the job alive and make its resource use visible:

Detach a process that survives logout: nohup python train.py > train.log 2>&1 &
Manage sessions across SSH disconnects: tmux new -s training, tmux attach -t training, Ctrl-b d to detach.
Give the job lower priority so your laptop remains usable: nice -n 10 python train.py
Find the files using disk right now: du -ah /workspace | sort -rh | head -20
Check which Python process is using the GPU: nvidia-smi + ps aux | grep python

Together, these commands keep a job alive across disconnects, lower its CPU priority when needed, and show which files or processes are consuming resources.

The reproducible activation contract

Create an activation script that each teammate and each CI job can source. This first chapter doesn't require PyTorch yet, so the script reports CUDA when torch is already installed.

terminal-10

cat > requirements.txt << 'EOF'
# Empty in this first chapter.
# Later chapters will add pinned runtime packages here.
EOF

cat > .env.example << 'EOF'
# Copy to .env locally. Never commit .env.
ACCESS_API_KEY=replace-me
HF_HOME=
EOF

cat > activate.sh << 'EOF'
#!/usr/bin/env bash

_activate_fail() {
  echo "ERROR: $1" >&2
}

_activate_main() {
  # Prefer an explicit interpreter so clones don't silently use 3.10 vs 3.12.
  # Optional: pin with a `.python-version` file and tools like uv or pyenv.
  PYTHON_BIN="${PYTHON_BIN:-python3}"
  if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
    _activate_fail "install python3 or set PYTHON_BIN=/path/to/python"
    return 1
  fi

  # 1. Create or reuse a local virtualenv
  if [[ ! -d .venv ]]; then
    "$PYTHON_BIN" -m venv .venv || {
      _activate_fail "could not create .venv"
      return 1
    }
  fi
  source .venv/bin/activate || {
    _activate_fail "could not activate .venv"
    return 1
  }

  # 2. Install declared dependencies only when requirements.txt has entries.
  # Prefer a locked install once you add packages: `uv lock` + `uv sync`,
  # or `pip-compile` + `pip install -r requirements.txt --require-hashes`.
  grep -Ev '^\s*(#|$)' requirements.txt >/dev/null 2>&1
  _requirements_status=$?
  if [[ $_requirements_status -eq 0 ]]; then
    python -m pip install -r requirements.txt || {
      _activate_fail "could not install requirements.txt"
      return 1
    }
  elif [[ $_requirements_status -gt 1 ]]; then
    _activate_fail "could not read requirements.txt"
    return 1
  fi
  unset _requirements_status

  # 3. Print the local environment without requiring GPU packages yet
  if ! python - << 'PY'
import os, sys
print("Python:", sys.version.split()[0])
if sys.version_info[:2] < (3, 12):
    print("WARNING: this project targets Python 3.12+; set PYTHON_BIN to a 3.12 interpreter")
visible_devices = os.environ.get("CUDA_VISIBLE_DEVICES")
print(
    "CUDA_VISIBLE_DEVICES:",
    visible_devices if visible_devices is not None else "<unset; no mask applied>",
)
try:
    import torch
except ModuleNotFoundError:
    print("PyTorch: not installed yet (OK for this chapter)")
else:
    print("PyTorch:", torch.__version__)
    print("CUDA available:", torch.cuda.is_available())
    if torch.cuda.is_available():
        print("GPU:", torch.cuda.get_device_name(0))
print("HF_HOME:", os.environ.get("HF_HOME", "(default ~/.cache/huggingface)"))
PY
  then
    _activate_fail "environment probe failed"
    return 1
  fi

  echo "Environment ready. Run './repro.sh' to execute the eval gate."
}

_activate_main
_activate_status=$?
unset -f _activate_main _activate_fail
if [[ $_activate_status -ne 0 ]]; then
  return "$_activate_status" 2>/dev/null || exit "$_activate_status"
fi
unset _activate_status
EOF
chmod +x activate.sh

The outer status check is essential for a sourced script. A helper that merely return 1s can fail while the rest of activate.sh continues and its final echo makes the source command look successful. Here every setup failure returns from _activate_main, and the file-level return propagates that nonzero status to source activate.sh without closing an interactive shell.

Document it in README.md:

README.md

## Quick start

git clone [email protected]:your-org/access-rag.git
cd access-rag
./scripts/install_hooks.sh
if command -v git-lfs >/dev/null 2>&1; then
  git lfs pull
fi
source activate.sh && ./repro.sh

Now a fresh engineer (or a fresh GPU box provisioned by your platform team) can reproduce the same 0.667 result through the tracked setup path.

Activation fail-early pipeline: tracked activate.sh and repro.sh create the environment, print Python CUDA and HF_HOME truth, stop on missing setup, and only then run the three-row eval gate that must print 0.667. — The activation contract is small but strict. Tracked files create environment, print machine truth, and route every clone to same repro command instead of hidden laptop state.

The failure modes you'll see in real life

Symptom	Most common cause	Fix that belongs in the repo
`CUDA not found` on the GPU box	Host driver or container runtime is missing, the image lacks a compatible CUDA stack, or `CUDA_VISIBLE_DEVICES` is explicitly empty/invalid and hides the GPU	Compare `nvidia-smi`, the printed `CUDA_VISIBLE_DEVICES`, and `torch.cuda.is_available()`; remove an accidental mask or choose a valid device index, then document the required image and runtime. An unset variable normally exposes all available devices.
`ModuleNotFoundError` for a package that worked on the laptop	`requirements.txt` is incomplete or uses unpinned versions	Rebuild from a clean env, pin direct deps (and prefer a lockfile such as `uv.lock` or hashed `requirements.txt`), commit the pins, and reinstall from that file
Eval returns 0.000 because the three-row JSONL is missing	`.gitignore` did not protect the generated cache directory that the author had on disk	Move the fixture to `eval/` and add the directory to the committed tree; don't rely on "I had it in my downloads folder"
Pre-commit hook doesn't run on a fresh clone	Hooks under `.git/hooks/` are local machine files, not tracked project files	Commit `scripts/pre-commit-ai-eval.sh` and `scripts/install_hooks.sh`, then run the installer after cloning
Pre-commit hook fails with "permission denied"	The hook script was installed without `chmod +x` or the clone was on a filesystem that strips execute bits	`chmod +x scripts/.sh .git/hooks/` + a one-line check in the installer
"It worked yesterday" after a `git pull`	teammate committed a new large model without LFS or changed the expected schema of the eval file	LFS tracking + a schema validation step in the scorer + `git diff` before each `git pull` on data files

Encode each diagnosis and prevention in the repo so the next clone fails clearly instead of requiring machine-specific guesswork.

The repo contract

The first real engineering loop is in place:

git clone + source activate.sh produces a working environment on any machine with the declared dependencies.
repro (or the pre-commit hook) guarantees that the tiny contract (three rows, 0.667) is still satisfied after each change.
gpu, ds, and the Linux session commands let you see what the hardware is doing.
The .gitignore + LFS rules + activation script travel with the code, so the next person doesn't have to reverse-engineer your laptop.

Self check: clone the repo into a fresh temporary directory and run ./scripts/install_hooks.sh && source activate.sh && ./repro.sh. The expected output is the same 0.667 score plus a visible environment summary. If the command needs a hidden laptop file, name that missing contract, add it to .env.example, README, LFS, or the activation script, and rerun the clean-clone check.

Reproducible change checklist

Before sharing a commit, inspect git status, run the repo-local eval, clone into a temporary directory, install tracked hooks, source activate.sh, and run ./repro.sh. Any dependency on an untracked file becomes a named repo contract before handoff.

Next Step

Continue to Docker for Reproducible AI

Your repo now carries its fixture, setup path, and eval gate to a clean clone. Docker adds the missing runtime contract so the same command runs with <span data-glossary="dependency-lockfile">pinned dependencies</span>, isolated <span data-glossary="secrets-management">secrets</span>, and a predictable filesystem on teammate and CI machines.

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Pro Git (2nd ed.)

Chacon, S. & Straub, B. · 2014

https://git-scm.com/book/en/v2

About large files on GitHub

GitHub · 2026

https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github

nvidia-smi documentation

NVIDIA · 2026

https://docs.nvidia.com/deploy/nvidia-smi/index.html

The Open Group Base Specifications: signal.h

The Open Group · 2018

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/signal.h.html

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Back to Topics

LearnComputing FoundationsGit, Shell, Linux for AI

⚙️EasyMLOps & Deployment

Git, Shell, Linux for AI

Build a reproducible AI repo with safe Git defaults, a clean-clone eval gate, shell diagnostics, and Linux process habits.

11 min read

Learning path

Step 1 of 177 in the full curriculum

Docker for Reproducible AI

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Start with the smallest useful repo

Create a new directory and initialize Git as you would for any real AI project.

terminal

mkdir access-rag && cd access-rag
git init

The .git directory is the repository's memory. Everything that follows will be tracked or explicitly ignored.

The .gitignore that protects AI work

Create .gitignore with the patterns that real LLM projects need:

.gitignore

# Python
__pycache__/
*.py[cod]
*$py.class
.venv/
env/
ENV/

# Environment & secrets (do not commit these)
.env
.env.local
*.pem
secrets/

# Generated model and vector artifacts that do not travel with the repo
checkpoints/
models/local/
*.pt
*.pth
chroma/
faiss_index/
*.db
*.sqlite3

# OS and editor noise
.DS_Store
.idea/
.vscode/
*.swp

# Evaluation caches that should be regenerated
eval_cache/
runs/
wandb/
mlruns/

# Do not ignore *.gguf, *.safetensors, or *.bin here.
# .gitattributes routes those extensions through Git LFS when configured.

terminal-2

cat > .gitattributes << 'EOF'
# Install Git LFS before committing model weights:
# git lfs track "*.gguf" "*.safetensors" "*.bin"
EOF

if command -v git-lfs >/dev/null 2>&1; then
  git lfs install
  git lfs track "*.gguf" "*.safetensors" "*.bin"
else
  echo "Git LFS is not installed. Safe for now: do not commit model weights yet."
fi

git add .gitattributes

Commit the skeleton.

terminal-3

git add .gitignore .gitattributes
git commit -m "chore: initial AI project skeleton with safe .gitignore and LFS"

The eval that must travel with the code

Place the three-row access-request evaluation file that the rest of the curriculum will reuse.

terminal-4

mkdir -p eval
cat > eval/access_requests.jsonl << 'EOF'
{"prompt": "Access request 101 status?", "expected": "approved", "prediction": "approved"}
{"prompt": "Access request 102 status?", "expected": "blocked", "prediction": "escalated"}
{"prompt": "Access request 103 status?", "expected": "restored", "prediction": "restored"}
EOF

This tiny file is the contract. The same three rows also live in assets/access_requests.jsonl so the runnable scorer below can open a real fixture without depending on your laptop path.

Add the provided scorer

scripts/score_access_requests.py

import json
import sys
from pathlib import Path

REQUIRED = ("prompt", "expected", "prediction")
EVAL_FILE = Path("eval/access_requests.jsonl")
ASSET_FILE = Path("assets/access_requests.jsonl")

path = EVAL_FILE if EVAL_FILE.exists() else ASSET_FILE
if not path.exists():
    print(f"ERROR: {path} missing; commit the eval fixture", file=sys.stderr)
    raise SystemExit(1)

rows = []
for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1):
    if not line.strip():
        continue
    try:
        row = json.loads(line)
    except json.JSONDecodeError as error:
        print(f"ERROR: line {line_number}: {error.msg}", file=sys.stderr)
        raise SystemExit(1)
    missing = [field for field in REQUIRED if field not in row]
    if missing:
        print(f"ERROR: line {line_number} missing {missing}", file=sys.stderr)
        raise SystemExit(1)
    rows.append(row)

if not rows:
    print("ERROR: eval file is empty", file=sys.stderr)
    raise SystemExit(1)

def normalize(label: str) -> str:
    return str(label).strip().lower()

correct = sum(normalize(row["expected"]) == normalize(row["prediction"]) for row in rows)
total = len(rows)
score = correct / total
print(f"Eval rows: {total}")
print(f"Exact-match accuracy on tiny fixture: {score:.3f} ({correct}/{total})")
# Count gate: fail below 2/3 correct (same rule Docker and Python reuse).
if correct * 3 < total * 2:
    print("Gate failed: score regressed below 2/3", file=sys.stderr)
    raise SystemExit(1)
print("Gate passed. You may commit.")

Output

Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.

The pre-commit gate that protects the score

Create a tiny executable that the pre-commit hook and clean-clone reproduction command will run.

terminal-5

mkdir -p scripts
cat > scripts/run_eval.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

EVAL_FILE="eval/access_requests.jsonl"
if [[ ! -f "$EVAL_FILE" ]]; then
  echo "ERROR: $EVAL_FILE missing. Did you forget to commit the fixture or pull the latest repo?"
  exit 1
fi

PYTHON_BIN="${PYTHON_BIN:-python3}"
if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
  echo "ERROR: Python 3 is required by the provided scorer. Install it or set PYTHON_BIN."
  exit 1
fi

if [[ -f scripts/score_access_requests.py ]]; then
  "$PYTHON_BIN" scripts/score_access_requests.py
else
  echo "ERROR: scripts/score_access_requests.py missing. Add the Python scorer from this chapter."
  exit 1
fi
EOF
chmod +x scripts/run_eval.sh

Create a repo-local reproduction command. This is important: shell aliases and Git hooks are local machine state, but repro.sh travels with the repo.

terminal-6

cat > repro.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

./scripts/run_eval.sh
EOF
chmod +x repro.sh

terminal-7

cat > scripts/pre-commit-ai-eval.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

echo "Running AI eval gate before commit..."
./scripts/run_eval.sh
echo "Eval gate passed."
EOF

cat > scripts/install_hooks.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

mkdir -p .git/hooks
cp scripts/pre-commit-ai-eval.sh .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "Installed .git/hooks/pre-commit"
EOF

chmod +x scripts/pre-commit-ai-eval.sh scripts/install_hooks.sh
./scripts/install_hooks.sh

Test it.

terminal-8

./repro.sh
git add scripts/score_access_requests.py scripts/run_eval.sh scripts/pre-commit-ai-eval.sh scripts/install_hooks.sh repro.sh eval/access_requests.jsonl
git commit -m "feat: add three-row eval and pre-commit gate that protects 0.667"

Output

Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.
Running AI eval gate before commit...
Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.
Eval gate passed.

If the scorer reports a regression or the file disappears, the commit is rejected before the broken state enters history.

Shell one-liners that make the invisible visible

Add a few reusable functions to ~/.zshrc or ~/.bashrc.

terminal-9

# GPU snapshot (works on NVIDIA, falls back gracefully)
gpu() {
  if command -v nvidia-smi >/dev/null 2>&1; then
    nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu --format=csv,noheader
  else
    echo "No NVIDIA GPU or nvidia-smi not in PATH"
  fi
}

# Dataset size at a glance
ds() {
  du -sh "${1:-.}" 2>/dev/null | awk '{print $1 "  " $2}'
  echo "JSONL rows: $(find "${1:-.}" -name '*.jsonl' -exec wc -l {} + 2>/dev/null | tail -1 | awk '{print $1}')"
}

# One-command reproduction of the current eval
repro() {
  if [[ -x ./repro.sh ]]; then
    ./repro.sh
  elif [[ -x ./scripts/run_eval.sh ]]; then
    ./scripts/run_eval.sh
  elif [[ -x ./reproduce.sh ]]; then
    ./reproduce.sh
  else
    echo "No reproducible entrypoint found (looked for repro.sh, scripts/run_eval.sh, or reproduce.sh)"
    return 1
  fi
}

Inspect data and processes without crashing the box

terminal-inspect

head -n 1 eval/access_requests.jsonl    # peek at the schema of one row
wc -l eval/access_requests.jsonl        # count rows without loading the file
grep -c '"expected"' eval/access_requests.jsonl  # how many rows have the field

The pipe | chains these into one pass. grep '"restored"' eval/access_requests.jsonl | wc -l filters, then counts, without ever holding the whole file in memory.

terminal-kill

nvidia-smi                       # read the PID in the bottom "Processes" table
kill 12345                       # SIGTERM (15): let the process flush and release VRAM
for _ in {1..10}; do             # give graceful shutdown up to ten seconds
  kill -0 12345 2>/dev/null || break
  sleep 1
done
kill -0 12345 2>/dev/null && kill -9 12345   # escalate only if it is still alive

Linux fundamentals that keep long jobs alive

Training and evaluation jobs can run for hours, often through an SSH session. These commands keep the job alive and make its resource use visible:

Detach a process that survives logout: nohup python train.py > train.log 2>&1 &
Manage sessions across SSH disconnects: tmux new -s training, tmux attach -t training, Ctrl-b d to detach.
Give the job lower priority so your laptop remains usable: nice -n 10 python train.py
Find the files using disk right now: du -ah /workspace | sort -rh | head -20
Check which Python process is using the GPU: nvidia-smi + ps aux | grep python

Together, these commands keep a job alive across disconnects, lower its CPU priority when needed, and show which files or processes are consuming resources.

The reproducible activation contract

Create an activation script that each teammate and each CI job can source. This first chapter doesn't require PyTorch yet, so the script reports CUDA when torch is already installed.

terminal-10

cat > requirements.txt << 'EOF'
# Empty in this first chapter.
# Later chapters will add pinned runtime packages here.
EOF

cat > .env.example << 'EOF'
# Copy to .env locally. Never commit .env.
ACCESS_API_KEY=replace-me
HF_HOME=
EOF

cat > activate.sh << 'EOF'
#!/usr/bin/env bash

_activate_fail() {
  echo "ERROR: $1" >&2
}

_activate_main() {
  # Prefer an explicit interpreter so clones don't silently use 3.10 vs 3.12.
  # Optional: pin with a `.python-version` file and tools like uv or pyenv.
  PYTHON_BIN="${PYTHON_BIN:-python3}"
  if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
    _activate_fail "install python3 or set PYTHON_BIN=/path/to/python"
    return 1
  fi

  # 1. Create or reuse a local virtualenv
  if [[ ! -d .venv ]]; then
    "$PYTHON_BIN" -m venv .venv || {
      _activate_fail "could not create .venv"
      return 1
    }
  fi
  source .venv/bin/activate || {
    _activate_fail "could not activate .venv"
    return 1
  }

  # 2. Install declared dependencies only when requirements.txt has entries.
  # Prefer a locked install once you add packages: `uv lock` + `uv sync`,
  # or `pip-compile` + `pip install -r requirements.txt --require-hashes`.
  grep -Ev '^\s*(#|$)' requirements.txt >/dev/null 2>&1
  _requirements_status=$?
  if [[ $_requirements_status -eq 0 ]]; then
    python -m pip install -r requirements.txt || {
      _activate_fail "could not install requirements.txt"
      return 1
    }
  elif [[ $_requirements_status -gt 1 ]]; then
    _activate_fail "could not read requirements.txt"
    return 1
  fi
  unset _requirements_status

  # 3. Print the local environment without requiring GPU packages yet
  if ! python - << 'PY'
import os, sys
print("Python:", sys.version.split()[0])
if sys.version_info[:2] < (3, 12):
    print("WARNING: this project targets Python 3.12+; set PYTHON_BIN to a 3.12 interpreter")
visible_devices = os.environ.get("CUDA_VISIBLE_DEVICES")
print(
    "CUDA_VISIBLE_DEVICES:",
    visible_devices if visible_devices is not None else "<unset; no mask applied>",
)
try:
    import torch
except ModuleNotFoundError:
    print("PyTorch: not installed yet (OK for this chapter)")
else:
    print("PyTorch:", torch.__version__)
    print("CUDA available:", torch.cuda.is_available())
    if torch.cuda.is_available():
        print("GPU:", torch.cuda.get_device_name(0))
print("HF_HOME:", os.environ.get("HF_HOME", "(default ~/.cache/huggingface)"))
PY
  then
    _activate_fail "environment probe failed"
    return 1
  fi

  echo "Environment ready. Run './repro.sh' to execute the eval gate."
}

_activate_main
_activate_status=$?
unset -f _activate_main _activate_fail
if [[ $_activate_status -ne 0 ]]; then
  return "$_activate_status" 2>/dev/null || exit "$_activate_status"
fi
unset _activate_status
EOF
chmod +x activate.sh

Document it in README.md:

README.md

## Quick start

git clone [email protected]:your-org/access-rag.git
cd access-rag
./scripts/install_hooks.sh
if command -v git-lfs >/dev/null 2>&1; then
  git lfs pull
fi
source activate.sh && ./repro.sh

Now a fresh engineer (or a fresh GPU box provisioned by your platform team) can reproduce the same 0.667 result through the tracked setup path.

The failure modes you'll see in real life

Symptom	Most common cause	Fix that belongs in the repo
`CUDA not found` on the GPU box	Host driver or container runtime is missing, the image lacks a compatible CUDA stack, or `CUDA_VISIBLE_DEVICES` is explicitly empty/invalid and hides the GPU	Compare `nvidia-smi`, the printed `CUDA_VISIBLE_DEVICES`, and `torch.cuda.is_available()`; remove an accidental mask or choose a valid device index, then document the required image and runtime. An unset variable normally exposes all available devices.
`ModuleNotFoundError` for a package that worked on the laptop	`requirements.txt` is incomplete or uses unpinned versions	Rebuild from a clean env, pin direct deps (and prefer a lockfile such as `uv.lock` or hashed `requirements.txt`), commit the pins, and reinstall from that file
Eval returns 0.000 because the three-row JSONL is missing	`.gitignore` did not protect the generated cache directory that the author had on disk	Move the fixture to `eval/` and add the directory to the committed tree; don't rely on "I had it in my downloads folder"
Pre-commit hook doesn't run on a fresh clone	Hooks under `.git/hooks/` are local machine files, not tracked project files	Commit `scripts/pre-commit-ai-eval.sh` and `scripts/install_hooks.sh`, then run the installer after cloning
Pre-commit hook fails with "permission denied"	The hook script was installed without `chmod +x` or the clone was on a filesystem that strips execute bits	`chmod +x scripts/.sh .git/hooks/` + a one-line check in the installer
"It worked yesterday" after a `git pull`	teammate committed a new large model without LFS or changed the expected schema of the eval file	LFS tracking + a schema validation step in the scorer + `git diff` before each `git pull` on data files

Encode each diagnosis and prevention in the repo so the next clone fails clearly instead of requiring machine-specific guesswork.

The repo contract

The first real engineering loop is in place:

git clone + source activate.sh produces a working environment on any machine with the declared dependencies.
repro (or the pre-commit hook) guarantees that the tiny contract (three rows, 0.667) is still satisfied after each change.
gpu, ds, and the Linux session commands let you see what the hardware is doing.
The .gitignore + LFS rules + activation script travel with the code, so the next person doesn't have to reverse-engineer your laptop.

Reproducible change checklist

Next Step

Continue to Docker for Reproducible AI

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Pro Git (2nd ed.)

Chacon, S. & Straub, B. · 2014

https://git-scm.com/book/en/v2

About large files on GitHub

GitHub · 2026

https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github

nvidia-smi documentation

NVIDIA · 2026

https://docs.nvidia.com/deploy/nvidia-smi/index.html

The Open Group Base Specifications: signal.h

The Open Group · 2018

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/signal.h.html

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Git, Shell, Linux for AI

Start with the smallest useful repo

The .gitignore that protects AI work

The eval that must travel with the code

Add the provided scorer

Why does the gate compare integer counts instead of testing whether a floating-point score equals `0.667`?

The pre-commit gate that protects the score

Shell one-liners that make the invisible visible

Inspect data and processes without crashing the box

You find a live training PID holding GPU memory. Which signal should you send first, and when should you escalate?

Linux fundamentals that keep long jobs alive

The reproducible activation contract

The failure modes you'll see in real life

The repo contract

Why is a clean temporary clone stronger evidence than rerunning `repro` in your working directory?

Reproducible change checklist

Mastery Check

Discussion

Git, Shell, Linux for AI

Start with the smallest useful repo

The .gitignore that protects AI work

The eval that must travel with the code

Add the provided scorer

Why does the gate compare integer counts instead of testing whether a floating-point score equals `0.667`?

The pre-commit gate that protects the score

Shell one-liners that make the invisible visible

Inspect data and processes without crashing the box

You find a live training PID holding GPU memory. Which signal should you send first, and when should you escalate?

Linux fundamentals that keep long jobs alive

The reproducible activation contract

The failure modes you'll see in real life

The repo contract

Why is a clean temporary clone stronger evidence than rerunning `repro` in your working directory?

Reproducible change checklist

Mastery Check

Discussion

Start with the smallest useful repo

The .gitignore that protects AI work

The eval that must travel with the code

Add the provided scorer

Why does the gate compare integer counts instead of testing whether a floating-point score equals 0.667?

Why does the gate compare integer counts instead of testing whether a floating-point score equals 0.667?

The pre-commit gate that protects the score

Shell one-liners that make the invisible visible

Inspect data and processes without crashing the box

You find a live training PID holding GPU memory. Which signal should you send first, and when should you escalate?

You find a live training PID holding GPU memory. Which signal should you send first, and when should you escalate?

Linux fundamentals that keep long jobs alive

The reproducible activation contract

The failure modes you'll see in real life

The repo contract

Why is a clean temporary clone stronger evidence than rerunning repro in your working directory?

Why is a clean temporary clone stronger evidence than rerunning repro in your working directory?

Reproducible change checklist

Mastery Check

Discussion

Start with the smallest useful repo

The .gitignore that protects AI work

The eval that must travel with the code

Add the provided scorer

Why does the gate compare integer counts instead of testing whether a floating-point score equals 0.667?

Why does the gate compare integer counts instead of testing whether a floating-point score equals 0.667?

The pre-commit gate that protects the score

Shell one-liners that make the invisible visible

Inspect data and processes without crashing the box

You find a live training PID holding GPU memory. Which signal should you send first, and when should you escalate?

You find a live training PID holding GPU memory. Which signal should you send first, and when should you escalate?

Linux fundamentals that keep long jobs alive

The reproducible activation contract

The failure modes you'll see in real life

The repo contract

Why is a clean temporary clone stronger evidence than rerunning repro in your working directory?

Why is a clean temporary clone stronger evidence than rerunning repro in your working directory?

Reproducible change checklist

Mastery Check

Discussion

Why does the gate compare integer counts instead of testing whether a floating-point score equals `0.667`?

Why is a clean temporary clone stronger evidence than rerunning `repro` in your working directory?

Why does the gate compare integer counts instead of testing whether a floating-point score equals `0.667`?

Why is a clean temporary clone stronger evidence than rerunning `repro` in your working directory?