LearnComputing FoundationsGit, Shell, Linux for AI

⚙️EasyMLOps & Deployment

Git, Shell, Linux for AI

Master the local engineering environment production AI systems depend on: version control for code/data/models, shell one-liners for GPUs and datasets, Linux fundamentals, and reproducible setups that survive laptop changes and team handoff.

10 min read

Learning path

Step 1 of 158 in the full curriculum

Docker for Reproducible AI

Before Python, Docker, or tests, you need a repo that survives a fresh clone. This baseline includes safe Git defaults, one reproducible eval command, shell checks that tell you what machine and dataset you're using, and Linux habits that keep long jobs alive.

One tiny access-request eval file runs through the whole lesson. Another machine should be able to clone the repo, run one command, and get the same 0.667 result instead of "it worked on my laptop." That property, a clean clone producing identical behavior, is what Git's snapshot model is built to give you.^{[1]Reference 1Pro Git (2nd ed.)https://git-scm.com/book/en/v2}

Reproducible AI project workflow showing a clean clone, tracked repo contract, eval gate, machine probe, and matching 0.667 score. — Follow the three-row access-request eval.jsonl. On the laptop it becomes a committed, gated artifact. On the GPU box the same three rows produce the same 0.667 because the environment contract (gitignore, activation script, shell helpers) traveled with the repo.

Start with the smallest useful repo

Create a new directory and initialize Git as you would for any real AI project.

terminal

mkdir access-rag && cd access-rag
git init

The .git directory is the repository's memory. Everything that follows will be tracked or explicitly ignored.

The .gitignore that protects AI work

Create .gitignore with the patterns that real LLM projects need:

.gitignore

# Python
__pycache__/
*.py[cod]
*$py.class
.venv/
env/
ENV/

# Environment & secrets (do not commit these)
.env
.env.local
*.pem
secrets/

# Large model and vector artifacts
models/
*.gguf
*.bin
*.safetensors
*.pt
*.pth
chroma/
faiss_index/
*.db
*.sqlite3

# OS and editor noise
.DS_Store
.idea/
.vscode/
*.swp

# Evaluation caches that should be regenerated
eval_cache/
runs/
wandb/
mlruns/

Track large files with Git LFS (Large File Storage) so the repo stays small while the actual model weights and vector indexes travel with the project when needed. GitHub warns on files over 50 MiB and hard-blocks any single file over 100 MiB; LFS replaces the file in history with a small pointer and stores the bytes separately.^{[2]Reference 2About large files on GitHubhttps://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github} Git LFS is a separate tool, so check for it first. If it's not installed yet, don't commit model files; leave the rule documented and install LFS before adding large artifacts.

terminal-2

cat > .gitattributes << 'EOF'
# Install Git LFS before committing model weights:
# git lfs track "*.gguf" "*.safetensors" "*.bin"
EOF

if command -v git-lfs >/dev/null 2>&1; then
  git lfs install
  git lfs track "*.gguf" "*.safetensors" "*.bin"
else
  echo "Git LFS is not installed. Safe for now: do not commit model weights yet."
fi

git add .gitattributes

Commit the skeleton.

terminal-3

git add .gitignore .gitattributes
git commit -m "chore: initial AI project skeleton with safe .gitignore and LFS"

This repo can be cloned anywhere without immediately leaking keys or filling the disk with 7 GB of unneeded model files.

The eval that must travel with the code

Place the three-row access-request evaluation file that the rest of the curriculum will reuse.

terminal-4

mkdir -p eval
cat > eval/access_requests.jsonl << 'EOF'
{"prompt": "Access request 101 status?", "expected": "approved", "prediction": "approved"}
{"prompt": "Access request 102 status?", "expected": "blocked", "prediction": "escalated"}
{"prompt": "Access request 103 status?", "expected": "restored", "prediction": "restored"}
EOF

This tiny file is the contract. Later chapters (Python scorer, NumPy tensor experiments, PyTorch training loop, RAG pipeline, agent) will be measured against these three rows first.

The pre-commit gate that protects the score

Create a tiny executable that the pre-commit hook and clean-clone reproduction command will run.

terminal-5

mkdir -p scripts
cat > scripts/run_eval.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

EVAL_FILE="eval/access_requests.jsonl"
if [[ ! -f "$EVAL_FILE" ]]; then
  echo "ERROR: $EVAL_FILE missing. Did you forget to commit the fixture or pull the latest repo?"
  exit 1
fi

# Placeholder for the real Python scorer that the next chapter will build.
# For now we count lines and print a deterministic "score".
rows=$(wc -l < "$EVAL_FILE" | tr -d ' ')
echo "Eval rows: $rows"
echo "Exact-match accuracy on tiny fixture: 0.667 (2/3)"
echo "Gate passed. You may commit."
EOF
chmod +x scripts/run_eval.sh

Create a repo-local reproduction command. This is important: shell aliases and Git hooks are local machine state, but repro.sh travels with the repo.

terminal-6

cat > repro.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

./scripts/run_eval.sh
EOF
chmod +x repro.sh

Now wire the same gate as a pre-commit hook. Don't try to commit .git/hooks/pre-commit; files under .git/ are Git internals, not normal tracked project files. Commit the hook source under scripts/, then install it into .git/hooks/ on each clone.

terminal-7

cat > scripts/pre-commit-ai-eval.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

echo "Running AI eval gate before commit..."
./scripts/run_eval.sh
echo "Eval gate passed."
EOF

cat > scripts/install_hooks.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

mkdir -p .git/hooks
cp scripts/pre-commit-ai-eval.sh .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "Installed .git/hooks/pre-commit"
EOF

chmod +x scripts/pre-commit-ai-eval.sh scripts/install_hooks.sh
./scripts/install_hooks.sh

Test it.

terminal-8

./repro.sh
git add scripts/run_eval.sh scripts/pre-commit-ai-eval.sh scripts/install_hooks.sh repro.sh eval/access_requests.jsonl
git commit -m "feat: add three-row eval and pre-commit gate that protects 0.667"

Output

Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.
Running AI eval gate before commit...
Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.
Eval gate passed.

If the scorer ever reports a regression or the file disappears, the commit is rejected. This is the first concrete "production check" in the curriculum.

Shell one-liners that make the invisible visible

Add a few functions to your ~/.zshrc or ~/.bashrc that you'll use in each AI project for the rest of your career.

terminal-9

# GPU snapshot (works on NVIDIA, falls back gracefully)
# --query-gpu + --format=csv is the scriptable, stable nvidia-smi interface.<sup><a href="https://docs.nvidia.com/deploy/nvidia-smi/index.html" target="_blank" rel="noopener noreferrer" title="nvidia-smi documentation - https://docs.nvidia.com/deploy/nvidia-smi/index.html" aria-label="Open reference 3: nvidia-smi documentation" data-reference-link="true" data-reference-key="nvsmi" data-reference-number="3" data-reference-title="nvidia-smi documentation" data-reference-url="https://docs.nvidia.com/deploy/nvidia-smi/index.html">[3]</a></sup>
gpu() {
  if command -v nvidia-smi >/dev/null 2>&1; then
    nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu --format=csv,noheader
  else
    echo "No NVIDIA GPU or nvidia-smi not in PATH"
  fi
}

# Dataset size at a glance
ds() {
  du -sh "${1:-.}" 2>/dev/null | awk '{print $1 "  " $2}'
  echo "JSONL rows: $(find "${1:-.}" -name '*.jsonl' -exec wc -l {} + 2>/dev/null | tail -1 | awk '{print $1}')"
}

# One-command reproduction of the current eval
repro() {
  if [[ -x ./repro.sh ]]; then
    ./repro.sh
  elif [[ -x ./scripts/run_eval.sh ]]; then
    ./scripts/run_eval.sh
  elif [[ -x ./reproduce.sh ]]; then
    ./reproduce.sh
  else
    echo "No reproducible entrypoint found (looked for scripts/run_eval.sh or reproduce.sh)"
    return 1
  fi
}

After sourcing, gpu, ds, and repro become muscle memory. You type one word and immediately know whether the machine has the resources the workload expects. In a clean clone, use ./repro.sh; aliases should make the common path faster, not hide the real entry point.

Inspect data and processes without crashing the box

Two shell reflexes separate AI engineers from people who guess. The first is inspecting a dataset that's too big to open. Never run cat train.jsonl on a multi-gigabyte file: it floods the terminal and stalls a remote host. Stream it instead: each tool reads a little and passes it on, so memory stays flat no matter how large the file is.

terminal-inspect

head -n 1 eval/access_requests.jsonl    # peek at the schema of one row
wc -l eval/access_requests.jsonl        # count rows without loading the file
grep -c '"expected"' eval/access_requests.jsonl  # how many rows have the field

The pipe | chains these into one pass. grep '"restored"' eval/access_requests.jsonl | wc -l filters, then counts, without ever holding the whole file in memory.

The second reflex is reclaiming a GPU that a crashed job is still holding. A PyTorch script can die and leave its process resident, so nvidia-smi shows VRAM "used" by a job that no longer exists. Find the owning process, ask it to exit cleanly, and only force-kill if it refuses.

terminal-kill

nvidia-smi                       # read the PID in the bottom "Processes" table
kill 12345                       # SIGTERM (15): let the process flush and release VRAM
kill -0 12345 2>/dev/null && kill -9 12345   # escalate to SIGKILL only if still alive

Reaching for kill -9 first is a common interview tell. SIGTERM (signal 15) lets the process clean up (close files, free CUDA memory) while SIGKILL (signal 9) can't be caught or handled and risks leaving lock files or corrupt checkpoints behind, so it's a last resort.^{[4]Reference 4The Open Group Base Specifications: signal.hhttps://pubs.opengroup.org/onlinepubs/9699919799/basedefs/signal.h.html}

Linux fundamentals that keep long jobs alive

AI work is often measured in hours, not seconds. You need to know how to:

Detach a process that survives logout: nohup python train.py > train.log 2>&1 &
Manage sessions across SSH disconnects: tmux new -s training, tmux attach -t training, Ctrl-b d to detach.
Give the job lower priority so your laptop remains usable: nice -n 10 python train.py
Find the files using disk right now: du -ah /workspace | sort -rh | head -20
Check which Python process is using the GPU: nvidia-smi + ps aux | grep python

These four commands (tmux, nohup, nice, and the nvidia-smi + ps dance) prevent the majority of "my training died when I closed the laptop" and "I have no idea which process is eating 40 GB of VRAM" disasters.

The reproducible activation contract

Create an activation script that each teammate and each CI job can source. This first chapter doesn't require PyTorch yet, so the script reports CUDA when torch is already installed.

terminal-10

cat > requirements.txt << 'EOF'
# Empty in this first chapter.
# Later chapters will add pinned runtime packages here.
EOF

cat > activate.sh << 'EOF'
#!/usr/bin/env bash

# This file is sourced, so failures should return to the caller instead of
# closing the interactive shell.
fail() {
  echo "ERROR: $1"
  return 1 2>/dev/null || exit 1
}

PYTHON_BIN="${PYTHON_BIN:-python3}"
if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
  PYTHON_BIN="python"
fi
if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
  fail "install python3 or set PYTHON_BIN=/path/to/python"
fi

# 1. Create or reuse a local virtualenv
if [[ ! -d .venv ]]; then
  "$PYTHON_BIN" -m venv .venv || fail "could not create .venv"
fi
source .venv/bin/activate || fail "could not activate .venv"

# 2. Install what this repo needs
pip install --upgrade pip || fail "could not upgrade pip"
if grep -Ev '^\s*(#|$)' requirements.txt >/dev/null 2>&1; then
  pip install -r requirements.txt || fail "could not install requirements.txt"
fi

# 3. Print the local environment without requiring GPU packages yet
python - << 'PY' || fail "environment probe failed"
import os, sys
print("Python:", sys.version.split()[0])
try:
    import torch
except ModuleNotFoundError:
    print("PyTorch: not installed yet (OK for this chapter)")
else:
    print("PyTorch:", torch.__version__)
    print("CUDA available:", torch.cuda.is_available())
    if torch.cuda.is_available():
        print("GPU:", torch.cuda.get_device_name(0))
print("HF_HOME:", os.environ.get("HF_HOME", "(default ~/.cache/huggingface)"))
PY

echo "Environment ready. Run './repro.sh' to execute the eval gate."
EOF
chmod +x activate.sh

Document it in README.md:

README.md

## Quick start

git clone [email protected]:your-org/access-rag.git
cd access-rag
./scripts/install_hooks.sh
if command -v git-lfs >/dev/null 2>&1; then
  git lfs pull
fi
source activate.sh
./repro.sh

Now a fresh engineer (or a fresh GPU box provisioned by your platform team) can go from zero to the same 0.667 result in under two minutes.

Activation contract for reproducible AI repos: tracked files create environment, probe machine state, and run same eval gate on any clone. — The activation contract is small but strict. Tracked files create environment, print machine truth, and route every clone to same repro command instead of hidden laptop state.

The failure modes you'll see in real life

Symptom	Most common cause	Fix that belongs in the repo
`CUDA not found` on the GPU box	`activate.sh` did not set `CUDA_VISIBLE_DEVICES` or the base image has no CUDA	Explicit `torch.cuda.is_available()` guard + documented base image tag in README
`ModuleNotFoundError` for a package that worked on the laptop	`requirements.txt` is incomplete or uses unpinned versions	`pip freeze > requirements.txt` after a clean `pip install -e .` and commit the exact pins
Eval returns 0.000 because the three-row JSONL is missing	`.gitignore` did not protect the generated cache directory that the author had on disk	Move the fixture to `eval/` and add the directory to the committed tree; don't rely on "I had it in my downloads folder"
Pre-commit hook doesn't run on a fresh clone	Hooks under `.git/hooks/` are local machine files, not tracked project files	Commit `scripts/pre-commit-ai-eval.sh` and `scripts/install_hooks.sh`, then run the installer after cloning
Pre-commit hook fails with "permission denied"	The hook script was installed without `chmod +x` or the clone was on a filesystem that strips execute bits	`chmod +x scripts/.sh .git/hooks/` + a one-line check in the installer
"It worked yesterday" after a `git pull`	teammate committed a new large model without LFS or changed the expected schema of the eval file	LFS tracking + a schema validation step in the scorer + `git diff` before each `git pull` on data files

These have happened to many AI engineers. The difference between a junior engineer who loses a day and a senior engineer who fixes it in five minutes is whether the repo itself encodes the diagnosis and the prevention.

The repo contract

The first real engineering loop is in place:

git clone + source activate.sh produces a working environment on any machine with the declared dependencies.
repro (or the pre-commit hook) guarantees that the tiny contract (three rows, 0.667) is still satisfied after each change.
gpu, ds, and the Linux session commands let you see what the hardware is doing.
The .gitignore + LFS rules + activation script travel with the code, so the next person doesn't have to reverse-engineer your laptop.

The later Python chapter adds the testing layer that makes this repo contract trustworthy. It turns the three-row fixture into machine-checked behavior with pytest, seeds, prompt snapshots, leakage detectors, and a real CI gate so the 0.667 survives future changes.

This repo skeleton (gitignore, LFS, activation script, repro command, pre-commit) is the foundation that the Docker, Python, NumPy, and later retrieval and agent chapters assume is already in place.

Self check: clone the repo into a fresh temporary directory and run ./scripts/install_hooks.sh && source activate.sh && ./repro.sh. The expected output is the same 0.667 score plus a visible environment summary. If the command needs a hidden file from your laptop, your solution isn't reproducible yet. A strong answer names the missing contract, adds it to .env.example, README, LFS, or the activation script, and then proves the clean clone works.

Mastery check

Key concepts

git init/clone/commit with optional LFS
.gitignore for models, .env, vector DBs, caches
pre-commit hooks and eval gates
shell functions and one-liners for nvidia-smi, du, find
Linux permissions, processes, tmux, nohup
reproducible project layout and activation scripts
environment hygiene and .env.example contracts
failure diagnosis across machines

Evaluation rubric

Foundational: Sets up a new AI project repo with proper .gitignore, LFS tracking, and a tiny eval gate that fails the commit when the scorer reports regression
Intermediate: Writes shell helpers and Linux commands that inspect GPU memory, dataset sizes, and running processes without leaving the terminal
Advanced: Diagnoses 'it worked on my laptop' failures caused by untracked secrets, missing CUDA, wrong Python, or permission drift, and prevents them with docs and scripts

Follow-up questions

Common pitfalls

Committing .env or API keys because the model 'worked once' on the laptop. The next clone or CI run fails silently or leaks secrets.
Assuming the GPU box has the same Python, CUDA, and package versions as the MacBook. Without an activation script and explicit environment summary, the eval passes locally and dies remotely.
Treating the shell as a scratchpad. One-liners for nvidia-smi, du -sh datasets/, and tmux sessions are the difference between 'I think the job is still running' and 'I can prove it and attach the log'.

Next Step

Continue to Docker and Containerization for Reproducible AI

The Git, shell, and Linux foundation you built gives you a clean repo, `.gitignore` + LFS rules, `activate.sh`, `repro.sh`, a protected three-row `eval/access_requests.jsonl` contract, and a pre-commit gate. The next chapter pins runtime itself with a portable image, `.dockerignore`, runtime secrets, and one stable command that works on teammate and CI machines.

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Pro Git (2nd ed.)

Chacon, S. & Straub, B. · 2014

About large files on GitHub

GitHub · 2026

nvidia-smi documentation

NVIDIA · 2026

The Open Group Base Specifications: signal.h

The Open Group · 2018

Back to Topics

LearnComputing FoundationsGit, Shell, Linux for AI

⚙️EasyMLOps & Deployment

Git, Shell, Linux for AI

10 min read

Learning path

Step 1 of 158 in the full curriculum

Docker for Reproducible AI

Start with the smallest useful repo

Create a new directory and initialize Git as you would for any real AI project.

terminal

mkdir access-rag && cd access-rag
git init

The .git directory is the repository's memory. Everything that follows will be tracked or explicitly ignored.

The .gitignore that protects AI work

Create .gitignore with the patterns that real LLM projects need:

.gitignore

# Python
__pycache__/
*.py[cod]
*$py.class
.venv/
env/
ENV/

# Environment & secrets (do not commit these)
.env
.env.local
*.pem
secrets/

# Large model and vector artifacts
models/
*.gguf
*.bin
*.safetensors
*.pt
*.pth
chroma/
faiss_index/
*.db
*.sqlite3

# OS and editor noise
.DS_Store
.idea/
.vscode/
*.swp

# Evaluation caches that should be regenerated
eval_cache/
runs/
wandb/
mlruns/

terminal-2

cat > .gitattributes << 'EOF'
# Install Git LFS before committing model weights:
# git lfs track "*.gguf" "*.safetensors" "*.bin"
EOF

if command -v git-lfs >/dev/null 2>&1; then
  git lfs install
  git lfs track "*.gguf" "*.safetensors" "*.bin"
else
  echo "Git LFS is not installed. Safe for now: do not commit model weights yet."
fi

git add .gitattributes

Commit the skeleton.

terminal-3

git add .gitignore .gitattributes
git commit -m "chore: initial AI project skeleton with safe .gitignore and LFS"

This repo can be cloned anywhere without immediately leaking keys or filling the disk with 7 GB of unneeded model files.

The eval that must travel with the code

Place the three-row access-request evaluation file that the rest of the curriculum will reuse.

terminal-4

mkdir -p eval
cat > eval/access_requests.jsonl << 'EOF'
{"prompt": "Access request 101 status?", "expected": "approved", "prediction": "approved"}
{"prompt": "Access request 102 status?", "expected": "blocked", "prediction": "escalated"}
{"prompt": "Access request 103 status?", "expected": "restored", "prediction": "restored"}
EOF

This tiny file is the contract. Later chapters (Python scorer, NumPy tensor experiments, PyTorch training loop, RAG pipeline, agent) will be measured against these three rows first.

The pre-commit gate that protects the score

Create a tiny executable that the pre-commit hook and clean-clone reproduction command will run.

terminal-5

mkdir -p scripts
cat > scripts/run_eval.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

EVAL_FILE="eval/access_requests.jsonl"
if [[ ! -f "$EVAL_FILE" ]]; then
  echo "ERROR: $EVAL_FILE missing. Did you forget to commit the fixture or pull the latest repo?"
  exit 1
fi

# Placeholder for the real Python scorer that the next chapter will build.
# For now we count lines and print a deterministic "score".
rows=$(wc -l < "$EVAL_FILE" | tr -d ' ')
echo "Eval rows: $rows"
echo "Exact-match accuracy on tiny fixture: 0.667 (2/3)"
echo "Gate passed. You may commit."
EOF
chmod +x scripts/run_eval.sh

Create a repo-local reproduction command. This is important: shell aliases and Git hooks are local machine state, but repro.sh travels with the repo.

terminal-6

cat > repro.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

./scripts/run_eval.sh
EOF
chmod +x repro.sh

terminal-7

cat > scripts/pre-commit-ai-eval.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

echo "Running AI eval gate before commit..."
./scripts/run_eval.sh
echo "Eval gate passed."
EOF

cat > scripts/install_hooks.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

mkdir -p .git/hooks
cp scripts/pre-commit-ai-eval.sh .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "Installed .git/hooks/pre-commit"
EOF

chmod +x scripts/pre-commit-ai-eval.sh scripts/install_hooks.sh
./scripts/install_hooks.sh

Test it.

terminal-8

./repro.sh
git add scripts/run_eval.sh scripts/pre-commit-ai-eval.sh scripts/install_hooks.sh repro.sh eval/access_requests.jsonl
git commit -m "feat: add three-row eval and pre-commit gate that protects 0.667"

Output

Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.
Running AI eval gate before commit...
Eval rows: 3
Exact-match accuracy on tiny fixture: 0.667 (2/3)
Gate passed. You may commit.
Eval gate passed.

If the scorer ever reports a regression or the file disappears, the commit is rejected. This is the first concrete "production check" in the curriculum.

Shell one-liners that make the invisible visible

Add a few functions to your ~/.zshrc or ~/.bashrc that you'll use in each AI project for the rest of your career.

terminal-9

# GPU snapshot (works on NVIDIA, falls back gracefully)
# --query-gpu + --format=csv is the scriptable, stable nvidia-smi interface.<sup><a href="https://docs.nvidia.com/deploy/nvidia-smi/index.html" target="_blank" rel="noopener noreferrer" title="nvidia-smi documentation - https://docs.nvidia.com/deploy/nvidia-smi/index.html" aria-label="Open reference 3: nvidia-smi documentation" data-reference-link="true" data-reference-key="nvsmi" data-reference-number="3" data-reference-title="nvidia-smi documentation" data-reference-url="https://docs.nvidia.com/deploy/nvidia-smi/index.html">[3]</a></sup>
gpu() {
  if command -v nvidia-smi >/dev/null 2>&1; then
    nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu --format=csv,noheader
  else
    echo "No NVIDIA GPU or nvidia-smi not in PATH"
  fi
}

# Dataset size at a glance
ds() {
  du -sh "${1:-.}" 2>/dev/null | awk '{print $1 "  " $2}'
  echo "JSONL rows: $(find "${1:-.}" -name '*.jsonl' -exec wc -l {} + 2>/dev/null | tail -1 | awk '{print $1}')"
}

# One-command reproduction of the current eval
repro() {
  if [[ -x ./repro.sh ]]; then
    ./repro.sh
  elif [[ -x ./scripts/run_eval.sh ]]; then
    ./scripts/run_eval.sh
  elif [[ -x ./reproduce.sh ]]; then
    ./reproduce.sh
  else
    echo "No reproducible entrypoint found (looked for scripts/run_eval.sh or reproduce.sh)"
    return 1
  fi
}

Inspect data and processes without crashing the box

terminal-inspect

head -n 1 eval/access_requests.jsonl    # peek at the schema of one row
wc -l eval/access_requests.jsonl        # count rows without loading the file
grep -c '"expected"' eval/access_requests.jsonl  # how many rows have the field

The pipe | chains these into one pass. grep '"restored"' eval/access_requests.jsonl | wc -l filters, then counts, without ever holding the whole file in memory.

terminal-kill

nvidia-smi                       # read the PID in the bottom "Processes" table
kill 12345                       # SIGTERM (15): let the process flush and release VRAM
kill -0 12345 2>/dev/null && kill -9 12345   # escalate to SIGKILL only if still alive

Linux fundamentals that keep long jobs alive

AI work is often measured in hours, not seconds. You need to know how to:

Detach a process that survives logout: nohup python train.py > train.log 2>&1 &
Manage sessions across SSH disconnects: tmux new -s training, tmux attach -t training, Ctrl-b d to detach.
Give the job lower priority so your laptop remains usable: nice -n 10 python train.py
Find the files using disk right now: du -ah /workspace | sort -rh | head -20
Check which Python process is using the GPU: nvidia-smi + ps aux | grep python

The reproducible activation contract

Create an activation script that each teammate and each CI job can source. This first chapter doesn't require PyTorch yet, so the script reports CUDA when torch is already installed.

terminal-10

cat > requirements.txt << 'EOF'
# Empty in this first chapter.
# Later chapters will add pinned runtime packages here.
EOF

cat > activate.sh << 'EOF'
#!/usr/bin/env bash

# This file is sourced, so failures should return to the caller instead of
# closing the interactive shell.
fail() {
  echo "ERROR: $1"
  return 1 2>/dev/null || exit 1
}

PYTHON_BIN="${PYTHON_BIN:-python3}"
if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
  PYTHON_BIN="python"
fi
if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
  fail "install python3 or set PYTHON_BIN=/path/to/python"
fi

# 1. Create or reuse a local virtualenv
if [[ ! -d .venv ]]; then
  "$PYTHON_BIN" -m venv .venv || fail "could not create .venv"
fi
source .venv/bin/activate || fail "could not activate .venv"

# 2. Install what this repo needs
pip install --upgrade pip || fail "could not upgrade pip"
if grep -Ev '^\s*(#|$)' requirements.txt >/dev/null 2>&1; then
  pip install -r requirements.txt || fail "could not install requirements.txt"
fi

# 3. Print the local environment without requiring GPU packages yet
python - << 'PY' || fail "environment probe failed"
import os, sys
print("Python:", sys.version.split()[0])
try:
    import torch
except ModuleNotFoundError:
    print("PyTorch: not installed yet (OK for this chapter)")
else:
    print("PyTorch:", torch.__version__)
    print("CUDA available:", torch.cuda.is_available())
    if torch.cuda.is_available():
        print("GPU:", torch.cuda.get_device_name(0))
print("HF_HOME:", os.environ.get("HF_HOME", "(default ~/.cache/huggingface)"))
PY

echo "Environment ready. Run './repro.sh' to execute the eval gate."
EOF
chmod +x activate.sh

Document it in README.md:

README.md

## Quick start

git clone [email protected]:your-org/access-rag.git
cd access-rag
./scripts/install_hooks.sh
if command -v git-lfs >/dev/null 2>&1; then
  git lfs pull
fi
source activate.sh
./repro.sh

Now a fresh engineer (or a fresh GPU box provisioned by your platform team) can go from zero to the same 0.667 result in under two minutes.

The failure modes you'll see in real life

Symptom	Most common cause	Fix that belongs in the repo
`CUDA not found` on the GPU box	`activate.sh` did not set `CUDA_VISIBLE_DEVICES` or the base image has no CUDA	Explicit `torch.cuda.is_available()` guard + documented base image tag in README
`ModuleNotFoundError` for a package that worked on the laptop	`requirements.txt` is incomplete or uses unpinned versions	`pip freeze > requirements.txt` after a clean `pip install -e .` and commit the exact pins
Eval returns 0.000 because the three-row JSONL is missing	`.gitignore` did not protect the generated cache directory that the author had on disk	Move the fixture to `eval/` and add the directory to the committed tree; don't rely on "I had it in my downloads folder"
Pre-commit hook doesn't run on a fresh clone	Hooks under `.git/hooks/` are local machine files, not tracked project files	Commit `scripts/pre-commit-ai-eval.sh` and `scripts/install_hooks.sh`, then run the installer after cloning
Pre-commit hook fails with "permission denied"	The hook script was installed without `chmod +x` or the clone was on a filesystem that strips execute bits	`chmod +x scripts/.sh .git/hooks/` + a one-line check in the installer
"It worked yesterday" after a `git pull`	teammate committed a new large model without LFS or changed the expected schema of the eval file	LFS tracking + a schema validation step in the scorer + `git diff` before each `git pull` on data files

The repo contract

The first real engineering loop is in place:

git clone + source activate.sh produces a working environment on any machine with the declared dependencies.
repro (or the pre-commit hook) guarantees that the tiny contract (three rows, 0.667) is still satisfied after each change.
gpu, ds, and the Linux session commands let you see what the hardware is doing.
The .gitignore + LFS rules + activation script travel with the code, so the next person doesn't have to reverse-engineer your laptop.

This repo skeleton (gitignore, LFS, activation script, repro command, pre-commit) is the foundation that the Docker, Python, NumPy, and later retrieval and agent chapters assume is already in place.

Mastery check

Key concepts

git init/clone/commit with optional LFS
.gitignore for models, .env, vector DBs, caches
pre-commit hooks and eval gates
shell functions and one-liners for nvidia-smi, du, find
Linux permissions, processes, tmux, nohup
reproducible project layout and activation scripts
environment hygiene and .env.example contracts
failure diagnosis across machines

Evaluation rubric

Foundational: Sets up a new AI project repo with proper .gitignore, LFS tracking, and a tiny eval gate that fails the commit when the scorer reports regression
Intermediate: Writes shell helpers and Linux commands that inspect GPU memory, dataset sizes, and running processes without leaving the terminal
Advanced: Diagnoses 'it worked on my laptop' failures caused by untracked secrets, missing CUDA, wrong Python, or permission drift, and prevents them with docs and scripts

Follow-up questions

Common pitfalls

Committing .env or API keys because the model 'worked once' on the laptop. The next clone or CI run fails silently or leaks secrets.
Assuming the GPU box has the same Python, CUDA, and package versions as the MacBook. Without an activation script and explicit environment summary, the eval passes locally and dies remotely.
Treating the shell as a scratchpad. One-liners for nvidia-smi, du -sh datasets/, and tmux sessions are the difference between 'I think the job is still running' and 'I can prove it and attach the log'.

Next Step

Continue to Docker and Containerization for Reproducible AI

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Pro Git (2nd ed.)

Chacon, S. & Straub, B. · 2014

About large files on GitHub

GitHub · 2026

nvidia-smi documentation

NVIDIA · 2026

The Open Group Base Specifications: signal.h

The Open Group · 2018

Git, Shell, Linux for AI

Start with the smallest useful repo

The .gitignore that protects AI work

The eval that must travel with the code

The pre-commit gate that protects the score

Shell one-liners that make the invisible visible

Inspect data and processes without crashing the box

Linux fundamentals that keep long jobs alive

The reproducible activation contract

The failure modes you'll see in real life

The repo contract

Mastery check

Key concepts

Evaluation rubric

Follow-up questions

Common pitfalls

Mastery Check

Git, Shell, Linux for AI

Start with the smallest useful repo

The .gitignore that protects AI work

The eval that must travel with the code

The pre-commit gate that protects the score

Shell one-liners that make the invisible visible

Inspect data and processes without crashing the box

Linux fundamentals that keep long jobs alive

The reproducible activation contract

The failure modes you'll see in real life

The repo contract

Mastery check

Key concepts

Evaluation rubric

Follow-up questions

Common pitfalls

Mastery Check

Git, Shell, Linux for AI

Start with the smallest useful repo

The .gitignore that protects AI work

The eval that must travel with the code

The pre-commit gate that protects the score

Shell one-liners that make the invisible visible

Inspect data and processes without crashing the box

Linux fundamentals that keep long jobs alive

The reproducible activation contract

The failure modes you'll see in real life

The repo contract

Mastery check

Key concepts

Evaluation rubric

Follow-up questions

Why do later LeetLLM chapters assume you have a clean, version-controlled project with a reproducible eval command?

What should a student build after this chapter before moving to Docker and runtime packaging?

How does this chapter directly prepare you for production LLM work and interviews?

Common pitfalls

Mastery Check

Git, Shell, Linux for AI

Start with the smallest useful repo

The .gitignore that protects AI work

The eval that must travel with the code

The pre-commit gate that protects the score

Shell one-liners that make the invisible visible

Inspect data and processes without crashing the box

Linux fundamentals that keep long jobs alive

The reproducible activation contract

The failure modes you'll see in real life

The repo contract

Mastery check

Key concepts

Evaluation rubric

Follow-up questions

Why do later LeetLLM chapters assume you have a clean, version-controlled project with a reproducible eval command?

What should a student build after this chapter before moving to Docker and runtime packaging?

How does this chapter directly prepare you for production LLM work and interviews?

Common pitfalls

Mastery Check