Git, Shell, Linux for AI

Your laptop runs an evaluation script, compares three saved model predictions against ground truth, and prints 0.667: two answers matched. A teammate clones your repo, runs the exact same script, and crashes immediately with a FileNotFoundError. The Python code arrived over Git, but the dataset it reads stayed behind on your hard drive.

We'll build a minimal project called access-rag that anyone can clone, configure, and execute reliably. Its evaluation (an eval) evaluates saved predictions against expected answers without invoking live APIs or needing secret tokens. Git carries the source code and fixtures, shell scripts define repeatable entrypoints, and a fresh clone proves that nothing was left behind on your laptop.

Meet the terminal, shell, and repository

The terminal provides your command window; the shell running inside it (typically Bash or Zsh) parses commands and launches processes. Linux is common on remote AI servers and multi-GPU clusters, so the command-line habits you build here transfer directly to those environments.

Use Bash or Zsh on macOS, Linux, or Windows Subsystem for Linux (WSL), with Git and Python 3.12 or newer installed. You don't need a physical GPU to complete this project; the GPU commands we'll inspect later illustrate routine remote-cluster diagnostics.

Before writing code, make your working path visible. A filesystem path locates a file or directory (. represents your current directory and .. points to its parent). We run pwd (print working directory) to check the current folder, mkdir to create our project directory, and cd to move into it. Run these commands line by line in your workspace:

Notice cd access-rag || exit 1. If the directory change fails, the shell halts with exit status 1 rather than quietly executing subsequent commands in the wrong folder. In shell conventions, exit status 0 signals success, while any nonzero code flags a failure. Whenever a pipeline crashes, always read the first failing line rather than the final traceback.

Git tracks changes by recording deliberate, immutable snapshots of files called commits. A directory managed by Git is a repository (or repo). Running git init -b main creates Git's internal bookkeeping directory at .git/ and designates main as your initial branch. A branch represents an independent line of development; we'll stay right on main. Leave everything inside .git/ to Git.

Before creating your first commit, assign an author name and email. These settings stay local to access-rag. (Add --global only if you want to use this identity across every repository on your machine):

git status confirms you're on main with no commits yet. Your project folder contains three distinct tiers: the working tree (the editable files on disk), the staging area or index (the selected changes staged for the next snapshot), and committed history. Running git add copies a file's exact current content into the staging area, and git commit permanently seals those staged contents into history.^{[1]Reference 1Pro Git (2nd ed.)https://git-scm.com/book/en/v2} If you edit a file after staging it, run git add again to stage the updated state.

That lifecycle explains the missing-file bug. A file can sit comfortably on your laptop's filesystem without ever entering the staging area or commit history. Before adding our eval code, establish a hard boundary: which files must travel through Git, and which files must never leave your local disk?

Ignore the files that shouldn't travel

Before adding the scorer, separate project inputs from local residue. Create .gitignore with patterns for secrets, caches, generated indexes, and model artifacts:

.gitignore keeps matching untracked files out of ordinary git add operations; it doesn't remove files already committed or rewrite Git history. It also isn't an access control: git add -f can stage an ignored file. If git ls-files '.env*' returns any file containing credentials, halt immediately: remove it from tracking with git rm --cached before sharing your branch. When an active credential leaks into commit history, rotate that key on your provider immediately. A subsequent commit that deletes the file still leaves the key readable in earlier Git revisions.^{[2]Reference 2Git: gitignore Documentationhttps://git-scm.com/docs/gitignore}

Use Git LFS only when a large artifact truly belongs in this repo. GitHub warns above 50 MiB and blocks ordinary Git files above 100 MiB; LFS keeps a small pointer in Git while storing the bytes separately.^{[3]Reference 3About large files on GitHubhttps://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github} Generated checkpoints and local model directories stay ignored.

Production datasets and frequently changing weights usually belong in object storage or a data registry, with a versioned manifest in Git. LFS quotas and file caps depend on the host and plan, so treat LFS as a pointer mechanism, not a dataset store.

An ignored path won't be picked up by LFS tracking. Keep selected LFS patterns out of .gitignore, then check whether the separate git-lfs tool is installed before adding weights:

Commit this skeleton before adding the fixture. Inspect the staged changes with git diff --cached, then record them. git commit saves locally; it doesn't upload anything to GitHub.

On a normal LFS checkout, Git materializes the object. Without LFS, or with GIT_LFS_SKIP_SMUDGE=1, a clone may contain pointer text instead. git lfs pull downloads and checks out missing objects after LFS is installed.^{[4]Reference 4git-lfs-pullhttps://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-pull.adoc} Commit .gitattributes when tracking patterns change, and document which workflows need those bytes.

Commit the three-row fixture

Our repository has directory structure, but no evaluation data yet. A fixture is a small, deterministic test dataset used for repeatable verification. We'll add three access-request examples, each pairing a user prompt with an expected classification label and a saved model prediction. The shell idiom cat > path << 'EOF' writes every line up to EOF into the target file, replacing any existing content:

Row 102 is the intentional miss (blocked vs escalated). Two of three labels match, so accuracy is $2 / 3$, printed to three decimal places as 0.667. Three fixed examples are enough to test the workflow, not enough to establish model quality. The downloadable fixture contains the same rows; the browser's runnable example reads that copy.

A scorer that fails out loud

Give the check a small program that reads JSONL, one JSON object per line, and computes exact-match accuracy. You don't need to understand every Python statement yet. Follow its three jobs: validate the rows, count matching labels, and report whether at least two thirds match. Python for AI Engineering later explains the functions, exceptions, and tests behind this work.

A gate is a check that blocks a later action when it fails. Valid rows at or above the threshold print a score and return status 0; missing, malformed, empty, or below-threshold input returns nonzero. Run mkdir -p scripts, then save the utility below as scripts/score_access_requests.py. The shell wrapper will invoke it on every clone.

Labels are normalized with strip().lower(), which removes surrounding whitespace and ignores letter case. The gate compares integer counts (correct * 3 < total * 2) instead of asking whether a rounded float equals 0.667. A score of 1.000 also passes; 0.667 is this fixture's expected output, not the only permitted score.

Read the printed count for what happened, then read the exit status for whether the command may continue. Errors use standard error (stderr), a separate output stream from ordinary results. Shell scripts can capture or redirect those streams independently.

The pre-commit gate that protects the score

The scorer works, but remembering to trigger it manually every time doesn't scale. We'll create an executable shell wrapper that both our local pre-commit hook and the clean-clone test can invoke.

The command a teammate needs to run must be a tracked script in the repo, never a personal shell alias. Aliases and installed .git/hooks/ binaries stay pinned to one person's laptop; repro.sh travels everywhere with the repository.

Now connect that command to Git via a pre-commit hook. A hook is an executable script that Git automatically triggers before specific lifecycle actions like committing or pushing. Git stores active hooks inside .git/hooks/, meaning they never travel in a clone; we keep the canonical hook source in scripts/ and provide a guarded installer that sets up the local hook on every new checkout.^{[5]Reference 5githookshttps://git-scm.com/docs/githooks} The installer verifies existing hooks so it won't silently overwrite other tooling.

Test it.

If the scorer reports a regression or the fixture disappears, the commit stops. This hook reads working-tree files, not the staged snapshot. A passing unstaged edit could therefore hide a failing version in the commit. Review git diff for unstaged edits and git diff --cached for the version you're about to commit, then test the committed version in a clone.

--no-verify can bypass a local hook, so it provides fast feedback rather than a security boundary. Run ./repro.sh in continuous integration (CI), an automated check of a committed version, too. That shared check catches a bypassed or missing local hook without borrowing working-tree edits.

Trace a regression through Git history

The gate tells you when behavior changes, but it doesn't name which file or commit broke it. When yesterday's passing 0.667 suddenly regresses to exit status 1, capture the terminal output and exit code before modifying any code:

git log names the snapshots that led here. git show reveals what the latest snapshot changed, while git status --short separates committed history from edits still sitting in the worktree. A green rerun with a dirty worktree is weaker evidence than a green commit with a clean status.

When the bad behavior could have entered anywhere in a long history, test the middle instead of reading every diff. git bisect checks one candidate commit at a time and narrows the range. Start from a clean worktree or set aside uncommitted edits, because bisect checks out candidate snapshots:

At each candidate, run the gate and read its output before classifying the commit:

Enter git bisect good if the check passes, git bisect bad if it reproduces the regression, or git bisect skip if an unrelated setup problem prevents testing. Repeat the test and classification at each new candidate until Git reports the first bad commit. Only then inspect it and return to your original branch:

The shell's zero-versus-nonzero status maps directly to git bisect good and git bisect bad. If a candidate commit can't execute because of a broken environment, use git bisect skip rather than falsely condemning that revision. Once Git identifies the first bad snapshot, running git show <commit> gives you the exact diff that introduced the bug.^{[1]Reference 1Pro Git (2nd ed.)https://git-scm.com/book/en/v2}

Personal shortcuts vs repo commands

The hook now runs on your development machine, but shortcuts defined in ~/.zshrc or ~/.bashrc stay strictly local. Shell functions can accelerate daily development on your personal machine; they can never serve as a repository's official setup contract.

For NVIDIA setups, nvidia-smi --query-gpu with --format=csv provides a scriptable interface for custom shell helpers.^{[6]Reference 6nvidia-smi documentationhttps://docs.nvidia.com/deploy/nvidia-smi/index.html} The functions below assume GNU utilities on a Linux GPU system. Command options often differ on macOS, which doesn't ship NVIDIA drivers; run these helpers on remote GPU instances, and rely on ./repro.sh as the portable repo entrypoint.

After saving the functions, run source ~/.zshrc or source ~/.bashrc once to reload the configuration. gpu, ds, and repro then become shortcuts. Keep ./repro.sh in setup docs and CI. If a clone needs the alias, the command still isn't reproducible.

Inspect data, then reclaim a GPU

Remote debugging on cloud nodes starts with a fundamental question: is the input dataset malformed, or is the training process hung? First inspect large datasets without flooding your SSH session. Running cat train.jsonl dumps millions of lines into your terminal buffer; start with a single line and targeted row counts instead:

The pipe operator | routes stdout from one process into stdin of the next. Running grep '"restored"' eval/access_requests.jsonl | wc -l streams lines through the filter and counts matches on the fly. Remember that wc -l counts newline characters (\n), not JSON objects: a final line lacking a trailing newline won't be counted, and blank lines will be counted. Similarly, grep performs plain string matching rather than schema validation. These utilities offer rapid checks, not replacements for full data validation.

Without pipefail, Bash normally uses the final command's status for a pipeline. That means wc could report success (status 0) even if grep couldn't open the file. With pipefail, a failure in an earlier stage makes the pipeline return nonzero too. Bash still waits for the pipeline to finish; this option doesn't immediately cancel the other stages.^{[7]Reference 7Bash Reference Manual: Pipelineshttps://www.gnu.org/software/bash/manual/html_node/Pipelines.html}

set -euo pipefail is useful, but it doesn't replace explicit error handling. -u catches unset variables; -e exits on many unhandled nonzero statuses. Commands tested by if, while, !, or parts of && / || lists have exceptions. Testing a function this way can also disable -e inside the function. Check important failures deliberately, and remember that grep returns 1 for a valid search with no matches.^{[8]Reference 8Bash Reference Manual: The Set Builtinhttps://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html}

For example, ask whether a label appears instead of treating every nonzero status as a crash. A nonexistent file is a different outcome from an existing file with no matching label:

If the output is surprising, verify location before blaming Python. A relative path such as eval/access_requests.jsonl starts from the current directory; an absolute path starts at /. Quote variables that contain paths so spaces don't split one path into several arguments. Then check the exact capability a command needs:

chmod +x scripts/run_eval.sh adds the executable bit. It doesn't make a script trustworthy, grant access to its input, or fix a filesystem mounted with execution disabled. Avoid chmod 777: it grants write access far beyond what this repo needs and hides the ownership problem.

Now diagnose a GPU that remains pinned after a training script hangs or an SSH session terminates abruptly. A detached PyTorch process can stay resident in VRAM while nvidia-smi reports active compute processes.^{[6]Reference 6nvidia-smi documentationhttps://docs.nvidia.com/deploy/nvidia-smi/index.html} A process ID (PID) identifies a running program; it isn't an immediate license to run kill -9. Check who owns it and what command launched it, request a graceful shutdown, and only force-kill if the process refuses to stop and you're authorized to terminate it.

Only signal a process you own or are authorized to operate. kill -0 checks whether a PID exists and whether you may signal it; it doesn't prove that the PID still belongs to the same program. Re-inspect owner and command before SIGKILL. If VRAM remains allocated but nvidia-smi shows no owning compute PID, there's no process ID to kill. Check container or PID-namespace visibility and other driver clients, then move to container-runtime or driver diagnosis.

Start with SIGTERM. Signal 15 gives the process a chance to close files and release CUDA memory. SIGKILL (signal 9) can't be caught or handled and can leave lock files or corrupt checkpoints, so reserve it for a process that refuses to exit.^{[9]Reference 9The Open Group Base Specifications: signal.hhttps://pubs.opengroup.org/onlinepubs/9699919799/basedefs/signal.h.html}

Keep a long job alive across SSH

./repro.sh finishes in a blink. A training job can run for hours through SSH, and closing the laptop shouldn't kill it. The choice depends on what evidence you'll need when you reconnect.

• nohup python train.py > train.log 2>&1 </dev/null & runs a noninteractive process that ignores hangup and writes a log. It won't give you a session to type into later.
• tmux new -s training gives you a detachable session. Detach with Ctrl-b d, reconnect, then run tmux attach -t training; the job keeps its terminal while you're away.^{[10]Reference 10tmux Getting Startedhttps://github.com/tmux/tmux/wiki/Getting-Started}
• nice -n 10 python train.py lowers CPU scheduling priority relative to the default. It doesn't cap RAM or GPU, and it doesn't keep a process alive after logout.^{[11]Reference 11GNU Coreutils Manualhttps://www.gnu.org/software/coreutils/manual/coreutils.html}
• When disk fills up, find heavy paths with du -ah /workspace | sort -rh | head -20. When you need the training PID, pgrep -af 'python.*train.py' finds candidates without matching the search command itself.

Use tmux when you may need to inspect or steer the run later. Use nohup for a noninteractive job whose progress you can inspect through its log. Neither protects against a machine reboot, an out-of-memory kill, or a cluster scheduler stopping the job. They also don't make its inputs or environment reproducible.

Git carried the scripts and fixture. Python still needs a declared, rebuildable environment on the next machine.

Rebuild Python on every clone

The next common onboarding failure is ModuleNotFoundError: the tracked Python code arrived, but python points to a different machine's site-packages. Python's built-in venv module creates isolated virtual environments, but compiled virtual environments aren't portable across paths or machines. Every fresh clone must construct its own .venv locally.^{[12]Reference 12venv: Creation of Virtual Environmentshttps://docs.python.org/3/library/venv.html} Our scorer relies solely on the standard library; the activation script below checks for CUDA availability if torch happens to be installed.

The outer status check matters because this file is sourced. A helper can return nonzero while later commands keep running and make setup look successful. _activate_main checks its important steps explicitly; the surrounding if captures its result before cleanup, and the file-level return passes that status back to the caller. Run source activate.sh && ./repro.sh to continue only after successful setup. If your calling shell has -e enabled, handle a setup failure in a conditional too rather than assuming a nonzero source will leave that shell open.

The scorer's dependency set is empty today. Once packages arrive, plain version ranges won't reproduce an install. Choose one complete contract: commit pyproject.toml plus uv.lock and run uv sync --locked, or commit a requirements file that pins every direct and transitive package and includes accepted artifact hashes for pip --require-hashes.^{[13]Reference 13Locking and Syncinghttps://docs.astral.sh/uv/concepts/projects/sync/[14]Reference 14Secure Installshttps://pip.pypa.io/en/stable/topics/secure-installs/} A lockfile constrains Python packages; it doesn't pin the OS, driver, or system libraries. Docker handles that boundary next.

.python-version suggests Python 3.12 to version managers; it doesn't install Python or enforce that version by itself. This lab checks for 3.12 or newer because it uses only the standard library. A project that depends on one exact Python release should pin and verify that release too.

.env.example demonstrates how to record a variable name without its secret value. This scorer doesn't need ACCESS_API_KEY; leave it empty. activate.sh deliberately doesn't source .env: sourcing treats every line as shell code. When an application does need a key, export trusted values through your shell or a dedicated environment loader and fail clearly when the required variable is absent.

Expected output: source activate.sh should print an executable inside access-rag/.venv, a Python version of 3.12 or newer, the current CUDA_VISIBLE_DEVICES state, and Environment ready. If setup prints ERROR, fix that first and don't run the eval from a half-created environment.

Create README.md with the same setup order another machine should follow:

Track the setup contract before testing a clone. If the hook is configured, committing these files runs the same eval gate first.

With ordinary status settings, no output from git status --short means the tracked files and staging area match the new commit and no unignored, untracked files were reported. Ignored files can still exist. Now test the stronger claim: did the required project files actually enter the commit?

When the clone still fails

Keep each diagnosis and its prevention in the repo. The next clone should name its missing contract instead of requiring machine-specific guesswork.

Prove it with a throwaway clone

The code, fixture, and setup scripts are committed. Now verify whether a completely isolated copy can run without borrowing any untracked files from your laptop. A valid clone must receive the fixture and scripts, build a fresh .venv, and install its local pre-commit hook. It requires no secret tokens for this eval. Both copies must output 0.667 when running ./repro.sh.

mktemp -d generates a clean, unique temporary directory. The clone receives committed files without borrowing the original worktree's untracked inputs or installed .git/hooks/. Calling the tracked ./repro.sh avoids relying on a personal alias. Adding --no-local uses regular Git transport instead of the local hardlink optimization; a local-path clone still doesn't need a network server.^{[16]Reference 16git-clonehttps://git-scm.com/docs/git-clone} Wrapping the commands in a subshell (...) keeps activation changes out of your active terminal.

Check all four results: activation prints its Python executable and version, the eval prints three rows and 0.667 (2/3), git status --short prints nothing, and the clone check exits 0. The ignored .venv may exist without making the tree dirty. A setup or eval failure stops the subshell instead of being hidden by a later successful git status.

This is a file-isolation test on the same machine, not a new machine. It still shares your installed tools, environment variables, network access, and operating system. If it needs an untracked project file, add the nonsecret input or its setup instructions to Git, commit, and rerun. Testing on another machine or in CI exposes the next layer of assumptions.

Before sharing a commit, inspect git status, run ./repro.sh, and repeat the temporary clone. Record any missing file, variable, package, permission, or machine capability in the tracked contract first.