HMAC File Integrity Checker (python, written by Codex)

Codex wrote this python project from the task below. It installed and ran on a clean Ubuntu 22.04 machine as written.

Task given to the agent:

TASK: HMAC File Integrity Checker

Write a program that computes and verifies HMAC-based integrity checksums for files and directories, creating a manifest of checksums that can later be used to detect unauthorized modifications.

FUNCTIONAL REQUIREMENTS:
- Accept a file or directory path, a secret key, and a mode (generate or verify) as command-line arguments
- Support multiple hash algorithms selectable via --algorithm flag: SHA-256 (default), SHA-384, SHA-512, SHA3-256
- generate mode: Compute HMAC for each file and save a manifest file containing file paths, HMAC values, file sizes, and timestamps
- verify mode: Read the manifest, recompute HMACs, and report which files are unchanged, modified, added (present but not in manifest), or missing (in manifest but deleted)
- For directories, recursively process all files and support --exclude flag with glob patterns to skip files (e.g., --exclude "*.log,*.tmp")
- Support a --output flag for the manifest file path (default: integrity_manifest.json)
- Print a colored summary to console showing verification results: passed files in green, modified in red, missing in yellow, new files in blue
- Compute a master HMAC over the entire manifest to detect tampering of the manifest file itself
- Support incremental updates: when generating with an existing manifest, only recompute HMACs for files whose modification time has changed via --incremental flag
- If no arguments are given, generate sample files in a temporary directory, create an integrity manifest, modify one file, delete another, add a new file, then run verification to demonstrate all detection capabilities
- Handle binary and text files correctly, permission errors gracefully

Create a complete Python project for a clean Ubuntu 22.04 machine with only Python 3.10+ installed. Include:
- Source code
- requirements.txt with all dependencies (direct and transitive) pinned to exact versions
- README.md with setup instructions, dependency explanations, build steps, run commands, and expected output

[]

# HMAC File Integrity Checker (Python)

Computes and verifies HMAC manifests for files/directories to detect unauthorized changes.

## Requirements

- Ubuntu 22.04
- Python 3.10+

## Dependencies

- Direct: none
- Transitive: none
- Runtime uses only Python standard library modules.

Pinned dependency file is `requirements.txt` (empty external set).

## Setup

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

## Run

```bash
python src/main.py <generate|verify> <path> <secret-key> [--algorithm SHA-256|SHA-384|SHA-512|SHA3-256] [--output integrity_manifest.json] [--exclude "*.log,*.tmp"] [--incremental]
```

Examples:

```bash
python src/main.py generate ./data my-secret --algorithm SHA-512 --output ./integrity_manifest.json
python src/main.py verify ./data my-secret --output ./integrity_manifest.json
python src/main.py generate ./data my-secret --exclude "*.log,*.tmp" --incremental
```

## Behavior

- `generate` writes a JSON manifest with file paths, per-file HMAC values, file size, mtime, timestamp, and a master HMAC over manifest content.
- `verify` reports `UNCHANGED`, `MODIFIED`, `MISSING`, and `ADDED` files in color.
- Supports recursive directory processing and glob exclusion patterns.
- Supports incremental generation by reusing unchanged file HMACs from existing manifests.
- Running without arguments executes a demo that creates files, mutates files, and verifies detection.

## Expected Output (verify summary)

```text
UNCHANGED ...
MODIFIED ...
MISSING ...
ADDED ...

Summary
  unchanged: N
  modified : N
  missing  : N
  added    : N
  manifest : valid|tampered
```

# No external dependencies required.
# Standard library only (argparse, hashlib, hmac, json, pathlib, fnmatch).

#!/usr/bin/env python3
from __future__ import annotations

import argparse
import fnmatch
import hashlib
import hmac
import json
import os
import tempfile
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Iterable


RESET = "\033[0m"
GREEN = "\033[32m"
RED = "\033[31m"
YELLOW = "\033[33m"
BLUE = "\033[34m"


def colorize(color: str, text: str) -> str:
    return f"{color}{text}{RESET}"


def now_iso() -> str:
    return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")


def iso_from_ms(ms: int) -> str:
    return datetime.fromtimestamp(ms / 1000.0, tz=timezone.utc).isoformat().replace("+00:00", "Z")


def normalize_algorithm(value: str) -> tuple[str, str]:
    v = value.strip().lower().replace("_", "-")
    mapping = {
        "sha-256": ("sha-256", "sha256"),
        "sha256": ("sha-256", "sha256"),
        "sha-384": ("sha-384", "sha384"),
        "sha384": ("sha-384", "sha384"),
        "sha-512": ("sha-512", "sha512"),
        "sha512": ("sha-512", "sha512"),
        "sha3-256": ("sha3-256", "sha3_256"),
    }
    if v not in mapping:
        raise ValueError("Unsupported algorithm. Use SHA-256, SHA-384, SHA-512, or SHA3-256.")
    return mapping[v]


def parse_excludes(raw: str) -> list[str]:
    return [x.strip() for x in raw.split(",") if x.strip()]


def should_exclude(rel_path: str, name: str, patterns: list[str]) -> bool:
    return any(fnmatch.fnmatch(rel_path, p) or fnmatch.fnmatch(name, p) for p in patterns)


def collect_files(target: Path, exclude_patterns: list[str]) -> tuple[list[Path], bool]:
    if target.is_file():
        return [target], False
    if not target.is_dir():
        raise ValueError("Target path must be a file or directory.")

    files: list[Path] = []
    for root, _, names in os.walk(target):
        for name in names:
            full = Path(root) / name
            rel = full.relative_to(target).as_posix()
            if should_exclude(rel, name, exclude_patterns):
                continue
            files.append(full)
    files.sort()
    return files, True


def relative_key(root: Path, file_path: Path, root_is_dir: bool) -> str:
    if root_is_dir:
        return file_path.relative_to(root).as_posix()
    return file_path.name


def hmac_bytes(data: bytes, secret: str, digest_name: str) -> str:
    return hmac.new(secret.encode("utf-8"), data, digestmod=digest_name).hexdigest()


def hmac_file(path: Path, secret: str, digest_name: str) -> str:
    hm = hmac.new(secret.encode("utf-8"), digestmod=digest_name)
    with path.open("rb") as f:
        while True:
            chunk = f.read(1024 * 1024)
            if not chunk:
                break
            hm.update(chunk)
    return hm.hexdigest()


def build_master_payload(manifest_no_master: dict) -> str:
    entries = sorted(manifest_no_master["entries"], key=lambda e: e["path"])
    lines = [
        str(manifest_no_master["version"]),
        manifest_no_master["algorithm"],
        manifest_no_master["rootPath"],
        manifest_no_master["generatedAt"],
    ]
    for e in entries:
        lines.append(f"{e['path']}|{e['size']}|{e['mtimeMs']}|{e['timestamp']}|{e['hmac']}")
    return "\n".join(lines)


def compute_master_hmac(manifest_no_master: dict, secret: str, digest_name: str) -> str:
    payload = build_master_payload(manifest_no_master).encode("utf-8")
    return hmac_bytes(payload, secret, digest_name)


@dataclass
class VerifyResult:
    unchanged: list[str]
    modified: list[str]
    missing: list[str]
    added: list[str]
    manifest_valid: bool


def generate_mode(target: Path, secret: str, algorithm_input: str, output: Path, excludes: list[str], incremental: bool) -> None:
    algorithm_label, digest_name = normalize_algorithm(algorithm_input)
    files, root_is_dir = collect_files(target, excludes)

    previous = {}
    if incremental and output.exists():
        prior = json.loads(output.read_text(encoding="utf-8"))
        for entry in prior.get("entries", []):
            previous[str(entry.get("path", ""))] = entry

    entries = []
    for file_path in files:
        stat = file_path.stat()
        rel = relative_key(target, file_path, root_is_dir)
        mtime_ms = int(stat.st_mtime * 1000)
        prior = previous.get(rel)
        if prior and int(prior.get("mtimeMs", -1)) == mtime_ms and int(prior.get("size", -1)) == stat.st_size:
            digest = str(prior.get("hmac", ""))
        else:
            digest = hmac_file(file_path, secret, digest_name)
        entries.append(
            {
                "path": rel,
                "hmac": digest,
                "size": stat.st_size,
                "mtimeMs": mtime_ms,
                "timestamp": iso_from_ms(mtime_ms),
            }
        )
    entries.sort(key=lambda e: e["path"])

    manifest_no_master = {
        "version": 1,
        "generatedAt": now_iso(),
        "rootPath": str(target.resolve()),
        "algorithm": algorithm_label,
        "entries": entries,
    }
    manifest = dict(manifest_no_master)
    manifest["masterHmac"] = compute_master_hmac(manifest_no_master, secret, digest_name)

    output.parent.mkdir(parents=True, exist_ok=True)
    output.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
    print(f"Manifest written: {output}")
    print(f"Processed files: {len(entries)}")
    print(f"Algorithm: {algorithm_label}")
    print(f"Incremental: {'enabled' if incremental else 'disabled'}")


def verify_mode(target: Path, secret: str, algorithm_override: str | None, output: Path, excludes: list[str]) -> VerifyResult:
    if not output.exists():
        raise FileNotFoundError(f"Manifest not found: {output}")
    manifest = json.loads(output.read_text(encoding="utf-8"))
    entries = manifest.get("entries")
    if not isinstance(entries, list):
        raise ValueError("Manifest is malformed: entries missing.")

    manifest_label, manifest_digest = normalize_algorithm(str(manifest.get("algorithm", "sha-256")))
    if algorithm_override:
        effective_label, effective_digest = normalize_algorithm(algorithm_override)
    else:
        effective_label, effective_digest = manifest_label, manifest_digest

    manifest_no_master = {
        "version": manifest.get("version", 1),
        "generatedAt": manifest.get("generatedAt", ""),
        "rootPath": manifest.get("rootPath", ""),
        "algorithm": manifest_label,
        "entries": entries,
    }
    expected_master = compute_master_hmac(manifest_no_master, secret, manifest_digest)
    manifest_valid = expected_master == manifest.get("masterHmac", "")

    files, root_is_dir = collect_files(target, excludes)
    current = {relative_key(target, f, root_is_dir): f for f in files}
    seen: set[str] = set()

    unchanged: list[str] = []
    modified: list[str] = []
    missing: list[str] = []
    for entry in entries:
        rel = str(entry.get("path", ""))
        file_path = current.get(rel)
        if file_path is None:
            missing.append(rel)
            continue
        seen.add(rel)
        digest = hmac_file(file_path, secret, effective_digest)
        if digest == entry.get("hmac"):
            unchanged.append(rel)
        else:
            modified.append(rel)

    added = sorted([k for k in current.keys() if k not in seen])
    unchanged.sort()
    modified.sort()
    missing.sort()

    if not manifest_valid:
        print(colorize(RED, "Manifest master HMAC mismatch: manifest may be tampered."))
    if algorithm_override and effective_label != manifest_label:
        print(colorize(YELLOW, f"Using --algorithm {effective_label} instead of manifest algorithm {manifest_label}."))
    for p in unchanged:
        print(colorize(GREEN, f"UNCHANGED {p}"))
    for p in modified:
        print(colorize(RED, f"MODIFIED  {p}"))
    for p in missing:
        print(colorize(YELLOW, f"MISSING   {p}"))
    for p in added:
        print(colorize(BLUE, f"ADDED     {p}"))

    print("\nSummary")
    print(f"  unchanged: {len(unchanged)}")
    print(f"  modified : {len(modified)}")
    print(f"  missing  : {len(missing)}")
    print(f"  added    : {len(added)}")
    print(f"  manifest : {'valid' if manifest_valid else 'tampered'}")

    return VerifyResult(unchanged, modified, missing, added, manifest_valid)


def run_demo() -> int:
    with tempfile.TemporaryDirectory(prefix="hmac-integrity-") as tmp:
        root = Path(tmp)
        (root / "alpha.txt").write_text("alpha\n", encoding="utf-8")
        (root / "beta.txt").write_text("beta\n", encoding="utf-8")
        (root / "nested").mkdir(parents=True, exist_ok=True)
        (root / "nested" / "gamma.bin").write_bytes(os.urandom(128))
        manifest = root / "integrity_manifest.json"
        secret = "demo-secret"

        print(f"Demo directory: {root}")
        generate_mode(root, secret, "sha-256", manifest, [], False)
        (root / "alpha.txt").write_text("alpha\ntampered\n", encoding="utf-8")
        (root / "beta.txt").unlink(missing_ok=True)
        (root / "new.txt").write_text("new file\n", encoding="utf-8")
        result = verify_mode(root, secret, "sha-256", manifest, [])
        ok = result.manifest_valid and not result.modified and not result.missing and not result.added
        print(f"Demo verification result: {'PASS' if ok else 'FAIL (expected for demo)'}")
    return 0


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="HMAC File Integrity Checker",
        epilog='Example: python src/main.py generate ./data my-secret --exclude "*.log,*.tmp" --incremental',
    )
    parser.add_argument("mode", nargs="?")
    parser.add_argument("target_path", nargs="?")
    parser.add_argument("secret_key", nargs="?")
    parser.add_argument("--algorithm", default="sha-256")
    parser.add_argument("--output", default="integrity_manifest.json")
    parser.add_argument("--exclude", default="")
    parser.add_argument("--incremental", action="store_true")
    return parser


def main() -> int:
    parser = build_parser()
    args = parser.parse_args()
    if not args.mode:
        return run_demo()

    if not args.target_path or not args.secret_key:
        parser.print_help()
        return 1

    mode = str(args.mode).lower()
    target = Path(args.target_path).resolve()
    output = Path(args.output).resolve()
    excludes = parse_excludes(args.exclude)

    try:
        if mode == "generate":
            generate_mode(target, args.secret_key, args.algorithm, output, excludes, args.incremental)
            return 0
        if mode == "verify":
            result = verify_mode(target, args.secret_key, args.algorithm, output, excludes)
            ok = result.manifest_valid and not result.modified and not result.missing and not result.added
            return 0 if ok else 2
        print("Error: mode must be generate or verify.")
        return 1
    except PermissionError:
        print("Error: permission denied while reading files.")
        return 1
    except FileNotFoundError as exc:
        print(f"Error: {exc}")
        return 1
    except ValueError as exc:
        print(f"Error: {exc}")
        return 1
    except Exception as exc:
        print(f"Error: {exc}")
        return 1


if __name__ == "__main__":
    raise SystemExit(main())