HTTP REST Client with Retry Logic (python, written by Codex)

Codex wrote this python project from the task below. It installed and ran on a clean Ubuntu 22.04 machine as written.

Task given to the agent:

TASK: HTTP REST Client with Retry Logic

Write a program that sends HTTP requests to REST APIs with configurable retry logic, response validation, request chaining, and comprehensive logging of request/response cycles.

FUNCTIONAL REQUIREMENTS:
- Accept a URL as a command-line argument with --method flag (GET, POST, PUT, PATCH, DELETE; default: GET)
- Support sending request body via --data flag (JSON string) or --file flag (read body from a file)
- Support custom headers via --header flag (repeatable, e.g., --header "Content-Type: application/json" --header "Authorization: Bearer token")
- Implement configurable retry logic: --retries flag (default 3), --backoff flag for backoff strategy (linear, exponential; default: exponential), --retry-delay flag for initial delay in seconds (default 1)
- Retry only on configurable HTTP status codes via --retry-on flag (default: 429, 500, 502, 503, 504)
- Support request timeout via --timeout flag (default 30 seconds)
- Support request chaining from a JSON recipe file via --chain flag: execute a sequence of requests where later requests can reference values from earlier responses using JSONPath expressions
- Log complete request/response cycles: method, URL, headers, body, status code, response headers, response body, timing, and retry attempts
- Support response validation via --expect-status flag (e.g., --expect-status 200) and --expect-json flag (JSONPath expression that must exist in the response)
- Print formatted response to console: status code, headers, and pretty-printed body (auto-detect JSON and format it)
- Save the full request/response log as JSON with --output flag (default: request_log.json)
- If no URL is given, demonstrate the client by sending requests to a public test API (httpbin.org or similar), showing GET, POST with JSON body, retry on simulated failure, and a simple request chain
- Handle errors: connection refused, DNS failure, SSL errors, malformed JSON responses, and redirect loops

Create a complete Python project for a clean Ubuntu 22.04 machine with only Python 3.10+ installed. Include:
- Source code
- requirements.txt with all dependencies (direct and transitive) pinned to exact versions
- README.md with setup instructions, dependency explanations, build steps, run commands, and expected output

[]

# HTTP REST Client with Retry Logic (Python)

Sends REST requests with retry/backoff policies, status-code retry filters, timeout controls, response validation via status and JSONPath, request chaining from JSON recipes, and full request/response logging.

## Requirements

- Ubuntu 22.04
- Python 3.10+

## Dependencies (Pinned)

- `requests==2.32.3`
- `jsonpath-ng==1.6.1`

## Setup

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

## Run

```bash
python src/main.py https://httpbin.org/get
python src/main.py https://httpbin.org/post --method POST --data '{"hello":"world"}' --header "Content-Type: application/json"
python src/main.py https://httpbin.org/status/503 --retries 3 --backoff exponential --retry-delay 1 --retry-on 503
python src/main.py https://httpbin.org/get --expect-status 200 --expect-json '$.url'
python src/main.py --chain chain.json
python src/main.py
```

## Notes

- `--header` is repeatable.
- `--data` accepts inline body text, `--file` reads request body from disk.
- `--chain` accepts a JSON array (or object with `requests` array).
- No-URL mode runs a built-in demo chain on `httpbin.org`.

certifi==2024.8.30
charset-normalizer==3.3.2
idna==3.8
jsonpath-ng==1.6.1
ply==3.11
requests==2.32.3
urllib3==2.2.2

#!/usr/bin/env python3
from __future__ import annotations

import argparse
import json
import time
from pathlib import Path

import requests
from jsonpath_ng.ext import parse as parse_jsonpath

DEMO_CHAIN = [
    {"method": "GET", "url": "https://httpbin.org/get", "expect_status": 200, "expect_json": "$.url"},
    {"method": "POST", "url": "https://httpbin.org/post", "data": {"message": "hello"}, "expect_status": 200, "expect_json": "$.json.message"},
    {"method": "GET", "url": "https://httpbin.org/status/503", "retries": 2, "retry_on": [503], "expect_status": 503},
    {"method": "GET", "url": "https://httpbin.org/anything/{{step1.$.json.message}}", "expect_status": 200},
]


def parse_headers(values: list[str]) -> dict[str, str]:
    out = {}
    for h in values:
        if ":" not in h:
            continue
        k, v = h.split(":", 1)
        out[k.strip()] = v.strip()
    return out


def parse_retry_on(raw: str | None) -> set[int]:
    text = raw or "429,500,502,503,504"
    out = set()
    for p in text.split(","):
        p = p.strip()
        if p:
            out.add(int(p))
    return out


def replace_refs(text: str, context: list[dict]) -> str:
    out = text
    while "{{step" in out and "}}" in out:
        start = out.find("{{step")
        end = out.find("}}", start)
        if start < 0 or end < 0:
            break
        token = out[start + 2:end]
        # token format: step<idx>.<jsonpath>
        prefix, expr = token.split(".", 1)
        idx = int(prefix[4:])
        body = context[idx].get("response_json")
        value = ""
        if body is not None:
            vals = [m.value for m in parse_jsonpath(expr).find(body)]
            if vals:
                value = str(vals[0])
        out = out[:start] + value + out[end + 2:]
    return out


def request_with_retry(spec: dict, defaults: dict, context: list[dict], logs: list[dict]) -> dict:
    method = str(spec.get("method", defaults["method"])).upper()
    url = replace_refs(str(spec["url"]), context)
    headers = {**defaults["headers"], **spec.get("headers", {})}
    retries = int(spec.get("retries", defaults["retries"]))
    retry_on = set(spec.get("retry_on", defaults["retry_on"]))
    backoff = str(spec.get("backoff", defaults["backoff"]))
    retry_delay = float(spec.get("retry_delay", defaults["retry_delay"]))
    timeout = float(spec.get("timeout", defaults["timeout"]))

    body = spec.get("data")
    if isinstance(body, str):
        body = replace_refs(body, context)
    elif body is not None:
        body = json.dumps(body)

    attempt = 0
    while True:
        t0 = time.perf_counter()
        try:
            resp = requests.request(method, url, headers=headers, data=body, timeout=timeout, allow_redirects=True)
            elapsed = (time.perf_counter() - t0) * 1000.0
            try:
                parsed_json = resp.json()
            except Exception:
                parsed_json = None
            entry = {
                "attempt": attempt,
                "method": method,
                "url": url,
                "request_headers": headers,
                "request_body": body,
                "status_code": resp.status_code,
                "response_headers": dict(resp.headers),
                "response_body": resp.text,
                "response_json": parsed_json,
                "timing_ms": round(elapsed, 2),
            }
            logs.append(entry)

            if resp.status_code in retry_on and attempt < retries:
                attempt += 1
                delay = retry_delay * (attempt if backoff == "linear" else (2 ** (attempt - 1)))
                time.sleep(delay)
                continue

            expected = spec.get("expect_status")
            if expected is not None and int(expected) != resp.status_code:
                raise RuntimeError(f"Expected status {expected}, got {resp.status_code}")
            expr = spec.get("expect_json")
            if expr:
                if parsed_json is None:
                    raise RuntimeError("Expected JSON response but response is not JSON")
                vals = [m.value for m in parse_jsonpath(str(expr)).find(parsed_json)]
                if not vals:
                    raise RuntimeError(f"JSONPath not found: {expr}")
            return entry
        except Exception as exc:
            elapsed = (time.perf_counter() - t0) * 1000.0
            logs.append({
                "attempt": attempt,
                "method": method,
                "url": url,
                "request_headers": headers,
                "request_body": body,
                "status_code": 0,
                "response_headers": {},
                "response_body": "",
                "response_json": None,
                "timing_ms": round(elapsed, 2),
                "error": str(exc),
            })
            if attempt >= retries:
                raise
            attempt += 1
            delay = retry_delay * (attempt if backoff == "linear" else (2 ** (attempt - 1)))
            time.sleep(delay)


def print_response(entry: dict) -> None:
    print(f"Status: {entry['status_code']}")
    print("Headers:")
    for k, v in (entry.get("response_headers") or {}).items():
        print(f"  {k}: {v}")
    print("Body:")
    if entry.get("response_json") is not None:
        print(json.dumps(entry["response_json"], indent=2))
    else:
        print(entry.get("response_body", ""))


def run_chain(chain: list[dict], defaults: dict, output: Path) -> None:
    logs = []
    context = []
    for step in chain:
        attempt_log = []
        result = request_with_retry(step, defaults, context, attempt_log)
        logs.append({"request": step, "attempts": attempt_log})
        context.append(result)
        print_response(result)
    output.write_text(json.dumps({"generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "chain": chain, "logs": logs}, indent=2), encoding="utf-8")
    print(f"Saved log: {output}")


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description="HTTP REST Client with Retry Logic")
    p.add_argument("url", nargs="?")
    p.add_argument("--method", default="GET")
    p.add_argument("--data")
    p.add_argument("--file")
    p.add_argument("--header", action="append", default=[])
    p.add_argument("--retries", type=int, default=3)
    p.add_argument("--backoff", choices=["linear", "exponential"], default="exponential")
    p.add_argument("--retry-delay", type=float, default=1.0)
    p.add_argument("--retry-on", default="429,500,502,503,504")
    p.add_argument("--timeout", type=float, default=30.0)
    p.add_argument("--chain")
    p.add_argument("--expect-status", type=int)
    p.add_argument("--expect-json")
    p.add_argument("--output", default="request_log.json")
    return p


def main() -> int:
    args = parser().parse_args()
    defaults = {
        "method": args.method,
        "headers": parse_headers(args.header),
        "retries": args.retries,
        "backoff": args.backoff,
        "retry_delay": args.retry_delay,
        "retry_on": parse_retry_on(args.retry_on),
        "timeout": args.timeout,
    }
    output = Path(args.output).resolve()

    if args.chain:
        chain_payload = json.loads(Path(args.chain).read_text(encoding="utf-8"))
        chain = chain_payload if isinstance(chain_payload, list) else chain_payload.get("requests", [])
        run_chain(chain, defaults, output)
        return 0

    if not args.url:
        run_chain(DEMO_CHAIN, defaults, output)
        return 0

    body = None
    if args.file:
        body = Path(args.file).read_text(encoding="utf-8")
    elif args.data is not None:
        body = args.data

    spec = {
        "method": args.method,
        "url": args.url,
        "headers": defaults["headers"],
        "data": body,
        "expect_status": args.expect_status,
        "expect_json": args.expect_json,
    }
    attempt_log = []
    result = request_with_retry(spec, defaults, [], attempt_log)
    print_response(result)
    output.write_text(json.dumps({"generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "request": spec, "attempts": attempt_log}, indent=2), encoding="utf-8")
    print(f"Saved log: {output}")
    return 0


if __name__ == "__main__":
    try:
        raise SystemExit(main())
    except Exception as exc:
        print(f"Error: {exc}")
        raise SystemExit(1)