HTTP File Downloader (java, written by Codex)

Codex wrote this java project from the task below. It installed and ran on a clean Ubuntu 22.04 machine as written.

Task given to the agent:

TASK: HTTP File Downloader

Write a program that downloads files from HTTP/HTTPS URLs with support for resumable downloads, progress tracking, concurrent downloads, and checksum verification.

FUNCTIONAL REQUIREMENTS:
- Accept one or more URLs as command-line arguments
- Display a progress bar during download showing: percentage complete, bytes downloaded, total size, download speed, and estimated time remaining
- Support resumable downloads via HTTP Range headers: if a download is interrupted, restarting with the same URL and output path should resume from where it stopped via --resume flag
- Support concurrent downloading of multiple files via --parallel flag with configurable thread count (--threads, default 4)
- Support downloading all URLs listed in a text file (one URL per line) via --list flag
- Verify downloaded file integrity via --checksum flag accepting algorithm:hash format (e.g., --checksum sha256:abc123...)
- Support custom HTTP headers via --header flag (e.g., --header "Authorization: Bearer token")
- Support following HTTP redirects (up to 10 hops) and report the final URL
- Set connection timeout via --timeout flag (default 30 seconds) and retry failed downloads via --retries flag (default 3) with exponential backoff
- Save files to a directory specified by --output flag (default: current directory), using the filename from the URL or Content-Disposition header
- Print a download summary to console: file name, size, time taken, average speed, and checksum verification result
- If no URLs are given, download a set of sample public domain text files from Project Gutenberg, display progress for each, and print a summary table
- Handle errors: DNS resolution failures, SSL certificate errors, HTTP 4xx/5xx responses, disk full, and network timeouts

Create a complete Java project for a clean Ubuntu 22.04 machine with only JDK 17+ installed. Include:
- Source code
- pom.xml with all dependencies (direct and transitive) pinned to exact versions
- README.md with setup instructions, dependency explanations, build steps, run commands, and expected output

[]

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.tmlr</groupId>
    <artifactId>http-file-downloader</artifactId>
    <version>1.0.0</version>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.13.0</version>
                <configuration>
                    <release>17</release>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.4.2</version>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>HttpFileDownloader</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

# HTTP File Downloader (Java)

Downloads one or many HTTP/HTTPS files with resumable range support, progress reporting, retries/backoff, redirect resolution, parallel mode, checksum verification, list-file input, and summary output.

## Requirements

- Ubuntu 22.04
- JDK 17+
- Maven 3.8+

## Dependencies

- No external runtime dependencies (uses Java 17 `HttpClient` and standard library only)

## Build

```bash
mvn clean package
```

## Run

```bash
java -cp target/http-file-downloader-1.0.0.jar HttpFileDownloader https://example.com/file.txt
java -cp target/http-file-downloader-1.0.0.jar HttpFileDownloader https://example.com/file.bin --resume --output ./downloads
java -cp target/http-file-downloader-1.0.0.jar HttpFileDownloader --list urls.txt --parallel --threads 4 --timeout 30 --retries 3
java -cp target/http-file-downloader-1.0.0.jar HttpFileDownloader https://example.com/file --checksum sha256:abc123...
java -cp target/http-file-downloader-1.0.0.jar HttpFileDownloader https://example.com/file --header "Authorization: Bearer TOKEN"
java -cp target/http-file-downloader-1.0.0.jar HttpFileDownloader
```

## Notes

- Redirects are followed manually up to 10 hops.
- `--resume` uses `Range` and appends only when server returns `206`.
- `--parallel` uses a fixed thread pool (`--threads`, default `4`).
- No-input mode downloads sample Project Gutenberg text files.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URLDecoder;
import java.net.http.HttpClient;
import java.net.http.HttpHeaders;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.MessageDigest;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

public final class HttpFileDownloader {
    private static final List<String> SAMPLE_URLS = List.of(
        "https://www.gutenberg.org/files/11/11-0.txt",
        "https://www.gutenberg.org/files/1342/1342-0.txt",
        "https://www.gutenberg.org/files/1661/1661-0.txt"
    );

    private record ParsedArgs(Map<String, String> options, Map<String, List<String>> lists, List<String> positional) {}
    private record Checksum(String algorithm, String expected) {}
    private record DownloadResult(
        String url,
        String finalUrl,
        String file,
        long size,
        double timeSec,
        double avgSpeed,
        String checksum
    ) {}
    private record ResponseMeta(HttpResponse<InputStream> response, String finalUrl) {}

    private HttpFileDownloader() {}

    public static void main(String[] args) {
        try {
            ParsedArgs parsed = parseArgs(args);
            List<String> urls = new ArrayList<>(parsed.positional);
            if (parsed.options.containsKey("list")) {
                Path listPath = Paths.get(parsed.options.get("list")).toAbsolutePath();
                try (BufferedReader br = Files.newBufferedReader(listPath, StandardCharsets.UTF_8)) {
                    String line;
                    while ((line = br.readLine()) != null) {
                        line = line.trim();
                        if (!line.isEmpty()) urls.add(line);
                    }
                }
            }
            if (urls.isEmpty()) urls.addAll(SAMPLE_URLS);

            Checksum checksum = parseChecksum(parsed.options.get("checksum"));
            List<DownloadResult> results = new ArrayList<>();

            boolean parallel = parsed.options.containsKey("parallel");
            int threads = Math.max(1, Integer.parseInt(parsed.options.getOrDefault("threads", "4")));
            if (parallel && urls.size() > 1) {
                ExecutorService pool = Executors.newFixedThreadPool(threads);
                try {
                    List<Future<DownloadResult>> futures = new ArrayList<>();
                    for (int i = 0; i < urls.size(); i++) {
                        final int idx = i;
                        final Checksum ck = (idx == 0 ? checksum : null);
                        futures.add(pool.submit(new DownloadTask(urls.get(i), idx, urls.size(), parsed, ck)));
                    }
                    for (Future<DownloadResult> f : futures) {
                        try {
                            results.add(f.get());
                        } catch (Exception e) {
                            results.add(new DownloadResult("n/a", "n/a", "n/a", 0, 0, 0, "failed: " + e.getMessage()));
                        }
                    }
                } finally {
                    pool.shutdown();
                }
            } else {
                for (int i = 0; i < urls.size(); i++) {
                    Checksum ck = (i == 0 ? checksum : null);
                    results.add(new DownloadTask(urls.get(i), i, urls.size(), parsed, ck).call());
                }
            }
            printSummary(results);
        } catch (Exception e) {
            System.err.println("Error: " + e.getMessage());
            System.exit(1);
        }
    }

    private static final class DownloadTask implements Callable<DownloadResult> {
        private final String url;
        private final int index;
        private final int total;
        private final ParsedArgs parsed;
        private final Checksum checksum;

        private DownloadTask(String url, int index, int total, ParsedArgs parsed, Checksum checksum) {
            this.url = url;
            this.index = index;
            this.total = total;
            this.parsed = parsed;
            this.checksum = checksum;
        }

        @Override
        public DownloadResult call() throws Exception {
            int timeoutSec = Integer.parseInt(parsed.options.getOrDefault("timeout", "30"));
            int retries = Integer.parseInt(parsed.options.getOrDefault("retries", "3"));
            boolean resume = parsed.options.containsKey("resume");
            Path outDir = Paths.get(parsed.options.getOrDefault("output", ".")).toAbsolutePath();
            Files.createDirectories(outDir);

            String guessed = filenameFromUrl(url);
            Path guessPath = outDir.resolve(guessed);
            long existing = (resume && Files.exists(guessPath)) ? Files.size(guessPath) : 0;

            Map<String, String> headers = parseHeaderMap(parsed.lists.getOrDefault("header", List.of()));
            if (existing > 0) headers.put("Range", "bytes=" + existing + "-");

            int attempt = 0;
            while (attempt <= retries) {
                long startNanos = System.nanoTime();
                try {
                    HttpClient client = HttpClient.newBuilder()
                        .connectTimeout(Duration.ofSeconds(timeoutSec))
                        .followRedirects(HttpClient.Redirect.NEVER)
                        .build();
                    ResponseMeta meta = fetchWithRedirects(client, url, headers, timeoutSec, 10);
                    HttpResponse<InputStream> response = meta.response();
                    int code = response.statusCode();
                    if (code >= 400) throw new IOException("HTTP " + code);

                    String cd = response.headers().firstValue("content-disposition").orElse("");
                    String fileName = filenameFromContentDisposition(cd).orElse(filenameFromUrl(meta.finalUrl()));
                    Path outPath = outDir.resolve(fileName);

                    boolean resumed = resume && existing > 0 && code == 206 && outPath.equals(guessPath);
                    long downloaded = resumed ? existing : 0;
                    long contentLength = response.headers().firstValueAsLong("content-length").orElse(0L);
                    long totalBytes = contentLength > 0 ? contentLength + downloaded : 0;

                    try (InputStream in = response.body();
                         OutputStream out = Files.newOutputStream(outPath, resumed
                             ? java.nio.file.StandardOpenOption.CREATE
                             : java.nio.file.StandardOpenOption.CREATE,
                             resumed
                                 ? java.nio.file.StandardOpenOption.APPEND
                                 : java.nio.file.StandardOpenOption.TRUNCATE_EXISTING)) {
                        byte[] buf = new byte[64 * 1024];
                        int r;
                        while ((r = in.read(buf)) >= 0) {
                            out.write(buf, 0, r);
                            downloaded += r;
                            double elapsed = Math.max(0.001, (System.nanoTime() - startNanos) / 1_000_000_000.0);
                            double speed = (downloaded - (resumed ? existing : 0)) / elapsed;
                            double pct = totalBytes > 0 ? (downloaded * 100.0 / totalBytes) : 0.0;
                            double eta = (speed > 0 && totalBytes > 0) ? ((totalBytes - downloaded) / speed) : Double.POSITIVE_INFINITY;
                            System.out.print("\r[" + (index + 1) + "/" + total + "] " + fileName + " | " +
                                String.format(Locale.ROOT, "%.1f", pct) + "% | " + formatBytes(downloaded) + "/" +
                                (totalBytes > 0 ? formatBytes(totalBytes) : "?") + " | " + formatBytes(speed) + "/s | ETA " +
                                formatEta(eta) + "   ");
                        }
                    }
                    System.out.println();

                    long size = Files.size(outPath);
                    double sec = (System.nanoTime() - startNanos) / 1_000_000_000.0;
                    String checksumResult = "not requested";
                    if (checksum != null) {
                        String got = hashFile(outPath, checksum.algorithm());
                        checksumResult = got.equalsIgnoreCase(checksum.expected()) ? "ok" : "mismatch (" + got + ")";
                    }

                    return new DownloadResult(
                        url,
                        meta.finalUrl(),
                        outPath.getFileName().toString(),
                        size,
                        sec,
                        size / Math.max(0.001, sec),
                        checksumResult
                    );
                } catch (Exception e) {
                    attempt += 1;
                    if (attempt > retries) throw e;
                    Thread.sleep((long) Math.pow(2, attempt - 1) * 1000L);
                }
            }
            throw new IOException("failed after retries");
        }
    }

    private static ResponseMeta fetchWithRedirects(HttpClient client, String url, Map<String, String> headers, int timeoutSec, int maxHops)
        throws IOException, InterruptedException, URISyntaxException {
        String current = url;
        for (int hop = 0; hop <= maxHops; hop++) {
            HttpRequest.Builder b = HttpRequest.newBuilder()
                .uri(new URI(current))
                .timeout(Duration.ofSeconds(timeoutSec))
                .GET();
            for (Map.Entry<String, String> h : headers.entrySet()) b.header(h.getKey(), h.getValue());
            HttpResponse<InputStream> res = client.send(b.build(), HttpResponse.BodyHandlers.ofInputStream());
            int code = res.statusCode();
            if (code >= 300 && code < 400) {
                Optional<String> loc = res.headers().firstValue("location");
                if (loc.isEmpty()) return new ResponseMeta(res, current);
                URI next = new URI(current).resolve(loc.get());
                current = next.toString();
                continue;
            }
            return new ResponseMeta(res, current);
        }
        throw new IOException("Redirect limit exceeded (10)");
    }

    private static ParsedArgs parseArgs(String[] args) {
        Map<String, String> options = new LinkedHashMap<>();
        Map<String, List<String>> lists = new HashMap<>();
        List<String> positional = new ArrayList<>();
        for (int i = 0; i < args.length; i++) {
            String token = args[i];
            if (token.startsWith("--")) {
                String key = token.substring(2);
                String value = "true";
                if (i + 1 < args.length && !args[i + 1].startsWith("--")) value = args[++i];
                if (key.equals("header")) {
                    lists.computeIfAbsent("header", k -> new ArrayList<>()).add(value);
                } else options.put(key, value);
            } else positional.add(token);
        }
        return new ParsedArgs(options, lists, positional);
    }

    private static Map<String, String> parseHeaderMap(List<String> values) {
        Map<String, String> out = new LinkedHashMap<>();
        for (String h : values) {
            int idx = h.indexOf(':');
            if (idx <= 0) continue;
            String k = h.substring(0, idx).trim();
            String v = h.substring(idx + 1).trim();
            if (!k.isEmpty()) out.put(k, v);
        }
        return out;
    }

    private static Checksum parseChecksum(String raw) {
        if (raw == null || raw.isBlank()) return null;
        int idx = raw.indexOf(':');
        if (idx <= 0) throw new IllegalArgumentException("Invalid --checksum. Use algorithm:hash");
        return new Checksum(raw.substring(0, idx).toLowerCase(Locale.ROOT), raw.substring(idx + 1).toLowerCase(Locale.ROOT));
    }

    private static String filenameFromUrl(String url) {
        try {
            URI u = URI.create(url);
            String p = u.getPath();
            if (p == null || p.isBlank() || p.endsWith("/")) return "download.bin";
            String name = Paths.get(p).getFileName().toString();
            return URLDecoder.decode(name, StandardCharsets.UTF_8);
        } catch (Exception ignored) {
            return "download.bin";
        }
    }

    private static Optional<String> filenameFromContentDisposition(String cd) {
        if (cd == null || cd.isBlank()) return Optional.empty();
        String lower = cd.toLowerCase(Locale.ROOT);
        int idx = lower.indexOf("filename=");
        if (idx < 0) return Optional.empty();
        String v = cd.substring(idx + "filename=".length()).trim().replace("\"", "");
        if (v.isBlank()) return Optional.empty();
        return Optional.of(URLDecoder.decode(v, StandardCharsets.UTF_8));
    }

    private static String hashFile(Path path, String algorithm) throws Exception {
        MessageDigest md = MessageDigest.getInstance(algorithm.toUpperCase(Locale.ROOT));
        try (InputStream in = Files.newInputStream(path)) {
            byte[] buf = new byte[1024 * 1024];
            int r;
            while ((r = in.read(buf)) >= 0) md.update(buf, 0, r);
        }
        byte[] out = md.digest();
        StringBuilder sb = new StringBuilder();
        for (byte b : out) sb.append(String.format("%02x", b));
        return sb.toString();
    }

    private static String formatBytes(double n) {
        if (!Double.isFinite(n) || n < 0) return "?";
        String[] units = {"B", "KB", "MB", "GB", "TB"};
        int i = 0;
        while (n >= 1024 && i < units.length - 1) {
            n /= 1024.0;
            i++;
        }
        return (i == 0 ? String.format(Locale.ROOT, "%.0f", n) : String.format(Locale.ROOT, "%.2f", n)) + " " + units[i];
    }

    private static String formatEta(double sec) {
        if (!Double.isFinite(sec) || sec < 0) return "--:--";
        int s = (int) Math.floor(sec % 60);
        int m = (int) Math.floor((sec / 60) % 60);
        int h = (int) Math.floor(sec / 3600);
        if (h > 0) return String.format(Locale.ROOT, "%d:%02d:%02d", h, m, s);
        return String.format(Locale.ROOT, "%02d:%02d", m, s);
    }

    private static void printSummary(List<DownloadResult> rows) {
        rows.sort(Comparator.comparing(r -> r.file));
        System.out.println("\nDownload Summary");
        System.out.println("File | Size | Time | Avg Speed | Checksum | Final URL");
        System.out.println("---- | ---- | ---- | --------- | -------- | ---------");
        for (DownloadResult r : rows) {
            System.out.printf(
                "%s | %s | %.2fs | %s/s | %s | %s%n",
                r.file,
                formatBytes(r.size),
                r.timeSec,
                formatBytes(r.avgSpeed),
                r.checksum,
                r.finalUrl
            );
        }
    }
}