Structured Log Processor (java, written by Claude Code)

Claude Code wrote this java project from the task below. It does not run on a clean Ubuntu 22.04 machine as written.

Task given to the agent:

TASK: Structured Log Processor

Write a program that parses, queries, transforms, and aggregates structured log data in JSON Lines format, supporting filtering, field extraction, statistical aggregation, and output formatting.

FUNCTIONAL REQUIREMENTS:
- Accept a log file path as a command-line argument (JSON Lines format: one JSON object per line)
- Support filtering log entries via --where flag with field comparisons (e.g., --where "level==ERROR" or --where "response_time>500" or --where "status!=200")
- Support multiple filters combined with AND logic; support OR logic via --or flag
- Support field selection via --fields flag (comma-separated list of field names to include in output)
- Support aggregation operations via --group-by and --aggregate flags: count, sum, avg, min, max, and percentile(N) grouped by a specified field (e.g., --group-by status --aggregate "count,avg:response_time")
- Support time-based aggregation: group by time windows (--time-window flag: 1m, 5m, 1h, 1d) on a specified timestamp field (--time-field flag)
- Support sorting via --sort flag (field name with optional :asc or :desc suffix)
- Support limiting output via --limit flag and skipping via --offset flag
- Support output in multiple formats via --format flag: json (default), csv, table (formatted console table), and jsonl (JSON Lines)
- Compute and display summary statistics for numeric fields: count, min, max, mean, median, p95, p99
- Support extracting unique values of a field via --distinct flag
- Print results to console by default
- Save results to a file via --output flag
- If no input file is given, generate a sample web server access log with 1000 entries containing fields (timestamp, method, path, status, response_time, user_agent, ip), then demonstrate: filtering ERROR entries, computing average response time grouped by HTTP method, finding the top 10 slowest requests, and computing hourly request counts
- Handle errors: malformed JSON lines (skip with warning and count), missing fields in filter expressions, type mismatches in comparisons, and very large files

Create a complete Java project for a clean Ubuntu 22.04 machine with only JDK 17+ installed. Include:
- Source code
- pom.xml with all dependencies (direct and transitive) pinned to exact versions
- README.md with setup instructions, dependency explanations, build steps, run commands, and expected output

[]

import com.google.gson.*;
import com.beust.jcommander.*;

import java.io.*;
import java.nio.file.*;
import java.time.*;
import java.time.format.*;
import java.util.*;
import java.util.stream.*;

/**
 * Structured Log Processor (Trial 2)
 * Parses/queries/aggregates JSON Lines logs with filtering, field selection,
 * and time-based aggregation using Gson and JCommander.
 */
public class LogProcessor {

    private static final Gson gson = new GsonBuilder().setPrettyPrinting().create();
    private static final Gson compactGson = new Gson();

    public static void main(String[] args) {
        MainArgs mainArgs = new MainArgs();
        QueryArgs queryArgs = new QueryArgs();
        CountByArgs countByArgs = new CountByArgs();
        StatsArgs statsArgs = new StatsArgs();
        TimeSeriesArgs tsArgs = new TimeSeriesArgs();
        TopArgs topArgs = new TopArgs();

        JCommander commander = JCommander.newBuilder()
            .addObject(mainArgs)
            .addCommand("query", queryArgs)
            .addCommand("count-by", countByArgs)
            .addCommand("stats", statsArgs)
            .addCommand("timeseries", tsArgs)
            .addCommand("top", topArgs)
            .build();
        commander.setProgramName("logproc");

        try {
            commander.parse(args);
        } catch (ParameterException e) {
            System.err.println(e.getMessage());
            commander.usage();
            System.exit(1);
        }

        String cmd = commander.getParsedCommand();
        if (cmd == null) {
            commander.usage();
            return;
        }

        try {
            switch (cmd) {
                case "query": executeQuery(queryArgs); break;
                case "count-by": executeCountBy(countByArgs); break;
                case "stats": executeStats(statsArgs); break;
                case "timeseries": executeTimeSeries(tsArgs); break;
                case "top": executeTop(topArgs); break;
            }
        } catch (IOException e) {
            System.err.println("Error: " + e.getMessage());
            System.exit(1);
        }
    }

    static JsonElement getNestedField(JsonObject obj, String path) {
        String[] parts = path.split("\\.");
        JsonElement current = obj;
        for (String part : parts) {
            if (current == null || !current.isJsonObject()) return null;
            JsonObject o = current.getAsJsonObject();
            if (!o.has(part)) return null;
            current = o.get(part);
        }
        return current;
    }

    static String elementToString(JsonElement el) {
        if (el == null || el.isJsonNull()) return "<null>";
        if (el.isJsonPrimitive()) return el.getAsString();
        return el.toString();
    }

    static boolean matchesFilter(JsonObject record, String filterExpr) {
        String[][] opDefs = {
            {"!=", "ne"}, {">=", "gte"}, {"<=", "lte"},
            {"==", "eq"}, {">", "gt"}, {"<", "lt"},
            {"contains", "contains"}, {"startswith", "startswith"}, {"endswith", "endswith"}
        };
        for (String[] opDef : opDefs) {
            String sym = opDef[0];
            int idx = filterExpr.indexOf(sym);
            if (idx > 0) {
                String field = filterExpr.substring(0, idx).trim();
                String value = filterExpr.substring(idx + sym.length()).trim();
                JsonElement fieldEl = getNestedField(record, field);
                if (fieldEl == null || fieldEl.isJsonNull()) return false;
                String fieldStr = elementToString(fieldEl);

                switch (opDef[1]) {
                    case "eq": return fieldStr.equals(value);
                    case "ne": return !fieldStr.equals(value);
                    case "contains": return fieldStr.toLowerCase().contains(value.toLowerCase());
                    case "startswith": return fieldStr.startsWith(value);
                    case "endswith": return fieldStr.endsWith(value);
                    case "gt": case "gte": case "lt": case "lte":
                        try {
                            double a = Double.parseDouble(fieldStr);
                            double b = Double.parseDouble(value);
                            switch (opDef[1]) {
                                case "gt": return a > b;
                                case "gte": return a >= b;
                                case "lt": return a < b;
                                case "lte": return a <= b;
                            }
                        } catch (NumberFormatException e) {
                            int cmp = fieldStr.compareTo(value);
                            switch (opDef[1]) {
                                case "gt": return cmp > 0;
                                case "gte": return cmp >= 0;
                                case "lt": return cmp < 0;
                                case "lte": return cmp <= 0;
                            }
                        }
                }
                break;
            }
        }
        return true;
    }

    static List<JsonObject> readAndFilter(String logfile, List<String> filters) throws IOException {
        List<JsonObject> results = new ArrayList<>();
        JsonParser parser = new JsonParser();
        try (BufferedReader reader = Files.newBufferedReader(Path.of(logfile))) {
            String line;
            while ((line = reader.readLine()) != null) {
                line = line.trim();
                if (line.isEmpty()) continue;
                try {
                    JsonObject obj = parser.parse(line).getAsJsonObject();
                    boolean passes = filters.stream().allMatch(f -> matchesFilter(obj, f));
                    if (passes) results.add(obj);
                } catch (JsonSyntaxException ignored) {}
            }
        }
        return results;
    }

    static JsonObject projectFields(JsonObject record, List<String> fields) {
        JsonObject result = new JsonObject();
        for (String field : fields) {
            JsonElement val = getNestedField(record, field);
            if (val != null) result.add(field, val);
        }
        return result;
    }

    static void executeQuery(QueryArgs args) throws IOException {
        List<JsonObject> records = readAndFilter(args.logfile, args.filters);
        List<String> fieldList = args.fields != null ?
            Arrays.stream(args.fields.split(",")).map(String::trim).collect(Collectors.toList()) : null;

        int count = 0;
        for (JsonObject rec : records) {
            if (args.limit > 0 && count >= args.limit) break;
            JsonObject output = (fieldList != null) ? projectFields(rec, fieldList) : rec;
            System.out.println(args.pretty ? gson.toJson(output) : compactGson.toJson(output));
            count++;
        }
        System.err.printf("%n--- %d/%d records matched ---%n", count, records.size());
    }

    static void executeCountBy(CountByArgs args) throws IOException {
        List<JsonObject> records = readAndFilter(args.logfile, args.filters);
        Map<String, Long> counts = new LinkedHashMap<>();
        for (JsonObject rec : records) {
            JsonElement val = getNestedField(rec, args.field);
            String key = elementToString(val);
            counts.merge(key, 1L, Long::sum);
        }
        counts.entrySet().stream()
            .sorted(Map.Entry.<String, Long>comparingByValue().reversed())
            .forEach(e -> System.out.printf("%s: %d%n", e.getKey(), e.getValue()));
    }

    static void executeStats(StatsArgs args) throws IOException {
        List<JsonObject> records = readAndFilter(args.logfile, args.filters);
        DoubleSummaryStatistics stats = records.stream()
            .map(r -> getNestedField(r, args.field))
            .filter(Objects::nonNull)
            .filter(e -> e.isJsonPrimitive() && e.getAsJsonPrimitive().isNumber())
            .mapToDouble(JsonElement::getAsDouble)
            .summaryStatistics();
        System.out.printf("count: %d%nmin: %.4f%nmax: %.4f%navg: %.4f%nsum: %.4f%n",
            stats.getCount(), stats.getMin(), stats.getMax(), stats.getAverage(), stats.getSum());
    }

    static void executeTimeSeries(TimeSeriesArgs args) throws IOException {
        List<JsonObject> records = readAndFilter(args.logfile, args.filters);
        Map<String, Long> buckets = new TreeMap<>();
        for (JsonObject rec : records) {
            JsonElement val = getNestedField(rec, args.timeField);
            if (val == null) continue;
            String timeStr = elementToString(val);
            LocalDateTime dt = parseDateTime(timeStr);
            if (dt == null) continue;
            String bucket = formatBucket(dt, args.interval);
            buckets.merge(bucket, 1L, Long::sum);
        }
        buckets.forEach((k, v) -> System.out.printf("%s: %d%n", k, v));
    }

    static void executeTop(TopArgs args) throws IOException {
        List<JsonObject> records = readAndFilter(args.logfile, new ArrayList<>());
        Map<String, Long> counts = new LinkedHashMap<>();
        for (JsonObject rec : records) {
            JsonElement val = getNestedField(rec, args.field);
            if (val != null && !val.isJsonNull()) {
                counts.merge(elementToString(val), 1L, Long::sum);
            }
        }
        counts.entrySet().stream()
            .sorted(Map.Entry.<String, Long>comparingByValue().reversed())
            .limit(args.top)
            .forEach(e -> System.out.printf("%s: %d%n", e.getKey(), e.getValue()));
    }

    static LocalDateTime parseDateTime(String s) {
        DateTimeFormatter[] fmts = {
            DateTimeFormatter.ISO_DATE_TIME,
            DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
        };
        for (DateTimeFormatter fmt : fmts) {
            try { return LocalDateTime.parse(s, fmt); } catch (Exception ignored) {}
        }
        try {
            return Instant.parse(s).atZone(ZoneId.systemDefault()).toLocalDateTime();
        } catch (Exception ignored) {}
        return null;
    }

    static String formatBucket(LocalDateTime dt, String interval) {
        switch (interval) {
            case "minute": return dt.format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm"));
            case "day": return dt.format(DateTimeFormatter.ofPattern("yyyy-MM-dd"));
            case "month": return dt.format(DateTimeFormatter.ofPattern("yyyy-MM"));
            default: return dt.format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:00"));
        }
    }

    @Parameters static class MainArgs {}

    @Parameters(commandDescription = "Query log records with filtering and field selection")
    static class QueryArgs {
        @Parameter(description = "logfile") List<String> main = new ArrayList<>();
        @Parameter(names = {"-f", "--filter"}, description = "Filter expressions") List<String> filters = new ArrayList<>();
        @Parameter(names = {"-s", "--fields"}, description = "Comma-separated field list") String fields;
        @Parameter(names = {"-l", "--limit"}, description = "Max records") int limit = 0;
        @Parameter(names = {"--pretty"}, description = "Pretty-print") boolean pretty = false;
        String getLogfile() { return main.isEmpty() ? null : main.get(0); }
        String logfile;
        @Parameter(names = "--logfile", hidden = true) void setLogfile(String lf) { this.logfile = lf; }
    }

    @Parameters(commandDescription = "Count records grouped by a field")
    static class CountByArgs {
        @Parameter(description = "logfile field") List<String> main = new ArrayList<>();
        @Parameter(names = {"-f", "--filter"}) List<String> filters = new ArrayList<>();
        String getLogfile() { return main.size() > 0 ? main.get(0) : null; }
        String logfile; String field;
    }

    @Parameters(commandDescription = "Compute numeric statistics for a field")
    static class StatsArgs {
        @Parameter(description = "logfile field") List<String> main = new ArrayList<>();
        @Parameter(names = {"-f", "--filter"}) List<String> filters = new ArrayList<>();
        String logfile; String field;
    }

    @Parameters(commandDescription = "Aggregate records by time intervals")
    static class TimeSeriesArgs {
        @Parameter(description = "logfile time_field") List<String> main = new ArrayList<>();
        @Parameter(names = {"-f", "--filter"}) List<String> filters = new ArrayList<>();
        @Parameter(names = {"-i", "--interval"}) String interval = "hour";
        String logfile; String timeField;
    }

    @Parameters(commandDescription = "Show top N most frequent values")
    static class TopArgs {
        @Parameter(description = "logfile field") List<String> main = new ArrayList<>();
        @Parameter(names = {"-n", "--top"}) int top = 10;
        String logfile; String field;
    }
}

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.logprocessor</groupId>
    <artifactId>structured-log-processor</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Structured Log Processor</name>
    <description>Parses/queries/aggregates JSON Lines logs using Gson and JCommander</description>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.10.1</version>
        </dependency>
        <dependency>
            <groupId>com.beust</groupId>
            <artifactId>jcommander</artifactId>
            <version>1.82</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.3.0</version>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>LogProcessor</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

# Structured Log Processor (Java - Trial 2)

## Description
Parses, queries, and aggregates JSON Lines log files with support for filtering,
field selection, and time-based aggregation. Built with Gson for JSON parsing
and JCommander for CLI argument handling.

## Dependencies
- **gson**: Google's JSON library for Java serialization/deserialization
- **jcommander**: Annotation-based CLI parameter parsing framework

## Usage

### Build
```bash
mvn clean package
```

### Query logs with filters
```bash
java -jar target/structured-log-processor-1.0-SNAPSHOT.jar query access.jsonl -f "level==ERROR" -s "timestamp,message" --pretty
```

### Count by field
```bash
java -jar target/structured-log-processor-1.0-SNAPSHOT.jar count-by access.jsonl level
```

### Numeric statistics
```bash
java -jar target/structured-log-processor-1.0-SNAPSHOT.jar stats access.jsonl response_time
```

### Time series aggregation
```bash
java -jar target/structured-log-processor-1.0-SNAPSHOT.jar timeseries access.jsonl timestamp -i hour
```

## Input Format
Expects JSON Lines format (one JSON object per line):
```json
{"timestamp": "2024-01-15T10:30:00Z", "level": "INFO", "message": "Request processed", "response_time": 42}
```