Structured Log Processor (java, written by Codex)

Codex wrote this java project from the task below. It installed and ran on a clean Ubuntu 22.04 machine as written.

Task given to the agent:

TASK: Structured Log Processor

Write a program that parses, queries, transforms, and aggregates structured log data in JSON Lines format, supporting filtering, field extraction, statistical aggregation, and output formatting.

FUNCTIONAL REQUIREMENTS:
- Accept a log file path as a command-line argument (JSON Lines format: one JSON object per line)
- Support filtering log entries via --where flag with field comparisons (e.g., --where "level==ERROR" or --where "response_time>500" or --where "status!=200")
- Support multiple filters combined with AND logic; support OR logic via --or flag
- Support field selection via --fields flag (comma-separated list of field names to include in output)
- Support aggregation operations via --group-by and --aggregate flags: count, sum, avg, min, max, and percentile(N) grouped by a specified field (e.g., --group-by status --aggregate "count,avg:response_time")
- Support time-based aggregation: group by time windows (--time-window flag: 1m, 5m, 1h, 1d) on a specified timestamp field (--time-field flag)
- Support sorting via --sort flag (field name with optional :asc or :desc suffix)
- Support limiting output via --limit flag and skipping via --offset flag
- Support output in multiple formats via --format flag: json (default), csv, table (formatted console table), and jsonl (JSON Lines)
- Compute and display summary statistics for numeric fields: count, min, max, mean, median, p95, p99
- Support extracting unique values of a field via --distinct flag
- Print results to console by default
- Save results to a file via --output flag
- If no input file is given, generate a sample web server access log with 1000 entries containing fields (timestamp, method, path, status, response_time, user_agent, ip), then demonstrate: filtering ERROR entries, computing average response time grouped by HTTP method, finding the top 10 slowest requests, and computing hourly request counts
- Handle errors: malformed JSON lines (skip with warning and count), missing fields in filter expressions, type mismatches in comparisons, and very large files

Create a complete Java project for a clean Ubuntu 22.04 machine with only JDK 17+ installed. Include:
- Source code
- pom.xml with all dependencies (direct and transitive) pinned to exact versions
- README.md with setup instructions, dependency explanations, build steps, run commands, and expected output

[]

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.logprocessor</groupId>
    <artifactId>structured-log-processor</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Structured Log Processor</name>
    <description>Parses/queries/aggregates JSON Lines logs with filtering, field selection, and time-based aggregation</description>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.16.1</version>
        </dependency>
        <dependency>
            <groupId>info.picocli</groupId>
            <artifactId>picocli</artifactId>
            <version>4.7.5</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.3.0</version>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>LogProcessor</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

# Structured Log Processor (Java - Trial 1)

## Description
Parses, queries, and aggregates JSON Lines log files with support for filtering,
field selection, and time-based aggregation. Built with Jackson for JSON parsing
and Picocli for CLI argument handling.

## Dependencies
- **jackson-databind**: High-performance JSON parsing and tree model traversal
- **picocli**: Annotation-based CLI framework with subcommand support

## Usage

### Build
```bash
mvn clean package
```

### Query logs with filters
```bash
java -jar target/structured-log-processor-1.0-SNAPSHOT.jar query access.jsonl -f "level==ERROR" -s "timestamp,message" --pretty
```

### Count by field
```bash
java -jar target/structured-log-processor-1.0-SNAPSHOT.jar count-by access.jsonl level
```

### Numeric statistics
```bash
java -jar target/structured-log-processor-1.0-SNAPSHOT.jar stats access.jsonl response_time
```

### Time series aggregation
```bash
java -jar target/structured-log-processor-1.0-SNAPSHOT.jar timeseries access.jsonl timestamp -i hour
```

### Top N values
```bash
java -jar target/structured-log-processor-1.0-SNAPSHOT.jar top access.jsonl status_code -n 5
```

## Input Format
Expects JSON Lines format (one JSON object per line):
```json
{"timestamp": "2024-01-15T10:30:00Z", "level": "INFO", "message": "Request processed", "response_time": 42}
```

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
import picocli.CommandLine.Parameters;

import java.io.*;
import java.nio.file.*;
import java.time.*;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeParseException;
import java.util.*;
import java.util.concurrent.Callable;
import java.util.stream.*;

/**
 * Structured Log Processor
 * Parses/queries/aggregates JSON Lines logs with filtering, field selection,
 * and time-based aggregation using Jackson and Picocli.
 */
@Command(name = "logproc", mixinStandardHelpOptions = true, version = "1.0",
        description = "Structured Log Processor - Query and analyze JSON Lines logs.",
        subcommands = {
            LogProcessor.QueryCommand.class,
            LogProcessor.CountByCommand.class,
            LogProcessor.StatsCommand.class,
            LogProcessor.TimeSeriesCommand.class,
            LogProcessor.TopCommand.class
        })
public class LogProcessor implements Callable<Integer> {

    private static final ObjectMapper mapper = new ObjectMapper();

    public static void main(String[] args) {
        int exitCode = new CommandLine(new LogProcessor()).execute(args);
        System.exit(exitCode);
    }

    @Override
    public Integer call() {
        CommandLine.usage(this, System.out);
        return 0;
    }

    static JsonNode getNestedField(JsonNode node, String fieldPath) {
        String[] parts = fieldPath.split("\\.");
        JsonNode current = node;
        for (String part : parts) {
            if (current == null || !current.has(part)) return null;
            current = current.get(part);
        }
        return current;
    }

    static boolean matchesFilter(JsonNode record, String filterExpr) {
        String[] operators = {"!=", ">=", "<=", "==", ">", "<", "contains", "startswith", "endswith"};
        for (String op : operators) {
            int idx = filterExpr.indexOf(op);
            if (idx > 0) {
                String field = filterExpr.substring(0, idx).trim();
                String value = filterExpr.substring(idx + op.length()).trim();
                JsonNode fieldVal = getNestedField(record, field);
                if (fieldVal == null) return false;
                String fieldStr = fieldVal.isTextual() ? fieldVal.asText() : fieldVal.toString();

                switch (op) {
                    case "==": return fieldStr.equals(value);
                    case "!=": return !fieldStr.equals(value);
                    case "contains": return fieldStr.toLowerCase().contains(value.toLowerCase());
                    case "startswith": return fieldStr.startsWith(value);
                    case "endswith": return fieldStr.endsWith(value);
                    case ">": case ">=": case "<": case "<=":
                        try {
                            double a = Double.parseDouble(fieldStr);
                            double b = Double.parseDouble(value);
                            switch (op) {
                                case ">": return a > b;
                                case ">=": return a >= b;
                                case "<": return a < b;
                                case "<=": return a <= b;
                            }
                        } catch (NumberFormatException e) {
                            return fieldStr.compareTo(value) > 0 && op.contains(">");
                        }
                }
                break;
            }
        }
        return true;
    }

    static List<JsonNode> readAndFilter(String logfile, List<String> filters) throws IOException {
        List<JsonNode> results = new ArrayList<>();
        try (BufferedReader reader = Files.newBufferedReader(Path.of(logfile))) {
            String line;
            while ((line = reader.readLine()) != null) {
                line = line.trim();
                if (line.isEmpty()) continue;
                try {
                    JsonNode node = mapper.readTree(line);
                    boolean match = true;
                    for (String f : filters) {
                        if (!matchesFilter(node, f)) { match = false; break; }
                    }
                    if (match) results.add(node);
                } catch (Exception ignored) {}
            }
        }
        return results;
    }

    static ObjectNode selectFields(JsonNode record, List<String> fields) {
        ObjectNode result = mapper.createObjectNode();
        for (String field : fields) {
            JsonNode val = getNestedField(record, field);
            if (val != null) result.set(field, val);
        }
        return result;
    }

    @Command(name = "query", description = "Query log records with filtering and field selection.")
    static class QueryCommand implements Callable<Integer> {
        @Parameters(index = "0", description = "Path to JSON Lines log file")
        String logfile;
        @Option(names = {"-f", "--filter"}, description = "Filter expressions")
        List<String> filters = new ArrayList<>();
        @Option(names = {"-s", "--fields"}, description = "Comma-separated field list")
        String fields;
        @Option(names = {"-l", "--limit"}, description = "Max records to return", defaultValue = "0")
        int limit;
        @Option(names = {"--pretty"}, description = "Pretty-print output")
        boolean pretty;

        @Override
        public Integer call() throws Exception {
            List<JsonNode> records = readAndFilter(logfile, filters);
            List<String> fieldList = fields != null ?
                Arrays.stream(fields.split(",")).map(String::trim).collect(Collectors.toList()) : null;

            int count = 0;
            for (JsonNode rec : records) {
                if (limit > 0 && count >= limit) break;
                JsonNode output = (fieldList != null) ? selectFields(rec, fieldList) : rec;
                String json = pretty ? mapper.writerWithDefaultPrettyPrinter().writeValueAsString(output)
                                     : mapper.writeValueAsString(output);
                System.out.println(json);
                count++;
            }
            System.err.printf("%n--- %d/%d records matched ---%n", count, records.size());
            return 0;
        }
    }

    @Command(name = "count-by", description = "Count records grouped by a field.")
    static class CountByCommand implements Callable<Integer> {
        @Parameters(index = "0", description = "Path to JSON Lines log file")
        String logfile;
        @Parameters(index = "1", description = "Field to group by")
        String field;
        @Option(names = {"-f", "--filter"}, description = "Filter expressions")
        List<String> filters = new ArrayList<>();

        @Override
        public Integer call() throws Exception {
            List<JsonNode> records = readAndFilter(logfile, filters);
            Map<String, Long> counts = new LinkedHashMap<>();
            for (JsonNode rec : records) {
                JsonNode val = getNestedField(rec, field);
                String key = (val != null) ? (val.isTextual() ? val.asText() : val.toString()) : "<null>";
                counts.merge(key, 1L, Long::sum);
            }
            counts.entrySet().stream()
                .sorted(Map.Entry.<String, Long>comparingByValue().reversed())
                .forEach(e -> System.out.printf("%s: %d%n", e.getKey(), e.getValue()));
            return 0;
        }
    }

    @Command(name = "stats", description = "Compute numeric statistics for a field.")
    static class StatsCommand implements Callable<Integer> {
        @Parameters(index = "0", description = "Path to JSON Lines log file")
        String logfile;
        @Parameters(index = "1", description = "Numeric field to analyze")
        String field;
        @Option(names = {"-f", "--filter"}, description = "Filter expressions")
        List<String> filters = new ArrayList<>();

        @Override
        public Integer call() throws Exception {
            List<JsonNode> records = readAndFilter(logfile, filters);
            DoubleSummaryStatistics stats = records.stream()
                .map(r -> getNestedField(r, field))
                .filter(Objects::nonNull)
                .filter(JsonNode::isNumber)
                .mapToDouble(JsonNode::asDouble)
                .summaryStatistics();
            System.out.printf("count: %d%n", stats.getCount());
            System.out.printf("min: %.4f%n", stats.getMin());
            System.out.printf("max: %.4f%n", stats.getMax());
            System.out.printf("avg: %.4f%n", stats.getAverage());
            System.out.printf("sum: %.4f%n", stats.getSum());
            return 0;
        }
    }

    @Command(name = "timeseries", description = "Aggregate records by time intervals.")
    static class TimeSeriesCommand implements Callable<Integer> {
        @Parameters(index = "0", description = "Path to JSON Lines log file")
        String logfile;
        @Parameters(index = "1", description = "Time field name")
        String timeField;
        @Option(names = {"-f", "--filter"}, description = "Filter expressions")
        List<String> filters = new ArrayList<>();
        @Option(names = {"-i", "--interval"}, description = "Time interval (minute|hour|day|month)", defaultValue = "hour")
        String interval;

        @Override
        public Integer call() throws Exception {
            List<JsonNode> records = readAndFilter(logfile, filters);
            Map<String, Long> buckets = new TreeMap<>();
            DateTimeFormatter[] parsers = {
                DateTimeFormatter.ISO_DATE_TIME, DateTimeFormatter.ISO_INSTANT,
                DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
            };
            for (JsonNode rec : records) {
                JsonNode val = getNestedField(rec, timeField);
                if (val == null) continue;
                String timeStr = val.isTextual() ? val.asText() : val.toString();
                LocalDateTime dt = null;
                for (DateTimeFormatter fmt : parsers) {
                    try {
                        dt = LocalDateTime.parse(timeStr, fmt);
                        break;
                    } catch (DateTimeParseException e) {
                        try {
                            dt = Instant.parse(timeStr).atZone(ZoneId.systemDefault()).toLocalDateTime();
                            break;
                        } catch (Exception ignored) {}
                    }
                }
                if (dt == null) continue;
                String bucket;
                switch (interval) {
                    case "minute": bucket = dt.format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm")); break;
                    case "day": bucket = dt.format(DateTimeFormatter.ofPattern("yyyy-MM-dd")); break;
                    case "month": bucket = dt.format(DateTimeFormatter.ofPattern("yyyy-MM")); break;
                    default: bucket = dt.format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:00")); break;
                }
                buckets.merge(bucket, 1L, Long::sum);
            }
            buckets.forEach((k, v) -> System.out.printf("%s: %d%n", k, v));
            return 0;
        }
    }

    @Command(name = "top", description = "Show top N most frequent values for a field.")
    static class TopCommand implements Callable<Integer> {
        @Parameters(index = "0", description = "Path to JSON Lines log file")
        String logfile;
        @Parameters(index = "1", description = "Field to analyze")
        String field;
        @Option(names = {"-n", "--top"}, description = "Number of top values", defaultValue = "10")
        int top;

        @Override
        public Integer call() throws Exception {
            List<JsonNode> records = readAndFilter(logfile, new ArrayList<>());
            Map<String, Long> counts = new LinkedHashMap<>();
            for (JsonNode rec : records) {
                JsonNode val = getNestedField(rec, field);
                if (val != null) {
                    String key = val.isTextual() ? val.asText() : val.toString();
                    counts.merge(key, 1L, Long::sum);
                }
            }
            counts.entrySet().stream()
                .sorted(Map.Entry.<String, Long>comparingByValue().reversed())
                .limit(top)
                .forEach(e -> System.out.printf("%s: %d%n", e.getKey(), e.getValue()));
            return 0;
        }
    }
}