EnvGap
Where it comes from
EnvGap grew out of our earlier work on dependency conflicts and on the gap between the dependencies a project claims and the ones it really uses. It reuses the clean per-language environments, the Sciunit tracing tools and the dependency normalization code from that work.
Earlier experiments ran generic install commands on each project. That tells you whether an install finished, not whether the reported problem was fixed, so a task is admitted here only after we reproduce its failure and a known fix in a fresh container.
Kinds of failure
- Misspecification: a declared dependency is wrong or no longer resolves.
- Underspecification: the project needs something the manifest leaves out, such as a system library or a compiler flag.
- Conflict: two requirements cannot be installed together.
- Security: the registry audit flags a package, or the project itself reports a security problem.
A task can carry more than one label, and each label keeps the rule and the text that triggered it. Stochastic is not a label; it is a way of running tasks several times.
Later history
An agent should solve a task from what existed on the day of the failure. We read every attempt in the listed runs for commands that reached the project's later history. If the output showed the agent the fix that was eventually merged, the task is scored as unresolved, even if the environment worked.