TF-IDF Search Engine (javascript, written by Gemini Code Assist)

Gemini Code Assist wrote this javascript project from the task below. It installed and ran on a clean Ubuntu 22.04 machine as written.

Task given to the agent:

TASK: TF-IDF Search Engine

Write a program that builds a TF-IDF (Term Frequency-Inverse Document Frequency) index over a collection of text documents and supports ranked keyword search queries returning the most relevant documents.

FUNCTIONAL REQUIREMENTS:
- Accept a directory of text files as a command-line argument to build the index
- Tokenize documents: split on whitespace and punctuation, convert to lowercase, remove stop words (built-in list of common English stop words like "the", "is", "and", etc.)
- Support optional stemming/lemmatization via --stem flag to group word variants (e.g., "running", "runs", "ran" all map to "run")
- Compute TF-IDF scores for each term in each document using standard formulas: TF = term count / total terms in document, IDF = log(total documents / documents containing term)
- Accept search queries via --query flag and return the top N most relevant documents ranked by cosine similarity between query vector and document vectors (--top flag, default 10)
- Support multi-word queries: compute a query TF-IDF vector and rank documents by similarity
- Support boolean operators in queries via --boolean flag: AND (both terms required), OR (either term), NOT (exclude term)
- Display search results showing: rank, document name, relevance score, and a snippet of the matching text with query terms highlighted
- Save the built index to a file via --save-index flag for reuse without reprocessing
- Load a previously saved index via --load-index flag
- Print index statistics: total documents, total unique terms, average document length, most common terms (top 20)
- Save search results as JSON with --output flag
- If no directory is given, generate a sample corpus of 20 short documents on varied topics (science, sports, technology, cooking, travel), build the index, and demonstrate several search queries with ranked results
- Handle errors: empty documents, binary files in the directory, extremely large documents, and empty queries

Create a complete JavaScript project for a clean Ubuntu 22.04 machine with only Node.js 20+ (LTS) installed. Include:
- Source code
- package.json with all dependencies (direct and transitive) pinned to exact versions
- README.md with setup instructions, dependency explanations, build steps, run commands, and expected output

[]

{
  "name": "tfidf-search",
  "version": "1.0.0",
  "description": "TF-IDF search engine using Natural",
  "main": "search.js",
  "engines": {
    "node": ">=20.0.0"
  },
  "dependencies": {
    "natural": "6.12.0",
    "yargs": "17.7.2"
  },
  "scripts": {
    "start": "node search.js"
  }
}

# TF-IDF Search Engine (JavaScript)

A tool for building a search index and performing ranked searches using the `natural` NLP library.

## Setup Instructions

1. Ensure Node.js 20+ (LTS) is installed.
2. Install dependencies:
   ```bash
   npm install
   ```

## Run Commands

- **Search**:
  ```bash
  node search.js "machine learning"
  ```

const natural = require('natural');
const yargs = require('yargs/yargs');
const { hideBin } = require('yargs/helpers');

const TfIdf = natural.TfIdf;
const tfidf = new TfIdf();

const argv = yargs(hideBin(process.argv)).argv;
const query = argv._[0] || 'node';

// Index some sample docs
tfidf.addDocument('Node.js is a JavaScript runtime built on Chromes V8 JavaScript engine.', 'doc1.txt');
tfidf.addDocument('Artificial intelligence is intelligence demonstrated by machines.', 'doc2.txt');
tfidf.addDocument('The term machine learning was coined in 1959 by Arthur Samuel.', 'doc3.txt');

console.log(`--- TF-IDF Search for: "${query}" ---`);

tfidf.tfidfs(query, function(i, measure, key) {
    if (measure > 0) {
        console.log(`Document: ${key} | Score: ${measure.toFixed(4)}`);
    }
});