HTML to Plain Text Extractor (python, written by Gemini Code Assist)

Gemini Code Assist wrote this python project from the task below. It installed and ran on a clean Ubuntu 22.04 machine as written.

Task given to the agent:

TASK: HTML to Plain Text Extractor

Write a program that converts HTML documents to clean plain text, intelligently handling formatting, tables, lists, and links while removing all markup and scripts.

FUNCTIONAL REQUIREMENTS:
- Accept an HTML file path as a command-line argument
- Strip all HTML tags, CSS styles, JavaScript, and comments while preserving readable text content
- Convert HTML formatting to plain text equivalents: headings become UPPERCASE with underlines, bold text is wrapped in *asterisks*, lists become indented with bullets (- ) or numbers (1.), horizontal rules become dashed lines
- Convert HTML tables to aligned plain text tables with column padding and separator rows
- Convert hyperlinks to "text [URL]" format, or optionally strip URLs via --no-urls flag
- Preserve paragraph spacing: consecutive block elements get blank line separators
- Handle HTML entities: decode & < >   — etc. to their text equivalents
- Support extracting text from only specific HTML elements via --selector flag (CSS selector syntax, e.g., --selector "article" or --selector ".content")
- Support extracting and listing all URLs found in the document via --extract-urls flag
- Set maximum line width via --width flag (default: 80 characters) with word wrapping
- Support batch conversion of multiple HTML files via --batch flag
- Print the plain text output to console by default
- Save to a file via --output flag (default: same base name with .txt extension)
- If no input is given, generate a sample HTML page with headings, paragraphs, links, tables, lists, images, inline styles, scripts, and HTML entities, then convert it and display both the original HTML and the extracted text
- Handle errors: malformed HTML (parse gracefully), encoding detection, and binary file detection

Create a complete Python project for a clean Ubuntu 22.04 machine with only Python 3.10+ installed. Include:
- Source code
- requirements.txt with all dependencies (direct and transitive) pinned to exact versions
- README.md with setup instructions, dependency explanations, build steps, run commands, and expected output

[]

import sys
import argparse
import os
import html2text
from bs4 import BeautifulSoup

def extract_text(html_path, output_path=None, no_urls=False):
    if not os.path.exists(html_path):
        print(f"Error: {html_path} not found.")
        return

    with open(html_path, 'r', encoding='utf-8') as f:
        html_content = f.read()
    
    h = html2text.HTML2Text()
    h.ignore_links = no_urls
    h.body_width = 80
    h.unicode_snob = True
    
    text = h.handle(html_content)
    
    if not output_path:
        output_path = os.path.splitext(html_path)[0] + ".txt"
        
    with open(output_path, 'w', encoding='utf-8') as f:
        f.write(text)
    
    print(f"Extracted plain text to: {output_path}")
    return text

def generate_sample(path):
    print(f"Generating sample HTML: {path}")
    sample_html = """<!DOCTYPE html>
<html>
<head><title>Sample Page</title></head>
<body>
    <h1>Main Heading</h1>
    <p>This is a paragraph with <b>bold text</b> and an <a href="https://example.com">external link</a>.</p>
    <h2>Sub-heading</h2>
    <ul>
        <li>First item</li>
        <li>Second item with <i>italics</i></li>
    </ul>
    <table>
        <tr><th>Name</th><th>Value</th></tr>
        <tr><td>Item 1</td><td>100</td></tr>
        <tr><td>Item 2</td><td>200</td></tr>
    </table>
    <script>console.log('Ignore me');</script>
    <style>.css { color: red; }</style>
</body>
</html>
"""
    with open(path, "w", encoding='utf-8') as f:
        f.write(sample_html)

def main():
    parser = argparse.ArgumentParser(description="HTML to Plain Text Extractor")
    parser.add_argument("input", nargs="?", help="Path to HTML file")
    parser.add_argument("--output", help="Output text file path")
    parser.add_argument("--no-urls", action="store_true", help="Strip URLs from links")
    
    args = parser.parse_args()
    
    input_file = args.input
    if not input_file:
        input_file = "sample.html"
        if not os.path.exists(input_file):
            generate_sample(input_file)
            
    extract_text(input_file, args.output, args.no_urls)

if __name__ == "__main__":
    main()

# HTML to Plain Text Extractor (Python)

A tool for converting HTML documents to clean, formatted plain text.

## Setup Instructions

1. Ensure Python 3.10+ is installed.
2. Create a virtual environment:
   ```bash
   python3 -m venv venv
   source venv/bin/activate
   ```
3. Install dependencies:
   ```bash
   pip install -r requirements.txt
   ```

## Run Commands

- **Convert HTML**:
  ```bash
  python html_extractor.py page.html
  ```
- **Strip URLs**:
  ```bash
  python html_extractor.py page.html --no-urls
  ```
- **Run Demo**:
  ```bash
  python html_extractor.py
  ```

## Features
- **Intelligent Formatting**: Converts headings, bold, and lists to text equivalents.
- **Table Support**: Renders HTML tables as aligned ASCII tables.
- **Script/Style Removal**: Automatically strips non-content blocks.

beautifulsoup4==4.12.3
lxml==5.1.0
html2text==2024.2.26