line-today-scrape

Prototype respectful crawler for https://today.line.me/th/ (Thai locale).

Overview

This project contains a conservative, policy-first crawler prototype written in Python. It demonstrates:

• Robots.txt fetching and policy enforcement
• Rate-limited async fetching
• HTML extraction (meta tags + JSON-LD fallback)
• Local storage of raw snapshots and parsed JSON

Note: This is a prototype. Always review and run responsibly.

Quickstart

1. Install dependencies (recommend using poetry or virtualenv)

   poetry install

2. Run the crawler in dry-run mode (fetch limited pages):

   python -m linetoday.cli --dry-run --limit 5

Files

• linetoday/robots.py - Robots & policy manager
• linetoday/fetcher.py - Async HTTP fetcher with rate limiting
• linetoday/frontier.py - URL frontier and canonicalization
• linetoday/extractor.py - Article extraction logic
• linetoday/storage.py - Local storage for snapshots and parsed JSON
• linetoday/cli.py - CLI entrypoint

License

Prototype for demonstration only.