79 lines
2.8 KiB
Markdown
79 lines
2.8 KiB
Markdown
# TODO - Developer Task Tracking
|
||
|
||
This file is used temporarily by the assistant to track progress on implementation tasks. It will be removed when all work is complete.
|
||
|
||
Milestone 0 – Bootstrap
|
||
- [x] M0-01: Create project skeleton directories and files
|
||
- [x] M0-02: Initialize Go module and pin go version
|
||
- [x] M0-03: Add .gitignore, .env.example, README
|
||
- [x] M0-04: Create Makefile (pending)
|
||
|
||
Milestone 1 – Config and Logging
|
||
- [x] M1-01: Implement logging setup using log/slog
|
||
- [x] M1-02: Implement configuration loader: env + flags + .env
|
||
- [x] M1-03: Define config schema and defaults
|
||
- [x] M1-04: Add config validation
|
||
- [ ] M1-05: Unit tests for config parsing precedence
|
||
|
||
Milestone 2 – Types and Parser
|
||
- [x] M2-01: Define normalized data types
|
||
- [x] M2-02: Define minimal Reddit API response structs
|
||
- [x] M2-03: Implement parser
|
||
- [x] M2-04: Implement deleted/removed filters
|
||
- [x] M2-05: Parser unit tests
|
||
|
||
Milestone 3 – Fetcher and Networking
|
||
- [x] M3-01: Build HTTP client
|
||
- [x] M3-02: Implement rate limiter
|
||
- [x] M3-03: Implement backoff with jitter
|
||
- [x] M3-04: URL builder for search.json
|
||
- [x] M3-05: Implement fetchPage
|
||
- [ ] M3-06: Fetcher tests
|
||
- [ ] M3-07: Implement metrics capture
|
||
|
||
Milestone 4 – Storage and Dedup
|
||
- [x] M4-01: Implement JSONL writer
|
||
- [x] M4-02: File naming and rotation
|
||
- [x] M4-03: Ensure output dir creation
|
||
- [x] M4-04: Implement dedup index
|
||
- [x] M4-05: Dedup persistence
|
||
- [ ] M4-06: Storage unit tests
|
||
|
||
Milestone 5 – Controller and Orchestration
|
||
- [x] M5-01: Implement controller orchestrator
|
||
- [x] M5-02: Pagination loop
|
||
- [x] M5-03: Integrate fetcher→parser→storage
|
||
- [x] M5-04: Progress reporting
|
||
- [x] M5-05: Graceful shutdown
|
||
- [x] M5-06: Summary report
|
||
- [x] M5-07: Wire CLI entrypoint
|
||
- [ ] M5-08: Error code taxonomy
|
||
- [ ] M5-09: Controller integration test
|
||
|
||
Milestone 6 – Nice-to-haves
|
||
- [ ] M6-01: Date-based subdir option
|
||
- [ ] M6-02: Optional compression on rollover
|
||
|
||
Milestone 7 – Performance
|
||
- [ ] M7-01: Performance runbook
|
||
- [ ] M7-02: Benchmark tuning
|
||
|
||
Milestone 8 – Docs and Release
|
||
- [ ] M8-01: README expansion
|
||
- [ ] M8-02: Cron examples
|
||
- [ ] M8-03: Sample data
|
||
- [ ] M8-04: CI steps
|
||
- [ ] M8-05: Tag and build release
|
||
|
||
Milestone 9 – Verification
|
||
- [ ] M9-01: Post-implementation checklist
|
||
|
||
|
||
Progress notes:
|
||
- Implemented project bootstrap, config loader, and logging.
|
||
- Implemented types and parser with unit tests for parser.
|
||
- Implemented fetcher client and page fetcher (rate limiting & Retry-After handling).
|
||
- Implemented storage (JSONL writer with rotation) and dedup index (persistent append-only file).
|
||
- Implemented controller orchestrator that coordinates fetching, parsing, deduping, and writing; supports graceful shutdown and basic retry/backoff.
|
||
- Remaining: add unit tests for fetcher & storage, wire CLI to controller, and add integration tests and metrics.
|