added icon downloader

This commit is contained in:
Joe Lothan 2026-05-17 22:09:03 -04:00
parent 8b5693b5c6
commit 5a2e37ae06
10 changed files with 829 additions and 68 deletions

View file

@ -220,7 +220,11 @@ WHERE url_path = '/'
**Prerequisite:** Unbound running as system resolver on the EC2 instance.
**Input:** `icons` table rows where `scan_state = 'unscanned'`
**Input:** `icons` table rows where `scan_state = 'unscanned'` and icon is worth downloading:
- All `favicon_ico` entries (always attempt)
- `link_rel` entries with no declared size (unknown, could be useful)
- `link_rel` entries with declared size ≤64x64
- Skip `link_rel` entries with declared size >64x64 (192x192, 180x180, 152x152, etc. — apple-touch-icon bloat we won't use at tab scale)
**Process:**
1. Claim batch (randomized to spread load across hosts):