added icon downloader
This commit is contained in:
parent
8b5693b5c6
commit
5a2e37ae06
10 changed files with 829 additions and 68 deletions
|
|
@ -220,7 +220,11 @@ WHERE url_path = '/'
|
|||
|
||||
**Prerequisite:** Unbound running as system resolver on the EC2 instance.
|
||||
|
||||
**Input:** `icons` table rows where `scan_state = 'unscanned'`
|
||||
**Input:** `icons` table rows where `scan_state = 'unscanned'` and icon is worth downloading:
|
||||
- All `favicon_ico` entries (always attempt)
|
||||
- `link_rel` entries with no declared size (unknown, could be useful)
|
||||
- `link_rel` entries with declared size ≤64x64
|
||||
- Skip `link_rel` entries with declared size >64x64 (192x192, 180x180, 152x152, etc. — apple-touch-icon bloat we won't use at tab scale)
|
||||
|
||||
**Process:**
|
||||
1. Claim batch (randomized to spread load across hosts):
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue