everytab/PLAN.md

# EveryTab Implementation Plan

This plan builds the system described in ARCHITECTURE.md in incremental steps. We start with 100K hosts to validate the pipeline end-to-end, then scale to the full ~30M.

Each step has a clear deliverable and validation criteria. Steps are sequential — each phase builds on the previous.

---

## Phase 0: Project Setup & AWS Infrastructure [COMPLETED]

### Step 0.1: Repository Structure [COMPLETED]

```
everytab/
├── design.md
├── ARCHITECTURE.md
├── PLAN.md
├── infra/
│   ├── main.tf              # Terraform: all AWS resources
│   ├── terraform.tfvars.example
│   ├── ec2-userdata.sh      # EC2 bootstrap (Go, DuckDB, Unbound)
│   └── README.md            # Setup steps
├── pipeline/
│   ├── 01_cc_index/
│   │   └── schema.sql      # Postgres table definitions
│   ├── 02_warc_parse/
│   ├── 03_icon_download/
│   ├── 04_best_icon/
│   ├── 05_bundle_gen/
│   └── 06_frontend/
├── frontend/
├── stats/                   # gitignored
└── go.mod
```

### Step 0.2: AWS Infrastructure (Terraform) [COMPLETED]

Infrastructure managed via `infra/main.tf`. Single file, uses `var.scanning` bool to switch phases:
- `terraform apply` — creates all scanning resources (EC2, RDS, S3 icons, S3 site, IAM, security groups)
- `terraform apply -var="scanning=false"` — destroys scanning resources, keeps site bucket
- `terraform destroy` — removes everything

Resources created:
- S3 `everytab-icons` (private), S3 `everytab-site` (for CloudFront later)
- RDS Postgres 16, db.t3.medium, 20GB gp3
- EC2 c5.xlarge, Amazon Linux 2023, 50GB gp3
- Security groups (SSH from home IP, RDS from EC2 only)
- IAM role + instance profile (S3 access only)
- SSH key (Terraform-managed ed25519)

### Step 0.3: EC2 Environment Setup [COMPLETED]

Bootstrap via `infra/ec2-userdata.sh`:
- Go 1.22+, DuckDB (httpfs + postgres extensions), Unbound (recursive resolver), psql, tmux
- Unbound configured as system resolver (systemd-resolved disabled)
- DATABASE_URL in .bashrc
- Schema applied: hosts + icons tables with indexes

---

## Phase 1: CC-Index Query (Stage 1)

### Step 1.1: Database Schema

Create the Postgres tables. Run via `psql`:

```sql
CREATE TABLE hosts (
    id SERIAL PRIMARY KEY,
    hostname TEXT NOT NULL UNIQUE,
    protocol TEXT NOT NULL,
    crawl_id TEXT NOT NULL,
    warc_filename TEXT NOT NULL,
    warc_record_offset BIGINT NOT NULL,
    warc_record_length INT NOT NULL,
    html_title TEXT,
    iframe_allowed BOOLEAN,
    best_icon_s3_key TEXT,
    parsed BOOLEAN DEFAULT FALSE
);

CREATE TABLE icons (
    id SERIAL PRIMARY KEY,
    host_id INT NOT NULL REFERENCES hosts(id),
    url TEXT NOT NULL,
    source TEXT NOT NULL,
    rel_type TEXT,
    rel_sizes TEXT,
    content_type TEXT,
    width INT,
    height INT,
    file_size INT,
    s3_key TEXT,
    scan_state TEXT DEFAULT 'unscanned',
    error TEXT
);

CREATE INDEX idx_hosts_parsed ON hosts(id) WHERE parsed = FALSE;
CREATE INDEX idx_icons_unscanned ON icons(id) WHERE scan_state = 'unscanned';
CREATE INDEX idx_icons_host_id ON icons(host_id);
```

**Done when:** Tables exist in RDS, schema matches ARCHITECTURE.md.

### Step 1.2: DuckDB CC-Index Query (100K limit) [COMPLETED]

Script: `pipeline/01_cc_index/query.sh`

Uses DuckDB with `aws` extension (credential chain) to read parquet directly from `s3://commoncrawl/.../*.parquet` glob, with the `postgres` extension to write results into RDS. Auto-detects latest crawl ID from the CC API.

Deduplication via `GROUP BY url_host_name` with `first(... ORDER BY ...)` aggregates (hash aggregation — more memory-efficient than window functions).

**Result:** 100K hosts, 77% https / 23% http, completed in 692s.

**Done when:** 100K hosts in the database with valid WARC coordinates.

### Step 1.3: Validate WARC Coordinates [COMPLETED]

Manually fetched WARC records with curl byte-range requests to `data.commoncrawl.org`. Confirmed valid WARC headers, HTTP response, and HTML with `<title>` and `<link rel="icon">` tags.

---

## Phase 2: WARC Parsing (Stage 2) [COMPLETED]

### Steps 2.1-2.3 [COMPLETED]

Binary: `pipeline/02_warc_parse/` (5 files: main.go, warc.go, parser.go, process.go, db.go, log.go)

**Architecture:**
- Fetches WARC records via AWS SDK S3 byte-range GetObject (using EC2 instance profile credentials)
- Parses WARC records with `github.com/nlnwa/gowarc/v3`
- Parses HTML with `golang.org/x/net/html` tokenizer (lenient, stops at `<body>`)
- Detects charset via `golang.org/x/net/html/charset` and converts to UTF-8
- Sanitizes titles with `strings.ToValidUTF8` as final safety net
- Concurrent goroutine pool with configurable concurrency
- Per-host log lines to stdout + optional log file
- Panic recovery per goroutine (logs PANIC, doesn't mark row as parsed)
- DB errors tracked and logged with `DB_ERROR:` prefix

**CLI:** `./warc_parse --db URL [--concurrency N] [--batch-size N] [--limit N] [--dry-run] [--log-file PATH] [--log-errors-only]`

**Result (100K hosts, concurrency 500):**
- Duration: 5m31s (~300 hosts/sec)
- Titles found: 93,384 (93%)
- Icons found: 201,780 (~2 per host)
- Iframe blocked: 17,855 (18%)
- Fetch errors: 3
- DB errors: 0
- Panics: 0

---

## Phase 3: Icon Download (Stage 3) [COMPLETED]

### Steps 3.1-3.3 [COMPLETED]

Binary: `pipeline/03_icon_download/` (6 files: main.go, download.go, image.go, s3.go, db.go, log.go)

**Architecture:**
- Channel-based work distribution: producer goroutine claims batches, N worker goroutines consume from buffered channel (no worker starvation)
- Shared `http.Transport` for connection pooling / TLS session reuse
- Content-addressed S3 storage (SHA-256 hash as key, dedup via HeadObject before upload)
- Magic byte validation (PNG, GIF, JPEG, ICO, BMP, WebP, SVG)
- ICO directory parsing for dimensions (picks largest ≤64x64)
- Filters to eligible icons only: `favicon_ico` + link_rel with no declared size or ≤64x64
- md5(id) shuffle in claim query to spread requests across hosts
- Panic recovery per worker, DB errors tracked and logged

**CLI:** `./icon_download --db URL [--s3-bucket NAME] [--concurrency N] [--batch-size N] [--timeout D] [--max-size N] [--limit N] [--dry-run] [--log-file PATH] [--log-errors-only]`

**Result (100K hosts, ~224K eligible icons):**
- Duration: 10m36s (351 icons/sec)
- Completed: 156,214 (70%)
- Failed: 67,459 (30% — mostly HTTP 404s from stale crawl data)
- Dedup hits: 55,771 (25% — shared Wix/WordPress/hosted platform favicons)
- Downloaded: 1.9GB
- DNS errors: 1,668 | Timeouts: 2,129 | HTTP errors: 47,565 | Invalid: 11,803 | Too large: 777
- DB errors: 0 | Panics: 0

---

## Phase 4: Best Icon Selection & Bundle Generation (Stages 4-5) [COMPLETED]

### Step 4.1: Best Icon Selection SQL [COMPLETED]

Script: `pipeline/04_best_icon/select.sql`

Selects the best icon per host using `DISTINCT ON` with priority ordering. Excludes SVGs (can't rasterize) and ≤2x2 icons (tracking pixels). See ARCHITECTURE.md for the full decision flow.

**Result:** 70,366 hosts got an icon (72%), 23,018 have title but no icon.

### Steps 4.2-4.4: Bundle Generator [COMPLETED]

Binary: `pipeline/05_bundle_gen/` (6 files: main.go, bundle.go, convert.go, db.go, s3.go, log.go)

**Architecture:**
- Queries all hosts with titles (randomized), concurrently downloads best icon from S3 icons bucket
- Uses `github.com/biessek/golang-ico` for ICO decoding (handles all bit depths including palette-based 1/4/8bpp)
- `image.Decode` handles PNG/GIF/JPEG/WebP/BMP/ICO via registered decoders. SVGs excluded.
- Icons >128px downscaled to 32x32 (nearest-neighbor). Icons ≤128px kept as-is.
- Re-encodes all icons as PNG, base64-encoded inline in bundle JSON.
- Panic recovery per icon conversion (malformed ICO files in the library)
- Concurrent S3 downloads with configurable concurrency (default 50)

**CLI:** `./bundle_gen --db URL [--icons-bucket NAME] [--site-bucket NAME] [--entries-per-bundle N] [--concurrency N] [--limit N] [--dry-run] [--output-dir DIR] [--log-file PATH] [--log-errors-only]`

**Result (93K hosts with titles, 70K with icons):**
- Duration: 1m30s
- Bundles created: 779 (120 entries each, last bundle partial)
- Total size: 165MB (avg 216KB per bundle)
- Convert errors: 1,263 (1,077 SVGs + 186 other — panics, truncated files, corrupt GIFs)
- S3: 779 JSON files in `everytab-site/tabs/`

---

## Phase 5: Frontend (Stage 6) [COMPLETED — v1]

### Steps 5.1-5.6 [COMPLETED]

Files: `frontend/index.html` and `frontend/site.js`

**Architecture:**
- Vanilla JS, no framework. Two files: HTML (with inline CSS) + JS.
- Fetches random bundle JSONs from `tabs/{N}.json`, renders tabs as rows filling the viewport.
- Seeded PRNG (`Date.now()` + mulberry32) — every visitor sees unique tab arrangement.
- Infinite scroll: loads more bundles as user approaches the bottom.
- Tracks loaded bundle IDs in a Set to avoid duplicates.

**Tab rendering:**
- Browser-specific tab styling via `navigator.userAgent` detection (Chrome, Firefox, Safari).
- Inactive tab appearance by default, selected/active style when iframe is open.
- Light mode default, auto-switches to dark mode via `prefers-color-scheme`.
- Bidirectional marquee: each row randomly scrolls left or right at different speeds (90-150s per cycle).
- Tabs duplicated in DOM for seamless marquee loop (`translateX(-50%)`).
- Hover shows full title as native tooltip.
- External link indicator (↗) on tabs that don't allow iframes.

**Iframe viewer:**
- Inline, not overlay — opens between tab rows, pushes content down (75vh height).
- Header shows favicon, title, external link, and close button.
- Sandboxed iframe (`allow-scripts allow-same-origin allow-forms`).
- Close via X button, Escape key.
- Only one viewer open at a time.

**`TOTAL_BUNDLES`** baked into HTML at build time. Build script (`pipeline/06_frontend/build.sh`) still TODO — currently hardcoded.

---

## Phase 6: Integration & End-to-End Test (100K) [COMPLETED]

### Steps 6.1-6.3 [COMPLETED]

Full clean end-to-end run from `terraform apply` to live site at everytab.site.

**Pipeline timing (100K hosts):**
| Stage | Duration |
|-------|----------|
| CC-Index query | 13m11s |
| WARC parsing | 4m55s |
| Icon download | 10m39s |
| Best icon selection | instant |
| Bundle generation | 1m32s |
| Frontend deploy | seconds |
| **Total pipeline** | **~31 minutes** |

**Loss funnel:**
```
100,000 hosts
 → 93,432 with titles (6.6% loss)
 → 70,551 with icons selected (24.4% loss)
 → 69,306 with icons in bundles (1.8% convert errors)
 → 779 bundles, 165MB total, avg 217KB per bundle
```

**Parameter review:**
- `ENTRIES_PER_BUNDLE = 120` — fills the screen well, kept as-is
- Icon download concurrency 200 — I/O bound at 350 icons/sec, increasing doesn't help
- Timeouts 10s — good balance, 2,261 timeouts (1%) is acceptable
- Icons look good on the live site

**Infrastructure notes:**
- c5.xlarge (8GB) needs 4GB swap for the DuckDB query — gets OOM killed without it
- DuckDB occasionally gets S3 503 (rate limit) — retry works
- `force_destroy = true` on icons bucket needed for clean teardown
- deploy.sh sed pattern must match `[0-9]*` not `.*` to avoid eating `</script>`

---

## Phase 7: Full-Scale Run (30M)

### Step 7.1: Remove Limits, Re-run CC-Index Query

Update the DuckDB query to remove `LIMIT 100000`. Re-run.

Considerations:
- If httpfs takes >1hr, switch to downloading the parquet files first
- May need to increase RDS storage (30M rows with WARC paths ≈ 5-10GB)
- Monitor DuckDB memory usage

**Validation:** `SELECT COUNT(*) FROM hosts;` shows ~30M rows.

### Step 7.2: Run WARC Parser at Scale

Run with full concurrency against 30M hosts. Expected time: 2-6 hours.

Monitor:
- Throughput (hosts/sec)
- Error rate stability (should plateau, not climb)
- Postgres connection pool health
- Memory usage

### Step 7.3: Run Icon Downloader at Scale

This is the long pole — expected 12-48 hours.

Monitor continuously:
- icons/sec rate
- DNS cache hit rate (check Unbound stats: `unbound-control stats`)
- S3 upload rate
- Error rate by type
- Completion percentage

If too slow (projected >48hrs):
- Consider increasing concurrency (if memory allows)
- Consider spinning up fleet (add more EC2 instances running the same binary)
- Check if DNS is the bottleneck (Unbound stats)
- Check if S3 uploads are the bottleneck (batch or reduce HEAD checks)

### Step 7.4: Best Icon Selection + Bundle Generation

Run at full scale. Expected: 1-2 hours total.

Monitor bundle sizes — verify they're in the expected range with `ENTRIES_PER_BUNDLE` from tuning.

### Step 7.5: Rebuild Frontend + Deploy

Run frontend build with the real bundle count. Invalidate CloudFront.

**Validation:** Visit the live site. Browse around. Check:
- Tab variety (seeing diverse sites, not just one TLD)
- Icon quality (no broken images, reasonable sizes)
- Performance (bundles load quickly, no jank)
- Stats page / stats.json looks correct

**Done when:** Full-scale site is live and working.

---

## Phase 8: Backup & Teardown

### Step 8.1: Backup RDS to Homelab

```bash
# On EC2 (fast connection to RDS):
pg_dump -Fc $DATABASE_URL > everytab_dump.pgfc

# Transfer to homelab (from EC2 or direct):
scp everytab_dump.pgfc homelab:/backups/everytab/

# On homelab, verify restore:
pg_restore -d everytab_local everytab_dump.pgfc
psql everytab_local -c "SELECT COUNT(*) FROM hosts; SELECT COUNT(*) FROM icons;"
```

### Step 8.2: Backup Icons S3 to Homelab

```bash
# From homelab (or EC2 as intermediary):
aws s3 sync s3://everytab-icons/ /backups/everytab/icons/

# Verify file count matches:
ls /backups/everytab/icons/ | wc -l
# Compare with: aws s3 ls s3://everytab-icons/ | wc -l
```

### Step 8.3: Verify & Teardown

After confirming backups:

```bash
# Verify the live site still works (it only depends on everytab-site + CloudFront)
curl -s https://your-cloudfront-domain.net/ | head

# Teardown scanning infrastructure:
aws rds delete-db-instance --db-instance-identifier everytab --skip-final-snapshot
aws s3 rb s3://everytab-icons --force
aws ec2 terminate-instances --instance-ids i-xxxxx
```

**Done when:** Only `everytab-site` S3 bucket + CloudFront remain running. Monthly cost: ~$2-4.

---

## Development Notes

### Execution Order

Phases are sequential: 0 → 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8. Frontend (Phase 5) uses real data from the 100K pipeline run. The only thing that can be developed ahead of time is writing Go code locally before EC2 is ready (compile-test locally, run on EC2).

### Progress & Observability

All Go programs have two output modes running simultaneously:

**Per-item log lines** (stdout, above the progress bar):
- WARC parser: `parsed: example.com 200 "Example Domai..." ok` or `parsed: broken.net 200 "" err:no_title`
- Icon downloader: `icon: https://example.com/favicon.ico 32x32 png 4.2KB ok` or `icon: https://fail.org/favicon.ico err:timeout`
- Bundle generator: `bundle: 0042.json 120 entries 247KB ok`

Each line is a short, fixed-format summary — hostname/URL, key result, and status. Keeps it scannable when running live.

**Log file** (`--log-file path/to/out.log`): If provided, mirror all per-item log lines to disk. For full-scale runs, consider using `--log-errors-only` flag to only write error lines to the log file (avoids filling disk with 30M success lines). Without `--log-file`, logs only go to stdout.

**Progress bar** (bottom of terminal, `schollz/progressbar`):
- Items processed / total items
- Processing rate (items/sec)
- ETA
- Error count

On completion, each program prints a summary line and writes its stats JSON (with started_at, finished_at, duration_seconds, and stage-specific counters).

### Testing Strategy

- **Dry-run flags** on all Go programs: print what would happen without mutating DB/S3
- **--limit flags** on all Go programs: process a small subset quickly
- **Spot-checks:** after each stage, manually verify 5-10 random entries
- **Stats files:** compare counts between stages to catch data loss
- **100K dev set:** full pipeline at small scale before committing to a 24hr+ full run

### Common Pitfalls to Watch For

- **DuckDB CC-Index path:** The exact S3 path to parquet files changes per crawl. Check Common Crawl's website for the latest crawl ID and index location.
- **WARC record format:** WARC records have a specific envelope format (WARC/1.0 header, blank line, HTTP response). Don't assume the HTTP response starts at byte 0.
- **Relative icon URLs:** `/favicon.ico` is relative to root, but `favicon.ico` (no leading slash) is relative to the page path. Since we only have root pages (`/`), both resolve the same. But `../icons/fav.png` could be tricky — handle gracefully or skip.
- **ICO files are complex:** The ICO container format can embed BMP (with a modified header) or PNG. Many "ICO" files are actually just PNGs renamed to .ico. Check magic bytes, not file extension.
- **SVG rasterization:** Go doesn't have great native SVG support. Consider shelling out to `rsvg-convert` or `librsvg`, or use a Go library like `github.com/nicholasgasior/goresvg`. This can be a follow-up if SVG icons are rare.
- **Postgres connection limits:** RDS db.t3.medium has max_connections ≈ 80. With 1000 goroutines, we need connection pooling (pgx pool handles this). Set pool max to ~40 connections.
- **S3 eventual consistency:** After uploading an icon, a HEAD request might not find it immediately. For dedup checks, handle "not found" gracefully (just upload again — idempotent since key is content hash).
- **CloudFront caching:** After deploying new bundles, invalidate `/*` or set short TTL during development. For production, use long TTLs (bundles are immutable between crawls).

---

## Progress Log

### Phase 0 — Completed 2026-05-17

**Changes from original plan:**
- Replaced shell scripts (`setup.sh`, `teardown.sh`) with Terraform (`infra/main.tf`). Single file, `var.scanning` bool switches between scanning and serving phases.
- SSH key is Terraform-managed (no passphrase, stored in state) rather than manually generated.
- CloudFront distribution deferred — not created in Phase 0, will add to Terraform when frontend is ready.
- Added `infra/README.md` with terse setup steps for future replication.

**Lessons learned:**
- Shell scripts with `2>/dev/null || echo "already exists"` swallow real errors. Terraform's declarative model avoids this entirely — errors are always surfaced.
- RDS requires a DB subnet group (2+ subnets in different AZs). The original shell script didn't create one, causing a silent failure. Terraform handles this dependency automatically.
- Amazon Linux 2023 uses `systemd-resolved` which manages `/etc/resolv.conf`. Must disable it before pointing resolv.conf at Unbound. `chattr +i` doesn't work on the symlink.
- AWS EC2 key pairs created via API don't support passphrases. Use `tls_private_key` in Terraform or generate locally with `ssh-keygen` + import.
- When an AWS key pair name already exists from a previous run, Terraform may not regenerate it. Use `-replace` to force recreation of the key + instance together.

### Phase 1 (Steps 1.1-1.2) — Completed 2026-05-17

**Changes from original plan:**
- Used DuckDB `aws` extension with `CREDENTIAL_CHAIN` instead of httpfs anonymous access. The commoncrawl S3 bucket requires authenticated requests.
- IAM role needed explicit `s3:GetObject` and `s3:ListBucket` on `arn:aws:s3:::commoncrawl/*` — the bucket doesn't allow cross-account access based on bucket policy alone.
- Used `GROUP BY` with `first(... ORDER BY ...)` instead of `ROW_NUMBER()` window function. More memory-efficient (hash aggregation vs sort), cleaner syntax.
- DuckDB can glob `s3://.../subset=warc/*.parquet` directly (300 files) — no need to fetch a file list or download parquet locally.
- Dropped the `url_port IN (80, 443)` filter — CC stores standard ports as NULL, not 80/443. Replaced with `url_port IS NULL`.

**Lessons learned:**
- DuckDB URL-encodes `=` in S3 paths (e.g., `crawl%3DCC-MAIN-2026-17`) but S3 decodes it correctly. The real issue was always IAM permissions, not path encoding.
- The `commoncrawl` S3 bucket requires valid AWS credentials for both GetObject and ListBucket. Anonymous access (unsigned requests) does not work. Any valid IAM identity works as long as their policy allows it.
- DuckDB's LIMIT can interact unexpectedly with GROUP BY — the optimizer may stop reading input early once it has enough groups. This wasn't our issue (it was the port filter) but worth noting for future queries.
- CC-Index stores `url_port` as NULL for standard ports (80/443), not as the integer. Always check actual column values before writing filters.
- c5.xlarge (8GB) is tight for this query — uses 6.4GB + swap. For the full 30M run, use c5.2xlarge (16GB).
- Query takes ~692s (11.5 min) for 100K output rows reading all 300 parquet files. Full run without LIMIT will be similar duration but more memory for the hash table.

### Phase 2 — Completed 2026-05-17

**Changes from original plan:**
- Used AWS SDK S3 GetObject for WARC byte-range requests instead of HTTPS to `data.commoncrawl.org`. The HTTPS endpoint rate-limits at ~100 concurrent connections (429s). S3 has no such limit.
- Removed progress bar — it interfered with per-host log lines. Replaced with clean stdout log lines + summary at end. Check DB for mid-run progress.
- Added `process.go` and `log.go` files (plan had 4 files, we have 6 — cleaner separation).
- Added charset detection + UTF-8 conversion (`golang.org/x/net/html/charset` + `golang.org/x/text/transform`) for international titles.
- Added `strings.ToValidUTF8` sanitization as final safety net for titles that still have invalid bytes after charset conversion.
- Panic recovery per goroutine — logs `PANIC:` prefix, doesn't mark row as parsed (retryable on next run).
- DB write errors tracked separately (`DB_ERROR:` prefix, counted in summary + stats JSON).

**Lessons learned:**
- `data.commoncrawl.org` aggressively rate-limits (403/429) at ~100 concurrent connections. Use S3 API directly for high-concurrency access.
- Many Chinese/Japanese sites serve GBK or other non-UTF-8 encodings without declaring it in Content-Type or `<meta>`. `charset.DetermineEncoding` catches most but not all. `strings.ToValidUTF8` as final sanitization prevents Postgres encoding errors.
- gowarc's `HttpHeader()` can return nil for malformed records — always nil-check library return values defensively.
- Increasing concurrency from 100 to 500 didn't improve throughput (~300 hosts/sec either way). The bottleneck is likely Postgres write latency or S3 per-connection bandwidth, not parallelism. Could investigate batch inserts for the full run.
- Progress bars and per-item log lines don't mix well in terminals. Pick one or write progress to a separate channel (file, stderr).

### Phase 3 — Completed 2026-05-18

**Changes from original plan:**
- Filtered eligible icons before downloading: skip link_rel icons with declared size >64x64 (apple-touch-icon bloat). Reduced download count from ~302K to ~224K.
- Channel-based worker pool instead of semaphore pattern — producer goroutine feeds work channel, N workers consume. No starvation between batch claims.
- Shared http.Transport for connection pooling (marginal benefit since hosts are unique, but reduces GC pressure).
- No progress bar — same approach as Phase 2 (log lines + summary).
- User-Agent set to `EveryTabBot/1.0` with link to `everytab.site/bot` for bot identification.

**Lessons learned:**
- 70% icon download success rate is expected — most failures are 404s from domains/pages that changed since the crawl. This is acceptable loss.
- 25% dedup rate — many hosted platforms (Wix, WordPress.com, Squarespace) serve identical default favicons. Content-addressed S3 storage handles this efficiently.
- `data.commoncrawl.org` rate-limits HTTPS but S3 does not — same pattern as WARC parsing. Use S3 API for all CC access.
- Favicon download is I/O bound (network latency to diverse hosts worldwide). Concurrency helps up to a point, then the long tail of slow/dead servers dominates. 351 icons/sec at 200 concurrency.
- Invalid image detection (magic bytes) catches ~5% of "successful" downloads that are actually HTML error pages served at `/favicon.ico`.

### Phase 4 — Completed 2026-05-18

**Changes from original plan:**
- Used `github.com/biessek/golang-ico` instead of hand-rolled ICO decoder. Handles all bit depths (1/4/8/24/32bpp) correctly. Eliminated ~20 ICO decode errors from the hand-rolled version.
- SVGs excluded from best-icon selection (can't rasterize without external deps). SVG-only hosts show up with no icon instead of failing at conversion time.
- Added ≤2x2 pixel exclusion from best-icon selection (tracking pixels / garbage favicons).
- Icons >128px downscaled to 32x32 during bundle generation. Icons ≤128px (including 80x80) kept as-is — browser CSS handles display scaling.
- Added panic recovery around icon conversion (the ICO library panics on some malformed files).
- Added concurrency for S3 icon downloads during bundle generation (was single-threaded, now 50 concurrent).

**Lessons learned:**
- Many hosts (28%) have no usable favicon at all — their /favicon.ico returns HTML or 404, and they have no link rel="icon". These appear in bundles title-only.
- The golang-ico library panics on certain malformed ICO files (index out of bounds). Third-party decoders need panic recovery wrappers.
- 80x80 icons are overwhelmingly one single default favicon shared by a hosting platform (~4,276 sites share one hash). Content-addressed storage handles this.
- Bundle sizes are very heterogeneous (39KB to 198KB) due to icon size variance. Average 216KB is well within our target.
- SVG favicons are ~3.5% of downloaded icons (5,128 out of 156K). Supporting SVG rasterization would recover ~1,077 hosts. Deferred to future improvement.

### Phase 5 — Completed 2026-05-18

**Changes from original plan:**
- Inline iframe viewer instead of full-screen overlay. Opens between tab rows, pushes content down (75vh).
- Browser-specific tab styling (Chrome/Firefox/Safari) via userAgent detection — original plan deferred this to v2.
- Light/dark mode via `prefers-color-scheme` — original plan just targeted Firefox dark theme.
- No progress bar in any Go program — per-item log lines + summary at end is the pattern across the project.
- `TOTAL_BUNDLES` hardcoded in HTML for now — build script (Step 5.6) still TODO.

**Lessons learned:**
- CSS marquee with alternating directions needs care: right-scrolling rows must start at `translateX(-50%)` and animate to `0`, not the reverse. Both directions use the same duplicated DOM structure.
- `width: max-content` on the tab row is essential — without it, flex container constrains to viewport width and percentage-based translateX is wrong.
- Tab hover expansion (removing max-width) causes layout shifts that make neighboring tabs impossible to click. Native tooltip (`tab.title`) is simpler and has no side effects.
- Hundreds of animating DOM elements cause frame drops on weaker GPUs. `will-change: transform` helps but slower animation speeds help more.

---

## Future Improvements

### Phase 6 — Completed 2026-05-18

**Changes from original plan:**
- Added 4GB swap file to EC2 bootstrap — DuckDB OOM kills without it on c5.xlarge.
- Added `force_destroy = true` to icons S3 bucket — terraform teardown fails otherwise when bucket has objects.
- Pipeline README with full sanity checks between each stage.
- Deploy script (`pipeline/06_frontend/deploy.sh`) automates frontend upload + CloudFront invalidation.
- CloudFront + ACM certificate + S3 bucket policy added to Terraform. Domain setup (Gandi ALIAS record) is one-time manual step.

**Lessons learned:**
- Pipeline is reproducible — second clean run produced nearly identical numbers to the first.
- DuckDB gets S3 503 errors occasionally (rate limiting). Retry works. May need `SET threads = 4` or retry logic for the full 30M run.
- ACM certificate validation is a chicken-and-egg with CloudFront — use `aws_acm_certificate_validation` resource to make Terraform wait for DNS validation before creating the distribution.
- deploy.sh sed must match `[0-9]*` not `.*` — the greedy match eats the closing `</script>` tag.
- Total wall-clock from `terraform apply` to live site: ~45 minutes (including bootstrap).

---

## Future Improvements

### Pipeline
- **WARC parser: retry on fetch errors** — Currently 3 fetch errors out of 100K (tolerable loss). Could add 1 retry with backoff for transient S3 errors.
- **WARC parser: batch DB inserts** — Currently one INSERT per icon. Using pgx batch or CopyFrom could improve DB write throughput and potentially unblock higher concurrency.
- **WARC parser: investigate throughput ceiling** — 300 hosts/sec at both 100 and 500 concurrency suggests a bottleneck. Profile to determine if it's S3 response latency, Postgres writes, or something else. For the full 30M run this determines wall-clock time (~28 hours at current rate).
- **CC-Index query: c5.2xlarge for full run** — 8GB is tight with 6.4GB usage + swap. 16GB instance for the 30M-host full run.
- **Encoding: investigate remaining garbled titles** — Some titles still show `<60>` in output (e.g., `BERGSTRANDS BAGERI <20>...`). These are pages that lie about their encoding. Could try more aggressive charset detection heuristics.
- **Icon download: retry transient failures** — DNS and timeout failures could benefit from a single retry. Would recover a small percentage of icons.
- **Icon download: download large link_rel icons** — Currently skipping declared sizes >64x64. Re-run with broader filter for future high-res projects.
- **Bundle gen: SVG rasterization** — ~1,077 hosts have SVG-only favicons. Could add `rsvg-convert` or a Go SVG library to rasterize these.
- **Bundle gen: smarter downscaling** — Currently nearest-neighbor to 32x32 for >128px icons. Could use bilinear/Lanczos for better quality, or preserve aspect ratio for non-square icons.

### Frontend
- **Performance: reduce DOM / animation cost** — Pause marquee animation on off-screen rows (IntersectionObserver). Virtualize rows to reduce total DOM element count.
- **Cross-browser tab styling** — Polish Chrome/Firefox/Safari tab appearances to more closely match real browser tabs. Test on actual browsers, use screenshots as reference.
- **Mobile layout** — Current design assumes desktop viewport. Need responsive tab sizing and touch-friendly interaction.
- **Build script** — `pipeline/06_frontend/build.sh` to inject TOTAL_BUNDLES and deploy to S3 + CloudFront invalidation.
- **Stats page** — Serve `stats.json` and render pipeline stats (host count, icon coverage, crawl date) on the site.