diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b1ee8aa..332a451 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -66,8 +66,8 @@ All resources in **us-east-1**. | Resource | Purpose | Lifecycle | |----------|---------|-----------| -| EC2 (c5.xlarge) + 1TB EBS | Run all pipeline stages, store icon archive | Scanning only | -| RDS Postgres (db.t3.medium) | Store hosts/icons metadata | Scanning only (backup to homelab, then delete) | +| EC2 (c5.2xlarge) + 1TB EBS | Run all pipeline stages, store icon archive | Scanning only | +| RDS Postgres (db.m5.large) | Store hosts/icons metadata | Scanning only (backup to homelab, then delete) | | S3 `everytab-site` | Static site: index.html, site.js, tabs/*.json | Permanent | | CloudFront | CDN for static site (Brotli compression enabled) | Permanent | | S3 `everytab-logs` | CloudFront access logs | Permanent | @@ -117,6 +117,7 @@ Icons are stored on local disk during scanning, not S3. The EBS volume holds the | s3_key | TEXT | SHA-256 hash of content (used as local file path, legacy column name) | | scan_state | TEXT DEFAULT 'unscanned' | `unscanned`, `in_progress`, `completed`, `failed` | | error | TEXT | Error message if failed | +| downloaded_at | TIMESTAMPTZ | When the icon was fetched (NULL if not yet downloaded) | **Indexes:** - `CREATE INDEX idx_icons_unscanned ON icons(id) WHERE scan_state = 'unscanned'` — partial index for work claiming. Only indexes unscanned rows; shrinks as work completes. Minimal write overhead since index only updates on transition OUT of 'unscanned'. @@ -199,17 +200,23 @@ WHERE url_path = '/' 7. Insert all discovered `link rel="icon"` entries into `icons` (any format: ICO, PNG, GIF, SVG, WebP, JPEG) 8. Update `hosts` row: html_title, iframe_allowed, parsed = TRUE -**Concurrency:** High — thousands of goroutines with a semaphore/pool. CC's S3 handles massive throughput. +**Architecture:** Three-stage pipeline: -**Error handling:** Malformed HTML → still extract what we can (partial title, partial icons). WARC fetch failure → log and skip (mark parsed = TRUE with NULL title to avoid retry loops). All errors logged with hostname for investigation. +``` +[DB fetcher] → hostCh → [500 workers] → resultCh → [DB writer with pgx.Batch] +``` + +1. **DB fetcher** (1 goroutine): continuously pages through unparsed hosts (batch size 5000), feeds `hostCh`. +2. **Workers** (500 goroutines, configurable): fetch WARC from S3, parse HTML, update stats, send successful results to `resultCh`. I/O-bound on S3 latency. +3. **DB writer** (1 goroutine): collects results, flushes every 100 using `pgx.Batch` (~400 queries per DB round-trip). S3 retry with 6 attempts and exponential backoff for transient 503s. + +**Error handling:** Malformed HTML → still extract what we can (partial title, partial icons). WARC fetch failure → log and skip (host stays `parsed = FALSE`, retryable on next run). Max 50 link_rel icons per host (defensive cap against adversarial pages). **Icon URL handling:** Relative URLs resolved against `{protocol}://{hostname}/`. Absolute URLs kept as-is. Data URIs ignored. -**No scan_state needed:** CC's S3 is highly reliable. The `parsed` boolean is sufficient. If the process crashes mid-batch, re-run picks up where it left off (unparsed rows). - **Cost:** $0 (same Open Data program). -**Stats emitted:** Rows processed, titles extracted, icons found (by source: favicon_ico vs link_rel), icon format distribution, iframe restrictions found, parse failures, rows with no title. +**Stats emitted:** Rows processed, titles extracted, no-title count, icons found, iframe restrictions, fetch/parse errors, DB errors, panics. ### Stage 3: Icon Download @@ -247,7 +254,7 @@ WHERE url_path = '/' - Update icons row: s3_key (the SHA-256 hash), content_type (from actual data, not HTTP header), width, height, file_size, scan_state = 'completed' - On failure: scan_state = 'failed', error = reason -**Concurrency:** Channel-based worker pool (default 200 workers, configurable). Producer goroutine feeds a buffered channel (buffer = batch size), N workers consume. No starvation between batch claims. +**Concurrency:** Channel-based worker pool (default 2500 workers, configurable). Producer goroutine feeds a buffered channel (buffer = batch size), shuffles each batch to avoid hitting the same host back-to-back. N workers consume from the channel. **Fast failure strategy:** - DNS failure → fail immediately (Unbound will cache NXDOMAIN) @@ -336,6 +343,54 @@ Bundles are written in-place (overwriting previous run). No delete-first step, s 3. **Verify backups:** confirm pg_dump restores cleanly on homelab, spot-check icon files 4. Tear down scanning infra: `terraform apply -var="scanning=false"` (deletes RDS, EC2, icons S3 bucket) +## Performance Characteristics + +Each pipeline stage has different bottlenecks. Understanding these explains the concurrency choices and why certain stages can't be sped up further on a single machine. + +### Stage 1: CC-Index Query +- **Download phase: network-bound.** `aws s3 sync` of ~166GB of parquet files. Throughput limited by EC2 network bandwidth (10 Gbps on c5.2xlarge). Takes ~10-15 minutes. +- **Query phase: memory-bound.** DuckDB loads the GROUP BY hash table into memory. At 30M output rows, the hash table approaches 16GB. `temp_directory` is set to EBS so DuckDB spills to NVMe efficiently (large sequential I/O) rather than relying on OS swap (random 4KB page faults). On c5.2xlarge (16GB RAM) with 8GB swap, the query completes without severe thrashing. +- **Not CPU-bound** — DuckDB's columnar scan is efficient, CPU cores are underutilized during the query. + +### Stage 2: WARC Parsing +- **CPU-bound + network I/O-bound (S3).** Each WARC fetch is a byte-range S3 GetObject request (~100-200ms round-trip), but TLS handshakes + gzip decompression + HTML parsing consume significant CPU. At 500 goroutines on 4 cores, CPU was at 100%. On c5.2xlarge (8 cores), more workers can actually compute simultaneously. +- **DB writes batched** via `pgx.Batch` — 500 results (~2000 queries) per round-trip. Non-burstable RDS (db.m5.large) provides consistent write performance. Burstable t3 instances throttle under sustained load and cause pipeline stalls via channel back-pressure. +- **Channel buffers sized to prevent stalls** — hostCh (20K) gives the DB fetcher enough runway between queries. resultCh (1K) absorbs write latency spikes. +- **S3 retry** with 6 attempts and exponential backoff handles transient 503s from the `commoncrawl` bucket. +- **Measured: 566 hosts/sec** at concurrency 500 on c5.xlarge (4 cores). Expected ~1000+ hosts/sec on c5.2xlarge (8 cores). + +### Stage 3: Icon Download +- **Network I/O-bound (internet).** Downloading from millions of different web servers worldwide. Latency varies wildly (1ms to 10s). The long tail of slow/dead servers dominates — most icons download in <500ms but timeouts (10s) hold workers. +- **The long pole of the pipeline** — longest stage at 30M scale. +- **5000 concurrent goroutines** to keep throughput high despite variable latency. Not CPU-bound (magic byte checks and SHA-256 are fast). Not DB-bound (one write per icon at ~1ms, self-smoothing due to random server latencies). +- **Memory is the concurrency limit** — each goroutine holds a TCP connection + TLS session + icon data buffer. At 5000 workers on c5.2xlarge (16GB), ~2-3GB for connection overhead — comfortable. +- **Disk I/O is negligible** — icons are small (median ~5KB), writes are sharded across directories. +- **DNS is cached** — Unbound's aggressive caching (1.7GB cache, 3600s min-TTL) means repeat TLD/nameserver lookups are instant. First-seen domains incur recursive resolution (~50-100ms) but this is pipelined with the HTTP request. +- **Measured: 439 icons/sec** at concurrency 1000 on c5.xlarge. Expected to improve significantly at 5000 concurrency on c5.2xlarge. + +### Stage 4: Best Icon Selection +- **CPU-bound (Postgres).** Single SQL query with `DISTINCT ON` and multi-column sort. Runs in seconds even at 30M — Postgres handles this efficiently with the `idx_icons_host_id` index. + +### Stage 5: Bundle Generation +- **CPU-bound (image conversion).** Decoding icons (especially ICO) and re-encoding as PNG is the bottleneck. 40 converter goroutines on c5.2xlarge (8 cores) keep all cores saturated. More goroutines don't help — they just compete for cores. +- **Disk I/O is secondary** — reading small icon files from the sharded directory. Usually cached in the OS page cache after first access. +- **S3 uploads are pipelined** — 10 upload workers hide the ~50-100ms PUT latency. The assembler serializes bundles while previous uploads are in flight. +- **DB reads are pipelined** — the fetcher goroutine prefetches pages while converters work, so workers never wait for DB. +- **Measured: 2,377 hosts/sec** at concurrency 20 on c5.xlarge (4 cores). Expected ~4500+ hosts/sec at concurrency 40 on c5.2xlarge. + +### Stage 6: Frontend Deploy +- **Network-bound.** 4 small file uploads to S3 + CloudFront invalidation. Seconds. + +### Summary: what would make each stage faster + +| Stage | Current bottleneck | To speed up further | +|-------|-------------------|-------------| +| CC-Index | Memory (DuckDB hash table spill) | Streaming dedup via INSERT ON CONFLICT, or more RAM | +| WARC parsing | CPU + S3 latency | More cores, or multiple EC2 instances | +| Icon download | Internet latency (slow/dead servers) | Multiple EC2 instances | +| Bundle gen | CPU (image decode/encode) | More cores, or better image libraries | +| Deploy | N/A | Already seconds | + ## DNS Architecture **Unbound** runs on the EC2 instance as the system DNS resolver. @@ -479,13 +534,13 @@ This is served publicly at `/stats.json` on the live site — interesting metada | Item | Estimate | |------|----------| -| EC2 c5.xlarge (~3-4 days) | $12-16 | -| EBS 1TB gp3 (~4 days) | $10 | -| RDS db.t3.medium (~4 days) | $4-6 | +| EC2 c5.2xlarge (~2-3 days) | $16-24 | +| EBS 1TB gp3 (~3 days) | $8 | +| RDS db.m5.large (~3 days) | $12-15 | | Common Crawl S3 reads (CC-Index + WARCs) | $0 (Open Data) | | Data transfer (icon downloads from internet, inbound) | $0 (inbound free) | -| Data transfer (backup to homelab, outbound) | $5-45 (depends on icon archive size) | -| **Total** | **~$31-77** | +| Data transfer (backup to homelab, outbound) | $5-59 (depends on icon archive size) | +| **Total** | **~$41-106** | ### Hosting Phase (Monthly Steady-State) diff --git a/PLAN.md b/PLAN.md index 3fdd40b..625cc27 100644 --- a/PLAN.md +++ b/PLAN.md @@ -606,25 +606,43 @@ Adversarial code review followed by performance improvements, validated with a 3 → 2,358 bundles, 603MB total ``` -## Phase 8: 30M Full Run (Single Machine) +## Phase 8: 30M Full Run [IN PROGRESS] -Full internet scan on one c5.xlarge. +Full internet scan. Upgraded to c5.2xlarge + db.m5.large after 300K run revealed bottlenecks. +### Infrastructure changes (mid-run) +- **EC2: c5.xlarge → c5.2xlarge** (8 vCPUs, 16GB RAM). WARC parsing was CPU-bound at 100% on 4 cores. Icon download was memory-limited at 2500 concurrent connections. Bundle gen CPU-bound. The extra cores and RAM benefit all three stages. +- **RDS: db.t3.medium → db.m5.large** (non-burstable, 2 vCPUs, 8GB RAM). The t3 burstable instance was getting CPU-throttled under sustained write load, causing the WARC parser's DB writer to stall and back-pressure workers. +- **Swap: 4GB → 8GB** (2x physical RAM as safety margin). +- **DuckDB temp_directory** set to EBS (`~/duckdb_temp`) instead of defaulting to tmpfs. DuckDB's managed spill-to-disk is far more efficient than OS swap — sequential large reads vs random 4KB page faults. + +### Tuning changes +- **WARC parser concurrency: 500 → 500** (kept, but now on 8 cores instead of 4 — more actual throughput) +- **WARC parser write batch: 100 → 500** (~2000 queries per DB round-trip). Fewer flushes = less back-pressure on workers. +- **WARC parser startup**: removed slow `COUNT(*) WHERE parsed = FALSE` query (scans 26M-row index, takes minutes). Not needed — fetcher discovers empty results naturally. +- **WARC parser channel buffers**: hostCh 5K → 20K, resultCh 500 → 1K. Prevents micro-stalls between DB fetcher queries. +- **Icon download concurrency: 1000 → 5000** (16GB RAM supports the connection overhead). +- **Icon download channel buffer**: 5K → 20K. +- **Bundle gen concurrency: 20 → 40** (8 cores × 5). +- **Bundle gen channel buffers**: 1.2K → 6K. +- **Debug logging** added to WARC parser fetcher and writer to diagnose stalls. + +### Run parameters - `--limit 0` for CC-Index query -- **Expected (extrapolated from 300K run):** - - CC-Index: ~10min (download) + ~15min (query) — possibly much longer at 30M due to swap thrashing - - WARC parsing: ~14-15hrs (566 hosts/sec) - - Icon download: ~50-60hrs (439 icons/sec at 1000 concurrency, the long pole — 2500 concurrency may improve this) - - Bundle gen: ~3.5hrs (2,377 hosts/sec) - - **Total: ~3 days** +- **CC-Index result:** 26,703,146 hosts (20.6M https, 6.1M http) - Run in tmux, monitor with `psql` queries from another session -- **Expected disk:** ~650GB for all icons (6.5GB per 300K × 100) -- **Cost:** ~$50 (EC2 + RDS + 1TB EBS for 3-4 days) +- **Expected disk:** ~650GB for all icons +- **Expected time:** ~2 days (with upgraded infra, down from ~3 days estimate) +- **Cost:** ~$65 (EC2 c5.2xlarge + RDS m5.large + 1TB EBS for 2-3 days) - After completion: deploy frontend, verify live site, backup icons + DB to homelab via rsync - **Stuck icon recovery** (if icon download crashes): `UPDATE icons SET scan_state = 'unscanned' WHERE scan_state = 'in_progress';` -### Consider c5.2xlarge for future runs -The CC-Index DuckDB query is memory-bound — at 30M the GROUP BY hash table exceeds 8GB and swap thrashing dominates query time. c5.2xlarge (16GB, 8 vCPUs) would eliminate swap pressure entirely and double CPU cores for bundle gen. Cost difference: $0.17/hr → $0.34/hr, but if it halves the CC-Index query time and speeds up bundle gen (CPU-bound), the total EC2 hours may decrease enough to break even. Also benefits WARC parsing (more headroom for 500+ goroutines) and icon download (more memory for 5000 concurrent connections). Worth testing on a future run. +### Lessons learned (during 30M run) +- **Burstable DB instances are unsuitable for pipeline workloads.** The t3.medium throttled under sustained writes, causing stalls that propagated through the entire WARC parser pipeline via channel back-pressure. Non-burstable m5 instances provide consistent performance. +- **WARC parsing is CPU-bound, not just I/O-bound.** At 500 goroutines on 4 cores, CPU was at 100% — TLS handshakes + gzip decompression + HTML parsing add up. More cores directly increases throughput. +- **Channel buffer sizing matters.** Small buffers (5K) caused micro-stalls every time the DB fetcher ran a query. 20K buffers give the fetcher enough runway to query without starving workers. +- **DuckDB temp_directory is critical at scale.** Without it, DuckDB spills to tmpfs (RAM-backed), which then swaps to disk via the OS — double indirection. Pointing temp_directory at EBS lets DuckDB manage spill efficiently with large sequential I/O. +- **COUNT(*) on large partial indexes is expensive.** The startup query `SELECT COUNT(*) FROM hosts WHERE parsed = FALSE` on 26M rows took minutes. Unnecessary — just start processing and discover completion naturally. ## Phase 9: Frontend Polish