updated PLAN.md and ARCHITECTURE.md with new instance type and performance concerns
This commit is contained in:
parent
b419b5bf6c
commit
baf657a8ed
2 changed files with 98 additions and 25 deletions
|
|
@ -66,8 +66,8 @@ All resources in **us-east-1**.
|
|||
|
||||
| Resource | Purpose | Lifecycle |
|
||||
|----------|---------|-----------|
|
||||
| EC2 (c5.xlarge) + 1TB EBS | Run all pipeline stages, store icon archive | Scanning only |
|
||||
| RDS Postgres (db.t3.medium) | Store hosts/icons metadata | Scanning only (backup to homelab, then delete) |
|
||||
| EC2 (c5.2xlarge) + 1TB EBS | Run all pipeline stages, store icon archive | Scanning only |
|
||||
| RDS Postgres (db.m5.large) | Store hosts/icons metadata | Scanning only (backup to homelab, then delete) |
|
||||
| S3 `everytab-site` | Static site: index.html, site.js, tabs/*.json | Permanent |
|
||||
| CloudFront | CDN for static site (Brotli compression enabled) | Permanent |
|
||||
| S3 `everytab-logs` | CloudFront access logs | Permanent |
|
||||
|
|
@ -117,6 +117,7 @@ Icons are stored on local disk during scanning, not S3. The EBS volume holds the
|
|||
| s3_key | TEXT | SHA-256 hash of content (used as local file path, legacy column name) |
|
||||
| scan_state | TEXT DEFAULT 'unscanned' | `unscanned`, `in_progress`, `completed`, `failed` |
|
||||
| error | TEXT | Error message if failed |
|
||||
| downloaded_at | TIMESTAMPTZ | When the icon was fetched (NULL if not yet downloaded) |
|
||||
|
||||
**Indexes:**
|
||||
- `CREATE INDEX idx_icons_unscanned ON icons(id) WHERE scan_state = 'unscanned'` — partial index for work claiming. Only indexes unscanned rows; shrinks as work completes. Minimal write overhead since index only updates on transition OUT of 'unscanned'.
|
||||
|
|
@ -199,17 +200,23 @@ WHERE url_path = '/'
|
|||
7. Insert all discovered `link rel="icon"` entries into `icons` (any format: ICO, PNG, GIF, SVG, WebP, JPEG)
|
||||
8. Update `hosts` row: html_title, iframe_allowed, parsed = TRUE
|
||||
|
||||
**Concurrency:** High — thousands of goroutines with a semaphore/pool. CC's S3 handles massive throughput.
|
||||
**Architecture:** Three-stage pipeline:
|
||||
|
||||
**Error handling:** Malformed HTML → still extract what we can (partial title, partial icons). WARC fetch failure → log and skip (mark parsed = TRUE with NULL title to avoid retry loops). All errors logged with hostname for investigation.
|
||||
```
|
||||
[DB fetcher] → hostCh → [500 workers] → resultCh → [DB writer with pgx.Batch]
|
||||
```
|
||||
|
||||
1. **DB fetcher** (1 goroutine): continuously pages through unparsed hosts (batch size 5000), feeds `hostCh`.
|
||||
2. **Workers** (500 goroutines, configurable): fetch WARC from S3, parse HTML, update stats, send successful results to `resultCh`. I/O-bound on S3 latency.
|
||||
3. **DB writer** (1 goroutine): collects results, flushes every 100 using `pgx.Batch` (~400 queries per DB round-trip). S3 retry with 6 attempts and exponential backoff for transient 503s.
|
||||
|
||||
**Error handling:** Malformed HTML → still extract what we can (partial title, partial icons). WARC fetch failure → log and skip (host stays `parsed = FALSE`, retryable on next run). Max 50 link_rel icons per host (defensive cap against adversarial pages).
|
||||
|
||||
**Icon URL handling:** Relative URLs resolved against `{protocol}://{hostname}/`. Absolute URLs kept as-is. Data URIs ignored.
|
||||
|
||||
**No scan_state needed:** CC's S3 is highly reliable. The `parsed` boolean is sufficient. If the process crashes mid-batch, re-run picks up where it left off (unparsed rows).
|
||||
|
||||
**Cost:** $0 (same Open Data program).
|
||||
|
||||
**Stats emitted:** Rows processed, titles extracted, icons found (by source: favicon_ico vs link_rel), icon format distribution, iframe restrictions found, parse failures, rows with no title.
|
||||
**Stats emitted:** Rows processed, titles extracted, no-title count, icons found, iframe restrictions, fetch/parse errors, DB errors, panics.
|
||||
|
||||
### Stage 3: Icon Download
|
||||
|
||||
|
|
@ -247,7 +254,7 @@ WHERE url_path = '/'
|
|||
- Update icons row: s3_key (the SHA-256 hash), content_type (from actual data, not HTTP header), width, height, file_size, scan_state = 'completed'
|
||||
- On failure: scan_state = 'failed', error = reason
|
||||
|
||||
**Concurrency:** Channel-based worker pool (default 200 workers, configurable). Producer goroutine feeds a buffered channel (buffer = batch size), N workers consume. No starvation between batch claims.
|
||||
**Concurrency:** Channel-based worker pool (default 2500 workers, configurable). Producer goroutine feeds a buffered channel (buffer = batch size), shuffles each batch to avoid hitting the same host back-to-back. N workers consume from the channel.
|
||||
|
||||
**Fast failure strategy:**
|
||||
- DNS failure → fail immediately (Unbound will cache NXDOMAIN)
|
||||
|
|
@ -336,6 +343,54 @@ Bundles are written in-place (overwriting previous run). No delete-first step, s
|
|||
3. **Verify backups:** confirm pg_dump restores cleanly on homelab, spot-check icon files
|
||||
4. Tear down scanning infra: `terraform apply -var="scanning=false"` (deletes RDS, EC2, icons S3 bucket)
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
Each pipeline stage has different bottlenecks. Understanding these explains the concurrency choices and why certain stages can't be sped up further on a single machine.
|
||||
|
||||
### Stage 1: CC-Index Query
|
||||
- **Download phase: network-bound.** `aws s3 sync` of ~166GB of parquet files. Throughput limited by EC2 network bandwidth (10 Gbps on c5.2xlarge). Takes ~10-15 minutes.
|
||||
- **Query phase: memory-bound.** DuckDB loads the GROUP BY hash table into memory. At 30M output rows, the hash table approaches 16GB. `temp_directory` is set to EBS so DuckDB spills to NVMe efficiently (large sequential I/O) rather than relying on OS swap (random 4KB page faults). On c5.2xlarge (16GB RAM) with 8GB swap, the query completes without severe thrashing.
|
||||
- **Not CPU-bound** — DuckDB's columnar scan is efficient, CPU cores are underutilized during the query.
|
||||
|
||||
### Stage 2: WARC Parsing
|
||||
- **CPU-bound + network I/O-bound (S3).** Each WARC fetch is a byte-range S3 GetObject request (~100-200ms round-trip), but TLS handshakes + gzip decompression + HTML parsing consume significant CPU. At 500 goroutines on 4 cores, CPU was at 100%. On c5.2xlarge (8 cores), more workers can actually compute simultaneously.
|
||||
- **DB writes batched** via `pgx.Batch` — 500 results (~2000 queries) per round-trip. Non-burstable RDS (db.m5.large) provides consistent write performance. Burstable t3 instances throttle under sustained load and cause pipeline stalls via channel back-pressure.
|
||||
- **Channel buffers sized to prevent stalls** — hostCh (20K) gives the DB fetcher enough runway between queries. resultCh (1K) absorbs write latency spikes.
|
||||
- **S3 retry** with 6 attempts and exponential backoff handles transient 503s from the `commoncrawl` bucket.
|
||||
- **Measured: 566 hosts/sec** at concurrency 500 on c5.xlarge (4 cores). Expected ~1000+ hosts/sec on c5.2xlarge (8 cores).
|
||||
|
||||
### Stage 3: Icon Download
|
||||
- **Network I/O-bound (internet).** Downloading from millions of different web servers worldwide. Latency varies wildly (1ms to 10s). The long tail of slow/dead servers dominates — most icons download in <500ms but timeouts (10s) hold workers.
|
||||
- **The long pole of the pipeline** — longest stage at 30M scale.
|
||||
- **5000 concurrent goroutines** to keep throughput high despite variable latency. Not CPU-bound (magic byte checks and SHA-256 are fast). Not DB-bound (one write per icon at ~1ms, self-smoothing due to random server latencies).
|
||||
- **Memory is the concurrency limit** — each goroutine holds a TCP connection + TLS session + icon data buffer. At 5000 workers on c5.2xlarge (16GB), ~2-3GB for connection overhead — comfortable.
|
||||
- **Disk I/O is negligible** — icons are small (median ~5KB), writes are sharded across directories.
|
||||
- **DNS is cached** — Unbound's aggressive caching (1.7GB cache, 3600s min-TTL) means repeat TLD/nameserver lookups are instant. First-seen domains incur recursive resolution (~50-100ms) but this is pipelined with the HTTP request.
|
||||
- **Measured: 439 icons/sec** at concurrency 1000 on c5.xlarge. Expected to improve significantly at 5000 concurrency on c5.2xlarge.
|
||||
|
||||
### Stage 4: Best Icon Selection
|
||||
- **CPU-bound (Postgres).** Single SQL query with `DISTINCT ON` and multi-column sort. Runs in seconds even at 30M — Postgres handles this efficiently with the `idx_icons_host_id` index.
|
||||
|
||||
### Stage 5: Bundle Generation
|
||||
- **CPU-bound (image conversion).** Decoding icons (especially ICO) and re-encoding as PNG is the bottleneck. 40 converter goroutines on c5.2xlarge (8 cores) keep all cores saturated. More goroutines don't help — they just compete for cores.
|
||||
- **Disk I/O is secondary** — reading small icon files from the sharded directory. Usually cached in the OS page cache after first access.
|
||||
- **S3 uploads are pipelined** — 10 upload workers hide the ~50-100ms PUT latency. The assembler serializes bundles while previous uploads are in flight.
|
||||
- **DB reads are pipelined** — the fetcher goroutine prefetches pages while converters work, so workers never wait for DB.
|
||||
- **Measured: 2,377 hosts/sec** at concurrency 20 on c5.xlarge (4 cores). Expected ~4500+ hosts/sec at concurrency 40 on c5.2xlarge.
|
||||
|
||||
### Stage 6: Frontend Deploy
|
||||
- **Network-bound.** 4 small file uploads to S3 + CloudFront invalidation. Seconds.
|
||||
|
||||
### Summary: what would make each stage faster
|
||||
|
||||
| Stage | Current bottleneck | To speed up further |
|
||||
|-------|-------------------|-------------|
|
||||
| CC-Index | Memory (DuckDB hash table spill) | Streaming dedup via INSERT ON CONFLICT, or more RAM |
|
||||
| WARC parsing | CPU + S3 latency | More cores, or multiple EC2 instances |
|
||||
| Icon download | Internet latency (slow/dead servers) | Multiple EC2 instances |
|
||||
| Bundle gen | CPU (image decode/encode) | More cores, or better image libraries |
|
||||
| Deploy | N/A | Already seconds |
|
||||
|
||||
## DNS Architecture
|
||||
|
||||
**Unbound** runs on the EC2 instance as the system DNS resolver.
|
||||
|
|
@ -479,13 +534,13 @@ This is served publicly at `/stats.json` on the live site — interesting metada
|
|||
|
||||
| Item | Estimate |
|
||||
|------|----------|
|
||||
| EC2 c5.xlarge (~3-4 days) | $12-16 |
|
||||
| EBS 1TB gp3 (~4 days) | $10 |
|
||||
| RDS db.t3.medium (~4 days) | $4-6 |
|
||||
| EC2 c5.2xlarge (~2-3 days) | $16-24 |
|
||||
| EBS 1TB gp3 (~3 days) | $8 |
|
||||
| RDS db.m5.large (~3 days) | $12-15 |
|
||||
| Common Crawl S3 reads (CC-Index + WARCs) | $0 (Open Data) |
|
||||
| Data transfer (icon downloads from internet, inbound) | $0 (inbound free) |
|
||||
| Data transfer (backup to homelab, outbound) | $5-45 (depends on icon archive size) |
|
||||
| **Total** | **~$31-77** |
|
||||
| Data transfer (backup to homelab, outbound) | $5-59 (depends on icon archive size) |
|
||||
| **Total** | **~$41-106** |
|
||||
|
||||
### Hosting Phase (Monthly Steady-State)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue