everytab/pipeline

Pipeline

Run these stages in order on the EC2 instance. Each stage is a single command.

Between stages, run the sanity checks to confirm data looks right before proceeding. All stages are idempotent — safe to re-run if interrupted.

Prerequisites

# Database URL in environment
export DATABASE_URL='postgres://everytab:PASS@RDS_ENDPOINT:5432/everytab'

# Schema created
psql $DATABASE_URL -f pipeline/01_cc_index/schema.sql

# Go binaries built on EC2
cd ~/everytab
go build -o ~/warc_parse ./pipeline/02_warc_parse/
go build -o ~/icon_download ./pipeline/03_icon_download/
go build -o ~/bundle_gen ./pipeline/05_bundle_gen/

Stage 1: CC-Index Query

Populates the hosts table from Common Crawl's columnar index.

./pipeline/01_cc_index/query.sh --db-url "$DATABASE_URL" --limit 100000
# Full run: --limit 0

Stage 2: WARC Parsing

Fetches WARC records from CC's S3, extracts titles, icons, and iframe headers.

./warc_parse --db "$DATABASE_URL" --log-file warc_parse.log

Stage 3: Icon Download

Downloads favicons from the live web, validates, uploads to S3.

./icon_download --db "$DATABASE_URL" --log-file icon_download.log --log-errors-only

Stage 4: Best Icon Selection

Picks the best icon per host for display.

psql $DATABASE_URL -f pipeline/04_best_icon/select.sql

Stage 5: Bundle Generation

Converts icons to PNG, assembles JSON bundles, uploads to S3.

./bundle_gen --db "$DATABASE_URL" --log-file bundle_gen.log --log-errors-only

Note the TOTAL_BUNDLES number from the summary — this gets baked into the frontend.

Stage 6: Frontend Deploy

From your local machine:

./pipeline/06_frontend/deploy.sh --total-bundles <NUMBER>

Stage 7: Backup to Homelab

After the site is deployed and verified, backup data before tearing down scanning infra.

What to backup:

Data	Location on EC2	Size estimate (30M)	Purpose
Database	RDS (pg_dump)	~5-10GB compressed	Full hosts + icons metadata, titles, WARC coordinates
Icons	~/icons/ directory	~500GB-1TB	Complete favicon archive, content-addressed by SHA-256
Stats	~/stats/*.json	<1MB	Pipeline timing and counts per stage
Logs	~/*.log	varies	Error logs for debugging

Backup commands:

# 1. Database dump (run on EC2, fast — dumps to local disk first)
pg_dump -Fc $DATABASE_URL > ~/everytab_dump.pgfc

# 2. Transfer database to homelab
rsync -avP ~/everytab_dump.pgfc homelab:/backups/everytab/

# 3. Transfer icons to homelab (this is the big one — 500GB+, will take hours)
rsync -avP ~/icons/ homelab:/backups/everytab/icons/

# 4. Transfer stats and logs
rsync -avP ~/stats/ homelab:/backups/everytab/stats/
rsync -avP ~/*.log homelab:/backups/everytab/logs/

Verify on homelab:

# Check database restores
pg_restore -d everytab_local /backups/everytab/everytab_dump.pgfc
psql everytab_local -c "SELECT COUNT(*) FROM hosts; SELECT COUNT(*) FROM icons;"

# Check icon count matches
find /backups/everytab/icons/ -type f | wc -l

Then tear down:

# From local machine
cd infra && terraform apply -var="scanning=false"

Note: The icons directory is the largest transfer. At home internet speeds (~300Mbps) transferring 500GB takes ~3-4 hours. Consider running rsync in tmux. If the transfer fails partway, rsync resumes where it left off.

Data transfer costs: AWS charges for outbound data. 500GB outbound = ~$45. For the 3M dev run (~50GB) it's ~$5. To avoid this, you could attach an EBS volume, snapshot it, and restore it in your homelab's AWS account — but rsync is simpler.

Full Reset (start over)

psql $DATABASE_URL -c "TRUNCATE hosts CASCADE;"
# Then run from Stage 1

Future: Automated Monthly Run

For automating with new Common Crawl releases, wrap stages 1-6 in a script that:

1. Truncates old data
2. Runs each stage, checking exit codes
3. Compares stats against previous run (alert if coverage drops significantly)
4. Deploys only if all stages succeed
5. Sends notification on completion or failure

The pipeline is already designed for this — each stage writes stats JSON, uses CLI flags for configuration, and is resumable on failure.