everytab/infra

Infrastructure Setup

Architecture

Two EC2 instances during scanning:

• c5.2xlarge (everytab) — compute: runs pipeline, stores icons on 1TB EBS
• i3.large (everytab-db) — database: runs Postgres on 475GB local NVMe (100K+ IOPS)

Both provisioned by Terraform with user_data scripts that auto-run on first boot:

• Compute: ec2-userdata.sh — installs Go, DuckDB, Unbound, swap; clones repo; builds binaries; applies DB schema
• Database: db-setup.sh — formats NVMe, installs Postgres, creates database + schema

Quick Start

Everything runs from your local machine unless noted.

# 1. Create infrastructure
cd infra
cp terraform.tfvars.example terraform.tfvars  # fill in your values (including repo_url)
terraform init
terraform apply

# 2. Save SSH key
terraform output -raw ssh_private_key > everytab-key && chmod 600 everytab-key

# 3. Wait ~3-5 minutes for both instances to auto-provision, then verify
ssh -i everytab-key ec2-user@$(terraform output -raw ec2_public_ip) \
  'pg_isready -h $(grep DATABASE_URL ~/.bashrc | cut -d@ -f2 | cut -d: -f1)'

If repo_url is set in tfvars, the compute instance automatically:

• Clones the repo
• Builds all Go binaries
• Waits for the DB to be ready
• Applies the schema

Running the Pipeline

SSH to the compute instance — everything is ready:

ssh -i everytab-key ec2-user@$(terraform output -raw ec2_public_ip)

# DATABASE_URL is already in .bashrc, binaries already built
# Start the pipeline (see pipeline/README.md for full guide)
./pipeline/01_cc_index/query.sh --db-url "$DATABASE_URL" --limit 0

Debugging (if auto-provision fails)

Check cloud-init logs on either instance:

# Compute instance
ssh -i everytab-key ec2-user@$(terraform output -raw ec2_public_ip) \
  'tail -30 /var/log/cloud-init-output.log'

# DB instance
ssh -i everytab-key ec2-user@$(terraform output -raw db_public_ip) \
  'tail -30 /var/log/cloud-init-output.log'

Pinning the EC2 AMI

The data.aws_ami lookup fetches the latest Amazon Linux 2023 AMI. Pin it to prevent instance replacement on unrelated changes:

aws ec2 describe-instances --filters "Name=tag:Name,Values=everytab" \
  --query "Reservations[0].Instances[0].ImageId" --output text

# Add to terraform.tfvars
echo 'ec2_ami = "ami-XXXXXXXXXXXX"' >> terraform.tfvars

Remove the line when you want fresh instances with the latest AMI.

Teardown

From the compute instance, back up before tearing down:

# Back up database
pg_dump $DATABASE_URL -Fc > ~/everytab_dump.pgfc

# Back up icons to homelab
rsync -avP ~/icons/ homelab:/backups/everytab/icons/

From your local machine:

# Destroy scanning infrastructure (keeps CloudFront + site bucket)
terraform apply -var="scanning=false"

# Or full destroy (including the live site)
terraform destroy

IMPORTANT: The i3's local NVMe is ephemeral — all data is lost on stop/terminate. Always pg_dump before teardown.