No description

Joe Lothan f45e4a6034 added warc parser		2026-05-17 20:25:59 -04:00
infra	allow ec2 to access common crawl s3	2026-05-17 18:22:41 -04:00
pipeline	added warc parser	2026-05-17 20:25:59 -04:00
.gitignore	added infra setup with terraform	2026-05-17 16:07:50 -04:00
ARCHITECTURE.md	added query.sh to read the cc-index from s3 parquet files and dump it into our psql db	2026-05-17 19:12:25 -04:00
design.md	don't use zdns, just use a local unbound to make things easier	2026-05-17 12:18:47 -04:00
go.mod	added warc parser	2026-05-17 20:25:59 -04:00
go.sum	added warc parser	2026-05-17 20:25:59 -04:00
PLAN.md	added query.sh to read the cc-index from s3 parquet files and dump it into our psql db	2026-05-17 19:12:25 -04:00