joe/everytab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

upped buffer sizes and switched to 2xlarge to increase speed

Browse source
This commit is contained in:
Joe Lothan 2026-05-20 12:59:12 -04:00
parent 1df9a234cf
commit 8dce702e8d
5 changed files with 10 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

2
pipeline/01_cc_index/query.sh
View file

@ -68,6 +68,8 @@ S3_PATH="s3://commoncrawl/cc-index/table/cc-main/warc/crawl=${CRAWL}/subset=warc
LOCAL_INDEX="$HOME/cc-index"
QUERY="
SET temp_directory = '${HOME}/duckdb_temp';
INSTALL postgres;
LOAD postgres;
ATTACH '${DB_URL}' AS pg (TYPE POSTGRES);