joe/everytab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

download cc-index to home not tmp (which is tmpfs)

Browse source
This commit is contained in:
Joe Lothan 2026-05-20 09:35:06 -04:00
parent 9308b5e039
commit 86cff37533
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
pipeline/01_cc_index/query.sh
View file

@ -65,7 +65,7 @@ if [ "$LIMIT" -gt 0 ] 2>/dev/null; then
fi fi
S3_PATH="s3://commoncrawl/cc-index/table/cc-main/warc/crawl=${CRAWL}/subset=warc/" S3_PATH="s3://commoncrawl/cc-index/table/cc-main/warc/crawl=${CRAWL}/subset=warc/"
LOCAL_INDEX="/tmp/cc-index" LOCAL_INDEX="$HOME/cc-index"
QUERY=" QUERY="
INSTALL postgres; INSTALL postgres;