Commit graph

Select branches

Hide pull requests

main

Mono Color

• 81f71bb354 don't show sites that don't allow embeddings main Joe Lothan 2026-06-02 01:22:20 -04:00
• 028f30cb77 even simpler about Joe Lothan 2026-05-28 09:25:47 -04:00
• 6a9436ece7 updated about Joe Lothan 2026-05-28 09:21:23 -04:00
• a60597157d auto open and size about page Joe Lothan 2026-05-27 22:17:37 -04:00
• bc06e79c6c fix horizontal scroll on mobile Joe Lothan 2026-05-27 20:48:06 -04:00
• b5787fae94 long url truncate with ... on mobile Joe Lothan 2026-05-27 11:36:04 -04:00
• 0b662019f3 no grey bar beneath the iframe Joe Lothan 2026-05-27 11:31:31 -04:00
• 2fb0e6706f auto open about tab at start Joe Lothan 2026-05-27 11:22:26 -04:00
• 758ab3080b interleve no icon hosts and icons hosts for an even mix Joe Lothan 2026-05-26 23:50:02 -04:00
• a799c05f81 order by downloaded time to improve ebs read performance Joe Lothan 2026-05-26 23:10:53 -04:00
• df2eaa251c shuffle bundle generation using a bucket approach in the assembler Joe Lothan 2026-05-26 09:11:15 -04:00
• 4b09a3219a added iotop and iftop for monitoring Joe Lothan 2026-05-26 09:10:51 -04:00
• 79881fce4b add one run.sh for the entire pipeline Joe Lothan 2026-05-26 02:09:57 -04:00
• 767083fb5e increase readahead to improve page cache use Joe Lothan 2026-05-26 02:07:38 -04:00
• 02437a677c updated mobile to show title above Joe Lothan 2026-05-26 02:02:21 -04:00
• 5d80e38249 removed shuffling of hosts to keep hostid for better bundling Joe Lothan 2026-05-26 01:42:07 -04:00
• 1343df1a41 revert to using host id ordering for bundle generation Joe Lothan 2026-05-26 01:33:14 -04:00
• 9266c1417d more iops cheese testing - sort by downloaded_at not host id Joe Lothan 2026-05-26 01:09:32 -04:00
• a30fe104a3 sort by host id to test iops cheese to improve icon read performance Joe Lothan 2026-05-26 00:43:11 -04:00
• 03f6406e18 added windows styling Joe Lothan 2026-05-26 00:04:01 -04:00
• 54ea6a662c added safari on mac Joe Lothan 2026-05-26 00:01:33 -04:00
• 53205ab661 bumped padding of bundles to 6 digits Joe Lothan 2026-05-25 23:56:13 -04:00
• 59d4d2c9e0 title goes above address bar on mobile Joe Lothan 2026-05-25 23:50:58 -04:00
• 2256055632 faster deploy to clean stale bundles, reading the live site Joe Lothan 2026-05-25 23:50:35 -04:00
• 6649c09d12 longer cache for the tabs/ s3 bucket Joe Lothan 2026-05-25 23:33:49 -04:00
• 8e3907505f deploy frontend from the ec2 at the end of the pipeline Joe Lothan 2026-05-25 23:21:50 -04:00
• 8d62832c1d tmux and htop on the db for performance monitoring Joe Lothan 2026-05-25 23:09:40 -04:00
• 3ea88790b5 frontend now looks like firefox and chrome tabs on linux Joe Lothan 2026-05-25 22:53:11 -04:00
• 7c4572aafb EVEN MORE CONCURRENCYYYY Joe Lothan 2026-05-25 22:00:53 -04:00
• 6c64ffcf94 upped concurrent s3 requests to speed up cc-index download Joe Lothan 2026-05-25 21:53:57 -04:00
• 4bfe165fac update infra README for cloud init Joe Lothan 2026-05-25 21:43:31 -04:00
• a92c838d23 fix ec2 provisioning for cloud init Joe Lothan 2026-05-25 21:43:12 -04:00
• 7c5573c24d add filter to squash warning Joe Lothan 2026-05-25 21:40:01 -04:00
• 33bd0a221e updated s3_key name to icon_hash Joe Lothan 2026-05-25 21:05:26 -04:00
• e308718eb2 remove icon s3 bucket, add log retention policy, make logs private explicitly Joe Lothan 2026-05-25 20:57:11 -04:00
• 8c005c4f6c two phase best icon selection with a temporary table Joe Lothan 2026-05-25 20:55:29 -04:00
• a819dabb57 updated number of async writers to warc_parse to accomidate faster db nvme write speeds Joe Lothan 2026-05-25 19:41:25 -04:00
• bfb7d8f883 updated gitignore Joe Lothan 2026-05-25 19:40:42 -04:00
• 1afbc41599 automated ec2 setup and build Joe Lothan 2026-05-25 18:29:37 -04:00
• bf8b932cdc switched from rds to i5 ec2 for nvme disk read/write speeds Joe Lothan 2026-05-25 18:17:07 -04:00
• c93d1736fe tune unbound to take up less memory for our use case Joe Lothan 2026-05-25 17:30:29 -04:00
• cb8d23842c better gated firefox specific code Joe Lothan 2026-05-25 17:18:28 -04:00
• 8ceb31bcbb increase bundlegen producer host amount to ensure workers aren't starved Joe Lothan 2026-05-25 16:21:49 -04:00
• 4c7a0f54f7 disable keepalives so connections stop after data transfer complete Joe Lothan 2026-05-25 16:20:31 -04:00
• ca90b7071e optimize db for bulk insert by turning off indexes and vacuum Joe Lothan 2026-05-25 14:16:40 -04:00
• eec486880a about everytab tab bolded title Joe Lothan 2026-05-21 01:02:08 -04:00
• 0f0acb642f fixed firefox marquee rollover flicker Joe Lothan 2026-05-21 00:56:50 -04:00
• fe3d5f7039 speed is fixed, no longer dependent on viewport width Joe Lothan 2026-05-21 00:44:00 -04:00
• b53fd7844b smoother marquee and fixed icon jitter/stutter in firefox Joe Lothan 2026-05-21 00:38:33 -04:00
• 4fa40c7b47 improved write efficency, though we are still bottlenecking on RDS - will switch to local postgres for future runs Joe Lothan 2026-05-20 22:38:23 -04:00
• baf657a8ed updated PLAN.md and ARCHITECTURE.md with new instance type and performance concerns Joe Lothan 2026-05-20 13:17:03 -04:00
• b419b5bf6c updated plan.md after 3M test Joe Lothan 2026-05-20 13:14:06 -04:00
• 8dce702e8d upped buffer sizes and switched to 2xlarge to increase speed Joe Lothan 2026-05-20 12:59:12 -04:00
• 1df9a234cf updated pipeline README to use compression and new flow Joe Lothan 2026-05-20 11:54:48 -04:00
• 6352b9253f upped swap to 8G Joe Lothan 2026-05-20 11:54:17 -04:00
• 024e0513ba upped icon downloading concurrency Joe Lothan 2026-05-20 11:00:17 -04:00
• 91f48f249a 1T for ec2 hd Joe Lothan 2026-05-20 10:19:02 -04:00
• ead6366ed0 up ulimit for more connection Joe Lothan 2026-05-20 10:18:48 -04:00
• 6d8ba61102 update warc parsing with new 3 stage producer, worker, consumer model, increasing speed and saturating cores Joe Lothan 2026-05-20 10:18:15 -04:00
• 0efec72e45 print every 100 bundles Joe Lothan 2026-05-20 10:17:35 -04:00
• 426abe1c90 upped concurrency of icon downloading Joe Lothan 2026-05-20 09:47:18 -04:00
• 3bc355e503 improved bundle cli output with progress Joe Lothan 2026-05-20 09:46:59 -04:00
• 86cff37533 download cc-index to home not tmp (which is tmpfs) Joe Lothan 2026-05-20 09:35:06 -04:00
• 9308b5e039 download cc-index first with aws cli instead of streaming it Joe Lothan 2026-05-20 08:14:22 -04:00
• 564919c5cc added downloaded_at timestamp to icon table Joe Lothan 2026-05-20 01:35:13 -04:00
• ec33b2e857 bump up s3 warc retries to 6 to avoid 503 errors Joe Lothan 2026-05-20 01:30:46 -04:00
• 081866f62e update bundle gen to use channels and goroutines to saturate disk and not block on db access + bundle coalesing and uploading Joe Lothan 2026-05-20 01:28:52 -04:00
• 902928235c updated best icon selection logic Joe Lothan 2026-05-20 01:15:08 -04:00
• 03e343a136 cap number of favicons to 50 per host Joe Lothan 2026-05-20 00:53:24 -04:00
• cd896427eb shuffle icon link batches before putting them in the channel Joe Lothan 2026-05-20 00:50:40 -04:00
• 27203ff085 updated bot rate Joe Lothan 2026-05-20 00:50:17 -04:00
• 963d9209ca cleaner dns error handling Joe Lothan 2026-05-20 00:35:55 -04:00
• c9ea462e97 check all CSP headers for iframe disallowing Joe Lothan 2026-05-20 00:32:56 -04:00
• a8177a1583 improve stats generation Joe Lothan 2026-05-20 00:31:38 -04:00
• 0c9ad5bfd6 count iframes only if there isn't an error Joe Lothan 2026-05-20 00:29:28 -04:00
• 3264288752 capped random favicons for frontend at 100 Joe Lothan 2026-05-20 00:17:12 -04:00
• 56ae26cbef added bmp decoder to bundler Joe Lothan 2026-05-20 00:11:53 -04:00
• 7d24b406aa redundant min Joe Lothan 2026-05-20 00:10:04 -04:00
• eb40995c60 just overwrite bundles, don't delete then re-add Joe Lothan 2026-05-20 00:09:53 -04:00
• d6ef34a1dc go mod tidy Joe Lothan 2026-05-20 00:07:48 -04:00
• 258c6c5f3a updated ARCHITECTURE.md Joe Lothan 2026-05-19 23:46:06 -04:00
• 2f1547a912 switched bundle host field to url to retain http Joe Lothan 2026-05-19 23:38:14 -04:00
• 7f36e99443 updated random value to double precision float Joe Lothan 2026-05-19 23:37:50 -04:00
• 41c0eb5c49 updated PLAN.md with another 3M run to test code changes Joe Lothan 2026-05-19 13:42:19 -04:00
• a28cd2b056 updated pipeline README Joe Lothan 2026-05-19 13:06:48 -04:00
• 1b665d1065 it's web, not Web according to APA style guides Joe Lothan 2026-05-19 12:25:45 -04:00
• 4ceefeec3d fixed typo Joe Lothan 2026-05-19 11:50:49 -04:00
• f7f564289c drop the space, it's cleaner Joe Lothan 2026-05-19 11:49:13 -04:00
• 3534f84b27 added about.html Joe Lothan 2026-05-19 11:42:09 -04:00
• 1d5b7bd374 added random_order to host table schema Joe Lothan 2026-05-19 10:47:05 -04:00
• e6d5d5175c fixed oom in bundle_gen and added randomOrder, still need a full redesign Joe Lothan 2026-05-19 10:46:40 -04:00
• cf17fc42b1 fixed icon downloading performance issues Joe Lothan 2026-05-19 10:32:34 -04:00
• 2745e75408 updated infra README about pinning AMI Joe Lothan 2026-05-19 10:23:29 -04:00
• 2f4e5b585d updated PLAN.md for future plans Joe Lothan 2026-05-19 08:34:42 -04:00
• 85b663a6e8 added logging to cloudfront Joe Lothan 2026-05-18 13:42:49 -04:00
• c7e33defa2 updated plan.md, finished integration test Joe Lothan 2026-05-18 12:49:34 -04:00
• 5b3f6a6870 switched from s3 to disk for saving icons Joe Lothan 2026-05-18 12:43:50 -04:00
• 113a261dae updated duckdb and added a swap file Joe Lothan 2026-05-18 02:10:15 -04:00
• 4436f43c6f force destroy bucket with icons Joe Lothan 2026-05-18 01:21:03 -04:00
• ddeb8bc504 fix TOTAL_BUNDLES sed command in deploy script Joe Lothan 2026-05-18 01:00:09 -04:00