joe/everytab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

updated PLAN.md, finished phase 5

Browse source
This commit is contained in:
Joe Lothan 2026-05-18 00:26:50 -04:00
parent 4963866427
commit e5035d9a28
1 changed files with 47 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

115
PLAN.md
View file

@ -213,80 +213,36 @@ Binary: `pipeline/05_bundle_gen/` (6 files: main.go, bundle.go, convert.go, db.g
--- ---
## Phase 5: Frontend (Stage 6) ## Phase 5: Frontend (Stage 6) [COMPLETED — v1]
Begins after Phase 4 is complete — we use real bundle data from the 100K pipeline run for frontend development. ### Steps 5.1-5.6 [COMPLETED]
### Step 5.1: Local Dev Server Files: `frontend/index.html` and `frontend/site.js`
Serve the generated bundles from S3 locally for frontend development: **Architecture:**
- Vanilla JS, no framework. Two files: HTML (with inline CSS) + JS.
- Fetches random bundle JSONs from `tabs/{N}.json`, renders tabs as rows filling the viewport.
- Seeded PRNG (`Date.now()` + mulberry32) — every visitor sees unique tab arrangement.
- Infinite scroll: loads more bundles as user approaches the bottom.
- Tracks loaded bundle IDs in a Set to avoid duplicates.
```bash **Tab rendering:**
# Sync a few bundles locally for testing - Browser-specific tab styling via `navigator.userAgent` detection (Chrome, Firefox, Safari).
aws s3 sync s3://everytab-site/tabs/ ./local-tabs/ --max-items 10 - Inactive tab appearance by default, selected/active style when iframe is open.
# Serve with any static file server - Light mode default, auto-switches to dark mode via `prefers-color-scheme`.
python -m http.server 8000 - Bidirectional marquee: each row randomly scrolls left or right at different speeds (90-150s per cycle).
``` - Tabs duplicated in DOM for seamless marquee loop (`translateX(-50%)`).
- Hover shows full title as native tooltip.
- External link indicator (↗) on tabs that don't allow iframes.
**Done when:** Can fetch real bundle JSON from a local dev server. **Iframe viewer:**
- Inline, not overlay — opens between tab rows, pushes content down (75vh height).
- Header shows favicon, title, external link, and close button.
- Sandboxed iframe (`allow-scripts allow-same-origin allow-forms`).
- Close via X button, Escape key.
- Only one viewer open at a time.
### Step 5.2: Basic Tab Rendering **`TOTAL_BUNDLES`** baked into HTML at build time. Build script (`pipeline/06_frontend/build.sh`) still TODO — currently hardcoded.
Build `frontend/index.html` and `frontend/site.js`:
1. HTML: minimal shell with a container div, inline CSS for tab styling
2. JS: fetch a bundle, render tabs as rows filling the viewport
3. Tab appearance: mimic Firefox tab shape (rounded top corners, slight border)
4. Each tab shows favicon (16x16 or 32x32 img from data URI) + truncated title
5. No-icon tabs show title only
Focus: get the visual density right. How many tabs fit across? How many rows fill the viewport? This determines `ENTRIES_PER_BUNDLE`.
**Done when:** Page renders tabs from a mock bundle. Visually looks like a page full of browser tabs.
### Step 5.3: Marquee Animation
Add horizontal marquee to each row:
- CSS `@keyframes` animation, translateX
- Each row at slightly different speed and direction (some left, some right)
- Smooth, subtle movement — not distracting, just enough to feel alive
- Rows need extra tabs beyond viewport width to avoid gaps during scroll
**Done when:** Rows scroll smoothly, no visual glitches at edges.
### Step 5.4: Interaction — Click, Iframe, Close
Implement tab click behavior:
1. If `iframe_ok`: show an overlay with iframe loading the site (`{protocol}://{hostname}`)
2. If `!iframe_ok`: open in new tab (`target="_blank"`, add rel="noopener")
3. Visual indicator on tabs that will open externally (small icon/badge)
4. Close overlay: X button + click-outside + Escape key
**Done when:** Clicking tabs works correctly for both iframe and external cases.
### Step 5.5: Infinite Scroll + Random Bundle Loading
Implement:
1. Seeded PRNG using `Date.now()` — generates deterministic sequence of bundle indices
2. On page load: fetch first bundle, render
3. Scroll detection: when user approaches bottom, fetch next random bundle
4. Track loaded bundle IDs in a Set (no duplicates)
5. Append new rows below existing ones
6. Handle edge case: all bundles loaded (unlikely with 50K+ bundles but handle gracefully)
`TOTAL_BUNDLES` is a constant baked into the JS at build time.
**Done when:** Infinite scroll works, new bundles load seamlessly, no duplicate bundles.
### Step 5.6: Frontend Build Script
Write `pipeline/06_frontend/build.sh`:
1. Read total bundle count (from pipeline output or S3)
2. Inject `const TOTAL_BUNDLES = {M};` into site.js
3. Copy index.html + site.js to S3 `everytab-site/`
4. Invalidate CloudFront (if distribution exists)
**Done when:** Build script produces deployable frontend with correct bundle count.
--- ---
@ -570,10 +526,26 @@ On completion, each program prints a summary line and writes its stats JSON (wit
- Bundle sizes are very heterogeneous (39KB to 198KB) due to icon size variance. Average 216KB is well within our target. - Bundle sizes are very heterogeneous (39KB to 198KB) due to icon size variance. Average 216KB is well within our target.
- SVG favicons are ~3.5% of downloaded icons (5,128 out of 156K). Supporting SVG rasterization would recover ~1,077 hosts. Deferred to future improvement. - SVG favicons are ~3.5% of downloaded icons (5,128 out of 156K). Supporting SVG rasterization would recover ~1,077 hosts. Deferred to future improvement.
### Phase 5 — Completed 2026-05-18
**Changes from original plan:**
- Inline iframe viewer instead of full-screen overlay. Opens between tab rows, pushes content down (75vh).
- Browser-specific tab styling (Chrome/Firefox/Safari) via userAgent detection — original plan deferred this to v2.
- Light/dark mode via `prefers-color-scheme` — original plan just targeted Firefox dark theme.
- No progress bar in any Go program — per-item log lines + summary at end is the pattern across the project.
- `TOTAL_BUNDLES` hardcoded in HTML for now — build script (Step 5.6) still TODO.
**Lessons learned:**
- CSS marquee with alternating directions needs care: right-scrolling rows must start at `translateX(-50%)` and animate to `0`, not the reverse. Both directions use the same duplicated DOM structure.
- `width: max-content` on the tab row is essential — without it, flex container constrains to viewport width and percentage-based translateX is wrong.
- Tab hover expansion (removing max-width) causes layout shifts that make neighboring tabs impossible to click. Native tooltip (`tab.title`) is simpler and has no side effects.
- Hundreds of animating DOM elements cause frame drops on weaker GPUs. `will-change: transform` helps but slower animation speeds help more.
--- ---
## Future Improvements ## Future Improvements
### Pipeline
- **WARC parser: retry on fetch errors** — Currently 3 fetch errors out of 100K (tolerable loss). Could add 1 retry with backoff for transient S3 errors. - **WARC parser: retry on fetch errors** — Currently 3 fetch errors out of 100K (tolerable loss). Could add 1 retry with backoff for transient S3 errors.
- **WARC parser: batch DB inserts** — Currently one INSERT per icon. Using pgx batch or CopyFrom could improve DB write throughput and potentially unblock higher concurrency. - **WARC parser: batch DB inserts** — Currently one INSERT per icon. Using pgx batch or CopyFrom could improve DB write throughput and potentially unblock higher concurrency.
- **WARC parser: investigate throughput ceiling** — 300 hosts/sec at both 100 and 500 concurrency suggests a bottleneck. Profile to determine if it's S3 response latency, Postgres writes, or something else. For the full 30M run this determines wall-clock time (~28 hours at current rate). - **WARC parser: investigate throughput ceiling** — 300 hosts/sec at both 100 and 500 concurrency suggests a bottleneck. Profile to determine if it's S3 response latency, Postgres writes, or something else. For the full 30M run this determines wall-clock time (~28 hours at current rate).
@ -583,3 +555,10 @@ On completion, each program prints a summary line and writes its stats JSON (wit
- **Icon download: download large link_rel icons** — Currently skipping declared sizes >64x64. Re-run with broader filter for future high-res projects. - **Icon download: download large link_rel icons** — Currently skipping declared sizes >64x64. Re-run with broader filter for future high-res projects.
- **Bundle gen: SVG rasterization** — ~1,077 hosts have SVG-only favicons. Could add `rsvg-convert` or a Go SVG library to rasterize these. - **Bundle gen: SVG rasterization** — ~1,077 hosts have SVG-only favicons. Could add `rsvg-convert` or a Go SVG library to rasterize these.
- **Bundle gen: smarter downscaling** — Currently nearest-neighbor to 32x32 for >128px icons. Could use bilinear/Lanczos for better quality, or preserve aspect ratio for non-square icons. - **Bundle gen: smarter downscaling** — Currently nearest-neighbor to 32x32 for >128px icons. Could use bilinear/Lanczos for better quality, or preserve aspect ratio for non-square icons.
### Frontend
- **Performance: reduce DOM / animation cost** — Pause marquee animation on off-screen rows (IntersectionObserver). Virtualize rows to reduce total DOM element count.
- **Cross-browser tab styling** — Polish Chrome/Firefox/Safari tab appearances to more closely match real browser tabs. Test on actual browsers, use screenshots as reference.
- **Mobile layout** — Current design assumes desktop viewport. Need responsive tab sizing and touch-friendly interaction.
- **Build script** — `pipeline/06_frontend/build.sh` to inject TOTAL_BUNDLES and deploy to S3 + CloudFront invalidation.
- **Stats page** — Serve `stats.json` and render pipeline stats (host count, icon coverage, crawl date) on the site.