ArchiveBox Hook Script Concurrency & Execution Plan

Overview

Snapshot.run() should enforce that snapshot hooks are run in 10 discrete, sequential "steps": 0*, 1*, 2*, 3*, 4*, 5*, 6*, 7*, 8*, 9*.

For every discovered hook script, ArchiveBox should create an ArchiveResult in queued state, then manage running them using retry_at and inline logic to enforce this ordering.

Design Decisions

ArchiveResult Schema

Add ArchiveResult.hook_name (CharField, nullable) - just filename, e.g., 'on_Snapshot__20_chrome_tab.bg.js'
Keep ArchiveResult.plugin - still important (plugin directory name)
Step number derived on-the-fly from hook_name via extract_step(hook_name) - not stored

Snapshot Schema

Add Snapshot.current_step (IntegerField 0-9, default=0)
Integrate with SnapshotMachine state transitions for step advancement

Hook Discovery & Execution

Snapshot.run() discovers all hooks upfront, creates one AR per hook with hook_name set
All ARs for a given step can be claimed and executed in parallel by workers
Workers claim ARs where extract_step(ar.hook_name) <= snapshot.current_step
Snapshot.advance_step_if_ready() increments current_step when:
- All foreground hooks in current step are finished (SUCCEEDED/FAILED/SKIPPED)
- Background hooks don't block advancement (they continue running)
- Called from SnapshotMachine state transitions

ArchiveResult.run() Behavior

If self.hook_name is set: run that single hook
If self.hook_name is None: discover all hooks for self.plugin and run sequentially
Background hooks detected by .bg. in filename (e.g., on_Snapshot__20_chrome_tab.bg.js)
Background hooks return immediately (ArchiveResult stays in STARTED state)
Foreground hooks wait for completion, update status from JSONL output

Hook Execution Flow

Within a step: Workers claim all ARs for current step in parallel
Foreground hooks (no .bg): ArchiveResult waits for completion, transitions to SUCCEEDED/FAILED/SKIPPED
Background hooks (.bg): ArchiveResult transitions to STARTED, hook continues running
Step advancement: Snapshot.advance_step_if_ready() checks:
- Are all foreground ARs in current step finished? (SUCCEEDED/FAILED/SKIPPED)
- Ignore ARs still in STARTED (background hooks)
- If yes, increment current_step
Snapshot sealing: When current_step=9 and all foreground hooks done, kill background hooks via Snapshot.cleanup()

Unnumbered Hooks

Extract step via re.search(r'__(\d{2})_', hook_name), default to 9 if no match
Log warning for unnumbered hooks
Purely runtime derivation - no stored field

Hook Numbering Convention

Hooks scripts are numbered 00 to 99 to control:

First digit (0-9): Which step they are part of
Second digit (0-9): Order within that step

Hook scripts are launched strictly sequentially based on their filename alphabetical order, and run in sets of several per step before moving on to the next step.

Naming Format:

on_{ModelName}__{run_order}_{human_readable_description}[.bg].{ext}

Examples:

on_Snapshot__00_this_would_run_first.sh
on_Snapshot__05_start_ytdlp_download.bg.sh
on_Snapshot__10_chrome_tab_opened.js
on_Snapshot__50_screenshot.js
on_Snapshot__53_media.bg.py

Background (.bg) vs Foreground Scripts

Foreground Scripts (no .bg suffix)

Launch in parallel with other hooks in their step
Step waits for all foreground hooks to complete or timeout
Get killed with SIGTERM if they exceed their PLUGINNAME_TIMEOUT
Step advances when all foreground hooks finish

Background Scripts (.bg suffix)

Launch in parallel with other hooks in their step
Do NOT block step progression - step can advance while they run
Continue running across step boundaries until complete or timeout
Get killed with SIGTERM when Snapshot transitions to SEALED (via Snapshot.cleanup())
Should exit naturally when work is complete (best case)

Important: A .bg script started in step 2 can keep running through steps 3, 4, 5... until the Snapshot seals or the hook exits naturally.

Execution Step Guidelines

These are naming conventions and guidelines, not enforced checkpoints. They provide semantic organization for plugin ordering:

Step 0: Pre-Setup

00-09: Initial setup, validation, feature detection

Step 1: Chrome Launch & Tab Creation

10-19: Browser/tab lifecycle setup
- Chrome browser launch
- Tab creation and CDP connection

Step 2: Navigation & Settlement

20-29: Page loading and settling
- Navigate to URL
- Wait for page load
- Initial response capture (responses, ssl, consolelog as .bg listeners)

Step 3: Page Adjustment

30-39: DOM manipulation before archiving
- Hide popups/banners
- Solve captchas
- Expand comments/details sections
- Inject custom CSS/JS
- Accessibility modifications

Step 4: Ready for Archiving

40-49: Final pre-archiving checks
- Verify page is fully adjusted
- Wait for any pending modifications

Step 5: DOM Extraction (Sequential, Non-BG)

50-59: Extractors that need exclusive DOM access
- singlefile (MUST NOT be .bg)
- screenshot (MUST NOT be .bg)
- pdf (MUST NOT be .bg)
- dom (MUST NOT be .bg)
- title
- headers
- readability
- mercury

These MUST run sequentially as they temporarily modify the DOM
during extraction, then revert it. Running in parallel would corrupt results.

Step 6: Post-DOM Extraction

60-69: Extractors that don't need DOM or run on downloaded files
- wget
- git
- media (.bg - can run for hours)
- gallerydl (.bg)
- forumdl (.bg)
- papersdl (.bg)

Step 7: Chrome Cleanup

70-79: Browser/tab teardown
- Close tabs
- Cleanup Chrome resources

Step 8: Post-Processing

80-89: Reprocess outputs from earlier extractors
- OCR of images
- Audio/video transcription
- URL parsing from downloaded content (rss, html, json, txt, csv, md)
- LLM analysis/summarization of outputs

Step 9: Indexing & Finalization

90-99: Save to indexes and finalize
- Index text content to Sonic/SQLite FTS
- Create symlinks
- Generate merkle trees
- Final status updates

Hook Script Interface

Input: CLI Arguments (NOT stdin)

Hooks receive configuration as CLI flags (CSV or JSON-encoded):

--url="https://example.com"
--snapshot-id="1234-5678-uuid"
--config='{"some_key": "some_value"}'
--plugins=git,media,favicon,title
--timeout=50
--enable-something

Input: Environment Variables

All configuration comes from env vars, defined in plugin_dir/config.json JSONSchema:

WGET_BINARY=/usr/bin/wget
WGET_TIMEOUT=60
WGET_USER_AGENT="Mozilla/5.0..."
WGET_EXTRA_ARGS="--no-check-certificate"
SAVE_WGET=True

Required: Every plugin must support PLUGINNAME_TIMEOUT for self-termination.

Output: Filesystem (CWD)

Hooks read/write files to:

$CWD: Their own output subdirectory (e.g., archive/snapshots/{id}/wget/)
$CWD/..: Parent directory (to read outputs from other hooks)

This allows hooks to:

Access files created by other hooks
Keep their outputs separate by default
Use semaphore files for coordination (if needed)

Output: JSONL to stdout

Hooks emit one JSONL line per database record they want to create or update:

{"type": "Tag", "name": "sci-fi"}
{"type": "ArchiveResult", "id": "1234-uuid", "status": "succeeded", "output_str": "wget/index.html"}
{"type": "Snapshot", "id": "5678-uuid", "title": "Example Page"}

See archivebox/misc/jsonl.py and model from_json() / from_jsonl() methods for full list of supported types and fields.

Output: stderr for Human Logs

Hooks should emit human-readable output or debug info to stderr. There are no guarantees this will be persisted long-term. Use stdout JSONL or filesystem for outputs that matter.

Cleanup: Delete Cruft

If hooks emit no meaningful long-term outputs, they should delete any temporary files themselves to avoid wasting space. However, the ArchiveResult DB row should be kept so we know:

It doesn't need to be retried
It isn't missing
What happened (status, error message)

Signal Handling: SIGINT/SIGTERM

Hooks are expected to listen for polite SIGINT/SIGTERM and finish hastily, then exit cleanly. Beyond that, they may be SIGKILL'd at ArchiveBox's discretion.

If hooks double-fork or spawn long-running processes: They must output a .pid file in their directory so zombies can be swept safely.

Hook Failure Modes & Retry Logic

Hooks can fail in several ways. ArchiveBox handles each differently:

1. Soft Failure (Record & Don't Retry)

Exit: 0 (success) JSONL: {"type": "ArchiveResult", "status": "failed", "output_str": "404 Not Found"}

This means: "I ran successfully, but the resource wasn't available." Don't retry this.

Use cases:

404 errors
Content not available
Feature not applicable to this URL

2. Hard Failure / Temporary Error (Retry Later)

Exit: Non-zero (1, 2, etc.) JSONL: None (or incomplete)

This means: "Something went wrong, I couldn't complete." Treat this ArchiveResult as "missing" and set retry_at for later.

Use cases:

500 server errors
Network timeouts
Binary not found / crashed
Transient errors

Behavior:

ArchiveBox sets retry_at on the ArchiveResult
Hook will be retried during next archivebox update

3. Partial Success (Update & Continue)

Exit: Non-zero JSONL: Partial records emitted before crash

Behavior:

Update ArchiveResult with whatever was emitted
Mark remaining work as "missing" with retry_at

4. Success (Record & Continue)

Exit: 0 JSONL: {"type": "ArchiveResult", "status": "succeeded", "output_str": "output/file.html"}

This is the happy path.

Error Handling Rules

DO NOT skip hooks based on failures
Continue to next hook regardless of foreground or background failures
Update ArchiveResults with whatever information is available
Set retry_at for "missing" or temporarily-failed hooks
Let background scripts continue even if foreground scripts fail

File Structure

archivebox/plugins/{plugin_name}/
├── config.json              # JSONSchema: env var config options
├── binaries.jsonl           # Runtime dependencies: apt|brew|pip|npm|env
├── on_Snapshot__XX_name.py  # Hook script (foreground)
├── on_Snapshot__XX_name.bg.py  # Hook script (background)
└── tests/
    └── test_name.py

Implementation Checklist

Phase 1: Schema Migration ✅

Add Snapshot.current_step (IntegerField 0-9, default=0)
Add ArchiveResult.hook_name (CharField, nullable) - just filename
Create migration: 0034_snapshot_current_step.py

Phase 2: Core Logic Updates ✅

Add extract_step(hook_name) utility in archivebox/hooks.py
- Extract first digit from __XX_ pattern
- Default to 9 for unnumbered hooks
Add is_background_hook(hook_name) utility in archivebox/hooks.py
- Check for .bg. in filename
Update Snapshot.create_pending_archiveresults() in archivebox/core/models.py:
- Discover all hooks (not plugins)
- Create one AR per hook with hook_name set
Update ArchiveResult.run() in archivebox/core/models.py:
- If hook_name set: run single hook
- If hook_name None: discover all plugin hooks (existing behavior)
Add Snapshot.advance_step_if_ready() method:
- Check if all foreground ARs in current step finished
- Increment current_step if ready
- Ignore background hooks (.bg) in completion check
Integrate with SnapshotMachine.is_finished() in archivebox/core/statemachines.py:
- Call advance_step_if_ready() before checking if done

Phase 3: Worker Coordination ✅

Update worker AR claiming query in archivebox/workers/worker.py:
- Filter: extract_step(ar.hook_name) <= snapshot.current_step
- Claims ARs in QUEUED state, checks step in Python before processing
- Orders by hook_name for deterministic execution within step

Phase 4: Hook Renumbering ✅

Renumber hooks per renumbering map below
Add .bg suffix to long-running hooks (media, gallerydl, forumdl, papersdl)
Move parse_* hooks to step 7 (70-79)
Test all hooks still work after renumbering

Migration Path

Natural Compatibility

No special migration needed:

Existing ARs with hook_name=None continue to work (discover all plugin hooks at runtime)
New ARs get hook_name set (single hook per AR)
ArchiveResult.run() handles both cases naturally
Unnumbered hooks default to step 9 (log warning)

Renumbering Map

Completed Renames:

# Step 5: DOM Extraction (sequential, non-background)
singlefile/on_Snapshot__37_singlefile.py      → singlefile/on_Snapshot__50_singlefile.py ✅
screenshot/on_Snapshot__34_screenshot.js      → screenshot/on_Snapshot__51_screenshot.js ✅
pdf/on_Snapshot__35_pdf.js                    → pdf/on_Snapshot__52_pdf.js ✅
dom/on_Snapshot__36_dom.js                    → dom/on_Snapshot__53_dom.js ✅
title/on_Snapshot__32_title.js                → title/on_Snapshot__54_title.js ✅
readability/on_Snapshot__52_readability.py    → readability/on_Snapshot__55_readability.py ✅
headers/on_Snapshot__33_headers.js            → headers/on_Snapshot__55_headers.js ✅
mercury/on_Snapshot__53_mercury.py            → mercury/on_Snapshot__56_mercury.py ✅
htmltotext/on_Snapshot__54_htmltotext.py      → htmltotext/on_Snapshot__57_htmltotext.py ✅

# Step 6: Post-DOM Extraction (background for long-running)
wget/on_Snapshot__50_wget.py                  → wget/on_Snapshot__61_wget.py ✅
git/on_Snapshot__12_git.py                    → git/on_Snapshot__62_git.py ✅
media/on_Snapshot__51_media.py                → media/on_Snapshot__63_media.bg.py ✅
gallerydl/on_Snapshot__52_gallerydl.py        → gallerydl/on_Snapshot__64_gallerydl.bg.py ✅
forumdl/on_Snapshot__53_forumdl.py            → forumdl/on_Snapshot__65_forumdl.bg.py ✅
papersdl/on_Snapshot__54_papersdl.py          → papersdl/on_Snapshot__66_papersdl.bg.py ✅

# Step 7: URL Extraction (parse_* hooks moved from step 6)
parse_html_urls/on_Snapshot__60_parse_html_urls.py      → parse_html_urls/on_Snapshot__70_parse_html_urls.py ✅
parse_txt_urls/on_Snapshot__62_parse_txt_urls.py        → parse_txt_urls/on_Snapshot__71_parse_txt_urls.py ✅
parse_rss_urls/on_Snapshot__61_parse_rss_urls.py        → parse_rss_urls/on_Snapshot__72_parse_rss_urls.py ✅
parse_netscape_urls/on_Snapshot__63_parse_netscape_urls.py → parse_netscape_urls/on_Snapshot__73_parse_netscape_urls.py ✅
parse_jsonl_urls/on_Snapshot__64_parse_jsonl_urls.py    → parse_jsonl_urls/on_Snapshot__74_parse_jsonl_urls.py ✅
parse_dom_outlinks/on_Snapshot__40_parse_dom_outlinks.js → parse_dom_outlinks/on_Snapshot__75_parse_dom_outlinks.js ✅

Testing Strategy

Unit Tests

Test hook ordering (00-99)
Test step grouping (first digit)
Test .bg vs foreground execution
Test timeout enforcement
Test JSONL parsing
Test failure modes & retry_at logic

Integration Tests

Test full Snapshot.run() with mixed hooks
Test .bg scripts running beyond step 99
Test zombie process cleanup
Test graceful SIGTERM handling
Test concurrent .bg script coordination

Performance Tests

Measure overhead of per-hook ArchiveResults
Test with 50+ concurrent .bg scripts
Test filesystem contention with many hooks

Open Questions

Q: Should we provide semaphore utilities?

A: No. Keep plugins decoupled. Let them use simple filesystem coordination if needed.

Q: What happens if ArchiveResult table gets huge?

A: We can delete old successful ArchiveResults periodically, or archive them to cold storage. The important data is in the filesystem outputs.

Q: Should naturally-exiting .bg scripts still be .bg?

A: Yes. The .bg suffix means "don't block step progression," not "run until step 99." Natural exit is the best case.

Examples

Foreground Hook (Sequential DOM Access)

#!/usr/bin/env python3
# archivebox/plugins/screenshot/on_Snapshot__51_screenshot.js

# Runs at step 5, blocks step progression until complete
# Gets killed if it exceeds SCREENSHOT_TIMEOUT

timeout = get_env_int('SCREENSHOT_TIMEOUT') or get_env_int('TIMEOUT', 60)

try:
    result = subprocess.run(cmd, capture_output=True, timeout=timeout)
    if result.returncode == 0:
        print(json.dumps({
            "type": "ArchiveResult",
            "status": "succeeded",
            "output_str": "screenshot.png"
        }))
        sys.exit(0)
    else:
        # Temporary failure - will be retried
        sys.exit(1)
except subprocess.TimeoutExpired:
    # Timeout - will be retried
    sys.exit(1)

Background Hook (Long-Running Download)

#!/usr/bin/env python3
# archivebox/plugins/ytdlp/on_Snapshot__63_ytdlp.bg.py

# Runs at step 6, doesn't block step progression
# Gets full YTDLP_TIMEOUT (e.g., 3600s) regardless of when step 99 completes

timeout = get_env_int('YTDLP_TIMEOUT') or get_env_int('TIMEOUT', 3600)

try:
    result = subprocess.run(['yt-dlp', url], capture_output=True, timeout=timeout)
    if result.returncode == 0:
        print(json.dumps({
            "type": "ArchiveResult",
            "status": "succeeded",
            "output_str": "media/"
        }))
        sys.exit(0)
    else:
        # Hard failure - don't retry
        print(json.dumps({
            "type": "ArchiveResult",
            "status": "failed",
            "output_str": "Video unavailable"
        }))
        sys.exit(0)  # Exit 0 to record the failure
except subprocess.TimeoutExpired:
    # Timeout - will be retried
    sys.exit(1)

Background Hook with Natural Exit

#!/usr/bin/env node
// archivebox/plugins/ssl/on_Snapshot__23_ssl.bg.js

// Sets up listener, captures SSL info, then exits naturally
// No SIGTERM handler needed - already exits when done

async function main() {
    const page = await connectToChrome();

    // Set up listener
    page.on('response', async (response) => {
        const securityDetails = response.securityDetails();
        if (securityDetails) {
            fs.writeFileSync('ssl.json', JSON.stringify(securityDetails));
        }
    });

    // Wait for navigation (done by other hook)
    await waitForNavigation();

    // Emit result
    console.log(JSON.stringify({
        type: 'ArchiveResult',
        status: 'succeeded',
        output_str: 'ssl.json'
    }));

    process.exit(0);  // Natural exit - no await indefinitely
}

main().catch(e => {
    console.error(`ERROR: ${e.message}`);
    process.exit(1);  // Will be retried
});

Summary

This plan provides:

✅ Clear execution ordering (10 steps, 00-99 numbering)
✅ Async support (.bg suffix)
✅ Independent timeout control per plugin
✅ Flexible failure handling & retry logic
✅ Streaming JSONL output for DB updates
✅ Simple filesystem-based coordination
✅ Backward compatibility during migration

The main implementation work is refactoring Snapshot.run() to enforce step ordering and manage .bg script lifecycles. Plugin renumbering is straightforward mechanical work.

TODO_hook_concurrency.md 19 KB Permalink History Raw