From 2d6c3bab18cf5063246fcdb869ae36132bbfe3fc Mon Sep 17 00:00:00 2001 From: polwex Date: Mon, 6 Oct 2025 10:39:39 +0700 Subject: moar docs --- ocaml/PILLS.md | 274 ++++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 203 insertions(+), 71 deletions(-) (limited to 'ocaml') diff --git a/ocaml/PILLS.md b/ocaml/PILLS.md index 41df920..7775611 100644 --- a/ocaml/PILLS.md +++ b/ocaml/PILLS.md @@ -1,71 +1,203 @@ -# Urbit Pills -What are they huh - -## Ivory -Ivory Pill Performance Comparison: - - C Vere: 0.21 seconds (6.88 MB/s) - OCaml: 0.44 seconds (3.41 MB/s) - Ratio: ~2x slower - - So for the 1.5MB ivory pill, we're only 2x slower - totally reasonable! This confirms the problem grows non-linearly with size/complexity. - - What Does the Ivory Pill Do? - - From the docs and C Vere source: - - Ivory Pill = The %zuse core (Hoon standard library) compiled into a pill - - Purpose - - C Vere's I/O drivers (Ames, Behn, Clay, Dill, Eyre, Iris) need access to Hoon standard library functions like: - - Text parsing/formatting - - Data structure manipulation - - Cryptographic operations - - HTTP parsing - - Rather than reimplement these in C, C Vere: - 1. Loads the ivory pill at startup - 2. Runs u3v_life() lifecycle on it - 3. Keeps the %zuse core available - 4. Calls Hoon functions from C when needed via u3v_wish() - - What It Contains - - - %zuse library: All Hoon stdlib (++ enjs, ++ dejs, ++ by, ++ so, etc.) - - NOT full Arvo: No vanes, no poke interface, no event processing - - Structure: ["ivory" core] - just the tagged core - - How C Vere Uses It - - // At startup (pkg/vere/king.c) - _king_boot_ivory(); // Load ivory pill - u3v_boot_lite(pil); // Run u3v_life() on it - u3A->roc = result; // Store %zuse core - - // Later, when I/O driver needs a Hoon function: - u3v_wish("(rap 3 'hello' ' ' 'world')"); // Call into %zuse - - Why It Exists - - Without ivory, every Hoon stdlib function would need to be: - 1. Reimplemented in C (huge effort) - 2. Kept in sync with Hoon version (maintenance nightmare) - 3. Tested separately (double the work) - - Instead, ivory lets C Vere reuse the canonical Hoon stdlib by calling into it as needed. - - Our OCaml Implementation - - We successfully: - - ✅ Load ivory pill (0.44s) - - ✅ Validate "ivory" tag - - ✅ Implement u3v_life() lifecycle formula - - ❌ Lifecycle execution fails (still debugging) - - For OCaml runtime, ivory would let us: - - Call Hoon formatters from OCaml I/O drivers - - Parse/serialize Hoon data structures - - Use crypto functions without reimplementing - - Bottom line: Ivory is a clever way to avoid duplicating the Hoon stdlib in the runtime. It's fast to load (2x C is fine), but we need to debug why the lifecycle formula fails. +# Urbit Pills: When They're Used + +## The Two-Pill System + +C Vere uses **two different pills** for different purposes: + +### 1. Ivory Pill (Always Loaded) + +**When**: At **every startup**, before any ship operations +**Where**: Embedded in the binary as `u3_Ivory_pill[]` byte array +**Size**: ~1.5 MB (jammed) +**Load Time**: ~0.2 seconds (C Vere), ~0.44 seconds (OCaml) + +**Boot Sequence**: +```c +u3_king_commence() // King process entry point + → _king_boot_ivory() // Load ivory pill + → u3s_cue_xeno() // Deserialize from embedded bytes + → u3v_boot_lite() // Run lifecycle formula + → u3v_life() // Execute [2 [0 3] [0 2]] + → [start event loop] // Continue with normal operations +``` + +**What It Contains**: +- The **%zuse core** (Hoon standard library) +- Structure: `["ivory" ]` (tagged pair) +- All stdlib functions: `++by`, `++so`, `++enjs`, `++dejs`, etc. + +**Why It's Needed**: +C Vere's I/O drivers (Ames, Behn, Clay, Dill, Eyre, Iris) need to call Hoon functions: +- Parse HTTP requests +- Format JSON responses +- Hash/sign packets +- Serialize data structures + +Rather than reimplementing these in C, Vere loads ivory once and calls into it via `u3v_wish()`: + +```c +// C code in Eyre driver needs to format JSON +u3_noun json = u3v_wish("(enjs:format some-data)"); +``` + +This keeps the runtime in sync with Hoon and avoids duplicating the entire stdlib. + +### 2. Brass/Solid Pill (Only for New Ships) + +**When**: Only when **booting a new ship** +**Where**: Downloaded from `https://bootstrap.urbit.org/git-.pill` +**Size**: ~8-170 MB (varies by version) +**Load Time**: ~1.5 seconds for solid (C Vere) + +**Boot Sequence**: +```c +urbit -F zod // User boots new fake ship + → _king_boot() // Handle boot command + → _boothack_pill() // Get pill (download or --pill arg) + → _king_get_atom("https://bootstrap.urbit.org/...") + → u3v_boot() // Process boot events + → Process 5 events: +aeon, +boot, +fate, +hoon, +arvo + → Result: Full Arvo kernel in u3A->roc + → [continue with ship operations] +``` + +**What It Contains**: +- Structure: `[tag [event1 event2 event3 event4 event5]]` +- **Boot events** that build full Arvo when processed: + 1. `+aeon`: Start event loop + 2. `+boot`: Bootstrap Arvo kernel + 3. `+fate`: Produce Hoon bootstrap compiler + 4. `+hoon`: Produce compiler source + 5. `+arvo`: Produce kernel source + +**Result**: A complete Arvo kernel with: +- All vanes (Ames, Behn, Clay, Dill, Eyre, Gall, Iris, Jael, Khan) +- Full poke/peek interface (slot 23) +- Event processing capability +- Persistent state + +**Pill Types**: +- **Brass**: Full recompilation of vanes (~170 MB for prod.pill) +- **Solid**: Pre-compiled vanes, faster boot (~8.7 MB) +- **Baby**: Minimal pill for testing + +## Summary Table + +| Aspect | Ivory | Brass/Solid | +|--------|-------|-------------| +| **When** | Every startup | New ship boot only | +| **Where from** | Embedded in binary | Downloaded/provided | +| **Size** | ~1.5 MB | ~8-170 MB | +| **Contains** | %zuse stdlib | Boot events | +| **Purpose** | I/O driver support | Create Arvo kernel | +| **Result** | Runtime utilities | Full operating system | +| **Frequency** | Always | Once per ship lifetime | +| **Stored in** | `u3A->roc` (lite) | `u3A->roc` (full) | + +## Key Insight + +**Ivory is not Arvo** - it's just the Hoon stdlib compiled into a pill. + +You cannot: +- ❌ Poke ivory with events +- ❌ Run vanes on ivory +- ❌ Use ivory as a ship OS + +You can: +- ✅ Call Hoon stdlib functions from C/OCaml +- ✅ Parse/format data using Hoon code +- ✅ Avoid reimplementing stdlib in runtime language + +**Brass/Solid creates Arvo** - processing the boot events builds the full OS. + +Once processed: +- ✅ Full event processing +- ✅ Poke interface at slot 23 +- ✅ All vanes running +- ✅ Persistent state + +## Performance Comparison + +### Ivory Pill (1.5 MB) +``` +C Vere: 0.21 seconds (6.88 MB/s) +OCaml: 0.44 seconds (3.41 MB/s) +Ratio: 2x slower ✅ Acceptable! +``` + +### Solid Pill (8.7 MB) +``` +C Vere: 1.45 seconds (6.0 MB/s) +OCaml: >300 seconds (0.03 MB/s) +Ratio: ~200x slower ❌ Blocker +``` + +The performance gap grows non-linearly with size/complexity, suggesting algorithmic issues in the OCaml cue implementation (see `ARVO_CHALLENGE.md`). + +## For OCaml Implementation + +### What We Have ✅ + +Ivory pill loading: +- Load ivory pill (0.44s vs C's 0.2s = 2x slower, acceptable) +- Validate "ivory" tag (`0x79726f7669` = "ivory") +- Implement `u3v_life()` lifecycle formula `[2 [0 3] [0 2]]` +- ❌ Lifecycle execution fails (Nock exits - still debugging) + +### What We Need ⏳ + +For full Arvo: +1. **Fast cue** for solid pill (currently ~200x slower, blocker) +2. **Process 5 boot events** to build Arvo +3. **Result**: Working Arvo kernel with poke interface + +### What Ivory Enables 🎯 + +Even without full Arvo, ivory pill lets us: +- Test I/O drivers with real Hoon code +- Implement `wish()` to call into Hoon +- Validate jam/cue correctness on real data +- Benchmark Nock performance on stdlib code + +But for **full ship operation** with vanes and event processing, we need solid pill boot. + +## Bootstrap Flow for New Ship + +``` +User runs: urbit -F zod + +1. King process starts + ├─ Load embedded ivory pill (0.2s) + ├─ Run u3v_life() on ivory + └─ u3A->roc = %zuse core + +2. User command triggers new ship boot + ├─ Download brass pill from bootstrap.urbit.org + ├─ Cue pill (1.5s for solid, 30s+ for brass) + ├─ Extract 5 boot events from pill + └─ Process events sequentially: + ├─ +aeon: Initialize event loop + ├─ +boot: Bootstrap Arvo + ├─ +fate: Create Hoon compiler + ├─ +hoon: Load compiler source + └─ +arvo: Load kernel source + +3. Result: Full Arvo kernel + ├─ u3A->roc now contains complete Arvo + ├─ All vanes loaded and running + ├─ Poke interface active at slot 23 + └─ Ship ready for normal operation + +4. Subsequent startups + ├─ Load ivory (still needed for I/O) + ├─ Load ship state from snapshot/eventlog + └─ No pill download needed +``` + +## Source Code References + +- Ivory embedded: `pkg/vere/ivory/ivory.c:2` (`u3_Ivory_pill[]`) +- Ivory boot: `pkg/vere/king.c:962` (`_king_boot_ivory()`) +- Lifecycle: `pkg/noun/vortex.c:26` (`u3v_life()`) +- Pill download: `pkg/vere/king.c:600` (bootstrap URL generation) +- Boot events: `pkg/vere/king.c:606` (`_boothack_pill()`) -- cgit v1.2.3