1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
|
Multi-Core Urbit Runtime - Architecture & Implementation Plan │
│ │
│ Vere Architecture (Current C Implementation) │
│ │
│ Core Components: │
│ 1. Loom (allocate.c 2K lines) - Single contiguous memory arena with persistence │
│ 2. Events (events.c 1.6K lines) - Snapshot/checkpoint system (mprotect-based dirty tracking) │
│ 3. Manage (manage.c 2.6K lines) - System lifecycle, memory management │
│ 4. King/Lord (lord.c, pier.c) - Process separation (serf runs Nock, king handles I/O) │
│ 5. I/O Drivers (vere/) - Ames, HTTP, timers, filesystem, etc. │
│ │
│ Key Constraint: Single-threaded execution due to: │
│ - Loom is a single shared memory space │
│ - No parallelism in Nock execution │
│ - Event processing is sequential │
│ │
│ OCaml Multi-Core Runtime with Eio - NEW Architecture │
│ │
│ 🚀 CORE INNOVATION: True Parallelism with OCaml 5 + Eio │
│ │
│ This is THE fundamental advancement over C Vere: │
│ - C Vere: Single-threaded, blocking I/O, sequential event processing │
│ - OCaml Overe: Multi-domain parallelism, async I/O with Eio, concurrent execution │
│ │
│ Eio Benefits: │
│ ✅ Effects-based async I/O (no callbacks, no monads) │
│ ✅ Structured concurrency (fibers with automatic cleanup) │
│ ✅ Multi-domain support (true parallelism across CPU cores) │
│ ✅ Cancellation and timeouts built-in │
│ ✅ Cross-platform (Linux, macOS, Windows via io_uring/kqueue/IOCP) │
│ │
│ Phase 1: Event-Driven Core with Eio │
│ │
│ Goal: Build Eio-based runtime that can process events with async I/O │
│ │
│ What We Need: │
│ 1. Event Log (lib/eventlog.ml) - Eio-based async persistence │
│ - Eio.Path for async file I/O │
│ - Append/replay using Eio.Stream for concurrency │
│ - Non-blocking writes, parallel reads │
│ │
│ 2. State Management (lib/state.ml) - Domain-safe state │
│ - Ship state (arvo kernel + vanes) │
│ - Atomic snapshots using Eio.Promise │
│ - GC-based memory (no loom!) with domain-local allocation │
│ │
│ 3. Eio Runtime (lib/runtime.ml) - THE KEY COMPONENT │
│ - Eio.Switch for structured concurrency │
│ - Fiber per I/O driver (ames, http, behn, unix, term) │
│ - Parallel event processing with domain pool │
│ - Eio.Stream for event queue (lock-free!) │
│ - Effect coordination using Eio capabilities │
│ │
│ 4. Async I/O Drivers (lib/io/) - All Eio-based! │
│ - Timer (Eio.Time.sleep) - non-blocking sleeps │
│ - Network (Eio.Net) - async UDP/TCP │
│ - Filesystem (Eio.Path) - async file ops │
│ - Each driver runs in own fiber │
│ │
│ Files to Create: │
│ - lib/eventlog.ml - Eio-based event persistence │
│ - lib/state.ml - Domain-safe state management │
│ - lib/runtime.ml - Eio runtime with fiber-per-driver │
│ - lib/effects.ml - Effect types (Eio-compatible) │
│ - lib/io/eio_*.ml - Eio-based I/O drivers │
│ - test/test_runtime.ml - Concurrent runtime tests │
│ │
│ Benefits: │
│ - Actually run Urbit code with TRUE PARALLELISM! │
│ - Non-blocking I/O across all drivers │
│ - Can handle thousands of concurrent connections │
│ - Foundation for multi-core Nock execution │
│ │
│ Phase 2: Multi-Domain Parallel Execution (THE GAME CHANGER!) │
│ │
│ Goal: Leverage OCaml 5 domains for CPU-parallel Nock execution │
│ │
│ Eio + Domains Strategy: │
│ │
│ 1. Domain Pool (lib/domain_pool.ml) │
│ - Pool of worker domains (one per CPU core) │
│ - Domainslib.Task for work distribution │
│ - Lock-free work queues (Eio.Stream) │
│ │
│ 2. Parallel Nock (lib/nock_parallel.ml) │
│ - Detect parallelizable computations │
│ - Fork/join using domains for opcode 10 hints │
│ - Parallel jet execution (pure computations) │
│ - Speculative execution with cancellation │
│ │
│ 3. Concurrent Event Processing │
│ - Read-only scry requests in parallel domains │
│ - Multiple pokes processed concurrently (when independent) │
│ - Effect handling parallelized across domains │
│ - Eio manages coordination automatically │
│ │
│ 4. Parallel Jets (lib/jets_parallel.ml) │
│ - Pure jets (hash, crypto, parsing) run in parallel │
│ - Batch operations across domains │
│ - Map/reduce style processing │
│ │
│ Implementation: │
│ - Use Eio.Domain_manager for domain spawning │
│ - Eio.Promise for domain result collection │
│ - Domain-local state for zero-copy optimization │
│ - Lock-free communication via Eio.Stream │
│ │
│ Performance Targets: │
│ - 10-100x throughput on multi-core (vs single-threaded C) │
│ - Sub-millisecond latency for parallel scry │
│ - Thousands of concurrent connections (Eio I/O) │
│ │
│ Phase 3: Advanced Multi-Core Optimizations │
│ │
│ 1. Lock-Free Data Structures │
│ - Kcas (Software transactional memory) │
│ - Lock-free hash tables for noun cache │
│ - Domain-local heaps for allocation │
│ │
│ 2. Concurrent GC Tuning │
│ - OCaml 5's domain-local minor heaps │
│ - Parallel major GC phases │
│ - Tune for noun workload │
│ │
│ 3. Eio I/O Optimizations │
│ - io_uring on Linux (kernel async I/O) │
│ - kqueue on macOS/BSD │
│ - IOCP on Windows │
│ - Zero-copy networking where possible │
│ │
│ 4. JIT Compilation (Future) │
│ - Generate OCaml from hot Nock paths │
│ - Compile to native code at runtime │
│ - Cache compiled code across restarts │
│ │
│ Recommended Next Steps (Piece by Piece with Eio) │
│ │
│ Step 1: Event Log with Eio (2-3 days) │
│ │
│ - Add eio, eio_main to dune dependencies │
│ - Eio-based file I/O for event log (Eio.Path) │
│ - Async append using Eio.Flow │
│ - Parallel replay with Eio.Fiber.fork │
│ - Test with jam/cue roundtrips in Eio context │
│ │
│ Step 2: Domain-Safe State (2-3 days) │
│ │
│ - Domain-local state structures (Atomic) │
│ - Load Arvo kernel using Eio file ops │
│ - Atomic snapshot with Eio.Promise │
│ - Test state persistence across domains │
│ │
│ Step 3: Eio Runtime with Fibers (3-4 days) - THE CORE! │
│ │
│ - Eio.Switch for structured concurrency │
│ - Eio.Stream event queue (lock-free!) │
│ - Fiber per I/O driver pattern │
│ - Process pokes with Eio coordination │
│ - Timer using Eio.Time (first I/O driver) │
│ - First working ship with async I/O! │
│ │
│ Step 4: Multi-Domain Parallelism (1-2 weeks) - THE BREAKTHROUGH! │
│ │
│ - Add domainslib dependency │
│ - Domain pool with Eio.Domain_manager │
│ - Parallel scry using domains │
│ - Parallel jet execution │
│ - Domain-local noun caches │
│ - Benchmark: 10x+ speedup on multi-core! │
│ │
│ Step 5: Full Async I/O (1-2 weeks) │
│ │
│ - Eio.Net for Ames (UDP) - thousands of concurrent ships │
│ - Eio.Net for HTTP - async request handling │
│ - Eio.Path for Clay - non-blocking filesystem │
│ - All drivers as concurrent fibers │
│ - io_uring on Linux for maximum performance │
│ │
│ Why This Approach? │
│ │
│ ✅ GAME CHANGING: First truly parallel Urbit runtime! │
│ ✅ Eio Architecture: Modern async I/O, 1000x more concurrent connections │
│ ✅ Multi-Core Native: 10-100x throughput on multi-CPU systems │
│ ✅ No Loom Limits: GC-based memory, domains scale independently │
│ ✅ Type Safe: OCaml prevents concurrency bugs at compile time │
│ ✅ Production Ready: Eio proven in high-performance systems │
│ │
│ This isn't just a port - it's a fundamental architectural leap forward! │
│ │
│ Start with Step 1 (Eio Event Log)? │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
═══════════════════════════════════════════════════════════════════════════════════════════════════
C to OCaml File Mapping
═══════════════════════════════════════════════════════════════════════════════════════════════════
COMPLETED PORTS ✅
─────────────────────────────────────────────────────────────────────────────────────────────────
Core Noun Operations:
vere/pkg/noun/nock.c (85K) → ocaml/lib/nock.ml ✅ COMPLETE
vere/pkg/ur/serial.c → ocaml/lib/serial.ml ✅ COMPLETE (jam/cue)
vere/pkg/ur/bitstream.c → ocaml/lib/bitstream.ml ✅ COMPLETE
[implicit type definitions] → ocaml/lib/noun.ml ✅ COMPLETE
PHASE 1: EVENT-DRIVEN RUNTIME (Next to Port)
─────────────────────────────────────────────────────────────────────────────────────────────────
Event Log & Persistence (Eio-based):
vere/pkg/noun/events.c (39K) → ocaml/lib/eventlog.ml 📋 Step 1
- Event log management with Eio.Path async file I/O
- Async append/replay using Eio.Stream
- Crash recovery with parallel reads
vere/pkg/vere/disk.c (52K) → ocaml/lib/eventlog.ml 📋 Step 1 (partial)
- Event storage (start with Eio files, LMDB later)
- Snapshot persistence via Eio async writes
vere/pkg/vere/db/lmdb.c → [use OCaml lmdb + Eio] 📋 Later
State Management (Domain-safe):
vere/pkg/noun/manage.c (54K) → ocaml/lib/state.ml 📋 Step 2
- Domain-safe state with Atomic operations
- Arvo state handling across domains
- Atomic snapshots using Eio.Promise
vere/pkg/noun/urth.c (23K) → ocaml/lib/state.ml 📋 Step 2 (partial)
- State save/restore with Eio
- Checkpoint system
Eio Runtime & Event Loop (THE CORE):
vere/pkg/vere/lord.c (29K) → ocaml/lib/runtime.ml 📋 Step 3
- Serf process (runs Nock) with Eio.Switch
- Fiber-based event processing loop
- Poke/peek with Eio coordination
vere/pkg/vere/pier.c (32K) → ocaml/lib/runtime.ml 📋 Step 3 (partial)
- Pier lifecycle with Eio.Switch
- Eio.Stream event queue (lock-free!)
- Multi-fiber effect coordination
vere/pkg/vere/newt.c (8.9K) → ocaml/lib/ipc.ml 📋 Step 3
- IPC protocol (newt) with Eio.Flow
- Async message framing
Effects System (Eio-compatible):
vere/pkg/vere/auto.c (8.5K) → ocaml/lib/effects.ml 📋 Step 3
- Effect types (Eio-compatible)
- Async effect dispatch via fibers
Async I/O Drivers (All Eio-based):
vere/pkg/vere/io/behn.c → ocaml/lib/io/behn.ml 📋 Step 3
- Timer driver using Eio.Time.sleep
- Non-blocking timer events
vere/pkg/vere/time.c (3.3K) → ocaml/lib/io/behn.ml 📋 Step 3
- Time utilities with Eio
PHASE 2: PARALLEL JETS & MULTI-CORE OPTIMIZATION (Step 4)
─────────────────────────────────────────────────────────────────────────────────────────────────
Multi-Domain Jet System:
vere/pkg/noun/jets.c (54K) → ocaml/lib/jets.ml 📋 Step 4
- Domain-aware jet dashboard
- Parallel jet registration
- Lock-free jet matching/lookup
vere/pkg/noun/jets/a/*.c → ocaml/lib/jets/a/*.ml 📋 Step 4
vere/pkg/noun/jets/b/*.c → ocaml/lib/jets/b/*.ml 📋 Step 4
vere/pkg/noun/jets/c/*.c → ocaml/lib/jets/c/*.ml 📋 Step 4
vere/pkg/noun/jets/d/*.c → ocaml/lib/jets/d/*.ml 📋 Step 4
vere/pkg/noun/jets/e/*.c → ocaml/lib/jets/e/*.ml 📋 Step 4
vere/pkg/noun/jets/f/*.c → ocaml/lib/jets/f/*.ml 📋 Step 4
- Pure jets run in parallel across domains
- Crypto, hashing, parsing - all parallelized
- Map/reduce style batch processing
Parallel Nock Execution:
[new implementation] → ocaml/lib/nock_parallel.ml 📋 Step 4
- Domain pool for parallel execution
- Fork/join on hint opcode 10
- Speculative execution with cancellation
Domain-Safe Data Structures:
vere/pkg/ur/hashcons.c → ocaml/lib/hashcons.ml 📋 Step 4
- Lock-free noun deduplication (Kcas)
- Domain-local caches
- Memory optimization
vere/pkg/noun/hashtable.c (31K) → ocaml/lib/hashtable_lockfree.ml 📋 Step 4
- Lock-free hash tables for noun lookup
- Domain-safe operations
PHASE 3: FULL ASYNC I/O DRIVERS (Step 5)
─────────────────────────────────────────────────────────────────────────────────────────────────
King Process (Eio-based):
vere/pkg/vere/king.c (37K) → ocaml/lib/king.ml 📋 Step 5
- I/O process using Eio.Switch
- All I/O as concurrent fibers
- Process separation from serf
Network I/O (Eio.Net):
vere/pkg/vere/io/ames.c → ocaml/lib/io/ames.ml 📋 Step 5
- Async UDP networking (Eio.Net.udp)
- Thousands of concurrent ships
- io_uring on Linux for max performance
vere/pkg/vere/io/ames/stun.c → ocaml/lib/io/ames_stun.ml 📋 Step 5
- Async STUN for NAT traversal
vere/pkg/vere/io/mesa/*.c → ocaml/lib/io/mesa/*.ml 📋 Step 5
- Mesa protocol with Eio
- Parallel packet processing
vere/pkg/vere/io/http.c → ocaml/lib/io/http.ml 📋 Step 5
- HTTP server (Eyre) with Eio.Net
- Concurrent request handling
- WebSocket support via fibers
vere/pkg/vere/io/cttp.c → ocaml/lib/io/cttp.ml 📋 Step 5
- Async HTTP client with Eio
Filesystem (Eio.Path):
vere/pkg/vere/io/unix.c → ocaml/lib/io/unix_fs.ml 📋 Step 5
- Clay filesystem with Eio.Path
- Async file watching (inotify/kqueue)
- Non-blocking file operations
Terminal (Eio):
vere/pkg/vere/io/term.c → ocaml/lib/io/term.ml 📋 Step 5
- Terminal I/O (Dill) with Eio
- Async terminal rendering
vere/pkg/vere/platform/*/ptty.c → ocaml/lib/io/term.ml 📋 Step 5
- Platform-specific PTY with Eio
Other I/O (Eio-based):
vere/pkg/vere/io/conn.c → ocaml/lib/io/conn.ml 📋 Step 5
- Spider/thread connections via fibers
vere/pkg/vere/io/lick.c → ocaml/lib/io/lick.ml 📋 Step 5
- IPC with external processes using Eio
MEMORY & LOOM (May Not Need Direct Ports)
─────────────────────────────────────────────────────────────────────────────────────────────────
Loom System:
vere/pkg/noun/allocate.c (41K) → N/A - OCaml uses GC ⊘ Not needed
- Single arena allocator
- OCaml's GC handles this automatically
vere/pkg/noun/imprison.c (15K) → N/A - OCaml uses GC ⊘ Not needed
- Memory locking
- OCaml's GC handles this
UTILITIES & SUPPORT
─────────────────────────────────────────────────────────────────────────────────────────────────
Noun Operations:
vere/pkg/noun/retrieve.c (38K) → ocaml/lib/noun_ops.ml 📋 As needed
- Noun traversal utilities
- Path lookup
vere/pkg/noun/vortex.c (7.5K) → ocaml/lib/state.ml 📋 As needed
- Arvo kernel interface
Tracing & Debugging:
vere/pkg/noun/trace.c (30K) → ocaml/lib/trace.ml 📋 Optional
- Nock tracing
- Debugging support
vere/pkg/noun/log.c (706) → ocaml/lib/log.ml 📋 Optional
- Logging utilities
Boot & Initialization:
vere/pkg/vere/main.c (82K) → ocaml/bin/overe.ml 📋 Later
- Main entry point
- Command-line interface
vere/pkg/vere/dawn.c (11K) → ocaml/lib/boot.ml 📋 Later
- Network boot (Azimuth)
vere/pkg/vere/mars.c (45K) → ocaml/lib/boot.ml 📋 Later
- Fake ship boot
vere/pkg/vere/ivory/ivory.c → ocaml/lib/ivory.ml 📋 Later
- Ivory (minimal kernel)
Platform Support:
vere/pkg/vere/platform/* → [use OCaml stdlib/Unix] 📋 As needed
- Platform-specific code
- OCaml abstracts most of this
LEGEND
─────────────────────────────────────────────────────────────────────────────────────────────────
✅ COMPLETE - Already ported and tested
📋 Step N - Part of current plan, priority order
📋 Future - Planned for later phases
📋 As needed - Port incrementally when required
📋 Optional - Nice to have, not critical
⊘ Not needed - OCaml handles differently, no port needed
═══════════════════════════════════════════════════════════════════════════════════════════════════
|
