diff options
Diffstat (limited to 'docs/core-academy/ca12.md')
| -rw-r--r-- | docs/core-academy/ca12.md | 454 |
1 files changed, 454 insertions, 0 deletions
diff --git a/docs/core-academy/ca12.md b/docs/core-academy/ca12.md new file mode 100644 index 0000000..6932784 --- /dev/null +++ b/docs/core-academy/ca12.md @@ -0,0 +1,454 @@ +--- +description: "This lesson covers I/O with Khan, Arvo's thread runner vane, and Lick, Arvo's IPC vane." +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# 13. Vanes VI: Khan, Lick + +*This lesson covers I/O with Khan, the thread-runner vane, and Lick, the IPC vane.* + +Khan and Lick are both interprocess communications vanes with slightly different philosophies. We will also discuss `conn.c`. In brief: + +1. `%khan` is a high-level thread interface, useful to both Arvo and external clients. +2. `%lick` is a low-level noun interface for domain sockets, treating Arvo as a server and earth software as a client. +3. `conn.c` provides full administrative control over Arvo and Vere. + + +## Khan & Threads {#khan-threads} + +Khan allows threads to be triggered from outside of Urbit. To start off with, what exactly is a thread? + +> A thread is a monadic function that takes arguments and produces a result. It may perform input and output while running, so it is not a pure function. +> A thread's strength is that it can easily perform complex IO operations. It uses what's often called the IO monad (plus the exception monad) to provide a natural framework for IO. +> A thread's weakness is that it's impermanent and may fail unexpectedly. In most of its intermediate states, it expects only a small number of events (usually one), so if it receives anything it didn't expect, it fails. When code is upgraded, it's impossible to upgrade a running thread, so it fails. + +Threads can be run using Gall's `%spider` agent or Khan. + +### `%spider`: Threads Before Khan {#spider-threads-before-khan} + +Arvo is an event handler for OS-level moves for vanes. Gall is an event handler for agent-level moves. Spider is an agent for transient thread-level operations. + +#### Thread Definition + +Ultimately, a `$thread` is a gate which accepts a `vase` and returns the `form` of a `strand` that produces a `vase`. In other words, the `$thread` doesn't (just) produce a result, it produces a strand that takes input and produces output from which a result can be extracted. This allows threads to chain friable computations together until a `%thread-done` is produced. + +```hoon ++$ thread $-(vase _*form:(strand ,vase)) +``` + +- `+form` is the mold of the strand. It weaves together the notions of input and output thus: + +```hoon +++ form (strand-form-raw a) +++ strand-form-raw + |* a=mold + $-(strand-input (strand-output-raw a)) ++$ strand-input [=bowl in=(unit input)] +++ strand-output-raw + |* a=mold + $~ [~ %done *a] + $: cards=(list card) + $= next + $% [%wait ~] + [%skip ~] + [%cont self=(strand-form-raw a)] + [%fail err=(pair term tang)] + [%done value=a] + == + == +``` + +`+strand` is more complicated. It's an “asynchronous transaction mold”, which is basically a union of four different monads. It's a wet gate producing a core from a mold. + +1. Reader for input. +2. Writer for cards. +3. Continuation for callbacks. +4. Exception. + +This gate also produces a number of critical handlers, such as: + +- `+form` is the main type of a strand computation. +- `+pure` is an identity computation, useful for binding. +- `+bind` is a combination of two computations. +- `+eval` maintains the monadic nature of the computation. + +A simple thread (like `/ted/code`) is simply a wrapper for some check like a scry: + +```hoon +/- spider +/+ strandio +=, strand=strand:spider +^- thread:spider +|= arg=vase +=/ m (strand ,vase) +^- form:m +;< =bowl:spider bind:m get-bowl:strandio +;< code=@p bind:m (scry:strandio @p /j/code/(scot %p our.bowl)) +(pure:m !>(code)) +``` + +- Read this thread with new eyes about the types involved. + +[`;<` micgal](../../hoon/rune/mic.md#micgal) serves to permit a sequence of computations in which each one depends on the output of the previous one. + +```hoon +;< mold bind expr1 expr2 +``` + +which desugars to: + +```hoon +%+ (bind mold) + expr1 +|= mold +expr2 +``` + +> `;<` can be used to glue a pipeline together to run an asynchronous function or event. This can be helpful when deferring parts of a computation based on external data. + +The `main-loop` pattern provides a way of providing a list of functions to try a value against, and seems like an interesting way of handling an arbitrary number of `%facts`. + +Threads can trigger daughter threads. `+handle-start-thread` does this by modifying Spider's thread `yarn`, but you need to see Spider now. + +- [“Fetch JSON”](../../urbit-os/base/threads/examples/get-json.md) +- [“Child Thread”](../../urbit-os/base/threads/examples/child-thread.md) +- [“Main Loop”](../../urbit-os/base/threads/examples/main-loop.md) +- [“Poke Agent”](../../urbit-os/base/threads/examples/poke-agent.md) +- [“Scry”](../../urbit-os/base/threads/examples/scry.md) +- [“Take Fact”](../../urbit-os/base/threads/examples/take-fact.md) + +#### `/app/spider` + +`/app/spider` tracks threads at the highest level using a “spider core” and a state manager. + +```hoon ++$ card card:agent:gall ++$ thread $-(vase shed:khan) ++$ tid @tatid ++$ input [=tid =cage] ++$ yarn (list tid) ++$ thread-form _*eval-form:eval:(strand ,vase) ++$ trying ?(%build %none) +:: ++$ state + $: starting=(map yarn [=trying =vase]) + running=(axal thread-form) + tid=(map tid yarn) + serving=(map tid [(unit @ta) =mark =desk]) + scrying=(jug tid [=wire =ship =path]) + == +``` + +Fundamentally, each thread is an invocation of a list of thread IDs and their startup state, threads currently in progress, and some handlers for remote scries, etc. + +- `starting` is the collection of threads pending successful execution. +- `running` contains currently-running threads identified by path. +- `tid` is a map for tracking child threads. +- `serving` has to do with the [HTTP API](../../urbit-os/base/threads/http-api.md) for threads, allowing you to use the Urbit ship like a function-as-a-service server. +- `scrying` is a map of sets of remote scries. + +Threads use a set of internal mark conventions (notably `%thread-done` and `%thread-fail`). + +```hoon +++ strand-output-raw + |* a=mold + $~ [~ %done *a] + $: cards=(list card) + $= next + $% [%wait ~] + [%skip ~] + [%cont self=(strand-form-raw a)] + [%fail err=(pair term tang)] + [%done value=a] + == + == +``` + +- `cards` is the set of cards to dispatch immediately. +- `%thread-wait` means to not move on but to stay awaiting a callback. +- `%thread-skip` is a drop because this should be handled elsewhere. +- `%thread-cont` means to continue the computation from a new callback. +- `%thread-fail` aborts a computation and doesn't send effects. +- `%thread-done` finishes a computation and sends effects. + +The `%spider-helper` core has all the logic to handle HTTP, start and conclude threads, build code, handle input, etc. For instance: + +```hoon +++ thread-done + |= [=yarn =vase silent=?] + ^- (quip card ^state) + :: %- (slog leaf+"strand {<yarn>} finished" (sell vase) ~) + =/ =tid (yarn-to-tid yarn) + =/ done-cards=(list card) + :~ [%give %fact ~[/thread-result/[tid]] %thread-done vase] + [%give %kick ~[/thread-result/[tid]] ~] + == + =^ http-cards state + (thread-http-response tid vase) + =^ scry-card state (cancel-scry tid silent) + =^ cards state (thread-clean yarn) + [:(weld done-cards cards http-cards scry-card) state] +``` + +It's not a proper `+abet` core. + +Spider supports a few auxiliary scries to monitor thread state, such as the set of currently running threads: + +```hoon +.^((list path) %gx /=spider=/tree/noun) +``` + +You can only subscribe to Spider for thread results. + +- Look at `/lib/strand`. What surprises you? + +- [“Spider API”](../../urbit-os/base/threads/api.md) + +### A New Interface {#a-new-interface} + +> Khan is the "control plane" and thread-runner vane. Its main purpose is to allow external applications to run [threads](../../urbit-os/base/threads) via a Unix Socket and receive the result. + +Khan was conceived as a way to control Urbit ships from the exterior using threads. The concept evolved a fair bit from proposal to implementation. In practice, Khan is essentially an interface wrapper for Spider-based threads, which produces a somewhat strange (but not unprecedented) situation in which a vane relies on a piece of userspace infrastructure to function correctly. + +Khan can be internally invoked (using a `cage`) or externally invoked (using a `page`). + +#### `/sys/lull` Definition + +```hoon +:: :::: +:::: ++khan :: (1i) threads + :: :::: +++ khan ^? + |% + +$ gift :: out result <-$ + $% [%arow p=(avow cage)] :: in-arvo result + [%avow p=(avow page)] :: external result + == :: + +$ task :: in request ->$ + $~ [%vega ~] :: + $% $>(%born vane-task) :: new unix process + [%done ~] :: socket closed + [%fard p=(fyrd cage)] :: in-arvo thread + [%fyrd p=(fyrd cast)] :: external thread + [%lard =bear =shed] :: inline thread + $>(%trim vane-task) :: trim state + $>(%vega vane-task) :: report upgrade + == :: + :: :: + ++ avow |$ [a] (each a goof) :: $fyrd result + +$ bear $@(desk beak) :: partial $beak + +$ cast (pair mark page) :: output mark + input + ++ fyrd |$ [a] [=bear name=term args=a] :: thread run request + :: :: + +$ shed _*form:(strand:rand ,vase) :: compute vase + -- ::khan +``` + +- [“Khan”](../../urbit-os/kernel/khan) + +While `%khan` hasn't been thoroughly documented yet (we expect some minor API changes, such as the more recent addition of [inline thread invocation](https://github.com/urbit/urbit/pull/5981)), there are examples of its use in ~midsum-salrux's [Tendiebot price bot](https://github.com/midsum-salrux/tendiebot/blob/master/desk/tendiebot/app/tendiebot.hoon) and [Faux Urbit–Discord bridge](https://github.com/midsum-salrux/faux). + +The basic conceit of Khan is that it instruments three ways to run a thread: + +- `%fard` runs a thread from within Arvo directly. +- `%fyrd` runs a thread from outside Arvo (a connexion with the runtime). +- `%lard` runs an inline thread (rather than from `/ted`). + +A `%fard` has the form: + +```hoon +:* %pass + /path-name :: path + %arvo %k %fard :: Arvo vane and %khan mode + %namespace :: desk? + %thread-name :: /ted/thread-name.hoon + %noun :: mark (always %noun for now) + !> :* :: thread arguments: + bowl :: bowl (entropy etc.) + other-info :: other arguments for thread + == +== +``` + +A `%lard` has the form: + +```hoon +=strandio -build-file %/lib/strandio/hoon +=sh |= message=@t + =/ m (strand:rand ,vase) + ;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>('hi')) + ;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>(message)) + (pure:m !>('product')) +|pass [%k %lard %base (sh 'the message')] +``` + +Since `/sys/vane/khan` is a vane, you receive its gifts in `+on-arvo`. + +- `[%arow p=(avow cage)]` is received in userspace. Note that it is a `cage`, or a pair of mark and vase. +- `[%avow p=(avow page)]` can only be received by an external process. It is a `page`, or a pair of mark and (unvased) data. + +Compare Spider and Khan: + +```hoon +:_ this +:~ [%pass /thread/[ta-now] %agent [our.bowl %spider] %watch /thread-result/[tid]] + [%pass /thread/[ta-now] %agent [our.bowl %spider] %poke %spider-start !>([~ `tid byk.bowl %foo !>(~)])] +== +:: +:_ this +:~ [%pass /thread[ta-now] %arvo %k %fard q.byk.bowl %foo %noun !>([bowl ~])] +== +``` + +As a vane, `/sys/vane/khan` is almost as simple as a vane can be: it simply `+call`s tasks and `+take`s gifts from Spider to dispatch back to its caller. + +Khan currently supports no scries. + +- [“Khan: API Reference”](../../urbit-os/kernel/khan/tasks.md) +- [“Developer Call: The Future of `%khan`”](https://www.youtube.com/watch?v=cdSFvFNFqpI) + +Speculatively, I believe that producing an improved CLI predicated on thread execution is feasible today on Urbit. Imagine a context which can dispatch moves either batched or singly, and queue return cards for processing. + +In fact, although the vane evolved from its initial conception, Khan was originally proposed under the theory that pre-written threads would be the easiest way to bundle, distribute, and manage scripts for hosting and maintenance. + + +## Lick {#lick} + +Although also dealing with interprocess communication, Lick was designed for a very different scenario than Khan: to allow external processes, in particular hardware drivers, to intercommunicate with Urbit. (This breached the Earth/Mars divide.) Thus `/sys/vane/lick` focuses on instrumenting a low-level noun interfaces over domain sockets. + +> Lick manages IPC ports, and the communication between Urbit applications and POSIX applications via these ports. Other vanes and applications ask Lick to open an IPC port, notify it when something is connected or disconnected, and transfer data between itself and the Unix application. + +Lick works by opening a Unix socket for a particular process, which allows serialized IPC communications. These involve a jammed noun so the receiving process needs to know how to communicate in nouns. + +> The IPC ports Lick creates are Unix domain sockets (`AF_UNIX` address family) of the `SOCK_STREAM` type. + +The connexions are made via filepaths in `.urb/dev` of the pier. + +The format is: + +``` +V.BBBB.JJJJ.JJJJ... +``` + +- `V` version +- `B` jam size in bytes (little endian) +- `J` jammed noun (little endian) + +> The process on the host OS must therefore strip the first 5 bytes, [`+cue`](../../hoon/stdlib/2p.md#cue) the jamfile, check the mark and (most likely) convert the noun into a native data structure. + +### `/sys/lull` Definition {#syslull-definition} + +```hoon +:: :::: +:::: ++lick :: (1j) IPC + :: :::: +++ lick ^? + |% + +$ gift :: out result <-$ + $% [%spin =name] :: open an IPC port + [%shut =name] :: close an IPC port + [%spit =name =mark =noun] :: spit a noun to the IPC port + [%soak =name =mark =noun] :: soak a noun from the IPC port + == + +$ task :: in request ->$ + $~ [%vega ~] :: + $% $>(%born vane-task) :: new unix process + $>(%trim vane-task) :: trim state + $>(%vega vane-task) :: report upgrade + [%spin =name] :: open an IPC port + [%shut =name] :: close an IPC port + [%spit =name =mark =noun] :: spit a noun to the IPC port + [%soak =name =mark =noun] :: soak a noun from the IPC port + == + :: + +$ name path + -- ::lick +``` + +To evaluate what `/sys/vane/lick` is doing, we need to look at Unix's IPC model briefly. IPC (“interprocess communication“) describes any way that two processes in an operating system's shared context have to communicate with each other. Lick focuses on [Unix domain sockets](https://en.wikipedia.org/wiki/Unix_domain_socket), which are just [communication endpoints](https://man7.org/linux/man-pages/man7/unix.7.html). +For instance, a valid use of `%lick` would use cards that look like this: + +```hoon +++ init [[%pass / %arvo %l %spin /control]~ this] +:: +++ on-arvo + |= [=wire =sign-arvo] + ?+ sign-arvo (on-arvo:def wire sign-arvo) + [%lick %soak *] + ?+ mark.sign-arvo [~ this] + :: + %connect + ~& > "connect" + :_ this [%pass /spit %arvo %l %spit /control %init area.state]~ + == == +:: +++ send-state + |= =state + ^- card:agent:gall + [%pass /spit %arvo %l %spit /control %state [slick:state face.state food.state live.state]] +``` + +- [~mopfel-winrux, `%slick`](https://github.com/mopfel-winrux/slick) + +The vane definition of `/sys/vane/lick` is even simpler than `/sys/vane/khan`: it has no `+abet` core and primarily communicates to the `unix-duct` in its state. The `owner` is a `duct` to handle the return `%soak`. + +Lick takes several scries: + +- [`%a` - Read ports](../../urbit-os/kernel/lick/scry.md#a---read-ports) +- [`%d` - Port owner](../../urbit-os/kernel/lick/scry.md#d---port-owner) +- [`%u` - Port existance](../../urbit-os/kernel/lick/scry.md#u---port-existance) + +Gall needs to wrap `%soak` and `%spit` to route properly. See e.g. `+ap-generic-take`. This lets multiple agents share sockets with the same name, and each agent can have its own folder. + +### `vere/io/lick.c` {#vereiolickc} + +The hardware counterpart of `/sys/lick` is contained in `vere/io/lick.c` aside from its callback registration. As with other parts of the runtime event loop and callback system, the primary connexion is made using `libuv`, in this case via an instance of a [`uv_pipe_t`](https://docs.libuv.org/en/v1.x/pipe.html) descriptor. + +- `_lick_ef_spit()` + - `_lick_send_noun()` + - `u3_newt_send()` +- `_lick_sock_cb()`, callback for connection from Earth. +- `_lick_moor_poke`, result of `%soak` from external process. + + +## `conn.c` {#connc} + +[`conn.c`](https://github.com/urbit/vere/blob/develop/pkg/vere/io/conn.c) is a driver in Vere. It is a part of the "King" (a.k.a. "Urth") process. It exposes a [Unix domain socket](https://en.wikipedia.org/wiki/Unix_domain_socket) at `/path/to/pier/.urb/conn.sock` for sending/receiving data from external processes. The point of `conn.c` is to provide administrative control over Arvo and Vere: read ephemeral or persistent state, enqueue events, and send arbitrary commands (pack, meld, mass, &c). + +(`conn.c` is only loosely related to `/sys/khan`. Its main connection is special-casing some inputs for Khan.) + +`conn.c` accepts a newt-encoded `+jam`med noun of the shape `[request-id command arguments]`, where: + +- `request-id` is a client-supplied atomic identifier with type `@`. It exists entirely for the benefit of the client, allowing responses to be matched to requests. (The poor Earthling's wire.) +- `command` is one of: + - `%peek`, namespace scry request into Arvo. + - `%peel`, emulated namespace scry request into Vere. + - `%ovum`, injection of a raw kernel move. + - `%fyrd`, direct shortcut to Khan command. + - `%urth`, subcommand to runtime to `%pack` or `%meld`. + +See particularly: + +- `_conn_moor_poke()` for the main message dispatcher. +- `_conn_peek_cb()` for the peek handler. + - `_conn_send_noun()` +- `_conn_read_peel()` for the `%peel` handler. + +- [“`conn.c` Usage Guide”](https://github.com/urbit/tools/wiki/conn.c-Usage-Guide) +- [Click](https://github.com/urbit/tools/tree/master/pkg/click) + + +## Exercises {#exercises} + +- Run these valid commands on a fakeship from the outside (following examples in the `conn.c` usage guide). + - Pack, meld, OTA, install, code, vats |