diff options
Diffstat (limited to 'docs/core-academy/ca11.md')
| -rw-r--r-- | docs/core-academy/ca11.md | 657 |
1 files changed, 657 insertions, 0 deletions
diff --git a/docs/core-academy/ca11.md b/docs/core-academy/ca11.md new file mode 100644 index 0000000..d3a52ec --- /dev/null +++ b/docs/core-academy/ca11.md @@ -0,0 +1,657 @@ +--- +description: "Covers Gall, Arvo's userspace application framework vane, the agent type, running agents, and userspace software updates." +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# 12. Vanes V: Gall and Userspace + +*This lesson covers Gall: the `$agent` type, running agents, and userspace software updates.* + +## Gall {#gall} + +Currently, the end user zone of Urbit, denoted “userspace”, is supplied primarily by the `/sys/vane/gall` vane. Much as Arvo acts as a dispatcher and state manager for a functional OS in terms of transactions between vanes, Gall acts as a dispatcher and state manager for longrunning daemons. These daemons are referred to as “agents”, and comprise the main way that users actually use Urbit. (Other parts of userspace include generators and threads.) + +Gall is responsible to manage userspace applications and their state, including subscribers. To fully understand agent userspace, we want to cover a few related topics: + +1. Agents +2. `/sys/vane/gall` +3. Treaty/Docket publication +4. Agent wrappers +5. Historic (dynamic) Gall + +We will cover threads in the lesson on Khan and Lick. We covered generators previously in the lesson on Dill and Dojo. + + +## Agents {#agents} + +> An agent is a piece of software that is primarily focused on maintaining and distributing a piece of state with a defined structure. It exposes an interface that lets programs read, subscribe to, and manipulate the state. Every event happens in an atomic transaction, so the state is never inconsistent. Since the state is permanent, when the agent is upgraded with a change to the structure of the state, the developer provides a migration function from the old state type to the new state type. + +What is an agent in practice? In contemporary static Gall, an agent is a core that hews to the definition: + +```hoon +:: :::: +:::: ++gall :: (1g) extensions + :: :::: +++ gall ^? + |% + :: + :: +agent: app core + :: + ++ agent + =< form + |% + +$ step (quip card form) + +$ card (wind note gift) + +$ note + $% [%agent [=ship name=term] =task] + [%arvo note-arvo] + [%pyre =tang] + :: + [%grow =spur =page] + [%tomb =case =spur] + [%cull =case =spur] + == + +$ task + $% [%watch =path] + [%watch-as =mark =path] + [%leave ~] + [%poke =cage] + [%poke-as =mark =cage] + == + +$ gift + $% [%fact paths=(list path) =cage] + [%kick paths=(list path) ship=(unit ship)] + [%watch-ack p=(unit tang)] + [%poke-ack p=(unit tang)] + == + +$ sign + $% [%poke-ack p=(unit tang)] + [%watch-ack p=(unit tang)] + [%fact =cage] + [%kick ~] + == + ++ form + $_ ^| + |_ bowl + ++ on-init + *(quip card _^|(..on-init)) + :: + ++ on-save + *vase + :: + ++ on-load + |~ old-state=vase + *(quip card _^|(..on-init)) + :: + ++ on-poke + |~ [mark vase] + *(quip card _^|(..on-init)) + :: + ++ on-watch + |~ path + *(quip card _^|(..on-init)) + :: + ++ on-leave + |~ path + *(quip card _^|(..on-init)) + :: + ++ on-peek + |~ path + *(unit (unit cage)) + :: + ++ on-agent + |~ [wire sign] + *(quip card _^|(..on-init)) + :: + ++ on-arvo + |~ [wire sign-arvo] + *(quip card _^|(..on-init)) + :: + ++ on-fail + |~ [term tang] + *(quip card _^|(..on-init)) + -- + -- + -- ::gall + +``` + +A Gall agent must have ten arms. (There's a fascinating bit of self-reference in the state definition going on, and to be honest I'm a little surprised that it works, but it is an iron core.) The definitions here are for `|~` barsig arms in a `^|` ketbar core. + +- [`|~` barsig](../../hoon/rune/bar.md#barsig) yields an iron gate. +- [`^|` ketbar](../../hoon/rune/ket.md#ketbar) yields an iron core from a gold core. + +So we must at last really grapple with the [core variance model](../hoon-school/R-metals.md) in Urbit. This is often notorious to understand because we don't have great metaphors or analogues to type variance in real life. Right now, what we need to understand is that an iron/contravariant core is _opaque_: That is, we use this to define an interface in which the argument can be less specific than the interface and the result can be more specific. Contravariance is useful for flexibility in input values (`sample`s). + +> An `%iron` core `i` has a write-only sample (payload head, `+6.i`) and an opaque context (payload tail, `+7.i`). A core `j` which nests within it must be a `%gold` or `%iron` core, such that `+6.i` nests within `+6.j`. Hence, **contravariant**. + +The archetypal Gall agents in `/sys/lull` are composed using iron gates since they will be used as examples for building actual agent cores. Likewise, the `+rs` and sister gates in `/sys/hoon` are built using iron doors with specified rounding behavior so when you actually use the core (like `+add:rs`) the core you are using has been built as an example. + +- How are the iron gate runes actually implemented in the Hoon type system? (See `+deem:nest:ut` and `+peel:ut`.) +- Try to implement an agent missing an arm, like `+on-fail`. +- Bonus question: what half-implemented rune produces an `%iron` core? + +We construct an agent explicitly in an `/app` file by applying `%- agent:gall` to a correctly-shaped core. + +- What does each arm produce? + +```hoon +++ on-init (quip card _agent) +++ on-save (vase) +++ on-load (quip card _agent) +++ on-poke (quip card _agent) +++ on-watch (quip card _agent) +++ on-leave (quip card _agent) +++ on-peek (unit (unit cage)) +++ on-agent (quip card _agent) +++ on-arvo (quip card _agent) +++ on-fail (quip card _agent) +``` + +Finally, we can take a gander at what that ubiquitous `+quip` is: + +```hoon +++ quip + |$ [item state] + [(list item) state] +``` + +(It's just a wrapper for `(list item) state`.) + +Basically, every arm must produces a list of effects and a state change, if any. + +- What does each arm expect? + +```hoon +++ on-init :: not a gate, only an arm +++ on-save :: not a gate, only an arm +++ on-load |= =vase +++ on-poke |= =cage +++ on-watch |= =path +++ on-leave |= =path +++ on-peek |= =path +++ on-agent |= [=wire =sign:agent:gall] +++ on-arvo |= [=wire =sign-arvo] +++ on-fail |= [=term =tang] +``` + +We'll need to differentiate the Gall `$sign`s and the Arvo `$sign`s in a moment. + +When `+ford:clay` reads in a Gall agent file from `/app`, it automatically composes cores together using `=>` tisgar. (This leads to a slightly disconcerting situation in which the cores are simply present serially in a file.) + +- Compare the definition of `$agent:shoe` with `$agent:gall`. How does this correctly extend the Gall agent definition for the type system? + + +## Vane {#vane} + +While Gall facilitates very complex userspace apps, the vane itself is rather modest, weighing in at less than half the size of Clay or Ames. Gall knows how to route events to the handler arms in a standard agent core, and it instruments upgrades and subscriptions. + +However, we have to consider Gall at two levels: the vane level, which manages top-level state like the set of running agents and queued moves, and the agent level, which manages agents as doors. + +Gall is a landlocked vane. It has no runtime counterpart. + +### `/sys/lull` Definition {#syslull-definition} + +```hoon +:: :::: +:::: ++gall :: (1g) extensions + :: :::: +++ gall ^? + |% + +$ boar (map [=wire =ship =term] nonce=@) :: and their nonces + +$ dude term :: server identity + +$ gill (pair ship term) :: general contact + +$ load (list [=dude =beak =agent]) :: loadout + +$ scar :: opaque duct + $: p=@ud :: bone sequence + q=(map duct bone) :: by duct + r=(map bone duct) :: by bone + == :: + +$ suss (trel dude @tas @da) :: config report + +$ well (pair desk term) :: + +$ deal + $% [%raw-poke =mark =noun] + task:agent + == + +$ unto + $% [%raw-fact =mark =noun] + sign:agent + == + -- ::gall +``` + +Most of the important types have been separated and are called out below. + +(Variance again: [`^?` ketwut](../../hoon/rune/ket.md#ketwut) is for a lead/bivariant core.) + +### Vane State {#vane-state} + +```hoon ++$ state + $: system-duct=duct + outstanding=(map [wire duct] (qeu remote-request)) + contacts=(set ship) + yokes=(map term yoke) + blocked=(map term (qeu blocked-move)) + =bug + == +``` + +- `system-duct` is the set of outbound moves to other vanes (like Ames for subscriptions) or remote agent contacts. +- `outstanding` is the outstanding request queue. +- `contacts` is the set of other ships with which we are in communication. +- `yokes` is the set of running agents. +- `blocked` is the set of moves to agents that haven't been started yet. +- `bug` is the debug print configuration. + +### Vane Moves {#vane-moves} + +```hoon +|% ++$ gift :: outgoing result + $% [%boon payload=*] :: ames response + [%done error=(unit error:ames)] :: ames message (n)ack + [%unto p=unto] :: + == :: ++$ task :: incoming request + $~ [%vega ~] :: + $% [%deal p=sock q=term r=deal] :: full transmission + [%sear =ship] :: clear pending queues + [%jolt =desk =dude] :: (re)start agent + [%idle =dude] :: suspend agent + [%load =load] :: load agent + [%nuke =dude] :: delete agent + [%doff dude=(unit dude) ship=(unit ship)] :: kill subscriptions + [%rake dude=(unit dude) all=?] :: reclaim old subs + $>(%init vane-task) :: set owner + $>(%trim vane-task) :: trim state + $>(%vega vane-task) :: report upgrade + $>(%plea vane-task) :: network request + [%spew veb=(list verb)] :: set verbosity + [%sift dudes=(list dude)] :: per agent + == :: +-- +``` + +### Agent State {#agent-state} + +```hoon ++$ bitt (map duct (pair ship path)) :: incoming subs ++$ boat (map [=wire =ship =term] [acked=? =path]) :: outgoing subs ++$ bowl :: standard app state + $: $: our=ship :: host + src=ship :: guest + dap=term :: agent + == :: + $: wex=boat :: outgoing subs + sup=bitt :: incoming subs + $= sky :: scry bindings + %+ map path :: + ((mop @ud (pair @da (each page @uvI))) lte) :: + == :: + $: act=@ud :: change number + eny=@uvJ :: entropy + now=@da :: current time + byk=beak :: load source + == == :: +``` + +Every agent needs two parts of its state: the `$bowl`, which is the information outside of the agent that Gall needs to communicate for the vane, and the + +### Agent Moves {#agent-moves} + +```hoon +|% ++$ card (wind note gift) ++$ note + $% [%agent [=ship name=term] =task] + [%arvo note-arvo] + [%pyre =tang] + :: + [%grow =spur =page] + [%tomb =case =spur] + [%cull =case =spur] + == ++$ task + $% [%watch =path] + [%watch-as =mark =path] + [%leave ~] + [%poke =cage] + [%poke-as =mark =cage] + == ++$ gift + $% [%fact paths=(list path) =cage] + [%kick paths=(list path) ship=(unit ship)] + [%watch-ack p=(unit tang)] + [%poke-ack p=(unit tang)] + == ++$ sign + $% [%poke-ack p=(unit tang)] + [%watch-ack p=(unit tang)] + [%fact =cage] + [%kick ~] + == +-- +``` + +A Gall `$card` differs from an Arvo card: + +```hoon +:: Arvo ++$ card (cask) :: tagged, untyped event +++ cask |$ [a] (pair mark a) :: marked data builder +:: +:: Gall ++$ card (wind note gift) +++ wind + |$ :: a: forward + :: b: reverse + :: + [a b] + $% :: %pass: advance + :: %slip: lateral + :: %give: retreat + :: + [%pass p=path q=a] + [%slip p=a] + [%give p=b] + == ++$ note + $% [%agent [=ship name=term] =task] + [%arvo note-arvo] + [%pyre =tang] + :: + [%grow =spur =page] + [%tomb =case =spur] + [%cull =case =spur] + == ++$ gift + $% [%fact paths=(list path) =cage] + [%kick paths=(list path) ship=(unit ship)] + [%watch-ack p=(unit tang)] + [%poke-ack p=(unit tang)] + == +``` + +Gall does not permit a `%slip`, so a card is either: + +- `[%pass path note]` +- `[%give gift]` + + +### Structure {#structure} + +The two main engine cores within `/sys/vane/gall` are: + +- `+mo` Arvo move handler +- `+ap` agent-level core + +The `+abet` pattern used in Gall prefixes each arm with the containing door abbreviation so you can remain more easily oriented within `/sys/vane/gall`. + +#### `+mo` Arvo move handler + +Many `+mo` calls resolve into `+ap` calls. It mainly sets things up around particular per-agent calls. + +#### `+ap` agent-level core + +To run an agent, we have to know the state of the agent, which includes its state and relevant bowl information: + +```hoon +:: $yoke: agent runner state +:: ++$ yoke + $% [%nuke sky=(map spur @ud)] + $: %live + control-duct=duct + run-nonce=@t + sub-nonce=_1 + =stats + =bitt + =boat + =boar + code=* + agent=(each agent vase) + =beak + marks=(map duct mark) + sky=(map spur path-state) + ken=(jug spar:ames wire) + == == +``` + +- `control-duct` is the duct of +- `run-nonce` is a unique nonce for each build. +- `sub-nonce` is global `%watch` nonce. +- `stats` +- `bitt` is the set of incoming subscriptions (for the `bowl`). +- `boat` is the set of outgoing subscriptions (for the `bowl`). +- `code` is the most recently loaded code as a noun. +- `agent` is the agent core, possibly as a vase. +- `beak` is the compilation source. +- `marks` is the map of mark conversion requests. +- `sky` is the map of scry bindings. +- `ken` is the map of sets of open `%keen` remote scry requests. + +A typical call from `+mo` to `+ap` will be predicated on `+ap` setting up a Gall agent with its state and processing the incoming move through the appropriate arm. + +For instance, this is the lifecycle of a scry call to Gall: + +- A scry handler (`roof`) produces a call to Gall's `+scry` arm. + - `+mo` as a door needs a duct and a set of moves. + - `+mo-peek` sets up a call to `+ap-peek` along the given path with the appropriate care. + - `+ap-abed` sets up the agent noun for evaluation. + - `+ap-yoke` loads the actual agent state; an agent is a door with a state and bowl sample. + - `+ap-peek` parses the scry path appropriately. + - `+ap-mule-peek` evaluates the code using `[9 2 0 1]` and `+mock` (see `ca01` for a refresher). + - `+ap-agent-core` sets up the agent core with its current bowl and state; this includes a `+on-peek` arm since we know the shape of the `$agent` core. + - `+ap-construct-bowl` produces the agent-ready bowl from Gall-level information. + +The lifecycle of a poke looks like this: + +- A move is injected targeting Gall's `+call` arm as a `%deal` (indicating that the move goes to an agent). + - `+call` dispatches to `+mo-handle-use` for an agent. + - `+mo-handle-local` is for running local agents. + - `+mo-apply` and `+mo-apply-sure` prepare to call `+ap`. + - `+ap-abed` sets up the agent noun for evaluation. + - `+ap-apply` dispatches several kinds of operations, including pokes. + - `+ap-poke` queues a `%poke-ack` (since it's first among moves) and calls `+ap-ingest`. + - `+ap-ingest` calls the agent arm. + - `+ap-handle-result` and `+ap-handle-peers` take care of watches etc. + - `+ap-agent-core` sets up the agent core with its current bowl and state; this includes a `+on-poke` arm since we know the shape of the `$agent` core. + - `+ap-abet` yields the list of cards to resolve back to `+mo`, but also the `$yoke`, which is the new agent state for Gall's `state`. + - `+mo-abet` finalizes. + +What about an agent modification like `|nuke`? Let's see the lifecycle of that call. + +- `/gen/hood/nuke` → `%kiln-nuke` + - `/lib/hood/kiln` → `+poke-nuke` + - `[%pass /nuke %arvo %g [%nuke dude]]` + - `/sys/vane/gall`: + - `+call` + - `+mo` + - `+mo-nuke` + - `+ap` + - `+ap-abed` + - `+ap-nuke` is where the real work is done. Review it. + - `+ap-ingest` + - `+ap-abet` + - `+mo-abet` + +Clay actually governs which agents can run on a given desk. How does `|install` instigate this? + +- See `+goad` in `/sys/vane/clay`. + +There is additionally some plumbing around Gall receiving responses in `+take` for the vane (`/sys`) versus for an agent (`/use`). + +We'll need to differentiate the Gall `$sign`s and the Arvo `$sign`s in a moment. + +### Scry interface {#scry-interface} + +Gall brokers two kinds of scries: vane scries and agent scries. + +> In order to hit the vane-level endpoints, the beginning of the the `spur` (e.g. the `path` after the `beak`) _must_ be a `%$` empty element. For example: + +```hoon +.^(desk %gd /=acme=/$) +.^((set [=dude:gall live=?]) %ge /=base=/$) +.^((list path) %gt /=acme=//foo) +``` + +- [`%d`: get desk of app](../../urbit-os/kernel/gall/scry.md#d-get-desk-of-app) +- [`%e`: running apps](../../urbit-os/kernel/gall/scry.md#e-running-apps) +- [`%f`: nonces of apps](../../urbit-os/kernel/gall/scry.md#f-nonces-of-apps) +- [`%n`: get nonce of subscription](../../urbit-os/kernel/gall/scry.md#n-get-nonce-of-subscription) +- [`%t`: remote scry subpaths](../../urbit-os/kernel/gall/scry.md#t-remote-scry-subpaths) +- [`%u`: check if installed](../../urbit-os/kernel/gall/scry.md#u-check-if-installed) +- [`%w`: latest revision of path](../../urbit-os/kernel/gall/scry.md#w-latest-revision-of-path) +- [`%x`: remote scry file](../../urbit-os/kernel/gall/scry.md#x-remote-scry-file) +- [`%z:` hash of value at path](../../urbit-os/kernel/gall/scry.md#z-hash-of-value-at-path) + +An agent scry has the form `/=agent=/path/to/scry` and may accept any care. `%x` cares must include a terminal mark in the path, however. + +Gall also dispatches scries to agents' `+on-peek` arms. This takes place via `+mo-peek`→`+ap-peek`→`+on-peek`→`+ap-mule-peek`. + +See `+scry` for details of both. + +- [“Gall Scry Reference”](../../urbit-os/kernel/gall/scry.md) + + +## Treaty and Docket {#treaty-and-docket} + +There are two ways to distribute nouns over Ames today: + +1. Mark a desk `|public` and use Clay to directly synchronize desks. +2. Use the `/app/treaty` agent from `%landscape` to discover and install agents. + +Landscape (formerly Grid) is formally a Tlon product. It primarily consists of two agents (and associated marks, libraries, etc.): + +- `/app/treaty` handles publishing and advertising application desks. +- `/app/docket` handles retrieving, validating, and installing application desks. + +Together they query a remote `%treaty` instance to `+install` a particular desk. + +- Examine `$alliance` in `/sur/treaty`. +- Examine `+publish:so` and `+watch:tr` in `/app/treaty`. +- Examine `+install:ch` in `/app/docket`. + +`/app/treaty` in particular has pretty tight construction and I commend its style to you. + +- [“Software Distribution · Userspace”](../userspace/dist/software-distribution.md) + +### Updates {#updates} + +> When Gall receives a newly rebuilt agent from Clay, it calls the gate produced by the `+on-load` arm of the new agent with the state extracted from the old agent. If there is a crash in any `+on-load` calls or in the handling of any effects they emit (which can include more agent activations), then the whole event crashes, canceling the commit. This effectively gives any agent the ability to abort a commit by crashing. + +- Gall's `+call` arm receives a `%load` move with a noun of a core built by `+ford:clay`. + - `+mo-core` is a handle to `+mo` because no `+abet` is needed. + - `+mo-load` installs agents pretty mechanically, by simply `+skim`ming over the `%live` agents. + - `+mo-receive-core` checks whether the agent is running. + - If it is, then `+ap` is invoked to update the agent: + - `+ap-abed` + - `+ap-reinstall` + - `+on-save:ap-agent-core` + - `+ap-install` is the install wrapper. + - `+ap-upgrade-state` + - `+on-init:ap-agent-core` + - `+on-load:ap-agent-core` + - `+ap-abet` + - `+mo-clear-queue` flushes the blocked tasks pending for a new agent. + - If it isn't, then we have to create it in `+ap`: + - `+ap-abed` + - `+ap-upgrade-state` + - `+on-init:ap-agent-core` + - `+on-load:ap-agent-core` + - `+mo-idle` puts the agent to sleep if it's in the kill list (to be retired due to `desk/bill`). + - `+mo-abet` finalizes moves and state changes. + +Note that this is independent of Treaty and Docket once the remote desk has been installed. + + +## Agent Wrappers as Core Modifiers {#agent-wrappers-as-core-modifiers} + +.An agent wrapper (like `dbug`) is a tool to wrap additional handlers around an agent core. These can wrap the internal Gall agent with new functionality by catching pokes and other standard moves, then re-dispatching to the normal arms if no special behavior is needed. + +```hoon +> :my-agent +dbug [%state 'value'] +``` + +- Examine `/lib/dbug`. + - In particular, how does the `+on-poke` wrapper arm work? + - How does the `+dbug` generator work? + +> Using agent transformers to extend agents is a very nice conceptual pattern. But in practice, there are three pretty big problems with it: +> 1. You need to edit the agent code yourself. +> 2. Stateful transformers can break the agent. +> 3. The agent's world will also get transformed. + +- [“Debugging Wrapper”](../userspace/examples/dbug.md) +- [~wicrum-wicrun, “Gall Middleware (Assembly 2022”](https://www.quartus.co/blog/gall-middleware-wicrum-wicrun-assembly-2022) + + +## Dynamic Gall {#dynamic-gall} + +An earlier incarnation of Gall, dynamic Gall, specified its arms in terms of the names of the move coming back or the mark of the poke coming in. (This is what I learned on, way back when.) + +For instance, this agent was an earlier version of the egg timer app: + +```hoon +:: |start %egg +:: :egg ~s5 +|% ++$ effect (pair bone syscall) ++$ syscall [%wait path @da] +-- +|_ [bowl:gall ~] +++ poke-noun + |= t=@dr + ^+ [*(list effect) +>.$] + :_ +>.$ :_ ~ + [ost %wait /egg-timer (add now t)] +++ wake + |= [=wire error=(unit tang)] + ^+ [*(list effect) +>.$] + ~& "Timer went off!" + [~ +>.$] +-- +``` + + +## Exercises {#exercises} + +- Your assignment is to produce a minimalist Gall-like agent handler: a userspace framework for producing “toy” agent-like applications. Let's call them “scamps”. + + The scamp's state is defined in a `$state` block at the top of its file, e.g.: + + ```hoon + +$ state + $: scores=(list @) + hi-score=@ + == + ``` + + Scamps do not support state upgrades, so no version tag is provided. + + A scamp requires the following arms for the developer: + + ```hoon + |% + ++ on-init + ++ on-poke + ++ on-peek + -- + ``` + + You should be able to poke and peek into a scamp. It has no subscription model. + + A scamp specification file is NOT implicitly chained with a running `=>` tisgal. Compose explicitly. + +As a final aside, I believe that building an `%aqua`/`%pyro` testbed along the lines of Gall should also be feasible for you at this point. |