brass!!

1 files changed, 502 insertions, 0 deletions

diff --git a/docs/core-academy/ca09.md b/docs/core-academy/ca09.md
new file mode 100644
index 0000000..09728e8
--- /dev/null
+++ b/docs/core-academy/ca09.md
@@ -0,0 +1,502 @@
+---
+description: "Core Academy lesson on HTTP vanes covering the Eyre vane's HTTP server functionality, the Iris vane's HTTP client operations, web request/response patterns and authentication mechanisms."
+layout:
+  title:
+    visible: true
+  description:
+    visible: false
+  tableOfContents:
+    visible: true
+  outline:
+    visible: true
+  pagination:
+    visible: true
+---
+
+# 10. Vanes III:  Eyre, Iris
+
+*This lesson covers Arvo's HTTP server vane (Eyre) and its HTTP client vane, Iris. We discuss how Arvo interacts with clients over HTTP.*
+
+## Servers {#servers}
+
+One of Urbit's primary use cases is to act as a “personal server”. To examine this statement, we need to consider what a server does. Etymologically, a server serves a service. Generally speaking, it is the locus of a computation and coordination process. A server program is a system daemon—and since Gall agents are essentially daemons in many respects, Urbit's execution model fulfills this niche nicely.
+
+Some servers are physical or logical devices which talk to other devices as clients. Internet webpage and application servers typically follow this model. Other servers are software processes that run on the same hardware or local network as the client process, e.g. mail servers, print servers, and file servers.
+
+![](https://storage.googleapis.com/media.urbit.org/core-academy/client-server.png)
+
+The two major operational models for servers are the _request–response_ model and the _publish–subscribe_ (pub-sub) model. The request–response pattern corresponds to pokes and gifts in Arvo terms, while the pub-sub pattern is supplied by subscriptions and updates.
+
+- A _client_ originates and submits requests, and receives responses.
+- A _server_ accepts requests and replies with responses.
+
+A client and a server need to agree on a communications protocol. There are many of these, but the basis for the World Wide Web is the HyperText Transfer Protocol (HTTP).
+
+### Serving a Web Page {#serving-a-web-page}
+
+Two of the simplest actions one can take with a basic web server are to simply post a web page to any clients and to respond to interactions with that web page. Some interactions take place purely in the client session (form entry in the browser before submission), but then are propagated to the server.
+
+#### Requests
+
+HTTP requests are like Gall agent pokes:  they are messages to trigger some action on the server. A method is specified (like `GET`, `PUT`, or `POST`) and the associated service-specific data follow.
+
+It consists of a block of request headers, a block of general headers, and a block of representation headers. These may by followed by the body.
+
+![](https://storage.googleapis.com/media.urbit.org/core-academy/http-request.png)
+
+- `GET` means a read-only request for information (like an Urbit scry but without the bound namespace).
+- `PUT` requests a state creation or update.
+- `POST` asks for the server to process data. (Both `PUT` and `POST` are analogous to Urbit pokes.)
+
+#### Responses
+
+A [server response to a web page](https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages) looks like this:
+
+![](https://storage.googleapis.com/media.urbit.org/core-academy/http-response.png)
+
+The [response code](https://en.wikipedia.org/wiki/HTTP_response_code) is normatively `200 OK` for a successful page access, but `404 Not Found` and other errors and special messages also occur frequently. (It would be very interesting if Urbit would implement `402 Payment requested`.)
+
+The actual mechanics of communicating both kinds of communications are wrapped by `/lib/server` and Eyre. Generally speaking, an agent will receive HTTP requests in `+on-poke`, and commonly includes a `+handle-http` arm to deal with `inbound-request:eyre` values. `/lib/server` has request header parsers and response handlers which make it easy to respond appropriately (e.g. `(send:server ~ [%login-redirect './apps/my-agent'])`).
+
+
+## Eyre {#eyre}
+
+Eyre is an HTTP server, which receives HTTP messages from Unix and produces HTTP messages in reply. Your agent can register endpoints which a browser or other tool can interact with. Eyre can be instrumented to work with threads and generators.
+
+HTTP requests include a method tag. While other methods exist, we are primarily interested in `POST`, `PUT`, and `GET` requests. Since we don't want to deal with client-side code yet, we're going to use `curl` to send requests here.
+
+- `POST` is only used with Eyre to obtain a cookie.
+
+```sh
+$ curl -i localhost:8080/~/login -X POST -d "password=lidlut-tabwed-pillex-ridrup"
+HTTP/1.1 204 ok
+Date: Tue, 19 Jul 2022 16:28:05 GMT
+Connection: keep-alive
+Server: urbit/vere-1.9
+set-cookie: urbauth-~nec=0v3.pis4a.sfdhv.f1p6i.lttba.gp93q; Path=/; Max-Age=604800
+```
+
+This cookie should be included in subsequent requests.
+
+- `PUT` requests are used to send actions to Eyre:  pokes, subscriptions, acks, unsubscribe requests, and channel deletions.
+
+- `GET` requests are used to connect to a channel and receive any pending events. (Remember how Urbit prefers a dataflow computing model?)
+
+### `/sys/lull` Definition {#syslull-definition}
+
+```hoon
+::                                                      ::::
+::::                    ++eyre                            ::  (1e) http-server
+  ::                                                    ::::
+++  eyre  ^?
+  |%
+  +$  gift
+    $%  $>(?(%boon %done) gift:ames)   ::  Ames responses
+        [%set-config =http-config]   ::  configure external HTTP server
+        [%sessions ses=(set @t)]     ::  valid auth cookies
+        [%response =http-event:http]   ::  response to event from Earth
+        [%bound accepted=? =binding]   ::  response to %connect or %serve
+        [%grow =path]                ::  notification on cache entry change
+    ==
+  ::
+  +$  task
+    $~  [%vega ~]
+    $%  $>(%init vane-task)        ::  initialize ourself with an identity
+        $>(%born vane-task)        ::  new unix process
+        $>(%plea vane-task)        ::  network request
+        $>(%trim vane-task)        ::  trim state (memory pressure)
+        $>(%vega vane-task)        ::  report upgrade
+        ::
+        [%live insecure=@ud secure=(unit @ud)]            ::  live HTTPS ports
+        [%rule =http-rule]         ::  update HTTP configuration
+        [%eauth-host host=(unit @t)]   ::  set base URL for eauth
+        [%request secure=? =address =request:http]        ::  handle inbound
+        [%request-local secure=? =address =request:http]  ::  handle backdoor
+        [%cancel-request ~]        ::  cancel previous request
+        [%connect =binding app=term]   ::  connects a binding to an app
+        [%serve =binding =generator]   ::  connect binding to generator
+        [%disconnect =binding]         ::  disconnect binding to generator
+        ::
+        [%code-changed ~]          ::  web login code changed
+        [%approve-origin =origin]  ::  accept CORS requests from origin
+        [%reject-origin =origin]   ::  reject CORS requests from origin
+        [%spew veb=@]              ::  set verbosity
+        [%set-response url=@t entry=(unit cache-entry)]   ::  cache mapping
+    ==
+--  ::eyre
+```
+
+Eyre is responsible for a few subsystems that facilitate userspace applications (unlike, say, Behn or Ames, most of what Eyre does is to support userspace).
+
+- Authentication
+- Channels
+- Threads and generators
+- HTTP request handling
+- Scry interface
+
+Several of these are handled by the `$action` dispatch system, invoked when a binding matches a known path.
+
+> A `$binding` is a system unique mapping for a path to match. A `$binding` must be system unique because we don't want two handlers for a path; what happens if there are two different actions for `[~ /]`?
+
+```hoon
+::  +binding: A rule to match a path.
+::
++$  binding
+  $:  site=(unit @t)        ::  site: the site to match (~ for your.urbit.org)
+      path=(list @t)        ::  path: matches this prefix path
+  ==
+::  +action: the action to take when a binding matches an incoming request
+::
++$  action
+  $%  [%gen =generator]     ::  dispatch to a generator
+      [%app app=term]       ::  dispatch to an application
+      [%authentication ~]   ::  internal authentication page
+      [%eauth ~]            ::  cross-ship authentication handling
+      [%logout ~]           ::  internal logout page
+      [%channel ~]          ::  gall channel system
+      [%scry ~]             ::  gall scry endpoint
+      [%name ~]             ::  respond w/ @p requester is authenticated as
+      [%host ~]             ::  respond w/ @p of the ship serving the response
+      [%four-oh-four ~]     ::  respond with the default file not found page
+  ==
+```
+
+`/sys/vane/eyre` is relatively more straightforward than (say) Ames. There is only one interface (search for `~%  %http-server`).
+
+#### Authentication
+
+Client sessions typically require a login. (This is not true for materials served to the clearweb, e.g. via `%blog`.)  A cookie is generated for each session in response to a login using `+code`.
+
+```hoon
+::  +authentication-state: state used in the login system
+::
++$  authentication-state
+  $:  sessions=(map @uv session)  ::  map of cookies to session information
+      visitors=(map @uv visitor)  ::  visitors: in-progress incoming eauth flows
+      visiting=(map ship logbook) ::  visiting: outgoing eauth state per ship
+      endpoint=[user=(unit @t) auth=(unit @t) =time]  ::  endpoint: hardcoded 
+                                                      ::  local eauth endpoint
+                                                      ::  for %syn and %ack
+  ==
+```
+
+(Visits are part of the EAuth system, _q.v._)
+
+Authentication is handled by `+authentication`.
+
+- See `+authentication` in `/sys/vane/eyre`. Locate where the session cookie is created and logged.
+
+Authentication is enforced by `+request-is-logged-in` and `+request-is-authenticated`.
+
+- See `+request-is-logged-in` in `/sys/vane/eyre`.
+
+#### Channels
+
+> Channels are the main method where a webpage communicates with Gall apps. Subscriptions and pokes are issues with `PUT` requests on a path, while `GET` requests on that same path open a persistent `EventSource` channel.
+> The `EventSource` API is a sequence number based API that browser provide which allow the server to push individual events to the browser over a connection held open. In case of reconnection, the browser will send a `'Last-Event-Id'` header to the server; the server then resends all events since then.
+
+An [`EventSource`](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) interface is a way to track server-sent events for a client session. The JS on the browser/client-side receives `text/event-stream` formatted events. So a channel is a given connection to a browser including the `EventSource` connection.
+
+```hoon
+::  channel: connection to the browser
+::
++$  channel
+  $:  mode=?(%json %jam)
+      =identity
+      state=(each timer duct)
+      next-id=@ud              ::  next-id: next sequence number to use
+      last-ack=@da             ::  last-ack: time of last client ack
+      events=(qeu [id=@ud request-id=@ud =channel-event])  ::  unacked events
+      unacked=(map @ud @ud)    ::  unacked event counts by request-id
+      subscriptions=(map @ud [ship=@p app=term =path duc=duct])  ::  gall subs
+      heartbeat=(unit timer)   ::  sse heartbeat timer
+  ==
+::  +session: server side data about a session
+::
++$  session
+  $:  =identity                ::  authentication level & id of this session
+      expiry-time=@da          ::  when this session expires
+      channels=(set @t)        ::  channels opened by this session
+  ==
+::  channel-state: state used in the channel system
+::
++$  channel-state
+  $:  session=(map @t channel)   ::  mapping b/w an arbitrary key to a channel
+      duct-to-key=(map duct @t)  ::  mapping from ducts to session key
+  ==
+::  channel-event: unacknowledged channel event, vaseless sign
+::
++$  channel-event
+  $%  $>(%poke-ack sign:agent:gall)
+      $>(%watch-ack sign:agent:gall)
+      $>(%kick sign:agent:gall)
+      [%fact =desk =mark =noun]
+  ==
+```
+
+Conventional channels communicate in JSON. Values passed into Urbit can be sent through a mark file to be transformed into a `%noun` or other type automatically. On the way out, a similar transformation can take the values back into MIME types.
+
+- Trace `%poke` and `%poke-json` in `/sys/vane/eyre`.
+- Examine [`/lib/schooner`](https://github.com/urbit/yard/blob/main/desk/lib/schooner.hoon) and the `/mar` files in [the `%yard` desk](https://github.com/urbit/yard). How does it handle JSON transformations?  What about binary types like `audio/mpeg` (MP3)?
+
+Noun channels make it possible for external applications to speak Urbit nouns. This means that you can communicate with an Urbit ship in a way other than using a JSON payload. The `content-type` is marked as `application/x-urb-jam`. Nouns are `+jam`med when sent into Eyre.
+
+- Locate where `x-urb-jam` is processed in `/sys/vane/eyre` and in `/mar/noun`.
+
+Thanks to the mark system and `+find-channel-mode`, it is straightforward on Urbit's side to implement noun channels. However, on the other side you need something that speaks nouns, such as [`noun.py`](https://github.com/urbit/tools/blob/master/pkg/pynoun/noun.py).
+
+- [“Guide:  Noun channels”](../../urbit-os/kernel/eyre/noun-channels.md)
+- [~nordus- mocwyl, “bird-brained guide to noun channels”](https://bird.howm.art/guide/noun-channels)
+
+#### HTTP request handling
+
+All else set aside, the real purpose of Eyre is to act as the HTTP server for an Urbit ship. Eyre maintains a server configuration. There is an `$inbound-request` type to receive an HTTP request, but the main HTTP types are actually in another arm, `+http`.
+
+A raw HTTP request handle happens like this:
+
+- Eyre subscribes to an app at `/http-response/[eyre-id]`.
+- Eyre pokes the app with `%handle-http-request` and the ID.
+- The app produces `%fact`s of `?(%http-response-header %http-response-data %http-response-cancel)`.
+
+```hoon
+:: +http-config: full http-server configuration
+::
++$  http-config
+  $:  secure=(unit [key=wain cert=wain])  ::  PEM-encoded RSA private key and 
+                                          ::  cert or cert chain
+      proxy=_|                            ::  reverse TCP proxy HTTP(s)
+      log=?                               ::  keep HTTP(s) access logs
+      redirect=?               ::  send 301 redirects to upgrade HTTP to HTTPS
+  ==
+:: +http-rule: update configuration
+::
++$  http-rule
+  $%  [%cert cert=(unit [key=wain cert=wain])]  ::  set/clear cert and keypair
+      [%turf action=?(%put %del) =turf]   ::  add/remove established dns binding
+  ==
+::  +address: client IP address
+::
++$  address
+  $%  [%ipv4 @if]
+      [%ipv6 @is]
+      ::  [%ames @p]
+  ==
+::  +inbound-request: +http-request and metadata
+::
++$  inbound-request
+  $:  authenticated=?          ::  has a valid session cookie
+      secure=?                 ::  whether this request was encrypted (https)
+      =address                 ::  the source address of this request
+      =request:http            ::  the http-request itself
+  ==
+```
+
+- Examine `+http`. Find the `request` and `response` handlers. In particular, see `$simple-payload`.
+- Examine `/lib/server`, which contains wrapper arms for mere mortals.
+
+In some ways, although this is the meat-and-potatoes of Eyre, it's all rather straightforward.
+
+To actually get a value into userspace, Eyre sends the `response` to Gall:
+
+- `+request-to-app` to dispatch an `%app` `$action` to Gall.
+- `+deal-as` to `%pass` to Gall.
+
+#### Threads and generators
+
+We can treat a (local) ship as a “serverless” function call for a client.
+
+A generator is a standalone computation based on arguments. Eyre supports generators explicitly:
+
+```hoon
++$  generator
+  $:  =desk           ::  desk on current ship that contains the generator
+      path=(list @t)  ::  path on :desk to the generator's hoon file
+      args=*          ::  arguments passed to the gate
+  ==
+```
+
+and runs them in `+request`, branch `%gen`.
+
+- How does the generator run?  Note the `$roof` and the `+mock` call.
+
+Having bound a generator `/gen/eyre-gen`
+
+```hoon
+|=  [[now=@da eny=@uvJ bec=beak] ~ ~]
+|=  [authenticated=? =request:http]
+^-  simple-payload:http
+=/  msg=@t
+  ?~  body.request
+    (scot %da now)
+  (cat 3 (cat 3 (scot %da now) 10) q.u.body.request)
+=/  data=octs
+  (as-octs:mimes:html msg)
+=/  =response-header:http
+  [200 ['Content-Type' 'text/plain']~]
+[response-header `data]
+```
+
+to an endpoint `/mygen`,
+
+```hoon
+|pass [%e [%serve `/mygen %base /gen/eyre-gen/hoon ~]]
+```
+
+a client may initiate a generator call by posting a `PUT` request thus:
+
+```sh
+curl -i http://localhost:8080/mygen --data 'blah blah blah'
+```
+
+- Examine the [“Eyre Guide:  Generators”](../../urbit-os/kernel/eyre/guide.md#generators) example.
+
+A thread is a transient standalone computation similar in some regards to a generator. Spider provides thread support using an Eyre binding.
+
+```hoon
+++  handle-http-request
+  ~/  %handle-http-request
+  |=  [eyre-id=@ta =inbound-request:eyre]
+  ^-  (quip card _state)
+  ?>  authenticated.inbound-request
+  =/  url  (parse-request-line:server url.request.inbound-request)
+  ?>  ?=([%spider @t @t @t @t ~] site.url)
+  =*  desk         i.t.site.url
+  =*  input-mark   i.t.t.site.url
+  =*  thread       i.t.t.t.site.url
+  =*  output-mark  i.t.t.t.t.site.url
+  =/  =tid         (new-thread-id thread)
+  =.  serving.state  (~(put by serving.state) tid [`eyre-id output-mark desk])
+  =/  tube  (convert-tube %json input-mark desk bowl)
+  ?>  ?=(^ body.request.inbound-request)
+  =/  body=json  (need (de-json:html q.u.body.request.inbound-request))
+  =/  input=vase  (slop !>(~) (tube !>(body)))
+  =/  boc  bec
+  =/  =start-args:spider  [~ `tid boc(q desk, r da+now.bowl) thread input]
+  (handle-start-thread start-args)
+```
+
+- Examine the [“HTTP API”](../../urbit-os/base/threads/http-api.md) example.
+
+#### EAuth
+
+Eyre's EAuth system “a mechanism by which HTTP clients may authenticate themselves as a specific urbit on HTTP endpoints served by any other urbit.”  In other words, you can provide a comet-like client to an arbitrary client.
+
+- How robust to collision is random EAuth assignment?  ($\frac{1}{2^{128}-2^{64}} \approx 3^{-39}$, or one in 100 undecillion)
+- If you are interested in investigating EAuth in detail, see `$visitor`, `$logbook`, `$eauth-plea`, `$eauth-boon` as well as the source description at [~palfun-foslup, “mirage (eauth)”](https://gist.github.com/Fang-/41ed84b2a6dd96ca67c6a5dbda1fd35d) and the app [~paldev, %chat-stream](https://pal.dev/lobby/).
+
+### SSL {#ssl}
+
+If you are working locally, you typically just have HTTP set up instead of secure HTTPS. SSL is a transport layer protocol formerly used for client–server encrypted channels, but now HTTPS actually uses TLS.
+
+The `%acme` agent configures a certificate if you have a domain set up to use with Urbit.
+
+- Read the `+install` arm in `/app/acme`.
+
+#### Scry interface
+
+Eyre exposes some information about bindings and connections, such as the sessions and cookies:
+
+```hoon
+.^(authentication-state:eyre %e /=authentication-state=)
+
+.^((list [binding:eyre duct action:eyre]) %e /=bindings=)
+```
+
+### Vere I/O Driver:  `vere/io/http.c` {#vere-io-driver-vereiohttpc}
+
+The runtime counterpart to Eyre is `vere/io/http.c`, which is the HTTP server.
+
+Vere's `http.c` uses `libh2o` as its HTTP server:
+
+> H2O is a new generation HTTP server that provides quicker response to users with less CPU utilization when compared to older generation of web servers. Designed from ground-up, the server takes full advantage of HTTP/2 features including prioritized content serving and server push
+
+Take especial note of the following functions:
+
+- `u3_http_io_init()` to start the HTTP server manager.
+- `_http_serv_listen_cb()`, the callback for receiving a value
+- `_http_serv_accept()`
+- `_http_seq_accept()` to process a new HTTP request
+- `_http_hgen_send()` to send an HTTP response
+
+
+## Iris {#iris}
+
+Iris is an HTTP client. It is not currently widely used since Urbit ships do not often serve as HTTP clients (rather as peers).
+
+### `/sys/lull` Definition {#syslull-definition}
+
+```hoon
+::  %iris http-client interface
+::
+++  iris  ^?
+  |%
+  ::  +gift: effects the client can emit
+  ::
+  +$  gift
+    $%  [%request id=@ud request=request:http]  ::  outbound http-request
+        [%cancel-request id=@ud]                ::  tell earth to cancel a 
+                                                ::  previous %request
+        [%http-response =client-response]       ::  response to the caller
+    ==
+  ::
+  +$  task
+    $~  [%vega ~]
+    $%  $>(%born vane-task)  ::  system started up; reset open connections
+        $>(%trim vane-task)  ::  trim state (in response to memory pressure)
+        $>(%vega vane-task)  ::  report upgrade
+        [%request =request:http =outbound-config]  ::  fetches a remote resource
+        [%cancel-request ~]  ::  cancels a previous fetch
+        [%receive id=@ud =http-event:http]  ::  receives http data from outside
+    ==
+  ::  +client-response: one or more client responses given to the caller
+  ::
+  +$  client-response
+    $%  $:  %progress        ::  periodic update along %fetch source duct
+            =response-header:http     ::  full transaction header
+            bytes-read=@ud            ::  bytes fetched so far
+            expected-size=(unit @ud)  ::  size if response had content-length
+            incremental=(unit octs)   ::  data received since last update
+        ==
+        [%finished =response-header:http full-file=(unit mime-data)] ::  final
+                                                                     ::  response
+        [%cancel ~]          ::  canceled by runtime system
+    ==
+  ::  mime-data: externally received but unvalidated mimed data
+  ::
+  +$  mime-data  [type=@t data=octs]
+  ::  +outbound-config: configuration for outbound http requests
+  ::
+  +$  outbound-config
+    $:  redirects=_5         ::  number of times to follow 300 before error
+        retries=_3           ::  number of retries before failing
+    ==
+  --
+```
+
+### Structure {#structure}
+
+`/sys/vane/iris` is quite short and legible. Most of the vane is either tracking connection state as a client or sending updates for data transmission progress.
+
+- Examine the [“Iris Guide:  Example”](../../urbit-os/kernel/iris/example.md) thread.
+- Use this as a springboard for examining how the response header and body are constructed in Urbit.
+
+Iris has relatively little information to expose at any given time, and has an extremely minimal scry interface:
+
+- `%i %x %whey`, show memory usage.
+
+### Vere I/O Driver:  `vere/io/cttp.c` {#vere-io-driver-vereiocttpc}
+
+The runtime counterpart to Iris is `vere/io/cttp.c`, which is the HTTP client.
+
+- Read `u3_cttp_io_init()`, which initializes the client manager state.
+
+Like Eyre, Vere's Iris uses [`libh2o`](https://powerdns.org/libh2o/) as its HTTP server/client library.
+
+- Read `_cttp_creq_on_body()`, the callback upon receiving a response body
+- `_cttp_creq_respond()`
+- `_cttp_http_client_receive()`
+
+
+## Exercise {#exercise}
+
+- Produce an app which allows a clearweb login. This can be done using EAuth but it would be interesting to implement standard username/password login as well.