http — HTTP client builtins

Response streams and idle timeouts

http_get, http_post and http_request accept idle_timeout: positive seconds (fractions allowed, at most one year), limiting the gap between received response bytes, including the initial wait for response headers. Connection establishment and acquisition of the complete response headers are each also bounded by this duration. It is optional for buffered calls; omitting it preserves their existing behaviour. timeout remains the whole-call cap (integer seconds, default 30). timeout: 0 disables the whole-call cap, while an explicitly supplied idle limit still applies. With timeout: 0, a server dripping a byte just inside idle_timeout can hold the call indefinitely; accepting that is the caller's trade-off.

$r = http_get($url, {timeout: 0, idle_timeout: 5})

http_stream(method, url, on_line[, body, headers, opts]) reads a response as it arrives. It uses the same body, headers and TLS options as http_request. Supply nil for unused slots. Its default idle limit is 30 seconds; set timeout: 0 for a stream that may run indefinitely while data keeps arriving. HTTP error statuses also deliver their response lines.

$r = http_stream("POST", "https://api.example.com/messages",
  fn($line)
    print($line);
    return $line != "data: [DONE]"
  end,
  json_encode({stream: true}),
  {"Content-Type": "application/json", Accept: "text/event-stream"},
  {timeout: 0, idle_timeout: 5})
if $r.status == 0 then print($r.error_code .. ": " .. $r.error) end

The callback receives one UTF-8 line without LF or CRLF, including empty lines and a final line without a newline. Framing recognises LF and CRLF; a bare CR is not a terminator. SSE event assembly (blank-line boundaries, data: fields and multi-line payloads) belongs to the caller. Return the boolean false to stop and close promptly; any other return value continues. A callback exception propagates after releasing the response. Invalid UTF-8 returns HTTP_TEXT. Each line is capped at 64 MiB; the stream has no aggregate body cap. Callbacks run sequentially and apply backpressure. Avoid slow callbacks: idle reads are measured when the response is polled, and HTTP deadlines are checked after each callback returns. The evaluator uses its own interrupt checkpoints inside callbacks; the HTTP reader never cancels a callback midway through its evaluator frame. Synchronous callback work cannot be pre-empted.

The result includes status, headers, final_url, duration_ms, error_code, error, bytes_received and stopped (true on callback stop). stopped is emitted only by http_stream. Buffered successes have exactly the established response keys; buffered failures with idle_timeout also include bytes_received for partial-response diagnostics. For streams, body/bytes contain only an unfinished line on failure; completed lines have already reached the callback. Successful streams return empty body/bytes. Byte counts include line terminators and any bytes already read ahead in the current chunk. Counts refer to decoded body bytes, excluding HTTP framing and compressed wire bytes. Buffered calls with an idle limit retain all received bytes in bytes and UTF-8 body (otherwise nil), including on failure, subject to the existing 64 MiB cap.

Operational failures are values: status: 0 with HTTP_IDLE_TIMEOUT for inactivity, HTTP_TIMEOUT for the whole-call cap, HTTP_STREAM_CUT for a body cut short (Content-Length or chunked framing), or HTTP_INTERRUPTED. Close-delimited EOF is a normal end: without framing, a client cannot tell an intentional close from a truncated application message. Received bytes remain counted even when an error follows. Inspect the result before trusting application-level completeness.

This callback API follows Mix's evaluator-level higher-order builtins. The active call owns the async response, so early return, callback errors, SIGINT/SIGTERM cancellation and retiring the evaluator drop the connection; there is no persistent handle to leak or to close in a finally block. Network capability is required, with FsRead additionally required for ca_file, just as for the buffered calls.

Implementation: the idle/stream path uses the workspace's async reqwest client. With no whole-call deadline, ureq 2's socket idle timeout does work. When both are configured, its total deadline overrides the socket read timeout. It also exposes a blocking response reader, so cancellation cannot drop an in-flight read promptly. Async reads let Mix enforce both budgets and drop the response on cancellation without a detached reader thread. Idle option validation is shared with file transfers. Calls without an idle option keep their established ureq path.

TLS uses the same explicit rustls configuration: the same webpki roots as the existing HTTP builtins, ca_file/ca_pem added to that set, and the same signature-checking but trust-bypassing verifier for ssl_verify: false. The CA options remain mutually exclusive with each other and with disabled verification. Reqwest inherits default-features: false and rustls-tls from the workspace; only gzip is advertised and transparently decoded, matching ureq's default features. Brotli is disabled. bytes_received counts the decoded body bytes, not compressed bytes on the wire. This change enables neither native-tls nor OpenSSL and uses the existing rustls 0.23 version. The build worker has verified a static-pie x86_64-unknown-linux-musl release build with -C target-feature=+crt-static and no shared libraries.

Redirects match http_get/http_post/http_request: ureq's configured budget of five requests permits four followed redirects and fails on a fifth 3xx. 301/302/303 switch methods other than GET/HEAD to GET; 307/308 follow only GET/HEAD/OPTIONS/TRACE, without replaying a request body. Authorization, Cookie and Content-Length are removed when following a redirect, including to the same host. The idle/stream path additionally removes a caller-set Host so the next request uses the destination's host. URL userinfo uses ureq's literal (not percent-decoded) Basic auth encoding; an explicit Authorization header takes precedence on the initial request. The final URL is reported. The separate file-transfer builtins disable redirects and return 3xx. These are distinct existing policies.

Streaming file download (0.97.0)

http_get_file(url, path[, opts]) -> map<http_file_response> streams GET to a unique sibling created exclusively. For a new destination it uses mode 0666 under the process umask, matching write_atomic. When the destination exists, staging starts at 0600, before any append prefix or incoming body is copied; the target's permission bits are applied through the open descriptor only after verification, just before publication. It hashes while writing, syncs the completed file, publishes atomically, then syncs the directory. The parent directory must exist. Defaults refuse an existing destination: Linux uses renameat2(RENAME_NOREPLACE), with atomic hard-link creation as the fallback. Unsupported filesystems fail; there is no check-then-rename fallback. overwrite: true permits atomic replacement of a regular file, preserving its current permission bits. Symlink replacement is refused.

Options are overwrite, append, expect_blake3 (64 lowercase hex digits), max_bytes, headers, and the upload timeout/TLS options below. Redirects are returned. Only a 200 or 204 installs a fresh file. HTTP error bodies may be returned in body/bytes (64 MiB cap), but never become the destination. Successful file bodies stay on disk; body is nil and bytes empty. From 0.97.0, 200/206 downloads require identity framing with Content-Length. Missing length or non-identity encoding returns status 0 with HTTP_IDENTITY_FRAMING and leaves the destination untouched. This also rejects ureq's transparent gzip decoding, which removes both Content-Encoding and Content-Length. Chunked/close-delimited 200/206 bodies are deliberately refused; blobd's lane supplies Content-Length. A body-free 204 remains supported.

append: true requires expect_blake3 for the whole final file and an existing regular prefix file. It copies and hashes the prefix into staging, requests Range: bytes=<prefix-size>-, and accepts only a 206 with one Content-Range starting at that size and ending at the representation's final byte. An ignored Range, wrong range, short read or hash mismatch leaves the prefix unchanged. Append and overwrite are mutually exclusive; append itself authorises replacing the prefix after verification. Keep the destination exclusive to this caller during append/overwrite. The builtin owns Range and Accept-Encoding (requests identity); these headers cannot be overridden.

max_bytes applies to the final size, including the prefix, and defaults to 9007199254740991 (the exact integer limit). bytes_written counts downloaded bytes written to staging; size includes the copied prefix. blake3 is the verified final digest, or nil on failure. The extra published boolean is true once publication succeeds, including the rare status-0 directory-sync failure after publication; callers must not interpret that failure as rollback. All earlier failures remove staging and preserve the destination.

Operational failures return status 0 with FILE_IO, FILE_EXISTS, HTTP_RANGE, HTTP_HASH_MISMATCH, HTTP_SHORT_READ, HTTP_BODY_LIMIT, or the HTTP transport/body codes. Invalid options raise. Network and FsWrite are unconditional; append:true or ca_file additionally requires FsRead. The same socket inactivity and cooperative deadline limitations apply as for uploads. Memory use for successful downloads stays bounded by a 64 KiB buffer.

Streaming file upload (0.97.0)

http_put_file(url, path[, opts]) -> map<http_file_response> streams a regular file with a fixed 64 KiB transfer buffer. Keep the source immutable during the call. range: {start, end} selects inclusive byte offsets; without it the whole file is sent, including an empty file. Content-Length is exact and an early EOF fails with HTTP_SHORT_READ. Sizes and offsets must be exact Mix integers (0 through 9007199254740991); invalid ranges raise OPTION_INVALID.

Options are method (PUT, POST, or PATCH, default PUT), range, headers, idle_timeout (seconds, default 30, positive), deadline (seconds, default 0 = disabled), ssl_verify, ca_file, and ca_pem. TLS options have the same validation and trust roots as the buffered HTTP builtins below. Unknown options raise. The builtin owns Content-Length and Transfer-Encoding; setting either header raises. Redirects are disabled: a 3xx is returned.

idle_timeout bounds socket inactivity (and connection establishment). deadline is checked cooperatively around streaming IO. It is best-effort during blocking DNS/IO, not a hard whole-call bound. Both accept fractional seconds, up to one year.

The distinct http_file_response has status, headers, bytes_written, size, blake3, body, bytes, final_url, duration_ms, error_code, and error, and published (false for uploads). For uploads, bytes_written counts source bytes consumed, not a server's durable acknowledgement; size is the selected window size and blake3 is its lowercase digest (nil on failure). Response bodies are capped at 64 MiB. Received HTTP responses retain their real status; IO/transport failures return status 0 and an error (FILE_IO, HTTP_SHORT_READ, or the HTTP codes below). Argument errors raise. Inspect the result even when the call does not raise.

A server can reject a large PUT/PATCH before consuming its body. If the socket fails while ureq is still writing, the caller can receive status 0 and lose the server's status and diagnostic body. For resumable blob uploads, recover via HEAD and its durable offset/state rather than assuming the chunk was accepted. Downloads refuse close-delimited 200/206 bodies without Content-Length, as required by the identity-framing contract above.

Network and FsRead are unconditional capabilities, including calls without options. Structured discovery and mix lint include both. The http Cargo feature itself enables BLAKE3; crypto is not required for these transfers.

The HTTP client builtins — a small, blocking HTTP/1.1 client built on ureq. Three calls cover the whole surface: http_get, http_post, and the any-verb http_request. They are feature-gated (the http cargo feature) in mixos-lib-mix; the shipped mix binary turns the feature on, so they are always present in the CLI. In the builtin catalogue they list under the system category (mix builtins system); one-line help for any name: mix what http_get.

Mental model: every call returns a map, never raises, for anything that happens on the network. A successful or HTTP-error response is {status, body, bytes}; a transport failure — including a timeout — is {status: 0, error}. You branch on status, you don't wrap the call in try/catch. (Argument mistakes — wrong arity, a bad opts map, a bytes header value — do raise: they are script bugs, not network weather.)

$r = http_get("https://example.com")
print("status=" .. $r["status"])
print("body_len=" .. length($r["body"]))
status=200
body_len=559

The four calls

http_get(url, [headers], [{timeout, idle_timeout, ssl_verify, ca_file, ca_pem}])                      -- GET
http_post(url, body, [headers], [{timeout, idle_timeout, ssl_verify, ca_file, ca_pem}])               -- POST with a body
http_request(method, url, [body], [headers], [{timeout, idle_timeout, ssl_verify, ca_file, ca_pem}])  -- any verb
http_post_multipart(url, fields, files, [headers], [opts])                              -- POST multipart/form-data
  • url — a string (anything else is stringified via the usual coercion).
  • headers — an optional map of header name → value (see Headers).
  • body — a string (sent as UTF-8) or a bytes buffer (sent raw); see Request bodies.
  • fields / files (http_post_multipart only, v0.103.12) — fields is a map of string form fields; files is a map of field_name → {filename, content_type?, data: bytes|buffer}. The multipart boundary is generated and the Content-Type header set accordingly — the generic webhook/file-upload slot, so a vendor upload is a script over this, never a core dependency (see Request bodies).
  • method (http_request only) — upper-cased internally, so "get", "Get", "GET" are equivalent. It must be a valid RFC 7230 token (no spaces / control bytes) or the call returns the error shape — this guards against request-line injection.
  • {timeout, idle_timeout, ssl_verify, ca_file, ca_pem} — an optional trailing opts map. timeout bounds the whole request (default 30 s, {timeout: 0} disables — see Timeouts). ssl_verify defaults to true; ssl_verify: false skips TLS certificate and hostname verification for that call (like curl -k) — see Skipping TLS verification.
  • ca_file / ca_pem (v0.29.0) — trust a private CA for this call: ca_file reads PEM certificate(s) from a path, ca_pem takes them inline (string/bytes/buffer). The certificates are ADDED to the default (Mozilla webpki) roots — chain building and hostname verification still run in full, so this is the right way to talk to an internal endpoint with a proper private-CA-issued cert (the readiness-check case), unlike ssl_verify: false which proves nothing. Mutually exclusive with each other and with ssl_verify: false; input capped at 4 MiB; a missing/unreadable file, invalid PEM, or PEM with no certificates raises a catchable HTTP_TLS structured error (see errors). In a capability-sandboxed embedder, ca_file additionally requires the fs-read class (declared as a conditional capability in the builtin metadata).

http_get takes 1–3 arguments, http_post 2–4, http_request 2–5. Too few raises a runtime error, and — unlike the Mix-wide minimum-arity convention — too many also raises, so a misplaced opts map can't be silently ignored and leave a call unbounded:

http_get()
-- Runtime error at line 1: http_get() expects at least 1 argument(s), got 0
http_post("https://example.com")
-- Runtime error at line 1: http_post() expects at least 2 argument(s), got 1
http_get($url, {}, {}, {})
-- Runtime error at line 1: http_get() expects at most 3 argument(s), got 4

The return shape

Every call resolves to one of two map shapes.

Success / HTTP-error response

keytypemeaning
statusnumberthe HTTP status code (200, 404, 503, …)
bodystring | nilthe body decoded as UTF-8; nil when the bytes are not valid UTF-8
bytesbytesthe raw response byte buffer, always present
headersmapresponse headers — lowercase names → list of string values (repeated fields preserved), e.g. $r.headers["content-type"] is ["text/html"] (v0.30.0)
final_urlstringthe URL after any redirects (v0.30.0)
duration_msnumberwall-clock request duration (v0.30.0)
error_codestring | nilnil on any HTTP response (incl. 4xx/5xx); an HTTP_* code only on a transport failure (v0.30.0)
errorstring | nilnil on an HTTP response; the message on a transport failure

The headers/final_url/duration_ms/error_code keys are additive — pre-0.30 scripts reading status/body/bytes are unaffected.

A 4xx / 5xx is a response, not a failure — it carries the real code plus whatever error payload the server sent, so REST callers can read both:

$r = http_get("https://example.com/does-not-exist-xyz")
print("status=" .. $r["status"])
print("body=" .. substr($r["body"], 0, 40))
status=404
body=<!doctype html><html lang="en"><head><ti

The documented success test is status == 200 (or a 2xx range check) — it is unaffected by error statuses landing in status rather than the error shape.

Transport failure — {status: 0, error}

Only a genuine transport error — DNS failure, TLS handshake, connection refused, a deadline expiry, a mid-stream read error, an invalid method, or an over-cap body — collapses to status == 0 with a human-readable error string and no body/bytes:

$r = http_get("http://127.0.0.1:1/")
print("status=" .. $r["status"])
print("error=" .. $r["error"])
status=0
error=http://127.0.0.1:1/: Connection Failed: Connect error: Connection refused (os error 111)

status: 0 is the one sentinel that means "no HTTP exchange happened" — a real server never returns code 0, so a single if $r["status"] == 0 cleanly separates "the network broke" from "the server answered". The canonical guard:

$r = http_get($url)
if $r["status"] == 0 then
  print("transport error: " .. $r["error"])
else
  -- $r["status"] is a real HTTP code; $r["body"]/$r["bytes"] are present
end

Text vs binary — body and bytes

bytes always carries the raw buffer. body is the UTF-8 decode of that buffer, or nil when the bytes are not valid UTF-8 — so a binary response (image, archive, anything high-bit) does not silently corrupt through a lossy decode. An empty body decodes to the empty string "", not nil — nil strictly means "not UTF-8". Reach for .body for JSON/text, .bytes for binary:

$r = http_get("https://www.google.com/favicon.ico")
print("status=" .. $r["status"])
print("body_is_nil=" .. ("" .. ($r["body"] == nil)))
print("bytes_present=" .. ("" .. (not is_empty($r["bytes"]))))
status=200
body_is_nil=true
bytes_present=true

For a text response, body is a normal string and bytes holds the same data as a raw buffer — write it straight to disk with write_file (a bytes value is written verbatim), or pass it to a bytes helper:

$r = http_get("https://example.com")
write_file("/tmp/example.html", $r["bytes"])

To measure the buffer use bytes_len($r["bytes"]) or, since v0.64.0, plain length($r["bytes"]) — both answer the byte count. The raw body can also be indexed, sliced, iterated and searched directly (bytes_find, bytes_split, bytes_starts_with); see io.

Headers

The optional headers argument is a map. Keys are header names, values are stringified the usual way. If you don't set a User-Agent, ureq's default (ureq/2.12.1) goes out — the examples here set their own because some APIs care:

$r = http_get("https://api.github.com/zen", { "User-Agent": "mix-docs" })
print("status=" .. $r["status"])
print("body=" .. $r["body"])
status=200
body=Encourage flow.

A bytes value is rejected (not stringified) so the internal <bytes:N> placeholder never ships over the wire — HTTP headers are text. If you genuinely need binary header metadata, encode it yourself:

http_get($url, { "X-Token": $raw_bytes })
-- Runtime error: http: header `X-Token` does not accept bytes; base64_encode($v) first

One carve-out: a sole map in this slot whose keys are all opts keys (timeout, idle_timeout, ssl_verify, ca_file, ca_pem) is read as the opts map, not as headers — see the trailing-map rule. Any real-world header map (carrying a non-opts key like Authorization) is unaffected.

Request bodies

http_post (and http_request's optional 3rd slot) send a body. A string is sent as UTF-8; a bytes buffer is sent raw (so a binary upload doesn't go through string coercion). Set Content-Type yourself — the client does not guess it:

$payload = "{\"hello\":\"world\"}"
$r = http_post("https://httpbin.org/post", $payload, { "Content-Type": "application/json" })
print("status=" .. $r["status"])
$j = json_parse($r["body"])
print("echoed=" .. $j["data"])
status=200
echoed={"hello":"world"}

http_request's body slot is nil-tolerant: an absent or nil 3rd argument sends a bodyless request (the right behaviour for GET/DELETE/ OPTIONS):

$r = http_request("GET", "https://example.com")   -- no 3rd arg → bodyless
print("status=" .. $r["status"])
status=200

HEAD reports the real status with an empty body — a bodyless response is not a transport failure:

$r = http_request("HEAD", "https://example.com")
print($r["status"] .. " / body=" .. length($r["body"]) .. " bytes")
200 / body=0 bytes

A HEAD response mirrors the headers a GET would return — including a Content-Encoding — but the server omits the body. The body is skipped rather than drained; before this fix, a HEAD from a server that sends Content-Encoding (example.com, github.com, …) collapsed to {status: 0, error} because the (empty) body was still run through a decoder.

An invalid method short-circuits to the error shape (no request is sent):

$r = http_request("BAD METHOD", "https://example.com")
print($r["status"] .. " / " .. $r["error"])
0 / http: invalid request method "BAD METHOD" (must be an RFC 7230 token)

Timeouts

Every call runs under a total-request deadline — 30 seconds by default — one wall-clock budget covering connect, TLS, request write, and response read (ureq itself sets no timeout; before mix 0.21 a stalled server could hang the evaluator, and a login shell, forever). Override it per call with the trailing opts map:

$r = http_get($url, {timeout: 5})            -- 5 s deadline
$r = http_get($url, $headers, {timeout: 5})  -- headers AND a deadline
$r = http_get($url, {timeout: 0})            -- no deadline (deliberately long transfer)

timeout takes a non-negative integer, in seconds. A fractional, negative, or non-number value is a loud runtime error, and so is any other key in the opts map — a typo can't silently produce an unbounded call:

http_get($url, {}, {timeot: 5})
-- Runtime error at line 1: http_get: unknown opt "timeot" (supported: timeout)
http_get($url, {timeout: 1.5})
-- Runtime error at line 1: http_get: timeout must be a non-negative integer, got 1.5

A deadline expiry is a transport failure, not a raise — the usual {status: 0, error} shape. (There is no timed_out key; that is a run_rc convention.)

$r = http_get("http://127.0.0.1:8098/", {timeout: 1})   -- a hanging server
print($r["status"] .. " / " .. $r["error"])
0 / http://127.0.0.1:8098/: Network Error: Network Error: Error encountered in the status line: timed out reading response

The trailing-map rule

The opts map rides in the last slot, after the optional headers (and, for http_request, body) slots. To keep the common case short, a sole trailing map whose keys are all opts keys (timeout, idle_timeout, ssl_verify, ca_file, ca_pem) is always read as the opts map — never as a headers map, and never as a request body:

http_get($url, {timeout: 1})                     -- deadline, not a `timeout` header
http_get($url, {ssl_verify: false})              -- skip TLS verification
http_post($url, $body, {timeout: 1})             -- deadline, not a header
http_request("GET", $url, {timeout: 1})          -- deadline, not a body
http_request("POST", $url, $body, {timeout: 1})  -- deadline, not a header

To genuinely send a literal timeout HTTP header, spell out the opts slot — an empty opts map keeps the 30 s default:

$r = http_get($url, {timeout: "60s"}, {})   -- sends the header `timeout: 60s`

A map carrying any non-opts key (e.g. Authorization) is a plain headers map (or, in http_request's 3rd slot, a stringified body), so real-world maps never trip the rule.

Skipping TLS verification

TLS is on for every https:// URL, and a bad certificate surfaces as a transport error ({status: 0, error}), not a panic. Some internal endpoints — Proxmox VE (:8006), Proxmox Backup Server (:8007), and other appliances — serve a self-signed certificate that no public CA vouches for. Rather than pin their CA, pass ssl_verify: false in the opts map to skip certificate and hostname verification for that call, exactly like curl -k:

$r = http_get("https://b1.example:8006/api2/json/version",
              {"Authorization": $tok}, {ssl_verify: false})

The TLS handshake still happens — the connection is encrypted and the peer's signatures are checked against the ring provider's algorithms — but the certificate-chain / hostname trust decision is bypassed. That removes protection against a man-in-the-middle, so use it only for endpoints you reach over a trusted path (a private/WireGuard network, localhost, an SSH tunnel). ssl_verify defaults to true; there is no way to disable verification globally — it is always per-call and explicit.

The 64 MiB body cap

A response body is buffered fully into memory and capped at 64 MiB (67_108_864 bytes). The cap exists so a huge or endlessly-streaming response can't OOM a process embedding the Mix evaluator. An over-cap body collapses to the transport-error shape:

{status: 0, error: "http: response body exceeds the 67108864 byte cap (64 MiB)"}

This is a hard limit, not a tunable — these builtins are for API calls and modest fetches, not for streaming multi-gigabyte downloads. For a large file, shell out to a streaming tool by full path (see run), which takes its own {timeout: seconds} opts map if you want the download bounded too:

run("/usr/bin/curl -fsSL https://example.com/big.iso -o /tmp/big.iso", {timeout: 600})

Parsing responses

body is just a string, so feed it to the data builtins. JSON is the common case — json_parse turns the body into a map/list with dotted / indexed field access:

$r = http_get("https://api.github.com/repos/markc/mixos",
  { "User-Agent": "mix-docs", "Accept": "application/vnd.github+json" })
if $r["status"] == 200 then
  $j = json_parse($r["body"])
  print("name=" .. $j["name"])
  print("default_branch=" .. $j["default_branch"])
end
name=mix
default_branch=main

For richer extraction use jq(...); for a numeric field, to_number(...). See data for the full parse / serialize surface (json_parse, jq, data_encode, the bytes_* helpers).

Worked example — a guarded JSON fetch helper

The idiom in one function — single status == 0 transport guard, then branch on the HTTP code, parse on success. It uses the pass-in / return / reassign discipline (no shared globals):

fn fetch_json($url)
  $r = http_get($url, { "User-Agent": "mix-docs", "Accept": "application/json" })
  if $r["status"] == 0 then
    return { ok: false, why: $r["error"] }
  end
  if $r["status"] != 200 then
    return { ok: false, why: "http " .. $r["status"] }
  end
  return { ok: true, data: json_parse($r["body"]) }
end

$res = fetch_json("https://api.github.com/zen")
if $res["ok"] then
  print("got: " .. ("" .. $res["data"]))
else
  print("failed: " .. $res["why"])
end

Notes & sharp edges

  • Sequential calls. Buffered calls return after collecting the body; http_stream runs callbacks during the read. The idle/stream path yields during network waits so cancellation can close it. All calls retain the default 30 s whole-call cap unless disabled.
  • No automatic retries, no cookie jar, no redirect-body replay — ureq's defaults apply (redirects are followed). Add your own retry loop if you need one.
  • TLS is on for https://. A bad certificate surfaces as a transport error (status: 0), not a panic.
  • Headers and methods are validated to block request-line / header injection: a non-token method and a bytes header value are both rejected in-band (the former as {status: 0, error}, the latter as a runtime error).
  • Don't confuse Mix-string $(...) with the URL. In a double-quoted Mix string, ${name} interpolates but $(...) is literal text — build URLs with .. concat or ${...}, e.g. "https://example.com/api/" .. $id. See strings.

Serving — http_serve, http_recv (v0.75.0)

The other direction, deliberately minimal. The boundary first: no TLS ever, and no dynamic handlers — certificates, vhosts, and request dispatch to code are [webd]'s job and serve-mode citizens' job. What lives here is the pair of shapes an agent keeps needing locally:

http_serve(root[, {port, host, duration, index, listing, render_md,
                   requests, spa, clean_urls}])
    -> requests served (BLOCKING)
http_recv(port[, {timeout, host, max, respond}])
    -> {method, path, query, headers, body, bytes, from_host, from_port} | nil

http_serve is the python -m http.server slot: serve a directory to a browser — docs preview, a build artifact, a report. GET and HEAD only; anything else answers 405. port: 0 (the default) binds an ephemeral port and prints the URL before serving, so the caller always learns where. host defaults to 127.0.0.1; exposing to the LAN is one explicit host: "0.0.0.0". duration bounds the run in seconds (0 = until interrupted) — an in-flight request gets at most a 10-second read grace past it, so one slow client cannot hold the server open; requests stops after N exchanges; listing opts into directory indexes; render_md (markdown feature) serves .md files as styled HTML — http_serve("docs/mix", {render_md: true}) renders the raw manual .md files (but not the site's HTML shell, whose assets live at the site root — serve the whole docs/ tree, not a subdirectory, if you want the interactive site).

The boolean options (listing, render_md, clean_urls) take a real bool only — "false", 1 or nil raise TYPE_MISMATCH before the listener binds, never a truthiness read.

Serving a subdirectory of a larger site? A page that references root-absolute assets (/web/style.css) 404s them when the served root is a subdirectory — the assets are above it. Serve the site root, or pre-render a self-contained copy (see docs/_build/gen-static-site.mix for the pattern: each page carries its own inline CSS and needs no external asset, so it works from any root, curl, wget, or file://).

Traversal is the defended edge: the request path is percent-decoded first, then the resolved file is canonicalised and must remain under the canonicalised root — ../, %2e%2e%2f, and symlinks inside the root pointing out all answer 403/404, and the tests prove each one against raw sockets (a polite HTTP client normalizes ../ away; an attacker's socket does not).

Serving a site with clean URLs, two shapes. A static server maps /bus/props-core to a file of that exact name and 404s when there isn't one — but real sites use extensionless URLs. Two options cover the two site shapes, and they compose (clean_urls is tried first — a specific page beats a generic fallback):

  • clean_urls: true — a would-be 404 on an extensionless path serves <path>.html when it exists (/bus/props-core → /bus/props-core.html). This mirrors GitHub Pages' pretty-URL behaviour, and is the fix for a pre-rendered page-per-route site — a real page or a router stub at every clean URL. A section page (bus.html) sitting beside its children directory (bus/) resolves too: /bus serves the page, /bus/props-core its child, and /bus/ (trailing slash) stays a directory request. http_serve("docs", {clean_urls: true}) serves the mixos.dev docs tree locally, fully styled, as Pages does.
  • spa: true (or spa: "shell.html") — a would-be 404 on an extensionless path serves ONE shell, so a single-shell SPA's client-side router boots on every deep URL (/users/42/settings → the shell → the router renders). Point it at any Vite/React build. The shell path is checked at startup, not per request.

Both are frontend-only, and the boundary is exact: a static SPA — assets from root, data from static files or a separate API — works completely; http_serve serves the app, never runs its backend. A path with an extension (/main.css, /logo.png) is never rewritten by either option, so a genuinely missing asset stays an honest 404 instead of silently becoming the HTML shell.

http_recv catches ONE request and answers it — the OAuth localhost-redirect wait, the webhook test, the "curl me when the job is done" rendezvous. It blocks up to timeout seconds (default 30, 0 = forever), returns nil on timeout (an ordinary answer — poll again), and hands back the full request: body as UTF-8 or nil, bytes always the truth, headers lowercase-keyed. respond shapes the reply ({status, body, content_type, headers}, default 200 ok); max caps the request body (default 1 MiB, refused with 413 beyond it).

-- wait for an OAuth redirect, answer the browser, extract the code
$r = http_recv(8910, {timeout: 120, respond: {body: "You can close this tab.\n"}})
if $r == nil then die "no redirect within 2 minutes" end
$code = parse_query($r["query"])["code"]

http_serve blocks until duration/requests ends it, or until Ctrl-C — SIGINT stops the accept loop cleanly and the call returns the count served so far, so an interactive http_serve(dir) is a normal "serve until I stop it" command.

Both are sequential, Connection: close, one exchange per connection — a browser and a script, not a load. The moment the words "certificate", "virtual host", or "route this to a function" appear, the answer is webd.

See also

  • data — json_parse, jq, data_encode, bytes_to_string for parsing and serializing HTTP payloads
  • run — shelling out (curl/wget) for streaming or oversized transfers
  • strings — building URLs and bodies (.. concat, interpolation rules)
  • functions — the pass-in / return / reassign idiom used in the helper above
  • errors — why these builtins return a shape instead of raising
  • builtins index — the full builtin catalogue
  • Bus messaging — in-mesh RPC (send/emit); HTTP is for the world outside the mesh
  • the manual index — every page in this manual
  • mix what http_get · mix what http_post · mix what http_request · mix builtins system · the mix repo

This page as Markdown · This page describes the current main branch of MixOS.