Cramming scrapscript into msgpack

This is a work-in-progress! Please email me at [email protected] if you have any suggestions, criticisms, threats, etc.

msgpack is a lovely little serialization format. As a JSON replacement, it saves bandwidth while preserving native language features (e.g. tuples, records, objects, dates).

By leveraging msgpack, every scrapscript program/expression can be crammed into "flat" scraps.

When using HTTP, set Content-Type to application/scrap.

Because flat scraps encode a full programming language, you can send unambiguous programs/queries across a wire:

"Joy" |> (name -> ok { name })

-- EXPR       0xc7201f
-- "Joy"      0xa34a6f79
-- name=name  0x81a46e616d65c704786e616d65
-- REC        0xc17b7d
-- OK         0xc10006
-- name       0xc704786e616d65
-- ->         0xc12d3e
-- |>         0xc17c3e

…and represent its evaluated result:

ok { name = "Joy" }

-- EXPR        0xc7101f
-- name="Joy"  0x81a46e616d65a34a6f79
-- REC         0xc17b7d
-- OK          0xc10006

This first-class serialization format confers non-obvious benefits:

• a compact format affords a canonical format for hashing
• when computation is cheaper than bandwidth, you can send programs instead of sending data
• both code & data can share the same tooling for compression, analysis, optimization, etc.
• cheaper to store programs on disk (or KV store) and load into memory
• it's a simple/convenient cross-platform compilation target
• with types, encoders/decoders can be magically inferred/generated/typechecked at compile-time
• msgpack is supported by 50+ languages/environments; all msgpack messages are already scraps, and scrap-specific extensions can be incrementally implemented in each host language

Each extID classifies its msgpack ext datatype. Each of these bytes mnemonically corresponds to ASCII characters (e.g. record is 0x7b for {).

Versioning remains an open question! Email me at [email protected] with your wildest ideas.

Literals

Expressions

Flat scrap expressions are written in reverse polish notation:

Within the expression, prepend each 2-byte operator with 0xc1.

Here are many of the normal computer operations you know and love:

Typing is tricky. For example, we want addition to work for floats and integers, but we also don't want to introduce ambiguity. We could enforce type-specific operators, but that might introduce complexity elsewhere. As usual, email [email protected] with thoughts.

Fancier operators are also available:

To keep things tiny, we also cram common expressions into the operator space:

References

Flat scraps distinguish between scope variables (76), native functions (72), and expression hashes (24):

Hashes point to other flat scraps in a scrapyard.

Structures

A scrapscript list corresponds to a plain msgpack array:

The other structures are represented as maps:

Because maps are ambiguously typed, use special operators to clarify the intended structure:

Evaluation

Astute observers may have noticed some strange operators like where and await:

To build a basic interpreter, push values on a stack and perform operations in-place. Bind function arguments and environment variables into a scoped map that can be copied from.

Blind copies are naive! I'm cooking up some unsafe "hint" operators for advanced memory management. Stay tuned.

And someday soon, we'll all be able to do irresponsible stuff like this:

// javascript
const fib = scrap.decode(await fetch("yard.scrap.land/tom/fib"));
console.log(fib(40)); // 102334155

// javascript
import fib from "yard.scrap.land/tom/fib";
console.log(fib(40)); // 102334155

// golang
import (
  "fmt"
  "yard.scrap.land/tom"
)
func main() {
  fmt.Println(tom.Fib(40)) // 102334155
}

# python
async with aiohttp.ClientSession() as sess:
    async with sess.get("yard.scrap.land/tom/fib") as res:
        fib = scrap.decode(await res.read())
        print(fib(40)) # 102334155

// javascript
const fib = n => {
  let phi = (1 + Math.sqrt(5)) / 2;
  let asymp = Math.pow(phi, n) / Math.sqrt(5);
  return Math.round(asymp);
};
await fetch("api.example.com", {
  headers: { "Content-Type": "application/scrap" },
  method: "POST",
  body: scrap.encode(fib),
});