# Data demand some damn respect.
Loyal bytes toil in memory, on disk, across wires, etc. They crunch our numbers;
we cast them to the garbage collector. _Et tu, brutus computus._
And so I've been thinking about "compassionate" compilation. How can we learn to
love our languages again? How can we reduce our energy footprint? How can we
tame the chaos monkeys? How can we reignite the joy of simple software? How can
we share our abundance as if recycling air?
## The Modern Stack
Modern web computing looks vaguely like this:
- TSX → unspeakable horror → HTML/CSS/JS
- HTML/CSS/JS on CDN → HTTP → HTML/CSS/JS into DOM → process layout and run JS →
virtual DOM
- Rust → ??? → LLVM IR → LLVM → server executable → Dockerfile → Docker image
- virtual DOM → DOM → browser → JSON → HTTP → JSON → server executable -> SQL →
PG protocol → query planner → PG tuples → MVCC magic → PG protocol → server
executable → JSON → HTTP → JSON → browser → DOM → virtual DOM
This is barely a map of the _happy path_. It skips error states, compiler
internals, OS, CI, package management, containers, garbage collection, CPU
caches, etc. How many allocations--
Nope. It's too much for my sad synapses. I cannot pretend to comprehend the
journey that my data take to load a single web app.
## A Convenient Package
Computers are crafted at the _nanometer_ scale; they are the pinnacle of human
precision. But software
[squanders this power](https://en.wikipedia.org/wiki/Wirth%27s_law). Our
precision engines evoke frustration, bugs, and inconsistency -- "did you try
turning it off and on again?". Codebases devolve into chimeras of LEGO, Lincoln
Logs, cement, popsicle sticks, duct tape -- software cannot succeed like this.
But to wrest LEGO and Lincoln Logs from y'all would be folly. I cannot coerce an
entire industry to converge on compatible standards. No, no -- please keep your
familiar tools/toys.
Here's my proposal: we choose one "good enough" data layout, and we squeeze it
for as much mileage as it can bear. Such a format must encode most popular
formats, respect bandwidth, support single-pass execution, allocate sparingly,
afford sublinear lookup (for lists/dicts), and remain simple to
understand/implement. Good news: our computing forefathers already solved these
problems -- we just need to cram them into a single convenient package.
**tl;dr** flat scraps = bootleg [MsgPack](https://msgpack.org) + bootleg
[IPFS](https://ipfs.tech) + bootleg
[XSD]() + bootleg
[FORTH]()
## Boring Data
Integers, floats, strings, bytes, arrays, maps, variants, types -- any
serialization format that encodes these data will do. CBOR and MsgPack are
reasonable starting points.
[^note]:
Flat scraps are still in the discovery phase! Meanwhile, check out
[Max Bernstein](https://bernsteinbear.com)'s
[elegant ~100 LOC scrapscript serializer](https://github.com/tekknolagi/scrapscript/blob/trunk/scrapscript.py#L1028-L1182).
Most popular markup/config/data formats (JSON, YAML, XML, CSS, HTML, CSV,
Dockerfile, etc.) can be efficiently encoded as CBOR/MsgPack (sketched here as
scrapscript types):
```
()
; csv :
list text
; css :
list css-rule
; css-rule :
| #style { selector : text, declarations : list decl }
| #at { name : text, prelude : text, block : maybe (list decl) }
; css-decl :
{ property : text, value : text, important : bool }
; html :
xml
; xml :
| #text text
| #float float
| #bool bool
| #object (list xml)
| #doctype { name : text, attrs : dict text xml }
| #tag { name : text, attrs : dict text xml, body : xml }
; yaml :
json
; json :
| #null
| #bool bool
| #int int
| #float float
| #text text
| #list (list json)
| #dict (dict text json)
; dockerfile :
{ base : (pair text text)
, instructions : list (
| #from (pair text text)
| #run text
| #copy (pair (list text) text)
| #workdir text
| #env (list (pair text text))
| #expose (list int)
| #volume (list text)
| #user text
| #entrypoint (list text)
| #cmd (list text)
| #comment text
)
}
```
CSS parsers should live in editors, not browsers. You can still write as much
CSS as you want -- just don't send it around as CSS _text_. Text is floppy. It's
hard to cache, hard to compress, hard to parse, hard to validate, etc. Send data
structures instead.
## Shareable Data
By shoving all these data structures into the same serialization format, all
software can reap the rewards of common infrastructure: shared compression,
shared schema validation, shared optimizations, shared versioning, etc.
Scrapscript (and flat scraps) extend "sharing" further. You can
[replace any "scrap" of data with its cryptographic hash](https://scrapscript.org/guide#Scrapyards)
to reference it and reuse it. For example, we could serialize HTML pages like
this:
```
hello world
```
Suddenly, without any syntax changes, every format acquires the powers of git
_at the semantic level_. Replace any chunk of JSON/CSV/etc with its hash and it
can hit a cache on the other side of the wire.
It gets better. We can use [scrap maps](https://scrapscript.org/guide#ScrapMap)
to name (and version) arbitrary hashes:
```