Omar Rizwan

Clause cards

(This note was originally an email to my GitHub sponsors.)

At Dynamicland, we depended on keyboards to program things. If you want to program something, you pull a keyboard off the shelf, put it on the table, and point it at a program (piece of paper), and you get kind of a normal-looking code editor that you can use to reprogram that object.

I don't know if we said this much publicly, but – we always resented that dependency. Once you have a keyboard and a code editor, it starts to feel like 'programming' at a gut level; it turns people off; there's this cliff where people who are perfectly comfortable playing with physical stuff balk at 'programming'. (and you get the opposite problem, too, where people who already know programming just go all into writing Lua code on the keyboard and the code editor, instead of taking advantage of the unique properties of the system)

so we had this long-standing dream of making some kind of keyboard-free, non-textual, (direct-manipulation?) programming experience.

And one time, we actually prototyped one such keyboard-free programming system. and it kind of worked! and it was pretty interesting! and no one's written about it, until now :-)

(i've found myself describing this system to multiple people recently; i started promising that i'd write a little memo or something about it, so here we are)

(This work was done in August 2019 by Bret Victor, Josh Horowitz, Luke Iannini, and me. This writeup will necessarily be a bit informal and from memory and from the photos and materials that I happen to have lying around, since I don't have a working instance of this system here.)

A clause card

This keyboard-free programming system is built around clause cards. A clause card is a square particle-board card with some stuff printed on it. Here's what a clause card looks like:

Maybe you can see that this card has a statement stored on it: /someone/ wishes ("<Page 24259>") is labelled (text)

(well, technically, that's a clause of a statement – thus, clause cards)

(the projection here is lower-resolution than I'm used to; not sure why)

Every clause card has two sides. One side is the When side; the other side is the Claim/Wish side. I'm going to talk about the When side first.

Clause card as querying tool

The first way you can think of a clause card is that it's a little search engine, a query interface for the computational world around it.

Here's a clause card that is holding the statement ("<Page 20706>") is a ("keyboard"):

Notice how it says 1 match below the clause card. Whenever you put a clause card on the table (When-side up), the card immediately and continuously queries the world for its statement. (Is there anyone out there saying that <Page 20706> is a keyboard?)

What that 1 match means is that there is in fact one keyboard – probably somewhere else on this table – called <Page 20706>. The statement in the When card has 1 match.

If we took that keyboard off the table, then we would see the match count decrease to 0 matches.

Creating a free variable

What if I want to look for any keyboard on the table, not just 20706? Say I have 3 keyboards out on the table right now; I want to see all 3 as query results.

Well, we have this query When ("<Page 20706>") is a ("keyboard") on our card. One way to think about it is that we want to replace ("<Page 20706>") with a free variable like X that can match anything, so our query would be something like When /X/ is a ("keyboard").

To that end, the clause card system comes with various tokens (⭐️, ♥️, 🍀, 🌔) that you can snap onto a clause card:

Let's snap a ♥️ token in to replace that <Page 20706>:

I put that ♥️ token in the first slot on the card – exactly where "<Page 20706>" used to be in the statement.

So now our clause card has gone from When ("<Page 20706>") is a keyboard to When ♥️ is a keyboard, where that heart is able to match any value (it's a free variable, or wildcard, or something like that).

And you can see that, now that we're looking for any keyboard on the table, we now have 3 matches instead of the one match – and we also get to see all the concrete results ("<Page 20706>", "<Page 14021>", "<Page 14013>") that our free variable (red ♥️) matched.

Statements

At any given time, a clause card can store one Realtalk statement clause, like the wish-to-label or is-a-keyboard examples from earlier.

(I guess it's kind of like how a text box can store a piece of text at a time?)

What is a statement? What are some examples of statements?

In the Dynamicland system, statements are just… out there, hundreds or thousands of them. All programs, including most of the 'standard library', communicate via statements, not by making function calls or invoking methods. Here are some other examples of statements that might be floating around in the environment in Realtalk:

  • Claim "<Page 1002>" is (5) inches from the left edge of ("the library table").

  • Claim "<Page 1002>" is at a (47) degree angle.

  • Wish "<Page 3003>" is labelled "Hello, world!".

  • Wish "<Processor realbox3>" plays sound "bell.wav".

  • Claim "<Keyboard A>" is pointing up at "<Page 3030>".

You could suck any of these statements into a clause card and match for them and/or generalize them. That means that the clause card, even just the querying side we've seen of it so far, is a very powerful tool in this environment.

The three slots

I haven't talked much about these 3 slots on the clause cards yet.1

I snapped a ♥️ token to one earlier, and there are other (🍀, 🌔, ⭐️) tokens that I could use. What are these slots? How do they work?

These 3 slots represent the first 3 'parameters' in the statement. For example, if a clause card holds the statement

"<Page 1002>" is (5) inches from the left edge of ("the library table").

then the slots would hold:

  1. "<Page 1002>"

  2. 5

  3. "the library table"

and putting a token (like ♥️ or ⭐️) on one of these slots would turn that parameter into a free variable. (then you could match any page that is 5 inches from the left edge of the library table / or any page 5 inches from the left edge of any table / etc)

There's something really satisfying about the token-snapping interaction. We glued little neodymium magnets under the paper and under the tokens (we had to laser-cut holes in the tokens and make sure the magnets had the right orientation on both sides), so you can literally snap a token to a slot, and it stays in place nicely. (Magnets are so great for creating these sort of interactions, like in the AirPods case; we should use them more.)

(The slots remind me a little of buttons/presets on a synthesizer or something – a finite number of items, unlike a menu on a screen that can be infinite and dynamic, but that can be a good thing, you can have much richer physicality because it is 3 actual slots that you've built out.)

Claim/Wish

We can do a lot more with clause cards than just querying for existing statements: a clause card can also create statements.

We've been looking at the When side of the clause card, but if you flip it over, the Claim/Wish side looks exactly the same, just with "CLAIM/WISH" instead of "WHEN" on that tab that sticks off the top.

When you flip over a card with a statement in it, the card still stores the same statement, but now it will emit that statement instead of querying for it.2

When you flip over a clause card from When to Claim/Wish, it might go from

When /someone/ wishes ("<Page 24259>") is labelled (text) (a query)

to

Wish ("<Page 24259>") is labelled (text) (a Wish)

and the card immediately and continuously emits that Wish statement.

The card's behavior changes from 'is someone putting this label on this page?' to 'put this label on this page'.

Once you can emit statements, you have the power to create all kinds of effects and relationships – you can put labels on things, you can play sounds, you can draw shapes, you can print stuff, (you can pretend to be a keyboard…)

I'll show an example shortly, where we'll draw a circle on every keyboard that's on the table.

Chaining clause cards

You can chain together multiple clause cards to make a program that

  1. queries for statements about the world (When cards), then

  2. emits statements to cause effects in the world (Claim/Wish cards)

Not all programs can be built this way, but this is still a pretty big and interesting class of programs.

Here, we chain the When card from earlier (When ♥️ is a keyboard) with a Wish card (Wish ♥️ draws shape ("circle") with options ({})):

Now this program finds every keyboard on the table, then draws a circle on each of those keyboards (and you should be able to see the circles on the keyboards above).

Here's the video where we make the program work live. Watch for how the circles appear on the keyboards.

The card chain is compiling to roughly this textual program:

When /heart/ is a keyboard:
    Wish (heart) draws shape ("circle") with options ({}).
End

You could have just typed that in, yes, but building this program as a card chain actually feels very different, and I think the card chains encourage a different and more elegant and idiomatic style of 'programming'.

You can interactively see the data you're operating on as you build the program. You put down a When, set up its query, see the match results, then add (join) another When, set up its query in turn, make sure you'll be operating on the right data… you put a separate Claim/Wish down on the side and play around with your desired effect with some concrete input data… then you join them all together into a single chain/program when you're ready.

You can split your program back up into subcomponents, along whatever boundary you want, just by pulling the card chain apart. It's far more fluid and open than cutting and pasting chunks of text.

Although we didn't fully put this system into production or regular use, we were able to build much more complicated programs than this, including an entire microphone/'materialized sound' program along the lines of what I described in "Against recognition". And there were so many interesting accidents and subprograms that emerged in the process, it felt like we got tons of little custom software instruments for free as we did it.

Input

I want to talk directly about input – how do you get things into a clause card? – because there are a lot of subtle and interesting questions/possibilities here.

More specifically: how do you put stuff in a clause card without needing to type it in with a keyboard?

There are two kinds of stuff you might want to put into a clause card, more or less: statements and parameters of statements.

Statements

There are a lot of statements in the world. Even if you narrow your focus to a specific page, there may be tens or hundreds of statements floating around about that page (its location, its size, the graphics being drawn on it, its relationships with other pages, the program it's running, …).

Here's a 'dictionary', "Statements about above", showing all the statements floating around about the dial object above it:

(video)

Some ways we came up with to input statements:

  • using a 'picker' tool to pick a statement right off a listing like the above (there are various other ways to get listings of statements emitted by or about a page) & feed it into a clause card - you can scroll or (literally) paginate that listing if there are too many statements

    • you can filter that listing for only the statements that are changing. If you think about it, this is actually a really deep interaction, because it creates a sort of "programming by example": if you want to create a program that reacts to the movement of a page, the first step is to wiggle the page to find the statement about that page's location. You act out an example of the thing that you want to react to.

  • pick a statement off a line of the source code of another program (if a program says "Hello, world", you can point at the line of code that makes that statement and copy it into your clause card)

  • (I think there were others, but they're slipping my mind at the moment)

Parameters

A parameter of a statement is something like "<Page 1002>", or 5, or "Hello, world", or <bell sound>.

Parameters are what live in the slots on the clause card.

We talked about how you can place a token like ♥️ or ⭐️ or whatever to make a parameter into a free variable, but you might also want to put some concrete data in there. Maybe you want to make a statement that literally says "Hello". How do you type the Hello?

You'd want to input a parameter if you're making a program that displays some custom text, or plays a new sound, or does things at a distance you want to specify – all kinds of reasons you might want a concrete value.

(We also often hacked in programmatic Lua expressions by typing them into parameters, to get around the limitations of clause cards, although this isn't a great practice.)

Some ways we came up with to input parameters (I don't remember how many of these we actually implemented):

  • a 'notepad' that you can point at a slot, that you can handwrite in and it OCRs what you wrote into a string literal

  • point a keyboard directly at a slot and type your expression in (this is kind of awkward, physically, but you get the idea)

  • pick and copy literals or expressions out of other statements in the environment

The feeling of bricolage

Why do I emphasize getting rid of the keyboard so much? even down to not wanting to type individual parameters or statements

I think the goal is to make programming feel a certain way: the 'bricolage computer' where you're just pulling stuff in from all around you and piecing it together with your hands and using various fine specialized tools to adjust it where needed, instead of opening a file and typing a Program like Athena from your head.

The keyboard is too powerful; it flattens everything you do on it and makes it all the same; it breaks the possibility of that feeling of immersion. You can do anything by typing in code on a keyboard. We need to stay away from it if we want to develop new ideas.

Hmm

I wanted to talk a little about why I think this is interesting, beyond the immediate context of 'DL had a dream of a non-textual programming system'.

Computational environment

First, I want to highlight that this programming system only works because it exists inside a computational environment.

like, yes, you could make an iPad app with this behavior and this interface, or you could even make a full projector-camera-fiducial-cards-magnets-tokens stack to reimplement this one application, but it wouldn't be situated in any larger environment; there wouldn't be any causes or effects in the world in the form of statements that you could draw from.

this system works because there's this consistent model of data and communication that's used by lots of interesting other software in the environment.3 (whereas in other computing environments, software is communicating by API endpoints, or function calls, or Unix commands, or method invocations)

(video)

Continuity between database and programming

Clause cards blur the traditional distinction between database (query UI) and programming – a 'program' here is really defined as a composition of database queries, instead of a sequence of instructions.

Statements (which are sort of like database rows) are also a much richer conception than variables or functions or whatever; they're a lot easier to work with and visualize. There's some structure to them, where you can show a table of results, and you can pass queries about them around and abstract parameters of them out. They naturally support multiplicities – you can work on 10 or 1 or 0 objects equally easily.

(remember the Riffle stuff? or using Datalog to make an IDE?)

Interactively building a program

you can see the data (the results of individual subqueries) as you build the program; you see right away if your assumptions about your input are wrong.

you don't even have to explicitly ask for the data (inspect variables, call up a debugger, insert a print statement) – query results (in other words, the input to your program) just appear underneath, ambiently, as you put down a When card and configure it.

Little UIs that compose together

many little UIs (the cards) that you can compose and decompose that each have coherent state, that are useful both individually and together in chains. little search engines

opposite of, like, an app on your phone, like Google Maps, where you can basically only see one thing at a time.

(this also makes it naturally multiplayer – multiple people can be working on their own cards and chains at the same time, then combine them into a larger program whenever and however they want)

(video)

Structured editing

the 'card' provides a usefully different grain from a whole program (or an individual character or line). it's like tweets as opposed to a blog post – each card is like a tweet, an addressable unit that you can link and reply and quote.

it's so much easier to manipulate cards than to manipulate chunks of text, because cards (and tokens on cards) are actually tangible physical objects. i guess that's the point of Dynamicland!

(you can imagine tying some subchain of cards together with string, or coloring them the same color if they belong together, so you can visually chunk your program)

(you can imagine, when you're done, taking a photo of your entire card chain, and then that becomes a 'redistributable binary' that is a single page, and you can always rematerialize it into individual cards later – or point at & edit the card thumbnails on the compiled page – if you want to modify the program)

Anyway

I wonder if there are other computational environments that are structured and rich enough, even inside your computer, that a 'programming system' like this could work:

it's funny – I call it a programming system, but it's not even that much about programming! there's no conditionals, there's no loops, there's no functions. it's just data and effects. but somehow that captures the interesting part of programming to me more than something like Scratch or Swift Playgrounds does, with all kinds of functions and if and for and so on

  1. How do you create a new clause card? How do you copy one? These are kind of subtle questions. It's not the same as copying most programs – should you be able to use the same tool that you use to copy a random program to copy a clause card? Do we need to carve out an exception in its behavior to make that happen? (We also needed to add a new technical capability, to make those slot circles print on the card instead of the usual program source code.)

    (You also need to do some physical assembly after the fact – gluing the printout to particleboard, gluing magnets under the slots. Shouldn't the system walk you through that?) ↩︎

  2. Fun question: how do you implement this persistence of data between the two sides? Each side needs to know somehow that it's connected to the other side so it can retain its data. Both sides of a clause card are dotframe computer programs like any other program in Dynamicland – they need to be generated and printed ~at the same time (you need to allocate two page numbers, then generate the source for both so they point at each other). This trick took a bit to figure out. ↩︎

  3. it's sort of like if you had a computer that was a single giant Excel spreadsheet, where even the internal data structures of the operating system are cells on the spreadsheet. ↩︎