Notes from Dynamicland: Geokit

Hey ___,

I know you've been curious about some of the stuff I've done at Dynamicland. It might be a while until you can make it out here and visit in person, so I'll try to explain one project of mine, Geokit, from top to bottom, to give you some of the flavor.

I'll discuss what Geokit actually is and what you do with it first, and then I'll explain how it's made. That explanation might help you understand how people make things in Dynamicland in general. It's the kind of detail that isn't covered in the zine or on the Dynamicland website.

I sort of hope you read this, come here yourself, and reimplement Geokit better.

What Geokit is

Geokit is a 'kit' or 'library' for building and viewing maps.

One Dynamicland researcher, Luke Iannini, writes about his experiences with Geokit:

The real estate group that owns the building Dynamicland lives in uses their purchasing power to influence the city to improve infrastructure to the underserved areas of Oakland they develop in. The head of this initiative was visiting the space, and she began describing her work while we were gathered around the lunch table, which happened to have Geokit spread across it looking at Oakland. Without breaking the flow of conversation, I dealt the transit card to display the bus routes, and she grabbed the “zoom and pan dial”¹ without any instruction to zoom in on a portion of West Oakland — noticing a huge hole in route coverage she’d never seen before where she knew there were tons of working families. We spent the next 15 minutes exploring as she taught me more about the city's transit details than I ever knew I wanted to know.

In another case at a party I learned that a person I’d just met grew up in the same city as me (Tucson, AZ) when she searched for it to print it out, and we then bonded deeply over looking at the incredibly stark racial divisions in North and South Tucson that we knew from experience but had never seen laid out in raw data.

I’ve never had experiences like these in any other medium. Buildings don’t tend to have detailed racial, transit and elevation maps of every city in the US on the walls. In theory you could pull this data up on your personal iPad, praying there aren’t any embarrassing notifications and hopefully remembering what you were even going to look at by the time you’re past the lock screen looking at 24 rainbow-gradient icons and red bubbles, but I am personally not in the habit of pulling out my devices in the middle of parties. In both of these cases these just-dynamic-enough maps were places at the party, just like the appetizers table and the piano, where people could casually gather, play and converse.

Geokit lives on a shelf when it's not in use, so I'll take it off the shelf and lay it out on a table:

Geokit in action

I'm going to pull up a map of San Francisco and zoom into Dolores Park.

I'll start by typing into the "Geomap above from place search" page, which turns the giant blank page above it into a map.

If you were here in person, I'd hand you the keyboard, and I bet you'd want to type your hometown in. But I'll just use San Francisco.

OK, now I want to zoom in to the park. I have a tool for zooming, the "Map magnifier with IR (infrared) dial" page (the smaller page on the right). When I put that page on the table, it becomes a zoomed-in 'inset map.'

I also have to put this rotating dial on the big map of SF, which controls both the location and zoom level of the inset map. Moving the dial pans the inset map, and rotating the dial zooms it in and out. Being able to directly move the map position is really satisfying.

In this next clip, I put the dial down and then pan over and zoom the inset map to Dolores Park in the Mission.

Now I'm curious about this part of the neighborhood. How diverse is it? How has it changed over time? Are there stark dividing lines? I'll apply some data layers: a color-coded demographic map ("racial dot map"), and then street lines and labels on top so the map is more understandable. I point them at the big map first, then point them at the zoomed map (just to show that both work).

Check out how the west (toward Castro) is bluer on the inset map, while the east (toward Mission) is more orange.² Imagine having multiple people at the table: each person might have their personal view on smaller pages, but the table can all share the big map at the center.

Zoom lenses

To be honest, this whole project started with a very clear image I had in my head. I wanted to make zoom lenses, where you could point a lens at a map to zoom into it, then point a lens at that lens... so I want to show you those lenses now.

You can see me playing with the OpenPTMap (public transport) tilelayer as I mess with zoom, applying it here and there at various zoom levels (it propagates down from wherever I put it). The layer shows some bus lines, as well as the J-Church light rail line.³ I'm trying to get the flexibility of these tools across to you: layers work everywhere and zoom works everywhere, and you can pull a tool out and use it wherever and whenever you want.

Here's my play on Powers of Ten in Geokit: starting from the Earth, I apply a zoom lens, then a lens to that lens, and so on, until I get down to our neighborhood in Oakland.

I can even print a map. I have a printed map of San Francisco here, for instance. I use exactly the same tools on this printout: I look at public transit lines, then demographics, then I zoom and pan around east of Golden Gate Park.

Even if Geokit isn't drawing any tiles on a map and it's just a printout, it's still a fully functioning map.

I've been using San Francisco as an example, but I could have entered any place in the world, and everything would have worked exactly the same.⁴

The one-page 'engine' page makes all these maps and tools work, and that little engine doesn't need to know anything about the stuff built above it. Basically, the engine takes a page representing some geographic coordinates, and it draws some map tiles from the Internet on top. I'll show how you might build that engine from scratch here in Dynamicland, step by step.

All the parts

These are all the 'game pieces' inside the kit. Each page is both a little computer program and an object that people and other pages can see and work with. Some pages are face-down, but most of them are face-up, which means they're running right now. (Face-up is the side with the colored dots.)

You often point one page at another to make it do something, like how the zoom lens points at the printed map, or how the keyboard points at the text box which points at the big map.

We have a few ways to get a map of a place:

• ⌨️type a place name into the text box to search for it
• use a premade map like my 📄 printed map of SF
• apply the 🔍 zoom lens to an existing map: the lens becomes a zoomed-in map of the green region it points at
• use the 🎛 zoom & pan dial ('loupe') page on an existing map: it zooms into wherever the dial is, and you rotate the dial to change the zoom level

These parts are the ones in the kit right now, but anyone here can make new parts: you could come here and write up a new kind of tilelayer or a new way to make maps.

That's why I call Geokit a kit and not an application: it's naturally extensible. You could come here, pull a keyboard off the shelf, point it at any part of Geokit, and just start editing it and printing new stuff for your own use. You might provide a new tileset to answer some question you have, or come up with a new zoom mechanism to derive maps. (I'll explain how to do both of those things in this writeup!) You have the same status that I do, and you don't have to open your laptop and download and recompile some source code if you want to change something.

Background on Realtalk

Dynamicland is a place – a land – in Oakland, California.

Realtalk is the programming system that runs inside Dynamicland. It includes a superset of the Lua programming language. You do 'pure' computation in Lua, but you use special Realtalk extensions to do all communication with other pages and with the outside world.

The Realtalk 'operating system' already knows about some wishes as part of its standard library, so we can say things like Wish (you) is labelled "Bella". and Wish (you) is highlighted "green". and they work right away. But we can also make up our own claims and wishes:

-- Made-up claim demo
Claim (you) blahblahblah.

-- Made-up claim when demo
When /page/ blahblahblah:
    Wish (page) is highlighted "blue".
End

Why is "Made-up claim demo" highlighted in blue? What will happen if I flip "Made-up claim when demo" face-down?

Realtalk and the rest of Dynamicland

I'm mainly going to discuss Realtalk here, so I'm writing in a form that will look almost like any other tutorial about how to do a programming project.

I'm not going to talk about the social experience of Dynamicland, the community, or things particular to the physical lab, but those things are integral to the Dynamicland concept! Knowledge transfer in Dynamicland usually happens those ways, not through writing like this. I just think others can do a better job of describing those things, and I want to fill the gap of documenting 'programming.' Check out @Dynamicland1 on Twitter for a broader view.

Making Geokit

I'll start by creating a map page representing San Francisco, a map page which won't actually do anything at first.

Then I'll build up the Geokit 'engine', which is about a page of code, in four steps. (The engine will make the map page work.)

1. A starter three-line Geokit 'engine page' that just paints that map blue.

2. A better engine which downloads a map tile from OpenStreetMap and displays it.

3. Support for multiple tile layers, from different sources, on one map.

4. Stitching more than one tile together to get exactly the right map view to fit the map page's area.

A map page that doesn't do anything (yet)

First, I'll make a San Francisco map page for Geokit to act on.

I want to make a claim that my map page represents the geographical area of San Francisco, so that Geokit knows it's supposed to draw San Francisco on it. I'm going to type in a claim whose syntax I made up: Claim (you) is geomap of bbox (... coordinates ...).

At Dynamicland, you make a new page by

1. pointing a keyboard at an existing page someone else wrote (ideally with code you want to reuse),
2. then patching its code in place to do what you want,
3. and then printing it out.

I'll do that to make my map page, although I'm not keeping any of the original code in this case. I also add a bunch of linebreaks so the printout covers the whole page.

Is this a map? Well, it's a page – a program, basically – which is making a claim about itself. (See Tabitha Yong's cheat sheet, shown earlier, for more about claims.)

-- San Francisco map
Claim (you) is geomap of bbox ({left=-122.527, bottom=37.664, right=-122.348, top=37.851}).

You can think of it as setting a dynamic 'property' that other pages can see: "I claim I'm a map representing the box with corners (122.527° W, 37.664° N) and (122.348° W, 37.851° N)." (Those longitude-latitude pairs are, very roughly, the bounds of San Francisco.)

You can make whatever claims you want, but there's no guarantee that anything will happen as a result. Right now, nobody cares about my geomap claim, so the map is inert. But soon, the Geokit 'engine' will be querying for all the pages that make that kind of claim, and it will give them map behavior.

1. Starting on the 'engine' page

Next, I'll start building the Geokit 'engine', which is a separate page: it'll look around for maps like the one I just made, then draw on them.

My first attempt is a minimal test to show how I can recognize and act on maps: it colors all maps solid blue.

-- Geokit 1st try: blue
-- On each geomap, draws blue.

When /map/ is geomap of box /bbox/:

    Wish (map) is highlighted "blue".

End

I'm using a wish here, which is a sort of broadcast request for something in the world to be different: in this case, whenever I see a map, I wish for it to be highlighted in blue.

Some other page happens to notice that wish and fulfill it using a projector, but I don't know or care about that mechanism. I'm not addressing the wish to anyone in particular. When I put this page together with the map, sure enough, the map turns blue.

To show you that the engine responds to every map, not just one, I made a map of New York as well. See for yourself: it turns blue the same way.

2. Drawing one map tile

OK. Now I want to draw actual map tiles from OpenStreetMap instead of that placeholder blue. I'll start with something small: draw one tile of SF at a fixed level of detail, big enough that it fits the page.

I'll show you (an abbreviated version of) how I edited our previous "paint maps blue" 1st try and ended up with this 2nd try, which draws an actual map tile. I'll show you the code next, so don't worry about squinting to read all the code in the video. (The projection is much clearer in person!)

I start by changing "blue" to "orange" and then "red" to show you how I can edit the page in place. Then I extend the code so that it downloads and draws a hard-coded tile at (11, 327, 791), which is San Francisco. Finally, I make the code more general. I copy and paste the utility function deg2num from another page, which I use to compute a tile based on the bbox encoded in the map page, along with a drawing size for that tile based on the page's physical height and width. Once it works, I print the page out, so I have a permanent, physical version.

This code is some of the most 'traditional' and complicated in the whole kit. Honestly, it's a little unidiomatic to have so much 'normal' code bunched together. It doesn't help that it has to interface with the outside world (the Internet and filesystem).

You might start by looking at the code block I highlighted, the When /map/ is geomap... block at the bottom, which runs for each map on the table:

local URL_TEMPLATE = "https://tile.openstreetmap.org/%d/%d/%d.png"
local ZOOM = 11
When /map/ is geomap of bbox /bbox/,
     /map/ has width /width/, /map/ has height /height/:

    local xtile, ytile = deg2num((bbox.left + bbox.right)/2,
                                 (bbox.top + bbox.bottom)/2,
                                 ZOOM)
    local url, path = get_tile_url_and_path(URL_TEMPLATE, ZOOM,
                                            math.floor(xtile),
                                            math.floor(ytile))
    local tilesize = math.min(width, height)
    When tile URL (url) is downloaded to path (path):
        local ill = new_illumination()
        ill:image { filename=path, scale=1,
                    x=0, y=0,
                    width=tilesize, height=tilesize }
        Wish (map) has illumination (ill).
    End

End

1. Have the When bind variables:

   • map: reference to the map page
   • bbox: the left, right, top, bottom coordinates of this geomap
   • width: the map page's usable width in inches (7.5)
   • height: the map page's usable height in inches (10)

2. Compute the geographic center of bbox and use deg2num to find the corresponding OSM tile coordinates xtile and ytile (327 and 791, respectively, in this case)

3. Construct the URL https://tile.openstreetmap.org/11/327/791.png and the download path shared/geokit-tiles/https___tile_openstreetmap_org_11_327_791_png.png

4. Compute the tilesize at which to render the tile image, which is the minimum of the page width and height. In this case, it's about 7.5 inches.

5. Once the tile is downloaded⁵, build an illumination (the built-in graphics library) that draws the tile image. Then Wish to draw that illumination on the map page.

I patched in some log statements to show you the live values of bbox, width = 7.491, height = 10.001, xtile = 327, and ytile = 791:

3. Modular tile layers

I can make a ten-line change to the engine and get a huge amount more flexibility. Instead of hard-coding that URL_TEMPLATE = "https://tile.openstreetmap.org/%d/%d/%d.png" in Geokit, I'll let pages make Wishes to draw any tilesets from any URLs they want.

To let the user specify what layers to draw, I've made individual tilelayer pages. Each page represents a layer: when you point a layer page at a map page, the layer wishes to draw that layer on the map.

Here's the source code for the "Demographic dots tilelayer", for example:

When (you) points "up" at /map/:
    Wish (map) shows tilelayer
        "http://demographics.virginia.edu/DotMap/tiles4/%d/%d/%d.png"
        with options ({max_zoom=13}).
End

I edited the engine so that it responds to these Wishes, rather than looking for any geomap and drawing that hard-coded URL_TEMPLATE from last time.

Here are all four tilelayers in action on the San Francisco map. Notice how I chain layers to put them on top of each other!

And here is the new engine page:

This 3rd-try Geokit is almost unchanged from the 2nd try, except for this section, from line 34 down to the extra clause in the key When block that handles maps:

When /someone/ wishes /page/ shows tilelayer /url_template/ with options /opts/,
     /page/ wishes /target/ shows tilelayer /something/ with options /something/,
     page ~= target:
    opts = rt.table(opts) -- Make a mutable copy of opts
    opts.priority = 1 + (opts.priority or 1)
    Wish (target) shows tilelayer (url_template) with options (opts).
End

local ZOOM = 11
When /someone/ wishes /map/ shows tilelayer /url_template/ with options /opts/,
     /map/ is geomap of bbox /bbox/,
     /map/ has width /width/, /map/ has height /height/:

Start from the bottom: the added When /someone/ wishes /map/ shows tilelayer /url_template/ with options /opts/ clause makes my map-drawing code run separately, for every tilelayer wish for each map, rather than running only once for each map.

It binds url_template to whatever the tilelayer page said, whereas I had a hard-coded URL_TEMPLATE constant last time. The tilelayer page code makes a wish like Wish (map) shows tilelayer "https://tile.openstreetmap.org/%d/%d/%d.png" with options ({}).", which results in url_template = "https://tile.openstreetmap.org/%d/%d/%d.png" and opts = {}.

The first When block (When /someone/ wishes /page/ shows tilelayer...), starting on line 34, is really interesting. It's what lets me put layers on top of each other by chaining them. When a tilelayer page (someone) is pointed at another tilelayer page (page), and that page is itself pointed at target, then someone's tilelayer wish is 'forwarded' to target, but with its priority boosted by 1 (higher priority means drawn on top, so someone's tilelayer gets drawn above page's tilelayer). That means that a tilelayer pointed directly at the map is drawn at the bottom, a tilelayer pointed at that tilelayer is in front of it, and so on. The outermost tilelayer is drawn on top.

A note on the Realtalk model

Suppose you were implementing this tilelayer manipulation in traditional software. You'd probably make a list of tilelayers, with (+) and (-) buttons. You might put a check box next to each tilelayer. You'd have to implement drag and drop, and you'd have to handle events every time the user reordered the layers.⁶

This step starts to illustrate the power of the Realtalk model, in a way that I've tweeted about before. Look at how I've handled layers. Each layer is a page, and you put the layer down and point it to enable it and set its position. Nobody had to implement a checkbox or handle a drag event or synchronize an array with a UI view.

When I first showed up at Dynamicland, I did a lot of what I think of as 'traditional programming.' That means using a page essentially as a computer monitor: you hack out a lot of Lua code, print it, and it renders something cool on that page. Maybe it takes input based on different regions of the page. I've seen programmers do this over and over on their first exposure to the system, and I fell into it too: I would say it took me about four months to understand how to really take advantage of Realtalk.

One way to think about Realtalk is that a lot of code in the outside world is about displaying lists and letting people pick from lists.

• Mapping software has to show a list of layers, and make them checkable and uncheckable, and make them draggable, and handle all the events, and remember what the state is.

• A photo app shows you a list of photos, and then you open one, and then you see a toolbox of operations you can apply, and then you pick one...

• Even command-line programs let you ask for a list of source branches or a list of disk partitions or a list of files or whatever, and have some set of subcommands you can apply to your object of choice.

In Realtalk, most of that code falls away – provided that you represent your objects as individual physical pages. Once you have physical pages, you automatically get the operations of the physical world: placing and picking up objects, moving and grouping objects in space, pointing objects at each other, and so on.

We have a rule of thumb that no program should be longer than a page (73 lines!) If you have a longer program, you're probably doing something wrong, and you should split it up into little pages that work together.

And there's a lighter rule of thumb floating around, that most code should be 'purple text' – Whens and Claims and Wishes, which the editor highlights in purple – not gobs of plain Lua code.

These rules suggest that there's a distinct Realtalk idiom, which is very different from just writing Lua for projectors and cameras. It's something like objects or actors (where the pages are objects), and reactive programming, and reusing functionality from the real world wherever possible.

I don't claim that Geokit is an ideal example: the engine page is pretty complex, and even my use of the projector to render maps is questionable, since any static map could have been printed instead. But the layer pages show a little bit of this power.

4. Fitting to exact coordinates

Finally, I want to get rid of that one-tile hack. If a map page is supposed to represent the box with corners (122.527° W, 37.664° N) and (122.348° W, 37.851° N), I want it to display a map of that exact area, not just some nearby tile image that OpenStreetMap serves up.

We need to figure out the right level of detail, get multiple tiles at that level, and then stitch the map together.

I've made this engine development into a GitHub gist as well, so you can look at the revisions to see how I got from step 1 to here.

I've only changed the bottom When block from the previous engine:

When /someone/ wishes /map/ shows tilelayer /url_template/ with options /opts/,
     /map/ is geomap of bbox /bbox/,
     /map/ has width /width/, /map/ has height /height/:

    -- Made-up formula to guess a good zoom from width of bbox.
    local zoom = math.log(360 / (bbox.right - bbox.left)) / math.log(2)
    zoom = math.ceil(zoom + width/16 - 1) -- experimentally determined width factor
    local max_zoom = opts.max_zoom or 17
    if zoom > max_zoom then zoom = max_zoom end
  
    local xtile_left, ytile_top = deg2num(bbox.left, bbox.top, zoom)
    local xtile_right, ytile_bottom = deg2num(bbox.right, bbox.bottom, zoom)

    local tilesize = width / (xtile_right - xtile_left)

    for xtile=math.floor(xtile_left),math.floor(xtile_right) do
        for ytile=math.floor(ytile_top),math.floor(ytile_bottom) do
            local url, path = get_tile_url_and_path(url_template, zoom, xtile, ytile)
            When tile URL (url) is downloaded to path (path):
                local ill = new_illumination()
                ill:image { filename=path, scale=1,
                            x=tilesize * (xtile - xtile_left),
                            y=tilesize * (ytile - ytile_top),
                            width=tilesize, height=tilesize }
                Wish (map) has illumination (ill) with priority (opts.priority or 1).
            End
        end
    end
End

1. Determine the zoom level zoom – higher zoom means more tiles are used to represent the same area. I should point out that there's no single correct zoom level to draw a particular map: it depends on how much detail you want to show. You probably want higher zoom the higher your display resolution is.

   Here, I take the logarithm of (360 / the distance between the left longitude and right longitude of the map), so a map covering a smaller geographical area gets higher zoom. Then I increase the zoom by a bit, based on the physical size of the map page (width / 16). For this San Francisco map, I end up with zoom = 11.

2. Compute the tilesize in inches at which to render each tile, such that the tiles exactly fit the page.⁷ In this case, we get something like tilesize = 7.44 inches.

3. Loop through all the tiles and download and render each one, just as we did when we drew a single tile previously.

With perfect fit working, maps of different sizes also work great. Here's the New York map I made earlier.

How tools work

I showed a lot of tools, like zooming and searching for places by name, at the beginning; how do they work?

You can implement all of them without changing the engine at all. I won't go into as much detail about them – maybe some other time – but they all fundamentally come down to the claim that a page is geomap of bbox (...something...). I've used constant coordinates for the bbox so far, but they can also be dynamic values, which is how we get dynamic maps.

Zoom lens

The zoom lens is surprisingly simple to implement, and it was the first tool I made. Wanting to make a zoom lens inspired the whole project, after all.

The lens draws the green rectangle on the map it's zooming into, then computes a bbox so it can make its own claim to be a geomap.

It inherits tilelayers from its parent, but scales their priorities down so the user can easily put layers on top. It also inherits highlighted points, which are an addon I'll discuss next.

Addons

I need some addons to support the rest of the tools.

Wish (...) is geomap containing bbox (...)

This addon handles Wish (...) is geomap containing bbox (...). It lets a page become a map of an area that doesn't exactly match its aspect ratio. I'll need this functionality soon, when I want to map an arbitrary place that the user types in.

For example, if you have a page with a 2:1 aspect ratio, but you want to make page a map of square San Francisco, you can do Wish (page) is geomap containing bbox (...SF coordinates...), and this addon will give page a claim for a wider bbox which still contains San Francisco.

If you already knew a bbox that matched the page's 2:1 aspect ratio, then you wouldn't need this wish addon: you could simply make the original Claim (page) is geomap of bbox (...). that we implemented in the engine.

Wish (...) highlights geopoint (...) with label (...)

This addon handles Wish (...) highlights geopoint (...) with label (...). You can label specific places on a map with this Wish.

Place search

Place search goes to Google Maps with whatever you typed, gets a suggested viewport back, and then makes a Wish (to the geomap-wish addon from earlier) to construct a map containing that viewport. It also sets the OSM tilelayer as a default.

Zoom and pan dial

This dial pan-and-zoom mechanic was suggested and implemented by some visiting designers from a technology company you've heard of. We showed off a few things in the morning, including a generic dial and my Geokit zoom lenses, and one designer (who "didn't program") immediately thought about how to combine the two.

How could we control the zoomed-in map directly and in context, instead of having the lens (which sits at a distance from the original map)? With that interaction idea and a couple of hours of work in the afternoon, we had a working prototype for this dial. And we were working at a table together, with pieces of paper, a marker for scribbling, a dial, and a keyboard, not with laptops.

I love the direct manipulation; just as hoped, moving the dial on the map to pan feels much nicer than moving the zoom lens out on the side.

The green number on the dial (from 0.0 to 1.0) is the zoom level, where 1.0 is the most zoomed-in and 0.0 is the least.

There's also a green rectangle, which represents the exact area that the inset map is showing. When I increase the zoom level, the green rectangle shrinks and the inset map zooms in.

The original dial was just a (small) normal page, which you'd put on top of the map page. But the light from the projector interfered with dot detection, especially with the racial dot tilelayer's bright colors in place, so it was unreliable. To avoid that problem, I'm using an IR (infrared) dot detector (/map/ contains tracking dots /dots/) which tracks the gray-looking retroreflective dots. (Ideally, the dial would be a normal page, though.)

One of the best things about Dynamicland is making peripherals and using them with the computer right away. We don't need to implement simulated inertia or make a virtual drag handle, since we simply track the physical dial.

The dot furthest from the average of the dots is the 'head' of the dial, which I use to figure out the dial's angle.

It'd also be interesting to draw the zoomed map on the dial itself; we've played with this a few times, but there are still some issues (interference, performance, rotating view properly).

Printing maps

Printing is very much a work in progress. You point the "Print static map" page at a map page, and it prints a static map of that place, with a Google Maps view on it.

There's a lot of fiddly calibration to do here, and it's not perfect right now. It goes to Google's Maps API and gets an image around the center of the map it's pointed at, guessing the right radii around that image to line up with the existing map. It injects a claim, so the printed map will be a valid Geokit map too.

It's been surprisingly useful as a debugging tool, since I can turn a misbehaving dynamic map, which is hard to keep stable, into a static, printed test case.

Conclusion

These pages are all there is! Everything I showed you at the beginning of this writeup was implemented by the actual pieces of paper in front of me, and now you've seen how all that code works.

Yeah, I'm relying on some 'built-in' functionality, but all that behavior is also implemented on pages here in Dynamicland. When you're here, you can walk over to a board and read through those system libraries. There is no underlying stack hidden on a computer hard drive or on GitHub. On this whiteboard, you can find the page that renders to the projector when I Wish (map) has illumination..., the page that makes a GET request when I query When URL (...) has contents /contents/:, and the page that parses those Claims and Wishes and Whens in the first place:

In Dynamicland, software is supposed to be this simple or simpler. Software in Dynamicland is meant to be continually read and tweaked and discarded by users; that's why each page has its source code printed on it.

Why maps?

Why maps? Personally, I've been interested in maps and cities and public transportation and infrastructure for a few years, and I haven't gotten to do much original work around that interest, so Dynamicland seemed like a good place to explore it. I was excited to build an idiomatic-feeling 'kit' with parts, instead of a one-off program like my earliest work here.

I think maps highlight the advantages of breaking out of the screen. We insult an entire tradition of large, information-rich maps when we cram them onto tiny phone screens, then crowd around them and make finger gestures to squeeze answers out bit by bit.

I figured that in Dynamicland, we could have maps that actually looked like maps. We could have multiple people sitting around a map at a table, sharing a large view, but also applying computational filters and derivations to get at their individual interests. You could explore many places and ideas at once, and keep track of them on the table in front of you, instead of constantly sacrificing context on a limited screen.

Dynamicland is great at spatial interfaces, and maps are spatial. You get tools like dot tracking, page tracking, whiskers, and rotation, as well as whatever else you can tie in from the physical world.

And maps involve real information from the world, which makes them more relatable than a visualization of some mathematical model. I've seen plenty of people whip out lenses and turn zoom knobs and poke at layers in an attempt to learn about the place where they grew up. Maps are an early opportunity to use this technology to ask real questions about the world, the same way that I use my phone or laptop to ask questions.

On documentation

I think a lot of the material and discussion about Dynamicland has been either too vague or too focused on the wrong details. It's been hard to talk about the place to people who haven't visited. You lapse into phrases that sound like platitudes.

I'm taking a different approach here. I want to be concrete about how you make things and about the details of the programming experience. I'm hoping that this detail will actually do a better job of communicating the bigger ideas.

I think there are a few reasons why there hasn't been much written about Realtalk, especially writing beyond 'first impressions':

• Written essays feel like an awkward medium for talking about Dynamicland. Because it's a physical space, its inhabitants are used to communicating in person, with demonstrations at hand. You wouldn't ordinarily write or read a document like this one; if you want to know how something works, you put it out on the table and read the code on the pages, and maybe ask someone in the space (since it is a shared space, people will be there!)

• Building external attention hasn't been a big goal: the focus has been on in-person outreach, fundraising, and development within the space.

• Even among people who've visited Dynamicland, not many actually know Realtalk well, so not many can write this kind of overview. As with any creative tool, becoming fluent takes time. It took me at least a few months.

• There may also be some embarrassment about having a traditional text-editor-oriented environment, given the earlier work of Bret and other researchers in the lab.

• There was a fear that people would fixate on details and copy them blindly while missing the values, as happened to the Xerox Alto GUI.

But focusing on these programming details still seems better than focusing on things like the dots and projectors and computer vision, which are obvious to anyone who visits and so get a ton of discussion. I see that stuff – dots, the nature of the page database, cameras and projectors, occlusion, comparisons to augmented reality – purely as implementation details, and I don't spend much time thinking about it; I haven't discussed it in this writeup at all!

The pages/wishes/claims/whens programming model is the true 'spirit of Realtalk', even if ideally we'd change programming practice in even more radical ways: get rid of the keyboards as well, or add more direct manipulation, or introduce better visibility for debugging... or abolish programming entirely?

In fact, I'd argue that the programming model is very under-discussed. It is more important than you may think from what you see online. A lot of what's new about Dynamicland is that it's a programming system! Compare projection-mapping demos you see from the 1990s and 2000s, from various academic and industrial labs. They may have similar interface techniques, and they may demo cleaner, but the implementation is some big blob of C++ that is written backstage. You can't walk in and do your own program. The pieces of paper in Dynamicland are not just display surfaces for a demo someone wrote on their PC: they're used to program the system directly.

You could implement Geokit in JavaScript or C++, and build your own projection mapping and computer vision rig, and you could implement any Dynamicland program that way, but it would be a lot of work. I certainly would never have done it. Dynamicland lets you take all that as given.

There's also lots of Dynamicland coverage that is just out of date! Using the system today (August, 2018) is a very different experience compared to using the system a year or even six months ago: there are thousands of example programs, computing surfaces are everywhere in the space, projectors are high-resolution enough to display detailed data and text, everything runs way faster, you edit with keyboards within the system instead of laptops – the fundamental design is intact, but the subjective experience has changed a lot, especially when you're making things.

I hope this writeup has given you a better idea of how Dynamicland works and what's powerful and new about it. I've talked about some of these ideas @rsnous on Twitter, and I'd like to write and release more in public – let me know what you want to hear about.

best,
Omar

My thanks to John Backus, Jason Brennan, Nicky Case, Mary Rose Cook, Daniel Feichtinger, Katie Gu, Jacob Haip, Joshua Horowitz, Luke Iannini, Lauren Killingsworth, Tristan Navarro, Lisa Neigut, Paul Sonnentag, Bret Victor, Roshan Vid, Elizabeth Yang, and Andy Zhao for their thoughtful comments and feedback on this writeup.

---

1. as I'll discuss later, made by a visiting designer who "didn't program" ↩︎

2. back during the 2010 U.S. Census, at least ↩︎

3. The blank white areas you see on some of the lenses are a bug in Geokit: they're areas where no public transport runs, so OpenPTMap doesn't provide any tile images (good vs. bad). If Geokit gets a 404 instead of a tile image, it renders the 404 as though it's a valid PNG – which appears as an opaque white tile, blocking out the tilelayers underneath. Those tilelayers underneath are supposed to still show, so that's a bug. ↩︎

4. except the U.S. Racial Dot Map, outside the U.S. ↩︎

5. Maybe you're wondering why we need to save the tile to a path on disk. Why not render the PNG data in the contents variable on line 20 directly? That's probably the right way; it's a quirk of the current illumination API that you have to provide the rendering engine an image filename instead of data. ↩︎

6. The operation of the word user is interesting when we talk about traditional software. For example, in a traditional application, if the programmer doesn't explicitly provide a dialog to add a new kind of tilelayer, the user can never add their own tilelayer. Even if the application is open-source, how many people are able or willing to go in and make that change? There's something fundamentally authoritarian going on: the programmer makes the application, and the user uses it. ↩︎

7. I'm assuming that the bounding box declared by a map page matches the map page's aspect ratio. That is, if you say you represent a 16:9 geographical area, you should be a 16:9 page. You could imagine a design that automatically refits the drawn region to the page size, but the Geokit core should be as simple as possible: that function can be built on top later. I have an add-on, "Wish is geomap containing", which does the automatic refitting for applications that need it. ↩︎