Visualizing Complexity

This post could be considered the third of three responding to Bret Victor’s trio of talks; the previous ones were Abstraction and Standardization and Media for Thinking the Eminently Thinkable.

Question: what makes a program complex? Answer: time.

There are multiple types of time. The obvious one, the one measured by stopwatches, I’ll call physical time. A small program promotes a tight feedback loop, where the effect of a change is visible in a few seconds. Large programs take longer to compile and run, as do those dealing with large amounts of data. To help offset this, programming languages developed threading and concurrency. An expensive task can be queued up, so it will happen at some point in the future. This sort of parallelism makes programs much harder to reason about and test.

Then there’s logical time. This includes event-based programming, which usually involves a GUI. A user’s clicks and drags become events to which the system responds. Code is called when something happens, not when the thing before it ends. Rather than a procedural recipe racing towards the final answer, these programs are state machines, looping indefinitely and branching in response to an arbitrary sequence of events.

Finally, for completeness, there’s developer time. Memories fade, requirements change, people join or leave the project, the technology itself changes. Like people, code deteriorates as it ages, although measures can be taken to mitigate the decline. Any large codebase has annoying “legacy code” kept around for “historical reasons” and “backwards compatibility”.

In his talk Drawing Dynamic Visualizations, Bret Victor presents a drawing tool where the user creates a program by direct manipulation. This program can be used to create custom pictures for multiple datasets (different measurements of the same variables). The results feel very “PowerPoint-y”, the sort of thing you’d present in a board room or paper scientific journal. The method of creation is new, but the results emulate old media.

If you’re reading this, Bret, and think I’m a bit obsessed, (1) thanks for noticing (2) this will likely be the last post about you for awhile.

Users can step through the drawing program, but it’s capable of running instantaneously (i.e. without time and therefore much complexity) . There’s no need for a visualization to wait for a network event, but a good visualization reacts to mouse motions, like hover, click and drag. Interactivity is a crucial component of visualization. Victor has made a great way to make charts for boardroom presentations, but he hasn’t allowed the user to interact with the data. The visualization is “dynamic” in that it accommodates new data, but it doesn’t react to the user. I’m not asking to drag a bar to set the data to that point; we’re working under the assumption that the underlying data is fixed. And I can abstract any color or magic number as a parameter, so I can set data-independent features of the graph like error bars dynamically. But without interactivity, we can only accomplish the first third of Shneiderman’s mantra:

Overview first, zoom and filter, then details-on-demand.

Without interactivity, we can’t zoom or filter or otherwise adjust the visualization to get our details on demand. All we have is an overview. Our visualization sits there, dead. This is not good enough. Data today is so complex that even the best static image isn’t going to elucidate subtle meanings. Instead we need interactivity so that the visualization reacts to the user, not just the designer. I should clarify straight away that I’m not talking about the interactivity Victor rails against in his paper Magic Ink, where the user has a specific, often mundane question she wants answered. (How long will I wait for the bus? What move theater should I go to?) I’m talking about systems where the user doesn’t know what she wants, and needs to explore it to develop an intuition or notice something strange.

There is a disconnect, an asymmetry, in Victor’s work. In Stop Drawing Dead Fish, we had a novel direct manipulation tool and interactivity with its results. In Drawing Dynamic Visualizations, we had a novel direct manipulation tool and dynamic data. In Media For Thinking The Unthinkable, specifically the part on signal processing, we had interactivity and dynamic data. Create with direct manipulation, load dynamic data, interact with the result: pick two.

How can we add interactivity to Victor’s tool? Let’s start with mouseover. As a function of the mouse position, which can be considered an input, hover is stateless. The tool can run geometric collision detection on its primitives and provide a boolean measurement (output) as to whether the shape is being hovered over. If you want to use this information to, say, change the shape’s color, you have to have shapes able to read their own outputs. This can lead to feedback loops. If we draw nothing on mouseover, then we’re not being moused over anymore, so we go back to drawing our shape, which is now being moused over…and suddenly our system has state, and “flickers” indefinitely. Worse, by creating a place for code that is executed only if certain external conditions are true, we create asynchronous, “jumpy” code. This is a large increase in physical and logical time.

Selecting a shape runs into similar problems, an additionally requires a global variable (internal, not a parameter) to keep track of the selection. It gets even worse if we want to add collision detection between objects, like a force-directed diagram. (Actually that’s not collisions, just proximity-based repulsion, but roughly the same idea.) Even if the system manages to detect when two subpictures touch, they now will need a place for code that is called when that is the case. Victor briefly demonstrated how physics can be simulated by using a guide vector to geometrically represent velocity, which we could presumably shift in reaction to the angle of line contact. But we’re starting to think of shapes as their own entities, having state represented in guides associated with them, rather than being drawn and forgotten about. This brings us into Object Oriented Programming, the traditional choice for this sort of interactivity. It’s a great paradigm but it’s quite foreign to Victor’s tool. (Although his Stop Drawing Dead Fish tool lets its user do exactly this. Maybe the line between data scientists and visual artists, or perhaps their tools, is blurring…)

There’s a brief moment where Victor abstracts a color from a shape, and makes it a parameter. This parameter starts off as numeric but changes to a color, likely in response to a keyboard command. If the tool already has a notion of types, that a number is not a color, then there’s a piece of complexity already imported from traditional programming. Types, OOP, collision detection, encapsulated mutable state – all of these ideas have their place in code. There are also a number of mainstays of drawing tools we could include, like gradients, z-order, manual kerning, and the dropper tool. Deciding which of these to include and which to omit, and how to integrate them into the tool, will require some careful design decisions. I wonder how many of these ideas we can pile into the tool before either the graphical interface or the layperson’s acuity collapses under the weight of the complexity.

I, like Victor, and frustrated at the state of software and computing. Most artists are able to react with their medium, to try something, revise it, and use mistakes as inspiration. Listen to how creators in fields as diverse as painting and carpentry discuss a problem. It’s always about sculpting the unruly workpiece material with capable and precise tools, man vs. nature. So why then are computers, where the tool and the medium are one and the same, so limiting that our society can’t see them as anything other than frustratingly complex? Instead of using the medium to find out what she wants to create, the creator has trouble merely realizing what’s in her head. We need a fundamental overhaul of the tools we use for thinking. These tools should let humans, data, and simulation work together. Victor produces tantalizing glimpses of such tools with ease.

Maybe there are true geniuses out there. People who effortlessly sift through complexity, and can make bold pronouncements with confidence. The rest of us have to fake it. We get mired in the cacophany of voices and ideas, paths to choose and tools to use. We do our best to make sense of them but always wonder, in a grass-is-greener sort of way, if we’re missing the point entirely.


2 responses to this post.

  1. I noticed that issue with Victor’s drawing tool as well, Max. It focused on interactivity for the developer, but not as much interactivity for the end-user. I wonder how easily you could extend the drawing tool to make the things that are created fully fledged manipulable objects.

    Victor’s thinking around being able to see the effect of code is definitely important, but I also wonder if there are other ways to interact with code than by manipulating text (or blocks). I mean, when I program a for-next or do-while loop in a language, I visualize the instructions looping in my head. Could one create a coding environment that was a bit like tying knots in a string? The language construct would focus on the paths data can travel in the language, rather than the symbols we use to denote those paths. You could of course, expand, and switch between representations, but I think a visual representation like this would make a lot of sense.


    • I briefly hinted above that carefully managed objects could encapsulate state, similar to the Stop Drawing Dead Fish talk. You don’t want OOP running amok of the existing style though. Even if we consider interactivity harmful (see: Magic Ink), the same techniques can be applied to incorporate the time and location, to provide the drawing with some autonomy.

      It’s true that in C-style syntax, a conditional and a loop don’t look very different: keyword (conditional) { commands; } You have to read the keyword to distinguish between if and while. But Victor is subscribing to Tufte’s maxim “Show the data”. The control flow may be obfuscated, but the data is completely invisible. Often, the hard part is not knowing to use an if statement but what the condition should be. Did I get it backwards? Off by one? Wrong when they’re equal? And so on. It’s great to provide multiple representations: in code, and in pictures. But I think your idea ( is valuable if for no other reason than to be completely separate from Victor. Here I am piling half-baked addendums and rejections on to his ideologically clear work (hence the last paragraph).

      Thanks for commenting – given the amount of quality work Victor has produced and put out for free, I’m really surprised the internet doesn’t appear to take much notice. (He doesn’t have a Wikipedia article, for starters.) Am I missing the fan club, or am I its president because no one else cares?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: