vis.design

Organized Complexity

August 2009

The project consisted of a custom multi-touch screen, developed using a technique called Frustrated Total Internal Refraction (FTIR), and a generative graphic software where an abstract pattern continually evolve according to the competition rules of the system and the influence of the direct manipulation of the users.

Organized complexity was developed as the final graduation project for my architecture degree. It was also showcased in multiple exhibitions, including the File Festival and the International Architecture Biennial of São Paulo, it was also ported to the iPhone and iPad and had a version available for download on the App Store.

Objective

The objective of the project was to apply the concepts of complexity and emergence to the creation of graphic images. Instead of directly designing one image, I wanted to create a system that generated images, a system with several interrelated parts that exchanged information in complex ways, and that from those relations allowed new images to emerge. Images that were not a direct representation of the intent of an author, but that resulted from the complexity of a system organizing itself as its internal pieces exchanged information and communicated with one another.

The project been presented at the FILE International Festival in 2009.

Information exchange

The generative visual system is composed of multiple small unities that can see the other unities around them and use the information about those to decide how to change their own visual appearance. Each unity has the shape of a circle, and can control its visual properties, they can change their size, color and opacity. Those changes are not random, they happen as the unities follow a set of rules.

Each unity can see what're the properties of their close neighbors, they know the color and size that the unities around them have chosen to display. The unities can change their color and size whenever they please, but the visuals of their neighbors directly affects how a they choose their own visual properties. If several of the unities around a circle are displaying a nice new red hue, the circle may want to follow them and also change its color to red. But if every unit around it is wearing the same color as him, he may want to change his color to an opposite one.

All unities of the system are constantly reassessing their color, size and opacity; and as they change their visual properties they influence their close neighbors into changing their own colors, sizes and opacity. They influence their neighbors into either copying them, or doing the opposite of what they're doing; either one may happen according to the overall tendency around them.

On top of this behavior defined by the exchange of information from unities that are close to one another, there's another input that can influence how the unities behave: the manipulation done by spectators through the multi-touch screen. Multiple spectators can simultaneously interact with the software, push units around, mixing or separating them, and seeing how that affects they change their color and size. The spectators can push different groups together and see how they react to their new neighbors, they can separate formations and see how each of their choices will now unfold, etc.

Multiple ways to influence the system through the multi-touch screen.

When holding a finger on the screen, a menu with two orbiting buttons appear, one of the buttons force all the units in a radius to be the same color and size and the other randomize the units. The radius of those actions can be changed and the resulting area of effect dragged around by moving the finger used to open the menu and the finger pressing one of the buttons. Those two actions bring a set of more radical transformation to the units, and allow the spectators to see how they react under more extreme situations.

Those user interactions create a playful playground where the spectators can freely push the unities around and see how the system reacts to it. The manipulations done by the spectators force the system into new states, which can lead to unexpected interactions between its units as they follow their rules for updating their graphical properties. In a way, the actions that the spectators can do on the system are never definitive, they push the system on a new direction which may then unfold and evolve in unexpected ways.

The multi-touch screen

The hardware of the system was a custom home made multi-touch screen that use a technique called Frustrated Total Internal Refraction (FTIR). The screen is composed of a strip of infrared LEDs wrapped around a thick acrylic glass. When turned on, the infrared light will bounce inside the acrylic because of its internal refraction, but when a object like a finger gets pressed against it the light that was trapped inside acrylic exit the pane in the opposite direction of the object. That means that a infrared camera positioned underneath the glass can see the blobs of the fingers when they get pushed against the acrylic, but cannot clearly see an object that is just hovering above the surface.

Schematics of how the hardware was organized and how the multi-touch screen works.

With the screen setup in a place without other sources of infrared emission (like our lovely sun), the infrared camera underneath the glass pane sees only a black image, when soft objects like fingers get pressed against the screen, the camera sees blobs of white light, because of the frustrated internal refraction. Computer vision technique is then used to track the position of those blobs and get their X and Y positions relative to the screen. Those positions are fed into the generative software and used to drive the interactions with the unities in real time.

The top of the acrylic is covered with a special sheet that scatter light so the image generated by the software can be projected back to it from underneath the glass surface. Because the infrared light and the projected visible light work on different light spectrum, they don't affect each other, the projected graphs don't disturb the multi-touch input and the infrared don't distort the projected image. The light from the projection and the infrared light is redirected underneath the glass using a mirror, so that the position of the internal equipment can be simplified.

Outputs from the system

Cohesive and pleasing images emerge from finely tunned set of interactions rules of the generative software. Watching the system evolve and its internal interactions unfold is an interesting experience, groups of unities form and dissipate, dissidences emerge and then sync, forming new groups that expand just to lately been broken apart by their own dissidents, in a continuous flow of transformation and evolution. The emerging images share a common style, but they are never the same, and its impossible to predict what new ones may come up as the system runs and the spectators push it to new directions.

Sample output from the system.

Appearances

This work was first presented as the final graduation project for my architecture degree, later it also got showcased on:

A port of the software was also available on the App Store for the iPhone and iPad.