Multitouch360

 - a Platform for Interactive Art Installations

Multitouch360 is a project by Johann Korndoerfer and Thorsten Blum that aims at creating a hemispherical multitouch display. It has now officially reached some degree of "finished", meaning that all essentials are working, no cheap workarounds remain in either hardware or software and the software framework is capable of letting anyone code cool stuff.

Watch the short video below:

This is my debug application running on Multitouch360, showing a wire hemisphere along with camera calibration markers and visualizing touch events. As you can see, touch recognition works reasonably well.

How does it work?

We use a technique called FTIR (Frustrated Total Internal Reflection) to detect touches. This means that an infrared camera looks at the hemisphere from the inside. The picture to the left is what the camera sees - you can see a shadow from the hand and five bright points at the tips of the fingers. Those bright points are created by the users' fingers touching the outer acrylic hemisphere (called the Wave Guide). The Wave Guide is made of clear acrylic and flooded with infrared light from 288 infrared LEDs set in the rim. The IR light can't escape the acrylic except when it is touched by an object with a sufficiently large refractive index, like water or human skin. Thus, the infrared light enters the finger, gets scattered and can be picked up by the camera.

Since you can't project anything onto clear acrylic, we needed a inner layer of matte acrylic (called the Diffusor) just inside the Wave Guide. This layer makes the bright points from the outer layer a bit blurry for the camera, but we would have done the blurring in software anyway. Using a mirror to save some space, we project a rendered image onto the Diffusor, again from below. You can see the whole setup in the image to the right.

The image that is projected onto the sphere gets distorted quite a bit, which has to be anticipated in software. The same applies for the other direction: touch locations have to be transformed back into 3d as well. Both the projection and the camera have to be calibrated, which can be done by clicking five positions with the mouse (calibrating the projector), then touching the same positions on the sphere (calibrating the camera). You can see a visualization of the gradient descent that solves the resulting optimization problem right at the beginning of the video above.

The Software

I made a nice little framework for Multitouch360 that lets you write multitouch applications. The debug application from the video above looks like this:

from multitouch360 import *

class DebugApplication(MultitouchApplication):

    def touch_down(self,touch):
        TouchIndicator(touch)    

    def touch_move(self,touch):
        pass

    def touch_up(self,touch):
        Explosion(touch.world_pos)

    def display(self):
        glutWireSphere(1,30,30)

DebugApplication()

And this is it. You can define textures, buttons, etc. easily and the weird math you need for handling things on a sphere is taken care of.

What do you do with it?

First, you can do something with it. If you know some Python and OpenGL, you can code your own application. The Multitouch360 framework will simulate the hemisphere on any computer and you can click on a virtual version of it with your mouse. If you are interested in doing something with it and you are located near Karlsruhe, Germany, please let me know. Keep in mind that anything that runs on Multitouch360 must be tailored to the spherical layout - thinking in 2d will make you wish you had a flat table, but embracing the hemispherical form lets you do some things that are impossible otherwise: planets, star maps, boobs or eyes are some of the things that may come to life. Games that involve not seeing what your opponent is doing on their side are another obvious candidate.

Second, i have made a few very simple demo applications that are hardly more than a hello-world, but may show what's possible. Watch people play Global Thermonuclear War in the video below, incinerating whole countries by touching them. The only winning move is not to play. (Application written in about three hours.)