A note to visitors: I'm currently not actively developing this project. It is due to the lack of feedback (I know too little electronic musicians who would spend the time to play with the prototype), and because the prototype was more than sufficient for my own meditative playing, that my interests shifted. I'm currently engaged with square things in another place, developing a big database and website (you'll have to register in the forum if you'd like to know more details).
Anyway, if you want to take this project further, you're welcome to do so. I uploaded the code and the MPLab project in its unfinished state. Some things like the number of input lines to use are still hardcoded, and the menu could be made better (one of the harder parts), so you probably have to tweak the code to make it work to your liking. To be honest, I'm not even sure if I left the code in a functioning state but I'm sure that at least the version I flashed last works here on my table. So at least the included object file should work ;).
As always, if you should need assistance, please send me a message.

Enlivening Sonic Landscapes

Sound in motion, with a more ergonomic MIDI interface.

When people talk about a MIDI controller, they typically mean an input device which has numerous knobs and sliders for playing and manipulating sounds on the computer. Most people seem to be happy with their knobs and sliders. I found out that I am not. Most software synthesizers I tried have a large number of parameters that I can play with while creating, live, if I want to. If I am able to - because those knobs limit me to turning only two of them at the same time, or four if I'm really skilled. So in fact there is no elegantly relaxed and continuous playing but stressful switching from one knob to the next. Given that so many creative heads use this de-facto-standard these days it was an astonishing discovery for me that they did not adopt something more able.

Knobs seem to limit the expressions of my body to only two dimensions - one knob in each hand. My hands could do much more, having at least three degrees of freedom each, and this does not count in my arms, legs, body and head. A gesture is composed of simultaneous motions in all three spacial dimensions. I want to include this freedom in my creative activity. I need a gesture capturing interface!

motion visualization
Imagine this: you have your headphones on. When you turn your hand to the right there are the dark and low sounds. Turn your hand to the left and you hear bright shining bells. The other hand lets you shift the sound left or right as you like or just make more space as you turn forward. You can slowly turn from one side to the other, from dark to bright, from closed to wide, and the sound in your head follows your motion. With your hands you can sculpt a sonic landscape that surrounds you. Not only left and right but differently in every direction.

The Plan

My incentive for this project came out of coincidence. From a website about sensors I learned that there exist small parts called "accelerometers" or "inertial sensors". Since gravity is a force it will exert an acceleration as well and therefore these sensors can be used to measure their orientation relative to the surface of the earth. These are not cheap but they are affordable. I've done electronics and programming since I was a kid and I recently started to play with microcontrollers so there's nothing to stop me from building my own special MIDI controller!

Notice that I'm not the first one to build a motion capture device (see links section). There are commercial products available that would do just the thing I want with much less effort. Why build your own then, you might ask. Well, if you want it quick and easy then go for the commercial thing. If you want a cusomizeable device and like the added pleasure to have made it yourself then do it. This website presents the results of my own project, along with ideas and code from other open websites. In the latter case the origin the presented material is based upon is cited. Once complete, this information should help you to build your own motion capturing interface or start a derivative work, given some basic skills. If you do so, bear in mind to respect the original authors of the information. Please let me know about your project and/or your suggestions.

Outline

rough block diagram
Scheme of the setup. Multiple sensor units (4 analog lines each) are plugged to the analog-to-MIDI box. A multiplexer connects one analog line at a time to the microcontroller analog input. The microcontroller does the calculation and sends MIDI messages to the personal computer.

Phase 1

Phase 2

Phase 3

While State 1 and 2 are necessary for the thing to be useful, these are additional possibilities. If I go there will depend upon if I'm still interested to continue the project.

Construction

Part Selection

Schematics

The project is currently in a testing phase, so the schematics presented here may change.
Parts of this work are licensed in GPL. It is mentioned in the repective documents where the GPL is applicable.

schematic thumbnail Schematics:
Controller unit 20130423 (PDF) (64-input version, a smaller one will come).

The multiplexer outputs connected in this order, the order of the signals received at the A/D input when counting the address from 0 upwards should be: U4 inputs X0 to X7, U5 inputs X0 to X7, and so on until U11 inputs X0 to X7.
R12 to R22 are connected in parallel to the on/off pot on the sensor module, which makes this pot act in a non-linear way. If you want linear behavior then leave these resistors away.

Notice that I never tested the device with so many inputs, and that the code probably has to be adapted, too.

Sensor module 20130423 (PDF).
Exchange the pot R1 by a 100k value and exchange R7 by a 330 Ohms resistor to limit the current to 10 mA when the suppress button is pressed (I did not test this!). R12 to R22 on the controller have to be 100k as well in that case.

Layouts:
Sensor module layout on proto-board 20071209 (PDF). This is not yet built and may change a bit in the next revisions.
Important bug notice: This revision does not contain R7 - if the On/Off pot is at or near zero and the suppress switch is pressed then the switch short-circuits the 3.3V power supply acros the pot, thereby likely burning the pot.

Source packet 20130423 (ZIP, KiCad and OpenOffice). I made a few changes to some symbols and had to draw new ones. Therefore the symbol lib file is included. I don't think that you have to install it but it might be useful to you.

Source Code

The source code is licensed with the GPL, so feel free to use and modify it.

MPLab project files, source files and (hopefully) working binary: Source 2009-01-15




Timeline

October 2007

Project is outlined, parts are selected.

ST Microelectronics kindly provided me with a sample of three dwarfs:

accelerometers with size comparison
The LIS3L06AL 3-axis accelerometer ICs come in a 5x5x1.6mm leadless surface mount package. Happy soldering!

sensor, mounted
I glued the sensors onto DIP8 sized pieces of proto-board in order to handle them. The connections have been made with a twisted pair of single wires of a strand. Take a small solder iron tip for this. Grounding of the self test pin and a bypass capacitor SMD are 'integrated'.

November 15, 2007

First working Prototype, does output XYZ CC messages for a single sensor. No multiplexer and no menues yet. Results are impressive. Playing with it is exciting and hypnotizing. Will upload synth recordings soon.

November 21, 2007

I experimented a bit, using the inertia sensor with only a single synth plug-in (the rich-in-sound Triple Cheese). I played several hours of drone music exploring only a single synth patch. Mapping the XYZ inputs to the tone and damp controllers and using a lot of feedback I can make morphing sonic landscapes in the head. I can literally push the sound in a specific direction or scupt it slowly with small circular movements. It's especially meditative with headphones. Listen to examples.

prototypes of microcontroller and sensor electronics
Prototype electronics finished, with a few changes to the original plan. On this picture you see the electronics in their current state (on top of my sketches). In the upper left corner there is the sensor module. Below is the MCU part with input multiplexer at the left side, the MCU with controls and LCD in the middle and the 3.3V regulator for the sensor voltage in the upper right corner.

The input buffer OPamp I thought of (LM358, I salvaged from old harddrive electronics) proved to be too slow for a switched input. I inserted a 250-cycle wait in the code instead, and it works so far. As the main operating control I want to use a rotary quadrature encoder I got out of a computer scrolling mouse, maybe with the original scrollwheel. It's the part next to the MCU with the protruding wooden stick.

prototype of sensor electronics
At the sensor module i plan to add 2 potentiometers and to exchange the on/off switch with a pot as well (only the on/off pot is implemented in this picture). So there will be a total of 6 analog outputs at each sensor module (X, Y, Z, pot 1-3).
Also, it will be useful to have at least 2 bright LEDs on the module, pointing into different directions (only one shown in the picture). I found out that the LED spots can give good visual feedback when positioning the device.
The module will also contain a 'suppress' button which while pressed will simply short circuit the on/off pot to ground, forcing MIDI output for this module to be off.

As the housing for the sensors I thought of taking a DB9 plug housing. It's just large enough to hold the sensor, switch and LEDs, and I found a cable that has separatable plug shells. But after I plan to add three pots and a button I will have to use something larger. Probably a DB15 shell will do :).
Now I'm working on the software part - menu structure and such.

December 9, 2007

I made a few sketches and uploaded them. There will be some explanation necessary. I'll do it when I have time for it.

schema of sensor module
A DIP8 Sensor, 3 Potentiometers, 2 Switches, 3 LEDs, 3 Resistors, external connectors and an 8-strand cable. Does all this fit into a 37x30mm DB15 plug shell? I think there is still room for more. In theory.

Jauary 8, 2008

No progress with the project during the past weeks because I'm occupied by other work.

Jauary 15, 2009

The project is currently on halt.

April 23, 2013

*sigh* Times are a-changin' and I moved on, away from too much tech stuff. Currently I have no plans of continuing this project. Finally I made it to install KiCad again and fix the bugs in the schematics ;).


Sound Samples

Notice that this is a digital interface device only which does not produce sound by itself. It is possible to attach any number of parameters of a synthesizer or effect unit to this interface. The possibilities are endless. Therefore, the following are examples of my own experimentation only and can in no way display the whole spectrum of the capability. The examples might even not apply to everyone's taste (it's siren-like drone music, my mother says "it hurts her in the heart" :) ). Think your own application!

Morphing Sound 2:06, 1.8MB OGG-Vorbis - example of seamlessly morphing sound. Controls: one 3-axis inertia sensor. Made in one take with the Triple Cheese synthesizer plugin with built-in delay + feedback. Additional effects applied: none. Sorry for the crackles.

Fast Motions 2:33, 2.1MB OGG-Vorbis - example sound with fast motions (throwing the sensor in various directions) and circular motion. Controls: one 3-axis inertia sensor. Made in one take with the Triple Cheese synthesizer plugin with built-in delay + feedback. Additional effects applied: none. Sorry for the crackles.

Drone 1 4:20, 2.7MB OGG-Vorbis - taken from a jam session. Controls: one 3-axis inertia sensor. Made in one take with the Triple Cheese synthesizer plugin with built-in delay + feedback. Additional effects applied: only an equalizer to reduce bass and add highs.
Here is the full version, a little edited. Headphones on, happy space-out: What a Drone 21:17, 32.5MB MP3.

Drone 2 2:13, 1.9MB OGG-Vorbis - taken from a jam session. Throwing the sound right and left. Controls: one 3-axis inertia sensor. Made in one take with the Triple Cheese synthesizer plugin with built-in delay + feedback. Additional effects applied: none.

Here are more meditational pieces, all a little edited:
rec27 12:46, 18.8MB MP3.
rec32a 13:37, 23.6MB MP3. A looong slowly changing drone.
rec36 24:53, 36.2MB MP3.
rec39a 4:44, 7.4MB MP3.

Videos

motion visualization

Below you find some videos that demonstrate the device in operation. These videos are standard DivX AVI files but in the case they don't play correctly on your machine I warmly recommend using VLC media player. That one plays just everything.

Demo: effect of orientation 2:40, 10.31MB DivX4.12/lameMP3 (XviD alternative) - two oscillators, distortion and reverb, shown separately and together. Controls: one 3-axis inertia sensor. Made with the Triple Cheese synthesizer plugin. Additional effects applied: none.

motion visualization A short piece 1:41, 6.58MB DivX4.12/lameMP3 (XviD alternative) - quick slow improvisation. Not always in sync. Controls: one 3-axis inertia sensor. Made with the Triple Cheese synthesizer plugin. Additional effects applied: none.

Demo: shaking and small circles 1:09, 4.5MB DivX4.12/lameMP3 (XviD alternative) - Controls: one 3-axis inertia sensor. Made with the Triple Cheese synthesizer plugin. Additional effects applied: none.

Creative Commons License
These example sounds and videos are licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 License. If you need a better/longer version don't hesitate to mail me ;-).

Links

Similar and Otherwise Interesting Gadgets:

BioControl
Drancing. Darren Kelly made a very similar device, only he has not been convinced by MIDI and made his own synth instead.
eobody, a sensor-to-MIDI box.
Metasaxophone. The author was not satisfied with the limited interface of a sax. And built MIDI controllers onto it.
MIDIsense
reactable. Fascinating but... let's start with the basics. ;)

References

Atmel
Microchip
MIDI protocol, compact explanation.
PICList - lots of hints and program snippets (see source code library). (Thanks to Rob Hamerling for his USART interrupt example program, it made me happy.)
ST Microelectronics