A note to visitors: I'm currently not actively developing this project. It is due to the lack of feedback (I know too little electronic musicians who would spend the time to play with the prototype), and because the prototype was more than sufficient for my own meditative playing, that my interests shifted. I'm currently engaged with square things in another place, developing a big database and website (you'll have to register in the forum if you'd like to know more details).
Anyway, if you want to take this project further, you're welcome to do so. I uploaded the code and the MPLab project in its unfinished state. Some things like the number of input lines to use are still hardcoded, and the menu could be made better (one of the harder parts), so you probably have to tweak the code to make it work to your liking. To be honest, I'm not even sure if I left the code in a functioning state but I'm sure that at least the version I flashed last works here on my table. So at least the included object file should work ;).
As always, if you should need assistance, please send me a message.
Enlivening Sonic Landscapes
Sound in motion, with a more ergonomic MIDI interface.
When people talk about a MIDI controller, they typically mean an input device which has numerous knobs and sliders for playing and manipulating sounds on the computer. Most people seem to be happy with their knobs and sliders. I found out that I am not. Most software synthesizers I tried have a large number of parameters that I can play with while creating, live, if I want to. If I am able to - because those knobs limit me to turning only two of them at the same time, or four if I'm really skilled. So in fact there is no elegantly relaxed and continuous playing but stressful switching from one knob to the next. Given that so many creative heads use this de-facto-standard these days it was an astonishing discovery for me that they did not adopt something more able.
Knobs seem to limit the expressions of my body to only two dimensions - one knob in each hand. My hands could do much more, having at least three degrees of freedom each, and this does not count in my arms, legs, body and head. A gesture is composed of simultaneous motions in all three spacial dimensions. I want to include this freedom in my creative activity. I need a gesture capturing interface!
My incentive for this project came out of coincidence. From a website about sensors I learned that there exist small parts called "accelerometers" or "inertial sensors". Since gravity is a force it will exert an acceleration as well and therefore these sensors can be used to measure their orientation relative to the surface of the earth. These are not cheap but they are affordable. I've done electronics and programming since I was a kid and I recently started to play with microcontrollers so there's nothing to stop me from building my own special MIDI controller!
Notice that I'm not the first one to build a motion capture device (see links section). There are commercial products available that would do just the thing I want with much less effort. Why build your own then, you might ask. Well, if you want it quick and easy then go for the commercial thing. If you want a cusomizeable device and like the added pleasure to have made it yourself then do it. This website presents the results of my own project, along with ideas and code from other open websites. In the latter case the origin the presented material is based upon is cited. Once complete, this information should help you to build your own motion capturing interface or start a derivative work, given some basic skills. If you do so, bear in mind to respect the original authors of the information. Please let me know about your project and/or your suggestions.
- The main part is a mobile analog-to-MIDI (A/M) box.
- It's able to handle multiple analog inputs, for at least 5 accelerometers.
- Small sensor modules to be strapped to any part of the body. They are detachable from the A/M box and the box recognizes their presence automatically.
- Each sensor module carries an accelerometer chip, an on/off switch and possibly a sensitivity switch. LEDs on the sensor modules indicate operation and sensitivity setting.
- MIDI response to accelerometers is configurable at the A/M box using a display and menus.
- Get the basics to work: LCD, A/D and MIDI output at the microcontroller. Learn some C coding in the progress :). (done)
- At this point, an acceleration/tilt in each axis of the sensor module should output continuous controller (CC) messages. (done)
- Compact MIDI output (use "running status"). (done)
- CC output configureable: zero-point, sensitivity, direction of movement. (done)
- Menu structure with an eye on quick calibration and output configuration. (done)
- Configuration and calibration values are written to non-volatile memory.
- Experiment with it and look for viable configurations.
- Get some feedback from fellows and website visitors. Find a good title for the website ;).
- A/M box can be configured to output high-resolution CC messages (2 CCs from 1 analog value, 10-bit sampling).
- Possibility to generate note on/off messages with a settable threshold of acceleration (shaking the sensor).
- If necessary, try to compact MIDI stream even more with variable temporal resolution. Slow movement of a sensor needs to generate fewer messages than fast movement.
- Let musicians and inexperienced people play with the thing and watch out for difficulties.
Phase 3While State 1 and 2 are necessary for the thing to be useful, these are additional possibilities. If I go there will depend upon if I'm still interested to continue the project.
- Investigate convergent signal processing so that a combination of movements in the X/Y/Z directions produces a single output. This could be combined with simple pattern recognition in order to have a whole gesture output a single note event.
- Store more than one set of configuration values in the A/M box.
- Investigate other sensor types: LDRs, Phototransistors, Flexion sensors.
- Wireless operation so that the performer has more feedom to move around and really dance.
- Build a second unit so that two or more performers can interact. Probably this requires a MIDI merger box to be built.
- Sensors: there are multiple possibilities. I want to go with a triple-axis device. These parts are mass-produced for shaking compensation in digital cameras. Therefore most types are miniatures. This helps to limit the range of choices to those I can actually handle and solder.
Analog or digital - doesn't really matter. With digital sensors and an on/off switch with every sensor I need more input lines than with analog multiplexed lines (for I/O and addressing). I could use an input multiplexer with digital sensor outputs but that would make the device similarly complex as with analog multiplexers. Analog input lines can be used for other types of sensors as well. Sensors with analog output are cheaper, and ST Microelectronics offers samples of a suitable analog part which is also available from a local dealer, so I go with that one.
Part of choice: LIS3L06AL, 3-axis, analog-output, sensitivity selection, SMD with 1mm pad spacing.
- Microcontroller: any device with enough I/O lines, at least one A/D channel and an UART (for serial MIDI out) will do. I started with PIC16 MCUs a while ago. Now I want to try a compiler. The Microchip C18 compiler is for PIC18 and above. Also with an Atmel AVR MCU, there are many compiler choices, among them a GNU C compiler. If you want to go completely open-source try that.
Part of choice: PIC18F2420, 28 Pins, plenty of I/O lines, relatively cheap.
- Display: any backlit LCD I can obtain quickly and cheaply.
- What else:
5x 4051 8-to-1 analog multiplexer for up to 32 analog input lines (9x 4051 would give 64 lines at a single A/D input of the MCU).
A low supply voltage OPAmp like the LM2904 for analog buffering, so that the A/D converter can run fast. LM317 voltage regulator to give 3.3V power for the accelerometers.
5-Pin, 180deg., female DIN socket.
MIDI cable (DIY).
USB cable which will be used as the power source. This eliminates the need for batteries or transformers, as long as the thing is not wireless.
Cables and connectors for the sensor modules.
Some resistors, LEDs, switches and other parts.
The project is currently in a testing phase, so the schematics presented here may change.
Parts of this work are licensed in GPL. It is mentioned in the repective documents where the GPL is applicable.
Controller unit 20130423 (PDF) (64-input version, a smaller one will come).
The multiplexer outputs connected in this order, the order of the signals received at the A/D input when counting the address from 0 upwards should be: U4 inputs X0 to X7, U5 inputs X0 to X7, and so on until U11 inputs X0 to X7.
R12 to R22 are connected in parallel to the on/off pot on the sensor module, which makes this pot act in a non-linear way. If you want linear behavior then leave these resistors away.
Notice that I never tested the device with so many inputs, and that the code probably has to be adapted, too.
Sensor module 20130423 (PDF).
Exchange the pot R1 by a 100k value and exchange R7 by a 330 Ohms resistor to limit the current to 10 mA when the suppress button is pressed (I did not test this!). R12 to R22 on the controller have to be 100k as well in that case.
Sensor module layout on proto-board 20071209 (PDF). This is not yet built and may change a bit in the next revisions.
Important bug notice: This revision does not contain R7 - if the On/Off pot is at or near zero and the suppress switch is pressed then the switch short-circuits the 3.3V power supply acros the pot, thereby likely burning the pot.
Source packet 20130423 (ZIP, KiCad and OpenOffice). I made a few changes to some symbols and had to draw new ones. Therefore the symbol lib file is included. I don't think that you have to install it but it might be useful to you.
The source code is licensed with the GPL, so feel free to use and modify it.
MPLab project files, source files and (hopefully) working binary: Source 2009-01-15
October 2007Project is outlined, parts are selected.
ST Microelectronics kindly provided me with a sample of three dwarfs:
November 15, 2007
First working Prototype, does output XYZ CC messages for a single sensor. No multiplexer and no menues yet. Results are impressive. Playing with it is exciting and hypnotizing. Will upload synth recordings soon.
November 21, 2007
I experimented a bit, using the inertia sensor with only a single synth plug-in (the rich-in-sound Triple Cheese). I played several hours of drone music exploring only a single synth patch. Mapping the XYZ inputs to the tone and damp controllers and using a lot of feedback I can make morphing sonic landscapes in the head. I can literally push the sound in a specific direction or scupt it slowly with small circular movements. It's especially meditative with headphones. Listen to examples.
The input buffer OPamp I thought of (LM358, I salvaged from old harddrive electronics) proved to be too slow for a switched input. I inserted a 250-cycle wait in the code instead, and it works so far. As the main operating control I want to use a rotary quadrature encoder I got out of a computer scrolling mouse, maybe with the original scrollwheel. It's the part next to the MCU with the protruding wooden stick.
Also, it will be useful to have at least 2 bright LEDs on the module, pointing into different directions (only one shown in the picture). I found out that the LED spots can give good visual feedback when positioning the device.
The module will also contain a 'suppress' button which while pressed will simply short circuit the on/off pot to ground, forcing MIDI output for this module to be off.
As the housing for the sensors I thought of taking a DB9 plug housing. It's just large enough to hold the sensor, switch and LEDs, and I found a cable that has separatable plug shells. But after I plan to add three pots and a button I will have to use something larger. Probably a DB15 shell will do :).
Now I'm working on the software part - menu structure and such.
December 9, 2007
I made a few sketches and uploaded them. There will be some explanation necessary. I'll do it when I have time for it.
Jauary 8, 2008
No progress with the project during the past weeks because I'm occupied by other work.
Jauary 15, 2009
The project is currently on halt.
April 23, 2013
*sigh* Times are a-changin' and I moved on, away from too much tech stuff. Currently I have no plans of continuing this project. Finally I made it to install KiCad again and fix the bugs in the schematics ;).
Notice that this is a digital interface device only which does not produce sound by itself. It is possible to attach any number of parameters of a synthesizer or effect unit to this interface. The possibilities are endless. Therefore, the following are examples of my own experimentation only and can in no way display the whole spectrum of the capability. The examples might even not apply to everyone's taste (it's siren-like drone music, my mother says "it hurts her in the heart" :) ). Think your own application!
Morphing Sound 2:06, 1.8MB OGG-Vorbis - example of seamlessly morphing sound. Controls: one 3-axis inertia sensor. Made in one take with the Triple Cheese synthesizer plugin with built-in delay + feedback. Additional effects applied: none. Sorry for the crackles.
Fast Motions 2:33, 2.1MB OGG-Vorbis - example sound with fast motions (throwing the sensor in various directions) and circular motion. Controls: one 3-axis inertia sensor. Made in one take with the Triple Cheese synthesizer plugin with built-in delay + feedback. Additional effects applied: none. Sorry for the crackles.
Drone 1 4:20, 2.7MB OGG-Vorbis - taken from a jam session. Controls: one 3-axis inertia sensor. Made in one take with the Triple Cheese synthesizer plugin with built-in delay + feedback. Additional effects applied: only an equalizer to reduce bass and add highs.
Here is the full version, a little edited. Headphones on, happy space-out: What a Drone 21:17, 32.5MB MP3.
Drone 2 2:13, 1.9MB OGG-Vorbis - taken from a jam session. Throwing the sound right and left. Controls: one 3-axis inertia sensor. Made in one take with the Triple Cheese synthesizer plugin with built-in delay + feedback. Additional effects applied: none.
Here are more meditational pieces, all a little edited:
rec27 12:46, 18.8MB MP3.
rec32a 13:37, 23.6MB MP3. A looong slowly changing drone.
rec36 24:53, 36.2MB MP3.
rec39a 4:44, 7.4MB MP3.
Below you find some videos that demonstrate the device in operation. These videos are standard DivX AVI files but in the case they don't play correctly on your machine I warmly recommend using VLC media player. That one plays just everything.
Demo: effect of orientation 2:40, 10.31MB DivX4.12/lameMP3 (XviD alternative) - two oscillators, distortion and reverb, shown separately and together. Controls: one 3-axis inertia sensor. Made with the Triple Cheese synthesizer plugin. Additional effects applied: none.
A short piece 1:41, 6.58MB DivX4.12/lameMP3 (XviD alternative) - quick slow improvisation. Not always in sync. Controls: one 3-axis inertia sensor. Made with the Triple Cheese synthesizer plugin. Additional effects applied: none.
Demo: shaking and small circles 1:09, 4.5MB DivX4.12/lameMP3 (XviD alternative) - Controls: one 3-axis inertia sensor. Made with the Triple Cheese synthesizer plugin. Additional effects applied: none.
Similar and Otherwise Interesting Gadgets:BioControl
Drancing. Darren Kelly made a very similar device, only he has not been convinced by MIDI and made his own synth instead.
eobody, a sensor-to-MIDI box.
Metasaxophone. The author was not satisfied with the limited interface of a sax. And built MIDI controllers onto it.
reactable. Fascinating but... let's start with the basics. ;)
MIDI protocol, compact explanation.
PICList - lots of hints and program snippets (see source code library). (Thanks to Rob Hamerling for his USART interrupt example program, it made me happy.)