Audio Processing With Image Units

As the Mac world looks forward to news of what's in store for the next major version of Mac OS X, we take a look back on an interesting technology, introduced in the last major version, whose audio-processing potential has so far been overlooked.

With Apple's Worldwide Developer Conference (WWDC) taking place at the beginning of August, the rumour mill has been grinding away in anticipation of the announcements CEO Steve Jobs might make in his opening keynote speech. While Apple have already pre-announced that Leopard, the next major version of Mac OS X (10.5), will be unveiled, there's speculation that the highly anticipated Intel-based successor to the Power Mac will also be shown.

Monalisa-AU enables you to process audio using Image Units intended for image processing by Apple's Core Image Technology. Ever wondered what a drum loop processed by a Triangle Tile filter would look and sound like? Now you can find out.

Core Improvements

Although Intel only introduced their Core processor in January this year, the processor that now powers Macbook Pros, Macbooks, iMacs and the Mac Mini, this month the company launched the Core 2 processor, which is part of the first family of chips to feature Intel's new Core micro-architecture. Simply put, the micro-architecture of a processor refers to its actual design, and, in addition to simply performing better and consuming less power, the new Core micro-architecture has two features in particular that will be useful to musicians: firstly, it offers support for EMT64 (Intel's 64-bit memory-addressing instructions), meaning that if and when Apple adopt the processor, the resulting Macs will be able to run 64-bit applications, which was previously only possible on G5-based iMacs and Power Macs. Samplers that can address large amounts of RAM are just around the corner...

Secondly, the Core micro-architecture introduces additional SSE-style instructions (termed by some as SSE4), the kind used heavily by audio developers to optimise performance for DSP algorithms, along with a more buzzword-friendly feature called Advanced Digital Media Boost (ADMB). SSE operations previously required two processor clock cycles to execute, and ADMB is best described by Intel's developer-friendly subtitle 'single-cycle SSE', meaning that an SSE instruction is fully executed in a single clock cycle.

In theory, this means that a Core 2 processor can now execute twice as many SSE instructions in the same amount of time which, for time-critical audio algorithms, will translate into a huge performance benefit. Coupled with the fact that EMT64 already offers more SSE registers (a register is the part of the processor that stores data being worked on by SSE instructions), the Core 2 processor could be a huge leap forward for those working with audio.

Come On See The Noize

One particularly interesting technology Apple introduced in Tiger was Core Image, which makes it easy for applications to handle image processing in the same way that Core Audio makes it easy for applications to support audio. And while Core Audio provides a plug-in framework known as Audio Units, allowing third parties to create additional audio effects and instruments, Core Image offers the Image Units plug-in framework, allowing third parties to create image-processing tools, like the filters in Photoshop, for example.

If you're thinking "this is all well and good but what does it have to do with music and audio?", don't worry: we haven't changed our name to Picture On Picture. Image processing and audio processing actually have quite a bit in common. If you break it down simply, digital audio is represented by the level of a signal at a point in time, while a digital image is represented by the value of a colour at a point in space. Because of this, both audio and image manipulation make use of signal processing techniques such as filtering; so what would happen if you processed audio using an image filter?

While audio processing with image filters might not be something that would naturally occur to you, some Japanese developers have come up with a free Audio Unit called Monalisa-Audio Unit (www.monalisa-au.org) that lets you do exactly that: process audio via an Image Unit. It's currently Power PC-only, and although many of the Image Units produce a distorted mess, some of the results can be quite useful — and at least it would give you a useful way of answering the next person who asks if something you're working on could sound a bit more purple!

Monalisa-Audio Unit is still in its infancy and has some limitations. Currently, buffer sizes of 1024 are recommended, as the plug-in uses fairly long look-ahead times in order to have a big enough block to make the Image Unit's processing produce an audible result. There are also some stability issues to be worked out, but the basic idea is really nice and this will easily become the most curiously destructive Audio Unit in your collection!

Audio Shading

The technology behind Image Units is rather neat. The actual processing routines are written in a dialect of the Open GL Shading Language (which is known as the Core Image kernel language) and are compiled just before they are needed by the system. What this means for the user is that Core Image targets the most suitable hardware on which to run the Image Unit's processing; if you don't have any suitable graphics hardware in your Mac, the processing runs on the regular system processor core (or cores). However, with a graphics card featuring a GPU (Graphics Processing Unit) that supports programmable shaders (and most modern graphics cards do), Core Image will offload the processing to your graphics card.

For those who aren't sure about the term 'shader', this is simply a word used in 3D computer graphics for an algorithm that defines how the surface of an object should be rendered, such as how light should be reflected. However, shaders can implement other types of image processing features for 2D graphics as well, such as those exploited by Core Image, and many people have carried out research as to how shaders on GPUs could be exploited for other types of algorithms, audio being an obvious example.

With the current division between the computer's processor cores or separate DSP chips for audio — either the algorithm runs on the computer's processor or a specific DSP chip — a similar system for audio plug-ins, where the same plug-in was compiled 'just in time' (as they say in the computer world) to take advantage of whatever hardware you had available would be a huge step forward. Sadly, it's not that easy; as in the GPU example, shaders are better suited to some audio algorithms than others.

However, the authors of Monalisa-Audio Unit have also experimented with a couple of Image Units that implement delay and band-pass filter audio algorithms in the Core Image kernel language. Using Monalisa-Audio Unit to run these Image Units, you can now experiment with audio-based algorithms running on a GPU. It's not necessarily groundbreaking computer research, but it is the first time most users will have been able to experiment with such technology so readily. The same authors also hope to develop a reverb algorithm as an Image Unit, which was a goal announced by a company called Bionic FX (www.bionicfx.com) back in 2004, although its web site appeared to have recently expired at the time of writing.

While technology like Monalisa-Audio Unit is still experimental, it shows promise for future methods of audio processing, taking some of the strain off the computer's main CPU, in addition to providing a way of processing audio with tools you would never have considered using before.