We cut through the jargon to explain how software instruments achieve their incredible realism.
Reading some recent SOS reviews of flagship orchestral sample libraries from the likes of Spitfire, EastWest and VSL, I was struck by just how much jargon is involved in describing their features. Anyone new to the idea of creating a realistic orchestral composition in their computer, or whose most recent encounter with ‘fake’ musical instruments involved comparatively simple hardware ROMplers and workstation keyboards, would struggle to understand what they’d be getting from a piece of software that could cost hundreds of pounds/dollars and consume an entire SSD drive on its own. So, in this article, I’ll try and explain how these things work, so that the next time you see a review of Gigantic Trombones Vol 9, Enchanted Violas or Sensual Hollywood Banjos, you’ll know what the heck we’re talking about.
The Basics
Most virtual instruments are built using samples — that is, audio recordings of the instrument you’re ‘virtualising’. (There are some exceptions, most notably those that use physical modelling; for more info see the ‘Getting Physical’ box later on). So let’s start with a simple hypothetical virtual instrument: a solo violin. Because we want this to be the best virtual violin it can possibly be, we’ll hire a world‑class session player to bring his Stradivarius into our studio, and play the clearest, crispest, most mellifluous Middle C known to humanity for us to record. Now, we take that sample and load it into a software sampler like Native Instruments’ Kontakt. Hit Middle C on your keyboard, and you’ll hear that lovely note again. Congratulations on your first virtual instrument!
You might eventually consider playing notes other than Middle C, but don’t worry: all software samplers have the ability to pitch‑shift audio, so that our violin sample can be played back at any other pitch. This works well enough for notes a few semitones either side of the recorded note, but stray too high or too low and it’ll start to sound quite unnatural. So let’s call in our session violinist again, and ask him to play another C, this time an octave higher. If we map the first C to Middle C on our keyboard, plus all the notes half an octave either side, and then do the same an octave up for the higher C, we’ll have two octaves of pretty convincing violin to play with.
Twenty years or so ago, when hard‑disk space was expensive and limited, that’s exactly how it was done. But with storage space so cheap and fast nowadays, we can afford to be uncompromising, so why not go the whole hog and ask our session player to play every single note in the violin’s range, for maximum realism?
Dynamics
This is exactly what most modern sample libraries do, and it allows us to play some relatively convincing solo violin lines from our MIDI keyboard, across the entire instrument’s range. But what if you want some notes to be played quietly and others loudly? You have two options. The first is to apply your MIDI keyboard’s velocity data to the synth‑style controls in your software sampler, to modulate the volume at which the samples play back. You can even add a low‑pass filter too, setting it so that lower‑velocity notes sound duller than higher‑velocity ones, mimicking the softer timbre you get when playing a real violin more quietly.
Again, this is very much how dynamics were handled many years ago, and it can work well enough, but since we’ve all got multi‑terabyte drives and our sample library is still only a few megabytes, why not get our session player in again and ask him to play every single note once again, but at three different volumes? Then we’ll tell our sampler to play the quiet version of each note for MIDI velocities below, say, 50; the medium‑volume samples from 50 up to 100; and then the loud samples for anything above that.
These are what’s known as velocity layers and, again, most modern sample libraries use this technique to enhance realism. If we were selling our virtual violin, we might boast about it having ‘three velocity layers’, but you can use more — indeed, virtual drum instruments in particular tend to use a great deal more (BFD Drums, a division of InMusic, use up to 80 velocity layers per drum kit element). You can start to see why some of them ship on their own hard drives!
Virtual drum instrument BFD3 features up to 80 velocity layers per kit piece.
To get the most from our humble three velocity layers, though, we can even combine them with the MIDI‑controlled filtering scheme I mentioned earlier. And, to avoid an obvious ‘switch’ in samples between notes played at a velocity of 49 and those played at 50, we can crossfade between them, for a smoother transition.
Round Robin
Our virtual instrument is now quite usable, but try playing repeated notes at the same velocity and your ears will tell you instantly that you’re just hearing the same sample over and over again, rather than a real violin. If this were a virtual drum instrument we’d complain about the ‘machine gun’ effect (it’s particularly audible on drum hits, but our virtual violin isn’t immune to the problem).
One way around this would be to put our programming hat on again and tell the sampler to apply a tiny amount of random pitch‑shift to successive samples, to mimic the imperfections that our human violinist might exhibit if we told him to play E7 eight times in a row. Better still, though, would be to drag our patient session violinist back into the studio, and get him to play every sample — each note on the violin, at each of our three velocity layers — all over again. Then we tell our sampler to alternate between samples from the first session and the new ones we’ve just recorded, and now the machine‑gun effect is eliminated.
This is known in the trade as a round robin. Our virtual violin has one round robin per note, because our session player got bored and went home before we could ask him to record more, but a fully-featured orchestral instrument could easily have half a dozen round robins — and some of EastWest’s virtual string instruments have as many as 16.
Now our virtual violin — which started with just a single sample — already has close to 300 (six distinct samples for each of the 50‑odd semitones in its range).
One potential, albeit usually pretty minor, issue with round robins is that when you hit Play at the start of your song, you won’t know for sure which of the round robin samples will play back. In practice this is rarely an issue (because sample library makers are pretty meticulous types and will have been consistent in recording one mezzopiano Middle C after another), but if you simply must hear exactly the same thing every time you play your composition, some orchestral instruments feature a ‘round robin reset’ button, to allow you to do just that.
EastWest’s Hollywood Strings 2. On the right you can see faders for the different microphone options, and on the left is the ‘round robin reset’ button, which ensures consistent playback.
Take A Bow
So far, we’ve only asked our violinist to play straightforward notes at a variety of pitches and volumes. But the violin is capable of much more than that. You can pluck it (or play ‘pizzicato’), you can play harmonics, you can hit the strings with the back of the bow (‘col legno’), you can use the bow over the fingerboard (‘sul tasto’) or near the bridge (‘sul ponticello’) for different timbres. The same is true for other instruments: a drummer might play with brushes or mallets instead of sticks; bassists can slap and pop (whether they should is another matter), woodwind players can use flutter tongue... In musical terms, as well as in sample library parlance, these techniques are known as articulations, and unlike for pitch and volume, there’s no realistic way to ‘fake’ these using MIDI Controller messages to tweak the sample playback. If we want these playing techniques to be available in our virtual violin, we’ll have to record them.
By now our beleaguered session violinist is quite used to our phone calls, so in he pops to record his pizzicatos, harmonics and fancy bowing tricks, with multiple velocity layers and round robins. Our sample library has grown commensurately, and now has a different patch for each of these articulations, selectable from a drop‑down menu in our sample player of choice. But wouldn’t it be nice if we could switch playing styles instantly, in the same way a real violinist might switch between bowing and plucking?
Using keyswitches, you can play or program a realistic violin part with multiple playing techniques, just by drawing in or hitting the occasional extra MIDI note.
This is where keyswitches come in. Many multi‑articulation sample libraries employ the very lowest or highest MIDI notes (those outside the range of the instrument) as switches, to tell the sampler which set of articulations to play. So, if you play C1 (the bottom C on an 88‑note keyboard), you won’t hear an unnaturally low violin note; rather, it tells the sampler to use your ‘default’ bowed sample set. Play D1, and it’ll switch to pizzicato, and so on. Using keyswitches, you can play or program a realistic violin part with multiple playing techniques, just by drawing in or hitting the occasional extra MIDI note.
Clearly, this is a bit of a fudge: if you don’t have an 88‑note keyboard, you’ll have to mash your ‘octave down’ button repeatedly to get to the right articulation keyswitch, before jumping back up again to the playable range. And even if you do have a full‑size keyboard, remembering which key activates which playing style is far from intuitive, given that the keyswitches are unlikely to be in the same key as the music you’re playing.
NI’s S Series controllers have a Light Guide feature, which can be used to show you at a glance where your keyswitches are.
Native Instruments’ S Series keyboard controllers offer a helping hand here, by illuminating the relevant keyswitches in colour so you can find them easily, and some DAWs have attempted to simplify things by including a dedicated keyswitch automation section (Expression Maps in Cubase, Sound Variations in Studio One), so that you can easily see and change your articulations, with useful names rather than simple note numbers, when editing your MIDI performances. Even so, some composers find it easier to just run multiple instances of an instrument, with one set to legato, one to pizzicato, one to sforzando, and so on. That does require a lot of extra CPU and RAM resources, though, so if your computer can’t handle that many instances, you’re stuck with keyswitches.
The Sound Variations window in PreSonus Studio One. Sound Variations are PreSonus’ way of making keyswitches a bit more intuitive to work with.
Mic Options
Our virtual violin is now feeling pretty complete: we can play and program a realistic‑sounding performance of our violin, using a number of different articulations via the keyswitches. But what if the recordings don’t fit our production? Perhaps the violin was recorded in mono when we’d prefer stereo; or it was captured too close‑up and lacks reverb; or the room we recorded it in didn’t have the best acoustics?
We could, of course, try to simulate a better acoustic using a reverb plug-in in our DAW. Or, we could build a reverb into our sampled instrument — and that’s something that many sample libraries now do, often using convolution reverb for maximum realism. But if we want true flexibility, the thing to do is to record the violin using multiple microphones in multiple positions, so that we can change the sound of the recording even after we’ve played or programmed our violin part.
Many large orchestral sample libraries offer multiple mic options — close mics, an ORTF array, a Decca Tree, distant omni mics, maybe a Mid‑Sides setup — and these can usually be blended together in a mixer section to offer the kind of ambience your production needs.
Speaking of ambience, many companies go to the additional trouble of using world‑renowned concert and recording venues to capture their samples: for example, Orchestral Tools recorded their Berlin Symphonic Strings library at the world‑famous Teldex Studio; Spitfire Audio are regular customers of Abbey Road (their Abbey Road series of instruments covers the complete orchestral line‑up); and Toontrack’s Legacy Of Rock expansion for Superior Drummer was captured at the legendary AIR Lyndhurst. And some virtual‑instrument makers have gone even further, by buying their own studios: EastWest own the Bill Putnam‑designed Western Recorders in LA (and naturally renamed it EastWest Studios), while Vienna Symphonic Library did much the same with scoring studio Synchronhalle (now Synchron Studios).
A photo of one of the recording sessions for Spitfire Audio’s Abbey Road One series of orchestral instruments.
Wrapping Up
And so, what started innocently enough as a single recording of Middle C on the violin has ballooned into a multi‑gigabyte behemoth!
- We’ve got samples for each of the violin’s 50 playable semitones
- 3 velocity layers for each of those samples
- Round robin alternatives for each of those again
- 6 or so different articulations (each with its own velocity layers and round robins)
- 10 different microphone signals for each note, articulation, velocity layer and round robin...
Rather like the ancient ‘grains of rice on a chessboard’ problem, our simple virtual violin now comprises around 18,000 audio files! It’s also got an awful lot of clever stuff going on under the hood, such as MIDI‑modulated filtering and pitch‑shifting, keyswitches mapped to the different articulations... And perhaps a nicely designed graphical user interface to hide it all behind.
So, next time you play a virtual violin, or program a sampled sousaphone, or boot up Superior Drummer, perhaps spare a thought for the recording engineers who had to capture all those files, the assistants who had to name them and keep track of them all, the software engineers who had to program the sample players — and, of course, the skilled musicians who played all those notes!
Physical Modelling
If recording 60,000 samples and spending weeks programming a sampler to play them just right in order to get your computer to sound like a violin sounds like a slightly brute‑force approach... Well, you’d be right. And there is another, arguably more elegant way to go about it.
Physical Modelling is the name for a type of synthesis that’s aimed at recreating the sound of acoustic instruments. Consider the violin: when a string is played, it vibrates, generating a pitched sound. But the vibrations of the string also travel through the bridge and into the soundboard, which resonates at a number of frequencies related to its shape and dimensions. The other strings might vibrate in sympathy too, with more string vibrations travelling through the neck of the violin, exciting the back and sides of the body, interacting with the other vibrations travelling through the bridge, cancelling here and reinforcing there... And as that complex pattern of sounds radiates from the violin and into the air for us to hear, radiation from one part of the instrument will cancel or reinforce with sound emanating from another, and so on.
Building up a mathematical model of everything that contributes to the sound of an acoustic instrument is an immensely complicated business, but many physical modelling instruments manage it remarkably well. Modartt’s Pianoteq 8 Pro, for example, offers realistic pianos, while Applied Acoustic Systems’ String Studio VS‑3 uses the physics of stringed instruments as the basis for its sounds.
Modartt’s acclaimed Pianoteq achieves highly realistic piano sounds without using a single sample.
Advantages?
So what advantages does physical modelling have over a sample‑based approach? Well, the obvious one is that a physically modelled instrument doesn’t require hundreds of gigabytes of audio samples. Indeed, most physical‑modelling‑based instruments weigh in at perhaps a couple of hundred megabytes, compared with the many hundreds of gigabytes in a modern, multi‑articulation sample library. The down side, though, is that the maths required by physical modelling is much more demanding of CPUs than the comparatively simple job of playing lots of audio files at the right time.
Model vs Sample?
The difference between physically modelled and sampled virtual instruments is somewhat analogous to algorithmic versus convolution reverb. Just as a convolution processor can give you the exact qualities of a specific acoustic, so a sample library made using a 1685 Stradivarius recorded in Abbey Road Studio 2 with a Neumann M49 will sound exactly like... A 1685 Stradivarius recorded in Abbey Road Studio 2 with a Neumann M49. But that realism comes at the cost of flexibility. In the same way that an algorithmic reverb will let you adjust things like room size, diffusion, damping and so on, physically modelled instruments usually let you adjust the parameters of the mathematical model itself — which allows for changes as subtle as a slightly duller or more strident violin, perhaps some more bow noise, or adjustable sympathetic resonance of the unplayed strings... or as radical as making your violin sound like it’s 16 feet long and made out of concrete. That kind of adjustability is largely absent from sampled instruments.
Some developers have sought to combine the two methods, for a ‘best of both’ approach — notably Audio Modeling, whose SWAM series woodwind instruments employ both sampling and physical modelling techniques.
Signature Sounds
In much the same way that Gibson have the Slash collection of Les Pauls, and Fender offer the Jimi Hendrix Signature Strat, many software instrument makers enlist the help of big‑name artists and composers when building their instruments. For example, Spitfire’s Hans Zimmer Strings promises the bombastic sound of the legendary Hollywood composer; while UJAM were founded by the man himself, and their Symphonic Elements Drums was recorded using Zimmer’s own extensive drum collection.
Other sample libraries employ renowned musicians in their sample capture (Native Instruments have just released a second Alicia Keys collaboration, and harpist Justyna Hebda features in Steinberg’s new Navia Harp), while others sell themselves on the pedigree of their engineers — for example EastWest’s Quantum Leap series, which was recorded by Grammy‑winning engineer Professor Keith O’Johnson.
Front Ends
Most of this article has been about recording samples, but what about playing them back?
In the hardware days, a sampler would do both jobs, but in software it’s more common for ‘samplers’ to only concern themselves with the playback side of things. Probably the best known one is Native Instruments’ Kontakt, and there’s a huge number of companies making third‑party content for it. Other major platforms include IK Multimedia’s SampleTank 4, Steinberg’s HALion and UVI’s Workstation 3 — all powerful sampling engines that have attracted legions of third‑party developers.
Cinesamples’ CineBrass instrument uses a custom GUI over Native Instruments’ Kontakt sampler. Shown here is the mixer page.
Some companies, such as Spitfire, VSL and Orchestral Tools, have built their own sample playback engines (the Spitfire Player, Synchron Player and Sine Player, respectively). These perform much the same function, but allow the devs to use their own proprietary technology, and to customise the user experience more than using an off‑the‑peg sampler would allow.
These are all powerful pieces of software with all the features that both developers and users need to wring maximum realism from a sample set, so the choice of whether to buy one instrument or another will often depend on which ‘ecosystem’ you’ve already bought into.
Further Trickery
Extensive sampling will get you most of the way to a realistic‑sounding virtual instrument, but there are some clever tricks to be played with sample manipulation and MIDI data too. Your MIDI keyboard’s pitch and mod wheels can be put to good use, allowing for real‑time playing of pitch glides and vibrato. The former is straightforward pitch‑shifting of the sample, while the latter is usually a simple LFO modulation of the sample playback speed, with the mod wheel controlling either the modulation amount, or speed, or both. While some sampled instruments will record vibrato notes as an additional articulation, the mod wheel/LFO approach gives you continuous adjustability, and allows for things like tempo‑sync’ing to your DAW project.
Another neat trick is the use of ‘release samples’. These are additional samples that capture what happens when a musician stops playing a note — for example, a guitarist might intuitively rest their finger on the strings to stop them ringing, resulting in a quiet but audible ‘thud’. Spectrasonics’ bass instrument Trillian includes such release noises, triggered by the Note Off MIDI messages that are generated when you release a key, and most virtual pianos do the same thing for the ‘hammer return’ sound when a key is released.
MIDI aftertouch can also be pressed into service, perhaps for controlling tremolo depth or speed, as an alternative means of controlling vibrato, or for adjusting playback volume so that you can play crescendos in real time.
Legato — a playing style whereby a run of notes is played ‘smoothly’, ie. without any gaps between them — is another area that’s ripe for playback tweakery. If we play our sampled violin in legato style on a MIDI keyboard, we’ll hear a bow attack for each note, which is not how a violinist would play legato. So what we can do is program our software sampler so that, if one note is played before the preceding note ends, it cuts off the beginning of the second note and crossfades into it from the first, giving us a smooth transition and approximating the sound of a violinist playing two notes with one smooth bow movement.
It should be noted that some of the more fully-featured software instruments now go the whole hog and record actual legato note transitions, so that when you jump from a C to a G, say, you’ll hear the whole C sample, then a brief sample of a violin jumping from C to G, and then the sustain portion of the regular G sample. Vienna Symphonic Library pioneered this ‘true legato’ approach back in 2002, and it now appears in a number of other orchestral instruments.
Similar trickery can also be used to achieve realistic trills, portamentos and glissandos.
Grooves & Players
A meticulously multisampled virtual instrument is all well and good if you have the keyboard skills to play it well, but for those who don’t, many software instruments now include a ‘performance engine’ to help you get the best out of them. For example, Native Instruments’ Action Strings 2 includes a number of preset phrases that you can trigger with just a single key strike, and even features an editor so that you can write your own.
Toontrack’s EZkeys 2 includes a virtual Bandmate, to help you generate suitable keyboard parts for your projects.
Virtual drum instruments likewise often ship with a collection of ‘grooves’: MIDI performances played by real drummers on electronic kits, which can be used to trigger the underlying samples. Spitfire Audio’s The Grange, for example, features real drum performances from the likes of Chad Smith, Roger Taylor and Andy Gangadeen, so you can incorporate those drummers’ performances in your productions while still having the flexibility to, say, swap out drum kit parts or tweak the MIDI notes to fit.
Toontrack have gone even further, with the Bandmate feature found in their EZdrummer 3 and EZkeys 2 instruments. Drag and drop an audio or MIDI file from your DAW into the Bandmate window of any of those instruments, and they’ll automatically generate a suitable accompaniment. Even better, you can share the auto‑generated drum part with with EZkeys 2, which will then use the drum groove as the basis for its own playing.
Apple’s Logic Pro DAW now also includes some ‘virtual session musicians’, which you can read about in this month’s Logic workshop.