Recursive Geek Love

Jo-Co, Merlin Mann, et al perform “Still Alive” on Rock Band.

Use Your Illusion

 

The Illusion of Music

New Scientist

23 February 2008

Daniel Levitin

 

IMAGINE that you stretch a pillowcase tightly across the opening of a bucket, and different people throw ping-pong balls at it from different distances. They can each throw as many balls as they like, and as often as they like. Your job is to figure out, just by looking at how the pillowcase moves up and down, how many people there are, who they are and whether they are walking towards you, away from you or standing still. This is essentially the problem your auditory system has to contend with when it uses the eardrum as the gateway to hearing.

 

Sound is transmitted through the air by molecules vibrating at certain frequencies. These bombard the eardrum, causing it to wiggle in and out depending on how hard they hit it (related to the volume, or amplitude, of the sound) and how fast they are vibrating (related to what we call pitch). But there is nothing in the molecules that tells the eardrum where they came from, or which ones are associated with which object. Voices may be mixed in with other voices, or the sounds of machines, wind and footsteps. Most of the time the input is incomplete or ambiguous. So how does the brain figure out, from this disorganised mixture of molecules beating against a membrane, what is out there in the world?

 

Most people assume that the world is just as they perceive it to be. Yet experiments have forced researchers, including myself, to confront the reality that this is not the case. What we actually hear is the end of a long chain of mental events that give rise to an impression - a mental image - of the physical world. Nowhere is this more striking than in the perceptual illusion in which our brain imposes structure and order on a sequence of sounds to create what we call music.

 

The chain of mental events begins with a process called feature extraction. The brain extracts basic, low-level features from the music, using specialised neural networks that decompose the signal into information about pitch, timbre, spatial location, loudness, reverberant environment, tone durations and the onset times for different notes (and for different components of tones). This bottom-up processing of basic elements occurs in the peripheral and phylogenetically older parts of our brains. Next comes a process called integration. Parts of the higher brain - mostly in the frontal cortex - receive the basic features from lower brain regions and work top-down to integrate them into a perceptual whole.

 

The brain faces three difficulties in feature extraction and integration. First, the information arriving at the sensory receptors is undifferentiated in terms of location, source and identity. Second, the information is ambiguous: different sounds can give rise to similar or identical patterns of activation on the eardrum. Third, the information is seldom complete. Parts of the sound may be masked by other sounds, or lost. The brain has to make a calculated guess about what is really out there. So, auditory perception is a process of inference. And when the sensory input is music, these inferences include several factors over and above the sounds themselves: what has come before in the piece of music we are hearing; what we remember will come next if the music is familiar; what we expect will come next if the genre or style is familiar; and any additional information we may have, such as a summary of the music that we have read, a sudden movement by a performer or a nudge by the person sitting next to us.

 

The brain thus constructs a representation of reality, based on both the component features of what we actually hear and our expectations of what we think we should be hearing. There are good evolutionary reasons for this - a perceptual system that can restore missing information can help us make quick decisions in threatening situations - but it is not without drawbacks. The top-down expectations can cause us to misperceive things by resetting some of the circuitry in the bottom-up processors. This is partly the neural basis for perceptual illusions such as the one demonstrated by cognitive psychologist Richard Warren from the University of Wisconsin. He recorded a sentence, “The bill was passed by both houses of the legislature”, cut out part of it from the recording tape and then replaced the missing piece with a burst of white noise (static) of the same duration. Nearly everyone who heard the altered recording reported that they heard both a sentence and static. Yet a large proportion of people couldn’t tell when the static occurred because the auditory system had filled in the missing speech information, so that the sentence seemed to be uninterrupted.

 

This filling-in phenomenon is not just a laboratory curiosity. Composers exploit the same principle, knowing that our perception of a melodic line will continue, even if part of it is obscured by other instruments. It also happens whenever we hear the lowest notes on the piano or double bass. We are not actually hearing 27.5 or 35 hertz, because those instruments are typically incapable of producing much energy at these ultra-low frequencies. Instead, our ears are filling in the information and giving us the illusion that the pitch is that low.

 

Most contemporary recordings contain another type of auditory illusion. Our brains use cues about the spectrum of the sound and the types of echoes to tell us about the auditory world around us, much as a mouse uses its whiskers to learn about the physical world around it.

 

Recording engineers have learned to mimic those cues to imbue recordings with a real-world, lifelike quality even when they are made in sterile recording studios. Artificial reverberation makes vocalists and lead guitars sound as if they are coming from the back of a concert hall, even when we are listening on headphones and the sound is an inch away from our ears. The same principles can also generate auditory tricks, such as making a guitar sound as if it is 10 feet wide and your ears are right where the soundhole should be.

 

Special effects

 

Recorded music allows us to experience other sensory impressions that we never actually have in the real world. Recording engineers and musicians create special effects that tickle our brains by stimulating neural circuits that evolved to discern important features of our auditory environment. For example, our brains can estimate the size of an enclosed space on the basis of the reverberation and echo present in the signal that hits our ears. Even though few of us understand the equations necessary to describe how one room differs from another, we can all tell whether we are standing in a small tiled bathroom, a medium-sized concert hall or a large church with high ceilings. And we can tell what size room the singer or speaker is in when we hear recordings of voices. Recording engineers exploit this ability to create what I call “hyper-realities”, playing with our perceptions of space in the auditory equivalent of the cinematographer’s trick of mounting a camera on the bumper of a speeding car.

 

“Recorded music allows us to experience sensory impressions that we never have in the real world”Another illusion involves timing. Our brains are exquisitely sensitive to timing information. We are able to localise objects in the world based on differences of only a few milliseconds between the time of arrival of a sound at one of our ears versus the other. Many of the special effects we love to hear in recorded music are based on this sensitivity. The sounds of jazz guitarist Pat Metheny or that of David Gilmour of Pink Floyd use multiple delays of the signal to give an otherworldly, haunting effect that triggers parts of our brains in ways that humans had never experienced before, simulating the sound of an enclosed cave with multiple echoes such as would never actually occur in the real world - the auditory equivalent of the barbershop mirrors that repeat infinitely.

 

Perhaps the ultimate illusion in music, however, is the illusion of structure and form. There is nothing in a sequence of notes themselves that creates the rich emotional associations we have with music, nothing about a scale, a chord or a chord sequence that intrinsically causes us to expect a resolution.

 

Our ability to make sense of music depends on experience and on neural structures that learn and can modify themselves with each new song or piece of music we hear, and with each new listen to music we are already familiar with. Our brains learn a kind of musical grammar that is specific to the music of our culture, just as we learn to speak the language of our culture. This becomes the basis for our understanding of music, and ultimately the basis for what we like in music, what music moves us, and how it moves us.

 

Top five musical illusions

 

In piano works such as Chopin’s Fantasy-Impromptu in C-sharp Minor, opus 66, or Sinding’s The Rustle of Spring, the notes go by so quickly that an illusory melody emerges. When the notes are close enough together in time, the melody “pops out” because the perceptual system binds them together, giving an emergent impression of tunefulness. Play the tune slowly and this disappears.

 

In a Sardinian style of a cappella singing studied by Bernard Lortat-Jacob at the Musée de l’Homme in Paris, a fifth female voice called the quintina (literally “fifth one” in Sardinian) emerges from four male voices when their harmony and timbres are just right. The voice is said to be that of the Virgin Mary coming to reward the singers for their piety, but in fact it is simply a misperception of the chord and its harmonics.

 

The Eagles’ song, One of These Nights, opens with a pattern played by bass and guitar that sounds like one instrument. The bass plays a single note, and the guitar adds a glissando, but the perceptual effect is of the bass sliding due to the gestalt principle of good continuation, which binds together two objects when the trajectory of one implies the continued trajectory of another.

 

Jazz pianist George Shearing created a new timbral effect by having a guitar (or in some cases, vibraphone) precisely match what he was playing on the piano. Listeners come away wondering, “What is that new instrument?”, when in reality it is two separate instruments whose sounds have perceptually fused.

 

In Lady Madonna, the Beatles sing into their cupped hands during an instrumental break and we could swear that there are saxophones playing. This perception is based on the unusual timbre they achieve, coupled with our expectation that saxophones should be playing in a song of this genre. (This is not to be confused with the actual saxophone solo that occurs in the song.)

 

They just don’t get it

 

History is littered with figures noted for their hopeless unmusicality. Ulysses S. Grant, the 18th president of the United States, had a tin ear and found music profoundly irritating; Che Guevara famously couldn’t distinguish one piece of music from another. Once, such people would have been described as “tone deaf”; today they are seen as much more interesting than that.

 

In the past few years it has become clear that the inability to hold a tune can sometimes be caused by a neurological condition called congenital amusia, which completely robs people of what is normally an instinctive and spontaneous appreciation of music. No wonder the condition has become a major research topic in the bid to understand the mysteries of how the brain handles music.

 

The first case report of “note deafness” appeared in 1878, and the literature is full of anecdotal accounts of people with a lifelong failure of music perception. It wasn’t until 2002, however, that the first proper study of congenital amusia was published. A team led by Isabelle Peretz of the University of Montreal in Canada reported the case of Monica, a woman in her early 40s who had always lacked even the most basic of musical abilities (Neuron, vol 33, p 185).

 

Peretz concluded that Monica’s problem was a failure to detect pitch changes in melodies. Played two notes in sequence, she could rarely tell whether the second was higher or lower than the first or had the same pitch. Most people can easily distinguish small differences in pitch - half a semitone, say - but for amusics, even a leap of an octave, equivalent to the first two notes of Somewhere Over The Rainbow, can be barely perceptible. Tones and semitones are the building blocks of melody, so no wonder amusics find music monotonous in more than one sense of the word.

 

Peretz and others have since documented dozens of similar cases. These people all have normal hearing, intelligence and memory, but absolutely no grasp of melody. For them, one tune sounds very much like another, familiar songs are unrecognisable without lyrics, and dissonant chords that cause most of us to wince elicit no response. Amusics cannot sing, though they often don’t recognise this. The condition is unusual but not particularly rare - the accepted figure is 4 per cent of the population - and it runs in families.

 

So what causes congenital amusia? According to Peretz, the best explanation is that the human brain is equipped with a specialised “module” for processing melody, which occasionally fails to develop properly. That would explain why amusia appears to affect musical perception alone. If correct, music, like language, is an innate human adaptation that was hard-wired into our brains by evolution.

 

AUDITORY CHEESECAKE?

 

Not everyone agrees with this view, however. Steven Pinker once famously described music as “auditory cheesecake” - pleasurable but with no adaptive function. What’s more, there is some evidence that amusia is not a purely musical deficit but is linked to problems with language or spatial processing. So perhaps amusia (and by extension, normal music perception) is rooted in the brain circuits that handle intonation in language, or that look after the concepts of “highness” and “lowness” central to our mental representations of melody.

 

Peretz’s group and others are now scanning the brains of amusics in search of anatomical anomalies that might lead them to the underlying problem. So far they have found some minor differences in the thickness of white matter in a brain area called the right inferior frontal gyrus - a region linked with musical pitch perception and melodic memory (Brain, vol 129, p 2562). They are also searching for the genes that make amusia heritable, in the hope of gaining new insight into abnormal brain development in amusia (The American Journal of Human Genetics, vol 81, p 582).

 

Another key question is whether congenital amusia is one condition or several. Some amusics like listening to music because they enjoy the rhythms, but Peretz’s team has found that around half their subjects have a problem with rhythm perception. This suggests there may be a related condition that wipes out timing as well as melody. There’s also the problem of “clatterers” - amusics to whom music sounds like a drainpipe being hit with a wrench. “Only a very few amusics hear clattering,” says Peretz. “For the majority, music is just confusing.” That has led some researchers to propose a separate disorder of music perception called dystimbria, which prevents people from perceiving musical “colour”, or timbre.

 

Whether amusia is one condition or many, the hope is that understanding it better will benefit those unfortunates excluded from the profound pleasure of music. Peretz thinks that with early intervention it might be possible to tap into the natural plasticity of the brain and stem some of the damage. “There’s no chance of helping adults,” she says. “We’ve tried. But with children, maybe.”

 

Five great auditory illusions 

24 February 2008

Mike Marshall

 

As part of our special issue on music, Daniel Levitin has written The Music Illusion, which looks at auditory illusions and how they can help us understand the workings of the human brain. Here we have compiled five of the most striking auditory illusions discovered so far.

 

We had a big pool to choose from, from the mysterious quintina (fifth voice) heard in some types of throat-singing, to the saxophone solo that isn’t on Lady Madonna (it’s actually the Beatles singing into their cupped hands) and the soaring guitar sound of Pink Floyd’s Dave Gilmour. Listen to our top 5 below, and read our explanations of the effects involved.

 

1 Barber’s shop illusion (Listen with headphones)

 

This is a demonstration of the stereo effect. Listening to it, you feel as though you are in a barber’s chair, with the barber moving around you, clipping away at your hair. As the barber “moves” to your right, the volume increases slightly in the right channel and decreases in the left. Similarly, increases in the volume of sound from the clippers give the impression that he is bringing them closer and closer to each ear. The illusion demonstrates our ability to locate sounds in space; by comparing the inputs to the two ears, we can work out where a sound is coming from.

 

2 Phantom words (Listen through stereo separated loudspeakers, best placed some distance apart)

 

This illusion was first demonstrated by Diana Deutsch at the University of California, San Diego. Building on the stereo effect described above, the recording features overlapping sequences of repeating words or phrases, located in different regions of stereo space. As you listen to it, you’ll start to pick out specific phrases. However, none of the phrases are really there. Your brain is constructing them, in a bid to make sense of a meaningless noise. Indeed, you may find that the phrases you hear are related to what’s on your mind – for example, people who are dieting often hear phrases associated with food.

 

3 Temporal induction of speech

 

Much of human perception is the result of the brain filling in gaps in the data from our senses. This means that if a part of an audio recording is missing, our brains will often work out what should have been there. In this recording by Richard Warren from the University of Wisconsin in Milwaukee, a spoken sentence is interrupted by a cough. One of the phonemes has actually been completely removed by the cough. But not only do most people hear the complete sentence, they generally find it very difficult to work out which phoneme has been deleted. If the phoneme is replaced by a period of silence, rather than a cough, the deletion is very obvious.

 

4 Scale illusion (Listen through stereo headphones, or stereo separated loudspeakers, best placed some distance apart)

 

Another effect first demonstrated by Diana Deutsch, this is an example of our brains “grouping” similar notes together. Two major scales are played: one ascending, one descending. However, the notes alternate from ear to ear – for instance, the right ear hears the first note of one scale, and then the second note of the other (see diagram, top right).

 

There are several ways in which people perceive these sounds, but the most common is to group the high and low notes together. Rather than hearing the two scales, people hear a descending and re-ascending melody in one ear, and an ascending and descending melody in the other. In other words, the brain reassigns some of the notes to a different ear in order to make a coherent melody. Right-handed people tend to hear the high melody in the right ear, and the low one in the left, while left-handers show a more diverse response.

 

5 Phantom melodies

 

Some pieces of music consist of high-speed arpeggios or other repeating patterns, which change only subtly. If they’re played fast enough, the brain picks up on the occasional notes that change, and links them together to form a melody. The melody disappears if the piece is played slowly.

 

Compare these recordings of Christian Sinding’s Frühlingsrauschen (”Rustle of Spring”). At the higher speed, the changing notes linger in your perception long enough to be linked into a melody, but at the lower speeds they’re too widely separated. (original recording: www.classicalmidi.co.uk / Slow recording courtesy of Karle-Philip Zamor)

 

 

 

More Kudos for RB Avatars

This is a pretty good post from Feminsta about the avatar design in Rock Band.

The Recent Update

Here’s the official list of features and fixes in the software patch now available for download.  This comes directly from Harmonix (www.rockband.com):

• (New!) Music Store: Rock Band now has a new feature that lets you preview and purchase songs from within the game! Accessible from the main menu, the Music Store lets you view, purchase and sort all available music based on various categories, view album art, listen to song previews, and check out extended information about the song like difficulty for each instrument.
• Revised Fan Caps: To allow Easy, Medium, and Hard players to progress further in the Band World Tour, we’ve increased the number of fans that you can earn before hitting the cap. Easy players can now travel across the Atlantic, Medium players have a wider range of venues they can play at, and Hard players on the Xbox 360 are now able to unlock the “One Million Fans” achievement.
• More diverse songs in Band World Tour: If you’ve ever cursed about having to play “Say it Ain’t So” or a Metallica track multiple times in the same hour, then you’ll be glad to know that we’ve tracked down and fixed the issues that triggered these very repetitive moments in Band World Tour.
• Improved phoneme recognition: We’ve improved the detection and scoring for phoneme recognition. If you had trouble on songs like “Timmy and the Lords of the Underworld” or “Blitzkrieg Bop”, you should have an easier time beating these songs now.
• Microphone Latency on PlayStation 3 improved: Our awesome team of audio programmers has found some optimizations that reduce microphone latency on the PlayStation 3 in certain situations.
• Xbox 360 Band Logos are now visible through Xbox LIVE: An issue with parental controls stopped band logos from being visible over Xbox LIVE. With this update, you can now view all of the awesome band logos that people around the world have created!
• Faster loading of downloadable content: Not only is the loading speed faster, but this information is now cached so that this loading time is a “once only” wait rather than something that happens every time you turn on the game.

I LOVE the new Music Store interface - so much better than the resident XBox one.  And I appreciate the faster load times and better phoneme recognition.  But the new patch frelled my calibration something fierce, and that was a total PITA to recalibrate.

I notice there are new clothes for sale - anyone know if that’s part of the patch or just something Harmonix pushes out occasionally?  Also, and I may be losing it, but I could swear that some of the songs have been altered.  I don’t know if, when you hit a certain level of stars/fans, the notes you have to play are different even on the same level, or if Harmonix is modifying the gameplay so the songs are level-correct.

I hate Metallica

After that friggin’ 30-minute-long Metallica song came up twice in two hours on random setlists, we did some selective surgery and deleted the Metallica pack.  I don’t like listening to Metallica, I don’t like playing Metallica, and I don’t like spending 10 minutes playing music that I don’t like to hear and that isn’t fun to play.  We now have (Unknown Song) on our Top 5 list under our band profile, but that’s Oh Kay.

Take a Cue from Wii

The Wii lets you load your gamer data (avatar, purchased items, stats, etc) onto the Wiimote, so you can take your Wiimote to a friend’s house and play there with your own avatar, rather than having to start from scratch.

Harmonix should do this for Rock Band - let you load your data onto your controller, so if each member of your band has an Xbox, you don’t have to have different bands at different houses.  We just had to start a new band for this very reason.  Now, it’s called “Extreme Rendition,” which is cool, but it’s sort of a pain to have to start all over.

Battle of the Rock Bands

We had a Rock Band Party on Saturday.  Rather than just having people over and randomly picking songs off of Band Quickplay, we got all organized and stuff.  Here’s what we did:

- Cool invitation, like a flyer for a skanky urban club

- Asked people to come as their favorite rock star

- Had two setups.  One was on the home theater in the nerd hole, the other was in the dining room on a 26″ monitor.  Both had two guitars, courtesy of GH3.

- Randomly sorted guests into bands of 3.  This way, they could select around their least good instrument.  Bands who could handle all four could determine whether they wanted the points from vocals or the overdrive from bass or whatever.

- Every band played the same 3-song set.  Each player could choose their own difficulty level.  Scoring was based on number of stars (in case of a tie, the actual points determined the winner).  We selected easy songs for the first set, and subsequent sets got harder and harder songs.  At the end of each set the band with the lowest score was eliminated, until we had two bands in the final set.

- Between sets we ate lunch and dinner, and had two sideline competitions.  1) Guitar Hero, where anyone could challenge me to a Tug of War on Hard.  Songs were picked randomly from the tough songs with serious guitar solos.  Whoever got the best score won the prize.  2) Beat the Wanker, where anyone could challenge the drummer (whose name is Wanker…I recommend naming an avatar The Meat for this one).  The “Random Song” feature of the game picked the song, and the challenger could play on any level.  Wanker played on no less than hard.  Number of stars won; ties went to Wanker.  It was not fair, but he almost bombed out on a Queens of the Stone Age song.

- We made t-shirts with a cool “God of Rock” design on the back and something on the front to designate what the person won for.  In addition to the above, we had a shirt for whoever acted most like a rock star, and whoever managed to get the most 100% scores.  The winning band also got silly little trophies from the party store.

After the contest, most people hung out for a another few hours, and we swapped people in and out on whatever instrument they wanted until the wee hours.  It was good fun.

The next day, 3 of the guests went to buy Rock Band and an Xbox.  One of them actually did plunk down the moolah, so that’s two whole kits we’ve helped sell by hosting RB days.  We should buy stock in EA/Harmonix.

Every Culture Has One

RockStar Sharon sent me this vid (Bagpipe Hero) and the concept that maybe the next game should feature a bagpipe controller. Talk about world music!

Not Really Rock Band

Blender.com has a fun article about the Top 20 Biggest Record Company Screwups.

excerpt:

Tomorrow Never Knows
#2 Decca Records A&R exec tells Fab Four, “No, thanks”
Dick Rowe was not the only record-label executive who passed on the Beatles in the early ’60s, but he was the only one who brushed off their manager, Brian Epstein, with the astute prediction that: “Groups with guitars are on their way out.” Epstein begged Rowe to reconsider, so Rowe hopped a train to Liverpool to check out the band live. When he arrived at the Cavern, he found a mob of kids trying to force their way into the club in the pouring rain. Annoyed, he smoked a cigarette, went home and signed Brian Poole and the Tremeloes instead.
Unintended consequence The Monkees

I Want My MTV!

The wireless drum controller for Rock Band seems to be backordered EVERYWHERE! How can I really shred when I have to watch out for the damn drum cable?

The wireless guitar will be released April 1. Pre-order, yo.