[Dixielandjazz] What will they think of next?

[Stephen G Barbone]

For Pianist, Software Is Replacing Sonatas

By JAMES BARRON - NY TIMES - August 21, 2010


The pianist Robert Taub was puttering around the house one afternoon  
in 2004 while his teen-age daughter was practicing for a violin lesson  
— a Schubert sonatina in A minor. His assessment of her playing was  
diplomatic: “She needed to be reminded about notes and rhythms.”

What followed was a brainstorm that explains why Mr. Taub — who made  
his reputation playing two distinctly different B’s, Beethoven and  
Milton Babbitt — has put his performing on hold, and why “software  
entrepreneur” now tops his résumé.

“I thought, wouldn’t it be wonderful if she could take a photograph of  
her page of music and hear it instantaneously,” he recalled. “She’d  
know what the right notes are, and what the right rhythms are, and she  
could imitate what she heard.”

Soon he was dreaming of a device — or maybe just software running on a  
computer — that could do everything he had learned to do in music  
theory class: read and play a printed musical score, and listen to a  
passage of music and transcribe it, down to the key signature, the  
tempo and the time signature. He said that a quick check showed that  
nothing then on the market could do all that.

So Mr. Taub started exploring the world of machine-learning  
technology. Before long he had organized a startup company and was  
spending more time on conference calls than at the piano.

“This is what I eat, dream, sleep,” he said. Now 54, he last performed  
in the summer of 2008 in Aspen, Colo.

The company, MuseAmi, now has half a dozen software engineers and  
about as many patents applied for. The company’s name (pronounced  
myooze-ah-MEE) is a play on words: “You become the music: muse am I,”  
he said. “But it’s also a musical friend.”

In recent years the world of software for musicians has exploded, with  
performers using their smartphones as metronomes or tuning devices.  
Mr. Taub has plunged in with MuseAmi’s first app, Improvox, which made  
its debut on iTunes last month, at $7.99 a download, and has sold a  
few thousand so far, Mr. Taub said. It is not the photo-and-play app  
he had dreamed of; he said he expected to introduce that by the end of  
the year.

Instead, Improvox promises to do much of what a well-equipped  
commercial recording studio can do, correcting notes that are sharp or  
flat. That is a task that was pioneered by Auto-Tune software, a plug- 
in for audio software used in recording studios. Being able to fix  
pitch or intonation problems is particularly helpful after a recording  
session, when the engineer and the producer discover that their big- 
name singer was slightly off.

MuseAmi uses different technology to correct pitch, and does so in  
real time. (Auto-Tune works on recordings that have already been  
made.) It can also generate harmonies, chosen by icons on its touch- 
pad screen. The icon that looks like Johann Sebastian Bach gives a  
singer a Baroque backup. The icon that looks like a barber pole adds  
three other voices for a barbershop quartet sound, with dominant  
chords. And there are other icons for other effects, but as Mr. Taub  
explained, “you don’t need to know any music theory.”

But in developing Improvox Mr. Taub had to learn a lot about software.  
First he went on a manhunt for software developers. He asked around at  
the Institute for Advanced Study in Princeton, N.J., where he had been  
an artist in residence. The name that kept coming up was that of Yann  
LeCun, a computer science researcher who had worked for AT&T Bell  
Laboratories. He was known for work involving something called  
convolutional neural networks, which he had developed for software  
that banks use to scan and read checks.

Mr. Taub cold-called Mr. LeCun. They met in a New Jersey diner, and  
Mr. LeCun “sketched out on a napkin how we were going to do it,” Mr.  
Taub recalled.

Mr. LeCun said they focused on technologies that they believed they  
could develop fairly quickly. “Some of those things drew on my  
experience with machine learning,” Mr. LeCun said. But he said he also  
drew on his own experience with wind instruments: he plays the  
recorder, the oboe and the crumhorn.

Mr. Taub found financing chiefly from Bob Stockman, whose merchant  
banking group has primarily financed medical-device companies. They  
knew each other because their children had attended the same preschool  
and because Mr. Stockman had gone to some of Mr. Taub’s concerts.

Mr. Taub began hiring a team of software engineers and developing  
algorithms. The team members had a “Watson, come here” moment for the  
read-and-play function when they photographed the sheet music for the  
theme from “The Godfather,” and the software played it back. But the  
record-and-play function and the pitch correction came first, and the  
initial reviews from musicians are positive.

“It rips the mask off the studio tricks for your everyday music fan,”  
said Billy Mann, a record producer and songwriter who has worked with  
Pink, Ricky Martin and the Backstreet Boys, among others. “If someone  
wants to sing and they want to know how reverb or phasing or flanging  
works, they can play with it. More than anything, I think it’s fun for  
someone who’s an amateur and wants to mess around with some of the  
high-end professional capabilities that people have in big studios,  
and do it on their phone.”

Christianne Orto, associate dean and director of recording and  
distance learning at the Manhattan School of Music, said it was aimed  
at the “entertainment ‘prosumer’ market.”

“It doesn’t tell you you’re sharp or flat” the way educational  
software might, it just makes the correction, she said. “And when it  
harmonizes, it’s not giving the harmonic structure underneath. It is  
somewhat glorified karaoke, but it could be a wonderful portal, a  
great first step for giving someone some confidence in the possibility  
of musical accomplishment.”

Mr. Taub has his own sense of accomplishment. Recalling that it had  
taken him eight years to learn Beethoven’s daunting “Hammerklavier”  
Sonata, he said, “This started a little less than five years ago now,  
and we already have some market traction.”