Interspeech 2014 Recap

This year's Interspeech was in Singapore.    Singapore is, in some ways, a very easy venue to travel to.  It's a modern, cosmopolitan city.  They speak English.  It's tropical, but you're never more than a hundred meters from air conditioning.   In other ways, it's so far.  Over 20 hours each way.  I like airplanes.  They're as magical as any technology we've got.  But 20 hours is a long time to sit still.  Think about how many steps you take in 20 hours.  How many different faces you see.  Then reduce that to about 200 steps, and 10 people.

"Because we are all poets or babies in the middle of the night, struggling with being." - Martin Amis "London Fields"

Interspeech 2014 was a well run conference.  The quality of papers was generally quite high. The venue easily handled the size of the event and the wifi was steady.  It was difficult to find enough food at the Welcome Reception, but easy to find enough beer.  The banquet was flawed -- segregating vegetarians is pretty rude -- but they all are, and at least there was enough food to go around, and everyone ate promptly.  And of course, there was Mambo and Jambo.  I'm not going to go into it here, but find someone who attended the opening ceremony and ask them to describe it.  Then don't believe them, and ask someone else to do the same.  It was "odd" at best.  But be sure, I'll be attending the Dresden opening ceremony to see how they one-up it.

Deep Learning
A few years ago, DNNs invaded speech conferences.  Deep Learning is still a significant buzz word, and a hot topic.  But for, the better of everyone involved, the intensity has cooled.  Now the interest in DNNs seems to have shifted into 1) understanding how they work, and how to best train them to a task and 2) Long Short-Term Memory units to model sequential data.  The latter really broke out at this years conference.  There were a number of papers finding that them to be an effective alternative to traditional recurrent nets trained with back-propagation through time.

BABEL
I've been involved with the IARPA-BABEL program, so my view is pretty biased on this front, but I felt like the presence of BABEL in this year's Interspeech was particularly large.  The program's central task is performing keyword search on low-resource languages. It has an aggressive evaluation schedule with an increased number of *new* languages involved each year.  There were at least two sessions devoted to keyword search, and papers evaluated by either BABEL-proper or the NIST OpenKWS challenge seemed to be all over the conference.  (Searching the paper index suggests that there are between 30 and 40 BABEL papers, and another 10 or so OpenKWS papers.)  It seems clear that this program has had a large impact on ASR and KWS research.  2014 was certainly the high-water mark here, as the program shrunk by 50% last year, but it's worth noting its effect on the field.

Some standout papers

I don't mean to suggest that these are "the best" papers, but they're ones that caught my eye for one reason or another.

Acoustic Modeling with Deep Neural Networks Using Raw Time Signal for LVCSR by Zoltán Tüske, Pavel Golik, Ralf Schlüter and Hermann Ney.   Part of the promise of Deep Learning is the ability to "learn feature representations" directly from data.  This is frequently touted as a description of what is happening in the first hidden layer of a deep net.  So, the logic goes, do we need MFCC/PLP/etc. features, or can we do speech recognition directly on a raw acoustic signal?  This is the first paper I'm aware of to affirmatively show that "yes, yes we can".  It requires a good amount of training data, and rectified linear (ReLU) neurons work better for this, but 1) it works competitively with traditional features and 2) many of the first hidden layer neurons can be shown to be learning a filterbank.  Very cool.

Backoff Inspired Features for Maximum Entropy Language Models by Fadi Biadsy, Keith Hall, Pedro Moreno and Brian Roark.  In n-gram language modeling, when a sequence of words A, B, C have never been observed, its n-gram probability P(C|A,B) can be approximated by the probability P(C|B).  But to be sensible about it, you've got to apply a backoff penalty.  Discriminative language models can seamlessly incorporate backoff features, F(A,B,C), F(B,C), etc. and learn appropriate weights.  The key insight in this paper is that when a discriminative model uses these backoff estimates, it incurs no penalty.  It's essentially overestimating the probability of uncommon n-grams in the context of common (n-k)-grams.  This paper seeks to fix this, and does.

Additional favorites some from my students:

• Word Embeddings for Speech Recognition by Samy Bengio and Georg Heigold.  Far and away the most popular poster at the conference.  This is high on my list to read closely.  It promises to learn a euclidean space into which word decoding can happen, so that words that sound similar are closer in space.  

• The Obligatory Contour Principle in African and European Varieties of French by Mathieu Avanzi, Guri Bordal and Gélase Nimbona.  An investigation of prosodic differences in dialects of French.  Very consistent with one of my student's dissertation projects.

• Improving Spoken Document Retrieval by Unsupervised Language Model Adaptation Using Utterance-Based Web Search by Robert Herms, Marc Ritter, Thomas Wilhelm-Stein, Maximilian Eibl.  A clever way of handling OOVs in spoken document retrieval.  Essentially the idea is if I recognize W[i-2], W[i-1], UNK, W[i+1], W[i+2], go do a web search for the context, find some matching documents, and augment the language model with them, then re-decode.   Kind of like distant supervision for spoken document IR.

• Learning Small-Size DNN with Output-Distribution-Based Criteria by Jinyu Li, Rui Zhao, Jui-Ting Huang, Yifan Gong.  How do you effectively train a small DNN without ruining its performance?  This paper out of MSR suggests training a large DNN, then using its output distribution to train the small one.  It'll take me some closer reading to fully understand why this works, but I'm intrigued.

• Canonical Correlation Analysis and Local Fisher Discriminant Analysis based Multi-View Acoustic Feature Reduction for Physical Load Prediction by Heysem Kaya, Tuğçe Özkaptan, Albert Ali Salah, Sadık Fikret Gürgen. We tried a number of dimensionality reduction approaches for this year's Paralinguistic Challenge, but didn't get particularly good results.  These guys did using CCA and LFDA.  Looking forward to reading this one as a more general feature reduction approach when the number of features are larger than the number of training instances.

Interspeech 2012 Recap

Portland proved to be a great venue for this year's Interspeech.  (Though people who attended ACL 2011 probably already could have guessed that.)

Setting up three simultaneous poster sessions in the parking garage may not sound like the mark of a good conference, but it was perfect.  There was loads of space between each presenter.  It allowed for all three sessions to be in the same place.  And the folks at the Hilton did a great job of making it fairly unrecognizable as a parking lot.  (In fact, Alejna Brugos didn't realize it until they were removing the carpets and "walls" on Thursday afternoon.)

Deep Neural Networks.
For "trends", there's really nothing hotter right now than Deep Neural Networks or Deep Belief Nets.  This isn't an area that I do research in, but the story goes more or less this.  Neural Networks with more than a few hidden layers don't train very well with back-propagation. Geoff Hinton and his group figured out how to overcome this limitation not too long ago.  (I think this 2006 paper explains it, but I can't be 100% sure.) Then at ASRU 2012 and ICASSP 2011 and 2012, the folks at Microsoft showed that you can use Deep Neural Networks to generate *very* useful front end features.  (Tara Sainath has a nice recap of ICASSP 2012 here.) Now, everyone wants a piece.

The field has expanded from Microsoft to include IBM and Stanford/Berkeley/Google and RWTH Aachen.  Joining them with posters on Deep Neural Nets for ASR are Tsinghua, CMU, Karlsruhe, NTT, INESC-ID, UWashington, and Georgia Tech.  At this point, there's no way to deny that this approach is receiving significant research attention.  The results seem to be holding up.  If only they didn't take so long to train...

Prominence Special Session.
I was particularly looking forward to the Special Session on Prominence.  On balance I was happy about the session.  It attracted work and discussion of prosody in a way that can sometimes feel diffuse and unfocused at a large conference like Interspeech.

I found this session to be surprising in a few ways.

It's been my understanding that "prominence" was used as a catch-all term to cover diverse prosodic phenomena including stress, emphasis, and pitch accenting.  The first surprising element of this Session came in a review of the paper I submitted to it.  The paper is on the use of automatically predicted pitch accents and intonational phrase boundaries to improve pronunciation modeling.  The review, while generally positive, found the paper to not be appropriate for a prominence session because it explored the use of "pitch accents" rather than "prominence".  I still haven't gotten a good explanation of the difference, and the reviews are blind.

A second surprise is that there seems to be a movement away from a phonological theory of prosody. Mark Hasegawa-Johnson and Jennifer Cole have been doing work over the last few years investigating how naive listeners perceive prominence.  They've consistently found that listeners respond to different qualities sometimes at different thresholds when assessing prominence.  I've found this line of research to be interesting and generally informative, but not a clear indictment of the theory that there perceptual and productive prosodic categories exist.    The panel (which I was a part of) on balance seemed comfortable with the idea that prominence is a continuous rather than categorical phenomenon.  This view was most directly expressed Denis Arnold who said approximately: focus can be categorical, stress can be categorical, while prominence is still continuous. I didn't understand this statement then, and still don't.  But again, this may be due to a different definition of prominence than I use.

The last surprise comes from finding out that there is a direction of pursuing language universals in prominence and prosody more broadly. Petra Wagner and Fabio Tamburini (the session organizers) are planning a workshop to investigate this.  In my experience, while the dimensions of prosodic variation may be used in multiple languages and some of these (e.g. increased intensity or duration) may be used to indicate prominence in all languages, it is extremely unlikely that either the communicative impacts of prosodic variation or its realization and perception are language-universal.   From that perspective, I'm not quite clear about what this line of research hopes to accomplish, but I'm curious about where it ends up.

Dynamic Decoding.
It appears that every year, I find myself sitting in on an oral session on a topic that I know very little about.  Last year it was the language identification session.  This year it was Dynamic Decoding.  I was most intrigued by this because I hadn't heard the term before.  When I asked someone what it was, they said "I don't know, Viterbi?".

I'm not quite sure this is a good enough distillation of the topic, but the papers in this session were about how to make on-the-fly (or post-training) modifications to language or pronunciation models.  This is a cool idea with clear practical importance -- how do you add words to a recognizer on a mobile device and have this appropriately incorporated into the LM and pronunciation model?  These two papers have some interesting WFST based approaches on this task.  I'll be curious to see learn more about this.  Also, if anyone has a more precise definition of this research area, I'd love to hear it.

Finally, some comments on two of the keynotes.

There were four keynotes at this year's Interspeech, two were about interesting inter/multi-disciplinary questions about how speech processing intersects with music and animal vocalization, respectively.

Chin-Hui Lee: An Information-Extraction Approach to Speech Analysis and Processing
A third was delivered by this years ISCA medalist, Chin-Hui Lee.  Prof. Lee's most famous accomplishment is MAP adaptation in acoustic modeling.  This is a researcher who spent a career treating speech recognition as a pattern matching problem.  This is a view embodied by the Fred Jelenik quote: "Every time I fire a linguist, the performance of the speech recognizer goes up".  What struck me, is that despite this view, in a talk summarizing a successful career, Prof. Lee presented a view of speech recognition that says that linguistic knowledge and speech science should be incorporated into the task.  This is an alternate perspective that has been investigated by a lot of talented researchers, including Hynek Hermansky, Jennifer Cole, Mark Hasegawa-Johnson, Alex Waibel, Hermann Ney (via speech-to-speech translation), Mari Ostendorf, Elizabeth Shriberg, Andreas Stolcke, Rene Beutler, Karen Livescu and many more (my apologies to anyone I missed).

I was struck by the evolution of perspective from someone who represents the statistical pattern matching approach to recognizing the potential importance of linguistic knowledge.

However, this talk was not so well received by some members of the audience for fairly obvious reasons.  Firstly, it over-played the importance of Prof. Lee's own contributions.  In a slide on "My contributions", virtually all major improvements to ASR over the last 20 years were mentioned including most styles of adaptation (including MAP), and virtually all major forms of discriminative training.  Secondly, it failed to recognize that the linguistic inspired approach that he was advocating for the future had been extensively researched by other talented peers.

On balance, I found it a compelling message.  In principle, it understandably rubbed some people the wrong way.

Michael Riley: Weighted Transducers in Speech and Language Processing
I should preface my comments about Michael Riley's keynote by saying that we worked together while I was interning at Google.  I'm a fan.  Michael has the rare quality of being the smartest guy in the room without letting anyone know until its genuinely useful.

The best part of this keynote was the history of the Weighted Finite State Transducer.  This was a great story that takes place largely at Bell Labs in the 90s and features Fernando Pereira, Mehryar Mohri and, naturally, Michael Riley.  This section was appropriately personal, while presenting this relevant recent history.  The WFST is so ubiquitous in speech and NLP applications that it's easy to forget that it's has a human context.

Much of the rest of the keynote felt like a 3 hour tutorial compressed into 40 minutes.  This involved showing algorithms, and example WFSTs and describing all of the things that they can be used for.  While a successful demonstration of the breadth of application, it was presented at such a pace that it was difficult to get anything out of it, if you didn't know it already.  I'd point the interested to the references found on the OpenFST page for more thorough tutorials that can be digested at your own pace.

Interspeech 2012 was successful and fun.  Portland and the Hilton (and it's solid wifi) were excellent hosts.  There was good work and as ever more than I could see.  If you have great or favorite papers that I missed, please let me know!

Interspeech 2011 Recap

Interspeech 2011 was held in Florence Italy about a week ago, August 28-August 31.  A vacation in Italy was too good to pass up on, so The Lady joined me, and we stayed until Labor Day.

I ended up sending a bulk of work to Interspeech, so spent more time than usual in sessions that I was presenting in rather than seeing a lot of papers.

Two interesting themes stood out to me this year.  Not for nothing, but these represent some novel ideas about speech science through understanding dialog and the speech engineering.

Entrainment
Julia Hirschberg, my former advisor, received the ISCA medal for her years of work in speech.  Her talk was on current work with Agustín Gravano and Ani Nenkova on entrainment.  Entrainment is the phenomenon by which when people are speaking to each other, their speech becomes more similar.  This can be realized in terms of the words that are used to describe a concept, as well as speaking rate, pitch, intensity.  How this happens isn't totally understood, and measures of entrainment are still being developed.  This research theme is still in its early phases, but I haven't seen an idea spread around a conference as quickly or as thoroughly as this did.  There were questions and discussions all over the place (like Tom Mitchell's keynote about fMRI data and word meaning) about this phenomenon.  The more engineering folks weren't as compelled by the utility of this in helping speech processing, but within the speech perception and production communities, and specifically the dialog and IVR folks, it was all the rage.  It'll be something to see how this develops.


I-vectors
I-vectors are a topic that I need to spend some time learning.  I hadn't heard of it before this conference, where there were no less than a dozen papers that used this approach.  Essentially the idea is this:  The location of mixture components in a GMM model are composed of (in at least one form) a UBM, a channel component, and a "interesting" component.  This "interesting" component can be the contribution of a particular speaker, or a language/dialect, or anything else you're trying to model.  Joint Factor Analysis is used to decompose the observation into these components in an unsupervised fashion.  It's in this part where my understanding of the math is still limited.  The crux is that the "interesting" component can be represented by a \[Dx\] transformation, where the dimensionality of x can be set by the user.  In comparison to a supervector representation, where the dimensionality of the supervector is constrained to be equal to the number of parameters (or means) of the GMM, i-vectors can be significantly smaller leading to better estimation and smaller models.  I'll be reading this tutorial by Howard Lei over the next few weeks to get up to speed.


There were few specific papers that stood out to me this conference.  I'm intrigued by Functional Data Analysis as a way to model continuous time-value observations. Michele Gubian gave a tutorial on this that I sadly missed, and included it in at least one paper, Predicting Taiwan Mandarin tone shapes from their duration by Chierh Chung and Michele Gubian.  This paper wasn't totally convincing in the utility of the technique, but there may be more appropriate applications.

It was a satisfying and inspiring conference, to be sure.  I think I was more interested in talking to people than in papers in particular this time.  If anyone has particular favorites that I missed, please use the comments to share or just email me.

Interspeech Tutorial

I'm giving my first tutorial at Interspeech on Saturday in Florence.

It's titled "More than Words can Say: Prosodic Analysis Techniques and Applications".

The (very likely) final version of the slides are available here.

Now off to Italy.  Interspeech Recap to come.

Interspeech 2010 Recap

Interspeech was in Makuhari, Japan last week.  Makuhari is about 40 minutes from Tokyo, and I'd say totally worth the commute.  The conference center was large and clean, and (after the first day) had functional wireless, but Makuhari offers a lot less than Tokyo does.

Interspeech is probably the speech conference with the broadest scope and largest draw.  This makes it a great place to learn what is going on in the field.

One of the things that was most striking about the work at Interspeech 2010 was the lack of a Hot Topic.  Acoustic modeling for automatic speech recognition is a mainstay of any speech conference, that was there in spades.   There was some nice work on prosody analysis.  Recognition of age, affect and gender were highlighted in the INTERSPEECH 2010 Paralinguistics Challenge, but outside the special session focussing on this, there wasn't an exceptional amount of work on these questions.  Despite the lack of a new major theme to emerge this year, there was some very high quality, interesting work.

Here is some of the work that I found particularly compelling.

• Married Couples' speech
  Sri Narayanan's group with other collaborators from USC and UCLA have collected a set of married couples' dialog speech during couple's therapy. So this is already compelling data to look at.  You've got naturally occurring emotional speech, which is a rare occurrence, and it's emotion in dialog.  They had (at least) 2 papers on this data at the conference, one looking at prosodic entrainment during these dialogs, and the other classifying qualities like blame, acceptance, and humor in either souse.  Both very compelling first looks at this data.  There are obviously some serious privacy issues with sharing this data, but hopefully it will be possible eventually.
  
  Automatic Classification of Married Couples’ Behavior using Audio Features Matthew Black, et al.
  
  Quantification of Prosodic Entrainment in Affective Spontaneous Spoken Interactions of Married Couples Chi-Chun Lee, et al.
  
• Ferret auditory cortex data for phone recognition
  Hynek Hermansky and colleagues have done a lot of compelling work on phone recognition.  To my eye, a lot of it has been banging away at techniques other than MFCC representations for speech recognition.  Some of them work better than others, obviously, but it's great to see that kind of scientific creativity applied to a core task for speech recognition.  This time the idea was to take "spectro temporal receptive fields" empirically observed from ferrets that have been trained to be accurate phone recognizers, and use these responses to train a phone classifier.  Yeah, that's right.  They used ferret neural activity to try to recognize human speech.  Way out there.  If that weren't compelling enough, the results are good!
  
• Prosodic Timing Analysis for Articulatory Re-Synthesis Using a Bank of Resonators with an Adaptive Oscillator Michael C. Brady
  A pet project has been to find a nice way to process rhythm in speech for prosodic analysis.  Most people use some statistic based on the intervocalic intervals, but this is unsatisfying.  While is captures the degree of stability of the speaking rate, it doesn't tell you anything about which syllables are evenly spaced, and which are anomalous.  This paper uses an adaptive oscillator to find the frequency that best describes the speech data.  One of the nicest results (that Michael didn't necessarily highlight) was that he found that deaccented words in his example utterance were not "on the beat".  In the near term I'm planning on replicating this approach for analyzing phrasing, on the idea that in addition to other acoustic resets, the prosodic timing resets at phrase boundaries. A very cool approach.
• Compressive Sensing
  There was a special session on compressive sensing that was populated mostly by IBM speech team folks.  I hadn't heard of compressive sensing before this conference, and it's always nice to learn a new technique.  At its core compressive sensing is an exemplar based learning algorithm.  Where it gets clever is that where k-means uses a fixed number, k, of exemplars to use with equal weight, and SVM use a fixed set of support vectors to make decisions, in compressive sensing a dynamic set of exemplars are used to classify each data point.  The set of candidate exemplars (possibly the whole training data set) are then weighted with some L1-ish regularization to drive most of the weights to zero -- selecting a subset of all candidates for classification.  Then a weighted k-means is performed using the selected exemplars and weights.  The dynamic selection and weighting of exemplars outperforms vanilla SVMs, but the process is fairly computationally expensive.

Interspeech 2011 is in Florence, and I can't wait -- now I've just got to get banging on some papers for it.