Speech recognition in restaurants: phone lines, menu words and accents

Automatic speech recognition (ASR) is software that turns speech into written words. In a restaurant it is the first step of every AI phone, drive-thru or kiosk order, and everything after it works only from the words it writes down. Three things make that harder than in a quiet office: the narrow sound of a phone call, menu words the software has not heard, and accents it heard less of in training.

This guide covers phone lines and indoor order points. Engine noise, wind and headsets at the lane are in our drive-thru speech guide, and the other parts of an AI order taker, with their vendor terms, are in conversational AI in plain terms.

How does speech recognition work?

A microphone or phone line turns sound into a stream of numbers. A model trained on recorded speech and matching transcripts works out the most likely words, often with a score for how sure it is. OpenAI's Whisper paper (December 2022), for example, describes models trained on 680,000 hours of transcribed audio.

The model learns from what it was fed. Speech that differs from its training audio, such as a narrower sound, a new word or a less common accent, produces more mistakes. The sections below take each in turn.

Why does a phone call sound different to the software?

A phone call can carry only part of the sound of a voice. The ITU's G.711 standard for digital telephone voice sets a sampling rate of 8,000 samples a second, and a signal sampled that often can hold sound up to 4,000 Hz at most. The ITU's G.722 standard, written for "higher quality speech", samples 16,000 times a second and covers 50 to 7,000 Hz. A call that runs through the narrower coding loses the upper part of the voice before the software hears it.

Research on phone speech shows how much the kind of conversation matters as well. In a 2016 Microsoft study on the NIST 2000 test set of recorded phone calls, professional transcribers got 5.9% of words wrong on calls between strangers talking on an assigned topic, and 11.3% on calls between friends and family. Microsoft's system scored 5.8% and 11.0%. An IBM paper in March 2017 reported 5.5% and 10.3% on the same two sets, and found human transcribers could do better than the Microsoft figures. Open-ended calls between friends and family roughly doubled the error rate for people and machines alike.

Two questions follow for your phone line. Ask the vendor what sampling rate and codec your calls arrive in, and whether call forwarding from your current number changes it. Our call-forwarding checklist covers the routing itself.

What does word error rate measure?

Word error rate (WER) is the score research uses for ASR. NIST's scoring tool sclite, part of its speech recognition scoring toolkit, lines the software's transcript up against a reference transcript written by a person. It counts words substituted, deleted and inserted, and divides each count by the number of words in the reference. The word error rate is those three shares added together. Every word counts the same, whether it is "the" or "pepperoni".

Two findings from the research matter when a vendor quotes a WER:

• Formatting can move the score. The Whisper authors found that transcripts a person would judge correct can still score badly over small differences in style. Their text clean-up before scoring cut WER by up to half on some datasets.
• A lower WER does not promise a better ticket. In a 2003 Microsoft Research paper, a system with a 46% higher word error rate made up to 17% fewer errors in the details it had to understand.

Here is a made-up phone order of 12 words: "two large pizzas one pepperoni one veggie and an order of wings".

What the software wrote	Errors	WER	Ticket
"2 large pizzas one pepperoni one veggie and an order of wings"	1 substitution ("2" for "two")	First transcript: 1 of 12 words, 8.3%	Right
"two large pizzas one pepperoni one veggie and an order of rings"	1 substitution ("rings" for "wings")	Second transcript: 1 of 12 words, 8.3%	Wrong item
"two large pizzas pepperoni veggie and order of wings"	3 deletions ("one", "one", "an")	Third transcript: 3 of 12 words, 25%	Right

The two transcripts with the same score produced a right ticket and a wrong one, and the worst score produced a right ticket. Our drive-thru guide shows the same effect with a burger order. Count ticket errors yourself, by item, size, modifier and quantity.

Why do menu words trip it up?

A recognizer picks likely words from what it learned. A house sauce, a dish named after the owner or a local spelling may be rare in its training data. Research systems handle this by giving the recognizer a list of expected phrases at the time of the call. In a 2018 Google paper, a model built to use such lists, including words it had not seen in training, cut word errors by as much as 68% relative to an older way of adding the list, across the paper's test tasks.

For you, the questions are how your menu names get onto that list, and who updates it when you add an item. Our guide to how an AI order becomes a ticket covers the phrase-list settings in Google's and Microsoft's services, and the menu preparation worksheet helps you list the names guests use.

How much do accents matter?

A lot, and unevenly. The Edinburgh International Accents of English Corpus (March 2023) recorded about 40 hours of video calls between friends, speaking many first- and second-language varieties of English. The best model the authors tested got 19.7% of words wrong on those calls, against 2.7% on clean, read US English. Every model they tested did worse on Indian, Jamaican and Nigerian English speakers.

Our drive-thru guide covers a 2020 Stanford-led study of five large speech services that found a similar gap between Black and white American speakers. The practical step is the same: test with the voices your restaurant actually hears, guests and staff both.

What changes at a kiosk or other order point?

A kiosk or counter microphone is not limited by a phone line, so the sound can be fuller. The problems move to the room: other guests talking nearby, music and kitchen noise. Our voice kiosk guide covers those. Maple's voice kiosk, in early access, uses a microphone built to pick out the guest in front and shows the cart on screen as the guest speaks, so mistakes can be fixed before paying.

How do you test speech recognition for your restaurant?

1. Write 20 test orders from your real ticket history, including your hardest dish names, a half-and-half or split item, and quantities such as "two twelve-inch".
2. Place each order on your real line: from a mobile phone, a landline, a speakerphone and a car's hands-free system.
3. Use at least four speakers with different accents, including staff whose first language is not English.
4. Ask the vendor for the written transcript of every test call.
5. Work out WER by hand for five calls using the method above, so you understand any figure the vendor quotes.
6. Score every call on the ticket: items, sizes, modifiers, quantities and pickup time.
7. Add a new menu item, wait for the update, and order it by name the same day.
8. Note what the system does when it is unsure: ask again, read back, guess or pass to staff.

Maple's phone ordering page asks restaurants to test the languages, menu terms, accents and interruptions their guests use before going live, and its Voice plan lists multilingual voice. Run the list above on any system you consider, Maple included.

Published by Maple, which sells AI phone answering and ordering to restaurants. This AI-assisted guide combines ITU telephone standards, NIST's scoring tool documentation and published research from Microsoft, IBM, Google, OpenAI and the University of Edinburgh with an original worked example and test plan. It does not report a test Maple ran or measure any vendor's recognition.