Speech Recognition Engineer Interview Questions: Complete Prep Guide 2026

Answer: CTC loss allows training sequence-to-sequence models without requiring frame-level alignments between audio and text. It works by:

• Introducing a "blank" token that represents no output
• Allowing multiple paths through the output sequence that collapse to the same final text
• Summing probabilities of all valid paths that produce the target sequence
• Using dynamic programming to compute this efficiently

Why it matters: CTC was revolutionary for ASR because you don't need phoneme-level timestamps—just the audio and final transcription.

Follow-up they might ask: "What are the limitations of CTC?" (Answer: Can't model output dependencies, blank token overhead, assumes conditional independence)

Answer:

WER: Measures errors at word level. Formula: (Substitutions + Deletions + Insertions) / Total Words

CER: Measures errors at character level. Same formula but applied to characters.

When to use:

• WER: English and other space-separated languages, end-user facing metrics
• CER: Languages without clear word boundaries (Chinese, Japanese), when word tokenization is unclear

Pro tip: Always report which metric you're using—a 5% WER sounds great until you realize the baseline was 2%.

Answer: End-to-end ASR models typically have three components:

1. Encoder: Converts audio features (mel-spectrograms) into high-level representations. Usually a stack of CNNs + RNNs or Transformers. Takes variable-length audio input.
2. Attention Mechanism: Learns to focus on relevant parts of the encoded audio when predicting each output token. Allows the model to "attend" to different parts of the audio at different times.
3. Decoder: Generates output sequence (characters or subwords) autoregressively. Uses previous predictions and attended encoder outputs.

Key advantage over traditional pipeline: Single neural network trained end-to-end, no need for separate acoustic model, pronunciation dictionary, and language model.

Answer: Beam search is a decoding algorithm that maintains the top K most likely partial hypotheses at each step:

1. Start with K=beam_width empty hypotheses
2. For each hypothesis, generate all possible next tokens
3. Score each extension (usually log probability)
4. Keep only the top K scored complete hypotheses
5. Repeat until end-of-sequence or max length

Tradeoffs:

• Larger beam (K=10-20): Better accuracy, slower inference, more memory
• Smaller beam (K=1-5): Faster inference, less memory, might miss optimal path
• K=1 (greedy): Fastest but often suboptimal

Production insight: Most systems use K=5-8 as a sweet spot. Beyond K=10, gains plateau.

Answer: Mel-spectrograms are time-frequency representations of audio that use the mel scale, which better matches human perception of sound.

How they're created:

1. Take raw audio waveform
2. Apply Short-Time Fourier Transform (STFT) to get spectrogram
3. Convert frequency axis to mel scale (logarithmic)
4. Often apply logarithm to amplitudes

Why mel scale? Humans perceive pitch logarithmically—doubling from 100Hz to 200Hz sounds like the same "distance" as 1000Hz to 2000Hz. Mel scale captures this.

Why use them? Better than raw audio (too high-dimensional) or linear spectrograms (don't match human perception).

Non-streaming (offline): Process entire audio file at once, can look forward and backward, higher accuracy.

Streaming (online): Process audio in real-time as it arrives, can only look backward (and limited lookahead), must maintain low latency.

Technical challenges of streaming:

• Latency: Must emit results within ~200-500ms for real-time feel
• Chunking: How to split audio while maintaining context?
• Look-ahead limitations: Can't use future context that works well offline
• Stability: Results shouldn't change after being emitted (no "flickering")
• State management: Need to maintain decoder state between chunks

Common solutions: RNN-Transducer architecture, limited lookahead windows, causal attention mechanisms.

Answer: Wav2Vec 2.0 is a self-supervised learning approach for speech that learns representations from raw audio without transcriptions.

How it works:

1. Masking: Randomly mask portions of the audio input (like BERT does for text)
2. Quantization: Discretize the audio into a finite set of representations
3. Contrastive Learning: Train the model to predict the correct quantized representation from a set of "distractors"
4. Fine-tuning: After pre-training, add a small CTC head and fine-tune on labeled data

Why it's important: Achieves strong results with as little as 10 minutes of labeled data for low-resource languages, vs. thousands of hours needed for traditional approaches.

Related work: HuBERT, WavLM, Data2Vec (similar ideas with variations)

def compute_wer(reference, hypothesis):
    """
    Compute Word Error Rate using Levenshtein distance.
    
    Args:
        reference: Ground truth string
        hypothesis: Predicted string
    
    Returns:
        WER as a float (0.0 to 1.0+)
    """
    ref_words = reference.split()
    hyp_words = hypothesis.split()
    
    # Build edit distance matrix
    d = [[0] * (len(hyp_words) + 1) for _ in range(len(ref_words) + 1)]
    
    # Initialize first row and column
    for i in range(len(ref_words) + 1):
        d[i][0] = i
    for j in range(len(hyp_words) + 1):
        d[0][j] = j
    
    # Fill matrix
    for i in range(1, len(ref_words) + 1):
        for j in range(1, len(hyp_words) + 1):
            if ref_words[i-1] == hyp_words[j-1]:
                d[i][j] = d[i-1][j-1]  # No error
            else:
                substitution = d[i-1][j-1] + 1
                insertion = d[i][j-1] + 1
                deletion = d[i-1][j] + 1
                d[i][j] = min(substitution, insertion, deletion)
    
    # WER = edit distance / reference length
    return d[len(ref_words)][len(hyp_words)] / len(ref_words) if ref_words else 0.0

# Test
ref = "the quick brown fox"
hyp = "the qwick brown fox"
print(f"WER: {compute_wer(ref, hyp):.2f}")  # 0.25 (1 error out of 4 words)

Follow-up questions:

• "How would you optimize this for very long sequences?" (Answer: Use NumPy, only keep two rows)
• "What if you need to return the specific errors?" (Answer: Backtrack through the matrix)

def greedy_ctc_decode(logits, blank_id=0):
    """
    Greedy CTC decoding: take argmax at each timestep, collapse repeats and blanks.
    
    Args:
        logits: (T, vocab_size) model outputs before softmax
        blank_id: ID of blank token (usually 0)
    
    Returns:
        List of predicted token IDs
    """
    import numpy as np
    
    # Get argmax at each timestep
    predictions = np.argmax(logits, axis=1)
    
    # Collapse: remove consecutive duplicates and blanks
    output = []
    previous = None
    
    for pred in predictions:
        # Skip if same as previous (collapse repeats)
        if pred == previous:
            continue
        # Skip blank tokens
        if pred == blank_id:
            previous = pred
            continue
        # Add to output
        output.append(pred)
        previous = pred
    
    return output

# Example usage
# logits shape: (50, 29) for 50 timesteps, 29 tokens (26 letters + 3 special)
# After greedy decode: might get [8, 5, 12, 12, 15] -> "hello"

Extension: "Now implement beam search CTC decoding" (significantly harder, usually just discuss approach)

import librosa
import numpy as np

def extract_mel_spectrogram(audio_path, sr=16000, n_mels=80, 
                           n_fft=400, hop_length=160):
    """
    Extract mel-spectrogram features from audio file.
    
    Args:
        audio_path: Path to audio file
        sr: Sample rate
        n_mels: Number of mel bands
        n_fft: FFT window size
        hop_length: Hop length for STFT
    
    Returns:
        Mel-spectrogram (n_mels, time)
    """
    # Load audio
    audio, _ = librosa.load(audio_path, sr=sr)
    
    # Compute mel-spectrogram
    mel_spec = librosa.feature.melspectrogram(
        y=audio,
        sr=sr,
        n_fft=n_fft,
        hop_length=hop_length,
        n_mels=n_mels,
        fmin=0,
        fmax=sr/2
    )
    
    # Convert to log scale (dB)
    mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max)
    
    return mel_spec_db

# Usage
features = extract_mel_spectrogram('speech.wav')
print(f"Feature shape: {features.shape}")  # (80, T)

Discussion points:

• "Why these specific hyperparameters?" (16kHz is standard for speech, 80 mels is common, hop of 10ms)
• "What's the time resolution?" (hop_length/sr = 10ms per frame)

Key components to discuss:

1. Wake Word Detection (on-device)

• Tiny neural network (1-5MB) running on device
• Always listening, very low power
• High recall (catch all wake words), lower precision OK
• Sends audio to cloud only after detection

2. ASR Service (cloud)

• Streaming ASR (RNN-T or similar)
• Auto-scaling based on load
• Regional deployment (latency)
• GPU inference servers
• Target latency: <200ms for first word

3. NLU (Intent Classification)

• Extract intent and entities from transcription
• Route to appropriate service (music, weather, etc.)
• Fast inference (CPU or small GPU)

4. Response Generation

• TTS for voice response
• Caching common responses
• Multiple voice options

Scale considerations:

• 10M concurrent: Need thousands of inference servers
• Load balancing: Geographic routing, queue management
• Cost optimization: Batch where possible, cache aggressively
• Monitoring: Latency p50/p95/p99, WER, uptime

Tradeoffs to discuss:

• On-device vs cloud processing (privacy vs accuracy)
• Model size vs accuracy (smaller = faster but less accurate)
• Streaming vs batch (latency vs throughput)

Architecture components:

1. Audio Ingestion

• WebSocket or WebRTC from client
• Audio chunking (1-5 second segments)
• Queue system (Kafka/RabbitMQ)

2. ASR Pipeline

• Streaming ASR (Whisper or similar)
• Speaker diarization (who spoke when)
• Punctuation restoration
• Word-level timestamps

3. Post-Processing

• Filler word removal (um, uh, like)
• Paragraph segmentation
• Named entity recognition
• Action item extraction (ML model)

4. Storage & Retrieval

• Audio in S3/cloud storage
• Transcripts in database (PostgreSQL)
• Full-text search (Elasticsearch)

Scale math:

• 1000 concurrent meetings, 1 hour avg = 1000 audio-hours/hour
• Real-time factor (RTF) = 0.2-0.5 (process 1 hour in 12-30 min)
• Need ~50-200 GPU instances for ASR
• Storage: 1000 meetings/day * 30 days * 100MB audio = 3TB/month

What they're really asking: Can you advocate for your ideas while staying collaborative?

Good answer structure (STAR method):

• Situation: "We were deciding between RNN-T and Transformer for streaming ASR..."
• Task: "I believed RNN-T was better for our latency requirements..."
• Action: "I prepared a doc with benchmark data, presented to the team, and we decided to prototype both..."
• Result: "RNN-T won, but the process helped us align on latency goals..."

Red flags to avoid: Being stubborn, not listening to others, making it personal

Why they ask: Speech tech moves fast. Can you adapt?

Good example topics:

• Learning Whisper when it came out and applying it to your use case
• Picking up Kaldi despite its steep learning curve
• Diving into self-supervised learning papers and implementing Wav2Vec

Key points to emphasize:

• How you approached learning (papers, code, experiments)
• Timeline (weeks not months)
• Concrete outcome (shipped feature, improved metric)

Bad answer: "It's a hot field" or "Good salary"

Good answer shows genuine interest:

• "I'm fascinated by how much context ASR requires—acoustic + linguistic + sometimes visual..."
• "Voice is the most natural interface, but we're still far from solving it..."
• "I built a project using Whisper and realized how challenging low-resource languages are..."
• "The combination of signal processing, deep learning, and linguistics is unique..."

Connect to company: "Your work on [specific product] aligns with my interest in [area]..."