Voice AI Test Platform (VATP)

Limitations and Extremes of Manual Voice Testing

Traditional voice testing relies heavily on manual execution, which is not only time-consuming and highly repetitive, but also makes it difficult to quantify complex noise environments. It comes with the following major limitations:

• Long-term Manual Monitoring:
  During long-term stress testing, testers suffer from decreased concentration due to fatigue, making it easy to miss critical, sporadic errors such as unexpected system crashes or unresponsive voice triggers.
• Results Reliant on Manual Judgment:
  There is a lack of uniform standards; the judgment criteria (Subjective Judgment) can vary from one tester to another.
• Low Efficiency in Repetitive Testing:
  Whenever a software version is updated, a complete regression test (Regression Test) must be conducted. Repeating the exact same voice commands manually is extremely tedious and time-consuming.
• Complex Acoustic Scenarios are Difficult to Simulate and Replicate:
  It is challenging for humans to accurately simulate composite environments that feature both “ambient background noise” and “multiple people talking simultaneously in the background.”

Allion VATP Voice Verification Solution

The Voice Assistant System automated testing solution developed by Allion combines artificial intelligence recognition technology with a fully automated control platform.

This solution aims to provide a “Closed-loop” testing environment. From simulating human voice commands and ambient noise simulation to determining the results after the voice assistant executes them, the entire process requires zero human intervention. This drastically increases testing efficiency by over 80% while ensuring high data consistency.

24/7 non-stop execution with automated report generation

Precise error positioning and automated regression testing

One-click switching of environmental parameters to reduce testing equipment wear and tear

Visual UI element checking and simulation of different pronunciation characteristics

Standardization of multiple languages and consistency in background noise

High-intensity stress testing to eliminate human operational errors

Enhanced test coverage and historical data comparison

Precise environmental control to simulate various scenarios such as being at home or outdoors

True Simulation of Critical User Scenarios to Meet Everyday Application Needs

Under Consumer Electronics Ecosystem Applications, VATP is capable of reconstructing environmental sound fields and simulating the following use cases:

• Home Environment Noise: Verifies the degree of impact that interfering sound sources have on the voice recognition of smart devices (Smart Devices).
• Café Environment Noise: Verifies the impact of AI PC noise cancellation functions on the success rate of voice recognition.
• Video Conferencing Use: Reconstructs acoustic conditions to identify the root cause when voice transmission interruptions occur.

VATP Voice Verification Core Architecture

To achieve authentic simulation of user scenarios, VATP must operate within an acoustic “Listening Room.” Through a tightly integrated space, software, and hardware architecture, it delivers complete and precise automated testing.

• Voice Command Control Module
• Visual Recognition Module
• Auditory Recognition Module
• Automated Software Central Control Platform

Acts as the “mouth” of the testing system. It features a built-in, high-quality human voice simulator that can simulate natural vocalizations across different genders, ages, tones, and multiple languages. Compared to traditional recorded playback, the AI voice output can adjust speech speed and emphasis more flexibly, testing the system’s tolerance toward various human pronunciation traits.

• Simulates actual users controlling smart voice devices through script orchestration.
• Any voice converted into a .wav audio file can be imported into VATP, allowing it to simulate different national voices or accents according to customer requirements.
• Depending on the application scenario, up to 4 speakers can operate simultaneously to simulate 4 different audio sources when multiple people are talking in various environments.

Acts as the “eyes” of the testing system. It monitors screens via a high-frame-rate camera and utilizes image recognition technology to immediately determine:

• Whether the screen correctly navigates to the corresponding page.
• Whether UI icons and text content appear correctly.
• The visual latency calculated from the end of the voice command to the screen presentation.
• Testing Capabilities:
  • HD-Resolution：1920*1080 @ 30FPS
  • Detect feedback of Product：OCR_Optical Character Recognition

• Through camera imagery and visual recognition technology, it automatically detects whether screen content displays as expected, achieving unmanned operation.
  • Input Voice Comment：How to get To Taichung HSR Station
  • OCR Test Result：Pass

Through microphone recording and speech-to-text recognition technology, it automatically detects whether the device’s output voice displays as expected, achieving unmanned operation.

MIC SPEC

• Capsule: 1/2″ low-noise pre-polarized condenser.
• Sample Rates: 44.1, 48, 88.2, 96, 176.4, 192 kHz
• Resolution: 32-bit ADC.
• Sensitivity: -34 dBfs@ 0 dB Gain
• Noise Floor: 105.3 dBfs (A) @ 0 dB Gain
• Max SPL: 125 dB SPL
• Polar Pattern: Omnidirectional.

Acts as the “ears” of the testing system. It receives response audio from the voice assistant and employs high-precision AI speech-to-text (STT) technology to analyze whether the voice assistant’s response content is correct, and whether audio clipping or dropouts occur. The testing capabilities are as follows:

This is the “brain” of the system. Allion’s exclusively developed automated central control software coordinates all the following modules:

• Script Editing
  Supports intuitive drag-and-drop script scheduling, allowing rapid onboarding without requiring programming skills.
• Synchronized Control
  Ensures that voice emission, noise simulation, and image capture are synchronized at a millisecond level.
• Data Reporting
  Automatically generates detailed reports including success rate statistics, fault screenshots, and log analyses to quickly locate problems.
• UI Language Support : English, Japanese

What Problems Does the VATP Voice Verification Solution Solve?

1. Ambient Noise V.S. Voice Recognition:
   Measures the wake-up rate and speech recognition accuracy (SRR) of voice assistants under simulated true-to-life environmental noises.
2. Voice Assistant Control Verification:
   Automatically executes sequential commands like “search” or “play” to verify whether smart voice assistant devices switch screens correctly.
3. Multi-Language Globalization Testing:
   Swiftly switches voice templates to accommodate global market adaptability.
4. System Stress and Stability:
   Conducts thousands of continuous voice interactions to monitor whether memory leaks (Memory Leak) cause voice function failures.
5. Professional Report Output:
   Provides an automated Log/Report generation system for you to record and trace the root cause of issues.
6. Precision and Efficiency Enhancements:
   Taking an 8-hour day of manual testing as an example, automated testing is three times more efficient and can better ensure the accuracy and reliability of test results.

Global Services – Allion Delivers Professional Localized Testing Expertise

To provide clients with higher-quality and broader service coverage, Allion offers localized, fast, and high-quality testing services across multiple global locations, including the United States, Japan, China, Europe and additional regions worldwide.

Contact Us

If you have any questions about our VATP testing services, please feel free to contact us or fill out the inquiry form. Our team will be glad to support you.