As we mentioned in our last article, smart speakers can be activated by voice to provide search and entertainment support as well as control of other smart devices in different environments. To complete the aforementioned tasks, a smart speaker needs to work with other devices; for example, it needs to connect to the internet via wireless access points (APs), it needs to synchronize with smart phones through Bluetooth®. However, we found several interoperability issues after paring up six smart speakers with 20 APs and 20 smart phones. In this article, we argue that, even for the products that have been certified by Wi-Fi and Bluetooth®, interoperability issues can occur during the complex matches among the various products in the market. We will share the test results from Allion’s Wi-Fi and Bluetooth® interoperability validation and explain the issues, including irrational latency, connection failures, inconsistent user experience, and functionality shortages caused by poor interoperability of products.
Voice Assistant’s Most Mature Application
Among the three applications mentioned previously, entertainment support (e.g. listening to music or radio) tends to be the most used application for smart speakers owners. According to a 2018 survey from Voicebot.ai, over 75% of the users listen to music through smart speaker support on a monthly basis. A Music Ally Report also states that “over 80% of the use case around these devices [i.e. smart speaker] is listening music.” (Everybody’s Talkin’: Smart Speakers & their impact on music consumption, p. 25).
Figure 1: The Data Cites from Voicebot AI.
Wi-Fi Interoperability Issues
One way that smart speakers play music is connecting them via Wi-Fi to digital music providers, such as Amazon Music, YouTube, and others. Nevertheless, Wi-Fi interoperability issues could occur due to the complex combinations between the various smart speakers and routers in the market. By simply pairing up the six most popular brands’ smart speakers (i.e. models from Amazon, Lenovo, Xiaomi, Alibaba, etc.) with 20 access points (i.e. the top-selling models from Apple, Google, Belkin, Netgear, etc), Allion’s experts have found irrational latency and connection failures on some of the matches. These issues can seriously decrease user satisfaction.
Figure 2: Test Bed (Wireless Access Points) shows in Allion’s Device Library which catalogs over 10 thousands devices.
Irrational Latency
When receiving a voice command from user, a smart speaker has to connect to its database via the internet to provide the appropriate reply. Therefore, we asked the 6 voice assistants (named P, M, I, A, D, and L) to play songs from assigned artists via the 20 routers (named from A to T) in 10 times and averaged their response periods (sec.). We also used Ping to measure the throughputs between all the APs and smart speakers.
Figure 3: Test “P, M, I, A, D, and L” smart speakers with 20 wireless routers (A-T)
Unfortunately, we found the average throughputs of Smart Speakers I, A, and D varied while connecting with different routers. For example, the average throughput range of Smart Speaker I were from 17.52 Mbps to 149.66 Mbps. When we extended the distance between the device-under-test (DUT) and the test beds, the average throughput range of Smart Speaker I would become 4.25 Mbps to 115.23 Mbps. Smart Speaker D had even larger gap under the longer distance test condition: 5.67 Mbps to 235.06 Mbps. The varied throughput values had negative correlations with the DUTs response time, causing different levels of latency to DUTs.
Connection Failure
Besides latency, the poor interoperability of smart speakers and routers can trigger connection failures. In order to provide a better entertainment experience, Brand P, for example, has developed an app to allow users to stream their media contents from different devices (such as PC or mobile phones) to smart speakers under the same Wi-Fi network. However, during the initial setup process, we found that after receiving a connection request from the app on mobile devices, a connection failure occurred between Router B and Smart Speaker P. The issue can cause the whole system to crash.
Bluetooth® Interoperability Issues
Some users are used to connecting smart speakers to smart phones via Bluetooth®, allowing them to synchronize personal media contents, such as music playlists,video, and more. However, we found functionality shortages and inconsistent user experience after paring up the 6 DUTs (smart speakers) with 20 mobile phones (models from Apple, Google, SONY, OPPO, etc.).
Figure 4: Test Bed (Mobile Phone) shows in Allion’s Device Library which catalogs over 10 thousands devices.
Inconsistent User Experience
During the Bluetooth® interoperability validations, we found that some Bluetooth® connections between smart speakers and mobile phones are able to be rebuilt once Bluetooth® is turned on on the mobile phones. Yet, the same phenomenon was not observed on some of the matches. For example, smart speakers and phones from the U.S., compared to those from China, were less likely to establish the connection automatically during our validation cases. The issue becomes critical since if the connection can be automatically re-established after the switches, it means that one of the devices kept sending connection requests to another. This could not only make the device consume more power to create inconsistent user experiences but also cause security concerns on some of the cases.
Figure 5: Smart Speakers P, M, I, A, D, and L with 20 Mobile Phones A to T
Functionality Shortage e
Besides issues discovered from establishing Bluetooth® connections, we found functionality shortages during the synchronizations of smart speakers and phones. For instance, the volume bar on Mobile Phone S was not able to adjust to the volume bar on Smart Speakers L, I, A, and D. The “next” and “previous” function on Smart Speaker I could neither work properly. The “next” function changed to “Fast Forward” and the “previous” function changed to “Rewind” when pairing with Smart Phone G.”
Potential Issues Revealed by Complex Matches
Allion’s experts have listed some root causes of the issues, including irrational latency (Wi-Fi), connection failures (Wi-Fi), inconsistent user experience (Bluetooth®), and functionality shortages (Bluetooth®) mentioned in the early paragraphs. For Wi-Fi interoperability issues, evidence shows that the incompatible designs of broadcast technologies might make the two devices (Smart Speaker P and Router B) fail to exchange information, causing connection failures. It also suggests that since chipsets of our test beds cover different manufacturers such as Qualcomm, Broadcom, MediaTek, and others, the uninteroperable chipsets could decelerate the connection speed of internet. For the inconsistent user experience and functionality shortages during the Bluetooth® connections between smart speakers and mobiles phones, evidence shows that it might be caused by the incompatible setting of devices’ Audio-Video Remote Control Profile (AVRCP) and Generic Access Protocol (GAP). Nevertheless, these issues could not be discovered during the compliance tests of Wi-Fi and Bluetooth®.
Conclusion
In conclusion, we have shared the test results from Allion’s Wi-Fi and Bluetooth® interoperability validation and provide some implications by analyzing the issues, including irrational latency, connection failures, inconsistent user experience, and functionality shortages, caused by poor interoperability of products. We argue that, even for the products that have been certified by Wi-Fi and Bluetooth®, interoperability issues can occur during the complex matches among the various products in the market.
Dedicated to seanmlessly integrating electronic devices into users’ lives, Allion’s experts specialize in finding interoperability issues by designing complex combinations from a variety of products. We are not only the Authorized Test Lab of Wi-Fi and Bluetooth®, but our device library also features thousands of products, including wireless routers, mobile phones, and IoT devices. We further provide technical consultations and debugging support. The one-stop solution will thus allow vendors to reduce delays and uncertainties during the manufacturing process.
Let us know if you are interested in any of our service: service@allion.com