Allion Exploratory Testing Helps You Find Potential Problems of Smart TV

Allion Labs

Nowadays, smart TVs are equipped with many functions created by means of complex development processes. If personnel only rely on the waterfall model of the quality assurance process, the testing schedules and contents may be easily affected by any delays in preceding links, unable to find major defects in products or fix them, which in turn leads to the dilemmas of delayed product launches or endless customer complaints.

In order to help customers solve the issues related to market risks, Allion Labs uses Agile Iteration combined with the Exploratory Testing model to conduct customized Weakness Analysis and strategic planning for TV products before, during, and after the development process, and even post-market launch. Allion Labs also works closely with the product development team during the testing period, providing suggestions if necessary, to allow more room for adjustment and then giving responses in subsequent product developments and testing schedules.

Figure 1: Comparison between “Waterfall” and “Agile Testing” structure processes

And what is Exploratory Testing? How do you build an efficient inspection strategy to make product inspection effortless yet effective?

Identifying Key problems Through Exploration and Analyze Key Issues Through Testing

If you’ve ever watched Formula 1 races, you have probably been impressed by the team strategies and professional division of work in the teams. For example, the performance tuning performed by mechanical engineers, the scenario planning prepared by the strategy team, the feedback by drivers and the lightning-fast pit stops for tire changes, or even sudden rain in the middle of the race, all of which are the trials for testing the team’s teamwork and ability to respond instantly.

Figure 2: Formula 1 Teamwork

The idea of Exploratory Testing is similar to that of a well-trained F1 professional team. Before the race, the strategy team investigates the current situation and develops the best strategy; during the race, the experienced driver executes tactics and improvises according to the real situation on the track; at the end, they analyze the results after the race, which serve as educational learning and reference materials for next race, and so on and so forth throughout the season. Through it all, the development and implementation of strategies are carried out in flexible processes, instead of rigid step-by-step actions! Therefore, efficient Exploratory Testing focuses on the practical experiences of both analysts and testers. Only by setting proper directions and combining with rich testing experience can we efficiently explore and identify key issues! Now that we have understood the basic concepts behind Exploratory Testing, we will further compare the benefits and application opportunities brought by Exploratory Testing.

Is Basic Testing Inadequate? Exploratory Testing Digs Deeper into Potential Product Problems

Suppose under the same premise based on the same context structure and testing time, we can see from the following test node simulation comparison: Compared with the basic tests used in the past, Exploratory Testing is conducted based on the problems found in products at the current stage, for performing deeper explorations through the tester’s experiences, by focusing on the key issues containing “high-risk directions” and then by uncovering severe problems hidden within these products.

Figure 3: Comparison of Exploratory Testing and Basic Testing simulations

Table 1: Comparison of Exploratory Testing and Basic Testing

As we can see from Table 1, Exploratory Testing is not a substitute for general basic testing. Instead, both testing methods need to complement each other to maximize their combined effect during product testing. Therefore, Exploratory Testing is recommended for the following situations:

1. Products under development: Perform Exploratory Testing on problem-prone parts after performing basic tests, to identify potential severe problems.
2. New product features or significant fixes: Perform Exploratory Testing on relevant features, to see if there is any potential problem.
3. Products launched on the market: Perform Exploratory Testing to improve quality effectively, based on customer complaints or user feedback.

The Four Stages of Exploratory Testing

As an outside third party, the Allion Labs team can provide the most professional Exploratory Testing service with its many years of experience in the TV testing field.

Step 1—Learn: Communicate and discuss with clients before conducting the tests, to understand current major issues through product specifications /test issue lists/user feedback.

Step 2—Design: After analyzing the data, draw up the testing strategies and scope targets for critical devices, scenarios, and checkpoints.

Step 3—Execute: Exploratory Testing is conducted by the team with rich testing experience, based on the testing strategies designed.

Step 4—Analyze: Provide analyses and recommendation reports based on testing results.

Figure 4: Allion’s Exploratory Test Services

Case Study

Testing Background

The client had already conducted basic testing on their TV products, but they still receive numerous customer complaints after product launch. Many problems had not been discovered before the product launch, so they decided to commission Allion Labs to conduct customized testing on their products, hoping to find out the key causes behind these quality defects.

Test Planning

Judging from customer complaints, we found that these products suffer from problems in the functions related to HDMI, Bluetooth, Wi-Fi, and others. The Allion Labs team immediately developed strategies and directions for Exploratory Testing on key equipment, scenarios, and inspection key points, based on the client’s lists of problems and market user feedback.

Figure 5: Allion Test Plans

Customer Complaints

1. After a Bluetooth headset is connected and used for a period of time, problems such as intermittent interruption, noise, or asynchronization begin to occur.
2. In some settings and combinations, TV speakers and audio equipment may have unstable sound output/switching and compatibility problems.
3. Under the operations when Bluetooth audio devices are connected or streaming platforms are switched or TV is turned on/off, etc., there may be blank screens or serious image problems.
4. The functions of “One Touch Play” and “Power on Device from TV” on HDMI-CEC will malfunction in some connection scenarios.
5. When some of the mainstream HDMI-CEC devices on the market are connected, sometimes abnormal identifications or abnormal overall device lists/out-of-sync displays will appear.
6. After the wireless network connection is turned on or off, there are connection abnormalities/status display abnormalities under certain conditions.
7. Back panel slots are poorly designed, unable to let some external plug-in devices be plugged in successfully.

Test Results

Overall, with only one DUT, Allion Labs spent only 5 working days identifying 26 problems and classified them into four levels according to problem severity (see the table below). Among them, 19 problems were Critical and Medium, accounting for 73% of the total proportion, which is quite unstable in terms of quality for a product that has been on the market for such a long time.

Figure 6: Exploratory Test Case

In terms of problem type distribution, in addition to HDMI, Wi-Fi, and Bluetooth, which were the main problems initially tested, a total of 15 problems were further identified with regard to Sound, UX/UI, and Resolution through Exploratory Testing, accounting for 58% of the total problems, including 5 Critical-level problems in Sound and Picture. The client felt so surprised at such results.

Figure 7: Exploratory Testing Case Study: Issue Type Distribution

With regards to mechanism design, because they never considered that the product should be equipped with some functions necessary for combining with other external plug-in devices (as shown in the following figure) during the process of product development and design planning, the product could not be used smoothly. In addition to affecting user experience, this problem in the worst case may even cause partial functional damage to the product under long-term usage. Although this issue is not classified as the level of Critical, it is still a major factor from the perspective of user experience.

Figure 8: Case Study of Mechanism Design 

Suggestions and Guidelines

Based on the testing results and our many years of testing experience, Allion Labs provides the following suggestions to the client:

Compatibility: If there are so many problems cropping up merely in the use of a few soundbars/AVR/headphones, it is likely that numerous compatibility problems have been already occurring on the market. We suggest that large-scale compatibility tests should be conducted on HDMI/Bluetooth audio or headphone devices (at least 150+ units) further,  to ensure product compatibility.

Basic Product Functions: Although the client had performed tests and inspections by themselves before launching these products, we discovered many problems with regard to basic HDMI protocol functions. We suggest that further investigation should be conducted regarding HDMI Logo certification testing, to find out the root cause of the problem.

Mechanism Design and User Interface: Some of the mechanism designs did not take into account compatibility issues with peripheral products, while user interfaces were not integrated or optimized. We suggest that relevant testing and design improvements should be conducted for the peripheral products with higher market shares as well as major competitors’ TV products, to enhance market competitiveness comprehensively.

From the case study mentioned above, if you endeavor to avoid numerous customer complaints and adverse impact on your company’s goodwill after product launch, we suggest that Exploratory Testing should be conducted at the product development stage, to identify potential problems and design flaws in the product as early as possible, so that the problems can be fixed in time before product launch, to reduce the possibilities of customer complaints.

With more than 30 years of experience in testing and verification, Allion Labs has been dedicated to the R&D of innovative testing services, to help customers solve various product problems. If you want to learn more or have related product testing needs, please feel free to contact us via our contact form or by email.