Technical Blogs

When Tele-Vision Becomes Inter-Vision: Smart TV Test Report Part II
2013 2013-06-27

No More Couch Potato

Now we can say that the future Internet technology and application will move toward the development of mobile entertainment devices and home entertainment devices. In “Mobile” oriented, mobile entertainment devices such as smartphone and tablet PC have significantly and successfully caused the big bang explosion in market; in “Home” oriented, lean-back media such as smart TV, connected TV and set-top box invade our living room gradually, and then users just sit down and veg out watching TV and browsing Internet. That is, the user scenario of TV is no longer passively receiving information, but users are able to take the initiate to select, browse and operate various TV applications and services.


The development of TV is perhaps not rapid as mobile devices; however, whether global Internet/system vendors or information/content suppliers are constantly exploring the potential business model. At CES 2011, 2012 and even 2013, all major tech vendors are talking about innovative features for their smart TVs in an attempt to differentiate them. Predictably, with the growth of smart TV technology and usability, the smoothness of human-computer interactions with menus and applications vitally contributes to the better user experience (UX). At this critical moment, Allion Labs, Inc. (Allion) particularly purchases some international brand smart TVs for comprehensive competitive analysis of their technology, interface, performance and usability. It can help users get their ideal smart TV with the most consideration and dominance; after that, there are no more couch potatoes left in the living room.


Smart TV Competitive Analysis

We select three Japanese-related & Korean-related smart TV models as the test targets (Brand A, Brand B and Brand C), and then implement functionality test and usability evaluation according to the following test scope.

I. Functional Signal Test of Basic I/O Interface

II. Power Consumption Test

III. Ecosystem Test

IV. User Experience

i. Graphic User Interface (GUI) Analysis

ii. Usability Test

iii. Smart Function and Brower Performance Test

Initially, we compare the announced product feature of three smart TVs (Table I).

2013-6-27 下午 04-37-00

2013-6-27 下午 04-37-42

In the traditional TV functions, we find that Brand A and Brand C are better than Brand B with years of advantages. But in the smart functions, Brand B has more innovative highlights and breakthroughs, including face recognition, voice activated, motion control and mobile phone synchronization.

 I. Functional Signal Test of Basic I/O Interface 

In this part, we implement functional signal tests of basic I/O interfaces, including Tuner, USB, HDMI, DLNA, Wi-Fi Direct and RF Signal. Then, we find that Brand A cannot discover the DLNA server from wireless network after executing the Wi-Fi Direct function; also, it cannot playback the JPG file (Photoshop) because of its lake of decoding picture file properly. For Brand B, it will appear the condition of “Channel Lost”, which can be improved by adjusting the antenna manually. For common end-users; however, they may not know how to rule out the similar problem. At HDMI test, “Vertical Line Noise” could be seen on Brand B. Finally, we find that Brand C cannot show “Program Rating” information; for example, no. 6 means Parental Guidance Suggested (PG), but Brand A and Brand B can show it correctly. At HDMI test, although Brand C has announced that it passed the HDMI certification, we still find one test item “TMDS Jitter Tolerance” failed. That is not just the problem of signal interference, but an inconsistency between the product development and specification certification. In addition, there is a similar issue happened at DLNA test, our test criteria is based on registered supported format on DLNA official website and these functions shall be under normal usage on TV. However, we find that some DLNA test items such as DMP mode and DMS mode are failed and cannot be decoded correctly. In other words, the certification process of Brand C needs to be reviewed carefully and strictly.

 II. Power Consumption Test 

Based on the default settings of three TVs, there are significant differences in their “Brightness Power Ratios”. Particularly, Brand B has the best brightness in the three models (Brand A: 202 nits/ Brand B: 497 nits/ Brand C: 262 nits), so it will indirectly affect the higher power consumption of Brand B. For the average of overall power consumption, Brand C consumes the least power; however, when we do different brightness adjustment and observe the power consumption, we find the overall brightness power ratio of Brand B is better than Brand A and Brand C(Table II).

2013-6-27 下午 04-38-21

 III. Ecosystem Test 

The function of traditional TV is only broadcasting, and the peripheral devices are mostly CD or DVD players. However, because smart TV has more interfaces which can connect to multiple peripheral devices, we choose different types of peripherals such as Blu-ray player, USB drive, USB keyboard and mouse to connect with three TVs for ecosystem test (Table III).

There are more compatible issues on Brand A. First, some Blu-ray players reject Brand A standby CEC command; next, Brand A is unable to read USB storage with NTFS file system, that means Brand A can’t playback the file size which is larger than 4GB (HD movies are averagely more than 4GB). For Brand C, the basic storage function of USB hard disk and pen drive is normal, but we find that it is unable to detect MTP storage from specific DSC; that is, the USB connection with external devices still has problems to be overcome. Besides, Brand A & Brand C are unable to recognize generic USB keyboard and mouse which can increase the usability and satisfaction for operating smart TV. It is one critical user experience issue due to a large number of interactive user scenarios with smart TV.

2013-6-27 下午 04-38-41

 IV. User Experience 

i. Graphic User Interface (GUI) Analysis

We all know that user experience (UX) is not totally new for high-tech industry, but how to optimize the interaction and usability of smart TV will be the key success factors. Like the development of mobile devices, smart TVs seem to integrate all features through Internet, smartphone, tablet PC and TV; meanwhile, the content and function layouts on large size screen becomes very important. Learning the successful interface model of mobile device or developing the brand-new smart TV interface is a question for all vendors, but it’s no doubt that the true feelings and feedbacks of real users are able to strengthen the usability and satisfaction of visual, audio or even emotional reflection, and then to seize the initiative in smart TV market. According to the following Figure I, we can find that Brand A, Brand B and Brand C have extremely different UIs and GUIs.

2013-6-27 下午 04-39-16

Brand A’s UI is basically a link to its own cloud entertainment network “SEN”, and is somewhat similar to Windows 8 Modern UI style with box icons. The left side shows TV signal program, the central and right sides are the blocks of application icons, so users can organize their favorite applications. In fact, UI and GUI of Brand A is kind of “Simple Style”, no extra option or icon to interfere with user’s visual cognitive process.

Otherwise, Brand B’s UI is similar to today’s Android smartphone interface, especially the layout of applications, and GUI is more diverse and rich. The upper-left side is TV signal program, the central block is for Brand B’s customized applications, including learning, sports and etc., and the below side is the arrangement for other applications.

Brand C’s UI has two-layers design, and the first layer is designed similar to the bookmark mode which includes TV signal program, applications, search, and settings, etc.. The left side shoes TV signal program, and when users click on each bookmark, it will appear the second layer UI; meanwhile, the TV signal program is presented as the background. As same as Brand B, the overall UI of Brand C is similar to today’s smartphone, and the layer-design enables users classify different functions and options.

Although Brand B’s and Brand C’s UIs seem much fancier, but for users who have been accustomed to the smartphone interface, it should not cause too big burden on the switch of mental model. For users who never used smartphone; nevertheless, the designers have to implement the usability test and evaluation to understand the UI and GUI are easy to use/learn or not.

ii. Usability Test

From these three different UI and GUI designs on smart TVs, we can propose one critical question, “the interface design of smart TV should be Simple or Diverse?” It is worth noting that the TV screen is not only larger than samrtphone or computer, but also its user scenario is unlike other electronic devices which are operated by users in front of the screen. In fact, smart TV is such a lean-back medium which has an image of user relaxing on a couch with vegging out browsing and playing applications, so the design of UI and GUI should be considered the interactive process between user scenario and visual cognition. For this issue, we implement the usability test for Brand A, Brand B and Brand C to observe the effect of three interface designs with remote control on user experience and human-computer interaction (Table IV).

2013-6-27 下午 04-39-54

With the overall interface usability test, System Usability Scale (SUS) and Subjective User Satisfaction ( Satisfaction) questionnaire results, we find that Brand A’s simple interface design emphasizes clear and visible style, allowing users to understand the locations of all function intuitively, so we can find that the speed and quality of task completion are rather better. Conversely, Brand B’s diverse interface design is similar to Android system, and the GUI is more vivid. Although it will increase users’ visual loading at the first sight, most of them gradually are able to get started with the adaptation after one task learning. However, Brand C and Brand B have the similar style of interface, but Brand C’s fluency and performance between remote control & TV feedback is relatively slow, resulting in user’s learnability is much lower, and user’s efficiency is longer as well; also, the overall SUS and Satisfaction feedbacks are the worst among three TVs.

Let’s view from the GUI, Brand A’s simple interface design also means too boring for users to catch their eyes, and most users feel its interface is not innovative and smart enough. On the other hand, Brand B’s diverse interface design gets better users’ satisfaction and feedback, which indirectly means that users’ objective task performances are not equal to the subjective emotional feelings. Although Brand C’s UI design can catch user’s eyes, but its performance and fluency cannot keep up with the pace of intelligent interfaces, but drag the entire TV usage.

Such the result is able to reflect our previous question, “Simple” or “Diverse” interface design indeed have different effect for users, and we also bring out the "Performance" and "Interface" of smart TV are as important as the first question. The perfect combination of performance and interface can make users fully experience the advanced technology. Based on the above contradictory results, vendors, especially in the design of smart TV interface, must do more researches and evaluations on such trade-offs. What smart TV interface can make users feel easy-to-use and emotional satisfaction to achieve the golden cross? Think outside-the-box with unusual design concept should be focused in the future development. In addition, one psychologist said, “The positive emotion is the key for learning, curiosity and creative thinking; that is, happiness can extend the thinking to be improved more creative.” If the interface indeed attracts users initially, the human-computer interaction process will be regarded as a positive motivation for learning even though it’s kind of difficult to use.

iii. Smart Function, Application and Brower Performance Test

In order to understand the performance of three smart TVs, we make some basic check of their smart functions (1st initiate time required and follow-up initiate time required), applications (YouTube and Skype) and browser access abilities (Table V).

2013-6-27 下午 04-40-57

According to Table V, there is no significant difference for Brand A and Brand B on 1st initiate time required of smart function, but at the follow-up initiate time required, Brand A is almost 1.1 seconds slower than Brand B because it should re-link to SEN every time while initiating. And the UI of Brand C has two layers design, so the switch between the TV screen and smart function is relatively faster with the average of 2 seconds. In fact, when users operate smart TV, the user scenario is possibly between watching TV program and using smart function; thus, the interface switch of TV should be more smooth and faster, or it will result in users wait too long to lose patience. And for the usage of applications such as YouTube and Skype, there are also no significant differences in Brand A and Brand B, but Brand C should spend more than 20 seconds on initialing the applications.

Next, we particularly use PC, mobile device, Brand A, Brand B and Brand C to access to some world-renowned websites, and check the correctness of the opened page (Figure II). In this test, we discover three critical issues. One is that Brand A’s and Brand C’s networking are faster than Brand B’s while accessing to Amazon, but we observe some website functions or advertising icons cannot be fully displayed on Brand A’s and Brand C’s browsers. After comparing with the webpage displays on PC and Brand B, we can confirm that there may be some shortages of Brand A’s and Brand C’s HTML or webpage syntax errors.

2013-6-27 下午 04-41-19

The second issue is that Brand A’s opened page would automatically switch to the mobile browser mode while accessing to Brand B’s official website. According to Figure III, we observe that Brand A’s opened page is significantly different from PC’s, Brand B’s, Brand C’s and even mobile device’s views; besides, the whole opened page has some blocks shift happened. All above mean that Brand A’s networking capability still needs to be adjusted to allow users have the similar and familiar browsing experience and operational feedback as PC.

2013-6-27 下午 04-41-36

Last, we find that Brand A’s and Brand C’s smart TVs have no Flash Player plug-in, so some media content of the webpage can’t display completely. And Brand A’s opened page automatically switches to the mobile browser mode again (Figure IV).

2013-6-27 下午 04-41-51

Artificial Intelligence or Artificial Smart

The development of artificial intelligence creates a breakthrough of contemporary high-technology, and more and more electronic devices are attached the so-called “Smart” to strengthen their own capabilities, such as smartphones, smart TV and even smart homes. Nevertheless, all those high-tech devices may bring various kinds of conveniences for human life, but if users have never been able to operate them correctly and effortlessly, or even been dominated by them, to be Smart or not to be Smart, that is the question. So far, the representative of Smart cannot depart from Internet, and it seems like electronic devices with networking features can be unlimitedly regarded as Smart devices. Actually, the electronic devices with Internet have broken through the limitations of space; through critical years of development, they have also created numbers of applications to meet users’ requirements.

As mentioned previously, the successful development of smartphone has brought the revolution of high-tech device usage, and has enhanced better and better user experience and interface. If we want this successful model to be transferred to smart TV, we must deeply research TV’s user scenario and current technological development. After all, the traditional remote control for users is still for watching TV programs, but it is not the suitable interface for networking or browsing. Therefore, many vendors have developed lots of operational modes, including voice activated, touch screen, mouse/keyboard connected and motion control. All above would like to provide users with more operational modes, so that the processes of watching TV and using smart TV can be more flexible to increase uses’ sensational pleasure.

However, only hardware upgraded is not enough, how to make the seamless connection to other devices has also become an innovative solution. Especially, there are thousands of electronic devices in our daily life, so the convergence of user experience will be able to make users feel one device in hand and ubiquitous world around. Take smartphone as an example, we can use it to remote control or share files on TV. Due to the OS of smartphone and smart TV has become similar, and the innovation of applications and wireless network technologies such as Wi-Fi and DLNA , smartphone replaces remote control is expectable, and makes up for the lack of the usability of remote control. In addition, the development of smart TV cannot just focus on the technology, user experience, such as easy-to-use and easy-to-learn, and diversified content are rather essential as well. If the technological development seems to be smarter without user’s real feedback, the device is only “Artificial Smart”. Therefore, through all aspects of user experience researches are able to solve the issues of unfriendly and unfamiliar user interface, so that “Artificial Intelligence” is not a slogan, but an ideal human technology.

Form the point of view, the development of smart TV involves technology, interface and content. As an expert in the field of smart TV validation, Allion combines our testing experience with user experience research to provide this report. We can not only ensure the best technical quality, but also propose the most considerable suggestions greatly contributed to the best user experience. The report, which takes three vendors’ smart TV as examples, wants to provide users with the deeply thinking on smart TV competitive analysis. What to expect when we are expecting.

Back to Article List