[Watch the Webinar: Are you really taking care of QoE in your mobile network?]
When studying rankings of mobile operators, you often see a map of the United States with 4G/LTE and 5G coverage claims. Network operators often tout maximum download and upload speeds, the reliability and availability of offered services, and data transport performance. These technical parameters describing quality of service (QoS) are the bedrock of a reliable mobile service, as a well-performing network is the basis for subscriber satisfaction.
But are they enough to reduce subscriber churn and to improve customer loyalty?
In the age of increasing data rate requirements from third-party media services such as Facebook and WhatsApp, and live video streaming on YouTube, it is imperative that large-scale benchmarking campaigns examine and weigh not only QoS but also the end user’s quality of experience (QoE).
A campaign to evaluate QoE in real-time field measurements requires a different setup than the typical before-and-after analysis of dedicated technical parameters and campaign key performance indicators (KPI).
Measurement campaigns based on QoE need:
- a comprehensive set of emulated use cases and QoE-centered KPIs that are supported by the measurement equipment
- a user profile based on real human user needs
- measurements that include all places and venues where subscribers typically use mobile services
In general, QoE-centered KPIs and measures are values that reflect what a user really perceives. One example is the mean opinion score (MOS) for voice and video quality. MOS is a perceptual measure. For voice, it indicates the naturalness of transmitted speech; for video, it expresses how a viewer rates the visual quality of a video.
Apart from MOS values, in telephony it is fundamental to rate the call setup time, i.e. the overall length of time required to establish a call and start a conversation. For video, this would be the time between requesting a video from a service and actually seeing it play.
For voice and video, QoE measures appear obvious. Measures describing the QoE of data or messaging services are also defined in our test solutions. These include the download time of highly popular web pages or files, such as new applications, and the time it takes to deliver messages or post pictures on a social media timeline.
Compared to plain technical metrics that linearly describe a technical value, such as dB or Mbit/s, measures quantifying quality of experience are non-linear. They are bounded by saturation areas where perception no longer changes.
With this in mind, it becomes clear that is not enough to test a network’s core services, such as telephony, SMS and data throughput. Popular media services, such as Facebook, WhatsApp and YouTube, also need to be considered since customers are increasingly using these types of services in their everyday routines.
Transparency, Fairness, Comparative Results
How can service providers guarantee transparency and fairness and obtain comparative results? On the one hand, they must evaluate all available networks under the same conditions. This means all operators need to be measured simultaneously, at the same location and under the same test conditions. On the other hand, to perform extensive drive and walk tests within a very short time frame, they need equipment that is highly reliable, up and running quickly and capable of continuously collecting masses of data for hours.
Beyond the reliability of the equipment, mobile benchmarking defines its own set of additional requirements. The basic idea is to compare different networks on a fair and equal basis. Fairness entails the discerning selection of tests, transparent descriptions of all test cases and the preferred use of standardized KPIs and metrics. Moreover, to guarantee congruent conditions across all network, all test cases must be the same and conducted synchronously.
Commercial Phones Measure Quality of Experience
Regardless of whether driving or walking, all of our measurement solutions use commercially available smartphones as measurement probes to ensure equivalent test procedures and results – and to be as close as possible to what end users perceive. To compare networks on a fair basis, the smartphones have to be state-of-the-art and support all the latest technologies and services.
Benchmarking, especially when done for public consumption and focused on QoE-metrics, requires bulletproof data. This is ensured by the consistent use of standardized measuring procedures wherever applicable.
Along with standardized procedures, full transparency is indispensable. Benchmarked operators have to know what was measured, how it was rated and how the final ranking was calculated. A clear rating procedure based on statistically significant amounts of data obtained in a transparent manner makes the ranking easy to understand and comprehend.
Measuring QoE in the field is a challenging task that requires specific knowledge and purpose-built test equipment. With continuing network densification and a blended RAN approach that leverages many different types of technologies for outdoors and indoor use, a small and flexible solution that is easy to use and can adapt to many different measurement environments and technologies is ideal.
The data collection tools have to ensure that the data is accurate, precise and reliable and that the measurement results can be repeated, validated and used for statistical purposes. This ensures that the test solution itself does not have an impact on the network measurement results.
Many factors influence accuracy and reliability of data acquisition and data analytics. To review how correct measurement equipment and techniques can enable the efficient delivery of better services and higher quality to customers, watch the Rohde & Schwarz webinar titled, “Are you really taking care of QoE in your mobile network.”