SMS Messages Filtering

SMS filtering experiment to help users filter spam, ads, and verification messages.

Timeline

2020 Nov - 2021 May

My Role

Product Designer

Responsibility

UX & UI design, User Survey

Overview

Objective

The team wanted to design a filtering feature for SMS in the Whoscall app. We implemented an experiment to observe user reactions and use it as a basis for optimization.

Impact

We launched MVP and opened it to 15% of users. This feature officially incorporate into the Whoscall premium after experiment completed. At the same time, it also increased 10% for the user report rate.

Role

I worked closely with PM, two engineers, and a QA engineer. I was responsible for the entire UX and UI design, from defining the problem to deliver the final mockups.

Challenges

This feature has many functional dependencies and scope limitations with other development teams. We need to communicate and define priorities to ensure that the MVP can be completed.

Background

Objective

Our company expected to release a "category" feature for Whoscall app to make the SMS message list cleaner than its previous, chronological list.
At the same time, our technical team trained a ML model for text recognition, hoping to apply this technology to Whoscall app. We decided to apply this technique to this feature.

SMS Photos
Timeline

We started the experiment with the MVP product and delivered to the business team within six months. We decided to experiment in 2 stages - First identify only spam, and then move to multiple categories in the second stage.

Process

1 Define Problems and hypothesis.
2 Usabiltiy testing and soluction.
3 Launch the stage 1 experiment.
4 Questionnaire survey and tracking.
5 Optimized to stage 2 MVP.
6 Deliver to our business & product team.
Define Problems

After interviewing our stakeholders and reviewing the questionnaires we received in the past, we found two problems:


1. Users thought that the existing SMS list is too messy, and most of them are useless.

In recent years, SMS had been used as one of the marketing methods. Users often received advertisements, promotions, and even malicious messages. This made users felt that is very cluttered and cannot found important messages.

SMS UI

2. Users cannot distinguish scams and malicious messages.

When users received too many unfamiliar messages, they will found it was difficult to distinguish scams, malicious messages, and felt distrust or anxiety about it. Some elders were even defrauded.

Goal & Hypothesis

We wanted to make a product that can actually be launched for experiments and can be flexibly expanded for future needs. Our team discussed the goals and hypotheses. After defining these, we ideated the features and arranged the priority.

Goal

Help users filter spam and malicious SMS messages.

Hypothesis
1

By marking the malicious message on the SMS dialog when the message was received, we can help users stay away from the malicious message.

2

By providing categories in the SMS list, the interface can be made more clear to help users find important or needed messages more easily.

SMS Features

we used the user journey map to ideated the feature. Then arranged the stages and backlogs according to the importance.

Usabiltiy Testing

We spent 3 days completing the usability test from 5 people by prototyping to knew how users perceive our product.

Testing Goals and What We Learned

- The user's perception of classification.
- When the list arrangement and usage method were changed, can users understand and used it normally?

Usability test

We collected the insights during the testing and then colored the cognitively incorrect and uncertain content to help the team better understand the user's feelings.

SMS UI
Stage 1 MVP

After several rounds of discussions, we determined what the MVP delivered in the first stage. This function was called "SMS Assistant", which mainly detects malicious messages and warns users.

Usability test

1. Automatically help identify spam SMS.

This new experience starts with the dialog of receiving SMS messages. When a user receives a message, it will be recognized through our ML model and warns the user when it is recognized as spam. We have defined the criteria for spam: fraudulent, advertising, or loan messages.

Usability test

Because the previous function has designed the dialog style of the link warning, we decided to use the existing components at this stage.


2. More clearer SMS list.

The messages will be divided into general messages or spam messages in the SMS logs, so users can focus on useful messages and avoid fraud.
We designed categories to be presented in tags, which can be more flexible in the future.

Usability test

We have also designed a moving function that allows users to move out the messages that they think it is not spam. This also help to optimize the accuracy of our ML model.

Questionnaire Survey & Tracking

One month after the first stage of the MVP was released to testing users, we conducted the questionnaire survey and data tracking to understand user satisfaction and suggestions for features.

Goals
Know the Functional Satisfaction

We sent a questionnaire to testing users to understand their satisfaction and the perception of the accuracy of identification.

Understand User Behavior

Tracking data to understand function usage. Such as rate of usage, recognition results, rate of and behavior of moving category, etc.

What We Learned

There are more than 12,000 users in this experiment, which is about 15% of the premium users in Taiwan. 80% are men, and the main age group is 31-50 years old.

1Each user received approximately 35 ~ 45 text messages. 35.25% of SMS were identified as spam.

3The overall function CSAT is 85%. They are satisfied because they "can know what is spam."

27% users have used the moving function, and 96% of the messages moved are spam.

4Users are not satisfied in: Not enough classifications, Wrong recognition.

Survey Feedbacks
Optimized to stage 2 MVP

After completing the questionnaire and data analysis, we sorted out some important items and considered the business model to define the new goals and design.

Goal

Helps to automatically classify SMS, so that users can manage the messages more effectively.

Hypothesis
1

Through the automatic classification of SMS, users can more easily understand the importance of messages.

2

Providing multiple categories can make the SMS list cleaner, which can improve the efficiency of checking or finding messages.

Stage 2 Design
Usability test

1. Automatically classify SMS into: General, Transactions, Promotions, Spam.

We will use the ML model to automatically classify SMS messages. The user can understand the security and type of the message through the category tag.

Usability test

Because the previous function has designed the dialog style of the link warning, we decided to use the existing components at this stage.


2. Make the classification more useful.

We added transaction messages and promotional messages, which are also consistent with iOS.
In addition, in order to reduce users’ unfamiliarity with the new interface, we have added the "All" category, where you can view all SMS in chronological list.

Usability test
Usability test

There is still a moving feature that allows users to switch messages to another category they want.

Outcome

Commercial Monetization

After completing this MVP, the commercial team hopes to officially incorporate this feature into the Whoscall premium. We also assisted in handing overall data and backlogs to the commercial team and ending this task.

95% Retention Rate

Among testing users, 95% are willing to continue using this SMS Assistant, which makes the team feel that the feature is beneficial to users.

+10% User Report Rate

This feature has added a moving and report function, although it was not the expected goal, we also increased the user report rate by 10%, which also helped the technical team to increase the amount of data.

1After adding the new categories, the proportion of overall moving dropped. Spam is still the majority of moved messages.

2The category most frequently used is General rather than All, which may mean that the classified list is useful.

Next Project

Smart Scooter Rental

Design scooter rental to make it more eco-friendly and convenient.