Kara McCroskey
End to end research for the full implementation of push notifications at Express Scripts
My Role: Researcher
​
Note: As a UX researcher at Express Scripts, I performed all parts of the research process: research planning, writing protocols, conducting research, data collection, data synthesis, presenting, and making recommendations to stakeholders. In my work, I apply myriad methodologies including, but not limited to: moderated in-depth interviews, contextual inquiry, diary studies, usability testing, surveys, RITE testing, tree tests, and card sorts.
​
The business team made the decision to implement push notifications in the native mobile app product; I was tasked with doing research for the initiative. Several business partners had expressed interest in launching various campaigns via push, so it seemed vital to conduct a full body of research for a successful implementation of the new communication channel. I partnered with a project manager on our Omnicomm team for parts of this work.
​
My Approach
Secondary & Generative research
Since the push notification channel was a brand new initiative, there was no previous internal research done on which to build. So I started by conducting secondary and generative research. I did a lit review of about 15 reputable, cited sources, to get a better understanding of UI trends, best practices, and some statistical data around users in the current push notification space. Below are a few of my significant findings:
-
Data showed that the primary reason digital users turned off push notifications is because they find them irrelevant or annoying
-
Although it's not the trend to include terms like "tap", they help identify notifications that deep-link into the app
-
Timing and frequency play a huge role in whether users opt in or out of push notifications; more than 1/day cause majority to turn them off​
​
​I then conducted generative research to better understand individual communication preferences, habits, and pain points. I recruited 12 people who were a mix of ESI members and members of other health insurance, who managed at least 3 long-term medications for structured 1:1 in-depth interviews. While the study was generative in nature (and therefore open-ended in what we could potentially learn) I hoped to gain insights into whether our members would truly use push notifications, as well as insights into how to best optimize the new channel.
I began the interview by asking participants to tell me about how they managed their prescription medications, asking questions specifically around how they received communications about their medications from their current pharmacy and healthcare provider. I was intentional in listening for what worked and didn't work about their current prescription management and communication methods.
I then used an interactive visual component - a collaborative figjam board (pictured below). On the board, I included:
-
several different (prescription- and benefit-related) notifications
-
icons representing varied communication channels, settings, frequency, and more
I shared the figjam board with the participant, and asked a more focused series of questions. For each notification, I moved the representative icons to reflect their answers. Below are some of my focused questions, though I allowed participants freedom to elaborate on their answers.
1. Through what communication channel would you want to receive the notification?
2. What would be your preference for a secondary channel (if required)?
3. How urgent do you regard the notification?
4. How would you categorize this notification?
​

Glimpse of a completed interactive interview figjam board.
Once I completed interviews, I copied the transcripts from the interviews (automatically transcribed in Userzoom) to my working figma file. Rather than using individual notes and affinity mapping, I color coded data according to common themes. Once I completed the coding, I organized the themes so that the team could better grasp which communications our members might want via push notification vs text, email, or phone call, user expectations around frequency, default states, and more.
Synthesizing raw data in a qualitative study

From my generative research, I learned several note-worthy insights including:
-
People consider notifications about medication orders, account alerts, and the like to be urgent, and therefore want to receive them via a disruptive channel (text, push notification) and, depending on the specific alert, via email as a back-up channel. Other notifications (vaccine updates, benefit updates) are primarily considered non-urgent, and therefore preferred to be received via a non-urgent channel.
-
People prefer receiving urgent notifications via text to push notification because push notifications a text remains on one's phone.
-
People largely didn't want sales-oriented notifications e.g: "switch to home-delivery and save money" and news-oriented notifications e.g: "new covid vaccine available through ESI pharmacy"; such notifications could cause members to turn off notifications altogether.
-
In terms of number of categories, the "sweet spot" was between 4 and 7 in order avoid cognitive overload.
Card Sort
The findings from the generative interviews gave us more clarity on members' expectations, needs, and behaviors around prescription and benefit communications. Next I engaged my content partners to create appropriate category labels for the push notification categories in the communication preferences screen. We wanted to test two versions - one with more (7) categories and one with fewer (5) categories, so I had my content partners develop two versions of final category labels; then I created and conducted a closed card sort for each version, with 75 participants per card sort.

Userzoom has built-in analytics to produce a results grid from a card sort. A results grid illuminates notifications that were easy for members to categorize (and clearly "belong" to a single category) vs those notifications that members placed in multiple categories by members (thus are more ambiguous to categorize.) Pictured is a results grid from one of the card sorts. The higher the percentage (darker blue) indicated for a notification, the more frequently it was categorized in the corresponding category.

Significant note-worthy insights from the card sort included:
-
There were themes in terms of how users categorized some notifications, giving insight into mental models
-
Rx-related notifications were challenging for users to categorize when presented with 7 categories, because of the similarity of "Order Status" and "Order Alerts"
​​
Usability Testing
In addition to understanding how users thought about our notifications (topically), we wanted to evaluate how members would manage and interact with push notifications we wanted to launch, so I conducted usability testing on a prototype and tested with 12 members. The prototype was developed by my designer with my input.
For one task, I had participants select their communication preferences, including selecting the push notification categories they wanted to be on. Upon observing this, I asked them about their reasoning behind their choices. I also followed up with: Which categories (pictured) would you turn on, vs which would you expect to be defaulted to "on"? I did this for both the pictured version and a version with only 5 categories, in order to refine our understanding of members' mental models of which notifications they deemed important and which they did not.
​
​
​
​
​
​
​
​
​
​
​
​
We also included in the prototype several interactive push notifications, applying varying language in the CTAs, in order to observe whether certain language better helped members identify actionable notifications (that deep-link into the app).
From my usability testing, some of my note-worthy insights were:​
-
Users varied in terms of how many, and which notifications they wanted to receive
-
Without"Tap" to learn more, or similar language, some testers did not assume push notifications deep-linked into the app
-
Adding push notifications to our communication preferences screen created some confusion for testers in terms of navigation
​
​
​
​
​
​
​
​
​
​
​
​
​​
​


While test participants claimed to understand what they were doing when making selections for Preferred Communication Method and Notifications on separate screens; however, during follow-up questioning, it became clear that not everyone understood that they had selected push as a secondary communication channel.
Below are screens from my presentation on my evaluative research.




In closing, collaborating with my Omnicomm partner, we provided the larger organization with actionable recommendations on myriad aspects of launching "push", including:
-
specific push notifications to launch, from most highly recommended / important to least recommended / important
-
placement of requesting member to allow push notifications from the ESI app
-
the appropriate number of categories to provide in communications preferences
-
default state of push notification categories in the communication preferences
-
communication preferences screen rework
-
timing of push notifications
-
most effective and / or appropriate CTA language
​
Significant Actionable Insights
In summary, among my most note-worthy actionable insights gathered throughout the phases of research I conducted were:
1. People want to be able to opt in/out of categories of push notifications on a granular level; the more granular category we can offer, the better.
Noting this, in terms of cognitive load, 7 categories is the maximum number of categories users want to manage (turn on or off); creating only 4 categories does not allow ESI to provide the amount of granularity people want.
Thus, my recommendation: ship MVP1 with 5 categories, so that there is room to add more categories when future business campaigns surface.
2. Users, by far, prefer receiving urgent notifications via text to push notification because push notifications are perceived to "go away" whereas a text remains on one's phone. (This was my hypothesis prior to testing).
My recommendation: send urgent alerts to members via text, and email (as a secondary channel). Note: While this represented the voice of the customer, it presented challenges for my business partners because text messages are much more expensive to employ than push notifications.
3. Impersonal, sales-oriented notifications e.g: "switch to home-delivery and save money" and news-oriented notifications e.g: "new covid vaccine available through ESI pharmacy" were perceived as annoying by testers, and would likely cause members to turn off notifications altogether. (Also a validated hypothesis).
My recommendation: These should be lowest "tier" category notifications to launch; default correlating categories to "off" if they must be sent.
4. Adding push notifications to our communication preferences presents information architecture issues.
My recommendation: re-think communication preferences screen design.