The problem we didn’t expect while testing our French newsletter
Hi! Charlotte and Élise here!
We recently learned an interesting lesson about assuming we know the problems of our users, just because we’re so close to our products. Turns out, that isn’t necessarily the case. Here are some diary entries about that helpful learning experience.
November 18, 2019
I’ve been noticing a problem with one of our newsletters.
We have two mailing lists that people can subscribe to: one in English and one in French. We’ve been using analytics to improve our performance every month, and seeing a lot of success with our English newsletter. Subscriptions for that one have been going way up! The problem is, we aren’t seeing the same steady growth with our French newsletter.
|
I’m happy things are resonating with our English audience, but I wonder why it’s not resonating with our French audience in the same way. We want the newsletter to have the same value for people regardless of what language they’re reading it in. I met with Élise today to talk about how we can figure out a way to improve the uptake of our French newsletter. We’re assuming the reason readership is low is because we’re missing the mark with our content and how we’re presenting it in French. Afterall, content is king, right? |
|
We decided we’re going to launch into content testing to fix this. We’ll sit down with French subscribers and have them go through the newsletters with us to get a better understanding of what is clear, what is helpful, and what is neither of those things.
We set up a meeting with one of our design researchers, Adrianne, to help us flesh out this plan. More to come!
- Charlotte
November 26, 2019
|
We’re pivoting a little after a great chat with Adrianne. She suggested we may want to test our assumption that content is the problem before we leap into fixing it. Her advice was to get more data on what the problem is from actual subscribers. Then we can choose what the best way is to tackle that problem. She said it could very well be content, but let’s let actual people using the service tell us that so we’re not trying to solve a problem that might not be a problem. |
Based on that helpful advice, Élise and I reconvened today to discuss a new plan of attack and to update our research plan. We talked about who we want to target for testing, came up with survey questions for each audience, and then put together a recruitment plan to ensure we get enough data/responses.
We’re pretty confident we know what the responses will be (hello, content!), and that it will lead us to come up with a content testing plan. But Adrianne’s totally right: better to have the data to point to!
- Charlotte
November 29, 2019
Charlotte and I took advantage of the last few days to do prep work. I finalized the survey questions, and Charlotte created the outline of the next newsletter. I think we have all the puzzle pieces lined up now!
|
First, we have a four-question survey that we’ll put into the French newsletter, with questions like “What makes you want to open a newsletter?” and “Do you usually read digital news in French or English?” Then we have a one-question survey for the English newsletter, asking francophones why they subscribed to the English mailing list and not the French one. The target audience in both cases is Francophone readers to help us understand what’s wrong with our French newsletter. |
|
Since there are fewer potential respondents in French (240 subscribers compared to 2,800 English subscribers), Charlotte had the idea of sending a targeted “We miss you” email blast (eblast) to people who, according to our MailChimp data, almost never open the newsletter. I’ll write a short paragraph inviting them to help us improve the French newsletter. We will include the survey and send it out a week before the monthly newsletter. The hope is we’ll get feedback from people who actively engage with the newsletter as well as people who subscribed but no longer open our emails.
I can’t wait to see what we learn.
- Élise
December 10, 2019
The “We miss you” eblast and subsequent newsletters have been built and sent out!
|
Five days ago we sent out the eblast with our survey to the less engaged French subscribers. We got 16 responses so far from a group of people who, data tells us, don’t typically engage with our emails. Not too shabby. Then today we sent out the two newsletters, English and French, with their respective surveys. We’re going to give people a few days to fill them out. I can’t wait to check out the responses! - Charlotte |
December 20, 2019
Today was a day of surprises.
Charlotte and I analyzed the survey responses. While both surveys offered great feedback, the most surprising insights came from the Francophones subscribed to the English newsletter. (This was the group we sent the one-question survey to, asking them why they subscribed to the English newsletter and not the French.) Here’s what we discovered:
a) 41% of them didn’t actually know there was a French newsletter.
b) 20% of them find French translations in government to be poor or prefer to read text in the original language it’s written.
c) 11% of them read in English because they can more easily share the content with their English-speaking colleagues.
Wow. There goes our theory of the content being the problem. Charlotte and I definitely didn’t expect that so many people were just not aware of the French newsletter! It’s a much smaller problem to tackle than testing and reworking the content. This is going to save us, what would have been, a lot of time spent on the wrong thing.
While we’ll continue to try and tackle the problem areas in the long-term, we developed some quick, short term ways to address the survey insights.
|
|
Moral of the story: never assume we know the answer to our audience’s problems, because we may not even know their real problems - only they do!
- Élise