Imagine you’re the Chief Marketing Officer of a fortune 100 company. Your team has just presented you with the results from a series of extensive market research deep-dive quantitative studies that cost well into seven figures. We’ll just say the goal of this research was to better understand a current key segment’s needs/wants, as well as that of another rapidly growing segment. The series involved tens of thousands of online interviews around the globe in strategic markets.
From sifting through the data, the team has strategic recommendations for each specific market, i.e., marketing campaigns, branding, messaging, etc. Implementing these recommendations will cost your marketing department nearly a quarter of their annual budget (hundreds of millions of dollars).
This is a huge decision but, as CMO, you trust the R&D team and the outside partners you’ve worked with for years. And, ultimately, you trust the data that has been collected, and feel you should follow through with overseeing the implementation of the recommendations.
But what if you knew that well over half your studies’ respondents belong to 3+ panels/platforms and are considered ‘professional respondents?’ Would that make you question whether those respondents were truly representative of the population you wanted to get insights from? Furthermore, what if you learned that the industry standard for fraud or bad respondents make up around 30-40 percent of completes collected from panels? Would that make you reconsider what you’re willing to invest in based on the data? What if your results are off?
According to a recent Coalition for Advancing Sampling Excellence (CASE) study that measured 10s of thousands of respondents from the top 20 panels in the industry:
- 4 in 10 respondents who entered a survey did not complete it
- Vendors had supplication rates anywhere from 4-24 percent
- 1 out of 4 respondents failed trap questions
- 30-40 percent of completes collected were deemed fraud/poor quality
The granular details of panel, data collection, and data quality measures implemented by panels can be a bit of a black box for many corporate researchers. CASE addresses exactly that. Per their website, CASE was created when an insights professional at Kraft Heinz shared his frustration at his inability to receive valid and reliable findings with Mary Beth Weber, a veteran corporate researcher. It occurred to Mary that users of standard research/intelligence seldom asked questions about the accuracy or quality of the data they were analyzing. As an oversight organization, CASE ensures quality for client-side researchers, and requires transparency and accountability from their partners.
Ultimately, the decline in sample quality has resulted in a lack of trust in quantitative findings by the end client. In response, clients are investing more in qualitative research, investing in their own audiences by building their own panels, and looking into alternative sample sources, among other actions.
How we got here isn’t salient for this discussion, but for those in research agencies, it is important to understand what you can do to mitigate fraud/bad respondents.
Alternative sample sources
Companies like Rakuten, Walmart, Amazon, Facebook and LinkedIn have immense amounts of 1st party data and have direct relationships with validated individuals (real people). They’re now entering the market research space with research solutions. Some of them have already begun rolling out their sample offering.
Additionally, there are several types of alternative sources ResponsivMR has recently partnered with. Some whose primary business is not sample or market research. They already have their own audiences, i.e., banking customers, loyalty networks, social media groups, other popular online communities, etc., and they see the benefit of providing those audiences with opportunities to take part in market research. In general, most of these “new” players will be more expensive than the current samples the industry currently relies on. That said, the ESOMAR conference recently stated that CPIs will have to go up regardless in the coming years.
At ResponsivMR, we’ve sought out and built partnerships with alternative sample sources. Often, these sources are focused on niche audiences (certain B2B targets, Gen Z, Gamers, etc.) we know, based on their business + engagement models +collected data points, will yield higher-quality respondents than traditional panels. When a client needs one of these niche audiences, we’re quick to recommend them.
Vetting current providers practices
This section isn’t meant to put current sample providers on blast. They care, they have researchers working internally to mitigate fraud. There are simply bad actors out there who will look to exploit systems/platforms for their own gain.
Understanding what your current providers are doing to prevent fraud/bad respondents is paramount. CASE came up with a series of questions asking pointed questions about what they do to prevent contamination from bad actors. These questions ask for granular details about device participation, dashboards, duplication on mobile devices, AI detection, etc.
The questions are likely familiar to your suppliers, so you’ll be able to glean a lot from their answers. How thoroughly did they answer the questions posed? How transparent are they? Does it sound like they’re really invested in preventing fraud and looking toward the future? How do their practices compare/differ from other partners’ answers?
At ResponsivMR, we send these questions to every partner we work with, and review the answers internally, as well as comparing with others. We work together with our sample partners and have our own proprietary software that scores each respondent from 0 – 100 (0 being no detection of fraud; 100 being fraudulent). For example, when we detect someone is using a suspicious IP address or a VPN, their score goes up. When their mouse movements seem like that of a bot, they get more points added. We can detect for other variables, as well. Our capabilities include detection at the front end, preventing people who have a higher score from entering the survey, and/or on the backend once the survey is complete.
We share these scores with our partners and measure the quality we get from them over time. When the scores get above a certain level, we address them with our partners and make a plan to get back on track. If issues persist, we let them know we must look elsewhere until the quality is properly addressed.
Making your own panel
We help many clients recruit, panelize and keep their respondents active. Creating your own panel is an expensive undertaking, especially at the onset, and can be taxing to maintain over time. Panel-builds make the most sense when there is a need to get frequent insights from a particular audience that is very important to a client. Benefits include, targeting relevant individuals rather than having to go through gen pop sampling, cost efficiency over time, faster turnaround, longitudinal studies, better data quality, brand loyalty and engagement, ease in executing iterative studies and enhanced control, i.e., implementing engagement/retention strategies, autonomy over design and implementation of interviews.
Behavioral data
Targeting respondents via behavioral data is something we’ve been tapping into more and more lately. This allows you to observe respondents’ behavior to confirm if they’re real or frauds. Behavioral data allows you to see past digital activity and verify users are doing X, Y and Z. For example, if you’re looking for people who’ve downloaded and use certain apps, watch certain content on YouTube or DisneyPlus, physically go to X Y and Z store, or have shopped in certain categories at Walmart or Amazon, among many other use cases. There are some targets where it makes so much sense to use behavioral data: fast food restaurant goers, online shoppers, gamers, etc. In many cases, their verified behavior qualifies them at the jump. So, why ask the questions if you already know? Because this can mitigate the recall gap (add hyperlink). The CPIs are higher than most traditional panel CPIs, but the data is better.
Gamification of Surveys
Respondent fatigue is real and can have an adverse effect on data quality. If your audience isn’t really engaged while taking your survey, can you expect them to give you their best, honest answers? To combat respondent fatigue and improve engagement, some researchers are incorporating gamification elements into surveys. This approach makes the data collection process more interactive and enjoyable, leading to higher response rates and more accurate data. We always review our clients’ surveys, looking for problematic areas, and for ways to optimize the survey. We then propose edits to make the survey experience more engaging for respondents, etc.
Looking to the future
AI Analysis:
AI is a powerful tool that will continue to shape how we analyze data, in addition to the many other facets we use to conduct market research. AI can be trained not only to look for patterns and analyze data. It can also allow for significant improvements in analysis of responses to eliminate non-human responses. This will take time, but companies in the industry are already digging deep to find the right iteration and right solution.
Identity validation:
Know your customer (KYC) approaches are requirements for certain kinds of research. However, that rigor is not there for most commercial market research. These approaches use biometric, and AI based identity verification solutions which offer secure and efficient means of confirming respondent identities in real time.
Blockchain for Data Integrity:
Blockchain technology is emerging as a potential solution for enhancing data integrity in sampling. By providing a secure and transparent way to record and verify responses, blockchain can help prevent data fraud and ensure the authenticity of sample data.
If you’re interested in learning more about acquiring high-quality data, techniques you can employ, exploring alternative sources or how to create a sample procurement strategy (overall or for a specific ad hoc project), I’d love to chat with you. Please email me at cduffey@responsivmr.com
Connor Duffey
Vice President, Client Services