I often use segmented quotas in my surveys, as the sample is collected via online panels. It is often important to accurately count the number of quota fails in a survey, as this can have an impact on the cost per interview.
I have noticed that there is no sure-fire way of determining this, as cases are simply marked as disqualified. I can infer that they were quota failed by checking that they were not disqualified for any other reason. However, this is not a guarantee, and could mask a problem that is causing respondents to be disqualified unintentionally.
Is there something I’m missing, or is this not an available feature? If not, how have other users side-stepped this problem?
Thanks in advance,
I generally create a raw-data export including the Disqualified responses. – I can set up filters on the export that match my segmented quota rules, but I’m not sure if that is what you are after?
Can you maybe elaborate on what this would look like ideally for you?
Hi Dominic. Cheers for your reply.
Your solution would allow me to see statistics for quotas in my data export, but not the number of quota fails. What I mean by this is those respondents who screened into the survey, but were assigned to a quota that is already full, and subsequently disqualified from the survey. In the SPSS data export, there is no way to differentiate between these respondents, and those who were screened out for not matching the requirements of the target audience (for example, being the wrong age, or not engaging in the activity we’re interested in, etc.). All of these respondents are marked as ‘disqualified’. As I mentioned above, it is possible to check their other responses in SPSS to determine why they were marked as disqualified, but this is not pool-proof.
Here’s an example… A survey is targeted towards gamers aged 18+, and uses segmented quotas based on age and gender. A respondent that is under 18 would be screened out and marked as disqualified, as would a respondent who is not a gamer. A respondent who is a gamer aged over 18, but who matches the profile of a quota group that is already full, would also be marked as disqualified. In the dataset, I can use this information to infer that those respondents aged 18+ who are gamers and have still been disqualified must have been quota fails. However, the vast majority of my surveys are not this simple, and as the complexity of the survey increases the chance of bugs increases. I may assume that a respondent was disqualified because their quota was full, but in actual fact there is a bug in my survey that means this respondent was incorrectly removed.
As far as I can tell, there are no solutions to this problem, but I wondered whether any other community members had experienced this, and if so, whether they had devised a workaround.
Alternatively, I can think of two fixes that SurveyGizmo could implement: mark quota fails as quota fails, not disqualified; or give users the ability to determine the point in their survey at which quotas are accomplished, giving me the ability to store a hidden value to identify these individuals.