Hello, I’ve been trying to get in touch with SG support for a few days now via live chat and email, to no avail. I really need a more thorough explanation of the “not enough data” error that can occur with conjoint survey data. I see the website gives a brief explanation of this, but it’s quite vague and I’m not 100% clear on it.
First off, it notes that this error “can happen if, in your entire data set, there is a card that was never selected” – but what exactly does that mean? Does the error occur whenever there’s a card that’s never selected by any respondent as preferred (or best/worst, depending on the questioning approach used) among the cards shown? Or does it occur whenever there’s a certain configuration that is never presented as one of the choice options over the course of the study? I cannot imagine it would be the former, because it will almost always be the case that certain options would never be selected – there will always be the random card that offers the fewest/worst features at the highest price, which will never get picked, in which case we’d get that error every time, no matter how many respondents we have.
Having this error occur when a certain combo/card isn’t generated as an option throughout the entire survey isn’t ideal either, since even for a relatively small attribute matrix it would take an unrealistic # of respondents/choice-sets to stand a fair chance of having every combination generated – but at least it would be somewhat less bleak than the former.
And when you say it “can happen” – does that mean it will always occur in this scenario, or just sometimes (and if the latter, when/why)? I’m trying to understand under what conditions I should anticipate the error occurring or not.
I’m also trying to figure out if anything can be done to work-around this error. From the test data I’ve generated so far, I see that this error affects both the on-screen version of the report and the Excel export. There is no attribute or level utility data included in the export in this case. So am I completely stuck? It looks like I can at least still export the raw conjoint data – the choice sets shown/selected. Is my only option to find a way (in my own statistical tool) to compute the part-worth utilities from that raw data? I have no idea how to do that, so that would be a real bummer – but if that’s my only recourse in cases where this error occurs, at least it will be good to know and I can start trying to figure it out.
I would appreciate any guidance with this issue as soon as you’re able. I was excited about the conjoint module available in SurveyGizmo, and that was one of the deciding factors for me to purchase SurveyGizmo over some other options – but this is a fairly severe limitation that really hurts, that I hadn’t anticipated.
Thanks in advance for any help you can provide.
I apologize for the trouble contacting SurveyGizmo Support, and I would be happy to help give an answer to this question.
First, let me give some background on this error. SurveyGizmo uses an open source statistical library called ChoiceModelR to calculate conjoint part-worths or utilities. When beginning to process your conjoint observations (your raw data), ChoiceModelR will generate some meta-data, including the number of times each choice was selected.
When I say “choice” here, I am referring to a “slot” where a card will be inserted. It does not matter what the actual combination of attributes is. In the image below, I have a Choice-Based Conjoint question with default settings, which is to show 3 cards per set, with a “None” option. Each of the three cards, plus the “None” option, are “choices”.
Again, regardless of what actual combination of attributes gets generated, the first choice is always “choice #1”, as far as ChoiceModelR is concerned. In our documentation when we refer to a “card” that was not selected, this is what is being referred to – a “choice”, or “slot”.
So, when ChoiceModelR begins to generate this meta-data, it will scan through your conjoint observations and count the number of times each choice was selected. There is a generic error (‘Error in ytab[1, i] : subscript out of bounds’) that will occur if it tries to count one of the choices that was never selected (for example, if no-one ever happened to select the card that appeared as Choice #1 for them). When this occurs, you will see the “Not enough data” error message – perhaps too simplistic of an error message, but it was rather difficult to convey the actual meaning succinctly.
(As an aside, in a properly designed Conjoint experiment with enough data to be statistically relevant, it is very, very unlikely this will happen, due to each “choice” being random. I believe this is why the error exists in the ChoiceModelR library — the creators assumed you would be processing a large enough data-set, hence the error message “Not enough data”).
I hope this helps to answer your question, and best of luck in your data collection adventures!