I'd just like someone to explain why 23andme can't tell people what their genes put them at risk for.
— Cocky McSwagsalot (@MoreAndAgain) December 6, 2013
Here was my initial answer:
@MoreAndAgain the problem is that they literally can't, not w/certainty. Errors in sequencing, errors in software, complexity of data.
— Wandering Scientist (@wandsci) December 6, 2013
And then we discussed some more from there. This discussion was mightily hampered by the fact that I have been, as I admitted in my tweets, an only mildly interested bystander to the entire story.
My basic position- which again, I admit is not as informed as it could be- is that I'd like to see the personal genomics industry succeed, but that I think they need to be subject to some sort of regulatory oversight. My reason for that second part is that I, someone with a PhD in biochemistry who has worked in genomics, do not feel qualified to evaluate 23andMe's claims, and want there to be an impartial third party who checks that their claims are legitimate. I do not want to do this myself, because doing so would be enormously time-consuming, even for people who have the necessary scientific and technical background to assess their claims.
The usual response to this is that if I want help understanding my genetic data, I can go to my doctor. I see two problems with this position:
1. I know from multiple conversations I have had with my own doctors (all of whom I like, respect, and think of as really good doctors) that I am generally more well-versed on genetics and biochemistry than they are. This is not surprising, given the details of our educational backgrounds. However, it means that I cannot generally expect my doctor to tell me anything more about a specific risk factor I might find in 23andMe's data than I already know. Perhaps the best I could hope for would be to be referred to a specialist if my data indicated something potentially concerning.
2. The aspects of 23andMe's business that I most want subjected to oversight go well beyond what a specific risk factor might mean. I want someone to check that they are accurately and straightforwardly reporting on the accuracy of their sequencing methods. I wonder how many of their customers understand that there is pretty much a statistical certainty that the genotype they received from 23 and Me has at least one incorrect SNP call.* This does not mean 23andMe is doing their sequencing wrong- far from it. It is just a consequence of the limits of our detection methods and the large number of SNP calls they are making. However, if 23andMe customers do not understand this, I blame 23andMe and I would be inclined to think that they were engaging in shady marketing practices.
I also want someone to assess the validity of the methods they are using to associate specific SNPs with specific traits. What data are they using? Are they just applying public data, or do they have some proprietary algorithms? How well do they transmit information about the inherent uncertainties in this data to their customers?
I also want 23andMe to have a documented process for preventing bugs in the software they write to assemble the data, such as the one Lukas Hartman found when he dug into his alarming 23andMe results.
I have not done my research, so I do not know how much of the above 23andMe is doing well right now. Maybe they are doing everything exactly how I would hope. However, if they are not subject to regulatory oversight, I have essentially zero trust that they will always do all the above and do it well. The necessary scientific and technical validation is expensive to do. Taking software bugs as seriously as I think they need to is also expensive.Writing clear, consumer friendly documentation of all of the limitations- and scrupulously ensuring that no marketing spin sneaks into this documentation- is very hard to do. We do not trust drug companies to do this. The packaging inserts that accompany prescription drugs are reviewed and approved by the FDA. Why should we trust a personal genomics company to do it?
None of this means I think 23andMe should close up shop. Far from it. I think there is great promise in what they are trying to do. FDA regulation is not a perfect solution, but neither, I think, is just treating this industry like any other online service and letting it run unregulated.
I promised @Moreandagain that I'd dig up some links that do a better job of describing the science and the concerns than I can do. I put a couple of the links above. Here are some more:
David Dobbs has a nuanced write up in the New Yorker.
Hank Greely, writing at the Stanford Center for Law and the Biosciences covers the legal basis of the FDA's actions, some of the history, and some of the concerns. One interesting tidbit I noticed in this article was the mention that the FDA has indicated it wants a risk-based method of regulating "laboratory developed tests." This sounds similar to what they do for software used in medical and other regulated circumstances (21 CFR Part 11, for those in the know- and yes, software needs to be regulated in some circumstances, because some bugs can be deadly). The risk-based method of implementing the regulations for software allows companies to perform (and document!) an assessment of the risks of the system, and then take appropriate mitigating actions based on the severity and likelihood of those risks, essentially customizing the regulation for the level of risk. Doing something similar for genetic tests seems like a reasonable thing to me.
Genotopia has a post arguing that the dispute between 23andMe and the FDA is not about access to your genetic data, but about the hype the company used to make you want that access.
Michael Eisen has a very good post about the regulation of genetic testing, arguing it should be regulated but not as a medical device.
I'm sure I've missed some great pieces about this. Feel free to rectify that in the comments.
*SNP = single nucleotide polymorphism. The SNP calls are what allow 23andMe to make any prediction about traits. Basically, a SNP is a site in the genome in which more than one nucleotide is found within the population. Some SNPs are silent, and change nothing in our phenotype (observed traits). Others cause benign differences (e.g., hair color). Others are implicated in diseases. Of the ones implicated in diseases, only a subset have a proposed mechanism for action, and only a subset of those have experimental validation for that mechanism. Our level of confidence that a SNP is related to a disease state increases as we move from a statistical correlation to a plausible proposed mechanism to experimental validation of that mechanism.The classic example of a SNP associated with a disease via a known, experimentally validated mechanism is the mutation in hemoglobin that causes sickle cell anemia.