Friday, December 06, 2013

Weekend Reading: The 23 and Me Edition

Today, I had a Twitter conversation with @Moreandagain about 23andMe and the FDA, because I replied to this tweet:




Here was my initial answer:




And then we discussed some more from there. This discussion was mightily hampered by the fact that I have been, as I admitted in my tweets, an only mildly interested bystander to the entire story.

My basic position- which again, I admit is not as informed as it could be- is that I'd like to see the personal genomics industry succeed, but that I think they need to be subject to some sort of regulatory oversight. My reason for that second part is that I, someone with a PhD in biochemistry who has worked in genomics, do not feel qualified to evaluate 23andMe's claims, and want there to be an impartial third party who checks that their claims are legitimate. I do not want to do this myself, because doing so would be enormously time-consuming, even for people who have the necessary scientific and technical background to assess their claims.

The usual response to this is that if I want help understanding my genetic data, I can go to my doctor. I see two problems with this position:

1. I know from multiple conversations I have had with my own doctors (all of whom I like, respect, and think of as really good doctors) that I am generally more well-versed on genetics and biochemistry than they are. This is not surprising, given the details of our educational backgrounds. However, it means that I cannot generally expect my doctor to tell me anything more about a specific risk factor I might find in 23andMe's data than I already know. Perhaps the best I could hope for would be to be referred to a specialist if my data indicated something potentially concerning.

2. The aspects of 23andMe's business that I most want subjected to oversight go well beyond what a specific risk factor might mean. I want someone to check that they are accurately and straightforwardly reporting on the accuracy of their sequencing methods. I wonder how many of their customers understand that there is pretty much a statistical certainty that the genotype they received from 23 and Me has at least one incorrect SNP call.* This does not mean 23andMe is doing their sequencing wrong- far from it. It is just a consequence of the limits of our detection methods and the large number of SNP calls they are making. However, if 23andMe customers do not understand this, I blame 23andMe and I would be inclined to think that they were engaging in shady marketing practices.

I also want someone to assess the validity of the methods they are using to associate specific SNPs with specific traits. What data are they using? Are they just applying public data, or do they have some proprietary algorithms? How well do they transmit information about the inherent uncertainties in this data to their customers?

I also want 23andMe to have a documented process for preventing bugs in the software they write to assemble the data, such as the one Lukas Hartman found when he dug into his alarming 23andMe results.

I have not done my research, so I do not know how much of the above 23andMe is doing well right now. Maybe they are doing everything exactly how I would hope. However, if they are not subject to regulatory oversight, I have essentially zero trust that they will always do all the above and do it well. The necessary scientific and technical validation is expensive to do. Taking software bugs as seriously as I think they need to is also expensive.Writing clear, consumer friendly documentation of all of the limitations- and scrupulously ensuring that no marketing spin sneaks into this documentation- is very hard to do. We do not trust drug companies to do this. The packaging inserts that accompany prescription drugs are reviewed and approved by the FDA. Why should we trust a personal genomics company to do it?

None of this means I think 23andMe should close up shop. Far from it. I think there is great promise in what they are trying to do. FDA regulation is not a perfect solution, but neither, I think, is just treating this industry like any other online service and letting it run unregulated.

I promised @Moreandagain that I'd dig up some links that do a better job of describing the science and the concerns than I can do. I put a couple of the links above. Here are some more:

David Dobbs has a nuanced write up in the New Yorker.

Hank Greely, writing at the Stanford Center for Law and the Biosciences covers the legal basis of the FDA's actions, some of the history, and some of the concerns. One interesting tidbit I noticed in this article was the mention that the FDA has indicated it wants a risk-based method of regulating "laboratory developed tests." This sounds similar to what they do for software used in medical and other regulated circumstances (21 CFR Part 11, for those in the know- and yes, software needs to be regulated in some circumstances, because some bugs can be deadly). The risk-based method of implementing the regulations for software allows companies to perform (and document!) an assessment of the risks of the system, and then take appropriate mitigating actions based on the severity and likelihood of those risks, essentially customizing the regulation for the level of risk. Doing something similar for genetic tests seems like a reasonable thing to me.

Genotopia has a post arguing that the dispute between 23andMe and the FDA is not about access to your genetic data, but about the hype the company used to make you want that access.

Michael Eisen has a very good post about the regulation of genetic testing, arguing it should be regulated but not as a medical device.

I'm sure I've missed some great pieces about this. Feel free to rectify that in the comments.

----------
*SNP = single nucleotide polymorphism. The SNP calls are what allow 23andMe to make any prediction about traits. Basically, a SNP is a site in the genome in which more than one nucleotide is found within the population. Some SNPs are silent, and change nothing in our phenotype (observed traits). Others cause benign differences (e.g., hair color). Others are implicated in diseases. Of the ones implicated in diseases, only a subset have a proposed mechanism for action, and only a subset of those have experimental validation for that mechanism. Our level of confidence that a SNP is related to a disease state increases as we move from a statistical correlation to a plausible proposed mechanism to experimental validation of that mechanism.The classic example of a SNP associated with a disease via a known, experimentally validated mechanism is the mutation in hemoglobin that causes sickle cell anemia.

11 comments:

  1. Nice post. So I used 23andMe in 2011 for $99 (a special deal then which I think is now approximately the current retail price). I have been beyond super happy with my results. Best $ I ever spent. They got all the basic stuff 100% right: hair color, eye color, the health issue it told me I'm at risk for is the one I actually have been diagnosed with, plus the ancestry stuff has been fascinating to discuss with my husband and parents (we all did this together), and I really appreciated the way the health risk info is presented: in terms of odds, with links to studies, and with conflicting studies provided alongside each other.

    Since I've tried and loved it, it honestly makes me sad to think that other folks might not be able to make this choice if they want to because… Consumer protection? The need for regulation to protect me from... knowing some new aspects about myself and my ancestors? FDA, for heaven's sake - there are bigger fish to fry, but clearly those bigger fish (BigTobacco, Fast Food cough cough) have better lobbyists.

    ReplyDelete
    Replies
    1. I'm curious about how they handle the inherent uncertainty of their data. Did the materials they sent you make it clear that it was almost certain that at least one of the SNP calls would be incorrect?

      The thing that I find most difficult to understand about this story is the fact that the FDA was discussing regulation with the company for years and years, and then suddenly the company stopped responding. That is a regulatory strategy bordering on insane.

      Delete
  2. So my mom had a group of coworkers who used it before we all did, and 1 of them had results come back with the incorrect eye color, so we knew going into it not to expect perfection by any means.

    I became a customer over 2 years ago so IIRC, they provided some typical disclaimers (which no one but my mom ever reads) and they never made any type of guarantees as to accuracy. Their rhetoric was about "high quality" testing not "accurate" testing, if memory serves. They were clear they were not looking at the entire genome, so things would be missed. They were clear this was for fun, to know yourself better, to discover your roots, meet distant cousins if you like, and was not medical advice - for that you should consult a physician, etc.

    Or maybe I'm remembering it like that because to me, it is obvious that lab errors exist, samples get mixed up, items get lost in the mail, there are many non-genetic determinants of health risk, etc. No guarantees. If anything serious had turned up my family and I would have immediately sought genetic counseling for some real testing knowing that false positives exist, etc.

    I don't think failing to respond to the FDA is as Cray Cray as a lot of folks seem to think. Sounds like a cutting edge jurisdictional strategy - because if you just submit to the FDA's authority you're saying they get to regulate you. You're admitting you're in the category of "Food" and "Drug." You're admitting a straw thingy you spit into is some potentially threatening medical "device." It all sounds a bit silly really.

    I can see how the company would rather let the federal courts decide, and keep it out of the hands of gov't bureaucrats, so peace out, FDA: now you're going to have to sue us. And I'm sure certain members of Congress will have something to say about this - oh to be a fly on the wall at FDA HQ.

    23andMe is already winning in the court of public opinion - people are pissed they can't access their own info. Think about the implications of that - we have HIPAA and FOIA - there are strong precedents for American citizens being entitled to their info. I could see this eventually play out as a 1st Am issue re: the individual right to speech and expression, free from govt interference. Easily. On top of that, this is an age where even dyed in the wool liberals don't blindly trust the government anymore, thanks to the NSA scandals. And no American wants to be told they can't have something that already belongs to them as part of their body and is perfectly safe. It just does not pass the sniff test.

    I feel bad for my BFF who wanted 23andMe as an Xmas present and can't have it.

    ReplyDelete
  3. I actually saw a dissertation defense last month regarding this. 23andme doesn't account for epigenetics and they offered health information that could be used to misguide patient behavior. In my medical device company, we provide diagnostics tools that have been tested with thousands of patients to prove accuracy and efficacy. We work with doctors so patients will not self-medicate after a bad diagnosis. If we put out a product that diagnosed a patient with certain probabilities we would NEVER go around the doctors. Patients are not fit to make major medical decisions (this is part of why I HATE drug commercials on TV). The FDA let 23andme know about this, and 23andme ignored them. That's part of the problem. When the FDA notifies me about a problem they have with us, we fix the problem immediately, working with the FDA the whole time. This is because we care about our patients, and we care that the proper information will be fed up the appropriate channels. 23andme is about business, plain and simple. And if you're going to claim some kind of medical relevance, you better be responsible and substantiate your claims. All 23andme has to do is some legal tap-dancing and they'll be back in business. What they show is clearly just caring about the business. The FDA can be a headache (trust me, every medical device I've put out has gone through them eventually), but they're necessary to prevent companies from making wild claims about their products or poisoning people with bad medicine. 23andme isn't any difference than any other diagnostic tool that can be misinterpreted (ultrasound, MRI, panels, etc). Their claims should be regulated, and they should not ignore the FDA when they express concern.

    ReplyDelete
    Replies
    1. "Patients are not fit to make major medical decisions"

      Wow. I could not disagree with you more.

      Delete
  4. See Richard Epstein, The FDA Strikes Again: Its ban on home testing kits is, as usual, likely to do more harm than good --

    "The FDA should have to show by clear and convincing evidence that 23andme leads to the dangerous results that the FDA claims by surveying customers of the firm. Once the FDA keeps a product off the market, downstream actions by individual consumers and their physicians cannot undo its mistakes."

    at

    http://www.pointoflaw.com/archives/2013/11/t.php

    See also Ronald Bailey "Let My Genes Go" --

    "Now let’s assume that a 23andMe customer tests positive for one of the BRCA mutations, what are they likely to do next? On the strength of their letter, FDA bureaucrats seem to believe that women with a positive test will immediately go look for surgeons to remove their breasts; start gobbling down tamoxifen pills to ward off cancer; or get more mammograms. Of course, these scenarios are ridiculous. What a customer would do is consult with her physician and most likely get another more comprehensive genetic test. That is exactly what Huffington Post blogger Jill Steinberg did when her 23andMe results came back showing that she carried one of the BRCA mutations. Her subsequent test confirmed the 23andMe finding and she eventually decided to have a double mastectomy."

    at

    http://reason.com/archives/2013/11/29/why-the-fda-should-leave-23andme-alone

    ReplyDelete
    Replies
    1. Anonymous5:38 AM

      You don't need 23 and me to get tested for BRCA mutations. Your doctor can order those. And will hopefully be more informed about type 1 and type 2 error than the average consumer.

      Delete
    2. My doctor is an expensive middleman. So is my insurer. Personally, if I wanted a valid test, I'd see a genetic counselor, not my doctor. BTW, I think the thing isn't so much that people don't know genetic counselors exist, it's that they have no idea how you'd go about speaking to one. (I'd phone a synagogue, but most people would ask a doctor.)

      Obviously real medical tests will NOT cost as little as $99 (hell, in medicine they're charging whatever they want these days for tests because costs are still unregulated). I won't get my results quickly, nor will they be delivered in a cool, user-friendly format. I would also have to wait for an appointment, wait in a waiting room, wait, wait, wait. And as @Cloud already said, my doctor most assuredly does not understand those results at an expert level anyway.

      It's about entertainment vs. the actual practice of medicine.

      Again, what's the harm to me in choosing 23andMe for fun and my own edification? When folks use the site there are all kinds of pop up windows "You should consult a doctor, do you want to continue? Click here." "This might upset some users, want to continue, click here."

      People can and do handle getting potentially upsetting facts in the privacy of their own home. What's next, I'll have to get all pregnancy, HIV, UTI, yeast infection, drug, and ovulation tests done at the doctor's office, so he can assure me with his awesome bedside manner because bad facts might upset me enough to harm myself and I can't be trusted with the truth of my own health info? The AMA would love that.

      Could I still do breast self-exams in my own shower? Because that's what 23andMe is analogous to. Finding a lump. It's DIY. It's clearly inexact. Maybe a case could be made that in the aggregate, we shouldn't do it because it leads to unnecessary test$ and intervention$. Screw the aggregate, I want to be able to play.

      Dang, I guess I have a lot to say on this topic. I'll go away now. I can't seriously be the only WandSci reader who has used 23andMe. There is something about having been an actual 23andMe user that changes the optics of this for me, and my read on this controversy that's hard to put into words. I'd say try it for yourself but now you can't.

      Delete
    3. @hush- you are welcome to say as much as you want! I find your perspective interesting. I think we'll be able to try it for ourselves again soon. Last I heard 23andMe was back talking to the FDA.

      I haven't read enough to have a firm opinion on whether those tests should be regulated as medical devices- probably not. But I do think their marketing claims and consumer documentation should be regulated. Otherwise, I don't see what protects consumers from 23andMe becoming less ethical than they seem to be now. It is the potential for misleading marketing and consumer information that worries me. They seem to have been ethical so far, but I don't see why we should trust that they (and any other company that chooses to compete in this space) will stay that way.

      I don't worry so much about self-medication (since the medications are themselves subject to regulation!) but I do wonder what would happen if (for instance) someone who thinks maybe they aren't as sharp as the used to be takes this test and gets a SNP call indicating a high Alzheimer's risk- or something like that. It isn't that I think they can't handle the info, it is that I want them to be really, really, really clear on the potentials for error, so that they don't do anything drastic before getting some sort of second opinion. And by drastic, I mean "commit suicide" since I personally know more than one person who has stated they would kill themselves if they thought they were getting Alzheimer's, and I believe they really would.

      I am, however, sympathetic to the frustration @Phindustry expresses. It is probably similar to how I feel watching to unregulated claims of the supplement industry. It is frustrating to work in a highly regulated industry to produce a product you believe will help people and to watch clearly inferior products get to skirt regulatory control, and then to have people think the unregulated company is the good guy and you're the evil money-grubbing bad guy because you product costs so much more than theirs. Doing the tests to make sure your claims are true is expensive! Now, I don't think what 23andMe offers is anywhere near as risky as some of the supplements out there, but I do not think it is without risk.

      Delete
    4. Anonymous7:39 AM

      It's probably the same situation as those u/s pictures you can get at the mall. They are for entertainment purposes only etc. and have tons of warnings about how they differ from seeing your doctor.

      There are very few instances of cut and dry genes like brac and huntingtons. If those are suspected, the doctor really is the best place--that is not entertainment. Just like all the warnings about the bad things an u/s can find. The mall is really not the place to find problems with your baby.

      Delete
  5. I used 23andme! But only for racial makeup...and to try and prove my parents aren't my real ones :).

    You're correct, some doctors are middlemen, and some patients are smart enough to use the information properly. But I've seen two studies in the OR in the past month alone where patient self-medication resulted in major surgery.

    The FDA has said the self-breast exam is a good early detection method. And BD Sensability had to get FDA approval for their breast exam monitoring device. 23andme just needed to get their claims correct and not ignore the FDA. I know how much the FDA can suck: I've launched multiple medical products in the US market, and I'm preparing three more after Euro and Canada studies. But if you're going to release something for patients, especially patient-specific, you can't ignore the FDA. It's the only way we have to protect patients. If 23andme wanted to stay in business they either should have listened to the FDA, changed their claims, or stay out of the US. It's what everyone else has to do. And it's the only way to keep patients safe, since those of us in the medical-industry complex have an allegiance first to patients. I had lunch with one of my company's lawyers and she told me 23andme wouldn't have to run countless studies, just change their marketing. She also likened them to running a blood culture for infection. The patient gives up their blood, and they're told what their bacterial levels are. We need to prove that a certain level dictates some kind of action. It's a diagnosis tool. I hope 23andme gets up and running under the proper guidelines since I believe this could be very useful, they just need to care about the people and make sure the information can't be used improperly.

    ReplyDelete

Sorry for the CAPTCHA, folks. The spammers were stealing too much of my time.