Reproducibility In Organic Chemistry

How One Chemist Playing Hooky From His Day Job Helped To Get A Paper Retracted From Chemistry’s Flagship Journal

[In the vein of the recent Chemical & Engineering News perspective on “how the internet has changed chemistry”, [see here for illuminating articles on access to databases, job searching, and education] it seems an opportune moment to revisit a recent example of how the Internet augmented the self-correcting nature of the scientific process, and helped to boot a poorly researched paper out of chemistry’s flagship journal before it hit publication.

Why publish this on a blog for sophomore organic chemistry students? Reproducibility of scientific experiments is in the news [psychology] [biomedical science].  It’s a great lesson for non-experts in how organic chemistry experiments are conducted… and a reminder that while bad papers sometimes sneak through, the organic chemistry literature has a lot of integrity.]

On July 21, 2009 a very… interesting article was published in the ASAP (“As Soon As Publishable”) section of the Journal of The American Chemical Society (JACS). The authors claimed that treating the following molecule (4-chlorobenzyhydrol) A with the base sodium hydride (NaH) resulted in the formation of the product B (4-chloroacetophenone) in 85% yield. The paper had several other very similar examples.

If you can read structural formulae, you can see that what’s happened here is that C-H had broken and we’ve formed a new C-O double bond (“π bond”) – a reaction known as oxidation. 

The authors claimed that sodium hydride was acting as the oxidant in this reaction.

The response of many chemists was to furrow an eyebrow and say… “huh?”. Since the very definition of oxidation is to break C-H bonds, and NaH is a reagent for forming bonds to H, it seems highly curious (to say the least) that this reagent might act as an oxidant.

In particular the authors were vague about how NaH could perform this task (a process that organic chemists refer to as the “reaction mechanism”), and even said that sodium metal (which should have produced the same anion as NaH) did not promote oxidation. In retrospect, that was a crucial clue that something else was going on.

Chapter 1: The Thread Hijack

Meanwhile, in one of the dark corners of the Internet where chemists hung out at the time, the blog Totally Synthetic, author Paul Docherty had just finished an article about a recent synthesis of Tricholomalides A and B by Sam Danishefsky’s lab at Sloan-Kettering.

The second comment in the thread took the discussion in a completely different direction, pointing to the “NaH as oxidant” paper.

Chapter 2: Liveblogging The Reaction

Docherty’s curiosity was piqued. Noting the simplicity of the example, and realizing that these same starting materials were present in his own laboratory, the next day (July 22) he took some time off whatever he was supposed to be doing at his day job at Arrow Pharmaceuticals to check this reaction out for himself.

At 10:40 a.m. the NaH was added to the alcohol under an atmosphere of nitrogen at 0°C and stirred.
“Cue fizzing”, said Docherty.

12:10 p.m. “‘Tis Orange!” After warming to room temperature, the reaction had turned orange [color is a parts-per-million phenomenon, so this can often be a misleading indication for reaction progress].
1:20 p.m. First Analysis The reaction was  analyzed by liquid-chromatography mass spectroscopy (LCMS), which showed both starting material and product to be present.

Chapter 3: The Result

The next day, Docherty took an NMR of the product mixture (a tool chemists use to determine the structure and purity of organic compounds) and was able to determine that the reaction proceeded in 15% yield – a far cry from the 85% yield reported in the paper. [The NMR shows strong peaks corresponding to starting material at 7.5 and 2.0 ppm; the remainder of the peaks are product]

Chapter 4: The Community Weighs In

While Docherty was live-blogging his reaction, others attempted to reproduce the paper’s results.

One commenter found that the reaction did not proceed at all in the absence of oxygen, which seemed to bear out the general sentiment that NaH was not in fact the oxidant in this reaction.

However, when oxygen was added, the same reaction went to completion, giving the desired product:

Others reported similar observations.

Note the timestamp on these posts. These results were obtained within 72 hours of the paper appearing online. 

Interestingly, one of the commenters referenced a paper from 1965 which showed that there was precedent for NaH (usually used as a suspension in mineral oil) oxidizing alcohols to carboxylic acids in the presence of air. In other words, the authors of the 2009 paper didn’t do their homework.

Step 5: “This Manuscript Has Been Withdrawn For Scientific Reasons”

Docherty’s results, as well as those of the commenters, suggested that the true “oxidant” in the reaction was not NaH as the authors of the paper claimed, but molecular oxygen.

After this, things quieted down.  There was no direct contact with the lab of the PI responsible for publishing the paper. However, a short period of time later, the following note appeared where the link to the paper formerly appeared.

“This manuscript has been withdrawn for scientific reasons”

Conclusion: 

What can we learn from this? Organic chemistry experiments have have several features which favor a high degree of reproducibility.

• Large sample size. Running a reaction on a millimole (1/1000 of a mole) of material still means you’re sampling 6 x 10²⁰ molecules [Docherty ran his experiment on 4 millimoles of material]. Compare this to psychology studies, which might be done on a few dozen individuals at the lower bound. Safe to say that no chemist has ever had to manipulate a p-value.
• Reasonable cost of most reagents (chemicals are generally cheap when compared with enzymes, antibodies, DNA sequences, etc)
• Short time frame – most experiments are complete within 24 h. The example above was replicated within 72h of the paper appearing online. This also compares favourably with biomedical studies, which can take months or even years to conduct (if you’re waiting for an organism with a knockout gene to reach a certain developmental stage, for instance)
• Widely held expertise – With no offence meant to Docherty, there was nothing particularly “special” or unique about his training in organic synthesis. He ran a reaction that thousands of other people in the world could have run just as expertly. What put him in a unique position was that he ran a blog read by thousands of other chemists, so his results were widely distributed. Contrast this to molecular biology, where success in reproducing an experiment might require considerable practice in conducting a particular experimental technique.
  “You can’t give me and Julia Child the same recipe and expect an equally good meal”, said Jeff Settleman of Calico Life Sciences (in the Science article mentioned above). When “Julia Child effects” arise in organic chemistry, there’s likely a good reason for it (see below).

• Minimal variables –  for organic chemistry, controlling  the temperature,  purity of reagents, purity of solvent, and atmosphere is usually enough.  Traces of water or oxygen can cause problems, especially on small scale, but these can generally be dealt with. Controlling the variables in biomedical papers, by contrastcan be a nightmare. Experiments are notoriously sensitive to small differences in protein sources, to contamination, or other weird factors (mice get stressed out by male researchers, for example).  For some examples on how easy it can be to mess up, here are some lessons in paranoia for making stock solutions. Let’s not even mention controlling variables in psychology. One researcher, being told that her results were much weaker upon replication, said that the reason was that the replicated study used American college students, while hers used Italians.

Given these, we can safely say that the integrity of results in the organic chemistry literature is high. If problematic research papers are found, attention will likely be drawn to them, and given the speed at which their results can be falsified, their half-life in the literature will be short. The Internet has increased this half-life dramatically. 

2013 Redux

If you look through the comments on the article at Totally Synthetic, you might note several grumblings about another reaction organic chemists found to be capricious – the oxidation of benzylic hydrogens with the reagent IBX.  In an interesting postscript to this story, in 2013 this reaction was tested by the anonymous authors of the website Blog Syn.  The authors also had difficulty reproducing the result at first, or as they put it, “it requires mojo“.

However, after discussions with one of the paper’s primary authors (who had one of his graduate students reproduce the work, in epic detail, with accompanying pictures), they were able to coax the reaction to produce the desired product.

A sharp eyed anonymous commenter pointed out that the “successful” runs of this reaction had greater trace water impurities in the IBX than “unsuccessful” runs, leading the commenter to suggest adding a small amount of H₂O to the reaction as an additive. And lo! the yield and reaction time increased considerably, leading the Blog Syn authors to suggest that the H₂O adduct of IBX, not IBX itself, was in fact the active oxidant.

I include this because it’s a great example of how “Julia Child Effects” (i.e. two people given the same recipe produce results of differing quality) arise for sound scientific reasons, and further exploration of variables can lead to a greater understanding of how reactions work.

The whole article might be one of the best chemistry blog posts ever published,  for sheer detail. Check it out.

On a final note, best wishes to Paul, wherever you may be, for everything you did in building Totally Synthetic. It is dearly missed.