Friday, July 24, 2015

A healthy dose of skepticism and the need for editors

As some of you know, I've been engaged in an interesting dialogue as a reviewer for Frontiers recently. This got me thinking about the role of editors in the process of publication, but also about how my own brain interprets experimental data. I was originally going to write a couple of posts, but I think they work together so now you just get a singular post that's slightly longer.

 Long story short, currently, I disagree with the way that the authors have analyzed their data and am waiting to actually "endorse" the publication at a Frontiers journal. If you snoop around in a couple of months I'm guessing that you'll be able to figure out what I'm talking about because at Frontiers the reviewers names are listed openly on the final PDF. This whole process has led me to rethink the way that Frontiers actually performs review (which takes place in an interactive forum where authors respond directly to comments from reviewers). I love the idea of open, non-anonymous review and am strongly in favor of making public a record of review for each paper. For reasons I'll elaborate on below, I think this system is slightly flawed.

Maybe I've just grown cynical over the years, but the first thing I do when I get awesome new data is to question how I screwed up. Was everything randomized? Did the strains get contaminated? Etc, etc...Ideally all of these questions are answered by experimental controls, but I'm good at thinking of extravagant and elaborate ways in which I'm wrong. Nature is often quite good at this too I've found, although that's the fun of biology (after a period of cursing the sky). Thanks in a large part to this  self-skepticism, I'm always thinking about the next ways to adequately control for experiments which leads me to wait to pull the trigger on submitting publications. My grad school and PD advisors helped to reign in these skeptical tendencies and slowroll of manuscript submissions just a bit by pointing out that nothing is ever perfect. The voices are always still there though.

These same tendencies act when I'm reviewing other papers. Sometimes things are easy to believe just by comparing summary stats to the reported data, but other times I'd like to see the primary data and dig my hands personally into the underlying statistical model/assumptions until I truly believe it. In many cases I have to actually ask to see this primary data, which is not great, but at least with anonymity I don't worry about directly questioning the author's abilities. I mean, inherently if you are asking for primary data because the stats seem wonky then you're implicitly questioning other people's abilities. When my name is not going to be known I don't worry as much about the social ramifications of it all and I sleep better at night.

I am way too over-critical of my own experiments. A little bit of skepticism is healthy, but too much self-skepticism as a scientist paralyzes your career. Even as a reviewer I worry about being over-critical and asking for tedious and minuscule changes that might not ultimately matter. When you are knee deep into reviewing a paper it's easy to lose sight of the bigger picture. This is where the editor comes in. Each time we review a paper, we make a list of critical and less than critical things that need to be "fixed" before publication. Oftentimes the editors will read these list from every reviewer and distill down the absolute requirements. Editors often have their own impression of what makes a publishable unit (that's for another post though, suffice it to say that's why direct track 2 submission to PNAS no longer exists). What I've come to think is that editors are absolutely required in the current publishing process. Reviewers and authors are on about the same level in the dynamic, but the editor inherently has an overriding sense of authority in the whole process. They can take reviewers comments and immediately disregard the ones that aren't critical. They can emphasize to authors exactly everything that needs to be done. The authority is key because both reviewers and authors are deferential to it. As a reviewer I'm not worried about asking too small a question because 1) everything I write in the review is important to me and 2) I know that the good editors will know when I'm being too specific are nit-picky.

With this Frontiers article I've had to respond to the authors that "I'd like to see the primary data". Having received many reviews in my career, I know exactly how this comment will be received. When it comes from a reviewer directly it seems nit-picky and maybe even a bit of a personal affront. If the editor agrees, there is a bit more weight to the comment. It felt weird having to directly comment to the authors that I wanted to see their data. They're a good lab and I worry that their impression of me (since they'll know my name after it's published) will change for the worse. These are things you can't control, but that's how it goes.

For all of you out there who have papers I'll review in the future, know that I'm even harder on myself. I'd like to think self-skepticism is part of what makes me good at my job though.

Wednesday, June 24, 2015

Metagenomics and "A feeling for the organism"

Evelyn Keller's biography of Barbara McClintock is entitled "A feeling for the organism". A few paragraphs in this last chapter sum up quite nicely how McClintock viewed the scientific enterprise and discoveries:

"Over and over again, she tells us one must have the time to look, the patience to "hear what the material has to say to you," the openness to "let it come to you." Above all, one must have "a feeling for the organism." One must understand "how it grows, understand its parts, understand when something is going wrong with it. [An organism] isn't just a piece of plastic, it's something that is constantly being affected by the environment, constantly showing attributes or disabilities in its growth. You have to be aware of all of that.... You need to know those plants well enough so that if anything changes, ... you [can] look at the plant and right away you know what this damage you see is from-something that scraped across it or something that bit it or something that the wind did." You need to have a feeling for every individual plant. "No two plants are exactly alike. They're all different, and as a consequence, you have to know that difference," she explains. "I start with the seedling, and I don't want to leave it. I don't feel I really know the story ifI don't watch the plant all the way along. So I know every plant in the field. I know them intimately, and I find it a great pleasure to know them." This intimate knowledge, made possible by years of close association with the organism she studies, is a prerequisite for her extraordinary perspicacity. "I have learned so much about the com plant that when I see things, I can interpret [them] right away." Both literally and figuratively, her "feeling for the organism"

I agree wholeheartedly. Others may science differently, but I make my living by looking and studying everything about the bacteria I work with. Going into each experiment I have an idea of what to expect (even if these "experiments" simply involve streaking out cultures from frozen). If you give me a genome of Pseudomonas syringae, I can tell you the main components you'll find . I can tell you how certain strains will grow (or won't), what the colonies will look like, how long they'll take to pop up, what color they'll be. It took me a few years to gather this intuition, but now that it's engrained I like to think I have an innate sense when something is "off". I liken this to a scientific Spidey-sense. The challenging part is truly knowing when to follow up on these odd results, when to store away for the future, and when to disregard them as uninteresting.

I was reminded of McClintock's "feeling for the organism" by a couple of stories from metagenomics that have popped up across my feeds. Before I say anything else, I don't intend to denigrate the quality of the science or data underlying these stories by any means. The work is solid, I just think we're starting to find some limitations in the power of "big science" and these holes usually pop up in the discussion sections of papers and press releases. The first of these stories was a tour de force looking at metagenomics of the NYC subway system. The authors reported a variety of interesting results, but the tag line that a lot of news outlets seemed to focus on were the presence of Yersinia pestis (plague) and Bacillus anthracis (anthrax) within the subway system. The limitations of these methods have been hashed out already (here and here), but I want to focus on the inherent lack of "a feeling for the organism" when dealing with metagenomic data. Studies of any open microbial ecosystems are going to find a diversity of taxa. Unless you bring in specialists, there is simply no way to know the ins and outs of each organism. In the case of the NYC subway metagenome, from my interpretation at least, the authors looked at only bits and pieces of the Yersinia and Bacillus genomes without capturing the whole picture. They had to do this because the story was so inherently large that you couldn't possibly investigate everything in depth. However, specialists with "a feeling" for either Yersinia or Bacillus could have provided a viewpoint on which directions (other genes to look at, levels of nucleotide diversity which seem a bit too high) to follow up on to truly demonstrate presence of these bugs within the subway.

Likewise, one part of the story on urban microbes in this piece caught my eye:

"Rodents are under study, too. White-footed mice (Peromyscus leucopus) in New York City carry more Helicobacter and Atopobium bacteria — associated with stomach ulcers and bacterial vaginosis in humans — than their suburban counterparts"

I worked and slaved over Helicobacter pylori cultures all through grad school. I simultaneously loved and hated that bug. It's finicky growth patterns are the reason I moved over to study the reliably growing Pseudomonas after grad school. Unless something has dramatically changed since I've been in the literature (which is completely possible), rodents are terrible hosts for H. pylori strains that cause stomach ulcers (general overview here). You can get a subset of H. pylori strains to grow in mice, but there are often a variety of genetic changes that take place that allow them to adapt (see here).  I wouldn't be surprised if there were multiple Helicobacter strains within mice in NYC, but my money is on the fact that they aren't Helicobacter pylori that could cause stomach ulcers.

These are the tradeoffs that are made when dealing with immense data sets, and I'm not quite sure how to fix this. No one has a feeling for ALL THE MICROBZ. If you have a fun/interesting story on microbiomes that focuses on a couple of taxa in bold, at least try to run your data and ideas past someone that truly has a feeling for these organisms before publishing the paper. If it holds up after that, more power to you.

Friday, June 19, 2015

Navigating the waters of NSF grant submission

*Disclaimer: What follows is  a post about structural biases I've perceived within the NSF Biology system. I think these biases are intrinsic but keep in mind I could be completely wrong (and if you have different views, please feel free to comment). They also aren't inherently bad or need to be fixed, they just exist based on the pool of reviewers/panelists and timing of the grant cycles. It's a bit rambling, but I'm hoping to provide at least a slightly useful insight or two.

Even before the new office smell has worn off, and in many cases before you've actually moved into your office, the thoughts of many PIs newly merging onto the tenure track are focused on grant writing. This isn't going to be a post about how to get NSF grants, but more along the lines of "things I've experienced writing grants across panels". Grant writing is truly an art. Something that I didn't truly appreciate before is that, as with any piece of art, each target audience has their own subjective opinions. I've had my lab for 4 1/2 year now and have written grants to DEB, MCB, and IOS. I've been fortunate enough to sit on preproposal and full proposal panels. One of the most difficult ongoing lessons I'm learning is that grants written to each of these are very different beasts.

1) Preproposals change the game. DEB and IOS require preproposals, MCB does not. I'll save most of the comments about pre vs. full proposals for other posts, but suffice it to say that writing a convincing preproposal takes a different skill set than writing a convincing full proposal. Since preproposals don't go out for external review, the fate of your grant is entirely influenced by the composition of the panel. In panels that get a lot of submissions focused on similar systems (at least from what I've seen at IOS where there are only a handful of well-worn symbiosis models) novelty can be a benefit. If you propose to work with a new/novel system, and the science makes sense, you can get some bonus points if every other grant is focused on model organisms. Furthermore, while there are certainly benefits to working with a model system, it's more likely that someone on the preproposal panel will know little details about the nuances of the organism and can call you out for poor experimental design. On the other hand, if you are proposing to work in a system that no one on the panel truly has expertise in, you better be able to convince them in four pages that the experiments are feasible. Depending on overlap of the panel's expertise with your own grant, there could be details missed during the preproposal discussion/reviews, and their will likely be subtle misinterpretations. It's just how it goes and feeds into the noise of the system. These things can be ironed out in the full proposal though because those will go out for external review. I get the feeling that DEB grants and review panels have a much higher variance in topic and system than IOS panels. If such a difference truly exists it definitely adds a new psychological layer into the process.

One last thing to mention in regards to the effects of preproposals. There is likely to at least be a little overlap between reviewers of your successful preproposal and your full proposal. I can't speak to anyone else on this, but when discussing full proposals I remembered the discussions surrounding the preproposals. I remembered perceived weaknesses and strengths and I tried to see how the authors dealt with these criticisms. I can't help but think that it's a good idea to dedicate some of your full proposal to laying out a response to your preproposal reviews.

2) Timing can matter for CAREER grants, especially since you have a choice about which panel to submit to. Submission of IOS/DEB full proposals occurs in summer and overlaps with CAREER award deadlines. Panels evaluating full proposals for both of these programs will also evaluate CAREER awards at the same time. Given the vetting of ideas that occurs due to preproposals, differences between CAREER grants and full proposals were often pretty glaring. It's also possible that you could have turned your non-invited preproposal into a CAREER grant, and that it would be reviewed by the same panel for both IOS/DEB.

In contrast, the normal deadlines for MCB panels that I've applied to are now in November. Therefore, if I submit a CAREER award to MCB there is no chance that the grant could be reviewed by the same panel that it would be as a regular submission. This matters because I've had some CAREER grants go to what I perceive as weird places at MCB (like Engineering panels) and they get evaluated very differently than they do at the regular November panels. Differences in criteria between regular and CAREER grants aside, the science may be essentially the same in the grants I've submitted but I get a feeling that there is much more variance in the CAREER reviews simply because the panel isn't quite the fit I imagine it to be. I think this also factors in because I'm not convinced that reviews of CAREER grants inform my writing of regular MCB grants (and vice versa), whereas I think you can get more traction out of reviews regardless of grant type at both IOS and DEB.

3) Funding rates are low regardless, but DEB (evolutionary processes at least, I can't speak to anything ecology) feels like an even steeper climb for microbiologists than for other biologists. The first few times that I had grants rejected from DEB, the POs made statements like "you have to convince frog biologists that your work is important". These comments were spot on and looking back I did a terrible job at describing how my work applied across systems. However, and I could be wrong about this although the few people I've asked back up my intuition, I'm not sure that grants from frog biologists at DEB get the reverse critique of "convincing microbiologists that your work is important". I'm not sure what this means, and certainly some great microbiology work gets funded through DEB, but it feels like there is a slightly implicit bias from the reviewers against microbial evolution work at DEB. There are some generally important evolutionary phenomena in bacteria (like rampant horizontal gene transfer) that simply don't apply across systems. Likewise, there are some generally important evolutionary phenomena in eukaryotes (sex ratio biases, diploidy) that don't really cleanly apply to bacteria. Given the broad makeup of review panels at DEB, I think it's just hard to get some types of microbial work funded through there even though in a world with unlimited funding it's the right place for it. It's possible that the reverse is true at IOS because most model symbiosis systems involve microbes.

Monday, January 19, 2015

My thoughts on "The type VI secretion system of Vibrio cholerae fosters horizontal gene transfer"

I was on a late Christmas break last week, when I caught wind of a newly published study (here) and associated write ups (best one by Ed Yong here) which suggested that natural transformation and type VI secretion (T6S) were linked in Vibrio cholerae. Given my research interests into microbe-microbe interactions, and my experience studying and writing about natural transformation and evolution, I was naturally intrigued. I was also wary, however, because these kinds of studies tend to oversell correlations and tend towards "just so" stories. Having now read the paper a couple of times, I actually think it's quite a good example of a microbial genetics story and much less so evolutionary biology story.

I won't go into the gory details too much, but the authors start out pointing out that little is known about regulation of the T6S system in Vibrio. The main take home result of this paper is that the T6S system operons are controlled by TfoX and also by quorum sensing through HapR and QstR. That's a solid story and worthy of publication in a pretty high tier journal. However, due to a happenstance of history moreso than anything else, TfoX is also said to be a master regulator of competence for natural transformation in Vibrio. This association arose because TfoX was originally identified as a regulator of competence in the presence of chitin. Looking back in hindsight, maybe TfoX should be referred to as a master regulator of pathways associated with chitin presence.

The authors decide to run with with regulatory association between T6S and competence, and test whether killing of cells by T6S facilitates horizontal gene transfer through natural transformation. As a way to suggest that there is a more evolutionary link between these processes, the authors set up experiments to demonstrate that genetic exchange dependent on T6S killing can occur. For this experiment, the authors test for the ability of their focal T6S wielding strain to be transformed by a kanamycin resistance gene integrated into the genome of another Vibrio cholerae strain. Surprise, surprise (/sarcasm) the experiment works and T6S facilitates genetic exchange. I say that snarkily because ANY process that releases DNA from cells can facilitate horizontal transfer by natural transformation...heat, lightning, whatever you can imagine. My problem here is that the authors place their finger on the scale to effectively rig an experiment whereby they will get the "sexy" result that would be undoubtedly overspun in press releases.  The problem, and this goes for a lot of papers (especially of the microbiome sort) is that just because something is possible under experimental conditions doesn't make that phenomenon evolutionarily relevant.  How did the authors bias this experiment and why am I annoyed enough to hastily craft a blog post?

1) Natural transformation frequency of genomic DNA is highly dependent on similarity of donor and recipient genomes. Transformation by plasmids is a bit different because these don't require recombination. The authors used two (relatively closely related) different Vibrio strains ensuring that recombination could occur. I doubt this experiment would have nearly the success rate (if at all) if different Vibrio species were used as prey. The chances of success fall with genomic divergence from the recipient strain. I have no clue of what the spectrum of other bacteria that live on planktonic crustaceans that would be killed by Vibrio, but the more diverse they are the less likely that T6S truly affects genetic exchange.

2) The type of selection matters. The authors set up the experiment with kanamycin resistance, because they can plate out strains onto antibiotics and strongly select for transformants. Not critiquing that part, and it's certainly how you'd do the experiment, but I'm not sure that such selective environments are representative life on crustaceans or in the ocean. For T6S to have evolved to significantly affect genetic exchange requires a constantly changing environment with strong selection pressures whereby prey strains can be more adapted than predator strains. To this point, laboratory experiments have begun to show that natural transformation can increase rates of adaptation, but generally only in "stressful" environments. It's possible that such conditions could consistently arise for Vibrio, but it's a hard sell.

3) Since T6S preferentially targets dissimilar strains, there is a much much much greater chance that transformation of DNA from prey cells would be detrimental than beneficial. Rosie Redfield has already made the case (here and here) that transformation of DNA from closely related strains is likely detrimental because transformable DNA will contain more deleterious alleles on average than living cells. Additionally, there is always the chance of incorporating alleles that lower the transformation rate and which can't easily be replaced once incorporated. Transformation of DNA from prey cells targeted by T6S systems introduces two related problems. Although transformable DNA won't inherently contain deleterious mutations (unlike Rosie's paper, cells are killed by other cells rather than by deleterious mutations) many of the genes within this pool will be diverged from those in the recipient genome. Therefore, it would be much (much much+++) more likely that predator cells would be transformed by alleles of housekeeping genes that wouldn't function efficiently when placed into a new genomic context than by beneficial genes (here although see here). Is it likely that Vibrio cells will grow equally well if you replace their copy of rpoD with that of Pseudomonas? Probably not. On average then, forgive the lack of a mathematical model but I could whip one up if you'd really like, it is probably much easier to lower fitness of Vibrio through transformation after killing by T6S than to increase fitness. Added to this, analogous to alleles that lower competence in Rosie's model, is that genes that render strains sensitive to killing by T6S will be overrepresented in the transformable DNA pool.

4) Last but not least...I can understand why authors and press releases would be spun to suggest a tight evolutionary link between T6S, competence, and genetic exchange. As Rosie has pointed out, it's a much cleaner evolutionary story to think that predator cells are killing prey for nutrition. Also see her comment on Ed's blog post (here). The authors chose to play up the genetic exchange angle rather than test whether DNA from killed cells could be used as a nutrient. They don't even mention that DNA (and proteins, and a bunch of other things from lysed cells) could be used as a nutrient even though they use the terms predator and prey. Now to bring everything full circle, TfoX is actually the ortholog of Sxy, the gene in Haemophilus influenzae that Rosie's nutrient research is focused on. C'mon folks, at least acknowledge the literature.

So in conclusion, it's a nice genetics story.

Monday, November 10, 2014

We're Recruiting a Graduate Student to Study Microbiomes Inside Fungi

The Baltrus lab is looking to recruit a graduate student to start in the Fall of 2015. We have funds to support this student for at least 2 years, but am very willing to work with students to apply for outside fellowships to cover expenses past this point. Outside of this, the School of Plant Sciences at U of A does have opportunities for TA support on a competitive basis. Students are also encouraged to apply for the ABBS program (here), which provides support for rotations during the first year of graduate school. There's also the chance the grant gets renewed in a couple of years, fingers crossed. As such we're interested in recruiting either a Masters or PhD level student, given implicit funding caveats described above.

The student is being recruited to work on an emerging model system for multi-host symbioses. The Arnold lab (a close collaborator on this project) recently described a phylogenetically diverse group of facultative bacterial symbionts, found within a phylogenetically diverse group of fungal endophytes, found within a diverse group of host plants (here). Yes, even fungi harbor microbiomes! Betsy's lab has also demonstrated that these bacterial symbionts can mediate fungal metabolism (here). To this point, all of the bacterial symbionts are able to be grown and maintained under standard laboratory conditions. We have since established a protocol to cure and reinfect fungi with different bacterial symbionts and have demonstrated that effects on fungal metabolism are 1) specific to the bacteria isolate 2) specific to the fungal host. We have also been able to obtain complete genomes for 12 diverse bacterial symbionts. The new graduate student will work with me to tease apart the molecular basis for symbiotic phenotypes using basic microbial genetics approaches coupled with comparative genomics and transcriptomics.

There are many, many open questions at both evolutionary and ecological levels within this system. I am particularly interesting in setting up laboratory evolution experiments using these bacterial and fungal strains. In addition to the experiments described above, I'm open to helping this graduate student develop new research directions within the context of this system and encouraging of experimental independence. The deadline for admissions to UA is December 15th, and additional details can be found at the Plant Sciences admissions page (here).

If you have any questions, please feel free to contact me by email.

EDIT: even if you don't have questions feel free to email me. It's really important that you get along with your grad school advisor, so set up this connection early!

Saturday, October 4, 2014

CAREER grant post mortem

I received the grant rejection email last week...and waited until today to look at the reviews. For a couple of reviewers I hit everything just right, and for a couple of other reviewers it was the opposite.

I'm not going to go through these line by line, but I think overall I got a fair shot. I could have gone into a lot more detail about what reviewer 2 wanted to see, and I actually have in previous iterations of this grant, but decided to not go to heavy on lots of work on gene duplications. Suffice it to say I grew up as a scientist in the EvoDevo program at Oregon, and my first paper is actually on gene duplications. Will admit to being a bit stung by the "overgeneralization" part, because anyone that knows me knows I am well aware of every nuance...but  in science these days, you win some you lose a lot more than some. The reviewers made good points, I just tried to go heavy on the outreach part of the grant and had to sacrifice some science to make the page limit. I would have also liked to have had some more preliminary data under my belt (KOs of some of the genes and phenotyping), but right now I've got two very capable undergrads working on that for next year. C'est la vie.

So, as a resource for everyone out there:

There were a total of 36 grants in my panel, MCB Genetic Mechanisms. 1 was ranked High priority, 21 (including this one) were ranked Medium priority, 10 were Low priority, and 4 were Non-competitive.

My grant can be found here
Reviews of the grant can be found here

Disqus for