Open preprint reviews by Anshul Kundaje

Fast, scalable prediction of deleterious noncoding variants from functional and population genomic data

Yi-Fei Huang, Brad Gulko, Adam Siepel

Review posted on 10th October 2016

Very nice paper. A few questions and clarifications.

1. Whats the negative set you used in the TFBS prediction evaluation (supp. fig 2). Its not clear from reading the methods.

2. Also was evaluation of each method done on held out chromosomes for that specific method i.e. chromosomes not used in training? E.g. DeepSEA holds out chr8 and 9 and trains on data from all other chromosomes for all data types across a range of cell types. So if you are evaluating performance of DeepSEA on sites in the training chromosomes, its not going to reflect test performance but rather training performance. Same goes for all other methods, unless you retrained them on all common training/test settings.
3. Also please avoid reporting auROCs for TFBS prediction evaluation or for that matter any unbalanced prediction problem on the genome. They can be very misleading. auROCs of >0.9 can translate to terrible auPRCs (< 0.2) and very poor recall at reasonable FDRs (e.g. < 1% recall at 50% FDR). Could you please report auPRCs and recall at reasonable FDR thresholds?

show less