Recent blog posts by Andrej Karpathy at Medium.com and Pete Warden at PeteWarden.com have caused a paradigm shift in the way I think about neural nets. Instead of thinking of them as powerful machine learning tools, the authors instead suggest that we should think of neural nets, and in particular, convolution deep nets, as ‘self-writing programs.’ Hence the term, “Software 2.0.”
It turns out that a large portion of real-world problems have the property that it is significantly easier to collect the data than to explicitly write the program. A large portion of programmers of tomorrow do not maintain complex software repositories, write intricate programs, or analyze their running times. They collect, clean, manipulate, label, analyze and visualize data that feeds neural networks. — Andrej Karpathy, Medium.com
I found this to be a dramatic reversal in my thinking about these techniques, but it opens up a deeper understanding and is much more intuitive. The fact is that combinations of artificial neurons can be used to model any logical operation. Therefore you can conceptualize training a neural net as searching programming space for an optimal program that behaves in the way you specify. You provide the inputs and desired outputs, and the model searches for the optimal program.
This stands in contrast to the “Software 1.0” paradigm where the programmer uses her skill and experience to conceptualize the right combination of specific instructions to produce the desired behavior. While it seems certain that Software 1.0 and 2.0 will co-exist for a long time, this new way of understanding deep learning is crucial and exciting, in my opinion.
Here are five different ways to gain an introduction to the topic of CNNs. Each approach is geared toward a different style of learning:
Visualize them in real time with your own inputs (this is amazing!)
Watch a lecture by the “godfather” of neural nets, Geoff Hinton.
Take a top-ranked online course on Deep Learning.
Learn the math behind them.
This blog named a “Top 75 in Bioinformatics” by Feedspot.com!
I made the list at #58. I’m proud of that fact, but I want to push into the top 30 on the internet. I plan to increase my rate of posting new articles and also up my game on content and analysis. Stay tuned!
I just got back from Great Lakes Bio 2017 (GLBIO2017) at the University of Illinois-Chicago (UIC) campus. It was a great meeting and I really enjoyed the quality of the research presented as well as the atmosphere of the campus and neighborhood.
I was very surprised by just how nice the Chicago “West Loop” neighborhood near Randolph Street and down towards Greektown really is. I had some great meals, including a memorable Italian dinner at Formentos.
But the purpose of this post is to briefly describe a few of my favorite talks from the meeting. So here goes, in no particular order:
Kevin White, Tempus Labs:
I was really impressed with Kevin White’s GLBIO2017 talk and demo of his company’s technology (despite the ongoing technical A/V issues!) Tempus labs is a clinical sequencing company but also an informatics company focused on cancer treatment that seeks to pull together all of the disparate pieces of patient data that float around in EHR databases and are oftentimes not connected in meaningful ways.
The company sequences patient samples (whole exome and whole genome) and then also hoovers up reams of patient EHR data using Optical Character Recognition (OCR), Natural Language Processing (NLP), and human expert curation to turn the free-form flat text of medical records from different clinics and systems into a form of “tidy data” that can be accessed from an internal database.
Then, clinical and genomic data are combined for each patient in a deep-learning system that looks at treatments and outcomes for other similar patients and presents the clinician with charts that show how patients in similar circumstances fared with varying treatments, given certain facts of genotype and tumor progression, etc… The system is pitched as “decision support” rather than artificial “decision making.” That is, a human doctor is still the primary decider of treatment for each patient, but the Tempus deep learning system will provide expert support and suggest probabilities for success at each critical care decision point.
The system also learns and identifies ongoing clinical trials, and will present relevant trials to the clinician so that patients can be informed of possibly beneficial trials that they can join.
Murat Eren, merenlab.org
Murat Eren’s talk on tracking microbial colonization in fecal microbiome transplantation (i.e., “poop pills”) was excellent and very exciting. Although the “n” was small (just 4 donors and 2 recipients) he showed some very interesting results from transferring fecal microbiota (FM) from healthy individuals to those with an inflammatory bowel disease.
Among the interesting results are the fact that he was able to assemble 97 metagenomes in the 4 donor samples. Following the recipients at 4 and 8-weeks post FM transplant showed that the microbial genomes could be classed into those that transfer and colonize permissively (both recipients), those that colonize one or the other recipient, and those that fail to colonize both. Taxa alone did not explain why some microbes colonized easily, while other failed to colonize.
He also showed that 8 weeks post FM transplant, the unhealthly recipients had improved symptoms but also showed that in a PCA analysis of the composition of the recipient gut and the healthy human gut from 151 human microbiome project (HMP) samples, the recipients moved into the “healthy” HMP cluster from being extreme outliers on day 0.
He also investigated differential gene function enrichment between the permissive colonizers and the microbes that never colonized recipient’s guts and found that sporulation genes may be a negative factor driving the failure (or success) of transplantation. He proposed that the recent and notable failure of the Seres microbiome drug in clinical trials may be owing to the fact that the company killed the live cultures in favor of more stable spore-forming strains when formulating the drug. His work would suggest that these strains are less successful at colonizing new hosts.
Bo Zhang, 3D genome browser
With the ever-increasing volume of genomic and regulatory data and the complexity of that data, there is a need for accessible interfaces to it. Bo Zhang’s group at Penn State has worked to make a new type of genome browser available that focuses on the 3D structure of the genome, pulling together disparate datatypes including chromatin interaction data, ChIP-Seq, RNA-Seq, etc… You can also browse a complete view of the regulatory landscape and 3D architecture of any region of the genome. You can also check the expression of any queried gene across hundreds of tissue/cell types measured by the ENCODE consortium. On the virtual 4C page, they provide multiple methods to link distal cis-regulatory elements with their potential target genes, including virtual 4C, ChIA-PET and cross-cell-type correlation of proximal and distal DHSs.
All in all, GLBIO2017 was a very enjoyable and informative meeting where I met a lot of great colleagues and learned much. I am looking forward to next year!
There is an interesting new post at “In the Pipeline” that summarizes the performance of Google’s “big data” project to track flu trends from search terms. In short, the predictive performance appears to be pretty bad so far, at least compared to what you might have expected given the hype around “big data.” The author raises some key points, including the importance of high-quality data, even in very large datasets. I particularly like this analogy:
“The quality of the data matters very, very, much, and quantity is no substitute. You can make a very large and complex structure out of toothpicks and scraps of wood, because those units are well-defined and solid. You cannot do the same with a pile of cotton balls and dryer lint, not even if you have an entire warehouse full of the stuff.” –In the Pipeline, March 24, 2014
Data filtering and modeling approaches will likely continue to improve, however, and I think this project is worth watching in the future.
People often assume that the world tomorrow will be pretty much like the world today. We all have an in-built bias towards linear thinking when we ponder the future. Although a linear bias was helpful for thousands of years of our evolution, today technology is changing at an exponential pace and in order to better anticipate future market opportunities and technology’s impact on society, it is crucial to think in terms of exponential trends. This is a point that renowned futurist Ray Kurzweil has made in his many books and speeches for the last several decades.
We all have an in-built bias towards linear thinking when we ponder the future.
One example of an exponential trend in biology (among many) is the cost per genome sequence (graph below). As recently as 2001, the cost to sequence a genome was an astronomical $100M. Between 2001 and 2007, the cost decreased exponentially (a straight line on a log plot), to the point where a genome in 2007 cost only $10M to sequence. Around 2007, a paradigm shift in technology massively accelerated this exponential process, and the cost decreased even faster than before, hitting just $10K in 2012.
The dramatic, exponential gains in price/performance of sequencing technology have unleashed a tidal wave of sequence data.
As economists are fond of saying, when the price falls, more is demanded. As a result of this massively reduced sequencing price, many more partial and complete genomes are being sequenced than ever before. The dramatic, exponential gains in price/performance of sequencing technology have unleashed a tidal wave of sequence data.