Deep learning capabilities and limitations

Deep learning capabilities

Neural networks have been around for a long time, but they did not perform very well and were quite computationally intensive, so the machine learning community devoted its energy to other approaches. Then, a few things changed. First, many of these approaches were parallelized, meaning that they could take advantage of GPUs, and making it feasible to use these approaches on larger data sets and more complex problems. Second, in 2012 Andrew Ng at Google took large volumes of image data, available from online resources, and made his neural networks larger and deeper, taking advantage of the compute power from parallelization, and showed that image classification could work remarkably well with these deep neural networks. This set off the deep learning community, and a slew of applications that take advantage of the power of deep networks to learn complex representations.

Recent applications that use deep neural networks highlight the capabilities of these approaches to a variety of domains. The depth of the networks increases both the number and complexity of features that can be learned by the system to drive high accuracy classification and regression. In addition, because these networks capture qualities of a dataset, they can be used to 'generate' new data which mimics the data on which it was trained. This means that learners can create new images of faces from looking at lots of examples of human faces during training.

For more info on the difference between AI, machine learning and deep learning go here: https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/

Limitations of Deep Learning

Deep Learning as a Black-box

The main criticism of deep learning is that it is used as a black box. The term "black box" implies that when somebody uses the method, they are only concerned with the input and output, and they do not know what happens to transform that input into the output. One goal of this site is to help transition from this black-box testing to white-box testing, wherein the user knows and understands the tools being used. We hope to do this by presenting some of the basic deep learning methodologies and explaining how they work as well as giving examples of their use in biological settings as well as other settings.

Model Selection in Deep Learning

Deep Learning Requires Larger Training Sets

Another limitation of deep learning comes from the training requirements. Deep learning models require much larger datasets than other machine learning models. To address this limitation, many models will up-sample their data and add random noise to generate extra training samples. Hopefully, as more and more biological data is collected, this limitation will become less relevant.

Along with requiring a larger training set, deep learning is also dependent on a balanced training set. To handle an imbalance in the data, preprocessing techniques can be used to either up-sample or down-sample different classes in order to balance the data. Another approach is to use cost sensitive functions to penalize some classes more than others, which can be thought of us penalizing a class error based on its proportion of the training set. Lastly, pretraining has been shown to help with imbalanced datasets. In this case, the model is first trained on a balanced set of similar data to get a rough estimation of the optimized weights. Afterwards, the real data is used to fine-tune the model.

Interpretability of Deep Learning

An important limitation of deep learning is the lack of interpretability for the models. This is extremely important to address when dealing with biology since more often than not, the accuracy is less important than the reason the model achieved that accuracy. For example, if you could predict if somebody was going to develop a disease at 100% accuracy, but you could not explain why that person would develop that disease, the mechanisms would still remain unknown. These mechanisms are often more important than the accuracy since downstream experiments using targeted studies and drug treatment rely on knowing what mechanisms are important.

References

1. Copeland M. What’s the Difference Between Artificial Intelligence, Machine Learning, and Deep Learning? NVIDIA. July 29, 2016. https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/. Accessed May 3, 2017.
2. Knight W. The Dark Secrets at the Heart of AI. MIT Technology Review. April 11, 2017. https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/. Accessed May 3, 2017.
3. Yao M. Understanding the limits of deep learning. VentureBeat. April 2, 2017. https://venturebeat.com/2017/04/02/understanding-the-limits-of-deep-learning/. Accessed May 3, 2017.
4. Min S, Lee B, Yoon S. Deep learning in bioinformatics. Brief Bioinform 2016. doi: 10.1093/bib/bbw068