Advances in machine learning, notably deep learning, have led computers to match or surpass human performance in several cognitive tasks including vision, speech and natural language processing. However, implementation of neural algorithms in conventional "von-Neumann" architectures are several orders of magnitude more area and power expensive than the biological brain. Hence, we need fundamentally new approaches to sustain the exponential growth in performance at high energy-efficiency. Exploring the new paradigm of computing necessitates a multi-disciplinary approach: exploration of new learning algorithms inspired from neuroscientific principles, developing network architectures best suited for such algorithms, new hardware techniques to achieve orders of improvement in energy consumption, and nanoscale devices that can closely mimic the neuronal and synaptic operations. In this talk, I will present our recent work on spike-based learning to achieve high energy efficiency with accuracy comparable to that of standard analog deep-learning techniques. Input coding from DVS cameras has been used to develop energy efficient hybrid SNN/ANN networks for optical flows, gesture recognition, and language translation. Additionally, we propose probabilistic neural and synaptic computing platforms that can leverage the underlying stochastic device physics of spin-devices. System-level simulations indicate ~100x improvement in energy consumption for such spintronic implementations over a corresponding CMOS implementation across different computing workloads. Complementary to the above device efforts, we have explored different local/global learning algorithms including stochastic learning with one-bit synapses that greatly reduces the storage/bandwidth requirement while maintaining competitive accuracy, and adaptive online learning that efficiently utilizes the limited memory and resource constraints to learn new information without catastrophically forgetting already learnt data.