One Apple Park Way

MS 929-3SIR

Cupertino, CA 95014

Hi! I am a senior machine learning engineer at Apple, a co-founder and CTO at Illumia AI, and involved with other stealth startups. Before this, I was a senior research scientist at Samsung Research, where I worked on efficient on-device AI models. I completed my Ph.D. at the Department of Electrical and Computer Engineering, Princeton University under the supervision of Niraj K. Jha. My main research areas are Machine Learning, Neuroscience-inspired AI, and Hardware-software Co-design. I did my undergraduate studies at the Department of Electrical Engineering, Indian Institute of Technology Delhi. I have also worked as a visiting researcher at the Embedded Systems Laboratory (ESL), Institute of Electrical Engineering at EPFL, Switzerland.

My publications can be seen here. For an up-to-date list, check out my Google Scholar. I have reviewed for top journals including IEEE TCAD, TEVC, TETC, TII, and Wiley SPE. I have also served as a reviewer for reputed conferences like CISS, CogSci, and ICML. View my CV here.

news

Jan 27, 2025	Joining Apple as a Senior ML Engineer
Jan 29, 2024	Joining Samsung AI Center as a Senior Research Scientist
Nov 30, 2023	Officially a Doctor -- I've Completed My PhD!
Sep 28, 2023	Received SEAS Award for Excellence
May 6, 2023	Joining Samsung AI Center as Summer Research Intern
Jun 22, 2022	Top Downloaded Article Award for survey paper on Smart Healthcare
May 23, 2022	Received Pramod Subramanyan *17 Early Career Graduate Award

selected publications

1. EdgeTran: Device-Aware Co-Search Of Transformers for Efficient Inference on Mobile Edge Platforms

   Tuli, Shikhar, and Jha, Niraj K.

   IEEE Transactions on Mobile Computing 2023

   Abs Bibtex CC 5

   Automated design of efficient transformer models has recently attracted significant attention from industry and academia. However, most works only focus on certain metrics while searching for the best-performing transformer architecture. Furthermore, running traditional, complex, and large transformer models on low-compute edge platforms is a challenging problem. In this work, we propose a framework, called ProTran, to profile the hardware performance measures for a design space of transformer architectures and a diverse set of edge devices. We use this profiler in conjunction with the proposed co-design technique to obtain the best-performing models that have high accuracy on the given task and minimize latency, energy consumption, and peak power draw to enable edge deployment. We refer to our framework for co-optimizing accuracy and hardware performance measures as EdgeTran. It searches for the best transformer model and edge device pair. Finally, we propose GPTran, a multi-stage block-level grow-and-prune post-processing step that further improves accuracy in a hardware-aware manner. The obtained transformer model is 2.8 smaller and has a 0.8% higher GLUE score than the baseline (BERT-Base). Inference with it on the selected edge device enables 15.0% lower latency, 10.0x lower energy, and 10.8x lower peak power draw compared to an off-the-shelf GPU.

   @article{tuli2023edgetran, author = {Tuli, Shikhar and Jha, Niraj K.}, citationlist = {http://scholar.google.com/scholar?cites=17381555235815555284&as_sdt=2005&sciodt=0,5&hl=en}, citations = {5}, journal = {IEEE Transactions on Mobile Computing}, selected = {true}, title = {EdgeTran: Device-Aware Co-Search Of Transformers for Efficient Inference on Mobile Edge Platforms}, year = {2023} }

2. BREATHE: Second-Order Gradients and Heteroscedastic Emulation based Design Space Exploration

   Tuli, Shikhar, and Jha, Niraj K.

   arXiv Preprint 2023

   Abs arXiv Bibtex CC 0

   Researchers constantly strive to explore larger and more complex search spaces in various scientific studies and physical experiments. However, such investigations often involve sophisticated simulators or time-consuming experiments that make exploring and observing new design samples challenging. Previous works that target such applications are typically sample-inefficient and restricted to vector search spaces. To address these limitations, this work proposes a constrained multi-objective optimization (MOO) framework, called BREATHE, that searches not only traditional vector-based design spaces but also graph-based design spaces to obtain best-performing graphs. It leverages second-order gradients and actively trains a heteroscedastic surrogate model for sample-efficient optimization. In a single-objective vector optimization application, it leads to 64.1% higher performance than the next-best baseline, random forest regression. In graph-based search, BREATHE outperforms the next-best baseline, i.e., a graphical version of Gaussian-process-based Bayesian optimization, with up to 64.9% higher performance. In a MOO task, it achieves up to 21.9× higher hypervolume than the state-of-the-art method, multi-objective Bayesian optimization (MOBOpt). BREATHE also outperforms the baseline methods on most standard MOO benchmark applications.

   @article{tuli2023breathe, arxiv = {2308.08666}, author = {Tuli, Shikhar and Jha, Niraj K.}, citations = {0}, journal = {arXiv Preprint}, selected = {true}, title = {BREATHE: Second-Order Gradients and Heteroscedastic Emulation based Design Space Exploration}, year = {2023} }

3. TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference

   Tuli, Shikhar, and Jha, Niraj K.

   IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 2023

   Abs Bibtex CC 6

   Automated co-design of machine learning models and evaluation hardware is critical for efficiently deploying such models at scale. Despite the state-of-the-art performance of transformer models, they are not yet ready for execution on resource-constrained hardware platforms. High memory requirements and low parallelizability of the transformer architecture exacerbate this problem. Recently-proposed accelerators attempt to optimize the throughput and energy consumption of transformer models. However, such works are either limited to a one-sided search of the model architecture or a restricted set of off-the-shelf devices. Furthermore, previous works only accelerate model inference and not training, which incurs substantially higher memory and compute resources, making the problem even more challenging. To address these limitations, this work proposes a dynamic training framework, called DynaProp, that speeds up the training process and reduces memory consumption. DynaProp is a low-overhead pruning method that prunes activations and gradients at runtime. To effectively execute this method on hardware for a diverse set of transformer architectures, we propose ELECTOR, a framework that simulates transformer inference and training on a design space of accelerators. We use this simulator in conjunction with the proposed co-design technique, called TransCODE, to obtain the best-performing models with high accuracy on the given task and minimize latency, energy consumption, and chip area. The obtained transformer-accelerator pair achieves 0.3% higher accuracy than the state-of-the-art pair while incurring 5.2x lower latency and 3.0x lower energy consumption.

   @article{tuli2023transcode, author = {Tuli, Shikhar and Jha, Niraj K.}, citationlist = {http://scholar.google.com/scholar?cites=15333846075507847242&as_sdt=2005&sciodt=0,5&hl=en}, citations = {6}, journal = {IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems}, selected = {true}, title = {TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference}, year = {2023} }

4. AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers

   Tuli, Shikhar, and Jha, Niraj K.

   IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 2023

   Abs Bibtex CC 29

   Self-attention-based transformer models have achieved tremendous success in the domain of natural language processing. Despite their efficacy, accelerating the transformer is challenging due to its quadratic computational complexity and large activation sizes. Existing transformer accelerators attempt to prune its tokens to reduce memory access, albeit with high compute overheads. Moreover, previous works directly operate on large matrices involved in the attention operation, which limits hardware utilization. In order to address these challenges, this work proposes a novel dynamic inference scheme, DynaTran, which prunes activations at runtime with low overhead, substantially reducing the number of ineffectual operations. This improves the throughput of transformer inference. We further propose tiling the matrices in transformer operations along with diverse dataflows to improve data reuse, thus enabling higher energy efficiency. To effectively implement these methods, we propose AccelTran, a novel accelerator architecture for transformers. Extensive experiments with different models and benchmarks demonstrate that DynaTran achieves higher accuracy than the state-of-the-art top-k hardware-aware pruning strategy while attaining up to 1.2x higher sparsity. One of our proposed accelerators, AccelTran-Edge, achieves 330Kx higher throughput with 93Kx lower energy requirement when compared to a Raspberry Pi device. On the other hand, AccelTran-Server achieves 5.73x higher throughput and 3.69x lower energy consumption compared to the state-of-the-art transformer co-processor, Energon.

   @article{tuli2023acceltran, author = {Tuli, Shikhar and Jha, Niraj K.}, citationlist = {http://scholar.google.com/scholar?cites=7364225806971428277&as_sdt=2005&sciodt=0,5&hl=en}, citations = {29}, journal = {IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems}, selected = {true}, title = {AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers}, year = {2023} }

5. CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework

   Tuli, Shikhar, Li, Chia-Hao, Sharma, Ritvik, and Jha, Niraj K.

   ACM Transactions on Embedded Computing Systems 2022

   Abs arXiv Code Bibtex CC 17

   Recently, automated co-design of machine learning (ML) models and accelerator architectures has attracted significant attention from both the industry and academia. However, most co-design frameworks either explore a limited search space or employ suboptimal exploration techniques for simultaneous design decision investigations of the ML model and the accelerator. Furthermore, training the ML model and simulating the accelerator performance is computationally expensive. To address these limitations, this work proposes a novel neural architecture and hardware accelerator co-design framework, called CODEBench. It comprises two new benchmarking sub-frameworks, CNNBench and AccelBench, which explore expanded design spaces of convolutional neural networks (CNNs) and CNN accelerators. CNNBench leverages an advanced search technique, BOSHNAS, to efficiently train a neural heteroscedastic surrogate model to converge to an optimal CNN architecture by employing second-order gradients. AccelBench performs cycle-accurate simulations for diverse accelerator architectures in a vast design space. With the proposed co-design method, called BOSHCODE, our best CNN-accelerator pair achieves 1.4% higher accuracy on the CIFAR-10 dataset compared to the state-of-the-art pair while enabling 59.1% lower latency and 60.8% lower energy consumption. On the ImageNet dataset, it achieves 3.7% higher Top1 accuracy at 43.8% lower latency and 11.2% lower energy consumption. CODEBench outperforms the state-of-the-art framework, i.e., Auto-NBA, by achieving 1.5% higher accuracy and 34.7x higher throughput while enabling 11.0x lower energy-delay product (EDP) and 4.0x lower chip area on CIFAR-10.

   @article{tuli2022codebench, arxiv = {2212.03965}, author = {Tuli, Shikhar and Li, Chia-Hao and Sharma, Ritvik and Jha, Niraj K.}, citationlist = {http://scholar.google.com/scholar?cites=17012787971509511244&as_sdt=2005&sciodt=0,5&hl=en}, citations = {17}, code = {https://github.com/jha-lab/codebench}, journal = {ACM Transactions on Embedded Computing Systems}, selected = {true}, title = {CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework}, year = {2022} }

6. DINI: Data Imputation using Neural Inversion for Edge Applications

   Tuli, Shikhar, and Jha, Niraj K.

   Nature Scientific Reports 2022

   Abs Code Bibtex CC 0

   The edge computing paradigm has recently drawn significant attention from industry and academia. Due to the advantages in quality-of-service metrics, namely, latency, bandwidth, energy efficiency, privacy, and security, deploying artificial intelligence (AI) models at the network edge has attracted widespread interest. Edge-AI has seen applications in diverse domains that involve large amounts of data. However, poor dataset quality plagues this compute regime owing to numerous data corruption sources, including missing data. As such systems are increasingly being deployed in mission-critical applications, mitigating the effects of corrupted data becomes important. In this work, we propose a strategy based on data imputation using neural inversion, DINI. It trains a surrogate model and runs data imputation in an interleaved fashion. Unlike previous works, DINI is a model-agnostic framework applicable to diverse deep learning architectures. DINI outperforms state-of-the-art methods by at least 10.7% in average imputation error. Applying DINI to mission-critical applications can increase prediction accuracy to up to 99% (F1 score of 0.99), resulting in significant gains compared to baseline methods.

   @article{tuli2022dini, author = {Tuli, Shikhar and Jha, Niraj K.}, citationlist = {#}, citations = {0}, code = {https://github.com/jha-lab/dini}, journal = {Nature Scientific Reports}, selected = {true}, title = {DINI: Data Imputation using Neural Inversion for Edge Applications}, year = {2022} }

7. FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?

   Tuli, Shikhar, Dedhia, Bhishma, Tuli, Shreshth, and Jha, Niraj K.

   Journal of Artificial Intelligence Research 2022

   Abs arXiv Code Bibtex CC 13

   The existence of a plethora of language models makes the problem of selecting the best one for a custom task challenging. Most state-of-the-art methods leverage transformer-based models (e.g., BERT) or their variants. Training such models and exploring their hyperparameter space, however, is computationally expensive. Prior work proposes several neural architecture search (NAS) methods that employ performance predictors (e.g., surrogate models) to address this issue; however, analysis has been limited to homogeneous models that use fixed dimensionality throughout the network. This leads to sub-optimal architectures. To address this limitation, we propose a suite of heterogeneous and flexible models, namely FlexiBERT, that have varied encoder layers with a diverse set of possible operations and different hidden dimensions. For better-posed surrogate modeling in this expanded design space, we propose a new graph-similarity-based embedding scheme. We also propose a novel NAS policy, called BOSHNAS, that leverages this new scheme, Bayesian modeling, and second-order optimization, to quickly train and use a neural surrogate model to converge to the optimal architecture. A comprehensive set of experiments shows that the proposed policy, when applied to the FlexiBERT design space, pushes the performance frontier upwards compared to traditional models. FlexiBERT-Mini, one of our proposed models, has 3% fewer parameters than BERT-Mini and achieves 8.9% higher GLUE score. A FlexiBERT model with equivalent performance as the best homogeneous model achieves 2.6x smaller size. FlexiBERT-Large, another proposed model, achieves state-of-the-art results, outperforming the baseline models by at least 5.7% on the GLUE benchmark.

   @article{tuli2022flexibert, arxiv = {2205.11656}, author = {Tuli, Shikhar and Dedhia, Bhishma and Tuli, Shreshth and Jha, Niraj K.}, citationlist = {http://scholar.google.com/scholar?cites=12913197364874416044&as_sdt=2005&sciodt=0,5&hl=en}, citations = {13}, code = {https://github.com/jha-lab/txf_design-space}, journal = {Journal of Artificial Intelligence Research}, selected = {true}, title = {FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?}, year = {2022} }

8. Generative Optimization Networks for Memory Efficient Data Generation

   Tuli, Shreshth, Tuli, Shikhar, Casale, Giuliano, and Jennings, Nicholas R

   NeurIPS 2021 - Workshop on ML for Systems 2021

   Abs arXiv Code Bibtex CC 7

   In standard generative deep learning models, such as autoencoders or GANs, the size of the parameter set is proportional to the complexity of the generated data distribution. A significant challenge is to deploy resource-hungry deep learning models in devices with limited memory to prevent system upgrade costs. To combat this, we propose a novel framework called generative optimization networks (GON) that is similar to GANs, but does not use a generator, significantly reducing its memory footprint. GONs use a single discriminator network and run optimization in the input space to generate new data samples, achieving an effective compromise between training time and memory consumption. GONs are most suited for data generation problems in limited memory settings. Here we illustrate their use for the problem of anomaly detection in memory-constrained edge devices arising from attacks or intrusion events. Specifically, we use a GON to calculate a reconstruction-based anomaly score for input time-series windows. Experiments on a Raspberry-Pi testbed with two existing and a new suite of datasets show that our framework gives up to 32% higher detection F1 scores and 58% lower memory consumption, with only 5% higher training overheads compared to the state-of-the-art.

   @article{tuli2021generative, arxiv = {2110.02912}, author = {Tuli, Shreshth and Tuli, Shikhar and Casale, Giuliano and Jennings, Nicholas R}, citationlist = {http://scholar.google.com/scholar?cites=15582971490951281448&as_sdt=2005&sciodt=0,5&hl=en}, citations = {7}, code = {https://github.com/imperial-qore/GON}, journal = {NeurIPS 2021 - Workshop on ML for Systems}, selected = {true}, title = {Generative Optimization Networks for Memory Efficient Data Generation}, year = {2021} }

9. Are Convolutional Neural Networks or Transformers more like human vision?

   Tuli, Shikhar, Dasgupta, Ishita, Grant, Erin, and Griffiths, Thomas L.

   Annual Meeting of the Cognitive Science Society (CogSci) 2021

   Abs arXiv Code Bibtex CC 247

   Modern machine learning models for computer vision exceed humans in accuracy on specific visual recognition tasks, notably on datasets like ImageNet. However, high accuracy can be achieved in many ways. The particular decision function found by a machine learning system is determined not only by the data to which the system is exposed, but also the inductive biases of the model, which are typically harder to characterize. In this work, we follow a recent trend of in-depth behavioral analyses of neural network models that go beyond accuracy as an evaluation metric by looking at patterns of errors. Our focus is on comparing a suite of standard Convolutional Neural Networks (CNNs) and a recently-proposed attention-based network, the Vision Transformer (ViT), which relaxes the translation-invariance constraint of CNNs and therefore represents a model with a weaker set of inductive biases. Attention-based networks have previously been shown to achieve higher accuracy than CNNs on vision tasks, and we demonstrate, using new metrics for examining error consistency with more granularity, that their errors are also more consistent with those of humans. These results have implications both for building more human-like vision models, as well as for understanding visual object recognition in humans.

   @article{tuli2021are, arxiv = {2105.07197}, author = {Tuli, Shikhar and Dasgupta, Ishita and Grant, Erin and Griffiths, Thomas L.}, citationlist = {http://scholar.google.com/scholar?cites=667529829221842193&as_sdt=2005&sciodt=0,5&hl=en}, citations = {247}, code = {https://github.com/shikhartuli/cnn_txf_bias}, journal = {Annual Meeting of the Cognitive Science Society (CogSci)}, selected = {true}, title = {Are Convolutional Neural Networks or Transformers more like human vision?}, year = {2021} }

10. AVAC: A Machine Learning based Adaptive RRAM Variability-Aware Controller for Edge Devices

    Tuli, Shikhar, and Tuli, Shreshth

    IEEE International Symposium on Circuits and Systems (ISCAS) 2020

    arXiv Bibtex CC 2

    @article{tuli2020avac, arxiv = {2005.03077}, author = {Tuli, Shikhar and Tuli, Shreshth}, citationlist = {http://scholar.google.com/scholar?cites=2777597957252907747&as_sdt=2005&sciodt=0,5&hl=en}, citations = {2}, journal = {IEEE International Symposium on Circuits and Systems (ISCAS)}, selected = {true}, title = {AVAC: A Machine Learning based Adaptive RRAM Variability-Aware Controller for Edge Devices}, year = {2020} }

11. RRAM-VAC: A variability-aware controller for RRAM-based memory architectures

    Tuli, Shikhar, Rios, Marco, Levisse, Alexandre, and Atienza, David

    Asia and South Pacific Design Automation Conference (ASPDAC) 2020

    Code Bibtex CC 7

    @article{tuli2020rram, author = {Tuli, Shikhar and Rios, Marco and Levisse, Alexandre and Atienza, David}, citationlist = {https://scholar.google.com/scholar?cites=7027936382486962646&as_sdt=5,31&sciodt=0,31&hl=en}, citations = {7}, code = {https://github.com/shikhartuli/rram-vac}, journal = {Asia and South Pacific Design Automation Conference (ASPDAC)}, pages = {181--186}, selected = {true}, title = {RRAM-VAC: A variability-aware controller for {RRAM}-based memory architectures}, year = {2020} }

12. Design of a Conventional-Transistor-Based Analog Integrated Circuit for On-Chip Learning in a Spiking Neural Network

    Tuli, Shikhar, and Bhowmik, Debanjan

    International Conference on Neuromorphic Systems (ICONS) 2020

    Bibtex CC 1

    @article{tuli2020design, articleno = {14}, author = {Tuli, Shikhar and Bhowmik, Debanjan}, citationlist = {https://scholar.google.com/scholar?cites=10186738064493107792&as_sdt=5,31&sciodt=0,31&hl=en}, citations = {1}, journal = {International Conference on Neuromorphic Systems (ICONS)}, numpages = {8}, selected = {true}, title = {Design of a Conventional-Transistor-Based Analog Integrated Circuit for On-Chip Learning in a Spiking Neural Network}, year = {2020} }

13. Predicting the Growth and Trend of COVID-19 Pandemic using Machine Learning and Cloud Computing

    Tuli, Shreshth, Tuli, Shikhar, Tuli, Rakesh, and Gill, Sukhpal Singh

    Internet of Things 2020

    medrXiv Code Bibtex CC 539

    @article{tuli2020predicting, author = {Tuli, Shreshth and Tuli, Shikhar and Tuli, Rakesh and Gill, Sukhpal Singh}, citationlist = {http://scholar.google.com/scholar?cites=832410515520312149&as_sdt=2005&sciodt=0,5&hl=en}, citations = {539}, code = {https://github.com/shreshthtuli/covid-19-prediction}, journal = {Internet of Things}, medrxiv = {10.1101/2020.05.06.20091900v1}, pages = {100222}, publisher = {Elsevier}, selected = {true}, title = {Predicting the Growth and Trend of COVID-19 Pandemic using Machine Learning and Cloud Computing}, year = {2020} }

My GitHub coding calendar.

The best way to reach me is via email. :)