I study compute-efficient learning in systems that improve as they condition on more context: meta-training of in-context learners, continual pretraining, and scaling analysis that characterizes their behaviour.
I'm a PhD student at Mila and Université de Montréal, advised by Irina Rish. Previously, I co-founded Nolano, where my work focused on low-bit pretraining and inference for foundation models, including the Spectra suite of ternary language models (ICLR 2025 Spotlight) and its scaling-law follow-up (ACL 2025). I hold an MS from UT Austin and a B.Tech from IIT Kharagpur, and was a research intern at Google, where I worked on encoder architectures for streaming sequence tagging.
Developed MegaTabICL, an exact reimplementation of TabICLv2 pretraining with 3.86× lower stage-1 cost, and used it for, to our knowledge, the first billion-parameter compute-optimal scaling study of tabular foundation models.
Implemented 4-bit LLaMA inference on CPU on top of ggml (March 2023). Proposed LoRD, one-shot low-rank decomposition for compressing code LLMs. Hi-NOLIN: English-Hindi continued pretraining of Pythia. Spectra: an open suite of ternary, quantized and FP16 language models trained on identical data, for controlled comparison of low-bit pretraining. TriTera (Spectra 1.1): scaling laws and inference kernels for ternary language models.
Proposed HEAR, a hybrid unidirectional-bidirectional encoder with adaptive restarts, which avoids re-encoding the full prefix at every new token in streaming sequence tagging.
Probed what subword embeddings encode about their constituent characters, and traced how pretraining induces that information.
Stance detection (NAACL 2021) and the causal direction of data collection in NLP (EMNLP 2021); contributed to JuliaText through Google Summer of Code.
A complete list is on Google Scholar.
MegaTabICL: Efficient Scaling of Tabular Foundation Model Pretraining [Blog: MegaTabICL: Efficient Scaling of Tabular Foundation Model Pretraining] [Paper: MegaTabICL: Efficient Scaling of Tabular Foundation Model Pretraining]
NeurIPS 2026 Workshop AXIOM (poster)
Scaling Laws and Efficient Inference for Ternary Language Models [Blog: Scaling Laws and Efficient Inference for Ternary Language Models] [Paper: Scaling Laws and Efficient Inference for Ternary Language Models]
ACL 2025
Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale [Blog: Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale] [Paper: Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale]
ICLR 2025 (Spotlight)
LoRD: Low-Rank Decomposition of Monolingual Code LLMs for One-Shot Compression [Blog: LoRD: Low-Rank Decomposition of Monolingual Code LLMs for One-Shot Compression] [Paper: LoRD: Low-Rank Decomposition of Monolingual Code LLMs for One-Shot Compression]
ICML 2024 Workshop on Foundation Models in the Wild
Efficient Encoders for Streaming Sequence Tagging [Blog: Efficient Encoders for Streaming Sequence Tagging] [Paper: Efficient Encoders for Streaming Sequence Tagging]
EACL 2023
Research Internship Project at Google, Mountain View.
What do tokens know about their characters and how do they know it? [Blog: What do tokens know about their characters and how do they know it?] [Paper: What do tokens know about their characters and how do they know it?] [Code: What do tokens know about their characters and how do they know it?]
NAACL 2022
* Equal contribution.
An exact reimplementation of TabICLv2 pretraining at 3.86× lower stage-1 cost, used for a billion-parameter compute-optimal scaling study.
Continue reading: MegaTabICL: Efficient Scaling of Tabular Foundation Model PretrainingScaling laws for ternary language models, the TriTera suite trained on 1.2T tokens, lossless 2-bit and 1.6-bit weight packing, and TriRun GPU kernels.
Continue reading: Scaling Laws and Efficient Inference for Ternary Language Models
An open suite of 54 ternary, quantized and FP16 language models from 99M to 3.9B parameters, all trained on the same 300B tokens.
Continue reading: Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale
An English-Hindi language model obtained by continued pretraining of Pythia from 7B to 9B parameters, released under Apache 2.0.
Continue reading: Hi-NOLIN: English-Hindi Continued Pretraining of PythiaStarCoder 16B compressed to 13B parameters by one-shot low-rank decomposition, with HumanEval pass@1 of 31.57% versus 31.67% for the original model.
Continue reading: LoRD: Low-Rank Decomposition of Monolingual Code LLMs for One-Shot CompressionLLaMA with 4-bit weights generating text on a MacBook CPU in C/C++, shown publicly on March 10, 2023 (UTC), two weeks after LLaMA was announced.
Continue reading: llama-int4-quant: 4-bit LLaMA Inference on a Laptop CPUHEAR: a hybrid unidirectional-bidirectional encoder with adaptive restarts that avoids re-encoding the full prefix at every new token.
Continue reading: Efficient Encoders for Streaming Sequence TaggingProbing what subword embeddings encode about their constituent characters, and how pretraining induces that information.
Continue reading: What do tokens know about their characters and how do they know it?
The sequence labelling models, word tokenizers and text analysis tools I worked on for JuliaText.
Continue reading: My Journey in Google Summer of Code