I study sample- and compute-efficient learning in systems that improve as they condition on more context: meta-training of in-context learners, continual pretraining, and scaling analysis that characterizes their behaviour.
I'm a PhD student at Mila and Université de Montréal, advised by Irina Rish. Previously, I co-founded Nolano, where my work focused on low-bit pretraining and inference for foundation models, including the Spectra suite of ternary language models (ICLR 2025 Spotlight) and TriTera (ACL 2025), which added scaling laws, training on 1.2T tokens, and fast CPU and GPU inference for ternary models. I hold an MS from UT Austin and a B.Tech from IIT Kharagpur, and was a research intern at Google, where I worked on encoder architectures for streaming sequence tagging.
Showed that most of the cost of pretraining tabular foundation models comes from training practices inherited from other modalities, and built MegaTabICL: a pretraining framework and stabilized BF16 recipe that match TabICLv2's training trajectory at 3.86× lower stage-1 cost. With it, ran, to our knowledge, the first billion-parameter compute-optimal scaling study of tabular foundation models, which shows the compute-optimal model size growing with the training budget.
Proposed HEAR, a hybrid unidirectional-bidirectional encoder with adaptive restarts for streaming sequence tagging. It reduces streaming compute by 58.9% on average across four tasks (up to 71.1%) while keeping offline accuracy and improving streaming exact match by up to 10 points.
Probed what subword embeddings encode about their constituent characters, and traced how pretraining induces that information.
Stance detection (NAACL 2021) and the causal direction of data collection in NLP (EMNLP 2021); contributed to JuliaText through Google Summer of Code.
A complete list is on Google Scholar.
MegaTabICL: Efficient Scaling of Tabular Foundation Model Pretraining [Blog: MegaTabICL: Efficient Scaling of Tabular Foundation Model Pretraining] [Paper: MegaTabICL: Efficient Scaling of Tabular Foundation Model Pretraining]
NeurIPS 2026 Workshop AXIOM (poster)
Scaling Laws and Efficient Inference for Ternary Language Models [Blog: Scaling Laws and Efficient Inference for Ternary Language Models] [Paper: Scaling Laws and Efficient Inference for Ternary Language Models]
ACL 2025
Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale [Blog: Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale] [Paper: Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale]
ICLR 2025 (Spotlight)
LoRD: Low-Rank Decomposition of Monolingual Code LLMs for One-Shot Compression [Blog: LoRD: Low-Rank Decomposition of Monolingual Code LLMs for One-Shot Compression] [Paper: LoRD: Low-Rank Decomposition of Monolingual Code LLMs for One-Shot Compression]
ICML 2024 Workshop on Foundation Models in the Wild
Efficient Encoders for Streaming Sequence Tagging [Blog: Efficient Encoders for Streaming Sequence Tagging] [Paper: Efficient Encoders for Streaming Sequence Tagging]
EACL 2023
Research Internship Project at Google, Mountain View.
What do tokens know about their characters and how do they know it? [Blog: What do tokens know about their characters and how do they know it?] [Paper: What do tokens know about their characters and how do they know it?] [Code: What do tokens know about their characters and how do they know it?]
NAACL 2022
* Equal contribution.
Why tabular foundation model pretraining is expensive, a pretraining framework and stabilized BF16 recipe that cut TabICLv2's stage-1 cost 3.86×, and, to our knowledge, the first billion-parameter compute-optimal scaling study of these models.
Continue reading: MegaTabICL: Efficient Scaling of Tabular Foundation Model PretrainingScaling laws for ternary language models, the TriTera suite trained on 1.2T tokens, lossless 2-bit and 1.6-bit weight packing, and TriRun GPU kernels.
Continue reading: Scaling Laws and Efficient Inference for Ternary Language Models
The first open suite of language models spanning multiple bit widths: 54 ternary, quantized and FP16 models from 99M to 3.9B parameters on the same 300B tokens. A 3.9B ternary model matches its FP16 counterpart across benchmarks while using fewer bits than an 830M FP16 model.
Continue reading: Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale
A 9B English-Hindi model trained on 3,072 GPUs of the Summit supercomputer. Continued pretraining from Pythia added Hindi while improving English and code, outperforming the larger Pythia 12B.
Continue reading: Hi-NOLIN: English-Hindi Continued Pretraining of PythiaStarCoder 16B compressed to 13B parameters by one-shot low-rank decomposition, with HumanEval pass@1 of 31.57% versus 31.67% for the original model.
Continue reading: LoRD: Low-Rank Decomposition of Monolingual Code LLMs for One-Shot CompressionLLaMA with 4-bit weights generating text on a MacBook CPU in C/C++, shown publicly on March 10, 2023 (UTC), two weeks after LLaMA was announced.
Continue reading: llama-int4-quant: 4-bit LLaMA Inference on a Laptop CPUHEAR: a hybrid unidirectional-bidirectional encoder with adaptive restarts that cuts streaming compute by 58.9% on average while improving streaming accuracy.
Continue reading: Efficient Encoders for Streaming Sequence TaggingProbing what subword embeddings encode about their constituent characters, and how pretraining induces that information.
Continue reading: What do tokens know about their characters and how do they know it?
The sequence labelling models, word tokenizers and text analysis tools I worked on for JuliaText.
Continue reading: My Journey in Google Summer of Code