About me

Hi There!

I am a Graduate Student at University of Texas at Austin. I am currently working on getting models more powerful than GPT-3 running on our personal device and solve real-world problems using it. (Spoilers: It will be done by mid-March'23).

Previously, I have done various research projects at Google (efficient neural models), IBM, ETH Zurich, University of Oregon and IIT Kharagpur. My publications are at EACL'23, NAACL'22 & 21, EMNLP, etc. But my most exciting work is what I am currently working on. Follow on Twitter to get notified once released.

As an undergraduate, I developed various Open Source softwares in Julia language surrounding the Machine Learning & NLP ecosystem.

News

• Jan 2023: My paper from my time at Google has been accepted at EACL’23. See you in May at Dubrovnik.
• August 2022: Concluded my Research Internship at Google Search on efficient NLU and prefix generalization. Details and paper coming soon!
• March 2022: One Oral accepted at NAACL 22, co-authored with Prof. Kyle Mahowald on analysing character information in LLMs like GPT.
• Novemeber 2021: One paper accepted in EMNLP 2021 on implication of causal direction of data in NLP.
• October 2021: My Computer Vision system on X-ray based lung diseased detection won the Tredence’s Last Mile Hacker Challenge.
• September 2021: Our system on GPT-3 for text simplication won the best design award at AllenNLP 2021 Hackathon.
• August 2021: I am happy to share that our team won the Cogito’s 2021 Hackathon on low-latency Speech Emotion Recognition.
• July 2021: I will be attending the weeklong Deepmind’s Eastern European ML School, look forward to interacting with everyone.
• June 2021: Had a wonderful time attending the virtual NAACL 2021 and presenting (3 + 3 minutes) my paper.
• May 2021: My Bachelor’s Thesis titled “Towards Target Aware Twitter Stance Detection” selected among Best Thesis
• April 2021: I will be joining the UT Austin’s Computer Science Department this fall as a Master’s student.
• March 2021: 1 first-author paper accepted at NAACL-HLT 2021.
• January 2021: Our submission to SemEval21 Task-2 stood runner up.

More

Publications and Talks

• "What do tokens know about their characters and how do they know it?"
  Ayush Kaushal and Kyle Mahowald
  Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2022)
  Code Pdf Cite More
Pre-trained language models (PLMs) that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information, despite lacking explicit access to the character composition of tokens. Here, studying a range of models (e.g., GPT- J, BERT, RoBERTa, GloVe), we probe what word pieces encode about character-level information by training classifiers to predict the presence or absence of a particular alphabetical character in a token, based on its embedding (e.g., probing whether the model embedding for “cat” encodes that it contains the character “a”). We find that these models robustly encode character-level information and, in general, larger models perform better at the task. We show that these results generalize to characters from non-Latin alphabets (Arabic, Devanagari, and Cyrillic). Then, through a series of experiments and analyses, we investigate the mechanisms through which PLMs acquire English-language character information during training and argue that this knowledge is acquired through multiple phenomena, including a systematic relationship between particular characters and particular parts of speech, as well as natural variability in the tokenization of related strings.




• "Causal Direction in Data Matters: Implications of Causal and Anticausal Learning in NLP",
  Zhijing Jin, Julius von Kügelgen, Jingwei Ni, Tejas Vaidhya, Ayush Kaushal, Mrinmaya Sachan and Bernhard Schoelkopf
  Long Paper at the 2021 Conference on Emperical Methods in Natural Language Processing (EMNLP 2021)
  Code Pdf Cite Poster More
The principle of independent causal mechanisms (ICM) states that generative processes of real world data consist of independent modules which do not influence or inform each other. While this idea has led to fruitful developments in the field of causal inference, it is not widely-known in the NLP community. In this work, we argue that the causal direction of the data collection process bears nontrivial implications that can explain a number of published NLP findings, such as differences in semi-supervised learning (SSL) and domain adaptation (DA) performance across different settings. We categorize common NLP tasks according to their causal direction and empirically assay the validity of the ICM principle for text data using minimum description length. We conduct an extensive meta-analysis of over 100 published SSL and 30 DA studies, and find that the results are consistent with our expectations based on causal insights. This work presents the first attempt to analyze the ICM principle in NLP, and provides constructive suggestions for future modeling choices. .




• "Asserting the Role of Targets for Detecting Stance in Tweets"
  Ayush Kaushal, Avirup Saha and Niloy Ganguly
  Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2021)
  Code Pdf Cite Slides Poster More
The stance detection task aims at detecting the stance of a tweet or a text for a target. These targets can be named entities or free-form sentences (claims). Though the task involves reasoning of the tweet with respect to a target, we find that it is possible to achieve high accuracy on several publicly available Twitter stance detection datasets without looking at the target sentence. Specifically, a simple tweet classification model achieved human-level performance on the WT–WT dataset and more than two-third accuracy on various other datasets. We investigate the existence of biases in such datasets to find the potential spurious correlations of sentiment-stance relations and lexcal choice associated with the stance category. Furthermore, we propose a new large dataset free of such biases and demonstrate its aptness on the existing stance detection systems. Our empirical findings show much scope for research on the stance detection task and proposes several considerations for creating future stance detection datasets.




• "Leveraging Event Specific and Chunk Span features to Extract COVID Events from tweets"
  Ayush Kaushal and Tejas Vaidhya
  Oral Presentation at the 6th Workshop on Noisy User-generated Text (W-NUT) at the 2020 Conference on Emperical Methods in Natural Language Processing (EMNLP 2020) Shared Task Winners
  Code Pdf Cite Slides Poster More
Twitter has acted as an important source of information during disasters and pandemic, especially during the times of COVID-19. In this paper, we describe our system entry for WNUT 2020 Shared Task-3. The task was aimed at automating the extraction of a variety of COVID-19 related events from Twitter, such as individuals who recently contracted the virus, someone with symptoms who were denied testing and believed remedies against the infection. The system consists of separate multi-task models for slot-filling subtasks and sentence-classification subtasks while leveraging the useful sentence-level information for the corresponding event. The system uses COVID-Twitter-Bert with attention-weighted pooling of candidate slot-chunk features to capture the useful information chunks. The system ranks 1st at the leader-board with F1 of 0.6598, without using any ensembles or additional datasets.




• "Multi-Staged Language Models for Extracting Measurements, their Attributes and Relations"
  Neel Karia*, Ayush Kaushal* and Faraaz Mallick*
  15th International Workshop on Semantic Evaluation (SemEval'21) at ACL-IJCNLP'21 Second Runner-up
  Code Pdf Cite Poster More
SemEval-2021 Task 8: MeasEval aims at improving the machine understanding of measurements in scientific texts through a set of entity and semantic relation extraction sub-tasks on identifying quantity spans along with various attributes and relationships. This paper describes our system, consisting of a three-stage pipeline, that leverages pre-trained language models to extract the quantity spans in the text, followed by intelligent templates to identify units and modifiers. Finally, it identifies the quantity attributes and their relations using language models boosted with a feature re-using hierarchical architecture and multi-task learning. Our submission significantly outperforms the baseline, with the best model from the post-evaluation phase delivering more than 100% increase on F1 (Overall) from the baseline.




• "Domain specific BERT representation for Named Entity Recognition of lab protocol."
  Tejas Vaidhya and Ayush Kaushal
  Proceedings of the 6th Workshop on Noisy User-generated Text (W-NUT) at the 2020 Conference on Emperical Methods in Natural Language Processing (EMNLP 2020)
  Code Pdf Cite Poster More
Supervised models trained to predict properties from representations, have been achieving high accuracy on a variety of tasks. For instance, the BERT family seems to work exceptionally well on the downstream task from NER tagging to the range of other linguistic tasks. But the vocabulary used in the medical field contains a lot of different tokens used only in the medical industry such as the name of different diseases, devices, organisms, medicines, etc. that makes it difficult for traditional BERT model to create contextualized embedding. In this paper, we are going to illustrate the System for Named Entity Tagging based on Bio-Bert. Experimental results show that our model gives substantial improvements over the baseline and stood the fourth runner up in terms of F1 score, and first runner up in terms of Recall among 13 teams with just 2.21 F1 score behind the best one.




• "Basic Tools for Tokenizing Natural Language in Julia."
  Ayush Kaushal, Lyndon White, Mike Innes, Rohit Kumar
  The Journal of Open Source Software (JOSS) 2020
  Code Pdf Cite Poster More
WordTokenizers.jl is a tool to help users of the Julia programming language work with natural language. WordTokenizers.jl provides a flexible API for defining fast tokenizers and sentence segmentors. Using this API several standard tokenizers and sentence segmenters have been implemented, allowing researchers and practitioners to focus on the higher details of their NLP tasks. WordTokenizers.jl uses a TokenBuffer API and its various lexers for fast word tokenization. TokenBuffer turns the string into a readable stream. A desired set of TokenBuffer lexers are used to read characters from the stream and flush out into an array of tokens. The package provides the following tokenizers made using this API. WordTokenizers.jl is currently being used by packages like TextAnalysis.jl, Transformers.jl and CorpusLoaders.jl for tokenizing text.




• "Towards Target Aware Twitter Stance Detection."
  Ayush Kaushal
  Bachelor's Thesis, IIT Kharagpur
  Code Pdf Slide Report More
The stance of a text is the attitude or standpoint of the author towards an entity or a debate topic. Stance Detection is the Natural Language Understanding (NLU) task to automatically detect the stance of a text for a given target. The targets are either named entities such as policies, politicians, or as free-form sentences such as popular debate topics, rumoured online claims or article headlines. Over the past few years, a considerable attention has been given to research on Stance Detection Systems. Several large scale datasets have also been proposed on a variety of domains to promote the progress of the field and to provide reliable benchmarks. In this project, we first study the role of the target entities in detecting stance and based on our findings move towards target aware stance detection systems. We mainly analyse the various publicly available stance detection datasets on tweets. In these datasets, the text is a tweet and targets are either fixed entities or naturally occurring free-form sentences. Empirical findings show that despite the task requiring significant amount of understanding of the target being considered, one can easily achieve impressive performance by doing simple target-agnostic classification over a tweet. Specifically, we observe that a simple tweet classification model achieved near-human upper bound performance on a large scale dataset (WT-WT). These classification models can also achieve more than two-third of accuracy on 6 other Twitter stance detection datasets. Then, an analysis of some of the datasets is carried out to explain the existence of such biases in such datasets. We find some spurious correlations between tweet sentiment and stance classes. These datasets additionally have a set of lexical choices associated with their respective stance category. Consequently, we propose novel methods to augment the var ious existing stance detection dataset to inhibit target oblivious models from performing well and evaluated the dataset’s usefulness on the stance detection baseline models. With these proper benchmarks, we propose and experiment with several target aware stance detection models and show that taking into consideration, the target aware cues, the models deliver state of the art performance. Finally, we create a leaderboard for the newly created datasets and make our code, datasets and pre-trained models publicly available.




• "Natural Language Processing in Julia."
  Ayush Kaushal
  JuliaCon 2020 Conference Full Talk (30 min)
  Code Talk Video More
The JuliaText ecosystem provides various packages for working with human languages. In this talk, I showed the usage of these JuliaText packages with Flux.jl for Natural Language Processing (NLP) with a focus on deep learning-based approaches. The attendees will gain working knowledge about how to apply the package for NLP in Julia. The talk will encompass Tokenizers, Word Embeddings, Recurrent Neural Networks and Transformer based Language models.

Other Selected Projects

Deep Learning Sequence Models

Natural Language ProcessingDeep Learning

Lossless Depropagandizing news

Natural Language ProcessingAI for Social Good

Explainable Rumour Detection

Natural Language ProcessingDeep Learning

Adversarial Visual Grounding

Deep LearningMultimodal Learning

Semantics for Stance Detection

Natural Language ProcessingDeep Learning

Contact