Pratyay Banerjee

Pratyay Banerjee

Natural Language Understanding | Multi-modal Reasoning | Question Answering

About Me

I am an Applied Scientist at Alexa AI, Sunnyvale, USA working with an awesome team of researchers focussed on generating and answering questions from the Web. My research interests are Natural Language Understanding, Question Answering and Multi-modal Reasoning. I am also working towards improving knowledge understanding from structured and unstructured knowledge bases, building neural models for retrieval, text and/or image-based question answering. Previously, I was a researcher at Arizona State University, Tempe, USA working with Dr. Chitta Baral in the Cognition and Intelligence Lab.

Updates

  • I have joined Alexa AI Web Information team and working on question answering and generation from the Web.

  • 2 of my work were accepted in EMNLP 2022, “Lexi: Self-Supervised Learning of the UI Language” and “Learning Action-Effect Dynamics for Hypothetical Vision-Language Reasoning Task”.

  • I successfully defended my Ph.D. dissertation on “Implicitly Supervised Neural Question Answering.” I will be joining Amazon Science in June 2022.

  • Three of our work was accepted in ACL and Findings of ACL. “Semantically Distributed Robust Optimization for Vision-and-Language Inference”, “To Find Waldo You Need Contextual Cues”, “Unsupervised Natural Language Inference Using PHL Triplet Generation”.

  • Our research was awarded the NSF grant Award: “Abstract 2132724 RI: Small: SM-An Active Approach for Data Engineering to Improve Vision-Language Tasks.”

  • “Weakly Supervised Visual-Retriever-Reader for Knowledge-based Question Answering” was accepted in The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021).

  • “Weakly Supervised Relative Spatial Reasoning for Visual Question Answering” was accepted in International Conference of Computer Vision (ICCV 2021). We weakly-supervise transformer-based VQA systems using two novel, unit normalized 3D-vision guided tasks, Centroid Estimation and Relative Position Estimation.

  • “Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction” was accepted in 59th Annual Meeting of the Association for Computational Linguistics (ACL 2021).

  • “Constructing Flow Graphs from Procedural Cybersecurity Texts” was accepted in Findings of ACL 2021.

  • “WeaQA: Weak Supervision via Captions for Visual Question Answering” was accepted in Findings of ACL 2021.

  • The journal version of KGNER, “Bio-Medical Named Entity Recognition via Knowledge Guidance and Question Answering” has been accepted in “ACM Transactions on Computing for Healthcare”.

  • “Self-Supervised Test-Time Learning for Reading Comprehension” was accepted in 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2021).

  • My work “Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering” was accepted in The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).

  • Our paper “MUTANT: A Training Paradigm for Out-of-Distribution Generalization in VQA” was accepted in The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).

  • Our paper “Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning” was accepted in The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).

  • Our paper “VQA-LOL: Visual Question Answering under the Lens of Logic” was accepted in 16th European Conference on Computer Vision (ECCV 2020).

  • My work “Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering” describes a framework for zero-shot question answering, and approaches early supervised models.

  • My work “Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question Answering” achieves state-of-the-art on OpenBookQA and QASC datasets.

  • Our dataset and task paper “Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning” is released. We look at a new task to generate commonsense enriched video captions, and approach the problems in a QA and Video-caption generation task.

  • Our survey paper “Natural Language QA Approaches using Reasoning with External Knowledge” is released in arxiv. We look at both symbolic and neural approaches to incorporate external knowledge to perform NLQA.

  • Our paper “KGNER: Knowledge Guided Named Entity Recognition” achieves state of the art F1 scores on 15 Bio-Medical NER datasets.

  • Our paper “VQA-LOL: Visual Question Answering under the Lens of Logic” is released. This work makes VQA models more robust to logical questions.

  • Our paper “Exploring ways to incorporate additional knowledge to improve Natural Language Commonsense Question Answering” shows ways to incorporate knowledge in neural language models.

  • My work on generating explanations given a question and correct answer, “Explanation ReGeneration using Language Models and Iterative Re-Ranking” ranked 2nd in the Shared Task in the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13).

  • Our paper “Careful Selection of Knowledge to solve Open Book Question Answering” was accepted in 57th Annual Meeting of the Association for Computational Linguistics.

Publications

  • “Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction” : 59th Annual Meeting of the Association for Computational Linguistics (ACL 2021).

  • “Constructing Flow Graphs from Procedural Cybersecurity Texts” : Findings of ACL 2021.

  • “WeaQA: Weak Supervision via Captions for Visual Question Answering” : Findings of ACL 2021.

  • “Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering” : The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).

  • “MUTANT: A Training Paradigm for Out-of-Distribution Generalization in VQA” : The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).

  • “Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning” : The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).

  • “VQA-LOL: Visual Question Answering under the Lens of Logic” : 16th European Conference on Computer Vision (ECCV 2020).

  • “Careful Selection of Knowledge to solve Open Book Question Answering” : 57th Annual Meeting of the Association for Computational Linguistics.

  • “Explanation ReGeneration using Language Models and Iterative Re-Ranking” : Graph-Based Methods for Natural Language Processing (TextGraphs-13).

Pre-prints

  • “Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question Answering” - arxiv
  • “Natural Language QA Approaches using Reasoning with External Knowledge” - arxiv
  • “Knowledge Guided Named Entity Recognition” - arxiv
  • “Exploring ways to incorporate additional knowledge to improve Natural Language Commonsense Question Answering” - arxiv

Education

  • Bachelor of Computer Science and Engineering, Jadavpur University, Kolkata, India, 2014
    • Advisor: Dr. Goutam Paul, ISI, Kolkata, India
  • Ph.D in Computer Science, Arizona State University, AZ, USA, May 2022
    • Advisor: Dr. Chitta Baral, ASU, USA

Projects

Code for most recent projects are available in my github.

  • BioNLP, ASU, Fall 2019:
    • Our work with Dr. Devarakonda on Knowledge Guided NER achieves state of the art F1 scores on 15 Bio-Medical NER datasets.
  • Knowledge Representation, ASU, Fall 2019:
    • We solved ASP Challenge 2019 Optimization problems using Clingo.
  • Mobile Computing, ASU, Spring 2019 :
    • Activity Classification using Myo Gesture Control Armband data through Machine Learning. We evaluated multiple feature generation techniques and several ML models.
  • Statistical Machine Learning, ASU, Spring 2019 :
    • Improving Information Retrieval for Knowledge Extraction and Open Book Question Answering. We tested multiple IR retrieval methods using different SML models and multiple kinds of features, including features from BERT.
  • Software Security, ASU, Fall 2018 :
    • We were team AISec , which stood 2nd overall and 1st among teams who had no prior experience to CTF Attack/Defence competitions.
  • Natural Language Processing, ASU, Fall 2018:
    • Improving BiDAF for SQUAD 2.0, which had questions with no answers.
  • Undergrad Thesis at ISI, Kolkata : Fall 2013 - Spring 2014 :
    • A Graph-Based FHE based on Homomorphic Bit Vector Encoding. Evaluated multiple schemes to exploit the hardness of graph path traversal algorithms and Homomorphic Bit Vector Encoding to derive a graph based FHE scheme.
  • Summer Internship at CNERG, IIT Kharagpur : Summer 2013 :
    • Detection and analysis of dynamic communities in time varying networks. We contrasted several community detection algorithms over time varying networks, built a test bed to analyze these algorithms. Derived useful insights of few community datasets, using these implementations.
  • Software Engineering Project, Jadavpur University : Summer 2012 :
    • To find the best algorithm to identify and demarcate ringworms; an image preprocessing, segmentation, comparative algorithm analysis and implementation project. We developed a software in MS Visual Studio suite, which processes any image and identifies possible ringworms