Executive Summary : | The majority of document processing in many countries and companies is done online, with images often containing critical information. This proposal focuses on the analysis of images in financially or security-critical situations, such as surveillance, risk analysis, and fraud analysis. Risk analysis requires reasoning with relations among objects and commonsense facts and rules. Humans possess commonsense knowledge and can fuzzily factor this knowledge along with sensory information from images and text. Automation of this process is difficult, so current systems bypass the process by learning functions that mimic the reasoning process behaviorally. Neuro-symbolic methods have been revisited due to their potential to exhibit more logical behavior than neural counterparts. Two generic challenges arise for neuro-symbolic systems: 1) how to acquire and represent the rules, and 2) how to learn from data and rules together. For certain expressiveness of rules, these problems have been addressed to some extent. Awasthi et al. 2020 (ICLR) bypassed rule representation altogether and model their effect behaviorally. Zhu et al. 2021 (NAACL) propose an iterative E-M method to efficiently update and learn rules while updating the representation learner.
However, when input is from images, there are several fundamental research questions: 1) Discretization in images is hard, as we often do not know all objects, relations, and rules about all objects. 2) Rules require objects and predicates, and solving these fundamental questions will lead to more advanced robust image analytics rich with human intuition and knowledge. |