Cognitive Process Simulation

A framework for simulating human comprehension process

- Proposed a framework to combine hierarchical Long short-term memory (LSTM) with knowledge graphs to encode and understand visual storytelling data.

Abstract

This work describes a framework that contains a visual narrative comprehension model with different levels of comic features to represent the comic context, a pipeline for processing, and an encoding of comic data with function portals to access the studied and learned knowledge. The comprehension model uses a hierarchical Long Short-Term Memory (LSTM) model adapted for encoding comic sequences to fit the multi-modality of comics. It transforms comic context and perceptual features into matrices to form a comic descriptor.