View article

Most studies of infant language acquisition have focused on the role of purely linguistic information as the central constraint. However, several researchers (eg Baldwin) have suggested that non-linguistic information, such as vision and talkers' attention, also plays a major role in language acquisition. In light of this, we implemented an embodied language learning system that explores the computational role of attention to build visually grounded lexicons. The central idea is to make use of visual perception as contextual information to facilitate word spotting, and utilize eye and head movements as deictic references to discover temporal correlations of data from multiple modalities. In the experiments, subjects were asked to perform three kinds of everyday activities (pouring water, stapling papers, and taking a lid off) while providing natural language descriptions of their behaviors. We collected speech data in concert …