View article

Humans possess the remarkable ability to give and follow natural language route instructions through largescale spaces. In this process, a director describes the actions and observations along the route, recalling the environment’s topology, metrical layout, and visual features. A follower interprets these descriptions, navigating by applying the instructions to the possibly unfamiliar environment. Furthermore, followers must account for mistakes, ambiguities, and omissions in the route description. To study how instructions are written and followed, we collected 756 free-form route instructions from six participants for 126 routes in three virtual environments. A second group of participants and a computational model (Marco) followed these instructions. Humans successfully reached the destination on 68% of the instructions and Marco followed 61% of the instructions. Marco’s performance was a strong predictor of human performance and ratings of individual instructions.