View article

Supporting natural language interface is crucial for accessing knowledge bases (KBs) due to the complexity of their schema. Many natural language interfaces to KBs (NLIKBs) have been built to simplify query formulation and facilitate advanced applications like personal intelligent assistant. Unfortunately, there is a lack of a comprehensive natural language question benchmark that is able to fairly evaluate different NLIKBs in terms of accuracy and runtime. In this paper, we propose a framework to construct comprehensive question benchmarks, where an array of question characteristics, including query size, commonness, modifier, answer cardinality, and paraphrasing, are made configurable. The benchmark constructed in this way delivers unprecedentedly fine-grained performance profiling, broken down by each question characteristic, which is important for performance comparison and future improvement.