Query Parsing Task Proposal for GeoCLEF 2007

Microsoft Research Asia

·        Introduction

A geographic query is usually composed of three components, “what”, “geo-relation” and “where”. How to parse queries and extract these components from them is a key problem for geographic information retrieval (GIR). Therefore, we propose to add a geographic query parsing task for GeoCLEF 2007.

·        Eligibility

[To be revised]

The contest is open to any party planning to attend CLEF 2007. Multiple submissions per group are allowed, since we will not provide feedback at the time of submission. Only the LAST submission before the deadline will be evaluated.

·        Query Structure

In the data set, a common query structure will be “what” + “geo-relation” + “where”. The keywords in the “what” component indicate what users want to search; “where” indicates the geographic area users are interested in; “geo-relation” stands for the relationship between “what” and “where”. There also exist non-geographic queries in the data set which need to be recognized.

For example, for a query “Restaurant in Beijing, China”, “what” = “Restaurant”, “where” = “Beijing, China”, and “geo-relation” = “IN”. For another query “Mountains in the south of United States”, “what” = “Mountains”, “where” = “United States”, and “geo-relation” = “SOUTH-OF”.

·        Task Description

1.      Detect whether the query is a geographic query or not. A geographic query is defined as a query which contains at least a “where” component. For example, “pizza in Seattle, WA” is a geographic query, while “Microsoft software” is not a geographic query. For non-geographic queries, further parsing is not needed.

2.      Extract the “where” component from the geographic query and output the corresponding latitude/longitude. For example, in the query “pizza in Seattle, WA”, “Seattle, WA” will be extracted and lat/long value (47.59, -122.33) will be output. Sometimes terms in the “where” component are ambiguous. In this case, the participant should output the lat/long value with the highest confidence. A few queries contain multiple locations, for example, “bus lines from US to Canada”. We will not include these queries in our test set..

3.      Extract the “geo-relation” component from the geographic query and normalize it into a pre-defined relation type. A suggested relation type list is shown in Table 1. If the relation type you find is not defined in Table 1, you should categorize it into “UNDEFINED”.

Table 1. Geo-relation Types

Example query




in Beijing


on the Long Island


of Beijing


near Beijing

next to Beijing


in or around Beijing

in and around Beijing


along the Rhine


at Beijing University


from Beijing


to Beijing


within d miles of Beijing


north of Beijing

in the north of Beijing


south of Beijing

in the south of Beijing


east of Beijing

in the east of Beijing


west of Beijing

in the west of Beijing


northeast of Beijing

in the northeast of Beijing


northwest of Beijing

in the northwest of Beijing


southeast of Beijing

in the southeast of Beijing


southwest of Beijing

in the southwest of Beijing


north to Beijing


south to Beijing


east to Beijing


west to Beijing


northeast to Beijing


northwest to Beijing


southeast to Beijing


southwest to Beijing


4.      Extract the “what” component from the geographic query and categorize it into one of three predefined types, which are listed below.

·        Datasets

We will provide a test set of 800,000 (to be decided) queries. All queries come from real search engine logs. Most of them will be geographical queries. A sample labeled set of 100 (to be decided) queries will also be provided as a training set.

·        Data Format

1.      Test Set

The test set will be provided in XML format. Each query has two attributes:  <QUERYNO> and <QUERY>.


<QUERY>Restaurant in Beijing, China</QUERY>


<QUERY>Real estate in Florida</QUERY>


<QUERY>Mountains in the south of United States</QUERY>

2.      Training Set and Results

The sample labeled set and the results should be in the following format. There are 4 more attributes: <LOCAL>, <WHAT>, <WHAT_TYPE>, <GEO-RELATION> and <WHERE>.


<QUERY>Restaurant in Beijing, China</QUERY>



<WHAT-TYPE> Yellow page</WHAT-TYPE>


<WHERE>Beijing, China</WHERE>

<LAT-LONG>40.24, 116.42</LAT-LONG>


<QUERY> Lottery in Florida</QUERY>






<LAT-LONG>28.38, -81.75</LAT-LONG>

If a submission does not contain all search queries, those queries not included will be treated as errors.

·        Evaluation Criterion

We will evaluate the submitted result based on several criterions, including precision, recall, and F1-score.

We will use multiple human editors to tag a subset of queries selected from the total test set. The collection of human editors is assumed to have the most complete knowledge about internet as compared with any individual end user. You will not know which queries will be used for evaluation and are asked to categorize all queries given.

The evaluation will run on the selected test queries and rank your results by how closely they match to the results from human editors. Here are the set of measures we will use to evaluate results submitted by the contestants: