Design and implementation of an intelligent annotation tool -- 2nd Project




  • Angewandte Informatik Bachelor
  • Angewandte Informatik Master
  • Komedia Bachelor
  • Komedia Master
  • Advanced programming skills (desired)


People use online search engines for many reasons, but frequently an online search is performed to assist in decision-making. On the individual level, for example the decisions that are supported by online content range from deciding on whether to buy a product or a service, over potentially life changing decisions, e.g. on health treatments, to decisions that bear importance on whole societies, like what socio-political agenda to support. Furthermore, decision-making searches are performed by representatives of organizations for professional reasons, e.g. by a journalist to write a newspaper article, by marketing experts to try and estimate the public opinion, by analysts and researchers in public and private sector to find relevant evidence for their particular institutional decisions, etc.

For decision-making people need arguments. An argument consists of several pieces of evidence called premises and a claim, also referred to as the conclusion of the argument. Premises can support or reject the claim. For example, consider the following claim or conclusion "Brexit will have a positive impact on the UK's economy". Some examples of its premises are: "goods exports from the UK to other countries rose 3.4% between June and July 2016", "exports to EU rose 9.1%, according to official figures", "unemployment has held at an 11-year low of 4.9%". Based on this argument a member of a bank may perform a critical investment, an owner of a company may decide on her investments, a journalist may write a report, etc. In other words to make an informed decision an individual needs to consider the pro and counter evidence for a claim of interest. Therefore, if a search is performed to support decision-making, it is arguments that people search for online.

Extracting arguments from texts is termed as Argument Mining and is a rather young but growing research field within Computational Linguistic and Computer Science. Developing argument mining systems require gold standard data — human annotated data that contains examples about claims and premises — to learn what arguments are and then can be used to perform argument extraction from unseen documents. However, obtaining such gold standard data is always challenging and is prone to high variations between human annotators — i.e. given a document annotators annotate different passages as argument and extract only very few examples where they agree. There are several reasons for this such as: lack of motivation, misunderstanding the task, lack of knowledge within the domain, confidence (low/high), no awareness about her progress, no comparison to peers, lack in presentation of task, use of unclear questions, etc.

In this project we aim to tackle the gold standard data generation challenge by developing an intelligent argument annotation tool for achieving high agreement among annotators. In summary the project should achieve the following aim:

Developing an online system that presents the tasks and allows annotators to perform the annotation task.
The system should help

  1. Preparing the annotator to the task and keeping her motivated till the end of the task.
  2. Keeping the annotator up to dake about her progress and how she performs compared to her peers.
  3. Tracking her activities in the entire annotation time.
  4. Generating summary reports and sending them to task owners — this can help to open communication to annotators when needed and also update task/question descriptions.
  5. Allowing multi-user access to the system.