Annotation as a Pivot in DH Experiments

Annotation Experiments

Annotating texts with categories predicted from a certain theory is far from trivial. Examples given in theoretic literature are usually clear cut and unambiguous, but “real” text is not. There is a gap between a theory and annotated texts. Bridging this gap can be done in an experiment-driven fashion.

  1. Start with an existing theory (or create a new one)
  2. Define categories, develop tests, give examples for annotators
  3. Let annotaters do their job (in parallel)
  4. Evaluate the annotations, change definitions and categories accordingly, go back to 2.
  5. After a number of iterations: Reflect on the relation between theory and annotation rules, incorporate insights from annotation into theory

Automatisation Experiments

Independent of the concrete implementation strategy (rule-based, supervised or unsupervised machine learning, …), at some point a certain amount of annotated test items is required. Test items are manual annotations of text and thus text instances for which we already know what the program we write should predict. If we make changes to our program (adding new rules, adding features, changing the algorithm), we can test against the test set and check whether the changes decrease or increase the performance.

  1. Annotated dataset (resulting from, e.g., an annotation experiment)
  2. Implement a program (that incorporates hypotheses on how to detect the category automatically)
  3. Evaluate the program against the data set, make an error analysis
  4. Make changes to the program based on the evaluation

Conclusions

After some annotation and in the following automatisation experiments have taken place, the prediction quality of the programs has (hopefully) reached a certain level on the annotated data. If, for instance, the program reaches a prediction quality of 90% it would not be perfect but could still be helpful towards big data questions as mentioned in the beginning.

--

--

mad ſcientiſt

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store