Download this Paper Open PDF in Browser

Automated Production of High-Volume, Real-Time Political Event Data

26 Pages Posted: 19 Jul 2010 Last revised: 31 Aug 2010

Philip A. Schrodt

Pennsylvania State University

Date Written: 2010


This paper summarizes the current state-of-the-art for generating high-volume, near-real-time event data using automated coding methods, based on recent efforts for the DARPA Integrated Crisis Early Warning System (ICEWS) and NSF-funded research. The ICEWS work expanded by more than two orders of magnitude previous automated coding efforts, coding of about 26-million sentences generated from 8-million stories condensed from around 30 gigabytes of text. The actual coding took six minutes. The paper is largely a general "how-to" guide to the pragmatic challenges and solutions to various elements of the process of generating event data using automated techniques. It also discusses a number of ways that this could be augmented with existing open-source natural language processing software to generate a third-generation event data coding system.

Keywords: event data, ICEWS, prediction, natural language processing, DARPA, open source

Suggested Citation

Schrodt, Philip A., Automated Production of High-Volume, Real-Time Political Event Data (2010). APSA 2010 Annual Meeting Paper. Available at SSRN:

Philip A. Schrodt (Contact Author)

Pennsylvania State University ( email )

University Park
State College, PA 16802
United States

Paper statistics

Abstract Views