A Systematic Evaluation of the Creative Writing Skills of Transformer Deep Neural Networks

Abstract

We present the results of a study that tests the creative writing abilities of Transformers (current state-of-the-art Deep Neural Networks for Natural Language Processing) with respect to humans. In our experiment, transformers are given a title, and their task is to invent a synopsis for a movie that matches the title. We collected 24,480 manual assessments on synopsis written by transformers and humans that, altogether, reveal that the synopsis generated by transformers are, in average, significantly better than their human counterparts in terms of readability, understandability, relevance with respect to the title, informativity and attractiveness. The only aspect where transformers match, but do not improve, human performance is creativity. Our results also indicate that, if assessors are informed of who is the author of the text (human or machine), machine-made synopsis receive lower scores, confirming that the interpretation of a creative text depends on the expectations of the reader with respect to the author. This is, to our knowledge, the first experiment on creative writing where transformers show true superhuman performance. Our result confirms the potential of transformers to assist creative writers, but also calls for a reflection on the methodological limitations and challenges of evaluating creative tasks.

Keywords: natural language processing, deep neural networks, transformers, creative text writing