A Systematic Evaluation of the Creative Writing Skills of Transformer Deep Neural Networks

26 Pages Posted: 8 Apr 2022

Date Written: February 24, 2022


We present the results of a study that tests the creative writing abilities of Transformers (current state-of-the-art Deep Neural Networks for Natural Language Processing) with respect to humans. In our experiment, transformers are given a title, and their task is to invent a synopsis for a movie that matches the title. We collected 24,480 manual assessments on synopsis written by transformers and humans that, altogether, reveal that the synopsis generated by transformers are, in average, significantly better than their human counterparts in terms of readability, understandability, relevance with respect to the title, informativity and attractiveness. The only aspect where transformers match, but do not improve, human performance is creativity. Our results also indicate that, if assessors are informed of who is the author of the text (human or machine), machine-made synopsis receive lower scores, confirming that the interpretation of a creative text depends on the expectations of the reader with respect to the author. This is, to our knowledge, the first experiment on creative writing where transformers show true superhuman performance. Our result confirms the potential of transformers to assist creative writers, but also calls for a reflection on the methodological limitations and challenges of evaluating creative tasks.

Keywords: natural language processing, deep neural networks, transformers, creative text writing

Suggested Citation

Marco, Guillermo and Gonzalo, Julio and Rello, Luz, A Systematic Evaluation of the Creative Writing Skills of Transformer Deep Neural Networks (February 24, 2022). Available at SSRN: https://ssrn.com/abstract=4042578 or http://dx.doi.org/10.2139/ssrn.4042578

Guillermo Marco

UNED ( email )


Julio Gonzalo (Contact Author)

UNED ( email )

ETSI Informática UNED
c/ Juan del Rosal 16
Madrid, Madrid 28040

HOME PAGE: http://nlp.uned.es

Luz Rello

IE Business School - IE University ( email )

Calle Maria de Molina 12
Madrid, Madrid 28006

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
PlumX Metrics