Researchers Develop AI That Evaluates Quality and Potential Popularity of Short Stories

Researchers have been trying to get artificial intelligence (AI) to understand and generate narratives for years. A big problem in this line of research has now been seemingly solved, as researchers from Disney and the University of Massachusetts at Boston have developed neural networks that can evaluate short stories and predict how popular they could be with a large audience.

AI that can Evaluate Short Stories

Though this AI can evaluate stories, it does not try to mimic professional, experienced critics. Instead, it manages to analyze the quality of the narrative and predict its future popularity, according to Boyang “Albert” Li, a research scientist at Disney Research:

“Our neural networks had some success in predicting the popularity of stories. You can’t yet use them to pick out winners for your local writing competition, but they can be used to guide future research.”

One of the challenges researchers had to face was the lack of large databases of stories that have been evaluated by humans, which could have been used to train the neural networks. To solve the problem, researchers turned to question-and-answer website Quora, as users on the platform regularly answer questions by describing their own experiences in the form of narratives.

Researchers picked as many as 55,000 answers from Quora, and classified about 28,000 as relevant narratives, averaging 369 words per answer. Researchers then used upvotes – given by users to helpful answers – to measure a story’s “proxy for narrative quality” as well as potential popularity.

To understand the complex semantics of the various analyzed stories and their characters, two different neural networks were developed. One looked at separate parts of each story, including the question that inspired it, while the other one evaluated “how meaning of the events and story regions emerged from the entire story” in looking at the story as a whole. Both AIs made predictions of which texts would be most popular with readers.

The two neural networks outperformed the baseline text evaluation system, but the holistic AI showed an 18 percent improvement over the one that focused on separate parts.

Possible Future Uses

This is only the first step in getting AIs to create narratives themselves, as they will need to understand what makes a good text before trying to create one themselves. In the future, Disney and other leading movie studios could easily use these AIs to choose one script over another based upon how well the AI says it will perform with an audience.

Markus Gross, vice president at Disney Research, stated:

“The ability to predict narrative quality impacts on both story creation and story understanding. To evaluate quality, the AI needs some level of understanding of the text. And if AIs are to create narratives, they need to be able to judge the quality of what they are producing”.

When matured, these neural networks could even be used to generate narratives for production, meaning that one day we could watch movies written by AI that knows exactly what will get us to sit on the edge of our seats.