Neural Net Poetry Meets Distant Reading · Princeton

![workshop logo cropped](images/workshop-logo-cropped.png)

### Neural Network Poetry Meets Distant Reading
Analysing Computer-Generated Echoes of Russian Literary History

[Boris Orekhov](http://nevmenandr.net/bo.php) ¹  ·  [Frank Fischer](https://www.hse.ru/en/org/persons/182492735) ¹ ²

¹ Higher School of Economics, Moscow 
² DARIAH-EU

[Slavic DH Workshop](https://cdh.princeton.edu/events/2019/05/slavic-dh-workshop-russian-literary-studies-digital-age/)  ·  Princeton University  🇺🇸  ·  28 May 2019, 3:30–5:00 p.m.

## Motto

*[The machine] is accused of destroying aesthetics. In fact, it destroys only aesthetic mysticism.*

— Leon Trotsky: Poetry, the Machine, and the Poetry of the Machine (1901)

### **Chapters**

1. Artificial Neural Networks and Poetry
2. Computer Poetry Before the Era of Neural Networks
3. Generating Russian Poetry with Neural Networks
5. Summary

---

# Chapter 1.

### Artificial Neural Networks and Poetry

### Why?

- in recent years, artificial neural networks have evolved from a concept known only to specialists to a popular news item
- the task of generating text is relevant in the context of chat bots:
 - Apple iOS (Siri, 2011)
 - Microsoft (Cortana, 2014)
 - Amazon (Alexa, 2014)
 - Google (Assistant, 2016)
 - Яндекс (Алиса, 2017)

### Programmers’ Interest

- computer-generated texts attracted the attention of programmers interested in the development of future technologies
- many of them decided to try out TensorFlow, Google’s publicly available free library for deep learning (November 2015), which simplified the task of creating neural networks and attracted many specialists

### Programmers’ Interest

- some of the programmers who wanted to try it out, used poems as testbed for the new technology
- on GitHub, we find dozens of examples of exercises in creating poetic texts
- poems, especially popular and canonised ones, are freely available online, occupying a key position in the cultural hierarchy

### NN Texts and Poetry

- poetic texts are often arranged as a chain of fairly self-contained statements and are not required to tell a holistic story
- look at the textbook lines by Joseph Brodsky:

A list of observations. The corner is warmer. 
A gaze will leave a mark on whatever it’s settled on. 
Glass is another manifestation of water. 
Man is more frightening than his skeleton.

- a poetic aphorism is expressed in every line
- the structure of the text partly resembles a dialogue, also consisting of individual (often short) replicas of communicants

### NN Texts and Poetry

- neural networks do not always succeed in generating grammatical and easily interpretable structures when it comes to replicas
- 20th-century poetry is also replete with experimental texts that reduce grammatical rules and conventions of semantic connectivity
- a non-professional view of poetry in general also suggests that poetry does not have to be „meaningful“, but **poetry „must be slightly stupid“**, as Pushkin famously wrote in a letter

### Postulates of H. P. Grice

- both the poetic text and text generated by neural networks are similar in that most of the postulates of H. P. Grice are violated:
 - statements belonging to these categories provide less information than necessary
 - are not necessarily true
 - incomprehensible and ambiguous

- the poetry of artificial neural networks turned out to be in an ensemble of other technologically relevant topics
- at the technical level, this interaction has already been reflected in scientific publications describing the language and mathematical details of the implementation of working models

---

# Chapter 2.

### Computer Poetry Before the Era of Neural Networks

### Prehistory

- the history of computer poetry began long before the current boom of neural networks
- the first computers appeared in the early 1940s (German engineer Konrad Zuse and American Howard Aiken, „Z3“ and „Mark I“)
- in the late 1950s poets began making use of them

### First Opus

- in 1959, German mathematician Theo Lutz programmed a computer to randomly combine phrases from the 16 chapters of Kafka’s unfinished novel „The Castle“, he called the resulting texts „stochastic“:

[NICHT](https://netzliteratur.net/lutz_schule.htm) JEDER BLICK IST NAH. KEIN DORF IST SPAET. 
EIN SCHLOSS IST FREI UND JEDER BAUER IST FERN. 
JEDER FREMDE IST FERN. EIN TAG IST SPAET. 
JEDES HAUS IST DUNKEL. EIN AUGE IST TIEF. 
NICHT JEDES SCHLOSS IST ALT. JEDER TAG IST ALT 
NICHT JEDER GAST IST WUETEND. EINE KIRCHE IST SCHMAL 
KEIN HAUS IST OFFEN UND NICHT JEDE KIRCHE IST STILL. 
NICHT JEDES AUGE IST WUETEND. KEIN BLICK IST NEU.

### 1960s

- during the 1960s, several more such poetic experiments were performed
- Brion Gysin in 1960 makes the computer randomly swap words in the phrase „I AM THAT I AM“
- in 1961, Nanni Balestrini creates the poem „Tape Mark I“ on the IBM 7070 computer, composed of parts of such works as the „Hiroshima Diary“ by Michihiko Hachiya and the „Tao Te Ching“:

l’accecante / globo / di fuoco 
si espande / rapidamente 
trenta volte / più luminoso / del sole 
quando raggiunge / la stratosfera 
la sommità / della nuvola 
assume / la ben nota forma / di fungo

- in the same year, Rul Gunzenhäuser’s „Weihnachtsgedicht“ (automatic poems) appeared:

Der Schnee ist kalt 
und jeder Friede ist tief 
und kein Christbaum ist leise 
oder jede Kerze ist weiss 
oder ein Friede ist kalt 
oder nicht jede Kerze ist rein 
und ein Engel ist rein 
und jeder Friede ist still 
oder jeder Friede ist weiss 
oder das Kind ist still 
ein Engel ist überall

### America and Europe

- if at the beginning of the 21st century North America is the centre of IT developments and the de-facto language of the industry is English, the primacy in cybernetic experiments with poetry half a century ago belonged to European artists

### Two Basic Principles

- the first one suggested the recombination of words or phrases of an already existing text (at the dawn of computer poetry, such text was more often a work in prose)
- the second is a random choice of words, the language and/or verse restrictions set by the programmer are imposed on the arrangement for the resulting text

### Randomness

- thus, the central category of this process becomes the category of **randomness**
- one of the artists of that era, Marc Adrian, says this directly: „For me, the neutrality of the machine is very important. It makes it much easier for the reader to find their own meanings associated with words, since their choice, size, and composition are randomised.“
- surprisingly, it is an accident, raised to an artistic absolute, although poets associate it with a computer, for a computer it is just a big technological problem

### Meanwhile in Russia

- known experiments with generating of poetic texts in Russian took place much later and date back to the time when personal computers started to spread
- L. Kaganov, in his 1996 diploma, cites as an example of the previous developments the POET.EXE program (in other sources, „Stikhoplyuy“, the author is M. I. Grinchuk), which randomly composes a text in verse from the dictionary of accentologically marked words
- the year of creation of the program, thus, is no later than 1996, perhaps, 1991

- other sources also mention the „Poe“ program, created in 1994 by S. Derzhane
- it composed the text according to grammatical patterns set by the user, obviously using the built-in dictionary and information about the place of stress
- L. A. Kaganov’s own program uses the original (prose) text to create „associations“, that is, rules governing the possibility of following words in a text

### Cyber Pushkin

- by 2002, the program „Cyber Pushkin“ appears
- its author S. Teterin admits that „he taught him [Cyber-Pushkin] everything he needed: a poetic rhythm, rules of rhyming, the specifics of the poetic lexicon“, which, apparently, it used to build an algorithm on a random selection of lexical items from a given list

### Russian Poetic Conservatism

- it is well seen that all these early experiences in Russian material imply such a comparatively depleted concept of poetic, which requires an obligatory syllabic and strict rhyme
- it is curious that the first poetic works created by computers in the late 1950s and early 1960s were the result of a freer understanding of poetry, not limited by the set of formal parameters reproduced by the culture of 19th century writing

### Permutational Art

- both these earlier and later experiments, which preceded the wide use of neural networks, are based on the idea of combinatorics of predetermined elements (words or ready-made statements)
- linking this idea to the philosophy of structuralism, Abraham Mole conceptualised it as „permutational art“
- one that „decomposes the world into ‘atoms’, of which then arbitrary structures are constituted“
- „permutational art is not only a game – it is a method of aesthetic research, since it opens a concrete path to the analysis and synthesis of a work of art“

---

# Chapter 3.

### Generating Russian Poetry with Neural Networks

![heatmap](images/nejrolirika.jpg)

https://iq.hse.ru/news/228812683.html

### What Changed?

- artificial neural networks radically changed the subject field of computer poetry
- accident in the process of generating text has faded into the background
- there is no need for an algorithmic description of the complex rules of language and poetics, according to which the automatic work should be built
- the main factors are not combinatorics and randomness, but, on the contrary, consistency, not arbitrariness of the structure, but structural dependencies of the elements

### Concentrated Form

- if Joseph Brodsky considered poetry to be a concentrated form of literature, the production of a neural network can be a concentrated form of representing poetry of a genre or period
- an example generated by a two-layer neural network with 512 neurons, trained on a corpus of Russian translations of ancient hexameters (Homer, Hesiod, Ovid, Virgil, etc.) of 5.351.336 characters (106.321 poetic lines):

Силу, к голубке хитон отличась, Гиоклей благородный. 
На Ликеи веселие слово кружает другого, 
Слишком попал бы и все повреждает она одиноко 
И возливаешь они рассудить — городские, проделать 
Кровью вкусили два дочь. На корабль он твухте твоей силы!

Sílu, k golúbke khitón otlichás’, Giokléy blagoródnyy. 
Ná Likéi veséliye slóvo kruzháyet drugógo, 
Slíshkom popál by i vsé povrezhdáyet oná odinóko 
Í vozliváyesh’ oní rassudít’ — gorodskíye, prodélat’ 
Króv’yu vkusíli dva dóch’. Na korábl’ on tvukhté tvoyey síly!

Power, to the dove the chiton is different, Giocleus the noble. 
To the Lyceum, the amusement, a word spins the other, 
Too much you would hit and she damages everything lonely. 
And you drink they to decide — urbanic, to do. 
With blood tasted, two daughter. To the ship he tbay your strength!

(Our „translation“. 😉)

### Stylistic Features

- correct reproduction of the meter
- the epithet „blagorodnyy“ („noble“) in postposition of a proper name as is common in translation of Greek hexameters
- generated non-existent words („kruzhat’“, „tvukhta“), which from the point of view of the model are likely to exist
- the word „Giokley“ („Giocleus“), which is absent in the onomasticon of the Russian hexameter (and generally in Russian texts), but actually resembles the names of ancient characters: Diokley (Diocles), Gippodam (Hippodamus), Egiokh (Aegiochos), Peley (Peleus)
- the grammatical inconsistencies („dva doch’“, „oni rassudit’“) are due to the size of the corpus

## Modern Example: Original Poem by N. M. Azarova

бор ёлки в воду бросил 
сил держать верхушки нет 
кинет дрожащих в озеро 
зеро ребячьей ряби голосов свободно 
дно тростником упало навзничь 
ничь — я — не — потревожена — от — бором 
Москва 
21 ноября 2003

### The neural network spawned the following text:

могу не озеро горизонта 
синих глубок луг на бахли на стаю 
болтлизись? 
слежу ночи кораллов гадаю облаками 
сэнимальной луна? 
Ялта 
19 сентября

- the generated texts look more authentic, because they do not tell about the event, but represent a number of lyrical motifs
- it is worth noting that correctly reproduced words (“lake”, “corals”, “moon”) are really peculiar to the poetics of N. M. Azarova
- the neural network seems to have authenticly modelled the distribution of letters, creating sound effects characteristic of N. M. Azarova’s verses

### Vysotsky

- a less successful experiment was the the network trained on a corpus of singer-songwriter and poet Vladimir Vysotsky
- since Vysotsky’s poetry usually does without „neo-futuristic“ linguistic irregularities and builds on narrative fragments, it is much more difficult to learn the original style here:

Песня Вопилась 
Да коровами про провода 
Состоит — нет, шерной и ненасытную свой слева тех.

---

# Summary

### Style vs. Narrative

- texts generated with the help of artificial neural networks reproduce part of the stylistic features of the training sample
- mainly those that are visible in small segments:
 - instrumentation
 - vocabulary
 - syntax
 - meter
- plot and larger narrative elements in general remain outside the reproducible as yet

Thanks.

– https://hum.hse.ru/digital/ –