Using RNN to predict shakesphearean text
In this tutorial, we will look at building a Recurrent Neural Network model to predict the Shakespearean text using Keras TensorFlow API in Python. We will train the GitHub Shakespearean text dataset using a custom-built RNN model to generate new text.
We will use a set of paragraphs from Shakespeare’s work as our training dataset. The dataset is in the GitHub, and the link is here.
IMPORTING THE LIBRARIES
import numpy as np import pandas as pd import matplotlib.pyplot as plt import tensorflow as tf from tensorflow import keras import sweetviz as sw import seaborn as sns sns.set()
We have used some of the commonly used libraries in deep learning. Here, you can see a new library called sweetviz, which automates exploratory data analysis and is very useful in analysing our training dataset.
shakespeare_url='https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt' filepath=keras.utils.get_file('shakespeare.txt',shakespeare_url) with open(filepath) as f: shakespeare_text=f.read()
Now, we have downloaded the dataset in our python notebook, and before we use it for training, we need to preprocess the text dataset.
PREPROCESSING THE DATASET
Tokenisation is the process in which the longer strings of text are broken into smaller chunks or tokens. Larger pieces of text can be tokenised into sentences, sentences into words. Further preprocessing is done after the tokenisation.
max_id=len(tokenizer.word_index) #Number of distinct characters dataset_size=tokenizer.document_count #Total number of characters [encoded]=np.array(tokenizer.texts_to_sequences([shakespeare_text]))-1 #Encoding the dataset
Here, we have used texts_to_sequences function, which basically removes filter like punctuation.
PREPARING THE DATASET
Note that we used tf.data.Dataset module which is generally useful for a large set of elements like text data from a book. The module usually follows a typical pattern.
- Create a source dataset from the input dataset
- Apply dataset transformations to preprocess the dataset
- Iterate over the dataset and process the elements
We have used from_tensor_slices function from the modules which take data from the tokenised array we created earlier and converts the data into tensor format over which transformations can be applied.
n_steps=100 window_length=n_steps+1 dataset=dataset.repeat().window(window_length,shift=1,drop_remainder=True)
Dataset.repeat() iterates over the dataset and repeats the dataset specified number of times. window() is like a sliding window that slides the window by a specified number each time the dataset iterates over itself.
dataset=dataset.flat_map(lambda window: window.batch(window_length)) batch_size=32 dataset=dataset.shuffle(10000).batch(batch_size) dataset=dataset.map(lambda windows: (windows[:,:-1],windows[:,1:])) dataset=dataset.map(lambda X_batch,Y_batch: (tf.one_hot(X_batch,depth=max_id),Y_batch)) dataset=dataset.prefetch(1)
Now that we have prepared our dataset for training let’s build our model.
BUILDING OUR MODEL
model=keras.models.Sequential() model.add(keras.layers.GRU(128,return_sequences=True,input_shape=[None,max_id])) model.add(keras.layers.GRU(128,return_sequences=True)) model.add(keras.layers.TimeDistributed(keras.layers.Dense(max_id,activation='softmax')))
Seems simple right. Generally, a single LSTM or GRU layer is powerful enough to build an RNN model. Here, we have used two GRU layers, and we have used a TimeDistributed Dense layer. Let’s compile and train our model.
model.compile(loss='sparse_categorical_crossentropy',optimizer='adam') history=model.fit(dataset,steps_per_epoch=train_size // batch_size,epochs=1)
31370/31370 [==============================] - 344s 11ms/step - loss: 1.0911
We have used ‘Adam’ optimizer here. You can also use other available optimizers to your need. Now that we have trained our model, we will test it using some text inputs.
TESTING OUR MODEL
def preprocess(texts): X=np.array(tokenizer.texts_to_sequences(texts))-1 return tf.one_hot(X,max_id)
def next_char(text,temperature=1): X_new=preprocess([text]) y_proba=model.predict(X_new)[0,-1:,:] rescaled_logits=tf.math.log(y_proba)/temperature char_id=tf.random.categorical(rescaled_logits,num_samples=1)+1 return tokenizer.sequences_to_texts(char_id.numpy())
def complete_text(text,n_chars=50,temperature=1): for _ in range(n_chars): text+=next_char(text,temperature) return text
We have defined some utility function as when text input is provided. It preprocesses and prepares it according to our defined model and predicts the next characters up to the specified number of characters. Let’s test it with some input text character.
t and our of men these me some contake and faith hi
justress to see 'twixts i reved be such sonleous ha
Now, let’s see how our model predictability varies with varying t=input temperature parameter.
Still in her in: what
S the other and in pr
In this tutorial, we have looked at how to train a Shakespearean text using a custom-built RNN model and test it using some text inputs. We also looked at how our model predictability varies with input temperature pa.