Text Generation with Keras and Tensorflow using LSTM and tokenization
Hello Everyone, today we will explore the world of writings! In this tutorial, we will be using the concept of Natural Language Processing (NLP) from Deep learning with TensorFlow and Keras API in Python.
What is NLP?
Well to put in simple terms, it is a predictive analysis technique. The machine analyzes the human language in terms of text and speeches.
We will generate text where a set of phrases are feed as input. The model will predict the next set of words from the previously gained knowledge from the trained dataset.
We will be using Shakespeare’s Caesar for the same.
Let’s start to code now!
IMPORTING PYTHON LIBRARIES
from keras.models import Sequential from keras.models import load_model import numpy as np import re from keras.layers import Dropout import pandas from keras.layers import Dense from keras.layers import LSTM from random import randint from keras.utils import to_categorical from keras.layers import Embedding
Now, we will download the required dataset. Rather than manually downloading the dataset let us use python’s nltk library to do the same. If one does not have nltk installed on their system, simply run pip install nltk on your terminal. After that follow the below commands
python; //run python on the terminal import nltk nltk.download('gutenberg')
We will be using the Gutenberg dataset that has around 3036 books of total of 14 authors and it has the one we require namely, ‘Caesar ‘.
Later print a certain set of book names so that we can get a sneak peak into the contents of the dataset we have.
import nltk nltk.download('gutenberg') from nltk.corpus import gutenberg as gut print(gut.fileids())
['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt', 'bible-kjv.txt', 'blake-poems.txt', 'bryant-stories.txt', 'burgess-busterbrown.txt', 'carroll-alice.txt', 'chesterton-ball.txt', 'chesterton-brown.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'shakespeare-caesar.txt', 'shakespeare-hamlet.txt', 'shakespeare-macbeth.txt', 'whitman-leaves.txt']
We will now view some contents of the caesar book and print the first 500 characters of the book present in the dataset.
from nltk.corpus.gutenberg import raw caesar_text = raw('shakespeare-caesar.txt') print(caesar_text[:500])
Check the output below:
[The Tragedie of Julius Caesar by William Shakespeare 1599] Actus Primus. Scoena Prima. Enter Flauius, Murellus, and certaine Commoners ouer the Stage. Flauius. Hence: home you idle Creatures, get you home: Is this a Holiday? What, know you not (Being Mechanicall) you ought not walke Vpon a labouring day, without the signe Of your Profession? Speake, what Trade art thou? Car. Why Sir, a Carpenter Mur. Where is thy Leather Apron, and thy Rule? What dost thou with thy best Apparrell on
From the output, we can observe that the text contains many numbers and special characters as well hence we need to filter them out so that the data can be processed in the way that the machine understands.
Define a filter function preprocess_text() to remove the special characters and punctuations from the text.
def preprocess_text(sent): # first we will remove all the punctuations that can cause problems for the model nonPunctuationsSentence = re.sub('[^a-zA-Z]', ' ', sent) # We will now remove all the single characters from the sentences nonCharSentence = re.sub(r"\s+[a-zA-Z]\s+", ' ', nonPunctuationsSentence ) # Spaces can also cause problem for the model noSpaceSentence = re.sub(r'\s+', ' ', nonCharSentence) return noSpaceSentence.lower()
Since we will be focusing only on the first 500 words so we will apply our filter function on first 500 words.
Analyze the output:
the tragedie of julius caesar by william shakespeare actus primus scoena prima enter flauius murellus and certaine commoners ouer the stage flauius hence home you idle creatures get you home is this holiday what know you not being mechanicall you ought not walke vpon labouring day without the signe of your profession speake what trade art thou car why sir carpenter mur where is thy leather apron and thy rule what dost thou with thy best apparrell on you sir what trade are you cobl truely sir in'
What is tokenization?
It is a method in which we split the given text into smaller units such as single words or numbers by locating the ending point of a word and the starting of the next following word.
Here, we will convert the individual words into integers for which we first need to convert the entire text into single words for which we need to use word_tokenize() from nltk.tokenize.
Let us also print the number of unique words present in the tokenized text.
from nltk.tokenize import word_tokenize as tokenizer caesar_text_words = (tokenizer(caesar_text)) n_words = len(caesar_text_words ) print('Total Words: %d' % n_words) unique_words = len(set(caesar_text_words )) print('Unique Words: %d' % unique_words)
Total Words: 19642 Unique Words: 3003
Now I have used the tokenizer provided by nltk so far. However, as the model is being trained on TensorFlow/Keras we will use the Keras Tokenizer to fit out caesar text.
from keras.preprocessing.text import Tokenizer tokenizer = Tokenizer(num_words=3003) tokenizer.fit_on_texts(caesar_text_words)
To keep track of all the words I made a variable to store the length of our vocabulary. Once that was done, I used the tokenizer class to convert all the words into the index.
vocab_size = len(tokenizer.word_index) + 1 word_2_index = tokenizer.word_index print(caesar_text_words) print(word_2_index[caesar_text_words])
You can see the sample output below:
Modifying the shape of data
As text generation is a field of deep learning that falls into the many-to-one sequence category, (since the input is a sequence of words and the output is a single word), we will make use of LSTM for this.
I have extensively touched on LSTM in one of my previous tutorials which you can find here ⇒ here
But first, we will need to modify the shape of the input sequences and the corresponding outputs for the same, so it is human-readable:
sequenceInput=  wordOutput=  seq_length= 100 for i in range(0, n_words - seq_length, 1): in_seque= caesar_text_words[i:i + seq_length] out_seque = caesar_text_words[i + seq_length] sequenceInput.append([word_2_index[word] for word in in_seque]) wordOutput.append(word_2_index[out_seque])
In the code above, I had declared two lists for input sequence and output words. I converted the words to index and store the first 100 in input sequences, such that the 101st will be treated as an output. The next iteration will shift the inputs from 100 to 101, and the output to 102nd word, this will go on.
[2, 886, 5, 1357, 12, 37, 1358, 1359, 471, 1360, 1361, 1362, 54, 472, 887, 1, 394, 888, 297, 2, 1363, 472, 230, 231, 4, 694, 889, 249, 4, 231, 8, 16, 1364, 29, 49, 4, 7, 272, 1365, 4, 553, 7, 358, 67, 1366, 71, 473, 2, 554, 5, 23, 1367, 98, 29, 395, 130, 32, 1368, 75, 137, 1369, 474, 108, 8, 63, 890, 1370, 1, 63, 891, 29, 555, 32, 22, 63, 232, 1371, 43, 4, 137, 29, 395, 33, 4, 892, 893, 137, 9, 359, 5, 1372, 1373, 62, 26, 28, 4, 72, 93, 894, 474]
This is a sample input sequence in terms of the index. Now for the next step, the data need to be shaped as per the input requirement of our model.
X = np.reshape(input_sequence, (len(input_sequence), input_seq_length, 1)) X = X / float(vocab_size) y = to_categorical(output_words)
We will use the traditional NumPy reshape function to make the index input-friendly. In the code below, I have normalized the data for easier information. And lastly, output_words was sent to categorical function from Keras library for training.
print("The shape for X :", X.shape) print("The shape for Y :", y.shape)
I will now print the shape of X and Y just for our reassurance, this step can be skipped but I highly recommend to cross-check the data before training.
The shape for X : (19542, 100, 1) The shape for Y : (19542, 3004)
I will define our model as a Sequential with 3 LSTM layers all with 800 hidden nodes and finish off the layers with a softmax activated dense layer.
model = Sequential() model.add(LSTM(500, input_shape=(X.shape, X.shape), return_sequences=True)) model.add(LSTM(300, return_sequences=True)) model.add(LSTM(200)) model.add(Dense(y.shape, activation='softmax')) model.summary() model.compile(loss='categorical_crossentropy', optimizer='adam')
I will also use adam as my optimizer as it is an all-rounder for all different types of models. You can use any kind of optimizer you for your usecase.
The model summary would look like this:
Model: "sequential_11" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= lstm_1 (LSTM) (None, 100, 500) 1004000 _________________________________________________________________ lstm_2 (LSTM) (None, 100, 300) 961200 _________________________________________________________________ lstm_3 (LSTM) (None, 200) 400800 _________________________________________________________________ dense_1 (Dense) (None, 3004) 603804 ================================================================= Total params: 2,969,804 Trainable params: 2,969,804 Non-trainable params: 0
Now to start the heavy work we will call the fit method in the model class to start the training.
model.fit(X, y, batch_size=64, epochs=10, verbose=1)
After the training, this will be the output if all goes good.
Epoch 9/10 17150/17150 [==============================] - 191s 11ms/step - loss: 6.6049 Epoch 10/10 17150/17150 [==============================] - 193s 11ms/step - loss: 6.6033
Testing the model:
For testing I used the following input request picked at random from the sonnet data set, this following function will pick and print your input sequence.
randomSeq_index = np.random.randint(0, len(sequenceInput)-1) randomSeq = sequenceInput[randomSeq_index] index_2_wordMap = map(reversed, word_2_index.items()) index_2_word = dict(index_2_wordMap) wordSequence = [index_2_word[indexValue] for indexValue in randomSeq] print(' '.join(wordSequence))
output of the above code for me is this:
that spare cassius he reades much he is great obseruer and he lookes quite through the deeds of men he loues no playes as thou dost antony he heares no musicke seldome he smiles and smiles in such sort as if he mock himselfe and scorn his spirit that could be mou.....my right hand for this
Note: This will differ for you on each run as it is randomised.
Now I will pick the 100 words range and feed it into our model to predict the following phrases.
for i in range(100): sampleInt = np.reshape(randomSeq, (1, len(randomSeq), 1)) sampleInt = sampleInt / float(vocab_size) indexPrediction = model.predict(sampleInt) wordIDPrediction = np.argmax(indexPrediction) inSeq = [index_2_word[Valueindex] for Valueindex in randomSeq] wordSequence.append(index_2_word[ wordIDPrediction]) randomSeq.append(wordIDPrediction) randomSeq = randomSeq[1:len(randomSeq)] final_output = "" for words in wordSequence: final_output = final_output + " " + words print(final_output)
For the above two sets of code blocks, I have used the previously explained functions only so be sure to reread the parts that seem confusing to you.
And the output looks like this:
that spare cassius he reades much he is great obseruer and he lookes quite through the deeds of men he loues no playes as thou dost antony he heares no musicke seldome he smiles and smiles in such sort as if he mock himselfe and scorn his spirit that could be mou to smile at any thing such men as he be neuer at hearts ease whiles they behold greater then themselues and therefore are they very dangerous rather tell thee what is to be fear then what feare for alwayes am caesar come on my right hand for this and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and
ouch, well honestly this is better than what we could have imagined as it is still really difficult to impersonate a normal human being let alone a legendary writer. But maybe with better data and more training, you can extract better output. Best of luck!