Build Simple Classifier using Low-Level APIs using TensorFlow in Python

Hello learners, today in this tutorial we will learn about building a simple classifier using low-level APIs using TensorFlow in Python.
The classifier is special kinds of hypotheses in an algorithm, with the help of these discrete value function we assign a class to points.APIs are some special programming rules for accessing web-based applications.
In this tutorial, we will specifically focus on making classifiers using low-level APIs. The low-level APIs provide us complete flexibility over the tools.

Simple Classifier using Low-Level APIs

First of all load the following libraries in the programming file. Make sure you have the latest version of the libraries on your system.

import numpy as np
import pandas as pd
import tensorflow as tf
from enum import Enum
from sklearn.datasets import load_iris

Using the following Activation function, Batch size, Epochs, Hidden neuron layers, optimizer, and the output neurons:

class HyperParams(Enum):
    ACTIVATION     = tf.nn.relu
    BATCH_SIZE     = 5
    EPOCHS         = 500
    HIDDEN_NEURONS = 10
    NORMALIZER     = tf.nn.softmax
    OUTPUT_NEURONS = 3
    OPTIMIZER      = tf.keras.optimizers.Adam

The next step is loading the data. We will use the Iris dataset.
For the partition of objects, I will use the xdat and ydat np.ndarray.

iris = load_iris()
xdat = iris.data
ydat = iris.target

To organize the data, we will use the following class which contains methods on batching, tensorizing, and partitioning.

class Data:

    def __init__(self, xdat: np.ndarray, ydat: np.ndarray, ratio: float = 0.3) -> Tuple:
        self.xdat  = xdat
        self.ydat  = ydat
        self.ratio = ratio

    def partition(self) -> None:
        scnt = self.xdat.shape[0] / np.unique(self.ydat).shape[0]
        ntst = int(self.xdat.shape[0] * self.ratio / (np.unique(self.ydat)).shape[0])
        idx  = np.random.choice(np.arange(0, self.ydat.shape[0] / np.unique(self.ydat).shape[0], dtype = int), ntst, replace = False)
        for i in np.arange(1, np.unique(self.ydat).shape[0]):
            idx = np.concatenate((idx, np.random.choice(np.arange((scnt * i), scnt * (i + 1), dtype = int), ntst, replace = False)))

        self.xtrn = self.xdat[np.where(~np.in1d(np.arange(0, self.ydat.shape[0]), idx))[0], :]
        self.ytrn = self.ydat[np.where(~np.in1d(np.arange(0, self.ydat.shape[0]), idx))[0]]
        self.xtst = self.xdat[idx, :]
        self.ytst = self.ydat[idx]

    def to_tensor(self, depth: int = 3) -> None:
        self.xtrn = tf.convert_to_tensor(self.xtrn, dtype = np.float32) 
        self.xtst = tf.convert_to_tensor(self.xtst, dtype = np.float32)
        self.ytrn = tf.convert_to_tensor(tf.one_hot(self.ytrn, depth = depth))
        self.ytst = tf.convert_to_tensor(tf.one_hot(self.ytst, depth = depth))
    
    def batch(self, num: int = 16) -> None:
        try:
            size = self.xtrn.shape[0] / num
            if self.xtrn.shape[0] % num != 0:
                sizes = [tf.floor(size).numpy().astype(int) for i in range(num)] + [self.xtrn.shape[0] % num]
            else:
                sizes = [tf.floor(size).numpy().astype(int) for i in range(num)]

            self.xtrn_batches = tf.split(self.xtrn, num_or_size_splits = sizes, axis = 0)
            self.ytrn_batches = tf.split(self.ytrn, num_or_size_splits = sizes, axis = 0)

            num = int(self.xtst.shape[0] / sizes[0])
            if self.xtst.shape[0] % sizes[0] != 0:
                sizes = [sizes[i] for i in range(num)] + [self.xtst.shape[0] % sizes[0]]
            else:
                sizes = [sizes[i] for i in range(num)]

            self.xtst_batches = tf.split(self.xtst, num_or_size_splits = sizes, axis = 0)
            self.ytst_batches = tf.split(self.ytst, num_or_size_splits = sizes, axis = 0)
        except:
            self.xtrn_batches = [self.xtrn]
            self.ytrn_batches = [self.ytrn]
            self.xtst_batches = [self.xtst]
            self.ytst_batches = [self.ytst]

Now the next step is to apply the code on the dataset that we have loaded.

data = Data(xdat, ydat)
data.partition()
data.to_tensor()
data.batch(HyperParams.BATCH_SIZE.value)

The Dense Layer:

We will be defining the Dense layer from scratch to use the low-level APIs. There are several ways to do this, Gaussian distribution is one of these.
Firstly we define the weight values and then we go ahead to define the feedforward. Below is the Python code:

class Dense:

    def __init__(self, i: int, o: int, f: Callable[[tf.Tensor], tf.Tensor], initializer: Callable = tf.random.normal) -> None:
        self.w = tf.Variable(initializer([i, o]))
        self.b = tf.Variable(initializer([o]))
        self.f = f

    def __call__(self, x: tf.Tensor) -> tf.Tensor:
        if callable(self.f):
            return self.f(tf.add(tf.matmul(x, self.w), self.b))
        else:
            return tf.add(tf.matmul(x, self.w), self.b)

Using the dense layer for the feedforward calculations.

layer = Dense(4, 3, tf.nn.relu)
layer(data.xtrn[1:3, :])

layer(data.xtrn[1:4, :])
<tf.Tensor: shape=(3, 3), dtype=float32, numpy=
array([[ 9.751311 ,  0.       ,  5.747163 ],
       [ 9.624053 ,  0.       ,  5.7506356],
       [10.481971 ,  0.       ,  6.19804  ]], dtype=float32)>

The Chaining process:

Now extending the layers to the come up with the MLP architecture we use the Chain for layers. Here it is advised to use the Keras Optimizers for better results.
Let me also give a gist of the following code. After declaring the layer we need to train on the data and hence optimize the weights. To do so we use the backpropagation method.

class Chain:

    def __init__(self, layers: List[Iterable[Dense]]) -> None:
        self.layers = layers
    
    def __call__(self, x: tf.Tensor) -> tf.Tensor:
        self.out = x; self.params = []
        for l in self.layers:
            self.out = l(self.out)
            self.params.append([l.w, l.b])
        
        self.params = [j for i in self.params for j in i]
        return self.out

    def backward(self, inputs: tf.Tensor, targets: tf.Tensor) -> None:
        grads = self.grad(inputs, targets)
        self.optimize(grads, 0.001)
    
    def loss(self, preds: tf.Tensor, targets: tf.Tensor) -> tf.Tensor:
        return tf.reduce_mean(
            tf.keras.losses.categorical_crossentropy(
                targets, preds
            )
        )
        
    def grad(self, inputs: tf.Tensor, targets: tf.Tensor) -> List:
        with tf.GradientTape() as g:
            error = self.loss(self(inputs), targets)
        
        return g.gradient(error, self.params)

    def optimize(self, grads: List[tf.Tensor], rate: float) -> None:
        opt = HyperParams.OPTIMIZER.value(learning_rate = rate)
        opt.apply_gradients(zip(grads, self.params))

After the weight calculation using the optimizers and the gradient calculations is done. Now it is time to chain up the layers together.

model = Chain([
    Dense(data.xtrn.shape[1], HyperParams.HIDDEN_NEURONS.value, HyperParams.ACTIVATION),
    Dense(HyperParams.HIDDEN_NEURONS.value, HyperParams.OUTPUT_NEURONS.value, HyperParams.NORMALIZER)
])

Now we use the following to call the model function by passing the desired values. Note that the output received here is on the untrained weights and may not be accurate. This is because we have not yet trained our model on the data, just written the code for it.

model(data.xtrn[1:3, :])

model(data.xtrn[1:6, :])
<tf.Tensor: shape=(5, 3), dtype=float32, numpy=
array([[1.6694264e-25, 8.5078453e-04, 9.9914920e-01],
       [4.2532067e-25, 1.1736125e-03, 9.9882632e-01],
       [9.0538706e-27, 7.7706709e-04, 9.9922287e-01],
       [7.0977163e-29, 4.3619360e-04, 9.9956375e-01],
       [2.9547798e-25, 1.4161889e-03, 9.9858379e-01]], dtype=float32)>

Training and finally testing the model:

After customizing out training procedure to our requirements, it is now time to train the model on the data and improve accuracy.

def accuracy(y, yhat):
    j = 0; correct = []
    for i in tf.argmax(y, 1):
        if i == tf.argmax(yhat[j]):
            correct.append(1)
        
        j += 1
    
    num = tf.cast(tf.reduce_sum(correct), dtype = tf.float32)
    den = tf.cast(y.shape[0], dtype = tf.float32)
    return num / den

Now the step is to define the number epochs, using loops we can parse through the data.

epoch_trn_loss = []
epoch_tst_loss = []
epoch_trn_accy = []
epoch_tst_accy = []
for j in range(HyperParams.EPOCHS.value):
    trn_loss = []; trn_accy = []
    for i in range(len(data.xtrn_batches)):
        model.backward(data.xtrn_batches[i], data.ytrn_batches[i])
        ypred = model(data.xtrn_batches[i])
        trn_loss.append(model.loss(ypred, data.ytrn_batches[i]))
        trn_accy.append(accuracy(data.ytrn_batches[i], ypred))

    trn_err = tf.reduce_mean(trn_loss).numpy()
    trn_acy = tf.reduce_mean(trn_accy).numpy()

    tst_loss = []; tst_accy = []
    for i in range(len(data.xtst_batches)):
        ypred = model(data.xtst_batches[i])
        tst_loss.append(model.loss(ypred, data.ytst_batches[i]))
        tst_accy.append(accuracy(data.ytst_batches[i], ypred))
    
    tst_err = tf.reduce_mean(tst_loss).numpy()
    tst_acy = tf.reduce_mean(tst_accy).numpy()
    
    epoch_trn_loss.append(trn_err)
    epoch_tst_loss.append(tst_err)
    epoch_trn_accy.append(trn_acy)
    epoch_tst_accy.append(tst_acy)
    
    if j % 25 == 0:
        print("Epoch: {0:5d} \t Error in Training: {1:.5f} \t Error in Testung: {2:.5f} \t Accuracy in Training: {3:.5f} \t Accuracy in Testing: {4:.5f}".format(j, trn_err, tst_err, trn_acy, tst_acy))
Epoch:     0 	 Error in Training: 5.84301 	 Error in Testing: 4.31555 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:    25 	 Error in Training: 5.84299 	 Error in Testing: 4.31554 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:    50 	 Error in Training: 5.84297 	 Error in Testing: 4.31554 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:    75 	 Error in Training: 5.84296 	 Error in Testing: 4.31553 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   100 	 Error in Training: 5.84294 	 Error in Testing: 4.31553 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   125 	 Error in Training: 5.84293 	 Error in Testing: 4.31552 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   150 	 Error in Training: 5.84291 	 Error in Testing: 4.31552 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   175 	 Error in Training: 5.84290 	 Error in Testing: 4.31551 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   200 	 Error in Training: 5.84288 	 Error in Testing: 4.31551 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   225 	 Error in Training: 5.84287 	 Error in Testing: 4.31551 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   250 	 Error in Training: 5.84285 	 Error in Testing: 4.31550 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   275 	 Error in Training: 5.84284 	 Error in Testing: 4.31550 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   300 	 Error in Training: 5.84283 	 Error in Testing: 4.31550 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   325 	 Error in Training: 5.84281 	 Error in Testing: 4.31549 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   350 	 Error in Training: 5.84280 	 Error in Testing: 4.31549 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   375 	 Error in Training: 5.84279 	 Error in Testing: 4.31549 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   400 	 Error in Training: 5.84278 	 Error in Testing: 4.31548 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   425 	 Error in Training: 5.84276 	 Error in Testing: 4.31548 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   450 	 Error in Training: 5.84275 	 Error in Testing: 4.31548 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794
Epoch:   475 	 Error in Training: 5.84274 	 Error in Testing: 4.31547 	 Accuracy in Training: 0.34286 	 Accuracy in Testing: 0.50794

The above code shows the output of the model over 500 epochs, with the results being displayed after every 25 epochs. We have now successfully created a basic classifier using low-level APIs using TensorFlow in Python.

Hope you enjoyed learning with me in this tutorial. Have a good day and happy learning.

Leave a Reply

Your email address will not be published. Required fields are marked *