Logistic Regression in TensorFlow

Logistic Regression is one of the most used Algorithms in Machine Learning. In this tutorial, Let’s Implement it using TensorFlow with Python programming.

Introduction

Unlike linear regression, logistic regression is used to predict categories. Clearly, we use the so-called logistic function or sigmoid. This function takes a value between 0 and 1. Indeed, we can define a threshold to predict the output. It helps us in solving supervised learning problems.

import matplotlib.pyplot as plt
import numpy as np
import math
x = np.linspace(-10, 10, 100) 
z=1/(1+np.exp(-x))
plt.plot(x, z) 
plt.xlabel("x") 
plt.ylabel("Sigmoid(X)") 
plt.show() 

Firstly, the weights are initialized with random values at the beginning of the training. Then, we find the error in our model with a cost function. We try to minimize it using gradient descent. Finally, the model is trained multiple times known as “Epochs”.

TensorFlow Implementation with Python

Now we are going to implement the Logistic Regression in TensorFlow with the Python code. So let’s follow the steps with me…

First of all, import libraries and datasets. Using the MNIST dataset for implementation.

import input_data
import tensorflow as tf
import matplotlib.pyplot as plt

#Import Dataset 
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

Next, we define model parameters.

training_epochs=25
learning_rate = 0.01
batch_size = 100
display_step = 1

The Model

By now, you would be familiar with the MNIST dataset. If not, you can learn about it from:

MNIST-Dataset

x = tf.placeholder("float", [None, 784])
y = tf.placeholder("float", [None, 10])
# Create model
# Set model weights
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
# Construct model
activation = tf.nn.softmax(tf.matmul(x, W) + b)

Our aim is to predict the digit written. Obviously, when programming, we will deal with vectors. To assign probabilities, we have used the softmax function. Firstly, we define a weight tensor of the size of 784×10. We could use the distance function for calculating error. But we should prefer cross-entropy with neural networks.

# Minimize error using cross entropy
cross_entropy = y*tf.log(activation)
cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(actv), reduction_indices=1))
# OPTIMIZER
learning_rate = 0.01
optm = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
#Used Gradient Descent Algorithm

Finally, we will compile the model and plot the output:

with tf.Session() as sess: 
      
    # Initializing the Variables 
    sess.run(init) 
    cost_history, accuracy_history = [], [] 
    # Iterating through all the epochs 
    for epoch in range(epochs): 
        cost_per_epoch = 0
          
        # Running the Optimizer 
        sess.run(optimizer, feed_dict = {X : x, Y : y}) 
          
        # Calculating cost on current Epoch 
        c = sess.run(cost, feed_dict = {X : x, Y : y}) 
          
        # Calculating accuracy on current Epoch 
        correct_prediction = tf.equal(tf.argmax(Y_hat, 1), 
                                          tf.argmax(Y, 1)) 
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, 
                                                 tf.float32)) 
          
        # Storing Cost and Accuracy to the history 
        cost_history.append(sum(sum(c))) 
        accuracy_history.append(accuracy.eval({X : x, Y : y}) * 100) 
          
        # Displaying result on current Epoch 
        if epoch % 100 == 0 and epoch != 0: 
            print("Epoch " + str(epoch) + " Cost: "
                            + str(cost_history[-1]))  
    Weight = sess.run(W) # Optimized Weight 
    Bias = sess.run(b)   # Optimized Bias  
    # Final Accuracy 
    correct_prediction = tf.equal(tf.argmax(Y_hat, 1), 
                                      tf.argmax(Y, 1)) 
    accuracy = tf.reduce_mean(tf.cast(correct_prediction,  
                                             tf.float32)) 
    print("\nAccuracy:", accuracy_history[-1], "%")

Why Logistic Regression?

For a machine learning beginner, understanding Logistic Regression is very important. You might be eventually using other algorithms like Support Vector Machines, KNN, K-Means, Decision Trees, etc. Logistic regression makes it easy to understand the basics of these algorithms. Not only it introduces you to supervised learning, but it also involves a lot of maths used in many machine learning models.

Feel Free to ask your doubts in the comments section.