Install Samba on Linux

Samba is an open-source software suite that provides file and print services to SMB/CIFS clients. It allows Linux computers to participate in a Windows-based network and supports accessing and sharing files and printers with Windows computers. Samba implements the SMB/CIFS protocol and can create file and printer shares on a Linux machine that can be accessed from Windows computers and access Windows files and printer shares from a Linux machine. This way, Samba enables seamless file and printer sharing between Windows and Linux computers in a mixed-operating system environment.

Install samba:

sudo apt install samba -y

On successful installation, check the smbd status

systemctl status smbd

Replace wamaitha with your computer username and change paths to suite your needs.

We want to share movies from Linux to Windows. Create a folder that will be the shared folder.

mkdir /media/wamaitha/data/movies

Give full permission to the folder.

chmod 777 /media/wamaitha/data/MOVIES

Create a user that will be associated with the file sharing:

 useradd sambauser

Create an smb password for this user

smbpasswd -a sambauser

Edit the samba config file using nano or any editor of your choice

nano /etc/samba/smb.conf

Scroll to the bottom of the config file and add the following:

[Movies]
path = /media/wamaitha/data/MOVIES
valid users = sambauser
read only = no
browsable = yes
public = yes
writable = yes
browsable = yes

Save the file and check if everything was correctly configured using the following command:

testparm

Check your IP address on the Linux machine using ifconfig:

ifconfig wlo1

On the output, the IP address is shown below:

Connecting on windows

On windows launch run by pressing the windows logo + R .

Type the IP address of the Linux machine from the ipconfig output with two slashes.

\\192.168.0.101

This will open the Linux-shared folders.

You can now access the files in the Movies folder on Linux from windows.

Resources:

https://www.samba.org/samba/what_is_samba.html

Advantages of using wget

• wget is built to cater to slow and/or unstable networks. The wget utility retries when a download fails and supports downloading from where the last download failed (if the server the user is downloading from supports Range headers).

• Furthermore, the utility allows the use of proxy servers which can aid in faster retrievals and lighter network loads.

• wget supports recursive downloads. A recursive download refers to the process of downloading a file and then using that file to download additional files, repeating this process until all necessary files have been downloaded. This is often used in downloading a website or other collection of files from the internet, where the initial file (e.g., a HTML page) contains links to other files (e.g., images, stylesheets, etc.) that are also needed to render the content fully. Therefore, wget can be tweaked to work as a website crawler.

• Last but not least, wget supports downloads using the FTP, HTTP, and HTTPS protocols, all internet protocols used to transfer data between computers. FTP (File Transfer Protocol) is a standard network protocol that transfers files from one host to another over a TCP-based network. FTP allows clients to upload, download, and manage files on a remote server. HTTP (Hypertext Transfer Protocol) is a standard protocol for sending and receiving information on the World Wide Web. It is the foundation of data communication on the web and is used to transmit data from a server to a client, such as a web browser. HTTPS (HTTP Secure) is a variant of HTTP that uses a secure SSL/TLS connection to encrypt data sent between a server and a client. This makes it more difficult for someone to intercept and read the data as it is transmitted. HTTPS is commonly used to protect sensitive information, such as login credentials and financial transactions.

The wget comes installed on most Linux distributions. To install it on Debian-based distributions use:

sudo apt install wget

The wget syntax is:

wget [options] [URL]

• wget invokes the wget utility

• [options] instruct wget on what to do with the URL provided. Some options have a long-form and short-form version.

• [URL] file or folder to be downloaded. This can also be a website link to download

Download Options

We will cover the following download options

1. Retrying when a connection fails.

2. Saving downloaded files with a different name.

3. Continuation of partially downloaded files.

4. Download to a specific folder.

5. Using proxies.

6. Limit download speeds.

7. Extracting from multiple URLs.

8. Mirror a webpage.

9. Extract entire websites.

10. Extract a website as if you were Googlebot.

Retrying when a connection fails.

• By default wget retries 20 times except for fatal errors. Examples of fatal errors are connection refused or 404 (not found) errors.

    # Long form 
    wget --tries=number http://example.com/myfile.zip
  
    # Short form
    wget -t number http://example.com/myfile.zip
  

  The number can be specified as 0 or inf for infinite retries

    # Long form 
    wget --tries=0 http://example.com/myfile.zip
  
    # Short form
    wget -t 0 http://example.com/myfile.zip
  

Saving downloaded files with a different name.

• The default behavior is to save the file with the name derived from the URL. Using the output file option, you can specify a new name for the file.

    # Long form
    wget -O newfilename.zip http://example.com/myfile.zip
    # short form
    wget --output-file=newfilename.zip http://example.com/myfile.zip
  

  In the code above, the file downloaded myfile.zip will be saved as newfilename.zip .

  However, it is important to note that the use of -O is not intended to mean simply “use the name newfilename instead of myfilezip as provided in the URL;”. Rather, it is analogous to shell redirection that works like :

    wget -O - http://example.com/myfile.zip > newfilename.zip
  

  newfilename.zip will be truncated, and all downloaded content will be saved there.

Continuation of partially downloaded files.

• The continue option allows the continuation of a partially downloaded file. This can be a file that was downloaded by another wget instance or another program.

    # Long form
    wget --continue http://example.com/myfile.zip
  
    # Short form
    wget -c http://example.com/myfile.zip
  

  Assuming there was a partial download with the name myfile.zip, wget assumes the local copy is the first portion of the file and attempts to retrieve the rest of the file from the server using an offset equal to the length of the local file. In the event, the file found locally, and that on the server are of equal size, wget prints an explanatory message

  

  Consequently, should the file on the server be smaller than the one locally, the assumption made is the file on the server is an updated version. However, if the file on the server is bigger than locally due to a new version, the download will continue and the result becomes a garbled file.

  The continue option only works with FTP servers and HTTP servers that support the Range header.

Download to a specific folder

To download a file with wget and save it to a specific folder, the -P or --directory-prefix option followed by the path to the directory where files are to be saved are used.

For example:

# Long Form
wget -directory-prefix=/path/to/directory http://example.com/myfile.zip

# Short form
wget -P /path/to/directory http://example.com/myfile.zip

Using proxies

To use a proxy with wget, the --proxy option followed by the proxy URL are used.

For example:

wget --proxy=http://myproxy.example.com:8080 http://example.com/myfile.zip

This downloads the file myfile.zip from http://example.com using the HTTP proxy at http://myproxy.example.com:8080.

If the proxy requires authentication, use the --proxy-user and --proxy-password options to specify the username and password.

For example:

wget --proxy=http://myproxy.example.com:8080 --proxy-user=myusername --proxy-password=mypassword http://example.com/myfile.zip

The --proxy-type option can also be used to specify the type of proxy being used. Valid values include http, socks4, and socks5.

For example:

wget --proxy=http://myproxy.example.com:8080 --proxy-type=socks5 http://example.com/myfile.zip

If all downloads require going through a proxy every time, modify the ~/.wgetrc file. You can do this with nano or your favorite text editor

nano ~/.wgetrc

Add the following lines to the file

use_proxy = on
http_proxy =  http://username:password@proxy.server.address:port/
https_proxy =  http://username:password@proxy.server.address:port/

Limit download speeds

The --limit-rate option is used to limit the download speed of wget. For example, to limit the download speed to 100KB/s:

wget --limit-rate=100k http://example.com/file.zip

The maximum download speed in bytes per second can be specified by using the B suffix. For example, to limit the download speed to 100KB/s:

wget --limit-rate=100000B http://example.com/file.zip

Extracting multiple URLs

To download multiple files with wget, specify multiple URLs on the command line, separated by spaces.

For example:

wget http://example.com/file1.zip http://example.com/file2.zip http://example.com/file3.zip

This will download the files file1.zip, file2.zip, and file3.zip . If you have a list of URLs in a file, you can use the -i or --input-file option to tell Wget to read the URLs from the file.

For example:

# Long form
wget --input-file=url-list.txt
# Short form
wget -i url-list.txt

Mirror a webpage

To create a mirror of a webpage, you can use the wget command-line utility with the -m or --mirror option. This option tells wget to download the webpage and all the files it references (such as images, stylesheets, etc.), recursively and save them to your local machine.

For example:

# Long form
wget --mirror http://example.com

# Short form
wget -m http://example.com

This downloads the webpage at http://example.comand all the files it references, and save them to the current working directory. The files will be saved in a directory structure that mirrors the structure of the website, with the home page saved as index.html and other pages saved in subdirectories as necessary.

You can also use the -k or --convert-links option to tell wget to convert the links in the downloaded HTML pages to point to the local copies of the files so that the downloaded copy of the website can be browsed offline.

For example:

# Long form
wget --mirror --convert-links http://example.com

# Short form
wget -m -k http://example.com

Extract entire websites

To download an entire website with wget, you can use the --recursive option to follow links and download all the files it finds, recursively.

For example:

wget --recursive http://example.com

Extract the website as Googlebot

To extract a website as if you were Googlebot using Wget, you can use the --user-agent option to set the user agent string to "Googlebot". For example:

wget --user-agent="Googlebot" http://example.com

The website will be downloaded using the user agent string "Googlebot", which tells the server that the request is coming from Google's web crawler. This can be useful if you want to see how Googlebot would render a website or if you want to download a website while bypassing any restrictions that may be in place based on the user agent.

You can also combine options demonstrated, for example:

wget --user-agent="Googlebot" -O example.html http://example.com

Resources:

https://www.gnu.org/software/wget/manual/wget.html#Examples

alias [alias-name]='[command]'

• alias: Invokes the alias command.

• alias-name: An alias is a user-defined string that acts as a reference to a command, with the exception of certain special characters and the words 'alias' and 'unalias'.

• command: Specifies the command the alias references.

For example:

1. alias ll='ls -al': This alias creates a new command called ll that is equivalent to running the ls -al command. You could use this alias to list the contents of a directory in long format by typing ll instead of ls -al.

2. alias df='df -h': This alias creates a new command called df that is equivalent to running the df -h command. The -h flag is used to display the sizes of files in "human-readable" format (e.g., in MB or GB instead of bytes).

3. alias grep='grep --color=auto': This alias creates a new command called grep that is equivalent to running the grep --color=auto command. The --color=auto flag tells grep to highlight the matching text in the output, making it easier to spot.

4. alias shutdown='sudo shutdown -h now': This alias creates a new command called shutdown that is equivalent to running the sudo shutdown -h now command. The -h flag tells the shutdown command to halt the system, and the now argument tells it to do so immediately.

5. alias update='sudo apt update && sudo apt upgrade -y': This alias creates a new command called update that is equivalent to running the sudo apt update and sudo apt upgrade -y commands. The update command updates the package manager's list of available packages, and the upgrade command installs any available updates for installed packages. The -y flag tells apt to assume "yes" as the answer to all prompts.

A few ways I use the alias command in my day to day development:

Copying trading bots from wine to my documents.

I currently work on Linux but need to do MQL5 programming from time to time. With MQL5, I can write trading bots for clients. However, to compile the MQL5 code to an .ex5 executable, I need to have MT5 installed. MT5 is a Windows and Mac program; therefore, I have to install wine on Linux. Wine is a compatibility layer that allows you to run Windows applications on Linux. It does this by translating the Windows application's calls into calls that the Linux operating system can understand and processing them on behalf of the application. This allows you to run Windows applications on Linux without installing a copy of the Windows operating system. Once the code is compiled into an executable, the files are stored in the wine drive. To copy the .ex5 to a specific folder, I would normally run the following command

cd /home/wamaitha/.wine/drive_c/Program\ Files/Deriv/MQL5

The MQL5 directory has a folder that holds all the trading bots I program. Therefore, I can copy all the .ex5 files to the EAS folder with the command below.

cp -r Experts/**/*.ex5 ~/Documents/FOREX/EAS

To make my work easier, I've written an alias command that does all this

alias fxeas="cd /home/wamaitha/.wine/drive_c/Program\ Files/Deriv/MQL5 && cp -r Experts/**/*.ex5 ~/Documents/FOREX/EAS"

Activating virtual environments

In Python when using virtualenv , to activate the environment on Linux we need to type the following command:

source venv/bin/activate

Then navigate back to the project folder

cd .. && cd ..

We can write an alias for this

alias activate="source venv/bin/activate && cd .. && cd .."

Permanent aliases

Using the alias command only applies to the current terminal session. To make the commands useful on other terminal sessions, you would need to edit the .bashrc file if running bash or the .zshrc if on zsh. I'm currently on zsh , therefore I add these commands at the end of the .zshrc file.

Launch the .zshrc using the command below

nano ~/.zshrc

Add the alias commands at the end of the file

alias activate="source venv/bin/activate && cd .. && cd .."
alias fxeas="cd /home/wamaitha/.wine/drive_c/Program\ Files/Deriv/MQL5 && cp -r Experts/**/*.ex5 ~/Documents/FOREX/EAS"

Run the command below to activate the changes for all terminal sessions.

source ~/.zshrc

A demo of what we will be building is shown below.

Tools used

1. Terraform

Terraform is an IAC tool widely adopted to create, modify and destroy servers. Normally you would log on to Linode and deploy a Linux Server manually while specifying how many vCPUs, RAM, and storage your project will need. Terraform allows users to write a file that, when executed, will deploy the server on Linode. Terraform also allows the user to modify the servers by scaling up or down different resources. For example, increasing the amount of RAM an existing server has. Terraform can consequently destroy all or some of the servers as needed.

2. Ansible

Ansible is a suite of software IAC tools for configuration management, deployments, and software provisioning. Ansible works the same way a puppet master and his dolls do. The puppet master pulls strings that control the doll(s) to make them move or do something. The puppet master, in this case, is called the control machine, while the dolls are the remote machines. The strings that connect the doll and the puppet's hand are SSH connections. Ansible is only installed on the control machine. Therefore, the remote machines do not need special configurations making Ansible agentless.

3. Github Actions

Github Actions is a CICD platform that allows users to automate the build, test, and deployment pipelines when an event happens. Github Actions works through workflows. A workflow is a configurable automated process running one or more jobs. A repository can have one or more workflows.

For example, we can have a workflow that runs tests using Jest on every push request. Yet another workflow can deploy the application on every merge to the master branch. Events trigger the workflows. An event on Github can be a pull request, push a commit, or open or closed issue. The workflow runs a series of steps known as a job. The workflow runs on servers provisioned by Github called runners. Github provisions runners for Linux, Windows, and macOS. Therefore, if you're building a program that's meant to be run on windows specifically, you can create a workflow that runs on the Windows environment.

4 . Docker

Docker solves the classical problem of working on my machine but not on my friend's machine. Docker does this by providing a virtualization layer on the operating system it's installed on. The software is delivered in packages called containers. The container is realized from a docker image. A docker image is an immutable(Cannot be changed) snapshot of the environment in the container that will run the software. Immutability means the image can be consistently deployed in any environment.

5. Docker Compose

Compose is a tool for defining and running multi-container Docker applications. Our example will have five microservices, each on its own docker container. Using docker-compose, we will configure and manage these services. Docker compose is the puppet master, while the containers are the dolls.

Project Overview

This guide will use Linode as the hosting provider to deploy a full-stack application using React for the front end and five microservices on the back end. The microservices are:

1. Kanye as a service - Kanye jokes being served as a Python application.

2. Dad Jokes as a service - dadjokes being served as a Python application.

3. My jokes as a service - custom jokes being served using NodeJs from a MongoDB microservice.

4. React is served with nginx as a web server.

5. MongoDB stores the jokes added to microservice 3.

Since we will not be provisioning infrastructure every time, we will run Terraform on the laptop and let Github Actions handle CICD with Ansible. The initial setup steps will be:

• Generate an API key on Linode.

• Install Terraform locally

• Generate ssh keys.

• Spin up a nanode using Terraform.

• Terraform logs on to the VM and copies the public key to the nanode.

Development steps once the infrastructure is up will be:

• Develop microservices in Docker containers.

• Manage containers using docker-compose.

• Push code to Github.

• The Github runner updates the code in the VM using Ansible. Ansible will:

  • Login to VM using the private key.

  • Copy the new code into the VM

  • Rebuild the docker images and spin the containers using docker-compose.

Prerequisites

• Github account and git installed on your laptop.

• Python, NodeJs, and Npm are installed on your laptop.

• Docker installed on your laptop.

• Javascript and Python knowledge. The focus will be on the DevOps aspects of the project.

• Linode account.

• Bash for executing bash scripts

The code on this blog was written on Ubuntu 22.04.1 LTS. You may need to do more tweaking to support your current OS.

Installations

Install terraform on Linux.

wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg && 
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list && sudo apt update && sudo apt install terraform

Install Ansible

python3 -m pip install  ansible

Install ssh-pass. Used to pass password to ssh-copy-id when Terraform logs in to the VM to copy the public key.

sudo apt-get install sshpass

Microservices

We will start with the backend microservices REST APIs.All services will be in the microservices folder.

Dad Jokes as a service.

The service runs a flask app using gunicorn. The folder structure is as shown below:

Project_Folder
└─── microservices
    └─── Dadjokes
         | .dockerignore
         | dadjokes.py
         | Dockerfile
         | gunicorn.sh
         | requirements.txt
         | wsgi.py
    | docker-compose.yml

Code in .dockerignore excludes the Dockerfile from being included during the copying of files to docker.

# .dockerignore
Dockerfile

Code in dadjokes.py

# dadjokes.py
from flask import Flask
import requests
import json
from flask_cors import CORS

app = Flask(__name__)
CORS(app)

@app.route("/dadjoke", methods=['GET'])
def random_dad_joke():
    random_dad_joke = requests.get("https://icanhazdadjoke.com/",
                                   headers={"Accept": "application/json"})
    random_dad_joke = json.loads(random_dad_joke.text)["joke"]

    return json.dumps(random_dad_joke)

Code in gunicorn.sh

#!/bin/sh
# gunicorn.sh
gunicorn -b :7000 --access-logfile - --error-logfile - wsgi:app

WSGI (Web Server Gateway Interface) is a specification for a common interface between web servers and web applications. It defines a standard way for a web server to communicate with a web application and for the web application to communicate with the web server.

The above command will start the gunicorn web server and run the WSGI application specified by wsgi:app. The wsgi part specifies the module that contains the application, and the app part specifies the name of the application within the module. In this case, the module is specified in the wsgi.py The server will listen for connections on the port 7000 and write access and error logs to the console. The -b flag specifies the address and port on which the server should listen for connections. Therefore the server will listen on port 7000 on all available network interfaces. The --access-logfile and --error-logfile flags specify the locations of the access and error log files, respectively. The - character indicates that the log output should be written to standard output (the console) rather than a file.

Code in wsgi.py

# wsgi.py
from dadjokes import app

if __name__ == "__main__":
    app.run(debug=True)

Code in requirements.txt

# requirements.txt
flask
requests
gunicorn
flask-cors

Code in the Dockerfile

# Dockerfile
FROM python:3.8-slim-buster

WORKDIR /app

COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

COPY . .

EXPOSE 7000
CMD ["./gunicorn.sh"]

The Dockerfile creates a Docker image that will be run as a container. When the Docker image is built using this Dockerfile, it will create a container that has the Python 3.8 environment and installs the packages listed in requirements.txt, and runs the gunicorn web server on port 7000.

The first line of the Dockerfile specifies the base image on which the container should be based. Here the base image is python:3.8-slim-buster, which is a minimal Python 3.8 installation on top of the Debian "Buster" operating system.

The WORKDIR instruction sets the current working directory for subsequent instructions in the Dockerfile. The working directory is set to /app.

The COPY instruction copies a file or directory from the host file system into the container. The requirements.txt file is copied from the host into the /app directory in the container.

The RUN instruction runs a command in the container. The command pip3 install -r requirements.txt is run, which installs the Python packages listed in the requirements.txt file.

The second COPY instruction copies the entire current directory (.) from the host into the /app directory in the container. This will include all the files and directories in the current directory, including the gunicorn.sh script.

The EXPOSE instruction specifies the port that the container should expose to the host. The container will expose port 7000.

The CMD instruction specifies the command that should be run when the container is started. The command is ./gunicorn.sh, which is a shell script that starts the gunicorn web server.

Add the service to docker-compose.yml.

version: '3.8'

services:
  dadjokes:
    image: dadjokes
    container_name: dadjokes
    build:
      context: ./DadJokes
    networks:
      - dadjokes
    ports:
      - "7000:7000"

networks:
  dadjokes:

The first line version: '3.8'specifies the version of the Docker Compose file format being used.

The services block defines the services that make up the application. The first service isdadjokes.

The image property specifies the name of the Docker image that should be used to create the container. The image name is dadjokes.

The container_name property specifies the name that should be given to the container when it is created. The specified container name is dadjokes.

The build block specifies the build options for the service. The context property specifies the directory that contains the Dockerfile used to build the image. For dadjokes the Dockerfile is located in the ./DadJokes directory.

The networks block specifies the networks that the container should be connected to. The container will be connected to a network called dadjokes.

The ports block specifies the port mapping between the container and the host. Port 7000 on the container will be mapped to port 7000 on the host. This means that connections to port 7000 on the host will be forwarded to the container, and the container will be able to receive connections on port 7000.

The networks section defines a network called dadjokes. When you run docker-compose up, Docker will create this network if it doesn't already exist. The dadjokes service is then connected to this network, allowing the containers to communicate with each other over the network.

Kanye as a service

The service runs a flask app using gunicorn. The structure of this service looks similar to what we have in the dadjokes service. However, the port in use is is 8080 , and the code kanye.py returns a Kanye Quote.

Project_Folder
    └─── microservices
        └─── DadJokes
        └─── Kanye # <----------------- we are here
            | .dockerignore
            | Dockerfile
            | gunicorn.sh
            | kanye.py
            | requirements.txt
            | wsgi.py
       | docker-compose.yml

Code in .dockerignore

Dockerfile

Code in Dockerfile

FROM python:3.8-slim-buster

WORKDIR /app

COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

COPY . .

EXPOSE 8080 # <---- different port
CMD ["./gunicorn.sh"]

Code in gunicorn.sh

#!/bin/sh
gunicorn -b :8080 --access-logfile - --error-logfile - wsgi:app

Code in kanye.py

from flask import Flask
import requests
import json
from flask_cors import CORS

app = Flask(__name__)
CORS(app)

@app.route("/kanye", methods=['GET'])
def random_kanye_joke():
    random_kanye_joke = requests.get("https://api.kanye.rest",
                                   headers={"Accept": "application/json"})
    random_kanye_joke = json.loads(random_kanye_joke.text)["quote"]

    return json.dumps(random_kanye_joke)

Code in requirements.txt

flask
requests
gunicorn
flask-cors

Code in wsgi.py

from kanye import app

if __name__ == "__main__":
    app.run(debug=True)

Add the service to the docker-compose.yml file

version: '3.8'

services:

  dadjokes:
    image: dadjokes
    container_name: dadjokes
    build:
      context: ./DadJokes
    networks:
      - dadjokes
    ports:
      - "7000:7000"

  kanye: # <-------------------- kanye service added
    image: kanye
    container_name: kanye
    build:
      context: ./Kanye
    networks:
      - kanye
    ports:
      - "8080:8080"

networks:
  dadjokes:
  kanye:

MongoDB as a service

The service is defined in the docker-compose.yml file.

version: '3.8'

services:

  dadjokes:
    image: dadjokes
    container_name: dadjokes
    build:
      context: ./DadJokes
    networks:
      - dadjokes
    ports:
      - "7000:7000"

  kanye:
    image: kanye
    container_name: kanye
    build:
      context: ./Kanye
    networks:
      - kanye
    ports:
      - "8080:8080"

  mongo: #<------------------------- add mongo
    container_name: mongo
    image: mongo:5.0
    ports:
      - "27017:27017"
    env_file:
      - .env    
    environment:
      - MONGO_INITDB_DATABASE=${MONGO_INITDB_DATABASE}
      - MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD}
    volumes:
      - mongo-database:/data/db
    networks:
      - mongonetwork

networks:
  dadjokes:
  kanye:
  mongonetwork:

volumes:
  mongo-database:

The service is based on the mongo:5.0 image and is run in a container named mongo. The container exposes port 27017 on the host machine, which is mapped to port 27017 in the container. Port 27017 is also the default port MongoDB runs on. This allows you to access the MongoDB database running in the container from the host machine.

The env_file and environment sections specify environment variables that will be passed to the container when it is started. The env_file section specifies a file called .env that contains environment variables, and the environment section specifies additional environment variables that will be passed to the container. These environment variables are used to configure the MongoDB database, such as the name of the database and the username and password of the root user.

The volumes section specifies a named volume called mongo-database, which is mapped to the /data/db directory in the container. This allows you to persist the data stored in the MongoDB database even if the container is stopped or removed.

Finally, the networks section specifies that the mongo container should be connected to a network called mongonetwork. This network allows the mongo container to communicate with other containers connected to the same network.

Create the .env file

Project_Folder
    └─── microservices
        └─── DadJokes
        └─── Kanye
       | docker-compose.yml 
       | .env # <----- add the .env

In the .env, add the environmental variables.

MONGO_INITDB_DATABASE=admin
MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD=example
MONGODB_CONNSTRING=mongodb://root:example@mongo:27017/test?directConnection=true&authSource=admin&replicaSet=replicaset&retryWrites=true

My Jokes Microservice

This microservice implements two endpoints in NodeJS. One endpoint will fetch data from the MongoDB service, while the other will feed data to the database.

The file structure for the microservice is as follows:

   microservices    
        └─── MyJokes   
            └─── Config
                | db.js
            └─── Controller
                | quoteController.js
            └─── Models
                | quoteModel.js
            └─── Routes
                | quoteRoute.js
            | .dockerignore
            | .gitignore
            | Dockerfile 
            | server.js

Initialize npm and install dependencies using npm. This will create package.json and package-lock.json files.

npm init -y && npm install colors && cors && express && mongoose

The db.js contains the connection to the mongo service. The connection string will be passed as an environment variable in docker-compose.yml.

const mongoose = require('mongoose')
const colors = require('colors')

const connectDB = async () => {
  try {

    const conn = await mongoose.connect(process.env.MONGODB_CONNSTRING)

    console.log(`MongoDB Connected: ${conn.connection.host}`.cyan.underline)
  } catch (error) {
    console.error(`Error: ${error.message}`.red.underline.bold)
    process.exit(1)
  }
}

module.exports = {connectDB}

The quoteModel.js contains the Mongoose schema of the data being saved to Mongo. The data will be saved to the Quote collection.

const mongoose = require('mongoose')

const quotesSchema = new mongoose.Schema({
    quote: {
        type: String,
        required: true
    },
},
    {
        timestamps: true,
    }
)

module.exports = mongoose.model('Quote', quotesSchema)

The quoteController.js contains business logic for interfacing with the database and performing CRUD operations.

const Quote = require('../Models/quoteModel')


// @desc    Fetch random quote
// @route   GET /api/quote
// @access  Public
const randomQuote = async (req, res) => {
    try {

        Quote.count().exec(function (err, count) {
            // Get a random entry
            const random = Math.floor(Math.random() * count)
            // Again query all users but only fetch one offset by our random #
            Quote.findOne().skip(random).exec(
                function (err, result) {
                    // Tada! random user
                    return res.json(result)
                })
        })

    } catch (error) {
        console.error(error)
        res.status(500).json({ message: 'Server Error' })
    }
}

const postQuote = async (req, res) => {
    try {

        console.log(req.body)

        const { quote } = req.body

        const newQuote = new Quote({
            quote
        })
        const createdQuote = await newQuote.save()
        res.json(createdQuote)
    } catch (error) {
        console.error(error)
        res.status(500).json({ message: 'Server Error' })
    }
}

module.exports = { randomQuote, postQuote }

The quoteRoute.js defines the API routes for the two controllers. Fetching a new quote is done using get while saving to the DB uses post.

const express = require('express')

const router = express.Router()

const { randomQuote,postQuote } = require('../Controller/quoteController.js')

router.route('/').get(randomQuote)

router.route('/new').post(postQuote)

module.exports = router

The server.js will start the express server on port 3030 after connecting to the DB and serving the API on localhost:3030

const  express = require('express')
const cor = require('cors')
const  {connectDB} = require('./Config/db.js')
const quoteRoute = require('./Routes/quoteRoute.js')
// Constants
const PORT = 3030;
// App
const app = express();
app.use(cor())
app.use(express.json());
app.use('/api/quote', quoteRoute)
const start = async () => {
  try {
    await connectDB()
    app.listen(PORT,  () => {
      console.log(`Running on ${PORT}`);
    });
  } catch (error) {
    console.log(error);
  }
}

start()

The .dockerignore has:

node_modules
Dockerfile

The Dockerfile Has:

FROM node:16

# Create app directory
WORKDIR /app

# Install app dependencies
COPY package.json .
RUN npm install
# If you are building your code for production
RUN npm ci --only=production

# Bundle app source
COPY . .
EXPOSE 3030

CMD [ "node", "server.js" ]

The Dockerfile begins by specifying the base image to use as the starting point for the image being built. The base image is node:16, which is a version of the Node.js runtime.

Next, the WORKDIR instruction sets the working directory for subsequent instructions. Any files or directories added to the image will be placed in the /app directory.

The COPY instruction copies the package.json file from the source directory (the directory containing the Dockerfile) to the /app directory in the image.

The RUN instruction then runs the npm install command, which installs the dependencies listed in the package.json file. The npm ci command is similar to npm install, but is intended to be used in automated environments like continuous integration and deployment, where the package-lock.json file is also checked into version control.

The COPY instruction then copies all files and directories in the source directory (including the package.json file and the dependencies that were just installed) to the /app directory in the image.

The EXPOSE instruction indicates that the container listens on the specified port at runtime. The container will listen on port 3030.

Finally, the CMD instruction specifies the command to run when the container is started. The command is node server.js, which runs the Node.js server.

Add the service to docker-compose.yml

version: '3.8'

services:

  dadjokes:
    image: dadjokes
    container_name: dadjokes
    build:
      context: ./DadJokes
    networks:
      - dadjokes
    ports:
      - "7000:7000"

  kanye:
    image: kanye
    container_name: kanye
    build:
      context: ./Kanye
    networks:
      - kanye
    ports:
      - "8080:8080"

  mongo:
    container_name: mongo
    image: mongo:5.0
    ports:
      - "27017:27017"
    env_file:
      - .env    
    environment:
      - MONGO_INITDB_DATABASE=${MONGO_INITDB_DATABASE}
      - MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD}
    volumes:
      - mongo-database:/data/db
    networks:
      - mongonetwork

  myjokes:
    container_name: myjokes
    image: myjokes
    build:
      context: ./MyJokes
    ports:
      - "3030:3030"
    networks:
      - myjokes
      - mongonetwork
    links:
      - mongo
    depends_on:
      - mongo
    env_file:
      - .env
    environment:
      - MONGODB_CONNSTRING=${MONGODB_CONNSTRING}

networks:
  dadjokes:
  kanye:
  myjokes:
  mongonetwork:

volumes:
  mongo-database:

The myjokes container uses an image named myjokes, which is built from the context in the ./MyJokes directory. The ports configuration maps the container's port 3030 to the host machine's port 3030.

The networks configuration specifies the networks that the container should be connected to. The container is connected to two networks: myjokes and mongonetwork. We connect to the mongonetwork to access the database.

The links configuration specifies any containers that the container depends on. The container depends on the mongo container.

The depends_on configuration specifies any containers that must be started before the container can be started. The container is only spin up after the mongo container is up.

The env_file configuration specifies a file containing environment variables that should be passed to the container. .env in this case.

The environment configuration specifies additional environment variables that should be passed to the container. The MONGODB_CONNSTRING environment variable is being set to the value of the MONGODB_CONNSTRING variable in the .env file.

React Microservice

The front end will be built using ReactJS. The front end will contain components that interact with all our backend microservices. The dashboard will have four components:

• Kanye-as-a-service component will display data from the Kanye quotes backend microservice.

• Dadjokes-as-a-service will display data from the dadjokes quotes backend microservice.

• Custom quotes will be what we create and store on the MongoDB microservice.

• An input component for adding custom quotes to the MongoDB microservice.

On the project folder, create a react app with npm

npx create-react-app frontend

The above command will create a frontend directory. Your current directory structure should now be as follows:

microservices
    └───Dadjokes
    └───frontend # <--- new react folder 
    └─── Kanye
    └───MyJokes
    | docker-compose.yml

Switch to the frontend directory

cd frontend

and install Tailwind CSS

npm install -D tailwindcss

Initialize Tailwind CSS

npx tailwindcss init

In the tailwind.config.js file created by running the above command, point the content to all subfolders within src folder.

// frontend/tailwind.config.js
/** @type {import('tailwindcss').Config} */
module.exports = {
  content: ["./src/**/*.js"],
  theme: {
    extend: {},
  },
  plugins: [],
}

Add tailwind to src/App.css .

/* frontend/src/App.css */
@tailwind base;
@tailwind components;
@tailwind utilities;

Add a script in package.json that compiles Tailwind to public/style.css

// frontend/package.json
...
  "scripts": {
    "start": "react-scripts start",
    "css": "npx tailwindcss -i ./src/App.css -o ./public/syle.css --watch",
    "dev": "npm run start && npm run css",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
....

Add the compiled CSS to the public/index.html file in the public directory.

<!--frontend/public/index.html  -->
<!DOCTYPE html>
<html lang="en">
<head>
...

<link rel="stylesheet" href="style.css"/>

...
</head>

Create the following files and folders in the src folder

frontend
   └─── src
        └─── Components # <-----folder
             | Dadjokes.js
             | Kanye.js
             | MyJokes.js
             | NewJoke.js
        └─── config # <-----folder
             | config.js

The config/config.js file contains all the URLs to REST APIs from the kanye, dadjokes and myjokes microservices. Its contents are:

export default {
    services: {
      kanye: process.env.REACT_APP_KANYE_SERVICE_URL ,
      dadjokes: process.env.REACT_APP_DADJOKES_SERVICE_URL,
      myjokes: process.env.REACT_APP_MYJOKES_SERVICE_URL

    }
  }

The Components/Dadjokes.js will use the dadjoke endpoint from the config file to fetch a dadjoke from the dadjoke container. Its contents are:

// frontend/Components/Dadjokes.js
import {useEffect,useState} from 'react'
import config from '../config/config'
const Dadjokes = () => {

    const [jokes, setJokes] = useState([])
    const [loading, setLoading] = useState(false)

    const getJokes = async (e = undefined) => {
        if(e) e.preventDefault()
        setLoading(true)
        const res = await fetch(config.services.dadjokes)
        const data = await res.json()
        setJokes(data)
        setLoading(false)
    }

    useEffect( () => {
        getJokes()
    }, [])



  return (
    <div className="flex font-mono">
    <div className="flex-none w-56 relative">
        <img src="images/leo.jpg" alt="" className="absolute inset-0 w-full h-full object-cover rounded-lg" loading="lazy" />
    </div>
    <form className="flex-auto p-6 m-10  ">
        <div className="flex flex-wrap border-green-600">

            <div className="w-full flex-none mt-2 order-1 text-2xl font-bold text-slate-600">
                {loading ? 'Loading...' : jokes}
            </div>
            <div className="text-sm font-medium text-slate-400">
                Dadjokes as a service
            </div>
        </div>

        <div className="flex space-x-4 mb-5 text-sm font-medium py-4">
        <div className="flex-auto flex space-x-4" onClick={(e)=>  getJokes(e)}>
                <button className="h-10 px-6 font-semibold rounded-full bg-violet-600 text-white">
                    New Dadjoke
                </button>

            </div>
            <button className="flex-none flex items-center justify-center w-9 h-9 rounded-full text-green-600 bg-violet-50" type="button" aria-label="Like">
                <svg width="20" height="20" fill="currentColor" aria-hidden="true">
                    <path fill-rule="evenodd" clip-rule="evenodd" d="M3.172 5.172a4 4 0 015.656 0L10 6.343l1.172-1.171a4 4 0 115.656 5.656L10 17.657l-6.828-6.829a4 4 0 010-5.656z" />
                </svg>
            </button>
        </div>
        <p className="text-sm text-slate-500">
            Coming from the Dadjokes microservice
        </p>
    </form>
</div>

        )
}

export default Dadjokes

The Components\Kanye.js will use the Kanye service endpoint from the config file to fetch a Kanye quote from the Kanye container. Its contents are:

// frontend/Components/Kanye.js
import {useEffect,useState} from 'react'
import config from '../config/config'

const Kanye = () => {
    const [jokes, setJokes] = useState([])
    const [loading, setLoading] = useState(false)

    const getJokes = async (e = undefined) => {
        if(e) e.preventDefault()
        setLoading(true)
        const res = await fetch(config.services.kanye)
        const data = await res.json()
        setJokes(data)
        setLoading(false)
    }


    useEffect( () => {
        getJokes()
    }, [])


    return (

            <div className="flex font-mono">
                <div className="flex-none w-56 relative">
                    <img src="images/kanye.jpg" alt="" className="absolute inset-0 w-full h-full object-cover rounded-lg" loading="lazy" />
                </div>
                <form className="flex-auto p-6 m-10  ">
                    <div className="flex flex-wrap border-green-600">

                        <div className="w-full flex-none mt-2 order-1 text-2xl font-bold text-slate-600">
                            {loading ? 'Loading...' : jokes}
                        </div>
                        <div className="text-sm font-medium text-slate-400">
                            Kanye as a service
                        </div>
                    </div>

                    <div className="flex space-x-4 mb-5 text-sm font-medium py-4">
                    <div className="flex-auto flex space-x-4" onClick={(e)=>  getJokes(e)}>
                            <button className="h-10 px-6 font-semibold rounded-full bg-violet-600 text-white" >
                                New Kanye
                            </button>

                        </div>
                        <button className="flex-none flex items-center justify-center w-9 h-9 rounded-full text-green-600 bg-violet-50" type="button" aria-label="Like">
                            <svg width="20" height="20" fill="currentColor" aria-hidden="true">
                                <path fill-rule="evenodd" clip-rule="evenodd" d="M3.172 5.172a4 4 0 015.656 0L10 6.343l1.172-1.171a4 4 0 115.656 5.656L10 17.657l-6.828-6.829a4 4 0 010-5.656z" />
                            </svg>
                        </button>
                    </div>
                    <p className="text-sm text-slate-500">
                        Coming from the Kanye microservice
                    </p>
                </form>
            </div>


    )
}

export default Kanye

The Components\MyJokes.js will use the myjokes service endpoint from the config file to fetch a joke from the MongoDB container. Its contents are:

// frontend/Components/MyJokes.js
import { useState ,useEffect} from 'react'
import config from '../config/config'
const MyJokes = () => {

    const [jokes, setJokes] = useState([])
    const [loading, setLoading] = useState(false)

    const getJokes = async (e = undefined) => {
        if(e) e.preventDefault()

        setLoading(true)
        const res = await fetch(config.services.myjokes)
        const data = await res.json()
        setJokes(data['quote'])
        setLoading(false)
    }

    useEffect( () => {
        getJokes()
    }, [])



    return (
        <div className="flex font-mono">
            <div className="flex-none w-56 relative">
                <img src="images/joker.jpg" alt="" className="absolute inset-0 w-full h-full object-cover rounded-lg" loading="lazy" />
            </div>
            <form className="flex-auto p-6 m-10  ">
                <div className="flex flex-wrap border-green-600">

                    <div className="w-full flex-none mt-2 order-1 text-2xl font-bold text-slate-600">
                       {loading ? 'Loading...' : jokes}

                    </div>
                    <div className="text-sm font-medium text-slate-400">
                        Jokes as a service
                    </div>
                </div>

                <div className="flex space-x-4 mb-5 text-sm font-medium py-4">
                    <div className="flex-auto flex space-x-4" onClick={(e)=>  getJokes(e)}>
                        <button className="h-10 px-6 font-semibold rounded-full bg-violet-600 text-white" >
                           Load New Joke 
                        </button>

                    </div>
                    <button className="flex-none flex items-center justify-center w-9 h-9 rounded-full text-green-600 bg-violet-50" type="button" aria-label="Like">
                        <svg width="20" height="20" fill="currentColor" aria-hidden="true">
                            <path fill-rule="evenodd" clip-rule="evenodd" d="M3.172 5.172a4 4 0 015.656 0L10 6.343l1.172-1.171a4 4 0 115.656 5.656L10 17.657l-6.828-6.829a4 4 0 010-5.656z" />
                        </svg>
                    </button>
                </div>
                <p className="text-sm text-slate-500">
                    Coming from the MyJoke microservice
                </p>
            </form>
        </div>

    )
}

export default MyJokes.js

The Components/NewJoke.js component has a form that submits jokes to the MongoDB service. Its contents are:

// frontend/Components/NewJokes.js
import { useState } from 'react'
import config from '../config/config'

const NewJoke = () => {
    const [joke, setJoke] = useState('')
    const [showModal, setShowModal] = useState(false)


    const handleSubmit = async (e) => {
        e.preventDefault()
        let formData = new FormData();
        formData.append('quote', joke);
        await fetch(`${config.services.myjokes}/new`,
            {
                method: "post",
                headers: {
                    'Accept': 'application/json',
                    'Content-Type': 'application/json'
                  },
                body: JSON.stringify({
                    quote: joke
                }),
            });
        setJoke('')
        setShowModal(false)

    }
    return (
        <div>



            <div className="flex" onClick={(e) => {
                e.preventDefault()
                setShowModal(!showModal)
            }}>
                <a href="/new" className="hover:border-blue-500 hover:border-solid hover:bg-white hover:text-blue-500 group w-full flex flex-col items-center justify-center rounded-md border-2 border-dashed border-slate-300 text-sm leading-6 text-slate-900 font-medium py-3">
                    <svg className="group-hover:text-blue-500 mb-1 text-slate-400" width="20" height="20" fill="currentColor" aria-hidden="true">
                        <path d="M10 5a1 1 0 0 1 1 1v3h3a1 1 0 1 1 0 2h-3v3a1 1 0 1 1-2 0v-3H6a1 1 0 1 1 0-2h3V6a1 1 0 0 1 1-1Z" />
                    </svg>
                    <h1 className='text-2xl'>New Joke</h1>
                </a>
            </div>
            {showModal &&
                <div className='w-100 flex justify-center py-10  w-full' >
                    <form onSubmit={handleSubmit}>
                        <h1 className='text-2xl'>Create New Joke</h1>
                        <div>

                            <label className="block">

                                <textarea rows="5" cols="50" className="block text-3xl text-slate-500
                                 border border-slate-500 rounded-md shadow-sm focus:border-blue-500 focus:ring-blue-500
                                 "
                                    value={joke}
                                    onChange={(e) => setJoke(e.target.value)}
                                >

                                </textarea>
                            </label>
                        </div>
                        <div>

                            <button className="h-10 px-6 font-semibold rounded-full bg-violet-600 text-white"
                                type='submit'
                            >
                                Save
                            </button>
                        </div>

                    </form>
                </div >

            }

        </div>
    )
}
export default NewJoke

All the components are rendered in frontend/App.js

// frontend/App.js
import Kanye from "./Components/Kanye";
import Dadjokes from "./Components/Dadjokes";
import MyJokes from "./Components/MyJokes";
import NewJoke from "./Components/NewJoke";
function App() {

  return (
    <div className="flex justify-center">

        <div className="flex justify-center flex-col space-y-4 px-10 w-1/2 py-10">
            <NewJoke />
            <MyJokes />
            <Dadjokes />
            <Kanye />
        </div>
    </div>
  );
}

export default App;

Create the following files in the frontend folder

  └─── frontend
            └─── nginx # <-- create this
                | nginx.conf # <-- create this
            └─── public
            └─── src
            | .dockerignore # <-- create this
            | .gitignore 
            | Dockerfile # <-- create this
            | package-lock.json
            | package.json
            | tailwind.config.js

Since we are creating a docker-compose for use in production, we will need to serve React with a production-grade server. We do this using nginx. We will create a multi-stage Dockerfile that will allow us to build the react app and then serve it using an nginx image. In the nginx, we will also set up a reverse proxy. A reverse proxy is a server that sits between client computers and backend servers and forwards incoming requests to the appropriate backend server.

The .dockerignore file has the following:

Dockerfile
node_modules

The Dockerfile has the following:

FROM node:16 as build

ARG KANYE_SERVICE_URL
ARG DADJOKES_SERVICE_URL
ARG MYJOKES_SERVICE_URL
ARG REACT_APP_HOST_IP_ADDRESS

ENV REACT_APP_HOST_IP_ADDRESS $REACT_APP_HOST_IP_ADDRESS
ENV REACT_APP_KANYE_SERVICE_URL $KANYE_SERVICE_URL
ENV REACT_APP_DADJOKES_SERVICE_URL $DADJOKES_SERVICE_URL
ENV REACT_APP_MYJOKES_SERVICE_URL $MYJOKES_SERVICE_URL

WORKDIR /app

COPY ./package.json /app/package.json
COPY ./package-lock.json /app/package-lock.json

RUN npm install

COPY . .

RUN npm run css
RUN npm run build

FROM nginx

COPY ./nginx/nginx.conf /etc/nginx/conf.d/default.conf

COPY --from=build /app/build /usr/share/nginx/html

This multi-stage Dockerfile builds the React application in the first stage. The first stage uses the node:16 image as the base image and installs the application's dependencies. It then copies the application code into the image and runs the css and build scripts to build the application. The css compiles Tailwind.

The second stage uses the nginx image as the base image and copies a custom Nginx configuration file into the image. It also copies the build artifacts from the first stage into the image.

The nginx.conf has the following:

server { 
 listen 80;
 server_name frontend;
 location / {
   # This would be the directory where your React app's static files are stored at
   root /usr/share/nginx/html;
   try_files $uri /index.html;
 }

 location /dadjoke {
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-NginX-Proxy true;
   proxy_pass http://dadjokes:7000/dadjoke;
   proxy_ssl_session_reuse off;
   proxy_set_header Host $http_host;
   proxy_cache_bypass $http_upgrade;
   proxy_redirect off;
 }


 location /kanye {
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-NginX-Proxy true;
   proxy_pass http://kanye:8080/kanye;
   proxy_ssl_session_reuse off;
   proxy_set_header Host $http_host;
   proxy_cache_bypass $http_upgrade;
   proxy_redirect off;
 }


 location /myjoke {
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-NginX-Proxy true;
   proxy_pass http://myjokes:3030/api/quote;
   proxy_ssl_session_reuse off;
   proxy_set_header Host $http_host;
   proxy_cache_bypass $http_upgrade;
   proxy_redirect off;
 }



}

All the configuration blocks for the Nginx web server that sets up a reverse proxy for the /dadjoke , /myjoke , /kanye / locations.

The proxy_set_header directives set various request headers that are forwarded to the backend server. The X-Real-IP and X-Forwarded-For headers contain the client's IP address and the X-NginX-Proxy header is set to true to indicate that the request is being proxied.

The proxy_pass directive specifies the URL of the backend server to which the request should be forwarded.

The proxy_ssl_session_reuse directive is set to off to disable SSL session reuse, which can improve security but may also reduce performance.

The proxy_set_header directive sets the Host header to the value of the $http_host variable, which contains the hostname of the original request.

The proxy_cache_bypass directive is set to the value of the $http_upgrade variable, which allows the reverse proxy to pass through WebSocket and HTTP/2 requests.

The proxy_redirect directive is set to off to disable automatic redirection of the client to the URL specified in the Location header of the backend server's response.

Run all the microservices

To run all the microservices using docker-compose run the following command on the microservices folder.

docker-compose up --force-recreate --build -d

docker-compose up is a command used to start and run a Docker Compose application. The --force-recreate flag forces the recreation of any containers that have been previously created, which can be useful if you have made changes to the application and want to ensure that the containers are recreated with the updated configuration.

The --build flag forces a rebuild of the images, even if they are already up to date. This can be useful if you have made changes to the application code or the Dockerfile and want to ensure that the images are rebuilt with the updated code.

The -d flag runs the containers in the background, allowing you to continue using the command prompt while the containers are running.

This command will start and run the Docker Compose application in the background, rebuilding the images if necessary and recreating any containers that have been previously created.

Navigate to localhost, and you should see the live react app.

Provisioning VM with Terraform

Create a terraform folder on the project root and the subsequent files, as shown below.

Project_Folder
    └─── microservices
    └─── terraform # <----- new folder
         | linode.tf
         | terraform.tfvars
         | variable.tf

Navigate to Linode dashboard and generate an API key.

Paste the key in the terraform.tfvars file. Also, pass a strong password that terraform will use as the default password on the VMs.

# variables.tf
linode-token="paste-token-here"
root_pass="a-very-strong-password"

Define variables in the variables.tf file. The variables are read from the variables.tfvars file.

# variables.tfvars
variable "linode-token" {}
variable "root_pass" { 
}
variable "region" {
  default = "us-southeast"
}

The linode.tf file is where all the magic happens.

# Linode Provider definition
terraform {
  required_providers {
    linode = {
      source  = "linode/linode"
      version = "1.27.1"
    }
  }
}

# Configure the Linode provider
provider "linode" {
  token = var.linode-token
}
# Create a Linode instance
resource "linode_instance" "example" {
  image      = "linode/ubuntu20.04"
  type       = "g6-nanode-1"
  region     = "us-east"
  label      = "Microservice-blog"
  private_ip = true
  root_pass  = var.root_pass
}
output "ip_address" {
  value = linode_instance.example.ip_address
}

The code defines a Terraform configuration that uses the Linode provider to create a Linode instance running Ubuntu 20.04. The Linode provider is required to be version 1.27.1 and is configured with a token that is stored in a variable called linode-token.

The configuration then creates a Linode instance resource, specifying the image to use (linode/ubuntu20.04), the type of instance to create (g6-nanode-1), the region where the instance will be located (us-east), and a label for the instance (Microservice-blog). The instance is also configured to have a private IP address and a root password that is stored in the root_pass variable.

Finally, the configuration defines an output called ip_address that returns the public IP address of the created Linode instance.

To avoid retyping or forgetting the commands, we will build a bash script that will call terraform and perform other operations.

Create a Scripts folder in the project root folder.

Project_Folder
    └─── microservices
    └─── terraform 
    └─── Scripts # <----- new folder
        | terraform_setup.sh

In terraform_setup.sh paste the code below.

#!/bin/bash
# create the directory name of the ssh keys
# $HOME is a bash variable that points to the home directory of the current user
echo "-----------------------Creating ssh keys-----------------------"
mkdir -p $HOME/.ssh/microservicekeys

# create the ssh keys
ssh-keygen -t rsa -b 4096 -C "global microservices" -f $HOME/.ssh/microservicekeys/id_rsa -N "" <<< y
echo "------Adjust permissions of generated key-files locally---"
chmod 0600 "$HOME/.ssh/microservicekeys/id_rsa" "$HOME/.ssh/microservicekeys/id_rsa.pub"

# get to root directory
cd .. 

# get into the terraform directory
echo "-----------------------Starting Terraform Execution-----------------------"
cd terraform

# Terraform destroy
echo "----------------------- Terraform Destroy -----------------------"
terraform destroy -auto-approve -lock=false

# initialize terraform
terraform init

# plan terraform
echo "----------------------- Terraform Plan -----------------------"
terraform plan -lock=false

# apply terraform
echo "----------------------- Terraform Apply -----------------------"
terraform apply -auto-approve -lock=false

# save the ip address of the instance in variable ip
echo "----------------------- Terraform Output -----------------------"
ip=$(terraform output -json | jq -r '.ip_address.value')


# get back to root directory
cd ..

# get into the terraform directory
cd terraform

echo "----------------------- Copying public Key to Instance -----------------------"

# Read the terraform.tfvars file
while read -r line
do
  # Split the line into a key and a value
  key=$(echo "$line" | cut -d'=' -f1)
  value=$(echo "$line" | cut -d'=' -f2)
  # If the key is "password", store the value in a bash variable
  if [ "$key" == "root_pass" ]; then
    # Delete the quotes from the value
    password=$(echo "$value" | tr -d '"')
  fi
done < terraform.tfvars

# get back to root directory
cd ..

# save the ip to a file
echo "----------------------- Saving IP Addresses -----------------------"

# create a file to store the ip address
touch inventory.ini
echo $ip > inventory.ini


while read line; 
do 
    echo "----------------Copying ssh keys to server id -> $line----------------"; 
    sshpass -p $password ssh-copy-id -i $HOME/.ssh/microservicekeys/id_rsa.pub -o PubkeyAuthentication=yes -o PasswordAuthentication=yes -o StrictHostKeyChecking=no root@$line
done <  inventory.ini

echo "----------------------- All done now! -----------------------"
echo "Copy the following private key to the github repository secrets"
cat $HOME/.ssh/microservicekeys/id_rsa

exit

exec bash

The script first creates a directory for the SSH keys and generates a new set of RSA keys using the ssh-keygen command. It then adjusts the permissions of the generated key files and moves to the root directory.

The script then enters the terraform directory and runs terraform init to initialize Terraform, and runs terraform plan to create a plan for creating the new Linode instance. The script then applies this plan using terraform apply.

After the Linode instance is created, the script retrieves its public IP address using terraform output and stores it in a file called inventory.ini. It then reads the terraform.tfvars file to retrieve the root password for the Linode instance and uses this password to copy the public SSH key to the instance using sshpass.

Finally, the script prints the private key to the terminal and exits.

Run the script while in the Scripts folder using:

./terraform_setup.sh

On Linode, you should see the instance created.

Automating deploys with Ansible

At this point, we can use the local machine (your laptop) as an ansible controller to ssh into the VM, install dependencies, and run docker-compose.

In the project folder create a playbook.yml , ansible.cfg and a run_ansible.sh file in the Scripts folder.

Project_Folder
    └─── microservices
    └─── terraform 
    └─── Scripts 
        | terraform_setup.sh
        | run_ansible.sh # <---create this
    | inventory.ini
    | ansible.cfg # <---create this
    | playbook.yml # <---create this

The ansible.cfg has the following:

[ssh_connection]
pipelining=True

This is a configuration block for the ssh_connection plugin in Ansible. The pipelining option is set to True, which means that Ansible will use an optimized command execution flow that allows multiple commands to be sent to the remote host without waiting for the previous command to finish. This can improve the performance of Ansible by reducing the number of network round-trips required to execute a playbook.

The playbook.yml file has the following

---
- name: Connect to linode and install dependencies and run docker-compose
  hosts: all
  become: true
  vars:
    source: "{{ source }}"

  tasks:
    - name: Install required system packages
      apt:
        pkg:
          - apt-transport-https
          - ca-certificates
          - curl
          - software-properties-common
          - python3-pip
          - virtualenv
          - python3-setuptools

        state: latest
        update_cache: true

    - name: Install Docker
      become: true
      apt:
        name: docker.io
        state: present
    - name: Install Docker Compose
      pip:
        name: docker-compose
      tags:
        - docker

    - name: Install Python
      apt:
        name: python3
        state: present
      tags:
        - python

    - name: Synchronize src and dest, excluding node_modules subfolders
      ansible.builtin.synchronize:
        src: "{{ source}}"
        dest: /home
        rsync_opts:
          - "--exclude=**/node_modules"
          - "--exclude=**/data"

    - name: Copy env
      copy:
            src:   "{{source}}/.env"
            dest: /home/microservices/.env

    - name: Run Docker Compose Build and run image
      command: docker-compose -f /home/microservices/docker-compose.yml up --force-recreate --build -d
      tags:
        - docker

This Ansible playbook connects to the Linode instance and installs some required system packages, Docker, Docker Compose, and Python. It then synchronizes the contents of the microservices directory with a destination on the Linode instance set to /home, excluding the node_modules subfolders. The playbook copies the.env from the microservices/.env to the /home/microservices/.env on the Linode instance. Finally, the playbook runs docker-compose to build and run our Docker images using the docker-compose.yml file.

The playbook is intended to be run on all hosts in the inventory.ini file and uses become to run tasks with root privileges. The source variable is a playbook variable used to specify the local directory that should be synchronized with the destination on the Linode instance.

The run_ansible.sh bash script has the following

#!/bin/bash
# this runs ansible on your laptop
# cd to root
cd ..
#  create environment variables from .env in the microservices folder file
cd microservices

# Read the .env file and export each key-value pair as an environment variable
while read line; do
  export $line
done < .env

# cd to root and run ansible
cd ..

ansible-playbook playbook.yml -u root --private-key=$HOME/.ssh/microservicekeys/id_rsa -i inventory.ini -vvv --extra-vars "source=microservices"

echo "----------------------- All done now! -----------------------"

exit
exec bash

The script first changes the current working directory to the root directory and then enters the microservices directory. It reads the .env file and exports each key-value pair as an environment variable.

The script then changes the current working directory to the root directory and runs the ansible-playbook command to execute the playbook. The playbook is run with verbose output enabled, and the source variable is passed as an extra variable. The source variable is specified as microservices The ansible-playbook command is run with the -u flag to specify the user to connect as (root in this case) and the --private-key flag to specify the private key for authentication. The -i flag is used to specify the inventory file to use (inventory.ini), and the -vvv flag enables verbose output.

Finally, the script prints a message to the terminal and exits.

Run the script while in the Scripts folder using

./run_ansible.sh

You should see the live react app if you navigate to your IP address.

Using Github Actions as the Controller Machine

Normally, you would push code to a VCS, and we want to leverage Github actions to automatically update code on the server on push. For this, we need to create a Github actions workflow.

Project_Folder
    └─── microservices
    └─── terraform 
    └─── Scripts 
    └─── .github # <---create this
         └─── workflows # <---create this
             | actions.yml # <---create this
    | inventory.ini
    | ansible.cfg 
    | playbook.yml

The actions.yml has the following:

name: Building microservices with Terraform, Ansible, docker, docker-compose, and Github Actions
on:
  push:
      branches:
        - "main"
jobs:
  provision:
    name: Run Ansible
    runs-on: ubuntu-latest
    steps:      
    - name: Checkout repository
      uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v2
    - name: Install dependencies
      run: |
        python -m pip install ansible
    - name: Add hosts to known_hosts
      env:
        SSH_AUTH_SOCK: /tmp/ssh_agent.sock

      run: |
        mkdir -p ~/.ssh
        touch ~/.ssh/known_hosts
        ssh-keyscan -f inventory.ini > known_hosts
        cat known_hosts >> ~/.ssh/known_hosts
        ssh-agent -a $SSH_AUTH_SOCK > /dev/null
        ssh-add - <<< "${{ secrets.SSH_PRIVATE_KEY }}"


    - name: Run ansible playbook
      env:
          SSH_AUTH_SOCK: /tmp/ssh_agent.sock              
      run: |
           cd microservices 
           echo MONGODB_CONNSTRING="${{secrets.MONGODB_CONNSTRING}}" >> .env
           echo MONGO_INITDB_DATABASE="${{secrets.MONGO_INITDB_DATABASE}}" >> .env
           echo MONGO_INITDB_ROOT_PASSWORD="${{secrets.MONGO_INITDB_ROOT_PASSWORD}}" >> .env
           echo MONGO_INITDB_ROOT_USERNAME="${{secrets.MONGO_INITDB_ROOT_USERNAME}}" >> .env
           cd .. 
           ansible-playbook playbook.yml -u root --private-key="${{ secrets.SSH_PRIVATE_KEY }}" -i inventory.ini -vvv  --extra-vars "source=microservices"

The workflow is triggered by a push to the main branch and runs on an Ubuntu machine.

The workflow consists of several steps that perform various tasks. The first step checks out the repository, and the second step sets up Python. The third step installs the dependencies required to run ansible, including the ansible package. The fourth step adds the hosts specified in the inventory.ini file to the known_hosts file, which is used to store the public keys of known hosts. This step also sets up an SSH agent to manage SSH keys and adds the private key stored in the SSH_PRIVATE_KEY secret to the agent.

The final step runs the ansible playbook, passing the private key stored in the SSH_PRIVATE_KEY secret as an argument to the --private-key flag. This step also reads the secrets stored in the repository and appends them to the .env file in the microservices directory. Finally, the ansible-playbook command is run to execute the playbook.

Add secrets in your .env to github secrets. Go to the setting tab on the Github repo

The SSH_PRIVATE_KEY is the private key generated by the terraform_setup.sh script. You can view it by running the following command:

cat $HOME/.ssh/microservicekeys/id_rsa

Once you push changes to github, you should have the code auto-deploy on the server

Resources:

Code used in this blog

https://www.terraform.io/use-cases/infrastructure-as-code?product_intent=terraform

https://docs.github.com/en/actions

For example:

Introduction

Npm registry is the largest Javascript Software library. Developers publish packages on the registry as tools. For example, if you are a react developer and need to code an image upload component, instead of coding the component from scratch, you can use a published component in the registry.

There are several advantages of open source:

1. A package that many people have used has been battle tested on most environments. For example, the component's contributors have ensured the image component works on most browsers.

2. The lead time to production reduces significantly. The time that would have been spent implementing and testing the component is allocated to other aspects of the software development process.

3. There are multiple solutions for the same problem. If one package doesnt meet the requirements, you can always try out others.

4. The code is open source; hence, you can always use the package and build on it to meet your requirements.

5. If you publish your package, as more people use it, you can improve the project from feedback from the community.

6. The community can contribute to the codebase by reporting bugs, solving bugs, and adding new features. Through this, critical vulnerabilities are easy to identify as more use the project.

7. More recently, companies have been supporting open-source contributors. Sponsorships make it possible to become an open-source contributor and earn from it.

Prerequisites

• Npm account. Create one from the npm website

• NodeJS and npm installed on your computer. Follow NodeJS installation instructions from the official documentation for your operating system. This guide uses the following versions

node v18.12.1
npm 8.19.2

Sheng to shembeteng

The current project folder looks as follows:

Project_Folder
  |__ index.js

The sheng to shembeteng logic relies on a regular expression. Going over the pseudo-code:

• Given the sentence :

 Luku yako. # Your drip, your style

• Split the sentence into an array of words

['Luku','yako']

• Loop over the words. On each word, run a regular expression that splits the word into an array with two parts. The first part holds the constants before the first vowel, while the second part holds the rest of the word from the first vowel.

Luku ---> ['L','uku']

Yako ---> ['Y','ako']

• On the second part of the word, extract the vowel and match the corresponding sheng injection from the following dictionary.

{
    'a':'mbata',
    'e':'mbete',
    'i':'mbiti',
    'o':'mboto',
    'u':'mbutu',
}

The full code for this implementation:

const shembetengInjections = {
    'a':'mbata',
    'e':'mbete',
    'i':'mbiti',
    'o':'mboto',
    'u':'mbutu',
}


export const shembeteng = (sentence) => {
    // split sentence into array of words
    const words = sentence.split(' ');

    // loop through each word
    const shembetengedSentence = words.map(word => {
        let shembetengPart = '';
        // split each word into array of letters
        const wordSplitOnFirstVowel = word.split(/([aeiou].*)/)

        if(wordSplitOnFirstVowel.length > 1){
            // the word has a vowel

            // get the specific vowel
            const vowel = wordSplitOnFirstVowel[1][0];

            // get the shembeteng injection
            const injection = shembetengInjections[vowel];
            // get the rest of the word
            const restOfWord = wordSplitOnFirstVowel[1].slice(1);
            // return the word with the injection
            shembetengPart = vowel + injection + restOfWord;
            return  wordSplitOnFirstVowel[0] + shembetengPart;
        }

        // the word has no vowel
        return word;

    })

    return shembetengedSentence.join(' ')
}

The injections do not always happen on the first vowel occurrence. This would make a good case for neural machine translation if we had a dataset to train a model. Since translation is not the goal of this article, we will leave the simple implementation as is.

Publishing to npm

On your terminal, initialize npm using npm init and follow the prompts. Make sure the entry point is the same name as the file containing the logic. In this case index.js

npm init

This will create a package.json file. Add the "type": "module" to the package.json above the scripts. Adding the "type": "module" means our package user will need to use the import syntax.

The final package.json is as follows:

{
  "name": "shembeteng",
  "version": "1.0.0",
  "description": "Sheng to shembeteng",
  "main": "index.js",
  "type": "module",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/wamaithaNyamu/shembeteng.git"
  },
  "keywords": [
    "Sheng",
    "shembeteng"
  ],
  "author": "Wamaitha Nyamu",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/wamaithaNyamu/shembeteng/issues"
  },
  "homepage": "https://github.com/wamaithaNyamu/shembeteng#readme"
}

On your terminal, login to npm using npm adduser and follow the prompts.

npm adduser

Publish the package using npm publish

npm publish

The package is now live and ready to be used by other developers.

Testing

To test the package. Create a new folder outside the project folder. In the new folder, run npm install shembeteng

 npm install shembeteng

This installs shemebteng from the registry and creates a package.json. Be sure to add the "type":"module" in the test folder.

{
  "type": "module",
  "dependencies": {
    "shembeteng": "^1.0.0"
  }
}

Create a file to test out the project. In the file, import shemebteng and test it out.

import { shembeteng } from "shembeteng";

const sentence = "Luku Yako";

const shemebetenged = shembeteng(sentence);

console.log(shemebetenged);

Updating the npm package

Our package is live, but we need to add a README on how to use the package and push the changes to npm. This would be the same process you would follow if you needed to update the code or release a new version. After changes have been made, change the version number on the package.json to 1.0.1. Then run npm publish to push the new changes to npm.

npm publish

Resources:

Code used for the article

Published npm package

https://docs.npmjs.com/about-npm

Login as root

To log in as root, retrieve the root password from Vultr on the specific instance.

On your terminal, log in as root using.

ssh root@ip_address

To demonstrate how permissions work on Linux, we create a testfolder in the new user's directory while logged in as root, then change permissions for the user in that folder.

Test folder

Navigate to the new user home directory

cd /home/wamaitha

Create the testfolder

mkdir testfolder

Navigate to the testfolder, create a text file hello.txt, and populate the txt file with "Hello from root".

# cd into folder
cd testfolder

# create file and populate with echo
echo "Hello from root!" > hello.txt

# view file content of the hello.txt using cat
cat hello.txt

Check folder permissions as shown below

ls -l

The permissions are sectioned into 4; file type, user permissions, group permissions, and other user permissions. The permissions are a combination of read, write and execute. No permissions are denoted by a dash(-),

[file type][user][group][other users]

Root's permissions on the testfolder are

drwxr-xr-x

• d - directory
• rwx - root has read , write and execute permissions
• r-x - the group associated with root has read and execute permissions only
• r-x - other users on the directory have read and execute permission only.

We want to add the user wamaitha to this directory and give them read and write permissions. By default, a new user is created and added to a group of the same name. Check this using the group's command

groups wamaitha

Permissions

Permissions on Linux are given using the chmod command followed by the permission number.

Permission numbers are:

• 0 = --- (no permissions at all)

• 1 = --x (execute permission only)

• 2 = -w- (write permission only)

• 3 = -wx (write and execute permission only)

• 4 = r- (read permission only)

• 5 = r-x (read and execute only)

• 6 = rw- (read and write permission only)

• 7 = rwx (read,write and execute permissions)

With this, we can combine these numbers to change permissions. Chmod accepts three numbers. The first number represents the user's permission. The second represents the group permissions, and the third represents permission for all other users.

chmod 744  testfolder

For succinctness, the command above will yield:

• read, write and execute permissions to the current user logged in (root) on the testfolder.
• read permissions to the group (all groups that are added to the folder).
• read permissions to all other users accessing the testfolder (other users like wamaitha).

Execute the command and check the permissions again.

Switch to the new user. If prompted for a password, use the password you used when creating the new user. The new user only has read permissions on the test folder. Thus any execution commands will be denied due to insufficient permissions.

su wamaitha

Resources

Classic SysAdmin: Understanding Linux File Permissions

Login to the Vultr instance or whichever ssh server you are using. This guide assumes the ssh server is an Ubuntu 22.04 LTS x64 instance.

Install tmux

tmux is an open-source terminal multiplexer for Unix-based systems. tmux is used for simultaneously running multiple terminal sessions. For example, you may need one terminal running the frontend server and another terminal running a backend server on a full-stack application.

Install tmux using:

sudo apt-get install tmux

Start a tmux session using the tmux command.

tmux

You can exit the tmux session using the exit command.

exit

To get back to the tmux session, list all available tmux sessions using tmux ls.

tmux ls

Re-launch the session using

tmux attach-session -t 0

Create a new user

Create a new user using the adduser command followed by the name you want to assign to the new user.

adduser wamaitha

Create a password for the user and fill in the rest of the details on the prompt.

Verify the user was created using awk. Awk is a programming language used on bash as a scripting tool for text processing. Awk can be used for pattern matching. Users are stored on the /etc/passwd file. The command below returns the names of the users on the system as the /etc/passwd file is quite verbose.

awk -F':' '{ print $1}' /etc/passwd

Move SSH keys

Create a .ssh directory on the new user's home directory

mkdir /home/wamaitha/.ssh

Move the ssh keys to the new folder

mv /root/.ssh/authorized_keys /home/wamaitha/.ssh

Change ownership of the .ssh directory from root to the new user.

chown -R wamaitha:wamaitha /home/wamaitha/.ssh

By default, new users are added to a group of the same name. Log out from tmux using exit, then log out from Vultr using exit.

Login as the new user using ssh with the username as follows

Use the same passphrase used while generating the ssh keys.

This guide uses Ubuntu 22.04.1 LTS(my laptop as the client machine) and a Vultr Ubuntu 22.04 LTS (server machine) to demonstrate the use of the SSH protocol. A summary of what to expect is as shown on the diagram below.

Generating SSH Keys with OpenSSH

OpenSSH ships on Ubuntu 22.04.1 LTS by default. Launch a terminal using Ctrl+Alt+t and follow the following steps:

Create a folder that will store the public and private keys. Create the folder vultrKeyFolder on the root folder using the command

mkdir ~/.ssh/vultrkeysfolder

Check if the folder was created successfully.

ls ~/.ssh/

Generate the keys using ssh-keygen. The -t flag specifies the type of key to generate. Options for ssh protocol 2 are dsa, ecdsa, ed25519, rsa, and rsa1. The -C flag stands for comment. Using the ed25519 generate the key using the command below

ssh-keygen -t ed25519 -C "Key for the vultr ssh demo on my hashnode blog"

The output prompts a location to save the key. Save the key on the folder we created on step 1 and give the keys a name; vultrkey for my case.

/home/user/.ssh/vultrkeysfolder/vultrkey

The next step asks for a passphrase. A passphrase protects the private key from someone who doesn't know the passphrase. If an attacker were to get hold of the key, they would not be able to use it without the passphrase. Use an easy-to-remember passphrase. If you do not wish to have a passphrase, press enter. A random art based on the key will be generated on success.

Two files will be generated. The file with the .pub extension is the public key, while the other is the private key. Your private key should always be kept a secret. The .pub key will be used on the server. You can check for the files generated using

ls ~/.ssh/vultrkeysfolder

Head over to Vultr and create an account. Get $100 Free to follow along. Click on Products on the left panel. Click on Instances, then click on Deploy Server.

Choose the Cloud Compute shared vCPU and the AMD High Performance option for the CPU & Storage Technology.

Choose a location that is closest to you. For the Server Image, choose Ubuntu 22.04 LTS x64

Select the least server size possible; however, you are free to choose whichever configurations that work for you.

Scrolling down, there is an SSH Keys section. Click on the Add New button.

This opens a new window as shown below

Going back to the terminal and open the file containing the public key using cat .

cat ~/.ssh/vultrkeysfolder/vultrkey.pub

Copy the public key.

Paste the key on Vultr and click on Add SSH Key.

The SSH Keys section will show an added key on the dashboard.

Click on the Deploy Now button to deploy the instance.

Once the instance deploys, the status will change to Running. You may need to give the instance around 10 minutes to boot fully.

Click on the instance and copy the IP Address.

On your terminal connect to the server using the command

ssh -i ~/.ssh/vultrkeysfolder/vultrkey root@202.182.111.47

Answer Yes to the prompt asking if you are sure you want to connect

Enter your passphrase, and you should be logged in to the Vultr instance.

Resources

https://www.ssh.com/academy/ssh/protocol

https://www.commandlinux.com/man-page/man1/ssh-keygen.1.html

• Of all adolescent incarcerations, the majority are from fatherless families. These adolescents are more susceptible to delinquency.
• Adults that grew up without a father are three times more likely to end up in jail by the time they clock 30 years.
• Poverty rates in female-led families are at 45.8%, while those with two parents are at 9.5%.

But wait, what does this have to do with Machine learning? Just like humans, machines can learn how to "do" things, good and bad. More importantly, who we learn from directly impacts the kind of humans we turn out to be. Garbage in, garbage out also apply to training machines to "think". You might have encountered a few use cases of machine learning, let's explore a few.

If you have a google photos account, you may notice that similar images are often grouped together. How does Google do it? With enough samples of your face, Google can correctly group other photos your in. Google AI has "learnt" how you look and can confidently identify you. This specific technique of using samples to learn is called supervised machine learning.

Go is an ancient board game from China from over 2,500 years ago. The game has more moves than there are atoms in the universe! Wining in Go requires multiple layers of strategic thinking. Therefore, computers cannot brute force and win this game. Yet, the team at Deepmind trained an AI (Alpha Go) that beat the world champion in Go. Alpha Go researchers implemented deep neural networks to achieve this great fit.

An example close to home is social media algorithms. Oh, the mighty algorithms! Ever search for a YouTube video on "how to do X", and then suddenly YouTube starts recommending videos on that search? Instagram works the same, too, search a page on shoes, and suddenly your feed is full of vendors selling shoes. These algorithms learn from what we search for and recommend products to us. Meta heavily relies on personalized recommendations to generate revenue for the company. Arguably all big tech is making billions of our search data.

It gets more interesting! Researchers from Harrisburg University developed a facial recognition system that predicts criminality. The AI was developed to assist law enforcement and military operations. It works by extracting facial features that predict criminality. The research also emphasized on the AI not being racially biased. Racial bias is perpetuated by having sample data that does not have a balanced representation of all races. However, the AI has an 80% accuracy. A very low accuracy for the stakes at hand. Imagine going to prison becomes your eyebrow matches that of a criminal!

In 2016, an AI chatbot was released by Microsoft on Twitter. Tay was the chatbot's name. Tay was designed to learn through interacting with Twitter users. Through interactions, Tay's personality would be likened to that of a racist within 24 hours of launch.

Tay was shut down, with Microsoft citing a coordinated attack.

These are just but a few of the many applications of machine learning. Sounds exciting? Good! Jenga school and Zindi will conduct a 5-day hackathon on Machine learning. Come one come all. Tell a friend to tell a friend. Come ready to learn and, most importantly, have fun with all the endless possibilities that Machine Learning offers! Find registration information on the poster below.

Happy hacking! You've got this!

For example, 2 and 4 are factors of 16 because 16÷ 2 = 8 exactly and 16 ÷ 4 = 4 exactly. The other factors of 16 are 1, 8, and 16.

Take 13 as an example; we would need to divide 13 by every number from 2 to 13 to check if we get a perfect division. We can do this using the modulo. The modulus operator (%) in python gives the remainder after division.

For example:

13 % 2 -> 1 # 13/2 = 6 remainder 1

13 % 3 -> 1 # 13/3 = 4 remainder 1

13 % 4 -> 1 # 13/4 = 3 remainder 1

13 % 5 -> 3 # 13/5 = 2 remainder 3

13 % 6 -> 1 # 13/6 = 2 remainder 1

13 % 7 -> 6 # 13/7 = 1 remainder 6

13 % 8 -> 5 # 13/8 = 1 remainder 5

13 % 9 -> 4 # 13/9 = 1 remainder 4

13 % 10 -> 3 # 13/10 = 1 remainder 3

13 % 11 -> 1 # 13/11 = 1 remainder 2

13 % 12 -> 1 # 13/12 = 1 remainder 1

13 % 13 -> 0 # 13/13 = 1 remainder 0

We can conclude that a number qualifies as a prime number when we only get 0 as the result of the modulus when we divide the number by itself. In code, we can implement this as follows:

def check_if_prime(x):
      for i in range(2, x):
          if x % i == 0:
             return print("not prime")              
          return print("Is prime")

The code works as expected for checking if a number is prime. We need to modify the code to return a list of prime numbers within a given range. For example, prime numbers between 1 and 100 are:

2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97

Find prime numbers within a range

Working with lists requires the use of nested for loops.

via GIPHY

The complete code is as shown below:

import time    

def check_if_prime(upper_range):

  if upper_range  < 2 :
     return []  
  # append all prime numbers
  list_of_primes = [2] 
  # outter loop from 2 to x
  for number in range(3, upper_range + 1):
      is_prime = True
      # check if the current value of j is prime
      for num in range(3, number):
          # check if j is prime by getting the modulus from 2 to j
          if number % num == 0:
             # if num is not prime break loop and moves to the next value in the range and change is_prime to false
             is_prime = False
             break
      # if  the number is prime, append it to the list
      if is_prime :
        list_of_primes.append(number)             
  return  list_of_primes         

tic = time.perf_counter() # start time
primes = check_if_prime(100000)
toc = time.perf_counter() # End Time

print(primes)
# TIME TAKEN
print(f"Build finished in {toc - tic:0.4f} seconds")

Going over the code above:

• The function takes in a number as a parameter. The parameter represents the maximum value in a range. For example, if we need to find the prime numbers between 0 and 1000, 1000 would be the upper_range

• Initialize a list that will hold all the prime numbers of the given range. The list initializes at 2 since 2 is the first prime number. Any number below two returns an empty list.

• Create an outer for loop that loops through 0 to the upper_range. We add one since the range function excludes the upper range.

For example, when finding the range from 0-5, the range function only works from 0-4. Therefore, if we need the 5 included, we need to add 1 -i.e., from 0-6.

• Initialize a boolean that tracks if the current digit is a prime. It initializes as True and changes to False when the modulus yields a zero.
• The inner for loop checks the specific if the digit is a prime number. This is the same implementation we had earlier on for a single number.
• In the event, the number is prime, the is_prime variable changes to False and is appended to the list_of_primes list.
• lastly, we measure how long this algorithm takes when the upper_range supplied is 100000. It takes 47.34 seconds.

Optimization

We can tweak the algorithm to be more efficient.

• All even numbers are not prime numbers since they are divisible by 2. Thus if a number is not divisible by 2, what's the point of dividing by 4,6, or 8? Essentially, we can eliminate even numbers.

• Consider the number 80. Below are all the factors of 80

    2 *  40 = 80
    4 *  20 = 80
    5 *  16 = 80
    8 * 10 = 80
  10 * 8 = 80 # from here, we are repeating the factors in a different order
  16 *  5 = 80
  20 *  4= 80
  40 *  2= 80

• Once we get to 8 *10, the factors don't change—the order changes with the bigger number starting. Coincidentally, the square root of 80 is 8.94427191. We can then reduce iteration up to the square root of the number when looking for primes.

• Lastly, we use the existing prime values to get the modulus in the inner loop.

def check_if_prime(upper_range):

    if upper_range  < 2 :
       return []

    list_of_primes=[2] 

    for number in range(3, upper_range+1, 2):
          is_prime = True
          square_root_stop = number**0.5
          for num in list_of_primes: # <---only use primes
                if num > square_root_stop: # <--- square root
                      break
                if number % num == 0:
                      is_prime = False
                      break
          if is_prime:
                list_of_primes.append(number)
    print(list_of_primes)
    return list_of_primes     

tic = time.perf_counter() # 
primes = check_if_prime(100000)
toc = time.perf_counter() # End Time
# Print the Difference Minutes and Seconds
print(f"Build finished in {(toc - tic)/60:0.0f} minutes {(toc - tic)%60:0.0f} seconds")
# For additional Precision
print(f"Build finished in {toc - tic:0.4f} seconds")

This reduces the runtime of our algorithm by 46 seconds! Amazing!

via GIPHY

A few examples of palindromes are :

- Madam
- Hannah
- Bob
- Never odd or even
- A man, a plan, a canal – Panama
- 121
- 99
- 22

When dealing with strings, it's rather straightforward. The reverse of the string should be equal to the original string.

An implementation of this in python, is as shown below

original_string = "Madam"
original_string = original_string.lower() # convert to lowercase
string_reversed = "".join(reversed(original_string))

Comparing both variables yields true.

The implementation works with one-word strings but needs some modifications to work with the sentences. The transformations needed would be :

1. Remove punctuations and special characters
2. Remove all spaces

The transformations are done using regex. Code implementation for these transformations is as shown below

import re # import regex

original_sentence = "A man, a plan, a canal – Panama "
original_sentence = original_sentence.lower() # convert to lowercase
original_sentence = re.sub('[^A-Za-z0-9]+', '', original_sentence) # remove all special characters and spaces
sentence_reversed = "".join(reversed(original_sentence))

Wait, but you said this is about numbers!

Buckle up, boys! A palindrome adventure awaits!

via GIPHY

The goal is to avoid converting the integer into a string to check if it qualifies as a palindrome. A few base cases we need to cover:

1 . Negative numbers cannot be palindromes

-121 -> 121-

2 . 10 is not a palindrome. In fact, any number ending with a zero is not a palindrome. When the zero moves to the ones position, a new number is formed.

3 . All numbers between 0 and 9 are palindromes

1010 -> 0101 # not read as the same number

Implementation for the base cases :

def check_if_palindrome(x):
# check if negative
    if x < 0 :
      return "Not a palindrome"
# check if ends with zero
    if x % 10 == 0:
       return "Not a palindrome"
 # check if x is between 0 and 9
    if(x > 0 && x <10):
       return "Is palindrome"

The % in python is used to get the modulus. In layman's terms, the modulus returns the remainder when we perform division.

For example, the division of 12 by 10 yields 1.2. 2 is the remainder of the division. Thus if we perform 12 % 10, we get 2.

This concept will be the backbone of our implementation for finding palindromes.

A few more examples for clarity :

100 % 10 -> 0
11 % 10 -> 1
13 % 6 -> 1
22 % 4 -> 2

via GIPHY

Look at the following examples. Do you notice a pattern?

import math # math module in python

121  % 10 -> 1
math.floor(121 / 10 )-> 12 

1771 % 10 -> 1
math.floor(1771 / 10 -> 177

2442 % 10 -> 2
math.floor(2442 / 10) -> 244

396693 % 10 -> 3
math.floor(396693 / 10) -> 39669

120 % 10 -> 0
math.floor(120 / 10) -> 12

Using 10 as the modulo yields the last digit, while getting the floor of the division with 10 yields all digits with the last digit eliminated.

We can then use two variables to keep track of both halves of the digit. The idea is to have both halves and compare them. If they are the same when reversed, we can identify a palindrome.

Let's first look at the even number 396693. We need an algorithm that splits the number into two, such that :

half_one = 396
half_two = 396 # 693 reversed

Hence in the first iteration, we get :

import math # math module in python

half_one  = math.floor(396693 / 10)
half_two  = 396693 % 10

On the next iteration, we need to use the value in half_one (39669) and eliminate the 9 at the end. The 9 needs to move to half_two(3) and have the value in half_two bump up to 39.

We start with half_two since we need the current value in half_one. Performing the % on 39669 yields 9. To get half_one's value to 39, we first need to multiply it by 10 to get 30. Then add 9 to 30.

Lastly, we can floor half_one by 10 to get 3966 . Our code thus becomes:

import math # math module in python

half_one  = math.floor(396693 / 10)
half_two  = 396693 % 10

half_two  = half_two * 10 + half_one % 10 # need to use the current value of half_one before it is changed
half_one  = math.floor(half_one / 10)

Performing this operation one more time :

import math # math module in python

half_one  = math.floor(396693 / 10)
half_two  = 396693 % 10

half_two  = half_two * 10 + half_one % 10 # need to use the current value of half_one before it is changed
half_one  = math.floor(half_one / 10)


half_two  = half_two * 10 + half_one % 10 # need to use the current value of half_one before it is changed
half_one  = math.floor(half_one / 10)

At this point, we can check if half_one is equivalent to half_two. We can generalise this to work for any even number as :

half_one = 396693
half_two = 0 # initialise as 0

while half_one > half_two :
      half_two  = half_two * 10 + half_one % 10 # need to use the current value of half_one before it is changed
      half_one  = math.floor(half_one / 10)

print(half_one, half_two)

When half_one and half_two are compared, we get a true value and conclude the number is a palindrome and vice versa.

Updating our palindrome function to accomodate even numbers :

def check_if_palindrome(x):
    # check if negative
    if x < 0 :
      return "Not a palindrome"

    # check if ends with zero
    if x % 10 == 0:
       return "Not a palindrome"

    # check if x is between 0 and 9
    if x > 0 and x <10:
       return "Is palindrome"

    # palindrome for even numbers
    half_one = x
    half_two = 0 # initialise as 0    
    while half_one > half_two :              
          half_two  = half_two * 10 + half_one % 10 # need to use the current value of half_one before it is changed
          half_one  = math.floor(half_one / 10)

    if half_one == half_two:
        return "Is palindrome"
    else:
        return "Not a palindrome"

We now need to cover base for odd numbers. Odd numbers are such that when the middle number is removed , we are left with an even number, and we repeat the process above for even numbers.

import math # math module in python

half_one  = math.floor(121/ 10)
half_two  = 121% 10

This yields :

Performing this operation again moves 2 to the second half. We want to always have the odd value on the second half. This ensures the code runs until all values omitting the middle value have been separated.

import math # math module in python

half_one  = math.floor(121/ 10)
half_two  = 121% 10

half_two  = half_two * 10 + half_one % 10 # need to use the current value of half_one before it is changed
half_one  = math.floor(half_one / 10)

At this point we need to drop the 2 on half_two and compare the values in half_one and half_two again to check if the number is a palindrome

import math # math module in python

half_one  = math.floor(121/ 10)
half_two  = 121% 10

half_two  = half_two * 10 + half_one % 10 # need to use the current value of half_one before it is changed
half_one  = math.floor(half_one / 10)

half_two = math.floor(half_two / 10)

At this point half_one and half_two only contain the value 1. Comparing them yields true which means 121 was a palindrome. This can be generalised as :

half_one = 121
half_two = 0 # initialise as 0

while half_one > half_two :


      half_two  = half_two * 10 + half_one % 10 # need to use the current value of half_one before it is changed
      half_one  = math.floor(half_one / 10)     

print(half_one, math.floor(half_two / 10) )

Hold up! We've seen this before! Therefore we can adjust our palindrome function to accommodate odd numbers as follows:

def check_if_palindrome(x):
    # check if negative
    if x < 0 :
      return "Not a palindrome"

    # check if ends with zero
    if x % 10 == 0:
       return "Not a palindrome"

    # check if x is between 0 and 9
    if x > 0 and x <10:
       return "Is palindrome"

    # palindrome for odd and  even numbers
    half_one = x
    half_two = 0 # initialise as 0    
    while half_one > half_two :              
          half_two  = half_two * 10 + half_one % 10 # need to use the current value of half_one before it is changed
          half_one  = math.floor(half_one / 10)

    if half_one == half_two or half_one == math.floor(half_two / 10) :
        return "Is palindrome"
    else:
        return "Not a palindrome"

Testing the handy dandy function

Yeap, that's all! You did it! I'm so proud of you! Thank you for being on this adventure with me.

via GIPHY

Perhaps now you're wondering, is this solution optimal? Space-time complexity? Worry not! I will be covering these complex topics in a future blog. So please subscribe to my newsletter or follow me here on hashnode to ensure you do not miss that!

By the end of this guide, you should be able to integrate the Mpesa express API in Nodejs and express. You will achieve this in several steps:

1. Authorize your Mpesa API by generating an access token for every API call

2. Invoke the STK push endpoint that prompts the user to impute their mpesa pin to authorize transactions

3. Have a callback that saves the transaction details to a database of your choice.

4. Confirm payments

Prerequisites:

• Safaricom developer account. Create one here

• Node js installed

• Postman

The project folder structure for this guide is as shown below.

Project_Folder
└───controllers
      | controllers.lipanampesa.js
└───middlewares
      | middlewares.generateAccessToken.js
└───routes
      | routes.lipanampesa.js
└───utils
      | utils.timestamp.js

| server.js
| package.json
| .env

Setup

Log on to the developer portal and create an app.

Click on show key/secret to reveal the test application's consumer key and secret key.

Save the consumer key and the secret key in the .env file

SAFARICOM_CONSUMER_SECRET=paste_secret_here
SAFARICOM_CONSUMER_KEY=paste_key_here

On the navbar, switch to the APIS tab, then click on the simulate button on the Mpesa express card

On the simulator on the right, choose the TestApp.

Save the business short code and the passkey in the .env

The complete .env now looks as follows

SAFARICOM_CONSUMER_SECRET=paste_secret_here
SAFARICOM_CONSUMER_KEY=paste_key_here
PASS_KEY=paste_passkey_here
BUSINESS_SHORT_CODE=paste_shortcode_here
PORT=3000 # port your server will run on

The project dependencies can be installed using npm on the root folder at the terminal using

 npm install cors dotenv express request ngrok

• Cors - package for providing a Connect/Express middleware that can be used to enable CORS .

• dotenv - loads environment variables from .env file

• express - minimalist web framework

• request - making HTTP calls

• ngrok - node wrapper for ngrok

Use Case

Let's dive into the code

via GIPHY

The code you will cover assumes the user will pay for an order identified by an Order_ID. You will initiate a payment request to the end user for that specific Order_ID. You will also implement a route that confirms the payment was made.

Server

Add the following code in the server.js

import express from "express";
import cors from "cors";
import 'dotenv/config'

// initialize express
const app = express()

// middlewares
app.use(express.json())
app.use(cors())

// import routes
import lipaNaMpesaRoutes from "./routes/routes.lipanampesa.js"
app.use('/api',lipaNaMpesaRoutes)

const port = process.env.PORT

app.listen(port, () => {
    console.log(`App listening on port ${port}`)
})

In the above code, you import relevant libraries, initialize express, and pass the express.json and the cors middleware. In this case, you then import the routes and start the server on the port number in the .env; Port 3000.

Routes

Moving to the routes.lipanampesa.js, the code in this file is as follows

import express from 'express'
const router = express.Router()
import {
    initiateSTKPush,
    stkPushCallback,
    confirmPayment
} from "../controllers/controllers.lipanampesa.js";


import {accessToken} from "../middlewares/middlewares.generateAccessToken.js";

router.route('/stkPush').post(accessToken,initiateSTKPush)
router.route('/stkPushCallback/:Order_ID').post(stkPushCallback)
router.route('/confirmPayment/:CheckoutRequestID').post(accessToken,confirmPayment)

export default router

In the routes file, you initialize the express router and import functions from the controllers.lipanampesa.js file (To be covered in the next section). You also import the accessToken middleware function. This middleware will be used for the authorization of all our Safaricom requests. Lastly, we create three post-request routes that will be used.

• The /stkPush route will initiate stk push popup on the users phone.

• The /stkPushCallback/:Order_ID route will be the route Safaricom sends the results of the stk push.

• The /confirmPayment/:CheckoutRequestID route will use the CheckoutRequestID to confirm payment details. The CheckoutRequestID comes from successfully executing the stk push.

Generating the Access Token

The code in the middlewares.generateAccessToken.js is as follows

import request from "request";
import 'dotenv/config'

export const accessToken = (req, res, next)=> {
    try{

        const url = "https://sandbox.safaricom.co.ke/oauth/v1/generate?grant_type=client_credentials"
        const auth = new Buffer.from(`${process.env.SAFARICOM_CONSUMER_KEY}:${process.env.SAFARICOM_CONSUMER_SECRET}`).toString('base64');

        request(
            {
                url: url,
                headers: {
                    "Authorization": "Basic " + auth
                }
            },
            (error, response, body) => {
                if (error) {
                    res.status(401).send({
                        "message": 'Something went wrong when trying to process your payment',
                        "error":error.message
                    })
                }
                else {
                    req.safaricom_access_token = JSON.parse(body).access_token
                    next()
                }
            }
        )
    }catch (error) {

        console.error("Access token error ", error)
        res.status(401).send({
            "message": 'Something went wrong when trying to process your payment',
            "error":error.message
        })
    }

}

The middleware requests https://sandbox.safaricom.co.ke/oauth/v1/generate?grant_type=client_credentials using the consumer key and consumer secret parsed to base64 as the authorization header. The request returns an access token that we append to the original request as req.safaricom_access_token to be used in the controller that the request will be passed along.

Utils

Before moving to the controllers, you will need a utility function that converts DateTime to the format YearMonthDayHourMinuteSecond

timestamp -> YearMonthDayHourMinuteSecond

For example, the date conversion is as shown below

Date -> 8/20/2022, 9:17:44 AM
Timestamp -> 20220820091744

The code to do this in the utils.timestamp.js is as follows

function parseDate(val) {
    return (val < 10) ? "0" + val : val;
}

export const  getTimestamp = () => {

    const dateString  = new Date().toLocaleString("en-us", {timeZone: "Africa/Nairobi"})
    const dateObject = new Date(dateString);
    const month  = parseDate(dateObject.getMonth() + 1);
    const day  = parseDate(dateObject.getDate());
    const hour = parseDate(dateObject.getHours());
    const minute = parseDate(dateObject.getMinutes());
    const second = parseDate(dateObject.getSeconds());
    return dateObject.getFullYear() + "" + month + "" + day + "" +
        hour + "" + minute + "" + second;
}

The parseDate is a helper function that appends a zero on values less than 10. For example, August the 8th month is converted to 08, 2A.M to 02, etc. Values greater than 10 remain unchanged.

The getTimestamp function gets the current local date time as a string and then converts it to a date object. We get the month, day, hour, minute, and second from the date object. Lastly, the function returns a timestamp in the form YearMonthDayHourMinuteSecond.

Controllers

The first function on the controllers.lipanampesa.js :

import request from "request";
import 'dotenv/config'
import {getTimestamp} from "../Utils/utils.timestamp.js";
import ngrok from 'ngrok'

// @desc initiate stk push
// @method POST
// @route /stkPush
// @access public
export const initiateSTKPush = async(req, res) => {
    try{

        const {amount, phone,Order_ID} = req.body
        const url = "https://sandbox.safaricom.co.ke/mpesa/stkpush/v1/processrequest"
        const auth = "Bearer " + req.safaricom_access_token

        const timestamp = getTimestamp()
        //shortcode + passkey + timestamp
        const password = new Buffer.from(process.env.BUSINESS_SHORT_CODE + process.env.PASS_KEY + timestamp).toString('base64')
        // create callback url
        const callback_url = await ngrok.connect(process.env.PORT);
        const api = ngrok.getApi();
        await api.listTunnels();


        console.log("callback ",callback_url)
        request(
            {
                url: url,
                method: "POST",
                headers: {
                    "Authorization": auth
                },
                json: {
                    "BusinessShortCode": process.env.BUSINESS_SHORT_CODE,
                    "Password": password,
                    "Timestamp": timestamp,
                    "TransactionType": "CustomerPayBillOnline",
                    "Amount": amount,
                    "PartyA": phone,
                    "PartyB": process.env.BUSINESS_SHORT_CODE,
                    "PhoneNumber": phone,
                    "CallBackURL": `${callback_url}/api/stkPushCallback/${Order_ID}`,
                    "AccountReference": "Wamaitha Online Shop",
                    "TransactionDesc": "Paid online"
                }
            },
            function (e, response, body) {
                if (e) {
                    console.error(e)
                    res.status(503).send({
                        message:"Error with the stk push",
                        error : e
                    })
                } else {
                    res.status(200).json(body)
                }
            }
        )
    }catch (e) {
        console.error("Error while trying to create LipaNaMpesa details",e)
        res.status(503).send({
            message:"Something went wrong while trying to create LipaNaMpesa details. Contact admin",
            error : e
        })
    }
}

The function will initiate the stk push to the end user. The code destructures the request body to extract the amount, phone, and Order_ID. You then send a request to https://sandbox.safaricom.co.ke/mpesa/stkpush/v1/processrequest using the access token as the authorization header and the shortcode + passkey + timestamp as the password. This request requires a callback URL which we create using ngrok. Ngrok exposes your server to the internet. You also initialize a tunnel listener that will log any logs from the callback function to the terminal.

Running this request on post man yields:

On the terminal, you will see a callback logged

Lastly, the phone number used will receive an stk push as shown

The second function on the controllers.lipanampesa.js :

// @desc callback route Safaricom will post transaction status
// @method POST
// @route /stkPushCallback/:Order_ID
// @access public
export const stkPushCallback = async(req, res) => {
    try{

    //    order id
        const {Order_ID} = req.params

        //callback details

        const {
            MerchantRequestID,
            CheckoutRequestID,
            ResultCode,
            ResultDesc,
            CallbackMetadata
                 }   = req.body.Body.stkCallback

    //     get the meta data from the meta
        const meta = Object.values(await CallbackMetadata.Item)
        const PhoneNumber = meta.find(o => o.Name === 'PhoneNumber').Value.toString()
        const Amount = meta.find(o => o.Name === 'Amount').Value.toString()
        const MpesaReceiptNumber = meta.find(o => o.Name === 'MpesaReceiptNumber').Value.toString()
        const TransactionDate = meta.find(o => o.Name === 'TransactionDate').Value.toString()

        // do something with the data
        console.log("-".repeat(20)," OUTPUT IN THE CALLBACK ", "-".repeat(20))
        console.log(`
            Order_ID : ${Order_ID},
            MerchantRequestID : ${MerchantRequestID},
            CheckoutRequestID: ${CheckoutRequestID},
            ResultCode: ${ResultCode},
            ResultDesc: ${ResultDesc},
            PhoneNumber : ${PhoneNumber},
            Amount: ${Amount}, 
            MpesaReceiptNumber: ${MpesaReceiptNumber},
            TransactionDate : ${TransactionDate}
        `)

        res.json(true)

    }catch (e) {
        console.error("Error while trying to update LipaNaMpesa details from the callback",e)
        res.status(503).send({
            message:"Something went wrong with the callback",
            error : e.message
        })
    }
}

The callback function simply destructures the request body and prints it to the console. At this point, you can save the response to any database you choose.

Rerunning the /stkpush endpoint on postman yields the following on the terminal:

The last part is confirming payments using the CheckoutRequestID.

The final function in the controller file is as shown

// @desc Check from safaricom servers the status of a transaction
// @method GET
// @route /confirmPayment/:CheckoutRequestID
// @access public
export const confirmPayment = async(req, res) => {
    try{


        const url = "https://sandbox.safaricom.co.ke/mpesa/stkpushquery/v1/query"
        const auth = "Bearer " + req.safaricom_access_token

        const timestamp = getTimestamp()
        //shortcode + passkey + timestamp
        const password = new Buffer.from(process.env.BUSINESS_SHORT_CODE + process.env.PASS_KEY + timestamp).toString('base64')


        request(
            {
                url: url,
                method: "POST",
                headers: {
                    "Authorization": auth
                },
                json: {
                    "BusinessShortCode":process.env.BUSINESS_SHORT_CODE,
                    "Password": password,
                    "Timestamp": timestamp,
                    "CheckoutRequestID": req.params.CheckoutRequestID,

                }
            },
            function (error, response, body) {
                if (error) {
                    console.log(error)
                    res.status(503).send({
                        message:"Something went wrong while trying to create LipaNaMpesa details. Contact admin",
                        error : error
                    })
                } else {
                    res.status(200).json(body)
                }
            }
        )
    }catch (e) {
        console.error("Error while trying to create LipaNaMpesa details",e)
        res.status(503).send({
            message:"Something went wrong while trying to create LipaNaMpesa details. Contact admin",
            error : e
        })
    }
}

The function takes a similar construct as the initiate stkpush function, but on the JSON body, we pass in the CheckoutRequestID, which we get from the request params. Executing the function on post man confirms the payment, as shown below

You did it!

via GIPHY

The full code can be found here

Why use type hints in python? Python being a dynamically typed language means that variable types are determined during run time. However, declaring the expected type can be beneficial in very large projects as the code becomes self-documenting and make it more readable. FAANG also expect their python developers to know how to use type hints in code. Google has a Python style guide that can be found here.

Initializing Lists

Lists are a built-in data type in Python used to store multiple items in a single variable and are initialized in two ways :

1. Using square brackets :

# Import the type hint module
# Any means the List will take in any type of data type
from typing import List, Any

# Initialised using square brackets

my_empty_list: List[Any] = []
print(my_empty_list)

my_list_of_integers: List[int] =  [1,2,3,4]
print(my_list_of_integers)

my_list_of_floats: List[float]  = [1.2, 3.4, 5.6]
print(my_list_of_floats)

my_list_of_strings: List[str]  =["this", "is" ,"a" ,"string"]
print(my_list_of_strings)

my_list_of_booleans: List[bool]  =[True,False,False,True]
print(my_list_of_booleans)

my_mixed_list: List[Any] = [1, "Hello", False, 1.23]
print(my_mixed_list)

Output:

[]
[1, 2, 3, 4]
[1.2, 3.4, 5.6]
['this', 'is', 'a', 'string']
[True, False, False, True]
[1, 'Hello', False, 1.23]

1. Using the list constructor :

# Import the type hint module
# Any means the List will take in any type of data type
from typing import List, Any

# Initialized using the list constructor 

my_empty_list: List[Any] = list()
print(my_empty_list)

my_list_of_integers: List[int] = list((1,2,3,4))
print(my_list_of_integers)

my_list_of_floats: List[float] =list ((1.2, 3.4, 5.6))
print(my_list_of_floats)

my_list_of_strings: List[str]=list (("this", "is" ,"a" ,"string"))
print(my_list_of_strings)

my_list_of_booleans: List[bool] =list((True, False,False,True))
print(my_list_of_booleans)

my_mixed_list: List[Any]= list((1, "Hello", False, 1.23))
print(my_mixed_list)

Output:

[]
[1, 2, 3, 4]
[1.2, 3.4, 5.6]
['this', 'is', 'a', 'string']
[True, False, False, True]
[1, 'Hello', False, 1.23]

Characteristics of list items

Items in a list have the following characteristics :

1. Order

The list items are said to be ordered when they follow each other in a set manner. By default, new items are added to the very end of the list

1. Can hold duplicates

Since the list has an order and every item in the list has a unique index, we can duplicate values within the list as much as we want.

For example ```py my_list = [1,2,2,5,6,3,4]

The two 2's are in different positions yet have the same value.

3. Can be changed

We can add, remove and change elements in a list. 

## Accessing items in a list

### Positive indexing

> Positive indexing means we start from the beginning of the list

Every item in the list has a unique index. Index values begin at zero, meaning that the first element in a list is in index zero. We use square brackets containing the index of the list we want. 

For example,

Given the list:
```py
# list of only integers
my_list: List[int] = [1,2,2,5,6,3,4]

We can access the first element in the list (1) using :

# list of only integers
my_list: List[int] = [1,2,2,5,6,3,4]
# item in the list of integers is of type int(integer)
first_item: int= my_list[0]
print(first_item)

The general access expression is :

list_item_you_want: List[type of list items the list will contain] = the_list[list_item_index]

Negative indexing

Negative indexing means we start from the end of the list.

We can also access the list items from the last element using negative indexing.

For example,

Given the list :

# List of only strings
adele_lyrics: List[str]  = ["Hello", "from" ,"the" ,"other", "side"]

To get the last element, we can use negative indexing : ```py

List of only strings

adele_lyrics: List[str] = ["Hello", "from" ,"the" ,"other", "side"]

The last element of the string which is of type str(string)

last_word: str = adele_lyrics[-1]

print(last_word) Output :sh side To get the second last element :py

List of only strings

adele_lyrics: List[str] = ["Hello", "from" ,"the" ,"other", "side"]

The second last element of the string which is of type str(string)

second_last_word: str = adele_lyrics[-2]

print(second_last_word) Output :sh other ```

Range of indexes

A range of indexes helps get a sublist from the list.

For example:

Given the list :

# List of fruits

fruits: List[str] = ["oranges", "grapefruits","mandarins ","limes","strawberries","raspberries","blueberries","watermelons","rockmelons ","passionfruit","banana",]

From the list of the fruits we can tell that we have at least three categories of fruits on the list :

1. Citrus fruits -> oranges to limes

2. Berries -> strawberries to blueberries

3. Melons -> watermelons to rockmelons

Using range indexing we can get the three categories. The range indexing general expression is :

sublist = the_list[start_index: end_index]

Note that the end_index is not inclusive. For example, a start index of 2 and an end index of 5 will only return elements from the second index to the fourth index.

To get the :

• Citrus fruits

# get the citrus fruits oranges(0) to limes(3)
# to include limes in the sublist we add one to the limes index
citrus_fruits: List[str] = fruits[0:4]
print(citrus_fruits)

Output:

['oranges', 'grapefruits', 'mandarins ', 'limes']

• Berries

# get the berries strawberries(4) to blueberries(6)
# to include blueberries in the sublist we add one to the blueberries index
berries_fruits: List[str] = fruits[4:7]
print(berries_fruits)

Output: ['strawberries', 'raspberries', 'blueberries']

• Melons

# get the melons watermelons to rockmelons
# to include rockmelons in the sublist we add one to the rockmelons index

melon_fruits: List[str] = fruits[7:9]
print(melon_fruits)

Output:

['watermelons', 'rockmelons ']

We can also use negative indexing to get the melons.

To include rockmelons in the sublist we can index from watermelons to the passionfruit index which can either be 9 or -2

# get the melons watermelons to rockmelons using negative indexing

melon_fruits: List[str] = fruits[7:-2]
print(melon_fruits)

Output:

['watermelons', 'rockmelons ']

Length of a list

There are instances when we may need to know how many items are present in a list.

Python has an inbuilt function for this :

numbers_list: List[int] = [0,1,2,3,4,5,6,7,8,9,10]
number_of_list_items: int = len(numbers_list)
print(number_of_list_items)

Output:

11

Example use case:

Given a list get all the values in a list that are in an even index.

def get_all_values_in_even_indexes(the_list: List[Any]) -> List[Any]:
    """ Get all values that are in even indexes in a list
    Args:
     the_list: List of values.

    Returns:
      List of all values in even indexes

    """
    # initialise empty list for even numbers
    even_list: List[Any] = []
    # loop over all the indexes in the list
    for index in range(len(the_list)):
        # if the index number when divided by 2 we get 0; ie its even we append the item to the even list
        if index % 2 == 0:
            even_list.append(the_list[index])
    print(even_list)
    return even_list


items_in_even_indexes = get_all_values_in_even_indexes(["Zero","One", "Two", "Three", "Four", "Five", "Six"])

Output

['Zero', 'Two', 'Four', 'Six']

List manipulations

Changing values in a list

To replace the value of a list item, we can refer to its index number and change the value.

Going back to the fruits example, to change limes to lemons we simply do:

fruits: List[str] = ["oranges", "grapefruits", "mandarins ", "limes", "strawberries", "raspberries", "blueberries",
                     "watermelons", "rockmelons ", "passionfruit", "banana"]
fruits[3]: str = 'lemons'

Output:

['oranges', 'grapefruits', 'mandarins ', 'lemons', 'strawberries', 'raspberries', 'blueberries', 'watermelons', 'rockmelons ', 'passionfruit', 'banana']

Using range indexing we can also change the values on a list. On the fruits list, we can change the berries to leafy greens as follows:

leafy_greens: List[str]= ["lettuce","spinach","silverbeet"]
fruits[4:7] = leafy_greens
print(fruits)

Output: sh ['oranges', 'grapefruits', 'mandarins ', 'lemons', 'lettuce', 'spinach', 'silverbeet', 'watermelons', 'rockmelons ', 'passionfruit', 'banana']

Adding items in a list

We can also insert items in an index without replacing the existing values. In the fruits example, we can add a new citrus fruit after lemons without replacing lettuce.

fruits.insert(4,"pomelo")
print(fruits)

Output :

['oranges', 'grapefruits', 'mandarins ', 'lemons', 'pomelo', 'lettuce', 'spinach', 'silverbeet', 'watermelons', 'rockmelons ', 'passionfruit', 'banana']

To add items at the end of a list we use the append() method:

Prerequisites

• You need to have python 3.5+ installed - Download python here

• Pip package installer- Download pip from here

• Venv for creating virtual environments - Download Venv from here

Working with projects that do not have a requirements.txt

I came across this issue when I was following machine learning tutorials on Microsft Learn. The GitHub repo only has the code with no requirements.txt. I was particularly interested in the code for extracting text from images and pdfs. The code looks as shown in the image below.

You can see there is no requirements.txt. We have to first clone the whole repo then navigate to the specific folder we want to work on

Start by cloning the whole repository.

git clone https://github.com/MicrosoftLearning/AI-102-AIEngineer.git

Then get into the specific folder we are interested in

cd AI-102-AIEngineer-master\20-ocr\Python\read-text\

The project structure is as shown below ``` AI-102-AIEngineer-master └───20-ocr | └───Python | | └───read-text | | | read-text.py | | | .env | | └───images | | | | Lincoln.jpg | | | | Note.jpg | | | | Rome.pdf

## Setting up an environment

Setting up an environment for a project ensures that the installs done for the project are within that environment without affecting global installs. This can also be useful when a specific package is only installed once on a specific project.
To set up a venv environment:

```shell
python -m venv name-of-environment

Name the environment folder that you wish. I'll name mine venv. My current folder structure now looks like ``` AI-102-AIEngineer-master └───20-ocr | └───Python | | └───read-text | | | read-text.py | | | .env | | | venv | | └───images | | | | Lincoln.jpg | | | | Note.jpg | | | | Rome.pdf

On windows activate the environment 
```shell
 cd venv\Scripts\ && activate && cd .. && cd ..

With the environment set, next, we need to install (Pigar)[https://pypi.org/project/pigar/]. Pigar is a tool used to generate requirements.txt for python projects. Install Pigar with pip

pip install pigar

To use pigar on our project simply run

pigar

You can find a generated requirements.txt on your project folder. Learn more about pigar from its documentation. With a requirements.txt, install all the dependencies using

pip install -r requirements.txt

Happy hacking!

I was recently applying for my passport and was required to provide passport numbers for my parents. Problem was, they both misplaced them and the only option left was to visit a government office to have the numbers manually retrieved since my parents applied for passports before digitization. I had to wait for an hour before they could trace the numbers. As I sat there waiting, I couldn't help but wonder if the government could leverage current technology to digitize the documents that are still in hard copy. Although the government will not be using telegram, the approach used here has an endpoint that makes it interoperable with any system that accepts REST endpoints.

For the visual learners, find a video walkthrough of this tutorial here

Demo

What we will be building 👇🏾

Prerequisites

• Familiarity with python

• Azure account - Create an azure account here

• Python 3.8+ - Install python from here

• Pip - Install python from here

• Telegram account - Install python from here

Project structure

ProjectFolder
│main.py   
│app.py   
│.env     
│.gitignore
 | requirements.txt
└───images
│   │   abc.png
│   │   chinua.jpg
│   │   story.pdf
│   │   wams.png

Creating an azure cognitive service

Signup for an azure account here

Once logged in, click on the plus create resource

Search for cognitive services

Select cognitive services

Create the resource

If you do not have a resource group created yet, click on the create new below the resource group input. On the name field, make sure to give a unique URL of your project. Azure will let you know when a name is taken.

I did not make any changes on the Identity or tags tab and went straight to the Review+Create tab and clicked on create at the very bottom.

You can check the progress of your deployment on the notification bar that can be accessed using the bell icon on the navbar to the right.

Viola! Once your deployment is done, click on Go to the resource.

We need two important things that will enable us to connect with azure from python. A key and our API endpoint. Click on manage keys .

This takes us to the page where we can access two keys and an endpoint URL.

Environmental variables

Azure Keys

On your IDE, paste any one of the two keys and the endpoint in your .env. Keep the format as follows:

COG_SERVICE_ENDPOINT=your_azure_endpoint_goes_here
COG_SERVICE_KEY=your_azure_key_goes_here

Telegram Token

Search the BotFather on telegram or access it from here There are very many fake accounts so please make sure the profile looks as below:

To create a new bot send a /newbot message to the botfather.

You might need to keep trying for an available username. Once you do, copy the token to your .env.

COG_SERVICE_ENDPOINT=your_azure_endpoint_goes_here
COG_SERVICE_KEY=your_azure_key_goes_here
TELEGRAM_TOKEN=your_telegram_token_goes_here

If you will push this code to a codebase, add the .env file to the .gitignore to avoid exposing your keys to the public.

Installing dependencies

Next up, we will install all the dependencies needed to run the project.

Add the following in your requirements.txt

Pillow == 8.3.1
matplotlib == 3.5.1
python_dotenv == 0.17.1
flask == 2.0.2
python-telegram-bot == 13.10
azure-cognitiveservices-vision-computervision==0.7.0
requests == 2.27.1

Open a terminal from the project folder and run : sh pip install -r requirements.txt This installs are the packages needed to run this project. A summary of what each package does:

• Pillow: Imaging Library

• Matplotlib:plotting package

• python_dotenv: Read key-value pairs from a .env file and set them as environment variables

• flask :A simple framework for building complex web applications.

• python-telegram-bot: Python wrapper for the official telegram API

• azure-cognitiveservices-vision-computervision :Microsoft Azure Cognitive Services Computer Vision Client Library for Python

• requests :Python HTTP for Humans. Now we are ready to write some code

via GIPHY

Difference between READ API and OCR API

The OCR API uses an older recognition model, supports only images, and executes synchronously, returning immediately with the detected text. The Read API uses an updated recognition model, takes an image or PDF document as the input, and extracts text asynchronously. But supports fewer languages The READ API also works page by page then line by line then word by word whereas the OCR works region by region then line by line then word by word. In terms of API response, the OCR API's response looks as follows:

{
  "language": "en",
  "textAngle": 0.00000,
  "orientation": "Up",
  "regions": [
    {
      "boundingBox": "462,379,497,75",
      "lines": [
        {
          "boundingBox": "462,379,497,74",
          "words": [
            {
              "boundingBox": "462,379,41,73",
              "text": "Hello"
            },
            {
              "boundingBox": "523,379,153,73",
              "text": "World!"
            }
          ]
        }
      ]
    }
  ]
}

The API response for the READ API looks as follows: JSON { "status": "succeeded", "createdDateTime": "2019-10-03T14:32:04Z", "lastUpdatedDateTime": "2019-10-03T14:38:14Z", "analyzeResult": { "version": "v3.0", "readResults": [ { "page": 1, "language": "en", "angle": 49.59, "width": 600, "height": 400, "unit": "pixel", "lines": [ { "boundingBox": [ 20,61,204,64,204,84,20,81], "text": "Hello world!", "words": [ { "boundingBox": [ 20,62,48,62,48,83,20,82], "text": "Hello", "confidence": 0.91 }, { "boundingBox": [ 51,62,105,63,105,83,51,83], "text": "world!", "confidence": 0.164 } ] } ] } ] } } Since both APIs provide bounding box coordinates, they can be used for visualization. For example, performing OCR on the abc.png in our images folders yields this:

Original image :

Image after OCR:

via GIPHY

You can read more from the official documentation on their differences. Let's get our hands dirty now, shall we?

Ask the user for input

For a start, we will create a function ask_user_for_input() that asks a user to choose whether they want to invoke the Read API or the OCR API. The GetTextRead() function and the GetTextOcr() function only print out a statement but we will build on them as we go. Your main.py should contain: ```py import os

def GetTextOcr(file_path): """ Takes in a file and does OCR on the document and returns a string of the text extracted during OCR. :param file_path: :return: string """ try: print("This function invokes the OCR API on", file_path) except Exception as ex: print(ex)

def GetTextRead(file_path): """ Takes in a file extracts handwritten or printed text on the document and returns a string of the text extracted during OCR. :param file_path: :return: string """ try: print("This function invokes the READ API on", file_path) except Exception as ex: print(ex)

def ask_user_for_input():

""" Asks the user for input from the command line

""" try:

# Menu for text reading functions print('1: Use OCR API\n2: Use Read API\n3: Read handwriting\nAny other key to quit') command = input('Enter a number:') if command == '1': image_file = os.path.join('images', 'chinua.png') GetTextOcr(image_file) elif command == '2': image_file = os.path.join('images', 'story.pdf') GetTextRead(image_file) elif command == '3': image_file = os.path.join('images', 'wams.png') GetTextRead(image_file)

except Exception as ex: print(ex)

ask_user_for_input()

Run main.py using:
```shell
python main.py

You should get prompts to enter a number

Authenticating the azure cognitive client

The library imports and the azure cognitive client authentication code is as follows: ```py import os import time from matplotlib import pyplot as plt from PIL import Image, ImageDraw

Import namespaces

from azure.cognitiveservices.vision.computervision import ComputerVisionClient from azure.cognitiveservices.vision.computervision.models import OperationStatusCodes from msrest.authentication import CognitiveServicesCredentials

from dotenv import load_dotenv

Get Configuration Settings

load_dotenv() cog_endpoint = os.getenv('COG_SERVICE_ENDPOINT') cog_key = os.getenv('COG_SERVICE_KEY')

Authenticate Computer Vision client

credential = CognitiveServicesCredentials(cog_key) cv_client = ComputerVisionClient(cog_endpoint, credential)

def GetTextOcr(file_path): """ Takes in a file does OCR on the document and returns a string of the text extracted during OCR.

.... ``` Let's go over it line by line.

• The os enables us to retrieve images from local directories.

• We will need our code to sleep for a few seconds at some point. The time module helps with that.

• Matplotlib offers a canvas for us to draw on

• Pillow (PIL) enables us to read and draw on our images. We will be drawing bounding boxes to show where the text was extracted. This will be especially useful when visualizing what the cognitive service is doing behind the scenes

• The azure ComputerVisionClient is responsible for all the OCR computations we will be doing

• The azure computer vision models are advanced models trained to not only work on printed text but also handwritten. As we will see later, the OCR API does very poorly on handwritten data while the READ API is competent in decoding printed and handwritten text.

• Last but not least, we import the CognitiveServicesCredentials that handles authenticating our code to azure using the key and endpoint we generated from the Azure portal.

• Finally, we import load_dotenv. A python package is used to retrieve environmental variables.

Now that we have all the necessary libraries In place, we need to authenticate our computer vision client. The code just before the GetTextOcr() should now be as follows:

import os
import time
from PIL import Image, ImageDraw
from matplotlib import pyplot as plt

# Import namespaces
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from azure.cognitiveservices.vision.computervision.models import OperationStatusCodes
from msrest.authentication import CognitiveServicesCredentials

from dotenv import load_dotenv

# Get Configuration Settings
load_dotenv()
cog_endpoint = os.getenv('COG_SERVICE_ENDPOINT')
cog_key = os.getenv('COG_SERVICE_KEY')

# Authenticate Computer Vision client
credential = CognitiveServicesCredentials(cog_key)
cv_client = ComputerVisionClient(cog_endpoint, credential)


def GetTextOcr(file_path):
    """
    Takes in a file does OCR on the document and returns a string
    of the text extracted during OCR.

To authenticate the client, we get our environmental variables from the .env.

Note: The variable name being retrieved from the .env must be the same one passed to os.getenv(). Case sensitivity is observed. For example, the COG_SERVICE_ENDPOINT in the env must be the same as what is passed in the py os.getenv('COG_SERVICE_ENDPOINT')

We then authenticate the computer vision by first passing our key to the CognitiveServicesCredentials that validate our key. That is then passed along to the ComputerVisionClient.

OCR API

We will start with the OCR API.

The full code for the GetTextOcr() looks as follows:

....
def GetTextOcr(file_path):
    """
    Takes in a file and does OCR on the document and returns a string
    of the text extracted during OCR.
    :param file_path:

    """
    try:
        print("This function invokes the OCR API on", file_path)
        # Use OCR API to read the text in the image
        with open(file_path, mode="rb") as image_data:
            ocr_results = cv_client.recognize_printed_text_in_stream(image_data)

        # Prepare image for drawing
        fig = plt.figure(figsize=(7, 7))
        img = Image.open(file_path)
        draw = ImageDraw.Draw(img)

        # All the words extracted will be stored as a list
        results = []

        # Process the text line by line
        for region in ocr_results.regions:
            for line in region.lines:

                # Show the position of the line of text
                l, t, w, h = list(map(int, line.bounding_box.split(',')))
                draw.rectangle(((l, t), (l + w, t + h)), outline='magenta', width=5)

                # Read the words in the line of text
                line_text = ''
                for word in line.words:
                    line_text += word.text + ' '
                print(line_text.rstrip())
                results.append(line_text.rstrip())

        # Save the image with the text locations highlighted if the image was ocrd
        if len(results) > 0:
            plt.axis('off')
            plt.imshow(img)
            # create output folder if doesnt exist
            if not os.path.exists('ocr-results'):
                os.makedirs('ocr-results')
            file_path = file_path.rsplit('\\', 1)[-1].rsplit('.', 1)[0]
            outputfile = f'ocr-results\\{file_path}-ocr_results.jpg'
            fig.savefig(outputfile)
            print('Results saved in', outputfile)
            # if there was no ocr decoded the results list will be empty
        if len(results) == 0:
            print(f'{file_path} IMAGE WAS NOT OCRD')


    except Exception as ex:
        print(ex)
....

Code break down: ```py

Use OCR API to read the text in the image

with open(file_path, mode="rb") as image_data: ocr_results = cv_client.recognize_printed_text_in_stream(image_data)

- The above code opens the image we are currently working on.
- Streams the image to the azure client that recognized the text in the image.
- The results from the stream are stored in ocr_results.

```py
        # Prepare image for drawing
        fig = plt.figure(figsize=(7, 7))
        img = Image.open(file_path)
        draw = ImageDraw.Draw(img)

Next, we initialize a matplotlib canvas (figure) and use the Draw() method from pillow to draw our image on the canvas.

     # All the words extracted will be stored as a list
        results = []

Initialize an empty list that will store all the words extracted from the image by the azure service.

        # Process the text line by line
        for region in ocr_results.regions:
            for line in region.lines:

                # Show the position of the line of text
                l, t, w, h = list(map(int, line.bounding_box.split(',')))
                draw.rectangle(((l, t), (l + w, t + h)), outline='magenta', width=5)

                # Read the words in the line of text
                line_text = ''
                for word in line.words:
                    line_text += word.text + ' '
                print(line_text.rstrip())
                results.append(line_text.rstrip())

The for loop starts by looping through each region from the ocr_results. Refer to the sample API response we highlighted when comparing the OCR and READ API. Every region has an array of lines in the region. Think of a line like a sentence. Every line comes with bounding boxes that we will be drawing on the image we duplicated on the matplotlib canvas. All the words extracted from each line are stored in the results list we had initialized. This means that if the image did not contain any words or the OCR API was unable to extract any words (remember that OCR performs poorly on handwritten text), the results list remains empty. This logic takes us to the next part of the code.

       # Save the image with the text locations highlighted if the image was ocrd
        if len(results) > 0:
            plt.axis('off')
            plt.imshow(img)
            # create output folder if doesnt exist
            if not os.path.exists('ocr-results'):
                os.makedirs('ocr-results')
            file_path = file_path.rsplit('\\', 1)[-1].rsplit('.', 1)[0]
            outputfile = f'ocr-results\\{file_path}-ocr_results.jpg'
            fig.savefig(outputfile)
            print('Results saved in', outputfile)
            # if there was no ocr decoded the results list will be empty
        if len(results) == 0:
            print(f'{file_path} IMAGE WAS NOT OCRD')

We first check the length of the results list. If the length is greater than 0, the image processed contained at least one line or one word that the API extracted. If this is the case, we save the image on our local directory. If no word was extracted on the image we simply print that the image did not contain any word the API could recognize. Run main.py and test the function:

Results were as we had seen on the abc.png. Try out an image of your liking by changing the file name in the function we created for asking the user for input initially.

via GIPHY

READ API

The GetTextRead() function implements the READ API. The full code for the function is as follows:

def GetTextRead(file_path):
    """
      Takes in a file extracts handwritten or printed text
       on the document and returns a string of the text extracted during OCR.
      :param file_path:
    """
    try:
        print("This function invokes the READ API on", file_path)
        results = []
        # Use Read API to read text in image
        with open(file_path, mode="rb") as image_data:
            read_op = cv_client.read_in_stream(image_data, raw=True)

        # Get the async operation ID so we can check for the results
        operation_location = read_op.headers["Operation-Location"]
        operation_id = operation_location.split("/")[-1]

        # Wait for the asynchronous operation to complete
        while True:
            read_results = cv_client.get_read_result(operation_id)
            if read_results.status not in [OperationStatusCodes.running, OperationStatusCodes.not_started]:
                break
            time.sleep(1)

        # If the operation was successfuly, process the text line by line
        if read_results.status == OperationStatusCodes.succeeded:
            for page in read_results.analyze_result.read_results:
                for line in page.lines:
                    print(line.text)
                    results.append(line.text)
        print('Reading text in {}\n'.format(file_path))

        return results
    except Exception as ex:
        print(ex)

Going over the code line by line:

results =[]

Initialise empty list that will store all the extracted text in the image or document ```py
# Use Read API to read text in image with open(file_path, mode="rb") as image_data: read_op = cv_client.read_in_stream(image_data, raw=True)

# Get the async operation ID so we can check for the results operation_location = read_op.headers["Operation-Location"] operation_id = operation_location.split("/")[-1]

# Wait for the asynchronous operation to complete while True: read_results = cv_client.get_read_result(operation_id) if read_results.status not in [OperationStatusCodes.running, OperationStatusCodes.not_started]: break time.sleep(1) ``` We mentioned that the READ API works asynchronously so we need a mechanism that waits for the whole process to complete. We first pass the image to our computer vision client. In order to make sure the READ API is done, we start a while loop that only exits when the status code is either SUCCEEDED or FAILED. You can learn more about the operational codes here. Once the code breaks, the results from the READ API will be stored in the read_results variable.

 # If the operation was successfuly, process the text line by line
        if read_results.status == OperationStatusCodes.succeeded:
            for page in read_results.analyze_result.read_results:
                for line in page.lines:
                    print(line.text)
                    results.append(line.text)
        print('Reading text in {}\n'.format(file_path))

        return results
    except Exception as ex:
        print(ex)

If the operation is successful, we extract all the words and append them in the results variable. Any errors will be printed to the console as exceptions. Since the READ API works on both documents and images, we will test it out using the story.pdf and the handwritten file wams.py.

Test it out :

python main.py

You should get the following results:

The full code for main.py:

import os
import time
from PIL import Image, ImageDraw
from matplotlib import pyplot as plt

# Import namespaces
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from azure.cognitiveservices.vision.computervision.models import OperationStatusCodes
from msrest.authentication import CognitiveServicesCredentials
from dotenv import load_dotenv


# Get Configuration Settings
load_dotenv()
cog_endpoint = os.getenv('COG_SERVICE_ENDPOINT')
cog_key = os.getenv('COG_SERVICE_KEY')

# Authenticate Computer Vision client
credential = CognitiveServicesCredentials(cog_key)
cv_client = ComputerVisionClient(cog_endpoint, credential)


def GetTextOcr(file_path):
    """
    Takes in a file and does OCR on the document and returns a string
    of the text extracted during OCR.
    :param file_path:

    """
    try:
        print("This function invokes the OCR API on", file_path)
        # Use OCR API to read text in image
        with open(file_path, mode="rb") as image_data:
            ocr_results = cv_client.recognize_printed_text_in_stream(image_data)

        # Prepare image for drawing
        fig = plt.figure(figsize=(7, 7))
        img = Image.open(file_path)
        draw = ImageDraw.Draw(img)

        # All the words extracted will be stored as a list
        results = []
        # Process the text line by line
        for region in ocr_results.regions:
            for line in region.lines:

                # Show the position of the line of text
                l, t, w, h = list(map(int, line.bounding_box.split(',')))
                draw.rectangle(((l, t), (l + w, t + h)), outline='magenta', width=5)

                # Read the words in the line of text
                line_text = ''
                for word in line.words:
                    line_text += word.text + ' '
                print(line_text.rstrip())
                results.append(line_text.rstrip())

        # Save the image with the text locations highlighted if the image was ocrd
        if len(results) > 0:
            plt.axis('off')
            plt.imshow(img)
            # create output folder if doesnt exist
            if not os.path.exists('ocr-results'):
                os.makedirs('ocr-results')
            file_path = file_path.rsplit('\\', 1)[-1].rsplit('.', 1)[0]
            outputfile = f'ocr-results\\{file_path}-ocr_results.jpg'
            fig.savefig(outputfile)
            print('Results saved in', outputfile)
            # if there was no ocr decoded the results list will be empty
        if len(results) == 0:
            print(f'{file_path} IMAGE WAS NOT OCRD')


    except Exception as ex:
        print(ex)


def GetTextRead(file_path):
    """
      Takes in a file and extracts handwritten or printed text
       on the document and returns a string of the text extracted during OCR.
      :param file_path:
    """
    try:
        print("This function invokes the READ API on", file_path)
        results = []
        # Use Read API to read text in image
        with open(file_path, mode="rb") as image_data:
            read_op = cv_client.read_in_stream(image_data, raw=True)

        # Get the async operation ID so we can check for the results
        operation_location = read_op.headers["Operation-Location"]
        operation_id = operation_location.split("/")[-1]

        # Wait for the asynchronous operation to complete
        while True:
            read_results = cv_client.get_read_result(operation_id)
            if read_results.status not in [OperationStatusCodes.running, OperationStatusCodes.not_started]:
                break
            time.sleep(1)

        # If the operation was successfuly, process the text line by line
        if read_results.status == OperationStatusCodes.succeeded:
            for page in read_results.analyze_result.read_results:
                for line in page.lines:
                    print(line.text)
                    results.append(line.text)
        print('Reading text in {}\n'.format(file_path))

        return results
    except Exception as ex:
        print(ex)


def ask_user_for_input():
    """
    Asks the user for input from the command line

    """
    try:

        # Menu for text reading functions
        print('1: Use OCR API\n2: Use Read API\n3: Read handwriting\nAny other key to quit')

        command = input('Enter a number:')
        if command == '1':
            file_path = os.path.join('images', 'abc.png')
            GetTextOcr(file_path)
        elif command == '2':
            file_path = os.path.join('images', 'story.pdf')
            GetTextRead(file_path)
        elif command == '3':
            file_path = os.path.join('images', 'wams.png')
            GetTextRead(file_path)


    except Exception as ex:
        print(ex)


ask_user_for_input()

via GIPHY

So far we've been hardcoding the image we are processing, but we need to make the awesome project accessible to others. We will implement a flask API endpoint that will be used by telegram. You can use the endpoint with any frontend framework that accepts REST

Our Flask endpoint will reside in app.py. The bare-bone code on app.py is :

import os
from dotenv import load_dotenv
from flask import Flask
from telegram.ext import Updater, MessageHandler, Filters
from telegram.ext import CommandHandler
from main import GetTextRead
load_dotenv()


TELEGRAM_TOKEN = os.getenv('TELEGRAM_TOKEN')
updater = Updater(token=TELEGRAM_TOKEN, use_context=True)
dispatcher = updater.dispatcher

app = Flask(__name__)


@app.route('/')
def extract_text_from_telegram(update, context):
    try:
        print("Uploading to telegram ...")
    except Exception as ex:
        update.message.reply_text("Upload an image with text")
        print(ex)


# set up the introductory statement for the bot when the /start command is invoked
def start(update, context):
    try:
        chat_id = update.effective_chat.id
        context.bot.send_message(chat_id=chat_id,
                                 text="Hello there, Wamaitha here. Thank you for registering for the event. \n"
                                      "This bot takes in a photo and applies the azure cognitive service to extract "
                                      "printed or handwritten text from images or pdfs. Have fun while at it! \n "
                                      "Connect with me  :\n"
                                      "Linkedin : https://www.linkedin.com/in/wamaithanyamu/\n"
                                      "Github : https://github.com/wamaithaNyamu \n"
                                      "Twitter : https://twitter.com/wamaithaNyamu \n")


    except Exception as ex:
        print(ex)


# run the start function when the user invokes the /start command
dispatcher.add_handler(CommandHandler("start", start))

# Handle messages from the user
dispatcher.add_handler(MessageHandler(Filters.all, extract_text_from_telegram))
updater.start_polling()


if __name__ == '__main__':
    app.run(debug=True)

Let's go over the code line by line :

• We import the os and dotenv that will be used to retrieve our environmental variable for the telegram token

• Import flask that will be used to make an endpoint

• Import telegram.ext and helper functions that will be used to communicate with python from telegram

• We will be using the READ API since its more comprehensive than the OCR API, as such we need to import it from our main.py

• The extract_text_from_telegram function will be populated with the logic needed to filter out messages sent by the user whether they uploaded an image or a pdf. More on this next! For now we only print a statement to the console

• The start function is what you see when you start the bot

• The dispatcher is what interacts with telegram. the first dispatcher handles the /start command from the user. The other dispatcher will handle any incoming messages from telegram and then invoke the extract_text_from_telegram function.

• Lastly we poll the updater to check if there is any message from the user so that we can process it

• We then run the flask app

To run the app.py as is :

On Unix Bash (Linux, Mac, etc.):

export FLASK_APP=app
flask run

On Windows CMD:

set FLASK_APP=app
flask run

On Windows PowerShell:

$env:FLASK_APP = "app"
flask run

The full code for our app.py is as follows: ```py import os import pathlib

from dotenv import load_dotenv from flask import Flask import requests from telegram.ext import Updater, MessageHandler, Filters from telegram.ext import CommandHandler

from our main.py

from main import GetTextRead

load_dotenv()

TELEGRAM_TOKEN = os.getenv('TELEGRAM_TOKEN') updater = Updater(token=TELEGRAM_TOKEN, use_context=True) dispatcher = updater.dispatcher

app = Flask(name)

def file_handler(update): try: print(update.message) file_id = '' if len(update.message.photo) == 0 and update.message.document.file_id: print("WE HAVE A FILE", update.message.document.file_id) file_id = update.message.document.file_id elif len(update.message.photo) > 0: print("WE HAVE AN IMAGE", update.message.photo[-1].file_id) file_id = update.message.photo[-1].file_id return file_id

except Exception as e: print("Handler exception",e)

@app.route('/') def extract_text_from_telegram(update, context): """

:param update: checks for updates from telegram :param context: :return: """ try:

file_name ='' print("Uploading to telegram ...", ) file_id = file_handler(update) print("FILE ID", file_id)

if file_id: update.message.reply_text("Processing file...") file_path = f'https://api.telegram.org/bot{TELEGRAM_TOKEN}/getFile?file_id={file_id}' img = requests.get(file_path).json() img = img['result']['file_path'] # img path: photos/file_35.jpg print("img path:", img) file_image = f'https://api.telegram.org/file/bot{TELEGRAM_TOKEN}/{img}' response = requests.get(file_image) # split_img path: file_35.jpg split_img = img.rsplit('/', 1)[-1] print("split_img path:", split_img)

# File Extension: .jpg file_extension = pathlib.Path(split_img).suffix print("File Extension: ", file_extension) # file_35.jpg-AgACAgQAAxkBAAIBS2H5FN20IK2ArluFlw_E_MiY2bw.jpg file_name = f'{split_img}-{file_id}.{file_extension}'

file = open(f'{file_name}', "wb") file.write(response.content) file.close()

user_image_text_results = GetTextRead(file_name) print("results:", user_image_text_results) if len(user_image_text_results) > 0: results = ' '.join(str(e) for e in user_image_text_results) print(results) update.message.reply_text("Thank you for your patience. This is what I extracted:") update.message.reply_text(results) else: update.message.reply_text( "Unfortunately, this image does not contain any printed or handwritten text. ")

else: update.message.reply_text("Upload an image with text")

if os.path.exists(file_name): os.remove(file_name) print(file_name, ' deleted!') else: print(f"The file {file_name} does not exist") except Exception as ex: update.message.reply_text("Upload an image with text") print(ex)

set up the introductory statement for the bot when the /start command is invoked

def start(update, context): """ /start command on telegram :param update: :param context: :return: """ try: chat_id = update.effective_chat.id context.bot.send_message(chat_id=chat_id, text="Demo time!!!")

except Exception as ex: print(ex)

run the start function when the user invokes the /start command

dispatcher.add_handler(CommandHandler("start", start))

dispatcher.add_handler(MessageHandler(Filters.all, extract_text_from_telegram)) updater.start_polling()

print

if name == 'main': app.run(debug=True)

Going over the app.y, we added code to the extract_text_from_telegram function and also made a utility function file_handler.

The file_handler function takes in the message from the user and determines whether the message has an image or a file and then returns an id of either the image id or the file id. 

In the extract_text_from_telegram function, we call on the file_hanlder function to give us an id of the current message being sent by the user. 

If there is a file id, we request a telegram for that file and store it on our local machine. The image or file is then passed to the Get_Read function that we worked on in the main.py. The extracted text we get from Azure is sent back to the user on telegram. If azure did not find any texts on the provided file or image, we let the user know that we found no text on the supplied file or image.

Lastly, the file or image we saved from telegram is then deleted from our system. 

<iframe src="https://giphy.com/embed/3o6Mbbs879ozZ9Yic0" width="480" height="366" frameBorder="0" class="giphy-embed" allowFullScreen></iframe><p><a href="https://giphy.com/gifs/season-7-the-simpsons-7x4-3o6Mbbs879ozZ9Yic0">via GIPHY</a></p>

Results from telegram:

![sample1.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1645532848269/-nTsQEk8m.png)


![sample2.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1645532866846/aN6n3nZhD.png)


![sample3.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1645532891711/DtdRTEFeS.png)


![sample4.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1645532902947/ZhTPipvHT.png)


![sample5.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1645532913522/vfYxBZMY8.png)

# HOSTING ON HEROKU

We need to make sure the rest of the world can use our bot even we shut down our computer. We will be hosting on heroku.


You will need to have the heroku CLI and git  installed on your system. If you do not have the CLI you can install it using npm:

```shell
npm install -g heroku

Installation instructions can be found here incase you get stuck.

Git installation instructions are found here

via GIPHY

Create the Procfile

Heroku uses a special file that defines the commands we want to run when heroku launches our application. In this case we need a Procfile that tells Heroku to start our flask app. Add the Procfile to your project root folder. The folder structure should now be as follows: ProjectFolder |main.py |app.py |.env |.gitignore | requirements.txt | Procfile └───images │ │ abc.png │ │ chinua.jpg │ │ story.pdf │ │ wams.png

The Procfile does not have a file extension

Add the following to your Procfile

web: python app.py

Create the Heroku App

Ensure you have the CLI and Git installed on your system

Create a Heroku app using the command

heroku create unique_name_for_your_bot

Alternatively, you can let Heroku create a unique name for your bot by running

heroku create

Running the command gives two URLs. One is the application URL and the other is a Git URL.

in your .env file add the application URL, we will need it later. Your .env should now look as follows:

COG_SERVICE_ENDPOINT=your_azure_endpoint_goes_here
COG_SERVICE_KEY=your_azure_key_goes_here
TELEGRAM_TOKEN=your_telegram_token_goes_here
BOT_URL=your_heroku_application_url_goes_here
PORT=5000

Python Buildpack

Since our application runs on Python, we need to set our Heroku environment to use python. Heroku does this using buildpacks. Add the python buildpack using:

heroku buildpacks:set heroku/python

Modifying the app.py file

Before we can deploy the code to Heroku we need to change a few things in our app.py code

...
dispatcher.add_handler(MessageHandler(Filters.all, extract_text_from_telegram))
# updater.start_polling() <--- comment this out

# add the webhook code
updater.start_webhook(listen="0.0.0.0",
                      port=int(os.environ.get('PORT', 8080)),
                      url_path=TELEGRAM_TOKEN,
                      webhook_url=os.getenv('BOT_URL') + TELEGRAM_TOKEN
                      )

We are switching from polling to webhooks. Polling asks telegram if the user has sent any new message every few seconds while a webhook waits until the user sends a message to act on it. Thus we are telling telegram to listen to any new messages from the Heroku application URL we provide.

Deploying

With git already installed, you need to first initialize git in the repo then push the code to Heroku. Run the following commands sequentially in your terminal:

Make sure to add .env file in your .gitignore

• Initialise git shell git init

• Add all files except those listed in the .gitignore to git shell git add .

• Commit shell git commit -m "Done with my telegram azure bot "

• Deploy

git push heroku main

Adding envs to Heroku

Log in to your Heroku dashboard and go to the application you created. it will be under the same name you gave Heroku or Heroku created for you. On the settings tab, scroll and click on Reveal Config vars.

From there just copy-paste all the vars in your .env.

Conclusion

via GIPHY

Good job! You did it. You should now have the app running on Heroku. Note that the free dynos on Heroku sleep when they are idle for a while thus you may find that the next time you use your telegram bot you may have delays. Worry not! The dynos take a minute to go back up online.

Full code can be found here

Expectations

The end goal is to make this a binary classification problem where given a set of inputs, the model either gives a prediction of approved or rejected. The approved prediction means Angela can go ahead and give the loan to the applicant while rejection means Angela will either have to revisit the decision manually or outright trust the model.

Loading the Data

For this problem, we will be using the data set on loan prediction on Kaggle. You can download the data from here

You will need to use scikit-learn version 0.23.2 for the project to run smoothly

pip install -U scikit-learn==0.23.2

Import the relevant libraries needed ```py

Importing important libraries

import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import plotly import plotly.express as px import plotly.figure_factory as ff from sklearn.utils import resample from sklearn.preprocessing import StandardScaler , MinMaxScaler from collections import Counter from scipy import stats

Classifiers

from sklearn.ensemble import AdaBoostClassifier , GradientBoostingClassifier , VotingClassifier , RandomForestClassifier from sklearn.linear_model import LogisticRegression , RidgeClassifier from sklearn.discriminant_analysis import LinearDiscriminantAnalysis from sklearn.model_selection import RepeatedStratifiedKFold from sklearn.neighbors import KNeighborsClassifier from sklearn.model_selection import GridSearchCV from sklearn.tree import DecisionTreeClassifier

from sklearn import tree from sklearn.naive_bayes import GaussianNB from xgboost import plot_importance from xgboost import XGBClassifier from sklearn.svm import SVC

#Model evaluation tools from sklearn.metrics import classification_report , accuracy_score , confusion_matrix from sklearn.metrics import accuracy_score,f1_score from sklearn.model_selection import cross_val_score

#Data processing functions from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split from sklearn import model_selection from sklearn.preprocessing import LabelEncoder le = LabelEncoder()

Load the data from the csv. Replace ”./path to csv” with where your data is located.

```py
    data = pd.read_csv("./path to csv")

We load the dataset as a pandas dataframe and store it in data. We can view our data as follows py data.head(4)

Exploratory data analysis We first draw a scatter plot of all variables against themselves in a bivariate fashion. py sns.pairplot(data) plt.show()

data.describe()

From the above results from the describe function, we can see that some columns have missing values as indicated by the count. The standard deviation(std) also tells us more on how the data is distributed. Low standard deviation means data points are clustered around the mean, and high standard deviation indicates data is more spread out. We might need to investigate further on the data points on the ApplicantIncome and CoapplicantIncome columns as the std is too high which might indicate presence of outliers. Before we can do any manipulations we need to understand our column datatype using the info() function. py data.info()

We have a dataset with 4 distinct datatype. The Loan_ID column does not help with our predictions, so we drop it. You will not qualify for a loan just because your loan id is a certain value.

data = data.drop(["Loan_ID"],axis=1)

Are graduates more eligible for loans than non-graduands?

                sns.catplot(x='Loan_Status',
                            y='LoanAmount',
                            hue="Education",
                            ci="sd",
                            palette="dark",
                            alpha=.6,
                            height=6,
                            data=data,
                            kind='bar',
                           );

Are women more likely to get loans than men?

  sns.catplot(x='Loan_Status',
                            y='LoanAmount',
                            hue="Gender",
                            ci="sd",
                            palette="dark",
                            alpha=.6,
                            height=6,
                            data=data,
                            kind='bar',
                    );

Are single people more likely to get loans than married people? ```py sns.catplot(x='Loan_Status', y='LoanAmount', hue="Married", ci="sd", palette="dark", alpha=.6, height=6, data=data, kind='bar', );

![maritalLoanStatus.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1643098374749/n7IoRKeoe.png)
From the std, we suspect having outliers in the Applicant Income and coapplicant income. Lets start by first visualizing the distribution of the data in the ApplicantIncome to verify that we indeed have outliers that are accounting for the high std.
```py

fig = px.scatter_matrix(data["ApplicantIncome"])
fig.update_layout(width=700,height=400)
fig.show()

The ApplicantIncome scatter plot visually confirms that the column has outliers. Next we work on the visualization of the CoapplicantIncome column.

fig = px.scatter_matrix(data["CoapplicantIncome"])
fig.update_layout(width=700,height=400)
fig.show()

As expected, the visual plot of the Coapplicantincome also shows our outliers.

To make work easier as the notebook grows, I will be writing all our model predictions on an external txt

import os
if os.path.exists("results.txt"):
   os.remove("results.txt")
else:
   print("The file does not exist")
def write_results_to_file(txt):

  f = open("results.txt","a")
  f.write(txt + '\n')
  f.close()

def print_line(name):
    write_results_to_file(f'------------------------------------------------------------{name}----------------------------------')
    print(f'------------------------------------------------------------{name}----------------------------------')

The above code just removes any old files named results.txt on our file system and creates one if it does not exist. The write_results_to_file appends a new line to the results.txt while the print_line adds a divider to the write_results_to_file file. Checking the distribution of the data

Ideally,to get the best possible results our data needs to be in a normal distribution. Data in a normal distribution observes the bell curve.

def check_distribution_of_column(column_name):
  fig = px.histogram(data[column_name],x =column_name,y = column_name)
  fig.update_layout(title=column_name)
  fig.show()

The function check_distribution_of_column takes in a column name and plots a histogram showing the distribution of data in that column. We iterate through all the columns in our data-frame calling on our handy function to draw the distribution.

for col in data.columns:
    print("Now showing the distribution of the " + col + " column:")
    check_distribution_of_column(col)

You should be having individual plots of each column. The loan amount plot looks like this

From the above graphs we can see that we have a mix of categorical variables and discrete variables. Categorical variables contain a finite number of categories or distinct groups. Categorical data might not have a logical order. For example, categorical predictors for our data-set are gender,marital status,self employed etc. Discrete variables are numeric variables that have a countable number of values between any two values for example the applicants income, coapplicantincome and loan amount. With all these, we can now begin some data cleaning.

Data Cleaning

From our EDA we have established we have some problems we need to fix with the data :

Deal with null values to ensure our data is of same length Deal with outliers Find a way of converting categorical variables to discrete variables since our model will expect to use discrete values. Find a way to make the data normally distributed

1. Dealing with null values We have a few options:

Drop all rows with at least one null value Replace the null value with either the mode, mean or median of the column. For the first approach, we do not have much data to work with as such I will be exploring the second option. Also dropping the null values doesnt necessarily mean that we will not be dropping more rows as we continue with the data cleaning. We might need to drop more rows in future and thus greatly reduce our dataset. Option 2 is our best bet.

# shows number of missing values by column
data.isnull().sum()

For the categorical variables, I will use the mode of the column. For example, we have more men in the dataset so the probability of someone with a null value having being a man was higher than it being a woman.

data["Gender"].fillna(data["Gender"].mode()[0],inplace=True)
data["Married"].fillna(data["Married"].mode()[0],inplace=True)
data["Self_Employed"].fillna(data["Self_Employed"].mode()[0],inplace=True)
data["Loan_Amount_Term"].fillna(data["Loan_Amount_Term"].mode()[0],inplace=True)
data["Dependents"].fillna(data["Dependents"].mode()[0],inplace=True)
data["Credit_History"].fillna(data["Credit_History"].mode()[0],inplace=True)

# shows number of missing values by column
data.isnull().sum()

Now checking for null values, we notice only the LoanAmount still has nulls which is a discrete variable.To fix this we will use the median since we had already established an outlier for this column. Thus using the mean might not give an accurate representation of roughly how much might have been borrowed.

data["LoanAmount"].fillna(data["LoanAmount"].median(),inplace=True)

All done with dealing with null values.

2. Dealing with outliers We have three discrete variables of which two have outliers. py data["ApplicantIncome"] = np.log(data["ApplicantIncome"]) #As "CoapplicantIncome" columns has some "0" values we will get log values except "0" since log 0 is undefined data["CoapplicantIncome"] = [np.log(i) if i!=0 else 0 for i in data["CoapplicantIncome"]] data["LoanAmount"] = np.log(data["LoanAmount"]) We apply the natural logarithm to all our discrete variables to normalize the data. Natural log transformations reduce skewness to our datasets. Compare our loan amount distribution now with what we had above.