Feb 21 2022 · Aleksandra Płońska, Piotr Płoński

Build Computer Vision Web App with Python, OpenCV and Mercury

Mercury convert Jupyter notebook to web appComputer Vision methods provide great flexibility for images processing. For example, you can easily write a computer program for converting any photo to an artistic sketch. In this article I will show you:

  • how to create Python code for converting images into sketches in Jupyter Notebook with the usage of OpenCV package,
  • how to convert the notebook into a web application with Mercury framework,
  • how to deploy the web app to Heroku (using free dyno).

The preview of the application is presented in the GIF below:

the sketch web app

Looks interesting? Let's start - you will learn a lot! 🤓 📚

Setup a GitHub repository

Please create a new repo for our project. You can do this by going to https://github.com/new. I've selected artistic-sketches-jupyter-mercury as the repository name. After repository creation, please click on a green Code button and copy the address to clone the repository.

git clone git@github.com:pplonski/artistic-sketches-jupyter-mercury.git

My repository is public and available on GitHub https://github.com/pplonski/artistic-sketches-jupyter-mercury.

Set virtual environment and install requirements

Let's change the directory to the project folder. We will use virtualenv to create a virtual environment for our python packages.

virtualenv sketchenv
source sketchenv/bin/activate

The sketchenv is the name of our virtual environment. Let's create requirements.txt file with needed packages:

mljar-mercury
opencv-python-headless
pillow
matplotlib

Please install packages by running:

pip install -r requirements.txt

To use the new virtual environment in the Jupyter run:

python -m ipykernel install --user --name=sketchenv

It is important to add a virtual environment as a kernel in Jupyter because it is not the same thing as the Jupyter kernel.

Create a new Notebook

Please start Jupyter Notebook with the following command:

jupyter notebook

Let's create a new notebook. ** Remember to select a kernel with the name of the virtual environment sketchenv.** Change notebook's name by clicking the Untitled at the top. I set sketch_app as a new notebook name.

Start with importing packages and setting the matplotlib plot options.

import os
import cv2
from PIL import Image
from matplotlib import pyplot as plt

import matplotlib as mpl
mpl.rcParams['figure.figsize'] = (12,12)
mpl.rcParams['axes.grid'] = False

Let's set new variables in the next cell:

image_path = "/home/piotr/Downloads/house.jpg"
output_dir = "output_images"

The image_path is a path for an image that will be converted. The output_dir is the directory name where a new image with sketch will be saved.

In the next cell I will define a method for image plotting in the notebook:

def imshow(image, title=None, fname=None):
    if len(image.shape) > 3:
        image = tf.squeeze(image, axis=0)

    plt.imshow(image)
    plt.axis('off')
    if title:
        plt.title(title)
    if fname:
        plt.savefig(fname)

Let's load the image and plot it:

img = cv2.imread(image_path)
RGB_img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
imshow(RGB_img)

You should see the house like in the image below:

Input image with house

Now let's apply the sequence of image transformation that will produce the artistic sketch of the image:

# image transformation
grayed = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
inverted = cv2.bitwise_not(grayed)
blurred = cv2.GaussianBlur(inverted, (19, 19), sigmaX=0, sigmaY=0)
def blend(x, y):
    return cv2.divide(x, 255 - y, scale=256)
final_result = blend(grayed, blurred)

# save the image
_ = cv2.imwrite(os.path.join(output_dir, "sketch_photo.jpg"), final_result)

# plot the image
RGB_img = cv2.cvtColor(final_result, cv2.COLOR_BGR2RGB)
imshow(RGB_img)

You can read about step-by-step image preprocessing on Behic Guven's article.

The image after transformation should look like this:

Output image with house

OK, you have the notebook ready, but how to share it with others?

Notebook code

Mercury for notebook sharing

The Mercury framework will help you with notebook sharing:

  • it transforms a notebook into a web application,
  • sharing is as easy as sending the link,
  • the input widgets for the notebook are created based on the YAML header,
  • user can provide her input and then execute the notebook,
  • the notebook's code can be hidden - to not scary non-coders.

The Mercury is already installed in the virtual environment (the mljar-mercury package). Let's start the web app from the notebook by running:

mercury watch sketch_app.ipynb

Please open the web browser and enter the address http://127.0.0.1:8000/. You should see your notebook's card:

Mercury watch the notebook - the main view

Please click on the notebook, and you will have it displayed with the warning message telling that YAML header is missing:

Mercury watch the notebook

Let's add the YAML header. Please add the RAW cell at the top of the notebook.

Add RAW cell to the notebook

The YAML header:

---
title: Convert Photo to Sketch 🖌️
description: Create sketch from uploaded a photo
show-code: False
params:
    image_path:
        input: file
        label: Please upload a photo
    output_dir:
        output: dir
---

The YAML header configuration:

  • title - the application title used in the home view and at the top of the sidebar in notebook view,
  • description - the text describing what the app is doing, it is used in the home view,
  • show-code - the variable which decides whether to show or hide the notebook's code,
  • params - the widgets added to the notebook.

There are two widgets added to the notebook:

  • image_path - the file widget. The variable image_path will be replaced with the path to the uploaded file.
  • output_dir - the variable pointing to the output directory. All files created in the output directory will be available for download.

The notebook has now the file upload widget, and here is what it looks like:

Notebook with YAML header in Mercury

Deploy to Heroku

Please set Heroku account and install Heroku CLI tools. You can create the Heroku app by running:

heroku create sketch-app-mercury

To make the application work on Heroku, we will need to add two more files. The Procfile with the command how to run our web app:

web: mercury run 0.0.0.0:$PORT

and Aptfile with additional system libraries that needs to be installed in order to use OpenCV package:

libsm6
libxrender1
libfontconfig1
libice6

To install additional system libraries, we need to add buildpack in Heroku:

https://github.com/heroku/heroku-buildpack-apt.git

Please open your Heroku dashboard and go to app settings. There please find the Buildpacks and add the above link.

Heroku buildpack

Please add and commit all project files to the repository. The application can be deployed to Heroku with the command:

git push heroku main

My app is running at https://sketch-app-mercury.herokuapp.com/.

The Mercury application

Conclusions

Thanks to tools like Python, OpenCV, and Jupyter you can easily create amazing scripts for images manipulation. The open-source framework - Mercury is the easiest way to share the notebook with non-coders. You can easily add input widgets to the notebook. What is more, the code can be hidden, and only outputs are displayed to the end-user. The deployment process is simple. The Mercury is built on top of the Django framework. Thanks to this, it can be deployed on any server in the cloud.