Schedule Jupyter Notebook to run daily

You have just created an excellent report in Jupyter Notebook and would like to run it daily. Are you looking for a scheduler solution that fulfills your requirements? It would be great to send executed notebook as a PDF by email. You would like to serve a website with an updated HTML view of the notebook and restricted access to only selected users. It would be fantastic to have an option to parametrize the notebook and execute it with customized values. There are several schedulers for Jupyter Notebook available. However, not all fulfill those requirements. This post will show you how to schedule your notebook to run daily with the open-source Mercury framework.

YAML header

The notebook is ready. You would like to run it daily. The example notebook that is producing financial report is presented below.

The code for the notebook is available in the GitHub repository. It has two variables that we will use later to parametrize our notebook:

ticker = "TSLA"
period = "3mo"

The simple YAML header that will execute the notebook every day from Monday to Friday at 9:00 is below:

---
title: Financial report
description: Stock financial report
schedule: '0 9 * * 1-5'
show-code: False
---

The keywords in the YAML header:

• title: Financial report is a title of our notebook, it will be displayed in the Mercury website,
• description: Stock financial report it a short description of notebook content, it will be displayed in the Mercury main site,
• schedule: '0 9 * * 1-5' controls the time at which the notebook will be executed,
• show-code: False hides the Python code from the resulting notebook.

After adding such YAML header in the first, raw cell in the notebook we can use Mercury for notebooks scheduling and results serving. Please execute the following command to start Mercury development server locally (installation documentation):

mercury run

The schedule keyword is accepting crontab string as paramter. I'm ususally using crontab.guru website to get proper string value:

Parametrized Notebook

Let's extend the YAML header with new parameters. The Mercury allows you to add widgets to the notebook. Users can control the variables ticker and period in User Interface with widgets (add configuration in the YAML to get widgets).

---
title: Financial report
description: Stock financial report
schedule: '0 9 * * 1-5'
show-code: False
params:
    ticker:
        input: select
        label: Select a ticker
        value: TSLA
        choices: [TSLA, TWTR, MSFT, SNOW, PLTR, NFLX]
    period:
        input: select
        label: Select period
        value: 3mo
        choices: [1mo, 2mo, 3mo, 6mo, 12mo, 24mo]
---

We added two widgets to the notebook, both with type select. The first is for ticker variable. There is TSLA default value and available choices are [TSLA, TWTR, MSFT, SNOW, PLTR, NFLX]. The default is 3mo for period variable with choices [1mo, 2mo, 3mo, 6mo, 12mo, 24mo]. The parametrized notebook in the Mercury website will look like in the image below:

Please check Mercury documentation for more information about available widgets.

The web application is working perfectly. Users can set new values for variables without touching the Python code and execute the notebook:

Email notifications

Let's send executed notebook as PDF file to the list of emails. The Mercury has a keyword notify for this. The example YAML header will look like:

---
title: Financial report
description: Stock financial report
schedule: '0 9 * * 1-5'
notify:
    on_success: contact@mljar.com
    attachment: pdf
show-code: False
params:
    ticker:
        input: select
        label: Select a ticker
        value: TSLA
        choices: [TSLA, TWTR, MSFT, SNOW, PLTR, NFLX]
    period:
        input: select
        label: Select period
        value: 3mo
        choices: [1mo, 2mo, 3mo, 6mo, 12mo, 24mo]
---

The notification email might be send for:

• successful notebook execution with notebook in PDF or HTML formats,
• failure execution (notify notebook's owner that there are some problems).

You can read more in Mercury scheduling documentation.

The example email with notebook attachement sent by Mercury website:

Restricted access

All notebooks are public by default in the Mercury website. There is an option to restrict access to only selected users with share keyword. The user management is Mercury's Pro feature available in the commercial license.

---
title: Financial report
description: Stock financial report
schedule: '0 9 * * 1-5'
notify:
    on_success: contact@mljar.com
    attachment: pdf
show-code: False
share: private # share notebook with authenticated users only!
params:
    ticker:
        input: select
        label: Select a ticker
        value: TSLA
        choices: [TSLA, TWTR, MSFT, SNOW, PLTR, NFLX]
    period:
        input: select
        label: Select period
        value: 3mo
        choices: [1mo, 2mo, 3mo, 6mo, 12mo, 24mo]
---

The access to the notebook might be set for selected users or groups. Please refer to the documentation for more information.

Summary

The open-source Mercury framework offers a service for automatic notebook scheduling and sharing. The scheduled notebook can be parametrized with widgets available in User Interface. The resulting notebooks can be exported to HTML or PDF files and sent to my email. Thanks to user management, the owner can restrict notebook access to selected users. There is an available commercial license for users looking for dedicated support, additional features, and private forks.