A JupyterLab Python Git Workflow

Focusing on ensuring having a stable development environment.

Here we walk through assembling (and then using) a JupyterLab Python Git workflow with an emphasis on ensuring that we have a stable, i.e., having consistent software versions, development environment.

note: Would not call this approach authoritative; rather it is something that I found that works for me (and I could not find anything else better).

The end result of our work will be a series of workflows (see below) starting with creating a new JupyterLab Python project.

Prerequisites

Here we assume the reader is already familiar with JupyterLab; if not the video Introduction to Jupyter Lab for Python is an excellent starting point. Our use of Python will be not much more than a “Hello World!” application; so not much here required by the reader. Here we also assume the reader is familiar with Git; if not the document gittutorial — A tutorial introduction to Git is helpful.

If you wish to follow along, you will need to have pyenv installed.

pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.

You will also need to have Git installed and access to a Git remote repository, e.g., a GitHub repository.

Empty Git Repository

Here we start with an empty Git repository; in my case I am using GitHub.

I follow the instructions “…or create a new repository on the command line”.

The commit for this step.

Python

The first thing we want to ensure is that we always use the same version of Python; as of today the latest version would be version 3.13.1.

We can accomplish this by installing and using pyenv.

pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.

Once we have installed pyenv, we start by installing Python 3.13.1 by running the following command.

$ pyenv install 3.13.1

We then run the following command within our local Git repository.

$ pyenv local 3.13.1

This creates a file, .python-version. With this file in place, we can run the following command within our local Git repository to ensure that we are using the proper version of Python for this project.

$ pyenv install

The commit for this step.

Virtual Environment

Next we need to create a virtual environment.

From Virtual Environments and Packages.

Python applications will often use packages and modules that don’t come as part of the standard library. Applications will sometimes need a specific version of a library, because the application may require that a particular bug has been fixed or the application may be written using an obsolete version of the library’s interface.

This means it may not be possible for one Python installation to meet the requirements of every application. If application A needs version 1.0 of a particular module but application B needs version 2.0, then the requirements are in conflict and installing either version 1.0 or 2.0 will leave one application unable to run.

The solution for this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.

We start by creating a file .gitignore to ensure that the .environment folder is never committed.

.environment/

Now and later, when we first clone this repository, we will need to run the following command once within our local Git repository.

$ python3 -m venv .environment

Now and each time we work with this repository, we need to run the following command within our local Git repository.

$ source .environment/bin/activate

The commit for this step.

Pipenv

Here we be using pipenv to manage our project’s package dependencies;

Now and later, when we first clone this repository, we will need to run the following command once within our local Git repository.

$ pip install pipenv==2024.4.1

The commit (just documentation) for this step.

JupyterLab

Now we can actually install JupyterLab by running the following command within our local Git repository.

$ pipenv install jupyterlab==4.3.5

Later, when we first clone this repository, we will need to run the following command once within our local Git repository to install all the project’s package dependencies (including jupyterlab).

$ pipenv install

We update our .gitignore to ignore a folder that JupyterLab internally uses.

.environment/
.ipynb_checkpoints/

Now and each time we work with this repository, we start JupyterLab by running the following command within our local Git repository.

$ jupyter lab

The commit for this step.

Jupytext

From Jupytext.

Have you always wished Jupyter notebooks were plain text documents? Wished you could edit them in your favorite IDE? And get clear and meaningful diffs when doing version control? Then, Jupytext may well be the tool you’re looking for!

Here we are interested in getting clear and meaningful diffs when doing version control.

We install Jupytext by running the following command within our local Git repository.

$ pipenv install jupytext==1.16.7

The Jupytext supports paired notebooks.

A convenient alternative to text notebooks are paired notebooks. These are a set of two files, say .ipynb and .py, that contain the same notebook, but in different formats.

By creating a jupytext.toml file in our repository, newly created notebooks (.ipynb files) are automatically paired with text notebooks (.py files).

# jupytext.toml at the root of your notebook directory
formats = "ipynb,py:percent"

The commit for this step.

note: I forgot to add the the .ipynb files to the our .gitignore file; fixed in a later commit.

.environment/
.ipynb_checkpoints/
**/*.ipynb

jupyterlab-git

Here we add another JupyterLab extension that gives a GUI Git experience.

$ pipenv install jupyterlab-git==0.51.0

The commit for this step.

Our First Notebook

From within JupyterLab, we right-click and select New folder to create a folder, notebooks; where will store all of our notebooks.

Opening (double-clicking) the notebooks folder, we right-click and select New Notebook to create a new notebook; notice that two files (.ipynb and .py) are created. Following the convention from the video Introduction to Jupyter Lab for Python, we will right-click the Untitled.ipynb file and select Rename to rename the file to 00_project_readme.ipynb.

For completeness, we can add a single Markdown cell to this notebook.

The commit for this step (includes the .gitignore fix from earlier and cleaning out the README.md file).

With this last change, we have created a template that we can use to create new JupyterLab Python projects.

New Project Workflow

We start by downloading the project template and uncompressing it; say into a project folder named my_jupyterlab_project. One way of doing this (Linux, MacOS) is:

$ curl -o project.tar.gz -L https://github.com/larkintuckerllc/hello-jupyterlab/archive/refs/tags/0.1.0.tar.gz
$ tar xvf project.tar.gz
$ rm project.tar.gz
$ mv hello-jupyterlab-0.1.0 my_jupyterlab_project

Next, we need to convert this project folder into a Git repository; here you will follow the steps appropriate for your Git remote repository (or just run the command git init in the project folder).

We perform a one-time setup of the project by running the following command within our newly created local Git repository.

$ pyenv install
$ python3 -m venv .environment
$ source .environment/bin/activate
$ pip install pipenv==2024.4.1
$ pipenv install

We can now start using JupyterLab by running the following command within our local Git repository.

$ jupyter lab

New Session Workflow

Later when we have a new terminal session, we can start using JupyterLab by running the following command within our local Git repository.

$ source .environment/bin/activate
$ jupyter lab

Make Changes Workflow

By default, JupyText does automatically not pair .py files into .ipynb (notebook) files (it does work the other way around though). Right-click the .py file and select Open With > Notebook. Going forward, these files will remain paired as long as the local Git repository exists.

We make a change to a notebook.

Choosing the Git tool, we can see which file(s) are changed.

Pressing the diff button on a changed file, shows the changes to the right.

Pressing the Stage this change (+) button on a changed file, stages it to be committed.

We supply a commit message and press the COMMIT button.

The small icon with the red dot in the top right of the Git pane indicates that the local Git repository is ahead of the remote Git repository; clicking it pushes the changes to the remote Git Repository.

Add Package Dependency Workflow

Undoubtedly, we will need to use Python packages in our notebooks. For example, one handy package is tqdm as JupyterLab is often used to run long operations (say for AI model training).

Instantly make your loops show a smart progress meter — just wrap any iterable with tqdm(iterable), and you’re done!

From the Launcher window, we select Terminal to open up a terminal console.

Just as we do in the new session workflow, we need to activate the virtual environment (has to be run from the root folder of our local Git repository).

$ source .environment/bin/activate

Next we need to know what the latest version of our package, here tqdm, is. The output today shows the latest version to be 4.67.1.

$ pip index versions tqdm

We then install tqdm using the following command.

$ pipenv install tqdm==4.67.1

Here you can see this makes changes to our Pipfile and Pipfile.lock.

note: It is important to always use this workflow to add package dependencies, even JupyterLab extensions, to our project. Using the JupyterLab Extension Manager tool to install package dependencies will not update the Pipfile and Pipfile.lock files which is how we are managing the project’s package dependencies.

Now we can use the package in our notebook.

That is about it; enjoy.