Create Your Custom, private Python Package That You Can PIP Install From Your Git Repository

Share your self-built Python package using your git repo.

Create Your Custom, private Python Package That You Can PIP Install From Your Git Repository
Distribute you code like this bee distributes pollen (image by Dimitry Grigoriev on Unsplash)

Hands-on Tutorials

Create Your Custom, private Python Package That You Can PIP Install From Your Git Repository

You’ve created some convenient script that you would like your coworkers or others to use. In many companies, code like this is copied and emailed to one another. Although email is a very accessible tool to share code, we’re not living in the 90s anymore so let’s distribute your code in a smart way.

This article tackles the problem described above in two steps: packaging and distribution. First, we’ll focus on transforming your code into a python package so that people can easily install it. Then we’ll put the package into a repository (like on Github or Bitbucket) so people have access to it. At the end of this article you:

  • understand the requirements of a Python package
  • are able to build a Python package or transform an existing project into a package
  • are able to pip install the self-built package from the repository
  • are able to update your package

Let’s code!


Goals and preparations

We work in a company that performs analytics for restaurants. The owner of a restaurant collects data about guests, meals, and prices. They send us a data file with a specific question like, “What type of guest eats the canard a l’orange?”, “Is our pumpkin soup overprices?”, and “Do we see an increase in customers since we’ve lowered prices on deserts?”

In my years of working here I’ve noticed I use a lot of the same code, just copying it over from previous assignments. Our goal is to create a “Toolbox” package that contains some very generic pieces of code that my coworkers and I can easily install with pip. Then, every time one of us thinks of another handy function, we can add it and update our package.

In order to achieve this, we’re going to first package our existing code in a Python package. Then we’ll focus on distributing the package so it’s available to my coworkers.


Packaging our code

First, we’ll take our functions and create a Python package out of it. This is necessary if you want to use pip for the installation. You can check out all of my folders and files in this repository.

Create and publish your own Python package
A short and simple guide on how to pip install your custom made package

We’ll walk through creating one in the steps below. Check out the article above on how to create a public Python package

1. Create a venv

Create a virtual environment and add a gitignore, otherwise, we’ll create an unnecessarily large package.

Virtual environments for absolute beginners — what is it and how to create one (+ examples)
A deep dive into Python virtual environments, pip and avoiding entangled dependencies

2. Create package folder

Create a folder with the name of your package. In my case, this is “toolbox”.
This will be the package we’re installing. We’ll create some files in the package folder:

  • toolbox/functions.py
    This file will hold some of the functions that we want to share. I’ve included 3 functions: listChunker, weirdCase and report.
  • toolbox/__init__.py
    This will tell Python that the toolbox folder is a python package. This file can also be used to import functions so that we can import listChunker from toolbox in addition to from toolbox.functions import listChunker. Creating this file is required but the content is optional
Why Python is so slow and how to speed it up
Take a look under the hood to see where Python’s bottlenecks lie

3. Create setup.py

This file is required for telling pip what your package needs for it to be installed. Let’s take a look at the setup.py that I’ve used.

Below we walk through the setup file lines that need a little more explanation.

  • Line 3: load README.md in a variable called long_description. This is optional.
  • Line 7: Give our package a name. Must match your package folder name
  • Line 8. Which is the version of our package. Pip uses this version to see if packages need updating so make sure to increment it if you want users to be able to update
  • Line 12 and 13: loads the README.md from line 3; line 13 indicates the format of the readme.
  • Line 14: URL to your repo
  • Line 15: optionally list some handy URLs
  • Line 18: how can users use your package? Check choosealicense.com
  • Line 19: list of all packages that need to be built: make sure this matches your package folder name
  • Line 20: list of packages that your package relies upon. Even though none of my functions use requests I’ve decided to include it for demonstration purposes. Including a package here makes sure that when pip installing the toolbox package, requests gets installed first so that toolbox can use it.
Thread Your Python Program with Two Lines of Code
Speed up your program by doing multiple things simultaneously

4. Other optional files

I’ve decided to include a README.md and a LICENSE file. These are simple text files that are not really required but a very nice addition.

Our repository is complete! Let’s find out how to distribute it!


Distributing our code via GitHub

Now that our package is created we can use a repository for distribution. First, we’ll create the repository and use it to pip install our package. Finally, we’ll update our package after we’ve changed the source code.

Destroying Duck Hunt with OpenCV — image analysis for beginners
Write code that will beat every Duck Hunt high score

First, create a repository. You can do this on any platform that uses Git (GitHub, BitBucket, etc). Then add all your files, make sure to gitignore unnecessary files, and push to the repo.


Pip install

Copy the URL of your repository. You can pip install your package with this URL like this:pip install git+https://github.com/mike-huls/toolbox.git

That’s it! Easy isn’t it? Also note that you can install from both a public (like toolbox) and a private repository!

Create a fast auto-documented, maintainable and easy-to-use Python API in 5 lines of code with…
Perfect for (unexperienced) developers who just need a complete, working, fast and secure API

Update your package

Imagine my coworker has come up with a new function and decided to commit it to the repository. I can use pip to update my package. Every time I call pip install git+https://github.com/mike-huls/toolbox.git pip checks the version number in the setup.py file. If my coworker remembered to increment the version then my package gets updated. Easy!


Including our package into a Docker container

There is a very easy trick to use your package in Docker:

Use git submodules to install a private, custom python package in a docker image
This is a complex title but I swear it’s not difficult

Other advantages

GitHub offers a place to document issues, have a nice readme on the ‘front page’, and even offers a wiki for when your package requires even more explanation.


Conclusion

As we’ve seen in this article, combining the power of Python packaging and Git offers a lot of advantages:

  1. Easy distribution, installation, and updates from one central source (one source of the truth)
  2. Version control on our package and the ability to collaborate
  3. Ability to update the package once it’s modified
  4. Ability to pip install and update packages from a private repository

I hope I’ve clarified a lot of the process of creating and distributing Python packages. If you have suggestions/clarifications please comment so I can improve this article. In the meantime, check out my other articles on all kinds of programming-related topics like these:

Happy coding!

— Mike

P.S: like what I’m doing? Follow me!

Join Medium with my referral link — Mike Huls
As a Medium member, a portion of your membership fee goes to writers you read, and you get full access to every story…