Create Your Custom, private Python Package That You Can PIP Install From Your Git Repository
Share your self-built Python package using your git repo.
Hands-on Tutorials
Create Your Custom, private Python Package That You Can PIP Install From Your Git Repository
You’ve created some convenient script that you would like your coworkers or others to use. In many companies, code like this is copied and emailed to one another. Although email is a very accessible tool to share code, we’re not living in the 90s anymore so let’s distribute your code in a smart way.
This article tackles the problem described above in two steps: packaging and distribution. First, we’ll focus on transforming your code into a python package so that people can easily install it. Then we’ll put the package into a repository (like on Github or Bitbucket) so people have access to it. At the end of this article you:
- understand the requirements of a Python package
- are able to build a Python package or transform an existing project into a package
- are able to pip install the self-built package from the repository
- are able to update your package
Let’s code!
Goals and preparations
We work in a company that performs analytics for restaurants. The owner of a restaurant collects data about guests, meals, and prices. They send us a data file with a specific question like, “What type of guest eats the canard a l’orange?”, “Is our pumpkin soup overprices?”, and “Do we see an increase in customers since we’ve lowered prices on deserts?”
In my years of working here I’ve noticed I use a lot of the same code, just copying it over from previous assignments. Our goal is to create a “Toolbox” package that contains some very generic pieces of code that my coworkers and I can easily install with pip. Then, every time one of us thinks of another handy function, we can add it and update our package.
In order to achieve this, we’re going to first package our existing code in a Python package. Then we’ll focus on distributing the package so it’s available to my coworkers.
Packaging our code
First, we’ll take our functions and create a Python package out of it. This is necessary if you want to use pip for the installation. You can check out all of my folders and files in this repository.
We’ll walk through creating one in the steps below. Check out the article above on how to create a public Python package
1. Create a venv
Create a virtual environment and add a gitignore, otherwise, we’ll create an unnecessarily large package.
2. Create package folder
Create a folder with the name of your package. In my case, this is “toolbox”.
This will be the package we’re installing. We’ll create some files in the package folder:
- toolbox/functions.py
This file will hold some of the functions that we want to share. I’ve included 3 functions:listChunker
,weirdCase
andreport
. - toolbox/__init__.py
This will tell Python that the toolbox folder is a python package. This file can also be used to import functions so that we canimport listChunker from toolbox
in addition tofrom toolbox.functions import listChunker
. Creating this file is required but the content is optional
3. Create setup.py
This file is required for telling pip what your package needs for it to be installed. Let’s take a look at the setup.py that I’ve used.
Below we walk through the setup file lines that need a little more explanation.
- Line 3: load README.md in a variable called long_description. This is optional.
- Line 7: Give our package a name. Must match your package folder name
- Line 8. Which is the version of our package. Pip uses this version to see if packages need updating so make sure to increment it if you want users to be able to update
- Line 12 and 13: loads the README.md from line 3; line 13 indicates the format of the readme.
- Line 14: URL to your repo
- Line 15: optionally list some handy URLs
- Line 18: how can users use your package? Check choosealicense.com
- Line 19: list of all packages that need to be built: make sure this matches your package folder name
- Line 20: list of packages that your package relies upon. Even though none of my functions use requests I’ve decided to include it for demonstration purposes. Including a package here makes sure that when pip installing the toolbox package, requests gets installed first so that toolbox can use it.
4. Other optional files
I’ve decided to include a README.md and a LICENSE file. These are simple text files that are not really required but a very nice addition.
Our repository is complete! Let’s find out how to distribute it!
Distributing our code via GitHub
Now that our package is created we can use a repository for distribution. First, we’ll create the repository and use it to pip install our package. Finally, we’ll update our package after we’ve changed the source code.
First, create a repository. You can do this on any platform that uses Git (GitHub, BitBucket, etc). Then add all your files, make sure to gitignore unnecessary files, and push to the repo.
Pip install
Copy the URL of your repository. You can pip install your package with this URL like this:pip install git+https://github.com/mike-huls/toolbox.git
That’s it! Easy isn’t it? Also note that you can install from both a public (like toolbox) and a private repository!
Update your package
Imagine my coworker has come up with a new function and decided to commit it to the repository. I can use pip to update my package. Every time I call pip install git+https://github.com/mike-huls/toolbox.git
pip checks the version number in the setup.py file. If my coworker remembered to increment the version then my package gets updated. Easy!
Including our package into a Docker container
There is a very easy trick to use your package in Docker:
Other advantages
GitHub offers a place to document issues, have a nice readme on the ‘front page’, and even offers a wiki for when your package requires even more explanation.
Conclusion
As we’ve seen in this article, combining the power of Python packaging and Git offers a lot of advantages:
- Easy distribution, installation, and updates from one central source (one source of the truth)
- Version control on our package and the ability to collaborate
- Ability to update the package once it’s modified
- Ability to pip install and update packages from a private repository
I hope I’ve clarified a lot of the process of creating and distributing Python packages. If you have suggestions/clarifications please comment so I can improve this article. In the meantime, check out my other articles on all kinds of programming-related topics like these:
- Multi-tasking in Python: speed up your program 10x by executing things simultaneously
- Create a fast auto-documented, maintainable and easy-to-use Python API in 5 lines of code with FastAPI
- Python to SQL — UPSERT Safely, Easily and Fast
- Create and publish your own Python package
- Create Your Custom, private Python Package That You Can PIP Install From Your Git Repository
- Virtual environments for absolute beginners — what is it and how to create one (+ examples)
- Dramatically improve your database insert speed with a simple upgrade
Happy coding!
— Mike
P.S: like what I’m doing? Follow me!