If we want our code to be reused it needs to be mobile. We need to be able to use and update it in different places. We also need others to be able to find it, download and install it. This process needs to be clear and simple regardless of dependencies, versions and OSs (with in the capabilities of the code).
To do this we want to be on the main distribution systems for code and packages, GitHub and PyPI. These are sources people are already used to using for download and installation. For instance we can download code from GitHub to extend or edit it by cloning:
git clone https://github.com/faulknerrainford/Example.git
and we can install a well built package from PyPI with:
pip install example
While being able to clone from a git repository can be set up simply by creating one and making it public installation from PyPI has slightly more requirments. This can make it seem complicated and unavailable to research software packages. The correct set up and tools though exist to make this process as simple as the installation and use of a single piece of software.
In python we can make use of the Poetry software to set up our project to build as a package from the start. It will then handle our dependencies, versions, build and distribution straight to PyPI, or our own server, for us.
Their own guides and documentation as well as download can be found on their website: https://python-poetry.org/
Here we will cover some of the basic usage of poetry and git including uploading to GitHub and PyPI.
Poetry can assist you from the start of your project to its distribution and even on going maintenance. To start with it will set up your project folder to encourage good file management and packaging. This works best from a blank start but can also be added to an existing project.
We can install poetry either by downloading from their website or using pip:
pip install poetry
Remember to do this locally if your working with a virtual environment. We can then set up a new project with:
poetry new Example
Which will require a set of questions answered in relation to dependencies and version. It will then generate a file structure:
Example
├── pyproject.toml
├── README.md
├── example
│ └── __init__.py
└── tests
└── __init__.py
This has the required README.md and the pyproject.toml which will allow us to build and publish. We have a source directory named for the package and a tests directory. This is a great structure for reminding us to write tests which are important for published packages to ensure correctness. Each of these has a __init__.py file to make it a python package and allow us to import modules and classes between files easily.
For an existing project Poetry does not give us a file structure but will set up the pyproject.toml to manage versions, dependancies and allow us to build the project. This can be done with Poetry installed by navigating to the project directory and running:
poetry init Example
Poetry tracks and installs dependencies for you. It makes it simpler to develop on multiple computers by allow the project to install all dependencies it needs even with out a full build. We can tell Poetry about dependencies we know we're going to need at set up. However it is common for new dependencies to be added during development. We add these to all relevant files using:
poetry add numpy
We add these so that new users can make sure they have all the correct packages needed to use your package. To make this simpler Poetry will sort it for you with:
poetry install
This will install all dependencies at the versions listed when they were added to Poetry.
Version control is important but versioning is correct as well. This is one place where often different files get out of sync. By using Poetry for versioning we can keep the consistent version in a single place. Poetry supports PEP 440 versioning using major, minor, patch and release note versioning.
Major - Big changes such as new features in the stable version of the code.
Minor - Small improvements in the stable version of the code
Patch - correction or bug fix
Release Notes - normally indicating stages of release
alpha - small limited release for checking
beta - wider release for general testing and reviews
None - an actual release for the general public
We can update it with a new version string:
poetry version 1.2.0a0
or with a bump rule such as a patch or major:
poetry version patch
For more on versioning see: Versioning
When we build with Poetry we get a source and wheels archive. Currently it only supports pure wheels. If want to just build we can run:
poetry build
Before we can publish to PyPI we need to set up an account which can be done here: https://pypi.org/
We then need to connect Poetry to this account.
poetry config pypi-token.pypi my-token
This is done with an API token you can generate from your PyPI account under Account Settings -> API tokens.
We can then publish with
poetry publish
If it is the first time its published Poetry will register the package for you. We can combine this with build:
poetry publish --build
Or if you want to check everything first without uploading you can do everything else with a dry-run
poetry publish --dry-run
The use of a git repository is recommended. In particular git is a fairly ubiquitous technology. It is important that version control is always used properly. Its only useful if it is up to date. For more information on version control and git in particular and its usage see Version Control.
For managing you repository and making it publicly available to be cloned by other users GitHub is a great free service. In particlar GitHub provides a feature called GitHub pages which provides a website to go with your repository. This can work well with Sphinx to providing API documentation for your project.