Modern Python developers use virtual environments, or “venvs,” to keep their projects and dependancies separate. One of the downsides of a virtual environment, though, is the size of a project directory. Each venv can take 10MB or more of disk space — and that’s just the base install, not including the libraries installed into the project. But now there is a language feature, and a package management system to go with it, that can cut down the size of your project’s footprint.
As of Python 3.8, Python has a feature that can automatically recognize the presence of a subdirectory in a project that stores packages associated exclusively with that project. Codified in PEP 582, Python allows a __pypackages__ directory to contain version-specific editions of packages that can be imported before packages from the base install of Python, or even a venv.
Up until recently, most Python developers made use of PEP 582-style package storage only “by hand.” But a recently developed package management tool, PDM — short for Python Development Master — lets you install packages to a project using the PEP 582 storage guidelines. The result is a project that is smaller, more portable, and less awkward to deal with.
Setting up PDM
PDM installs in Python 3.7 or higher. It’s best to install PDM into the user directory accessed by your Python installation, rather than in the Python installation itself. The PDM documentation explains how to do this. Alternatively, pip install --user pdm is a reliable, automatic way to get the same result.
Note that one crucial part of setting up PDM is enabling the use of PEP 582 behaviors. This can be done automatically, or by manually modifying PYTHONPATH.
Once you've installed PDM properly, you should be able to run the command pdm on the command line. If you're dealing with multiple Python installations on Windows, you can (and should) use py <version> -m pdm to trigger PDM from the proper Python version.
Running a project that uses PDM-installed modules should be transparent. Just execute Python files using the interpreter that PDM was installed into.
Managing project dependencies with PDM
To set up a project to use PDM, go to the root of the project directory in the console and type pdm init. You’ll be asked a few questions about the project, after which your project will have a pyproject.toml file and a __pypackages__ directory.
If you're using source control (e.g., Git), be sure to include pyproject.toml in the repository, but exclude the __pypackages__ directory and the .lock files generated by PDM.
To add dependencies to the project, use pdm add <dependency>. You can list more than one dependency, and you should specify them using the standard in PEP 508. In other words, they should look like the dependency entries in a standard requirements.txt file. To remove a dependency, use pdm remove <dependency>.
Two things worth keeping in mind when adding or removing dependencies:
- Each time you add or remove a dependency, PDM recomputes the dependency graph for the project, which may take some time.
- When you remove a dependency, any dependencies that depend solely on the one you remove are not automatically removed, but they can be cleaned up semi-automatically. (More on this later.)
To list the dependencies in a project, use pdm list. Or you can use pdm list --graph to show dependencies in a tree, so that you can see at a glance which packages depend on which other packages.
To ensure all dependencies are installed as needed, you can use one of two commands:
pdm install, the more commonly used of the two, should be your first choice. It will create the lock file — the description of package versions to use with the project — and then install the dependencies from that.pdm syncuses a slightly different strategy. If a lock file doesn’t already exist, it will throw an error; otherwise, it will install the dependencies listed in the existing lock file. This is useful if you want to install only from an existing lock file, without recomputing dependencies.
Managing optional and development dependencies with PDM
By default any dependencies installed to a PDM-managed project are installed without indicating their function in the project. For instance, if you install the black code formatter using pdm add black, PDM won’t identify it as a dependency needed exclusively for development versus one used for runtime.
If you want to tag dependencies for specific uses, you can use the -dG (“dependency group”) option, along with a group name, to group them. For instance, we could use pdm add -dG dev black to install black to a subgroup of dependencies labeled dev.
You can also specify that a given dependency should be installed only for production use by using pdm add -d <dependency>. For instance, if we used the module regex in the production application, but not in development, we would add it with pdm add -d regex.
Updating and cleaning PDM dependencies
PDM-controlled dependencies can be updated all at once by typing pdm update, updated individually by using a package name (pdm update <package>), or updated as a group by using the group name (pdm update -G <groupname>).
By default, packages pinned to a specific version will be kept to that version. You can forcibly update pinned packages to their latest versions by passing the --update-eager flag.
If you want to remove packages that are no longer needed, run pdm sync --clean.
Caveats when using PDM
PEP 582 is still relatively new. As a result, using PDM comes with a couple of caveats. First, many IDEs do not automatically detect or use __pypackages__ directories in a project. The PDM documentation explains how to configure various editors to recognize the directory for a project.
Second, some packages with archaisms in their internal build and install processes may not set up properly with PDM. If that happens, you may need to manually install them to your __pypackages__ directory with pip install -t, and they won’t be manageable through PDM.