How to manage Python projects with Poetry

There should be one—and preferably only one—obvious way to do it. —Tim Peters, Zen of Python

While that quote is excellent, Python doesn’t always adhere to it in principle. One area where Python has fallen short of the ideal is project management. For too long, managing Python projects involved a mishmash of tools and methodologies. However, a few clean and simple toolsets are emerging. One of them is Poetry.

Poetry brings to Python the kind of all-in-one project management capability that Go and Rust have long enjoyed. Poetry offers deterministic dependencies with specific package versions, so projects build consistently in different places. Poetry also makes it easier to build, package, and publish projects and libraries to PyPI, so that others can share the fruits of your Python labors.

This article walks through using Poetry for Python development projects. You'll learn how to set up Poetry, how to use Poetry to configure project dependencies and virtual environments, and how to avoid some of the pitfalls that come with Poetry’s unique way of doing things.

How to set up Poetry in Python

Poetry is deliberately unlike other Python dependency and project management tools, beginning with setup. Instead of using pip, Poetry uses a custom installer. The installer adds the Poetry application to your user’s profile directory, so it can be used with any Python installation in your system, present or future.

Although you can use pip install poetry to install Poetry in a specific Python installation, this isn’t recommended for two reasons. First, it might conflict with other system files. Second, it makes using Poetry consistently with different versions of Python and different virtual environments difficult.

Create a Poetry-managed Python project

Once you have Poetry installed, you can create a new Poetry-managed project directory simply by typing poetry new <project_name>. This command creates a subdirectory named <project_name> and populates it with a project scaffold.

The Poetry project scaffold includes the following:

• pyproject.toml: The definition file for the project. Poetry manages this definition for you. If you know what you’re doing, you can edit the file directly, but most of the time you won’t need to. Developers should use the pyproject.toml standard for all new Python projects going forward, whenever possible.
• README.rst: An empty README file in ReStructuredText format, the file format used for Python documentation. (Nothing says that you must use .rst format for your docs; you can use Markdown for simpler cases.)
• tests: A subdirectory with scaffolding for unit tests. If you aren’t in the habit of writing tests for your new projects, you should be!
• A subdirectory with the project name that contains the code for your project.

Manage Python virtual environments in Poetry

Probably the first thing you’ll want with a new Poetry project is a Python virtual environment. True to form, Poetry has its own distinct way of handling virtual environments. Instead of placing virtual environments inside the project directory, Poetry puts them in a centralized cache directory that varies according to the operating system:

• Unix: ~/.cache/pypoetry/virtualenvs
• MacOS: ~/Library/Caches/pypoetry/virtualenvs
• Windows: C:\Users\<username>\AppData\Local\pypoetry\Cache\virtualenvs or %LOCALAPPDATA%\pypoetry\Cache\virtualenvs

The advantage to Poetry’s approach is the ability to share virtual environments across projects whenever it makes sense. But doing so requires altering your work habits.

To set up a virtual environment in Poetry, go to the project's directory and type poetry env use python (or poetry env use py on Windows). Poetry will create a new virtual environment, store it in the cache directory, and display a randomly generated name for the virtual environment.

For added convenience, Poetry also installs any dependencies listed in the project’s pyproject.toml file. You will find this super useful should you ever want to copy a Poetry project from somewhere else and get it set up on your system. 

Note that if you run poetry env use python in a project directory that already has a Poetry-assigned virtual environment, Poetry will activate that virtual environment in the context of the CLI session.

Next, you'll want to use your Poetry-managed virtual environments to work with your IDE. Many Python IDEs now automatically detect the presence of a Poetry environment. Visual Studio Code, for instance, can do this, although if you set up the environment in VS Code's terminal, you may need to reload the project window.  (Type >reload window in the command palette to find this command.) 

If you don’t want Poetry to manage your virtual environments, you can disable the behavior with this command:

poetry config virtualenvs.create false

Just keep in mind that disabling this feature defeats one of Poetry's big advantages.

Add dependencies to a Python project in Poetry

You can add dependencies in a Poetry project in one of two ways: as a general dependency used by the project in all circumstances, or as part of a dependency group.

Dependency groups let you create sets of dependencies within a project, each for different uses. For instance, you could create a dependency set specifically for developers with coding tools like black, mypy, or docutils.

To add dependencies to a project generally, use poetry add <dependency_name>. To add dependencies for a given group, use poetry add --group <group_name> <dependency_name>.

Note that the poetry add command works much like pip install in that you can specify either a package name or a Git path (e.g., git+https://github.com/developer/project.git#branchname). You can also configure Poetry to use private repos.

Once dependencies are resolved and installed, Poetry creates a file named poetry.lock in the project directory. This file is a manifest of all of the downloaded dependencies, and should be saved along with the rest of your project. Then anyone who pulls a copy of the project from source control will get the same versions of all required packages.

Now you’re ready to begin the project in earnest. All you have to remember from this point forward is to use Poetry—and only Poetry—to manage all dependencies and virtual environments for the project.

Removing a Poetry virtual environment

By default, a Poetry-managed virtual environment lives in a central directory on the system, away from any projects associated with it. As a result, deleting a project directory does not also delete the corresponding virtual environment, as Poetry doesn't track such things automatically. You have to remove centrally managed virtual environments yourself.

To do this, go to the root directory of the Poetry project in question and issue the command poetry env remove python (or poetry env remove py, for Windows). Note that if you've overridden Poetry's default behavior and placed the environment in your project directory, you can just delete the environment manually, but again doing this defeats one of Poetry's best features.