Rolling out Repodata v3

Enjoy optional dependency groups (aka “extras”), conditional dependencies and package flags with the new “v3” repodata!

At prefix we have been driving one of the biggest changes to the Conda ecosystem since its inception: new repodata keys that allow a much more flexible selection of packages and their dependencies.

What is “v3”?

Every conda channel ships a repodata.json that lists the available packages and their metadata. Today’s format has essentially been frozen since conda’s early days: a flat list of packages, each with a single depends array and a build string that has to encode everything about a variant.

“v3” is the next revision of that format. It keeps the base package record but adds a few new keys so packages can express three things:

• extras, optional dependency groups a consumer can opt into (CEP 44),

• conditional dependencies, dependencies that only apply when a condition holds (CEP 43),

• flags, variant tags you can select on without pattern matching the build string (CEP 45).

A package advertises the revision with a repodata_revision: 3 marker. Clients that don’t understand v3 keep reading the base record and simply ignore the new fields.

Extras

If you come from the PyPI ecosystem, you will already be quite familiar with “extras” to select additional dependencies for a given Python package. For example, for sqlalchemy you can select database backends using sqlalchemy[postgres, sqlite], which automatically pulls in drivers for the two specified databases.

In conda-forge we had to work around this by creating additional packages (e.g. sqlalchemy-postgres and sqlalchemy-sqlite). Those packages then pin sqlalchemy and add the extra dependencies.

With “v3” we don’t need to create these empty packages anymore. We can express extras directly in the repodata! In a conda MatchSpec you request one or more groups with the extras bracket key:

sqlalchemy[extras=[postgres, sqlite]]

(The list items can be quoted or unquoted, so extras=["postgres", "sqlite"] works too.)

To define the extra groups in a recipe, add an extras mapping to the requirements section. Each key is a group name, and each value is a list of dependencies in the exact same syntax as run:

package:
  name: sciency
  version: 1.0.0

requirements:
  run:
    - numpy
  extras:
    plot:
      - matplotlib

    linalg:
      - numpy
      - scipy

    # groups can reference each other
    all:
      - sciency[extras=[plot, linalg]]

Group names follow CEP 44: lowercase letters, digits and ._+-, up to 64 characters. So plot, linalg and all are fine, but Plot or my group are not.

The same is possible in a pixi.toml when defining a package:

[package.extra-dependencies.plot]
matplotlib = "*"

[package.extra-dependencies.linalg]
numpy = "*"
scipy = "*"

[package.extra-dependencies.all]
sciency = { version = "*", extras = ["linalg", "plot"] }

And when you depend on such a package, request the groups you want with the extras field:

[dependencies]
sciency = { version = "*", extras = ["plot", "linalg"] }

Conditional Dependencies

Another use case that PyPI already supports is conditional dependencies. For example, you might want to add pywin32 only on Windows even though you publish only a single noarch package.

Because if is already taken in recipes, we settled on using when in a MatchSpec:

requirements:
  run:
    - numpy
    - pywin32[when="__win"]

You can put any MatchSpec in the when condition: a virtual package like __win or __cuda, a regular package with a version, or several of them combined with and, or, not and parentheses:

- pytorch[when="__cuda >=12 and python >=3.10"]

In Pixi you write these with TOML. The simplest form is a string, which takes exactly the same MatchSpec:

[dependencies]
pywin32 = { version = "*", when = "__win" }

For anything more than a version match, say a build string, use the expanded table form:

[dependencies]
# Only install `numpy` when `python` is at least 3.12.
numpy = { version = "*", when = { package = "python", version = ">=3.12" } }

And to combine conditions we use all (logical AND) and any (logical OR), each taking a list of any of the forms above:

[dependencies]
# Needs *both* a CUDA >=12 stack and python >=3.10.
some-gpu-helper = { version = "*", when = { all = ["__cuda >=12", "python >=3.10"] } }

Flags

Flags are a new way of selecting variants. Instead of having to use regexes or globs to match the build string, the “flags” mechanism makes it easy to select a variant based on named tags. This is most useful for compiled packages with multiple build variants.

To define the flags that your variant carries, add them to the flags list in the build section. Because flags are conditional and templated like any other recipe list, the cleanest way to emit a flag only for some variants is the if/then form:

package:
  name: pytorch
  version: 2.13.0

build:
  flags:
    - if: cuda_compiler_version != "None"
      then: gpu
    - blas:${{ blas }}

A flag is a short lowercase tag matching ^[a-z0-9_]+(:[a-z0-9_]+)?$, so it may have a single :-separated suffix, as in blas:mkl or blas:openblas. Flags that depend on a variant variable participate in the variant hash, so each flag combination gets its own build.

Now a downstream user can use a simple matcher to select only packages where the flag matches:

pytorch[flags=[gpu, blas:*]]

The * is a glob, so blas:* matches blas:mkl, blas:openblas, and so on. In Pixi TOML files this looks like:

[dependencies]
pytorch = { version = "*", flags = ["gpu", "blas:*"] }

This is a clear improvement over the only previous alternative, which was trying to select variants with globs in the build strings, e.g. pytorch=*=*cuda*.

What ends up in the repodata

A v3 build tags its index.json (and hence the repodata record) with the revision and the new fields:

{
  "repodata_revision": 3,
  "flags": ["gpu", "blas:openblas"],
  "extra_depends": {
    "plot": ["matplotlib >=3.8"]
  }
}

flags and extra_depends are omitted when empty. Conditional dependencies live inside the regular depends entries, carrying their when= predicate along.

Rollout in conda-forge

To build these packages you currently need to pass --v3 to rattler-build, which prevents v3 packages from entering conda-forge just yet. Without the flag, any recipe or MatchSpec that uses a v3-only field is rejected at parse time, and it never silently downgrades. Pixi, on the other hand, automatically detects whether you have any of the new fields set and turns v3 on for you. V3 fields work totally fine for any source package!

Older clients that don’t understand v3 will still install the base package, but they will silently miss conditional dependencies, flag-based selection and optional groups. So make sure your toolchain (recent rattler / pixi / conda) is v3-aware before you publish.

If you run your own channel on prefix.dev you can enable the upload of v3 packages by ticking the following box:

We are working on the last remaining CEP, clarifying the update strategy. Once this is finalized we will move v3 out of beta on our server and work with the community to bring v3 to everyone. Modern clients that do not understand v3 or future fields will be able to emit a warning once this update strategy is finalized.

Learn more

• rattler-build, V3 packages (beta): how to produce v3 packages, the full recipe fields and MatchSpec syntax.

• rattler-build, recipe reference: the requirements section, including extras and the v3 bracket keys.

• Pixi, package specifications: extras, flags and when in a pixi.toml.

• Pixi, dependency types: declaring extra-dependencies and conditional dependencies when building a package.

• CEP 45 (flags), CEP 44 (extras), CEP 43 (conditional dependencies).