supakeen's homepage
The Fedora 45 Sausage Factory
This is a walkthrough of how Fedora turns source code and packages into the
artifacts you download and install. It follows the a package from a packager’s
git push to a composed release: ISOs, cloud images, container images, and OSTree
deployments.
The walkthrough describes how the Fedora ‘sausage’ is created as of Fedora 45,
things change all the time; I hope to have time to update this document every
cycle or every few cycles of Fedora releases so there’s both history and people
can find up to date information.
In fact this document is currently still living as Fedora 45 has not been released
yet so this post likely will get updated until the release. There’s a few change
proposals in flight that change some of the bits and bobs here. Specifically how
the boot.iso is produced which is a decently large part of this post.
Thus now is also the right time to let me know if I got anything wrong, please
e-mail me at cmdr@supakeen.com if you find any errors.
Starting Line: dist-git
Things starts with a packager pushing a commit to a package.
Fedora stores the source definitions of every package in individual Git
repositories at src.fedoraproject.org. Each
repo contains an RPM spec file, any downstream patches, and a sources file
that points to upstream tarballs stored in a separate lookaside cache. The
large binary files stay out of Git; the other files exist inside of Git and
thus have full version control.
Packagers usually interact with these repos through fedpkg, a CLI that wraps
the common operations: cloning repos, uploading source tarballs, submitting builds,
and creating updates. The important thing about fedpkg build is what it
actually does: it constructs a URL pointing at a specific commit in the Git
repo and hands that to Koji, the build system. The build is fully reproducible
from that commit hash.
Branches map to releases: rawhide for development, f44 for Fedora 44.
The hosting is split between Pagure (src.fedoraproject.org) and Forgejo
(forge.fedoraproject.org), with an ongoing
migration from the former to the latter.
Packagers can do work without using fedpkg, but for the purposes of this
post it’s easier to assume fedpkg is in use since it abstract away some of
the bits that are irrelevant to the reader.
Building Packages: Koji
When fedpkg build submits that Git URL, Koji takes over. Koji is Fedora’s
build system. It has been around since Fedora 7, and it builds essentially
everything.
Koji follows a hub-and-spoke architecture. The hub is a passive XML-RPC server
that sits in front of a PostgreSQL database. Builder daemons poll the hub for
work, create a fresh Mock
chroot environment for each build, run the build, and upload the results. Every
build starts from a clean room. You can never get a different result because
someone installed something on the builder last week.
The organizational model is built around tags. A tag is a named collection
of builds. Build targets map an incoming build request to two tags: a build tag
(which defines the buildroot, the packages available during the build) and a
destination tag (where the finished build lands). Tags support multiple
inheritance, so you can layer a Fedora 44 tag on top of a base tag without
duplicating everything.
Koji doesn’t just build RPMs. Through its plugin system and content generators,
it also orchestrates image builds: Kiwi images via the kiwiBuild task type,
Image Builder artifacts via imageBuilderBuild, and OSTree composes via
runroot tasks. We’ll get to those.
Gating Updates: Bodhi
A fresh RPM build sitting in Koji doesn’t automatically reach users. For
branched releases (anything that isn’t Rawhide), it goes through
Bodhi, Fedora’s update management system.
Bodhi gates the release of updates through a feedback and testing cycle. A
packager submits an update containing one or more builds. That update goes
through a sequence of states: pending, testing, stable. Users and automated
tests provide karma (+1 / -1). When an update hits +3 karma or spends enough
days in testing, it will be automatically pushed to stable. If an update hits
+1 karma the maintainer can manually push to stable (+2 for critical path).
Behind the scenes, Bodhi manages all of this through Koji tags. When an update
moves from testing to stable, Bodhi moves the build from the
f44-updates-testing tag to the f44-updates tag. It then invokes Pungi to
compose the update repository, the actual yum/dnf repo that users pull from
when they run dnf upgrade.
Critical path packages, the ones your system needs to boot and function, get
stricter requirements: 14 days in testing instead of 7, and more karma needed.
Bodhi also integrates with Greenwave and ResultsDB for CI test gating, so
automated test failures can block an update from reaching stable.
For Rawhide, these processes are configured differently. Non-critical path
packages go stable immediately unless there’s a gating policy (either because
the package is in the critical path or because the package itself has one) it
still applies and an update can be held back until things pass.
Composing a Release: Pungi
Individual RPMs, even with Bodhi’s gating, are just packages. Turning them into
something you can download and install, an ISO, a cloud image, a repository,
is the job of Pungi.
Pungi is the compose orchestrator. It doesn’t do much of the heavy lifting
itself. Instead, it coordinates the tools that do, ensuring everything is built
from the same consistent set of packages.
The name is a reference to the pungi,
a snake-charming instrument. It “charms” Anaconda,
Fedora’s installer. The pun has survived since 2006.
A compose starts when pungi-koji runs, either triggered by cron for nightly
Rawhide composes, or manually for milestone releases. It loads a configuration
file (for Fedora, that’s fedora.conf in the
pungi-fedora repository)
and runs through a sequence of phases.
Freezing the Package Set
The first real work Pungi does is snapshot the set of packages from a Koji tag.
This is the Pkgset phase, and it’s critical: every subsequent phase works
from this frozen set. If someone submits a new build to Koji while the compose
is running, it won’t sneak into the compose. The tag-based snapshot makes the
entire compose auditable. You can always determine exactly which versions of
which packages ended up in any given compose.
Deciding What Goes Where: comps and variants
With the packages frozen, Pungi needs to figure out which packages belong in
which product. This is where two XML inputs come in.
comps (fedora-comps) defines package groups, collections of related
packages like gnome-desktop, server-product, or core. Each package in a
group has a type: mandatory, default, optional, or conditional. This is the
same grouping system that powers dnf group install and Anaconda’s Software
Selection screen.
variants XML (variants-fedora.xml in pungi-fedora) defines the products
in a compose: Everything, Server, Workstation, KDE, Silverblue, and so on.
Each variant lists which comps groups it includes and which architectures it
supports. Most variants in modern Fedora are is_empty="true", meaning they
produce only images (via Kiwi, Image Builder, or OSTree) using packages from
the Everything variant’s repository. Only Everything and Server actually run
the package gathering pipeline.
Building Boot Images
The Buildinstall phase runs lorax to create
boot.iso, the image that boots the Anaconda installer. Lorax installs a
minimal package set into a temporary root, configures the Anaconda environment,
compresses it into install.img, and wraps it with a bootloader into an ISO.
One lorax task runs per variant-architecture combination, typically in a Koji
runroot environment for isolation.
The resulting boot.iso is the foundation for installer ISOs. It doesn’t contain
the packages themselves. Those get layered on top in the next step to produce
a dvd.iso.
Producing ISOs
The Createiso phase takes the boot.iso from Buildinstall, overlays packages
and repository metadata on top of it using xorriso, and produces the final
installer ISOs. For Fedora’s production compose, this means primarily the
Server dvd.iso. Most other variants get their media from Kiwi or Image Builder
instead.
Building Images with Kiwi
Kiwi is the primary image-building tool
in Fedora’s compose. It builds cloud images (AWS, Azure, GCP, generic), Vagrant
boxes, container base images, WSL images, and live ISOs for most desktop spins.
Kiwi was originally created at SUSE around 2005 and is a two-step builder:
prepare a root filesystem by installing packages, then package it into the
target format. Image definitions are XML files validated against a RELAX NG
schema. Fedora’s definitions live in the
fedora-kiwi-descriptions
repository, organized under teams: teams/cloud/, teams/kde/,
teams/spins/, teams/wsl/.
Kiwi integrates with Koji through a built-in plugin providing the kiwiBuild
task type. The parent task checks out the image description from SCM, parses
the XML, and spawns one subtask per architecture. Each subtask creates a Mock
buildroot, replaces all repository definitions in the XML with Koji-controlled
repos (so the build uses the compose’s frozen package set), runs
kiwi-ng system build, and uploads the results.
Building Images with Image Builder
Image Builder handles the
ostree-based and bootc-based artifacts: Atomic Desktop ISOs, Fedora IoT,
Fedora Minimal, and boot.iso.
Under the hood, Image Builder is built on top of
osbuild, a pipeline-based build engine
that processes JSON manifests. Each manifest describes a directed acyclic graph
of pipelines, and each pipeline runs sequential stages, small executables that
do one thing (install packages, configure a bootloader, create a partition
table, embed an OSTree commit). There are currently 176 stages. Stages run
in bubblewrap-sandboxed environments with filesystem and network isolation.
The image-builder CLI is stateless: it reads distribution definitions, resolves
package dependencies, generates an osbuild manifest, and runs osbuild directly.
Image Builder integrates with Koji through
koji-image-builder, a
separate Koji plugin providing the imageBuilderBuild task type. Each build
runs image-builder inside a Mock buildroot on a Koji worker.
Building Atomic Desktops: rpm-ostree
Fedora Silverblue, Kinoite, Sway Atomic, Budgie Atomic, and COSMIC Atomic are
all built differently from traditional package-based variants. Instead of
shipping a pile of RPMs for the user to install, these variants ship a
versioned, checksummed filesystem tree, an OSTree commit, that the client
replicates atomically.
rpm-ostree is the tool that makes this
work. It’s a hybrid image/package system, combining
libostree (atomic OS tree deployment)
with RPM (package management via libdnf).
On the compose side, rpm-ostree compose tree takes a treefile (a YAML
specification listing packages, configuration, and post-processing scripts) and
a set of RPM repositories, and produces an OSTree commit. Pungi’s OSTree phase
invokes this command in a Koji runroot environment for each Atomic Desktop
variant.
The treefiles live in the
workstation-ostree-config
repository. They use a layered include system:
silverblue-ostree.yaml includes silverblue-common.yaml, which includes
common.yaml, which conditionally includes fedora.yaml and architecture-
specific files. The comps-sync.py script bridges the gap between comps
groups and treefile package lists. It parses the comps XML, extracts the
default and mandatory packages from groups like gnome-desktop and
workstation-product, and writes them into generated YAML files. This way,
the desktop environment packages stay in sync with what comps defines, while
the treefile adds ostree-specific packages (bootupd, composefs) and applies
exclusions for packages that are Flatpak’d or incompatible with ostree.
Compose Metadata: productmd
Every compose produces a set of metadata files that describe what’s in it.
These are defined by productmd,
a Python library that standardizes the format.
The key files are:
composeinfo.json, release identity, variant definitions, filesystem paths
images.json, every image with its type, format, checksums, and size
rpms.json, every RPM organized by variant and architecture
.treeinfo, per-variant metadata consumed by Anaconda to find installation
trees
Pungi creates these as it goes. At the start, it generates a compose ID (e.g.,
Fedora-Rawhide-20260723.n.0). Each image-producing phase adds Image objects.
The Gather phase populates the RPMs manifest. At finalization, everything gets
written out.
Downstream consumers rely on these files heavily. Anaconda reads .treeinfo
to discover where to find packages and boot images during installation. Bodhi
inspects compose metadata for update management. openQA reads images.json
to locate images for testing. Mirror management tools query the metadata to
decide what to sync.
Testing: openQA
Once a compose finishes, openQA picks it
up for automated testing. openQA is an automated test system for operating
systems, originally developed by SUSE and adapted for Fedora by Adam
Williamson and the Fedora QA team.
[Adam
Williamson](https://www.happyassassin.net/)
openQA boots the compose’s images in virtual machines and runs through test
scenarios: installation workflows (interactive, kickstart, various disk
layouts), desktop functionality, upgrade paths, and basic system validation.
Tests are written as a combination of test scripts (“needles” for visual
matching and Perl test modules for interaction) and are maintained in the
os-autoinst-distri-fedora
repository.
Every nightly Rawhide compose and every milestone compose (Beta, Final)
triggers an openQA run. Results feed into the release validation process,
and blocking bugs found by openQA can hold a release. The system also tests
update composes from Bodhi, catching regressions before they reach users.
The Sausage
Here’s how the phases flow in a typical Fedora compose:
The barriers between steps matter. Step 3 must complete before step 4 starts,
because Createiso needs boot.iso, the image phases need package repos, and
ImageBuilder needs OSTree commits.
Governance: Changes Process
Major modifications to Fedora, whether new tools, default changes, or mass
rebuilds, go through the Changes process.
This is the governance mechanism that tracks proposals like “use Kiwi for cloud
images” or “switch boot.iso to Image Builder.”
Changes come in two flavors. System-Wide changes affect critical path
packages or system defaults and need detailed proposals, contingency plans,
test plans, and FESCo approval. Self-Contained changes are scoped to
packages the proposer owns and get lighter review.
The lifecycle is: draft a wiki page, submit it, get reviewed by the Change
Wrangler, get voted on by FESCo (the 9-member elected engineering steering
committee), implement in Rawhide, and meet completion checkpoints tied to the
release schedule. Miss the Beta Freeze deadline and your change gets deferred
to the next release automatically.