Introduction to linux kernel development process


Written by Paolo Ciarrocchi, November 2005

                               ---- Overview ----

Linux kernel development process is quite complex and not very well documented
so I decided to try to summarize it in the hope to be useful for the community.
2.6.x kernels are the base stable releases released by Linus.
The highest numbered release is the most recent.
If regressions or other serious flaws are found then a -stable fix patch
will be released (see below) on top of this base. Once a new 2.6.x base
kernel is released, a patch is made available that is a delta between the
previous 2.6.x kernel and the new one.

                             --- 2.6.x kernels ---

2.6.x kernels are maintained by Linus Torvalds,
it's development is as follow:
- As soon a new kernel is released a two weeks windows is open, during
  this period of time maintainers can submit big diffs to Linus, usually
  the patched sited in -mm kernels for a few weeks.
  Preferred way to submit big changes is using GIT
  (more information about GIT at and
- After two weeks a -rc1 kernel is released and now is possible to
  push only patches that do not include new functionalities that could
  affect the stability of the whole kernel. Please note that a whole
  new driver (or filesystem) might be accepted
  after -rc1 because there is no risk of causing regressions with
  such a change
- A new -rc is released whenever Linus deems the current git
(the kernel's source management) tree to be in a reasonably sane state
  adequate for testing. The goal is to release a new -rc kernel every week.
- Process continues until the kernel is considered "ready", the
  process should last around 6 weeks ( 6 kernels per year should be
  released) an usually includes 4/5 rc releases.
There a numbers of tools used by the community to measure the quality and the
performance of a kernel.
A couple of examples are:
Copy and paste from the project description:
"We are a group of dedicated Linux kernel engineers taking on the challenge
  of testing the Linux kernel. In order to track performance, we are running
  a large set of benchmarks covering core components of the Linux kernel
  (virtual memory management, I/O subsystem, process scheduler, file system,
  network, device driver, etc). Benchmarks are run on a variety of platforms
  every week, testing the latest snapshot of Linus' git development tree.
  Comprehensive performance data from our tests will be hosted here for
  easy access."
  It's the official bugzilla instance used as an online kernel bug tracking.
  Users are invited to report all the bugs they hit using this tool.
But it's worth to mention what Andrew Morton wrote on lkml:
"Nobody knows when a kernel will be released, because it's released
  according to perceived bug status, not according to a preconceived

                     --- 2.6.x.y kernels, a.k.a -stable ---

Kernels with 4 digit versions are -stable kernels. They contain small(ish)
critical fixes for security problems or significant regressions discovered
in a given 2.6.x kernel.

This is the recommended branch for users who want the most recent stable
kernel and are not interested in helping test development/experimental

If no 2.6.x.y kernel is available, then the highest numbered 2.6.x kernel is
the current stable kernel.

2.6.x.y are maintained by the "stable" team (stable at kernel dot
org), are released almost every week.

Rules on what kind of patches are accepted, and what ones are not, into
the "-stable" tree:
- It must be obviously correct and tested.
- It can not bigger than 100 lines, with context.
- It must fix only one thing.
- It must fix a real bug that bothers people (not a, "This could be a
  problem..." type thing.)
- It must fix a problem that causes a build error (but not for things
  marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
  security issue, or some "oh, that's not good" issue.  In short,
  something critical.
- No "theoretical race condition" issues, unless an explanation of how
  the race can be exploited.
- It can not contain any "trivial" fixes in it (spelling changes,
  whitespace cleanups, etc.)
- It must be accepted by the relevant subsystem maintainer.
- It must follow Documentation/SubmittingPatches rules.

Procedure for submitting patches to the -stable tree:
- Send the patch, after verifying that it follows the above rules, to
- The sender will receive an ack when the patch has been accepted into
  the queue, or a nak if the patch is rejected.  This response might
  take a few days, according to the developer's schedules.
- If accepted, the patch will be added to the -stable queue, for review
  by other developers.
- Security patches should not be sent to this alias, but instead to the

Review cycle:
- When the -stable maintainers decide for a review cycle, the patches
  will be sent to the review committee, and the maintainer of the
  affected area of the patch (unless the submitter is the maintainer of
  the area) and CC: to the linux-kernel mailing list.
- The review committee has 48 hours in which to ack or nak the patch.
- If the patch is rejected by a member of the committee, or linux-kernel
  members object to the patch by bringing up issues that the maintainer
  and members did not realize, the patch will be dropped from the
- At the end of the review cycle, the acked patches will be added to
  the latest -stable release, and a new -stable release will happen.
- Security patches will be accepted into the -stable tree directly from
  the security kernel team, and not go through the normal review cycle.
  Contact the kernel security team for more details on this procedure.

Review committee:
- This will be made up of a number of kernel developers who have
  volunteered for this task, and a few that haven't.

                            --- The -mm kernels ---

These are experimental kernels released by Andrew Morton.

The -mm tree serves as a sort of proving ground for new features and other
experimental patches.
Once a patch has proved its worth in -mm for a while Andrew pushes it on to
Linus for inclusion in mainline.

Although it's encouraged that patches flow to Linus via the -mm tree, this
is not always enforced.
Subsystem maintainers (or individuals) sometimes push their patches directly
to Linus, even though (or after) they have been merged and tested in -mm (or
sometimes even without prior testing in -mm).

This branch is in constant flux and contains many experimental features, a
lot of debugging patches not appropriate for mainline etc and is the most
experimental of the branches described in this document.

These kernels are not appropriate for use on systems that are supposed to be
stable and they are more risky to run than any of the other branches (make
sure you have up-to-date backups - that goes for any experimental kernel but
even more so for -mm kernels).

These kernels in addition to all the other experimental patches they contain
usually also contain any changes in the mainline -git kernels available at
the time of release.

The -mm kernels are not released on a fixed schedule, but usually a few -mm
kernels are released in between each -rc kernel (1 to 3 is common).

                            --- The -git kernels ---

These are daily snapshots of Linus' kernel tree (managed in a git
repository, hence the name).

These patches are usually released daily and represent the current state of
Linus' tree. They are more experimental than -rc kernels since they are
generated automatically without even a cursory glance to see if they are

I hope you enjoy reading this article and I really deserve
special thanks to Tony Luck and Jesper Juhl for their suggestions!!

Part of this article is from the document written by Jesper Juhl,
you can find it here: