Blame view
Documentation/process/2.Process.rst
23.7 KB
f7c9fe4b1 doc: development-... |
1 2 3 4 |
.. _development_process: How the development process works ================================= |
75b021468 Add the developme... |
5 6 7 8 9 10 11 |
Linux kernel development in the early 1990's was a pretty loose affair, with relatively small numbers of users and developers involved. With a user base in the millions and with some 2,000 developers involved over the course of one year, the kernel has since had to evolve a number of processes to keep development happening smoothly. A solid understanding of how the process works is required in order to be an effective part of it. |
f7c9fe4b1 doc: development-... |
12 13 |
The big picture --------------- |
75b021468 Add the developme... |
14 15 16 17 |
The kernel developers use a loosely time-based release process, with a new major kernel release happening every two or three months. The recent release history looks like this: |
f7c9fe4b1 doc: development-... |
18 |
====== ================= |
fb0e0ffe7 Documentation: br... |
19 20 21 22 23 24 |
5.0 March 3, 2019 5.1 May 5, 2019 5.2 July 7, 2019 5.3 September 15, 2019 5.4 November 24, 2019 5.5 January 6, 2020 |
f7c9fe4b1 doc: development-... |
25 |
====== ================= |
75b021468 Add the developme... |
26 |
|
fb0e0ffe7 Documentation: br... |
27 28 29 30 |
Every 5.x release is a major kernel release with new features, internal API changes, and more. A typical release can contain about 13,000 changesets with changes to several hundred thousand lines of code. 5.x is the leading edge of Linux kernel development; the kernel uses a |
75b021468 Add the developme... |
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
rolling development model which is continually integrating major changes. A relatively straightforward discipline is followed with regard to the merging of patches for each release. At the beginning of each development cycle, the "merge window" is said to be open. At that time, code which is deemed to be sufficiently stable (and which is accepted by the development community) is merged into the mainline kernel. The bulk of changes for a new development cycle (and all of the major changes) will be merged during this time, at a rate approaching 1,000 changes ("patches," or "changesets") per day. (As an aside, it is worth noting that the changes integrated during the merge window do not come out of thin air; they have been collected, tested, and staged ahead of time. How that process works will be described in detail later on). |
5c050fb96 docs: update the ... |
46 47 |
The merge window lasts for approximately two weeks. At the end of this time, Linus Torvalds will declare that the window is closed and release the |
fb0e0ffe7 Documentation: br... |
48 |
first of the "rc" kernels. For the kernel which is destined to be 5.6, |
5c050fb96 docs: update the ... |
49 |
for example, the release which happens at the end of the merge window will |
fb0e0ffe7 Documentation: br... |
50 |
be called 5.6-rc1. The -rc1 release is the signal that the time to |
5c050fb96 docs: update the ... |
51 52 |
merge new features has passed, and that the time to stabilize the next kernel has begun. |
75b021468 Add the developme... |
53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
Over the next six to ten weeks, only patches which fix problems should be submitted to the mainline. On occasion a more significant change will be allowed, but such occasions are rare; developers who try to merge new features outside of the merge window tend to get an unfriendly reception. As a general rule, if you miss the merge window for a given feature, the best thing to do is to wait for the next development cycle. (An occasional exception is made for drivers for previously-unsupported hardware; if they touch no in-tree code, they cannot cause regressions and should be safe to add at any time). As fixes make their way into the mainline, the patch rate will slow over time. Linus releases new -rc kernels about once a week; a normal series will get up to somewhere between -rc6 and -rc9 before the kernel is |
fb0e0ffe7 Documentation: br... |
67 |
considered to be sufficiently stable and the final release is made. |
75b021468 Add the developme... |
68 |
At that point the whole process starts over again. |
fb0e0ffe7 Documentation: br... |
69 70 |
As an example, here is how the 5.4 development cycle went (all dates in 2019): |
5c050fb96 docs: update the ... |
71 |
|
f7c9fe4b1 doc: development-... |
72 |
============== =============================== |
fb0e0ffe7 Documentation: br... |
73 74 75 76 77 78 79 80 81 82 |
September 15 5.3 stable release September 30 5.4-rc1, merge window closes October 6 5.4-rc2 October 13 5.4-rc3 October 20 5.4-rc4 October 27 5.4-rc5 November 3 5.4-rc6 November 10 5.4-rc7 November 17 5.4-rc8 November 24 5.4 stable release |
f7c9fe4b1 doc: development-... |
83 |
============== =============================== |
75b021468 Add the developme... |
84 85 86 87 88 89 90 |
How do the developers decide when to close the development cycle and create the stable release? The most significant metric used is the list of regressions from previous releases. No bugs are welcome, but those which break systems which worked in the past are considered to be especially serious. For this reason, patches which cause regressions are looked upon unfavorably and are quite likely to be reverted during the stabilization |
5c050fb96 docs: update the ... |
91 |
period. |
75b021468 Add the developme... |
92 93 94 95 96 97 |
The developers' goal is to fix all known regressions before the stable release is made. In the real world, this kind of perfection is hard to achieve; there are just too many variables in a project of this size. There comes a point where delaying the final release just makes the problem worse; the pile of changes waiting for the next merge window will grow |
fb0e0ffe7 Documentation: br... |
98 |
larger, creating even more regressions the next time around. So most 5.x |
75b021468 Add the developme... |
99 100 101 102 |
kernels go out with a handful of known regressions though, hopefully, none of them are serious. Once a stable release is made, its ongoing maintenance is passed off to the |
fb0e0ffe7 Documentation: br... |
103 104 105 106 107 108 109 |
"stable team," currently Greg Kroah-Hartman. The stable team will release occasional updates to the stable release using the 5.x.y numbering scheme. To be considered for an update release, a patch must (1) fix a significant bug, and (2) already be merged into the mainline for the next development kernel. Kernels will typically receive stable updates for a little more than one development cycle past their initial release. So, for example, the 5.2 kernel's history looked like this (all dates in 2019): |
5c050fb96 docs: update the ... |
110 |
|
f7c9fe4b1 doc: development-... |
111 |
============== =============================== |
cca73e494 docs: Correct the... |
112 |
July 7 5.2 stable release |
fb0e0ffe7 Documentation: br... |
113 114 115 116 117 |
July 14 5.2.1 July 21 5.2.2 July 26 5.2.3 July 28 5.2.4 July 31 5.2.5 |
8962e40c1 docs: update kern... |
118 |
... ... |
fb0e0ffe7 Documentation: br... |
119 |
October 11 5.2.21 |
f7c9fe4b1 doc: development-... |
120 |
============== =============================== |
5c050fb96 docs: update the ... |
121 |
|
fb0e0ffe7 Documentation: br... |
122 |
5.2.21 was the final stable update of the 5.2 release. |
5c050fb96 docs: update the ... |
123 124 125 126 |
Some kernels are designated "long term" kernels; they will receive support for a longer period. As of this writing, the current long term kernels and their maintainers are: |
fb0e0ffe7 Documentation: br... |
127 128 129 130 131 132 133 134 |
====== ================================ ======================= 3.16 Ben Hutchings (very long-term kernel) 4.4 Greg Kroah-Hartman & Sasha Levin (very long-term kernel) 4.9 Greg Kroah-Hartman & Sasha Levin 4.14 Greg Kroah-Hartman & Sasha Levin 4.19 Greg Kroah-Hartman & Sasha Levin 5.4 Greg Kroah-Hartman & Sasha Levin ====== ================================ ======================= |
5c050fb96 docs: update the ... |
135 136 137 138 139 |
The selection of a kernel for long-term support is purely a matter of a maintainer having the need and the time to maintain that release. There are no known plans for long-term support for any specific upcoming release. |
75b021468 Add the developme... |
140 |
|
f7c9fe4b1 doc: development-... |
141 142 |
The lifecycle of a patch ------------------------ |
75b021468 Add the developme... |
143 144 145 146 147 148 149 150 |
Patches do not go directly from the developer's keyboard into the mainline kernel. There is, instead, a somewhat involved (if somewhat informal) process designed to ensure that each patch is reviewed for quality and that each patch implements a change which is desirable to have in the mainline. This process can happen quickly for minor fixes, or, in the case of large and controversial changes, go on for years. Much developer frustration comes from a lack of understanding of this process or from attempts to |
5c050fb96 docs: update the ... |
151 |
circumvent it. |
75b021468 Add the developme... |
152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 |
In the hopes of reducing that frustration, this document will describe how a patch gets into the kernel. What follows below is an introduction which describes the process in a somewhat idealized way. A much more detailed treatment will come in later sections. The stages that a patch goes through are, generally: - Design. This is where the real requirements for the patch - and the way those requirements will be met - are laid out. Design work is often done without involving the community, but it is better to do this work in the open if at all possible; it can save a lot of time redesigning things later. - Early review. Patches are posted to the relevant mailing list, and developers on that list reply with any comments they may have. This process should turn up any major problems with a patch if all goes well. - Wider review. When the patch is getting close to ready for mainline |
ef0eba477 Documentation/dev... |
172 |
inclusion, it should be accepted by a relevant subsystem maintainer - |
75b021468 Add the developme... |
173 174 |
though this acceptance is not a guarantee that the patch will make it all the way to the mainline. The patch will show up in the maintainer's |
e4fabad30 Documentation/dev... |
175 |
subsystem tree and into the -next trees (described below). When the |
75b021468 Add the developme... |
176 177 178 |
process works, this step leads to more extensive review of the patch and the discovery of any problems resulting from the integration of this patch with work being done by others. |
ef0eba477 Documentation/dev... |
179 180 181 182 183 184 185 186 |
- Please note that most maintainers also have day jobs, so merging your patch may not be their highest priority. If your patch is getting feedback about changes that are needed, you should either make those changes or justify why they should not be made. If your patch has no review complaints but is not being merged by its appropriate subsystem or driver maintainer, you should be persistent in updating the patch to the current kernel so that it applies cleanly and keep sending it for review and merging. |
75b021468 Add the developme... |
187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 |
- Merging into the mainline. Eventually, a successful patch will be merged into the mainline repository managed by Linus Torvalds. More comments and/or problems may surface at this time; it is important that the developer be responsive to these and fix any issues which arise. - Stable release. The number of users potentially affected by the patch is now large, so, once again, new problems may arise. - Long-term maintenance. While it is certainly possible for a developer to forget about code after merging it, that sort of behavior tends to leave a poor impression in the development community. Merging code eliminates some of the maintenance burden, in that others will fix problems caused by API changes. But the original developer should continue to take responsibility for the code if it is to remain useful in the longer term. One of the largest mistakes made by kernel developers (or their employers) is to try to cut the process down to a single "merging into the mainline" step. This approach invariably leads to frustration for everybody involved. |
f7c9fe4b1 doc: development-... |
207 208 |
How patches get into the Kernel ------------------------------- |
75b021468 Add the developme... |
209 210 |
There is exactly one person who can merge patches into the mainline kernel |
fb0e0ffe7 Documentation: br... |
211 212 213 214 215 216 |
repository: Linus Torvalds. But, for example, of the over 9,500 patches which went into the 2.6.38 kernel, only 112 (around 1.3%) were directly chosen by Linus himself. The kernel project has long since grown to a size where no single developer could possibly inspect and select every patch unassisted. The way the kernel developers have addressed this growth is through the use of a lieutenant system built around a chain of trust. |
75b021468 Add the developme... |
217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 |
The kernel code base is logically broken down into a set of subsystems: networking, specific architecture support, memory management, video devices, etc. Most subsystems have a designated maintainer, a developer who has overall responsibility for the code within that subsystem. These subsystem maintainers are the gatekeepers (in a loose way) for the portion of the kernel they manage; they are the ones who will (usually) accept a patch for inclusion into the mainline kernel. Subsystem maintainers each manage their own version of the kernel source tree, usually (but certainly not always) using the git source management tool. Tools like git (and related tools like quilt or mercurial) allow maintainers to track a list of patches, including authorship information and other metadata. At any given time, the maintainer can identify which patches in his or her repository are not found in the mainline. When the merge window opens, top-level maintainers will ask Linus to "pull" the patches they have selected for merging from their repositories. If Linus agrees, the stream of patches will flow up into his repository, becoming part of the mainline kernel. The amount of attention that Linus pays to specific patches received in a pull operation varies. It is clear that, sometimes, he looks quite closely. But, as a general rule, Linus trusts the subsystem maintainers to not send bad patches upstream. Subsystem maintainers, in turn, can pull patches from other maintainers. For example, the networking tree is built from patches which accumulated first in trees dedicated to network device drivers, wireless networking, etc. This chain of repositories can be arbitrarily long, though it rarely exceeds two or three links. Since each maintainer in the chain trusts those managing lower-level trees, this process is known as the "chain of |
5c050fb96 docs: update the ... |
247 |
trust." |
75b021468 Add the developme... |
248 249 250 251 |
Clearly, in a system like this, getting patches into the kernel depends on finding the right maintainer. Sending patches directly to Linus is not normally the right way to go. |
f7c9fe4b1 doc: development-... |
252 253 |
Next trees ---------- |
75b021468 Add the developme... |
254 255 256 257 258 259 260 261 262 263 264 265 |
The chain of subsystem trees guides the flow of patches into the kernel, but it also raises an interesting question: what if somebody wants to look at all of the patches which are being prepared for the next merge window? Developers will be interested in what other changes are pending to see whether there are any conflicts to worry about; a patch which changes a core kernel function prototype, for example, will conflict with any other patches which use the older form of that function. Reviewers and testers want access to the changes in their integrated form before all of those changes land in the mainline kernel. One could pull changes from all of the interesting subsystem trees, but that would be a big and error-prone job. |
e4fabad30 Documentation/dev... |
266 |
The answer comes in the form of -next trees, where subsystem trees are |
75b021468 Add the developme... |
267 268 269 |
collected for testing and review. The older of these trees, maintained by Andrew Morton, is called "-mm" (for memory management, which is how it got started). The -mm tree integrates patches from a long list of subsystem |
5c050fb96 docs: update the ... |
270 |
trees; it also has some patches aimed at helping with debugging. |
75b021468 Add the developme... |
271 272 273 274 275 276 277 278 279 |
Beyond that, -mm contains a significant collection of patches which have been selected by Andrew directly. These patches may have been posted on a mailing list, or they may apply to a part of the kernel for which there is no designated subsystem tree. As a result, -mm operates as a sort of subsystem tree of last resort; if there is no other obvious path for a patch into the mainline, it is likely to end up in -mm. Miscellaneous patches which accumulate in -mm will eventually either be forwarded on to an appropriate subsystem tree or be sent directly to Linus. In a typical |
5c050fb96 docs: update the ... |
280 281 |
development cycle, approximately 5-10% of the patches going into the mainline get there via -mm. |
75b021468 Add the developme... |
282 |
|
e6a591e58 Documentation/dev... |
283 284 |
The current -mm patch is available in the "mmotm" (-mm of the moment) directory at: |
75b021468 Add the developme... |
285 |
|
e7b4311eb Replace HTTP link... |
286 |
https://www.ozlabs.org/~akpm/mmotm/ |
75b021468 Add the developme... |
287 288 289 |
Use of the MMOTM tree is likely to be a frustrating experience, though; there is a definite chance that it will not even compile. |
5c050fb96 docs: update the ... |
290 |
The primary tree for next-cycle patch merging is linux-next, maintained by |
75b021468 Add the developme... |
291 292 293 294 |
Stephen Rothwell. The linux-next tree is, by design, a snapshot of what the mainline is expected to look like after the next merge window closes. Linux-next trees are announced on the linux-kernel and linux-next mailing lists when they are assembled; they can be downloaded from: |
e7b4311eb Replace HTTP link... |
295 |
https://www.kernel.org/pub/linux/kernel/next/ |
75b021468 Add the developme... |
296 |
|
5c050fb96 docs: update the ... |
297 298 299 |
Linux-next has become an integral part of the kernel development process; all patches merged during a given merge window should really have found their way into linux-next some time before the merge window opens. |
75b021468 Add the developme... |
300 |
|
f7c9fe4b1 doc: development-... |
301 302 |
Staging trees ------------- |
f830673f6 Documentation/dev... |
303 |
|
5c050fb96 docs: update the ... |
304 |
The kernel source tree contains the drivers/staging/ directory, where |
f830673f6 Documentation/dev... |
305 306 307 308 309 310 |
many sub-directories for drivers or filesystems that are on their way to being added to the kernel tree live. They remain in drivers/staging while they still need more work; once complete, they can be moved into the kernel proper. This is a way to keep track of drivers that aren't up to Linux kernel coding or quality standards, but people may want to use them and track development. |
5c050fb96 docs: update the ... |
311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 |
Greg Kroah-Hartman currently maintains the staging tree. Drivers that still need work are sent to him, with each driver having its own subdirectory in drivers/staging/. Along with the driver source files, a TODO file should be present in the directory as well. The TODO file lists the pending work that the driver needs for acceptance into the kernel proper, as well as a list of people that should be Cc'd for any patches to the driver. Current rules require that drivers contributed to staging must, at a minimum, compile properly. Staging can be a relatively easy way to get new drivers into the mainline where, with luck, they will come to the attention of other developers and improve quickly. Entry into staging is not the end of the story, though; code in staging which is not seeing regular progress will eventually be removed. Distributors also tend to be relatively reluctant to enable staging drivers. So staging is, at best, a stop on the way toward becoming a proper mainline driver. |
75b021468 Add the developme... |
327 |
|
f7c9fe4b1 doc: development-... |
328 329 |
Tools ----- |
75b021468 Add the developme... |
330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 |
As can be seen from the above text, the kernel development process depends heavily on the ability to herd collections of patches in various directions. The whole thing would not work anywhere near as well as it does without suitably powerful tools. Tutorials on how to use these tools are well beyond the scope of this document, but there is space for a few pointers. By far the dominant source code management system used by the kernel community is git. Git is one of a number of distributed version control systems being developed in the free software community. It is well tuned for kernel development, in that it performs quite well when dealing with large repositories and large numbers of patches. It also has a reputation for being difficult to learn and use, though it has gotten better over time. Some sort of familiarity with git is almost a requirement for kernel developers; even if they do not use it for their own work, they'll need git to keep up with what other developers (and the mainline) are doing. Git is now packaged by almost all Linux distributions. There is a home |
ef0eba477 Documentation/dev... |
349 |
page at: |
75b021468 Add the developme... |
350 |
|
e7b4311eb Replace HTTP link... |
351 |
https://git-scm.com/ |
75b021468 Add the developme... |
352 |
|
5c050fb96 docs: update the ... |
353 |
That page has pointers to documentation and tutorials. |
75b021468 Add the developme... |
354 355 356 |
Among the kernel developers who do not use git, the most popular choice is almost certainly Mercurial: |
e7b4311eb Replace HTTP link... |
357 |
https://www.selenic.com/mercurial/ |
75b021468 Add the developme... |
358 359 360 361 362 |
Mercurial shares many features with git, but it provides an interface which many find easier to use. The other tool worth knowing about is Quilt: |
e7b4311eb Replace HTTP link... |
363 |
https://savannah.nongnu.org/projects/quilt/ |
75b021468 Add the developme... |
364 365 366 367 368 369 370 |
Quilt is a patch management system, rather than a source code management system. It does not track history over time; it is, instead, oriented toward tracking a specific set of changes against an evolving code base. Some major subsystem maintainers use quilt to manage patches intended to go upstream. For the management of certain kinds of trees (-mm, for example), quilt is the best tool for the job. |
f7c9fe4b1 doc: development-... |
371 372 |
Mailing lists ------------- |
75b021468 Add the developme... |
373 374 375 376 377 378 379 380 381 382 383 384 385 386 |
A great deal of Linux kernel development work is done by way of mailing lists. It is hard to be a fully-functioning member of the community without joining at least one list somewhere. But Linux mailing lists also represent a potential hazard to developers, who risk getting buried under a load of electronic mail, running afoul of the conventions used on the Linux lists, or both. Most kernel mailing lists are run on vger.kernel.org; the master list can be found at: http://vger.kernel.org/vger-lists.html There are lists hosted elsewhere, though; a number of them are at |
f67281a72 Documentation: pr... |
387 |
redhat.com/mailman/listinfo. |
75b021468 Add the developme... |
388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 |
The core mailing list for kernel development is, of course, linux-kernel. This list is an intimidating place to be; volume can reach 500 messages per day, the amount of noise is high, the conversation can be severely technical, and participants are not always concerned with showing a high degree of politeness. But there is no other place where the kernel development community comes together as a whole; developers who avoid this list will miss important information. There are a few hints which can help with linux-kernel survival: - Have the list delivered to a separate folder, rather than your main mailbox. One must be able to ignore the stream for sustained periods of time. - Do not try to follow every conversation - nobody else does. It is important to filter on both the topic of interest (though note that long-running conversations can drift away from the original subject without changing the email subject line) and the people who are |
5c050fb96 docs: update the ... |
407 |
participating. |
75b021468 Add the developme... |
408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 |
- Do not feed the trolls. If somebody is trying to stir up an angry response, ignore them. - When responding to linux-kernel email (or that on other lists) preserve the Cc: header for all involved. In the absence of a strong reason (such as an explicit request), you should never remove recipients. Always make sure that the person you are responding to is in the Cc: list. This convention also makes it unnecessary to explicitly ask to be copied on replies to your postings. - Search the list archives (and the net as a whole) before asking questions. Some developers can get impatient with people who clearly have not done their homework. - Avoid top-posting (the practice of putting your answer above the quoted text you are responding to). It makes your response harder to read and makes a poor impression. - Ask on the correct mailing list. Linux-kernel may be the general meeting point, but it is not the best place to find developers from all subsystems. The last point - finding the correct mailing list - is a common place for beginning developers to go wrong. Somebody who asks a networking-related question on linux-kernel will almost certainly receive a polite suggestion to ask on the netdev list instead, as that is the list frequented by most networking developers. Other lists exist for the SCSI, video4linux, IDE, filesystem, etc. subsystems. The best place to look for mailing lists is in the MAINTAINERS file packaged with the kernel source. |
f7c9fe4b1 doc: development-... |
438 439 |
Getting started with Kernel development --------------------------------------- |
75b021468 Add the developme... |
440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 |
Questions about how to get started with the kernel development process are common - from both individuals and companies. Equally common are missteps which make the beginning of the relationship harder than it has to be. Companies often look to hire well-known developers to get a development group started. This can, in fact, be an effective technique. But it also tends to be expensive and does not do much to grow the pool of experienced kernel developers. It is possible to bring in-house developers up to speed on Linux kernel development, given the investment of a bit of time. Taking this time can endow an employer with a group of developers who understand the kernel and the company both, and who can help to train others as well. Over the medium term, this is often the more profitable approach. Individual developers are often, understandably, at a loss for a place to start. Beginning with a large project can be intimidating; one often wants to test the waters with something smaller first. This is the point where some developers jump into the creation of patches fixing spelling errors or minor coding style issues. Unfortunately, such patches create a level of noise which is distracting for the development community as a whole, so, increasingly, they are looked down upon. New developers wishing to introduce themselves to the community will not get the sort of reception they wish for by these means. Andrew Morton gives this advice for aspiring kernel developers |
f7c9fe4b1 doc: development-... |
465 |
:: |
75b021468 Add the developme... |
466 467 468 469 470 |
The #1 project for all kernel beginners should surely be "make sure that the kernel runs perfectly at all times on all machines which you can lay your hands on". Usually the way to do this is to work with others on getting things fixed up (this can require persistence!) but that's fine - it's a part of kernel development. |
e7b4311eb Replace HTTP link... |
471 |
(https://lwn.net/Articles/283982/). |
75b021468 Add the developme... |
472 473 474 475 476 477 |
In the absence of obvious problems to fix, developers are advised to look at the current lists of regressions and open bugs in general. There is never any shortage of issues in need of fixing; by addressing these issues, developers will gain experience with the process while, at the same time, building respect with the rest of the development community. |