Linux creator Linus Torvalds joined Jim Zemlin, executive director of the Linux Foundation, on stage at the Open Source Leadership Summit for a candid conversation on maintaining open source projects. After 25 years of maintaining the Linux kernal, Torvalds has developed a network of trust that enables the project to iterate quickly with its 2.5 month release cycle.
The current pace of Linux development sees 10K lines of code added, 8K lines subtracted, and 1800 lines modified every single day. Torvalds described how Linux contributors learned to modularize the code and development model so they can accomplish more in parallel, while working independently. He used to read through all the patches but quickly found himself becoming a bottleneck to progress. Ensuring code quality on the project is no longer about micromanaging all the patches but about finding the people Torvalds can trust.
“We have a very strong network and that’s why we can have 1,000 people involved in every release,” Torvalds said. “There’s this whole human potential there that we use very extensively to make sure the process works. And it’s taken 25 years to get to where we are on a process level, too.”
Torvalds described how the project revolves around individual contributors building a trusted reputation with maintainers and how one moves from casual contributor status to become part of the network of trust.
“When people move between companies, the trust does not stay with the company,” Torvalds said. “The trust stays with the person. Obviously people do not stay around forever. People come and go. One of the reasons people sometimes think it’s hard to get into the kernel, is because you think you can just come in and shoot off a patch and go away, but that’s not how it works. What really takes time is to shoot off a lot of small patches until you reach the point where the maintainers trust you and at that point you have become something more than the guy who sends patches. You’ve become part of the whole network of trust.”
Increasing levels of trust among contributors allows maintainers to oversee a rapid pace of development. Ten years ago, when Torvalds wanted to move an 8-week release schedule, he said everyone knew he was pushing for something unrealistically short. They have been cranking out releases every 2.5 months since then and Torvalds said it “solved so many things we didn’t even know needed to be solved.” Linux maintainers found their way to a successful development model by experimenting until it clicked.
“If you want to do something new, something that is really pushing the envelope, nobody can really say ‘This is the way to do it,'” Torvalds said. “What you need to do is a lot of trial and error. I often compare software development to biological processes where really it is evolution. It is not intelligent design. I’m there in the middle of the thing and I can tell you, it is absolutely not intelligent design. You have a lot of people with ideas of where things need to go and then you need to have a marketplace where you can try them out. And that’s what open source allows – all these different groups to go off, do their own thing, and then come back and say I did this, and it actually worked.”
Zemlin asked him if he ever experienced an “a-ha moment” where he realized that Linux was much bigger than what he thought he started. Torvalds said that although a couple of moments have taken him by surprise, the biggest moment was six months after releasing it when it went past being a personal project. He suddenly didn’t even know the people sending him patches. Twenty five years later, it’s the people doing the work who still inspire him.
“I’m a huge believer in the 99% perspiration, 1% inspiration thing,” Torvalds said. “The innovation that this industry talks about so much is bullshit. Anybody can innovate. Don’t do this big, ‘Think different’ innovation thing. Screw that; it’s meaningless. 99% of it is ‘Get the work done.’ That’s my least favorite part of the technology news cycle: the constant innovation and new ideas, ‘This will revolutionize,’ all that hype – that’s not where the real work is. The real work is in the details.”
Not all open source projects could be successful with the same model and leadership that Torvalds has brought to Linux development, but the kernel is arguably one of the most active and successful GPL-licensed projects. Zemlin noted that nearly 900 organizations participate in all the different Linux Foundation programs, creating code in almost every aspect of computing from mobile phones to cars to home appliances. It wouldn’t be possible without Torvalds’ laser focus on continuing the work he started.
“I’m obviously one of those people who likes to concentrate on one project,” Torvalds said. “I don’t like flitting from one idea to another. You need the people who just flutter about and come up with ideas, but they’re not the really useful ones. They end up being the ones who maybe give the ideas to the ones doing the work. What anybody should take away from this talk – the people who actually do the work are the ones you should really listen to. These days I don’t actually do the work anymore – I merge other people’s work. But I would like conferences, too, to be less about the visionary innovation thing and more about the day-to-day what are my problems and how do I want to solve them.”
Check out the video below to watch the full session with Linus Torvalds and Jim Zemlin: