What follows is an opinion blog from members of the NGI Commons consortium (Nicholas Gates, from OpenForum Europe, and Jan Krewer, from Open Future) regarding the recent exposure of the XZ utils backdoor. If you would like to provide feedback, please get in touch with us through the website.
The recent exposure of the XZ utils backdoor rattled the open source and cybersecurity communities. While it (once again) demonstrated the major advantages of open source software in terms of security over proprietary software – considering that the flaw was discovered relatively quickly and that the community was able to react as fast by sharing information and taking action – it also exposed the limits of a system that has to rely largely on the efforts of volunteers.
A growing number of experts argue that these limitations should be addressed through increased public support, including public funding. At the same time, it is essential to preserve and strengthen the fundamentals on which most open source projects rely: collaboration, freedom of use, standardisation of use, etc. We therefore believe that more intentional governance of key open source software (OSS) packages being used as digital commons in security infrastructure might help to codify community norms and overcome the problem of burnout, issues we saw at play in the XZ utils case.
What happened with the XZ utils backdoor?
XZ utils is a data compression utility, used widely in many GNU/Linux distributions. It was found to have a backdoor put in it by a maintainer, the so-called Jia Tan, who gained trust in the community maintaining XZ utils over a period of years while (allegedly) acting on behalf of a state-backed body. Jia Tan’s subterfuge – which was shortly set to many distributions of Linux – was discovered only by the dogged investigation of a lone PostgresSQL developer and member of the Openwall security community, Andres Freund.
As a commentary by Alex Stamos noted, had XZ utils been patched into global Linux distributions, it could have been: “… the most widespread and effective backdoor ever planted in any software product” and could have “given its creators a master key to (…) hundreds of millions of computers around the world”. This potential ‘oversight’ in the code would have had wide-ranging consequences for cybersecurity, digital sovereignty, and national security globally.
While the fundamentals of OSS governance remain largely unquestioned by this case and have been shown to be worthy of preservation, reports on the interactions behind the introduction of the vulnerability into the XZ utility show the limits of the maintainer model. Many maintainers lack resources and time to support their projects as much as they require, often leading to stress and burnout.For many, these insights will recall the now-famous meme on modern digital infrastructure, which notes how major infrastructure often has dependencies on the small-scale work of a handful of volunteers, requiring more funding to improve security practices at scale.
Why does that matter?
When thinking about how OSS communities are organised, it’s important to remember that the OSS ecosystem has become increasingly vast and complex. Some 96% of commercial code contains open source and 76% of code in general is open source, according to the 2023 OSSRA report. The existence of private foundations and a large number of big tech companies contributing to the provision of OSS through funding or direct contributions has continued to increase in recent years.
That said, this can make it hard to map everything that’s being used as part of security infrastructure in the first place. There are even innovative solutions to track one’s reliance on certain open source software to contribute back to the maintenance of key components in a value chain (e.g. Drips, FossFunders, GitHub Sponsors, etc), but these are often imperfect solutions at best.
Moreover, these efforts seem to be insufficient to address the fundamental problem of maintenance shortage (e.g. resources and attention) in the field. The current scale of open source development requires more public engagement and public funding, not unquestioned faith in the best members of the open source community to ensure its security. There must be specific prescriptions for strengthening OSS governance and helping to overcome the challenges of continued engagement and burnout.
These observations have led to several calls for stronger collective responsibility and public investments in the area, similar to what the Sovereign Tech Fund is doing in Germany. There have also been recent calls in the UK, for example. The objective of such efforts would be to identify and support key software components from the perspective of the general public interest, such as cybersecurity, sovereignty, industrial dependency, government services, or the overall resilience of the Internet stack.
A digital commons approach for reinforcing OSS security
The security of open source software is enabled in part by what has been coined as Linus’ law – the idea that ‘given enough eyeballs, all bugs are shallow’. This law – at least as an adage, if not pure social science – still holds true, considering that it was the openness of the XZ utils that allowed Freund to discover and expose the backdoor.
But we would also note that this example shows that OSS security also depends in large part on the health of communities that are increasingly burned out and potentially vulnerable to being exploited for nefarious purposes. Such capacities require more than just ‘more eyeballs’ or additional software development skills.
This is, according to Tobie Langel, the key problem today: we do not recognise the distinct role of open source maintainers, it is assumed that open source software development is only about developing new code and adding new features. Discussions on security of open source software require root cause solutions for addressing potential vulnerabilities in decentralised software development, including the threat of social engineering.
The policy debates on the cybersecurity of OSS should also take into account the critical differences in the organisation and governance behind an OSS project. The governance of OSS packages differs greatly, from huge foundations with vast communities such as the Apache Foundation to smaller informal projects collaborating on specific tools on Github.
In order to better understand the varying social realities between open source projects, an increasing level of attention is given to the concept of digital commons. Digital commons are commonly defined as ‘non-rivalrous and non-exclusive resources defined by distributed and communal production, ownership and governance of informational capacities and technology. This concept is not only used to describe resources with non-exclusive access licences, but resources that are actually managed by a community that self-organises to define its governance as well as access and user rights.
We argue that thinking about specific OSS packages and their communities more intentionally as digital commons and codifying stronger rules and norms for their governance might help overcome potential exploitation of the maintainer model, creating more opportunities for checks and balances. OSS projects that are self-organised but have well established rules and governance models can reduce single dependencies through distributed production.
This is, of course, hard, when we don’t know exactly who is using what, or when some packages don’t have anything resembling a community, just a single person. But we argue that adding some more definition or structure to models for OSS governance is critical for continuing to support the communities doing the hard work of maintenance. Software resources alone are different from the ones backed-up by vibrant communities, and they require people coming together in a more defined way to codify rules and reinforce shared responsibility for their long-term sustainability.
Continuing to promote robust governance mechanisms through more resources and attention should lead to shared obligations around maintenance in the face of the threat of social engineering. That said, it requires public support, and the maturity (and needs) of maintainer communities needs to be taken into account by industries and governments who depend on OSS.
But how do we get towards that reality?
The role of the public sector
A majority of policies around technology in general are associate software development primarily with innovation, rather than maintenance. Only recently, have conversations started to focus more on how to take care of the existing infrastructures and their security. The burnt-out OSS maintainers show us that this needs to change even more rapidly, and what happened with XZ utils is yet another reminder.
Existing foundations may soon start moving more towards separating activities related to maintenance, legal advisory, community building, or communication from software development. Innovative solutions to identify dependencies and current attempts to better define what OSS components should be considered as infrastructure are very encouraging for helping researchers figure out what needs support in the first place.
Once identified, we must not forget the role of public funding in supporting open source digital commons. For example, the experience of the French government’s funding of scikit-learn underlines the importance of funding the maintenance of existing OSS like scikit-learn that is used by millions of researchers and engineers daily, rather than simply funding the development of new features or tools.
When combined with initiatives to change how digital commons are publicly supported, either through long-term funding or capacity building measures, we see the potential for creating more participation in open source communities for key software packages for cybersecurity, particularly those being used in Linux distributions (on which a lot of cybersecurity infrastructure runs).
The NGI Commons project will actively participate in these efforts by trying to imagine a more coherent European public and private funding landscape for resources that are critical for Europe’s digital sovereignty in the future. The project will engage with digital commons communities to better identify their needs and current gaps in policy initiatives aimed at supporting their development. Furthermore, considering the widespread use and ubiquity that digital commons have in our societies, we believe these issues must be fully incorporated into the European Commission’s work, in particular in the context of the negotiations for the EU’s next Multiannual Financial Framework (MFF).