This blog post covers “A Toolkit for Measuring the Impacts of Public Funding on Open Source Software Development”, authored by Cailean Osborne, Paul Sharratt, Dawn Foster, and Mirko Boehm. You can view the preprint here.
Introduction
Open source software (OSS) has become critical digital infrastructure, powering approximately 96% of codebases and constituting up to 90% of commercial software stacks. Traditionally maintained by volunteer communities, recent high-profile security vulnerabilities, such as Log4Shell in 2021 or the xz utils backdoor this year, have highlighted the unsustainability of relying solely on volunteer labour for maintaining such crucial infrastructure.
In light of these events, among others, the understanding of the roles and responsibilities of the public sector in supporting and sustaining critical OSS projects has evolved, with governments increasingly stepping in to fund OSS development, security, and maintenance.
Beyond strengthening the security and resilience of software supply chains, public funding for OSS development serves multiple policy objectives, such as bolstering digital sovereignty, aiding the growth of domestic software markets, and boosting national competitiveness in science and innovation.
Notable examples of public funding programmes have included the Open Technology Fund in the USA; the Next Generation Internet (NGI) initiative by the European Commission, which has provided €140 million for over 1,200 projects between 2019 and 2024; and Germany’s Sovereign Tech Agency (previously Sovereign Tech Fund), which invests around €19 million annually in the security and maintenance of critical open source infrastructure.
The Challenge of Measuring Funding Impacts
While public funding for OSS development is increasing, we have a limited understanding of its actual impacts, nor do we have consensus on how to meaningfully measure the impacts. This creates challenges for policymakers who need to justify public spending, fund managers who want to optimise their funding strategies, OSS projects trying to demonstrate the value of funding, and researchers studying the relationship between funding and OSS development.
The difficulty stems from several interconnected factors. OSS projects and ecosystems are highly diverse, and funding approaches and objectives vary significantly. Impacts can be technological, economic, and social, with effects that may be direct or indirect, internal to projects and their communities or external to their ecosystems of dependents and users, as well as positive or negative. Furthermore, changes may manifest over different time horizons, and causal attribution is complicated by the non-randomness of funding and no shortage of confounding factors.
A Practical Toolkit for Measuring the Impacts of OSS Funding
To address these challenges, we’ve developed a toolkit that provides practical guidance for measuring the impact of public funding on OSS development.
The toolkit is not prescriptive, nor is it exhaustive; rather, we aim to equip various stakeholders with practical insights that may inform the evaluation of OSS funding programmes and measurement of impacts.
Guiding Motto and Questions
The toolkit is guided by the following motto: If we can meaningfully (quantitatively and/or qualitatively) measure the impacts of public funding on OSS development, then we can work towards building an evidence base that can inform effective and public-interest funding strategies.
Beyond this motto, the toolkit aims to encourage relevant stakeholders to ask themselves the following questions:
- What kinds of social, economic, and technological impacts—both positive and negative, direct and indirect—can funding have on OSS projects and their broader ecosystems over different time horizons?
- How do different funding approaches influence OSS project outcomes and community dynamics, and what are the relative merits and drawbacks of various funding approaches?
- What are potential unintended consequences of funding that need to be identified and mitigated?
- How, if at all, can we create meaningful metrics that capture the multiplier effects of funding and convey the return on public investment?
- How can or should we define and measure “success” in OSS funding?
Account for Funding Objectives
A key lesson from the CHAOSS project is that every OSS project is different, and metrics should always be interpreted with the needs of the project and its context taken into account. Similarly, when considering the impacts of funding on an OSS project, one should start by taking into account the funding objectives.
Different funding instruments serve different purposes, from innovation to maintenance, from project-specific goals to ecosystem-wide improvements. Understanding the specific objectives helps align impact measurements with expected outcomes, rather than fishing for impacts in the wild.
Account for Context: Project Life Stages, Social Structures, and Cost Factors
Context is crucial when measuring the impact of OSS funding. Projects at different life stages with different contributor and user community sizes have vastly different needs – a new prototype might need funding for initial development, while a mature project might require support for security improvements. Additionally, regional and organisational cost factors play a crucial role, as similar budget allocations across different organisations and regions may result in vastly different full-time equivalent personnel, and remuneration doesn’t always correlate with contribution quality.
Consider Multiple Impact Dimensions
When assessing the impacts of OSS funding, there may be a tendency to focus on technological impacts, which may in part be due to the technological nature of OSS development or the relative ease of measuring technological impacts due to data availability from OSS repositories.
However, drawing on social models of OSS development and community health methodologies, the toolkit highlights three key impact dimensions: technological, economic, and social. Technological impacts include improvements in code quality, security enhancements, and development velocity. Economic impacts encompass job creation, project revenue generation, and cost savings for adopters. Social impacts include community growth, contributor diversity, and improvements in project governance.
Each dimension can have both direct and indirect effects. For instance, funding might directly enable a project to hire maintainers (direct economic impact) while indirectly fostering the growth of a commercial ecosystem around the project (indirect economic impact). Similarly, impacts can be internal to the project and its community of contributors or external to its ecosystem of dependents and users. What is more, the impacts do not follow a linear or unidirectional scale; funding can lead to both improvements and degradations across different metrics. This multidirectional nature of impacts means that measurement frameworks need to be flexible to capture both positive and negative changes, rather than assuming funding inherently leads to improvements or that metrics only move in one direction.
Choose Appropriate Methodological Approaches
The toolkit outlines three main methodological approaches for measuring impact. Quantitative methods, such as repository mining, econometric modelling, and time series analysis, offer scalability and clear metrics but may oversimplify complex dynamics. Meanwhile, qualitative methods, including case studies, interviews, and participant observation, provide rich contextual insights but are resource-intensive and limited in scalability.
Mixed-methods approaches combine both quantitative and qualitative techniques, allowing for both depth and breadth in understanding impact. This combination helps validate findings through triangulation and provides a more complete picture of how funding affects OSS development. We offer three examples of mixed-methods approaches that sequence the qualitative and quantitative approaches differently, each with merits and limitations.
Consider Temporal Aspects
Impact should be measured across different time horizons. Short-term effects (under one year) might include immediate improvements in development velocity or bug fixing. Medium-term impacts (one to three years) often involve adaptation and growth in project capabilities. Finally, long-term changes (over three years) might encompass fundamental shifts in project sustainability or ecosystem dynamics, but are the hardest to measure.
Options for and Hazards of Multiplier Effect Estimations
Multiplier effects — measurements of how much additional value is generated beyond an initial investment — provide useful quantitative evidence for policymakers and funders. While subjective or qualitative assessments of funding impact can be valuable, decision-makers often require clear numerical evidence about return on investment to justify public spending and optimise funding strategies.
The estimation of multiplier effects of OSS funding must start by recognising that the collaborative nature of OSS development creates unique dynamics that differ from traditional R&D contexts in academic and corporate labs. For example, when a funded developer improves code, that improvement becomes instantly accessible to all users and contributors, creating cascading benefits as others build upon and adapt the work. This instant availability can amplify positive effects far beyond what’s typical in proprietary development. We see this in practice: the NGI initiative found that each funded contributor supported a community of roughly 50 contributors, while in the European Union €1 billion invested in OSS by companies in 2018 is estimated to have generated up to €95 billion for the EU’s GDP that year.
However, measuring multiplier effects requires careful consideration and carries risks of oversimplification and norm-setting. Time lags between funding and observable impacts often exceed typical measurement periods, and it’s challenging to isolate funding effects from other influences on project activity. Oversimplified multiplier calculations might create misleading incentives or overlook negative effects. The toolkit, therefore, advocates for nuanced approaches that consider both positive and negative multiplier effects while acknowledging the complex nature of OSS development.
Open Questions and Future Work
Several important questions remain unanswered in our understanding of OSS funding impacts. We need better insight into the long-term effects of funding, particularly after it ends for projects. The relative effectiveness of different funding models also requires more study, as does the potential for unintended consequences such as changes in volunteer motivations or the creation of unsustainable dependencies on funding.
We also need to better understand how funding influences innovation and collaboration patterns in the broader OSS ecosystem. A key challenge lies in developing standardised impact evaluation frameworks while maintaining enough flexibility to account for the diverse contexts of different OSS projects and communities.
Help Us Improve the Toolkit
This toolkit is designed to be practical and useful for various stakeholders in the OSS funding ecosystem. However, we recognise that different funders — whether public sector organisations, companies, or foundations – may have different needs. For this reason, we’re actively seeking feedback to make the toolkit more useful. The name of the game is to iteratively improve this toolkit to enhance its responsiveness to the needs of various funders.
We’ll also be hosting workshops (e.g. “Funding the FOSS Ecosystem” devroom at FOSDEM 2025) and community discussions to gather input and iterate on the toolkit. If you’re involved in OSS funding, whether as a funder, recipient, or researcher, we’d love to hear your thoughts on how this toolkit could better serve your needs. Get involved by reading the preprint, contacting the authors and/or participating in upcoming events. You can view the preprint here.
A Toolkit for Measuring the Impacts of Public Funding on Open Source Software Development
