From ab402b22eee0f3e18b4f1afa78479f638c3c93f7 Mon Sep 17 00:00:00 2001 From: Thomas Vuillaume Date: Thu, 22 Oct 2026 13:52:14 +0200 Subject: [PATCH 1/3] reorganize quality dimension section --- source/01-introduction/index.md | 4 ++-- source/02-framework/index.md | 19 +++++++++++++++---- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/source/01-introduction/index.md b/source/01-introduction/index.md index 5aeb073..36c6837 100644 --- a/source/01-introduction/index.md +++ b/source/01-introduction/index.md @@ -4,7 +4,7 @@ The goal of this document is to provide an overview of the considerations and perspectives around research software quality being taken by the EVERSE project and to act as a guide to internal activity as well as a statement to those outside of the project of the approach being taken. The main scope is to connect various perspectives to software quality, excellence, and good practices. -The Framework is structured around four main pillars, detailed in Section 2: Technical aspects (Section 2.1), FAIR Software (Section 2.2), Open Source Software (Section 2.3) and Sustainability (Section 2.4). +The Framework is structured around quality dimensions and indicators, detailed in Section 2: Technical dimensions (Section 2.1), FAIR Software (Section 2.2), Open Source Software (Section 2.3), Sustainability (Section 2.4) and Community (Section 2.5). To address the diverse needs of the research software community, the Framework offers four distinct views, which are detailed in Section 3: @@ -27,7 +27,7 @@ For additional context and examples, see: Defining Research Software: a controve ### 1.2.2 Research Software Quality -We define "research software quality" as the degree to which software supports reliable, efficient, maintainable, and trustworthy research. Quality encompasses multiple dimensions including correctness, performance, maintainability, usability, robustness, and reproducibility, etc. +We define "research software quality" as the degree to which software supports reliable, efficient, maintainable, and trustworthy research. Quality encompasses multiple dimensions including correctness, performance, maintainability, usability, robustness, and reproducibility, etc. These are fundamental technical aspects of software quality in general considered by software engineers regardless of whether they work in academia or industry. However, research software may place different emphasis on these dimensions and also focuses on other aspects. Quality research software enables researchers to trust, share, and build upon computational work, making science transparent and verifiable. FAIR software principles (Findable, Accessible, Interoperable, Reusable) represent a crucial subset of quality, ensuring software can be discovered, understood, and exercised by others whilst supporting long-term reproducibility and research integrity. diff --git a/source/02-framework/index.md b/source/02-framework/index.md index 47bbe8a..97be7ad 100644 --- a/source/02-framework/index.md +++ b/source/02-framework/index.md @@ -1,9 +1,20 @@ -# 2 EVERSE Reference Framework +# 2 Quality Dimensions and Indicators for Research Software -Research software quality assessment combines insights from the two worlds of software engineering and research. Because of software's fundamental characteristics, many quality aspects centre on technical dimensions that software engineers and developers apply universally, regardless of whether they work in academia or industry. However, research software may place different emphasis on these dimensions compared to typical industry applications. Additionally, research contexts introduce specific quality requirements that extend beyond traditional software engineering concerns. +EVERSE structures the discussion of research software quality around three concepts: quality dimensions, quality indicators and quality tools. This section defines them and then goes through each dimension. -In EVERSE, we structure the discussion around research software quality with three main concepts: Research Quality Dimensions, Research Software Quality Indicators and Research Software quality tools. Research Software Quality Dimensions represent common criteria relevant for assessing software quality, at a conceptual level (e.g., documentation, performance, etc.). At a lower level of granularity, a software quality indicator represents a specific aspect of software that can be measured (e.g., FAIRness, test coverage, documentation coverage, etc.). Finally, a research software quality tool is the instrument that measures a software quality indicator against specific criteria or, alternatively, helps improve in one particular aspect (such as linters). Figure 1 shows an example of this division with an example: the "Maintainability" dimension, as defined in ISO/IEC 25010 may be associated with an indicator to check whether continuous integration tests exist in a source code repository (e.g., as defined by the OpenSSF community). Then an assessment tool like the OpenSSF Scorecard may implement the indicator by looking at whether the project runs tests before pull requests are merged in GitHub. However, a different quality assessment tool may check the same indicator against GitLab repositories (not supported by OpenSSF Scorecard). +**Quality dimensions** are broad categories of quality, defined at a conceptual level (e.g., maintainability, security, FAIRness). They give a shared vocabulary and a way to sort everything else: practices, indicators and tools are all attached to one or more dimensions. Dimensions are not measured directly. + +**Quality indicators** are specific, checkable properties of a piece of software or of its development process (e.g., "software has a license", "software has continuous integration tests", "test coverage follows community conventions"). An indicator belongs to one or more dimensions and has a defined way of being assessed, which can be a yes/no check, a score or a threshold. The EVERSE indicators repository holds the canonical list: [https://everse.software/indicators/](https://everse.software/indicators/). In this document, indicators are referred to by their identifier, e.g. `software_has_license`. + +**Quality tools** are the software, services or guides that act on indicators. We distinguish two roles: + +- **Tools that measure an indicator** inspect the software, its repository or its metadata and report a value for the indicator, without changing anything. Examples: howfairis checks whether a repository has a license and a citation file; Pylint reports linting issues; Gitleaks scans the history for leaked credentials. +- **Tools that improve an indicator** help developers change the software or its process so that the indicator value gets better. Examples: cffinit generates a `CITATION.cff` file; GitHub Actions runs the tests on every change; pixi records and locks dependencies. + +Some tools do both. Ruff reports linting issues and fixes many of them; Dependabot detects vulnerable dependencies and opens pull requests to update them. The distinction still matters: a measuring tool can provide an assessment, while an improving tool can help you implement. Running a measuring tool on a regular basis is itself a practice that some indicators capture, such as `uses_tool_for_warnings_and_mistakes`. The EVERSE Tech Radar ([https://everse.software/TechRadar/](https://everse.software/TechRadar/)) records, for each tool, which indicators it measures and which it improves. + +Figure 1 shows how the three concepts fit together. The "Maintainability" dimension, as defined in ISO/IEC 25010, may be associated with an indicator that checks whether continuous integration tests exist in a source code repository (e.g., as defined by the OpenSSF community). An assessment tool like the OpenSSF Scorecard may measure this indicator by checking whether the project runs tests before pull requests are merged on GitHub. A different tool may measure the same indicator on GitLab repositories, which OpenSSF Scorecard does not support. A CI service such as GitLab CI/CD does not measure anything, but it is how a project improves that indicator. ![Figure 1: Main research software quality concepts, together with an example](../figures/figure1-quality-concepts.png){ width=90% } -The EVERSE Framework encompasses thirteen main dimensions of research software quality, derived from established standards and research software best practices. These consist of nine technical dimensions based on ISO/IEC standards, plus FAIRness, Open Source Software, Sustainability, and Community dimensions that reflect the unique requirements of research software. Each dimension is described in the sections that follow. +The EVERSE Framework encompasses thirteen dimensions of research software quality, derived from established standards and research software best practices. Nine are technical dimensions taken from ISO/IEC 25010 (Section 2.1). The other four, FAIRness (Section 2.2), Open Source Software (Section 2.3), Sustainability (Section 2.4) and Community (Section 2.5), reflect requirements specific to research software. For each dimension, we describe what it covers, list "good enough" practices, and give examples of indicators together with tools that measure or improve them. These examples are illustrative, not exhaustive; the EVERSE indicators repository and the Tech Radar are the up-to-date references. From e75d6ffffec421e18009bc91298638222637a18c Mon Sep 17 00:00:00 2001 From: Thomas Vuillaume Date: Thu, 22 Oct 2026 13:53:07 +0200 Subject: [PATCH 2/3] add examples indicators to quality dimensions --- source/02-framework/community.md | 7 +++++++ source/02-framework/fair.md | 9 +++++++++ source/02-framework/open-source-software.md | 6 ++++++ source/02-framework/sustainability.md | 8 ++++++++ 4 files changed, 30 insertions(+) diff --git a/source/02-framework/community.md b/source/02-framework/community.md index 8a426c0..c41d217 100644 --- a/source/02-framework/community.md +++ b/source/02-framework/community.md @@ -118,4 +118,11 @@ The following is a non-exhaustive list of tools that can help assess or improve - All Contributors (https://allcontributors.org/) - recognising diverse contribution types - CRediT (Contributor Roles Taxonomy) - standardised contribution roles +**Examples of indicators and related tools:** + +- `has_active_contributors` (Project has active contributors beyond core maintainers): measured by GrimoireLab, Augur and GitHub Insights; All Contributors helps by making non-code contributions visible. +- `response_timeframe_ok` (Responses to issues and/or pull requests are done within a defined timeframe): measured by GrimoireLab and Augur. +- `has_active_communication_channels` (Project has active communication channels): improved by setting up Discourse, Gitter or a mailing list. +- `has_contribution_guidelines` (Software repository has contribution guidelines): checked by GitHub's community profile; Open Source Guides and Community Rule help write them. + Note that community health is deeply interconnected with sustainability and open source practices. Effective community development requires not just tools but also intentional practices, inclusive cultures, and sustained engagement that evolve with the research software and its user base. diff --git a/source/02-framework/fair.md b/source/02-framework/fair.md index f617b11..1c8baf9 100644 --- a/source/02-framework/fair.md +++ b/source/02-framework/fair.md @@ -61,4 +61,13 @@ The tools listed below fall into two main categories: automated assessment tools - Five recommendations for FAIR software (https://fair-software.nl/, does not explicitly use FAIR4RS) - FAIR data self assessment tool (https://ardc.edu.au/resource/fair-data-self-assessment-tool/) +**Examples of indicators and related tools:** + +- `software_has_license` (Software has license): measured by howfairis, F-UJI and RSFC; improved by REUSE and Choose a License. +- `software_has_citation` (Software uses citation): measured by howfairis and SOMEF; improved by cffinit, which generates a `CITATION.cff` file. +- `descriptive_metadata` (Software has descriptive metadata) and `codemeta_completeness` (CodeMeta completeness): measured by SOMEF; improved by the CodeMeta Generator and Somesy. +- `persistent_and_unique_identifier` (Software has persistent and unique identifier): measured by RSFC and the FAIRsoft Evaluator; improved by the Zenodo GitHub integration, which assigns a DOI to each release. +- `archived_in_software_heritage` (Software is archived in Software Heritage): measured by RSFC; improved by Software Heritage's "save code now" feature. +- `listed_in_registry` (Software is listed in a registry): measured by howfairis. + [^2]: FAIR Principles for Research Software (FAIR4RS Principles), https://zenodo.org/records/6623556. diff --git a/source/02-framework/open-source-software.md b/source/02-framework/open-source-software.md index dd4e8ff..c9c004f 100644 --- a/source/02-framework/open-source-software.md +++ b/source/02-framework/open-source-software.md @@ -60,4 +60,10 @@ The following is a non-exhaustive list of tools that can help assess or improve - Zenodo (https://zenodo.org/) - open repository for research software publication - Software Heritage (https://www.softwareheritage.org/) - universal archive for software source code +**Examples of indicators and related tools:** + +- `software_has_license` (Software has license): measured by howfairis, RSFC and SCANOSS; improved by Choose a License and REUSE. +- `has_contribution_guidelines` (Software repository has contribution guidelines): improved by project templates such as the Fair Python Cookiecutter, which ship a `CONTRIBUTING` file; GitHub's community profile checks for its presence. +- `has_active_contributors` (Project has active contributors beyond core maintainers) and `response_timeframe_ok` (Responses to issues and/or pull requests are done within a defined timeframe): measured by community health dashboards such as GrimoireLab and Augur. + Note that these dimensions overlap partially with other dimensions (e.g. source code availability could also be categorized under FAIR, and Integration with OSS ecosystems could be seen as just a specification of the Technical Dimension of Compatibility). Despite this, the open source aspect is important enough to warrant separate attention. It allows those involved in research software to better integrate into the much broader world of open source software outside of academia. Speaking the same language as the OSS world makes it easier to get outside support, attract outside talent into academia, and vice versa improves career possibilities for RSEs outside of academia. diff --git a/source/02-framework/sustainability.md b/source/02-framework/sustainability.md index a582358..0447e9a 100644 --- a/source/02-framework/sustainability.md +++ b/source/02-framework/sustainability.md @@ -92,4 +92,12 @@ The following is a non-exhaustive list of tools that can help assess or improve There are many open-source tools available in this space to support these practices. +**Examples of indicators and related tools:** + +- `dependency_management` (Software has dependency management solution): improved by pixi, Poetry, Renovate and Dependabot; measured by SCANOSS. +- `repository_workflows` (Software has ci/cd workflows in its repository): improved by GitHub Actions and GitLab CI/CD; measured by RSFC and SOMEF. +- `software_test_coverage` (Software has sufficient test coverage): measured by gcov and pytest (with its coverage plugin). +- `has_releases` (Software has releases): improved by Git tags and the Zenodo GitHub integration; measured by RSFC and SOMEF. +- `archived_in_scholarly_repository` (Software is archived in a scholarly repository): improved by Zenodo and Software Heritage. + [^3]: For governance frameworks, see Code for Society (https://www.codeforsociety.org/incubator/resources/governance-bibliography) and Community Rule (https://communityrule.info/). From 4b078209acd1f5108528f1b2c640071a5eb148ce Mon Sep 17 00:00:00 2001 From: Thomas Vuillaume Date: Thu, 22 Oct 2026 13:53:40 +0200 Subject: [PATCH 3/3] Add support for unnumbered tables and break long identifiers in LaTeX template --- templates/everse.latex | 3 +++ 1 file changed, 3 insertions(+) diff --git a/templates/everse.latex b/templates/everse.latex index 6288982..33442c2 100644 --- a/templates/everse.latex +++ b/templates/everse.latex @@ -57,6 +57,9 @@ \makeatother \usepackage{longtable,booktabs,array,calc} +\newcounter{none} % for unnumbered tables (pandoc >= 3.x) +% Let long identifiers such as indicator names break after underscores +\AtBeginDocument{\let\oldunderscore\_\renewcommand{\_}{\oldunderscore\allowbreak}} \providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}} \usepackage{fancyhdr}