Science cluster & challenges
Summary
Diffraction experiments generate vast amounts of raw data in diverse formats, complicating their reuse and interoperability across research domains. The MC-ReDD project responds to this challenge by focusing on the expanded imgCIF format, which allows easy transfer and processing of the metadata associated with image data, without needing to transfer the actual data images. The project will create a publicly-available, easy-to-use tool for semi, or fully automatic construction of imgCIF files from raw data sets. This will facilitate the interoperability and reusability benefits offered by the imgCIF scheme so as to ensure trust in raw data re-use by specialists from other domains.
Challenge
Open Science project, Cross-domain/Cross-RI
Raw diffraction data are captured using numerous binary data formats and varying metadata structures, making it difficult to achieve interoperability or reusability. The large size of these datasets, often over 100 GB, further complicates data sharing and validation. Current methods for handling raw data lack standardisation, hindering transparent communication between researchers, and making it harder to trust and reuse data across disciplines.
Solution
MC-ReDD aims to build on the text-based imgCIF format to create a publicly-available, easy-to-use tool for semi or fully automatic construction of imgCIF files from raw data sets allowing easy transfer and processing of the metadata associated with image data, without necessarily needing to transfer the entire dataset. This scheme offers, for the first time, a way to transparently communicate rich information about raw data in a standardised, robust, machine-readable fashion, allowing third-party raw data services to be provided on the open web.
Scientific Impact
While enhancing interoperability and reusability of raw diffraction data across scientific domains, the project aims to make the tools available as an Open Science Service within the EOSC Web of FAIR data and Services, or PaNdata software catalogue, as well as in the form of an open service hosted by the IUCr journals website. The imgCIF files can become part of open data collections hosted at Metadata Catalogues, enabling cross searching with other EOSC components. The imgCIF file thus represents a FAIR Digital object.
Results
- imgCIF_Creator: Command-line tool (Python) to convert raw diffraction image data from various sources, supplied by online repository reference or from local storage , extract metadata information, complement if necessary and produce imgCIF output files with comprehensive metadata and links to the actual image data.
- imgCIF_Creator web frontend: Streamlit (Python) web user interface to the imgCIf creator backend. Deployment on a web-server in order to provide a complete web-service including for imgCIF workflow including imgCIF file validation is currently being done.
Publications
- MC-ReDD: Metadata Collection and validation for Re-use of raw Diffraction Data - project poster 2nd OSCARS AGM 2026, DOI: https://doi.org/10.5281/zenodo.18805700
Events
- 24-26 March 2025 | Berlin, Germany - McReDD presentation as part of a presentation regarding European XFEL activities on FAIR and open data at the DAPHNE4NFDI annual meeting 2025
- 23 May 2025 | Online - PaNOSC-OSCARS projects mentorship meeting - Online meeting bringing together PaNOSC-related projects funded under the first OSCARS open call. The aim was to identify mentorship needs for adopting PaNOSC outcomes and to discuss potential sustainability challenges. MC-ReDD was presented as a flash talk.
- 28 June 2025 | Schenefeld, Germany - European XFEL Open Day - Introducing European XFEL and a wide spectrum of its activities to the public. MC-ReDD was presented as a poster in the context of showcasing funded projects and their benefits for society.
- 28 January 2026 | Hamburg, Germany - McReDD presentation at the DAPHNE4NFDI satellite workshop on “Making data FAIR and AI-ready”, to the DESY user meeting 2026.
Other material
- McReDD workflow infographics - https://syncandshare.xfel.eu/index.php/s/aDFZ8zroQ8zSaGt
- Source code: https://github.com/COMCIFS/imgCIF_Creator
Principal investigator
Dr. Loes M.J. Kroon-Batenburg has been assistant professor at the Faculty of Science of the Utrecht University from 2003-2022, working on accurate data processing methods in crystallography, with focus on problematic and complicated diffraction patterns. She served on the International Union of Crystallography (IUCr) Diffraction Data Deposition Working Group (DDDWG) from 2011-2017, investigating the possibilities and practicalities of raw diffraction data archiving and metadata aspects. She has been the main-editor for the IUCrData journal section Raw Data Letters since 2022.
