The imgCIF format - recommended by the IUCr Commission on the Crystallographic Information Framework (COMCIFS) - offers a standardised, comprehensive way to store diffraction metadata alongside references to the original image data, so results can be reproduced with established data reduction software. In practice, however, assembling an imgCIF file by hand means tracking down scattered metadata across multiple sources - a task that is difficult for non-experts.
The OSCARS-funded project MC-ReDD – Metadata Collection and validation for Re-use of raw Diffraction Data – led by Loes Kroon-Batenburg (IUCr CommDat working group), with partners at European XFEL and the IUCr, set out to make this process automatic.
The team built imgCIF_Creator, a semi-automatic workflow that:
- Downloads raw diffraction data directly from public repositories where crystallographers deposit their images.
- Analyses file headers and completes missing metadata using the cctbx and dxtbx libraries.
- Generates a standards-compliant imgCIF file, complete with references back to the original image data.
The tool is available in two forms. imgCIF_Creator itself is a command-line program, installable via Conda, designed for authors working with local image files before depositing their raw data. It also powers a public web service hosted by the IUCr (imgcif.iucr.org), built with a Streamlit interface, which lets users generate an imgCIF file either from local DIALS-expt files or directly from a repository URL - with metadata fetched, extracted and converted automatically.
The project also explored extending imgCIF to describe Serial Femtosecond Crystallography (SFX) experiments conducted at X-ray Free-Electron Laser (XFEL) sources; a technical framework was outlined, though implementation wasn't completed within the project's timeframe.
Both the core Python tool and the web frontend are openly available on GitHub: github.com/COMCIFS/imgCIF_Creator.
A streamlined path to reusable crystallographic data
By removing the manual burden of metadata collection, MC-ReDD is designed to encourage more crystallographers to publish their raw diffraction data alongside their results, including “dark” data that doesn't lead to immediate publication, but which, properly annotated, can support long-term reuse and future analysis methods, including AI/ML approaches applied to historic datasets.
Given the reliance of life sciences, pharmaceuticals, electronics, batteries and wider materials industries on structural insight, more open and reproducible crystallographic data stands to benefit a broad applied research community well beyond crystallography itself.
- Final poster: DOI https://doi.org/10.5281/zenodo.21452655
- Project webpage: https://oscars-project.eu/projects/mc-redd-metadata-capture-and-validation-re-use-raw-diffraction-data
- A scientific article is being prepared as part of the project's deliverables and will be published on Zenodo.