Benchmarking (defined as “the process of comparing a practice’s performance with an external standard”) will provide transplant professionals with a means to assess how their unit is performing compared to the wider BMT community. In addition, it can reassure patients and payers with respect to the clinical performance and cost-effectiveness of their clinical BMT teams and services. Regulators see that benchmarking is an additional component to ensuring safe care of patients.
One obstacle to the adoption of benchmarking by the BMT community is achieving a fair and accurate methodology based on meaningful measures accepted by all stakeholders and a reliable source of clinical data. If the need and importance for benchmarking is accepted, the next question is how this can be implemented.
There are particular challenges in this respect. EBMT/JACIE covers a diverse collection of predominantly (but not exclusively) European countries each with their own individual health services serving the demands for BMT according to population, geography, political and economic aspects. Across EBMT, overall BMT caseload per centre is variable with annual numbers ranging from 5 patients (JACIE minimum for accreditation) in small, often autologous-only centres, to several hundred patients in the largest centres. Although national BMT and other professional societies have developed benchmarking exercises for survival outcomes in populous and developed countries e.g. UK, Italy, Switzerland, Germany, France and Belgium, these systems are not always comprehensive, consistent and cross-comparable, while in other countries, where BMT centres may be few in number and limited in size, there may be geographic and economic challenges in developing, resourcing and implementing robust ‘bespoke’ national benchmarking systems. Therefore, there is a clear opportunity for a single international system to answer these needs based on expert consensus.
The EBMT registry now contains over half a million BMT registrations and is currently being upgraded to a new platform called MACRO from Elsevier. This electronic data capture software has been modified to meet EBMT needs and forms part of the EBMT ‘Project 2020’. Data registration is a mandatory feature of full EBMT membership and is audited through JACIE accreditation. ‘Project 2020’ therefore offers an opportunity to incorporate benchmarking of transplant outcomes, sufficient at least in the initial phases for centres in all countries to meet the assessment required for the primary benchmarking JACIE standard (B4.7.5).
Therefore, a work package “Clinical quality assurance of patient survival outcomes by international benchmarking” was incorporated as an intrinsic part of ‘Project 2020’. Following initial discussions within meetings of the EBMT Board and EBMT Annual Meeting in Marseille in 2017-18, a statistical model for benchmarking was resourced and approved for development by the Leiden University Medical Centre (LUMC) team. Subsequently, a multi-national group of senior BMT clinicians with broad expertise, registry managers, JACIE officers and biostatisticians from LUMC and national societies are meeting systematically on a monthly basis from the Lisbon Annual Meeting in March 2018 to June 2019. CIBMTR and Australasian (ABMTRR) representatives were also co-opted in order to access their experience.
The principal objective of the Clinical Outcomes Group ‘Benchmarking Project’ is to provide a neutral platform to allow centres to compare their performance in the interest of improving care for their patients. Through using benchmarking as a tool for continuous improvement will not only identifying lower-performing centres and engaging with them to identify possible causes for poorer results but also to learn from those centres that consistently achieve high performance. To meet FACT-JACIE standards (B4.7.5), the goals are to provide centres with the means by which they can assess their 1-year post-transplant survival across all member centres irrespective of country.
The benchmarking mechanisms will consider a series of risk factors (called "case mix" variables) that should be integrated into the statistical models to remove bias in the comparison of centres related to different patient population characteristics.
There is a broader aspiration to contribute to the evidence base for benchmarking in HSCT.
Since work started in the first half of 2017, the benchmarking project has made important progress. After agreeing Terms of Reference, discussions progressed either by e-mail or telephone conference with periodic in-person meetings (approximate timeline in figure).
Statistical aspects of project implementation are being led by the team from the Department of Medical Statistics & BioInformatics, LUMC, with Hein Putter, Theodor Balan (programming), Erik van Zwet and Ronald Brand (as advisor in his retirement). There is input from the EBMT Statistical Committee via Myriam Labopin, along with national statistical representatives. The main work so far has focused on constructing case-mix-corrected funnel plots to compare centers to the national or European average performance with respect to 1-year and 5-year overall survival. Funnel plots are a very well-known tool for performance evaluation. The output will be a performance ranking of centers.
As part of the early development work, the LUMC methods have been applied to data from British and Italian centers for comparison with the existing BSBMT and GITMO mechanisms. It is important to note that the goal is not to supplant existing performance evaluations but to supplement them and to facilitate comparisons across Europe. This validation phase should conclude in March 2018.
Later in 2018 an expert group will be formed to discuss model assumptions, interpretation and limitations of the results, and consider the impact of the introduction of a benchmarking system and a process will be opened in which EBMT will decide how the benchmarking information should be delivered to the centres and how to prevent misinterpretation of results. Finally a period of formal consultation with the EBMT membership, including JACIE accredited centres, is planned for later stages of development before finalisation over 2019 to 2020.
John Snowden, Chair of Clinical Outcomes Group, Chair JACIE, and EBMT Board/Scientific Council member with Portfolio of Registry, Sheffield, UK
Nigel Brook, 2020 Project Manager, EBMT Central Registry Office, London, UK
Eoin McGrath, JACIE Operations Manager, EBMT Executive Office, Barcelona, Spain
Other Clinical Outcomes Group Members:
Erik van Zwet, Member, Member Statistical Committee, Holland
Hein Putter, Professor, Department of Medical Statistics and Bioinformatics, Holland
Ronald Brand, Prof.em. Good Research Data Management (GRDM), Biostatistician, Dep. of Biomedical Data Sciences, Section Advanced Data Management, Holland
Myriam Labopin, Chair of EBMT Statistical Committee, Paris, France
Carmen Ruiz de Elvira, Head of the EBMT Registry, London, UK
Kim Orchard, Chair of JACIE Accreditation Committee
Riccardo Saccardi, JACIE Medical Director, Florence, Italy
Rafael Duarte, EBMT Secretary, Spain
Alois Gratwohl, Physician, EBMT Honorary Member, Switzerland
Jakob Passweg & Helen Baldomero, EBMT/WBMT Transplant Activity Survey office Swiss National Transplant Registry SBST, Basel, Switzerland
Elena Oldani, Data Management, Secretary and Italian National BMT Registry – GITMO, Italy
Francesca Bonifazi, Principal Investigator, Secretary and Italian National BMT Registry – GITMO, Italy
Francis Ayuk Ayuketang, Physician, University Hospital Eppendorf
Julia Perry, Head of BSBMT Data Registry, UK
Rachel Pearce, Statistician, BSBMT Data Registry, UK
Noel Milpied, Hématologie clinique et thérapie celllulaire, France
Yves Beguin, Chair of Belgian Transplant Registry, Belgium
Doug Rizzo, Senior Scientific Director, CIBMTR (co-opted)
Steve Spellman, Scientific Director, CIBMTR (co-opted)
John Moore & Leonie Wilcox, Australasian Bone Marrow Transplant Recipient Registry (ABMTRR)
Per Ljungman, Chair of the Registry Committee, Sweden