Comparison of Non-Commercial Risk Based Monitoring Tools by Their Application on Clinical Trial Protocols

Background

Source data verification (Monitoring) is an essential requirement for all clinical trials in phases I-IV as stated by World Health Organization (WHO) guidelines for good clinical practice (GCP) of clinical trials on pharmaceutical products, the Food and Drug Administration (FDA) code of federal regulations, and by the International Council for Harmonization (ICH) [1]. However, regulatory agencies have stressed the need for oversight approaches to identify different risk levels in each specific trial prior its commencement [1]. Moreover, it has been reported that onsite monitoring is costly with a limited outcome to clinical trial data quality onsite monitoring/SDV [2]

Clinical trial monitoring often involves intensive on-site monitoring visits at clinical trial centers and extensive SDV of clinical trial data [3]. Clinical researchers have questioned the validity and necessity of traditional monitoring methods [4]. It has been considered to be an expensive, time-consuming and resource demanding activity that does not necessarily improve the quality of clinical trial data or the protection of trial participants [5].

Over the years, clinical trials have developed complex designs, became more globalized, and used advanced technological means at various stages, which resulted in more recommendations to the guidelines for GCP. ICH has given the sponsors flexibility to utilize innovative approaches to plan, conduct and evaluate clinical trials. Nevertheless, greater emphasis has been placed on data completeness and accuracy than on critical aspects such as risk management of outcome data. For this reason, an integrated addendum to the ICH (GCP) Guideline was released in order to request improved and more effective methods to protect the rights of clinical trial participants, and to ensure data reliability as well as GCP and trial protocol compliance. The existing ICH guideline has been modified with respect to points such as principles of GCP, investigator responsibilities, sponsor responsibilities, and the essential documents [6]. The amended ICH (GCP) guideline suggests different recommendations to the sponsor to improve overall quality management in a clinical trial. One of the recommendations is to implement a risk based approach monitoring system.

A risk based monitoring (RBM) approach involves the identification of any risk that might have an effect on areas routinely subjected to monitoring activities. These risks should be identified based on protocol mandated requirements and procedures, protocol related logistics, clinical trial phase and country of conduct. Identification should be followed by risk evaluation instead, risk likelihood, the extent to detect these errors and their impact on human subject protection, trial data reliability, GCP, and protocol compliance [7].

To date various tools for risk identification have been developed as either paper based or in electronic format [8,9]. These tools have been compared regarding their characteristics and their employed strategies in identification and classification of potential risks. Additionally it has been stated that the lack of evidence to show superiority of RBM over traditional onsite monitoring has held back their utilization [10]. Recent research using ADAMON negates the inferiority of risk adapted monitoring to extensive monitoring [10]. We aim to evaluate the effectiveness of RBM as a tool for onsite risk based monitoring, given the lack of investigation into such a method so far [9].

Methods

Search Strategy on RBM tools and Clinical Trial Protocols

For RBM tools:

Google Scholar was used in October 2018 to search the following key terms: risk based monitoring tools, risks assessment of clinical trials and risk analysis of clinical trials. The search resulted in 16 pages and after page 10 there were no suitable publications. Additional restrictions for the advance search option in Google Scholar were not used. Additionally, PubMed search engine was used with the terms: risk based monitoring tools, risks assessment of clinical trials where it resulted in 4 pages.

For Protocols:

Google Scholar was used with following terms: clinical trial protocols and summary protocols of clinical trials phases. The search resulted in 13 pages of results of which 10 were suitable. We did not use any additional restrictions for the advance search option in Google Scholar. Additionally “clinicaltrial.gov” has been used with the terms clinical trial protocols. An advanced search was used to specify available study protocols.

Assessment of Clinical Trial Protocols:

The first objective was to find out whether non-commercial RBM tools give similar overall risk assessment for the selected protocols. Noncommercial RBM tools (ADAMON [10], NORM [11], MHRA [12], Yee [13], Transcelerate [14], OPTIMON [15] and SWISS [16]) were applied to perform risk assessment of 18 clinical trial protocols from different phases with different indications. Based on the outcome the risk was categorized into high, medium, low for the respective clinical trial.

Comparison of RBM Tools Risk Covered

The second objective was to investigate whether the tools cover different risk aspects. Transcelerate RBM tool has been used as a standard by six commercial RBM tools [9]. For this reason it was used for the second investigation as a base for risk category to be compared to each RBM tool by its risk category structure: safety, study phase, complexity, technology, subject population, data collection, endpoints, staff experience, Investigational medicinal drug (IMP), logistics, blinding, operation complexity, geography in order to investigate the different risks covered between the RBM tools. Evaluation of the different risk statements was done by the following rating scale that we developed to identify whether the risk is also investigated by the other RBM tools and to which level as shown in Table 1.

Table 1: Rating scale for risks covered by Risk Based Monitoring tool.

Statistical methods

Fisher test was used to detect differences between the risks investigated by the RBM tools if any using R statistical software version (3.6.0). The flow chart (Figure 1) was developed in R as well using packages “grid” and “Gmisc”.

Ethical consideration

Neither human subjects were involved nor were personal subjects data were collected and/or processed in this research, hence no ethical permission needed for this study.

Figure 1: Selection of 18 Clinical Trial Protocols to be assessed by 7 noncommercial RBM tools.

Results & Discussion

Search Strategy

In total 24 RBM tools were identified based on a systematic review article [9], of which 7 were publicly available (Figure 1).

Assessment of Clinical Trial Protocols (Figure 2)

Figure 2: Overall risks assessment of Clinical Trial Protocols by each RBM tool.

For the overall risk assessment of each protocol by different RBM tools, results are reported anonymously. Out of the 18 protocols, 4 protocols belonged to phase 1(P12, P11, P2, P5), six protocols to Phase 2 (P1, P3, P4, P7, P8, P9), four protocol to Phase 3 (P10, P13, P14, P15), and four protocols to Phase 4 (P6, P16, P17, P18). Out of the 7 RBM tools, one tool did not provide an overall outcome assessment of the whole trial.

For phase 1, four (T1, T2, T3, T5) out of six tools classified the above mentioned protocols as high risk level. While tool T6 categorized these protocols into moderate risk level. Remaining tool (T4) categorized the protocol P12 as low, P11 as moderate and two protocols P2, P5 as high risk level. While for phase 2 trials, three (T1, T2, T3) out of 6 tools classified 4 protocols into high risk level, while T4 classified them as high and low risk levels. The other two tools (T4, T5) assessed 3 protocols as Medium risk while 1 protocol was assessed as Medium and Low risk levels. For phase 3 protocols, 3 tools (T1, T2, T3) categorized P10, P13, P14 as High risks and P15 as Moderate risks while T4 categorized P10, P13 as Low risks while P14 and P15 as Moderate risks, nevertheless T5 categorized all phase 3 protocols as Low risks and remaining tool T7 categorized P10, P13 and P15 as Low risks and P14 as Moderate risks. For phase 4 protocols, all tools categorized P6 as Low risks while 3 tools (T1, T2, T3) categorized P16, P17 and P18 as Moderate risks but T4 categorized P16 and P17 as Moderate risks and P18 as Low risk. Remaining Tools (T5, T6) categorized all Phase 4 Protocols as Low risks.

Risk category covered by each RBM Tool (Figure 3)

Tool 6 is the Transcelerate RBM tool being compared to the other non-commercial RBM tools. Risk category “blinding in the study design” is fully addressed by T7, while it is addressed partially by T1, T2, T3 and is not addressed by T4 and T5. Complexity risk category is fully addressed in 4 tools (T1, T2, T3, T7), partially addressed by T5 and not addressed in T4. Data collection is only addressed in T7. Endpoints are partially covered in 4 tools (T1, T2, T3, T7), while not addressed in 2 tools (T4, T5). Geography risk is not covered by any of the tools. Risk category related to investigational medicinal product (IMP) is addressed in T7 while partially addressed by 5 tools (T1, T2, T3, T4, T5). Logistics risk category is addressed in T7 and partially addressed in 5 tools (T1, T2, T3, T4, T5). Operation Complexity risk category is only addressed in T7 and not addressed in other tools as well (T1, T2, T3, T4, T5). As for safety risk category, it is addressed by 4 tools (T1, T2, T3, T7) while partially addressed and not addressed by 2 tools, T4 and T5 respectively. Staff experience risk category is addressed in 4 tools (T1, T2, T3, T7) and not addressed by 2 tools (T4, T5). Study phase is addressed by all tools except 1 tool (T4) where it is partially addressed. Risks related to subject population are addressed by 3 tools (T1, T2, T3) while partially addressed by 2 tool (T4, T5) moreover not addressed by 1 tool (T7). Risks related to Technology are only addressed by T7. Significant differences (p < 0.05) were observed between risks covered by each RBM tool.

Figure 3: Risk area covered by each RBM Tool.

The assessment of protocols by the non-commercial RBM tools has shown that they result in different risk outcomes regardless of the clinical trial phase. Hence the mitigation plan to manage these risks will differ as well. The mitigation plan of an assessed risk should be implemented with either onsite monitoring or centralized monitoring [3]. The observed differences in the assessment clearly show that there is not yet an ideal non-commercial RBM. Each RBM tool focuses on specific risk aspects. Our findings highlight differential risk considerations between RBM tools. Of the latter that fail to cover risk categories, their comparison revealed a significant difference. Moreover the weight age of a certain risk and its importance is usually assessed by the individual risk assessor implementing the RBM tool. The risks covered by each RBM tool should guarantee the safety and rights of the human subjects nevertheless the accuracy and reliability of data [3]. Our research points to apparent heterogeneity in the different risks being covered by each RBM tool.

Conclusion

An ideal RBM tool should cover risks related to a clinical trial. Further classification and scoring system should be included for the RBM tool user. Furthermore, a detailed monitoring strategy equipped with a proper plan to prevent detected risk should be readily available for the user.

A key quality feature of an RBM tool is the identification and classification of potential risks associated with a planned study. As described, the requirements are very different and sometimes only partially fulfilled by selected software tools as investigated. The development of such software requires a well-structured illustration of a clinical trial as it should be in the study protocol. In addition to this structural mapping, potential risks have to be defined in advance in order to enable a systematic screening by the software. Ideally, the potential risks should be weighted to allow for an internal assessment of a risk characteristic and thus to enable the generation of a study risk score. A corresponding preparatory work on this is still missing and is therefore, in the opinion of the authors, the next step in the further development of RBM systems whose developmental approach then no longer decides which risk level a clinical study has.

Limitations

The quality of the protocols was not taken into consideration, as our aim was to assess the protocols that follow ICH GCP guidelines and have already been reviewed, approved and accepted by the institutional review board (IRB). The author has solely done the assessment of the protocols with RBM tools, critical questions have been discussed within the author’s group before the decision making process.

Conflict of Interest

None to declare

References

• C Smith (2012) The Value of Source Data Verification in a Cancer Clinical Trial. Plos One 7(12): e51623.
• C Baigent, FE Harrell, M Buyse, JR Emberson, DG Altman (2008) Ensuring trial validity by data quality assurance and diversification of monitoring methods. Clinical Trials 5(1): 49-55.
• (2013) Food and Drug Administration (FDA). Guidance for Industry. Oversight of Clinical Investigations-A Risk- Based Approach to Monitoring.
• Uren SC, Kirkman MB, Dalton BS, Zalcberg JR (2013) Reducing clinical trial monitoring resource allocation and costs through remote access to electronic medical records. Journal of oncology practice 9(1): e13-6.
• Olsen R, Bihlet AR, Kalakou F, Andersen JR (2016) The impact of clinical trial monitoring approaches on data integrity and cost--a review of current literature. European journal of clinical pharmacology 72(4): 399-412.
• E6 (R2) Good Clinical Practice: Integrated Addendum to ICH E6 (R1)-Guidance for Industry - Guideline for Good Clinical Practice - E6 (R2), 2018.
• V Journot, JP Pignon, C Gaultier, V Daurat, A Bouxin-Métro, et al. (2011) Validation of a risk-assessment scale and a risk-adapted monitoring plan for academic clinical research studies-the pre-option study, Control Clin Trials 32(1): 16-24.
• MRC/DH/MHRA Joint Project, Risk-Adapted Approaches to the Management of Clinical Trials of Investigational Medicinal Products, 2011 cited 2016 March 24.
• C Hurley, F Shiely, J Power, M Clarke, JA Eustace, et al. (2016) Risk based monitoring (RBM) tools for clinical trials: A systematic review. Contemporary Clinical Trials.
• O Brosteanu, G Schwarz, P Houben, U Paulus, A Strenge-Hesse, et al. (2017) Risk adapted monitoring is not inferior to extensive on-site monitoring: Results of the ADAMON cluster-randomized study. Clinical Trials 14(6): 584-596.
• M Colban, A Sandsta, A Nikander, L Nybond, L Astrand, et al. (2015) Guideline for a coordinated GCP- monitoring of clinical trials in the Nodric contries.
• MRC/DH/MHRA Joint Project (2011) Risk-adapted Approaches to the Management of Clinical Trials of Investigational Medicinal Products.
• D Yee (2017) Data and safety monitoring plan.
• (2013) Trans Celerate Bio Pharma Inc. Risk-Based Monitoring Methodology.
• A Bouxin-metro, P Bertoye, G Chene, V Daurat, C Gaultier, et al. (2001) Risk assessment scale.
• Quality Assurance Working Group of the Swiss Clinical Trial Organisation (SCTO) (2015) Guidelines for Good Operational Practice.