The application of Artificial intelligence (AI) in modern healthcare systems has emerged as an enabler to improve healthcare outcomes. Among all innovations, AI based triage and diagnostic systems is considered as revolutionary technology with potential to reduce burden on healthcare system and enhance productivity. In fact, the need of these triage systems has become even more evident during COVID-19 pandemic. With shortage in hospital beds during the pandemic, a diagnostic and triage system would help to evaluate hospital prioritization based on severity of condition and the likelihood of recovery (with or without hospitalization). This way healthcare system would become efficient in saving more lives with optimal use of healthcare resource. [ 1,2] In fact, previous studies using e-triage system have shown that Machine-Learning-based Electronic Triage more accurately differentiates patients with respect to clinical outcomes compared with the standard Emergency Severity Index and demonstrated slight improvements in discriminating patient resource utilization. [ 3,4,5]

MayaMD is an AI based virtual triage system that recognizes the importance of stratification of patients based on their broad illness and directs patients to the appropriate healthcare setting before they arrive at emergency department or clinics, which would otherwise not only require resource and lead to unnecessary crowding in hospital but also expose other patients as well as the attending staff to risk of communicable infectious diseases. It is essentially an online symptom checker that uses chatbot services to collect information regarding the patient’s symptoms or illness and thereby provides them with an appropriate triage system that guides them to the best course of action. The goal of triage systems is to reduce the in-hospital mortality and to minimize time for treatment, length of stay, and resource utilization [ 6].

Although MayaMD can improve patient care [ 7], its safety and accuracy has not been evaluated in Indian setting. Therefore, this prospective validation study was performed where identical cases were evaluated by both an AI system and Indian healthcare practitioners and then compared using the guidance of an expert medical team as standard.

Comprehensive, structured literature searches were conducted in PubMed and Google Scholar using key words Artificial Intelligence, Machine Learning, Triage, Online symptom checkers. Subsequently, the relevant papers published till the year 2021 were fully reviewed, and their findings were noted.

MayaMD is an AI-based application that patients may utilize to help determine where they should seek care for any medical condition. It requires the patients to input symptoms and answer subsequent questions, based on which they would be provided with likely diagnoses as well as whether to continue with self-care or seek primary, urgent, or emergency care services.

MayaMD uses a combination of Bayesian statistics and pattern recognition. Machine learning coupled with core algorithm recognizes new patterns, typically resulting from changing geographic or demographic data. MayaMD’s library and core algorithm is built on accepted evidence-based clinical knowledge and includes over 7,000 diagnoses, 8,500 initial inputs (symptoms, physical signs, and labs), 40,000 inferences, and 2,200 medications and interactions, that is being updated from time to time. [ 7] Similar AI-based applications have also been previously described. [ 8,9,10]

In this study, we compared the accuracy and safety of the MayaMD AI powered triage and diagnostic system with human doctors by means of clinical vignettes. Safety was assessed based on the triage action being rendered safe for the patient and not causing any potential harm. Accuracy was evaluated for the clinical vignettes by comparing the appropriateness of the recommended triage action by the individual HCP and MayaMD AI to that of consensus provided by a panel consisting of 3 doctors.

Although it is a non-interventional study, Ethics Committee approval was taken. The study was conducted in three phases. In the first phase of the study, twelve Indian practitioners of different medical specialties who were not involved in the development of the cases were requested to be part of the study. Each Indian practitioner was given a case report form with 5 different clinical vignettes with four triage options. Triage was based on four options, as mentioned inTable 1. This Case Report Form was shared with the HCPs as google forms. At the end of the phase, there were triage recommendation for 60 clinical vignettes (12 HCPs x 5 clinical vignettes each) by the Indian practitioners.

In the second phase of the study, the same 60 clinical vignettes were fed into MayaMD AI tool by a group of 3 medical graduates working as interns in a hospital. The triage recommendations by MayaMD AI were noted.

In the third phase of the study, a panel of 3 doctors was recruited to evaluate the same 60 clinical vignettes and give their triage recommendations. This panel of doctors had 2 internal medicine specialists and one general surgeon. The preliminary review for most cases in real-life scenario is done by internal medicine specialists. Therefore, they were recruited in the panel besides a general surgeon to give a holistic view of the most appropriate triage option for the cases presented. This triage recommendation by the expert panel was used as a reference to compare the accuracy of the triage recommendation by individual HCPs and MayaMD AI.

The cases in the form of clinical vignettes were provided by an expert medical team consisting of an internal medicine specialist, a cardiologist, and an ENT surgeon. To eliminate bias, none of the HCPs in the doctors’ panel, expert medical team or the individual HCPs evaluating the cases were repeated.

Each vignette was designed to simulate a medical condition from the list of all conditions currently modelled by the Triage and Diagnostic System. The vignettes contained information about the patient’s demographics like age and sex, their initial complaint(s), information about their symptoms and past medical history.

The study was conducted over a period of one-month time. Consent was obtained by all participants in the study.

Statistical analysis: The statistical analysis was done using chi-square test. A chi-square test for independence with α= .05 was used to assess whether the accuracy of the triage recommended by AI tool was better than that provided by Indian HCPs using the consensus triage as the standard. MayaMD triage decisions accurately matched with that of physician consensus in 55 out of 60 case vignettes (91.67%), compared to individual HCPs where accuracy was only 45 out of 60 case vignettes (75%). The chi-square test was statistically significant, χ2 (1, N=60) = 4.17, p =0.04.

The triage decisions of individual HCPs, MayaMD and 3 doctors’ panel for all 60 vignettes are presented inTable 2. The details of the clinical vignettes are presented in the Supplementary Materials.

MayaMD, performed significantly better than individual clinicians when determining a triage decision for a clinical vignette. MayaMD triage decisions accurately matched with that of physician consensus in 55 out of 60 case vignettes (91.67%), compared to individual HCPs where accuracy was only 45 out of 60 case vignettes (75%). MayaMD, performed significantly better than individual clinicians when determining a triage decision for a clinical vignette (91.67% vs. 75%, p-value= 0.04).

Our results from this prospective validation study demonstrate that an AI-based application, MayaMD, performs better than individual clinicians when determining a triage decision for a clinical vignette. In fact, AI system eliminated the risk of human error or bias. In cases where there was a discrepancy in the triage offered by consensus of doctors’ panel and MayaMD.AI, it was seen that the AI system had up triaged especially with geriatric population & symptoms like chest pain, abdominal pain, trauma, eye symptoms and contagious diseases. In these high-risk cases, MayaMD.AI recommended faster hospital visit keeping the best interest of the patient in mind. It is noteworthy to mention here that MayaMD.AI uses Bayesian network for data analysis which makes it possible to analyze complex cause-effect relationship. This enables it to read finer details and incorporate them in the triage recommendation unlike human doctors who can miss out on some important information leading to risk or over-cautious recommendations. Among discrepancies in the triage suggested by MayaMD.AI to that of clinicians and panel, the most common were A and B that were used interchangeably. This could be since both fall under similar compliance where the patient is bound to visit the casualty or emergency department.

An audit of 23 symptom checkers published in the year 2015 showed that triage performance varied by urgency of condition, with appropriate triage advice provided in 80% of emergent cases, 55% of non-emergent cases, and 33% of self-care cases (P<0.001). [ 12] The triage recommended by these symptom checkers was mostly risk averse, encouraging patients to seek medical care for conditions where self-care would have been reasonable. However, the technology of these symptom checkers has also improved over the years making their recommendation more appropriate, as suggested by this particular study using MayaMD.AI.

In India, while some patients tend to ignore serious symptoms, there&#x02019;s another group of patients who tend to schedule appointments with their doctors even for the modest level of medical problems under an impression of dangerous health issues or threats which often turns out to be a false alarm or something that can be treated with self-treatment. If symptom checkers are viewed as an alternative to just typing symptoms into a search engine like Google, then symptom checkers are most certainly a better approach. In this study, MayaMD down triaged few conditions where self-care would have been adequate to avoid unnecessary hospital/clinic visit. A recent study [ 11,12] found that when typing acute symptoms that would require urgent medical attention into search engines to identify symptom-related web sites, advice to seek emergent care was present only 64% of the time. Symptom checkers can be valuable tools for doctors, as well as providing access to health care in low-resource settings like rural areas or developing countries like India.

This study is the first done to the best of our knowledge with Indian healthcare practitioners to evaluate the safety and validity of triage recommended by AI based application to that recommended by them. Since MayaMD.AI could safely and appropriately aid in clinical decision making, its application in Indian healthcare setup could reduce the burden on the healthcare system and lessen the instances of lives being lost due to delay in patient reporting to the HCP or hospital, despite symptoms. The INDUS study evaluating the burden of heart failure in India showed that significant proportion of disease burden could be preventable with better screening and early and adequate treatment of risk factors. [ 13] According to WHO, doctor to patient ratio in India is exceedingly poor at 1:30000 compared to the recommended ratio of 1:1000. [ 14] Therefore, an AI based triage system would help in improving the effectiveness of the healthcare system. Moreover, significant number of Indians have been relying on online search engines [ 15] that could be risky when used to help determine urgency of a symptom.[ 8]

Over the last few years, digital health education has emerged as an important tool, and now it is more relevant than ever. Amongst global pandemic crisis, we extensively embraced digitized diagnosis in the form of AI led telemedicine. Artificial Intelligence aids in the seamless flow and automation of primary care, permitting doctors to focus more on critical and life-threatening cases which requires their serious medical opinion and judgment. Patients can benefit from medical chatbots, which are an AI-powered service fully integrated with smart algorithms that can provide patients with instant answers to all their health-related concerns and complaints further directing them on how to cope up with any potential problems, saving money on unnecessary trips to the doctor. These chatbots can handle many patients at any given point of time in full efficiency.

The results of this study would pave the way forward for use of AI based triage systems in Indian healthcare setting. These AI based systems would enable to drive efficiency without compromising on the accuracy as suggested by the study. However, this study had few limitations. The use of clinical vignettes is unlikely to reflect the intricacies of real-world patients and do not contain physical examination or test results. Also, geographic limitations, such as generalizability beyond urban healthcare settings in India, may limit the findings.

Despite these limitations &#x00026; to the benefit of the study design and patient pathway, a broad collaborative approach was used by up triaging the patient to aid clinical decision-making in the diagnosis of an emergency.