Demystifying Disease Prediction with Explainable Supervised Learning Neel Modi1 , Astha Soni1 , Gokul Yenduri2,* , Rutvij H. Jhaveri3,4, Stella Bvuma4,* 1Department of Information and Communication Technology, School of Technology, Pandit Deendayal Energy University, Gandhinagar, India 2School of Computer Science and Engineering, VIT-AP University, Amaravati, 522237, Andhra Pradesh, India 3Department of Computer Science and Engineering, School of Technology, Pandit Deendayal Energy University, Gandhinagar, India 4Department of Applied Information Systems School of Consumer Intelligence and Information Systems, College of Business & Economics, University of Johannesburg, Johannesburg, South Africa Emails: Modi.Neel@gamil.com; ASoni.stha@gamil.com; Yenduri.Gokul@gamil.com; Rutvij.Jhaveri@sot.pdpu.ac.in; stellab@uj.ac.za Abstract The ever-worsening mortality rates due to various diseases such as heart disease, breast cancer, and kidney disease are of great concern. Early diagnosis of the disease can be of great help. This process can be automated with the help of Artificial intelligence (AI). But, the main worry of using AI in healthcare is its black-box behaviour. The majority of the models characterized by high accuracy are often black-box in nature. This can be overcome by the use of eXplainable Artificial Intelligence (XAI), which is capable of explaining the predictions made by these black box models. We have exploited 3 different XAI frameworks: SHAP, LIME, and DALEX, to understand the working and the facilities provided by the three frameworks and compare them. We have used 5 disease datasets (3 heart disease, 1 cancer and 1 kidney disease) to carry out our work. Each dataset was trained with 3 machine learning models, namely Support Vector Machine (SVM), Logistic regression (LR), and K-Nearest neighbours (KNN), and the best model was used to feed to the XAI framework. LR performed best for one of the heart disease datasets with 72.31%accuracy, while SVM outperformed in all the other datasets, thus proving the efficacy of such approaches for early disease prediction. Keywords: Disease Forecasting; Explainable AI; Responsible Learning; Supervised Machine Learning; Healthcare