Embedded Files
cSmartML: A Meta Learning-Based Framework for Automated Selection and Hyperparameter Tuning forClustering
Novel technologies in automated machine learning ease the complexity of algorithm selection and hyper-parameter optimization. However, these are usually restricted to supervised learning tasks such as classification and regression, while unsupervised learning remains a largely unexplored problem. In this project, we offer a solution for automating machine learning specifically for the case of unsupervised learning with clustering, in a domain-agnostic manner. This is achieved through a combination of state-of-the-art processes based on meta-learning for algorithm and evaluation criteria selection, and evolutionary algorithm for hyper-parameter tuning. We introduce a robust and scalable interactive tool, named cSmartML, built on scikit-learn with 8 clustering algorithms. In order to capture more than a single measure of goodness of the output clustering solution, cSmartML optimizes multiple objective functions. A pareto-approach evaluates each objective simultaneously for each clustering solution. On each of the 27 real and synthetic benchmark datasets, we show that the performance of cSmartML is often much better than using standard selection and hyper-parameter optimization methods. In addition, experimentation reveals that cSmartML takes advantage of the defined objective functions on multi-objective functions framework.
Novel technologies in automated machine learning ease the complexity of algorithm selection and hyper-parameter optimization. However, these are usually restricted to supervised learning tasks such as classification and regression, while unsupervised learning remains a largely unexplored problem. In this project, we offer a solution for automating machine learning specifically for the case of unsupervised learning with clustering, in a domain-agnostic manner. This is achieved through a combination of state-of-the-art processes based on meta-learning for algorithm and evaluation criteria selection, and evolutionary algorithm for hyper-parameter tuning. We introduce a robust and scalable interactive tool, named cSmartML, built on scikit-learn with 8 clustering algorithms. In order to capture more than a single measure of goodness of the output clustering solution, cSmartML optimizes multiple objective functions. A pareto-approach evaluates each objective simultaneously for each clustering solution. On each of the 27 real and synthetic benchmark datasets, we show that the performance of cSmartML is often much better than using standard selection and hyper-parameter optimization methods. In addition, experimentation reveals that cSmartML takes advantage of the defined objective functions on multi-objective functions framework.
Project Publications:
R. El Shawi, H. Lekunze , S. Sakr. cSmartML: A Meta Learning-Based Framework for Automated Selection and Hyperparameter Tuning for Clustering. IEEE BigData 2021.[to appear]
Automated Selection and Hyperparameter Optimization for Supervised Tasks
Automated Selection and Hyperparameter Optimization for Supervised Tasks
A major obstacle for developing machine learning models using big data is the challenging and time consuming process of identifying and training an adequate predictive model. Therefore, machine learning mode building is a highly iterative exploratory process where most scientists work hard to find the best model or algorithm that meets their performance requirement. In practice, there is no one-model-fits-all solutions, thus, there is no single model or algorithm that can handle all data set varieties and changes in data that may occur over time. All machine learning algorithms require user defined inputs to achieve a balance between accuracy and generalizability, this is referred to as hyperparameter optimization. This iterative and explorative nature of the building of distributed process is prohibitively expensive with big datasets. In this project, we addressed different issues about the hyperparamter optimization problem including the scalability, and controlability.
A major obstacle for developing machine learning models using big data is the challenging and time consuming process of identifying and training an adequate predictive model. Therefore, machine learning mode building is a highly iterative exploratory process where most scientists work hard to find the best model or algorithm that meets their performance requirement. In practice, there is no one-model-fits-all solutions, thus, there is no single model or algorithm that can handle all data set varieties and changes in data that may occur over time. All machine learning algorithms require user defined inputs to achieve a balance between accuracy and generalizability, this is referred to as hyperparameter optimization. This iterative and explorative nature of the building of distributed process is prohibitively expensive with big datasets. In this project, we addressed different issues about the hyperparamter optimization problem including the scalability, and controlability.
We developed D-SmartML, a distributed AutoML framework on top of Apache Spark, a distributed data processing framework. Our framework is equipped with a meta learning mechanism for automated algorithm selection and supports three different automated hyper-parameter tuning techniques: distributed grid search, distributed random search and distributed hyperband optimization.
We developed D-SmartML, a distributed AutoML framework on top of Apache Spark, a distributed data processing framework. Our framework is equipped with a meta learning mechanism for automated algorithm selection and supports three different automated hyper-parameter tuning techniques: distributed grid search, distributed random search and distributed hyperband optimization.
In addition, we focused in this project focused on answering several crucial questions for the AutoML process including: 1) Which classifiers are expected to be the best performing on a given dataset? 2) Can we predict the training time of a classifier? 3) Which classifiers are worth investing a larger portion of the time budget to improve their performance by tuning them? In our Meta-Learning process, we used 200 datasets with different characteristics on a wide set of metafeatures. In addition, we used 30 classifiers from two popular machine learning libraries, namely, Weka and Scikit-learn. Our results and Meta-Models have been obtained in a fully automated way. The methodology and results of our framework can be easily embeded/utilized by any AutoML system.
Project Publications:
S. Amashukeli, R. Elshawi, S. Sakr. iSmartML: An Interactive and User-Guided Framework for Automated Machine Learning.
A. Abd Elrahman, M. El Helw, R. Elshawi, S. Sakr. D-SmartML: A Distributed Automated Machine Learning Framework. In2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS) 2020 Nov 1 (pp. 1215-1218). IEEE.
S. Dyrmishi, R. Elshawi, S. Sakr. A decision support framework for automl systems: A meta-learning approach. In 2019 International Conference on Data Mining Workshops (ICDMW) 2019 Nov 8 (pp. 97-106). IEEE.
R. Elshawi, S. Sakr. Automated Machine Learning: Techniques and Frameworks. InEuropean Big Data Management and Analytics Summer School 2019 Jun 30 (pp. 40-69). Springer, Cham.
R. Elshawi, M. Maher, S. Sakr. Automated machine learning: State-of-the-art and open challenges. arXiv preprint arXiv:1906.02287. 2019 Jun 5.
Machine learning Interpretability in Healthcare
Machine learning Interpretability in Healthcare
Although complex machine learning models (e.g., Random Forest, Neural Networks) are commonly outperforming the traditional and simple interpretable models (e.g., Linear Regression, Decision Tree), in the healthcare domain, clinicians find it hard to understand and trust these complex models due to the lack of intuition and explanation of their predictions. With the new General Data Protection Regulation (GDPR), the importance for plausibility and verifiability of the predictions made by machine learning models has become essential. Hence, interpretability techniques for machine learning models are an area focus of this project. In general, the main aim of these interpretability techniques is to shed light and provide insights into the prediction process of the machine learning models and to be able to explain how the results from the prediction was generated. A major problem in this context is that both the quality of the interpretability techniques and trust of the machine learning model predictions are challenging to measure. In this project , we propose fundamental quantitative measures for assessing the quality of interpretability techniques . In addition, we present a comprehensive experimental evaluation of six recent and popular local model agnostic interpretability techniques.
Although complex machine learning models (e.g., Random Forest, Neural Networks) are commonly outperforming the traditional and simple interpretable models (e.g., Linear Regression, Decision Tree), in the healthcare domain, clinicians find it hard to understand and trust these complex models due to the lack of intuition and explanation of their predictions. With the new General Data Protection Regulation (GDPR), the importance for plausibility and verifiability of the predictions made by machine learning models has become essential. Hence, interpretability techniques for machine learning models are an area focus of this project. In general, the main aim of these interpretability techniques is to shed light and provide insights into the prediction process of the machine learning models and to be able to explain how the results from the prediction was generated. A major problem in this context is that both the quality of the interpretability techniques and trust of the machine learning model predictions are challenging to measure. In this project , we propose fundamental quantitative measures for assessing the quality of interpretability techniques . In addition, we present a comprehensive experimental evaluation of six recent and popular local model agnostic interpretability techniques.
Project Publications:
R. ElShawi, Y. Sherif, M. Al‐Mallah, S. Sakr. Interpretability in healthcare: A comparative study of local machine learning interpretability techniques. Computational Intelligence. 2021 Nov;37(4):1633-50.
R. Elshawi, MH Al-Mallah, S. Sakr. On the interpretability of machine learning-based model for predicting hypertension. BMC medical informatics and decision making. 2019 Dec;19(1):1-32.
R. El Shawi, M. Al-Mallah, S. Sakr. Interpretability in HealthCare A Comparative Study of Local Machine Learning Interpretability Techniques. In2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS) 2019 Jun 1 (pp. 275-280). IEEE Computer Society.
Interpretability of Black box models
Interpretability of Black box models
In this project we focus on different kinds on Interpretability; model specific and model agnostic techniques. We developed ILIME, a novel technique that explains the prediction of any supervised learning-based prediction model by relying on an interpretation mechanism that is based on the most influencing instances for the prediction of the instance to be explained. We demonstrate the effectiveness of our approach by explaining different models on different datasets. Our experiments show that ILIME outperforms a state-of-the-art baseline technique, LIME, in terms of the quality of the explanation and the accuracy in mimicking the behaviour of the black-box model. In addition, we present a global attribution technique that aggregates the local explanations generated from ILIME into few global explanations that can mimic the behaviour of the black-box model globally in a simple way.
In this project we focus on different kinds on Interpretability; model specific and model agnostic techniques. We developed ILIME, a novel technique that explains the prediction of any supervised learning-based prediction model by relying on an interpretation mechanism that is based on the most influencing instances for the prediction of the instance to be explained. We demonstrate the effectiveness of our approach by explaining different models on different datasets. Our experiments show that ILIME outperforms a state-of-the-art baseline technique, LIME, in terms of the quality of the explanation and the accuracy in mimicking the behaviour of the black-box model. In addition, we present a global attribution technique that aggregates the local explanations generated from ILIME into few global explanations that can mimic the behaviour of the black-box model globally in a simple way.
Currently, deep learning models have been widely used in different application domains due to its notable performance. Explaining the decisions made by deep learning models is important for end-users to enable them to comprehend and diagnose the trustworthiness of the model. Most of the current interpretability techniques provide explanations in the form of importance score for the input pixels or features. However, summarizing such importance scores for input features to provide human-interpretable solutions is challenging. To this end, in this project, we developed Automated Concept-based Decision Tree Explanations (ACDTE), a novel framework provides human-understandable concept-based explanations for classification networks. Our framework provides end users with the flexibility of customising the model explanations by allowing them to choose the concepts of interest among a set of automatically extracted visual human-understandable concepts and infer such concepts from the hidden layer activations. Then, such concepts are interpreted through a shallow decision tree that includes concepts deem important to the model. In addition, ACDTE generates counterfactual explanations that state the minimum number of concepts to be changed to change the prediction.
Project Publications:
R. ElShawi, Y Sherif, M. Al-Mallah, S. Sakr. ILIME: Local and Global Interpretable Model-Agnostic Explainer of Black-Box Decision. InEuropean Conference on Advances in Databases and Information Systems 2019 Sep 8 (pp. 53-68). Springer, Cham.
R. El Shawi, Y. Sherif, S. Sakr. Towards Automated Concept-based Decision TreeExplanations for CNNs. InEDBT 2021 (pp. 379-384).
DLBench: Benchmarking of Deep Learning Frameworks
DLBench: Benchmarking of Deep Learning Frameworks
Source Code: Github Repository
Deep Learning (DL) has achieved remarkable progress over the last decade on various tasks such as image recognition, speech recognition, and natural language processing. In practice, the increasing popularity of deep learning frameworks calls for benchmarking studies that can effectively evaluate and understand the performance characteristics of these systems. In this work, we conduct an extensive experimental evaluation and analysis of six popular deep learning frameworks, namely, TensorFlow, MXNet, PyTorch, Theano, Chainer, and Keras, using three types of DL architectures Convolutional Neural Networks (CNN), Faster Region-based Convolutional Neural Networks (Faster R-CNN), and Long Short Term Memory (LSTM). Our experimental evaluation considers different aspects for its comparison including accuracy, training time, convergence and resource consumption patterns. Our experiments have been conducted on both CPU and GPU environments using different datasets. We report and analyze the performance characteristics of the studied frameworks. In addition, we report a set of insights and important lessons that we have learned from conducting our experiments.
Deep Learning (DL) has achieved remarkable progress over the last decade on various tasks such as image recognition, speech recognition, and natural language processing. In practice, the increasing popularity of deep learning frameworks calls for benchmarking studies that can effectively evaluate and understand the performance characteristics of these systems. In this work, we conduct an extensive experimental evaluation and analysis of six popular deep learning frameworks, namely, TensorFlow, MXNet, PyTorch, Theano, Chainer, and Keras, using three types of DL architectures Convolutional Neural Networks (CNN), Faster Region-based Convolutional Neural Networks (Faster R-CNN), and Long Short Term Memory (LSTM). Our experimental evaluation considers different aspects for its comparison including accuracy, training time, convergence and resource consumption patterns. Our experiments have been conducted on both CPU and GPU environments using different datasets. We report and analyze the performance characteristics of the studied frameworks. In addition, we report a set of insights and important lessons that we have learned from conducting our experiments.
Project Publications:
R. Elshawi, A. Wahab, Barnawi, S. Sakr. DLBench: a comprehensive experimental evaluation of deep learning frameworks. Cluster Computing. 2021 Feb 7:1-22.
N. Mahmoud, Essam Y, R. Elshawi, S. Sakr . DLBench: an experimental evaluation of deep learning frameworks. In2019 IEEE International Congress on Big Data (BigDataCongress) 2019 Jul 8 (pp. 149-156). IEEE.
Page updated
Google Sites
Report abuse