Ensembles of Convolutional Neural Networks with Different Activation Functions for Small to Medium Size Biomedical Datasets

[abstract]

Because activation functions play a crucial role in the training of such deep learners as the Convolutional Neural Network (CNN), the development of more efficient and better-performing activation functions has been the focus of much recent research. Most activation functions introduce different nonlinearities when mapping the outputs of one layer to the inputs of another. This diversity makes activation functions an excellent candidate for generating ensembles of deep learners. In this study, we not only develop an ensemble of CNNs trained using different activation functions, but we also introduce a novel activation function. The objective is to improve the performance of CNNs on small to medium-sized biomedical datasets. Experimental results demonstrate that the best ensemble proposed in this chapter outperforms CNNs trained with the standard ReLU as well as with each of the other tested stand-alone activation functions (p-value of 0.01). For a more comprehensive evaluation, we test our approach on thirteen different biomedical datasets using three CNN architectures: Vgg16, ResNet50, and DenseNet. The MATLAB code used in this study will be available at https://github.com/LorisNanni.

Keywords:Deep learning; Bioimage classification.