Earth observation by planes and satellites provide imagery data that may complement survey and register data. Convolutional neural networks (CNNs) are a class of deep learning algorithms that can automatically learn features from image data. DeepSolaris was a European project where CBS applied this technique for the first time to exploit aerial image data. The main aim of the project was to train CNNs to detect solar panels, as a first step towards the more general question of how much power is generated by solar panels in Europe. This paper is a follow-up study where we investigate how to improve model performance and show the sensitivity of the model to noise, sample size, class imbalance, class weights, data augmentation and weight initialization, offering provisional best practices for a quick start in exploiting earth observation.