Accurate and efficient orchard tree inventories play a crucial role in obtaining up-to-date information for effective treatments and crop insurance purposes. Surveying orchard trees, including counting, locating, and assessing their health status, is vital for predicting production volumes and facilitating orchard management. However, traditional manual inventories are labor-intensive, expensive, and prone to errors. Motivated by the recent advances in UAV imagery and computer vision methods, we propose a new framework for individual tree detection and health assessment. The proposed approach follows a two-stage process. First, we build a tree detection model based on a hard negative mining strategy using RGB UAV images. In the second stage, we address the health classification problem using two methods. We present a classical machine learning approach by exploring the use of multi-band imagery-derived vegetation indices. We also propose a new convolutional autoencoder-based architecture mainly designed to extract the relevant features for tree health classification. The performed experiments demonstrate the robustness of the proposed framework for orchard tree health assessment from UAV images. In particular, our framework achieves an F1-score of 86.24% for tree detection and an overall accuracy of 98.06% for tree health assessment. Moreover, our work could be generalized for a wide range of UAV applications involving a detection/classification process.