Obtaining accurate information of defective areas of infrastructures helps to perform repair actions more efficiently. Recently, LiDAR scanners are used for the inspection of surface defects. Moreover, machine learning methods have attracted the attention of researchers for semantic segmentation and classification based on point cloud data. Although much work has been done in the area of computer vision based on images, research on machine learning methods for point cloud semantic segmentation is still in its early stages, and the current available deep learning methods for semantic segmentation of the concrete surface defects are based on converting point clouds to images or voxels. This paper proposes an approach for detecting concrete surface defects (i.e. cracks and spalls) using a Dynamic Graph Convolutional Neural Network (Dynamic Graph CNN) model. The proposed method is applied to a point cloud dataset from four concrete bridges in Montreal. The experimental results show the usefulness and robustness of the proposed method in detecting concrete surface defects from 3D point cloud data. Based on the sensitivity analysis of the model using three cases defined with different number of input points, the best test results show the detection recall for cracks and spalls are 55.20% and 89.77%, respectively.