Autonomous mapping for mobile robot is the premise of completing intelligent behavior.To improve the intelligence and intuitive user interaction of robot, maps are needed to achieve the semantics beyond geometry and appearance. This paper studies the 3D semantic map construction method, which fuses the pixellevel image semantic segmentation based on Deep Residual Networks(DRN) and Simultaneous Localization And Mapping(SLAM).Firstly, the combined median filter algorithmis used to restore the depth of the map. The improved Iterator Closest Point (ICP) algorithm is employed to estimate camera pose and loopback detection based on random ferns is proposed for 3D scene reconstruction. Then, the optimized DRN is utilized to achieve more accurate semantic prediction and segmentation. Finally, the predicted semantic classification labels are migrated to the 3D model by Bayesian based incremental transfer strategy to generate a globally consistent 3D semantic map.Experimental results show that the proposed method can build the realtime 3D semantic map in the real and complicated environment.