Semantic information can help the robot to better understand unknown environment and lay the foundation for more advanced humancomputer interaction and more complicated task. To enable mobile robot to build semantic map in real time, a light deep learning model is developed for object detection on embedded computer Jetson TX1. The interframe optical flow information in the video stream is used to reduce the missing rate of object detection algorithm, which is called motion guided propagation (MGP) algorithm. A realtime depth map restoration algorithm based on CUDA is utilized because the depth map generated by Kinect has black hole and black border. SLAM technology is employed in this paper for robot location, navigation and mapping. On this basis, Bayesian inference framework is integrated with measurement information of environment and object detection information to complete the building of semantic map. Experiments show that the proposed method can enable the mobile robot to build the semantic map in real time in the real, complicated indoor environment.