Aiming at the radial distortion in fisheye image, a method to correct the fisheye image by computing the radial distortion function is proposed in this study. First, the coordination of the image center and radius utilized for correction is obtained through Hough Transform according to characteristics of fisheye lens. The radial distortion calibration is accomplished by using the cylindrical checkerboard template which is proposed in this paper. The relation between the distance from the sampling points on radial direction of the image to the image center and the viewing angle of object points of the sampling points is achieved by fitting a polynomial. Then, the fisheye image is recovered to the spherical projective plane. Finally, the planar perspective image which is accord with human version is realized by using a planar projective model. A corrected 360° panorama is obtained by a cylindrical projective plane. The experimental results show that the proposed method is effective for correcting the radial distortion.