Optical telescope is an important tool to obtain optical information of distant objects, and has wide application in astronomical observation, remote sensing, and optical surveillance. Resolution is an important indicator of a telescope's ability to observe objects in detail, and the traditional way to improve the resolution of a telescope is to build a larger aperture telescope, which leads to a significant increase in construction and maintenance costs. In this paper, an optic-neural network joint optimization method is proposed. The point diffusion function of the telescopic system is equivalent to a single-core convolution layer, which is integrated into the front end of the image super-resolution reconstruction network for joint training, and the point diffusion function obtained by phase mask reconstruction training is introduced into the optical path, so as to achieve collaborative optimization of the two, and effectively improve the resolution of the observed image. In this paper, a high-performance generative adversarial network is constructed, whose training parameters are smaller than the existing unsupervised networks, and the reconstruction speed is much faster than the existing unsupervised networks. This network adopts double discriminator architecture to improve the ability to extract detailed features. The designed cascade residuals make full use of the extracted feature information at all levels, expand the information propagation path, and improve the reconstruction efficiency. The simulation results show that compared with the simple deep learning method, the PSNR and SSIM of the super-resolution image reconstructed by the joint optimization method in this paper are increased by 3.98 and 0.06 respectively in the simulation data set, and the image details are rich and easy to distinguish. Verification experiments show that the fringe image reconstructed by the joint optimization method in this paper has the highest contrast and is easier to distinguish.