To address the challenges of close-range, ultra-large-scale measurement in on-orbit photogrammetry and variations in the internal parameters of the measurement system, a novel fisheye camera on-orbit calibration method based on starlight vectors is proposed. First, an improved star pattern recognition algorithm is developed based on the state identification method. The algorithm is less sensitive to image distortion and effectively reduces redundancy in fisheye star pattern recognition using a four-star triangle joint decision method, thereby improving recognition speed. Second, to address the loss of recognition accuracy and the limited acquisition of full-field data for calibration caused by variations in the internal parameters of the on-orbit fisheye camera, a self-calibration bundle adjustment method is developed. This method integrates star-pattern recognition with a regional iterative expansion strategy. The algorithm progressively expands the field-of-view region and iteratively optimizes the camera′s internal and external parameters by incorporating recognition results from each region, gradually improving the overall recognition rate until the calibration requirements are satisfied. Finally, experimental validation is carried out using a multi-attitude star pattern dataset constructed from 246 fisheye images of the celestial sphere. Ground-based experiments show that when the recognition threshold is relatively large (0.25°), the improved star pattern recognition algorithm requires only 36.91% of the time consumed by the state identification method, demonstrating a significant improvement in efficiency. Moreover, when the internal parameters of the camera change considerably (principal distance variation of 0.15 mm), the proposed algorithm achieves full-field star recognition, with the recognition rate reaching 98.6%, thereby satisfying the requirement for full-field data in camera calibration. After calibration, the root mean square error of the reprojected star point image coordinates is 1/5 pixels. The experimental results indicate that the proposed algorithm enables high-precision self-calibration of measurement system parameters, providing an effective solution and reference data for addressing the calibration challenges of system parameters in on-orbit visual measurement applications.