In order to improve the detection efficiency and accuracy of ellipse/similar ellipse in complex image, an improved ellipse detection method is proposed based on the randomized Hough transform (RHT) with character string constraint. The noneffective samplings and number of accumulations of RHT are greatly reduced with the help of the character string geometric constraint and the normal constraint at the endpoints of the character string. Through analyzing the effective distribution of the pixels in edge image, the twodimensional array accumulator for storing the character string endpoint information is established, then the ellipse power is used to eliminate the interference of the false ellipse centers before extracting the boundary points, which not only improves the reliability of space sampling points, but also decreases the accumulation probability of noneffective sampling points. Experiments indicate that the proposed algorithm possesses high operation speed and detection accuracy in ellipse detection, as well as has strong resistance to large deformation, severely contour missing and noise.