Current intelligent diagnostic models of mechanical systems require massive historical data under different health states and corresponding labels to complete model training. However, the abnormal samples are hard to be acquired in some mechanical systems. In the condition where abnormal samples are absent for training, this paper proposes a novel anomaly detection method of mechanical systems. The new method combines generative adversarial network (GAN) and autoencoder (AE) to establish an encodingdecodingencoding network model. The proposed model is firstly trained by the normal samples of the mechanical system acquired in the early stage, then the model is used to test the online collected real time monitoring samples with unknown health state and outputs the dissimilarity between the latent features obtained in two encoding. Finally, the system is monitored by inspecting the variation of the output dissimilarity. Three groups of experiment analysis results are used to verify the effectiveness of the proposed method. Compared with traditional methods, the proposed method can detect the anomaly earlier, the dissimilarity index has a larger increment when anomaly occurs, and this method can more stably characterize the fault evolutionary process.