The hand-crafted features in the traditional anomaly detection algorithms can not represent the appearance and motion patterns in a unified way for different scenes. In this paper, we propose a novel anomaly detection algorithm based on the convolutional variational auto-encoder (ConVAE). Firstly, a ConVAE which takes the raw frames series as input is constructed to extract the deep features of the scene. These deep features can represent the appearance and motion patterns more specifically. And then multiple Gaussian models are employed to fit the deep feature vectors of the corresponding receptive fields. The fitted Gaussian models which correspond to the receptive fields are used to decide the deep feature of the corresponding receptive fields from the test sample is anomalous or not. The proposed anomaly detection algorithm is evaluated the UCSD anomaly detection datasets. Experimental results show that the area under curve (AUC) of the proposed method are 95. 7% and 69. 9% in frame-level and pixel-level, respectively.