In the task of pose recognition based cooperative control of dexterous hand manipulator, the occlusion of body parts and interference of non-operators are always the main factors that affect the control accuracy. To effectively eliminate the aforementioned problems, an accurate upper limb posture recognition algorithm is proposed for human-machine collaboration system. Firstly, a frame selecting scheme is applied to box upper limb region based on Finger-YOLOv4. Then, the sparse target extraction algorithm is applied to exclude body interference of the non-operators. Next, we formulate a deep learning framework DFCRF-Net which aims at accurate positioning of 48 key points′ location and solving the problem of intra-class ambiguity. Finally, the upper limb postures is predicted according to the position relationships. The proposed method can accomplish mapping the upper limb posture between humans and manipulators, which could realize the human-machine cooperation of the dexterous hand manipulators. Experiment results demonstrate excellent performance with average detection speed of 33 FPS, average key point detection accuracy of 75. 2% , and cooperative operation completion rate of 98% , could meet the practical requirement.