According to the Incident and Trafficking Database (ITDB) of IAEA, nuclear security incidents occur in the world about once every three days, with theft of nuclear and radioactive materials being the most common. However, currently there are few effective approaches to monitoring and preventing the theft of nuclear materials. To this end, we propose a novel approach to improve the prevention and detection performance of physical protection systems (PPS) by employing deep learning-based object detection and human pose estimation models to analyze images captured by surveillance cameras and detect the acts of theft of nuclear materials in real time. We first deploy the YOLOv3 model for the object detection task to recognize and localize the storage containers of radioactive materials (e.g., polythene bottles for medical radioisotopes storage), lockers (e.g., cupboards, refrigerators), and tools (e.g., scissors, screwdriver). Subsequently, we detect persons and extract two-dimensional joint positions of the persons in the image using a pre-trained human pose estimation model, i.e., OpenPose. Finally, we dynamically measure the distance between the objects and the key points of the detected persons and set the thresholds to achieve real-time detection of whether the act is malicious. The performance of the proposed approach was experimentally evaluated for four types of security/malicious act detection: (1) removing a storage container of radioactive material from the locker and returning it (legal action), (2) removing a storage container of radioactive material from the locker without returning it (illegal), (3) damaging the locker using a scissor (illegal), and (4) damaging the locker using a screwdriver (illegal). The test results indicate that our approach was capable of detecting the malicious acts of theft of nuclear materials with high precision and recall rate while ensuring real-time performance. Further task of this study are considered to extend the malicious detection behavior and performance improvements based on time series analysis.