Con-API’s Safety Monitoring Research
Enhancing construction site safety through intelligent detection, prediction, and intervention systems.
Our research in Safety Monitoring focuses on combining computer vision, AR/VR solutions, semantic analysis, and AI-powered alerting systems to proactively detect and mitigate worker hazards on job sites. Our work encompasses identifying unsafe activities to assisting with compliance checks. This division aims to create a safer, more responsive, and more transparent construction environment.
Research themes:
- Automated hazard and near-miss detection
- Vision-based safety compliance monitoring
- Natural language processing for safety documentation and checklists
- AR/VR safety training and intervention
- AI ethics in construction safety technologies
Featured Representative Works
1. Tsai, W. L., Le, P. L., Ho, W. F., Chi, N. W., Lin, J. J.*, Tang, S., & Hsieh, S. H. (2025). Construction safety inspection with contrastive language-image pre-training (CLIP) image captioning and attention. Automation in Construction, 169, 105863. (SCI) ❐
The developed mobile application preview from Figure 4 and The attention map with generated captions predicted by the model during testing from Table 7.
This study introduces an AI-powered safety inspection framework that utilizes contrastive language-image pre-training (CLIP) and attention mapping to automate the detection and description of unsafe site conditions from images. A mobile application is then programmed to package the entire framework into a single process, improving site inspection efficiency.
First, A CLIP model is first fine-tuned to produce CLIP embedded images and attributes. Then, a prefix captioning pipeline then concatenates the caption, attribute, and image embeddings of the input image. Finally, a GPT-2 Language model can then automatically generate safety observations from construction site image. By fine-tuning CLIP for nine distinct violation categories, the system achieves 73.7% average classification accuracy, outperforming baseline methods by over 40%.
The resulting mobile application allows inspectors to quickly capture images, receive AI-generated violation captions, and compile safety reports. User feedback suggests that the app streamlines inspections, reduces reporting time, and improves documentation quality.
2. Le, T.-H., & Lin Jacob, J. (2024). Reality and BIM Model-Driven Near-Miss Alerting Framework for Construction Equipment Using AR Interface. In Computing in Civil Engineering 2023 (pp. 257-265). (ASCE) ❐
The AR/VR Alert system proposed by the authors during testing. A simple “Heavy Vehicle approaching from behind” scenario is used.
The authors propose an AR-based safety alert system for real-time collaboration between field and office staff in identifying and addressing near-miss incidents. The system is designed to prevent struck-by accidents and near-misses involving construction equipment. A 3D reality-synchronized virtual environment of the job site is created from integrating BIM models with real-time AR localization, enabling equipment movement tracking and proximity-based hazard detection.
The system will continuously detect nearby moving objects, evaluate potential collision risks, and delivers real-time AR visual alerts to workers. Live tests in construction scenarios shows that the proposed AR technology can significantly improve situational awareness, enabling both on-site workers and off-site supervisors to collaboratively identify and mitigate hazards before they escalate.
Conference Highlights
Lai, Y. H., Lin, J. J., and S. H. Hsieh (2022). “Falling from Height Prevention Framework with BIM and Semantic Point Cloud,” Proceedings of the 26th Symposium on Construction Engineering and Management, Paper No. 159, July 22, 2022, Taoyuan, Taiwan.
Le, T. H. and Lin, J. J. (2023). “Reality and BIM model-driven near-miss alerting framework for construction equipment using AR interface,” Proceedings of ASCE International Conference on Computing in Civil Engineering (i3CE 2023), June 25-28, 2023, Corvallis, OR, USA.
Le, P.-L., Lin, Jacob J. (2024). Technical Requirements for Applying Digital Technologies in Monitoring Unsafe Activities during the Construction Phase. In International conference on construction engineering and project management (pp. 431-438). Korea Institute of Construction Engineering and Management.
Project Involvement within this field
以語意分析模型檢驗工程設計成果一致性
Examine the consistency of engineering design with semantic analysis models.
2022.Feb – 2023.Feb. PI
Sponsor: Sinotech Engineering Consultant
以自然語言處理擷取施工規範查驗項目並輔助查驗表單的設計與檢核
Employing natural language processing to capture construction specification inspection items and assist in the design and verification of inspection forms.
2023.Jun – 2024.Jun. Co-PI
Sponsor: National Science and Technology Council (NSTC)