Quality Assurance Framework for Multimodal Assessment Datasets on AI Risk Factors

This paper introduces a quality assurance procedure for building and refining high-quality datasets to assess various risk factors associated with generative AI. The method operates through two key phases: (1) establishing a quality assurance framework and (2) its implementation. The establishment phase involves i) organizing a quality assurance team, ii) defining and specifying quality assurance metrics, iii) determining the appropriate timing for each metric, and iv) designing and developing the details of quality assurance methods. Based on the established framework, the implementation phase outlines assessment processes conducted over N iterative cycles. Our framework incorporates daily checks for structural accuracy, evaluation accuracy, content validity, and effectiveness, as well as weekly checks to ensure diversity across content, evaluation, and risk dimensions. This paper also presents the results of real-world applications of the proposed procedure, demonstrating its usefulness in improving dataset quality and addressing challenges in generative AI risk assessment.