An Analysis of Unsafe Responses with Magic Expressions across Large Language Models

With advancements in AI models' ability to understand input, the risk of generating harmful outputs in response to malicious prompts has grown. This has led to increased awareness among researchers regarding AI safety, resulting in diverse ongoing investigations. However, existing studies have primarily relied on coarse-grained datasets to define risk factors, with limited availability of high-quality datasets that address fine-grained risks. To address this gap, we developed a high-quality, Korean AI safety evaluation dataset with overlooked risk factors in current LLMs. This dataset was curated with the involvement of human annotators to ensure its quality and relevance. We found that adding simple modifications, such as magic expressions, increases the likelihood of bypassing model guardrails. The analysis further revealed that this effect varies across different categories of risk factors. Additionally, we evaluated the harmfulness of LLM-generated outputs by measuring the frequency of risk-related keywords in the responses. This approach used prompt-based evaluation methods to quantify the degree of risk, providing a structured framework for assessing the potential dangers of model outputs.