Content Safety in Foundry Control Plane

Content Safety models have been specifically trained and tested in the following languages: English, German, Spanish, Japanese, French, Italian, Portuguese, and Chinese. The service can work in other languages as well, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.

Custom categories currently work well in English only. You can use other languages with your own dataset, but the quality might vary.

Some Content Safety features are only available in certain regions. See the features available in each region.

The system monitors across four harm categories: hate, sexual, violence, and self-harm.

Yes, you can adjust severity thresholds for each harm category filter.

Yes, you can use the Content Safety custom categories API to create your own content filters. By providing examples, you can train the filter to detect and block undesired content specific to your defined custom categories.

Prompt shields enhance the security of generative AI systems by defending against prompt injection attacks:

• Direct prompt attacks (jailbreaks): Users try to manipulate the AI system and bypass safety protocols by creating prompts that attempt to alter system rules or trick the model into executing restricted actions.
• Indirect attacks: Third-party content, like documents or emails, contains hidden instructions to exploit the AI system, such as embedded commands an AI might unknowingly execute.

Groundedness detection identifies and corrects the ungrounded outputs of generative AI models, ensuring they’re based on provided source materials. This helps to prevent the generation of fabricated or false information. Using a custom language model, groundedness detection evaluates claims against source data and mitigates AI hallucinations.

Protected material detection for text identifies and blocks known text content, such as lyrics, articles, recipes, and selected web content, from appearing in AI-generated outputs.

Protected material detection for code detects and prevents the output of known code. It checks for matches against public source code in GitHub repositories. Additionally, the code referencing capability powered by GitHub Copilot enables developers to locate repositories for exploring and discovering relevant code.

The content filtering system inside Azure OpenAl is powered by Content Safety. It's designed to detect and prevent the output of harmful content in both input prompts and output completions. It works alongside core models, including GPT and DALL-E.