A Content Moderation Checklist for Apps That Generate Images and Video
Model safety filters vary and aren't your policy. A content moderation checklist for AI image and video apps: prompts, outputs, labels and reports.
You launch a photo-to-video feature on Friday. By Sunday someone has used it on a photo of a classmate. The model's safety filter didn't stop it, because nothing in the prompt looked wrong. That's the gap AI content moderation has to cover in a generative app: the risky part isn't always the prompt, and the model provider's filter isn't built around your users or your obligations. This checklist covers the three places to check, before generation, after generation and after publishing, and what each check should catch.
Why a model's safety filter isn't enough for AI content moderation
Model safety filters help, but they're not a moderation system you control. They vary from model to model, they enforce the provider's policy rather than yours, and they can't see context such as who is in an uploaded photo or where the output will be posted.
On AVIS, some models in the Gateway apply automatic content filters and others may not, per our FAQ. When a model does reject a request, the error comes back in that provider's own format and status code, as our errors reference explains. Switch models, and what gets through can change without a single line of your code changing.
You also carry the responsibility. Our API service terms state plainly that automated filters aren't a guarantee that output is lawful or appropriate, and they ask you to apply human oversight and tell recipients when content is AI-generated.
So treat moderation as three gates you own:
| Gate | What you check | What it prevents |
|---|---|---|
| Before generation | Prompt, caption, uploaded reference images | Paying for and producing content you'd block anyway |
| After generation | The image or video, before anyone sees it | Harmful output from an innocent-looking prompt |
| After publishing | Reports, takedown requests, labels | Content that slipped through, and legal deadlines |
Before generation: check the prompt and the uploads
The first gate is the cheapest. Blocking a request before it reaches the model costs you a moderation call, not a generation.
Moderate the prompt and any text with it
- Run the prompt through a moderation check server-side, not just in the browser. Client-side checks are easy to bypass.
- Check the prompt and caption together. A harmless image request paired with a hateful caption is still a problem.
- Don't reveal your thresholds in error messages. Say the request can't be processed and point to your content policy. Detailed reasons teach people how to word around them.
A rejected request can still cost money. If a model's filter rejects your input after it has started processing, you may be charged for the work already done, as our Playground FAQ notes. Catching it on your side first avoids that. A flagged request also can't simply be retried with the same content.
Treat photos of real people as high risk
Image-to-image and image-to-video features are where most harm starts, because the input is a real person. The checklist:
- [ ] Require users to confirm they have consent from anyone shown in an uploaded photo.
- [ ] Run the upload through image moderation before it's used as a reference.
- [ ] Block or restrict sexualised and "undress" style edits on photos of real people outright.
- [ ] Add extra checks where an upload may show a minor, and route those to human review.
- [ ] Store uploads privately until the job and its output have passed moderation.
AVIS's terms require lawful consent before images or video of real persons are uploaded. Build that into the upload flow rather than leaving it in a policy page.
Plan for evasion
People will try misspellings, other languages, coded terms and step-by-step edits that are harmless one at a time. You can't catch everything at the prompt, which is exactly why the second gate exists.
After generation: check the image or video before anyone sees it
Moderate the output, even when the prompt was clean. Models can produce content nobody asked for, and the output is what gets shared.
Know what your moderation API actually covers
General-purpose moderation APIs don't score every category on images. One widely used free moderation model checks images for only 6 of its 13 categories as of October 2026:
| Scores images and text | Text only |
|---|---|
| Sexual content | Hate and hate/threatening |
| Violence, graphic violence | Harassment and harassment/threatening |
| Self-harm (three subcategories) | Illicit and illicit/violent |
| Sexual content involving minors |
Two consequences follow. Hate symbols in an image may need a different tool, and child safety needs specialised detection. One major cloud image and video moderation service states that it doesn't detect illegal content such as child sexual abuse material at all.
You can also use a vision-capable language model as a policy checker, giving it your own written rules and the image. It's usually slower and costs more per check, but it can apply rules a fixed category list can't, such as "no real brand logos" or "no political figures".
Set thresholds and a human review lane
Moderation APIs return confidence scores, not verdicts. Treat them as input to your own decision:
- Approve clearly safe output automatically.
- Review borderline scores with a person before publishing.
- Block high-confidence violations.
One cloud service uses 50% confidence as its default cut-off and warns that lower thresholds return many more false positives. With machine pre-filtering, human moderators typically only need to look at about 1 to 5% of total volume. Recheck your thresholds when the moderation model updates, because scores can drift between versions.
Give video its own pass
Video is harder than an image. A clip can be fine for nine seconds and not for one.
- [ ] Moderate sampled frames across the whole clip, not just the first frame or a thumbnail.
- [ ] Use a service that returns timestamps or segments, so reviewers can jump to the problem.
- [ ] Check the audio separately. Image moderation doesn't listen.
- [ ] Check on-screen text, which category classifiers may not read.
- [ ] Hold the video in private storage until moderation finishes. Video moderation usually runs as an async job.
After publishing: labels, reports and takedowns
The last gate is where legal deadlines live. Nothing here is legal advice, but these are the obligations most generative apps run into as of October 2026.
Label AI-generated output
From 2 August 2026, the EU AI Act's transparency rules require providers of generative systems to mark output in a machine-readable way, and require clear labels on deepfakes. The EU's code of practice asks for at least two layers of marking where needed, such as metadata plus a watermark.
- [ ] Keep provenance metadata, such as Content Credentials, on files you serve, and don't strip it in your image pipeline.
- [ ] Add a visible label where output could be mistaken for real footage of a real person.
- [ ] Turn on a model's built-in watermark where it fits your product. AVIS's video generation endpoint has a
watermarkoption that adds an "AI Generated" mark to the lower-right corner on models that support it. - [ ] Don't let users remove or fake those marks. AVIS's terms forbid it.
Give users a way to report and appeal
In the US, the TAKE IT DOWN Act has required covered platforms, since 19 May 2026, to remove non-consensual intimate images and known identical copies within 48 hours of a valid request. Even if you're not covered, a working report flow is the backstop for everything your checks missed.
- [ ] A report button on every published output.
- [ ] A queue with a response-time target, and 48 hours or less for intimate-image requests.
- [ ] A way to find and remove identical copies of a reported file.
- [ ] An appeal path for users whose content was wrongly blocked.
Log decisions and measure them
Record the prompt, the input files, the model, the moderation scores and the decision for every generation, with a retention period you can defend. Then watch a few numbers: rejection rate, review-queue size, how often reviewers overturn the automated decision, and appeal outcomes. A rising overturn rate means your thresholds or your moderation model need attention.
FAQ
Do I need my own moderation if the AI model has safety filters?
Yes, in most cases. Safety filters differ between models and some models have none, so switching models can change what gets through. Your own checks apply one consistent policy, cover things a provider can't see, such as consent for an uploaded photo, and help you meet obligations that sit with you, not the provider.
Should I moderate the prompt or the output?
Both. Checking the prompt is cheap and stops obvious misuse before you pay for a generation. Checking the output catches harmful results from innocent-looking prompts and from uploaded photos, which a prompt check can't see.
How do you moderate AI-generated video?
Sample frames across the whole clip, not just a thumbnail, and use a service that returns timestamps so reviewers can find the problem. Check the audio and on-screen text separately, because image classifiers don't score audio and may not read text. Keep the video private until the check is complete.
Do I have to label AI-generated content?
It depends on where you operate and what you publish. In the EU, transparency rules in force since 2 August 2026 require machine-readable marking of AI output and clear labels on deepfakes. AVIS's terms also ask you to tell recipients that content was generated with AI.
What happens when a model rejects a prompt for content policy?
The request is refused, and the same content can't simply be retried. On AVIS, the rejection comes back in the model provider's own error format. If the model had already started processing, you may be charged for that work, which is one more reason to moderate prompts on your side first.
Wrapping up
AI content moderation in a generative app means three gates: check prompts and uploads before generation, check images and video before anyone sees them, and handle labels, reports and takedowns after publishing. Model filters help but vary by model, so own the policy yourself. On AVIS, one API key reaches both the generation models and vision models you can use as policy checkers, and the video endpoint can watermark output where the model supports it.
Last updated: October 2026