Content moderation is crucial for maintaining online safety and upholding the standards of websites and social media platforms. It protects users from inappropriate content and ensures their well-being in digital spaces. For advertisers, content moderation helps protect their brands from negative associations and contributes to brand elevation and revenue growth. In industries like finance and healthcare, content moderation plays a critical role in safeguarding sensitive personal and health information, enhancing digital security for users, and preserving privacy.
To improve content moderation, we introduce a novel method that utilizes multi-modal pre-training and a large language model (LLM) for image data. With multi-modal pre-training, the model can answer questions about the image content, allowing users to chat with the image to confirm if it violates any policies. The LLM generates the final decision, including safe/unsafe labels and category type, and can provide output in structured JSON format.
We use BLIP-2 as the multi-modal pre-training method, which is known for its performance in visual question answering, image captioning, and image text retrieval. For the LLM, we use Llama 2, an open-source model that outperforms existing language models on various benchmarks.
However, there are challenges in content moderation. Traditional human-based moderation is unable to keep up with the growing volume of user-generated content, resulting in a poor user experience and high costs. Machine learning-powered content moderation has emerged as a solution, but it requires careful consideration. Challenges include acquiring labeled data, ensuring model generalization, maintaining operational efficiency, providing explainability, and addressing the adversarial nature of content moderation.
To address these challenges, we propose using BLIP-2, a multi-modal model that combines computer vision and natural language processing. This model can handle and integrate data from multiple sources and tasks, such as image captioning, image text retrieval, and visual question answering. BLIP-2 achieves state-of-the-art performance and demonstrates zero-shot image-to-text generation capabilities.
In the solution architecture, we deploy BLIP-2 to an Amazon SageMaker endpoint and use it in combination with an LLM for content moderation. The solution requires an AWS account with appropriate permissions and the creation of a SageMaker domain. By leveraging these technologies, organizations can improve the efficiency and effectiveness of content moderation, focus resources on strategic tasks, and mitigate brand risks and legal liabilities.