Google’s contractors working to enhance the performance of its Gemini AI have been comparing its outputs with those from Anthropic’s Claude AI model, as seen in internal communications reviewed by TechCrunch. These comparisons are part of an ongoing evaluation process where contractors assess the quality of responses from both models based on multiple criteria, including truthfulness, verbosity, and overall quality.
While Google declined to comment on whether it had secured permission to use Claude in these tests, the process of benchmarking AI models against each other is not unusual in the tech industry. However, it’s typically done using industry-standard benchmarks rather than direct comparison with a competitor’s AI.
In their evaluation, contractors are given up to 30 minutes per prompt to determine whether Gemini or Claude provides the better answer. Interestingly, some contractors began noticing references to Claude in the internal platform where comparisons were made, and one of the outputs specifically identified itself as “Claude, created by Anthropic.”
Some contractors observed that Claude’s responses were focused more on safety than Gemini’s. For example, Claude's safety features were noted as being stricter compared to Gemini’s, with Claude refusing to answer certain prompts it deemed unsafe, such as role-playing a different AI assistant. In one instance, Claude even avoided responding to a prompt, while Gemini’s response was flagged for safety violations related to nudity and bondage.
Notably, Anthropic’s commercial terms of service prohibit using Claude to build competing products or services without approval. Google, a major investor in Anthropic, is reportedly using Claude as part of its efforts to refine its own AI systems.
This internal testing and comparison underscore the fierce competition in the AI industry, where companies are constantly seeking ways to improve their models and meet the growing demand for more advanced and reliable AI systems.
Summary:
Google’s Gemini AI is being tested against Anthropic’s Claude model.
Contractors are assessing the responses based on accuracy, safety, and other factors.
Google has not confirmed if it received permission to use Claude for comparison.
Claude’s safety settings are notably stricter than Gemini’s, with some responses flagged for safety violations.
Anthropic’s terms prohibit the use of Claude in creating competing AI models without approval.
