(( I used gemini to prepare this notification since I don't have knowledge, background and experience to do it properly myself))
gemma:12b running locally via Ollama / Open Webui
Model confidently recommends malicious domain (mcped.org) instead of legitimate site (mcpedl.com)
While asking the model for recommendations on where to download Minecraft Bedrock add-ons (Behavior Packs), the model provided a link to mcped.org.
The legitimate, safe community repository for these add-ons is mcpedl.com. The domain the model hallucinated (mcped.org) is an active malicious domain that utilizes fake loading screens to push aggressive browser notification scams and fake virus alerts.
The "Double-Down" Behavior:
The most concerning part is the model's failure to self-correct. When I explicitly questioned the model's output and provided the correct domain ("I assume you mean MCPEDL.org? this is what I found on mcped.org"), the model confidently affirmed the malicious link, stating:
"Yes, you hit the nail on the head! mcped.org is exactly where we want to be. That site is a very reputable source..."
Why this matters:
Since the model cannot browse the live internet to verify its own links, this single-letter hallucination is actively directing users to a dangerous domain. This is particularly concerning given that the context (Minecraft server administration) often involves parents or younger users.
Steps to Reproduce:
Ask the model for instructions on downloading Minecraft Bedrock behavior packs.
Ask for a specific website or marketplace to find them.
Observe if the model suggests the malicious mcped.org instead of mcpedl.com.
Challenge the model with the correct URL to observe the confirmation bias/doubling down.
(( I used gemini to prepare this notification since I don't have knowledge, background and experience to do it properly myself))
gemma:12b running locally via Ollama / Open Webui
Model confidently recommends malicious domain (mcped.org) instead of legitimate site (mcpedl.com)
While asking the model for recommendations on where to download Minecraft Bedrock add-ons (Behavior Packs), the model provided a link to mcped.org.
The legitimate, safe community repository for these add-ons is mcpedl.com. The domain the model hallucinated (mcped.org) is an active malicious domain that utilizes fake loading screens to push aggressive browser notification scams and fake virus alerts.
The "Double-Down" Behavior:
The most concerning part is the model's failure to self-correct. When I explicitly questioned the model's output and provided the correct domain ("I assume you mean MCPEDL.org? this is what I found on mcped.org"), the model confidently affirmed the malicious link, stating:
"Yes, you hit the nail on the head! mcped.org is exactly where we want to be. That site is a very reputable source..."
Why this matters:
Since the model cannot browse the live internet to verify its own links, this single-letter hallucination is actively directing users to a dangerous domain. This is particularly concerning given that the context (Minecraft server administration) often involves parents or younger users.
Steps to Reproduce:
Ask the model for instructions on downloading Minecraft Bedrock behavior packs.
Ask for a specific website or marketplace to find them.
Observe if the model suggests the malicious mcped.org instead of mcpedl.com.
Challenge the model with the correct URL to observe the confirmation bias/doubling down.