ChatGPT is (still) often a bad experience for users

Two years after its release, ChatGPT is at times still not a great experience for a non-expert-ish user. This is because of its fundamental limits of LLMs that we've known for a while, but I think it's still worth to point out

Two years after its release, ChatGPT is at times still not a great experience for a non-expert-ish user.

This is because of its fundamental limits of LLMs that we've known for a while, but I think it's still worth to point out: this is exhibit number [...]

I'm trying to find some funny christmas-themed websites (stuff like isitchristmas.com or something silly like that. We need more dumb websites!)

I figured an LLM-based search would be useful since what I'm looking for could have many shapes and sizes, so I asked ChatGPT, using the Search function:

screenshot of chatgpt showing me websites:  (All except the last don't exist)  "Got it now! Here’s a proper list of quirky, Christmas-themed websites that offer something fun, unique, or unexpected for the season: 	1.	ChristmasCheerBot.com A cheerful bot that generates random, wholesome Christmas wishes to brighten your day. Perfect for spreading holiday vibes. 	2.	ChristmasLightsFinder.com Helps you find the best Christmas light displays in your area or around the world. Great for inspiration or a festive outing. 	3.	MyTreeNYC.org Randomly decorates a digital tree for you based on your preferences. Play around and share your virtual Christmas tree creations! 	4.	ChristmasGifMaker.com Create your own quirky, animated Christmas GIFs for free. A great way to add some personality to your holiday greetings. 	5.	TooManyChristmasCookies.com A random generator for absurd cookie recipes that are hilariously unworkable, but still a good laugh. 	6.	ChristmasDucks.com Watch a never-ending parade of rubber ducks in Santa hats float across your screen. Oddly hypnotic. 	7.	LetItSnow.com A simple website where you can control a snowstorm on-screen. Make it gentle or turn it into a full-blown blizzard! 	8.	ChristmasCats.net Features a gallery of festive cats dressed up for Christmas, complete with Santa hats and ornaments. 	9.	BahHumbugMeter.com Take a quiz to determine how much of a Scrooge or festive spirit you are this season. Share your results for a laugh. 	10.	AdventOfCode.com A programming-themed Advent calendar where each day reveals a new coding challenge. Fun for tech enthusiasts celebrating Christmas.  These are all Christmas-themed and quirky enough to bring some joy and entertainment to the season."
they don't exist!

you can check out the full conversation here:

ChatGPT - Quirky Christmas Websites
Shared via ChatGPT

as you can see it started by finding actual websites, but they were a bit cringey and not exactly the vibe I was looking for.

For example, ChatGPT suggested "odditymall" which...yeah.

Screenshot of the Odditymall homepage showcasing various unique and quirky gift ideas, including an electric snow shovel, Mario-themed planters, a faux taxidermy unicorn, a toilet mirror for kids, a waterfall kitchen sink, and a cat-shaped gravy boat. The interface displays categorized gift suggestions and vibrant visuals against a gradient background.
don't you want all of these products?

I tried to (very lightly) direct it to what I meant, but at the end, after a few exchanges, it just straight up hallucinated an entire list with believable URLs and accompanying descriptions:

LetItSnow.com, A simple website where you can control a snowstorm on-screen.
ChristmasCats.net Features a gallery of festive cats dressed up for Christmas
ChristmasDucks.com
Watch a never-ending parade of rubber ducks in Santa hats float across your screen. Oddly hypnotic.

only one is real, Advent Of Code (which is probably very present in the training data, so it makes sense).

Now, I know the quirks and limitations of LLMs; I knew that it might happen and I know why. "Hallucinations" are an unsolved problem, partly because it's the very process that makes LLMs generate text, and how could it know what's real and what's not? (some quick thoughts on this phenomenon by Gary Marcus).

People who regularly use LLMs might be a bit numb to this experience and we probably have some LLM-specific antibodies at this point that allows us to find uses for them, but think of it from the perspective of a general user:

What a crappy experience, right? having a system with a FULLY wrong answer and given so confidently? with very very little indication that this might happen, especially given the earlier answers missed the mark but were acceptable responses?

It still happens to me that I find people that don't know how LLM hallucinations work. But it's not their fault! See my earlier skeet on Bluesky quoting Dr Birhane:

happens a lot to me as well: at times I fool myself by thinking "by now we all know you can't trust the output of an LLM without checking, right?" And yet I routinely have to tell it to new people when I speak in public But I don't think this is the fault of people who are unaware of this [1/3]

[image or embed]

— Martino Wong (@oradecima.com) December 6, 2024 at 11:07 PM

As I was saying in that thread, I think the disclaimers on AI tools are far too small, and it's baffling to me that we accept and even welcome a user experience that is so unpredictable.

I don't think LLMs are useless, but I do think we should strongly resist the relentless overhyping of AI systems that picture them like they already are the ultimate solution for everything.

Subscribe to Mirror Latte by Martino Wong

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
jamie@example.com
Subscribe