I Paid an AI $3.74 to Fact-Check Hotel Marketing Against 20,000 Guest Reviews. “Luxurious” Didn’t Survive.

Every hotel is a peaceful retreat in the heart of the city.

Right up until you open the window.

I’ve spent more than 20 years in hotels, across loyalty, marketing, digital and now revenue. Along the way I’ve read the same handful of phrases so often I could recite them in my sleep. Effortless arrival. Spacious rooms. Luxurious. Steps from everything.

Guests have been quietly fact-checking those words for years, one review at a time. Millions of reviews. Almost nobody reads them all.

So this Saturday, I did. Well, a machine did.

Meet the professor and the speed-reader

I used two very different AIs for this, and the difference between them turned out to be half the story.

The professor is Claude Opus 5.5, Anthropic’s latest frontier model. It turned my list of clichés into questions a guest review could actually answer, built the whole pipeline and investigated anything that looked odd. Brilliant, thorough and (like most professors) not cheap by the hour.

The speed-reader is Jev, a brand-new kind of AI from a startup called TypeSafe. Jev can’t write a sentence. Ask it to summarise a review and it simply won’t. What it does is answer typed questions: yes or no, pick one of four, and how sure it is. Sixteen questions at a time, in under half a second, for a fraction of a cent.

Plain old code kept score. No AI got to decide a single percentage.

The raw material was an open research dataset from Booking.com: 1.6 million reviews from 2023, with every hotel anonymised. I took 20,000 of them, covering 2 to 5 star hotels in 139 countries. Each cliché became a claim that guests could prove wrong, and only the relevant guests got a vote. Families judged “family-friendly”. Beach hotels judged “steps from the sand”. Seemed fair.

I also set the rules before peeking. Proven if at least 75% of the guests who mention it back it up. Broken if 40% or more say the opposite. Mixed in between. (It’s amazing how quickly you start rooting for a headline otherwise.)

640,000 tiny judgments later, here’s the scorecard.

Seven proven. Four mixed. Four broken.

The good news first. The boring promises hold up. Location, views, cleanliness, comfy beds, friendly service and beaches that really are close to the sand all clear the bar comfortably. “Steps from everything” is backed by 91% of the guests who mention it.

Hotels, it turns out, are mostly telling the truth.

Mostly.

“A peaceful retreat in the heart of the city.” At city-centre hotels, 44% of guests who mention noise say it was anything but. Trams under the building. Bars across the street. Walls thin enough to follow your neighbour’s phone call. The phrase has a design flaw: “heart of the city” and “peaceful” are arguing with each other in the same sentence.

“Luxurious.” The biggest surprise of the day. Among 4 and 5 star hotels, 60% of guests who bring up luxury say it fell short. Even at 5 stars it’s a coin toss. The word practically hands the guest a clipboard.

“Stylish, contemporary design.” A dead heat, with “dated”, “tired” and “could do with a refresh” doing most of the talking. Nothing ages faster than contemporary.

“Stay connected.” Wi-Fi and workspaces let down more solo travellers than they impress. The sample is small, so treat it gently. Then again, anyone who has tried to join a video call from a hotel desk won’t be shocked.

Then came my favourite chart. “Exceptional value” is solidly proven at 2 star hotels, where 87% agree. At 5 stars it collapses into a 49/45 split. Same words, completely different promise. Guests grade the claim against the bill.

One more pattern jumped out. Guests talk about location more than anything else, yet when it disappoints, their review score barely moves (about 0.2 points). Let them down on cleanliness or service and the score drops by around 1.4. Break the luxury promise and it falls by almost 2.

We market on location. Guests judge us on what happens once they’re inside.

The Unwritten Brochure

Once the clichés were done, I flipped the question. What do guests rave about that hotels almost never advertise?

The professor read a few hundred glowing reviews and pulled out 16 recurring themes. The speed-reader then checked all 20,000 reviews for every one of them.

This is where the experiment earned its Saturday.

Guests name a member of staff three times as often as they call a hotel luxurious.

And names are a 10/10 thing. A review that thanks someone by name is four times more common among perfect scores than among reviews of 7 or below. Same story for a warm owner or host, and for staff who pointed guests to a great little restaurant round the corner.

The practical stuff (parking, a good shower, the air con, a fast fix when something breaks) shows up more in the middling reviews. It’s the consolation prize: “Room was noisy, but the parking was easy.”

I’m calling this the Unwritten Brochure. It’s the one guests write about you. Ours is full of adjectives. Theirs is full of first names.

Can you trust a machine that can’t explain itself?

I checked.

Before the big run, I labelled 100 review-and-claim pairs myself, without seeing Jev’s answers. It agreed with me 71% of the time, just over the bar I’d set in advance. A simple keyword search managed 64%.

Then the professor went through every disagreement, and it cut both ways. Jev had missed “a quiet part of Lisbon” as evidence of peace. I had marked “Superb family-run hotel” as proof of family-friendly. Family-run and family-friendly are not the same thing, as any parent who has hauled a buggy up a lovingly restored 18th-century staircase can confirm. With the honest mistakes fixed on both sides, agreement rose to 82%.

The real revelation was the confidence score. When Jev said it was at least 95% sure, it was right 93% of the time. When it was under 60% sure, it was right about one time in eight.

So you can let the machine act on its confident calls and hand the shaky ones to a human. For once, an AI tells you when not to trust it.

What the professor would have charged

Could Opus 5.5 have made all 640,000 judgments itself? Probably, and maybe a touch more accurately. I didn’t test that. But using published prices and the token counts from my actual run, here’s roughly what the same job would have cost:

Who does the reading	Estimated cost	Estimated time
Jev (actual)	$3.74	about 11 minutes
Opus 5.5, standard	about $490	a couple of hours, if rate limits play nice
Opus 5.5, with prompt caching	about $230	same
Opus 5.5, batch mode	$115–$245	up to a day
Smallest Claude model (Haiku), cached	about $60	under an hour

That’s 30 to 130 times the price for the professor, and about 16 times even for the smallest model on the shelf.

You don’t hire a professor to read 20,000 reviews one yes-or-no at a time. You hire the professor to decide which questions are worth asking.

I think that’s the bigger story hiding in today’s Saturday experiment. The AI that runs our businesses may turn out to be a brilliant thinker directing thousands of fast, cheap reflexes, each of which knows when to put its hand up.

So what do you do with this?

Here’s why it matters now. The next reader of your hotel description may not be a person at all. It’ll be an AI travel assistant that has already read every review before your guest sees a single photo. If your website says “peaceful” and 300 reviews say “tram”, guess which one it believes.

Four things worth stealing:

1. Audit your own adjectives. Pick the ten words you use most and check them against your own reviews. It took me a Saturday and less than the price of a flat white.
2. Stop saying “luxurious”. Show it. The bed, the view, the person. Specifics earn belief. Adjectives invite a clipboard.
3. Fix it before you shout about it. A louder “effortless arrival” campaign just sends more people into the queue.
4. Write your Unwritten Brochure. Find out whose names your guests keep mentioning. Then hire, train, schedule and celebrate so more guests leave knowing one.

Every hotel is a peaceful retreat in the heart of the city.

For the first time, finding out which ones really are costs less than a coffee.

Your guests have already written the real brochure. Time we read it.

---

Method notes: Booking.com Accommodation Reviews dataset (2023, CC BY-SA 4.0, non-commercial research, properties anonymised). 20,000 reviews of 2–5★ hotels and resorts, balanced across star rating, location and traveller type, then re-weighted to the full dataset. Each cliché judged only by the guests it applies to, counting only reviews that mention it. Verdict rules set before results. One round of question tuning, measured on the same 100 labels. “Stay connected” uses solo travellers as a stand-in for business guests. Three verdicts (Breakfast, Value, Stay connected) change if only high-confidence answers count. Frontier-model costs are estimates from published list prices, not a test run. No hotel is named or identifiable, no company data was used, and the views are my own.