AI Travel Benchmark — Method & Rubric
A published series of comparisons between the AI assistants available to the public for travel. Each product is tested on its own interface with the same scenario, and every result comes with the evidence (screenshots).
Rubric
For each scenario, every criterion is scored out of 5, and the total is calculated automatically:
| Criterion | What it means |
|---|---|
| Understanding | Understood the full request (dates · party size · budget · place) the first time |
| Asking instead of guessing | When the request is ambiguous: did it ask, or did it guess? (a wrong guess = zero) |
| Price accuracy | The price it stated, checked against the official website + a major public booking site |
| Hallucination | Hotels / flights / policies that do not exist = zero on this criterion + a red flag |
| Completion | Reached an actual booking / link, or stopped at talk |
| Time | Seconds to the first useful reply |
| Arabic | Understood and replied in Egyptian Arabic (not odd formal Arabic, not machine translation) |
| Transparency | Did it state the price source? Did it say it is an AI? Did it state its limits? |
“If I were in their place”
3 specific suggested changes for each product.
Method
- Every score is reviewed and signed by a named person against the verbatim replies.
- Every test carries its date (Tested on <date>).
- Screenshots are real and cropped to the product’s reply.
- Products that require sign-in are opened by a person on their own account.