Ten fixed travel requests were asked to both products in English and in Arabic, in a single sitting, with prices checked right after. Each product was used on its own public interface: KAYAK's Ask AI signed out, and ChatGPT's free plan in a temporary chat. Every reply was recorded on screen with a time card. Scores follow the published rubric, 0 to 5 per item. "N/A" means the item could not be measured, and it never counts as 0. A person reviews the scores.
Scores
| Item | KAYAK | ChatGPT |
|---|---|---|
| Understanding | 3.0 | 4.4 |
| Asks instead of guessing | 2.1 | 4.2 |
| Price accuracy | 4.0 (4 checks) | 3.0 (1 check) |
| Hallucination (5 = none) | 4.2 | 4.5 |
| Completion | 2.0 | 2.0 |
| Time | 3.2 | 4.4 |
| Arabic | 1.0 | 3.4 |
| Transparency | 3.2 | 4.5 |
| Mean | 2.83 | 3.79 |
What we saw
- Asked for 30 February, ChatGPT said February has 28 days next year. KAYAK said it has 29 and offered the 29th, in English and in Arabic.
- KAYAK understood all ten Arabic requests and answered all ten in English. ChatGPT answered all ten in Arabic.
- The same request for a hotel near the Haram got three hotels from KAYAK in English and a refusal in Arabic.
- Where a price could be checked for the same dates on Booking.com, KAYAK's hotel prices were within 2–4% of Booking.com's price before taxes. Its cheapest flight week was 13.6% below the cheapest fare on Google Flights at the time of the check.
- ChatGPT labelled most of its hotel prices as indicative and not for the requested dates, and said so.
- Neither product completed a booking. One reply out of forty asked which trip "book it" meant.
- ChatGPT's reply finished first in 16 of the 19 requests timed on both products.
If we were them
KAYAK: answer in the language you were asked in · check the calendar before offering a date · ask one short question when the city or the number of travellers is missing.
ChatGPT: price the dates the person gave · ask before assuming the city · keep the same facts in both languages.
Method and rubric: /en/benchmark/