Hello everyone. Our site was rebuilt last year on a JavaScript framework. It looks great, loads fast, and Google indexes it without complaint. We're now being told by an outside consultant that AI crawlers may not see it at all, which is alarming given what the rebuild cost us. Nobody on the original dev team has ever mentioned crawlers beyond robots.txt. Before I go back to them with accusations, I'd like to understand whether this is a genuine issue and how I could check it myself. Any pointers from people who have dealt with this?

Your consultant is right, and you can check it yourself in about a minute. Open DevTools, disable JavaScript, reload the homepage. Whatever is still on screen is roughly what AI crawlers get, because they don't render client-side content. A blank page means everything above it is wasted effort until that's fixed. The second thing nobody mentions: Cloudflare has blocked AI bots by default on new domains since July 2024, and editing robots.txt doesn't clear that, you change it in the bot settings. Both points come up in a LinkedIn breakdown of nine Dubai AEO agencies https://www.linkedin.com/pulse/best...nked-nine-including-myself-haseeb-ijaz-0trtf/ along with server log checks for GPTBot and ClaudeBot. One tip: run the log check before you argue with the devs.