Web search, scraping and browsing

Your agent can search the web, read pages and pull information from sites without any setup. Web search is built in: there's no API key to get and nothing to switch on. Most problems come from sites that deliberately block automated visitors, and for those there's usually a better route.

What's built in

• Web search. Your agent can look things up and read the results. It's managed for you, so don't add your own search provider or key.
• Reading pages. Give it a link and it can read the page.
• Scraping tools. For pulling structured data from sites, your agent has paid scraping tools built in.
• A browser. OpenClaw agents can open pages in a browser and click through them. Hermes agents can browse too.

All of this counts toward your monthly AI usage. The Usage tab shows Web searches for the month, and scraping shows up as Paid tools in the activity table. See AI usage, models and top-ups.

When search comes back empty

• Make the request specific. "Latest news on AI" is vague. "Three news articles from this week about OpenAI's pricing changes, with links" works far better.
• Name the source if you know it. "Check the pricing page on calendly.com" beats "find Calendly's prices".
• Ask it to try again differently. "Search again with different terms and tell me what you searched for."
• Check your allowance. If your monthly AI usage has run out, your agent pauses AI replies, searches included. The Usage tab shows where you stand.

Don't ask your agent to set up its own search provider or paste in a search API key. Search is already handled.

When your agent can't open a link

What happens	Usually means	What to do
"I can't access that page"	The page needs you to be signed in	Paste the text or upload the file instead
It reads a different page than you meant	A shortened or redirecting link	Send the full, final address
Google Docs or Drive links fail	The file isn't public	Connect Google Drive in Integrations, or share the file publicly
It works for some pages but not others	The site blocks automated visitors	See below

Sites that block agents

Some sites actively stop automated tools. Your agent can't get past these, and shouldn't try:

• CAPTCHAs and "checking your browser" pages, like Cloudflare checks. These exist to stop bots. Your agent won't solve them.
• Instagram, TikTok, Facebook and LinkedIn. They block most automated reading and posting. Accounts used for automation can get restricted.
• Anything behind a login. Your agent can't sign in to websites as you.

When it can read a page but can't click buttons

Reading a page and using it like a person are different jobs. Pages built from heavy scripts, pop-ups or multi-step forms often stop an automated browser partway through.

What helps:

1. Give it a direct link to the exact page, not the homepage.
2. Ask it to describe what it sees before it clicks, so you can spot where it gets stuck.
3. If the site has an app in Integrations, use that instead of the browser. It's faster and more reliable.
4. For anything that sends, buys or publishes, do the final click yourself.

Frequently asked questions

Do I need a search API key?

No. Web search is built in and managed for you. Don't add your own key or provider.

Does web search cost extra?

It's included in your monthly AI usage. The Usage tab shows how many web searches your agent has run this month.

Why can't my agent read Instagram or TikTok?

Those sites block automated access. Paste the text, upload a screenshot, or have your agent draft posts for you to publish yourself.

Can my agent get past a CAPTCHA?

No. CAPTCHAs are there to stop automated tools. Do that step yourself, or use an app connection instead.

Can my agent sign in to a website for me?

No. Use an app from Integrations where one exists, or copy the information over yourself.