The organisation's site is usually already there
The website address given at signup is crawled automatically the moment the account is approved — provided the crawl-consent box was ticked on the waitlist form. It becomes the organisation's first knowledge base, and it also gives the Builder a brief on the business, so new agents sound right from the first draft. Without consent, nothing is crawled: the base shows Consent withheld until the site is added manually.
Add another site
Knowledge → Add knowledge, then the Website tab (the panel opens on
whichever half the organisation is missing). A bare domain is fine —
https:// is added automatically.
Tick I'm authorised to have this site crawled. The button stays disabled until it is ticked.
The crawl starts straight away, with live progress on the list — see Knowledge bases.
One base per site: adding a host that already exists is refused. How many bases each plan includes is in Plans & limits.
What the crawler reads — and skips
The crawler reads the text of the site's pages — cleaned, summarised and
indexed — and picks out contact details like emails and phone numbers as it
goes. On public sites it honours robots.txt. Crawls are capped at a
sensible size, and each base holds up to 100 MB of extracted text.
Password-protected sites
A knowledge base can sit behind a login — a staff area or an intranet. This site needs a login, ticked when adding the site, reveals the login type — Username & password form or HTTP Basic auth — and credential fields; form logins can also take a Login page address. If the login fails, the crawl fails with a check-your-credentials error on the row.
The password is stored encrypted, used only by the crawler, and never shown
again — the base simply carries a Private badge. Because it is the
organisation's own consented content, authenticated crawls don't apply
robots.txt.
Keeping it fresh
Website bases are re-crawled automatically on a schedule. After a site update that should reach agents now, Re-crawl on the row queues a fresh crawl immediately.
Withdrawing consent
Crawl consent is revocable at any time. Deleting the base — Delete, then Confirm delete — withdraws it: everything stored from the crawl is purged permanently, and the site is not crawled again unless it is re-added.