Crawl settings
Most crawls run fine on the defaults. When you need more control, the crawl bar and Preferences let you shape exactly what gets fetched and how each page is scored.
Crawl controls
Set these on the crawl bar or in Preferences before you start. Start and Pages live on the toolbar; the rest are under Preferences.
| Setting | What it does | Default |
|---|---|---|
| Start URL | The URL where the crawl begins. | — |
| Pages | Maximum number of URLs to crawl. Enter a number, or leave blank / 0 / "All" to crawl every URL discovered. | 200 |
| Threads | Concurrent workers fetching pages (1–20). Higher is faster but heavier on the target site. | 5 |
| Timeout | Seconds to wait for each page before giving up. | 15 |
| User-Agent | The user-agent string GEOBee sends with each request. | GEOBee/1.0 (+AEO-GEO audit) |
| Lighthouse CLI | Path to the Lighthouse binary, used only by deep scans. | lighthouse (on PATH) |
Path filters
In Preferences you can scope a crawl with Include and Exclude path patterns — one glob per line. When a URL matches both, exclude wins. Use this to skip faceted URLs, staging areas, or endless parameter pages, or to focus a crawl on a single section of a large site.
Robots and sitemaps
GEOBee always respects robots.txt: disallowed URLs are not fetched and are
recorded with a "Disallowed by robots.txt" note rather than a fake score. It also discovers
and parses your sitemap, so you can filter results down to the URLs that are actually listed
in it.
Scoring rules
The Scoring Rules tab in Preferences exposes the thresholds behind several SEO checks. The defaults reproduce standard SEO scoring; change them to match your own house style. Adjustments apply to your next crawl, not the one already on screen.
| Threshold | Controls | Default |
|---|---|---|
| Title length | Min and max characters for a healthy <title>. | 30 / 60 |
| Meta description length | Min and max characters for the meta description. | 50 / 160 |
| H1 max length | Longest H1 before it is flagged. | 70 |
| Thin-content floor | Word count below which a page counts as thin. | 100 |
| Link-count cap | On-page links above which link count is flagged. | 100 |
Use Reset to defaults in that tab to return every threshold to its standard value.