Aboundexbot Web Crawler FAQ


What is Aboundexbot and what is it doing on my site?
Aboundexbot is an indexing crawler for our search engine that specializes on using self-descriptive language to categorize and index websites.


Why doesn't Aboundexbot visit all pages on my site?
Aboundexbot starts with a shallow scan of each website to find self-descriptive keyword information about website/company. The crawler will go deeper into a website to find more information by analyzing links and their surrounding text.


Why does Aboundexbot sometimes access pages from my server that do not exist?
Other website(s) may link to non-existing pages on your server. Part of the Aboundexbot scan will follow external links from a website which can lead to URLS that produce a 404(not found) error in your server logs.
JavaScript misinterpretation by the crawler can also cause this.


Why is Aboundexbot asking for a file called robots.txt that isn't on my server?
robots.txt is a standard document that can tell crawlers not to index some or all information from your web server as well as a crawl delay of how often a crawler may pull pages. Aboundexbot obeys all robots.txt files and restrictions.


How do i stop Aboundexbot from indexing my website?
If you have a concern about Aboundexbot, we hope you give us a chance to address it via the email below but if you need to block Aboundexbot, the robots.txt file will allow you to accomplish that goal.

To block Aboundexbot from your entire web site you add this to your robots.txt file:

User-agent: Aboundexbot
Disallow: /


Who can i contact with questions or concerns about Aboundexbot?
Please email admin (at) aboundex.com with any questions or concerns.