WHAT IS INDEXBOT?

Indexbot is the primary web crawler of the Open Web Index. It's a crawler designed to crawl a big portion of the internet to create an independent web index, which could be useful for research projects or independent search engines.

HOW CAN I BLOCK/UNBLOCK YOUR BOT?

Our Indexbot respects robots.txt files. If you wish to block our bot, simply disallow our specific bot in the robots.txt file of your website:

If you want to unblock/allow our bot, simply allow our specific bot in the robots.txt file of your website:

IS THIS A COMMERCIAL PROJECT?

No, the Open Web Index is designed to be a publicly available, free to use, and open source index of the internet. We may ask for support or funding to cover the costs of indexing the web and saving our large datasets.

CAN I GET MY DATA REMOVED?

If our bot captured personal information or illegal/copyrighted/licensed material please contact us immediately. Please list all effected files. Our bot only captures content, which is publicly available on the internet.