The HifiSharkBot is the web crawling robot of the Hifi Shark search engine. The purpose of HifiSharkBot is to discover new and updated for sale listings concerning second hand hifi products to be added to the Hifi Shark index. Due to the relative rarity of sites concerned with second hand hifi equipment the crawling process is based on manual selection of sites rather than site discovery by large scale web-crawling.
The crawling process is designed to be gentle. The goal of the HifiSharkBot is merely to discover the existence of listings and retrieve a URL than can be used for faithful linking by the Hifi Shark search engine. Therefore the crawling process neither pursues links to actual listings nor index information about such listings that is not available directly from pages that list available ads.
When possible the following information is indexed:
- The title of the listing.
- The URL of the listing.
- The listing price - if available.
- The listing creation date - if available.
- A thumbnail URL - if available.
HifiSharkBot and your site
The HifiSharkBot runs two kinds of processes periodically - minor updates, concerned with new listings only, and major updates, concerned with new listings as well as expired ones. For most sites, a minor update will only access very few pages, i.e. between 1 and 20, whereas a major update will access many. The minor updates run at an interval of minutes whereas major updates are performed once a day.
The HifiSharkBot is run from a single machine with a stable IP and reports an easily recognizable User-Agent to your site. If you want to prevent HifiSharkBot from crawling content on your site, you must use robots.txt to block access to files and directories on your server.
If you have questions or would like to make a request, e.g. a change in the crawl rate and/or frequency, please contact us by email or by filling out our webform.