Flowsearch open crawling - the rules

* You run the official Flowsearch crawler (the fscontrib program with the sgcrawl crawler). You get a personal token from the operator.
* The crawler asks for work and gets a slice of sites. It may only crawl those sites; the server rejects pages from any other site.
* The crawler obeys robots.txt, waits between requests and identifies itself in its User-Agent. A page the server finds is forbidden by robots.txt counts against you.
* Your computer's IP address is what the crawled sites see. Do not run the crawler on a network where that is a problem.
* Only plain text is accepted: no HTML, no scripts, no personal data of private people. The server may drop pages that break these rules.
* Your first batches are held back until the server has fetched a sample of them again and the text matches. After that your pages are stored at once, and a
  sample is still checked. If samples do not match, your pages are dropped and the account can be suspended.
* There is a daily limit of pages for every account. The operator can revoke a token at any time.
* Pages are used for the Flowsearch search index. To have a site removed, contact the operator.