ErikSisd:
Potentially daft question: what's the purpose of all these bots on the site? I.e. bingbot, lsfbot, Ezooms, Feedfetcher, etc.
I can see how a bot might be there to record or map a site, but how do they get access to a members-only site? Why do we seem multiple identical bot at the same time (i.e. 3 bingbots as I write)? And why are they allowed?
Most of them, as flopsy has pointed out, are gathering data for search engines. Some of them perform a useful function and others are merely annoying. They visit most sites, we just happen to record when they're here so I can see what they're up to.
They don't have access to the LSF as such, they merely see whatever a guest who isn't logged into the site would see which of course includes most of the forum.
Lsfbot is our own bot and is an entirely different kind of thing. It merely makes sure the chat room remains in existence, that is when it hasn't decide to wander off in search of more interesting conversation.
We see multiple instances of some bots (especially the annoying Microsoft Bing bot) because they are basically exploring multiple links on the site at the same time, probably the various forum threads, each of which is an HTML URL in its own right.
As to why we allow them - well we want to appear in the Google search index and I suppose the various other search engine indexes too. It is possible to prevent them from indexing the site but only if they are 'well behaved' and follow the agreed protocol where site owners can determine which bots they give access to.
