28 Jun
2010
28 Jun
'10
10:38 a.m.
On 2010-06-15 20:33, Hans-Christoph Steiner wrote:
I'd like to change the robots.txt for puredata.info so that it allows everything, and search engines can index it. Are we going by lazy consensus on this?
i would prefer if robots would not constantly pull 4GB of data. this is mainly relevant for media-data (patches, movies, presentations, ...) which is probably not so relevant for searchbots either.
so i would consent on a robots.txt that does not allow everything for everyone but rather restrict bots to text-pages.
fmasdr IOhannes