Received: with LISTAR (v1.0.0; list gopher);
 Mon, 15 Jan 2001 14:14:21 -0600 (CST)
Return-Path: <md@Linux.IT>
Delivered-To: gopher@complete.org
Received: from attila.bofh.it (attila.bofh.it [213.92.8.2])
	by pi.glockenspiel.complete.org (Postfix) with ESMTP id D8BAC3B8FD
	for <gopher@complete.org>; Mon, 15 Jan 2001 14:14:20 -0600 (CST)
Received: by attila.bofh.it (Postfix, from userid 10)
	id EA73A60443; Mon, 15 Jan 2001 21:10:37 +0100 (CET)
Received: by wonderland.linux.it (Postfix/Md, from userid 1001)
	id F199C17DFC; Mon, 15 Jan 2001 19:47:41 +0100 (CET)
Date: Mon, 15 Jan 2001 19:47:41 +0100
From: Marco d'Itri <md@Linux.IT>
To: gopher@complete.org
Subject: [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)
Message-ID: <20010115194741.B2647@wonderland.linux.it>
References: <87lmscq5op.fsf@complete.org>
 <200101151713.JAA08378@stockholm.ptloma.edu>
Mime-Version: 1.0
Content-type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.3.12i
In-Reply-To: <200101151713.JAA08378@stockholm.ptloma.edu>;
 from spectre@stockholm.ptloma.edu on Mon, Jan 15, 2001 at 09:13:34AM -0800
Content-Transfer-Encoding: 8bit
X-archive-position: 107
X-listar-version: Listar v1.0.0
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
X-original-sender: md@Linux.IT
Precedence: bulk
Reply-to: gopher@complete.org
X-list: gopher


On Jan 15, Cameron Kaiser <spectre@stockholm.ptloma.edu> wrote:

I agree server administrators need something like a /robots.txt file
which can be downloaded once and cached by the robot. The only other
pratical solution is to modify the server and make it *automatically*
add robots information in the relevant menu.
I can't put other files in the mirror directories, and excluding only
the top level entry in the hope nobody will reference anything below
it is not an acceptable solution.

 >I'm still waiting for Marco to tell me those regexes so I can hard-code them.
You should look at the root menu of gopher.bofh.it. There are two
entries for the debian archive and more entries for the NNTP gateway.

All gateway entries execute the gateway program, so I think it would be
a good idea to exclude *all* program-generated files on all hosts.

E.g.:

Path=exec:-T it.fan.marco-ditri:/gonnrp/gonnrp.pl

-- 
ciao,
Marco