Received: with LISTAR (v1.0.0; list gopher); Sat, 13 Jan 2001 18:29:59 -0600 (CST) Return-Path: Delivered-To: gopher@complete.org Received: from stockholm.ptloma.edu (stockholm.ptloma.edu [199.106.86.50]) by pi.glockenspiel.complete.org (Postfix) with ESMTP id B5BC83B807 for ; Sat, 13 Jan 2001 18:29:57 -0600 (CST) Received: (from spectre@localhost) by stockholm.ptloma.edu (8.9.1/8.9.1) id QAA13514 for gopher@complete.org; Sat, 13 Jan 2001 16:28:12 -0800 From: Cameron Kaiser Message-Id: <200101140028.QAA13514@stockholm.ptloma.edu> Subject: [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database) In-Reply-To: <20010114002128.D4061@wonderland.linux.it> from Marco d'Itri at "Jan 14, 1 00:21:28 am" To: gopher@complete.org Date: Sat, 13 Jan 2001 16:28:12 -0800 (PST) X-Mailer: ELM [version 2.4ME+ PL39 (25)] MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 8bit X-archive-position: 89 X-listar-version: Listar v1.0.0 Sender: gopher-bounce@complete.org Errors-to: gopher-bounce@complete.org X-original-sender: spectre@stockholm.ptloma.edu Precedence: bulk Reply-to: gopher@complete.org X-list: gopher >>Good point. I am actually trying to think of a way like the HTTP robots.txt >>that can more or less transparently tell V-2 what to stay out of. Suggestions? > I don't know well gopher yet, but you'd better find one before starting > to index my site, or your database will be filled with crap. I'm sure Dave and John will have some ideas, but for now just mail me offlist with some regexes that are off-limits and I'll hardcode them for the present. I appreciate it :-) -- ----------------------------- personal page: http://www.armory.com/~spectre/ -- Cameron Kaiser, Point Loma Nazarene University * ckaiser@stockholm.ptloma.edu -- People who buy computers from TV commercials *deserve* the Pentium. --------