Received: with ECARTIS (v1.0.0; list gopher);
Wed, 12 Oct 2005 14:24:05 -0500 (CDT)
Received: from mx.freeshell.org
([192.94.73.21] helo=sdf.lonestar.org ident=root)
by glockenspiel.complete.org with esmtp
(Exim 4.50)
id 1EPmCk-0005iZ-Ob
for gopher@complete.org; Wed, 12 Oct 2005 14:24:04 -0500
Received: from sdf.lonestar.org (IDENT:newmanbe@sdf.lonestar.org
[192.94.73.1])
by sdf.lonestar.org (8.13.1/8.12.10) with ESMTP id j9CJNgFc005000
for <gopher@complete.org>; Wed, 12 Oct 2005 19:23:42 GMT
Received: (from newmanbe@localhost)
by sdf.lonestar.org (8.13.1/8.12.8/Submit) id j9CJNgu7016350
for gopher@complete.org; Wed, 12 Oct 2005 14:23:42 -0500 (CDT)
Date: Wed, 12 Oct 2005 14:23:42 -0500
From: Benn Newman <newmanbe@sdf.lonestar.org>
To: gopher@complete.org
Subject: [gopher] Re: New Gopher Wayback Machine Bot
Message-ID: <20051012192342.GA9832@SDF.LONESTAR.ORG>
References: <20051012180132.GA19083@complete.org>
<20051012185141.GA21016@complete.org>
Mime-Version: 1.0
Content-type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20051012185141.GA21016@complete.org>
User-Agent: Mutt/1.4.2.1i
X-Spam-Status: No (score 0.9): AWL=0.861, FORGED_RCVD_HELO=0.05
X-Virus-Scanned: by Exiscan on glockenspiel.complete.org at Wed,
12 Oct 2005 14:24:04 -0500
Content-Transfer-Encoding: 8bit
X-archive-position: 1109
X-ecartis-version: Ecartis v1.0.0
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
X-original-sender: newmanbe@sdf.lonestar.org
Precedence: bulk
Reply-to: gopher@complete.org
List-help: <mailto:ecartis@complete.org?Subject=help>
List-unsubscribe: <mailto:gopher-request@complete.org?Subject=unsubscribe>
List-software: Ecartis version 1.0.0
List-Id: Gopher <gopher.complete.org>
X-List-ID: Gopher <gopher.complete.org>
List-subscribe: <mailto:gopher-request@complete.org?Subject=subscribe>
List-owner: <mailto:jgoerzen@complete.org>
List-post: <mailto:gopher@complete.org>
List-archive: <http://www.complete.org/mailinglists/archives/>
X-list: gopher
I know that Veronica-2 obeys robots.txt. Does the bot have its own client name (or whatever it is that
robots.txt calls it? I have content on my server that is easily replaceable (a mirror) and there
is no reason to do a full text of it. Last time I updated [my not publically avaliable] JUGHEAD, the
'gophermap' file was already eight megabytes and I have added content since then.
On Wed, Oct 12, 2005 at 01:51:41PM -0500, John Goerzen wrote:
> On Wed, Oct 12, 2005 at 01:01:32PM -0500, John Goerzen wrote:
> > This bot is now running on my laptop and is spidering away. The IP that
> > you see connections from will vary depending on where my laptop is at a
> > given time ;-)
>
> Speaking of this, do we have any equivolent of robots.txt? Cameron, do
> you have some sort of exclude list or anything?
>
> --
> John Goerzen
> Author, Foundations of Python Network Programming
> http://www.amazon.com/exec/obidos/tg/detail/-/1590593715
>
>
--
Benn Newman
newmanbe@sdf.lonestar.org | gopher://igneous-rock.homeunix.net
Bradley's Bromide:
If computers get too powerful, we can organize them into a
committee -- that will do them in.