org.egothor.robot

Class RobotsTXT

java.lang.Object

org.egothor.robot.module.Page

org.egothor.robot.RobotsTXT

public class RobotsTXT
extends Page

This class parses /robots.txt files and provides the respective test function. The format was summarized and suggested by M. Koster, and the original file may be accessed here . The format and semantics of the "/robots.txt" file are as follows: The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains lines of the form "<field>: <optionalspace><value><optionalspace>". The field name is case insensitive. Comments can be included in file using UNIX bourne shell conventions: the '#' character is used to indicate that preceding space (if any) and the remainder of the line up to the line termination is discarded. Lines containing only a comment are discarded completely, and therefore do not indicate a record boundary. The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below. Unrecognised headers are ignored.

• User-agent: The value of this field is the name of the robot the record is describing access policy for. If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record. The robot should be liberal in interpreting this field. A case insensitive substring match of the name without version information is recommended. If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records. It is not allowed to have multiple such records in the "/robots.txt" file.
• Disallow: The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html.

Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record. The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.

Author:

Leo Galambos

Field Summary

Fields

Modifier and Type	Field and Description
long	nextConn

Fields inherited from class org.egothor.robot.module.Page

id, length, response

Constructor Summary

Constructors

Constructor and Description
RobotsTXT(DataInput in)

Method Summary

Modifier and Type	Method and Description
static RobotsTXT	EMPTY()
boolean	equalTo(RobotsTXT r)
static RobotsTXT	FULL()
int	hashCode()
void	initialize(long time, String name, int rank) Sets this object as a fresh and new with lastModif=0 and nextConn=time.
void	load(DataInput in)
static RobotsTXT	parse(BufferedReader r) Analyses a robots.txt file.
void	print()
void	store(DataOutput out) Stores this structure to a stream.
String	toString()
boolean	valid(String uri) Tests whether the given URL fulfils the rules.

Methods inherited from class org.egothor.robot.module.Page

dropped, isAlreadyProcessed, toStringBuilder

Methods inherited from class java.lang.Object

equals, getClass, notify, notifyAll, wait, wait, wait

Field Detail

nextConn

public long nextConn

Constructor Detail

RobotsTXT

public RobotsTXT(DataInput in)
          throws IOException

Throws:

IOException

Method Detail

EMPTY

public static RobotsTXT EMPTY()

FULL

public static RobotsTXT FULL()

print

public void print()

equalTo

public boolean equalTo(RobotsTXT r)

hashCode

public int hashCode()

Overrides:

hashCode in class Object

parse

public static RobotsTXT parse(BufferedReader r)

Analyses a robots.txt file.

Parameters:

r - The file

Returns:

The respective RobotsTXT structure

store

public void store(DataOutput out)
           throws IOException

Stores this structure to a stream.

Overrides:

store in class Page

Parameters:

out - The stream

Throws:

IOException - When I/O fails

load

public void load(DataInput in)
          throws IOException

Overrides:

load in class Page

Throws:

IOException

valid

public boolean valid(String uri)

Tests whether the given URL fulfils the rules.

Parameters:

uri - The URL

Returns:

true iff the URL is valid

initialize

public void initialize(long time,
                       String name,
                       int rank)

Sets this object as a fresh and new with lastModif=0 and nextConn=time.

toString

public String toString()

Overrides:

toString in class Object