Previous  Up  Next

Module Restful.Robots

module Robots: sig .. end

Functions for generating responses to requests for /robots.txt.

A /robots.txt response is a sequence of records; each record consists of a User-agent field and one or more Disallow fields. You can construct a corresponding algebraic data type and Restful.Robots.some will generate a response from it.

type user_agent =
| User_agent of string

The type of User-agent fields.

type disallow =
| Disallow of string

The type of Disallow fields.

type record = user_agent * disallow *
       disallow list

The type of robots.txt records.

The following functions each take and return a function compatible with Restful.SERVICE.main, modifying it so that it automatically processes /robots.txt requests.

val all : ('a -> 'b -> Netcgi.cgi -> Nethttp.http_status) ->
       'a -> 'b -> Netcgi.cgi -> Nethttp.http_status

all main: convert main into a function that also answers /robots.txt requests with a response that allows all robots to access all URI's.

val none : ('a -> 'b -> Netcgi.cgi -> Nethttp.http_status) ->
       'a -> 'b -> Netcgi.cgi -> Nethttp.http_status

none main: convert main into a function that also answers /robots.txt requests with a response that refuses all robots access to any URI's.

val some : ?text:string ->
       ?file:string ->
       ?robots:(user_agent * disallow *
                disallow list)
               list ->
       ('a -> 'b -> Netcgi.cgi -> Nethttp.http_status) ->
       'a -> 'b -> Netcgi.cgi -> Nethttp.http_status

some ?text ?file ?robots main: convert main into a function that also answers /robots.txt requests with a customized response. Exactly one of ~text, ~file or ~robots must be provided.

The difference between expressing your robots.txt with ~text and ~robots is that with ~robots Restful will statically guarantee you a syntactically-correct robots.txt; in addition, with ~robots you can obviously generate a robots.txt algorithmically.

text : the complete text of your robots.txt response
file : the name of a file containing your robots.txt response
robots : list of records expressing your robots.txt response