More doc.

[idzebra-moved-to-github.git] / doc / zebra.sgml

<!doctype linuxdoc system>

<!--
  $Id: zebra.sgml,v 1.2 1995-11-28 14:25:23 adam Exp $
-->

<article>
<title>Zebra server Administrators's Guide and Reference
<author>Index Data, <tt/info@index.ping.dk/
<date>$Revision: 1.2 $
<abstract>
This document describes Zebra &mdash; an information retrieval server
that uses the Z39.50 protocol.
</abstract>

<toc>

<sect>Introduction

<p>
- gils
- UNI*X (not NT yet)
- goals

<sect>Compiling the software

<p>

Zebra uses the YAZ package in order to provide Z39.50 access. So you
must compile YAZ before going further. Specifically, Zebra uses
the YAZ header files in <tt>yaz/include/..</tt> and its public library
<tt>yaz/lib/libyaz.a</tt>.

As with YAZ, an ANSI C compiler is required in order to compile the Zebra
server system &mdash; GNU C works fine.

Unpack the Zebra software. You might put Zebra in the same directory level
as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-</tt>.., then
Zebra is placed in ..<tt>/src/zebra-</tt>.

Edit the top-level <tt>Makefile</tt> in the Zebra directory in which
you specify the location of YAZ by setting make variables.
The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with
MOSI support. Some systems, such as Solaris, have separate socket
libraries and for those systems you need to specify the
<tt>NETLIB</tt> variable.

When finished editing the <tt>Makefile</tt> type:
<tscreen><verb>
$ make
</verb></tscreen>

If successful, two executables have been created in the sub-directory
index.
<descrip>
<tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
<tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
</descrip>

<sect>Quick getting started

<p>
This section will get you started quickly! We will try to index a few
GILS records that are included with the Zebra distribution. Go to the
<tt>test</tt> subdirectory. There you will find a configuration
file named <tt>zebra.cfg</tt> with the following contents:
<tscreen><verb>
# Where are the YAZ tables located.
profilePath: /usr/local/yaz

# Files that describe the attribute sets supported.
attset: bib1.att
attset: gils.att
</verb></tscreen>

Now, edit the file and set <tt>profilePath</tt> to the path of the
YAZ profile tables (sub directory <tt>tab</tt> of YAZ).

The 48 test records are located in the sub directory <tt>records</tt>.
To index these, type:
<tscreen><verb>
$ ../index/zebraidx -t grs update records
</verb></tscreen>

In the command above the option <tt>-t</tt> specified the record
type &mdash; in this case <tt>grs</tt>. The word <tt>update</tt> followed
by a directory root updates all files below that directory.

If your indexing command went successful, you are now ready to
setup a server. To start a server on port 2100, type:
<tscreen><verb>
$ ../index/zebrasrv tcp:@:2100
</verb></tscreen>

The Zebra server just made has one database called Default. It will
return either USMARC/GRS or SUTRS depending on your client.

<sect>Administrating Zebra

<p>

Unlike many other retrieval systems, Zebra offers incremental
modifications of an existing index. Needless to say, these facilities
makes the administration of Zebra somewhat more complicated.

Normally, when Zebra modifies the index it reads a number of records
that you specify.
Depending on your specifications and on the contents of each record
one the following scenarios occur:
<descrip>
<tag>Insert</tag> The record is indexed as if it never occurred
before. Either the Zebra system doesn't know how to identify the record or
Zebra can identify the record but didn't find it to be already indexed.
<tag>Modify</tag> The record has already been indexed. In this case
either the contents of the record or the location (file) of the record
indicates that it has been indexed before.
<tag>Delete</tag> The record is deleted from the index. As in the
update-case it must be able to identify the record.
</descrip>

Please note, that in both the modify- and delete- case the Zebra
system must be able to make a key that identifies the record in
question.

To administrate the Zebra retrieval system, you run the
<tt>zebraidx</tt> program. This program supports a number of options
which are preceded by a minus, and a few commands (not preceded by
minus).

Both the Zebra administrative tool and the Z39.50 server share a
set of indexing files and a global configuration file. The
name of the configuration file defaults to <tt>zebra.cfg</tt>.
The configuration file includes specifications on how to index
various kinds of records and where the other configuration files
are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
be run in the same directory where the configuration file if you do
not indicate the location of the configuration file by option
<tt>-c</tt>.

<sect1>Record types
<p>
Indexing is a record-per-record process, in which
either insert/modify/delete will occur. Before a record is indexed
search keys are extracted from whatever might be the layout the
original record (sgml,html,text, etc..). The Zebra system 
currently only support GILS records and simple text records.
To specify a particular extraction process, use either the
command line option <tt>-t</tt> or specify a
<tt>recordType</tt> setting in the configuration file.

<sect1>The Zebra configuration file
<p>
The Zebra configuration file, read by <tt>zebraidx</tt> and
<tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
by <tt>-c</tt> option.

You can edit the configuration file with a normal text editor.
Setting names and values are seperated by colons in the file. Lines
starting with a hash sign (<tt/#/) are treated as comments.

A set of records that share common characteristics are called a group.
When <tt>zebraidx</tt> is run and you wish to address a given group
you specify that group with the <tt>-g</tt> option. In this case
settings that have the group name as their prefix will be used
by <tt>zebraidx</tt> and not default values.

The group is written before the option itself separated by a dot.
For example to set the record type for group <tt/public/ to <tt/grs/
you would write:
<tscreen><verb>
public.recordType: grs
</verb></tscreen>

To set the default value of the record type to text write:
<tscreen><verb>
recordType: text
</verb></tscreen>

The configuration settings are summarized below. They will be
explained further in the following chapters.

<descrip>
<tag><it>group</it>recordType<it>name</it></tag>
 Specifies how records with the file extension <it>name</it> should
 be handled by the indexer. This option may also be specified
 as a command line option (<tt>-t</tt>).
<tag><it>group</it>recordId</tag>
 Specifies how the record is to be identified when updated.
<tag><it>group</it>database</tag>
 Specifies the Z39.50 database.
<tag><it>group</it>storeKeys</tag>
 Specifies whether key information should be saved for a given
 group of records. If you plan to update/delete this type of
 record a later this should be specified as 1; otherwise it
 should be 0 (default).
<tag><it>group</it>storeData</tag>
 Specifies whether original copy of record should be stored internally
 in the Zebra system indexes. If your externally indexed files
 are temporary this option should certainly be true (1); otherwise
 false (0).
<tag>register</tag> 
 Specifies the location of the indexes.
<tag>profilePath</tag>
 Specifies the location of profile specification paths.
<tag>attset</tag> 
 Specifies the filename(s) of attribute specification files.
</descrip>

<sect1>Locating records
<p>
The default behaviour of the Zebra system is to reference the
records from their original location, i.e. where they were
read an indexed.

If your records files are temporary for example if you retrieve
them from the outside or if they where temporarily mounted on a CD-ROM,
you may want the Zebra server to make a copy of them. To do this
you specify 1 (true) in the <tt>storedata</tt> setting. When
the Z39.50 search engine retrieves recoreds they will be read from an
internal structure of the index.

<sect1>Indexing with no record Ids

<p>
If you have a set of records that you <em/never/ wish to delete
or modify you may find 'indexing with no records Ids' convenient.
This indexing method uses less space than the other methods and
is simple to use. 

To use this method, you simply don't specify the <tt>recordId</tt> 
for the group of files you index. To add a set of records you use
<tt>zebraidx</tt> with the <tt>update</tt> command. The
<tt>update</tt> command will always add the records to the index
becuase Zebra doesn't know how to match the new set of records with
existing records.

Consider a system, in which you have a group of text files called
<tt>simple</tt>. That group of records should belong to a Z39.50 database
called <tt>textbase</tt>. The following configuration file suffice:

<tscreen><verb>
profilePath: /usr/local/yaz
attset: bib1.att
attset: gils.att
simple.recordType: text
simple.database: textbase
</verb></tscreen>

<sect1>Indexing with file record Ids

<p>
If you have a set of external records that you wish to index you may
use the file key feature of the Zebra system. In short, the file key
feature, mirrors a directory and the files below efficiently. To
perform indexing of a directory with file keys, you specify the top-level
directory after the <tt>update</tt> command. The command will recursively
traverse the directories and compare each with whatever have been
indexed before in the same directory. If a file is new (not in
the previous version of the directory) it is inserted;
if a file was already indexed and it has been modified
since the last insertion the index is also modified; if a file is missing
since the last visit it is deleted.

The resulting system is easy to administer. To delete a record
you simply have to delete the corresponding file (with <tt/rm/). 
To force update of a given file, you may use the <tt>touch</tt>
command. And to add files create new files (or directories with files).
For your changes to take effect you must run <tt>zebraidx</tt> with
the same directory root again.

To use this method, you must specify <tt>file</tt> as the value
of <tt>recordId</tt> the configuration file. In the configuration
also set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra
indexer must save additional information in order to
modify/delete the record at a later time.

For example, to update group <tt>esdd</tt> records below
<tt>/home/grs</tt> you could type:
<tscreen><verb>
$ zebraidx -g esdd update /home/grs
</verb></tscreen>

The corresponding configuration file includes:
<tscreen><verb>
esdd.recordId: file
esdd.recordType: grs
esdd.storeKeys: 1
</verb></tscreen>

<em>Important note: You cannot start by a group of records with simple
indexing (no record ids as in the previous section) and then, later,
use file record Ids. The Zebra must know from the very beginning that
the group of files are indexed with file record Ids.
</em>

<sect1>Indexing with general record Ids
<p>
When using this method you specify an (almost) arbritrary record key
based on the contents of the record itself and other system
information. If you have a group of records that have an external
id attached, this method is convenient. For example, the record may
contain a title or an unique id. In either case you specify the 
Z39.50 attribute-set and attribute-use location in which this
information is stored.

As before, the record id is defined by the <tt>recordId</tt> setting
in the configuration file. The value of the record id consists of one
or more tokens separated by space. The resulting id is formed
by concatenating the tokens and separete them by ascii value (1).

There are three kinds of tokens:
<descrip>
<tag>Internal record info</tag> The token refers to a key that is
extracted from the record. The syntax of this token is
 <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the
attribute set number and <em/use/ is the use value of the attribute.
<tag>System variable</tag> The system variables are preceded by
<tt>$</tt> and immediately followed by the system variable name, which
may one of
 <descrip>
 <tag>group</tag> Group name.
 <tag>database</tag> Current database specified.
 <tag>filename</tag> Name of file that contain the record.
 <tag>type</tag> Record type.
 </descrip>
<tag>Constant string</tag> A string used as part of id &mdash; surrounded
 by single- or double quotes.
</descrip>

The test GILS records that comes with Zebra, contain a unique id
in the Control-Identifier field. This field is mapped to the Bib-1
use attribute 1007. To use this field as a record id, specify
<tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the
configuration file. If you have other record types that doesn't
contain an id in the same field, you might add the record type
in the record id of the gils records as well, to prevent match
of other types of records. In this case the recordId might be
set, as in:
<tscreen><verb>
gils.recordId: $type (1,1007)
</verb></tscreen>

As for the file record id case described in the previous section
updating your system is simply a matter of running <tt>zebraidx</tt>
with the <tt>update</tt> command. However, the update of with general
keys is considerably slower than with file record ids, since all files
visited must be (re)indexded. 

<sect1>Register location

<p>
Normally, the index files that form dictionaries, inverted
file, record info, etc. are stored in the current directory in which you run
<tt>zebraidx</tt>. If you wish to store those, possibly large, files
somewhere else, you must set the <tt>register</tt> setting in the
configuration file. Furhtermore, the Zebra system handles indexes that
span multiple file systems, which is usefull if a large number of
records are indexed.

The value <tt>register</tt> of register is a sequence of tokens.
Each token takes the form:
<tscreen>
<em>dir</em><tt>:</tt><em>size</em>. 
</tscreen>
The <em>dir</em> specifies a directory in which index files will be
stored and the <em>size</em> specifies the maximum size of all
files in that directory. The Zebra indexer system fill each directory
in the order specified and use the next specified directories as needed.
The <em>size</em> is an integer possibly followed by a quantity
code, <tt>M</tt> for Mega bytes, <tt>K</tt> for Kilo bytes.

If you have two spare disks :) and the first disk is mounted
on <tt>/d1</tt> and has 200 Mega bytes free space after formatting and the
second, mounted on <tt>/d2</tt> has 300 Mega bytes free, you could
specify the following in you configuration file:
<tscreen><verb>
register: /d1:200M /d2:300M
</verb></tscreen>

<sect>The Z39.50 Server

<p>

<sect1>Running the server
<p>
Starting the server and how to configure the server.

<sect1>How the server handles queries
<p>
What elements of Bib-1 are supported and where are result sets
stored.

<sect>About Index Data
<p>
Copyright &copy; 1995, Index Data.

Permission to use, copy, modify, distribute, and sell this software and
its documentation, in whole or in part, for any purpose, is hereby granted,
provided that:

1. This copyright and permission notice appear in all copies of the
software and its documentation. Notices of copyright or attribution
which appear at the beginning of any file must remain unchanged.

2. The names of Index Data or the individual authors may not be used to
endorse or promote products derived from this software without specific
prior written permission.

THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR
NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF
LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
OF THIS SOFTWARE.

<sect>About Index Data

<p>
Index Data is a consulting and software-development enterprise that
specialises in library and information management systems. Our
interests and expertise span a broad range of related fields, and one
of our primary, long-term objectives is the development of a powerful
information management
system with open network interfaces and hypermedia capabilities.

We make this software available free of charge, on a fairly unrestrictive
license; as a service to the networking community, and to further the
development of quality software for open network communication.

We'll be happy to answer questions about the software, and about ourselves
in general.

<tscreen>
Index Data&nl
Ryesgade 3&nl
2200 K&oslash;benhavn N&nl
DENMARK
</tscreen>

<p>
<tscreen><verb>
Phone: +45 3536 3672
Fax  : +45 3536 0449
Email: info@index.ping.dk
</verb></tscreen>

<sect>References

<p>

</article>