More work.

[idzebra-moved-to-github.git] / doc / zebra.sgml

<!doctype linuxdoc system>

<!--
  $Id: zebra.sgml,v 1.7 1995-11-28 16:47:24 adam Exp $
-->

<article>
<title>Zebra Server - Administrators's Guide and Reference
<author>Index Data, <tt/info@index.ping.dk/
<date>$Revision: 1.7 $
<abstract>
The Zebra information server combines a versatile fielded/free-text
search engine with a Z39.50-1995 frontend to provide a powerful and flexible
information management system. This document explains the procedure for
installing and configuring the system, and outlines the possibilities
for managing data and providing Z39.50
services using the software.
</abstract>

<toc>

<sect>Introduction

<sect1>Overview

<p>
The Zebra system is a fielded free-text indexing and retrieval engine with a
Z39.50 frontend. You can use any commercial or freeware Z39.50 client
to access data stored in Zebra.

The Zebra server is our first step towards the development of a fully
configurable, open information system. Eventually, it will be paired
off with a powerful Z39.50 client to support complex information
management tasks within almost any application domain. We're making
the server available now because it's no fun to be in the open
information retrieval business all by yourself. We want to allow
people with interesting data to make their things
available in interesting ways, without having to start out
by implementing yet another protocol stack from scratch.

This document is an introduction to the Zebra system. It will tell you
how to compile the software, and how to prepare your first database.
It also explains how the server can be configured to give you the
functionality that you need.

You should read <it/Specifying and Using Application (Database)
Profiles/, which is bundled with the YAZ documentation, to learn how
records are formatted, and how you can configure Zebra to handle
different types of Z39.50 application profiles.

<sect1>Features

<p>
This is a listing of some of the most important features of the
system.

<itemize>

<item>
Supports updating - records can be added and deleted without
rebuilding the index.

<item>
Supports large databases - files for indices, etc. can be
automatically partitioned over multiple disks.

<item>
Supports arbitrarily complex records - base input format is an
SGML-like syntax which allows nested (structured) data elements, as
well as variant forms of data.

<item>
Supports boolean queries as well as relevance-ranking (free-text)
searching. Right truncation and masking in terms are supported, as
well as full regular expressions.

<item>
Supports multiple concrete syntaxes
for record exchange (depending on the configuration): GRS-1, SUTRS,
ISO2709 (*MARC). Records can be mapped between record syntaxes and
schema on the fly.

<item>
Protocol support:

<itemize>

<item>
Protocol facilities: Init, Search, Retrieve, Browse.

<item>
Piggy-backed presents are honored in the search-request.

<item>
Named result sets are supported.

<item>
Easily configured to support different application profiles, with
tables for attribute sets, tag sets, and abstract syntaxes.
Additional tables control facilities such as element mappings to
different schema (eg., GILS-to-USMARC).

<item>
Complex composition specifications using Espec-1 are partially
supported (simple element requests only).

<item>
Element Set Names are established the Espec-1 capability of the
system, and are given in configuration files as simple element
requests (and possibly variant requests).

<item>
Some variant support (not fully implemented yet).

<item>
Using the YAZ toolkit for the protocol implementation, the
server can utilise a plug-in XTI/mOSI implementation (not included) to
provide SR services over an OSI stack, as well as Z39.50 over TCP/IP.

</itemize>

</itemize>

<sect1>Future Work

<p>
This is an early alfa-release of the software, to allow you to look at
it - try it out, and assess whether it can be of use to you. We expect
this version to be followed by a succession of beta-releases until we
arrive at a stable first version.

These are some of the plans that we have for the software in the near
and far future, approximately ordered after their relative importance.
Items marked with an
asterisk will be implemented before the
last beta release.

<itemize>

<item>
*Allow the system to handle additional input formats. Specifically
MARC records and general, structured ASCII records (such as mail/news
files) parameterized by regular expressions.

<item>
*Complete the support for variants. Finalize support for the WAIS
retrieval methodology.

<item>
*Finalize the data element <it/include/ facility to support multimedia
data elements in records.

<item>
*Port the system to Windows NT.

<item>
Add robust database updating - tolerant to crashes or hard interrupts
during register updating.

<item>
Add online updating, to permit register updating while users are
accessing the system.

<item>
Add index and data compression to save disk space.

<item>
Add more sophisticated relevance ranking mechanisms. Add support for soundex
and stemming. Add relevance feedback support.

<item>
Add Explain support.

<item>
Add support for very large records by implementing segmentation and
variant pieces.

<item>
Support the Item Update extended service of the protocol.

<item>
The Zebra search engine supports approximate string matching in the
index. We'd like to find a way to support and control this from RPN.

<item>
We want to add a management system that allows you to
control your databases and configuration tables from a graphical
interface. We'll probably use Tcl/Tk to stay platform-independent.

</itemize>

Programmers thrive on user feedback. If you are interested in a facility that
you don't see mentioned here, or if there's something you think we
could do better, please drop us a mail. If you think it's all really
neat, you're of course welcome to drop us a line saying that, too.
<sect>Introduction

<sect>Compiling the software

<p>
Zebra uses the YAZ package to implement Z39.50, so you
have to compile YAZ before going further. Specifically, Zebra uses
the YAZ header files in <tt>yaz/include/..</tt> and its public library
<tt>yaz/lib/libyaz.a</tt>.

As with YAZ, an ANSI C compiler is required in order to compile the Zebra
server system &mdash; GNU C works fine.

Unpack the Zebra software. You might put Zebra in the same directory level
as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-</tt>.., then
Zebra is placed in ..<tt>/src/zebra-</tt>.

Edit the top-level <tt>Makefile</tt> in the Zebra directory in which
you specify the location of YAZ by setting make variables.
The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with
MOSI support. Some systems, such as Solaris, have separate socket
libraries and for those systems you need to specify the
<tt>NETLIB</tt> variable.

When you are done editing the <tt>Makefile</tt> type:
<tscreen><verb>
$ make
</verb></tscreen>

If successful, two executables have been created in the sub-directory
<tt/index/.
<descrip>
<tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
<tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
</descrip>

<sect>Quick Start

<p>
This section will get you started quickly! We will try to index a few sample
GILS records that are included with the Zebra distribution. Go to the
<tt>test</tt> subdirectory. There you will find a configuration
file named <tt>zebra.cfg</tt> with the following contents:
<tscreen><verb>
# Where are the YAZ tables located.
profilePath: /usr/local/yaz

# Files that describe the attribute sets supported.
attset: bib1.att
attset: gils.att
</verb></tscreen>

Now, edit the file and set <tt>profilePath</tt> to the path of the
YAZ profile tables (sub directory <tt>tab</tt> of YAZ).

The 48 test records are located in the sub directory <tt>records</tt>.
To index these, type:
<tscreen><verb>
$ ../index/zebraidx -t grs update records
</verb></tscreen>

In the command above the option <tt>-t</tt> specified the record
type &mdash; in this case <tt>grs</tt>. The word <tt>update</tt> followed
by a directory root updates all files below that directory node.

If your indexing command went successful, you are now ready to
fire up a server. To start a server on port 2100, type:
<tscreen><verb>
$ ../index/zebrasrv tcp:@:2100
</verb></tscreen>

The Zebra index that you've just made has one database called Default. It will
return either USMARC, GRS-1, or SUTRS depending on what your client asks
for.

To test the server, you can use any Z39.50 client (1992 or later). For
instance, you can use the demo client that comes with YAZ: Just cd to
the <tt/client/ subdirectory of the YAZ distribution and type:

<tscreen><verb>
$ client tcp:localhost:2100
</verb></tscreen>

When the client has connected, you can type:

<tscreen><verb>
Z> find surficial
Z> show 1
</verb></tscreen>

To try other retrieval formats for the same record, try:

<tscreen><verb>
Z>format sutrs
Z>show 1
Z>format grs-1
Z>show 1
</verb></tscreen>

If you've made it this far, there's a reasonably good chance that
you've made it through the compilation OK.

<sect>Administrating Zebra

<p>

Unlike many other retrieval systems, Zebra offers incremental
modifications of an existing index. Needless to say, these facilities
make the administration of Zebra a bit more complicated than
systems that use the &dquot;index-it-all&dquot; approach.

Normally, when Zebra modifies the index it reads a number of records
that you specify.
Depending on your specifications and on the contents of each record
one the following events take place for each record:
<descrip>
<tag>Insert</tag> The record is indexed as if it never occurred
before. Either the Zebra system doesn't know how to identify the record or
Zebra can identify the record but didn't find it to be already indexed.
<tag>Modify</tag> The record has already been indexed. In this case
either the contents of the record or the location (file) of the record
indicates that it has been indexed before.
<tag>Delete</tag> The record is deleted from the index. As in the
update-case it must be able to identify the record.
</descrip>

Please note that in both the modify- and delete- case the Zebra
indexer must be able to make a unique key that identifies the record in
question.

To administrate the Zebra retrieval system, you run the
<tt>zebraidx</tt> program. This program supports a number of options
which are preceded by a minus, and a few commands (not preceded by
minus).

Both the Zebra administrative tool and the Z39.50 server share a
set of index files and a global configuration file. The
name of the configuration file defaults to <tt>zebra.cfg</tt>.
The configuration file includes specifications on how to index
various kinds of records and where the other configuration files
are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
be run in the same directory where the configuration file if you do
not indicate the location of the configuration file by option
<tt>-c</tt>.

<sect1>Record types
<p>
Indexing is a record-per-record process, in which
either insert/modify/delete will occur. Before a record is indexed
search keys are extracted from whatever might be the layout the
original record (sgml,html,text, etc..). The Zebra system 
currently only supports SGML-like, structured records and unstructured text
records.
To specify a particular extraction process, use either the
command line option <tt>-t</tt> or specify a
<tt>recordType</tt> setting in the configuration file.

<sect1>The Zebra Configuration File
<p>
The Zebra configuration file, read by <tt>zebraidx</tt> and
<tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
by <tt>-c</tt> option.

You can edit the configuration file with a normal text editor.
Setting names and values are seperated by colons in the file. Lines
starting with a hash sign (<tt/#/) are treated as comments.

A set of records that share common characteristics are called a group.
When <tt>zebraidx</tt> is run and you wish to address a given group
you specify that group with the <tt>-g</tt> option. In this case
settings that have the group name as their prefix will be used
by <tt>zebraidx</tt> and not default values. The default values have no prefix.

The group is written before the option itself separated by a dot.
For instance, to set the record type for group <tt/public/ to <tt/grs/ (structured records)
you would write:

<tscreen><verb>
public.recordType: grs
</verb></tscreen>

To set the default value of the record type to text write:

<tscreen><verb>
recordType: text
</verb></tscreen>

The configuration settings are summarized below. They will be
explained further in the following sections.

<descrip>
<tag><it>group</it>recordType<it>name</it></tag>
 Specifies how records with the file extension <it>name</it> should
 be handled by the indexer. This option may also be specified
 as a command line option (<tt>-t</tt>).
<tag><it>group</it>recordId</tag>
 Specifies how the record is to be identified when updated.
<tag><it>group</it>database</tag>
 Specifies the Z39.50 database.
<tag><it>group</it>storeKeys</tag>
 Specifies whether key information should be saved for a given
 group of records. If you plan to update/delete this type of
 records later this should be specified as 1; otherwise it
 should be 0 (default).
<tag><it>group</it>storeData</tag>
 Specifies whether the records should be stored internally
 in the Zebra system tables. If you want to maintain the raw records yourself,
 this option should be false (0). If you want Zebra to take care of the records
 for you, it should be true(1).
<tag>register</tag> 
 Specifies the location of the various files that Zebra uses to represent
 your system.
<tag>profilePath</tag>
 Specifies the location of profile specification paths.
<tag>attset</tag> 
 Specifies the filename(s) of attribute set files for use in searching.
</descrip>

<sect1>Locating Records
<p>
The default behaviour of the Zebra system is to reference the
records from their original location, i.e. where they were found when you
ran <tt/zebraidx/.

If your records files are temporary - for example if you retrieve
them from the outside, or if they where temporarily mounted on a CD-ROM,
you may want Zebra to make a copy of them. To do this,
you specify 1 (true) in the <tt>storedata</tt> setting. When
the Z39.50 server retrieves records they will be read from the
internal file structures of the system.

<sect1>Indexing with no Record IDs (Simple Indexing)

<p>
If you have a set of records that you <em/never/ wish to delete
or modify you may find &dquot;indexing without records IDs&dquot; convenient.
This indexing method uses less space than the other methods and
is simple to use. 

To use this method, you simply don't provide the <tt>recordId</tt> entry
for the group of files that you index. To add a set of records you use
<tt>zebraidx</tt> with the <tt>update</tt> command. The
<tt>update</tt> command will always add all of the records to the index
becuase Zebra doesn't know how to match the new set of records with
existing records.

Consider a system in which you have a group of text files called
<tt>simple</tt>. That group of records should belong to a Z39.50 database
called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice:

<tscreen><verb>
profilePath: /usr/local/yaz
attset: bib1.att
attset: gils.att
simple.recordType: text
simple.database: textbase
</verb></tscreen>

Since the existing records in an index can not be addressed by their
IDs, it is impossible to delete records when using this method.

<sect1>Indexing with File Record IDs

<p>
If you have a set of external records that you wish to index you may
use the file key feature of the Zebra system. In short, the file key
feature mirrors a directory structure and its files efficiently. To
perform indexing of a directory with file keys, you specify the top-level
directory after the <tt>update</tt> command. The command will recursively
traverse the directories and compare each with whatever have been
indexed before in the same directory. If a file is new (not in
the previous version of the directory) it is inserted;
if a file was already indexed and it has been modified
since the last insertion the index is also modified; if a file is missing
since the last visit it is deleted from the index.

The resulting system is easy to administer. To delete a record
you simply have to delete the corresponding file (with <tt/rm/). 
To force update of a given file, you may use the <tt>touch</tt>
command. And to add files create new files (or directories with files).
For your changes to take effect you must run <tt>zebraidx</tt> with
the same directory root again.

To use this method, you must specify <tt>file</tt> as the value
of <tt>recordId</tt> in the configuration file. In the configuration
also set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra
indexer must save additional information per record in order to
modify/delete the records at a later time.

For example, to update group <tt>esdd</tt> records below
<tt>/home/grs</tt> you could type:
<tscreen><verb>
$ zebraidx -g esdd update /home/grs
</verb></tscreen>

The corresponding configuration file includes:
<tscreen><verb>
esdd.recordId: file
esdd.recordType: grs
esdd.storeKeys: 1
</verb></tscreen>

<em>Important note: You cannot start out with a group of records with simple
indexing (no record IDs as in the previous section) and then later
enable file record Ids. Zebra must know from the first time that you
index the group that
the files should be indexed with file record IDs.
</em>

You cannot explicitly delete records when using this method. Instead
you have to delete the files from the file system (or remove them)
and then run <tt>zebraidx</tt> with the <tt>update</tt> again.

<sect1>Indexing with General Record IDs
<p>
When using this method you specify an (almost) arbritrary record key
based on the contents of the record itself and other system
information. If you have a group of records that have an external
ID associated with each records, this method is convenient. For
example, the record may contain a title or a unique ID-number. In either
case you specify the Z39.50 attribute set and use-attribute location
in which this information is stored.

As before, the record ID is defined by the <tt>recordId</tt> setting
in the configuration file. The value of the record ID specification
consists of one or more tokens separated by whitespace. The resulting
ID is
represented in the index by concatenating the tokens and separating them by
ASCII value (1).

There are three kinds of tokens:
<descrip>
<tag>Internal record info</tag> The token refers to a key that is
extracted from the record. The syntax of this token is
 <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the
attribute set ordinal number and <em/use/ is the use value of the attribute.
<tag>System variable</tag> The system variables are preceded by
<verb>$</verb> and immediately followed by the system variable name, which
may one of
 <descrip>
 <tag>group</tag> Group name.
 <tag>database</tag> Current database specified.
 <tag>type</tag> Record type.
 </descrip>
<tag>Constant string</tag> A string used as part of id &mdash; surrounded
 by single- or double quotes.
</descrip>

The test GILS records that comes with the Zebra distribution contain a
unique ID
in the Control-Identifier field. This field is mapped to the Bib-1
use attribute 1007. To use this field as a record id, specify
<tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the
configuration file. If you have other record types that don't
contain an ID in the same field, you might add the record type
in the record id of the gils records as well, to prevent matches
of other types of records. In this case the recordId might be
set like this:
<tscreen><verb>
gils.recordId: $type (1,1007)
</verb></tscreen>

As for the file record id case described in the previous section
updating your system is simply a matter of running <tt>zebraidx</tt>
with the <tt>update</tt> command. However, the update with general
keys is considerably slower than with file record IDs, since all files
visited must be (re)read to find their IDs. 

You may have noticed that when using the general record IDs
method, you can only add or modify existing records with the <tt>update</tt>
command. If you wish to delete records, you must use another command,
<tt>delete</tt>, which a root directory as a parameter. This will remove
all records that match the files below the root directory.

<sect1>Register location

<p>
Normally, the index files that form dictionaries, inverted
files, record info, etc., are stored in the directory where you run
<tt>zebraidx</tt>. If you wish to store these, possibly large, files
somewhere else, you must add the <tt>register</tt> entry to the
configuration file. Furthermore, the Zebra system allows its file
structures to
span multiple file systems, which is useful if a very large number of
records are stored.

The value <tt>register</tt> of register is a sequence of tokens.
Each token takes the form:
<tscreen>
<em>dir</em><tt>:</tt><em>size</em>. 
</tscreen>
The <em>dir</em> specifies a directory in which index files will be
stored and the <em>size</em> specifies the maximum size of all
files in that directory. The Zebra indexer system fill each directory
in the order specified and use the next specified directories as needed.
The <em>size</em> is an integer followed by a qualifier
code, <tt>M</tt> for megabytes, <tt>k</tt> for kilobytes.

For instance, if you have two spare disks :) and the first disk is mounted
on <tt>/d1</tt> and has 200 Mb of free space and the
second, mounted on <tt>/d2</tt> has 300 Mb, you could
put this entry in your configuration file:
<tscreen><verb>
register: /d1:200M /d2:300M
</verb></tscreen>

<sect>The Z39.50 Server

<p>

<sect1>Running the server
<p>
The server <tt>zebrasrv</tt> supports the same set of options as the 
test server <tt>ztest</tt> that comes with YAZ. As for the 
<tt>zebraidx</tt> the option <tt>-c</tt> specifies the configuration
filename. When the Zebra server is executed with its normal log level it 
prints (not too detailed) information about the incoming queries. 
This is useful if you don't happen to know what attributes your client sends.

Note that the server doesn't support the static mode (-S). 

<sect>License

<p>
Copyright &copy; 1995, Index Data.

All rights reserved.

Use and redistribution in source or binary form, with or without
modification, of any or all of this software and documentation is
permitted, provided that the following conditions are met:

1. This copyright and permission notice appear with all copies of the
software and its documentation. Notices of copyright or attribution
which appear at the beginning of any file must remain unchanged.

2. The names of Index Data or the individual authors may not be used to
endorse or promote products derived from this software without specific
prior written permission.

3. Source code or binary versions of this software and its documentation
may be used in not-for-profit applications. For profit aplications -
including marketing a product based in whole or in part on this software,
or providing for-pay database services - must obtain a commercial
license from Index Data.

THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR
NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF
LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
OF THIS SOFTWARE.

<sect>About Index Data

<p>
Index Data is a consulting and software-development enterprise that
specialises in library and information management systems. Our
interests and expertise span a broad range of related fields, and one
of our primary, long-term objectives is the development of a powerful
information management
system with open network interfaces and hypermedia capabilities.

We make this software available free of charge for noncommercial
purposes, as a service to the networking community, and to further
the development of quality software for open network communication.

We'll be happy to answer questions about the software, and about ourselves
in general.

<tscreen>
Index Data&nl
Ryesgade 3&nl
DK-2200 K&oslash;benhavn N&nl
</tscreen>

<p>
<tscreen><verb>
Phone: +45 3536 3672
Fax  : +45 3536 0449
Email: info@index.ping.dk
</verb></tscreen>

The <it>Random House College Dictionary</it>, 1975 edition
offers this definition of the 
word &dquot;Zebra&dquot;:

<it>
Zebra, n., any of several horselike, African mammals of the genus Equus,
having a characteristic pattern of black or dark-brown stripes on
a whitish background.
</it>

</article>