<!doctype linuxdoc system>
<!--
- $Id: zebra.sgml,v 1.1 1995-11-27 14:23:53 adam Exp $
+ $Id: zebra.sgml,v 1.2 1995-11-28 14:25:23 adam Exp $
-->
<article>
<title>Zebra server Administrators's Guide and Reference
<author>Index Data, <tt/info@index.ping.dk/
-<date>$Revision: 1.1 $
+<date>$Revision: 1.2 $
<abstract>
This document describes Zebra — an information retrieval server
that uses the Z39.50 protocol.
server system — GNU C works fine.
Unpack the Zebra software. You might put Zebra in the same directory level
-as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-</tt>.. Zebra
-is placed in ..<tt>/src/zebra-</tt>.
+as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-</tt>.., then
+Zebra is placed in ..<tt>/src/zebra-</tt>.
Edit the top-level <tt>Makefile</tt> in the Zebra directory in which
you specify the location of YAZ by setting make variables.
The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with
MOSI support. Some systems, such as Solaris, have separate socket
-libraries and for such systems you need to specify the
+libraries and for those systems you need to specify the
<tt>NETLIB</tt> variable.
-When finished with the editing of the <tt>Makefile</tt> type:
+When finished editing the <tt>Makefile</tt> type:
<tscreen><verb>
$ make
</verb></tscreen>
<tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
</descrip>
+<sect>Quick getting started
+
+<p>
+This section will get you started quickly! We will try to index a few
+GILS records that are included with the Zebra distribution. Go to the
+<tt>test</tt> subdirectory. There you will find a configuration
+file named <tt>zebra.cfg</tt> with the following contents:
+<tscreen><verb>
+# Where are the YAZ tables located.
+profilePath: /usr/local/yaz
+
+# Files that describe the attribute sets supported.
+attset: bib1.att
+attset: gils.att
+</verb></tscreen>
+
+Now, edit the file and set <tt>profilePath</tt> to the path of the
+YAZ profile tables (sub directory <tt>tab</tt> of YAZ).
+
+The 48 test records are located in the sub directory <tt>records</tt>.
+To index these, type:
+<tscreen><verb>
+$ ../index/zebraidx -t grs update records
+</verb></tscreen>
+
+In the command above the option <tt>-t</tt> specified the record
+type — in this case <tt>grs</tt>. The word <tt>update</tt> followed
+by a directory root updates all files below that directory.
+
+If your indexing command went successful, you are now ready to
+setup a server. To start a server on port 2100, type:
+<tscreen><verb>
+$ ../index/zebrasrv tcp:@:2100
+</verb></tscreen>
+
+The Zebra server just made has one database called Default. It will
+return either USMARC/GRS or SUTRS depending on your client.
+
<sect>Administrating Zebra
<p>
-Unlike many other retrieval systems, the Zebra offers incremental
+Unlike many other retrieval systems, Zebra offers incremental
modifications of an existing index. Needless to say, these facilities
makes the administration of Zebra somewhat more complicated.
-Normally, when Zebra modifies the index it reads a number of records.
+Normally, when Zebra modifies the index it reads a number of records
+that you specify.
Depending on your specifications and on the contents of each record
-one the following situations occur:
+one the following scenarios occur:
<descrip>
<tag>Insert</tag> The record is indexed as if it never occurred
before. Either the Zebra system doesn't know how to identify the record or
Zebra can identify the record but didn't find it to be already indexed.
-<tag>Update</tag> The record has already been indexed. In this case
+<tag>Modify</tag> The record has already been indexed. In this case
either the contents of the record or the location (file) of the record
indicates that it has been indexed before.
<tag>Delete</tag> The record is deleted from the index. As in the
update-case it must be able to identify the record.
</descrip>
-Please note, that in both the update- and delete- case the Zebra
+Please note, that in both the modify- and delete- case the Zebra
system must be able to make a key that identifies the record in
question.
To administrate the Zebra retrieval system, you run the
<tt>zebraidx</tt> program. This program supports a number of options
-which are preceded by a minus, and a set of commands (not preceded by
+which are preceded by a minus, and a few commands (not preceded by
minus).
Both the Zebra administrative tool and the Z39.50 server share a
various kinds of records and where the other configuration files
are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
be run in the same directory where the configuration file if you do
-not indicate the location of the configuration file by option <tt>-c</tt>.
-
-<sect1>Indexing
+not indicate the location of the configuration file by option
+<tt>-c</tt>.
+<sect1>Record types
<p>
-As said before, indexing is a record-per-record process, in which
-either insert/update/delete will occur. Before a record is indexed
+Indexing is a record-per-record process, in which
+either insert/modify/delete will occur. Before a record is indexed
search keys are extracted from whatever might be the layout the
-original record (sgml,html,text, etc..). To specify a particular
-extraction process, specify either the command line option <tt>-t</tt>
-or specify a <tt>fileExtension</tt> line in the configuration file.
-
-<sect2>Simple indexing
+original record (sgml,html,text, etc..). The Zebra system
+currently only support GILS records and simple text records.
+To specify a particular extraction process, use either the
+command line option <tt>-t</tt> or specify a
+<tt>recordType</tt> setting in the configuration file.
+<sect1>The Zebra configuration file
<p>
-In the simple case, in which you wish to index a set of files and you
-run the <tt>zebraidx</tt> program with the <tt>add</tt> command.
+The Zebra configuration file, read by <tt>zebraidx</tt> and
+<tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
+by <tt>-c</tt> option.
+
+You can edit the configuration file with a normal text editor.
+Setting names and values are seperated by colons in the file. Lines
+starting with a hash sign (<tt/#/) are treated as comments.
-For example to index the files below the directory <tt>/home/grs</tt>
-as GRS records you would type:
+A set of records that share common characteristics are called a group.
+When <tt>zebraidx</tt> is run and you wish to address a given group
+you specify that group with the <tt>-g</tt> option. In this case
+settings that have the group name as their prefix will be used
+by <tt>zebraidx</tt> and not default values.
+
+The group is written before the option itself separated by a dot.
+For example to set the record type for group <tt/public/ to <tt/grs/
+you would write:
<tscreen><verb>
-$ zebraidx -t grs add /home/grs
+public.recordType: grs
</verb></tscreen>
-The <tt>-t</tt> options indicates that the files should be treated
-as GRS records. The <tt>zebra.cfg</tt> must be located in the current
-directory and the configuation must specify where the profile tables
-and attribute specification files that comes with YAZ are located.
-For example, if your version of YAZ is located <tt>/home/yaz</tt>
-the configation file should include the following:
+To set the default value of the record type to text write:
<tscreen><verb>
-profilePath: /home/yaz/tab
-attset: bib1.att
-attset: gils.att
+recordType: text
</verb></tscreen>
-The <tt>attset</tt>-lines specify the locations of attribute set
-specification files.
+The configuration settings are summarized below. They will be
+explained further in the following chapters.
+
+<descrip>
+<tag><it>group</it>recordType<it>name</it></tag>
+ Specifies how records with the file extension <it>name</it> should
+ be handled by the indexer. This option may also be specified
+ as a command line option (<tt>-t</tt>).
+<tag><it>group</it>recordId</tag>
+ Specifies how the record is to be identified when updated.
+<tag><it>group</it>database</tag>
+ Specifies the Z39.50 database.
+<tag><it>group</it>storeKeys</tag>
+ Specifies whether key information should be saved for a given
+ group of records. If you plan to update/delete this type of
+ record a later this should be specified as 1; otherwise it
+ should be 0 (default).
+<tag><it>group</it>storeData</tag>
+ Specifies whether original copy of record should be stored internally
+ in the Zebra system indexes. If your externally indexed files
+ are temporary this option should certainly be true (1); otherwise
+ false (0).
+<tag>register</tag>
+ Specifies the location of the indexes.
+<tag>profilePath</tag>
+ Specifies the location of profile specification paths.
+<tag>attset</tag>
+ Specifies the filename(s) of attribute specification files.
+</descrip>
+
+<sect1>Locating records
+<p>
+The default behaviour of the Zebra system is to reference the
+records from their original location, i.e. where they were
+read an indexed.
+
+If your records files are temporary for example if you retrieve
+them from the outside or if they where temporarily mounted on a CD-ROM,
+you may want the Zebra server to make a copy of them. To do this
+you specify 1 (true) in the <tt>storedata</tt> setting. When
+the Z39.50 search engine retrieves recoreds they will be read from an
+internal structure of the index.
-At a later time, you can add another set of records with the
-<tt>add</tt> command. However, records will always be inserted
-and not updated because, we didn't specify a match criteria for
-the record(s).
+<sect1>Indexing with no record Ids
-One other property of the system just created, is that records are stored
-externally, i.e. the system reads the records when they are
-retrieved from their original location by the Z39.50 server.
-Therefore, you shouldn't delete files that you indexed this way.
+<p>
+If you have a set of records that you <em/never/ wish to delete
+or modify you may find 'indexing with no records Ids' convenient.
+This indexing method uses less space than the other methods and
+is simple to use.
+
+To use this method, you simply don't specify the <tt>recordId</tt>
+for the group of files you index. To add a set of records you use
+<tt>zebraidx</tt> with the <tt>update</tt> command. The
+<tt>update</tt> command will always add the records to the index
+becuase Zebra doesn't know how to match the new set of records with
+existing records.
+
+Consider a system, in which you have a group of text files called
+<tt>simple</tt>. That group of records should belong to a Z39.50 database
+called <tt>textbase</tt>. The following configuration file suffice:
+
+<tscreen><verb>
+profilePath: /usr/local/yaz
+attset: bib1.att
+attset: gils.att
+simple.recordType: text
+simple.database: textbase
+</verb></tscreen>
-<sect2>Indexing with file keys
+<sect1>Indexing with file record Ids
<p>
If you have a set of external records that you wish to index you may
use the file key feature of the Zebra system. In short, the file key
feature, mirrors a directory and the files below efficiently. To
-perform indexing of a directory with file keys you specify the top-level
-directory after the <tt>dir</tt> command. The command will recursively
+perform indexing of a directory with file keys, you specify the top-level
+directory after the <tt>update</tt> command. The command will recursively
traverse the directories and compare each with whatever have been
indexed before in the same directory. If a file is new (not in
the previous version of the directory) it is inserted;
if a file was already indexed and it has been modified
-since last insertion it is updated; if a file is missing since last
-visit it is deleted.
+since the last insertion the index is also modified; if a file is missing
+since the last visit it is deleted.
+
+The resulting system is easy to administer. To delete a record
+you simply have to delete the corresponding file (with <tt/rm/).
+To force update of a given file, you may use the <tt>touch</tt>
+command. And to add files create new files (or directories with files).
+For your changes to take effect you must run <tt>zebraidx</tt> with
+the same directory root again.
+
+To use this method, you must specify <tt>file</tt> as the value
+of <tt>recordId</tt> the configuration file. In the configuration
+also set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra
+indexer must save additional information in order to
+modify/delete the record at a later time.
+
+For example, to update group <tt>esdd</tt> records below
+<tt>/home/grs</tt> you could type:
+<tscreen><verb>
+$ zebraidx -g esdd update /home/grs
+</verb></tscreen>
-For example, to index the GRS files below <tt>/home/grs</tt> you
-would type:
+The corresponding configuration file includes:
<tscreen><verb>
-$ zebraidx -t grs dir /home/grs
+esdd.recordId: file
+esdd.recordType: grs
+esdd.storeKeys: 1
</verb></tscreen>
-<sect2>Indexing with general keys
+<em>Important note: You cannot start by a group of records with simple
+indexing (no record ids as in the previous section) and then, later,
+use file record Ids. The Zebra must know from the very beginning that
+the group of files are indexed with file record Ids.
+</em>
+<sect1>Indexing with general record Ids
<p>
-About the fileMatch thing.
+When using this method you specify an (almost) arbritrary record key
+based on the contents of the record itself and other system
+information. If you have a group of records that have an external
+id attached, this method is convenient. For example, the record may
+contain a title or an unique id. In either case you specify the
+Z39.50 attribute-set and attribute-use location in which this
+information is stored.
+
+As before, the record id is defined by the <tt>recordId</tt> setting
+in the configuration file. The value of the record id consists of one
+or more tokens separated by space. The resulting id is formed
+by concatenating the tokens and separete them by ascii value (1).
+
+There are three kinds of tokens:
+<descrip>
+<tag>Internal record info</tag> The token refers to a key that is
+extracted from the record. The syntax of this token is
+ <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the
+attribute set number and <em/use/ is the use value of the attribute.
+<tag>System variable</tag> The system variables are preceded by
+<tt>$</tt> and immediately followed by the system variable name, which
+may one of
+ <descrip>
+ <tag>group</tag> Group name.
+ <tag>database</tag> Current database specified.
+ <tag>filename</tag> Name of file that contain the record.
+ <tag>type</tag> Record type.
+ </descrip>
+<tag>Constant string</tag> A string used as part of id — surrounded
+ by single- or double quotes.
+</descrip>
-<sect1>File areas
+The test GILS records that comes with Zebra, contain a unique id
+in the Control-Identifier field. This field is mapped to the Bib-1
+use attribute 1007. To use this field as a record id, specify
+<tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the
+configuration file. If you have other record types that doesn't
+contain an id in the same field, you might add the record type
+in the record id of the gils records as well, to prevent match
+of other types of records. In this case the recordId might be
+set, as in:
+<tscreen><verb>
+gils.recordId: $type (1,1007)
+</verb></tscreen>
-<p>
-About the register thing.
+As for the file record id case described in the previous section
+updating your system is simply a matter of running <tt>zebraidx</tt>
+with the <tt>update</tt> command. However, the update of with general
+keys is considerably slower than with file record ids, since all files
+visited must be (re)indexded.
-<sect1>The Zebra configuration file
+<sect1>Register location
<p>
-The Zebra configuration file, read by <tt>zebraidx</tt> and
-<tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
-by <tt>-c</tt> option.
-
-<descrip>
-<tag><it>group</it>fileExtension<it>name</it></tag>
- Specifies how records with the file extension <it>name</it> should
- be handled by the indexer.
-<tag><it>group</it>fileMatch</tag>
- Specifies the key information that identifies the record.
-<tag><it>group</it>database</tag>
- Specifies the Z39.50 database of a group of records.
-<tag><it>group</it>storeKeys</tag>
- Specifies whether key information should be saved for a given
- group of records. If you plan to update/delete this type of
- record a later this should be specified as 1; otherwise it
- should be 0 (default).
-<tag><it>group</it>storeData</tag>
- Specifies whether original copy of record should be stored internally
- in the Zebra system indexes. If your externally indexed files
- are temporary this option should certainly be true (1); otherwise
- false (0).
-<tag>register</tag>
- Specifies the location of the indexes.
-<tag>profilePath</tag>
- Specifies the location of profile specification paths.
-<tag>attset</tag>
- Specifies the filename(s) of attribute specification files.
-</descrip>
+Normally, the index files that form dictionaries, inverted
+file, record info, etc. are stored in the current directory in which you run
+<tt>zebraidx</tt>. If you wish to store those, possibly large, files
+somewhere else, you must set the <tt>register</tt> setting in the
+configuration file. Furhtermore, the Zebra system handles indexes that
+span multiple file systems, which is usefull if a large number of
+records are indexed.
+
+The value <tt>register</tt> of register is a sequence of tokens.
+Each token takes the form:
+<tscreen>
+<em>dir</em><tt>:</tt><em>size</em>.
+</tscreen>
+The <em>dir</em> specifies a directory in which index files will be
+stored and the <em>size</em> specifies the maximum size of all
+files in that directory. The Zebra indexer system fill each directory
+in the order specified and use the next specified directories as needed.
+The <em>size</em> is an integer possibly followed by a quantity
+code, <tt>M</tt> for Mega bytes, <tt>K</tt> for Kilo bytes.
+
+If you have two spare disks :) and the first disk is mounted
+on <tt>/d1</tt> and has 200 Mega bytes free space after formatting and the
+second, mounted on <tt>/d2</tt> has 300 Mega bytes free, you could
+specify the following in you configuration file:
+<tscreen><verb>
+register: /d1:200M /d2:300M
+</verb></tscreen>
<sect>The Z39.50 Server
<p>
-<descrip>
-<tag>1 Ousterhout, John K.:</tag>
-Tcl and the Tk Toolkit. Addison-Wesley Company Inc (ISBN
-0-201-63337-X). Source and documentation
-can be found in <tt>URL:ftp://ftp.cs.berkeley.edu/pub/tcl</tt>
-and mirrors.
-<tag>2 Furniss, Peter:</tag>
-RFC 1698: Octet Sequences for Upper-Layer OSI to Support
-Basic Communications Applications.
-</descrip>
-
</article>
projects / idzebra-moved-to-github.git / commitdiff