[idzebra-moved-to-github.git] / doc / administration.xml

<chapter id="quick-start">
<title>Quick Start </title>

<para>
In this section, we will test the system by indexing a small set of sample
GILS records that are included with the software distribution. Go to the
<literal remap="tt">test/gils</literal> subdirectory of the distribution archive. There you will
find a configuration
file named <literal remap="tt">zebra.cfg</literal> with the following contents:

<screen>
# Where are the YAZ tables located.
profilePath: ../../../yaz/tab ../../tab

# Files that describe the attribute sets supported.
attset: bib1.att
attset: gils.att
</screen>

</para>

<para>
Now, edit the file and set <literal remap="tt">profilePath</literal> to the path of the
YAZ profile tables (sub directory <literal remap="tt">tab</literal> of the YAZ distribution
archive).
</para>

<para>
The 48 test records are located in the sub directory <literal remap="tt">records</literal>.
To index these, type:

<screen>
$ ../../index/zebraidx -t grs.sgml update records
</screen>

</para>

<para>
In the command above the option <literal remap="tt">-t</literal> specified the record
type &mdash; in this case <literal remap="tt">grs.sgml</literal>. The word <literal remap="tt">update</literal> followed
by a directory root updates all files below that directory node.
</para>

<para>
If your indexing command was successful, you are now ready to
fire up a server. To start a server on port 2100, type:

<screen>
$ ../../index/zebrasrv tcp:@:2100
</screen>

</para>

<para>
The Zebra index that you have just created has a single database
named <literal remap="tt">Default</literal>. The database contains records structured according to
the GILS profile, and the server will
return records in either either USMARC, GRS-1, or SUTRS depending
on what your client asks
for.
</para>

<para>
To test the server, you can use any Z39.50 client (1992 or later). For
instance, you can use the demo client that comes with YAZ: Just cd to
the <literal remap="tt">client</literal> subdirectory of the YAZ distribution and type:
</para>

<para>

<screen>
$ client tcp:localhost:2100
</screen>

</para>

<para>
When the client has connected, you can type:
</para>

<para>

<screen>
Z&#62; find surficial
Z&#62; show 1
</screen>

</para>

<para>
The default retrieval syntax for the client is USMARC. To try other
formats for the same record, try:
</para>

<para>

<screen>
Z&#62;format sutrs
Z&#62;show 1
Z&#62;format grs-1
Z&#62;show 1
Z&#62;elements B
Z&#62;show 1
</screen>

</para>

<para>
<emphasis remap="it">NOTE: You may notice that more fields are returned when your
client requests SUTRS or GRS-1 records. When retrieving GILS records,
this is normal - not all of the GILS data elements have mappings in
the USMARC record format.</emphasis>
</para>

<para>
If you've made it this far, there's a good chance that
you've got through the compilation OK.
</para>

</chapter>

<chapter id="administration">
<title>Administrating Zebra</title>

<para>
Unlike many simpler retrieval systems, Zebra supports safe, incremental
updates to an existing index.
</para>

<para>
Normally, when Zebra modifies the index it reads a number of records
that you specify.
Depending on your specifications and on the contents of each record
one the following events take place for each record:
<variablelist>

<varlistentry>
<term>Insert</term>
<listitem>
<para>
The record is indexed as if it never occurred
before. Either the Zebra system doesn't know how to identify the record or
Zebra can identify the record but didn't find it to be already indexed.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Modify</term>
<listitem>
<para>
The record has already been indexed. In this case
either the contents of the record or the location (file) of the record
indicates that it has been indexed before.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Delete</term>
<listitem>
<para>
The record is deleted from the index. As in the
update-case it must be able to identify the record.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>

<para>
Please note that in both the modify- and delete- case the Zebra
indexer must be able to generate a unique key that identifies the record in
question (more on this below).
</para>

<para>
To administrate the Zebra retrieval system, you run the
<literal remap="tt">zebraidx</literal> program. This program supports a number of options
which are preceded by a minus, and a few commands (not preceded by
minus).
</para>

<para>
Both the Zebra administrative tool and the Z39.50 server share a
set of index files and a global configuration file. The
name of the configuration file defaults to <literal remap="tt">zebra.cfg</literal>.
The configuration file includes specifications on how to index
various kinds of records and where the other configuration files
are located. <literal remap="tt">zebrasrv</literal> and <literal remap="tt">zebraidx</literal> <emphasis>must</emphasis>
be run in the directory where the configuration file lives unless you
indicate the location of the configuration file by option
<literal remap="tt">-c</literal>.
</para>

<sect1 id="record-types">
<title>Record Types</title>

<para>
Indexing is a per-record process, in which either insert/modify/delete
will occur. Before a record is indexed search keys are extracted from
whatever might be the layout the original record (sgml,html,text, etc..).
The Zebra system currently supports two fundamantal types of records:
structured and simple text.
To specify a particular extraction process, use either the
command line option <literal remap="tt">-t</literal> or specify a
<literal remap="tt">recordType</literal> setting in the configuration file.
</para>

</sect1>

<sect1 id="configuration-file">
<title>The Zebra Configuration File</title>

<para>
The Zebra configuration file, read by <literal remap="tt">zebraidx</literal> and
<literal remap="tt">zebrasrv</literal> defaults to <literal remap="tt">zebra.cfg</literal> unless specified
by <literal remap="tt">-c</literal> option.
</para>

<para>
You can edit the configuration file with a normal text editor.
parameter names and values are seperated by colons in the file. Lines
starting with a hash sign (<literal remap="tt">&num;</literal>) are treated as comments.
</para>

<para>
If you manage different sets of records that share common
characteristics, you can organize the configuration settings for each
type into "groups".
When <literal remap="tt">zebraidx</literal> is run and you wish to address a given group
you specify the group name with the <literal remap="tt">-g</literal> option. In this case
settings that have the group name as their prefix will be used
by <literal remap="tt">zebraidx</literal>. If no <literal remap="tt">-g</literal> option is specified, the settings
with no prefix are used.
</para>

<para>
In the configuration file, the group name is placed before the option
name itself, separated by a dot (.). For instance, to set the record type
for group <literal remap="tt">public</literal> to <literal remap="tt">grs.sgml</literal> (the SGML-like format for structured
records) you would write:
</para>

<para>

<screen>
public.recordType: grs.sgml
</screen>

</para>

<para>
To set the default value of the record type to <literal remap="tt">text</literal> write:
</para>

<para>

<screen>
recordType: text
</screen>

</para>

<para>
The available configuration settings are summarized below. They will be
explained further in the following sections.
</para>

<para>
<variablelist>

<varlistentry>
<term><emphasis remap="it">group</emphasis>.recordType&lsqb;<emphasis remap="it">.name</emphasis>&rsqb;</term>
<listitem>
<para>
Specifies how records with the file extension <emphasis remap="it">name</emphasis> should
be handled by the indexer. This option may also be specified
as a command line option (<literal remap="tt">-t</literal>). Note that if you do not
specify a <emphasis remap="it">name</emphasis>, the setting applies to all files. In general,
the record type specifier consists of the elements (each
element separated by dot), <emphasis remap="it">fundamental-type</emphasis>,
<emphasis remap="it">file-read-type</emphasis> and arguments. Currently, two
fundamental types exist, <literal remap="tt">text</literal> and <literal remap="tt">grs</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis remap="it">group</emphasis>.recordId</term>
<listitem>
<para>
Specifies how the records are to be identified when updated. See
section <xref linkend="locating-records"/>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis remap="it">group</emphasis>.database</term>
<listitem>
<para>
Specifies the Z39.50 database name.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis remap="it">group</emphasis>.storeKeys</term>
<listitem>
<para>
Specifies whether key information should be saved for a given
group of records. If you plan to update/delete this type of
records later this should be specified as 1; otherwise it
should be 0 (default), to save register space. See section
<xref linkend="file-ids"/>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis remap="it">group</emphasis>.storeData</term>
<listitem>
<para>
Specifies whether the records should be stored internally
in the Zebra system files. If you want to maintain the raw records yourself,
this option should be false (0). If you want Zebra to take care of the records
for you, it should be true(1).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>register</term>
<listitem>
<para>
Specifies the location of the various register files that Zebra uses
to represent your databases. See section
<xref linkend="register-location"/>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>shadow</term>
<listitem>
<para>
Enables the <emphasis remap="it">safe update</emphasis> facility of Zebra, and tells the system
where to place the required, temporary files. See section
<xref linkend="shadow-registers"/>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>lockDir</term>
<listitem>
<para>
Directory in which various lock files are stored.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>keyTmpDir</term>
<listitem>
<para>
Directory in which temporary files used during zebraidx' update
phase are stored. 
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>setTmpDir</term>
<listitem>
<para>
Specifies the directory that the server uses for temporary result sets.
If not specified <literal remap="tt">/tmp</literal> will be used.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>profilePath</term>
<listitem>
<para>
Specifies the location of profile specification files.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>attset</term>
<listitem>
<para>
Specifies the filename(s) of attribute set files for use in
searching. At least the Bib-1 set should be loaded (<literal remap="tt">bib1.att</literal>).
The <literal remap="tt">profilePath</literal> setting is used to look for the specified files.
See section <xref linkend="attset-files"/>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>memMax</term>
<listitem>
<para>
Specifies size of internal memory to use for the zebraidx program. The
amount is given in megabytes - default is 4 (4 MB).
</para>
</listitem>
</varlistentry>
</variablelist>
</para>

</sect1>

<sect1 id="locating-records">
<title>Locating Records</title>

<para>
The default behaviour of the Zebra system is to reference the
records from their original location, i.e. where they were found when you
ran <literal remap="tt">zebraidx</literal>. That is, when a client wishes to retrieve a record
following a search operation, the files are accessed from the place
where you originally put them - if you remove the files (without
running <literal remap="tt">zebraidx</literal> again, the client will receive a diagnostic
message.
</para>

<para>
If your input files are not permanent - for example if you retrieve
your records from an outside source, or if they were temporarily
mounted on a CD-ROM drive,
you may want Zebra to make an internal copy of them. To do this,
you specify 1 (true) in the <literal remap="tt">storeData</literal> setting. When
the Z39.50 server retrieves the records they will be read from the
internal file structures of the system.
</para>

</sect1>

<sect1 id="simple-indexing">
<title>Indexing with no Record IDs (Simple Indexing)</title>

<para>
If you have a set of records that are not expected to change over time
you may can build your database without record IDs.
This indexing method uses less space than the other methods and
is simple to use. 
</para>

<para>
To use this method, you simply omit the <literal remap="tt">recordId</literal> entry
for the group of files that you index. To add a set of records you use
<literal remap="tt">zebraidx</literal> with the <literal remap="tt">update</literal> command. The
<literal remap="tt">update</literal> command will always add all of the records that it
encounters to the index - whether they have already been indexed or
not. If the set of indexed files change, you should delete all of the
index files, and build a new index from scratch.
</para>

<para>
Consider a system in which you have a group of text files called
<literal remap="tt">simple</literal>. That group of records should belong to a Z39.50 database
called <literal remap="tt">textbase</literal>. The following <literal remap="tt">zebra.cfg</literal> file will suffice:
</para>

<para>

<screen>
profilePath: /usr/local/yaz
attset: bib1.att
simple.recordType: text
simple.database: textbase
</screen>

</para>

<para>
Since the existing records in an index can not be addressed by their
IDs, it is impossible to delete or modify records when using this method.
</para>

</sect1>

<sect1 id="file-ids">
<title>Indexing with File Record IDs</title>

<para>
If you have a set of files that regularly change over time: Old files
are deleted, new ones are added, or existing files are modified, you
can benefit from using the <emphasis remap="it">file ID</emphasis> indexing methodology. Examples
of this type of database might include an index of WWW resources, or a
USENET news spool area. Briefly speaking, the file key methodology
uses the directory paths of the individual records as a unique
identifier for each record. To perform indexing of a directory with
file keys, again, you specify the top-level directory after the
<literal remap="tt">update</literal> command. The command will recursively traverse the
directories and compare each one with whatever have been indexed before in
that same directory. If a file is new (not in the previous version of
the directory) it is inserted into the registers; if a file was
already indexed and it has been modified since the last update,
the index is also modified; if a file has been removed since the last
visit, it is deleted from the index.
</para>

<para>
The resulting system is easy to administrate. To delete a record you
simply have to delete the corresponding file (say, with the <literal remap="tt">rm</literal>
command). And to add records you create new files (or directories with
files). For your changes to take effect in the register you must run
<literal remap="tt">zebraidx update</literal> with the same directory root again. This mode
of operation requires more disk space than simpler indexing methods,
but it makes it easier for you to keep the index in sync with a
frequently changing set of data. If you combine this system with the
<emphasis remap="it">safe update</emphasis> facility (see below), you never have to take your
server offline for maintenance or register updating purposes.
</para>

<para>
To enable indexing with pathname IDs, you must specify <literal remap="tt">file</literal> as
the value of <literal remap="tt">recordId</literal> in the configuration file. In addition,
you should set <literal remap="tt">storeKeys</literal> to <literal remap="tt">1</literal>, since the Zebra
indexer must save additional information about the contents of each record
in order to modify the indices correctly at a later time.
</para>

<para>
For example, to update records of group <literal remap="tt">esdd</literal> located below
<literal remap="tt">/data1/records/</literal> you should type:

<screen>
$ zebraidx -g esdd update /data1/records
</screen>

</para>

<para>
The corresponding configuration file includes:

<screen>
esdd.recordId: file
esdd.recordType: grs.sgml
esdd.storeKeys: 1
</screen>

</para>

<para>
<emphasis>Important note: You cannot start out with a group of records with simple
indexing (no record IDs as in the previous section) and then later
enable file record Ids. Zebra must know from the first time that you
index the group that
the files should be indexed with file record IDs.</emphasis>
</para>

<para>
You cannot explicitly delete records when using this method (using the
<emphasis remap="bf">delete</emphasis> command to <literal remap="tt">zebraidx</literal>. Instead
you have to delete the files from the file system (or move them to a
different location)
and then run <literal remap="tt">zebraidx</literal> with the <emphasis remap="bf">update</emphasis> command.
</para>

</sect1>

<sect1 id="generic-ids">
<title>Indexing with General Record IDs</title>

<para>
When using this method you construct an (almost) arbritrary, internal
record key based on the contents of the record itself and other system
information. If you have a group of records that explicitly associates
an ID with each record, this method is convenient. For example, the
record format may contain a title or a ID-number - unique within the group.
In either case you specify the Z39.50 attribute set and use-attribute
location in which this information is stored, and the system looks at
that field to determine the identity of the record.
</para>

<para>
As before, the record ID is defined by the <literal remap="tt">recordId</literal> setting
in the configuration file. The value of the record ID specification
consists of one or more tokens separated by whitespace. The resulting
ID is
represented in the index by concatenating the tokens and separating them by
ASCII value (1).
</para>

<para>
There are three kinds of tokens:
<variablelist>

<varlistentry>
<term>Internal record info</term>
<listitem>
<para>
The token refers to a key that is
extracted from the record. The syntax of this token is
<literal remap="tt">(</literal> <emphasis>set</emphasis> <literal remap="tt">,</literal> <emphasis>use</emphasis> <literal remap="tt">)</literal>, where <emphasis>set</emphasis> is the
attribute set name <emphasis>use</emphasis> is the name or value of the attribute.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>System variable</term>
<listitem>
<para>
The system variables are preceded by

<screen>
$
</screen>
 and immediately followed by the system variable name, which
may one of
<variablelist>

<varlistentry>
<term>group</term>
<listitem>
<para>
Group name.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>database</term>
<listitem>
<para>
Current database specified.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>type</term>
<listitem>
<para>
Record type.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Constant string</term>
<listitem>
<para>
A string used as part of the ID &mdash; surrounded
by single- or double quotes.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>

<para>
For instance, the sample GILS records that come with the Zebra
distribution contain a unique ID in the data tagged Control-Identifier.
The data is mapped to the Bib-1 use attribute Identifier-standard
(code 1007). To use this field as a record id, specify
<literal remap="tt">(bib1,Identifier-standard)</literal> as the value of the
<literal remap="tt">recordId</literal> in the configuration file.
If you have other record types that uses the same field for a
different purpose, you might add the record type
(or group or database name) to the record id of the gils
records as well, to prevent matches with other types of records.
In this case the recordId might be set like this:

<screen>
gils.recordId: $type (bib1,Identifier-standard)
</screen>

</para>

<para>
(see section <xref linkend="data-model"/>
for details of how the mapping between elements of your records and
searchable attributes is established).
</para>

<para>
As for the file record ID case described in the previous section,
updating your system is simply a matter of running <literal remap="tt">zebraidx</literal>
with the <literal remap="tt">update</literal> command. However, the update with general
keys is considerably slower than with file record IDs, since all files
visited must be (re)read to discover their IDs. 
</para>

<para>
As you might expect, when using the general record IDs
method, you can only add or modify existing records with the <literal remap="tt">update</literal>
command. If you wish to delete records, you must use the,
<literal remap="tt">delete</literal> command, with a directory as a parameter.
This will remove all records that match the files below that root
directory.
</para>

</sect1>

<sect1 id="register-location">
<title>Register Location</title>

<para>
Normally, the index files that form dictionaries, inverted
files, record info, etc., are stored in the directory where you run
<literal remap="tt">zebraidx</literal>. If you wish to store these, possibly large, files
somewhere else, you must add the <literal remap="tt">register</literal> entry to the
<literal remap="tt">zebra.cfg</literal> file. Furthermore, the Zebra system allows its file
structures to
span multiple file systems, which is useful for managing very large
databases. 
</para>

<para>
The value of the <literal remap="tt">register</literal> setting is a sequence of tokens.
Each token takes the form:

<screen>
<emphasis>dir</emphasis><literal remap="tt">:</literal><emphasis>size</emphasis>. 
</screen>

The <emphasis>dir</emphasis> specifies a directory in which index files will be
stored and the <emphasis>size</emphasis> specifies the maximum size of all
files in that directory. The Zebra indexer system fills each directory
in the order specified and use the next specified directories as needed.
The <emphasis>size</emphasis> is an integer followed by a qualifier
code, <literal remap="tt">M</literal> for megabytes, <literal remap="tt">k</literal> for kilobytes.
</para>

<para>
For instance, if you have allocated two disks for your register, and
the first disk is mounted
on <literal remap="tt">/d1</literal> and has 200 Mb of free space and the
second, mounted on <literal remap="tt">/d2</literal> has 300 Mb, you could
put this entry in your configuration file:

<screen>
register: /d1:200M /d2:300M
</screen>

</para>

<para>
Note that Zebra does not verify that the amount of space specified is
actually available on the directory (file system) specified - it is
your responsibility to ensure that enough space is available, and that
other applications do not attempt to use the free space. In a large production system,
it is recommended that you allocate one or more filesystem exclusively
to the Zebra register files.
</para>

</sect1>

<sect1 id="shadow-registers">
<title>Safe Updating - Using Shadow Registers</title>

<sect2>
<title>Description</title>

<para>
The Zebra server supports <emphasis remap="it">updating</emphasis> of the index structures. That is,
you can add, modify, or remove records from databases managed by Zebra
without rebuilding the entire index. Since this process involves
modifying structured files with various references between blocks of
data in the files, the update process is inherently sensitive to
system crashes, or to process interruptions: Anything but a
successfully completed update process will leave the register files in
an unknown state, and you will essentially have no recourse but to
re-index everything, or to restore the register files from a backup
medium. Further, while the update process is active, users cannot be
allowed to access the system, as the contents of the register files
may change unpredictably.
</para>

<para>
You can solve these problems by enabling the shadow register system in
Zebra. During the updating procedure, <literal remap="tt">zebraidx</literal> will temporarily
write changes to the involved files in a set of "shadow
files", without modifying the files that are accessed by the
active server processes. If the update procedure is interrupted by a
system crash or a signal, you simply repeat the procedure - the
register files have not been changed or damaged, and the partially
written shadow files are automatically deleted before the new updating
procedure commences.
</para>

<para>
At the end of the updating procedure (or in a separate operation, if
you so desire), the system enters a "commit mode". First,
any active server processes are forced to access those blocks that
have been changed from the shadow files rather than from the main
register files; the unmodified blocks are still accessed at their
normal location (the shadow files are not a complete copy of the
register files - they only contain those parts that have actually been
modified). If the commit process is interrupted at any point during the
commit process, the server processes will continue to access the
shadow files until you can repeat the commit procedure and complete
the writing of data to the main register files. You can perform
multiple update operations to the registers before you commit the
changes to the system files, or you can execute the commit operation
at the end of each update operation. When the commit phase has
completed successfully, any running server processes are instructed to
switch their operations to the new, operational register, and the
temporary shadow files are deleted.
</para>

</sect2>

<sect2>
<title>How to Use Shadow Register Files</title>

<para>
The first step is to allocate space on your system for the shadow
files. You do this by adding a <literal remap="tt">shadow</literal> entry to the <literal remap="tt">zebra.cfg</literal>
file. The syntax of the <literal remap="tt">shadow</literal> entry is exactly the same as for
the <literal remap="tt">register</literal> entry (see section <xref linkend="register-location"/>). The location of the shadow area should be
<emphasis remap="it">different</emphasis> from the location of the main register area (if you
have specified one - remember that if you provide no <literal remap="tt">register</literal>
setting, the default register area is the
working directory of the server and indexing processes).
</para>

<para>
The following excerpt from a <literal remap="tt">zebra.cfg</literal> file shows one example of
a setup that configures both the main register location and the shadow
file area. Note that two directories or partitions have been set aside
for the shadow file area. You can specify any number of directories
for each of the file areas, but remember that there should be no
overlaps between the directories used for the main registers and the
shadow files, respectively.
</para>

<para>

<screen>
register: /d1:500M

shadow: /scratch1:100M /scratch2:200M
</screen>

</para>

<para>
When shadow files are enabled, an extra command is available at the
<literal remap="tt">zebraidx</literal> command line. In order to make changes to the system
take effect for the users, you'll have to submit a
"commit" command after a (sequence of) update
operation(s). You can ask the indexer to commit the changes
immediately after the update operation:
</para>

<para>

<screen>
$ zebraidx update /d1/records update /d2/more-records commit
</screen>

</para>

<para>
Or you can execute multiple updates before committing the changes:
</para>

<para>

<screen>
$ zebraidx -g books update /d1/records update /d2/more-records
$ zebraidx -g fun update /d3/fun-records
$ zebraidx commit
</screen>

</para>

<para>
If one of the update operations above had been interrupted, the commit
operation on the last line would fail: <literal remap="tt">zebraidx</literal> will not let you
commit changes that would destroy the running register. You'll have to
rerun all of the update operations since your last commit operation,
before you can commit the new changes.
</para>

<para>
Similarly, if the commit operation fails, <literal remap="tt">zebraidx</literal> will not let
you start a new update operation before you have successfully repeated
the commit operation. The server processes will keep accessing the
shadow files rather than the (possibly damaged) blocks of the main
register files until the commit operation has successfully completed.
</para>

<para>
You should be aware that update operations may take slightly longer
when the shadow register system is enabled, since more file access
operations are involved. Further, while the disk space required for
the shadow register data is modest for a small update operation, you
may prefer to disable the system if you are adding a very large number
of records to an already very large database (we use the terms
<emphasis remap="it">large</emphasis> and <emphasis remap="it">modest</emphasis> very loosely here, since every
application will have a different perception of size). To update the system
without the use of the the shadow files, simply run <literal remap="tt">zebraidx</literal> with
the <literal remap="tt">-n</literal> option (note that you do not have to execute the
<emphasis remap="bf">commit</emphasis> command of <literal remap="tt">zebraidx</literal> when you temporarily disable the
use of the shadow registers in this fashion. Note also that, just as
when the shadow registers are not enabled, server processes will be
barred from accessing the main register while the update procedure
takes place.
</para>

</sect2>

</sect1>

</chapter>

<chapter id="zebraidx">
<title>Running the Maintenance Interface (zebraidx)</title>

<para>
The following is a complete reference to the command line interface to
the <literal remap="tt">zebraidx</literal> application.
</para>

<para>
<emphasis remap="bf">Syntax</emphasis>

<screen>
$ zebraidx &lsqb;options&rsqb; command &lsqb;directory&rsqb; ...
</screen>

<emphasis remap="bf">Options</emphasis>
<variablelist>

<varlistentry>
<term>-t <emphasis remap="it">type</emphasis></term>
<listitem>
<para>
Update all files as <emphasis remap="it">type</emphasis>. Currently, the
types supported are <literal remap="tt">text</literal> and <literal remap="tt">grs</literal><emphasis remap="it">.subtype</emphasis>. If no
<emphasis remap="it">subtype</emphasis> is provided for the GRS (General Record Structure) type,
the canonical input format is assumed (see section <xref linkend="local-representation"/>). Generally, it
is probably advisable to specify the record types in the
<literal remap="tt">zebra.cfg</literal> file
(see section <xref linkend="record-types"/>), to avoid
confusion at subsequent updates.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-c <emphasis remap="it">config-file</emphasis></term>
<listitem>
<para>
Read the configuration file
<emphasis remap="it">config-file</emphasis> instead of <literal remap="tt">zebra.cfg</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-g <emphasis remap="it">group</emphasis></term>
<listitem>
<para>
Update the files according to the group
settings for <emphasis remap="it">group</emphasis> (see section
<xref linkend="configuration-file"/>).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-d <emphasis remap="it">database</emphasis></term>
<listitem>
<para>
The records located should be associated
with the database name <emphasis remap="it">database</emphasis> for access through the Z39.50
server.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-m <emphasis remap="it">mbytes</emphasis></term>
<listitem>
<para>
Use <emphasis remap="it">mbytes</emphasis> of megabytes before flushing
keys to background storage. This setting affects performance when
updating large databases.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-n</term>
<listitem>
<para>
Disable the use of shadow registers for this operation
(see section <xref linkend="shadow-registers"/>).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-s</term>
<listitem>
<para>
Show analysis of the indexing process. The maintenance
program works in a read-only mode and doesn't change the state
of the index. This options is very useful when you wish to test a
new profile.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-V</term>
<listitem>
<para>
Show Zebra version.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-v <emphasis remap="it">level</emphasis></term>
<listitem>
<para>
Set the log level to <emphasis remap="it">level</emphasis>. <emphasis remap="it">level</emphasis>
should be one of <literal remap="tt">none</literal>, <literal remap="tt">debug</literal>, and <literal remap="tt">all</literal>.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>

<para>
<emphasis remap="bf">Commands</emphasis>
<variablelist>

<varlistentry>
<term>Update <emphasis remap="it">directory</emphasis></term>
<listitem>
<para>
Update the register with the files
contained in <emphasis remap="it">directory</emphasis>. If no directory is provided, a list of
files is read from <literal remap="tt">stdin</literal>.
See section <xref linkend="administration"/>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Delete <emphasis remap="it">directory</emphasis></term>
<listitem>
<para>
Remove the records corresponding to
the files found under <emphasis remap="it">directory</emphasis> from the register.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Commit</term>
<listitem>
<para>
Write the changes resulting from the last <emphasis remap="bf">update</emphasis>
commands to the register. This command is only available if the use of
shadow register files is enabled (see section
<xref linkend="shadow-registers"/>).
</para>
</listitem>
</varlistentry>
</variablelist>
</para>

</chapter>

<chapter id="server">
<title>The Z39.50 Server</title>

<sect1 id="zebrasrv">
<title>Running the Z39.50 Server (zebrasrv)</title>

<para>
<emphasis remap="bf">Syntax</emphasis>

<screen>
zebrasrv &lsqb;options&rsqb; &lsqb;listener-address ...&rsqb;
</screen>

</para>

<para>
<emphasis remap="bf">Options</emphasis>
<variablelist>

<varlistentry>
<term>-a <emphasis remap="it">APDU file</emphasis></term>
<listitem>
<para>
Specify a file for dumping PDUs (for diagnostic purposes).
The special name "-" sends output to <literal>stderr</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-c <emphasis remap="it">config-file</emphasis></term>
<listitem>
<para>
Read configuration information from <emphasis remap="it">config-file</emphasis>. The default configuration is <literal remap="tt">./zebra.cfg</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-S</term>
<listitem>
<para>
Don't fork on connection requests. This can be useful for
symbolic-level debugging. The server can only accept a single
connection in this mode.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-s</term>
<listitem>
<para>
Use the SR protocol.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-z</term>
<listitem>
<para>
Use the Z39.50 protocol (default). These two options complement
eachother. You can use both multiple times on the same command
line, between listener-specifications (see below). This way, you
can set up the server to listen for connections in both protocols
concurrently, on different local ports.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-l <emphasis remap="it">logfile</emphasis></term>
<listitem>
<para>
Specify an output file for the diagnostic
messages. The default is to write this information to <literal remap="tt">stderr</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-v <emphasis remap="it">log-level</emphasis></term>
<listitem>
<para>
The log level. Use a comma-separated list of members of the set
&lcub;fatal,debug,warn,log,all,none&rcub;.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-u <emphasis remap="it">username</emphasis></term>
<listitem>
<para>
Set user ID. Sets the real UID of the server process to that of the
given <emphasis remap="it">username</emphasis>. It's useful if you aren't comfortable with having the
server run as root, but you need to start it as such to bind a
privileged port.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-w <emphasis remap="it">working-directory</emphasis></term>
<listitem>
<para>
Change working directory.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-i</term>
<listitem>
<para>
Run under the Internet superserver, <literal remap="tt">inetd</literal>. Make
sure you use the logfile option <literal remap="tt">-l</literal> in conjunction with this
mode and specify the <literal remap="tt">-l</literal> option before any other options.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-t <emphasis remap="it">timeout</emphasis></term>
<listitem>
<para>
Set the idle session timeout (default 60 minutes).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-k <emphasis remap="it">kilobytes</emphasis></term>
<listitem>
<para>
Set the (approximate) maximum size of
present response messages. Default is 1024 Kb (1 Mb).
</para>
</listitem>
</varlistentry>
</variablelist>
</para>

<para>
A <emphasis remap="it">listener-address</emphasis> consists of a transport mode followed by a
colon (:) followed by a listener address. The transport mode is
either <literal remap="tt">osi</literal> or <literal remap="tt">tcp</literal>.
</para>

<para>
For TCP, an address has the form
</para>

<para>

<screen>
hostname | IP-number &lsqb;: portnumber&rsqb;
</screen>

</para>

<para>
The port number defaults to 210 (standard Z39.50 port).
</para>

<para>
For OSI (only available if the server is compiled with XTI/mOSI
support enabled), the address form is
</para>

<para>

<screen>
&lsqb;t-selector /&rsqb; hostname | IP-number &lsqb;: portnumber&rsqb;
</screen>

</para>

<para>
The transport selector is given as a string of hex digits (with an even
number of digits). The default port number is 102 (RFC1006 port).
</para>

<para>
Examples
</para>

<para>

<screen>
tcp:dranet.dra.com

osi:0402/dbserver.osiworld.com:3000
</screen>

</para>

<para>
In both cases, the special hostname "@" is mapped to
the address INADDR_ANY, which causes the server to listen on any local
interface. To start the server listening on the registered ports for
Z39.50 and SR over OSI/RFC1006, and to drop root privileges once the
ports are bound, execute the server like this (from a root shell):
</para>

<para>

<screen>
zebrasrv -u daemon tcp:@ -s osi:@
</screen>

</para>

<para>
You can replace <literal remap="tt">daemon</literal> with another user, eg. your own account, or
a dedicated IR server account.
</para>

<para>
The default behavior for <literal remap="tt">zebrasrv</literal> is to establish a single TCP/IP
listener, for the Z39.50 protocol, on port 9999.
</para>

</sect1>

<sect1 id="protocol-support">
<title>Z39.50 Protocol Support and Behavior</title>

<sect2>
<title>Initialization</title>

<para>
During initialization, the server will negotiate to version 3 of the
Z39.50 protocol, and the option bits for Search, Present, Scan,
NamedResultSets, and concurrentOperations will be set, if requested by
the client. The maximum PDU size is negotiated down to a maximum of
1Mb by default.
</para>

</sect2>

<sect2 id="search">
<title>Search</title>

<para>
The supported query type are 1 and 101. All operators are currently
supported with the restriction that only proximity units of type "word" are
supported for the proximity operator.
Queries can be arbitrarily complex.
Named result sets are supported, and result sets can be used as operands
without limitations.
Searches may span multiple databases.
</para>

<para>
The server has full support for piggy-backed present requests (see
also the following section).
</para>

<para>
<emphasis remap="bf">Use</emphasis> attributes are interpreted according to the attribute sets which
have been loaded in the <literal remap="tt">zebra.cfg</literal> file, and are matched against
specific fields as specified in the <literal remap="tt">.abs</literal> file which describes the
profile of the records which have been loaded. If no <emphasis remap="bf">Use</emphasis>
attribute is provided, a default of Bib-1 <emphasis remap="bf">Any</emphasis> is assumed.
</para>

<para>
If a <emphasis remap="bf">Structure</emphasis> attribute of <emphasis remap="bf">Phrase</emphasis> is used in conjunction with a
<emphasis remap="bf">Completeness</emphasis> attribute of <emphasis remap="bf">Complete (Sub)field</emphasis>, the term is
matched against the contents of the phrase (long word) register, if one
exists for the given <emphasis remap="bf">Use</emphasis> attribute.
A phrase register is created for those fields in the <literal remap="tt">.abs</literal>
file that contains a <literal remap="tt">p</literal>-specifier.
</para>

<para>
If <emphasis remap="bf">Structure</emphasis>=<emphasis remap="bf">Phrase</emphasis> is used in conjunction with
<emphasis remap="bf">Incomplete Field</emphasis> - the default value for <emphasis remap="bf">Completeness</emphasis>, the
search is directed against the normal word registers, but if the term
contains multiple words, the term will only match if all of the words
are found immediately adjacent, and in the given order.
The word search is performed on those fields that are indexed as
type <literal remap="tt">w</literal> in the <literal remap="tt">.abs</literal> file.
</para>

<para>
If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">Word List</emphasis>,
<emphasis remap="bf">Free-form Text</emphasis>, or <emphasis remap="bf">Document Text</emphasis>, the term is treated as a
natural-language, relevance-ranked query.
This search type uses the word register, i.e. those fields
that are indexed as type <literal remap="tt">w</literal> in the <literal remap="tt">.abs</literal> file.
</para>

<para>
If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">Numeric String</emphasis> the
term is treated as an integer. The search is performed on those
fields that are indexed as type <literal remap="tt">n</literal> in the <literal remap="tt">.abs</literal> file.
</para>

<para>
If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">URx</emphasis> the
term is treated as a URX (URL) entity. The search is performed on those
fields that are indexed as type <literal remap="tt">u</literal> in the <literal remap="tt">.abs</literal> file.
</para>

<para>
If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">Local Number</emphasis> the
term is treated as native Zebra Record Identifier.
</para>

<para>
If the <emphasis remap="bf">Relation</emphasis> attribute is <emphasis remap="bf">Equals</emphasis> (default), the term is
matched in a normal fashion (modulo truncation and processing of
individual words, if required). If <emphasis remap="bf">Relation</emphasis> is <emphasis remap="bf">Less Than</emphasis>,
<emphasis remap="bf">Less Than or Equal</emphasis>, <emphasis remap="bf">Greater than</emphasis>, or <emphasis remap="bf">Greater than or
Equal</emphasis>, the term is assumed to be numerical, and a standard regular
expression is constructed to match the given expression. If
<emphasis remap="bf">Relation</emphasis> is <emphasis remap="bf">Relevance</emphasis>, the standard natural-language query
processor is invoked.
</para>

<para>
For the <emphasis remap="bf">Truncation</emphasis> attribute, <emphasis remap="bf">No Truncation</emphasis> is the default.
<emphasis remap="bf">Left Truncation</emphasis> is not supported. <emphasis remap="bf">Process &num;</emphasis> is supported, as
is <emphasis remap="bf">Regxp-1</emphasis>. <emphasis remap="bf">Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
search. As a default, a single error (deletion, insertion,
replacement) is accepted when terms are matched against the register
contents.
</para>

<sect3>
<title>Regular expressions</title>

<para>
Each term in a query is interpreted as a regular expression if
the truncation value is either <emphasis remap="bf">Regxp-1</emphasis> (102) or <emphasis remap="bf">Regxp-2</emphasis> (103).
Both query types follow the same syntax with the operands:
<variablelist>

<varlistentry>
<term>x</term>
<listitem>
<para>
Matches the character <emphasis remap="it">x</emphasis>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>.</term>
<listitem>
<para>
Matches any character.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal remap="tt">[</literal>..<literal remap="tt">]</literal></term>
<listitem>
<para>
Matches the set of characters specified;
such as <literal remap="tt">[abc]</literal> or <literal remap="tt">[a-c]</literal>.
</para>
</listitem>
</varlistentry>
</variablelist>
and the operators:
<variablelist>

<varlistentry>
<term>x*</term>
<listitem>
<para>
Matches <emphasis remap="it">x</emphasis> zero or more times. Priority: high.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>x+</term>
<listitem>
<para>
Matches <emphasis remap="it">x</emphasis> one or more times. Priority: high.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>x?</term>
<listitem>
<para>
Matches <emphasis remap="it">x</emphasis> once or twice. Priority: high.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>xy</term>
<listitem>
<para>
Matches <emphasis remap="it">x</emphasis>, then <emphasis remap="it">y</emphasis>. Priority: medium.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>x&verbar;y</term>
<listitem>
<para>
Matches either <emphasis remap="it">x</emphasis> or <emphasis remap="it">y</emphasis>. Priority: low.
</para>
</listitem>
</varlistentry>
</variablelist>
The order of evaluation may be changed by using parentheses.
</para>

<para>
If the first character of the <emphasis remap="bf">Regxp-2</emphasis> query is a plus character
(<literal remap="tt">+</literal>) it marks the beginning of a section with non-standard
specifiers. The next plus character marks the end of the section.
Currently Zebra only supports one specifier, the error tolerance,
which consists one digit. 
</para>

<para>
Since the plus operator is normally a suffix operator the addition to
the query syntax doesn't violate the syntax for standard regular
expressions.
</para>

</sect3>

<sect3>
<title>Query examples</title>

<para>
Phrase search for <emphasis remap="bf">information retrieval</emphasis> in the title-register:

<screen>
 @attr 1=4 "information retrieval"
</screen>

</para>

<para>
Ranked search for the same thing:

<screen>
 @attr 1=4 @attr 2=102 "Information retrieval"
</screen>

</para>

<para>
Phrase search with a regular expression:

<screen>
 @attr 1=4 @attr 5=102 "informat.* retrieval"
</screen>

</para>

<para>
Ranked search with a regular expression:

<screen>
 @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
</screen>

</para>

<para>
In the GILS schema (<literal remap="tt">gils.abs</literal>), the west-bounding-coordinate is
indexed as type <literal remap="tt">n</literal>, and is therefore searched by specifying
<emphasis remap="bf">structure</emphasis>=<emphasis remap="bf">Numeric String</emphasis>.
To match all those records with west-bounding-coordinate greater
than -114 we use the following query:

<screen>
 @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
</screen>

</para>

</sect3>

</sect2>

<sect2>
<title>Present</title>

<para>
The present facility is supported in a standard fashion. The requested
record syntax is matched against the ones supported by the profile of
each record retrieved. If no record syntax is given, SUTRS is the
default. The requested element set name, again, is matched against any
provided by the relevant record profiles.
</para>

</sect2>

<sect2>
<title>Scan</title>

<para>
The attribute combinations provided with the termListAndStartPoint are
processed in the same way as operands in a query (see above).
Currently, only the term and the globalOccurrences are returned with
the termInfo structure.
</para>

</sect2>

<sect2>
<title>Sort</title>

<para>
Z39.50 specifies three diffent types of sort criterias.
Of these Zebra supports the attribute specification type in which
case the use attribute specifies the "Sort register".
Sort registers are created for those fields that are of type "sort" in
the default.idx file. 
The corresponding character mapping file in default.idx specifies the
ordinal of each character used in the actual sort.
</para>

<para>
Z39.50 allows the client to specify sorting on one or more input
result sets and one output result set.
Zebra supports sorting on one result set only which may or may not
be the same as the output result set.
</para>

</sect2>

<sect2>
<title>Close</title>

<para>
If a Close PDU is received, the server will respond with a Close PDU
with reason=FINISHED, no matter which protocol version was negotiated
during initialization. If the protocol version is 3 or more, the
server will generate a Close PDU under certain circumstances,
including a session timeout (60 minutes by default), and certain kinds of
protocol errors. Once a Close PDU has been sent, the protocol
association is considered broken, and the transport connection will be
closed immediately upon receipt of further data, or following a short
timeout.
</para>

</sect2>

</sect1>

</chapter>

<chapter id="record-model">
<title>The Record Model</title>

<para>
The Zebra system is designed to support a wide range of data management
applications. The system can be configured to handle virtually any
kind of structured data. Each record in the system is associated with
a <emphasis remap="it">record schema</emphasis> which lends context to the data elements of the
record. Any number of record schema can coexist in the system.
Although it may be wise to use only a single schema within
one database, the system poses no such restrictions.
</para>

<para>
The record model described in this chapter applies to the fundamental,
structured
record type <literal remap="tt">grs</literal> as introduced in
section <xref linkend="record-types"/>.
</para>

<para>
Records pass through three different states during processing in the
system.
</para>

<para>

<itemizedlist>
<listitem>

<para>
When records are accessed by the system, they are represented
in their local, or native format. This might be SGML or HTML files,
News or Mail archives, MARC records. If the system doesn't already
know how to read the type of data you need to store, you can set up an
input filter by preparing conversion rules based on regular
expressions and possibly augmented by a flexible scripting language (Tcl). The input filter
produces as output an internal representation:

</para>
</listitem>
<listitem>

<para>
When records are processed by the system, they are represented
in a tree-structure, constructed by tagged data elements hanging off a
root node. The tagged elements may contain data or yet more tagged
elements in a recursive structure. The system performs various
actions on this tree structure (indexing, element selection, schema
mapping, etc.),

</para>
</listitem>
<listitem>

<para>
Before transmitting records to the client, they are first
converted from the internal structure to a form suitable for exchange
over the network - according to the Z39.50 standard.
</para>
</listitem>

</itemizedlist>

</para>

<sect1 id="local-representation">
<title>Local Representation</title>

<para>
As mentioned earlier, Zebra places few restrictions on the type of
data that you can index and manage. Generally, whatever the form of
the data, it is parsed by an input filter specific to that format, and
turned into an internal structure that Zebra knows how to handle. This
process takes place whenever the record is accessed - for indexing and
retrieval.
</para>

<para>
The RecordType parameter in the <literal remap="tt">zebra.cfg</literal> file, or the <literal remap="tt">-t</literal>
option to the indexer tells Zebra how to process input records. Two
basic types of processing are available - raw text and structured
data. Raw text is just that, and it is selected by providing the
argument <emphasis remap="bf">text</emphasis> to Zebra. Structured records are all handled
internally using the basic mechanisms described in the subsequent
sections. Zebra can read structured records in many different formats.
How this is done is governed by additional parameters after the
"grs" keyboard, separated by "." characters.
</para>

<para>
Three basic subtypes to the <emphasis remap="bf">grs</emphasis> type are currently available:
</para>

<para>
<variablelist>

<varlistentry>
<term>grs.sgml</term>
<listitem>
<para>
This is the canonical input format &mdash;
described below. It is a simple SGML-like syntax.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>grs.regx.<emphasis remap="it">filter</emphasis></term>
<listitem>
<para>
This enables a user-supplied input
filter. The mechanisms of these filters are described below.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>grs.marc.<emphasis remap="it">abstract syntax</emphasis></term>
<listitem>
<para>
This allows Zebra to read
records in the ISO2709 (MARC) encoding standard. In this case, the
last paramemeter <emphasis remap="it">abstract syntax</emphasis> names the .abs file (see below)
which describes the specific MARC structure of the input record as
well as the indexing rules.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>

<sect2>
<title>Canonical Input Format</title>

<para>
Although input data can take any form, it is sometimes useful to
describe the record processing capabilities of the system in terms of
a single, canonical input format that gives access to the full
spectrum of structure and flexibility in the system. In Zebra, this
canonical format is an "SGML-like" syntax.
</para>

<para>
To use the canonical format specify <literal remap="tt">grs.sgml</literal> as the record
type,
</para>

<para>
Consider a record describing an information resource (such a record is
sometimes known as a <emphasis remap="it">locator record</emphasis>). It might contain a field
describing the distributor of the information resource, which might in
turn be partitioned into various fields providing details about the
distributor, like this:
</para>

<para>

<screen>
&#60;Distributor&#62;
    &#60;Name&#62; USGS/WRD &#60;/Name&#62;
    &#60;Organization&#62; USGS/WRD &#60;/Organization&#62;
    &#60;Street-Address&#62;
        U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
    &#60;/Street-Address&#62;
    &#60;City&#62; ALBUQUERQUE &#60;/City&#62;
    &#60;State&#62; NM &#60;/State&#62;
    &#60;Zip-Code&#62; 87102 &#60;/Zip-Code&#62;
    &#60;Country&#62; USA &#60;/Country&#62;
    &#60;Telephone&#62; (505) 766-5560 &#60;/Telephone&#62;
&#60;/Distributor&#62;
</screen>

</para>

<para>
<emphasis remap="it">NOTE: The indentation used above is used to illustrate how Zebra
interprets the markup. The indentation, in itself, has no
significance to the parser for the canonical input format, which
discards superfluous whitespace.</emphasis>
</para>

<para>
The keywords surrounded by &lt;...&gt; are <emphasis remap="it">tags</emphasis>, while the
sections of text in between are the <emphasis remap="it">data elements</emphasis>. A data element
is characterized by its location in the tree that is made up by the
nested elements. Each element is terminated by a closing tag -
beginning with <literal remap="tt">&#60;</literal>/, and containing the same symbolic tag-name as
the corresponding opening tag. The general closing tag - <literal remap="tt">&#60;</literal>&gt;/ -
terminates the element started by the last opening tag. The
structuring of elements is significant. The element <emphasis remap="bf">Telephone</emphasis>,
for instance, may be indexed and presented to the client differently,
depending on whether it appears inside the <emphasis remap="bf">Distributor</emphasis> element,
or some other, structured data element such a <emphasis remap="bf">Supplier</emphasis> element.
</para>

<sect3>
<title>Record Root</title>

<para>
The first tag in a record describes the root node of the tree that
makes up the total record. In the canonical input format, the root tag
should contain the name of the schema that lends context to the
elements of the record (see section
<xref linkend="internal-representation"/>).
The following is a GILS record that
contains only a single element (strictly speaking, that makes it an
illegal GILS record, since the GILS profile includes several mandatory
elements - Zebra does not validate the contents of a record against
the Z39.50 profile, however - it merely attempts to match up elements
of a local representation with the given schema):
</para>

<para>

<screen>
&#60;gils&#62;
    &#60;title&#62;Zen and the Art of Motorcycle Maintenance&#60;/title&#62;
&#60;/gils&#62;
</screen>

</para>

</sect3>

<sect3>
<title>Variants</title>

<para>
Zebra allows you to provide individual data elements in a number of
<emphasis remap="it">variant forms</emphasis>. Examples of variant forms are textual data
elements which might appear in different languages, and images which
may appear in different formats or layouts. The variant system in
Zebra is
essentially a representation of the variant mechanism of
Z39.50-1995.
</para>

<para>
The following is an example of a title element which occurs in two
different languages.
</para>

<para>

<screen>
&#60;title&#62;
  &#60;var lang lang "eng"&#62;
    Zen and the Art of Motorcycle Maintenance&#60;/&#62;
  &#60;var lang lang "dan"&#62;
    Zen og Kunsten at Vedligeholde en Motorcykel&#60;/&#62;
&#60;/title&#62;
</screen>

</para>

<para>
The syntax of the <emphasis remap="it">variant element</emphasis> is <literal remap="tt">&lt;var class
type value&gt;</literal>. The available values for the <emphasis remap="it">class</emphasis> and
<emphasis remap="it">type</emphasis> fields are given by the variant set that is associated with the
current schema (see section <xref linkend="variant-set"/>).
</para>

<para>
Variant elements are terminated by the general end-tag &#60;/&#62;, by
the variant end-tag &#60;/var&#62;, by the appearance of another variant
tag with the same <emphasis remap="it">class</emphasis> and <emphasis remap="it">value</emphasis> settings, or by the
appearance of another, normal tag. In other words, the end-tags for
the variants used in the example above could have been saved.
</para>

<para>
Variant elements can be nested. The element
</para>

<para>

<screen>
&#60;title&#62;
  &#60;var lang lang "eng"&#62;&#60;var body iana "text/plain"&#62;
    Zen and the Art of Motorcycle Maintenance
&#60;/title&#62;
</screen>

</para>

<para>
Associates two variant components to the variant list for the title
element.
</para>

<para>
Given the nesting rules described above, we could write
</para>

<para>

<screen>
&#60;title&#62;
  &#60;var body iana "text/plain&#62;
    &#60;var lang lang "eng"&#62;
      Zen and the Art of Motorcycle Maintenance
    &#60;var lang lang "dan"&#62;
      Zen og Kunsten at Vedligeholde en Motorcykel
&#60;/title&#62;
</screen>

</para>

<para>
The title element above comes in two variants. Both have the IANA body
type "text/plain", but one is in English, and the other in
Danish. The client, using the element selection mechanism of Z39.50,
can retrieve information about the available variant forms of data
elements, or it can select specific variants based on the requirements
of the end-user.
</para>

</sect3>

</sect2>

<sect2>
<title>Input Filters</title>

<para>
In order to handle general input formats, Zebra allows the
operator to define filters which read individual records in their native format
and produce an internal representation that the system can
work with.
</para>

<para>
Input filters are ASCII files, generally with the suffix <literal remap="tt">.flt</literal>.
The system looks for the files in the directories given in the
<emphasis remap="bf">profilePath</emphasis> setting in the <literal remap="tt">zebra.cfg</literal> files. The record type
for the filter is <literal remap="tt">grs.regx.</literal><emphasis remap="it">filter-filename</emphasis>
(fundamental type <literal remap="tt">grs</literal>, file read type <literal remap="tt">regx</literal>, argument
<emphasis remap="it">filter-filename</emphasis>).
</para>

<para>
Generally, an input filter consists of a sequence of rules, where each
rule consists of a sequence of expressions, followed by an action. The
expressions are evaluated against the contents of the input record,
and the actions normally contribute to the generation of an internal
representation of the record.
</para>

<para>
An expression can be either of the following:
</para>

<para>
<variablelist>

<varlistentry>
<term>INIT</term>
<listitem>
<para>
The action associated with this expression is evaluated
exactly once in the lifetime of the application, before any records
are read. It can be used in conjunction with an action that
initializes tables or other resources that are used in the processing
of input records.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>BEGIN</term>
<listitem>
<para>
Matches the beginning of the record. It can be used to
initialize variables, etc. Typically, the <emphasis remap="bf">BEGIN</emphasis> rule is also used
to establish the root node of the record.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>END</term>
<listitem>
<para>
Matches the end of the record - when all of the contents
of the record has been processed.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>/pattern/</term>
<listitem>
<para>
Matches a string of characters from the input
record.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>BODY</term>
<listitem>
<para>
This keyword may only be used between two patterns. It
matches everything between (not including) those patterns.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>FINISH</term>
<listitem>
<para>
The expression asssociated with this pattern is evaluated
once, before the application terminates. It can be used to release
system resources - typically ones allocated in the <emphasis remap="bf">INIT</emphasis> step.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>

<para>
An action is surrounded by curly braces (&lcub;...&rcub;), and consists of a
sequence of statements. Statements may be separated by newlines or
semicolons (;). Within actions, the strings that matched the
expressions immediately preceding the action can be referred to as
&dollar;0, &dollar;1, &dollar;2, etc.
</para>

<para>
The available statements are:
</para>

<para>
<variablelist>

<varlistentry>
<term>begin <emphasis remap="it">type &lsqb;parameter ... &rsqb;</emphasis></term>
<listitem>
<para>
Begin a new
data element. The type is one of the following:
<variablelist>

<varlistentry>
<term>record</term>
<listitem>
<para>
Begin a new record. The followingparameter should be the
name of the schema that describes the structure of the record, eg.
<literal remap="tt">gils</literal> or <literal remap="tt">wais</literal> (see below). The <literal remap="tt">begin record</literal> call should
precede
any other use of the <emphasis remap="bf">begin</emphasis> statement.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>element</term>
<listitem>
<para>
Begin a new tagged element. The parameter is the
name of the tag. If the tag is not matched anywhere in the tagsets
referenced by the current schema, it is treated as a local string
tag.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>variant</term>
<listitem>
<para>
Begin a new node in a variant tree. The parameters are
<emphasis remap="it">class type value</emphasis>.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>data</term>
<listitem>
<para>
Create a data element. The concatenated arguments make
up the value of the data element. The option <literal remap="tt">-text</literal> signals that
the layout (whitespace) of the data should be retained for
transmission. The option <literal remap="tt">-element</literal> <emphasis remap="it">tag</emphasis> wraps the data up in
the <emphasis remap="it">tag</emphasis>. The use of the <literal remap="tt">-element</literal> option is equivalent to
preceding the command with a <emphasis remap="bf">begin element</emphasis> command, and following
it with the <emphasis remap="bf">end</emphasis> command.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>end <emphasis remap="it">&lsqb;type&rsqb;</emphasis></term>
<listitem>
<para>
Close a tagged element. If no parameter is given,
the last element on the stack is terminated. The first parameter, if
any, is a type name, similar to the <emphasis remap="bf">begin</emphasis> statement. For the
<emphasis remap="bf">element</emphasis> type, a tag name can be provided to terminate a specific tag.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>

<para>
The following input filter reads a Usenet news file, producing a
record in the WAIS schema. Note that the body of a news posting is
separated from the list of headers by a blank line (or rather a
sequence of two newline characters.
</para>

<para>

<screen>
BEGIN                { begin record wais }

/^From:/ BODY /$/    { data -element name $1 }
/^Subject:/ BODY /$/ { data -element title $1 }
/^Date:/ BODY /$/    { data -element lastModified $1 }
/\n\n/ BODY END      {
                        begin element bodyOfDisplay
                        begin variant body iana "text/plain"
                        data -text $1
                        end record
                     }
</screen>

</para>

<para>
If Zebra is compiled with support for Tcl (Tool Command Language)
enabled, the statements described above are supplemented with a complete
scripting environment, including control structures (conditional
expressions and loop constructs), and powerful string manipulation
mechanisms for modifying the elements of a record. Tcl is a popular
scripting environment, with several tutorials available both online
and in hardcopy.
</para>

<para>
<emphasis remap="it">NOTE: Tcl support is not currently available, but will be
included with one of the next alpha or beta releases.</emphasis>
</para>

<para>
<emphasis remap="it">NOTE: Variant support is not currently available in the input
filter, but will be included with one of the next alpha or beta
releases.</emphasis>
</para>

</sect2>

</sect1>

<sect1 id="internal-representation">
<title>Internal Representation</title>

<para>
When records are manipulated by the system, they're represented in a
tree-structure, with data elements at the leaf nodes, and tags or
variant components at the non-leaf nodes. The root-node identifies the
schema that lends context to the tagging and structuring of the
record. Imagine a simple record, consisting of a 'title' element and
an 'author' element:
</para>

<para>

<screen>
        TITLE     "Zen and the Art of Motorcycle Maintenance"
ROOT 
        AUTHOR    "Robert Pirsig"
</screen>

</para>

<para>
A slightly more complex record would have the author element consist
of two elements, a surname and a first name:
</para>

<para>

<screen>
        TITLE     "Zen and the Art of Motorcycle Maintenance"
ROOT  
                  FIRST-NAME "Robert"
        AUTHOR
                  SURNAME    "Pirsig"
</screen>

</para>

<para>
The root of the record will refer to the record schema that describes
the structuring of this particular record. The schema defines the
element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
well as the structuring (SURNAME should appear below AUTHOR, etc.). In
addition, the schema establishes element set names that are used by
the client to request a subset of the elements of a given record. The
schema may also establish rules for converting the record to a
different schema, by stating, for each element, a mapping to a
different tag path.
</para>

<sect2>
<title>Tagged Elements</title>

<para>
A data element is characterized by its tag, and its position in the
structure of the record. For instance, while the tag "telephone
number" may be used different places in a record, we may need to
distinguish between these occurrences, both for searching and
presentation purposes. For instance, while the phone numbers for the
"customer" and the "service provider" are both
representatives for the same type of resource (a telephone number), it
is essential that they be kept separate. The record schema provides
the structure of the record, and names each data element (defined by
the sequence of tags - the tag path - by which the element can be
reached from the root of the record).
</para>

</sect2>

<sect2>
<title>Variants</title>

<para>
The children of a tag node may be either more tag nodes, a data node
(possibly accompanied by tag nodes),
or a tree of variant nodes. The children of  variant nodes are either
more variant nodes or a data node (possibly accompanied by more
variant nodes). Each leaf node, which is normally a
data node, corresponds to a <emphasis remap="it">variant form</emphasis> of the tagged element
identified by the tag which parents the variant tree. The following
title element occurs in two different languages:
</para>

<para>

<screen>
      VARIANT LANG=ENG  "War and Peace"
TITLE
      VARIANT LANG=DAN  "Krig og Fred"
</screen>

</para>

<para>
Which of the two elements are transmitted to the client by the server
depends on the specifications provided by the client, if any.
</para>

<para>
In practice, each variant node is associated with a triple of class,
type, value, corresponding to the variant mechanism of Z39.50.
</para>

</sect2>

<sect2>
<title>Data Elements</title>

<para>
Data nodes have no children (they are always leaf nodes in the record
tree).
</para>

<para>
<emphasis remap="it">NOTE: Documentation needs extension here about types of nodes - numerical,
textual, etc., plus the various types of inclusion notes.</emphasis>
</para>

</sect2>

</sect1>

<sect1 id="data-model">
<title>Configuring Your Data Model</title>

<para>
The following sections describe the configuration files that govern
the internal management of data records. The system searches for the files
in the directories specified by the <emphasis remap="bf">profilePath</emphasis> setting in the
<literal remap="tt">zebra.cfg</literal> file.
</para>

<sect2>
<title>The Abstract Syntax</title>

<para>
The abstract syntax definition (also known as an Abstract Record
Structure, or ARS) is the focal point of the
record schema description. For a given schema, the ABS file may state any
or all of the following:
</para>

<para>

<itemizedlist>
<listitem>

<para>
The object identifier of the Z39.50 schema associated
with the ARS, so that it can be referred to by the client.

</para>
</listitem>
<listitem>

<para>
The attribute set (which can possibly be a compound of multiple
sets) which applies in the profile. This is used when indexing and
searching the records belonging to the given profile.

</para>
</listitem>
<listitem>

<para>
The Tag set (again, this can consist of several different sets).
This is used when reading the records from a file, to recognize the
different tags, and when transmitting the record to the client -
mapping the tags to their numerical representation, if they are
known.

</para>
</listitem>
<listitem>

<para>
The variant set which is used in the profile. This provides a
vocabulary for specifying the <emphasis remap="it">forms</emphasis> of data that appear inside
the records.

</para>
</listitem>
<listitem>

<para>
Element set names, which are a shorthand way for the client to
ask for a subset of the data elements contained in a record. Element
set names, in the retrieval module, are mapped to <emphasis remap="it">element
specifications</emphasis>, which contain information equivalent to the
<emphasis remap="it">Espec-1</emphasis> syntax of Z39.50.

</para>
</listitem>
<listitem>

<para>
Map tables, which may specify mappings to <emphasis remap="it">other</emphasis> database
profiles, if desired.

</para>
</listitem>
<listitem>

<para>
Possibly, a set of rules describing the mapping of elements to a
MARC representation.

</para>
</listitem>
<listitem>

<para>
A list of element descriptions (this is the actual ARS of the
schema, in Z39.50 terms), which lists the ways in which the various
tags can be used and organized hierarchically.
</para>
</listitem>

</itemizedlist>

</para>

<para>
Several of the entries above simply refer to other files, which
describe the given objects.
</para>

</sect2>

<sect2>
<title>The Configuration Files</title>

<para>
This section describes the syntax and use of the various tables which
are used by the retrieval module.
</para>

<para>
The number of different file types may appear daunting at first, but
each type corresponds fairly clearly to a single aspect of the Z39.50
retrieval facilities. Further, the average database administrator,
who is simply reusing an existing profile for which tables already
exist, shouldn't have to worry too much about the contents of these tables.
</para>

<para>
Generally, the files are simple ASCII files, which can be maintained
using any text editor. Blank lines, and lines beginning with a (&num;) are
ignored. Any characters on a line followed by a (&num;) are also ignored.
All other
lines contain <emphasis remap="it">directives</emphasis>, which provide some setting or value
to the system. Generally, settings are characterized by a single
keyword, identifying the setting, followed by a number of parameters.
Some settings are repeatable (r), while others may occur only once in a
file. Some settings are optional (o), whicle others again are
mandatory (m).
</para>

</sect2>

<sect2>
<title>The Abstract Syntax (.abs) Files</title>

<para>
The name of this file type is slightly misleading in Z39.50 terms,
since, apart from the actual abstract syntax of the profile, it also
includes most of the other definitions that go into a database
profile.
</para>

<para>
When a record in the canonical, SGML-like format is read from a file
or from the database, the first tag of the file should reference the
profile that governs the layout of the record. If the first tag of the
record is, say, <literal remap="tt">&lt;gils&gt;</literal>, the system will look for the profile
definition in the file <literal remap="tt">gils.abs</literal>. Profile definitions are cached,
so they only have to be read once during the lifespan of the current
process. 
</para>

<para>
When writing your own input filters, the <emphasis remap="bf">record-begin</emphasis> command
introduces the profile, and should always be called first thing when
introducing a new record.
</para>

<para>
The file may contain the following directives:
</para>

<para>
<variablelist>

<varlistentry>
<term>name <emphasis remap="it">symbolic-name</emphasis></term>
<listitem>
<para>
(m) This provides a shorthand name or
description for the profile. Mostly useful for diagnostic purposes.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>reference <emphasis remap="it">OID-name</emphasis></term>
<listitem>
<para>
(m) The reference name of the OID for
the profile. The reference names can be found in the <emphasis remap="bf">util</emphasis>
module of <emphasis remap="bf">YAZ</emphasis>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>attset <emphasis remap="it">filename</emphasis></term>
<listitem>
<para>
(m) The attribute set that is used for
indexing and searching records belonging to this profile.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>tagset <emphasis remap="it">filename</emphasis></term>
<listitem>
<para>
(o) The tag set (if any) that describe
that fields of the records.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>varset <emphasis remap="it">filename</emphasis></term>
<listitem>
<para>
(o) The variant set used in the profile.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>maptab <emphasis remap="it">filename</emphasis></term>
<listitem>
<para>
(o,r) This points to a
conversion table that might be used if the client asks for the record
in a different schema from the native one.
</para>
</listitem></varlistentry>
<varlistentry>
<term>marc <emphasis remap="it">filename</emphasis></term>
<listitem>
<para>
(o) Points to a file containing parameters
for representing the record contents in the ISO2709 syntax. Read the
description of the MARC representation facility below.
</para>
</listitem></varlistentry>
<varlistentry>
<term>esetname <emphasis remap="it">name filename</emphasis></term>
<listitem>
<para>
(o,r) Associates the
given element set name with an element selection file. If an (@) is
given in place of the filename, this corresponds to a null mapping for
the given element set name.
</para>
</listitem></varlistentry>
<varlistentry>
<term>any <emphasis remap="it">tags</emphasis></term>
<listitem>
<para>
(o) This directive specifies a list of
attributes which should be appended to the attribute list given for each
element. The effect is to make every single element in the abstract
syntax searchable by way of the given attributes. This directive
provides an efficient way of supporting free-text searching across all
elements. However, it does increase the size of the index
significantly. The attributes can be qualified with a structure, as in
the <emphasis remap="bf">elm</emphasis> directive below.
</para>
</listitem></varlistentry>
<varlistentry>
<term>elm <emphasis remap="it">path name attributes</emphasis></term>
<listitem>
<para>
(o,r) Adds an element
to the abstract record syntax of the schema. The <emphasis remap="it">path</emphasis> follows the
syntax which is suggested by the Z39.50 document - that is, a sequence
of tags separated by slashes (/). Each tag is given as a
comma-separated pair of tag type and -value surrounded by parenthesis.
The <emphasis remap="it">name</emphasis> is the name of the element, and the <emphasis remap="it">attributes</emphasis>
specifies which attributes to use when indexing the element in a
comma-separated list. A ! in
place of the attribute name is equivalent to specifying an attribute
name identical to the element name. A - in place of the attribute name
specifies that no indexing is to take place for the given element. The
attributes can be qualified with <emphasis remap="it">field types</emphasis> to specify which
character set should govern the indexing procedure for that field. The
same data element may be indexed into several different fields, using
different character set definitions. See the section
<xref linkend="field-structure-and-character-sets"/>.
The default field type is "w" for
<emphasis remap="it">word</emphasis>.
</para>
</listitem></varlistentry>
</variablelist>
</para>

<para>
<emphasis remap="it">NOTE: The mechanism for controlling indexing is not adequate for
complex databases, and will probably be moved into a separate
configuration table eventually.</emphasis>
</para>

<para>
The following is an excerpt from the abstract syntax file for the GILS
profile.
</para>

<para>

<screen>
name gils
reference GILS-schema
attset gils.att
tagset gils.tag
varset var1.var

maptab gils-usmarc.map

# Element set names

esetname VARIANT gils-variant.est  # for WAIS-compliance
esetname B gils-b.est
esetname G gils-g.est
esetname F @

elm (1,10)              rank                        -
elm (1,12)              url                         -
elm (1,14)              localControlNumber     Local-number
elm (1,16)              dateOfLastModification Date/time-last-modified
elm (2,1)               title                       w:!,p:!
elm (4,1)               controlIdentifier      Identifier-standard
elm (2,6)               abstract               Abstract
elm (4,51)              purpose                     !
elm (4,52)              originator                  - 
elm (4,53)              accessConstraints           !
elm (4,54)              useConstraints              !
elm (4,70)              availability                -
elm (4,70)/(4,90)       distributor                 -
elm (4,70)/(4,90)/(2,7) distributorName             !
elm (4,70)/(4,90)/(2,10 distributorOrganization     !
elm (4,70)/(4,90)/(4,2) distributorStreetAddress    !
elm (4,70)/(4,90)/(4,3) distributorCity             !
</screen>

</para>

</sect2>

<sect2 id="attset-files">
<title>The Attribute Set (.att) Files</title>

<para>
This file type describes the <emphasis remap="bf">Use</emphasis> elements of an attribute set.
It contains the following directives. 
</para>

<para>
<variablelist>

<varlistentry>
<term>name <emphasis remap="it">symbolic-name</emphasis></term>
<listitem>
<para>
(m) This provides a shorthand name or
description for the attribute set. Mostly useful for diagnostic purposes.
</para>
</listitem></varlistentry>
<varlistentry>
<term>reference <emphasis remap="it">OID-name</emphasis></term>
<listitem>
<para>
(m) The reference name of the OID for
the attribute set. The reference names can be found in the <emphasis remap="bf">util</emphasis>
module of <emphasis remap="bf">YAZ</emphasis>.
</para>
</listitem></varlistentry>
<varlistentry>
<term>ordinal <emphasis remap="it">integer</emphasis></term>
<listitem>
<para>
(m) This value will be used to represent the
attribute set in the index. Care should be taken that each attribute
set has a unique ordinal value.
</para>
</listitem></varlistentry>
<varlistentry>
<term>include <emphasis remap="it">filename</emphasis></term>
<listitem>
<para>
(o,r) This directive is used to
include another attribute set as a part of the current one. This is
used when a new attribute set is defined as an extension to another
set. For instance, many new attribute sets are defined as extensions
to the <emphasis remap="bf">bib-1</emphasis> set. This is an important feature of the retrieval
system of Z39.50, as it ensures the highest possible level of
interoperability, as those access points of your database which are
derived from the external set (say, bib-1) can be used even by clients
who are unaware of the new set.
</para>
</listitem></varlistentry>
<varlistentry>
<term>att <emphasis remap="it">att-value att-name &lsqb;local-value&rsqb;</emphasis></term>
<listitem>
<para>
(o,r) This
repeatable directive introduces a new attribute to the set. The
attribute value is stored in the index (unless a <emphasis remap="it">local-value</emphasis> is
given, in which case this is stored). The name is used to refer to the
attribute from the <emphasis remap="it">abstract syntax</emphasis>. 
</para>
</listitem></varlistentry>
</variablelist>
</para>

<para>
This is an excerpt from the GILS attribute set definition. Notice how
the file describing the <emphasis remap="it">bib-1</emphasis> attribute set is referenced.
</para>

<para>

<screen>
name gils
reference GILS-attset
include bib1.att
ordinal 2

att 2001                distributorName
att 2002                indextermsControlled
att 2003                purpose
att 2004                accessConstraints
att 2005                useConstraints
</screen>

</para>

</sect2>

<sect2>
<title>The Tag Set (.tag) Files</title>

<para>
This file type defines the tagset of the profile, possibly by
referencing other tag sets (most tag sets, for instance, will include
tagsetG and tagsetM from the Z39.50 specification. The file may
contain the following directives.
</para>

<para>
<variablelist>

<varlistentry>
<term>name <emphasis remap="it">symbolic-name</emphasis></term>
<listitem>
<para>
(m) This provides a shorthand name or
description for the tag set. Mostly useful for diagnostic purposes.
</para>
</listitem></varlistentry>
<varlistentry>
<term>reference <emphasis remap="it">OID-name</emphasis></term>
<listitem>
<para>
(o) The reference name of the OID for
the tag set. The reference names can be found in the <emphasis remap="bf">util</emphasis>
module of <emphasis remap="bf">YAZ</emphasis>. The directive is optional, since not all tag sets
are registered outside of their schema.
</para>
</listitem></varlistentry>
<varlistentry>
<term>type <emphasis remap="it">integer</emphasis></term>
<listitem>
<para>
(m) The type number of the tagset within the schema
profile (note: this specification really should belong to the .abs
file. This will be fixed in a future release).
</para>
</listitem></varlistentry>
<varlistentry>
<term>include <emphasis remap="it">filename</emphasis></term>
<listitem>
<para>
(o,r) This directive is used
to include the definitions of other tag sets into the current one.
</para>
</listitem></varlistentry>
<varlistentry>
<term>tag <emphasis remap="it">number names type</emphasis></term>
<listitem>
<para>
(o,r) Introduces a new
tag to the set. The <emphasis remap="it">number</emphasis> is the tag number as used in the protocol
(there is currently no mechanism for specifying string tags at this
point, but this would be quick work to add). The <emphasis remap="it">names</emphasis> parameter
is a list of names by which the tag should be recognized in the input
file format. The names should be separated by slashes (/). The
<emphasis remap="it">type</emphasis> is th recommended datatype of the tag. It should be one of
the following:

<itemizedlist>
<listitem>

<para>
structured
</para>
</listitem>
<listitem>

<para>
string
</para>
</listitem>
<listitem>

<para>
numeric
</para>
</listitem>
<listitem>

<para>
bool
</para>
</listitem>
<listitem>

<para>
oid
</para>
</listitem>
<listitem>

<para>
generalizedtime
</para>
</listitem>
<listitem>

<para>
intunit
</para>
</listitem>
<listitem>

<para>
int
</para>
</listitem>
<listitem>

<para>
octetstring
</para>
</listitem>
<listitem>

<para>
null
</para>
</listitem>

</itemizedlist>

</para>
</listitem></varlistentry>
</variablelist>
</para>

<para>
The following is an excerpt from the TagsetG definition file.
</para>

<para>

<screen>
name tagsetg
reference TagsetG
type 2

tag     1       title           string
tag     2       author          string
tag     3       publicationPlace string
tag     4       publicationDate string
tag     5       documentId      string
tag     6       abstract        string
tag     7       name            string
tag     8       date            generalizedtime
tag     9       bodyOfDisplay   string
tag     10      organization    string
</screen>

</para>

</sect2>

<sect2 id="variant-set">
<title>The Variant Set (.var) Files</title>

<para>
The variant set file is a straightforward representation of the
variant set definitions associated with the protocol. At present, only
the <emphasis remap="it">Variant-1</emphasis> set is known.
</para>

<para>
These are the directives allowed in the file.
</para>

<para>
<variablelist>

<varlistentry>
<term>name <emphasis remap="it">symbolic-name</emphasis></term>
<listitem>
<para>
(m) This provides a shorthand name or
description for the variant set. Mostly useful for diagnostic purposes.
</para>
</listitem></varlistentry>
<varlistentry>
<term>reference <emphasis remap="it">OID-name</emphasis></term>
<listitem>
<para>
(o) The reference name of the OID for
the variant set, if one is required. The reference names can be found
in the <emphasis remap="bf">util</emphasis> module of <emphasis remap="bf">YAZ</emphasis>.
</para>
</listitem></varlistentry>
<varlistentry>
<term>class <emphasis remap="it">integer class-name</emphasis></term>
<listitem>
<para>
(m,r) Introduces a new
class to the variant set.
</para>
</listitem></varlistentry>
<varlistentry>
<term>type <emphasis remap="it">integer type-name datatype</emphasis></term>
<listitem>
<para>
(m,r) Addes a
new type to the current class (the one introduced by the most recent
<emphasis remap="bf">class</emphasis> directive). The type names belong to the same name space as
the one used in the tag set definition file.
</para>
</listitem></varlistentry>
</variablelist>
</para>

<para>
The following is an excerpt from the file describing the variant set
<emphasis remap="it">Variant-1</emphasis>.
</para>

<para>

<screen>
name variant-1
reference Variant-1

class 1 variantId

  type  1       variantId               octetstring

class 2 body

  type  1       iana                    string
  type  2       z39.50                  string
  type  3       other                   string
</screen>

</para>

</sect2>

<sect2>
<title>The Element Set (.est) Files</title>

<para>
The element set specification files describe a selection of a subset
of the elements of a database record. The element selection mechanism
is equivalent to the one supplied by the <emphasis remap="it">Espec-1</emphasis> syntax of the
Z39.50 specification. In fact, the internal representation of an
element set specification is identical to the <emphasis remap="it">Espec-1</emphasis> structure,
and we'll refer you to the description of that structure for most of
the detailed semantics of the directives below.
</para>

<para>
<emphasis remap="it">NOTE: Not all of the Espec-1 functionality has been implemented yet.
The fields that are mentioned below all work as expected, unless
otherwise is noted.</emphasis>
</para>

<para>
The directives available in the element set file are as follows:
</para>

<para>
<variablelist>

<varlistentry>
<term>defaultVariantSetId <emphasis remap="it">OID-name</emphasis></term>
<listitem>
<para>
(o) If variants are used in
the following, this should provide the name of the variantset used
(it's not currently possible to specify a different set in the
individual variant request). In almost all cases (certainly all
profiles known to us), the name <literal remap="tt">Variant-1</literal> should be given here.
</para>
</listitem></varlistentry>
<varlistentry>
<term>defaultVariantRequest <emphasis remap="it">variant-request</emphasis></term>
<listitem>
<para>
(o) This directive
provides a default variant request for
use when the individual element requests (see below) do not contain a
variant request. Variant requests consist of a blank-separated list of
variant components. A variant compont is a comma-separated,
parenthesized triple of variant class, type, and value (the two former
values being represented as integers). The value can currently only be
entered as a string (this will change to depend on the definition of
the variant in question). The special value (@) is interpreted as a
null value, however.
</para>
</listitem></varlistentry>
<varlistentry>
<term>simpleElement <emphasis remap="it">path &lsqb;'variant' variant-request&rsqb;</emphasis></term>
<listitem>
<para>
(o,r) This corresponds to a simple element request in <emphasis remap="it">Espec-1</emphasis>. The
path consists of a sequence of tag-selectors, where each of these can
consist of either:
</para>

<para>

<itemizedlist>
<listitem>

<para>
A simple tag, consisting of a comma-separated type-value pair in
parenthesis, possibly followed by a colon (:) followed by an
occurrences-specification (see below). The tag-value can be a number
or a string. If the first character is an apostrophe ('), this forces
the value to be interpreted as a string, even if it appears to be numerical.

</para>
</listitem>
<listitem>

<para>
A WildThing, represented as a question mark (?), possibly
followed by a colon (:) followed by an occurrences specification (see
below).

</para>
</listitem>
<listitem>

<para>
A WildPath, represented as an asterisk (*). Note that the last
element of the path should not be a wildPath (wildpaths don't work in
this version).
</para>
</listitem>

</itemizedlist>

</para>

<para>
The occurrences-specification can be either the string <literal remap="tt">all</literal>, the
string <literal remap="tt">last</literal>, or an explicit value-range. The value-range is
represented as an integer (the starting point), possibly followed by a
plus (+) and a second integer (the number of elements, default being
one).
</para>

<para>
The variant-request has the same syntax as the defaultVariantRequest
above. Note that it may sometimes be useful to give an empty variant
request, simply to disable the default for a specific set of fields
(we aren't certain if this is proper <emphasis remap="it">Espec-1</emphasis>, but it works in
this implementation).
</para>
</listitem></varlistentry>
</variablelist>
</para>

<para>
The following is an example of an element specification belonging to
the GILS profile.
</para>

<para>

<screen>
simpleelement (1,10)
simpleelement (1,12)
simpleelement (2,1)
simpleelement (1,14)
simpleelement (4,1)
simpleelement (4,52)
</screen>

</para>

</sect2>

<sect2 id="schema-mapping">
<title>The Schema Mapping (.map) Files</title>

<para>
Sometimes, the client might want to receive a database record in
a schema that differs from the native schema of the record. For
instance, a client might only know how to process WAIS records, while
the database record is represented in a more specific schema, such as
GILS. In this module, a mapping of data to one of the MARC formats is
also thought of as a schema mapping (mapping the elements of the
record into fields consistent with the given MARC specification, prior
to actually converting the data to the ISO2709). This use of the
object identifier for USMARC as a schema identifier represents an
overloading of the OID which might not be entirely proper. However,
it represents the dual role of schema and record syntax which
is assumed by the MARC family in Z39.50.
</para>

<para>
<emphasis remap="it">NOTE: The schema-mapping functions are so far limited to a
straightforward mapping of elements. This should be extended with
mechanisms for conversions of the element contents, and conditional
mappings of elements based on the record contents.</emphasis>
</para>

<para>
These are the directives of the schema mapping file format:
</para>

<para>
<variablelist>

<varlistentry>
<term>targetName <emphasis remap="it">name</emphasis></term>
<listitem>
<para>
(m) A symbolic name for the target schema
of the table. Useful mostly for diagnostic purposes.
</para>
</listitem></varlistentry>
<varlistentry>
<term>targetRef <emphasis remap="it">OID-name</emphasis></term>
<listitem>
<para>
(m) An OID name for the target schema.
This is used, for instance, by a server receiving a request to present
a record in a different schema from the native one. The name, again,
is found in the <emphasis remap="bf">oid</emphasis> module of <emphasis remap="bf">YAZ</emphasis>.
</para>
</listitem></varlistentry>
<varlistentry>
<term>map <emphasis remap="it">element-name target-path</emphasis></term>
<listitem>
<para>
(o,r) Adds
an element mapping rule to the table.
</para>
</listitem></varlistentry>
</variablelist>
</para>

</sect2>

<sect2>
<title>The MARC (ISO2709) Representation (.mar) Files</title>

<para>
This file provides rules for representing a record in the ISO2709
format. The rules pertain mostly to the values of the constant-length
header of the record.
</para>

<para>
<emphasis remap="it">NOTE: This will be described better. We're in the process of
re-evaluating and most likely changing the way that MARC records are
handled by the system.</emphasis>
</para>

</sect2>

<sect2 id="field-structure-and-character-sets">
<title>Field Structure and Character Sets
</title>

<para>
In order to provide a flexible approach to national character set
handling, Zebra allows the administrator to configure the set up the
system to handle any 8-bit character set &mdash; including sets that
require multi-octet diacritics or other multi-octet characters. The
definition of a character set includes a specification of the
permissible values, their sort order (this affects the display in the
SCAN function), and relationships between upper- and lowercase
characters. Finally, the definition includes the specification of
space characters for the set.
</para>

<para>
The operator can define different character sets for different fields,
typical examples being standard text fields, numerical fields, and
special-purpose fields such as WWW-style linkages (URx).
</para>

<para>
The field types, and hence character sets, are associated with data
elements by the .abs files (see above). The file <literal remap="tt">default.idx</literal>
provides the association between field type codes (as used in the .abs
files) and the character map files (with the .chr suffix). The format
of the .idx file is as follows
</para>

<para>
<variablelist>

<varlistentry>
<term>index <emphasis remap="it">field type code</emphasis></term>
<listitem>
<para>
This directive introduces a new
search index code. The argument is a one-character code to be used in the
.abs files to select this particular index type. An index, roughly,
corresponds to a particular structure attribute during search. Refer
to section <xref linkend="search"/>.
</para>
</listitem></varlistentry>
<varlistentry>
<term>sort <emphasis remap="it">field code type</emphasis></term>
<listitem>
<para>
This directive introduces a 
sort index. The argument is a one-character code to be used in the
.abs fie to select this particular index type. The corresponding
use attribute must be used in the sort request to refer to this
particular sort index. The corresponding character map (see below)
is used in the sort process.
</para>
</listitem></varlistentry>
<varlistentry>
<term>completeness <emphasis remap="it">boolean</emphasis></term>
<listitem>
<para>
This directive enables or disables
complete field indexing. The value of the <emphasis remap="it">boolean</emphasis> should be 0
(disable) or 1. If completeness is enabled, the index entry will
contain the complete contents of the field (up to a limit), with words
(non-space characters) separated by single space characters
(normalized to " " on display). When completeness is
disabled, each word is indexed as a separate entry. Complete subfield
indexing is most useful for fields which are typically browsed (eg.
titles, authors, or subjects), or instances where a match on a
complete subfield is essential (eg. exact title searching). For fields
where completeness is disabled, the search engine will interpret a
search containing space characters as a word proximity search.
</para>
</listitem></varlistentry>
<varlistentry>
<term>charmap <emphasis remap="it">filename</emphasis></term>
<listitem>
<para>
This is the filename of the character
map to be used for this index for field type.
</para>
</listitem></varlistentry>
</variablelist>
</para>

<para>
The contents of the character map files are structured as follows:
</para>

<para>
<variablelist>

<varlistentry>
<term>lowercase <emphasis remap="it">value-set</emphasis></term>
<listitem>
<para>
This directive introduces the basic
value set of the field type. The format is an ordered list (without
spaces) of the characters which may occur in "words" of
the given type. The order of the entries in the list determines the
sort order of the index. In addition to single characters, the
following combinations are legal:
</para>

<para>

<itemizedlist>
<listitem>

<para>
Backslashes may be used to introduce three-digit octal, or
two-digit hex representations of single characters (preceded by <literal remap="tt">x</literal>).
In addition, the combinations
\\, \\r, \\n, \\t, \\s (space &mdash; remember that real space-characters
may ot occur in the value definition), and \\ are recognised,
with their usual interpretation.

</para>
</listitem>
<listitem>

<para>
Curly braces &lcub;&rcub; may be used to enclose ranges of single
characters (possibly using the escape convention described in the
preceding point), eg. &lcub;a-z&rcub; to entroduce the standard range of ASCII
characters. Note that the interpretation of such a range depends on
the concrete representation in your local, physical character set.

</para>
</listitem>
<listitem>

<para>
paranthesises () may be used to enclose multi-byte characters -
eg. diacritics or special national combinations (eg. Spanish
"ll"). When found in the input stream (or a search term),
these characters are viewed and sorted as a single character, with a
sorting value depending on the position of the group in the value
statement.
</para>
</listitem>

</itemizedlist>

</para>
</listitem></varlistentry>
<varlistentry>
<term>uppercase <emphasis remap="it">value-set</emphasis></term>
<listitem>
<para>
This directive introduces the
upper-case equivalencis to the value set (if any). The number and
order of the entries in the list should be the same as in the
<literal remap="tt">lowercase</literal> directive.
</para>
</listitem></varlistentry>
<varlistentry>
<term>space <emphasis remap="it">value-set</emphasis></term>
<listitem>
<para>
This directive introduces the character
which separate words in the input stream. Depending on the
completeness mode of the field in question, these characters either
terminate an index entry, or delimit individual "words" in
the input stream. The order of the elements is not significant &mdash;
otherwise the representation is the same as for the <literal remap="tt">upercase</literal> and
<literal remap="tt">lowercase</literal> directives.
</para>
</listitem></varlistentry>
<varlistentry>
<term>map <emphasis remap="it">value-set</emphasis> <emphasis remap="it">target</emphasis></term>
<listitem>
<para>
This directive introduces a
mapping between each of the members of the value-set on the left to
the character on the right. The character on the right must occur in
the value set (the <literal remap="tt">lowercase</literal> directive) of the character set, but
it may be a paranthesis-enclosed multi-octet character. This directive
may be used to map diacritics to their base characters, or to map
HTML-style character-representations to their natural form, etc.
</para>
</listitem></varlistentry>
</variablelist>
</para>

</sect2>

</sect1>

<sect1 id="formats">
<title>Exchange Formats</title>

<para>
Converting records from the internal structure to en exchange format
is largely an automatic process. Currently, the following exchange
formats are supported:
</para>

<para>

<itemizedlist>
<listitem>

<para>
GRS-1. The internal representation is based on GRS-1, so the
conversion here is straightforward. The system will create
applied variant and supported variant lists as required, if a record
contains variant information.

</para>
</listitem>
<listitem>

<para>
SUTRS. Again, the mapping is fairly straighforward. Indentation
is used to show the hierarchical structure of the record. All
"GRS" type records support both the GRS-1 and SUTRS
representations.

</para>
</listitem>
<listitem>

<para>
ISO2709-based formats (USMARC, etc.). Only records with a
two-level structure (corresponding to fields and subfields) can be
directly mapped to ISO2709. For records with a different structuring
(eg., GILS), the representation in a structure like USMARC involves a
schema-mapping (see section <xref linkend="schema-mapping"/>), to an
"implied" USMARC schema (implied,
because there is no formal schema which specifies the use of the
USMARC fields outside of ISO2709). The resultant, two-level record is
then mapped directly from the internal representation to ISO2709. See
the GILS schema definition files for a detailed example of this
approach.

</para>
</listitem>
<listitem>

<para>
Explain. This representation is only available for records
belonging to the Explain schema.

</para>
</listitem>
<listitem>

<para>
Summary.  This ASN-1 based structure is only available for records
belonging to the Summary schema - or schema which provide a mapping
to this schema (see the description of the schema mapping facility
above).

</para>
</listitem>
<listitem>

<para>
SOIF. Support for this syntax is experimental, and is currently
keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
abstract syntaxes can be mapped to the SOIF format, although nested
elements are represented by concatenation of the tag names at each
level.

</para>
</listitem>

</itemizedlist>

</para>

</sect1>

</chapter>