<!doctype linuxdoc system>
<!--
- $Id: zebra.sgml,v 1.2 1995-11-28 14:25:23 adam Exp $
+ $Id: zebra.sgml,v 1.45 2000-02-10 10:19:47 adam Exp $
-->
<article>
-<title>Zebra server Administrators's Guide and Reference
-<author>Index Data, <tt/info@index.ping.dk/
-<date>$Revision: 1.2 $
+<title>Zebra Server - Administrators's Guide and Reference
+<author><htmlurl url="http://www.indexdata.dk/" name="Index Data">,
+<tt><htmlurl url="mailto:info@indexdata.dk" name="info@indexdata.dk"></>
+<date>$Revision: 1.45 $
<abstract>
-This document describes Zebra — an information retrieval server
-that uses the Z39.50 protocol.
+
+
+The Zebra server combines a versatile fielded/free-text
+indexing/search engine with a Z39.50-1995 frontend to provide a powerful and flexible
+information mining tool. This document explains the procedure for
+installing and configuring Zebra, and outlines the possibilities
+for managing data and providing Z39.50
+services with the software. Zebra is a free version of the Index Data Z'mbol
+information system, and it excludes some functionality such as incremental
+database updating and support for large databases.
</abstract>
<toc>
<sect>Introduction
+<sect1>Overview
+
<p>
-- gils
-- UNI*X (not NT yet)
-- goals
+Zebra is a fielded free-text indexing and retrieval engine with a
+Z39.50 frontend. You can use any commercial or freeware Z39.50 client
+to access data stored in Zebra.
+
+Zebra server can be used at the core of a Z39.50-based information retrieval
+framework. We're making
+the server available now to allow researchers and small organisations to
+share their information in the best possible way. We believe that Z39.50
+currently represents one of the best ways of sharing information with others, and
+we would like to encourage as many people as possible to do so.
+This document is a guide to using Zebra. It will tell you
+how to compile the software, and how to prepare your first database.
+It also explains how the server can be configured to give you the
+functionality that you need.
+
+If you find the software interesting, you should join the support
+mailing-list by sending email to <tt/zebra-request@indexdata.dk/.
+
+If you are interested in running a commercial service, if you wish to run large
+databases, or if you wish to make incremental updates to your databases even
+while users are accessing your system, then you might be interested in the Z'mbol
+Information Server which is available from <htmlurl
+url="http://www.indexdata.dk/zmbol/" name="Index Data"> or Fretwell-Downing
+Informatics. Z'mbol is a complete and supported package which offers many
+exciting possibilities that we have not been able to fit into this package.
+
+<sect1>Features
-<sect>Compiling the software
+<p>
+This is a list of some of the most important features of the
+system.
+
+<itemize>
+
+<item>
+Supports arbitrarily complex records - base input format is an
+XML-like syntax which allows nested (structured) data elements, as
+well as variant forms of data.
+
+<item>
+Supports random storage formats. A system of input filters driven by
+regular expressions allows you to easily process most ASCII-based
+data formats. SGML/XML, ISO2709 (MARC), and raw text are also supported.
+
+<item>
+Supports boolean queries as well as relevance-ranking (free-text)
+searching. Right truncation and masking in terms are supported, as
+well as full regular expressions.
+
+<item>
+Supports multiple concrete syntaxes
+for record exchange (depending on the configuration): GRS-1, SUTRS,
+ISO2709 (*MARC), XML. Records can be mapped between record syntaxes and
+schema on the fly.
+
+<item>
+Supports approximate matching in registers (ie. spelling mistakes,
+etc).
+
+<item> Supports a subset of the Z39.50 Explain Facility. Zebra's Explain database
+is automatically updated when a set of records is loaded into Zebra.
+
+</itemize>
<p>
+Protocol support:
+
+<itemize>
+
+<item>
+Protocol facilities: Init, Search, Retrieve, Browse, Sort, Close, and Explain.
+
+<item>
+Piggy-backed presents are honored in the search-request.
+
+<item>
+Named result sets are supported.
-Zebra uses the YAZ package in order to provide Z39.50 access. So you
-must compile YAZ before going further. Specifically, Zebra uses
-the YAZ header files in <tt>yaz/include/..</tt> and its public library
-<tt>yaz/lib/libyaz.a</tt>.
+<item>
+Easily configured to support different application profiles, with
+tables for attribute sets, tag sets, and abstract syntaxes.
+Additional tables control facilities such as element mappings to
+different schema (eg., GILS-to-USMARC).
-As with YAZ, an ANSI C compiler is required in order to compile the Zebra
-server system — GNU C works fine.
+<item>
+Complex composition specifications using Espec-1 are partially
+supported (simple element requests only).
-Unpack the Zebra software. You might put Zebra in the same directory level
-as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-</tt>.., then
-Zebra is placed in ..<tt>/src/zebra-</tt>.
+<item>
+Element Set Names are defined using the Espec-1 capability of the
+system, and are given in configuration files as simple element
+requests (and possibly variant requests).
+
+<item>
+Zebra runs on most Unix-like systems as well as Windows NT - a binary
+distribution for Windows NT is forthcoming - so far, the installation
+requires Microsoft Visual C++ to compile the system (we use version 6.0).
+
+</itemize>
+
+<sect1>Future Work
+
+<p>
-Edit the top-level <tt>Makefile</tt> in the Zebra directory in which
-you specify the location of YAZ by setting make variables.
-The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with
-MOSI support. Some systems, such as Solaris, have separate socket
-libraries and for those systems you need to specify the
-<tt>NETLIB</tt> variable.
+These are some of the plans that we have for the software in the near
+and far future, approximately ordered after their relative importance.
+Items marked with an
+asterisk will be implemented before the
+last beta release.
-When finished editing the <tt>Makefile</tt> type:
+<itemize>
+
+<item>
+*Complete the support for variants.
+
+<item>
+*Finalize the data element <it/include/ facility to support multimedia
+data elements in records.
+
+<item>
+Add more sophisticated relevance ranking mechanisms. Add support for soundex
+and stemming. Add relevance <it/feedback/ support.
+
+<item>
+Complete EXPLAIN support.
+
+<item>
+We want to add a management system that allows you to
+control your databases and configuration tables from a graphical
+interface. We'll probably use Tcl/Tk to stay platform-independent.
+
+</itemize>
+
+Programmers thrive on user feedback. If you are interested in a facility that
+you don't see mentioned here, or if there's something you think we
+could do better, please drop us a mail. If you think it's all really
+neat, you're welcome to drop us a line saying that, too. You'll find
+contact info at the end of this file.
+
+<sect>Compiling the software
+<p>
+You need the
+<bf><htmlurl url="http://www.indexdata.dk/yaz/" name="YAZ"></>
+package in order to compile this software. We suggest you
+unpack <bf/YAZ/ in the same directory as Zebra. Running
+./configure (UNIX Only) and running make (nmake on WIN32) is
+in usully what it takes to compile YAZ.
+
+<sect1>UNIX
+<p>
+An ANSI C compiler is required to compile the Zebra
+server system — <tt/gcc/ works very well if your own system doesn't
+provide an adequate compiler.
+
+Unpack the distribution archive. The <tt>configure</tt> shell script
+attempts to guess correct values for various system-dependent variables
+used during compilation. It uses those values to create a 'Makefile' in
+each directory of Zebra.
+
+To run the configure script type:
+<tscreen><verb>
+ ./configure
+</verb></tscreen>
+
+The configure script attempts to use the C compiler specified by
+the <tt>CC</tt> environment variable. If not set, GNU C
+will be used if it is available. The <tt>CFLAGS</tt> environment variable
+holds options to be passed to the C compiler. If you're using a
+Bourne-compatible shell you may pass something like this:
<tscreen><verb>
-$ make
+ CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
</verb></tscreen>
+To customize Zebra the configure script accepts a set of options. The
+most important are
+<descrip>
+<tag><tt>--prefix </tt>path</tag> Specifies installation prefix. This is
+only needed if you run <tt>make install</tt> later to perform a
+"system" installation. The prefix is <tt>/usr/local</tt> if not
+specified.
+<tag><tt>--with-tclconfig </tt>path</tag> If Tcl is installed on
+the system you can tell configure where Tcl's <tt>tclConfig.sh</tt>
+installed. The <tt>tclConfig.sh</tt> include information about settings
+required to link with Tcl's libraries. If you don't specify this
+option, configure will see if Tcl's shell <tt>tclsh</tt> is in your
+path and if it is, it will guess where the equivalent tclConfig.sh
+is located. If tclsh is not found in your path and this option is not
+given Zebra will not include Tcl support.
+<tag><tt>--with-yazconfig </tt>path</tag> This options allows you to
+specify the path of YAZ's <tt>yaz-config</tt>. Therefore this option
+forces Zebra to use a particular version of YAZ. YAZ version 1.5 and
+later creates a script <tt>yaz-config</tt> that includes information
+on compiler settings needed to link with it.
+</descrip>
+
+When configured build the software by typing:
+<tscreen><verb>
+ make
+</verb></tscreen>
+
+As an option you may type <tt>make depend</tt> to create
+source file dependencies for the package. This is only needed,
+however, if you modify the source code later.
+
If successful, two executables have been created in the sub-directory
-index.
+<tt>bin</tt>.
<descrip>
<tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
<tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
</descrip>
-<sect>Quick getting started
+<p>
+The next step is optional and is only needed if you wish to install
+zebra in system directories such as /usr/bin, /usr/lib, etc.
+
+To perform this step, type
+<tscreen><verb>
+ make install
+</verb></tscreen>
+
+The executables will be installed in prefix/bin, and profile
+tables will be installed in prefix/lib/zebra/tab. Here prefix
+represents the prefix as specified -- default being /usr/local.
+
+<sect1>WIN32
<p>
-This section will get you started quickly! We will try to index a few
-GILS records that are included with the Zebra distribution. Go to the
-<tt>test</tt> subdirectory. There you will find a configuration
+Zebra is shipped with "makefiles" for the NMAKE tool that comes
+with Visual C++.
+
+Start an MS-DOS prompt and switch the sub directory <tt>WIN</tt> where
+the file <tt>makefile</tt> is located. Customize the installation
+by editing the <tt>makefile</tt> file (for example by using wordpad).
+
+The following summarises the most important settings in that file.
+
+<descrip>
+<tag><tt>YAZDIR</tt></tag> Specifies where YAZ is located.
+<tag><tt>DEBUG</tt></tag> If set to 1, the software is
+compiled with debugging libraries. If set to 0, the software
+is compiled with release (non-debugging) libraries.
+<tag>BZIP2</tag> A group of settings (<tt>BZIP2LIB</tt>,..)
+that must be defined if BZIP2 compression support is desired.
+</descrip>
+
+When satisfied with the settings in the makefile type
+<tscreen><verb>
+nmake
+</verb></tscreen>
+
+If compilation was successful the executables <tt>zebraidx.exe</tt>
+and <tt>zebrasrv.exe</tt> are put in the sub directory <tt>BIN</tt>.
+
+<sect>Quick Start
+<p>
+In this section, we will test the system by indexing a small set of sample
+GILS records that are included with the software distribution. Go to the
+<tt>test/gils</tt> subdirectory of the distribution archive. There you will
+find a configuration
file named <tt>zebra.cfg</tt> with the following contents:
<tscreen><verb>
-# Where are the YAZ tables located.
-profilePath: /usr/local/yaz
+# Where the schema files, attribute files, etc. are located.
+profilePath: .:../../tab:../../../yaz/tab
# Files that describe the attribute sets supported.
+attset: explain.att
attset: bib1.att
attset: gils.att
</verb></tscreen>
Now, edit the file and set <tt>profilePath</tt> to the path of the
-YAZ profile tables (sub directory <tt>tab</tt> of YAZ).
+YAZ profile tables (sub directory <tt>tab</tt> of the YAZ distribution
+archive).
The 48 test records are located in the sub directory <tt>records</tt>.
To index these, type:
<tscreen><verb>
-$ ../index/zebraidx -t grs update records
+$ ../../bin/zebraidx -t grs.sgml update records
</verb></tscreen>
In the command above the option <tt>-t</tt> specified the record
-type — in this case <tt>grs</tt>. The word <tt>update</tt> followed
-by a directory root updates all files below that directory.
+type — in this case <tt>grs.sgml</tt>. The word <tt>update</tt> followed
+by a directory root updates all files below that directory node.
-If your indexing command went successful, you are now ready to
-setup a server. To start a server on port 2100, type:
+If your indexing command was successful, you are now ready to
+fire up a server. To start a server on port 2100, type:
<tscreen><verb>
-$ ../index/zebrasrv tcp:@:2100
+$ ../../bin/zebrasrv tcp:@:2100
</verb></tscreen>
-The Zebra server just made has one database called Default. It will
-return either USMARC/GRS or SUTRS depending on your client.
+The Zebra index that you have just created has a single database
+named <tt/Default/. The database contains records structured according to
+the GILS profile, and the server will
+return records in either either XML, USMARC, GRS-1, or SUTRS depending
+on what your client asks for.
-<sect>Administrating Zebra
+To test the server, you can use any Z39.50 client (1992 or later). For
+instance, you can use the demo client that comes with YAZ: Just cd to
+the <tt/client/ subdirectory of the YAZ distribution and type:
-<p>
+<tscreen><verb>
+$ ./yaz-client tcp:localhost:2100
+</verb></tscreen>
-Unlike many other retrieval systems, Zebra offers incremental
-modifications of an existing index. Needless to say, these facilities
-makes the administration of Zebra somewhat more complicated.
+When the client has connected, you can type:
-Normally, when Zebra modifies the index it reads a number of records
-that you specify.
-Depending on your specifications and on the contents of each record
-one the following scenarios occur:
-<descrip>
-<tag>Insert</tag> The record is indexed as if it never occurred
-before. Either the Zebra system doesn't know how to identify the record or
-Zebra can identify the record but didn't find it to be already indexed.
-<tag>Modify</tag> The record has already been indexed. In this case
-either the contents of the record or the location (file) of the record
-indicates that it has been indexed before.
-<tag>Delete</tag> The record is deleted from the index. As in the
-update-case it must be able to identify the record.
-</descrip>
+<tscreen><verb>
+Z> find surficial
+Z> show 1
+</verb></tscreen>
-Please note, that in both the modify- and delete- case the Zebra
-system must be able to make a key that identifies the record in
-question.
+The default retrieval syntax for the client is USMARC. To try other
+formats for the same record, try:
-To administrate the Zebra retrieval system, you run the
+<tscreen><verb>
+Z>format sutrs
+Z>show 1
+Z>format grs-1
+Z>show 1
+Z>format xml
+Z>show 1
+Z>elements B
+Z>show 1
+</verb></tscreen>
+
+<it>NOTE: You may notice that more fields are returned when your
+client requests SUTRS or GRS-1 records. When retrieving GILS records,
+this is normal - not all of the GILS data elements have mappings in
+the USMARC record format.</it>
+
+If you've made it this far, there's a good chance that
+you've got through the compilation OK.
+
+<sect>Administrating Zebra<label id="administrating">
+
+<p>
+
+To administrate Zebra, you run the
<tt>zebraidx</tt> program. This program supports a number of options
which are preceded by a minus, and a few commands (not preceded by
minus).
Both the Zebra administrative tool and the Z39.50 server share a
-set of indexing files and a global configuration file. The
+set of index files and a global configuration file. The
name of the configuration file defaults to <tt>zebra.cfg</tt>.
The configuration file includes specifications on how to index
various kinds of records and where the other configuration files
are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
-be run in the same directory where the configuration file if you do
-not indicate the location of the configuration file by option
+be run in the directory where the configuration file lives unless you
+indicate the location of the configuration file by option
<tt>-c</tt>.
-<sect1>Record types
+<sect1>Record Types<label id="record-types">
<p>
-Indexing is a record-per-record process, in which
-either insert/modify/delete will occur. Before a record is indexed
-search keys are extracted from whatever might be the layout the
-original record (sgml,html,text, etc..). The Zebra system
-currently only support GILS records and simple text records.
+Indexing is a per-record process. Before a record is indexed search
+keys are extracted from whatever might be the layout the original
+record (sgml,html,text, etc..).
+The Zebra system currently supports two fundamantal types of records:
+structured and simple text.
To specify a particular extraction process, use either the
command line option <tt>-t</tt> or specify a
<tt>recordType</tt> setting in the configuration file.
-<sect1>The Zebra configuration file
+<sect1>The Zebra Configuration File<label id="configuration-file">
<p>
The Zebra configuration file, read by <tt>zebraidx</tt> and
<tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
by <tt>-c</tt> option.
You can edit the configuration file with a normal text editor.
-Setting names and values are seperated by colons in the file. Lines
-starting with a hash sign (<tt/#/) are treated as comments.
+Parameter names and values are seperated by colons in the file. Lines
+starting with a hash sign (<tt/#/) are treated as comments.
-A set of records that share common characteristics are called a group.
+If you manage different sets of records that share common
+characteristics, you can organize the configuration settings for each
+type into &dquot;groups&dquot;.
When <tt>zebraidx</tt> is run and you wish to address a given group
-you specify that group with the <tt>-g</tt> option. In this case
+you specify the group name with the <tt>-g</tt> option. In this case
settings that have the group name as their prefix will be used
-by <tt>zebraidx</tt> and not default values.
+by <tt>zebraidx</tt>. If no <tt/-g/ option is specified, the settings
+with no prefix are used.
+
+In the configuration file, the group name is placed before the option
+name itself, separated by a dot (.). For instance, to set the record type
+for group <tt/public/ to <tt/grs.sgml/ (the SGML-like format for structured
+records) you would write:
-The group is written before the option itself separated by a dot.
-For example to set the record type for group <tt/public/ to <tt/grs/
-you would write:
<tscreen><verb>
-public.recordType: grs
+public.recordType: grs.sgml
</verb></tscreen>
-To set the default value of the record type to text write:
+To set the default value of the record type to <tt/text/ write:
+
<tscreen><verb>
recordType: text
</verb></tscreen>
-The configuration settings are summarized below. They will be
-explained further in the following chapters.
+The available configuration settings are summarized below. They will be
+explained further in the following sections.
<descrip>
-<tag><it>group</it>recordType<it>name</it></tag>
+<tag><it>group</it>.recordType[<it>.name</it>]</tag>
Specifies how records with the file extension <it>name</it> should
be handled by the indexer. This option may also be specified
- as a command line option (<tt>-t</tt>).
-<tag><it>group</it>recordId</tag>
- Specifies how the record is to be identified when updated.
-<tag><it>group</it>database</tag>
- Specifies the Z39.50 database.
-<tag><it>group</it>storeKeys</tag>
+ as a command line option (<tt>-t</tt>). Note that if you do not
+ specify a <it/name/, the setting applies to all files. In general,
+ the record type specifier consists of the elements (each
+ element separated by dot), <it>fundamental-type</it>,
+ <it>file-read-type</it> and arguments. Currently, two
+ fundamental types exist, <tt>text</tt> and <tt>grs</tt>.
+ <tag><it>group</it>.recordId</tag>
+ Specifies how the records are to be identified when updated. See
+section <ref id="locating-records" name="Locating Records">.
+<tag><it>group</it>.database</tag>
+ Specifies the Z39.50 database name.
+<tag><it>group</it>.storeKeys</tag>
Specifies whether key information should be saved for a given
group of records. If you plan to update/delete this type of
- record a later this should be specified as 1; otherwise it
- should be 0 (default).
-<tag><it>group</it>storeData</tag>
- Specifies whether original copy of record should be stored internally
- in the Zebra system indexes. If your externally indexed files
- are temporary this option should certainly be true (1); otherwise
- false (0).
-<tag>register</tag>
- Specifies the location of the indexes.
+ records later this should be specified as 1; otherwise it
+ should be 0 (default), to save register space. See section
+<ref id="file-ids" name="Indexing With File Record IDs">.
+<tag><it>group</it>.storeData</tag>
+ Specifies whether the records should be stored internally
+ in the Zebra system files. If you want to maintain the raw records yourself,
+ this option should be false (0). If you want Zebra to take care of the records
+ for you, it should be true(1).
+<tag>lockDir</tag>
+ Directory in which various lock files are stored.
+<tag>keyTmpDir</tag>
+ Directory in which temporary files used during zebraidx' update
+ phase are stored.
+<tag>setTmpDir</tag>
+ Specifies the directory that the server uses for temporary result sets.
+ If not specified <tt>/tmp</tt> will be used.
<tag>profilePath</tag>
- Specifies the location of profile specification paths.
+ Specifies the location of profile specification files.
<tag>attset</tag>
- Specifies the filename(s) of attribute specification files.
+ Specifies the filename(s) of attribute set files for use in
+ searching. At least the Bib-1 set should be loaded (<tt/bib1.att/).
+ The <tt/profilePath/ setting is used to look for the specified files.
+ See section <ref id="attset-files" name="The Attribute Set Files">
+<tag>memMax</tag>
+ Specifies size of internal memory to use for the zebraidx program. The
+ amount is given in megabytes - default is 4 (4 MB).
</descrip>
-
-<sect1>Locating records
+<sect1>Locating Records<label id="locating-records">
<p>
The default behaviour of the Zebra system is to reference the
-records from their original location, i.e. where they were
-read an indexed.
-
-If your records files are temporary for example if you retrieve
-them from the outside or if they where temporarily mounted on a CD-ROM,
-you may want the Zebra server to make a copy of them. To do this
-you specify 1 (true) in the <tt>storedata</tt> setting. When
-the Z39.50 search engine retrieves recoreds they will be read from an
-internal structure of the index.
-
-<sect1>Indexing with no record Ids
+records from their original location, i.e. where they were found when you
+ran <tt/zebraidx/. That is, when a client wishes to retrieve a record
+following a search operation, the files are accessed from the place
+where you originally put them - if you remove the files (without
+running <tt/zebraidx/ again, the client will receive a diagnostic
+message.
+
+If your input files are not permanent - for example if you retrieve
+your records from an outside source, or if they were temporarily
+mounted on a CD-ROM drive,
+you may want Zebra to make an internal copy of them. To do this,
+you specify 1 (true) in the <tt>storeData</tt> setting. When
+the Z39.50 server retrieves the records they will be read from the
+internal file structures of the system.
+
+<sect1>Indexing example
<p>
-If you have a set of records that you <em/never/ wish to delete
-or modify you may find 'indexing with no records Ids' convenient.
-This indexing method uses less space than the other methods and
-is simple to use.
-
-To use this method, you simply don't specify the <tt>recordId</tt>
-for the group of files you index. To add a set of records you use
-<tt>zebraidx</tt> with the <tt>update</tt> command. The
-<tt>update</tt> command will always add the records to the index
-becuase Zebra doesn't know how to match the new set of records with
-existing records.
-
-Consider a system, in which you have a group of text files called
+Consider a system in which you have a group of text files called
<tt>simple</tt>. That group of records should belong to a Z39.50 database
-called <tt>textbase</tt>. The following configuration file suffice:
+called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice:
<tscreen><verb>
-profilePath: /usr/local/yaz
+profilePath: /usr/lib/yaz/tab:/usr/lib/zebra/tab
+attset: explain.att
attset: bib1.att
-attset: gils.att
simple.recordType: text
simple.database: textbase
</verb></tscreen>
-<sect1>Indexing with file record Ids
+<sect>Running the Maintenance Interface (zebraidx)
+
+<p>
+The following is a complete reference to the command line interface to
+the <tt/zebraidx/ application.
+
+<bf/Syntax/
+<tscreen><verb>
+$ zebraidx [options] command [directory] ...
+</verb></tscreen>
+<bf/Options/
+<descrip>
+<tag>-t <it/type/</tag>Update all files as <it/type/. Currently, the
+types supported are <tt/text/ and <tt/grs/<it/.subtype/. If no
+<it/subtype/ is provided for the GRS (General Record Structure) type,
+the canonical input format is assumed (see section <ref
+id="local-representation" name="Local Representation">). Generally, it
+is probably advisable to specify the record types in the
+<tt/zebra.cfg/ file (see section <ref id="record-types" name="Record
+Types">), to avoid confusion at subsequent updates.
+
+<tag>-c <it/config-file/</tag>Read the configuration file
+<it/config-file/ instead of <tt/zebra.cfg/.
+
+<tag>-g <it/group/</tag>Update the files according to the group
+settings for <it/group/ (see section <ref id="configuration-file"
+name="The Zebra Configuration File">).
+
+<tag>-d <it/database/</tag>The records located should be associated
+with the database name <it/database/ for access through the Z39.50
+server.
+
+<tag>-m <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing
+keys to background storage. This setting affects performance when
+updating large databases.
+
+<tag>-s</tag>Show analysis of the indexing process. The maintenance
+program works in a read-only mode and doesn't change the state
+of the index. This options is very useful when you wish to test a
+new profile.
+
+<tag>-V</tag>Show Zebra version.
+
+<tag>-v <it/level/</tag>Set the log level to <it/level/. <it/level/
+should be one of <tt/none/, <tt/debug/, and <tt/all/.
+
+</descrip>
+
+<bf/Commands/
+<descrip>
+<tag>Update <it/directory/</tag>Update the register with the files
+contained in <it/directory/. If no directory is provided, a list of
+files is read from <tt/stdin/. See section <ref
+id="administrating" name="Administrating Zebra">.
+
+</descrip>
+
+<sect>The Z39.50 Server
+
+<sect1>Running the Z39.50 Server (zebrasrv)
<p>
-If you have a set of external records that you wish to index you may
-use the file key feature of the Zebra system. In short, the file key
-feature, mirrors a directory and the files below efficiently. To
-perform indexing of a directory with file keys, you specify the top-level
-directory after the <tt>update</tt> command. The command will recursively
-traverse the directories and compare each with whatever have been
-indexed before in the same directory. If a file is new (not in
-the previous version of the directory) it is inserted;
-if a file was already indexed and it has been modified
-since the last insertion the index is also modified; if a file is missing
-since the last visit it is deleted.
+<bf/Syntax/
+<tscreen><verb>
+zebrasrv [options] [listener-address ...]
+</verb></tscreen>
+
+<bf/Options/
+<descrip>
+<tag>-a <it/APDU file/</tag> Specify a file for dumping PDUs (for diagnostic purposes).
+The special name &dquot;-&dquot; sends output to <tt/stderr/.
+
+<tag>-c <it/config-file/</tag> Read configuration information from <it/config-file/. The default configuration is <tt>./zebra.cfg</tt>.
+
+<tag/-S/Don't fork on connection requests. This can be useful for
+symbolic-level debugging. The server can only accept a single
+connection in this mode.
+
+<tag>-l <it/logfile/</tag>Specify an output file for the diagnostic
+messages. The default is to write this information to <tt/stderr/.
-The resulting system is easy to administer. To delete a record
-you simply have to delete the corresponding file (with <tt/rm/).
-To force update of a given file, you may use the <tt>touch</tt>
-command. And to add files create new files (or directories with files).
-For your changes to take effect you must run <tt>zebraidx</tt> with
-the same directory root again.
+<tag>-v <it/log-level/</tag>The log level. Use a comma-separated list of members of the set
+{fatal,debug,warn,log,all,none}.
-To use this method, you must specify <tt>file</tt> as the value
-of <tt>recordId</tt> the configuration file. In the configuration
-also set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra
-indexer must save additional information in order to
-modify/delete the record at a later time.
+<tag>-u <it/username/</tag>Set user ID. Sets the real UID of the server process to that of the
+given <it/username/. It's useful if you aren't comfortable with having the
+server run as root, but you need to start it as such to bind a
+privileged port.
+
+<tag>-w <it/working-directory/</tag>Change working directory.
+
+<tag>-i</tag>Run under the Internet superserver, <tt/inetd/. Make
+sure you use the logfile option <tt/-l/ in conjunction with this
+mode and specify the <tt/-l/ option before any other options.
+
+<tag>-t <it/timeout/</tag>Set the idle session timeout (default 60 minutes).
+
+<tag>-k <it/kilobytes/</tag>Set the (approximate) maximum size of
+present response messages. Default is 1024 Kb (1 Mb).
+</descrip>
+
+A <it/listener-address/ consists of a transport mode followed by a
+colon (:) followed by a listener address. The transport mode is
+either <tt/osi/ or <tt/tcp/.
+
+For TCP, an address has the form
-For example, to update group <tt>esdd</tt> records below
-<tt>/home/grs</tt> you could type:
<tscreen><verb>
-$ zebraidx -g esdd update /home/grs
+hostname | IP-number [: portnumber]
</verb></tscreen>
-The corresponding configuration file includes:
+The port number defaults to 210 (standard Z39.50 port).
+
+The special hostname &dquot;@&dquot; is mapped to
+the address INADDR_ANY, which causes the server to listen on any local
+interface. To start the server listening on the registered port for
+Z39.50, and to drop root privileges once the
+port is bound, execute the server like this (from a root shell):
+
<tscreen><verb>
-esdd.recordId: file
-esdd.recordType: grs
-esdd.storeKeys: 1
+zebrasrv -u daemon tcp:@
</verb></tscreen>
-<em>Important note: You cannot start by a group of records with simple
-indexing (no record ids as in the previous section) and then, later,
-use file record Ids. The Zebra must know from the very beginning that
-the group of files are indexed with file record Ids.
-</em>
+You can replace <tt/daemon/ with another user, eg. your own account, or
+a dedicated IR server account.
+
+The default behavior for <tt/zebrasrv/ is to establish a single TCP/IP
+listener, for the Z39.50 protocol, on port 9999.
+
+<sect1>Z39.50 Protocol Support and Behavior
+
+<sect2>Initialization
-<sect1>Indexing with general record Ids
<p>
-When using this method you specify an (almost) arbritrary record key
-based on the contents of the record itself and other system
-information. If you have a group of records that have an external
-id attached, this method is convenient. For example, the record may
-contain a title or an unique id. In either case you specify the
-Z39.50 attribute-set and attribute-use location in which this
-information is stored.
+During initialization, the server will negotiate to version 3 of the
+Z39.50 protocol (unless the client specifies a lower version), and the option bits for Search, Present, Scan,
+NamedResultSets, and concurrentOperations will be set, if requested by
+the client. The maximum PDU size is negotiated down to a maximum of
+1Mb by default.
+
+<sect2>Search<label id="search">
-As before, the record id is defined by the <tt>recordId</tt> setting
-in the configuration file. The value of the record id consists of one
-or more tokens separated by space. The resulting id is formed
-by concatenating the tokens and separete them by ascii value (1).
+<p>
+The supported query type are 1 and 101. All operators are currently
+supported with the restriction that only proximity units of type "word" are
+supported for the proximity operator.
+Queries can be arbitrarily complex.
+Named result sets are supported, and result sets can be used as operands
+without limitations.
+Searches may span multiple databases.
+
+The server has full support for piggy-backed present requests (see
+also the following section).
+
+<bf/Use/ attributes are interpreted according to the attribute sets which
+have been loaded in the <tt/zebra.cfg/ file, and are matched against
+specific fields as specified in the <tt/.abs/ file which describes the
+profile of the records which have been loaded. If no <bf/Use/
+attribute is provided, a default of Bib-1 <bf/Any/ is assumed.
+
+If a <bf/Structure/ attribute of <bf/Phrase/ is used in conjunction with a
+<bf/Completeness/ attribute of <bf/Complete (Sub)field/, the term is
+matched against the contents of the phrase (long word) register, if one
+exists for the given <bf/Use/ attribute.
+A phrase register is created for those fields in the <tt/.abs/
+file that contains a <tt/p/-specifier.
+
+If <bf/Structure/=<bf/Phrase/ is used in conjunction with
+<bf/Incomplete Field/ - the default value for <bf/Completeness/, the
+search is directed against the normal word registers, but if the term
+contains multiple words, the term will only match if all of the words
+are found immediately adjacent, and in the given order.
+The word search is performed on those fields that are indexed as
+type <tt/w/ in the <tt/.abs/ file.
+
+If the <bf/Structure/ attribute is <bf/Word List/,
+<bf/Free-form Text/, or <bf/Document Text/, the term is treated as a
+natural-language, relevance-ranked query.
+This search type uses the word register, i.e. those fields
+that are indexed as type <tt/w/ in the <tt/.abs/ file.
+
+If the <bf/Structure/ attribute is <bf/Numeric String/ the
+term is treated as an integer. The search is performed on those
+fields that are indexed as type <tt/n/ in the <tt/.abs/ file.
+
+If the <bf/Structure/ attribute is <bf/URx/ the
+term is treated as a URX (URL) entity. The search is performed on those
+fields that are indexed as type <tt/u/ in the <tt/.abs/ file.
+
+If the <bf/Structure/ attribute is <bf/Local Number/ the
+term is treated as native Zebra Record Identifier.
+
+If the <bf/Relation/ attribute is <bf/Equals/ (default), the term is
+matched in a normal fashion (modulo truncation and processing of
+individual words, if required). If <bf/Relation/ is <bf/Less Than/,
+<bf/Less Than or Equal/, <bf/Greater than/, or <bf/Greater than or
+Equal/, the term is assumed to be numerical, and a standard regular
+expression is constructed to match the given expression. If
+<bf/Relation/ is <bf/Relevance/, the standard natural-language query
+processor is invoked.
+
+For the <bf/Truncation/ attribute, <bf/No Truncation/ is the default.
+<bf/Left Truncation/ is not supported. <bf/Process #/ is supported, as
+is <bf/Regxp-1/. <bf/Regxp-2/ enables the fault-tolerant (fuzzy)
+search. As a default, a single error (deletion, insertion,
+replacement) is accepted when terms are matched against the register
+contents.
+
+<sect3>Regular expressions
+<p>
-There are three kinds of tokens:
+Each term in a query is interpreted as a regular expression if
+the truncation value is either <bf/Regxp-1/ (102) or <bf/Regxp-2/ (103).
+Both query types follow the same syntax with the operands:
<descrip>
-<tag>Internal record info</tag> The token refers to a key that is
-extracted from the record. The syntax of this token is
- <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the
-attribute set number and <em/use/ is the use value of the attribute.
-<tag>System variable</tag> The system variables are preceded by
-<tt>$</tt> and immediately followed by the system variable name, which
-may one of
- <descrip>
- <tag>group</tag> Group name.
- <tag>database</tag> Current database specified.
- <tag>filename</tag> Name of file that contain the record.
- <tag>type</tag> Record type.
- </descrip>
-<tag>Constant string</tag> A string used as part of id — surrounded
- by single- or double quotes.
+<tag/x/ Matches the character <it/x/.
+<tag/./ Matches any character.
+<tag><tt/[/..<tt/]/</tag> Matches the set of characters specified;
+ such as <tt/[abc]/ or <tt/[a-c]/.
</descrip>
+and the operators:
+<descrip>
+<tag/x*/ Matches <it/x/ zero or more times. Priority: high.
+<tag/x+/ Matches <it/x/ one or more times. Priority: high.
+<tag/x?/ Matches <it/x/ once or twice. Priority: high.
+<tag/xy/ Matches <it/x/, then <it/y/. Priority: medium.
+<tag/x|y/ Matches either <it/x/ or <it/y/. Priority: low.
+</descrip>
+The order of evaluation may be changed by using parentheses.
+
+If the first character of the <bf/Regxp-2/ query is a plus character
+(<tt/+/) it marks the beginning of a section with non-standard
+specifiers. The next plus character marks the end of the section.
+Currently Zebra only supports one specifier, the error tolerance,
+which consists one digit.
+
+Since the plus operator is normally a suffix operator the addition to
+the query syntax doesn't violate the syntax for standard regular
+expressions.
+
+<sect3>Query examples
+<p>
+
+Phrase search for <bf/information retrieval/ in the title-register:
+<verb>
+ @attr 1=4 "information retrieval"
+</verb>
+
+Ranked search for the same thing:
+<verb>
+ @attr 1=4 @attr 2=102 "Information retrieval"
+</verb>
+
+Phrase search with a regular expression:
+<verb>
+ @attr 1=4 @attr 5=102 "informat.* retrieval"
+</verb>
+
+Ranked search with a regular expression:
+<verb>
+ @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
+</verb>
+
+In the GILS schema (<tt/gils.abs/), the west-bounding-coordinate is
+indexed as type <tt/n/, and is therefore searched by specifying
+<bf/structure/=<bf/Numeric String/.
+To match all those records with west-bounding-coordinate greater
+than -114 we use the following query:
+<verb>
+ @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
+</verb>
+
+<sect2>Present
+<p>
+The present facility is supported in a standard fashion. The requested
+record syntax is matched against the ones supported by the profile of
+each record retrieved. If no record syntax is given, SUTRS is the
+default. The requested element set name, again, is matched against any
+provided by the relevant record profiles.
+
+<sect2>Scan
+
+<p>
+The attribute combinations provided with the TermListAndStartPoint are
+processed in the same way as operands in a query (see above).
+Currently, only the term and the globalOccurrences are returned with
+the TermInfo structure.
+
+<sect2>Sort
+
+<p>
+Z39.50 specifies three diffent types of sort criterias.
+Of these Zebra supports the attribute specification type in which
+case the use attribute specifies the "Sort register".
+Sort registers are created for those fields that are of type "sort" in
+the default.idx file.
+The corresponding character mapping file in default.idx specifies the
+ordinal of each character used in the actual sort.
+
+Z39.50 allows the client to specify sorting on one or more input
+result sets and one output result set.
+Zebra supports sorting on one result set only which may or may not
+be the same as the output result set.
+
+<sect2>Close
+
+<p>
+If a Close PDU is received, the server will respond with a Close PDU
+with reason=FINISHED, no matter which protocol version was negotiated
+during initialization. If the protocol version is 3 or more, the
+server will generate a Close PDU under certain circumstances,
+including a session timeout (60 minutes by default), and certain kinds of
+protocol errors. Once a Close PDU has been sent, the protocol
+association is considered broken, and the transport connection will be
+closed immediately upon receipt of further data, or following a short
+timeout.
+
+<sect>The Record Model
+
+<p>
+Zebra is designed to support a wide range of data management
+applications. The system can be configured to handle virtually any
+kind of structured data. Each record in the system is associated with
+a <it/record schema/ which lends context to the data elements of the
+record. Any number of record schema can coexist in the system.
+Although it may be wise to use only a single schema within
+one database, the system poses no such restrictions.
+
+The record model described in this chapter applies to the fundamental,
+structured
+record type <tt>grs</tt> as introduced in
+section <ref id="record-types" name="Record Types">.
+
+Records pass through three different states during processing in the
+system.
+
+<itemize>
+<item>When records are accessed by the system, they are represented
+in their local, or native format. This might be SGML or HTML files,
+News or Mail archives, MARC records. If the system doesn't already
+know how to read the type of data you need to store, you can set up an
+input filter by preparing conversion rules based on regular
+expressions and possibly augmented by a flexible scripting language (Tcl). The input filter
+produces as output an internal representation:
+
+<item>When records are processed by the system, they are represented
+in a tree-structure, constructed by tagged data elements hanging off a
+root node. The tagged elements may contain data or yet more tagged
+elements in a recursive structure. The system performs various
+actions on this tree structure (indexing, element selection, schema
+mapping, etc.),
+
+<item>Before transmitting records to the client, they are first
+converted from the internal structure to a form suitable for exchange
+over the network - according to the Z39.50 standard.
+</itemize>
+
+<sect1>Local Representation<label id="local-representation">
+
+<p>
+As mentioned earlier, Zebra places few restrictions on the type of
+data that you can index and manage. Generally, whatever the form of
+the data, it is parsed by an input filter specific to that format, and
+turned into an internal structure that Zebra knows how to handle. This
+process takes place whenever the record is accessed - for indexing and
+retrieval.
+
+<p>
+The RecordType parameter in the <tt/zebra.cfg/ file, or the <tt/-t/
+option to the indexer tells Zebra how to process input records. Two
+basic types of processing are available - raw text and structured
+data. Raw text is just that, and it is selected by providing the
+argument <bf/text/ to Zebra. Structured records are all handled
+internally using the basic mechanisms described in the subsequent
+sections. Zebra can read structured records in many different formats.
+How this is done is governed by additional parameters after the
+&dquot;grs&dquot; keyboard, separated by &dquot;.&dquot; characters.
+
+Three basic subtypes to the <bf/grs/ type are currently available:
+
+<descrip>
+<tag>grs.sgml</tag>This is the canonical input format —
+described below. It is a simple SGML-like syntax.
+
+<tag>grs.regx.<it/filter/</tag>This enables a user-supplied input
+filter. The mechanisms of these filters are described below.
+
+<tag>grs.tcl.<it/filter/</tag>This enables a user-supplied input
+filter with Tcl rules (only availble if zebra is compiled with Tcl
+support).
+
+<tag>grs.marc.<it/abstract syntax/</tag>This allows Zebra to read
+records in the ISO2709 (MARC) encoding standard. In this case, the
+last paramemeter <it/abstract syntax/ names the .abs file (see below)
+which describes the specific MARC structure of the input record as
+well as the indexing rules.
+</descrip>
+
+<sect2>Canonical Input Format
+
+<p>
+Although input data can take any form, it is sometimes useful to
+describe the record processing capabilities of the system in terms of
+a single, canonical input format that gives access to the full
+spectrum of structure and flexibility in the system. In Zebra, this
+canonical format is an &dquot;SGML-like&dquot; syntax.
+
+To use the canonical format specify <tt>grs.sgml</tt> as the record
+type,
+
+Consider a record describing an information resource (such a record is
+sometimes known as a <it/locator record/). It might contain a field
+describing the distributor of the information resource, which might in
+turn be partitioned into various fields providing details about the
+distributor, like this:
-The test GILS records that comes with Zebra, contain a unique id
-in the Control-Identifier field. This field is mapped to the Bib-1
-use attribute 1007. To use this field as a record id, specify
-<tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the
-configuration file. If you have other record types that doesn't
-contain an id in the same field, you might add the record type
-in the record id of the gils records as well, to prevent match
-of other types of records. In this case the recordId might be
-set, as in:
<tscreen><verb>
-gils.recordId: $type (1,1007)
+<Distributor>
+ <Name> USGS/WRD &etago;Name>
+ <Organization> USGS/WRD &etago;Organization>
+ <Street-Address>
+ U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
+ &etago;Street-Address>
+ <City> ALBUQUERQUE &etago;City>
+ <State> NM &etago;State>
+ <Zip-Code> 87102 &etago;Zip-Code>
+ <Country> USA &etago;Country>
+ <Telephone> (505) 766-5560 &etago;Telephone>
+&etago;Distributor>
</verb></tscreen>
-As for the file record id case described in the previous section
-updating your system is simply a matter of running <tt>zebraidx</tt>
-with the <tt>update</tt> command. However, the update of with general
-keys is considerably slower than with file record ids, since all files
-visited must be (re)indexded.
+<it>NOTE: The indentation used above is used to illustrate how Zebra
+interprets the markup. The indentation, in itself, has no
+significance to the parser for the canonical input format, which
+discards superfluous whitespace.</it>
+
+The keywords surrounded by <...> are <it/tags/, while the
+sections of text in between are the <it/data elements/. A data element
+is characterized by its location in the tree that is made up by the
+nested elements. Each element is terminated by a closing tag -
+beginning with <tt/&etago;/, and containing the same symbolic tag-name as
+the corresponding opening tag. The general closing tag - <tt/&etago;>/ -
+terminates the element started by the last opening tag. The
+structuring of elements is significant. The element <bf/Telephone/,
+for instance, may be indexed and presented to the client differently,
+depending on whether it appears inside the <bf/Distributor/ element,
+or some other, structured data element such a <bf/Supplier/ element.
+
+<sect3>Record Root
+
+<p>
+The first tag in a record describes the root node of the tree that
+makes up the total record. In the canonical input format, the root tag
+should contain the name of the schema that lends context to the
+elements of the record (see section <ref id="internal-representation"
+name="Internal Representation">). The following is a GILS record that
+contains only a single element (strictly speaking, that makes it an
+illegal GILS record, since the GILS profile includes several mandatory
+elements - Zebra does not validate the contents of a record against
+the Z39.50 profile, however - it merely attempts to match up elements
+of a local representation with the given schema):
+
+<tscreen><verb>
+<gils>
+ <title>Zen and the Art of Motorcycle Maintenance&etago;title>
+&etago;gils>
+</verb></tscreen>
-<sect1>Register location
+<sect3>Variants
<p>
-Normally, the index files that form dictionaries, inverted
-file, record info, etc. are stored in the current directory in which you run
-<tt>zebraidx</tt>. If you wish to store those, possibly large, files
-somewhere else, you must set the <tt>register</tt> setting in the
-configuration file. Furhtermore, the Zebra system handles indexes that
-span multiple file systems, which is usefull if a large number of
-records are indexed.
+Zebra allows you to provide individual data elements in a number of
+<it/variant forms/. Examples of variant forms are textual data
+elements which might appear in different languages, and images which
+may appear in different formats or layouts. The variant system in
+Zebra is
+essentially a representation of the variant mechanism of
+Z39.50-1995.
-The value <tt>register</tt> of register is a sequence of tokens.
-Each token takes the form:
-<tscreen>
-<em>dir</em><tt>:</tt><em>size</em>.
-</tscreen>
-The <em>dir</em> specifies a directory in which index files will be
-stored and the <em>size</em> specifies the maximum size of all
-files in that directory. The Zebra indexer system fill each directory
-in the order specified and use the next specified directories as needed.
-The <em>size</em> is an integer possibly followed by a quantity
-code, <tt>M</tt> for Mega bytes, <tt>K</tt> for Kilo bytes.
+The following is an example of a title element which occurs in two
+different languages.
-If you have two spare disks :) and the first disk is mounted
-on <tt>/d1</tt> and has 200 Mega bytes free space after formatting and the
-second, mounted on <tt>/d2</tt> has 300 Mega bytes free, you could
-specify the following in you configuration file:
<tscreen><verb>
-register: /d1:200M /d2:300M
+<title>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance&etago;>
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel&etago;>
+&etago;title>
</verb></tscreen>
-<sect>The Z39.50 Server
+The syntax of the <it/variant element/ is <tt><<bf/var/ <it/class
+type value/></tt>. The available values for the <it/class/ and
+<it/type/ fields are given by the variant set that is associated with the
+current schema (see section <ref id="variant-set" name="Variant Set
+File">).
+
+Variant elements are terminated by the general end-tag &etago;>, by
+the variant end-tag &etago;var>, by the appearance of another variant
+tag with the same <it/class/ and <it/value/ settings, or by the
+appearance of another, normal tag. In other words, the end-tags for
+the variants used in the example above could have been saved.
+
+Variant elements can be nested. The element
+
+<tscreen><verb>
+<title>
+ <var lang lang "eng"><var body iana "text/plain">
+ Zen and the Art of Motorcycle Maintenance
+&etago;title>
+</verb></tscreen>
+
+Associates two variant components to the variant list for the title
+element.
+
+Given the nesting rules described above, we could write
+
+<tscreen><verb>
+<title>
+ <var body iana "text/plain>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel
+&etago;title>
+</verb></tscreen>
+
+The title element above comes in two variants. Both have the IANA body
+type &dquot;text/plain&dquot;, but one is in English, and the other in
+Danish. The client, using the element selection mechanism of Z39.50,
+can retrieve information about the available variant forms of data
+elements, or it can select specific variants based on the requirements
+of the end-user.
+
+<sect2>Input Filters
<p>
+In order to handle general input formats, Zebra allows the
+operator to define filters which read individual records in their native format
+and produce an internal representation that the system can
+work with.
+
+Input filters are ASCII files, generally with the suffix <tt/.flt/.
+The system looks for the files in the directories given in the
+<bf/profilePath/ setting in the <tt/zebra.cfg/ files. The record type
+for the filter is <tt>grs.regx.</tt><it>filter-filename</it>
+(fundamental type <tt>grs</tt>, file read type <tt>regx</tt>, argument
+<it>filter-filename</it>).
+
+Generally, an input filter consists of a sequence of rules, where each
+rule consists of a sequence of expressions, followed by an action. The
+expressions are evaluated against the contents of the input record,
+and the actions normally contribute to the generation of an internal
+representation of the record.
+
+An expression can be either of the following:
+
+<descrip>
+<tag/INIT/The action associated with this expression is evaluated
+exactly once in the lifetime of the application, before any records
+are read. It can be used in conjunction with an action that
+initializes tables or other resources that are used in the processing
+of input records.
+
+<tag/BEGIN/Matches the beginning of the record. It can be used to
+initialize variables, etc. Typically, the <bf/BEGIN/ rule is also used
+to establish the root node of the record.
+
+<tag/END/Matches the end of the record - when all of the contents
+of the record has been processed.
+
+<tag>/pattern/</tag>Matches a string of characters from the input
+record.
+
+<tag/BODY/This keyword may only be used between two patterns. It
+matches everything between (not including) those patterns.
+
+<tag/FINISH/THe expression asssociated with this pattern is evaluated
+once, before the application terminates. It can be used to release
+system resources - typically ones allocated in the <bf/INIT/ step.
+
+</descrip>
+
+An action is surrounded by curly braces ({...}), and consists of a
+sequence of statements. Statements may be separated by newlines or
+semicolons (;). Within actions, the strings that matched the
+expressions immediately preceding the action can be referred to as
+$0, $1, $2, etc.
+
+The available statements are:
+
+<descrip>
+
+<tag>begin <it/type [parameter ... ]/</tag>Begin a new
+data element. The type is one of the following:
+<descrip>
+<tag/record/Begin a new record. The followingparameter should be the
+name of the schema that describes the structure of the record, eg.
+<tt/gils/ or <tt/wais/ (see below). The <tt/begin record/ call should
+precede
+any other use of the <bf/begin/ statement.
+
+<tag/element/Begin a new tagged element. The parameter is the
+name of the tag. If the tag is not matched anywhere in the tagsets
+referenced by the current schema, it is treated as a local string
+tag.
+
+<tag/variant/Begin a new node in a variant tree. The parameters are
+<it/class type value/.
+
+</descrip>
+
+<tag/data/Create a data element. The concatenated arguments make
+up the value of the data element. The option <tt/-text/ signals that
+the layout (whitespace) of the data should be retained for
+transmission. The option <tt/-element/ <it/tag/ wraps the data up in
+the <it/tag/. The use of the <tt/-element/ option is equivalent to
+preceding the command with a <bf/begin element/ command, and following
+it with the <bf/end/ command.
+
+<tag>end <it/[type]/</tag>Close a tagged element. If no parameter is given,
+the last element on the stack is terminated. The first parameter, if
+any, is a type name, similar to the <bf/begin/ statement. For the
+<bf/element/ type, a tag name can be provided to terminate a specific tag.
+
+</descrip>
+
+The following input filter reads a Usenet news file, producing a
+record in the WAIS schema. Note that the body of a news posting is
+separated from the list of headers by a blank line (or rather a
+sequence of two newline characters.
+
+<tscreen><verb>
+BEGIN { begin record wais }
+
+/^From:/ BODY /$/ { data -element name $1 }
+/^Subject:/ BODY /$/ { data -element title $1 }
+/^Date:/ BODY /$/ { data -element lastModified $1 }
+/\n\n/ BODY END {
+ begin element bodyOfDisplay
+ begin variant body iana "text/plain"
+ data -text $1
+ end record
+ }
+</verb></tscreen>
+
+If Zebra is compiled with support for Tcl (Tool Command Language)
+enabled, the statements described above are supplemented with a complete
+scripting environment, including control structures (conditional
+expressions and loop constructs), and powerful string manipulation
+mechanisms for modifying the elements of a record. Tcl is a popular
+scripting environment, with several tutorials available both online
+and in hardcopy.
+
+<it>NOTE: Variant support is not currently available in the input
+filter, but will be included with one of the next
+releases.</it>
+
+<sect1>Internal Representation<label id="internal-representation">
-<sect1>Running the server
<p>
-Starting the server and how to configure the server.
+When records are manipulated by the system, they're represented in a
+tree-structure, with data elements at the leaf nodes, and tags or
+variant components at the non-leaf nodes. The root-node identifies the
+schema that lends context to the tagging and structuring of the
+record. Imagine a simple record, consisting of a 'title' element and
+an 'author' element:
+
+<tscreen><verb>
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ROOT
+ AUTHOR "Robert Pirsig"
+</verb></tscreen>
+
+A slightly more complex record would have the author element consist
+of two elements, a surname and a first name:
+
+<tscreen><verb>
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ROOT
+ FIRST-NAME "Robert"
+ AUTHOR
+ SURNAME "Pirsig"
+</verb></tscreen>
+
+The root of the record will refer to the record schema that describes
+the structuring of this particular record. The schema defines the
+element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
+well as the structuring (SURNAME should appear below AUTHOR, etc.). In
+addition, the schema establishes element set names that are used by
+the client to request a subset of the elements of a given record. The
+schema may also establish rules for converting the record to a
+different schema, by stating, for each element, a mapping to a
+different tag path.
+
+<sect2>Tagged Elements
+
+<p>
+A data element is characterized by its tag, and its position in the
+structure of the record. For instance, while the tag &dquot;telephone
+number&dquot; may be used different places in a record, we may need to
+distinguish between these occurrences, both for searching and
+presentation purposes. For instance, while the phone numbers for the
+&dquot;customer&dquot; and the &dquot;service provider&dquot; are both
+representatives for the same type of resource (a telephone number), it
+is essential that they be kept separate. The record schema provides
+the structure of the record, and names each data element (defined by
+the sequence of tags - the tag path - by which the element can be
+reached from the root of the record).
+
+<sect2>Variants
+
+<p>
+The children of a tag node may be either more tag nodes, a data node
+(possibly accompanied by tag nodes),
+or a tree of variant nodes. The children of variant nodes are either
+more variant nodes or a data node (possibly accompanied by more
+variant nodes). Each leaf node, which is normally a
+data node, corresponds to a <it/variant form/ of the tagged element
+identified by the tag which parents the variant tree. The following
+title element occurs in two different languages:
+
+<tscreen><verb>
+ VARIANT LANG=ENG "War and Peace"
+TITLE
+ VARIANT LANG=DAN "Krig og Fred"
+</verb></tscreen>
+
+Which of the two elements are transmitted to the client by the server
+depends on the specifications provided by the client, if any.
+
+In practice, each variant node is associated with a triple of class,
+type, value, corresponding to the variant mechanism of Z39.50.
+
+<sect2>Data Elements
+
+<p>
+Data nodes have no children (they are always leaf nodes in the record
+tree).
+
+<it>NOTE: Documentation needs extension here about types of nodes - numerical,
+textual, etc., plus the various types of inclusion notes.</it>
+
+<sect1>Configuring Your Data Model<label id="data-model">
-<sect1>How the server handles queries
<p>
-What elements of Bib-1 are supported and where are result sets
-stored.
+The following sections describe the configuration files that govern
+the internal management of data records. The system searches for the files
+in the directories specified by the <bf/profilePath/ setting in the
+<tt/zebra.cfg/ file.
-<sect>About Index Data
+<sect2>About Object Identifers
<p>
-Copyright © 1995, Index Data.
+When Object Identifiers (or OID's) need to be specified in the following
+a named OID reference or a raw OID reference may be used. For the named
+OID's refer to the source file <tt>util/oid.c</tt> from YAZ. The raw
+canonical OID's are specified in dot-notation (for example
+1.2.840.10003.3.1000.81.1).
-Permission to use, copy, modify, distribute, and sell this software and
-its documentation, in whole or in part, for any purpose, is hereby granted,
-provided that:
+<sect2>The Abstract Syntax
-1. This copyright and permission notice appear in all copies of the
-software and its documentation. Notices of copyright or attribution
-which appear at the beginning of any file must remain unchanged.
+<p>
+The abstract syntax definition (also known as an Abstract Record
+Structure, or ARS) is the focal point of the
+record schema description. For a given schema, the ABS file may state any
+or all of the following:
+
+<itemize>
+<item>The object identifier of the Z39.50 schema associated
+with the ARS, so that it can be referred to by the client.
+
+<item>The attribute set (which can possibly be a compound of multiple
+sets) which applies in the profile. This is used when indexing and
+searching the records belonging to the given profile.
+
+<item>The Tag set (again, this can consist of several different sets).
+This is used when reading the records from a file, to recognize the
+different tags, and when transmitting the record to the client -
+mapping the tags to their numerical representation, if they are
+known.
+
+<item>The variant set which is used in the profile. This provides a
+vocabulary for specifying the <it/forms/ of data that appear inside
+the records.
+
+<item>Element set names, which are a shorthand way for the client to
+ask for a subset of the data elements contained in a record. Element
+set names, in the retrieval module, are mapped to <it/element
+specifications/, which contain information equivalent to the
+<it/Espec-1/ syntax of Z39.50.
-2. The names of Index Data or the individual authors may not be used to
-endorse or promote products derived from this software without specific
-prior written permission.
+<item>Map tables, which may specify mappings to <it/other/ database
+profiles, if desired.
-THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
-EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
-WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
-IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
-INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES
-WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR
-NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF
-LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
-OF THIS SOFTWARE.
+<item>Possibly, a set of rules describing the mapping of elements to a
+MARC representation.
-<sect>About Index Data
+<item>A list of element descriptions (this is the actual ARS of the
+schema, in Z39.50 terms), which lists the ways in which the various
+tags can be used and organized hierarchically.
+</itemize>
+
+Several of the entries above simply refer to other files, which
+describe the given objects.
+
+<sect2>The Configuration Files
+
+<p>
+This section describes the syntax and use of the various tables which
+are used by the retrieval module.
+
+The number of different file types may appear daunting at first, but
+each type corresponds fairly clearly to a single aspect of the Z39.50
+retrieval facilities. Further, the average database administrator,
+who is simply reusing an existing profile for which tables already
+exist, shouldn't have to worry too much about the contents of these tables.
+
+Generally, the files are simple ASCII files, which can be maintained
+using any text editor. Blank lines, and lines beginning with a (#) are
+ignored. Any characters on a line followed by a (#) are also ignored.
+All other
+lines contain <it/directives/, which provide some setting or value
+to the system. Generally, settings are characterized by a single
+keyword, identifying the setting, followed by a number of parameters.
+Some settings are repeatable (r), while others may occur only once in a
+file. Some settings are optional (o), whicle others again are
+mandatory (m).
+
+<sect2>The Abstract Syntax (.abs) Files
+
+<p>
+The name of this file type is slightly misleading in Z39.50 terms,
+since, apart from the actual abstract syntax of the profile, it also
+includes most of the other definitions that go into a database
+profile.
+
+When a record in the canonical, SGML-like format is read from a file
+or from the database, the first tag of the file should reference the
+profile that governs the layout of the record. If the first tag of the
+record is, say, <tt><gils></tt>, the system will look for the profile
+definition in the file <tt/gils.abs/. Profile definitions are cached,
+so they only have to be read once during the lifespan of the current
+process.
+
+When writing your own input filters, the <bf/record-begin/ command
+introduces the profile, and should always be called first thing when
+introducing a new record.
+
+The file may contain the following directives:
+
+<descrip>
+<tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
+description for the profile. Mostly useful for diagnostic purposes.
+
+<tag>reference <it/OID-name/</tag> (m) The OID for
+the profile (name or dotted-numerical list).
+
+<tag>attset <it/filename/</tag> (m) The attribute set that is used for
+indexing and searching records belonging to this profile.
+
+<tag>tagset <it/filename/ [<it/type/]</tag> (o) The tag
+set (if any) that describe that fields of the records. The type, which
+is optional, specifies the tag type. If not given, the type-specifier
+in the Tag Set files is used.
+
+<tag>varset <it/filename/</tag> (o) The variant set used in the profile.
+
+<tag>maptab <it/filename/</tag> (o,r) This points to a
+conversion table that might be used if the client asks for the record
+in a different schema from the native one.
+
+<tag>marc <it/filename/</tag> (o) Points to a file containing parameters
+for representing the record contents in the ISO2709 syntax. Read the
+description of the MARC representation facility below.
+
+<tag>esetname <it/name filename/</tag> (o,r) Associates the
+given element set name with an element selection file. If an (@) is
+given in place of the filename, this corresponds to a null mapping for
+the given element set name.
+
+<tag>any <it/tags/</tag> (o) This directive specifies a list of
+attributes which should be appended to the attribute list given for each
+element. The effect is to make every single element in the abstract
+syntax searchable by way of the given attributes. This directive
+provides an efficient way of supporting free-text searching across all
+elements. However, it does increase the size of the index
+significantly. The attributes can be qualified with a structure, as in
+the <bf/elm/ directive below.
+
+<tag>elm <it/path name attributes/</tag> (o,r) Adds an element
+to the abstract record syntax of the schema. The <it/path/ follows the
+syntax which is suggested by the Z39.50 document - that is, a sequence
+of tags separated by slashes (/). Each tag is given as a
+comma-separated pair of tag type and -value surrounded by parenthesis.
+The <it/name/ is the name of the element, and the <it/attributes/
+specifies which attributes to use when indexing the element in a
+comma-separated list. A ! in
+place of the attribute name is equivalent to specifying an attribute
+name identical to the element name. A - in place of the attribute name
+specifies that no indexing is to take place for the given element. The
+attributes can be qualified with <it/field types/ to specify which
+character set should govern the indexing procedure for that field. The
+same data element may be indexed into several different fields, using
+different character set definitions. See the section
+<ref id="field structure and character sets"
+name="Field Structure and Character Sets">.
+The default field type is &dquot;w&dquot; for
+<it/word/.
+</descrip>
+
+The following is an excerpt from the abstract syntax file for the GILS
+profile.
+
+<tscreen><verb>
+name gils
+reference GILS-schema
+attset gils.att
+tagset gils.tag
+varset var1.var
+
+maptab gils-usmarc.map
+
+# Element set names
+
+esetname VARIANT gils-variant.est # for WAIS-compliance
+esetname B gils-b.est
+esetname G gils-g.est
+esetname F @
+
+elm (1,10) rank -
+elm (1,12) url -
+elm (1,14) localControlNumber Local-number
+elm (1,16) dateOfLastModification Date/time-last-modified
+elm (2,1) Title w:!,p:!
+elm (4,1) controlIdentifier Identifier-standard
+elm (2,6) abstract Abstract
+elm (4,51) purpose !
+elm (4,52) originator -
+elm (4,53) accessConstraints !
+elm (4,54) useConstraints !
+elm (4,70) availability -
+elm (4,70)/(4,90) distributor -
+elm (4,70)/(4,90)/(2,7) distributorName !
+elm (4,70)/(4,90)/(2,10 distributorOrganization !
+elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
+elm (4,70)/(4,90)/(4,3) distributorCity !
+</verb></tscreen>
+
+<sect2>The Attribute Set (.att) Files<label id="attset-files">
+
+<p>
+This file type describes the <bf/Use/ elements of an attribute set.
+It contains the following directives.
+
+<descrip>
+
+<tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
+description for the attribute set. Mostly useful for diagnostic purposes.
+
+<tag>reference <it/OID-name/</tag> (m) The reference name of the OID for
+the attribute set.
+
+<tag>include <it/filename/</tag> (o,r) This directive is used to
+include another attribute set as a part of the current one. This is
+used when a new attribute set is defined as an extension to another
+set. For instance, many new attribute sets are defined as extensions
+to the <bf/bib-1/ set. This is an important feature of the retrieval
+system of Z39.50, as it ensures the highest possible level of
+interoperability, as those access points of your database which are
+derived from the external set (say, bib-1) can be used even by clients
+who are unaware of the new set.
+
+<tag>att <it/att-value att-name [local-value]/</tag> (o,r) This
+repeatable directive introduces a new attribute to the set. The
+attribute value is stored in the index (unless a <it/local-value/ is
+given, in which case this is stored). The name is used to refer to the
+attribute from the <it/abstract syntax/. </descrip>
+
+This is an excerpt from the GILS attribute set definition. Notice how
+the file describing the <it/bib-1/ attribute set is referenced.
+
+<tscreen><verb>
+name gils
+reference GILS-attset
+include bib1.att
+
+att 2001 distributorName
+att 2002 indexTermsControlled
+att 2003 purpose
+att 2004 accessConstraints
+att 2005 useConstraints
+</verb></tscreen>
+
+<sect2>The Tag Set (.tag) Files
+
+<p>
+This file type defines the tagset of the profile, possibly by
+referencing other tag sets (most tag sets, for instance, will include
+tagsetG and tagsetM from the Z39.50 specification. The file may
+contain the following directives.
+
+<descrip>
+<tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
+description for the tag set. Mostly useful for diagnostic purposes.
+
+<tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
+the tag set. The directive is optional, since not all tag sets are
+registered outside of their schema.
+
+<tag>type <it/integer/</tag> (m) The type number of the tagset within the schema
+profile (note: this specification really should belong to the .abs
+file. This will be fixed in a future release).
+
+<tag>include <it/filename/</tag> (o,r) This directive is used
+to include the definitions of other tag sets into the current one.
+
+<tag>tag <it/number names type/</tag> (o,r) Introduces a new
+tag to the set. The <it/number/ is the tag number as used in the protocol
+(there is currently no mechanism for specifying string tags at this
+point, but this would be quick work to add). The <it/names/ parameter
+is a list of names by which the tag should be recognized in the input
+file format. The names should be separated by slashes (/). The
+<it/type/ is th recommended datatype of the tag. It should be one of
+the following:
+<itemize>
+<item>structured
+<item>string
+<item>numeric
+<item>bool
+<item>oid
+<item>generalizedtime
+<item>intunit
+<item>int
+<item>octetstring
+<item>null
+</itemize>
+</descrip>
+
+The following is an excerpt from the TagsetG definition file.
+
+<tscreen><verb>
+name tagsetg
+reference TagsetG
+type 2
+
+tag 1 title string
+tag 2 author string
+tag 3 publicationPlace string
+tag 4 publicationDate string
+tag 5 documentId string
+tag 6 abstract string
+tag 7 name string
+tag 8 date generalizedtime
+tag 9 bodyOfDisplay string
+tag 10 organization string
+</verb></tscreen>
+
+<sect2>The Variant Set (.var) Files<label id="variant-set">
+
+<p>
+The variant set file is a straightforward representation of the
+variant set definitions associated with the protocol. At present, only
+the <it/Variant-1/ set is known.
+
+These are the directives allowed in the file.
+
+<descrip>
+<tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
+description for the variant set. Mostly useful for diagnostic purposes.
+
+<tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
+the variant set, if one is required.
+
+<tag>class <it/integer class-name/</tag> (m,r) Introduces a new
+class to the variant set.
+
+<tag>type <it/integer type-name datatype/</tag> (m,r) Addes a
+new type to the current class (the one introduced by the most recent
+<bf/class/ directive). The type names belong to the same name space as
+the one used in the tag set definition file.
+</descrip>
+
+The following is an excerpt from the file describing the variant set
+<it/Variant-1/.
+
+<tscreen><verb>
+name variant-1
+reference Variant-1
+
+class 1 variantId
+
+ type 1 variantId octetstring
+
+class 2 body
+
+ type 1 iana string
+ type 2 z39.50 string
+ type 3 other string
+</verb></tscreen>
+
+<sect2>The Element Set (.est) Files
+
+<p>
+The element set specification files describe a selection of a subset
+of the elements of a database record. The element selection mechanism
+is equivalent to the one supplied by the <it/Espec-1/ syntax of the
+Z39.50 specification. In fact, the internal representation of an
+element set specification is identical to the <it/Espec-1/ structure,
+and we'll refer you to the description of that structure for most of
+the detailed semantics of the directives below.
+
+<it>
+NOTE: Not all of the Espec-1 functionality has been implemented yet.
+The fields that are mentioned below all work as expected, unless
+otherwise is noted.
+</it>
+
+The directives available in the element set file are as follows:
+
+<descrip>
+<tag>defaultVariantSetId <it/OID-name/</tag> (o) If variants are used in
+the following, this should provide the name of the variantset used
+(it's not currently possible to specify a different set in the
+individual variant request). In almost all cases (certainly all
+profiles known to us), the name <tt/Variant-1/ should be given here.
+
+<tag>defaultVariantRequest <it/variant-request/</tag> (o) This directive
+provides a default variant request for
+use when the individual element requests (see below) do not contain a
+variant request. Variant requests consist of a blank-separated list of
+variant components. A variant compont is a comma-separated,
+parenthesized triple of variant class, type, and value (the two former
+values being represented as integers). The value can currently only be
+entered as a string (this will change to depend on the definition of
+the variant in question). The special value (@) is interpreted as a
+null value, however.
+
+<tag>simpleElement <it/path ['variant' variant-request]/</tag>
+(o,r) This corresponds to a simple element request in <it/Espec-1/. The
+path consists of a sequence of tag-selectors, where each of these can
+consist of either:
+
+<itemize>
+<item>A simple tag, consisting of a comma-separated type-value pair in
+parenthesis, possibly followed by a colon (:) followed by an
+occurrences-specification (see below). The tag-value can be a number
+or a string. If the first character is an apostrophe ('), this forces
+the value to be interpreted as a string, even if it appears to be numerical.
+
+<item>A WildThing, represented as a question mark (?), possibly
+followed by a colon (:) followed by an occurrences specification (see
+below).
+
+<item>A WildPath, represented as an asterisk (*). Note that the last
+element of the path should not be a wildPath (wildpaths don't work in
+this version).
+</itemize>
+
+The occurrences-specification can be either the string <tt/all/, the
+string <tt/last/, or an explicit value-range. The value-range is
+represented as an integer (the starting point), possibly followed by a
+plus (+) and a second integer (the number of elements, default being
+one).
+
+The variant-request has the same syntax as the defaultVariantRequest
+above. Note that it may sometimes be useful to give an empty variant
+request, simply to disable the default for a specific set of fields
+(we aren't certain if this is proper <it/Espec-1/, but it works in
+this implementation).
+</descrip>
+
+The following is an example of an element specification belonging to
+the GILS profile.
+
+<tscreen><verb>
+simpleelement (1,10)
+simpleelement (1,12)
+simpleelement (2,1)
+simpleelement (1,14)
+simpleelement (4,1)
+simpleelement (4,52)
+</verb></tscreen>
+
+<sect2>The Schema Mapping (.map) Files<label id="schema-mapping">
+
+<p>
+Sometimes, the client might want to receive a database record in
+a schema that differs from the native schema of the record. For
+instance, a client might only know how to process WAIS records, while
+the database record is represented in a more specific schema, such as
+GILS. In this module, a mapping of data to one of the MARC formats is
+also thought of as a schema mapping (mapping the elements of the
+record into fields consistent with the given MARC specification, prior
+to actually converting the data to the ISO2709). This use of the
+object identifier for USMARC as a schema identifier represents an
+overloading of the OID which might not be entirely proper. However,
+it represents the dual role of schema and record syntax which
+is assumed by the MARC family in Z39.50.
+
+<it>
+NOTE: The schema-mapping functions are so far limited to a
+straightforward mapping of elements. This should be extended with
+mechanisms for conversions of the element contents, and conditional
+mappings of elements based on the record contents.
+</it>
+
+These are the directives of the schema mapping file format:
+
+<descrip>
+<tag>targetName <it/name/</tag> (m) A symbolic name for the target schema
+of the table. Useful mostly for diagnostic purposes.
+
+<tag>targetRef <it/OID-name/</tag> (m) An OID name for the target schema.
+This is used, for instance, by a server receiving a request to present
+a record in a different schema from the native one.
+
+<tag>map <it/element-name target-path/</tag> (o,r) Adds
+an element mapping rule to the table.
+</descrip>
+
+<sect2>The MARC (ISO2709) Representation (.mar) Files
+
+<p>
+This file provides rules for representing a record in the ISO2709
+format. The rules pertain mostly to the values of the constant-length
+header of the record.
+
+<it>NOTE: This will be described better. We're in the process of
+re-evaluating and most likely changing the way that MARC records are
+handled by the system.</it>
+
+<sect2>Field Structure and Character Sets
+<label id="field structure and character sets">
+
+<p>
+In order to provide a flexible approach to national character set
+handling, Zebra allows the administrator to configure the set up the
+system to handle any 8-bit character set — including sets that
+require multi-octet diacritics or other multi-octet characters. The
+definition of a character set includes a specification of the
+permissible values, their sort order (this affects the display in the
+SCAN function), and relationships between upper- and lowercase
+characters. Finally, the definition includes the specification of
+space characters for the set.
+
+The operator can define different character sets for different fields,
+typical examples being standard text fields, numerical fields, and
+special-purpose fields such as WWW-style linkages (URx).
+
+The field types, and hence character sets, are associated with data
+elements by the .abs files (see above). The file <tt/default.idx/
+provides the association between field type codes (as used in the .abs
+files) and the character map files (with the .chr suffix). The format
+of the .idx file is as follows
+
+<descrip>
+<tag>index <it/field type code/</tag>This directive introduces a new
+search index code. The argument is a one-character code to be used in the
+.abs files to select this particular index type. An index, roughly,
+corresponds to a particular structure attribute during search. Refer
+to section <ref id="search" name="Search">.
+
+<tag>sort <it/field code type/</tag>This directive introduces a
+sort index. The argument is a one-character code to be used in the
+.abs fie to select this particular index type. The corresponding
+use attribute must be used in the sort request to refer to this
+particular sort index. The corresponding character map (see below)
+is used in the sort process.
+
+<tag>completeness <it/boolean/</tag>This directive enables or disables
+complete field indexing. The value of the <it/boolean/ should be 0
+(disable) or 1. If completeness is enabled, the index entry will
+contain the complete contents of the field (up to a limit), with words
+(non-space characters) separated by single space characters
+(normalized to &dquot; &dquot; on display). When completeness is
+disabled, each word is indexed as a separate entry. Complete subfield
+indexing is most useful for fields which are typically browsed (eg.
+titles, authors, or subjects), or instances where a match on a
+complete subfield is essential (eg. exact title searching). For fields
+where completeness is disabled, the search engine will interpret a
+search containing space characters as a word proximity search.
+
+<tag>charmap <it/filename/</tag> This is the filename of the character
+map to be used for this index for field type.
+</descrip>
+
+The contents of the character map files are structured as follows:
+
+<descrip>
+<tag>lowercase <it/value-set/</tag>This directive introduces the basic
+value set of the field type. The format is an ordered list (without
+spaces) of the characters which may occur in &dquot;words&dquot; of
+the given type. The order of the entries in the list determines the
+sort order of the index. In addition to single characters, the
+following combinations are legal:
+
+<itemize>
+<item>Backslashes may be used to introduce three-digit octal, or
+two-digit hex representations of single characters (preceded by <tt/x/).
+In addition, the combinations
+\\, \\r, \\n, \\t, \\s (space — remember that real space-characters
+may ot occur in the value definition), and \\ are recognised,
+with their usual interpretation.
+
+<item>Curly braces {} may be used to enclose ranges of single
+characters (possibly using the escape convention described in the
+preceding point), eg. {a-z} to entroduce the standard range of ASCII
+characters. Note that the interpretation of such a range depends on
+the concrete representation in your local, physical character set.
+
+<item>Paranthesises () may be used to enclose multi-byte characters -
+eg. diacritics or special national combinations (eg. Spanish
+&dquot;ll&dquot;). When found in the input stream (or a search term),
+these characters are viewed and sorted as a single character, with a
+sorting value depending on the position of the group in the value
+statement.
+</itemize>
+
+<tag>uppercase <it/value-set/</tag>This directive introduces the
+upper-case equivalencis to the value set (if any). The number and
+order of the entries in the list should be the same as in the
+<tt/lowercase/ directive.
+
+<tag>space <it/value-set/</tag>This directive introduces the character
+which separate words in the input stream. Depending on the
+completeness mode of the field in question, these characters either
+terminate an index entry, or delimit individual &dquot;words&dquot; in
+the input stream. The order of the elements is not significant —
+otherwise the representation is the same as for the <tt/upercase/ and
+<tt/lowercase/ directives.
+
+<tag>map <it/value-set/ <it/target/</tag>This directive introduces a
+mapping between each of the members of the value-set on the left to
+the character on the right. The character on the right must occur in
+the value set (the <tt/lowercase/ directive) of the character set, but
+it may be a paranthesis-enclosed multi-octet character. This directive
+may be used to map diacritics to their base characters, or to map
+HTML-style character-representations to their natural form, etc.
+</descrip>
+
+<sect1>Exchange Formats
+
+<p>
+Converting records from the internal structure to en exchange format
+is largely an automatic process. Currently, the following exchange
+formats are supported:
+
+<itemize>
+<item>GRS-1. The internal representation is based on GRS-1, so the
+conversion here is straightforward. The system will create
+applied variant and supported variant lists as required, if a record
+contains variant information.
+
+<item>SUTRS. Again, the mapping is fairly straighforward. Indentation
+is used to show the hierarchical structure of the record. All
+&dquot;GRS&dquot; type records support both the GRS-1 and SUTRS
+representations.
+
+<item>ISO2709-based formats (USMARC, etc.). Only records with a
+two-level structure (corresponding to fields and subfields) can be
+directly mapped to ISO2709. For records with a different structuring
+(eg., GILS), the representation in a structure like USMARC involves a
+schema-mapping (see section <ref id="schema-mapping" name="Schema
+Mapping">), to an &dquot;implied&dquot; USMARC schema (implied,
+because there is no formal schema which specifies the use of the
+USMARC fields outside of ISO2709). The resultant, two-level record is
+then mapped directly from the internal representation to ISO2709. See
+the GILS schema definition files for a detailed example of this
+approach.
+
+<item>Explain. This representation is only available for records
+belonging to the Explain schema.
+
+<item>Summary. This ASN-1 based structure is only available for records
+belonging to the Summary schema - or schema which provide a mapping
+to this schema (see the description of the schema mapping facility
+above).
+
+<item>SOIF. Support for this syntax is experimental, and is currently
+keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
+abstract syntaxes can be mapped to the SOIF format, although nested
+elements are represented by concatenation of the tag names at each
+level.
+
+<item>XML. The use of XML as a transfer syntax in Z39.50 is not yet widely established
+so the use of it here must be characterised as somewhat experimental. The
+tag-names used are taken from the tag-set in use, except for local string tags
+where the tag itself is passed through unchanged.
+
+</itemize>
+
+<sect>License
+
+<p>
+Zebra
+Copyright (c) 1995-2000 Index Data ApS.
+
+All rights reserved.
+
+Use and redistribution in source or binary form, with or without
+modification, of any or all of this software and documentation is
+permitted, provided that the following Conditions 1 to 6 set out below
+are met.
+
+1. Unless prior specific written permission is obtained this copyright
+and permission notice appear with all copies of the software and its
+documentation. Notices of copyright or attribution which appear at the
+beginning of any file must remain unchanged.
+
+2. The names of Index Data or the individual authors may not be used
+to endorse or promote products derived from this software without
+specific prior written permission.
+
+3. Source code or binary versions of this software and its documentation
+may be used freely in not for profit applications limited to databases
+of 100,000 records maximum. Other applications - such as publishing over
+100,000 records, providing for-pay services, distributing a product based
+in whole or in part on this software or its documentation, or generally
+distributing this software or its documentation under a different license
+require a commercial license from Index Data.
+
+4. The software may be installed and used for evaluation purposes in
+conjunction with such commercially licensed applications for a trial
+period no longer than 60 days.
+
+5. Unless a prior specific written agreement is obtained THIS SOFTWARE
+IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND, EXPRESS, IMPLIED,
+OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY WARRANTY OF
+MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL
+INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL, INDIRECT OR
+CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES WHATSOEVER RESULTING
+FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR NOT ADVISED OF THE
+POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF LIABILITY, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+6. Commercial licenses and support agreements for Zebra and related
+Index Data products such as Z'bol (c) - and written agreements
+relating to these Conditions may be obtained only from Index Data
+or its appointed agents as follows:
+
+Index Data: www.indexdata.dk
+Fretwell-Downing Informatics: www.fdgroup.co.uk
+Fretwell-Downing Informatics USA: www.fdi.com
+
+<sect>About Index Data and the Zebra Server
<p>
Index Data is a consulting and software-development enterprise that
-specialises in library and information management systems. Our
+specialises in information management and retrieval applications. Our
interests and expertise span a broad range of related fields, and one
of our primary, long-term objectives is the development of a powerful
information management
-system with open network interfaces and hypermedia capabilities.
+system with open network interfaces and hypermedia capabilities. Zebra is an
+important component in this strategy.
+
+We make this software available free of charge for not-for-profit
+purposes, as a service to the networking community, and to further
+the development and use of quality software for open network
+communication. We encourage your comments and questions if you have ideas, things
+you would like to see in future versions, or things you would like to
+contribute.
+
+If you like this software, and would like to use all or part of it in
+a commercial product, or to provide a commercial database service,
+please contact us. The Z'mbol Information System represents the commercial
+variant of Zebra. It includes full support; additional functionality and
+performance-boosting features, and it has what we think is a very exciting
+development path.
-We make this software available free of charge, on a fairly unrestrictive
-license; as a service to the networking community, and to further the
-development of quality software for open network communication.
-
-We'll be happy to answer questions about the software, and about ourselves
-in general.
-
-<tscreen>
-Index Data&nl
-Ryesgade 3&nl
-2200 København N&nl
-DENMARK
-</tscreen>
+<tscreen><verb>
+Index Data
+Ryesgade 3
+DK-2200 Copenhagen N
+</verb></tscreen>
<p>
<tscreen><verb>
Phone: +45 3536 3672
Fax : +45 3536 0449
-Email: info@index.ping.dk
+Email: info@indexdata.dk
</verb></tscreen>
-<sect>References
+The <it>Random House College Dictionary</it>, 1975 edition
+offers this definition of the
+word &dquot;Zebra&dquot;:
-<p>
+<it>
+Zebra, n., any of several horselike, African mammals of the genus Equus,
+having a characteristic pattern of black or dark-brown stripes on
+a whitish background.
+</it>
</article>
projects / idzebra-moved-to-github.git / blobdiff