X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Fzebra.sgml;h=0317e2ce2eaf30ca77afaad46bc26919b7a2bd2c;hb=c8bb72d81cc3496fdfc7143e6fa5216fdb1a60f9;hp=d512faa4b23ef031cf296e9c41619922c5fd13a1;hpb=5c743078c8796354d7ff0593cec2e5631d1dc350;p=idzebra-moved-to-github.git diff --git a/doc/zebra.sgml b/doc/zebra.sgml index d512faa..0317e2c 100644 --- a/doc/zebra.sgml +++ b/doc/zebra.sgml @@ -1,454 +1,2002 @@
-Zebra server Administrators's Guide and Reference -<author>Index Data, <tt/info@index.ping.dk/ -<date>$Revision: 1.2 $ +<title>Zebra Server - Administrators's Guide and Reference +<author><htmlurl url="http://www.indexdata.dk/" name="Index Data">, +<tt><htmlurl url="mailto:info@indexdata.dk" name="info@indexdata.dk"></> +<date>$Revision: 1.47 $ <abstract> -This document describes Zebra — an information retrieval server -that uses the Z39.50 protocol. + + +The Zebra server combines a versatile fielded/free-text +indexing/search engine with a Z39.50-1995 frontend to provide a powerful and flexible +information mining tool. This document explains the procedure for +installing and configuring Zebra, and outlines the possibilities +for managing data and providing Z39.50 +services with the software. Zebra is a free version of the Index Data Z'mbol +information system, and it excludes some functionality such as incremental +database updating and support for large databases. </abstract> <toc> <sect>Introduction +<sect1>Overview + <p> -- gils -- UNI*X (not NT yet) -- goals +Zebra is a fielded free-text indexing and retrieval engine with a +Z39.50 frontend. You can use any commercial or freeware Z39.50 client +to access data stored in Zebra. + +Zebra server can be used at the core of a Z39.50-based information retrieval +framework. We're making +the server available now to allow researchers and small organisations to +share their information in the best possible way. We believe that Z39.50 +currently represents one of the best ways of sharing information with others, and +we would like to encourage as many people as possible to do so. +This document is a guide to using Zebra. It will tell you +how to compile the software, and how to prepare your first database. +It also explains how the server can be configured to give you the +functionality that you need. + +If you find the software interesting, you should join the support +mailing-list by sending email to <tt/zebra-request@indexdata.dk/. + +If you are interested in running a commercial service, if you wish to run large +databases, or if you wish to make incremental updates to your databases even +while users are accessing your system, then you might be interested in the Z'mbol +Information Server which is available from <htmlurl +url="http://www.indexdata.dk/zmbol/" name="Index Data"> or Fretwell-Downing +Informatics. Z'mbol is a complete and supported package which offers many +exciting possibilities that we have not been able to fit into this package. + +<sect1>Features -<sect>Compiling the software +<p> +This is a list of some of the most important features of the +system. + +<itemize> + +<item> +Supports arbitrarily complex records - base input format is an +XML-like syntax which allows nested (structured) data elements, as +well as variant forms of data. + +<item> +Supports random storage formats. A system of input filters driven by +regular expressions allows you to easily process most ASCII-based +data formats. SGML/XML, ISO2709 (MARC), and raw text are also supported. + +<item> +Supports boolean queries as well as relevance-ranking (free-text) +searching. Right truncation and masking in terms are supported, as +well as full regular expressions. + +<item> +Supports multiple concrete syntaxes +for record exchange (depending on the configuration): GRS-1, SUTRS, +ISO2709 (*MARC), XML. Records can be mapped between record syntaxes and +schema on the fly. + +<item> +Supports approximate matching in registers (ie. spelling mistakes, +etc). + +<item> Supports a subset of the Z39.50 Explain Facility. Zebra's Explain database +is automatically updated when a set of records is loaded into Zebra. + +</itemize> <p> +Protocol support: + +<itemize> + +<item> +Protocol facilities: Init, Search, Retrieve, Browse, Sort, Close, and Explain. + +<item> +Piggy-backed presents are honored in the search-request. + +<item> +Named result sets are supported. -Zebra uses the YAZ package in order to provide Z39.50 access. So you -must compile YAZ before going further. Specifically, Zebra uses -the YAZ header files in <tt>yaz/include/..</tt> and its public library -<tt>yaz/lib/libyaz.a</tt>. +<item> +Easily configured to support different application profiles, with +tables for attribute sets, tag sets, and abstract syntaxes. +Additional tables control facilities such as element mappings to +different schema (eg., GILS-to-USMARC). -As with YAZ, an ANSI C compiler is required in order to compile the Zebra -server system — GNU C works fine. +<item> +Complex composition specifications using Espec-1 are partially +supported (simple element requests only). -Unpack the Zebra software. You might put Zebra in the same directory level -as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-</tt>.., then -Zebra is placed in ..<tt>/src/zebra-</tt>. +<item> +Element Set Names are defined using the Espec-1 capability of the +system, and are given in configuration files as simple element +requests (and possibly variant requests). + +<item> +Zebra runs on most Unix-like systems as well as Windows NT - a binary +distribution for Windows NT is forthcoming - so far, the installation +requires Microsoft Visual C++ to compile the system (we use version 6.0). + +</itemize> + +<sect1>Future Work + +<p> -Edit the top-level <tt>Makefile</tt> in the Zebra directory in which -you specify the location of YAZ by setting make variables. -The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with -MOSI support. Some systems, such as Solaris, have separate socket -libraries and for those systems you need to specify the -<tt>NETLIB</tt> variable. +These are some of the plans that we have for the software in the near +and far future, approximately ordered after their relative importance. +Items marked with an +asterisk will be implemented before the +last beta release. -When finished editing the <tt>Makefile</tt> type: +<itemize> + +<item> +*Complete the support for variants. + +<item> +*Finalize the data element <it/include/ facility to support multimedia +data elements in records. + +<item> +Add more sophisticated relevance ranking mechanisms. Add support for soundex +and stemming. Add relevance <it/feedback/ support. + +<item> +Complete EXPLAIN support. + +<item> +We want to add a management system that allows you to +control your databases and configuration tables from a graphical +interface. We'll probably use Tcl/Tk to stay platform-independent. + +</itemize> + +Programmers thrive on user feedback. If you are interested in a facility that +you don't see mentioned here, or if there's something you think we +could do better, please drop us a mail. If you think it's all really +neat, you're welcome to drop us a line saying that, too. You'll find +contact info at the end of this file. + +<sect>Compiling the software +<p> +You need the +<bf><htmlurl url="http://www.indexdata.dk/yaz/" name="YAZ"></> +package in order to compile this software. We suggest you +unpack <bf/YAZ/ in the same directory as Zebra. Running +./configure (UNIX Only) and running make (nmake on WIN32) is +in usully what it takes to compile YAZ. + +<sect1>UNIX +<p> +An ANSI C compiler is required to compile the Zebra +server system — <tt/gcc/ works very well if your own system doesn't +provide an adequate compiler. + +Unpack the distribution archive. The <tt>configure</tt> shell script +attempts to guess correct values for various system-dependent variables +used during compilation. It uses those values to create a 'Makefile' in +each directory of Zebra. + +To run the configure script type: +<tscreen><verb> + ./configure +</verb></tscreen> + +The configure script attempts to use the C compiler specified by +the <tt>CC</tt> environment variable. If not set, GNU C +will be used if it is available. The <tt>CFLAGS</tt> environment variable +holds options to be passed to the C compiler. If you're using a +Bourne-compatible shell you may pass something like this: <tscreen><verb> -$ make + CC=/opt/ccs/bin/cc CFLAGS=-O ./configure </verb></tscreen> +To customize Zebra the configure script accepts a set of options. The +most important are +<descrip> +<tag><tt>-</tt><tt>-prefix </tt>path</tag> Specifies installation prefix. This is +only needed if you run <tt>make install</tt> later to perform a +"system" installation. The prefix is <tt>/usr/local</tt> if not +specified. +<tag><tt>-</tt><tt>-with-tclconfig=</tt>DIR</tag> If Tcl is installed on +the system you can tell configure in which directory Tcl's +<tt>tclConfig.sh</tt> is stored. The <tt>tclConfig.sh</tt> include +information about settings required to link with Tcl's libraries. +If you don't specify this option, configure will see if Tcl's shell +<tt>tclsh</tt> is in your path and if it is, it will guess where +the equivalent tclConfig.sh is located. If tclsh is not found in +your path and this option is not given Zebra will not include Tcl support. +<tag><tt>-</tt><tt>-with-yazconfig=</tt>DIR</tag> This options allows you to +specify the directory that contains YAZ's <tt>yaz-config</tt>. +This options is useful if you wish to compile Zebra with a specific +version of YAZ. YAZ version 1.5 and later creates a script +<tt>yaz-config</tt> that includes information on compiler settings +needed to link with it. +</descrip> + +When configured build the software by typing: +<tscreen><verb> + make +</verb></tscreen> + +As an option you may type <tt>make depend</tt> to create +source file dependencies for the package. This is only needed, +however, if you modify the source code later. + If successful, two executables have been created in the sub-directory -index. +<tt>bin</tt>. <descrip> <tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine. <tag><tt>zebraidx</tt></tag> The administrative tool for the search index. </descrip> -<sect>Quick getting started +<p> +The next step is optional and is only needed if you wish to install +zebra in system directories such as /usr/bin, /usr/lib, etc. + +To perform this step, type +<tscreen><verb> + make install +</verb></tscreen> + +The executables will be installed in prefix/bin, and profile +tables will be installed in prefix/lib/zebra/tab. Here prefix +represents the prefix as specified -- default being /usr/local. + +<sect1>WIN32 <p> -This section will get you started quickly! We will try to index a few -GILS records that are included with the Zebra distribution. Go to the -<tt>test</tt> subdirectory. There you will find a configuration +Zebra is shipped with "makefiles" for the NMAKE tool that comes +with Visual C++. + +Start an MS-DOS prompt and switch the sub directory <tt>WIN</tt> where +the file <tt>makefile</tt> is located. Customize the installation +by editing the <tt>makefile</tt> file (for example by using wordpad). + +The following summarises the most important settings in that file. + +<descrip> +<tag><tt>YAZDIR</tt></tag> Specifies where YAZ is located. +<tag><tt>DEBUG</tt></tag> If set to 1, the software is +compiled with debugging libraries. If set to 0, the software +is compiled with release (non-debugging) libraries. +<tag>BZIP2</tag> A group of settings (<tt>BZIP2LIB</tt>,..) +that must be defined if BZIP2 compression support is desired. +</descrip> + +When satisfied with the settings in the makefile type +<tscreen><verb> +nmake +</verb></tscreen> + +If compilation was successful the executables <tt>zebraidx.exe</tt> +and <tt>zebrasrv.exe</tt> are put in the sub directory <tt>BIN</tt>. + +<sect>Quick Start +<p> +In this section, we will test the system by indexing a small set of sample +GILS records that are included with the software distribution. Go to the +<tt>test/gils</tt> subdirectory of the distribution archive. There you will +find a configuration file named <tt>zebra.cfg</tt> with the following contents: <tscreen><verb> -# Where are the YAZ tables located. -profilePath: /usr/local/yaz +# Where the schema files, attribute files, etc. are located. +profilePath: .:../../tab:../../../yaz/tab # Files that describe the attribute sets supported. +attset: explain.att attset: bib1.att attset: gils.att </verb></tscreen> Now, edit the file and set <tt>profilePath</tt> to the path of the -YAZ profile tables (sub directory <tt>tab</tt> of YAZ). +YAZ profile tables (sub directory <tt>tab</tt> of the YAZ distribution +archive). The 48 test records are located in the sub directory <tt>records</tt>. To index these, type: <tscreen><verb> -$ ../index/zebraidx -t grs update records +$ ../../bin/zebraidx -t grs.sgml update records </verb></tscreen> In the command above the option <tt>-t</tt> specified the record -type — in this case <tt>grs</tt>. The word <tt>update</tt> followed -by a directory root updates all files below that directory. +type — in this case <tt>grs.sgml</tt>. The word <tt>update</tt> followed +by a directory root updates all files below that directory node. -If your indexing command went successful, you are now ready to -setup a server. To start a server on port 2100, type: +If your indexing command was successful, you are now ready to +fire up a server. To start a server on port 2100, type: <tscreen><verb> -$ ../index/zebrasrv tcp:@:2100 +$ ../../bin/zebrasrv tcp:@:2100 </verb></tscreen> -The Zebra server just made has one database called Default. It will -return either USMARC/GRS or SUTRS depending on your client. +The Zebra index that you have just created has a single database +named <tt/Default/. The database contains records structured according to +the GILS profile, and the server will +return records in either either XML, USMARC, GRS-1, or SUTRS depending +on what your client asks for. -<sect>Administrating Zebra +To test the server, you can use any Z39.50 client (1992 or later). For +instance, you can use the demo client that comes with YAZ: Just cd to +the <tt/client/ subdirectory of the YAZ distribution and type: -<p> +<tscreen><verb> +$ ./yaz-client tcp:localhost:2100 +</verb></tscreen> -Unlike many other retrieval systems, Zebra offers incremental -modifications of an existing index. Needless to say, these facilities -makes the administration of Zebra somewhat more complicated. +When the client has connected, you can type: -Normally, when Zebra modifies the index it reads a number of records -that you specify. -Depending on your specifications and on the contents of each record -one the following scenarios occur: -<descrip> -<tag>Insert</tag> The record is indexed as if it never occurred -before. Either the Zebra system doesn't know how to identify the record or -Zebra can identify the record but didn't find it to be already indexed. -<tag>Modify</tag> The record has already been indexed. In this case -either the contents of the record or the location (file) of the record -indicates that it has been indexed before. -<tag>Delete</tag> The record is deleted from the index. As in the -update-case it must be able to identify the record. -</descrip> +<tscreen><verb> +Z> find surficial +Z> show 1 +</verb></tscreen> -Please note, that in both the modify- and delete- case the Zebra -system must be able to make a key that identifies the record in -question. +The default retrieval syntax for the client is USMARC. To try other +formats for the same record, try: -To administrate the Zebra retrieval system, you run the +<tscreen><verb> +Z>format sutrs +Z>show 1 +Z>format grs-1 +Z>show 1 +Z>format xml +Z>show 1 +Z>elements B +Z>show 1 +</verb></tscreen> + +<it>NOTE: You may notice that more fields are returned when your +client requests SUTRS or GRS-1 records. When retrieving GILS records, +this is normal - not all of the GILS data elements have mappings in +the USMARC record format.</it> + +If you've made it this far, there's a good chance that +you've got through the compilation OK. + +<sect>Administrating Zebra<label id="administrating"> + +<p> + +To administrate Zebra, you run the <tt>zebraidx</tt> program. This program supports a number of options which are preceded by a minus, and a few commands (not preceded by minus). Both the Zebra administrative tool and the Z39.50 server share a -set of indexing files and a global configuration file. The +set of index files and a global configuration file. The name of the configuration file defaults to <tt>zebra.cfg</tt>. The configuration file includes specifications on how to index various kinds of records and where the other configuration files are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em> -be run in the same directory where the configuration file if you do -not indicate the location of the configuration file by option +be run in the directory where the configuration file lives unless you +indicate the location of the configuration file by option <tt>-c</tt>. -<sect1>Record types +<sect1>Record Types<label id="record-types"> <p> -Indexing is a record-per-record process, in which -either insert/modify/delete will occur. Before a record is indexed -search keys are extracted from whatever might be the layout the -original record (sgml,html,text, etc..). The Zebra system -currently only support GILS records and simple text records. +Indexing is a per-record process. Before a record is indexed search +keys are extracted from whatever might be the layout the original +record (sgml,html,text, etc..). +The Zebra system currently supports two fundamantal types of records: +structured and simple text. To specify a particular extraction process, use either the command line option <tt>-t</tt> or specify a <tt>recordType</tt> setting in the configuration file. -<sect1>The Zebra configuration file +<sect1>The Zebra Configuration File<label id="configuration-file"> <p> The Zebra configuration file, read by <tt>zebraidx</tt> and <tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified by <tt>-c</tt> option. You can edit the configuration file with a normal text editor. -Setting names and values are seperated by colons in the file. Lines -starting with a hash sign (<tt/#/) are treated as comments. +Parameter names and values are seperated by colons in the file. Lines +starting with a hash sign (<tt/#/) are treated as comments. -A set of records that share common characteristics are called a group. +If you manage different sets of records that share common +characteristics, you can organize the configuration settings for each +type into &dquot;groups&dquot;. When <tt>zebraidx</tt> is run and you wish to address a given group -you specify that group with the <tt>-g</tt> option. In this case +you specify the group name with the <tt>-g</tt> option. In this case settings that have the group name as their prefix will be used -by <tt>zebraidx</tt> and not default values. +by <tt>zebraidx</tt>. If no <tt/-g/ option is specified, the settings +with no prefix are used. + +In the configuration file, the group name is placed before the option +name itself, separated by a dot (.). For instance, to set the record type +for group <tt/public/ to <tt/grs.sgml/ (the SGML-like format for structured +records) you would write: -The group is written before the option itself separated by a dot. -For example to set the record type for group <tt/public/ to <tt/grs/ -you would write: <tscreen><verb> -public.recordType: grs +public.recordType: grs.sgml </verb></tscreen> -To set the default value of the record type to text write: +To set the default value of the record type to <tt/text/ write: + <tscreen><verb> recordType: text </verb></tscreen> -The configuration settings are summarized below. They will be -explained further in the following chapters. +The available configuration settings are summarized below. They will be +explained further in the following sections. <descrip> -<tag><it>group</it>recordType<it>name</it></tag> +<tag><it>group</it>.recordType[<it>.name</it>]</tag> Specifies how records with the file extension <it>name</it> should be handled by the indexer. This option may also be specified - as a command line option (<tt>-t</tt>). -<tag><it>group</it>recordId</tag> - Specifies how the record is to be identified when updated. -<tag><it>group</it>database</tag> - Specifies the Z39.50 database. -<tag><it>group</it>storeKeys</tag> + as a command line option (<tt>-t</tt>). Note that if you do not + specify a <it/name/, the setting applies to all files. In general, + the record type specifier consists of the elements (each + element separated by dot), <it>fundamental-type</it>, + <it>file-read-type</it> and arguments. Currently, two + fundamental types exist, <tt>text</tt> and <tt>grs</tt>. + <tag><it>group</it>.recordId</tag> + Specifies how the records are to be identified when updated. See +section <ref id="locating-records" name="Locating Records">. +<tag><it>group</it>.database</tag> + Specifies the Z39.50 database name. +<tag><it>group</it>.storeKeys</tag> Specifies whether key information should be saved for a given group of records. If you plan to update/delete this type of - record a later this should be specified as 1; otherwise it - should be 0 (default). -<tag><it>group</it>storeData</tag> - Specifies whether original copy of record should be stored internally - in the Zebra system indexes. If your externally indexed files - are temporary this option should certainly be true (1); otherwise - false (0). -<tag>register</tag> - Specifies the location of the indexes. + records later this should be specified as 1; otherwise it + should be 0 (default), to save register space. +<tag><it>group</it>.storeData</tag> + Specifies whether the records should be stored internally + in the Zebra system files. If you want to maintain the raw records yourself, + this option should be false (0). If you want Zebra to take care of the records + for you, it should be true(1). +<tag>lockDir</tag> + Directory in which various lock files are stored. +<tag>keyTmpDir</tag> + Directory in which temporary files used during zebraidx' update + phase are stored. +<tag>setTmpDir</tag> + Specifies the directory that the server uses for temporary result sets. + If not specified <tt>/tmp</tt> will be used. <tag>profilePath</tag> - Specifies the location of profile specification paths. + Specifies the location of profile specification files. <tag>attset</tag> - Specifies the filename(s) of attribute specification files. + Specifies the filename(s) of attribute set files for use in + searching. At least the Bib-1 set should be loaded (<tt/bib1.att/). + The <tt/profilePath/ setting is used to look for the specified files. + See section <ref id="attset-files" name="The Attribute Set Files"> +<tag>memMax</tag> + Specifies size of internal memory to use for the zebraidx program. The + amount is given in megabytes - default is 4 (4 MB). </descrip> - -<sect1>Locating records +<sect1>Locating Records<label id="locating-records"> <p> The default behaviour of the Zebra system is to reference the -records from their original location, i.e. where they were -read an indexed. - -If your records files are temporary for example if you retrieve -them from the outside or if they where temporarily mounted on a CD-ROM, -you may want the Zebra server to make a copy of them. To do this -you specify 1 (true) in the <tt>storedata</tt> setting. When -the Z39.50 search engine retrieves recoreds they will be read from an -internal structure of the index. - -<sect1>Indexing with no record Ids +records from their original location, i.e. where they were found when you +ran <tt/zebraidx/. That is, when a client wishes to retrieve a record +following a search operation, the files are accessed from the place +where you originally put them - if you remove the files (without +running <tt/zebraidx/ again, the client will receive a diagnostic +message. + +If your input files are not permanent - for example if you retrieve +your records from an outside source, or if they were temporarily +mounted on a CD-ROM drive, +you may want Zebra to make an internal copy of them. To do this, +you specify 1 (true) in the <tt>storeData</tt> setting. When +the Z39.50 server retrieves the records they will be read from the +internal file structures of the system. + +<sect1>Indexing example <p> -If you have a set of records that you <em/never/ wish to delete -or modify you may find 'indexing with no records Ids' convenient. -This indexing method uses less space than the other methods and -is simple to use. - -To use this method, you simply don't specify the <tt>recordId</tt> -for the group of files you index. To add a set of records you use -<tt>zebraidx</tt> with the <tt>update</tt> command. The -<tt>update</tt> command will always add the records to the index -becuase Zebra doesn't know how to match the new set of records with -existing records. - -Consider a system, in which you have a group of text files called +Consider a system in which you have a group of text files called <tt>simple</tt>. That group of records should belong to a Z39.50 database -called <tt>textbase</tt>. The following configuration file suffice: +called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice: <tscreen><verb> -profilePath: /usr/local/yaz +profilePath: /usr/lib/yaz/tab:/usr/lib/zebra/tab +attset: explain.att attset: bib1.att -attset: gils.att simple.recordType: text simple.database: textbase </verb></tscreen> -<sect1>Indexing with file record Ids +<sect>Running the Maintenance Interface (zebraidx) + +<p> +The following is a complete reference to the command line interface to +the <tt/zebraidx/ application. + +<bf/Syntax/ +<tscreen><verb> +$ zebraidx [options] command [directory] ... +</verb></tscreen> +<bf/Options/ +<descrip> +<tag>-t <it/type/</tag>Update all files as <it/type/. Currently, the +types supported are <tt/text/ and <tt/grs/<it/.subtype/. If no +<it/subtype/ is provided for the GRS (General Record Structure) type, +the canonical input format is assumed (see section <ref +id="local-representation" name="Local Representation">). Generally, it +is probably advisable to specify the record types in the +<tt/zebra.cfg/ file (see section <ref id="record-types" name="Record +Types">), to avoid confusion at subsequent updates. + +<tag>-c <it/config-file/</tag>Read the configuration file +<it/config-file/ instead of <tt/zebra.cfg/. + +<tag>-g <it/group/</tag>Update the files according to the group +settings for <it/group/ (see section <ref id="configuration-file" +name="The Zebra Configuration File">). + +<tag>-d <it/database/</tag>The records located should be associated +with the database name <it/database/ for access through the Z39.50 +server. + +<tag>-m <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing +keys to background storage. This setting affects performance when +updating large databases. + +<tag>-s</tag>Show analysis of the indexing process. The maintenance +program works in a read-only mode and doesn't change the state +of the index. This options is very useful when you wish to test a +new profile. + +<tag>-V</tag>Show Zebra version. + +<tag>-v <it/level/</tag>Set the log level to <it/level/. <it/level/ +should be one of <tt/none/, <tt/debug/, and <tt/all/. + +</descrip> + +<bf/Commands/ +<descrip> +<tag>Update <it/directory/</tag>Update the register with the files +contained in <it/directory/. If no directory is provided, a list of +files is read from <tt/stdin/. See section <ref +id="administrating" name="Administrating Zebra">. + +</descrip> + +<sect>The Z39.50 Server + +<sect1>Running the Z39.50 Server (zebrasrv) <p> -If you have a set of external records that you wish to index you may -use the file key feature of the Zebra system. In short, the file key -feature, mirrors a directory and the files below efficiently. To -perform indexing of a directory with file keys, you specify the top-level -directory after the <tt>update</tt> command. The command will recursively -traverse the directories and compare each with whatever have been -indexed before in the same directory. If a file is new (not in -the previous version of the directory) it is inserted; -if a file was already indexed and it has been modified -since the last insertion the index is also modified; if a file is missing -since the last visit it is deleted. +<bf/Syntax/ +<tscreen><verb> +zebrasrv [options] [listener-address ...] +</verb></tscreen> + +<bf/Options/ +<descrip> +<tag>-a <it/APDU file/</tag> Specify a file for dumping PDUs (for diagnostic purposes). +The special name &dquot;-&dquot; sends output to <tt/stderr/. + +<tag>-c <it/config-file/</tag> Read configuration information from <it/config-file/. The default configuration is <tt>./zebra.cfg</tt>. + +<tag/-S/Don't fork on connection requests. This can be useful for +symbolic-level debugging. The server can only accept a single +connection in this mode. + +<tag>-l <it/logfile/</tag>Specify an output file for the diagnostic +messages. The default is to write this information to <tt/stderr/. -The resulting system is easy to administer. To delete a record -you simply have to delete the corresponding file (with <tt/rm/). -To force update of a given file, you may use the <tt>touch</tt> -command. And to add files create new files (or directories with files). -For your changes to take effect you must run <tt>zebraidx</tt> with -the same directory root again. +<tag>-v <it/log-level/</tag>The log level. Use a comma-separated list of members of the set +{fatal,debug,warn,log,all,none}. -To use this method, you must specify <tt>file</tt> as the value -of <tt>recordId</tt> the configuration file. In the configuration -also set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra -indexer must save additional information in order to -modify/delete the record at a later time. +<tag>-u <it/username/</tag>Set user ID. Sets the real UID of the server process to that of the +given <it/username/. It's useful if you aren't comfortable with having the +server run as root, but you need to start it as such to bind a +privileged port. + +<tag>-w <it/working-directory/</tag>Change working directory. + +<tag>-i</tag>Run under the Internet superserver, <tt/inetd/. Make +sure you use the logfile option <tt/-l/ in conjunction with this +mode and specify the <tt/-l/ option before any other options. + +<tag>-t <it/timeout/</tag>Set the idle session timeout (default 60 minutes). + +<tag>-k <it/kilobytes/</tag>Set the (approximate) maximum size of +present response messages. Default is 1024 Kb (1 Mb). +</descrip> + +A <it/listener-address/ consists of a transport mode followed by a +colon (:) followed by a listener address. The transport mode is +either <tt/osi/ or <tt/tcp/. + +For TCP, an address has the form -For example, to update group <tt>esdd</tt> records below -<tt>/home/grs</tt> you could type: <tscreen><verb> -$ zebraidx -g esdd update /home/grs +hostname | IP-number [: portnumber] </verb></tscreen> -The corresponding configuration file includes: +The port number defaults to 210 (standard Z39.50 port). + +The special hostname &dquot;@&dquot; is mapped to +the address INADDR_ANY, which causes the server to listen on any local +interface. To start the server listening on the registered port for +Z39.50, and to drop root privileges once the +port is bound, execute the server like this (from a root shell): + <tscreen><verb> -esdd.recordId: file -esdd.recordType: grs -esdd.storeKeys: 1 +zebrasrv -u daemon tcp:@ </verb></tscreen> -<em>Important note: You cannot start by a group of records with simple -indexing (no record ids as in the previous section) and then, later, -use file record Ids. The Zebra must know from the very beginning that -the group of files are indexed with file record Ids. -</em> +You can replace <tt/daemon/ with another user, eg. your own account, or +a dedicated IR server account. + +The default behavior for <tt/zebrasrv/ is to establish a single TCP/IP +listener, for the Z39.50 protocol, on port 9999. + +<sect1>Z39.50 Protocol Support and Behavior + +<sect2>Initialization -<sect1>Indexing with general record Ids <p> -When using this method you specify an (almost) arbritrary record key -based on the contents of the record itself and other system -information. If you have a group of records that have an external -id attached, this method is convenient. For example, the record may -contain a title or an unique id. In either case you specify the -Z39.50 attribute-set and attribute-use location in which this -information is stored. +During initialization, the server will negotiate to version 3 of the +Z39.50 protocol (unless the client specifies a lower version), and the option bits for Search, Present, Scan, +NamedResultSets, and concurrentOperations will be set, if requested by +the client. The maximum PDU size is negotiated down to a maximum of +1Mb by default. + +<sect2>Search<label id="search"> -As before, the record id is defined by the <tt>recordId</tt> setting -in the configuration file. The value of the record id consists of one -or more tokens separated by space. The resulting id is formed -by concatenating the tokens and separete them by ascii value (1). +<p> +The supported query type are 1 and 101. All operators are currently +supported with the restriction that only proximity units of type "word" are +supported for the proximity operator. +Queries can be arbitrarily complex. +Named result sets are supported, and result sets can be used as operands +without limitations. +Searches may span multiple databases. + +The server has full support for piggy-backed present requests (see +also the following section). + +<bf/Use/ attributes are interpreted according to the attribute sets which +have been loaded in the <tt/zebra.cfg/ file, and are matched against +specific fields as specified in the <tt/.abs/ file which describes the +profile of the records which have been loaded. If no <bf/Use/ +attribute is provided, a default of Bib-1 <bf/Any/ is assumed. + +If a <bf/Structure/ attribute of <bf/Phrase/ is used in conjunction with a +<bf/Completeness/ attribute of <bf/Complete (Sub)field/, the term is +matched against the contents of the phrase (long word) register, if one +exists for the given <bf/Use/ attribute. +A phrase register is created for those fields in the <tt/.abs/ +file that contains a <tt/p/-specifier. + +If <bf/Structure/=<bf/Phrase/ is used in conjunction with +<bf/Incomplete Field/ - the default value for <bf/Completeness/, the +search is directed against the normal word registers, but if the term +contains multiple words, the term will only match if all of the words +are found immediately adjacent, and in the given order. +The word search is performed on those fields that are indexed as +type <tt/w/ in the <tt/.abs/ file. + +If the <bf/Structure/ attribute is <bf/Word List/, +<bf/Free-form Text/, or <bf/Document Text/, the term is treated as a +natural-language, relevance-ranked query. +This search type uses the word register, i.e. those fields +that are indexed as type <tt/w/ in the <tt/.abs/ file. + +If the <bf/Structure/ attribute is <bf/Numeric String/ the +term is treated as an integer. The search is performed on those +fields that are indexed as type <tt/n/ in the <tt/.abs/ file. + +If the <bf/Structure/ attribute is <bf/URx/ the +term is treated as a URX (URL) entity. The search is performed on those +fields that are indexed as type <tt/u/ in the <tt/.abs/ file. + +If the <bf/Structure/ attribute is <bf/Local Number/ the +term is treated as native Zebra Record Identifier. + +If the <bf/Relation/ attribute is <bf/Equals/ (default), the term is +matched in a normal fashion (modulo truncation and processing of +individual words, if required). If <bf/Relation/ is <bf/Less Than/, +<bf/Less Than or Equal/, <bf/Greater than/, or <bf/Greater than or +Equal/, the term is assumed to be numerical, and a standard regular +expression is constructed to match the given expression. If +<bf/Relation/ is <bf/Relevance/, the standard natural-language query +processor is invoked. + +For the <bf/Truncation/ attribute, <bf/No Truncation/ is the default. +<bf/Left Truncation/ is not supported. <bf/Process #/ is supported, as +is <bf/Regxp-1/. <bf/Regxp-2/ enables the fault-tolerant (fuzzy) +search. As a default, a single error (deletion, insertion, +replacement) is accepted when terms are matched against the register +contents. + +<sect3>Regular expressions +<p> -There are three kinds of tokens: +Each term in a query is interpreted as a regular expression if +the truncation value is either <bf/Regxp-1/ (102) or <bf/Regxp-2/ (103). +Both query types follow the same syntax with the operands: <descrip> -<tag>Internal record info</tag> The token refers to a key that is -extracted from the record. The syntax of this token is - <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the -attribute set number and <em/use/ is the use value of the attribute. -<tag>System variable</tag> The system variables are preceded by -<tt>$</tt> and immediately followed by the system variable name, which -may one of - <descrip> - <tag>group</tag> Group name. - <tag>database</tag> Current database specified. - <tag>filename</tag> Name of file that contain the record. - <tag>type</tag> Record type. - </descrip> -<tag>Constant string</tag> A string used as part of id — surrounded - by single- or double quotes. +<tag/x/ Matches the character <it/x/. +<tag/./ Matches any character. +<tag><tt/[/..<tt/]/</tag> Matches the set of characters specified; + such as <tt/[abc]/ or <tt/[a-c]/. </descrip> +and the operators: +<descrip> +<tag/x*/ Matches <it/x/ zero or more times. Priority: high. +<tag/x+/ Matches <it/x/ one or more times. Priority: high. +<tag/x?/ Matches <it/x/ once or twice. Priority: high. +<tag/xy/ Matches <it/x/, then <it/y/. Priority: medium. +<tag/x|y/ Matches either <it/x/ or <it/y/. Priority: low. +</descrip> +The order of evaluation may be changed by using parentheses. + +If the first character of the <bf/Regxp-2/ query is a plus character +(<tt/+/) it marks the beginning of a section with non-standard +specifiers. The next plus character marks the end of the section. +Currently Zebra only supports one specifier, the error tolerance, +which consists one digit. + +Since the plus operator is normally a suffix operator the addition to +the query syntax doesn't violate the syntax for standard regular +expressions. + +<sect3>Query examples +<p> + +Phrase search for <bf/information retrieval/ in the title-register: +<verb> + @attr 1=4 "information retrieval" +</verb> + +Ranked search for the same thing: +<verb> + @attr 1=4 @attr 2=102 "Information retrieval" +</verb> + +Phrase search with a regular expression: +<verb> + @attr 1=4 @attr 5=102 "informat.* retrieval" +</verb> + +Ranked search with a regular expression: +<verb> + @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval" +</verb> + +In the GILS schema (<tt/gils.abs/), the west-bounding-coordinate is +indexed as type <tt/n/, and is therefore searched by specifying +<bf/structure/=<bf/Numeric String/. +To match all those records with west-bounding-coordinate greater +than -114 we use the following query: +<verb> + @attr 4=109 @attr 2=5 @attr gils 1=2038 -114 +</verb> + +<sect2>Present +<p> +The present facility is supported in a standard fashion. The requested +record syntax is matched against the ones supported by the profile of +each record retrieved. If no record syntax is given, SUTRS is the +default. The requested element set name, again, is matched against any +provided by the relevant record profiles. + +<sect2>Scan + +<p> +The attribute combinations provided with the TermListAndStartPoint are +processed in the same way as operands in a query (see above). +Currently, only the term and the globalOccurrences are returned with +the TermInfo structure. + +<sect2>Sort + +<p> +Z39.50 specifies three diffent types of sort criterias. +Of these Zebra supports the attribute specification type in which +case the use attribute specifies the "Sort register". +Sort registers are created for those fields that are of type "sort" in +the default.idx file. +The corresponding character mapping file in default.idx specifies the +ordinal of each character used in the actual sort. + +Z39.50 allows the client to specify sorting on one or more input +result sets and one output result set. +Zebra supports sorting on one result set only which may or may not +be the same as the output result set. + +<sect2>Close + +<p> +If a Close PDU is received, the server will respond with a Close PDU +with reason=FINISHED, no matter which protocol version was negotiated +during initialization. If the protocol version is 3 or more, the +server will generate a Close PDU under certain circumstances, +including a session timeout (60 minutes by default), and certain kinds of +protocol errors. Once a Close PDU has been sent, the protocol +association is considered broken, and the transport connection will be +closed immediately upon receipt of further data, or following a short +timeout. + +<sect>The Record Model + +<p> +Zebra is designed to support a wide range of data management +applications. The system can be configured to handle virtually any +kind of structured data. Each record in the system is associated with +a <it/record schema/ which lends context to the data elements of the +record. Any number of record schema can coexist in the system. +Although it may be wise to use only a single schema within +one database, the system poses no such restrictions. + +The record model described in this chapter applies to the fundamental, +structured +record type <tt>grs</tt> as introduced in +section <ref id="record-types" name="Record Types">. + +Records pass through three different states during processing in the +system. + +<itemize> +<item>When records are accessed by the system, they are represented +in their local, or native format. This might be SGML or HTML files, +News or Mail archives, MARC records. If the system doesn't already +know how to read the type of data you need to store, you can set up an +input filter by preparing conversion rules based on regular +expressions and possibly augmented by a flexible scripting language (Tcl). The input filter +produces as output an internal representation: + +<item>When records are processed by the system, they are represented +in a tree-structure, constructed by tagged data elements hanging off a +root node. The tagged elements may contain data or yet more tagged +elements in a recursive structure. The system performs various +actions on this tree structure (indexing, element selection, schema +mapping, etc.), + +<item>Before transmitting records to the client, they are first +converted from the internal structure to a form suitable for exchange +over the network - according to the Z39.50 standard. +</itemize> + +<sect1>Local Representation<label id="local-representation"> + +<p> +As mentioned earlier, Zebra places few restrictions on the type of +data that you can index and manage. Generally, whatever the form of +the data, it is parsed by an input filter specific to that format, and +turned into an internal structure that Zebra knows how to handle. This +process takes place whenever the record is accessed - for indexing and +retrieval. + +<p> +The RecordType parameter in the <tt/zebra.cfg/ file, or the <tt/-t/ +option to the indexer tells Zebra how to process input records. Two +basic types of processing are available - raw text and structured +data. Raw text is just that, and it is selected by providing the +argument <bf/text/ to Zebra. Structured records are all handled +internally using the basic mechanisms described in the subsequent +sections. Zebra can read structured records in many different formats. +How this is done is governed by additional parameters after the +&dquot;grs&dquot; keyboard, separated by &dquot;.&dquot; characters. + +Three basic subtypes to the <bf/grs/ type are currently available: + +<descrip> +<tag>grs.sgml</tag>This is the canonical input format — +described below. It is a simple SGML-like syntax. + +<tag>grs.regx.<it/filter/</tag>This enables a user-supplied input +filter. The mechanisms of these filters are described below. + +<tag>grs.tcl.<it/filter/</tag>This enables a user-supplied input +filter with Tcl rules (only availble if zebra is compiled with Tcl +support). + +<tag>grs.marc.<it/abstract syntax/</tag>This allows Zebra to read +records in the ISO2709 (MARC) encoding standard. In this case, the +last paramemeter <it/abstract syntax/ names the .abs file (see below) +which describes the specific MARC structure of the input record as +well as the indexing rules. +</descrip> + +<sect2>Canonical Input Format + +<p> +Although input data can take any form, it is sometimes useful to +describe the record processing capabilities of the system in terms of +a single, canonical input format that gives access to the full +spectrum of structure and flexibility in the system. In Zebra, this +canonical format is an &dquot;SGML-like&dquot; syntax. + +To use the canonical format specify <tt>grs.sgml</tt> as the record +type, + +Consider a record describing an information resource (such a record is +sometimes known as a <it/locator record/). It might contain a field +describing the distributor of the information resource, which might in +turn be partitioned into various fields providing details about the +distributor, like this: -The test GILS records that comes with Zebra, contain a unique id -in the Control-Identifier field. This field is mapped to the Bib-1 -use attribute 1007. To use this field as a record id, specify -<tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the -configuration file. If you have other record types that doesn't -contain an id in the same field, you might add the record type -in the record id of the gils records as well, to prevent match -of other types of records. In this case the recordId might be -set, as in: <tscreen><verb> -gils.recordId: $type (1,1007) +<Distributor> + <Name> USGS/WRD &etago;Name> + <Organization> USGS/WRD &etago;Organization> + <Street-Address> + U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW + &etago;Street-Address> + <City> ALBUQUERQUE &etago;City> + <State> NM &etago;State> + <Zip-Code> 87102 &etago;Zip-Code> + <Country> USA &etago;Country> + <Telephone> (505) 766-5560 &etago;Telephone> +&etago;Distributor> </verb></tscreen> -As for the file record id case described in the previous section -updating your system is simply a matter of running <tt>zebraidx</tt> -with the <tt>update</tt> command. However, the update of with general -keys is considerably slower than with file record ids, since all files -visited must be (re)indexded. +<it>NOTE: The indentation used above is used to illustrate how Zebra +interprets the markup. The indentation, in itself, has no +significance to the parser for the canonical input format, which +discards superfluous whitespace.</it> + +The keywords surrounded by <...> are <it/tags/, while the +sections of text in between are the <it/data elements/. A data element +is characterized by its location in the tree that is made up by the +nested elements. Each element is terminated by a closing tag - +beginning with <tt/&etago;/, and containing the same symbolic tag-name as +the corresponding opening tag. The general closing tag - <tt/&etago;>/ - +terminates the element started by the last opening tag. The +structuring of elements is significant. The element <bf/Telephone/, +for instance, may be indexed and presented to the client differently, +depending on whether it appears inside the <bf/Distributor/ element, +or some other, structured data element such a <bf/Supplier/ element. + +<sect3>Record Root + +<p> +The first tag in a record describes the root node of the tree that +makes up the total record. In the canonical input format, the root tag +should contain the name of the schema that lends context to the +elements of the record (see section <ref id="internal-representation" +name="Internal Representation">). The following is a GILS record that +contains only a single element (strictly speaking, that makes it an +illegal GILS record, since the GILS profile includes several mandatory +elements - Zebra does not validate the contents of a record against +the Z39.50 profile, however - it merely attempts to match up elements +of a local representation with the given schema): + +<tscreen><verb> +<gils> + <title>Zen and the Art of Motorcycle Maintenance&etago;title> +&etago;gils> +</verb></tscreen> -<sect1>Register location +<sect3>Variants <p> -Normally, the index files that form dictionaries, inverted -file, record info, etc. are stored in the current directory in which you run -<tt>zebraidx</tt>. If you wish to store those, possibly large, files -somewhere else, you must set the <tt>register</tt> setting in the -configuration file. Furhtermore, the Zebra system handles indexes that -span multiple file systems, which is usefull if a large number of -records are indexed. +Zebra allows you to provide individual data elements in a number of +<it/variant forms/. Examples of variant forms are textual data +elements which might appear in different languages, and images which +may appear in different formats or layouts. The variant system in +Zebra is +essentially a representation of the variant mechanism of +Z39.50-1995. -The value <tt>register</tt> of register is a sequence of tokens. -Each token takes the form: -<tscreen> -<em>dir</em><tt>:</tt><em>size</em>. -</tscreen> -The <em>dir</em> specifies a directory in which index files will be -stored and the <em>size</em> specifies the maximum size of all -files in that directory. The Zebra indexer system fill each directory -in the order specified and use the next specified directories as needed. -The <em>size</em> is an integer possibly followed by a quantity -code, <tt>M</tt> for Mega bytes, <tt>K</tt> for Kilo bytes. +The following is an example of a title element which occurs in two +different languages. -If you have two spare disks :) and the first disk is mounted -on <tt>/d1</tt> and has 200 Mega bytes free space after formatting and the -second, mounted on <tt>/d2</tt> has 300 Mega bytes free, you could -specify the following in you configuration file: <tscreen><verb> -register: /d1:200M /d2:300M +<title> + <var lang lang "eng"> + Zen and the Art of Motorcycle Maintenance&etago;> + <var lang lang "dan"> + Zen og Kunsten at Vedligeholde en Motorcykel&etago;> +&etago;title> </verb></tscreen> -<sect>The Z39.50 Server +The syntax of the <it/variant element/ is <tt><<bf/var/ <it/class +type value/></tt>. The available values for the <it/class/ and +<it/type/ fields are given by the variant set that is associated with the +current schema (see section <ref id="variant-set" name="Variant Set +File">). + +Variant elements are terminated by the general end-tag &etago;>, by +the variant end-tag &etago;var>, by the appearance of another variant +tag with the same <it/class/ and <it/value/ settings, or by the +appearance of another, normal tag. In other words, the end-tags for +the variants used in the example above could have been saved. + +Variant elements can be nested. The element + +<tscreen><verb> +<title> + <var lang lang "eng"><var body iana "text/plain"> + Zen and the Art of Motorcycle Maintenance +&etago;title> +</verb></tscreen> + +Associates two variant components to the variant list for the title +element. + +Given the nesting rules described above, we could write + +<tscreen><verb> +<title> + <var body iana "text/plain> + <var lang lang "eng"> + Zen and the Art of Motorcycle Maintenance + <var lang lang "dan"> + Zen og Kunsten at Vedligeholde en Motorcykel +&etago;title> +</verb></tscreen> + +The title element above comes in two variants. Both have the IANA body +type &dquot;text/plain&dquot;, but one is in English, and the other in +Danish. The client, using the element selection mechanism of Z39.50, +can retrieve information about the available variant forms of data +elements, or it can select specific variants based on the requirements +of the end-user. + +<sect2>Input Filters <p> +In order to handle general input formats, Zebra allows the +operator to define filters which read individual records in their native format +and produce an internal representation that the system can +work with. + +Input filters are ASCII files, generally with the suffix <tt/.flt/. +The system looks for the files in the directories given in the +<bf/profilePath/ setting in the <tt/zebra.cfg/ files. The record type +for the filter is <tt>grs.regx.</tt><it>filter-filename</it> +(fundamental type <tt>grs</tt>, file read type <tt>regx</tt>, argument +<it>filter-filename</it>). + +Generally, an input filter consists of a sequence of rules, where each +rule consists of a sequence of expressions, followed by an action. The +expressions are evaluated against the contents of the input record, +and the actions normally contribute to the generation of an internal +representation of the record. + +An expression can be either of the following: + +<descrip> +<tag/INIT/The action associated with this expression is evaluated +exactly once in the lifetime of the application, before any records +are read. It can be used in conjunction with an action that +initializes tables or other resources that are used in the processing +of input records. + +<tag/BEGIN/Matches the beginning of the record. It can be used to +initialize variables, etc. Typically, the <bf/BEGIN/ rule is also used +to establish the root node of the record. + +<tag/END/Matches the end of the record - when all of the contents +of the record has been processed. + +<tag>/pattern/</tag>Matches a string of characters from the input +record. + +<tag/BODY/This keyword may only be used between two patterns. It +matches everything between (not including) those patterns. + +<tag/FINISH/THe expression asssociated with this pattern is evaluated +once, before the application terminates. It can be used to release +system resources - typically ones allocated in the <bf/INIT/ step. + +</descrip> + +An action is surrounded by curly braces ({...}), and consists of a +sequence of statements. Statements may be separated by newlines or +semicolons (;). Within actions, the strings that matched the +expressions immediately preceding the action can be referred to as +$0, $1, $2, etc. + +The available statements are: + +<descrip> + +<tag>begin <it/type [parameter ... ]/</tag>Begin a new +data element. The type is one of the following: +<descrip> +<tag/record/Begin a new record. The followingparameter should be the +name of the schema that describes the structure of the record, eg. +<tt/gils/ or <tt/wais/ (see below). The <tt/begin record/ call should +precede +any other use of the <bf/begin/ statement. + +<tag/element/Begin a new tagged element. The parameter is the +name of the tag. If the tag is not matched anywhere in the tagsets +referenced by the current schema, it is treated as a local string +tag. + +<tag/variant/Begin a new node in a variant tree. The parameters are +<it/class type value/. + +</descrip> + +<tag/data/Create a data element. The concatenated arguments make +up the value of the data element. The option <tt/-text/ signals that +the layout (whitespace) of the data should be retained for +transmission. The option <tt/-element/ <it/tag/ wraps the data up in +the <it/tag/. The use of the <tt/-element/ option is equivalent to +preceding the command with a <bf/begin element/ command, and following +it with the <bf/end/ command. + +<tag>end <it/[type]/</tag>Close a tagged element. If no parameter is given, +the last element on the stack is terminated. The first parameter, if +any, is a type name, similar to the <bf/begin/ statement. For the +<bf/element/ type, a tag name can be provided to terminate a specific tag. + +</descrip> + +The following input filter reads a Usenet news file, producing a +record in the WAIS schema. Note that the body of a news posting is +separated from the list of headers by a blank line (or rather a +sequence of two newline characters. + +<tscreen><verb> +BEGIN { begin record wais } + +/^From:/ BODY /$/ { data -element name $1 } +/^Subject:/ BODY /$/ { data -element title $1 } +/^Date:/ BODY /$/ { data -element lastModified $1 } +/\n\n/ BODY END { + begin element bodyOfDisplay + begin variant body iana "text/plain" + data -text $1 + end record + } +</verb></tscreen> + +If Zebra is compiled with support for Tcl (Tool Command Language) +enabled, the statements described above are supplemented with a complete +scripting environment, including control structures (conditional +expressions and loop constructs), and powerful string manipulation +mechanisms for modifying the elements of a record. Tcl is a popular +scripting environment, with several tutorials available both online +and in hardcopy. + +<it>NOTE: Variant support is not currently available in the input +filter, but will be included with one of the next +releases.</it> + +<sect1>Internal Representation<label id="internal-representation"> -<sect1>Running the server <p> -Starting the server and how to configure the server. +When records are manipulated by the system, they're represented in a +tree-structure, with data elements at the leaf nodes, and tags or +variant components at the non-leaf nodes. The root-node identifies the +schema that lends context to the tagging and structuring of the +record. Imagine a simple record, consisting of a 'title' element and +an 'author' element: + +<tscreen><verb> + TITLE "Zen and the Art of Motorcycle Maintenance" +ROOT + AUTHOR "Robert Pirsig" +</verb></tscreen> + +A slightly more complex record would have the author element consist +of two elements, a surname and a first name: + +<tscreen><verb> + TITLE "Zen and the Art of Motorcycle Maintenance" +ROOT + FIRST-NAME "Robert" + AUTHOR + SURNAME "Pirsig" +</verb></tscreen> + +The root of the record will refer to the record schema that describes +the structuring of this particular record. The schema defines the +element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as +well as the structuring (SURNAME should appear below AUTHOR, etc.). In +addition, the schema establishes element set names that are used by +the client to request a subset of the elements of a given record. The +schema may also establish rules for converting the record to a +different schema, by stating, for each element, a mapping to a +different tag path. + +<sect2>Tagged Elements + +<p> +A data element is characterized by its tag, and its position in the +structure of the record. For instance, while the tag &dquot;telephone +number&dquot; may be used different places in a record, we may need to +distinguish between these occurrences, both for searching and +presentation purposes. For instance, while the phone numbers for the +&dquot;customer&dquot; and the &dquot;service provider&dquot; are both +representatives for the same type of resource (a telephone number), it +is essential that they be kept separate. The record schema provides +the structure of the record, and names each data element (defined by +the sequence of tags - the tag path - by which the element can be +reached from the root of the record). + +<sect2>Variants + +<p> +The children of a tag node may be either more tag nodes, a data node +(possibly accompanied by tag nodes), +or a tree of variant nodes. The children of variant nodes are either +more variant nodes or a data node (possibly accompanied by more +variant nodes). Each leaf node, which is normally a +data node, corresponds to a <it/variant form/ of the tagged element +identified by the tag which parents the variant tree. The following +title element occurs in two different languages: + +<tscreen><verb> + VARIANT LANG=ENG "War and Peace" +TITLE + VARIANT LANG=DAN "Krig og Fred" +</verb></tscreen> + +Which of the two elements are transmitted to the client by the server +depends on the specifications provided by the client, if any. + +In practice, each variant node is associated with a triple of class, +type, value, corresponding to the variant mechanism of Z39.50. + +<sect2>Data Elements + +<p> +Data nodes have no children (they are always leaf nodes in the record +tree). + +<it>NOTE: Documentation needs extension here about types of nodes - numerical, +textual, etc., plus the various types of inclusion notes.</it> + +<sect1>Configuring Your Data Model<label id="data-model"> -<sect1>How the server handles queries <p> -What elements of Bib-1 are supported and where are result sets -stored. +The following sections describe the configuration files that govern +the internal management of data records. The system searches for the files +in the directories specified by the <bf/profilePath/ setting in the +<tt/zebra.cfg/ file. -<sect>About Index Data +<sect2>About Object Identifers <p> -Copyright © 1995, Index Data. +When Object Identifiers (or OID's) need to be specified in the following +a named OID reference or a raw OID reference may be used. For the named +OID's refer to the source file <tt>util/oid.c</tt> from YAZ. The raw +canonical OID's are specified in dot-notation (for example +1.2.840.10003.3.1000.81.1). -Permission to use, copy, modify, distribute, and sell this software and -its documentation, in whole or in part, for any purpose, is hereby granted, -provided that: +<sect2>The Abstract Syntax -1. This copyright and permission notice appear in all copies of the -software and its documentation. Notices of copyright or attribution -which appear at the beginning of any file must remain unchanged. +<p> +The abstract syntax definition (also known as an Abstract Record +Structure, or ARS) is the focal point of the +record schema description. For a given schema, the ABS file may state any +or all of the following: + +<itemize> +<item>The object identifier of the Z39.50 schema associated +with the ARS, so that it can be referred to by the client. + +<item>The attribute set (which can possibly be a compound of multiple +sets) which applies in the profile. This is used when indexing and +searching the records belonging to the given profile. + +<item>The Tag set (again, this can consist of several different sets). +This is used when reading the records from a file, to recognize the +different tags, and when transmitting the record to the client - +mapping the tags to their numerical representation, if they are +known. + +<item>The variant set which is used in the profile. This provides a +vocabulary for specifying the <it/forms/ of data that appear inside +the records. + +<item>Element set names, which are a shorthand way for the client to +ask for a subset of the data elements contained in a record. Element +set names, in the retrieval module, are mapped to <it/element +specifications/, which contain information equivalent to the +<it/Espec-1/ syntax of Z39.50. -2. The names of Index Data or the individual authors may not be used to -endorse or promote products derived from this software without specific -prior written permission. +<item>Map tables, which may specify mappings to <it/other/ database +profiles, if desired. -THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND, -EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY -WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. -IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL, -INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES -WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR -NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF -LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE -OF THIS SOFTWARE. +<item>Possibly, a set of rules describing the mapping of elements to a +MARC representation. -<sect>About Index Data +<item>A list of element descriptions (this is the actual ARS of the +schema, in Z39.50 terms), which lists the ways in which the various +tags can be used and organized hierarchically. +</itemize> + +Several of the entries above simply refer to other files, which +describe the given objects. + +<sect2>The Configuration Files + +<p> +This section describes the syntax and use of the various tables which +are used by the retrieval module. + +The number of different file types may appear daunting at first, but +each type corresponds fairly clearly to a single aspect of the Z39.50 +retrieval facilities. Further, the average database administrator, +who is simply reusing an existing profile for which tables already +exist, shouldn't have to worry too much about the contents of these tables. + +Generally, the files are simple ASCII files, which can be maintained +using any text editor. Blank lines, and lines beginning with a (#) are +ignored. Any characters on a line followed by a (#) are also ignored. +All other +lines contain <it/directives/, which provide some setting or value +to the system. Generally, settings are characterized by a single +keyword, identifying the setting, followed by a number of parameters. +Some settings are repeatable (r), while others may occur only once in a +file. Some settings are optional (o), whicle others again are +mandatory (m). + +<sect2>The Abstract Syntax (.abs) Files + +<p> +The name of this file type is slightly misleading in Z39.50 terms, +since, apart from the actual abstract syntax of the profile, it also +includes most of the other definitions that go into a database +profile. + +When a record in the canonical, SGML-like format is read from a file +or from the database, the first tag of the file should reference the +profile that governs the layout of the record. If the first tag of the +record is, say, <tt><gils></tt>, the system will look for the profile +definition in the file <tt/gils.abs/. Profile definitions are cached, +so they only have to be read once during the lifespan of the current +process. + +When writing your own input filters, the <bf/record-begin/ command +introduces the profile, and should always be called first thing when +introducing a new record. + +The file may contain the following directives: + +<descrip> +<tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or +description for the profile. Mostly useful for diagnostic purposes. + +<tag>reference <it/OID-name/</tag> (m) The OID for +the profile (name or dotted-numerical list). + +<tag>attset <it/filename/</tag> (m) The attribute set that is used for +indexing and searching records belonging to this profile. + +<tag>tagset <it/filename/ [<it/type/]</tag> (o) The tag +set (if any) that describe that fields of the records. The type, which +is optional, specifies the tag type. If not given, the type-specifier +in the Tag Set files is used. + +<tag>varset <it/filename/</tag> (o) The variant set used in the profile. + +<tag>maptab <it/filename/</tag> (o,r) This points to a +conversion table that might be used if the client asks for the record +in a different schema from the native one. + +<tag>marc <it/filename/</tag> (o) Points to a file containing parameters +for representing the record contents in the ISO2709 syntax. Read the +description of the MARC representation facility below. + +<tag>esetname <it/name filename/</tag> (o,r) Associates the +given element set name with an element selection file. If an (@) is +given in place of the filename, this corresponds to a null mapping for +the given element set name. + +<tag>any <it/tags/</tag> (o) This directive specifies a list of +attributes which should be appended to the attribute list given for each +element. The effect is to make every single element in the abstract +syntax searchable by way of the given attributes. This directive +provides an efficient way of supporting free-text searching across all +elements. However, it does increase the size of the index +significantly. The attributes can be qualified with a structure, as in +the <bf/elm/ directive below. + +<tag>elm <it/path name attributes/</tag> (o,r) Adds an element +to the abstract record syntax of the schema. The <it/path/ follows the +syntax which is suggested by the Z39.50 document - that is, a sequence +of tags separated by slashes (/). Each tag is given as a +comma-separated pair of tag type and -value surrounded by parenthesis. +The <it/name/ is the name of the element, and the <it/attributes/ +specifies which attributes to use when indexing the element in a +comma-separated list. A ! in +place of the attribute name is equivalent to specifying an attribute +name identical to the element name. A - in place of the attribute name +specifies that no indexing is to take place for the given element. The +attributes can be qualified with <it/field types/ to specify which +character set should govern the indexing procedure for that field. The +same data element may be indexed into several different fields, using +different character set definitions. See the section +<ref id="field structure and character sets" +name="Field Structure and Character Sets">. +The default field type is &dquot;w&dquot; for +<it/word/. +</descrip> + +The following is an excerpt from the abstract syntax file for the GILS +profile. + +<tscreen><verb> +name gils +reference GILS-schema +attset gils.att +tagset gils.tag +varset var1.var + +maptab gils-usmarc.map + +# Element set names + +esetname VARIANT gils-variant.est # for WAIS-compliance +esetname B gils-b.est +esetname G gils-g.est +esetname F @ + +elm (1,10) rank - +elm (1,12) url - +elm (1,14) localControlNumber Local-number +elm (1,16) dateOfLastModification Date/time-last-modified +elm (2,1) Title w:!,p:! +elm (4,1) controlIdentifier Identifier-standard +elm (2,6) abstract Abstract +elm (4,51) purpose ! +elm (4,52) originator - +elm (4,53) accessConstraints ! +elm (4,54) useConstraints ! +elm (4,70) availability - +elm (4,70)/(4,90) distributor - +elm (4,70)/(4,90)/(2,7) distributorName ! +elm (4,70)/(4,90)/(2,10 distributorOrganization ! +elm (4,70)/(4,90)/(4,2) distributorStreetAddress ! +elm (4,70)/(4,90)/(4,3) distributorCity ! +</verb></tscreen> + +<sect2>The Attribute Set (.att) Files<label id="attset-files"> + +<p> +This file type describes the <bf/Use/ elements of an attribute set. +It contains the following directives. + +<descrip> + +<tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or +description for the attribute set. Mostly useful for diagnostic purposes. + +<tag>reference <it/OID-name/</tag> (m) The reference name of the OID for +the attribute set. + +<tag>include <it/filename/</tag> (o,r) This directive is used to +include another attribute set as a part of the current one. This is +used when a new attribute set is defined as an extension to another +set. For instance, many new attribute sets are defined as extensions +to the <bf/bib-1/ set. This is an important feature of the retrieval +system of Z39.50, as it ensures the highest possible level of +interoperability, as those access points of your database which are +derived from the external set (say, bib-1) can be used even by clients +who are unaware of the new set. + +<tag>att <it/att-value att-name [local-value]/</tag> (o,r) This +repeatable directive introduces a new attribute to the set. The +attribute value is stored in the index (unless a <it/local-value/ is +given, in which case this is stored). The name is used to refer to the +attribute from the <it/abstract syntax/. </descrip> + +This is an excerpt from the GILS attribute set definition. Notice how +the file describing the <it/bib-1/ attribute set is referenced. + +<tscreen><verb> +name gils +reference GILS-attset +include bib1.att + +att 2001 distributorName +att 2002 indexTermsControlled +att 2003 purpose +att 2004 accessConstraints +att 2005 useConstraints +</verb></tscreen> + +<sect2>The Tag Set (.tag) Files + +<p> +This file type defines the tagset of the profile, possibly by +referencing other tag sets (most tag sets, for instance, will include +tagsetG and tagsetM from the Z39.50 specification. The file may +contain the following directives. + +<descrip> +<tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or +description for the tag set. Mostly useful for diagnostic purposes. + +<tag>reference <it/OID-name/</tag> (o) The reference name of the OID for +the tag set. The directive is optional, since not all tag sets are +registered outside of their schema. + +<tag>type <it/integer/</tag> (m) The type number of the tagset within the schema +profile (note: this specification really should belong to the .abs +file. This will be fixed in a future release). + +<tag>include <it/filename/</tag> (o,r) This directive is used +to include the definitions of other tag sets into the current one. + +<tag>tag <it/number names type/</tag> (o,r) Introduces a new +tag to the set. The <it/number/ is the tag number as used in the protocol +(there is currently no mechanism for specifying string tags at this +point, but this would be quick work to add). The <it/names/ parameter +is a list of names by which the tag should be recognized in the input +file format. The names should be separated by slashes (/). The +<it/type/ is th recommended datatype of the tag. It should be one of +the following: +<itemize> +<item>structured +<item>string +<item>numeric +<item>bool +<item>oid +<item>generalizedtime +<item>intunit +<item>int +<item>octetstring +<item>null +</itemize> +</descrip> + +The following is an excerpt from the TagsetG definition file. + +<tscreen><verb> +name tagsetg +reference TagsetG +type 2 + +tag 1 title string +tag 2 author string +tag 3 publicationPlace string +tag 4 publicationDate string +tag 5 documentId string +tag 6 abstract string +tag 7 name string +tag 8 date generalizedtime +tag 9 bodyOfDisplay string +tag 10 organization string +</verb></tscreen> + +<sect2>The Variant Set (.var) Files<label id="variant-set"> + +<p> +The variant set file is a straightforward representation of the +variant set definitions associated with the protocol. At present, only +the <it/Variant-1/ set is known. + +These are the directives allowed in the file. + +<descrip> +<tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or +description for the variant set. Mostly useful for diagnostic purposes. + +<tag>reference <it/OID-name/</tag> (o) The reference name of the OID for +the variant set, if one is required. + +<tag>class <it/integer class-name/</tag> (m,r) Introduces a new +class to the variant set. + +<tag>type <it/integer type-name datatype/</tag> (m,r) Addes a +new type to the current class (the one introduced by the most recent +<bf/class/ directive). The type names belong to the same name space as +the one used in the tag set definition file. +</descrip> + +The following is an excerpt from the file describing the variant set +<it/Variant-1/. + +<tscreen><verb> +name variant-1 +reference Variant-1 + +class 1 variantId + + type 1 variantId octetstring + +class 2 body + + type 1 iana string + type 2 z39.50 string + type 3 other string +</verb></tscreen> + +<sect2>The Element Set (.est) Files + +<p> +The element set specification files describe a selection of a subset +of the elements of a database record. The element selection mechanism +is equivalent to the one supplied by the <it/Espec-1/ syntax of the +Z39.50 specification. In fact, the internal representation of an +element set specification is identical to the <it/Espec-1/ structure, +and we'll refer you to the description of that structure for most of +the detailed semantics of the directives below. + +<it> +NOTE: Not all of the Espec-1 functionality has been implemented yet. +The fields that are mentioned below all work as expected, unless +otherwise is noted. +</it> + +The directives available in the element set file are as follows: + +<descrip> +<tag>defaultVariantSetId <it/OID-name/</tag> (o) If variants are used in +the following, this should provide the name of the variantset used +(it's not currently possible to specify a different set in the +individual variant request). In almost all cases (certainly all +profiles known to us), the name <tt/Variant-1/ should be given here. + +<tag>defaultVariantRequest <it/variant-request/</tag> (o) This directive +provides a default variant request for +use when the individual element requests (see below) do not contain a +variant request. Variant requests consist of a blank-separated list of +variant components. A variant compont is a comma-separated, +parenthesized triple of variant class, type, and value (the two former +values being represented as integers). The value can currently only be +entered as a string (this will change to depend on the definition of +the variant in question). The special value (@) is interpreted as a +null value, however. + +<tag>simpleElement <it/path ['variant' variant-request]/</tag> +(o,r) This corresponds to a simple element request in <it/Espec-1/. The +path consists of a sequence of tag-selectors, where each of these can +consist of either: + +<itemize> +<item>A simple tag, consisting of a comma-separated type-value pair in +parenthesis, possibly followed by a colon (:) followed by an +occurrences-specification (see below). The tag-value can be a number +or a string. If the first character is an apostrophe ('), this forces +the value to be interpreted as a string, even if it appears to be numerical. + +<item>A WildThing, represented as a question mark (?), possibly +followed by a colon (:) followed by an occurrences specification (see +below). + +<item>A WildPath, represented as an asterisk (*). Note that the last +element of the path should not be a wildPath (wildpaths don't work in +this version). +</itemize> + +The occurrences-specification can be either the string <tt/all/, the +string <tt/last/, or an explicit value-range. The value-range is +represented as an integer (the starting point), possibly followed by a +plus (+) and a second integer (the number of elements, default being +one). + +The variant-request has the same syntax as the defaultVariantRequest +above. Note that it may sometimes be useful to give an empty variant +request, simply to disable the default for a specific set of fields +(we aren't certain if this is proper <it/Espec-1/, but it works in +this implementation). +</descrip> + +The following is an example of an element specification belonging to +the GILS profile. + +<tscreen><verb> +simpleelement (1,10) +simpleelement (1,12) +simpleelement (2,1) +simpleelement (1,14) +simpleelement (4,1) +simpleelement (4,52) +</verb></tscreen> + +<sect2>The Schema Mapping (.map) Files<label id="schema-mapping"> + +<p> +Sometimes, the client might want to receive a database record in +a schema that differs from the native schema of the record. For +instance, a client might only know how to process WAIS records, while +the database record is represented in a more specific schema, such as +GILS. In this module, a mapping of data to one of the MARC formats is +also thought of as a schema mapping (mapping the elements of the +record into fields consistent with the given MARC specification, prior +to actually converting the data to the ISO2709). This use of the +object identifier for USMARC as a schema identifier represents an +overloading of the OID which might not be entirely proper. However, +it represents the dual role of schema and record syntax which +is assumed by the MARC family in Z39.50. + +<it> +NOTE: The schema-mapping functions are so far limited to a +straightforward mapping of elements. This should be extended with +mechanisms for conversions of the element contents, and conditional +mappings of elements based on the record contents. +</it> + +These are the directives of the schema mapping file format: + +<descrip> +<tag>targetName <it/name/</tag> (m) A symbolic name for the target schema +of the table. Useful mostly for diagnostic purposes. + +<tag>targetRef <it/OID-name/</tag> (m) An OID name for the target schema. +This is used, for instance, by a server receiving a request to present +a record in a different schema from the native one. + +<tag>map <it/element-name target-path/</tag> (o,r) Adds +an element mapping rule to the table. +</descrip> + +<sect2>The MARC (ISO2709) Representation (.mar) Files + +<p> +This file provides rules for representing a record in the ISO2709 +format. The rules pertain mostly to the values of the constant-length +header of the record. + +<it>NOTE: This will be described better. We're in the process of +re-evaluating and most likely changing the way that MARC records are +handled by the system.</it> + +<sect2>Field Structure and Character Sets +<label id="field structure and character sets"> + +<p> +In order to provide a flexible approach to national character set +handling, Zebra allows the administrator to configure the set up the +system to handle any 8-bit character set — including sets that +require multi-octet diacritics or other multi-octet characters. The +definition of a character set includes a specification of the +permissible values, their sort order (this affects the display in the +SCAN function), and relationships between upper- and lowercase +characters. Finally, the definition includes the specification of +space characters for the set. + +The operator can define different character sets for different fields, +typical examples being standard text fields, numerical fields, and +special-purpose fields such as WWW-style linkages (URx). + +The field types, and hence character sets, are associated with data +elements by the .abs files (see above). The file <tt/default.idx/ +provides the association between field type codes (as used in the .abs +files) and the character map files (with the .chr suffix). The format +of the .idx file is as follows + +<descrip> +<tag>index <it/field type code/</tag>This directive introduces a new +search index code. The argument is a one-character code to be used in the +.abs files to select this particular index type. An index, roughly, +corresponds to a particular structure attribute during search. Refer +to section <ref id="search" name="Search">. + +<tag>sort <it/field code type/</tag>This directive introduces a +sort index. The argument is a one-character code to be used in the +.abs fie to select this particular index type. The corresponding +use attribute must be used in the sort request to refer to this +particular sort index. The corresponding character map (see below) +is used in the sort process. + +<tag>completeness <it/boolean/</tag>This directive enables or disables +complete field indexing. The value of the <it/boolean/ should be 0 +(disable) or 1. If completeness is enabled, the index entry will +contain the complete contents of the field (up to a limit), with words +(non-space characters) separated by single space characters +(normalized to &dquot; &dquot; on display). When completeness is +disabled, each word is indexed as a separate entry. Complete subfield +indexing is most useful for fields which are typically browsed (eg. +titles, authors, or subjects), or instances where a match on a +complete subfield is essential (eg. exact title searching). For fields +where completeness is disabled, the search engine will interpret a +search containing space characters as a word proximity search. + +<tag>charmap <it/filename/</tag> This is the filename of the character +map to be used for this index for field type. +</descrip> + +The contents of the character map files are structured as follows: + +<descrip> +<tag>lowercase <it/value-set/</tag>This directive introduces the basic +value set of the field type. The format is an ordered list (without +spaces) of the characters which may occur in &dquot;words&dquot; of +the given type. The order of the entries in the list determines the +sort order of the index. In addition to single characters, the +following combinations are legal: + +<itemize> +<item>Backslashes may be used to introduce three-digit octal, or +two-digit hex representations of single characters (preceded by <tt/x/). +In addition, the combinations +\\, \\r, \\n, \\t, \\s (space — remember that real space-characters +may ot occur in the value definition), and \\ are recognised, +with their usual interpretation. + +<item>Curly braces {} may be used to enclose ranges of single +characters (possibly using the escape convention described in the +preceding point), eg. {a-z} to entroduce the standard range of ASCII +characters. Note that the interpretation of such a range depends on +the concrete representation in your local, physical character set. + +<item>Paranthesises () may be used to enclose multi-byte characters - +eg. diacritics or special national combinations (eg. Spanish +&dquot;ll&dquot;). When found in the input stream (or a search term), +these characters are viewed and sorted as a single character, with a +sorting value depending on the position of the group in the value +statement. +</itemize> + +<tag>uppercase <it/value-set/</tag>This directive introduces the +upper-case equivalencis to the value set (if any). The number and +order of the entries in the list should be the same as in the +<tt/lowercase/ directive. + +<tag>space <it/value-set/</tag>This directive introduces the character +which separate words in the input stream. Depending on the +completeness mode of the field in question, these characters either +terminate an index entry, or delimit individual &dquot;words&dquot; in +the input stream. The order of the elements is not significant — +otherwise the representation is the same as for the <tt/upercase/ and +<tt/lowercase/ directives. + +<tag>map <it/value-set/ <it/target/</tag>This directive introduces a +mapping between each of the members of the value-set on the left to +the character on the right. The character on the right must occur in +the value set (the <tt/lowercase/ directive) of the character set, but +it may be a paranthesis-enclosed multi-octet character. This directive +may be used to map diacritics to their base characters, or to map +HTML-style character-representations to their natural form, etc. +</descrip> + +<sect1>Exchange Formats + +<p> +Converting records from the internal structure to en exchange format +is largely an automatic process. Currently, the following exchange +formats are supported: + +<itemize> +<item>GRS-1. The internal representation is based on GRS-1, so the +conversion here is straightforward. The system will create +applied variant and supported variant lists as required, if a record +contains variant information. + +<item>SUTRS. Again, the mapping is fairly straighforward. Indentation +is used to show the hierarchical structure of the record. All +&dquot;GRS&dquot; type records support both the GRS-1 and SUTRS +representations. + +<item>ISO2709-based formats (USMARC, etc.). Only records with a +two-level structure (corresponding to fields and subfields) can be +directly mapped to ISO2709. For records with a different structuring +(eg., GILS), the representation in a structure like USMARC involves a +schema-mapping (see section <ref id="schema-mapping" name="Schema +Mapping">), to an &dquot;implied&dquot; USMARC schema (implied, +because there is no formal schema which specifies the use of the +USMARC fields outside of ISO2709). The resultant, two-level record is +then mapped directly from the internal representation to ISO2709. See +the GILS schema definition files for a detailed example of this +approach. + +<item>Explain. This representation is only available for records +belonging to the Explain schema. + +<item>Summary. This ASN-1 based structure is only available for records +belonging to the Summary schema - or schema which provide a mapping +to this schema (see the description of the schema mapping facility +above). + +<item>SOIF. Support for this syntax is experimental, and is currently +keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All +abstract syntaxes can be mapped to the SOIF format, although nested +elements are represented by concatenation of the tag names at each +level. + +<item>XML. The use of XML as a transfer syntax in Z39.50 is not yet widely established +so the use of it here must be characterised as somewhat experimental. The +tag-names used are taken from the tag-set in use, except for local string tags +where the tag itself is passed through unchanged. + +</itemize> + +<sect>License + +<p> +Zebra +Copyright (c) 1995-2000 Index Data ApS. + +All rights reserved. + +Use and redistribution in source or binary form, with or without +modification, of any or all of this software and documentation is +permitted, provided that the following Conditions 1 to 6 set out below +are met. + +1. Unless prior specific written permission is obtained this copyright +and permission notice appear with all copies of the software and its +documentation. Notices of copyright or attribution which appear at the +beginning of any file must remain unchanged. + +2. The names of Index Data or the individual authors may not be used +to endorse or promote products derived from this software without +specific prior written permission. + +3. Source code or binary versions of this software and its documentation +may be used freely in not for profit applications limited to databases +of 100,000 records maximum. Other applications - such as publishing over +100,000 records, providing for-pay services, distributing a product based +in whole or in part on this software or its documentation, or generally +distributing this software or its documentation under a different license +require a commercial license from Index Data. + +4. The software may be installed and used for evaluation purposes in +conjunction with such commercially licensed applications for a trial +period no longer than 60 days. + +5. Unless a prior specific written agreement is obtained THIS SOFTWARE +IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND, EXPRESS, IMPLIED, +OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY WARRANTY OF +MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL +INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL, INDIRECT OR +CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES WHATSOEVER RESULTING +FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR NOT ADVISED OF THE +POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF LIABILITY, ARISING OUT OF +OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +6. Commercial licenses and support agreements for Zebra and related +Index Data products such as Z'bol (c) - and written agreements +relating to these Conditions may be obtained only from Index Data +or its appointed agents as follows: + +Index Data: www.indexdata.dk +Fretwell-Downing Informatics: www.fdgroup.co.uk +Fretwell-Downing Informatics USA: www.fdi.com + +<sect>About Index Data and the Zebra Server <p> Index Data is a consulting and software-development enterprise that -specialises in library and information management systems. Our +specialises in information management and retrieval applications. Our interests and expertise span a broad range of related fields, and one of our primary, long-term objectives is the development of a powerful information management -system with open network interfaces and hypermedia capabilities. +system with open network interfaces and hypermedia capabilities. Zebra is an +important component in this strategy. + +We make this software available free of charge for not-for-profit +purposes, as a service to the networking community, and to further +the development and use of quality software for open network +communication. We encourage your comments and questions if you have ideas, things +you would like to see in future versions, or things you would like to +contribute. + +If you like this software, and would like to use all or part of it in +a commercial product, or to provide a commercial database service, +please contact us. The Z'mbol Information System represents the commercial +variant of Zebra. It includes full support; additional functionality and +performance-boosting features, and it has what we think is a very exciting +development path. -We make this software available free of charge, on a fairly unrestrictive -license; as a service to the networking community, and to further the -development of quality software for open network communication. - -We'll be happy to answer questions about the software, and about ourselves -in general. - -<tscreen> -Index Data&nl -Ryesgade 3&nl -2200 København N&nl -DENMARK -</tscreen> +<tscreen><verb> +Index Data +Ryesgade 3 +DK-2200 Copenhagen N +</verb></tscreen> <p> <tscreen><verb> Phone: +45 3536 3672 Fax : +45 3536 0449 -Email: info@index.ping.dk +Email: info@indexdata.dk </verb></tscreen> -<sect>References +The <it>Random House College Dictionary</it>, 1975 edition +offers this definition of the +word &dquot;Zebra&dquot;: -<p> +<it> +Zebra, n., any of several horselike, African mammals of the genus Equus, +having a characteristic pattern of black or dark-brown stripes on +a whitish background. +</it> </article>