Zebra Server - Administrators's Guide and Reference <author>Index Data, <tt/info@index.ping.dk/ <date>$Revision: 1.7 $ <abstract> The Zebra information server combines a versatile fielded/free-text search engine with a Z39.50-1995 frontend to provide a powerful and flexible information management system. This document explains the procedure for installing and configuring the system, and outlines the possibilities for managing data and providing Z39.50 services using the software. </abstract> <toc> <sect>Introduction <sect1>Overview The Zebra system is a fielded free-text indexing and retrieval engine with a Z39.50 frontend. You can use any commercial or freeware Z39.50 client to access data stored in Zebra. The Zebra server is our first step towards the development of a fully configurable, open information system. Eventually, it will be paired off with a powerful Z39.50 client to support complex information management tasks within almost any application domain. We're making the server available now because it's no fun to be in the open information retrieval business all by yourself. We want to allow people with interesting data to make their things available in interesting ways, without having to start out by implementing yet another protocol stack from scratch. This document is an introduction to the Zebra system. It will tell you how to compile the software, and how to prepare your first database. It also explains how the server can be configured to give you the functionality that you need. You should read <it/Specifying and Using Application (Database) Profiles/, which is bundled with the YAZ documentation, to learn how records are formatted, and how you can configure Zebra to handle different types of Z39.50 application profiles. <sect1>Features This is a listing of some of the most important features of the system. <itemize> <item> Supports updating - records can be added and deleted without rebuilding the index. <item> Supports large databases - files for indices, etc. can be automatically partitioned over multiple disks. <item> Supports arbitrarily complex records - base input format is an SGML-like syntax which allows nested (structured) data elements, as well as variant forms of data. <item> Supports boolean queries as well as relevance-ranking (free-text) searching. Right truncation and masking in terms are supported, as well as full regular expressions. <item> Supports multiple concrete syntaxes for record exchange (depending on the configuration): GRS-1, SUTRS, ISO2709 (*MARC). Records can be mapped between record syntaxes and schema on the fly. <item> Protocol support: <itemize> <item> Protocol facilities: Init, Search, Retrieve, Browse. <item> Piggy-backed presents are honored in the search-request. <item> Named result sets are supported. <item> Easily configured to support different application profiles, with tables for attribute sets, tag sets, and abstract syntaxes. Additional tables control facilities such as element mappings to different schema (eg., GILS-to-USMARC). <item> Complex composition specifications using Espec-1 are partially supported (simple element requests only). <item> Element Set Names are established the Espec-1 capability of the system, and are given in configuration files as simple element requests (and possibly variant requests). <item> Some variant support (not fully implemented yet). <item> Using the YAZ toolkit for the protocol implementation, the server can utilise a plug-in XTI/mOSI implementation (not included) to provide SR services over an OSI stack, as well as Z39.50 over TCP/IP. </itemize> </itemize> <sect1>Future Work This is an early alfa-release of the software, to allow you to look at it - try it out, and assess whether it can be of use to you. We expect this version to be followed by a succession of beta-releases until we arrive at a stable first version. These are some of the plans that we have for the software in the near and far future, approximately ordered after their relative importance. Items marked with an asterisk will be implemented before the last beta release. <itemize> <item> *Allow the system to handle additional input formats. Specifically MARC records and general, structured ASCII records (such as mail/news files) parameterized by regular expressions. <item> *Complete the support for variants. Finalize support for the WAIS retrieval methodology. <item> *Finalize the data element <it/include/ facility to support multimedia data elements in records. <item> *Port the system to Windows NT. <item> Add robust database updating - tolerant to crashes or hard interrupts during register updating. <item> Add online updating, to permit register updating while users are accessing the system. <item> Add index and data compression to save disk space. <item> Add more sophisticated relevance ranking mechanisms. Add support for soundex and stemming. Add relevance feedback support. <item> Add Explain support. <item> Add support for very large records by implementing segmentation and variant pieces. <item> Support the Item Update extended service of the protocol. <item> The Zebra search engine supports approximate string matching in the index. We'd like to find a way to support and control this from RPN. <item> We want to add a management system that allows you to control your databases and configuration tables from a graphical interface. We'll probably use Tcl/Tk to stay platform-independent. </itemize> Programmers thrive on user feedback. If you are interested in a facility that you don't see mentioned here, or if there's something you think we could do better, please drop us a mail. If you think it's all really neat, you're of course welcome to drop us a line saying that, too. <sect>Introduction <sect>Compiling the software Zebra uses the YAZ package to implement Z39.50, so you have to compile YAZ before going further. Specifically, Zebra uses the YAZ header files in <tt>yaz/include/..</tt> and its public library <tt>yaz/lib/libyaz.a</tt>. As with YAZ, an ANSI C compiler is required in order to compile the Zebra server system — GNU C works fine. Unpack the Zebra software. You might put Zebra in the same directory level as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-</tt>.., then Zebra is placed in ..<tt>/src/zebra-</tt>. Edit the top-level <tt>Makefile</tt> in the Zebra directory in which you specify the location of YAZ by setting make variables. The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with MOSI support. Some systems, such as Solaris, have separate socket libraries and for those systems you need to specify the <tt>NETLIB</tt> variable. When you are done editing the <tt>Makefile</tt> type: <tscreen><verb> $ make </verb></tscreen> If successful, two executables have been created in the sub-directory <tt/index/. <descrip> <tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine. <tag><tt>zebraidx</tt></tag> The administrative tool for the search index. </descrip> <sect>Quick Start This section will get you started quickly! We will try to index a few sample GILS records that are included with the Zebra distribution. Go to the <tt>test</tt> subdirectory. There you will find a configuration file named <tt>zebra.cfg</tt> with the following contents: <tscreen><verb> # Where are the YAZ tables located. profilePath: /usr/local/yaz # Files that describe the attribute sets supported. attset: bib1.att attset: gils.att </verb></tscreen> Now, edit the file and set <tt>profilePath</tt> to the path of the YAZ profile tables (sub directory <tt>tab</tt> of YAZ). The 48 test records are located in the sub directory <tt>records</tt>. To index these, type: <tscreen><verb> $ ../index/zebraidx -t grs update records </verb></tscreen> In the command above the option <tt>-t</tt> specified the record type — in this case <tt>grs</tt>. The word <tt>update</tt> followed by a directory root updates all files below that directory node. If your indexing command went successful, you are now ready to fire up a server. To start a server on port 2100, type: <tscreen><verb> $ ../index/zebrasrv tcp:@:2100 </verb></tscreen> The Zebra index that you've just made has one database called Default. It will return either USMARC, GRS-1, or SUTRS depending on what your client asks for. To test the server, you can use any Z39.50 client (1992 or later). For instance, you can use the demo client that comes with YAZ: Just cd to the <tt/client/ subdirectory of the YAZ distribution and type: <tscreen><verb> $ client tcp:localhost:2100 </verb></tscreen> When the client has connected, you can type: <tscreen><verb> Z> find surficial Z> show 1 </verb></tscreen> To try other retrieval formats for the same record, try: <tscreen><verb> Z>format sutrs Z>show 1 Z>format grs-1 Z>show 1 </verb></tscreen> If you've made it this far, there's a reasonably good chance that you've made it through the compilation OK. <sect>Administrating Zebra Unlike many other retrieval systems, Zebra offers incremental modifications of an existing index. Needless to say, these facilities make the administration of Zebra a bit more complicated than systems that use the &dquot;index-it-all&dquot; approach. Normally, when Zebra modifies the index it reads a number of records that you specify. Depending on your specifications and on the contents of each record one the following events take place for each record: <descrip> <tag>Insert</tag> The record is indexed as if it never occurred before. Either the Zebra system doesn't know how to identify the record or Zebra can identify the record but didn't find it to be already indexed. <tag>Modify</tag> The record has already been indexed. In this case either the contents of the record or the location (file) of the record indicates that it has been indexed before. <tag>Delete</tag> The record is deleted from the index. As in the update-case it must be able to identify the record. </descrip> Please note that in both the modify- and delete- case the Zebra indexer must be able to make a unique key that identifies the record in question. To administrate the Zebra retrieval system, you run the <tt>zebraidx</tt> program. This program supports a number of options which are preceded by a minus, and a few commands (not preceded by minus). Both the Zebra administrative tool and the Z39.50 server share a set of index files and a global configuration file. The name of the configuration file defaults to <tt>zebra.cfg</tt>. The configuration file includes specifications on how to index various kinds of records and where the other configuration files are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> must be run in the same directory where the configuration file if you do not indicate the location of the configuration file by option <tt>-c</tt>. <sect1>Record types Indexing is a record-per-record process, in which either insert/modify/delete will occur. Before a record is indexed search keys are extracted from whatever might be the layout the original record (sgml,html,text, etc..). The Zebra system currently only supports SGML-like, structured records and unstructured text records. To specify a particular extraction process, use either the command line option <tt>-t</tt> or specify a <tt>recordType</tt> setting in the configuration file. <sect1>The Zebra Configuration File The Zebra configuration file, read by <tt>zebraidx</tt> and <tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified by <tt>-c</tt> option. You can edit the configuration file with a normal text editor. Setting names and values are seperated by colons in the file. Lines starting with a hash sign (<tt/#/) are treated as comments. A set of records that share common characteristics are called a group. When <tt>zebraidx</tt> is run and you wish to address a given group you specify that group with the <tt>-g</tt> option. In this case settings that have the group name as their prefix will be used by <tt>zebraidx</tt> and not default values. The default values have no prefix. The group is written before the option itself separated by a dot. For instance, to set the record type for group <tt/public/ to <tt/grs/ (structured records) you would write: <tscreen><verb> public.recordType: grs </verb></tscreen> To set the default value of the record type to text write: <tscreen><verb> recordType: text </verb></tscreen> The configuration settings are summarized below. They will be explained further in the following sections. <descrip> <tag><it>group</it>recordType<it>name</it></tag> Specifies how records with the file extension <it>name</it> should be handled by the indexer. This option may also be specified as a command line option (<tt>-t</tt>). <tag><it>group</it>recordId</tag> Specifies how the record is to be identified when updated. <tag><it>group</it>database</tag> Specifies the Z39.50 database. <tag><it>group</it>storeKeys</tag> Specifies whether key information should be saved for a given group of records. If you plan to update/delete this type of records later this should be specified as 1; otherwise it should be 0 (default). <tag><it>group</it>storeData</tag> Specifies whether the records should be stored internally in the Zebra system tables. If you want to maintain the raw records yourself, this option should be false (0). If you want Zebra to take care of the records for you, it should be true(1). <tag>register</tag> Specifies the location of the various files that Zebra uses to represent your system. <tag>profilePath</tag> Specifies the location of profile specification paths. <tag>attset</tag> Specifies the filename(s) of attribute set files for use in searching. </descrip> <sect1>Locating Records The default behaviour of the Zebra system is to reference the records from their original location, i.e. where they were found when you ran <tt/zebraidx/. If your records files are temporary - for example if you retrieve them from the outside, or if they where temporarily mounted on a CD-ROM, you may want Zebra to make a copy of them. To do this, you specify 1 (true) in the <tt>storedata</tt> setting. When the Z39.50 server retrieves records they will be read from the internal file structures of the system. <sect1>Indexing with no Record IDs (Simple Indexing) If you have a set of records that you <em/never/ wish to delete or modify you may find &dquot;indexing without records IDs&dquot; convenient. This indexing method uses less space than the other methods and is simple to use. To use this method, you simply don't provide the <tt>recordId</tt> entry for the group of files that you index. To add a set of records you use <tt>zebraidx</tt> with the <tt>update</tt> command. The <tt>update</tt> command will always add all of the records to the index becuase Zebra doesn't know how to match the new set of records with existing records. Consider a system in which you have a group of text files called <tt>simple</tt>. That group of records should belong to a Z39.50 database called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice: <tscreen><verb> profilePath: /usr/local/yaz attset: bib1.att attset: gils.att simple.recordType: text simple.database: textbase </verb></tscreen> Since the existing records in an index can not be addressed by their IDs, it is impossible to delete records when using this method. <sect1>Indexing with File Record IDs If you have a set of external records that you wish to index you may use the file key feature of the Zebra system. In short, the file key feature mirrors a directory structure and its files efficiently. To perform indexing of a directory with file keys, you specify the top-level directory after the <tt>update</tt> command. The command will recursively traverse the directories and compare each with whatever have been indexed before in the same directory. If a file is new (not in the previous version of the directory) it is inserted; if a file was already indexed and it has been modified since the last insertion the index is also modified; if a file is missing since the last visit it is deleted from the index. The resulting system is easy to administer. To delete a record you simply have to delete the corresponding file (with <tt/rm/). To force update of a given file, you may use the <tt>touch</tt> command. And to add files create new files (or directories with files). For your changes to take effect you must run <tt>zebraidx</tt> with the same directory root again. To use this method, you must specify <tt>file</tt> as the value of <tt>recordId</tt> in the configuration file. In the configuration also set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra indexer must save additional information per record in order to modify/delete the records at a later time. For example, to update group <tt>esdd</tt> records below <tt>/home/grs</tt> you could type: <tscreen><verb> $ zebraidx -g esdd update /home/grs </verb></tscreen> The corresponding configuration file includes: <tscreen><verb> esdd.recordId: file esdd.recordType: grs esdd.storeKeys: 1 </verb></tscreen> Important note: You cannot start out with a group of records with simple indexing (no record IDs as in the previous section) and then later enable file record Ids. Zebra must know from the first time that you index the group that the files should be indexed with file record IDs. You cannot explicitly delete records when using this method. Instead you have to delete the files from the file system (or remove them) and then run <tt>zebraidx</tt> with the <tt>update</tt> again. <sect1>Indexing with General Record IDs When using this method you specify an (almost) arbritrary record key based on the contents of the record itself and other system information. If you have a group of records that have an external ID associated with each records, this method is convenient. For example, the record may contain a title or a unique ID-number. In either case you specify the Z39.50 attribute set and use-attribute location in which this information is stored. As before, the record ID is defined by the <tt>recordId</tt> setting in the configuration file. The value of the record ID specification consists of one or more tokens separated by whitespace. The resulting ID is represented in the index by concatenating the tokens and separating them by ASCII value (1). There are three kinds of tokens: <descrip> <tag>Internal record info</tag> The token refers to a key that is extracted from the record. The syntax of this token is <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the attribute set ordinal number and <em/use/ is the use value of the attribute. <tag>System variable</tag> The system variables are preceded by <verb>$</verb> and immediately followed by the system variable name, which may one of <descrip> <tag>group</tag> Group name. <tag>database</tag> Current database specified. <tag>type</tag> Record type. </descrip> <tag>Constant string</tag> A string used as part of id — surrounded by single- or double quotes. </descrip> The test GILS records that comes with the Zebra distribution contain a unique ID in the Control-Identifier field. This field is mapped to the Bib-1 use attribute 1007. To use this field as a record id, specify <tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the configuration file. If you have other record types that don't contain an ID in the same field, you might add the record type in the record id of the gils records as well, to prevent matches of other types of records. In this case the recordId might be set like this: <tscreen><verb> gils.recordId: $type (1,1007) </verb></tscreen> As for the file record id case described in the previous section updating your system is simply a matter of running <tt>zebraidx</tt> with the <tt>update</tt> command. However, the update with general keys is considerably slower than with file record IDs, since all files visited must be (re)read to find their IDs. You may have noticed that when using the general record IDs method, you can only add or modify existing records with the <tt>update</tt> command. If you wish to delete records, you must use another command, <tt>delete</tt>, which a root directory as a parameter. This will remove all records that match the files below the root directory. <sect1>Register location Normally, the index files that form dictionaries, inverted files, record info, etc., are stored in the directory where you run <tt>zebraidx</tt>. If you wish to store these, possibly large, files somewhere else, you must add the <tt>register</tt> entry to the configuration file. Furthermore, the Zebra system allows its file structures to span multiple file systems, which is useful if a very large number of records are stored. The value <tt>register</tt> of register is a sequence of tokens. Each token takes the form: <tscreen> dir<tt>:</tt>size. </tscreen> The dir specifies a directory in which index files will be stored and the size specifies the maximum size of all files in that directory. The Zebra indexer system fill each directory in the order specified and use the next specified directories as needed. The size is an integer followed by a qualifier code, <tt>M</tt> for megabytes, <tt>k</tt> for kilobytes. For instance, if you have two spare disks :) and the first disk is mounted on <tt>/d1</tt> and has 200 Mb of free space and the second, mounted on <tt>/d2</tt> has 300 Mb, you could put this entry in your configuration file: <tscreen><verb> register: /d1:200M /d2:300M </verb></tscreen> <sect>The Z39.50 Server <sect1>Running the server The server <tt>zebrasrv</tt> supports the same set of options as the test server <tt>ztest</tt> that comes with YAZ. As for the <tt>zebraidx</tt> the option <tt>-c</tt> specifies the configuration filename. When the Zebra server is executed with its normal log level it prints (not too detailed) information about the incoming queries. This is useful if you don't happen to know what attributes your client sends. Note that the server doesn't support the static mode (-S). <sect>License Copyright © 1995, Index Data. All rights reserved. Use and redistribution in source or binary form, with or without modification, of any or all of this software and documentation is permitted, provided that the following conditions are met: 1. This copyright and permission notice appear with all copies of the software and its documentation. Notices of copyright or attribution which appear at the beginning of any file must remain unchanged. 2. The names of Index Data or the individual authors may not be used to endorse or promote products derived from this software without specific prior written permission. 3. Source code or binary versions of this software and its documentation may be used in not-for-profit applications. For profit aplications - including marketing a product based in whole or in part on this software, or providing for-pay database services - must obtain a commercial license from Index Data. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND, EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL, INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. <sect>About Index Data Index Data is a consulting and software-development enterprise that specialises in library and information management systems. Our interests and expertise span a broad range of related fields, and one of our primary, long-term objectives is the development of a powerful information management system with open network interfaces and hypermedia capabilities. We make this software available free of charge for noncommercial purposes, as a service to the networking community, and to further the development of quality software for open network communication. We'll be happy to answer questions about the software, and about ourselves in general. <tscreen> Index Data&nl Ryesgade 3&nl DK-2200 København N&nl </tscreen> <tscreen><verb> Phone: +45 3536 3672 Fax : +45 3536 0449 Email: info@index.ping.dk </verb></tscreen> The <it>Random House College Dictionary</it>, 1975 edition offers this definition of the word &dquot;Zebra&dquot;: <it> Zebra, n., any of several horselike, African mammals of the genus Equus, having a characteristic pattern of black or dark-brown stripes on a whitish background. </it> </article>