1 <!doctype linuxdoc system>
4 $Id: zebra.sgml,v 1.4 1995-11-28 16:10:45 quinn Exp $
8 <title>Zebra server Administrators's Guide and Reference
9 <author>Index Data, <tt/info@index.ping.dk/
10 <date>$Revision: 1.4 $
12 The Zebra information server combines a versatile fielded/free-text
13 search engine with a Z39.50-1995 frontend to provide a powerful and flexible
14 information management system. This document explains the procedure for
15 installing and configuring the system, and outlines the possibilities
16 for managing data and providing Z39.50
17 services using the software.
32 This is a listing of some of the most important features of the
38 Supports updating - records can be added and deleted without
42 Supports large databases - files for indices, etc. can be
43 automatically partitioned over multiple disks.
46 Supports arbitrarily complex records - base input format is an
47 SGML-like syntax which allows nested (structured) data elements, as
48 well as variant forms of data.
51 Supports boolean queries as well as relevance-ranking (free-text)
52 searching. Right truncation and masking in terms are supported, as
53 well as full regular expressions.
56 Supports multiple concrete syntaxes
57 for record exchange (depending on the configuration): GRS-1, SUTRS,
58 ISO2709 (*MARC). Records can be mapped between record syntaxes and
67 Protocol facilities: Init, Search, Retrieve, Browse.
70 Piggy-backed presents are honored in the search-request.
73 Named result sets are supported.
76 Easily configured to support different application profiles, with
77 tables for attribute sets, tag sets, and abstract syntaxes.
78 Additional tables control facilities such as element mappings to
79 different schema (eg., GILS-to-USMARC).
82 Complex composition specifications using Espec-1 are partially
83 supported (simple element requests only).
86 Element Set Names are established the Espec-1 capability of the
87 system, and are given in configuration files as simple element
88 requests (and possibly variant requests).
91 Some variant support (not fully implemented yet).
94 Using the YAZ toolkit for the protocol implementation, the
95 server can utilise a plug-in XTI/mOSI implementation (not included) to
96 provide SR services over an OSI stack, as well as Z39.50 over TCP/IP.
105 This is an early alfa-release of the software, to allow you to look at
106 it - try it out, and assess whether it can be of use to you. We expect
107 this version to be followed by a succession of beta-releases until we
108 arrive at a stable first version.
110 These are some of the plans that we have for the software in the near
111 and far future, approximately ordered after their relative importance.
113 asterisk will be implemented before the
119 *Allow the system to handle additional input formats. Specifically
120 MARC records and general, structured ASCII records (such as mail/news
121 files) parameterized by regular expressions.
124 *Complete the support for variants. Finalize support for the WAIS
125 retrieval methodology.
128 *Finalize the data element <it/include/ facility to support multimedia
129 data elements in records.
132 *Port the system to Windows NT.
135 Add robust database updating - tolerant to crashes or hard interrupts
136 during register updating.
139 Add online updating, to permit register updating while users are
140 accessing the system.
143 Add index and data compression to save disk space.
146 Add more sophisticated relevance ranking mechanisms. Add support for soundex
147 and stemming. Add relevance feedback support.
153 Add support for very large records by implementing segmentation and
157 Support the Item Update extended service of the protocol.
160 The Zebra search engine supports approximate string matching in the
161 index. We'd like to find a way to support and control this from RPN.
165 Programmers thrive on user feedback. If you are interested in a facility that
166 you don't see mentioned here, or if there's something you think we
167 could do better, please drop us a mail. If you think it's all really
168 neat, you're of course welcome to drop us a line saying that, too.
171 <sect>Compiling the software
174 Zebra uses the YAZ package to implement Z39.50, so you
175 have to compile YAZ before going further. Specifically, Zebra uses
176 the YAZ header files in <tt>yaz/include/..</tt> and its public library
177 <tt>yaz/lib/libyaz.a</tt>.
179 As with YAZ, an ANSI C compiler is required in order to compile the Zebra
180 server system — GNU C works fine.
182 Unpack the Zebra software. You might put Zebra in the same directory level
183 as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-</tt>.., then
184 Zebra is placed in ..<tt>/src/zebra-</tt>.
186 Edit the top-level <tt>Makefile</tt> in the Zebra directory in which
187 you specify the location of YAZ by setting make variables.
188 The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with
189 MOSI support. Some systems, such as Solaris, have separate socket
190 libraries and for those systems you need to specify the
191 <tt>NETLIB</tt> variable.
193 When you are done editing the <tt>Makefile</tt> type:
198 If successful, two executables have been created in the sub-directory
201 <tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
202 <tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
208 This section will get you started quickly! We will try to index a few sample
209 GILS records that are included with the Zebra distribution. Go to the
210 <tt>test</tt> subdirectory. There you will find a configuration
211 file named <tt>zebra.cfg</tt> with the following contents:
213 # Where are the YAZ tables located.
214 profilePath: /usr/local/yaz
216 # Files that describe the attribute sets supported.
221 Now, edit the file and set <tt>profilePath</tt> to the path of the
222 YAZ profile tables (sub directory <tt>tab</tt> of YAZ).
224 The 48 test records are located in the sub directory <tt>records</tt>.
225 To index these, type:
227 $ ../index/zebraidx -t grs update records
230 In the command above the option <tt>-t</tt> specified the record
231 type — in this case <tt>grs</tt>. The word <tt>update</tt> followed
232 by a directory root updates all files below that directory node.
234 If your indexing command went successful, you are now ready to
235 fire up a server. To start a server on port 2100, type:
237 $ ../index/zebrasrv tcp:@:2100
240 The Zebra index that you've just made has one database called Default. It will
241 return either USMARC, GRS-1, or SUTRS depending on what your client asks
244 To test the server, you can use any Z39.50 client (1992 or later). For
245 instance, you can use the demo client that comes with YAZ: Just cd to
246 the <tt/client/ subdirectory of the YAZ distribution and type:
249 $ client tcp:localhost:2100
252 When the client has connected, you can type:
259 To try other retrieval formats for the same record, try:
268 If you've made it this far, there's a reasonably good chance that
269 you've made it through the compilation OK.
271 <sect>Administrating Zebra
275 Unlike many other retrieval systems, Zebra offers incremental
276 modifications of an existing index. Needless to say, these facilities
277 make the administration of Zebra a bit more complicated than
278 systems that use the &dquot;index-it-all&dquot; approach.
280 Normally, when Zebra modifies the index it reads a number of records
282 Depending on your specifications and on the contents of each record
283 one the following events take place for each record:
285 <tag>Insert</tag> The record is indexed as if it never occurred
286 before. Either the Zebra system doesn't know how to identify the record or
287 Zebra can identify the record but didn't find it to be already indexed.
288 <tag>Modify</tag> The record has already been indexed. In this case
289 either the contents of the record or the location (file) of the record
290 indicates that it has been indexed before.
291 <tag>Delete</tag> The record is deleted from the index. As in the
292 update-case it must be able to identify the record.
295 Please note that in both the modify- and delete- case the Zebra
296 indexer must be able to make a unique key that identifies the record in
299 To administrate the Zebra retrieval system, you run the
300 <tt>zebraidx</tt> program. This program supports a number of options
301 which are preceded by a minus, and a few commands (not preceded by
304 Both the Zebra administrative tool and the Z39.50 server share a
305 set of index files and a global configuration file. The
306 name of the configuration file defaults to <tt>zebra.cfg</tt>.
307 The configuration file includes specifications on how to index
308 various kinds of records and where the other configuration files
309 are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
310 be run in the same directory where the configuration file if you do
311 not indicate the location of the configuration file by option
316 Indexing is a record-per-record process, in which
317 either insert/modify/delete will occur. Before a record is indexed
318 search keys are extracted from whatever might be the layout the
319 original record (sgml,html,text, etc..). The Zebra system
320 currently only supports SGML-like, structured records and unstructured text
322 To specify a particular extraction process, use either the
323 command line option <tt>-t</tt> or specify a
324 <tt>recordType</tt> setting in the configuration file.
326 <sect1>The Zebra Configuration File
328 The Zebra configuration file, read by <tt>zebraidx</tt> and
329 <tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
330 by <tt>-c</tt> option.
332 You can edit the configuration file with a normal text editor.
333 Setting names and values are seperated by colons in the file. Lines
334 starting with a hash sign (<tt/#/) are treated as comments.
336 A set of records that share common characteristics are called a group.
337 When <tt>zebraidx</tt> is run and you wish to address a given group
338 you specify that group with the <tt>-g</tt> option. In this case
339 settings that have the group name as their prefix will be used
340 by <tt>zebraidx</tt> and not default values. The default values have no prefix.
342 The group is written before the option itself separated by a dot.
343 For instance, to set the record type for group <tt/public/ to <tt/grs/ (structured records)
347 public.recordType: grs
350 To set the default value of the record type to text write:
356 The configuration settings are summarized below. They will be
357 explained further in the following sections.
360 <tag><it>group</it>recordType<it>name</it></tag>
361 Specifies how records with the file extension <it>name</it> should
362 be handled by the indexer. This option may also be specified
363 as a command line option (<tt>-t</tt>).
364 <tag><it>group</it>recordId</tag>
365 Specifies how the record is to be identified when updated.
366 <tag><it>group</it>database</tag>
367 Specifies the Z39.50 database.
368 <tag><it>group</it>storeKeys</tag>
369 Specifies whether key information should be saved for a given
370 group of records. If you plan to update/delete this type of
371 records later this should be specified as 1; otherwise it
372 should be 0 (default).
373 <tag><it>group</it>storeData</tag>
374 Specifies whether the records should be stored internally
375 in the Zebra system tables. If you want to maintain the raw records yourself,
376 this option should be false (0). If you want Zebra to take care of the records
377 for you, it should be true(1).
379 Specifies the location of the various files that Zebra uses to represent
381 <tag>profilePath</tag>
382 Specifies the location of profile specification paths.
384 Specifies the filename(s) of attribute set files for use in searching.
387 <sect1>Locating Records
389 The default behaviour of the Zebra system is to reference the
390 records from their original location, i.e. where they were found when you
393 If your records files are temporary - for example if you retrieve
394 them from the outside, or if they where temporarily mounted on a CD-ROM,
395 you may want Zebra to make a copy of them. To do this,
396 you specify 1 (true) in the <tt>storedata</tt> setting. When
397 the Z39.50 server retrieves records they will be read from the
398 internal file structures of the system.
400 <sect1>Indexing with no Record IDs (Simple Indexing)
403 If you have a set of records that you <em/never/ wish to delete
404 or modify you may find &dquot;indexing without records IDs&dquot; convenient.
405 This indexing method uses less space than the other methods and
408 To use this method, you simply don't provide the <tt>recordId</tt> entry
409 for the group of files that you index. To add a set of records you use
410 <tt>zebraidx</tt> with the <tt>update</tt> command. The
411 <tt>update</tt> command will always add all of the records to the index
412 becuase Zebra doesn't know how to match the new set of records with
415 Consider a system in which you have a group of text files called
416 <tt>simple</tt>. That group of records should belong to a Z39.50 database
417 called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice:
420 profilePath: /usr/local/yaz
423 simple.recordType: text
424 simple.database: textbase
427 <sect1>Indexing with File Record IDs
430 If you have a set of external records that you wish to index you may
431 use the file key feature of the Zebra system. In short, the file key
432 feature mirrors a directory structure and its files efficiently. To
433 perform indexing of a directory with file keys, you specify the top-level
434 directory after the <tt>update</tt> command. The command will recursively
435 traverse the directories and compare each with whatever have been
436 indexed before in the same directory. If a file is new (not in
437 the previous version of the directory) it is inserted;
438 if a file was already indexed and it has been modified
439 since the last insertion the index is also modified; if a file is missing
440 since the last visit it is deleted from the index.
442 The resulting system is easy to administer. To delete a record
443 you simply have to delete the corresponding file (with <tt/rm/).
444 To force update of a given file, you may use the <tt>touch</tt>
445 command. And to add files create new files (or directories with files).
446 For your changes to take effect you must run <tt>zebraidx</tt> with
447 the same directory root again.
449 To use this method, you must specify <tt>file</tt> as the value
450 of <tt>recordId</tt> in the configuration file. In the configuration
451 also set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra
452 indexer must save additional information per record in order to
453 modify/delete the records at a later time.
455 For example, to update group <tt>esdd</tt> records below
456 <tt>/home/grs</tt> you could type:
458 $ zebraidx -g esdd update /home/grs
461 The corresponding configuration file includes:
468 <em>Important note: You cannot start out with a group of records with simple
469 indexing (no record IDs as in the previous section) and then later
470 enable file record Ids. Zebra must know from the first time that you
472 the files should be indexed with file record IDs.
475 <sect1>Indexing with General Record IDs
477 When using this method you specify an (almost) arbritrary record key
478 based on the contents of the record itself and other system
479 information. If you have a group of records that have an external
480 ID associated with each records, this method is convenient. For
481 example, the record may contain a title or a unique ID-number. In either
482 case you specify the Z39.50 attribute set and use-attribute location
483 in which this information is stored.
485 As before, the record ID is defined by the <tt>recordId</tt> setting
486 in the configuration file. The value of the record ID specification
487 consists of one or more tokens separated by whitespace. The resulting
489 represented in the index by concatenating the tokens and separating them by
492 There are three kinds of tokens:
494 <tag>Internal record info</tag> The token refers to a key that is
495 extracted from the record. The syntax of this token is
496 <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the
497 attribute set ordinal number and <em/use/ is the use value of the attribute.
498 <tag>System variable</tag> The system variables are preceded by
499 <tt>$</tt> and immediately followed by the system variable name, which
502 <tag>group</tag> Group name.
503 <tag>database</tag> Current database specified.
504 <tag>type</tag> Record type.
506 <tag>Constant string</tag> A string used as part of id — surrounded
507 by single- or double quotes.
510 The test GILS records that comes with the Zebra distribution contain a
512 in the Control-Identifier field. This field is mapped to the Bib-1
513 use attribute 1007. To use this field as a record id, specify
514 <tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the
515 configuration file. If you have other record types that don't
516 contain an ID in the same field, you might add the record type
517 in the record id of the gils records as well, to prevent matches
518 of other types of records. In this case the recordId might be
521 gils.recordId: $type (1,1007)
524 As for the file record id case described in the previous section
525 updating your system is simply a matter of running <tt>zebraidx</tt>
526 with the <tt>update</tt> command. However, the update with general
527 keys is considerably slower than with file record IDs, since all files
528 visited must be (re)read to find their IDs.
530 <sect1>Register location
533 Normally, the index files that form dictionaries, inverted
534 files, record info, etc., are stored in the directory where you run
535 <tt>zebraidx</tt>. If you wish to store these, possibly large, files
536 somewhere else, you must add the <tt>register</tt> entry to the
537 configuration file. Furthermore, the Zebra system allows indexes to
538 span multiple file systems, which is useful if a very large number of
541 The value <tt>register</tt> of register is a sequence of tokens.
542 Each token takes the form:
544 <em>dir</em><tt>:</tt><em>size</em>.
546 The <em>dir</em> specifies a directory in which index files will be
547 stored and the <em>size</em> specifies the maximum size of all
548 files in that directory. The Zebra indexer system fill each directory
549 in the order specified and use the next specified directories as needed.
550 The <em>size</em> is an integer followed by a qualifier
551 code, <tt>M</tt> for megabytes, <tt>k</tt> for kilobytes.
553 For instance, if you have two spare disks :) and the first disk is mounted
554 on <tt>/d1</tt> and has 200 Mb of free space and the
555 second, mounted on <tt>/d2</tt> has 300 Mb, you could
556 put this entry in your configuration file:
558 register: /d1:200M /d2:300M
561 <sect>The Z39.50 Server
565 <sect1>Running the server
567 The server <tt>zebrasrv</tt> supports the same set of options as the
568 test server <tt>ztest</tt> that comes with YAZ. As for the
569 <tt>zebraidx</tt> the option <tt>-c</tt> specifies the configuration
570 filename. When the Zebra server is executed with its normal log level it
571 prints (not too detailed) information about the incoming queries.
572 This is useful if you don't happen to know what attributes your client sends.
574 Note that the server doesn't support the static mode (-S).
576 <sect1>How the server handles queries
578 What elements of Bib-1 are supported and where are result sets
584 Copyright © 1995, Index Data.
588 Use and redistribution in source or binary form, with or without
589 modification, of any or all of this software and documentation is
590 permitted, provided that the following conditions are met:
592 1. This copyright and permission notice appear with all copies of the
593 software and its documentation. Notices of copyright or attribution
594 which appear at the beginning of any file must remain unchanged.
596 2. The names of Index Data or the individual authors may not be used to
597 endorse or promote products derived from this software without specific
598 prior written permission.
600 3. Source code or binary versions of this software and its documentation
601 may be used in not-for-profit applications. For profit aplications -
602 including marketing a product based in whole or in part on this software,
603 or providing for-pay database services - must obtain a commercial
604 license from Index Data.
606 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
607 EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
608 WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
609 IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
610 INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES
611 WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR
612 NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF
613 LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
616 <sect>About Index Data
619 Index Data is a consulting and software-development enterprise that
620 specialises in library and information management systems. Our
621 interests and expertise span a broad range of related fields, and one
622 of our primary, long-term objectives is the development of a powerful
623 information management
624 system with open network interfaces and hypermedia capabilities.
626 We make this software available free of charge for noncommercial
627 purposes, as a service to the networking community, and to further
628 the development of quality software for open network communication.
630 We'll be happy to answer questions about the software, and about ourselves
636 DK-2200 København N&nl
643 Email: info@index.ping.dk
646 The <it>Random House College Dictionary</it>, 1975 edition
647 offers this definition of the
648 word &dquot;Zebra&dquot;:
651 Zebra, n., any of several horselike, African mammals of the genus Equus,
652 having a characteristic pattern of black or dark-brown stripes on
653 a whitish background.