<!doctype linuxdoc system>
<!--
- $Id: zebra.sgml,v 1.22 1996-03-20 09:20:39 adam Exp $
+ $Id: zebra.sgml,v 1.40 1998-02-10 12:03:05 adam Exp $
-->
<article>
<title>Zebra Server - Administrators's Guide and Reference
-<author><htmlurl url="http://www.indexdata.dk/" name="Index Data">, <tt><htmlurl url="mailto:info@index.ping.dk" name="info@index.ping.dk"></>
-<date>$Revision: 1.22 $
+<author><htmlurl url="http://www.indexdata.dk/" name="Index Data">,
+<tt><htmlurl url="mailto:info@indexdata.dk" name="info@indexdata.dk"></>
+<date>$Revision: 1.40 $
<abstract>
The Zebra information server combines a versatile fielded/free-text
search engine with a Z39.50-1995 frontend to provide a powerful and flexible
functionality that you need.
If you find the software interesting, you should join the support
-mailing-list by sending Email to <tt/zebra-request@index.ping.dk/.
+mailing-list by sending email to <tt/zebra-request@indexdata.dk/.
<sect1>Features
<p>
-This is a listof some of the most important features of the
+This is a list of some of the most important features of the
system.
<itemize>
well as variant forms of data.
<item>
+Supports random storage formats. A system of input filters driven by
+regular expressions allows you to easily process most ASCII-based
+data formats. SGML, ISO2709 (MARC), and raw text are also supported.
+
+<item>
Supports boolean queries as well as relevance-ranking (free-text)
searching. Right truncation and masking in terms are supported, as
well as full regular expressions.
schema on the fly.
<item>
+Supports approximate matching in registers (ie. spelling mistakes,
+etc).
+
+</itemize>
+
+<p>
Protocol support:
<itemize>
<item>
-Protocol facilities: Init, Search, Retrieve, Browse.
+Protocol facilities: Init, Search, Retrieve, Browse and Sort.
<item>
Piggy-backed presents are honored in the search-request.
server can utilise a plug-in XTI/mOSI implementation (not included) to
provide SR services over an OSI stack, as well as Z39.50 over TCP/IP.
-</itemize>
+<item>
+Zebra runs on most Unix-like systems as well as Windows NT - a binary
+distribution for Windows NT is forthcoming - so far, the installation
+requires MSVC++ to compile the system (we use version 5.0).
</itemize>
<sect1>Future Work
<p>
-This is an alfa-release of the software, to allow you to look at
-it - try it out, and assess whether it can be of use to you. We expect
-this version to be followed by a succession of beta-releases until we
-arrive at a stable first version.
+This is a beta-release of the software, to allow you to look at
+it - try it out, and assess whether it can be of use to you.
These are some of the plans that we have for the software in the near
and far future, approximately ordered after their relative importance.
<itemize>
<item>
-*Allow the system to handle other input formats. Specifically
-MARC records and general, structured ASCII records (such as mail/news
-files) parameterized by regular expressions.
-
-<item>
-*Complete the support for variants. Finalize support for the WAIS
-retrieval methodology.
+*Complete the support for variants.
<item>
*Finalize the data element <it/include/ facility to support multimedia
data elements in records.
<item>
-*Port the system to Windows NT.
-
-<item>
-Add index and data compression to save disk space.
-
-<item>
Add more sophisticated relevance ranking mechanisms. Add support for soundex
-and stemming. Add relevance feedback support.
+and stemming. Add relevance <it/feedback/ support.
<item>
-Add Explain support.
+Complete EXPLAIN support.
<item>
Add support for very large records by implementing segmentation and/or
Support the Item Update extended service of the protocol.
<item>
-The Zebra search engine supports approximate string matching in the
-index. We'd like to find a way to support and control this from RPN.
-
-<item>
We want to add a management system that allows you to
control your databases and configuration tables from a graphical
interface. We'll probably use Tcl/Tk to stay platform-independent.
<sect>Compiling the software
<p>
-Zebra uses the YAZ package to implement Z39.50, so you
-have to compile YAZ before going further. Specifically, Zebra uses
-the YAZ header files in <tt>yaz/include/..</tt> and its public library
-<tt>yaz/lib/libyaz.a</tt>.
-
-As with YAZ, an ANSI C compiler is required in order to compile the Zebra
+An ANSI C compiler is required to compile the Zebra
server system — <tt/gcc/ works fine if your own system doesn't
provide an adequate compiler.
-Unpack the Zebra software. You might put Zebra in the same directory level
-as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-xxx</tt>, then
-Zebra is placed in ..<tt>/src/zebra-yyy</tt>.
-
-Edit the top-level <tt>Makefile</tt> in the Zebra directory in which
-you specify the location of YAZ by setting make variables.
-The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with
-MOSI support. Some systems, such as Solaris, have separate socket
-libraries and for those systems you need to specify the
-<tt>NETLIB</tt> variable.
+Unpack the distribution archive. In some cases, you may want to edit
+the top-level <tt/Makefile/, eg. to select a different C compiler, or
+to specify machine-specific libraries in the <bf/ELIBS/ variable.
When you are done editing the <tt>Makefile</tt> type:
<tscreen><verb>
<tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
</descrip>
-<sect>Quick Start
-
+<sect>Quick Start
<p>
In this section, we will test the system by indexing a small set of sample
GILS records that are included with the software distribution. Go to the
-<tt>test</tt> subdirectory of the distribution archive. There you will
+<tt>test/gils</tt> subdirectory of the distribution archive. There you will
find a configuration
file named <tt>zebra.cfg</tt> with the following contents:
<tscreen><verb>
# Where are the YAZ tables located.
-profilePath: ../../yaz/tab ../tab
+profilePath: ../../../yaz/tab ../../tab
# Files that describe the attribute sets supported.
attset: bib1.att
The 48 test records are located in the sub directory <tt>records</tt>.
To index these, type:
<tscreen><verb>
-$ ../index/zebraidx -t grs update records
+$ ../../index/zebraidx -t grs.sgml update records
</verb></tscreen>
In the command above the option <tt>-t</tt> specified the record
-type — in this case <tt>grs</tt>. The word <tt>update</tt> followed
+type — in this case <tt>grs.sgml</tt>. The word <tt>update</tt> followed
by a directory root updates all files below that directory node.
If your indexing command was successful, you are now ready to
fire up a server. To start a server on port 2100, type:
<tscreen><verb>
-$ ../index/zebrasrv tcp:@:2100
+$ ../../index/zebrasrv tcp:@:2100
</verb></tscreen>
The Zebra index that you have just created has a single database
-named <ztt/Default/. The database contains records structured according to
+named <tt/Default/. The database contains records structured according to
the GILS profile, and the server will
return records in either either USMARC, GRS-1, or SUTRS depending
on what your client asks
<sect1>Record Types<label id="record-types">
<p>
-Indexing is a per-record process, in which
-either insert/modify/delete will occur. Before a record is indexed
-search keys are extracted from whatever might be the layout the
-original record (sgml,html,text, etc..). The Zebra system
-currently only supports SGML-like, structured records and unstructured text
-records.
+Indexing is a per-record process, in which either insert/modify/delete
+will occur. Before a record is indexed search keys are extracted from
+whatever might be the layout the original record (sgml,html,text, etc..).
+The Zebra system currently supports two fundamantal types of records:
+structured and simple text.
To specify a particular extraction process, use either the
command line option <tt>-t</tt> or specify a
<tt>recordType</tt> setting in the configuration file.
with no prefix are used.
In the configuration file, the group name is placed before the option
-name
-itself, separated by a dot (.). For instance, to set the record type
-for group <tt/public/ to <tt/grs/ (the common format for structured
+name itself, separated by a dot (.). For instance, to set the record type
+for group <tt/public/ to <tt/grs.sgml/ (the SGML-like format for structured
records) you would write:
<tscreen><verb>
-public.recordType: grs
+public.recordType: grs.sgml
</verb></tscreen>
To set the default value of the record type to <tt/text/ write:
Specifies how records with the file extension <it>name</it> should
be handled by the indexer. This option may also be specified
as a command line option (<tt>-t</tt>). Note that if you do not
- specify a <it/name/, the setting applies to all files.
-<tag><it>group</it>.recordId</tag>
+ specify a <it/name/, the setting applies to all files. In general,
+ the record type specifier consists of the elements (each
+ element separated by dot), <it>fundamental-type</it>,
+ <it>file-read-type</it> and arguments. Currently, two
+ fundamental types exist, <tt>text</tt> and <tt>grs</tt>.
+ <tag><it>group</it>.recordId</tag>
Specifies how the records are to be identified when updated. See
section <ref id="locating-records" name="Locating Records">.
<tag><it>group</it>.database</tag>
Enables the <it/safe update/ facility of Zebra, and tells the system
where to place the required, temporary files. See section
<ref id="shadow-registers" name="Safe Updating - Using Shadow Registers">.
-<tag>tempSetPath</tag>
+<tag>lockDir</tag>
+ Directory in which various lock files are stored.
+<tag>keyTmpDir</tag>
+ Directory in which temporary files used during zebraidx' update
+ phase are stored.
+<tag>setTmpDir</tag>
Specifies the directory that the server uses for temporary result sets.
If not specified <tt>/tmp</tt> will be used.
<tag>profilePath</tag>
searching. At least the Bib-1 set should be loaded (<tt/bib1.att/).
The <tt/profilePath/ setting is used to look for the specified files.
See section <ref id="attset-files" name="The Attribute Set Files">
+<tag>memMax</tag>
+ Specifies size of internal memory to use for the zebraidx program. The
+ amount is given in megabytes - default is 4 (4 MB).
</descrip>
-
-<sect1>Locating Records<label="locating-records">
+<sect1>Locating Records<label id="locating-records">
<p>
The default behaviour of the Zebra system is to reference the
records from their original location, i.e. where they were found when you
<sect1>Indexing with no Record IDs (Simple Indexing)
<p>
-If you have a set of records that is not expected to change over time
+If you have a set of records that are not expected to change over time
you may can build your database without record IDs.
This indexing method uses less space than the other methods and
is simple to use.
-To use this method, you simply don't provide the <tt>recordId</tt> entry
+To use this method, you simply omit the <tt>recordId</tt> entry
for the group of files that you index. To add a set of records you use
<tt>zebraidx</tt> with the <tt>update</tt> command. The
<tt>update</tt> command will always add all of the records that it
The corresponding configuration file includes:
<tscreen><verb>
esdd.recordId: file
-esdd.recordType: grs
+esdd.recordType: grs.sgml
esdd.storeKeys: 1
</verb></tscreen>
<bf/Options/
<descrip>
<tag>-t <it/type/</tag>Update all files as <it/type/. Currently, the
-types supported are <tt/text/ and <tt/grs/<it/.filter/. If no
-<it/filter/ is provided for the GRS (General Record Structure) type,
+types supported are <tt/text/ and <tt/grs/<it/.subtype/. If no
+<it/subtype/ is provided for the GRS (General Record Structure) type,
the canonical input format is assumed (see section <ref
id="local-representation" name="Local Representation">). Generally, it
is probably advisable to specify the record types in the
-<tt/zebra.cfg/ file (see section <ref id="record-types" name="Record Types">).
+<tt/zebra.cfg/ file (see section <ref id="record-types" name="Record
+Types">), to avoid confusion at subsequent updates.
<tag>-c <it/config-file/</tag>Read the configuration file
<it/config-file/ instead of <tt/zebra.cfg/.
with the database name <it/database/ for access through the Z39.50
server.
-<tag>-d <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing
+<tag>-m <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing
keys to background storage. This setting affects performance when
updating large databases.
(see section <ref id="shadow-registers" name="Robust Updating - Using
Shadow Registers">).
+<tag>-s</tag>Show analysis of the indexing process. The maintenance
+program works in a read-only mode and doesn't change the state
+of the index. This options is very useful when you wish to test a
+new profile.
+
+<tag>-V</tag>Show Zebra version.
+
<tag>-v <it/level/</tag>Set the log level to <it/level/. <it/level/
should be one of <tt/none/, <tt/debug/, and <tt/all/.
</descrip>
-<sect>Running the Z39.50 Server (zebrasrv)
+<sect>The Z39.50 Server
+
+<sect1>Running the Z39.50 Server (zebrasrv)
<p>
<bf/Syntax/
<tag>-w <it/working-directory/</tag>Change working directory.
-<tag/-i/Run under the Internet superserver, <tt/inetd/.
+<tag>-i</tag>Run under the Internet superserver, <tt/inetd/. Make
+sure you use the logfile option <tt/-l/ in conjunction with this
+mode and specify the <tt/-l/ option before any other options.
+
+<tag>-t <it/timeout/</tag>Set the idle session timeout (default 60 minutes).
+
+<tag>-k <it/kilobytes/</tag>Set the (approximate) maximum size of
+present response messages. Default is 1024 Kb (1 Mb).
</descrip>
A <it/listener-address/ consists of a transport mode followed by a
The default behavior for <tt/zebrasrv/ is to establish a single TCP/IP
listener, for the Z39.50 protocol, on port 9999.
+<sect1>Z39.50 Protocol Support and Behavior
+
+<sect2>Initialization
+
+<p>
+During initialization, the server will negotiate to version 3 of the
+Z39.50 protocol, and the option bits for Search, Present, Scan,
+NamedResultSets, and concurrentOperations will be set, if requested by
+the client. The maximum PDU size is negotiated down to a maximum of
+1Mb by default.
+
+<sect2>Search<label id="search">
+
+<p>
+The supported query type are 1 and 101. All operators are currently
+supported with the restriction that only proximity units of type "word" are
+supported for the proximity operator.
+Queries can be arbitrarily complex.
+Named result sets are supported, and result sets can be used as operands
+without limitations.
+Searches may span multiple databases.
+
+The server has full support for piggy-backed present requests (see
+also the following section).
+
+<bf/Use/ attributes are interpreted according to the attribute sets which
+have been loaded in the <tt/zebra.cfg/ file, and are matched against
+specific fields as specified in the <tt/.abs/ file which describes the
+profile of the records which have been loaded. If no <bf/Use/
+attribute is provided, a default of Bib-1 <bf/Any/ is assumed.
+
+If a <bf/Structure/ attribute of <bf/Phrase/ is used in conjunction with a
+<bf/Completeness/ attribute of <bf/Complete (Sub)field/, the term is
+matched against the contents of the phrase (long word) register, if one
+exists for the given <bf/Use/ attribute.
+A phrase register is created for those fields in the <tt/.abs/
+file that contains a <tt/p/-specifier.
+
+If <bf/Structure/=<bf/Phrase/ is used in conjunction with
+<bf/Incomplete Field/ - the default value for <bf/Completeness/, the
+search is directed against the normal word registers, but if the term
+contains multiple words, the term will only match if all of the words
+are found immediately adjacent, and in the given order.
+The word search is performed on those fields that are indexed as
+type <tt/w/ in the <tt/.abs/ file.
+
+If the <bf/Structure/ attribute is <bf/Word List/,
+<bf/Free-form Text/, or <bf/Document Text/, the term is treated as a
+natural-language, relevance-ranked query.
+This search type uses the word register, i.e. those fields
+that are indexed as type <tt/w/ in the <tt/.abs/ file.
+
+If the <bf/Structure/ attribute is <bf/Numeric String/ the
+term is treated as an integer. The search is performed on those
+fields that are indexed as type <tt/n/ in the <tt/.abs/ file.
+
+If the <bf/Structure/ attribute is <bf/URx/ the
+term is treated as a URX (URL) entity. The search is performed on those
+fields that are indexed as type <tt/u/ in the <tt/.abs/ file.
+
+If the <bf/Structure/ attribute is <bf/Local Number/ the
+term is treated as native Zebra Record Identifier.
+
+If the <bf/Relation/ attribute is <bf/Equals/ (default), the term is
+matched in a normal fashion (modulo truncation and processing of
+individual words, if required). If <bf/Relation/ is <bf/Less Than/,
+<bf/Less Than or Equal/, <bf/Greater than/, or <bf/Greater than or
+Equal/, the term is assumed to be numerical, and a standard regular
+expression is constructed to match the given expression. If
+<bf/Relation/ is <bf/Relevance/, the standard natural-language query
+processor is invoked.
+
+For the <bf/Truncation/ attribute, <bf/No Truncation/ is the default.
+<bf/Left Truncation/ is not supported. <bf/Process #/ is supported, as
+is <bf/Regxp-1/. <bf/Regxp-2/ enables the fault-tolerant (fuzzy)
+search. As a default, a single error (deletion, insertion,
+replacement) is accepted when terms are matched against the register
+contents.
+
+<sect3>Regular expressions
+<p>
+
+Each term in a query is interpreted as a regular expression if
+the truncation value is either <bf/Regxp-1/ (102) or <bf/Regxp-2/ (103).
+Both query types follow the same syntax with the operands:
+<descrip>
+<tag/x/ Matches the character <it/x/.
+<tag/./ Matches any character.
+<tag><tt/[/..<tt/]/</tag> Matches the set of characters specified;
+ such as <tt/[abc]/ or <tt/[a-c]/.
+</descrip>
+and the operators:
+<descrip>
+<tag/x*/ Matches <it/x/ zero or more times. Priority: high.
+<tag/x+/ Matches <it/x/ one or more times. Priority: high.
+<tag/x?/ Matches <it/x/ once or twice. Priority: high.
+<tag/xy/ Matches <it/x/, then <it/y/. Priority: medium.
+<tag/x|y/ Matches either <it/x/ or <it/y/. Priority: low.
+</descrip>
+The order of evaluation may be changed by using parentheses.
+
+If the first character of the <bf/Regxp-2/ query is a plus character
+(<tt/+/) it marks the beginning of a section with non-standard
+specifiers. The next plus character marks the end of the section.
+Currently Zebra only supports one specifier, the error tolerance,
+which consists one digit.
+
+Since the plus operator is normally a suffix operator the addition to
+the query syntax doesn't violate the syntax for standard regular
+expressions.
+
+<sect3>Query examples
+<p>
+
+Phrase search for <bf/information retrieval/ in the title-register:
+<verb>
+ @attr 1=4 "information retrieval"
+</verb>
+
+Ranked search for the same thing:
+<verb>
+ @attr 1=4 @attr 2=102 "Information retrieval"
+</verb>
+
+Phrase search with a regular expression:
+<verb>
+ @attr 1=4 @attr 5=102 "informat.* retrieval"
+</verb>
+
+Ranked search with a regular expression:
+<verb>
+ @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
+</verb>
+
+In the GILS schema (<tt/gils.abs/), the west-bounding-coordinate is
+indexed as type <tt/n/, and is therefore searched by specifying
+<bf/structure/=<bf/Numeric String/.
+To match all those records with west-bounding-coordinate greater
+than -114 we use the following query:
+<verb>
+ @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
+</verb>
+
+<sect2>Present
+<p>
+The present facility is supported in a standard fashion. The requested
+record syntax is matched against the ones supported by the profile of
+each record retrieved. If no record syntax is given, SUTRS is the
+default. The requested element set name, again, is matched against any
+provided by the relevant record profiles.
+
+<sect2>Scan
+
+<p>
+The attribute combinations provided with the TermListAndStartPoint are
+processed in the same way as operands in a query (see above).
+Currently, only the term and the globalOccurrences are returned with
+the TermInfo structure.
+
+<sect2>Sort
+
+<p>
+Z39.50 specifies three diffent types of sort criterias.
+Of these Zebra supports the attribute specification type in which
+case the use attribute specifies the "Sort register".
+Sort registers are created for those fields that are of type "sort" in
+the default.idx file.
+The corresponding character mapping file in default.idx specifies the
+ordinal of each character used in the actual sort.
+
+Z39.50 allows the client to specify sorting on one or more input
+result sets and one output result set.
+Zebra supports sorting on one result set only which may or may not
+be the same as the output result set.
+
+<sect2>Close
+
+<p>
+If a Close PDU is received, the server will respond with a Close PDU
+with reason=FINISHED, no matter which protocol version was negotiated
+during initialization. If the protocol version is 3 or more, the
+server will generate a Close PDU under certain circumstances,
+including a session timeout (60 minutes by default), and certain kinds of
+protocol errors. Once a Close PDU has been sent, the protocol
+association is considered broken, and the transport connection will be
+closed immediately upon receipt of further data, or following a short
+timeout.
+
<sect>The Record Model
<p>
Although it may be wise to use only a single schema within
one database, the system poses no such restrictions.
+The record model described in this chapter applies to the fundamental,
+structured
+record type <tt>grs</tt> as introduced in
+section <ref id="record-types" name="Record Types">.
+
Records pass through three different states during processing in the
system.
News or Mail archives, MARC records. If the system doesn't already
know how to read the type of data you need to store, you can set up an
input filter by preparing conversion rules based on regular
-expressions and a flexible scripting language (Tcl). The input filter
+expressions and possibly augmented by a flexible scripting language (Tcl). The input filter
produces as output an internal representation:
<item>When records are processed by the system, they are represented
process takes place whenever the record is accessed - for indexing and
retrieval.
+<p>
+The RecordType parameter in the <tt/zebra.cfg/ file, or the <tt/-t/
+option to the indexer tells Zebra how to process input records. Two
+basic types of processing are available - raw text and structured
+data. Raw text is just that, and it is selected by providing the
+argument <bf/text/ to Zebra. Structured records are all handled
+internally using the basic mechanisms described in the subsequent
+sections. Zebra can read structured records in many different formats.
+How this is done is governed by additional parameters after the
+&dquot;grs&dquot; keyboard, separated by &dquot;.&dquot; characters.
+
+Three basic subtypes to the <bf/grs/ type are currently available:
+
+<descrip>
+<tag>grs.sgml</tag>This is the canonical input format —
+described below. It is a simple SGML-like syntax.
+
+<tag>grs.regx.<it/filter/</tag>This enables a user-supplied input
+filter. The mechanisms of these filters are described below.
+
+<tag>grs.marc.<it/abstract syntax/</tag>This allows Zebra to read
+records in the ISO2709 (MARC) encoding standard. In this case, the
+last paramemeter <it/abstract syntax/ names the .abs file (see below)
+which describes the specific MARC structure of the input record as
+well as the indexing rules.
+</descrip>
+
<sect2>Canonical Input Format
<p>
spectrum of structure and flexibility in the system. In Zebra, this
canonical format is an &dquot;SGML-like&dquot; syntax.
+To use the canonical format specify <tt>grs.sgml</tt> as the record
+type,
+
Consider a record describing an information resource (such a record is
sometimes known as a <it/locator record/). It might contain a field
describing the distributor of the information resource, which might in
sections of text in between are the <it/data elements/. A data element
is characterized by its location in the tree that is made up by the
nested elements. Each element is terminated by a closing tag -
-beginning with &etago;, and containing the same symbolic tag-name as
-the corresponding opening tag. The general closing tag - &etago;> -
+beginning with <tt/&etago;/, and containing the same symbolic tag-name as
+the corresponding opening tag. The general closing tag - <tt/&etago;>/ -
terminates the element started by the last opening tag. The
structuring of elements is significant. The element <bf/Telephone/,
for instance, may be indexed and presented to the client differently,
Input filters are ASCII files, generally with the suffix <tt/.flt/.
The system looks for the files in the directories given in the
-<bf/profilePath/ setting in the <tt/zebra.cfg/ file.
+<bf/profilePath/ setting in the <tt/zebra.cfg/ files. The record type
+for the filter is <tt>grs.regx.</tt><it>filter-filename</it>
+(fundamental type <tt>grs</tt>, file read type <tt>regx</tt>, argument
+<it>filter-filename</it>).
Generally, an input filter consists of a sequence of rules, where each
rule consists of a sequence of expressions, followed by an action. The
given in place of the filename, this corresponds to a null mapping for
the given element set name.
-<tag>elm <it/path name attribute/</tag> (o,r) Adds an element
+<tag>any <it/tags/</tag> (o) This directive specifies a list of
+attributes which should be appended to the attribute list given for each
+element. The effect is to make every single element in the abstract
+syntax searchable by way of the given attributes. This directive
+provides an efficient way of supporting free-text searching across all
+elements. However, it does increase the size of the index
+significantly. The attributes can be qualified with a structure, as in
+the <bf/elm/ directive below.
+
+<tag>elm <it/path name attributes/</tag> (o,r) Adds an element
to the abstract record syntax of the schema. The <it/path/ follows the
syntax which is suggested by the Z39.50 document - that is, a sequence
of tags separated by slashes (/). Each tag is given as a
comma-separated pair of tag type and -value surrounded by parenthesis.
-The <it/name/ is the name of the element, and the <it/attribute/
-specifies what attribute to use when indexing the element. A ! in
+The <it/name/ is the name of the element, and the <it/attributes/
+specifies which attributes to use when indexing the element in a
+comma-separated list. A ! in
place of the attribute name is equivalent to specifying an attribute
name identical to the element name. A - in place of the attribute name
-specifies that no indexing is to take place for the given element.
+specifies that no indexing is to take place for the given element. The
+attributes can be qualified with <it/field types/ to specify which
+character set should govern the indexing procedure for that field. The
+same data element may be indexed into several different fields, using
+different character set definitions. See the section
+<ref id="field structure and character sets"
+name="Field Structure and Character Sets">.
+The default field type is &dquot;w&dquot; for
+<it/word/.
</descrip>
<it>
elm (1,12) url -
elm (1,14) localControlNumber Local-number
elm (1,16) dateOfLastModification Date/time-last-modified
-elm (2,1) Title !
+elm (2,1) Title w:!,p:!
elm (4,1) controlIdentifier Identifier-standard
elm (2,6) abstract Abstract
elm (4,51) purpose !
module of <bf/YAZ/. The directive is optional, since not all tag sets
are registered outside of their schema.
-<tag>type <it/integer/</tag> (m) The type number of the tag within the schema
-profile.
+<tag>type <it/integer/</tag> (m) The type number of the tagset within the schema
+profile (note: this specification really should belong to the .abs
+file. This will be fixed in a future release).
<tag>include <it/filename/</tag> (o,r) This directive is used
to include the definitions of other tag sets into the current one.
re-evaluating and most likely changing the way that MARC records are
handled by the system.</it>
+<sect2>Field Structure and Character Sets
+<label id="field structure and character sets">
+
+<p>
+In order to provide a flexible approach to national character set
+handling, Zebra allows the administrator to configure the set up the
+system to handle any 8-bit character set — including sets that
+require multi-octet diacritics or other multi-octet characters. The
+definition of a character set includes a specification of the
+permissible values, their sort order (this affects the display in the
+SCAN function), and relationships between upper- and lowercase
+characters. Finally, the definition includes the specification of
+space characters for the set.
+
+The operator can define different character sets for different fields,
+typical examples being standard text fields, numerical fields, and
+special-purpose fields such as WWW-style linkages (URx).
+
+The field types, and hence character sets, are associated with data
+elements by the .abs files (see above). The file <tt/default.idx/
+provides the association between field type codes (as used in the .abs
+files) and the character map files (with the .chr suffix). The format
+of the .idx file is as follows
+
+<descrip>
+<tag>index <it/field type code/</tag>This directive introduces a new
+search index code. The argument is a one-character code to be used in the
+.abs files to select this particular index type. An index, roughly,
+corresponds to a particular structure attribute during search. Refer
+to section <ref id="search" name="Search">.
+
+<tag>sort <it/field code type/</tag>This directive introduces a
+sort index. The argument is a one-character code to be used in the
+.abs fie to select this particular index type. The corresponding
+use attribute must be used in the sort request to refer to this
+particular sort index. The corresponding character map (see below)
+is used in the sort process.
+
+<tag>completeness <it/boolean/</tag>This directive enables or disables
+complete field indexing. The value of the <it/boolean/ should be 0
+(disable) or 1. If completeness is enabled, the index entry will
+contain the complete contents of the field (up to a limit), with words
+(non-space characters) separated by single space characters
+(normalized to &dquot; &dquot; on display). When completeness is
+disabled, each word is indexed as a separate entry. Complete subfield
+indexing is most useful for fields which are typically browsed (eg.
+titles, authors, or subjects), or instances where a match on a
+complete subfield is essential (eg. exact title searching). For fields
+where completeness is disabled, the search engine will interpret a
+search containing space characters as a word proximity search.
+
+<tag>charmap <it/filename/</tag> This is the filename of the character
+map to be used for this index for field type.
+</descrip>
+
+The contents of the character map files are structured as follows:
+
+<descrip>
+<tag>lowercase <it/value-set/</tag>This directive introduces the basic
+value set of the field type. The format is an ordered list (without
+spaces) of the characters which may occur in &dquot;words&dquot; of
+the given type. The order of the entries in the list determines the
+sort order of the index. In addition to single characters, the
+following combinations are legal:
+
+<itemize>
+<item>Backslashes may be used to introduce three-digit octal, or
+two-digit hex representations of single characters (preceded by <tt/x/).
+In addition, the combinations
+\\, \\r, \\n, \\t, \\s (space — remember that real space-characters
+may ot occur in the value definition), and \\ are recognised,
+with their usual interpretation.
+
+<item>Curly braces {} may be used to enclose ranges of single
+characters (possibly using the escape convention described in the
+preceding point), eg. {a-z} to entroduce the standard range of ASCII
+characters. Note that the interpretation of such a range depends on
+the concrete representation in your local, physical character set.
+
+<item>Paranthesises () may be used to enclose multi-byte characters -
+eg. diacritics or special national combinations (eg. Spanish
+&dquot;ll&dquot;). When found in the input stream (or a search term),
+these characters are viewed and sorted as a single character, with a
+sorting value depending on the position of the group in the value
+statement.
+</itemize>
+
+<tag>uppercase <it/value-set/</tag>This directive introduces the
+upper-case equivalencis to the value set (if any). The number and
+order of the entries in the list should be the same as in the
+<tt/lowercase/ directive.
+
+<tag>space <it/value-set/</tag>This directive introduces the character
+which separate words in the input stream. Depending on the
+completeness mode of the field in question, these characters either
+terminate an index entry, or delimit individual &dquot;words&dquot; in
+the input stream. The order of the elements is not significant —
+otherwise the representation is the same as for the <tt/upercase/ and
+<tt/lowercase/ directives.
+
+<tag>map <it/value-set/ <it/target/</tag>This directive introduces a
+mapping between each of the members of the value-set on the left to
+the character on the right. The character on the right must occur in
+the value set (the <tt/lowercase/ directive) of the character set, but
+it may be a paranthesis-enclosed multi-octet character. This directive
+may be used to map diacritics to their base characters, or to map
+HTML-style character-representations to their natural form, etc.
+</descrip>
+
<sect1>Exchange Formats
<p>
contains variant information.
<item>SUTRS. Again, the mapping is fairly straighforward. Indentation
-is used to show the hierarchical structure of the record.
+is used to show the hierarchical structure of the record. All
+&dquot;GRS&dquot; type records support both the GRS-1 and SUTRS
+representations.
<item>ISO2709-based formats (USMARC, etc.). Only records with a
two-level structure (corresponding to fields and subfields) can be
<item>Explain. This representation is only available for records
belonging to the Explain schema.
+<item>Summary. This ASN-1 based structure is only available for records
+belonging to the Summary schema - or schema which provide a mapping
+to this schema (see the description of the schema mapping facility
+above).
+
+<item>SOIF. Support for this syntax is experimental, and is currently
+keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
+abstract syntaxes can be mapped to the SOIF format, although nested
+elements are represented by concatenation of the tag names at each
+level.
+
</itemize>
<sect>License
<p>
-Copyright © 1995, Index Data.
+Copyright © 1995-1998 Index Data.
All rights reserved.
purpose, our consulting services and expert knowledge of the software
is available to you at favorable rates.
-<tscreen>
-Index Data&nl
-Ryesgade 3&nl
-DK-2200 København N&nl
-</tscreen>
+<tscreen><verb>
+Index Data
+Ryesgade 3
+DK-2200 Copenhagen N
+</verb></tscreen>
<p>
<tscreen><verb>
Phone: +45 3536 3672
Fax : +45 3536 0449
-Email: info@index.ping.dk
+Email: info@indexdata.dk
</verb></tscreen>
The <it>Random House College Dictionary</it>, 1975 edition
projects / idzebra-moved-to-github.git / blobdiff