1 <chapter id="record-model">
2 <!-- $Id: recordmodel.xml,v 1.21 2005-01-04 19:10:29 quinn Exp $ -->
3 <title>The Record Model</title>
6 The Zebra system is designed to support a wide range of data management
7 applications. The system can be configured to handle virtually any
8 kind of structured data. Each record in the system is associated with
9 a <emphasis>record schema</emphasis> which lends context to the data
10 elements of the record.
11 Any number of record schemas can coexist in the system.
12 Although it may be wise to use only a single schema within
13 one database, the system poses no such restrictions.
17 The record model described in this chapter applies to the fundamental,
19 record type <literal>grs</literal>, introduced in
20 <xref linkend="record-types"/>.
22 FIXME - Need to describe the simple string-tag model, or at least
28 Records pass through three different states during processing in the
38 When records are accessed by the system, they are represented
39 in their local, or native format. This might be SGML or HTML files,
40 News or Mail archives, MARC records. If the system doesn't already
41 know how to read the type of data you need to store, you can set up an
42 input filter by preparing conversion rules based on regular
43 expressions and possibly augmented by a flexible scripting language
45 The input filter produces as output an internal representation,
53 When records are processed by the system, they are represented
54 in a tree-structure, constructed by tagged data elements hanging off a
55 root node. The tagged elements may contain data or yet more tagged
56 elements in a recursive structure. The system performs various
57 actions on this tree structure (indexing, element selection, schema
65 Before transmitting records to the client, they are first
66 converted from the internal structure to a form suitable for exchange
67 over the network - according to the Z39.50 standard.
75 <sect1 id="local-representation">
76 <title>Local Representation</title>
79 As mentioned earlier, Zebra places few restrictions on the type of
80 data that you can index and manage. Generally, whatever the form of
81 the data, it is parsed by an input filter specific to that format, and
82 turned into an internal structure that Zebra knows how to handle. This
83 process takes place whenever the record is accessed - for indexing and
88 The RecordType parameter in the <literal>zebra.cfg</literal> file, or
89 the <literal>-t</literal> option to the indexer tells Zebra how to
90 process input records.
91 Two basic types of processing are available - raw text and structured
92 data. Raw text is just that, and it is selected by providing the
93 argument <emphasis>text</emphasis> to Zebra. Structured records are
94 all handled internally using the basic mechanisms described in the
96 Zebra can read structured records in many different formats.
97 How this is done is governed by additional parameters after the
98 "grs" keyword, separated by "." characters.
102 Four basic subtypes to the <emphasis>grs</emphasis> type are
109 <term>grs.sgml</term>
112 This is the canonical input format —
113 described below. It is a simple SGML-like syntax.
118 <term>grs.regx.<emphasis>filter</emphasis></term>
121 This enables a user-supplied input
122 filter. The mechanisms of these filters are described below.
127 <term>grs.tcl.<emphasis>filter</emphasis></term>
130 Similar to grs.regx but using Tcl for rules.
135 <term>grs.marc.<emphasis>abstract syntax</emphasis></term>
138 This allows Zebra to read
139 records in the ISO2709 (MARC) encoding standard. In this case, the
140 last parameter <emphasis>abstract syntax</emphasis> names the
141 <literal>.abs</literal> file (see below)
142 which describes the specific MARC structure of the input record as
143 well as the indexing rules.
151 This filter reads XML records. Only one record per file
152 is supported. The filter is only available if Zebra/YAZ
153 is compiled with EXPAT support.
162 <title>Canonical Input Format</title>
165 Although input data can take any form, it is sometimes useful to
166 describe the record processing capabilities of the system in terms of
167 a single, canonical input format that gives access to the full
168 spectrum of structure and flexibility in the system. In Zebra, this
169 canonical format is an "SGML-like" syntax.
173 To use the canonical format specify <literal>grs.sgml</literal> as
178 Consider a record describing an information resource (such a record is
179 sometimes known as a <emphasis>locator record</emphasis>).
180 It might contain a field describing the distributor of the
181 information resource, which might in turn be partitioned into
182 various fields providing details about the distributor, like this:
188 <Distributor>
189 <Name> USGS/WRD </Name>
190 <Organization> USGS/WRD </Organization>
191 <Street-Address>
192 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
193 </Street-Address>
194 <City> ALBUQUERQUE </City>
195 <State> NM </State>
196 <Zip-Code> 87102 </Zip-Code>
197 <Country> USA </Country>
198 <Telephone> (505) 766-5560 </Telephone>
199 </Distributor>
204 <!-- There is no indentation in the example above! -H
207 The indentation used above is used to illustrate how Zebra
208 interprets the mark-up. The indentation, in itself, has no
209 significance to the parser for the canonical input format, which
210 discards superfluous whitespace.
216 The keywords surrounded by <...> are
217 <emphasis>tags</emphasis>, while the sections of text
218 in between are the <emphasis>data elements</emphasis>.
219 A data element is characterized by its location in the tree
220 that is made up by the nested elements.
221 Each element is terminated by a closing tag - beginning
222 with <literal><</literal>/, and containing the same symbolic
223 tag-name as the corresponding opening tag.
224 The general closing tag - <literal></></literal> -
225 terminates the element started by the last opening tag. The
226 structuring of elements is significant.
227 The element <emphasis>Telephone</emphasis>,
228 for instance, may be indexed and presented to the client differently,
229 depending on whether it appears inside the
230 <emphasis>Distributor</emphasis> element, or some other,
231 structured data element such a <emphasis>Supplier</emphasis> element.
235 <title>Record Root</title>
238 The first tag in a record describes the root node of the tree that
239 makes up the total record. In the canonical input format, the root tag
240 should contain the name of the schema that lends context to the
241 elements of the record
242 (see <xref linkend="internal-representation"/>).
243 The following is a GILS record that
244 contains only a single element (strictly speaking, that makes it an
245 illegal GILS record, since the GILS profile includes several mandatory
246 elements - Zebra does not validate the contents of a record against
247 the Z39.50 profile, however - it merely attempts to match up elements
248 of a local representation with the given schema):
255 <title>Zen and the Art of Motorcycle Maintenance</title>
263 <sect3><!-- ### we shouldn't make such a big deal about this -->
264 <title>Variants</title>
267 Zebra allows you to provide individual data elements in a number of
268 <emphasis>variant forms</emphasis>. Examples of variant forms are
269 textual data elements which might appear in different languages, and
270 images which may appear in different formats or layouts.
271 The variant system in Zebra is essentially a representation of
272 the variant mechanism of Z39.50-1995.
276 The following is an example of a title element which occurs in two
284 <var lang lang "eng">
285 Zen and the Art of Motorcycle Maintenance</>
286 <var lang lang "dan">
287 Zen og Kunsten at Vedligeholde en Motorcykel</>
294 The syntax of the <emphasis>variant element</emphasis> is
295 <literal><var class type value></literal>.
296 The available values for the <emphasis>class</emphasis> and
297 <emphasis>type</emphasis> fields are given by the variant set
298 that is associated with the current schema
299 (see <xref linkend="variant-set"/>).
303 Variant elements are terminated by the general end-tag </>, by
304 the variant end-tag </var>, by the appearance of another variant
305 tag with the same <emphasis>class</emphasis> and
306 <emphasis>value</emphasis> settings, or by the
307 appearance of another, normal tag. In other words, the end-tags for
308 the variants used in the example above could have been omitted.
312 Variant elements can be nested. The element
319 <var lang lang "eng"><var body iana "text/plain">
320 Zen and the Art of Motorcycle Maintenance
327 Associates two variant components to the variant list for the title
332 Given the nesting rules described above, we could write
339 <var body iana "text/plain>
340 <var lang lang "eng">
341 Zen and the Art of Motorcycle Maintenance
342 <var lang lang "dan">
343 Zen og Kunsten at Vedligeholde en Motorcykel
350 The title element above comes in two variants. Both have the IANA body
351 type "text/plain", but one is in English, and the other in
352 Danish. The client, using the element selection mechanism of Z39.50,
353 can retrieve information about the available variant forms of data
354 elements, or it can select specific variants based on the requirements
363 <title>Input Filters</title>
366 In order to handle general input formats, Zebra allows the
367 operator to define filters which read individual records in their
368 native format and produce an internal representation that the system
373 Input filters are ASCII files, generally with the suffix
374 <literal>.flt</literal>.
375 The system looks for the files in the directories given in the
376 <emphasis>profilePath</emphasis> setting in the
377 <literal>zebra.cfg</literal> files.
378 The record type for the filter is
379 <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
380 (fundamental type <literal>grs</literal>, file read
381 type <literal>regx</literal>, argument
382 <emphasis>filter-filename</emphasis>).
386 Generally, an input filter consists of a sequence of rules, where each
387 rule consists of a sequence of expressions, followed by an action. The
388 expressions are evaluated against the contents of the input record,
389 and the actions normally contribute to the generation of an internal
390 representation of the record.
394 An expression can be either of the following:
404 The action associated with this expression is evaluated
405 exactly once in the lifetime of the application, before any records
406 are read. It can be used in conjunction with an action that
407 initializes tables or other resources that are used in the processing
416 Matches the beginning of the record. It can be used to
417 initialize variables, etc. Typically, the
418 <emphasis>BEGIN</emphasis> rule is also used
419 to establish the root node of the record.
427 Matches the end of the record - when all of the contents
428 of the record has been processed.
433 <term>/pattern/</term>
436 Matches a string of characters from the input record.
444 This keyword may only be used between two patterns.
445 It matches everything between (not including) those patterns.
453 The expression associated with this pattern is evaluated
454 once, before the application terminates. It can be used to release
455 system resources - typically ones allocated in the
456 <emphasis>INIT</emphasis> step.
464 An action is surrounded by curly braces ({...}), and
465 consists of a sequence of statements. Statements may be separated
466 by newlines or semicolons (;).
467 Within actions, the strings that matched the expressions
468 immediately preceding the action can be referred to as
473 The available statements are:
480 <term>begin <replaceable>type [parameter ... ]</replaceable></term>
484 data element. The <replaceable>type</replaceable> is one of
492 Begin a new record. The following parameter should be the
493 name of the schema that describes the structure of the record, eg.
494 <literal>gils</literal> or <literal>wais</literal> (see below).
495 The <literal>begin record</literal> call should precede
496 any other use of the <replaceable>begin</replaceable> statement.
504 Begin a new tagged element. The parameter is the
505 name of the tag. If the tag is not matched anywhere in the tagsets
506 referenced by the current schema, it is treated as a local string
515 Begin a new node in a variant tree. The parameters are
516 <replaceable>class type value</replaceable>.
525 <term>data <replaceable>parameter</replaceable></term>
528 Create a data element. The concatenated arguments make
529 up the value of the data element.
530 The option <literal>-text</literal> signals that
531 the layout (whitespace) of the data should be retained for
533 The option <literal>-element</literal>
534 <replaceable>tag</replaceable> wraps the data up in
535 the <replaceable>tag</replaceable>.
536 The use of the <literal>-element</literal> option is equivalent to
537 preceding the command with a <replaceable>begin
538 element</replaceable> command, and following
539 it with the <replaceable>end</replaceable> command.
544 <term>end <replaceable>[type]</replaceable></term>
547 Close a tagged element. If no parameter is given,
548 the last element on the stack is terminated.
549 The first parameter, if any, is a type name, similar
550 to the <replaceable>begin</replaceable> statement.
551 For the <replaceable>element</replaceable> type, a tag
552 name can be provided to terminate a specific tag.
558 <term>unread <replaceable>no</replaceable></term>
561 Move the input pointer to the offset of first character that
562 match rule given by <replaceable>no</replaceable>.
563 The first rule from left-to-right is numbered zero,
564 the second rule is named 1 and so on.
573 The following input filter reads a Usenet news file, producing a
574 record in the WAIS schema. Note that the body of a news posting is
575 separated from the list of headers by a blank line (or rather a
576 sequence of two newline characters.
582 BEGIN { begin record wais }
584 /^From:/ BODY /$/ { data -element name $1 }
585 /^Subject:/ BODY /$/ { data -element title $1 }
586 /^Date:/ BODY /$/ { data -element lastModified $1 }
588 begin element bodyOfDisplay
589 begin variant body iana "text/plain"
598 If Zebra is compiled with support for Tcl enabled, the statements
599 described above are supplemented with a complete
600 scripting environment, including control structures (conditional
601 expressions and loop constructs), and powerful string manipulation
602 mechanisms for modifying the elements of a record.
609 <sect1 id="internal-representation">
610 <title>Internal Representation</title>
613 When records are manipulated by the system, they're represented in a
614 tree-structure, with data elements at the leaf nodes, and tags or
615 variant components at the non-leaf nodes. The root-node identifies the
616 schema that lends context to the tagging and structuring of the
617 record. Imagine a simple record, consisting of a 'title' element and
625 TITLE "Zen and the Art of Motorcycle Maintenance"
626 AUTHOR "Robert Pirsig"
632 A slightly more complex record would have the author element consist
633 of two elements, a surname and a first name:
640 TITLE "Zen and the Art of Motorcycle Maintenance"
649 The root of the record will refer to the record schema that describes
650 the structuring of this particular record. The schema defines the
651 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
652 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
653 addition, the schema establishes element set names that are used by
654 the client to request a subset of the elements of a given record. The
655 schema may also establish rules for converting the record to a
656 different schema, by stating, for each element, a mapping to a
661 <title>Tagged Elements</title>
664 A data element is characterized by its tag, and its position in the
665 structure of the record. For instance, while the tag "telephone
666 number" may be used different places in a record, we may need to
667 distinguish between these occurrences, both for searching and
668 presentation purposes. For instance, while the phone numbers for the
669 "customer" and the "service provider" are both
670 representatives for the same type of resource (a telephone number), it
671 is essential that they be kept separate. The record schema provides
672 the structure of the record, and names each data element (defined by
673 the sequence of tags - the tag path - by which the element can be
674 reached from the root of the record).
680 <title>Variants</title>
683 The children of a tag node may be either more tag nodes, a data node
684 (possibly accompanied by tag nodes),
685 or a tree of variant nodes. The children of variant nodes are either
686 more variant nodes or a data node (possibly accompanied by more
687 variant nodes). Each leaf node, which is normally a
688 data node, corresponds to a <emphasis>variant form</emphasis> of the
689 tagged element identified by the tag which parents the variant tree.
690 The following title element occurs in two different languages:
696 VARIANT LANG=ENG "War and Peace"
698 VARIANT LANG=DAN "Krig og Fred"
704 Which of the two elements are transmitted to the client by the server
705 depends on the specifications provided by the client, if any.
709 In practice, each variant node is associated with a triple of class,
710 type, value, corresponding to the variant mechanism of Z39.50.
716 <title>Data Elements</title>
719 Data nodes have no children (they are always leaf nodes in the record
724 FIXME! Documentation needs extension here about types of nodes - numerical,
725 textual, etc., plus the various types of inclusion notes.
733 <sect1 id="data-model">
734 <title>Configuring Your Data Model</title>
737 The following sections describe the configuration files that govern
738 the internal management of data records. The system searches for the files
739 in the directories specified by the <emphasis>profilePath</emphasis>
740 setting in the <literal>zebra.cfg</literal> file.
744 <title>The Abstract Syntax</title>
747 The abstract syntax definition (also known as an Abstract Record
748 Structure, or ARS) is the focal point of the
749 record schema description. For a given schema, the ABS file may state any
750 or all of the following:
754 FIXME - Need a diagram here, or a simple explanation how it all hangs together -H
763 The object identifier of the Z39.50 schema associated
764 with the ARS, so that it can be referred to by the client.
770 The attribute set (which can possibly be a compound of multiple
771 sets) which applies in the profile. This is used when indexing and
772 searching the records belonging to the given profile.
778 The tag set (again, this can consist of several different sets).
779 This is used when reading the records from a file, to recognize the
780 different tags, and when transmitting the record to the client -
781 mapping the tags to their numerical representation, if they are
788 The variant set which is used in the profile. This provides a
789 vocabulary for specifying the <emphasis>forms</emphasis> of
790 data that appear inside the records.
796 Element set names, which are a shorthand way for the client to
797 ask for a subset of the data elements contained in a record. Element
798 set names, in the retrieval module, are mapped to <emphasis>element
799 specifications</emphasis>, which contain information equivalent to the
800 <emphasis>Espec-1</emphasis> syntax of Z39.50.
806 Map tables, which may specify mappings to
807 <emphasis>other</emphasis> database profiles, if desired.
813 Possibly, a set of rules describing the mapping of elements to a
821 A list of element descriptions (this is the actual ARS of the
822 schema, in Z39.50 terms), which lists the ways in which the various
823 tags can be used and organized hierarchically.
832 Several of the entries above simply refer to other files, which
833 describe the given objects.
839 <title>The Configuration Files</title>
842 This section describes the syntax and use of the various tables which
843 are used by the retrieval module.
847 The number of different file types may appear daunting at first, but
848 each type corresponds fairly clearly to a single aspect of the Z39.50
849 retrieval facilities. Further, the average database administrator,
850 who is simply reusing an existing profile for which tables already
851 exist, shouldn't have to worry too much about the contents of these tables.
855 Generally, the files are simple ASCII files, which can be maintained
856 using any text editor. Blank lines, and lines beginning with a (#) are
857 ignored. Any characters on a line followed by a (#) are also ignored.
858 All other lines contain <emphasis>directives</emphasis>, which provide
859 some setting or value to the system.
860 Generally, settings are characterized by a single
861 keyword, identifying the setting, followed by a number of parameters.
862 Some settings are repeatable (r), while others may occur only once in a
863 file. Some settings are optional (o), while others again are
869 <sect2 id="abs-file">
870 <title>The Abstract Syntax (.abs) Files</title>
873 The name of this file type is slightly misleading in Z39.50 terms,
874 since, apart from the actual abstract syntax of the profile, it also
875 includes most of the other definitions that go into a database
880 When a record in the canonical, SGML-like format is read from a file
881 or from the database, the first tag of the file should reference the
882 profile that governs the layout of the record. If the first tag of the
883 record is, say, <literal><gils></literal>, the system will look
884 for the profile definition in the file <literal>gils.abs</literal>.
885 Profile definitions are cached, so they only have to be read once
886 during the lifespan of the current process.
890 When writing your own input filters, the
891 <emphasis>record-begin</emphasis> command
892 introduces the profile, and should always be called first thing when
893 introducing a new record.
897 The file may contain the following directives:
904 <term>name <replaceable>symbolic-name</replaceable></term>
907 (m) This provides a shorthand name or
908 description for the profile. Mostly useful for diagnostic purposes.
913 <term>reference <replaceable>OID-name</replaceable></term>
916 (m) The reference name of the OID for the profile.
917 The reference names can be found in the <emphasis>util</emphasis>
923 <term>attset <replaceable>filename</replaceable></term>
926 (m) The attribute set that is used for
927 indexing and searching records belonging to this profile.
932 <term>tagset <replaceable>filename</replaceable></term>
935 (o) The tag set (if any) that describe
936 that fields of the records.
941 <term>varset <replaceable>filename</replaceable></term>
944 (o) The variant set used in the profile.
949 <term>maptab <replaceable>filename</replaceable></term>
952 (o,r) This points to a
953 conversion table that might be used if the client asks for the record
954 in a different schema from the native one.
959 <term>marc <replaceable>filename</replaceable></term>
962 (o) Points to a file containing parameters
963 for representing the record contents in the ISO2709 syntax.
964 Read the description of the MARC representation facility below.
969 <term>esetname <replaceable>name filename</replaceable></term>
973 given element set name with an element selection file. If an (@) is
974 given in place of the filename, this corresponds to a null mapping for
975 the given element set name.
980 <term>any <replaceable>tags</replaceable></term>
983 (o) This directive specifies a list of attributes
984 which should be appended to the attribute list given for each
985 element. The effect is to make every single element in the abstract
986 syntax searchable by way of the given attributes. This directive
987 provides an efficient way of supporting free-text searching across all
988 elements. However, it does increase the size of the index
989 significantly. The attributes can be qualified with a structure, as in
990 the <replaceable>elm</replaceable> directive below.
995 <term>elm <replaceable>path name attributes</replaceable></term>
998 (o,r) Adds an element to the abstract record syntax of the schema.
999 The <replaceable>path</replaceable> follows the
1000 syntax which is suggested by the Z39.50 document - that is, a sequence
1001 of tags separated by slashes (/). Each tag is given as a
1002 comma-separated pair of tag type and -value surrounded by parenthesis.
1003 The <replaceable>name</replaceable> is the name of the element, and
1004 the <replaceable>attributes</replaceable>
1005 specifies which attributes to use when indexing the element in a
1006 comma-separated list.
1007 A ! in place of the attribute name is equivalent to
1008 specifying an attribute name identical to the element name.
1009 A - in place of the attribute name
1010 specifies that no indexing is to take place for the given element.
1011 The attributes can be qualified with <replaceable>field
1012 types</replaceable> to specify which
1013 character set should govern the indexing procedure for that field.
1014 The same data element may be indexed into several different
1015 fields, using different character set definitions.
1016 See the <xref linkend="field-structure-and-character-sets"/>.
1017 The default field type is <literal>w</literal> for
1018 <emphasis>word</emphasis>.
1024 <term>xelm <replaceable>xpath attributes</replaceable></term>
1027 Specifies indexing for record nodes given by
1028 <replaceable>xpath</replaceable>. Unlike directive
1029 elm, this directive allows you to index attribute
1030 contents. The <replaceable>xpath</replaceable> uses
1031 a syntax similar to XPath. The <replaceable>attributes</replaceable>
1032 have same syntax and meaning as directive elm, except that operator
1033 ! refers to the nodes selected by <replaceable>xpath</replaceable>.
1035 xelm / !:w default index
1036 xelm // !:w additional index
1037 xelm /gils/title/@att myatt:w index attribute @att in myatt
1038 xelm title/@att myatt:w same meaning.
1044 <term>melm <replaceable>field$subfield attributes</replaceable></term>
1047 This directive is specifically for MARC-formatted records,
1048 ingested either in the form of MARCXML documents, or in the
1049 ISO2709/Z39.2 format using the grs.marcxml input filter. You can
1050 specify indexing rules for any subfield, or you can leave off the
1051 <replaceable>$subfield</replaceable> part and specify default rules
1052 for all subfields of the given field (note: default rules should come
1053 after any subfield-specific rules in the configuration file). The
1054 <replaceable>attributes</replaceable> have the same syntax and meaning
1055 as for the 'elm' directive above.
1060 <term>encoding <replaceable>encodingname</replaceable></term>
1063 This directive specifies character encoding for external records.
1064 For records such as XML that specifies encoding within the
1065 file via a header this directive is ignored.
1066 If neither this directive is given, nor an encoding is set
1067 within external records, ISO-8859-1 encoding is assumed.
1072 <term>xpath <literal>enable</literal>/<literal>disable</literal></term>
1075 If this directive is followed by <literal>enable</literal>,
1076 then extra indexing is performed to allow for XPath-like queries.
1077 If this directive is not specified - equivalent to
1078 <literal>disable</literal> - no extra XPath-indexing is performed.
1085 <term>systag <replaceable>systemtag</replaceable> <replaceable>element</replaceable></term>
1088 This directive maps system information to an element during
1089 retrieval. This information is dynamically created. The
1090 following system tags are defined
1096 Size of record in bytes. By default this
1097 is mapped to element <literal>size</literal>.
1106 Score/rank of record. By default this
1107 is mapped to element <literal>rank</literal>.
1108 If no score was calculated for the record (non-ranked
1109 searched) search this directive is ignored.
1118 Zebra's system number (record ID) for the
1119 record. By default this is mapped to element
1120 <literal>localControlNumber</literal>.
1125 If you do not want a particular system tag to be applied,
1126 then set the resulting element to something undefined in the
1127 abs file (such as <literal>none</literal>).
1133 <!-- Mike's version -->
1137 <replaceable>systemTag</replaceable>
1138 <replaceable>actualTag</replaceable>
1142 Specifies what information, if any, Zebra should
1143 automatically include in retrieval records for the
1144 ``system fields'' that it supports.
1145 <replaceable>systemTag</replaceable> may
1146 be any of the following:
1149 <term><literal>rank</literal></term>
1151 An integer indicating the relevance-ranking score
1152 assigned to the record.
1156 <term><literal>sysno</literal></term>
1158 An automatically generated identifier for the record,
1159 unique within this database. It is represented by the
1160 <literal><localControlNumber></literal> element in
1161 XML and the <literal>(1,14)</literal> tag in GRS-1.
1165 <term><literal>size</literal></term>
1167 The size, in bytes, of the retrieved record.
1173 The <replaceable>actualTag</replaceable> parameter may be
1174 <literal>none</literal> to indicate that the named element
1175 should be omitted from retrieval records.
1184 The mechanism for controlling indexing is not adequate for
1185 complex databases, and will probably be moved into a separate
1186 configuration table eventually.
1191 The following is an excerpt from the abstract syntax file for the GILS
1199 reference GILS-schema
1204 maptab gils-usmarc.map
1208 esetname VARIANT gils-variant.est # for WAIS-compliance
1209 esetname B gils-b.est
1210 esetname G gils-g.est
1215 elm (1,14) localControlNumber Local-number
1216 elm (1,16) dateOfLastModification Date/time-last-modified
1217 elm (2,1) title w:!,p:!
1218 elm (4,1) controlIdentifier Identifier-standard
1219 elm (2,6) abstract Abstract
1220 elm (4,51) purpose !
1221 elm (4,52) originator -
1222 elm (4,53) accessConstraints !
1223 elm (4,54) useConstraints !
1224 elm (4,70) availability -
1225 elm (4,70)/(4,90) distributor -
1226 elm (4,70)/(4,90)/(2,7) distributorName !
1227 elm (4,70)/(4,90)/(2,10) distributorOrganization !
1228 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1229 elm (4,70)/(4,90)/(4,3) distributorCity !
1236 <sect2 id="attset-files">
1237 <title>The Attribute Set (.att) Files</title>
1240 This file type describes the <replaceable>Use</replaceable> elements of
1242 It contains the following directives.
1248 <term>name <replaceable>symbolic-name</replaceable></term>
1251 (m) This provides a shorthand name or
1252 description for the attribute set.
1253 Mostly useful for diagnostic purposes.
1255 </listitem></varlistentry>
1257 <term>reference <replaceable>OID-name</replaceable></term>
1260 (m) The reference name of the OID for
1262 The reference names can be found in the <replaceable>util</replaceable>
1263 module of <replaceable>YAZ</replaceable>.
1265 </listitem></varlistentry>
1267 <term>include <replaceable>filename</replaceable></term>
1270 (o,r) This directive is used to
1271 include another attribute set as a part of the current one. This is
1272 used when a new attribute set is defined as an extension to another
1273 set. For instance, many new attribute sets are defined as extensions
1274 to the <replaceable>bib-1</replaceable> set.
1275 This is an important feature of the retrieval
1276 system of Z39.50, as it ensures the highest possible level of
1277 interoperability, as those access points of your database which are
1278 derived from the external set (say, bib-1) can be used even by clients
1279 who are unaware of the new set.
1281 </listitem></varlistentry>
1284 <replaceable>att-value att-name [local-value]</replaceable></term>
1288 repeatable directive introduces a new attribute to the set. The
1289 attribute value is stored in the index (unless a
1290 <replaceable>local-value</replaceable> is
1291 given, in which case this is stored). The name is used to refer to the
1292 attribute from the <replaceable>abstract syntax</replaceable>.
1294 </listitem></varlistentry>
1299 This is an excerpt from the GILS attribute set definition.
1300 Notice how the file describing the <emphasis>bib-1</emphasis>
1301 attribute set is referenced.
1308 reference GILS-attset
1311 att 2001 distributorName
1312 att 2002 indextermsControlled
1314 att 2004 accessConstraints
1315 att 2005 useConstraints
1323 <title>The Tag Set (.tag) Files</title>
1326 This file type defines the tagset of the profile, possibly by
1327 referencing other tag sets (most tag sets, for instance, will include
1328 tagsetG and tagsetM from the Z39.50 specification. The file may
1329 contain the following directives.
1336 <term>name <emphasis>symbolic-name</emphasis></term>
1339 (m) This provides a shorthand name or
1340 description for the tag set. Mostly useful for diagnostic purposes.
1342 </listitem></varlistentry>
1344 <term>reference <emphasis>OID-name</emphasis></term>
1347 (o) The reference name of the OID for the tag set.
1348 The reference names can be found in the <emphasis>util</emphasis>
1349 module of <emphasis>YAZ</emphasis>.
1350 The directive is optional, since not all tag sets
1351 are registered outside of their schema.
1353 </listitem></varlistentry>
1355 <term>type <emphasis>integer</emphasis></term>
1358 (m) The type number of the tagset within the schema
1359 profile (note: this specification really should belong to the .abs
1360 file. This will be fixed in a future release).
1362 </listitem></varlistentry>
1364 <term>include <emphasis>filename</emphasis></term>
1367 (o,r) This directive is used
1368 to include the definitions of other tag sets into the current one.
1370 </listitem></varlistentry>
1372 <term>tag <emphasis>number names type</emphasis></term>
1375 (o,r) Introduces a new tag to the set.
1376 The <emphasis>number</emphasis> is the tag number as used
1377 in the protocol (there is currently no mechanism for
1378 specifying string tags at this point, but this would be quick
1380 The <emphasis>names</emphasis> parameter is a list of names
1381 by which the tag should be recognized in the input file format.
1382 The names should be separated by slashes (/).
1383 The <emphasis>type</emphasis> is the recommended data type of
1385 It should be one of the following:
1451 </listitem></varlistentry>
1456 The following is an excerpt from the TagsetG definition file.
1467 tag 3 publicationPlace string
1468 tag 4 publicationDate string
1469 tag 5 documentId string
1470 tag 6 abstract string
1472 tag 8 date generalizedtime
1473 tag 9 bodyOfDisplay string
1474 tag 10 organization string
1480 <sect2 id="variant-set">
1481 <title>The Variant Set (.var) Files</title>
1484 The variant set file is a straightforward representation of the
1485 variant set definitions associated with the protocol. At present, only
1486 the <emphasis>Variant-1</emphasis> set is known.
1490 These are the directives allowed in the file.
1497 <term>name <emphasis>symbolic-name</emphasis></term>
1500 (m) This provides a shorthand name or
1501 description for the variant set. Mostly useful for diagnostic purposes.
1503 </listitem></varlistentry>
1505 <term>reference <emphasis>OID-name</emphasis></term>
1508 (o) The reference name of the OID for
1509 the variant set, if one is required. The reference names can be found
1510 in the <emphasis>util</emphasis> module of <emphasis>YAZ</emphasis>.
1512 </listitem></varlistentry>
1514 <term>class <emphasis>integer class-name</emphasis></term>
1517 (m,r) Introduces a new
1518 class to the variant set.
1520 </listitem></varlistentry>
1522 <term>type <emphasis>integer type-name datatype</emphasis></term>
1526 new type to the current class (the one introduced by the most recent
1527 <emphasis>class</emphasis> directive).
1528 The type names belong to the same name space as the one used
1529 in the tag set definition file.
1531 </listitem></varlistentry>
1536 The following is an excerpt from the file describing the variant set
1537 <emphasis>Variant-1</emphasis>.
1548 type 1 variantId octetstring
1553 type 2 z39.50 string
1562 <title>The Element Set (.est) Files</title>
1565 The element set specification files describe a selection of a subset
1566 of the elements of a database record. The element selection mechanism
1567 is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
1568 syntax of the Z39.50 specification.
1569 In fact, the internal representation of an element set
1570 specification is identical to the <emphasis>Espec-1</emphasis> structure,
1571 and we'll refer you to the description of that structure for most of
1572 the detailed semantics of the directives below.
1577 Not all of the Espec-1 functionality has been implemented yet.
1578 The fields that are mentioned below all work as expected, unless
1584 The directives available in the element set file are as follows:
1590 <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
1593 (o) If variants are used in
1594 the following, this should provide the name of the variantset used
1595 (it's not currently possible to specify a different set in the
1596 individual variant request). In almost all cases (certainly all
1597 profiles known to us), the name
1598 <literal>Variant-1</literal> should be given here.
1600 </listitem></varlistentry>
1602 <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
1606 provides a default variant request for
1607 use when the individual element requests (see below) do not contain a
1608 variant request. Variant requests consist of a blank-separated list of
1609 variant components. A variant compont is a comma-separated,
1610 parenthesized triple of variant class, type, and value (the two former
1611 values being represented as integers). The value can currently only be
1612 entered as a string (this will change to depend on the definition of
1613 the variant in question). The special value (@) is interpreted as a
1614 null value, however.
1616 </listitem></varlistentry>
1619 <emphasis>path ['variant' variant-request]</emphasis></term>
1622 (o,r) This corresponds to a simple element request
1623 in <emphasis>Espec-1</emphasis>.
1624 The path consists of a sequence of tag-selectors, where each of
1625 these can consist of either:
1632 A simple tag, consisting of a comma-separated type-value pair in
1633 parenthesis, possibly followed by a colon (:) followed by an
1634 occurrences-specification (see below). The tag-value can be a number
1635 or a string. If the first character is an apostrophe ('), this
1636 forces the value to be interpreted as a string, even if it
1637 appears to be numerical.
1643 A WildThing, represented as a question mark (?), possibly
1644 followed by a colon (:) followed by an occurrences
1645 specification (see below).
1651 A WildPath, represented as an asterisk (*). Note that the last
1652 element of the path should not be a wildPath (wildpaths don't
1653 work in this version).
1662 The occurrences-specification can be either the string
1663 <literal>all</literal>, the string <literal>last</literal>, or
1664 an explicit value-range. The value-range is represented as
1665 an integer (the starting point), possibly followed by a
1666 plus (+) and a second integer (the number of elements, default
1671 The variant-request has the same syntax as the defaultVariantRequest
1672 above. Note that it may sometimes be useful to give an empty variant
1673 request, simply to disable the default for a specific set of fields
1674 (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
1675 but it works in this implementation).
1677 </listitem></varlistentry>
1682 The following is an example of an element specification belonging to
1689 simpleelement (1,10)
1690 simpleelement (1,12)
1692 simpleelement (1,14)
1694 simpleelement (4,52)
1701 <sect2 id="schema-mapping">
1702 <title>The Schema Mapping (.map) Files</title>
1705 Sometimes, the client might want to receive a database record in
1706 a schema that differs from the native schema of the record. For
1707 instance, a client might only know how to process WAIS records, while
1708 the database record is represented in a more specific schema, such as
1709 GILS. In this module, a mapping of data to one of the MARC formats is
1710 also thought of as a schema mapping (mapping the elements of the
1711 record into fields consistent with the given MARC specification, prior
1712 to actually converting the data to the ISO2709). This use of the
1713 object identifier for USMARC as a schema identifier represents an
1714 overloading of the OID which might not be entirely proper. However,
1715 it represents the dual role of schema and record syntax which
1716 is assumed by the MARC family in Z39.50.
1720 <emphasis>NOTE: FIXME! The schema-mapping functions are so far limited to a
1721 straightforward mapping of elements. This should be extended with
1722 mechanisms for conversions of the element contents, and conditional
1723 mappings of elements based on the record contents.</emphasis>
1727 These are the directives of the schema mapping file format:
1734 <term>targetName <emphasis>name</emphasis></term>
1737 (m) A symbolic name for the target schema
1738 of the table. Useful mostly for diagnostic purposes.
1740 </listitem></varlistentry>
1742 <term>targetRef <emphasis>OID-name</emphasis></term>
1745 (m) An OID name for the target schema.
1746 This is used, for instance, by a server receiving a request to present
1747 a record in a different schema from the native one.
1748 The name, again, is found in the <emphasis>oid</emphasis>
1749 module of <emphasis>YAZ</emphasis>.
1751 </listitem></varlistentry>
1753 <term>map <emphasis>element-name target-path</emphasis></term>
1757 an element mapping rule to the table.
1759 </listitem></varlistentry>
1766 <title>The MARC (ISO2709) Representation (.mar) Files</title>
1769 This file provides rules for representing a record in the ISO2709
1770 format. The rules pertain mostly to the values of the constant-length
1771 header of the record.
1775 NOTE: FIXME! This will be described better. We're in the process of
1776 re-evaluating and most likely changing the way that MARC records are
1777 handled by the system.</emphasis>
1782 <sect2 id="field-structure-and-character-sets">
1783 <title>Field Structure and Character Sets
1787 In order to provide a flexible approach to national character set
1788 handling, Zebra allows the administrator to configure the set up the
1789 system to handle any 8-bit character set — including sets that
1790 require multi-octet diacritics or other multi-octet characters. The
1791 definition of a character set includes a specification of the
1792 permissible values, their sort order (this affects the display in the
1793 SCAN function), and relationships between upper- and lowercase
1794 characters. Finally, the definition includes the specification of
1795 space characters for the set.
1799 The operator can define different character sets for different fields,
1800 typical examples being standard text fields, numerical fields, and
1801 special-purpose fields such as WWW-style linkages (URx).
1804 <sect3 id="default-idx-file">
1805 <title>The default.idx file</title>
1807 The field types, and hence character sets, are associated with data
1808 elements by the .abs files (see above).
1809 The file <literal>default.idx</literal>
1810 provides the association between field type codes (as used in the .abs
1811 files) and the character map files (with the .chr suffix). The format
1812 of the .idx file is as follows
1819 <term>index <emphasis>field type code</emphasis></term>
1822 This directive introduces a new search index code.
1823 The argument is a one-character code to be used in the
1824 .abs files to select this particular index type. An index, roughly,
1825 corresponds to a particular structure attribute during search. Refer
1826 to <xref linkend="search"/>.
1828 </listitem></varlistentry>
1830 <term>sort <emphasis>field code type</emphasis></term>
1833 This directive introduces a
1834 sort index. The argument is a one-character code to be used in the
1835 .abs fie to select this particular index type. The corresponding
1836 use attribute must be used in the sort request to refer to this
1837 particular sort index. The corresponding character map (see below)
1838 is used in the sort process.
1840 </listitem></varlistentry>
1842 <term>completeness <emphasis>boolean</emphasis></term>
1845 This directive enables or disables complete field indexing.
1846 The value of the <emphasis>boolean</emphasis> should be 0
1847 (disable) or 1. If completeness is enabled, the index entry will
1848 contain the complete contents of the field (up to a limit), with words
1849 (non-space characters) separated by single space characters
1850 (normalized to " " on display). When completeness is
1851 disabled, each word is indexed as a separate entry. Complete subfield
1852 indexing is most useful for fields which are typically browsed (eg.
1853 titles, authors, or subjects), or instances where a match on a
1854 complete subfield is essential (eg. exact title searching). For fields
1855 where completeness is disabled, the search engine will interpret a
1856 search containing space characters as a word proximity search.
1858 </listitem></varlistentry>
1860 <term>charmap <emphasis>filename</emphasis></term>
1863 This is the filename of the character
1864 map to be used for this index for field type.
1866 </listitem></varlistentry>
1871 <sect3 id="character-map-files">
1872 <title>The character map file format</title>
1874 The contents of the character map files are structured as follows:
1881 <term>lowercase <emphasis>value-set</emphasis></term>
1884 This directive introduces the basic value set of the field type.
1885 The format is an ordered list (without spaces) of the
1886 characters which may occur in "words" of the given type.
1887 The order of the entries in the list determines the
1888 sort order of the index. In addition to single characters, the
1889 following combinations are legal:
1897 Backslashes may be used to introduce three-digit octal, or
1898 two-digit hex representations of single characters
1899 (preceded by <literal>x</literal>).
1900 In addition, the combinations
1901 \\, \\r, \\n, \\t, \\s (space — remember that real
1902 space-characters may not occur in the value definition), and
1903 \\ are recognized, with their usual interpretation.
1909 Curly braces {} may be used to enclose ranges of single
1910 characters (possibly using the escape convention described in the
1911 preceding point), eg. {a-z} to introduce the
1912 standard range of ASCII characters.
1913 Note that the interpretation of such a range depends on
1914 the concrete representation in your local, physical character set.
1920 paranthesises () may be used to enclose multi-byte characters -
1921 eg. diacritics or special national combinations (eg. Spanish
1922 "ll"). When found in the input stream (or a search term),
1923 these characters are viewed and sorted as a single character, with a
1924 sorting value depending on the position of the group in the value
1932 </listitem></varlistentry>
1934 <term>uppercase <emphasis>value-set</emphasis></term>
1937 This directive introduces the
1938 upper-case equivalencis to the value set (if any). The number and
1939 order of the entries in the list should be the same as in the
1940 <literal>lowercase</literal> directive.
1942 </listitem></varlistentry>
1944 <term>space <emphasis>value-set</emphasis></term>
1947 This directive introduces the character
1948 which separate words in the input stream. Depending on the
1949 completeness mode of the field in question, these characters either
1950 terminate an index entry, or delimit individual "words" in
1951 the input stream. The order of the elements is not significant —
1952 otherwise the representation is the same as for the
1953 <literal>uppercase</literal> and <literal>lowercase</literal>
1956 </listitem></varlistentry>
1958 <term>map <emphasis>value-set</emphasis>
1959 <emphasis>target</emphasis></term>
1962 This directive introduces a
1963 mapping between each of the members of the value-set on the left to
1964 the character on the right. The character on the right must occur in
1965 the value set (the <literal>lowercase</literal> directive) of
1966 the character set, but
1967 it may be a paranthesis-enclosed multi-octet character. This directive
1968 may be used to map diacritics to their base characters, or to map
1969 HTML-style character-representations to their natural form, etc. The map directive
1970 can also be used to ignore leading articles in searching and/or sorting, and to perform
1971 other special transformations. See section <xref linkend="leading-articles"/>.
1973 </listitem></varlistentry>
1977 <sect3 id="leading-articles">
1978 <title>Ignoring leading articles</title>
1980 In addition to specifying sort orders, space (blank) handling, and upper/lowercase folding,
1981 you can also use the character map files to make Zebra ignore leading articles in sorting
1982 records, or when doing complete field searching.
1985 This is done using the <literal>map</literal> directive in the character map file. In a
1986 nutshell, what you do is map certain sequences of characters, when they occur <emphasis>
1987 in the beginning of a field</emphasis>, to a space. Assuming that the character "@" is
1988 defined as a space character in your file, you can do:
1993 The effect of these directives is to map either 'the' or 'The', followed by a space
1994 character, to a space. The hat ^ character denotes beginning-of-field only when
1995 complete-subfield indexing or sort indexing is taking place; otherwise, it is treated just
1996 as any other character.
1999 Because the <literal>default.idx</literal> file can be used to associate different
2000 character maps with different indexing types -- and you can create additional indexing
2001 types, should the need arise -- it is possible to specify that leading articles should be
2002 ignored either in sorting, in complete-field searching, or both.
2005 If you ignore certain prefixes in sorting, then these will be eliminated from the index,
2006 and sorting will take place as if they weren't there. However, if you set the system up
2007 to ignore certain prefixes in <emphasis>searching</emphasis>, then these are deleted both
2008 from the indexes and from query terms, when the client specifies complete-field
2009 searching. This has the effect that a search for 'the science journal' and 'science
2010 journal' would both produce the same results.
2016 <sect1 id="formats">
2017 <title>Exchange Formats</title>
2020 Converting records from the internal structure to en exchange format
2021 is largely an automatic process. Currently, the following exchange
2022 formats are supported:
2029 GRS-1. The internal representation is based on GRS-1/XML, so the
2030 conversion here is straightforward. The system will create
2031 applied variant and supported variant lists as required, if a record
2032 contains variant information.
2038 XML. The internal representation is based on GRS-1/XML so
2039 the mapping is trivial. Note that XML schemas, preprocessing
2040 instructions and comments are not part of the internal representation
2041 and therefore will never be part of a generated XML record.
2042 Future versions of the Zebra will support that.
2048 SUTRS. Again, the mapping is fairly straightforward. Indentation
2049 is used to show the hierarchical structure of the record. All
2050 "GRS" type records support both the GRS-1 and SUTRS
2052 <!-- FIXME - What is SUTRS - should be expanded here -->
2058 ISO2709-based formats (USMARC, etc.). Only records with a
2059 two-level structure (corresponding to fields and subfields) can be
2060 directly mapped to ISO2709. For records with a different structuring
2061 (eg., GILS), the representation in a structure like USMARC involves a
2062 schema-mapping (see <xref linkend="schema-mapping"/>), to an
2063 "implied" USMARC schema (implied,
2064 because there is no formal schema which specifies the use of the
2065 USMARC fields outside of ISO2709). The resultant, two-level record is
2066 then mapped directly from the internal representation to ISO2709. See
2067 the GILS schema definition files for a detailed example of this
2074 Explain. This representation is only available for records
2075 belonging to the Explain schema.
2081 Summary. This ASN-1 based structure is only available for records
2082 belonging to the Summary schema - or schema which provide a mapping
2083 to this schema (see the description of the schema mapping facility
2090 SOIF. Support for this syntax is experimental, and is currently
2091 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
2092 abstract syntaxes can be mapped to the SOIF format, although nested
2093 elements are represented by concatenation of the tag names at each
2095 <!-- FIXME - Is this used anywhere ? -H -->
2104 <!-- Keep this comment at the end of the file
2109 sgml-minimize-attributes:nil
2110 sgml-always-quote-attributes:t
2113 sgml-parent-document: "zebra.xml"
2114 sgml-local-catalogs: nil
2115 sgml-namecase-general:t
projects / idzebra-moved-to-github.git / blob