2 <title>&acro.grs1; Record Model and Filter Modules</title>
6 The functionality of this record model has been improved and
7 replaced by the DOM &acro.xml; record model. See
8 <xref linkend="record-model-domxml"/>.
13 The record model described in this chapter applies to the fundamental,
15 record type <literal>grs</literal>, introduced in
16 <xref linkend="componentmodulesgrs"/>.
20 <section id="grs-filters">
21 <title>&acro.grs1; Record Filters</title>
23 Many basic subtypes of the <emphasis>grs</emphasis> type are
30 <term><literal>grs.sgml</literal></term>
33 This is the canonical input format
34 described <xref linkend="grs-canonical-format"/>. It is using
35 simple &acro.sgml;-like syntax.
40 <term><literal>grs.marc.</literal><replaceable>type</replaceable></term>
43 This allows &zebra; to read
44 records in the ISO2709 (&acro.marc;) encoding standard.
45 Last parameter <replaceable>type</replaceable> names the
46 <literal>.abs</literal> file (see below)
47 which describes the specific &acro.marc; structure of the input record as
48 well as the indexing rules.
50 <para>The <literal>grs.marc</literal> uses an internal representation
51 which is not &acro.xml; conformant. In particular &acro.marc; tags are
52 presented as elements with the same name. And &acro.xml; elements
53 may not start with digits. Therefore this filter is only
54 suitable for systems returning &acro.grs1; and &acro.marc; records. For &acro.xml;
55 use <literal>grs.marcxml</literal> filter instead (see below).
58 The loadable <literal>grs.marc</literal> filter module
59 is packaged in the GNU/Debian package
60 <literal>libidzebra2.0-mod-grs-marc</literal>
65 <term><literal>grs.marcxml.</literal><replaceable>type</replaceable></term>
68 This allows &zebra; to read ISO2709 encoded records.
69 Last parameter <replaceable>type</replaceable> names the
70 <literal>.abs</literal> file (see below)
71 which describes the specific &acro.marc; structure of the input record as
72 well as the indexing rules.
75 The internal representation for <literal>grs.marcxml</literal>
76 is the same as for <ulink url="&url.marcxml;">&acro.marcxml;</ulink>.
77 It slightly more complicated to work with than
78 <literal>grs.marc</literal> but &acro.xml; conformant.
81 The loadable <literal>grs.marcxml</literal> filter module
82 is also contained in the GNU/Debian package
83 <literal>libidzebra2.0-mod-grs-marc</literal>
88 <term><literal>grs.xml</literal></term>
91 This filter reads &acro.xml; records and uses
92 <ulink url="http://expat.sourceforge.net/">Expat</ulink> to
93 parse them and convert them into ID&zebra;'s internal
94 <literal>grs</literal> record model.
95 Only one record per file is supported, due to the fact &acro.xml; does
96 not allow two documents to "follow" each other (there is no way
97 to know when a document is finished).
98 This filter is only available if &zebra; is compiled with EXPAT support.
101 The loadable <literal>grs.xml</literal> filter module
102 is packaged in the GNU/Debian package
103 <literal>libidzebra2.0-mod-grs-xml</literal>
108 <term><literal>grs.regx.</literal><replaceable>filter</replaceable></term>
111 This enables a user-supplied Regular Expressions input
112 filter described in <xref linkend="grs-regx-tcl"/>.
115 The loadable <literal>grs.regx</literal> filter module
116 is packaged in the GNU/Debian package
117 <literal>libidzebra2.0-mod-grs-regx</literal>
122 <term><literal>grs.tcl.</literal><replaceable>filter</replaceable></term>
125 Similar to grs.regx but using Tcl for rules, described in
126 <xref linkend="grs-regx-tcl"/>.
129 The loadable <literal>grs.tcl</literal> filter module
130 is also packaged in the GNU/Debian package
131 <literal>libidzebra2.0-mod-grs-regx</literal>
139 <section id="grs-canonical-format">
140 <title>&acro.grs1; Canonical Input Format</title>
143 Although input data can take any form, it is sometimes useful to
144 describe the record processing capabilities of the system in terms of
145 a single, canonical input format that gives access to the full
146 spectrum of structure and flexibility in the system. In &zebra;, this
147 canonical format is an "&acro.sgml;-like" syntax.
151 To use the canonical format specify <literal>grs.sgml</literal> as
156 Consider a record describing an information resource (such a record is
157 sometimes known as a <emphasis>locator record</emphasis>).
158 It might contain a field describing the distributor of the
159 information resource, which might in turn be partitioned into
160 various fields providing details about the distributor, like this:
166 <Distributor>
167 <Name> USGS/WRD </Name>
168 <Organization> USGS/WRD </Organization>
169 <Street-Address>
170 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
171 </Street-Address>
172 <City> ALBUQUERQUE </City>
173 <State> NM </State>
174 <Zip-Code> 87102 </Zip-Code>
175 <Country> USA </Country>
176 <Telephone> (505) 766-5560 </Telephone>
177 </Distributor>
182 <!-- There is no indentation in the example above! -H
185 The indentation used above is used to illustrate how &zebra;
186 interprets the mark-up. The indentation, in itself, has no
187 significance to the parser for the canonical input format, which
188 discards superfluous whitespace.
194 The keywords surrounded by <...> are
195 <emphasis>tags</emphasis>, while the sections of text
196 in between are the <emphasis>data elements</emphasis>.
197 A data element is characterized by its location in the tree
198 that is made up by the nested elements.
199 Each element is terminated by a closing tag - beginning
200 with <literal><</literal>/, and containing the same symbolic
201 tag-name as the corresponding opening tag.
202 The general closing tag - <literal></></literal> -
203 terminates the element started by the last opening tag. The
204 structuring of elements is significant.
205 The element <emphasis>Telephone</emphasis>,
206 for instance, may be indexed and presented to the client differently,
207 depending on whether it appears inside the
208 <emphasis>Distributor</emphasis> element, or some other,
209 structured data element such a <emphasis>Supplier</emphasis> element.
212 <section id="grs-record-root">
213 <title>Record Root</title>
216 The first tag in a record describes the root node of the tree that
217 makes up the total record. In the canonical input format, the root tag
218 should contain the name of the schema that lends context to the
219 elements of the record
220 (see <xref linkend="grs-internal-representation"/>).
221 The following is a GILS record that
222 contains only a single element (strictly speaking, that makes it an
223 illegal GILS record, since the GILS profile includes several mandatory
224 elements - &zebra; does not validate the contents of a record against
225 the &acro.z3950; profile, however - it merely attempts to match up elements
226 of a local representation with the given schema):
233 <title>Zen and the Art of Motorcycle Maintenance</title>
241 <section id="grs-variants">
242 <title>Variants</title>
245 &zebra; allows you to provide individual data elements in a number of
246 <emphasis>variant forms</emphasis>. Examples of variant forms are
247 textual data elements which might appear in different languages, and
248 images which may appear in different formats or layouts.
249 The variant system in &zebra; is essentially a representation of
250 the variant mechanism of &acro.z3950;-1995.
254 The following is an example of a title element which occurs in two
262 <var lang lang "eng">
263 Zen and the Art of Motorcycle Maintenance</>
264 <var lang lang "dan">
265 Zen og Kunsten at Vedligeholde en Motorcykel</>
272 The syntax of the <emphasis>variant element</emphasis> is
273 <literal><var class type value></literal>.
274 The available values for the <emphasis>class</emphasis> and
275 <emphasis>type</emphasis> fields are given by the variant set
276 that is associated with the current schema
277 (see <xref linkend="grs-variants"/>).
281 Variant elements are terminated by the general end-tag </>, by
282 the variant end-tag </var>, by the appearance of another variant
283 tag with the same <emphasis>class</emphasis> and
284 <emphasis>value</emphasis> settings, or by the
285 appearance of another, normal tag. In other words, the end-tags for
286 the variants used in the example above could have been omitted.
290 Variant elements can be nested. The element
297 <var lang lang "eng"><var body iana "text/plain">
298 Zen and the Art of Motorcycle Maintenance
305 Associates two variant components to the variant list for the title
310 Given the nesting rules described above, we could write
317 <var body iana "text/plain>
318 <var lang lang "eng">
319 Zen and the Art of Motorcycle Maintenance
320 <var lang lang "dan">
321 Zen og Kunsten at Vedligeholde en Motorcykel
328 The title element above comes in two variants. Both have the IANA body
329 type "text/plain", but one is in English, and the other in
330 Danish. The client, using the element selection mechanism of &acro.z3950;,
331 can retrieve information about the available variant forms of data
332 elements, or it can select specific variants based on the requirements
340 <section id="grs-regx-tcl">
341 <title>&acro.grs1; REGX And TCL Input Filters</title>
344 In order to handle general input formats, &zebra; allows the
345 operator to define filters which read individual records in their
346 native format and produce an internal representation that the system
351 Input filters are ASCII files, generally with the suffix
352 <literal>.flt</literal>.
353 The system looks for the files in the directories given in the
354 <emphasis>profilePath</emphasis> setting in the
355 <literal>zebra.cfg</literal> files.
356 The record type for the filter is
357 <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
358 (fundamental type <literal>grs</literal>, file read
359 type <literal>regx</literal>, argument
360 <emphasis>filter-filename</emphasis>).
364 Generally, an input filter consists of a sequence of rules, where each
365 rule consists of a sequence of expressions, followed by an action. The
366 expressions are evaluated against the contents of the input record,
367 and the actions normally contribute to the generation of an internal
368 representation of the record.
372 An expression can be either of the following:
379 <term><literal>INIT</literal></term>
382 The action associated with this expression is evaluated
383 exactly once in the lifetime of the application, before any records
384 are read. It can be used in conjunction with an action that
385 initializes tables or other resources that are used in the processing
391 <term><literal>BEGIN</literal></term>
394 Matches the beginning of the record. It can be used to
395 initialize variables, etc. Typically, the
396 <emphasis>BEGIN</emphasis> rule is also used
397 to establish the root node of the record.
402 <term><literal>END</literal></term>
405 Matches the end of the record - when all of the contents
406 of the record has been processed.
412 <literal>/</literal><replaceable>reg</replaceable><literal>/</literal>
416 Matches regular expression pattern <replaceable>reg</replaceable>
417 from the input record. The operators supported are the same
418 as for regular expression queries. Refer to
419 <xref linkend="querymodel-regular"/>.
424 <term><literal>BODY</literal></term>
427 This keyword may only be used between two patterns.
428 It matches everything between (not including) those patterns.
433 <term><literal>FINISH</literal></term>
436 The expression associated with this pattern is evaluated
437 once, before the application terminates. It can be used to release
438 system resources - typically ones allocated in the
439 <emphasis>INIT</emphasis> step.
447 An action is surrounded by curly braces ({...}), and
448 consists of a sequence of statements. Statements may be separated
449 by newlines or semicolons (;).
450 Within actions, the strings that matched the expressions
451 immediately preceding the action can be referred to as
456 The available statements are:
463 <term>begin <replaceable>type [parameter ... ]</replaceable></term>
467 data element. The <replaceable>type</replaceable> is one of
475 Begin a new record. The following parameter should be the
476 name of the schema that describes the structure of the record, e.g.,
477 <literal>gils</literal> or <literal>wais</literal> (see below).
478 The <literal>begin record</literal> call should precede
479 any other use of the <replaceable>begin</replaceable> statement.
487 Begin a new tagged element. The parameter is the
488 name of the tag. If the tag is not matched anywhere in the tagsets
489 referenced by the current schema, it is treated as a local string
498 Begin a new node in a variant tree. The parameters are
499 <replaceable>class type value</replaceable>.
508 <term>data <replaceable>parameter</replaceable></term>
511 Create a data element. The concatenated arguments make
512 up the value of the data element.
513 The option <literal>-text</literal> signals that
514 the layout (whitespace) of the data should be retained for
516 The option <literal>-element</literal>
517 <replaceable>tag</replaceable> wraps the data up in
518 the <replaceable>tag</replaceable>.
519 The use of the <literal>-element</literal> option is equivalent to
520 preceding the command with a <replaceable>begin
521 element</replaceable> command, and following
522 it with the <replaceable>end</replaceable> command.
527 <term>end <replaceable>[type]</replaceable></term>
530 Close a tagged element. If no parameter is given,
531 the last element on the stack is terminated.
532 The first parameter, if any, is a type name, similar
533 to the <replaceable>begin</replaceable> statement.
534 For the <replaceable>element</replaceable> type, a tag
535 name can be provided to terminate a specific tag.
541 <term>unread <replaceable>no</replaceable></term>
544 Move the input pointer to the offset of first character that
545 match rule given by <replaceable>no</replaceable>.
546 The first rule from left-to-right is numbered zero,
547 the second rule is named 1 and so on.
556 The following input filter reads a Usenet news file, producing a
557 record in the WAIS schema. Note that the body of a news posting is
558 separated from the list of headers by a blank line (or rather a
559 sequence of two newline characters.
565 BEGIN { begin record wais }
567 /^From:/ BODY /$/ { data -element name $1 }
568 /^Subject:/ BODY /$/ { data -element title $1 }
569 /^Date:/ BODY /$/ { data -element lastModified $1 }
571 begin element bodyOfDisplay
572 begin variant body iana "text/plain"
581 If &zebra; is compiled with support for Tcl enabled, the statements
582 described above are supplemented with a complete
583 scripting environment, including control structures (conditional
584 expressions and loop constructs), and powerful string manipulation
585 mechanisms for modifying the elements of a record.
592 <section id="grs-internal-representation">
593 <title>&acro.grs1; Internal Record Representation</title>
596 When records are manipulated by the system, they're represented in a
597 tree-structure, with data elements at the leaf nodes, and tags or
598 variant components at the non-leaf nodes. The root-node identifies the
599 schema that lends context to the tagging and structuring of the
600 record. Imagine a simple record, consisting of a 'title' element and
608 TITLE "Zen and the Art of Motorcycle Maintenance"
609 AUTHOR "Robert Pirsig"
615 A slightly more complex record would have the author element consist
616 of two elements, a surname and a first name:
623 TITLE "Zen and the Art of Motorcycle Maintenance"
632 The root of the record will refer to the record schema that describes
633 the structuring of this particular record. The schema defines the
634 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
635 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
636 addition, the schema establishes element set names that are used by
637 the client to request a subset of the elements of a given record. The
638 schema may also establish rules for converting the record to a
639 different schema, by stating, for each element, a mapping to a
643 <section id="grs-tagged-elements">
644 <title>Tagged Elements</title>
647 A data element is characterized by its tag, and its position in the
648 structure of the record. For instance, while the tag "telephone
649 number" may be used different places in a record, we may need to
650 distinguish between these occurrences, both for searching and
651 presentation purposes. For instance, while the phone numbers for the
652 "customer" and the "service provider" are both
653 representatives for the same type of resource (a telephone number), it
654 is essential that they be kept separate. The record schema provides
655 the structure of the record, and names each data element (defined by
656 the sequence of tags - the tag path - by which the element can be
657 reached from the root of the record).
662 <section id="grs-variant-details">
663 <title>Variants</title>
666 The children of a tag node may be either more tag nodes, a data node
667 (possibly accompanied by tag nodes),
668 or a tree of variant nodes. The children of variant nodes are either
669 more variant nodes or a data node (possibly accompanied by more
670 variant nodes). Each leaf node, which is normally a
671 data node, corresponds to a <emphasis>variant form</emphasis> of the
672 tagged element identified by the tag which parents the variant tree.
673 The following title element occurs in two different languages:
679 VARIANT LANG=ENG "War and Peace"
681 VARIANT LANG=DAN "Krig og Fred"
687 Which of the two elements are transmitted to the client by the server
688 depends on the specifications provided by the client, if any.
692 In practice, each variant node is associated with a triple of class,
693 type, value, corresponding to the variant mechanism of &acro.z3950;.
698 <section id="grs-data-elements">
699 <title>Data Elements</title>
702 Data nodes have no children (they are always leaf nodes in the record
707 FIXME! Documentation needs extension here about types of nodes - numerical,
708 textual, etc., plus the various types of inclusion notes.
716 <section id="grs-conf">
717 <title>&acro.grs1; Record Model Configuration</title>
720 The following sections describe the configuration files that govern
721 the internal management of <literal>grs</literal> records.
722 The system searches for the files
723 in the directories specified by the <emphasis>profilePath</emphasis>
724 setting in the <literal>zebra.cfg</literal> file.
727 <section id="grs-abstract-syntax">
728 <title>The Abstract Syntax</title>
731 The abstract syntax definition (also known as an Abstract Record
732 Structure, or ARS) is the focal point of the
733 record schema description. For a given schema, the ABS file may state any
734 or all of the following:
738 FIXME - Need a diagram here, or a simple explanation how it all hangs together -H
747 The object identifier of the &acro.z3950; schema associated
748 with the ARS, so that it can be referred to by the client.
754 The attribute set (which can possibly be a compound of multiple
755 sets) which applies in the profile. This is used when indexing and
756 searching the records belonging to the given profile.
762 The tag set (again, this can consist of several different sets).
763 This is used when reading the records from a file, to recognize the
764 different tags, and when transmitting the record to the client -
765 mapping the tags to their numerical representation, if they are
772 The variant set which is used in the profile. This provides a
773 vocabulary for specifying the <emphasis>forms</emphasis> of
774 data that appear inside the records.
780 Element set names, which are a shorthand way for the client to
781 ask for a subset of the data elements contained in a record. Element
782 set names, in the retrieval module, are mapped to <emphasis>element
783 specifications</emphasis>, which contain information equivalent to the
784 <emphasis>Espec-1</emphasis> syntax of &acro.z3950;.
790 Map tables, which may specify mappings to
791 <emphasis>other</emphasis> database profiles, if desired.
797 Possibly, a set of rules describing the mapping of elements to a
798 &acro.marc; representation.
805 A list of element descriptions (this is the actual ARS of the
806 schema, in &acro.z3950; terms), which lists the ways in which the various
807 tags can be used and organized hierarchically.
816 Several of the entries above simply refer to other files, which
817 describe the given objects.
822 <section id="grs-configuration-files">
823 <title>The Configuration Files</title>
826 This section describes the syntax and use of the various tables which
827 are used by the retrieval module.
831 The number of different file types may appear daunting at first, but
832 each type corresponds fairly clearly to a single aspect of the &acro.z3950;
833 retrieval facilities. Further, the average database administrator,
834 who is simply reusing an existing profile for which tables already
835 exist, shouldn't have to worry too much about the contents of these tables.
839 Generally, the files are simple ASCII files, which can be maintained
840 using any text editor. Blank lines, and lines beginning with a (#) are
841 ignored. Any characters on a line followed by a (#) are also ignored.
842 All other lines contain <emphasis>directives</emphasis>, which provide
843 some setting or value to the system.
844 Generally, settings are characterized by a single
845 keyword, identifying the setting, followed by a number of parameters.
846 Some settings are repeatable (r), while others may occur only once in a
847 file. Some settings are optional (o), while others again are
853 <section id="abs-file">
854 <title>The Abstract Syntax (.abs) Files</title>
857 The name of this file type is slightly misleading in &acro.z3950; terms,
858 since, apart from the actual abstract syntax of the profile, it also
859 includes most of the other definitions that go into a database
864 When a record in the canonical, &acro.sgml;-like format is read from a file
865 or from the database, the first tag of the file should reference the
866 profile that governs the layout of the record. If the first tag of the
867 record is, say, <literal><gils></literal>, the system will look
868 for the profile definition in the file <literal>gils.abs</literal>.
869 Profile definitions are cached, so they only have to be read once
870 during the lifespan of the current process.
874 When writing your own input filters, the
875 <emphasis>record-begin</emphasis> command
876 introduces the profile, and should always be called first thing when
877 introducing a new record.
881 The file may contain the following directives:
888 <term>name <replaceable>symbolic-name</replaceable></term>
891 (m) This provides a shorthand name or
892 description for the profile. Mostly useful for diagnostic purposes.
897 <term>reference <replaceable>OID-name</replaceable></term>
900 (m) The reference name of the OID for the profile.
901 The reference names can be found in the <emphasis>util</emphasis>
907 <term>attset <replaceable>filename</replaceable></term>
910 (m) The attribute set that is used for
911 indexing and searching records belonging to this profile.
916 <term>tagset <replaceable>filename</replaceable></term>
919 (o) The tag set (if any) that describe
920 that fields of the records.
925 <term>varset <replaceable>filename</replaceable></term>
928 (o) The variant set used in the profile.
933 <term>maptab <replaceable>filename</replaceable></term>
936 (o,r) This points to a
937 conversion table that might be used if the client asks for the record
938 in a different schema from the native one.
943 <term>marc <replaceable>filename</replaceable></term>
946 (o) Points to a file containing parameters
947 for representing the record contents in the ISO2709 syntax.
948 Read the description of the &acro.marc; representation facility below.
953 <term>esetname <replaceable>name filename</replaceable></term>
957 given element set name with an element selection file. If an (@) is
958 given in place of the filename, this corresponds to a null mapping for
959 the given element set name.
964 <term>all <replaceable>tags</replaceable></term>
967 (o) This directive specifies a list of attributes
968 which should be appended to the attribute list given for each
969 element. The effect is to make every single element in the abstract
970 syntax searchable by way of the given attributes. This directive
971 provides an efficient way of supporting free-text searching across all
972 elements. However, it does increase the size of the index
973 significantly. The attributes can be qualified with a structure, as in
974 the <replaceable>elm</replaceable> directive below.
979 <term>elm <replaceable>path name attributes</replaceable></term>
982 (o,r) Adds an element to the abstract record syntax of the schema.
983 The <replaceable>path</replaceable> follows the
984 syntax which is suggested by the &acro.z3950; document - that is, a sequence
985 of tags separated by slashes (/). Each tag is given as a
986 comma-separated pair of tag type and -value surrounded by parenthesis.
987 The <replaceable>name</replaceable> is the name of the element, and
988 the <replaceable>attributes</replaceable>
989 specifies which attributes to use when indexing the element in a
990 comma-separated list.
991 A <literal>!</literal> in place of the attribute name is equivalent
992 to specifying an attribute name identical to the element name.
993 A <literal>-</literal> in place of the attribute name
994 specifies that no indexing is to take place for the given element.
995 The attributes can be qualified with <replaceable>field
996 types</replaceable> to specify which
997 character set should govern the indexing procedure for that field.
998 The same data element may be indexed into several different
999 fields, using different character set definitions.
1000 See the <xref linkend="fields-and-charsets"/>.
1001 The default field type is <literal>w</literal> for
1002 <emphasis>word</emphasis>.
1008 <term>xelm <replaceable>xpath attributes</replaceable></term>
1011 Specifies indexing for record nodes given by
1012 <replaceable>xpath</replaceable>. Unlike directive
1013 elm, this directive allows you to index attribute
1014 contents. The <replaceable>xpath</replaceable> uses
1015 a syntax similar to XPath. The <replaceable>attributes</replaceable>
1016 have same syntax and meaning as directive elm, except that operator
1017 ! refers to the nodes selected by <replaceable>xpath</replaceable>.
1019 xelm / !:w default index
1020 xelm // !:w additional index
1021 xelm /gils/title/@att myatt:w index attribute @att in myatt
1022 xelm title/@att myatt:w same meaning.
1028 <term>melm <replaceable>field$subfield attributes</replaceable></term>
1031 This directive is specifically for &acro.marc;-formatted records,
1032 ingested either in the form of &acro.marcxml; documents, or in the
1033 ISO2709/Z39.2 format using the grs.marcxml input filter. You can
1034 specify indexing rules for any subfield, or you can leave off the
1035 <replaceable>$subfield</replaceable> part and specify default rules
1036 for all subfields of the given field (note: default rules should come
1037 after any subfield-specific rules in the configuration file). The
1038 <replaceable>attributes</replaceable> have the same syntax and meaning
1039 as for the 'elm' directive above.
1044 <term>encoding <replaceable>encodingname</replaceable></term>
1047 This directive specifies character encoding for external records.
1048 For records such as &acro.xml; that specifies encoding within the
1049 file via a header this directive is ignored.
1050 If neither this directive is given, nor an encoding is set
1051 within external records, ISO-8859-1 encoding is assumed.
1056 <term>xpath <literal>enable</literal>/<literal>disable</literal></term>
1059 If this directive is followed by <literal>enable</literal>,
1060 then extra indexing is performed to allow for XPath-like queries.
1061 If this directive is not specified - equivalent to
1062 <literal>disable</literal> - no extra XPath-indexing is performed.
1069 <term>systag <replaceable>systemtag</replaceable> <replaceable>element</replaceable></term>
1072 This directive maps system information to an element during
1073 retrieval. This information is dynamically created. The
1074 following system tags are defined
1080 Size of record in bytes. By default this
1081 is mapped to element <literal>size</literal>.
1090 Score/rank of record. By default this
1091 is mapped to element <literal>rank</literal>.
1092 If no score was calculated for the record (non-ranked
1093 searched) search this directive is ignored.
1102 &zebra;'s system number (record ID) for the
1103 record. By default this is mapped to element
1104 <literal>localControlNumber</literal>.
1109 If you do not want a particular system tag to be applied,
1110 then set the resulting element to something undefined in the
1111 abs file (such as <literal>none</literal>).
1117 <!-- Mike's version -->
1121 <replaceable>systemTag</replaceable>
1122 <replaceable>actualTag</replaceable>
1126 Specifies what information, if any, &zebra; should
1127 automatically include in retrieval records for the
1128 ``system fields'' that it supports.
1129 <replaceable>systemTag</replaceable> may
1130 be any of the following:
1133 <term><literal>rank</literal></term>
1135 An integer indicating the relevance-ranking score
1136 assigned to the record.
1140 <term><literal>sysno</literal></term>
1142 An automatically generated identifier for the record,
1143 unique within this database. It is represented by the
1144 <literal><localControlNumber></literal> element in
1145 &acro.xml; and the <literal>(1,14)</literal> tag in &acro.grs1;.
1149 <term><literal>size</literal></term>
1151 The size, in bytes, of the retrieved record.
1157 The <replaceable>actualTag</replaceable> parameter may be
1158 <literal>none</literal> to indicate that the named element
1159 should be omitted from retrieval records.
1168 The mechanism for controlling indexing is not adequate for
1169 complex databases, and will probably be moved into a separate
1170 configuration table eventually.
1175 The following is an excerpt from the abstract syntax file for the GILS
1183 reference GILS-schema
1188 maptab gils-usmarc.map
1192 esetname VARIANT gils-variant.est # for WAIS-compliance
1193 esetname B gils-b.est
1194 esetname G gils-g.est
1199 elm (1,14) localControlNumber Local-number
1200 elm (1,16) dateOfLastModification Date/time-last-modified
1201 elm (2,1) title w:!,p:!
1202 elm (4,1) controlIdentifier Identifier-standard
1203 elm (2,6) abstract Abstract
1204 elm (4,51) purpose !
1205 elm (4,52) originator -
1206 elm (4,53) accessConstraints !
1207 elm (4,54) useConstraints !
1208 elm (4,70) availability -
1209 elm (4,70)/(4,90) distributor -
1210 elm (4,70)/(4,90)/(2,7) distributorName !
1211 elm (4,70)/(4,90)/(2,10) distributorOrganization !
1212 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1213 elm (4,70)/(4,90)/(4,3) distributorCity !
1220 <section id="attset-files">
1221 <title>The Attribute Set (.att) Files</title>
1224 This file type describes the <replaceable>Use</replaceable> elements of
1226 It contains the following directives.
1232 <term>name <replaceable>symbolic-name</replaceable></term>
1235 (m) This provides a shorthand name or
1236 description for the attribute set.
1237 Mostly useful for diagnostic purposes.
1239 </listitem></varlistentry>
1241 <term>reference <replaceable>OID-name</replaceable></term>
1244 (m) The reference name of the OID for
1246 The reference names can be found in the <replaceable>util</replaceable>
1247 module of <replaceable>&yaz;</replaceable>.
1249 </listitem></varlistentry>
1251 <term>include <replaceable>filename</replaceable></term>
1254 (o,r) This directive is used to
1255 include another attribute set as a part of the current one. This is
1256 used when a new attribute set is defined as an extension to another
1257 set. For instance, many new attribute sets are defined as extensions
1258 to the <replaceable>bib-1</replaceable> set.
1259 This is an important feature of the retrieval
1260 system of &acro.z3950;, as it ensures the highest possible level of
1261 interoperability, as those access points of your database which are
1262 derived from the external set (say, bib-1) can be used even by clients
1263 who are unaware of the new set.
1265 </listitem></varlistentry>
1268 <replaceable>att-value att-name [local-value]</replaceable></term>
1272 repeatable directive introduces a new attribute to the set. The
1273 attribute value is stored in the index (unless a
1274 <replaceable>local-value</replaceable> is
1275 given, in which case this is stored). The name is used to refer to the
1276 attribute from the <replaceable>abstract syntax</replaceable>.
1278 </listitem></varlistentry>
1283 This is an excerpt from the GILS attribute set definition.
1284 Notice how the file describing the <emphasis>bib-1</emphasis>
1285 attribute set is referenced.
1292 reference GILS-attset
1295 att 2001 distributorName
1296 att 2002 indextermsControlled
1298 att 2004 accessConstraints
1299 att 2005 useConstraints
1306 <section id="grs-tag-files">
1307 <title>The Tag Set (.tag) Files</title>
1310 This file type defines the tagset of the profile, possibly by
1311 referencing other tag sets (most tag sets, for instance, will include
1312 tagsetG and tagsetM from the &acro.z3950; specification. The file may
1313 contain the following directives.
1320 <term>name <emphasis>symbolic-name</emphasis></term>
1323 (m) This provides a shorthand name or
1324 description for the tag set. Mostly useful for diagnostic purposes.
1326 </listitem></varlistentry>
1328 <term>reference <emphasis>OID-name</emphasis></term>
1331 (o) The reference name of the OID for the tag set.
1332 The reference names can be found in the <emphasis>util</emphasis>
1333 module of <emphasis>&yaz;</emphasis>.
1334 The directive is optional, since not all tag sets
1335 are registered outside of their schema.
1337 </listitem></varlistentry>
1339 <term>type <emphasis>integer</emphasis></term>
1342 (m) The type number of the tagset within the schema
1343 profile (note: this specification really should belong to the .abs
1344 file. This will be fixed in a future release).
1346 </listitem></varlistentry>
1348 <term>include <emphasis>filename</emphasis></term>
1351 (o,r) This directive is used
1352 to include the definitions of other tag sets into the current one.
1354 </listitem></varlistentry>
1356 <term>tag <emphasis>number names type</emphasis></term>
1359 (o,r) Introduces a new tag to the set.
1360 The <emphasis>number</emphasis> is the tag number as used
1361 in the protocol (there is currently no mechanism for
1362 specifying string tags at this point, but this would be quick
1364 The <emphasis>names</emphasis> parameter is a list of names
1365 by which the tag should be recognized in the input file format.
1366 The names should be separated by slashes (/).
1367 The <emphasis>type</emphasis> is the recommended data type of
1369 It should be one of the following:
1435 </listitem></varlistentry>
1440 The following is an excerpt from the TagsetG definition file.
1451 tag 3 publicationPlace string
1452 tag 4 publicationDate string
1453 tag 5 documentId string
1454 tag 6 abstract string
1456 tag 8 date generalizedtime
1457 tag 9 bodyOfDisplay string
1458 tag 10 organization string
1464 <section id="grs-var-files">
1465 <title>The Variant Set (.var) Files</title>
1468 The variant set file is a straightforward representation of the
1469 variant set definitions associated with the protocol. At present, only
1470 the <emphasis>Variant-1</emphasis> set is known.
1474 These are the directives allowed in the file.
1481 <term>name <emphasis>symbolic-name</emphasis></term>
1484 (m) This provides a shorthand name or
1485 description for the variant set. Mostly useful for diagnostic purposes.
1487 </listitem></varlistentry>
1489 <term>reference <emphasis>OID-name</emphasis></term>
1492 (o) The reference name of the OID for
1493 the variant set, if one is required. The reference names can be found
1494 in the <emphasis>util</emphasis> module of <emphasis>&yaz;</emphasis>.
1496 </listitem></varlistentry>
1498 <term>class <emphasis>integer class-name</emphasis></term>
1501 (m,r) Introduces a new
1502 class to the variant set.
1504 </listitem></varlistentry>
1506 <term>type <emphasis>integer type-name datatype</emphasis></term>
1510 new type to the current class (the one introduced by the most recent
1511 <emphasis>class</emphasis> directive).
1512 The type names belong to the same name space as the one used
1513 in the tag set definition file.
1515 </listitem></varlistentry>
1520 The following is an excerpt from the file describing the variant set
1521 <emphasis>Variant-1</emphasis>.
1532 type 1 variantId octetstring
1537 type 2 z39.50 string
1545 <section id="grs-est-files">
1546 <title>The Element Set (.est) Files</title>
1549 The element set specification files describe a selection of a subset
1550 of the elements of a database record. The element selection mechanism
1551 is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
1552 syntax of the &acro.z3950; specification.
1553 In fact, the internal representation of an element set
1554 specification is identical to the <emphasis>Espec-1</emphasis> structure,
1555 and we'll refer you to the description of that structure for most of
1556 the detailed semantics of the directives below.
1561 Not all of the Espec-1 functionality has been implemented yet.
1562 The fields that are mentioned below all work as expected, unless
1568 The directives available in the element set file are as follows:
1574 <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
1577 (o) If variants are used in
1578 the following, this should provide the name of the variantset used
1579 (it's not currently possible to specify a different set in the
1580 individual variant request). In almost all cases (certainly all
1581 profiles known to us), the name
1582 <literal>Variant-1</literal> should be given here.
1584 </listitem></varlistentry>
1586 <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
1590 provides a default variant request for
1591 use when the individual element requests (see below) do not contain a
1592 variant request. Variant requests consist of a blank-separated list of
1593 variant components. A variant component is a comma-separated,
1594 parenthesized triple of variant class, type, and value (the two former
1595 values being represented as integers). The value can currently only be
1596 entered as a string (this will change to depend on the definition of
1597 the variant in question). The special value (@) is interpreted as a
1598 null value, however.
1600 </listitem></varlistentry>
1603 <emphasis>path ['variant' variant-request]</emphasis></term>
1606 (o,r) This corresponds to a simple element request
1607 in <emphasis>Espec-1</emphasis>.
1608 The path consists of a sequence of tag-selectors, where each of
1609 these can consist of either:
1616 A simple tag, consisting of a comma-separated type-value pair in
1617 parenthesis, possibly followed by a colon (:) followed by an
1618 occurrences-specification (see below). The tag-value can be a number
1619 or a string. If the first character is an apostrophe ('), this
1620 forces the value to be interpreted as a string, even if it
1621 appears to be numerical.
1627 A WildThing, represented as a question mark (?), possibly
1628 followed by a colon (:) followed by an occurrences
1629 specification (see below).
1635 A WildPath, represented as an asterisk (*). Note that the last
1636 element of the path should not be a wildPath (wildpaths don't
1637 work in this version).
1646 The occurrences-specification can be either the string
1647 <literal>all</literal>, the string <literal>last</literal>, or
1648 an explicit value-range. The value-range is represented as
1649 an integer (the starting point), possibly followed by a
1650 plus (+) and a second integer (the number of elements, default
1655 The variant-request has the same syntax as the defaultVariantRequest
1656 above. Note that it may sometimes be useful to give an empty variant
1657 request, simply to disable the default for a specific set of fields
1658 (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
1659 but it works in this implementation).
1661 </listitem></varlistentry>
1666 The following is an example of an element specification belonging to
1673 simpleelement (1,10)
1674 simpleelement (1,12)
1676 simpleelement (1,14)
1678 simpleelement (4,52)
1685 <section id="schema-mapping">
1686 <title>The Schema Mapping (.map) Files</title>
1689 Sometimes, the client might want to receive a database record in
1690 a schema that differs from the native schema of the record. For
1691 instance, a client might only know how to process WAIS records, while
1692 the database record is represented in a more specific schema, such as
1693 GILS. In this module, a mapping of data to one of the &acro.marc; formats is
1694 also thought of as a schema mapping (mapping the elements of the
1695 record into fields consistent with the given &acro.marc; specification, prior
1696 to actually converting the data to the ISO2709). This use of the
1697 object identifier for &acro.usmarc; as a schema identifier represents an
1698 overloading of the OID which might not be entirely proper. However,
1699 it represents the dual role of schema and record syntax which
1700 is assumed by the &acro.marc; family in &acro.z3950;.
1704 <emphasis>NOTE: FIXME! The schema-mapping functions are so far limited to a
1705 straightforward mapping of elements. This should be extended with
1706 mechanisms for conversions of the element contents, and conditional
1707 mappings of elements based on the record contents.</emphasis>
1711 These are the directives of the schema mapping file format:
1718 <term>targetName <emphasis>name</emphasis></term>
1721 (m) A symbolic name for the target schema
1722 of the table. Useful mostly for diagnostic purposes.
1724 </listitem></varlistentry>
1726 <term>targetRef <emphasis>OID-name</emphasis></term>
1729 (m) An OID name for the target schema.
1730 This is used, for instance, by a server receiving a request to present
1731 a record in a different schema from the native one.
1732 The name, again, is found in the <emphasis>oid</emphasis>
1733 module of <emphasis>&yaz;</emphasis>.
1735 </listitem></varlistentry>
1737 <term>map <emphasis>element-name target-path</emphasis></term>
1741 an element mapping rule to the table.
1743 </listitem></varlistentry>
1749 <section id="grs-mar-files">
1750 <title>The &acro.marc; (ISO2709) Representation (.mar) Files</title>
1753 This file provides rules for representing a record in the ISO2709
1754 format. The rules pertain mostly to the values of the constant-length
1755 header of the record.
1759 NOTE: FIXME! This will be described better. We're in the process of
1760 re-evaluating and most likely changing the way that &acro.marc; records are
1761 handled by the system.</emphasis>
1767 <section id="grs-exchange-formats">
1768 <title>&acro.grs1; Exchange Formats</title>
1771 Converting records from the internal structure to an exchange format
1772 is largely an automatic process. Currently, the following exchange
1773 formats are supported:
1780 &acro.grs1;. The internal representation is based on &acro.grs1;/&acro.xml;, so the
1781 conversion here is straightforward. The system will create
1782 applied variant and supported variant lists as required, if a record
1783 contains variant information.
1789 &acro.xml;. The internal representation is based on &acro.grs1;/&acro.xml; so
1790 the mapping is trivial. Note that &acro.xml; schemas, preprocessing
1791 instructions and comments are not part of the internal representation
1792 and therefore will never be part of a generated &acro.xml; record.
1793 Future versions of the &zebra; will support that.
1799 &acro.sutrs;. Again, the mapping is fairly straightforward. Indentation
1800 is used to show the hierarchical structure of the record. All
1801 "&acro.grs1;" type records support both the &acro.grs1; and &acro.sutrs;
1803 <!-- FIXME - What is &acro.sutrs; - should be expanded here -->
1809 ISO2709-based formats (&acro.usmarc;, etc.). Only records with a
1810 two-level structure (corresponding to fields and subfields) can be
1811 directly mapped to ISO2709. For records with a different structuring
1812 (e.g., GILS), the representation in a structure like &acro.usmarc; involves a
1813 schema-mapping (see <xref linkend="schema-mapping"/>), to an
1814 "implied" &acro.usmarc; schema (implied,
1815 because there is no formal schema which specifies the use of the
1816 &acro.usmarc; fields outside of ISO2709). The resultant, two-level record is
1817 then mapped directly from the internal representation to ISO2709. See
1818 the GILS schema definition files for a detailed example of this
1825 Explain. This representation is only available for records
1826 belonging to the Explain schema.
1832 Summary. This ASN-1 based structure is only available for records
1833 belonging to the Summary schema - or schema which provide a mapping
1834 to this schema (see the description of the schema mapping facility
1839 <!-- FIXME - Is this used anywhere ? -H -->
1842 SOIF. Support for this syntax is experimental, and is currently
1843 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
1844 abstract syntaxes can be mapped to the SOIF format, although nested
1845 elements are represented by concatenation of the tag names at each
1854 <section id="grs-extended-marc-indexing">
1855 <title>Extended indexing of &acro.marc; records</title>
1857 <para>Extended indexing of &acro.marc; records will help you if you need index a
1858 combination of subfields, or index only a part of the whole field,
1859 or use during indexing process embedded fields of &acro.marc; record.
1862 <para>Extended indexing of &acro.marc; records additionally allows:
1866 <para>to index data in LEADER of &acro.marc; record</para>
1870 <para>to index data in control fields (with fixed length)</para>
1874 <para>to use during indexing the values of indicators</para>
1878 <para>to index linked fields for UNI&acro.marc; based formats</para>
1884 <note><para>In compare with simple indexing process the extended indexing
1885 may increase (about 2-3 times) the time of indexing process for &acro.marc;
1886 records.</para></note>
1888 <section id="formula">
1889 <title>The index-formula</title>
1891 <para>At the beginning, we have to define the term
1892 <emphasis>index-formula</emphasis> for &acro.marc; records. This term helps
1893 to understand the notation of extended indexing of &acro.marc; records by &zebra;.
1894 Our definition is based on the document
1895 <ulink url="http://www.rba.ru/rusmarc/soft/Z39-50.htm">"The table
1896 of conformity for &acro.z3950; use attributes and R&acro.usmarc; fields"</ulink>.
1897 The document is available only in Russian language.</para>
1900 The <emphasis>index-formula</emphasis> is the combination of
1901 subfields presented in such way:
1905 71-00$a, $g, $h ($c){.$b ($c)} , (1)
1909 We know that &zebra; supports a &acro.bib1; attribute - right truncation.
1910 In this case, the <emphasis>index-formula</emphasis> (1) consists from
1911 forms, defined in the same way as (1)</para>
1920 <para>The original &acro.marc; record may be without some elements, which included in <emphasis>index-formula</emphasis>.
1924 <para>This notation includes such operands as:
1929 <listitem><para>It means whitespace character.</para></listitem>
1934 <listitem><para>The position may contain any value, defined by
1936 For example, <emphasis>index-formula</emphasis></para>
1942 <para>includes</para>
1956 <para>The repeatable elements are defined in figure-brackets {}.
1958 <emphasis>index-formula</emphasis></para>
1961 71-00$a, $g, $h ($c){.$b ($c)} , (3)
1964 <para>includes</para>
1967 71-00$a, $g, $h ($c). $b ($c)
1968 71-00$a, $g, $h ($c). $b ($c). $b ($c)
1969 71-00$a, $g, $h ($c). $b ($c). $b ($c). $b ($c)
1978 All another operands are the same as accepted in &acro.marc; world.
1984 <section id="notation">
1985 <title>Notation of <emphasis>index-formula</emphasis> for &zebra;</title>
1988 <para>Extended indexing overloads <literal>path</literal> of
1989 <literal>elm</literal> definition in abstract syntax file of &zebra;
1990 (<literal>.abs</literal> file). It means that names beginning with
1991 <literal>"mc-"</literal> are interpreted by &zebra; as
1992 <emphasis>index-formula</emphasis>. The database index is created and
1993 linked with <emphasis>access point</emphasis> (&acro.bib1; use attribute)
1994 according to this formula.</para>
1996 <para>For example, <emphasis>index-formula</emphasis></para>
1999 71-00$a, $g, $h ($c){.$b ($c)} , (4)
2002 <para>in <literal>.abs</literal> file looks like:</para>
2005 mc-71.00_$a,_$g,_$h_(_$c_){.$b_(_$c_)}
2009 <para>The notation of <emphasis>index-formula</emphasis> uses the operands:
2014 <listitem><para>It means whitespace character.</para></listitem>
2019 <listitem><para>The position may contain any value, defined by
2020 &acro.marc; format. For example,
2021 <emphasis>index-formula</emphasis></para>
2027 <para>matches <literal>mc-70._1_$a,_$g_</literal> and includes</para>
2039 <listitem><para>The repeatable elements are defined in
2040 figure-brackets {}. For example,
2041 <emphasis>index-formula</emphasis></para>
2044 71#00$a, $g, $h ($c) {.$b ($c)} , (6)
2048 <literal>mc-71.00_$a,_$g,_$h_(_$c_){.$b_(_$c_)}</literal> and
2052 71.00_$a,_$g,_$h_(_$c_).$b_(_$c_)
2053 71.00_$a,_$g,_$h_(_$c_).$b_(_$c_).$b_(_$c_)
2054 71.00_$a,_$g,_$h_(_$c_).$b_(_$c_).$b_(_$c_).$b_(_$c_)
2060 <term><...></term>
2061 <listitem><para>Embedded <emphasis>index-formula</emphasis> (for
2062 linked fields) is between <>. For example,
2063 <emphasis>index-formula</emphasis>
2067 4--#-$170-#1$a, $g ($c) , (7)
2071 <literal>mc-4.._._$1<70._1_$a,_$g_(_$c_)>_</literal> and
2075 463_._$1<70._1_$a,_$g_(_$c_)>_
2084 <para>All another operands are the same as accepted in &acro.marc; world.</para>
2087 <section id="grs-examples">
2088 <title>Examples</title>
2095 <para>indexing LEADER</para>
2097 <para>You need to use keyword "ldr" to index leader. For example,
2098 indexing data from 6th and 7th position of LEADER</para>
2101 elm mc-ldr[6] Record-type !
2102 elm mc-ldr[7] Bib-level !
2109 <para>indexing data from control fields</para>
2111 <para>indexing date (the time added to database)</para>
2114 elm mc-008[0-5] Date/time-added-to-db !
2117 <para>or for R&acro.usmarc; (this data included in 100th field)</para>
2120 elm mc-100___$a[0-7]_ Date/time-added-to-db !
2127 <para>using indicators while indexing</para>
2129 <para>For R&acro.usmarc; <emphasis>index-formula</emphasis>
2130 <literal>70-#1$a, $g</literal> matches</para>
2133 elm 70._1_$a,_$g_ Author !:w,!:p
2136 <para>When &zebra; finds a field according to
2137 <literal>"70."</literal> pattern it checks the indicators. In this
2138 case the value of first indicator doesn't mater, but the value of
2139 second one must be whitespace, in another case a field is not
2145 <para>indexing embedded (linked) fields for UNI&acro.marc; based
2148 <para>For R&acro.usmarc; <emphasis>index-formula</emphasis>
2149 <literal>4--#-$170-#1$a, $g ($c)</literal> matches</para>
2152 elm mc-4.._._$1<70._1_$a,_$g_(_$c_)>_ Author !:w,!:p
2155 <para>Data are extracted from record if the field matches to
2156 <literal>"4.._."</literal> pattern and data in linked field
2158 <emphasis>index-formula</emphasis>
2159 <literal>70._1_$a,_$g_(_$c_)</literal>.</para>
2173 <!-- Keep this comment at the end of the file
2178 sgml-minimize-attributes:nil
2179 sgml-always-quote-attributes:t
2182 sgml-parent-document: "zebra.xml"
2183 sgml-local-catalogs: nil
2184 sgml-namecase-general:t