1 <chapter id="record-model-grs">
2 <!-- $Id: recordmodel-grs.xml,v 1.1 2006-02-15 11:07:47 marc Exp $ -->
3 <title>GRS Record Model and Filter Modules</title>
7 The record model described in this chapter applies to the fundamental,
9 record type <literal>grs</literal>, introduced in
10 <xref linkend="componentmodulesgrs"/>.
14 <sect1 id="grs-record-filters">
15 <title>GRS Record Filters</title>
17 Many basic subtypes of the <emphasis>grs</emphasis> type are
27 This is the canonical input format
28 described <xref linkend="grs-canonical-format"/>. It is using
29 simple SGML-like syntax.
33 <literal>libidzebra1.4-mod-grs-sgml not packaged yet ??</literal>
39 <term>grs.marc<!--.<emphasis>abstract syntax</emphasis>--></term>
42 This allows Zebra to read
43 records in the ISO2709 (MARC) encoding standard.
44 <!-- In this case, the
45 last parameter <emphasis>abstract syntax</emphasis> names the
46 <literal>.abs</literal> file (see below)
47 which describes the specific MARC structure of the input record as
48 well as the indexing rules. -->
51 The loadable <literal>grs.marc</literal> filter module
52 is packaged in the GNU/Debian package
53 <literal>libidzebra1.4-mod-grs-marc</literal>
58 <term>grs.marcxml<!--.<emphasis>abstract syntax</emphasis>--></term>
61 This allows Zebra to read
62 records in the ISO2709??? (MARCXML) encoding standard.
65 The loadable <literal>grs.marcxml</literal> filter module
66 is also contained in the GNU/Debian package
67 <literal>libidzebra1.4-mod-grs-marc</literal>
72 <term>grs.danbib</term>
75 The <literal>grs.danbib</literal> filter parses DanBib
76 records, a danish MARC record variant called DANMARC.
77 DanBib is the Danish Union Catalogue hosted by the
78 Danish Bibliographic Centre (DBC).
80 <para>The loadable <literal>grs.danbib</literal> filter module
81 is packages in the GNU/Debian package
82 <literal>libidzebra1.4-mod-grs-danbib</literal>.
90 This filter reads XML records and uses <ulink url="http://expat.sourceforge.net/">Expat</ulink> to
91 parse them and convert them into IDZebra's internal
92 <literal>grs</literal> record model.
93 Only one record per file
94 is supported. The filter is only available if Zebra/YAZ
95 is compiled with EXPAT support.
98 The loadable <literal>grs.xml</literal> filter module
99 is packagged in the GNU/Debian package
100 <literal>libidzebra1.4-mod-grs-xml</literal>
105 <term>grs.regx<!--.<emphasis>filter</emphasis>--></term>
108 This enables a user-supplied Regular Expressions input
110 <xref linkend="grs-regx-tcl"/>.
113 The loadable <literal>grs.regx</literal> filter module
114 is packaged in the GNU/Debian package
115 <literal>libidzebra1.4-mod-grs-regx</literal>
120 <term>grs.tcl<!--.<emphasis>filter</emphasis>--></term>
123 Similar to grs.regx but using Tcl for rules, described in
124 <xref linkend="grs-regx-tcl"/>.
127 The loadable <literal>grs.tcl</literal> filter module
128 is also packaged in the GNU/Debian package
129 <literal>libidzebra1.4-mod-grs-regx</literal>
137 <sect2 id="grs-canonical-format">
138 <title>GRS Canonical Input Format</title>
141 Although input data can take any form, it is sometimes useful to
142 describe the record processing capabilities of the system in terms of
143 a single, canonical input format that gives access to the full
144 spectrum of structure and flexibility in the system. In Zebra, this
145 canonical format is an "SGML-like" syntax.
149 To use the canonical format specify <literal>grs.sgml</literal> as
154 Consider a record describing an information resource (such a record is
155 sometimes known as a <emphasis>locator record</emphasis>).
156 It might contain a field describing the distributor of the
157 information resource, which might in turn be partitioned into
158 various fields providing details about the distributor, like this:
164 <Distributor>
165 <Name> USGS/WRD </Name>
166 <Organization> USGS/WRD </Organization>
167 <Street-Address>
168 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
169 </Street-Address>
170 <City> ALBUQUERQUE </City>
171 <State> NM </State>
172 <Zip-Code> 87102 </Zip-Code>
173 <Country> USA </Country>
174 <Telephone> (505) 766-5560 </Telephone>
175 </Distributor>
180 <!-- There is no indentation in the example above! -H
183 The indentation used above is used to illustrate how Zebra
184 interprets the mark-up. The indentation, in itself, has no
185 significance to the parser for the canonical input format, which
186 discards superfluous whitespace.
192 The keywords surrounded by <...> are
193 <emphasis>tags</emphasis>, while the sections of text
194 in between are the <emphasis>data elements</emphasis>.
195 A data element is characterized by its location in the tree
196 that is made up by the nested elements.
197 Each element is terminated by a closing tag - beginning
198 with <literal><</literal>/, and containing the same symbolic
199 tag-name as the corresponding opening tag.
200 The general closing tag - <literal></></literal> -
201 terminates the element started by the last opening tag. The
202 structuring of elements is significant.
203 The element <emphasis>Telephone</emphasis>,
204 for instance, may be indexed and presented to the client differently,
205 depending on whether it appears inside the
206 <emphasis>Distributor</emphasis> element, or some other,
207 structured data element such a <emphasis>Supplier</emphasis> element.
211 <title>Record Root</title>
214 The first tag in a record describes the root node of the tree that
215 makes up the total record. In the canonical input format, the root tag
216 should contain the name of the schema that lends context to the
217 elements of the record
218 (see <xref linkend="grs-internal-representation"/>).
219 The following is a GILS record that
220 contains only a single element (strictly speaking, that makes it an
221 illegal GILS record, since the GILS profile includes several mandatory
222 elements - Zebra does not validate the contents of a record against
223 the Z39.50 profile, however - it merely attempts to match up elements
224 of a local representation with the given schema):
231 <title>Zen and the Art of Motorcycle Maintenance</title>
239 <sect3><!-- ### we shouldn't make such a big deal about this -->
240 <title>Variants</title>
243 Zebra allows you to provide individual data elements in a number of
244 <emphasis>variant forms</emphasis>. Examples of variant forms are
245 textual data elements which might appear in different languages, and
246 images which may appear in different formats or layouts.
247 The variant system in Zebra is essentially a representation of
248 the variant mechanism of Z39.50-1995.
252 The following is an example of a title element which occurs in two
260 <var lang lang "eng">
261 Zen and the Art of Motorcycle Maintenance</>
262 <var lang lang "dan">
263 Zen og Kunsten at Vedligeholde en Motorcykel</>
270 The syntax of the <emphasis>variant element</emphasis> is
271 <literal><var class type value></literal>.
272 The available values for the <emphasis>class</emphasis> and
273 <emphasis>type</emphasis> fields are given by the variant set
274 that is associated with the current schema
275 (see <xref linkend="variant-set"/>).
279 Variant elements are terminated by the general end-tag </>, by
280 the variant end-tag </var>, by the appearance of another variant
281 tag with the same <emphasis>class</emphasis> and
282 <emphasis>value</emphasis> settings, or by the
283 appearance of another, normal tag. In other words, the end-tags for
284 the variants used in the example above could have been omitted.
288 Variant elements can be nested. The element
295 <var lang lang "eng"><var body iana "text/plain">
296 Zen and the Art of Motorcycle Maintenance
303 Associates two variant components to the variant list for the title
308 Given the nesting rules described above, we could write
315 <var body iana "text/plain>
316 <var lang lang "eng">
317 Zen and the Art of Motorcycle Maintenance
318 <var lang lang "dan">
319 Zen og Kunsten at Vedligeholde en Motorcykel
326 The title element above comes in two variants. Both have the IANA body
327 type "text/plain", but one is in English, and the other in
328 Danish. The client, using the element selection mechanism of Z39.50,
329 can retrieve information about the available variant forms of data
330 elements, or it can select specific variants based on the requirements
338 <sect2 id="grs-regx-tcl">
339 <title>GRS REGX And TCL Input Filters</title>
342 In order to handle general input formats, Zebra allows the
343 operator to define filters which read individual records in their
344 native format and produce an internal representation that the system
349 Input filters are ASCII files, generally with the suffix
350 <literal>.flt</literal>.
351 The system looks for the files in the directories given in the
352 <emphasis>profilePath</emphasis> setting in the
353 <literal>zebra.cfg</literal> files.
354 The record type for the filter is
355 <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
356 (fundamental type <literal>grs</literal>, file read
357 type <literal>regx</literal>, argument
358 <emphasis>filter-filename</emphasis>).
362 Generally, an input filter consists of a sequence of rules, where each
363 rule consists of a sequence of expressions, followed by an action. The
364 expressions are evaluated against the contents of the input record,
365 and the actions normally contribute to the generation of an internal
366 representation of the record.
370 An expression can be either of the following:
380 The action associated with this expression is evaluated
381 exactly once in the lifetime of the application, before any records
382 are read. It can be used in conjunction with an action that
383 initializes tables or other resources that are used in the processing
392 Matches the beginning of the record. It can be used to
393 initialize variables, etc. Typically, the
394 <emphasis>BEGIN</emphasis> rule is also used
395 to establish the root node of the record.
403 Matches the end of the record - when all of the contents
404 of the record has been processed.
409 <term>/pattern/</term>
412 Matches a string of characters from the input record.
420 This keyword may only be used between two patterns.
421 It matches everything between (not including) those patterns.
429 The expression associated with this pattern is evaluated
430 once, before the application terminates. It can be used to release
431 system resources - typically ones allocated in the
432 <emphasis>INIT</emphasis> step.
440 An action is surrounded by curly braces ({...}), and
441 consists of a sequence of statements. Statements may be separated
442 by newlines or semicolons (;).
443 Within actions, the strings that matched the expressions
444 immediately preceding the action can be referred to as
449 The available statements are:
456 <term>begin <replaceable>type [parameter ... ]</replaceable></term>
460 data element. The <replaceable>type</replaceable> is one of
468 Begin a new record. The following parameter should be the
469 name of the schema that describes the structure of the record, eg.
470 <literal>gils</literal> or <literal>wais</literal> (see below).
471 The <literal>begin record</literal> call should precede
472 any other use of the <replaceable>begin</replaceable> statement.
480 Begin a new tagged element. The parameter is the
481 name of the tag. If the tag is not matched anywhere in the tagsets
482 referenced by the current schema, it is treated as a local string
491 Begin a new node in a variant tree. The parameters are
492 <replaceable>class type value</replaceable>.
501 <term>data <replaceable>parameter</replaceable></term>
504 Create a data element. The concatenated arguments make
505 up the value of the data element.
506 The option <literal>-text</literal> signals that
507 the layout (whitespace) of the data should be retained for
509 The option <literal>-element</literal>
510 <replaceable>tag</replaceable> wraps the data up in
511 the <replaceable>tag</replaceable>.
512 The use of the <literal>-element</literal> option is equivalent to
513 preceding the command with a <replaceable>begin
514 element</replaceable> command, and following
515 it with the <replaceable>end</replaceable> command.
520 <term>end <replaceable>[type]</replaceable></term>
523 Close a tagged element. If no parameter is given,
524 the last element on the stack is terminated.
525 The first parameter, if any, is a type name, similar
526 to the <replaceable>begin</replaceable> statement.
527 For the <replaceable>element</replaceable> type, a tag
528 name can be provided to terminate a specific tag.
534 <term>unread <replaceable>no</replaceable></term>
537 Move the input pointer to the offset of first character that
538 match rule given by <replaceable>no</replaceable>.
539 The first rule from left-to-right is numbered zero,
540 the second rule is named 1 and so on.
549 The following input filter reads a Usenet news file, producing a
550 record in the WAIS schema. Note that the body of a news posting is
551 separated from the list of headers by a blank line (or rather a
552 sequence of two newline characters.
558 BEGIN { begin record wais }
560 /^From:/ BODY /$/ { data -element name $1 }
561 /^Subject:/ BODY /$/ { data -element title $1 }
562 /^Date:/ BODY /$/ { data -element lastModified $1 }
564 begin element bodyOfDisplay
565 begin variant body iana "text/plain"
574 If Zebra is compiled with support for Tcl enabled, the statements
575 described above are supplemented with a complete
576 scripting environment, including control structures (conditional
577 expressions and loop constructs), and powerful string manipulation
578 mechanisms for modifying the elements of a record.
585 <sect1 id="grs-internal-representation">
586 <title>GRS Internal Record Representation</title>
589 When records are manipulated by the system, they're represented in a
590 tree-structure, with data elements at the leaf nodes, and tags or
591 variant components at the non-leaf nodes. The root-node identifies the
592 schema that lends context to the tagging and structuring of the
593 record. Imagine a simple record, consisting of a 'title' element and
601 TITLE "Zen and the Art of Motorcycle Maintenance"
602 AUTHOR "Robert Pirsig"
608 A slightly more complex record would have the author element consist
609 of two elements, a surname and a first name:
616 TITLE "Zen and the Art of Motorcycle Maintenance"
625 The root of the record will refer to the record schema that describes
626 the structuring of this particular record. The schema defines the
627 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
628 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
629 addition, the schema establishes element set names that are used by
630 the client to request a subset of the elements of a given record. The
631 schema may also establish rules for converting the record to a
632 different schema, by stating, for each element, a mapping to a
637 <title>Tagged Elements</title>
640 A data element is characterized by its tag, and its position in the
641 structure of the record. For instance, while the tag "telephone
642 number" may be used different places in a record, we may need to
643 distinguish between these occurrences, both for searching and
644 presentation purposes. For instance, while the phone numbers for the
645 "customer" and the "service provider" are both
646 representatives for the same type of resource (a telephone number), it
647 is essential that they be kept separate. The record schema provides
648 the structure of the record, and names each data element (defined by
649 the sequence of tags - the tag path - by which the element can be
650 reached from the root of the record).
656 <title>Variants</title>
659 The children of a tag node may be either more tag nodes, a data node
660 (possibly accompanied by tag nodes),
661 or a tree of variant nodes. The children of variant nodes are either
662 more variant nodes or a data node (possibly accompanied by more
663 variant nodes). Each leaf node, which is normally a
664 data node, corresponds to a <emphasis>variant form</emphasis> of the
665 tagged element identified by the tag which parents the variant tree.
666 The following title element occurs in two different languages:
672 VARIANT LANG=ENG "War and Peace"
674 VARIANT LANG=DAN "Krig og Fred"
680 Which of the two elements are transmitted to the client by the server
681 depends on the specifications provided by the client, if any.
685 In practice, each variant node is associated with a triple of class,
686 type, value, corresponding to the variant mechanism of Z39.50.
692 <title>Data Elements</title>
695 Data nodes have no children (they are always leaf nodes in the record
700 FIXME! Documentation needs extension here about types of nodes - numerical,
701 textual, etc., plus the various types of inclusion notes.
709 <sect1 id="record-model-grs-conf">
710 <title>GRS Record Model Configuration</title>
713 The following sections describe the configuration files that govern
714 the internal management of <literal>grs</literal> records.
715 The system searches for the files
716 in the directories specified by the <emphasis>profilePath</emphasis>
717 setting in the <literal>zebra.cfg</literal> file.
721 <title>The Abstract Syntax</title>
724 The abstract syntax definition (also known as an Abstract Record
725 Structure, or ARS) is the focal point of the
726 record schema description. For a given schema, the ABS file may state any
727 or all of the following:
731 FIXME - Need a diagram here, or a simple explanation how it all hangs together -H
740 The object identifier of the Z39.50 schema associated
741 with the ARS, so that it can be referred to by the client.
747 The attribute set (which can possibly be a compound of multiple
748 sets) which applies in the profile. This is used when indexing and
749 searching the records belonging to the given profile.
755 The tag set (again, this can consist of several different sets).
756 This is used when reading the records from a file, to recognize the
757 different tags, and when transmitting the record to the client -
758 mapping the tags to their numerical representation, if they are
765 The variant set which is used in the profile. This provides a
766 vocabulary for specifying the <emphasis>forms</emphasis> of
767 data that appear inside the records.
773 Element set names, which are a shorthand way for the client to
774 ask for a subset of the data elements contained in a record. Element
775 set names, in the retrieval module, are mapped to <emphasis>element
776 specifications</emphasis>, which contain information equivalent to the
777 <emphasis>Espec-1</emphasis> syntax of Z39.50.
783 Map tables, which may specify mappings to
784 <emphasis>other</emphasis> database profiles, if desired.
790 Possibly, a set of rules describing the mapping of elements to a
798 A list of element descriptions (this is the actual ARS of the
799 schema, in Z39.50 terms), which lists the ways in which the various
800 tags can be used and organized hierarchically.
809 Several of the entries above simply refer to other files, which
810 describe the given objects.
816 <title>The Configuration Files</title>
819 This section describes the syntax and use of the various tables which
820 are used by the retrieval module.
824 The number of different file types may appear daunting at first, but
825 each type corresponds fairly clearly to a single aspect of the Z39.50
826 retrieval facilities. Further, the average database administrator,
827 who is simply reusing an existing profile for which tables already
828 exist, shouldn't have to worry too much about the contents of these tables.
832 Generally, the files are simple ASCII files, which can be maintained
833 using any text editor. Blank lines, and lines beginning with a (#) are
834 ignored. Any characters on a line followed by a (#) are also ignored.
835 All other lines contain <emphasis>directives</emphasis>, which provide
836 some setting or value to the system.
837 Generally, settings are characterized by a single
838 keyword, identifying the setting, followed by a number of parameters.
839 Some settings are repeatable (r), while others may occur only once in a
840 file. Some settings are optional (o), while others again are
846 <sect2 id="abs-file">
847 <title>The Abstract Syntax (.abs) Files</title>
850 The name of this file type is slightly misleading in Z39.50 terms,
851 since, apart from the actual abstract syntax of the profile, it also
852 includes most of the other definitions that go into a database
857 When a record in the canonical, SGML-like format is read from a file
858 or from the database, the first tag of the file should reference the
859 profile that governs the layout of the record. If the first tag of the
860 record is, say, <literal><gils></literal>, the system will look
861 for the profile definition in the file <literal>gils.abs</literal>.
862 Profile definitions are cached, so they only have to be read once
863 during the lifespan of the current process.
867 When writing your own input filters, the
868 <emphasis>record-begin</emphasis> command
869 introduces the profile, and should always be called first thing when
870 introducing a new record.
874 The file may contain the following directives:
881 <term>name <replaceable>symbolic-name</replaceable></term>
884 (m) This provides a shorthand name or
885 description for the profile. Mostly useful for diagnostic purposes.
890 <term>reference <replaceable>OID-name</replaceable></term>
893 (m) The reference name of the OID for the profile.
894 The reference names can be found in the <emphasis>util</emphasis>
900 <term>attset <replaceable>filename</replaceable></term>
903 (m) The attribute set that is used for
904 indexing and searching records belonging to this profile.
909 <term>tagset <replaceable>filename</replaceable></term>
912 (o) The tag set (if any) that describe
913 that fields of the records.
918 <term>varset <replaceable>filename</replaceable></term>
921 (o) The variant set used in the profile.
926 <term>maptab <replaceable>filename</replaceable></term>
929 (o,r) This points to a
930 conversion table that might be used if the client asks for the record
931 in a different schema from the native one.
936 <term>marc <replaceable>filename</replaceable></term>
939 (o) Points to a file containing parameters
940 for representing the record contents in the ISO2709 syntax.
941 Read the description of the MARC representation facility below.
946 <term>esetname <replaceable>name filename</replaceable></term>
950 given element set name with an element selection file. If an (@) is
951 given in place of the filename, this corresponds to a null mapping for
952 the given element set name.
957 <term>any <replaceable>tags</replaceable></term>
960 (o) This directive specifies a list of attributes
961 which should be appended to the attribute list given for each
962 element. The effect is to make every single element in the abstract
963 syntax searchable by way of the given attributes. This directive
964 provides an efficient way of supporting free-text searching across all
965 elements. However, it does increase the size of the index
966 significantly. The attributes can be qualified with a structure, as in
967 the <replaceable>elm</replaceable> directive below.
972 <term>elm <replaceable>path name attributes</replaceable></term>
975 (o,r) Adds an element to the abstract record syntax of the schema.
976 The <replaceable>path</replaceable> follows the
977 syntax which is suggested by the Z39.50 document - that is, a sequence
978 of tags separated by slashes (/). Each tag is given as a
979 comma-separated pair of tag type and -value surrounded by parenthesis.
980 The <replaceable>name</replaceable> is the name of the element, and
981 the <replaceable>attributes</replaceable>
982 specifies which attributes to use when indexing the element in a
983 comma-separated list.
984 A ! in place of the attribute name is equivalent to
985 specifying an attribute name identical to the element name.
986 A - in place of the attribute name
987 specifies that no indexing is to take place for the given element.
988 The attributes can be qualified with <replaceable>field
989 types</replaceable> to specify which
990 character set should govern the indexing procedure for that field.
991 The same data element may be indexed into several different
992 fields, using different character set definitions.
993 See the <xref linkend="field-structure-and-character-sets"/>.
994 The default field type is <literal>w</literal> for
995 <emphasis>word</emphasis>.
1001 <term>xelm <replaceable>xpath attributes</replaceable></term>
1004 Specifies indexing for record nodes given by
1005 <replaceable>xpath</replaceable>. Unlike directive
1006 elm, this directive allows you to index attribute
1007 contents. The <replaceable>xpath</replaceable> uses
1008 a syntax similar to XPath. The <replaceable>attributes</replaceable>
1009 have same syntax and meaning as directive elm, except that operator
1010 ! refers to the nodes selected by <replaceable>xpath</replaceable>.
1012 xelm / !:w default index
1013 xelm // !:w additional index
1014 xelm /gils/title/@att myatt:w index attribute @att in myatt
1015 xelm title/@att myatt:w same meaning.
1021 <term>melm <replaceable>field$subfield attributes</replaceable></term>
1024 This directive is specifically for MARC-formatted records,
1025 ingested either in the form of MARCXML documents, or in the
1026 ISO2709/Z39.2 format using the grs.marcxml input filter. You can
1027 specify indexing rules for any subfield, or you can leave off the
1028 <replaceable>$subfield</replaceable> part and specify default rules
1029 for all subfields of the given field (note: default rules should come
1030 after any subfield-specific rules in the configuration file). The
1031 <replaceable>attributes</replaceable> have the same syntax and meaning
1032 as for the 'elm' directive above.
1037 <term>encoding <replaceable>encodingname</replaceable></term>
1040 This directive specifies character encoding for external records.
1041 For records such as XML that specifies encoding within the
1042 file via a header this directive is ignored.
1043 If neither this directive is given, nor an encoding is set
1044 within external records, ISO-8859-1 encoding is assumed.
1049 <term>xpath <literal>enable</literal>/<literal>disable</literal></term>
1052 If this directive is followed by <literal>enable</literal>,
1053 then extra indexing is performed to allow for XPath-like queries.
1054 If this directive is not specified - equivalent to
1055 <literal>disable</literal> - no extra XPath-indexing is performed.
1062 <term>systag <replaceable>systemtag</replaceable> <replaceable>element</replaceable></term>
1065 This directive maps system information to an element during
1066 retrieval. This information is dynamically created. The
1067 following system tags are defined
1073 Size of record in bytes. By default this
1074 is mapped to element <literal>size</literal>.
1083 Score/rank of record. By default this
1084 is mapped to element <literal>rank</literal>.
1085 If no score was calculated for the record (non-ranked
1086 searched) search this directive is ignored.
1095 Zebra's system number (record ID) for the
1096 record. By default this is mapped to element
1097 <literal>localControlNumber</literal>.
1102 If you do not want a particular system tag to be applied,
1103 then set the resulting element to something undefined in the
1104 abs file (such as <literal>none</literal>).
1110 <!-- Mike's version -->
1114 <replaceable>systemTag</replaceable>
1115 <replaceable>actualTag</replaceable>
1119 Specifies what information, if any, Zebra should
1120 automatically include in retrieval records for the
1121 ``system fields'' that it supports.
1122 <replaceable>systemTag</replaceable> may
1123 be any of the following:
1126 <term><literal>rank</literal></term>
1128 An integer indicating the relevance-ranking score
1129 assigned to the record.
1133 <term><literal>sysno</literal></term>
1135 An automatically generated identifier for the record,
1136 unique within this database. It is represented by the
1137 <literal><localControlNumber></literal> element in
1138 XML and the <literal>(1,14)</literal> tag in GRS-1.
1142 <term><literal>size</literal></term>
1144 The size, in bytes, of the retrieved record.
1150 The <replaceable>actualTag</replaceable> parameter may be
1151 <literal>none</literal> to indicate that the named element
1152 should be omitted from retrieval records.
1161 The mechanism for controlling indexing is not adequate for
1162 complex databases, and will probably be moved into a separate
1163 configuration table eventually.
1168 The following is an excerpt from the abstract syntax file for the GILS
1176 reference GILS-schema
1181 maptab gils-usmarc.map
1185 esetname VARIANT gils-variant.est # for WAIS-compliance
1186 esetname B gils-b.est
1187 esetname G gils-g.est
1192 elm (1,14) localControlNumber Local-number
1193 elm (1,16) dateOfLastModification Date/time-last-modified
1194 elm (2,1) title w:!,p:!
1195 elm (4,1) controlIdentifier Identifier-standard
1196 elm (2,6) abstract Abstract
1197 elm (4,51) purpose !
1198 elm (4,52) originator -
1199 elm (4,53) accessConstraints !
1200 elm (4,54) useConstraints !
1201 elm (4,70) availability -
1202 elm (4,70)/(4,90) distributor -
1203 elm (4,70)/(4,90)/(2,7) distributorName !
1204 elm (4,70)/(4,90)/(2,10) distributorOrganization !
1205 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1206 elm (4,70)/(4,90)/(4,3) distributorCity !
1213 <sect2 id="attset-files">
1214 <title>The Attribute Set (.att) Files</title>
1217 This file type describes the <replaceable>Use</replaceable> elements of
1219 It contains the following directives.
1225 <term>name <replaceable>symbolic-name</replaceable></term>
1228 (m) This provides a shorthand name or
1229 description for the attribute set.
1230 Mostly useful for diagnostic purposes.
1232 </listitem></varlistentry>
1234 <term>reference <replaceable>OID-name</replaceable></term>
1237 (m) The reference name of the OID for
1239 The reference names can be found in the <replaceable>util</replaceable>
1240 module of <replaceable>YAZ</replaceable>.
1242 </listitem></varlistentry>
1244 <term>include <replaceable>filename</replaceable></term>
1247 (o,r) This directive is used to
1248 include another attribute set as a part of the current one. This is
1249 used when a new attribute set is defined as an extension to another
1250 set. For instance, many new attribute sets are defined as extensions
1251 to the <replaceable>bib-1</replaceable> set.
1252 This is an important feature of the retrieval
1253 system of Z39.50, as it ensures the highest possible level of
1254 interoperability, as those access points of your database which are
1255 derived from the external set (say, bib-1) can be used even by clients
1256 who are unaware of the new set.
1258 </listitem></varlistentry>
1261 <replaceable>att-value att-name [local-value]</replaceable></term>
1265 repeatable directive introduces a new attribute to the set. The
1266 attribute value is stored in the index (unless a
1267 <replaceable>local-value</replaceable> is
1268 given, in which case this is stored). The name is used to refer to the
1269 attribute from the <replaceable>abstract syntax</replaceable>.
1271 </listitem></varlistentry>
1276 This is an excerpt from the GILS attribute set definition.
1277 Notice how the file describing the <emphasis>bib-1</emphasis>
1278 attribute set is referenced.
1285 reference GILS-attset
1288 att 2001 distributorName
1289 att 2002 indextermsControlled
1291 att 2004 accessConstraints
1292 att 2005 useConstraints
1300 <title>The Tag Set (.tag) Files</title>
1303 This file type defines the tagset of the profile, possibly by
1304 referencing other tag sets (most tag sets, for instance, will include
1305 tagsetG and tagsetM from the Z39.50 specification. The file may
1306 contain the following directives.
1313 <term>name <emphasis>symbolic-name</emphasis></term>
1316 (m) This provides a shorthand name or
1317 description for the tag set. Mostly useful for diagnostic purposes.
1319 </listitem></varlistentry>
1321 <term>reference <emphasis>OID-name</emphasis></term>
1324 (o) The reference name of the OID for the tag set.
1325 The reference names can be found in the <emphasis>util</emphasis>
1326 module of <emphasis>YAZ</emphasis>.
1327 The directive is optional, since not all tag sets
1328 are registered outside of their schema.
1330 </listitem></varlistentry>
1332 <term>type <emphasis>integer</emphasis></term>
1335 (m) The type number of the tagset within the schema
1336 profile (note: this specification really should belong to the .abs
1337 file. This will be fixed in a future release).
1339 </listitem></varlistentry>
1341 <term>include <emphasis>filename</emphasis></term>
1344 (o,r) This directive is used
1345 to include the definitions of other tag sets into the current one.
1347 </listitem></varlistentry>
1349 <term>tag <emphasis>number names type</emphasis></term>
1352 (o,r) Introduces a new tag to the set.
1353 The <emphasis>number</emphasis> is the tag number as used
1354 in the protocol (there is currently no mechanism for
1355 specifying string tags at this point, but this would be quick
1357 The <emphasis>names</emphasis> parameter is a list of names
1358 by which the tag should be recognized in the input file format.
1359 The names should be separated by slashes (/).
1360 The <emphasis>type</emphasis> is the recommended data type of
1362 It should be one of the following:
1428 </listitem></varlistentry>
1433 The following is an excerpt from the TagsetG definition file.
1444 tag 3 publicationPlace string
1445 tag 4 publicationDate string
1446 tag 5 documentId string
1447 tag 6 abstract string
1449 tag 8 date generalizedtime
1450 tag 9 bodyOfDisplay string
1451 tag 10 organization string
1457 <sect2 id="variant-set">
1458 <title>The Variant Set (.var) Files</title>
1461 The variant set file is a straightforward representation of the
1462 variant set definitions associated with the protocol. At present, only
1463 the <emphasis>Variant-1</emphasis> set is known.
1467 These are the directives allowed in the file.
1474 <term>name <emphasis>symbolic-name</emphasis></term>
1477 (m) This provides a shorthand name or
1478 description for the variant set. Mostly useful for diagnostic purposes.
1480 </listitem></varlistentry>
1482 <term>reference <emphasis>OID-name</emphasis></term>
1485 (o) The reference name of the OID for
1486 the variant set, if one is required. The reference names can be found
1487 in the <emphasis>util</emphasis> module of <emphasis>YAZ</emphasis>.
1489 </listitem></varlistentry>
1491 <term>class <emphasis>integer class-name</emphasis></term>
1494 (m,r) Introduces a new
1495 class to the variant set.
1497 </listitem></varlistentry>
1499 <term>type <emphasis>integer type-name datatype</emphasis></term>
1503 new type to the current class (the one introduced by the most recent
1504 <emphasis>class</emphasis> directive).
1505 The type names belong to the same name space as the one used
1506 in the tag set definition file.
1508 </listitem></varlistentry>
1513 The following is an excerpt from the file describing the variant set
1514 <emphasis>Variant-1</emphasis>.
1525 type 1 variantId octetstring
1530 type 2 z39.50 string
1539 <title>The Element Set (.est) Files</title>
1542 The element set specification files describe a selection of a subset
1543 of the elements of a database record. The element selection mechanism
1544 is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
1545 syntax of the Z39.50 specification.
1546 In fact, the internal representation of an element set
1547 specification is identical to the <emphasis>Espec-1</emphasis> structure,
1548 and we'll refer you to the description of that structure for most of
1549 the detailed semantics of the directives below.
1554 Not all of the Espec-1 functionality has been implemented yet.
1555 The fields that are mentioned below all work as expected, unless
1561 The directives available in the element set file are as follows:
1567 <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
1570 (o) If variants are used in
1571 the following, this should provide the name of the variantset used
1572 (it's not currently possible to specify a different set in the
1573 individual variant request). In almost all cases (certainly all
1574 profiles known to us), the name
1575 <literal>Variant-1</literal> should be given here.
1577 </listitem></varlistentry>
1579 <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
1583 provides a default variant request for
1584 use when the individual element requests (see below) do not contain a
1585 variant request. Variant requests consist of a blank-separated list of
1586 variant components. A variant compont is a comma-separated,
1587 parenthesized triple of variant class, type, and value (the two former
1588 values being represented as integers). The value can currently only be
1589 entered as a string (this will change to depend on the definition of
1590 the variant in question). The special value (@) is interpreted as a
1591 null value, however.
1593 </listitem></varlistentry>
1596 <emphasis>path ['variant' variant-request]</emphasis></term>
1599 (o,r) This corresponds to a simple element request
1600 in <emphasis>Espec-1</emphasis>.
1601 The path consists of a sequence of tag-selectors, where each of
1602 these can consist of either:
1609 A simple tag, consisting of a comma-separated type-value pair in
1610 parenthesis, possibly followed by a colon (:) followed by an
1611 occurrences-specification (see below). The tag-value can be a number
1612 or a string. If the first character is an apostrophe ('), this
1613 forces the value to be interpreted as a string, even if it
1614 appears to be numerical.
1620 A WildThing, represented as a question mark (?), possibly
1621 followed by a colon (:) followed by an occurrences
1622 specification (see below).
1628 A WildPath, represented as an asterisk (*). Note that the last
1629 element of the path should not be a wildPath (wildpaths don't
1630 work in this version).
1639 The occurrences-specification can be either the string
1640 <literal>all</literal>, the string <literal>last</literal>, or
1641 an explicit value-range. The value-range is represented as
1642 an integer (the starting point), possibly followed by a
1643 plus (+) and a second integer (the number of elements, default
1648 The variant-request has the same syntax as the defaultVariantRequest
1649 above. Note that it may sometimes be useful to give an empty variant
1650 request, simply to disable the default for a specific set of fields
1651 (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
1652 but it works in this implementation).
1654 </listitem></varlistentry>
1659 The following is an example of an element specification belonging to
1666 simpleelement (1,10)
1667 simpleelement (1,12)
1669 simpleelement (1,14)
1671 simpleelement (4,52)
1678 <sect2 id="schema-mapping">
1679 <title>The Schema Mapping (.map) Files</title>
1682 Sometimes, the client might want to receive a database record in
1683 a schema that differs from the native schema of the record. For
1684 instance, a client might only know how to process WAIS records, while
1685 the database record is represented in a more specific schema, such as
1686 GILS. In this module, a mapping of data to one of the MARC formats is
1687 also thought of as a schema mapping (mapping the elements of the
1688 record into fields consistent with the given MARC specification, prior
1689 to actually converting the data to the ISO2709). This use of the
1690 object identifier for USMARC as a schema identifier represents an
1691 overloading of the OID which might not be entirely proper. However,
1692 it represents the dual role of schema and record syntax which
1693 is assumed by the MARC family in Z39.50.
1697 <emphasis>NOTE: FIXME! The schema-mapping functions are so far limited to a
1698 straightforward mapping of elements. This should be extended with
1699 mechanisms for conversions of the element contents, and conditional
1700 mappings of elements based on the record contents.</emphasis>
1704 These are the directives of the schema mapping file format:
1711 <term>targetName <emphasis>name</emphasis></term>
1714 (m) A symbolic name for the target schema
1715 of the table. Useful mostly for diagnostic purposes.
1717 </listitem></varlistentry>
1719 <term>targetRef <emphasis>OID-name</emphasis></term>
1722 (m) An OID name for the target schema.
1723 This is used, for instance, by a server receiving a request to present
1724 a record in a different schema from the native one.
1725 The name, again, is found in the <emphasis>oid</emphasis>
1726 module of <emphasis>YAZ</emphasis>.
1728 </listitem></varlistentry>
1730 <term>map <emphasis>element-name target-path</emphasis></term>
1734 an element mapping rule to the table.
1736 </listitem></varlistentry>
1743 <title>The MARC (ISO2709) Representation (.mar) Files</title>
1746 This file provides rules for representing a record in the ISO2709
1747 format. The rules pertain mostly to the values of the constant-length
1748 header of the record.
1752 NOTE: FIXME! This will be described better. We're in the process of
1753 re-evaluating and most likely changing the way that MARC records are
1754 handled by the system.</emphasis>
1759 <sect2 id="field-structure-and-character-sets">
1760 <title>Field Structure and Character Sets
1764 In order to provide a flexible approach to national character set
1765 handling, Zebra allows the administrator to configure the set up the
1766 system to handle any 8-bit character set — including sets that
1767 require multi-octet diacritics or other multi-octet characters. The
1768 definition of a character set includes a specification of the
1769 permissible values, their sort order (this affects the display in the
1770 SCAN function), and relationships between upper- and lowercase
1771 characters. Finally, the definition includes the specification of
1772 space characters for the set.
1776 The operator can define different character sets for different fields,
1777 typical examples being standard text fields, numerical fields, and
1778 special-purpose fields such as WWW-style linkages (URx).
1781 <sect3 id="default-idx-file">
1782 <title>The default.idx file</title>
1784 The field types, and hence character sets, are associated with data
1785 elements by the .abs files (see above).
1786 The file <literal>default.idx</literal>
1787 provides the association between field type codes (as used in the .abs
1788 files) and the character map files (with the .chr suffix). The format
1789 of the .idx file is as follows
1796 <term>index <emphasis>field type code</emphasis></term>
1799 This directive introduces a new search index code.
1800 The argument is a one-character code to be used in the
1801 .abs files to select this particular index type. An index, roughly,
1802 corresponds to a particular structure attribute during search. Refer
1803 to <xref linkend="search"/>.
1805 </listitem></varlistentry>
1807 <term>sort <emphasis>field code type</emphasis></term>
1810 This directive introduces a
1811 sort index. The argument is a one-character code to be used in the
1812 .abs fie to select this particular index type. The corresponding
1813 use attribute must be used in the sort request to refer to this
1814 particular sort index. The corresponding character map (see below)
1815 is used in the sort process.
1817 </listitem></varlistentry>
1819 <term>completeness <emphasis>boolean</emphasis></term>
1822 This directive enables or disables complete field indexing.
1823 The value of the <emphasis>boolean</emphasis> should be 0
1824 (disable) or 1. If completeness is enabled, the index entry will
1825 contain the complete contents of the field (up to a limit), with words
1826 (non-space characters) separated by single space characters
1827 (normalized to " " on display). When completeness is
1828 disabled, each word is indexed as a separate entry. Complete subfield
1829 indexing is most useful for fields which are typically browsed (eg.
1830 titles, authors, or subjects), or instances where a match on a
1831 complete subfield is essential (eg. exact title searching). For fields
1832 where completeness is disabled, the search engine will interpret a
1833 search containing space characters as a word proximity search.
1835 </listitem></varlistentry>
1837 <term>charmap <emphasis>filename</emphasis></term>
1840 This is the filename of the character
1841 map to be used for this index for field type.
1843 </listitem></varlistentry>
1848 <sect3 id="character-map-files">
1849 <title>The character map file format</title>
1851 The contents of the character map files are structured as follows:
1858 <term>lowercase <emphasis>value-set</emphasis></term>
1861 This directive introduces the basic value set of the field type.
1862 The format is an ordered list (without spaces) of the
1863 characters which may occur in "words" of the given type.
1864 The order of the entries in the list determines the
1865 sort order of the index. In addition to single characters, the
1866 following combinations are legal:
1874 Backslashes may be used to introduce three-digit octal, or
1875 two-digit hex representations of single characters
1876 (preceded by <literal>x</literal>).
1877 In addition, the combinations
1878 \\, \\r, \\n, \\t, \\s (space — remember that real
1879 space-characters may not occur in the value definition), and
1880 \\ are recognized, with their usual interpretation.
1886 Curly braces {} may be used to enclose ranges of single
1887 characters (possibly using the escape convention described in the
1888 preceding point), eg. {a-z} to introduce the
1889 standard range of ASCII characters.
1890 Note that the interpretation of such a range depends on
1891 the concrete representation in your local, physical character set.
1897 paranthesises () may be used to enclose multi-byte characters -
1898 eg. diacritics or special national combinations (eg. Spanish
1899 "ll"). When found in the input stream (or a search term),
1900 these characters are viewed and sorted as a single character, with a
1901 sorting value depending on the position of the group in the value
1909 </listitem></varlistentry>
1911 <term>uppercase <emphasis>value-set</emphasis></term>
1914 This directive introduces the
1915 upper-case equivalencis to the value set (if any). The number and
1916 order of the entries in the list should be the same as in the
1917 <literal>lowercase</literal> directive.
1919 </listitem></varlistentry>
1921 <term>space <emphasis>value-set</emphasis></term>
1924 This directive introduces the character
1925 which separate words in the input stream. Depending on the
1926 completeness mode of the field in question, these characters either
1927 terminate an index entry, or delimit individual "words" in
1928 the input stream. The order of the elements is not significant —
1929 otherwise the representation is the same as for the
1930 <literal>uppercase</literal> and <literal>lowercase</literal>
1933 </listitem></varlistentry>
1935 <term>map <emphasis>value-set</emphasis>
1936 <emphasis>target</emphasis></term>
1939 This directive introduces a mapping between each of the
1940 members of the value-set on the left to the character on the
1941 right. The character on the right must occur in the value
1942 set (the <literal>lowercase</literal> directive) of the
1943 character set, but it may be a paranthesis-enclosed
1944 multi-octet character. This directive may be used to map
1945 diacritics to their base characters, or to map HTML-style
1946 character-representations to their natural form, etc. The
1947 map directive can also be used to ignore leading articles in
1948 searching and/or sorting, and to perform other special
1949 transformations. See section <xref
1950 linkend="leading-articles"/>.
1952 </listitem></varlistentry>
1956 <sect3 id="leading-articles">
1957 <title>Ignoring leading articles</title>
1959 In addition to specifying sort orders, space (blank) handling,
1960 and upper/lowercase folding, you can also use the character map
1961 files to make Zebra ignore leading articles in sorting records,
1962 or when doing complete field searching.
1965 This is done using the <literal>map</literal> directive in the
1966 character map file. In a nutshell, what you do is map certain
1967 sequences of characters, when they occur <emphasis> in the
1968 beginning of a field</emphasis>, to a space. Assuming that the
1969 character "@" is defined as a space character in your file, you
1975 The effect of these directives is to map either 'the' or 'The',
1976 followed by a space character, to a space. The hat ^ character
1977 denotes beginning-of-field only when complete-subfield indexing
1978 or sort indexing is taking place; otherwise, it is treated just
1979 as any other character.
1982 Because the <literal>default.idx</literal> file can be used to
1983 associate different character maps with different indexing types
1984 -- and you can create additional indexing types, should the need
1985 arise -- it is possible to specify that leading articles should
1986 be ignored either in sorting, in complete-field searching, or
1990 If you ignore certain prefixes in sorting, then these will be
1991 eliminated from the index, and sorting will take place as if
1992 they weren't there. However, if you set the system up to ignore
1993 certain prefixes in <emphasis>searching</emphasis>, then these
1994 are deleted both from the indexes and from query terms, when the
1995 client specifies complete-field searching. This has the effect
1996 that a search for 'the science journal' and 'science journal'
1997 would both produce the same results.
2003 <sect1 id="grs-exchange-formats">
2004 <title>GRS Exchange Formats</title>
2007 Converting records from the internal structure to an exchange format
2008 is largely an automatic process. Currently, the following exchange
2009 formats are supported:
2016 GRS-1. The internal representation is based on GRS-1/XML, so the
2017 conversion here is straightforward. The system will create
2018 applied variant and supported variant lists as required, if a record
2019 contains variant information.
2025 XML. The internal representation is based on GRS-1/XML so
2026 the mapping is trivial. Note that XML schemas, preprocessing
2027 instructions and comments are not part of the internal representation
2028 and therefore will never be part of a generated XML record.
2029 Future versions of the Zebra will support that.
2035 SUTRS. Again, the mapping is fairly straightforward. Indentation
2036 is used to show the hierarchical structure of the record. All
2037 "GRS" type records support both the GRS-1 and SUTRS
2039 <!-- FIXME - What is SUTRS - should be expanded here -->
2045 ISO2709-based formats (USMARC, etc.). Only records with a
2046 two-level structure (corresponding to fields and subfields) can be
2047 directly mapped to ISO2709. For records with a different structuring
2048 (eg., GILS), the representation in a structure like USMARC involves a
2049 schema-mapping (see <xref linkend="schema-mapping"/>), to an
2050 "implied" USMARC schema (implied,
2051 because there is no formal schema which specifies the use of the
2052 USMARC fields outside of ISO2709). The resultant, two-level record is
2053 then mapped directly from the internal representation to ISO2709. See
2054 the GILS schema definition files for a detailed example of this
2061 Explain. This representation is only available for records
2062 belonging to the Explain schema.
2068 Summary. This ASN-1 based structure is only available for records
2069 belonging to the Summary schema - or schema which provide a mapping
2070 to this schema (see the description of the schema mapping facility
2075 <!-- FIXME - Is this used anywhere ? -H -->
2078 SOIF. Support for this syntax is experimental, and is currently
2079 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
2080 abstract syntaxes can be mapped to the SOIF format, although nested
2081 elements are represented by concatenation of the tag names at each
2091 <!-- Keep this comment at the end of the file
2096 sgml-minimize-attributes:nil
2097 sgml-always-quote-attributes:t
2100 sgml-parent-document: "zebra.xml"
2101 sgml-local-catalogs: nil
2102 sgml-namecase-general:t
projects / idzebra-moved-to-github.git / blob