1 <chapter id="record-model">
2 <!-- $Id: recordmodel.xml,v 1.11 2002-10-20 14:02:03 mike Exp $ -->
3 <title>The Record Model</title>
6 The Zebra system is designed to support a wide range of data management
7 applications. The system can be configured to handle virtually any
8 kind of structured data. Each record in the system is associated with
9 a <emphasis>record schema</emphasis> which lends context to the data
10 elements of the record.
11 Any number of record schemas can coexist in the system.
12 Although it may be wise to use only a single schema within
13 one database, the system poses no such restrictions.
17 The record model described in this chapter applies to the fundamental,
19 record type <literal>grs</literal>, introduced in
20 <xref linkend="record-types"/>.
22 FIXME - Need to describe the simple string-tag model, or at least
28 Records pass through three different states during processing in the
38 When records are accessed by the system, they are represented
39 in their local, or native format. This might be SGML or HTML files,
40 News or Mail archives, MARC records. If the system doesn't already
41 know how to read the type of data you need to store, you can set up an
42 input filter by preparing conversion rules based on regular
43 expressions and possibly augmented by a flexible scripting language
45 The input filter produces as output an internal representation:
52 When records are processed by the system, they are represented
53 in a tree-structure, constructed by tagged data elements hanging off a
54 root node. The tagged elements may contain data or yet more tagged
55 elements in a recursive structure. The system performs various
56 actions on this tree structure (indexing, element selection, schema
64 Before transmitting records to the client, they are first
65 converted from the internal structure to a form suitable for exchange
66 over the network - according to the Z39.50 standard.
74 <sect1 id="local-representation">
75 <title>Local Representation</title>
78 As mentioned earlier, Zebra places few restrictions on the type of
79 data that you can index and manage. Generally, whatever the form of
80 the data, it is parsed by an input filter specific to that format, and
81 turned into an internal structure that Zebra knows how to handle. This
82 process takes place whenever the record is accessed - for indexing and
87 The RecordType parameter in the <literal>zebra.cfg</literal> file, or
88 the <literal>-t</literal> option to the indexer tells Zebra how to
89 process input records.
90 Two basic types of processing are available - raw text and structured
91 data. Raw text is just that, and it is selected by providing the
92 argument <emphasis>text</emphasis> to Zebra. Structured records are
93 all handled internally using the basic mechanisms described in the
95 Zebra can read structured records in many different formats.
96 How this is done is governed by additional parameters after the
97 "grs" keyboard, separated by "." characters.
101 Four basic subtypes to the <emphasis>grs</emphasis> type are
108 <term>grs.sgml</term>
111 This is the canonical input format —
112 described below. It is a simple SGML-like syntax.
117 <term>grs.regx.<emphasis>filter</emphasis></term>
120 This enables a user-supplied input
121 filter. The mechanisms of these filters are described below.
126 <term>grs.tcl.<emphasis>filter</emphasis></term>
129 Similar to grs.regx but using Tcl for rules.
134 <term>grs.marc.<emphasis>abstract syntax</emphasis></term>
137 This allows Zebra to read
138 records in the ISO2709 (MARC) encoding standard. In this case, the
139 last parameter <emphasis>abstract syntax</emphasis> names the
140 <literal>.abs</literal> file (see below)
141 which describes the specific MARC structure of the input record as
142 well as the indexing rules.
150 This filter reads XML records. Only one record per file
151 is supported. The filter is only available if Zebra/YAZ
152 is compiled with EXPAT support.
161 <title>Canonical Input Format</title>
164 Although input data can take any form, it is sometimes useful to
165 describe the record processing capabilities of the system in terms of
166 a single, canonical input format that gives access to the full
167 spectrum of structure and flexibility in the system. In Zebra, this
168 canonical format is an "SGML-like" syntax.
172 To use the canonical format specify <literal>grs.sgml</literal> as
177 Consider a record describing an information resource (such a record is
178 sometimes known as a <emphasis>locator record</emphasis>).
179 It might contain a field describing the distributor of the
180 information resource, which might in turn be partitioned into
181 various fields providing details about the distributor, like this:
187 <Distributor>
188 <Name> USGS/WRD </Name>
189 <Organization> USGS/WRD </Organization>
190 <Street-Address>
191 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
192 </Street-Address>
193 <City> ALBUQUERQUE </City>
194 <State> NM </State>
195 <Zip-Code> 87102 </Zip-Code>
196 <Country> USA </Country>
197 <Telephone> (505) 766-5560 </Telephone>
198 </Distributor>
203 <!-- There is no indentation in the example above! -H
206 The indentation used above is used to illustrate how Zebra
207 interprets the mark-up. The indentation, in itself, has no
208 significance to the parser for the canonical input format, which
209 discards superfluous whitespace.
215 The keywords surrounded by <...> are
216 <emphasis>tags</emphasis>, while the sections of text
217 in between are the <emphasis>data elements</emphasis>.
218 A data element is characterized by its location in the tree
219 that is made up by the nested elements.
220 Each element is terminated by a closing tag - beginning
221 with <literal><</literal>/, and containing the same symbolic
222 tag-name as the corresponding opening tag.
223 The general closing tag - <literal><</literal>/> -
224 terminates the element started by the last opening tag. The
225 structuring of elements is significant.
226 The element <emphasis>Telephone</emphasis>,
227 for instance, may be indexed and presented to the client differently,
228 depending on whether it appears inside the
229 <emphasis>Distributor</emphasis> element, or some other,
230 structured data element such a <emphasis>Supplier</emphasis> element.
234 <title>Record Root</title>
237 The first tag in a record describes the root node of the tree that
238 makes up the total record. In the canonical input format, the root tag
239 should contain the name of the schema that lends context to the
240 elements of the record
241 (see <xref linkend="internal-representation"/>).
242 The following is a GILS record that
243 contains only a single element (strictly speaking, that makes it an
244 illegal GILS record, since the GILS profile includes several mandatory
245 elements - Zebra does not validate the contents of a record against
246 the Z39.50 profile, however - it merely attempts to match up elements
247 of a local representation with the given schema):
254 <title>Zen and the Art of Motorcycle Maintenance</title>
262 <sect3><!-- ### we shouldn't make such a big deal about this -->
263 <title>Variants</title>
266 Zebra allows you to provide individual data elements in a number of
267 <emphasis>variant forms</emphasis>. Examples of variant forms are
268 textual data elements which might appear in different languages, and
269 images which may appear in different formats or layouts.
270 The variant system in Zebra is essentially a representation of
271 the variant mechanism of Z39.50-1995.
275 The following is an example of a title element which occurs in two
283 <var lang lang "eng">
284 Zen and the Art of Motorcycle Maintenance</>
285 <var lang lang "dan">
286 Zen og Kunsten at Vedligeholde en Motorcykel</>
293 The syntax of the <emphasis>variant element</emphasis> is
294 <literal><var class type value></literal>.
295 The available values for the <emphasis>class</emphasis> and
296 <emphasis>type</emphasis> fields are given by the variant set
297 that is associated with the current schema
298 (see <xref linkend="variant-set"/>).
302 Variant elements are terminated by the general end-tag </>, by
303 the variant end-tag </var>, by the appearance of another variant
304 tag with the same <emphasis>class</emphasis> and
305 <emphasis>value</emphasis> settings, or by the
306 appearance of another, normal tag. In other words, the end-tags for
307 the variants used in the example above could have been omitted.
311 Variant elements can be nested. The element
318 <var lang lang "eng"><var body iana "text/plain">
319 Zen and the Art of Motorcycle Maintenance
326 Associates two variant components to the variant list for the title
331 Given the nesting rules described above, we could write
338 <var body iana "text/plain>
339 <var lang lang "eng">
340 Zen and the Art of Motorcycle Maintenance
341 <var lang lang "dan">
342 Zen og Kunsten at Vedligeholde en Motorcykel
349 The title element above comes in two variants. Both have the IANA body
350 type "text/plain", but one is in English, and the other in
351 Danish. The client, using the element selection mechanism of Z39.50,
352 can retrieve information about the available variant forms of data
353 elements, or it can select specific variants based on the requirements
362 <title>Input Filters</title>
365 In order to handle general input formats, Zebra allows the
366 operator to define filters which read individual records in their
367 native format and produce an internal representation that the system
372 Input filters are ASCII files, generally with the suffix
373 <literal>.flt</literal>.
374 The system looks for the files in the directories given in the
375 <emphasis>profilePath</emphasis> setting in the
376 <literal>zebra.cfg</literal> files.
377 The record type for the filter is
378 <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
379 (fundamental type <literal>grs</literal>, file read
380 type <literal>regx</literal>, argument
381 <emphasis>filter-filename</emphasis>).
385 Generally, an input filter consists of a sequence of rules, where each
386 rule consists of a sequence of expressions, followed by an action. The
387 expressions are evaluated against the contents of the input record,
388 and the actions normally contribute to the generation of an internal
389 representation of the record.
393 An expression can be either of the following:
403 The action associated with this expression is evaluated
404 exactly once in the lifetime of the application, before any records
405 are read. It can be used in conjunction with an action that
406 initializes tables or other resources that are used in the processing
415 Matches the beginning of the record. It can be used to
416 initialize variables, etc. Typically, the
417 <emphasis>BEGIN</emphasis> rule is also used
418 to establish the root node of the record.
426 Matches the end of the record - when all of the contents
427 of the record has been processed.
432 <term>/pattern/</term>
435 Matches a string of characters from the input record.
443 This keyword may only be used between two patterns.
444 It matches everything between (not including) those patterns.
452 The expression associated with this pattern is evaluated
453 once, before the application terminates. It can be used to release
454 system resources - typically ones allocated in the
455 <emphasis>INIT</emphasis> step.
463 An action is surrounded by curly braces ({...}), and
464 consists of a sequence of statements. Statements may be separated
465 by newlines or semicolons (;).
466 Within actions, the strings that matched the expressions
467 immediately preceding the action can be referred to as
472 The available statements are:
479 <term>begin <emphasis>type [parameter ... ]</emphasis></term>
483 data element. The type is one of the following:
490 Begin a new record. The following parameter should be the
491 name of the schema that describes the structure of the record, eg.
492 <literal>gils</literal> or <literal>wais</literal> (see below).
493 The <literal>begin record</literal> call should precede
494 any other use of the <emphasis>begin</emphasis> statement.
502 Begin a new tagged element. The parameter is the
503 name of the tag. If the tag is not matched anywhere in the tagsets
504 referenced by the current schema, it is treated as a local string
513 Begin a new node in a variant tree. The parameters are
514 <emphasis>class type value</emphasis>.
526 Create a data element. The concatenated arguments make
527 up the value of the data element.
528 The option <literal>-text</literal> signals that
529 the layout (whitespace) of the data should be retained for
531 The option <literal>-element</literal>
532 <emphasis>tag</emphasis> wraps the data up in
533 the <emphasis>tag</emphasis>.
534 The use of the <literal>-element</literal> option is equivalent to
535 preceding the command with a <emphasis>begin
536 element</emphasis> command, and following
537 it with the <emphasis>end</emphasis> command.
542 <term>end <emphasis>[type]</emphasis></term>
545 Close a tagged element. If no parameter is given,
546 the last element on the stack is terminated.
547 The first parameter, if any, is a type name, similar
548 to the <emphasis>begin</emphasis> statement.
549 For the <emphasis>element</emphasis> type, a tag
550 name can be provided to terminate a specific tag.
558 The following input filter reads a Usenet news file, producing a
559 record in the WAIS schema. Note that the body of a news posting is
560 separated from the list of headers by a blank line (or rather a
561 sequence of two newline characters.
567 BEGIN { begin record wais }
569 /^From:/ BODY /$/ { data -element name $1 }
570 /^Subject:/ BODY /$/ { data -element title $1 }
571 /^Date:/ BODY /$/ { data -element lastModified $1 }
573 begin element bodyOfDisplay
574 begin variant body iana "text/plain"
583 If Zebra is compiled with support for Tcl (Tool Command Language)
584 enabled, the statements described above are supplemented with a complete
585 scripting environment, including control structures (conditional
586 expressions and loop constructs), and powerful string manipulation
587 mechanisms for modifying the elements of a record. Tcl is a popular
588 scripting environment, with several tutorials available both online
596 <sect1 id="internal-representation">
597 <title>Internal Representation</title>
600 When records are manipulated by the system, they're represented in a
601 tree-structure, with data elements at the leaf nodes, and tags or
602 variant components at the non-leaf nodes. The root-node identifies the
603 schema that lends context to the tagging and structuring of the
604 record. Imagine a simple record, consisting of a 'title' element and
612 TITLE "Zen and the Art of Motorcycle Maintenance"
613 AUTHOR "Robert Pirsig"
619 A slightly more complex record would have the author element consist
620 of two elements, a surname and a first name:
627 TITLE "Zen and the Art of Motorcycle Maintenance"
636 The root of the record will refer to the record schema that describes
637 the structuring of this particular record. The schema defines the
638 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
639 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
640 addition, the schema establishes element set names that are used by
641 the client to request a subset of the elements of a given record. The
642 schema may also establish rules for converting the record to a
643 different schema, by stating, for each element, a mapping to a
648 <title>Tagged Elements</title>
651 A data element is characterized by its tag, and its position in the
652 structure of the record. For instance, while the tag "telephone
653 number" may be used different places in a record, we may need to
654 distinguish between these occurrences, both for searching and
655 presentation purposes. For instance, while the phone numbers for the
656 "customer" and the "service provider" are both
657 representatives for the same type of resource (a telephone number), it
658 is essential that they be kept separate. The record schema provides
659 the structure of the record, and names each data element (defined by
660 the sequence of tags - the tag path - by which the element can be
661 reached from the root of the record).
667 <title>Variants</title>
670 The children of a tag node may be either more tag nodes, a data node
671 (possibly accompanied by tag nodes),
672 or a tree of variant nodes. The children of variant nodes are either
673 more variant nodes or a data node (possibly accompanied by more
674 variant nodes). Each leaf node, which is normally a
675 data node, corresponds to a <emphasis>variant form</emphasis> of the
676 tagged element identified by the tag which parents the variant tree.
677 The following title element occurs in two different languages:
683 VARIANT LANG=ENG "War and Peace"
685 VARIANT LANG=DAN "Krig og Fred"
691 Which of the two elements are transmitted to the client by the server
692 depends on the specifications provided by the client, if any.
696 In practice, each variant node is associated with a triple of class,
697 type, value, corresponding to the variant mechanism of Z39.50.
703 <title>Data Elements</title>
706 Data nodes have no children (they are always leaf nodes in the record
711 FIXME! Documentation needs extension here about types of nodes - numerical,
712 textual, etc., plus the various types of inclusion notes.
720 <sect1 id="data-model">
721 <title>Configuring Your Data Model</title>
724 The following sections describe the configuration files that govern
725 the internal management of data records. The system searches for the files
726 in the directories specified by the <emphasis>profilePath</emphasis>
727 setting in the <literal>zebra.cfg</literal> file.
731 <title>The Abstract Syntax</title>
734 The abstract syntax definition (also known as an Abstract Record
735 Structure, or ARS) is the focal point of the
736 record schema description. For a given schema, the ABS file may state any
737 or all of the following:
741 FIXME - Need a diagram here, or a simple explanation how it all hangs together -H
750 The object identifier of the Z39.50 schema associated
751 with the ARS, so that it can be referred to by the client.
757 The attribute set (which can possibly be a compound of multiple
758 sets) which applies in the profile. This is used when indexing and
759 searching the records belonging to the given profile.
765 The Tag set (again, this can consist of several different sets).
766 This is used when reading the records from a file, to recognize the
767 different tags, and when transmitting the record to the client -
768 mapping the tags to their numerical representation, if they are
775 The variant set which is used in the profile. This provides a
776 vocabulary for specifying the <emphasis>forms</emphasis> of
777 data that appear inside the records.
783 Element set names, which are a shorthand way for the client to
784 ask for a subset of the data elements contained in a record. Element
785 set names, in the retrieval module, are mapped to <emphasis>element
786 specifications</emphasis>, which contain information equivalent to the
787 <emphasis>Espec-1</emphasis> syntax of Z39.50.
793 Map tables, which may specify mappings to
794 <emphasis>other</emphasis> database profiles, if desired.
800 Possibly, a set of rules describing the mapping of elements to a
808 A list of element descriptions (this is the actual ARS of the
809 schema, in Z39.50 terms), which lists the ways in which the various
810 tags can be used and organized hierarchically.
819 Several of the entries above simply refer to other files, which
820 describe the given objects.
826 <title>The Configuration Files</title>
829 This section describes the syntax and use of the various tables which
830 are used by the retrieval module.
834 The number of different file types may appear daunting at first, but
835 each type corresponds fairly clearly to a single aspect of the Z39.50
836 retrieval facilities. Further, the average database administrator,
837 who is simply reusing an existing profile for which tables already
838 exist, shouldn't have to worry too much about the contents of these tables.
842 Generally, the files are simple ASCII files, which can be maintained
843 using any text editor. Blank lines, and lines beginning with a (#) are
844 ignored. Any characters on a line followed by a (#) are also ignored.
845 All other lines contain <emphasis>directives</emphasis>, which provide
846 some setting or value to the system.
847 Generally, settings are characterized by a single
848 keyword, identifying the setting, followed by a number of parameters.
849 Some settings are repeatable (r), while others may occur only once in a
850 file. Some settings are optional (o), while others again are
856 <sect2 id="abs-file">
857 <title>The Abstract Syntax (.abs) Files</title>
860 The name of this file type is slightly misleading in Z39.50 terms,
861 since, apart from the actual abstract syntax of the profile, it also
862 includes most of the other definitions that go into a database
867 When a record in the canonical, SGML-like format is read from a file
868 or from the database, the first tag of the file should reference the
869 profile that governs the layout of the record. If the first tag of the
870 record is, say, <literal><gils></literal>, the system will look
871 for the profile definition in the file <literal>gils.abs</literal>.
872 Profile definitions are cached, so they only have to be read once
873 during the lifespan of the current process.
877 When writing your own input filters, the
878 <emphasis>record-begin</emphasis> command
879 introduces the profile, and should always be called first thing when
880 introducing a new record.
884 The file may contain the following directives:
891 <term>name <emphasis>symbolic-name</emphasis></term>
894 (m) This provides a shorthand name or
895 description for the profile. Mostly useful for diagnostic purposes.
900 <term>reference <emphasis>OID-name</emphasis></term>
903 (m) The reference name of the OID for the profile.
904 The reference names can be found in the <emphasis>util</emphasis>
905 module of <emphasis>YAZ</emphasis>.
910 <term>attset <emphasis>filename</emphasis></term>
913 (m) The attribute set that is used for
914 indexing and searching records belonging to this profile.
919 <term>tagset <emphasis>filename</emphasis></term>
922 (o) The tag set (if any) that describe
923 that fields of the records.
928 <term>varset <emphasis>filename</emphasis></term>
931 (o) The variant set used in the profile.
936 <term>maptab <emphasis>filename</emphasis></term>
939 (o,r) This points to a
940 conversion table that might be used if the client asks for the record
941 in a different schema from the native one.
943 </listitem></varlistentry>
945 <term>marc <emphasis>filename</emphasis></term>
948 (o) Points to a file containing parameters
949 for representing the record contents in the ISO2709 syntax. Read the
950 description of the MARC representation facility below.
952 </listitem></varlistentry>
954 <term>esetname <emphasis>name filename</emphasis></term>
958 given element set name with an element selection file. If an (@) is
959 given in place of the filename, this corresponds to a null mapping for
960 the given element set name.
962 </listitem></varlistentry>
964 <term>any <emphasis>tags</emphasis></term>
967 (o) This directive specifies a list of attributes
968 which should be appended to the attribute list given for each
969 element. The effect is to make every single element in the abstract
970 syntax searchable by way of the given attributes. This directive
971 provides an efficient way of supporting free-text searching across all
972 elements. However, it does increase the size of the index
973 significantly. The attributes can be qualified with a structure, as in
974 the <emphasis>elm</emphasis> directive below.
976 </listitem></varlistentry>
978 <term>elm <emphasis>path name attributes</emphasis></term>
981 (o,r) Adds an element to the abstract record syntax of the schema.
982 The <emphasis>path</emphasis> follows the
983 syntax which is suggested by the Z39.50 document - that is, a sequence
984 of tags separated by slashes (/). Each tag is given as a
985 comma-separated pair of tag type and -value surrounded by parenthesis.
986 The <emphasis>name</emphasis> is the name of the element, and
987 the <emphasis>attributes</emphasis>
988 specifies which attributes to use when indexing the element in a
989 comma-separated list.
990 A ! in place of the attribute name is equivalent to
991 specifying an attribute name identical to the element name.
992 A - in place of the attribute name
993 specifies that no indexing is to take place for the given element.
994 The attributes can be qualified with <emphasis>field
995 types</emphasis> to specify which
996 character set should govern the indexing procedure for that field.
997 The same data element may be indexed into several different
998 fields, using different character set definitions.
999 See the <xref linkend="field-structure-and-character-sets"/>.
1000 The default field type is "w" for <emphasis>word</emphasis>.
1002 </listitem></varlistentry>
1004 <term>encoding <emphasis>encodingname</emphasis></term>
1007 This directive specifies character encoding for external records.
1008 For records such as XML that specifies encoding within the
1009 file via a header this directive is ignored.
1010 If neither this directive is given, nor an encoding is set
1011 within external records, ISO-8859-1 encoding is assmed.
1016 <term>xpath <emphasis>enable/disable</emphasis></term>
1019 If this directive is followed by <literal>enable</literal>,
1020 then extra indexing is performed to allow for XPath-like queries.
1021 If this directive is not specified - equivalent to
1022 <literal>disable</literal> - no extra XPath-indexing is performed.
1031 The mechanism for controlling indexing is not adequate for
1032 complex databases, and will probably be moved into a separate
1033 configuration table eventually.
1038 The following is an excerpt from the abstract syntax file for the GILS
1046 reference GILS-schema
1051 maptab gils-usmarc.map
1055 esetname VARIANT gils-variant.est # for WAIS-compliance
1056 esetname B gils-b.est
1057 esetname G gils-g.est
1062 elm (1,14) localControlNumber Local-number
1063 elm (1,16) dateOfLastModification Date/time-last-modified
1064 elm (2,1) title w:!,p:!
1065 elm (4,1) controlIdentifier Identifier-standard
1066 elm (2,6) abstract Abstract
1067 elm (4,51) purpose !
1068 elm (4,52) originator -
1069 elm (4,53) accessConstraints !
1070 elm (4,54) useConstraints !
1071 elm (4,70) availability -
1072 elm (4,70)/(4,90) distributor -
1073 elm (4,70)/(4,90)/(2,7) distributorName !
1074 elm (4,70)/(4,90)/(2,10 distributorOrganization !
1075 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1076 elm (4,70)/(4,90)/(4,3) distributorCity !
1083 <sect2 id="attset-files">
1084 <title>The Attribute Set (.att) Files</title>
1087 This file type describes the <emphasis>Use</emphasis> elements of
1089 It contains the following directives.
1095 <term>name <emphasis>symbolic-name</emphasis></term>
1098 (m) This provides a shorthand name or
1099 description for the attribute set.
1100 Mostly useful for diagnostic purposes.
1102 </listitem></varlistentry>
1104 <term>reference <emphasis>OID-name</emphasis></term>
1107 (m) The reference name of the OID for
1109 The reference names can be found in the <emphasis>util</emphasis>
1110 module of <emphasis>YAZ</emphasis>.
1112 </listitem></varlistentry>
1114 <term>include <emphasis>filename</emphasis></term>
1117 (o,r) This directive is used to
1118 include another attribute set as a part of the current one. This is
1119 used when a new attribute set is defined as an extension to another
1120 set. For instance, many new attribute sets are defined as extensions
1121 to the <emphasis>bib-1</emphasis> set.
1122 This is an important feature of the retrieval
1123 system of Z39.50, as it ensures the highest possible level of
1124 interoperability, as those access points of your database which are
1125 derived from the external set (say, bib-1) can be used even by clients
1126 who are unaware of the new set.
1128 </listitem></varlistentry>
1131 <emphasis>att-value att-name [local-value]</emphasis></term>
1135 repeatable directive introduces a new attribute to the set. The
1136 attribute value is stored in the index (unless a
1137 <emphasis>local-value</emphasis> is
1138 given, in which case this is stored). The name is used to refer to the
1139 attribute from the <emphasis>abstract syntax</emphasis>.
1141 </listitem></varlistentry>
1146 This is an excerpt from the GILS attribute set definition.
1147 Notice how the file describing the <emphasis>bib-1</emphasis>
1148 attribute set is referenced.
1155 reference GILS-attset
1158 att 2001 distributorName
1159 att 2002 indextermsControlled
1161 att 2004 accessConstraints
1162 att 2005 useConstraints
1170 <title>The Tag Set (.tag) Files</title>
1173 This file type defines the tagset of the profile, possibly by
1174 referencing other tag sets (most tag sets, for instance, will include
1175 tagsetG and tagsetM from the Z39.50 specification. The file may
1176 contain the following directives.
1183 <term>name <emphasis>symbolic-name</emphasis></term>
1186 (m) This provides a shorthand name or
1187 description for the tag set. Mostly useful for diagnostic purposes.
1189 </listitem></varlistentry>
1191 <term>reference <emphasis>OID-name</emphasis></term>
1194 (o) The reference name of the OID for the tag set.
1195 The reference names can be found in the <emphasis>util</emphasis>
1196 module of <emphasis>YAZ</emphasis>.
1197 The directive is optional, since not all tag sets
1198 are registered outside of their schema.
1200 </listitem></varlistentry>
1202 <term>type <emphasis>integer</emphasis></term>
1205 (m) The type number of the tagset within the schema
1206 profile (note: this specification really should belong to the .abs
1207 file. This will be fixed in a future release).
1209 </listitem></varlistentry>
1211 <term>include <emphasis>filename</emphasis></term>
1214 (o,r) This directive is used
1215 to include the definitions of other tag sets into the current one.
1217 </listitem></varlistentry>
1219 <term>tag <emphasis>number names type</emphasis></term>
1222 (o,r) Introduces a new tag to the set.
1223 The <emphasis>number</emphasis> is the tag number as used
1224 in the protocol (there is currently no mechanism for
1225 specifying string tags at this point, but this would be quick
1227 The <emphasis>names</emphasis> parameter is a list of names
1228 by which the tag should be recognized in the input file format.
1229 The names should be separated by slashes (/).
1230 The <emphasis>type</emphasis> is the recommended data type of
1232 It should be one of the following:
1298 </listitem></varlistentry>
1303 The following is an excerpt from the TagsetG definition file.
1314 tag 3 publicationPlace string
1315 tag 4 publicationDate string
1316 tag 5 documentId string
1317 tag 6 abstract string
1319 tag 8 date generalizedtime
1320 tag 9 bodyOfDisplay string
1321 tag 10 organization string
1327 <sect2 id="variant-set">
1328 <title>The Variant Set (.var) Files</title>
1331 The variant set file is a straightforward representation of the
1332 variant set definitions associated with the protocol. At present, only
1333 the <emphasis>Variant-1</emphasis> set is known.
1337 These are the directives allowed in the file.
1344 <term>name <emphasis>symbolic-name</emphasis></term>
1347 (m) This provides a shorthand name or
1348 description for the variant set. Mostly useful for diagnostic purposes.
1350 </listitem></varlistentry>
1352 <term>reference <emphasis>OID-name</emphasis></term>
1355 (o) The reference name of the OID for
1356 the variant set, if one is required. The reference names can be found
1357 in the <emphasis>util</emphasis> module of <emphasis>YAZ</emphasis>.
1359 </listitem></varlistentry>
1361 <term>class <emphasis>integer class-name</emphasis></term>
1364 (m,r) Introduces a new
1365 class to the variant set.
1367 </listitem></varlistentry>
1369 <term>type <emphasis>integer type-name datatype</emphasis></term>
1373 new type to the current class (the one introduced by the most recent
1374 <emphasis>class</emphasis> directive).
1375 The type names belong to the same name space as the one used
1376 in the tag set definition file.
1378 </listitem></varlistentry>
1383 The following is an excerpt from the file describing the variant set
1384 <emphasis>Variant-1</emphasis>.
1395 type 1 variantId octetstring
1400 type 2 z39.50 string
1409 <title>The Element Set (.est) Files</title>
1412 The element set specification files describe a selection of a subset
1413 of the elements of a database record. The element selection mechanism
1414 is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
1415 syntax of the Z39.50 specification.
1416 In fact, the internal representation of an element set
1417 specification is identical to the <emphasis>Espec-1</emphasis> structure,
1418 and we'll refer you to the description of that structure for most of
1419 the detailed semantics of the directives below.
1424 Not all of the Espec-1 functionality has been implemented yet.
1425 The fields that are mentioned below all work as expected, unless
1431 The directives available in the element set file are as follows:
1437 <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
1440 (o) If variants are used in
1441 the following, this should provide the name of the variantset used
1442 (it's not currently possible to specify a different set in the
1443 individual variant request). In almost all cases (certainly all
1444 profiles known to us), the name
1445 <literal>Variant-1</literal> should be given here.
1447 </listitem></varlistentry>
1449 <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
1453 provides a default variant request for
1454 use when the individual element requests (see below) do not contain a
1455 variant request. Variant requests consist of a blank-separated list of
1456 variant components. A variant compont is a comma-separated,
1457 parenthesized triple of variant class, type, and value (the two former
1458 values being represented as integers). The value can currently only be
1459 entered as a string (this will change to depend on the definition of
1460 the variant in question). The special value (@) is interpreted as a
1461 null value, however.
1463 </listitem></varlistentry>
1466 <emphasis>path ['variant' variant-request]</emphasis></term>
1469 (o,r) This corresponds to a simple element request
1470 in <emphasis>Espec-1</emphasis>.
1471 The path consists of a sequence of tag-selectors, where each of
1472 these can consist of either:
1479 A simple tag, consisting of a comma-separated type-value pair in
1480 parenthesis, possibly followed by a colon (:) followed by an
1481 occurrences-specification (see below). The tag-value can be a number
1482 or a string. If the first character is an apostrophe ('), this
1483 forces the value to be interpreted as a string, even if it
1484 appears to be numerical.
1490 A WildThing, represented as a question mark (?), possibly
1491 followed by a colon (:) followed by an occurrences
1492 specification (see below).
1498 A WildPath, represented as an asterisk (*). Note that the last
1499 element of the path should not be a wildPath (wildpaths don't
1500 work in this version).
1509 The occurrences-specification can be either the string
1510 <literal>all</literal>, the string <literal>last</literal>, or
1511 an explicit value-range. The value-range is represented as
1512 an integer (the starting point), possibly followed by a
1513 plus (+) and a second integer (the number of elements, default
1518 The variant-request has the same syntax as the defaultVariantRequest
1519 above. Note that it may sometimes be useful to give an empty variant
1520 request, simply to disable the default for a specific set of fields
1521 (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
1522 but it works in this implementation).
1524 </listitem></varlistentry>
1529 The following is an example of an element specification belonging to
1536 simpleelement (1,10)
1537 simpleelement (1,12)
1539 simpleelement (1,14)
1541 simpleelement (4,52)
1548 <sect2 id="schema-mapping">
1549 <title>The Schema Mapping (.map) Files</title>
1552 Sometimes, the client might want to receive a database record in
1553 a schema that differs from the native schema of the record. For
1554 instance, a client might only know how to process WAIS records, while
1555 the database record is represented in a more specific schema, such as
1556 GILS. In this module, a mapping of data to one of the MARC formats is
1557 also thought of as a schema mapping (mapping the elements of the
1558 record into fields consistent with the given MARC specification, prior
1559 to actually converting the data to the ISO2709). This use of the
1560 object identifier for USMARC as a schema identifier represents an
1561 overloading of the OID which might not be entirely proper. However,
1562 it represents the dual role of schema and record syntax which
1563 is assumed by the MARC family in Z39.50.
1567 <emphasis>NOTE: FIXME! The schema-mapping functions are so far limited to a
1568 straightforward mapping of elements. This should be extended with
1569 mechanisms for conversions of the element contents, and conditional
1570 mappings of elements based on the record contents.</emphasis>
1574 These are the directives of the schema mapping file format:
1581 <term>targetName <emphasis>name</emphasis></term>
1584 (m) A symbolic name for the target schema
1585 of the table. Useful mostly for diagnostic purposes.
1587 </listitem></varlistentry>
1589 <term>targetRef <emphasis>OID-name</emphasis></term>
1592 (m) An OID name for the target schema.
1593 This is used, for instance, by a server receiving a request to present
1594 a record in a different schema from the native one.
1595 The name, again, is found in the <emphasis>oid</emphasis>
1596 module of <emphasis>YAZ</emphasis>.
1598 </listitem></varlistentry>
1600 <term>map <emphasis>element-name target-path</emphasis></term>
1604 an element mapping rule to the table.
1606 </listitem></varlistentry>
1613 <title>The MARC (ISO2709) Representation (.mar) Files</title>
1616 This file provides rules for representing a record in the ISO2709
1617 format. The rules pertain mostly to the values of the constant-length
1618 header of the record.
1622 NOTE: FIXME! This will be described better. We're in the process of
1623 re-evaluating and most likely changing the way that MARC records are
1624 handled by the system.</emphasis>
1629 <sect2 id="field-structure-and-character-sets">
1630 <title>Field Structure and Character Sets
1634 In order to provide a flexible approach to national character set
1635 handling, Zebra allows the administrator to configure the set up the
1636 system to handle any 8-bit character set — including sets that
1637 require multi-octet diacritics or other multi-octet characters. The
1638 definition of a character set includes a specification of the
1639 permissible values, their sort order (this affects the display in the
1640 SCAN function), and relationships between upper- and lowercase
1641 characters. Finally, the definition includes the specification of
1642 space characters for the set.
1646 The operator can define different character sets for different fields,
1647 typical examples being standard text fields, numerical fields, and
1648 special-purpose fields such as WWW-style linkages (URx).
1652 The field types, and hence character sets, are associated with data
1653 elements by the .abs files (see above).
1654 The file <literal>default.idx</literal>
1655 provides the association between field type codes (as used in the .abs
1656 files) and the character map files (with the .chr suffix). The format
1657 of the .idx file is as follows
1664 <term>index <emphasis>field type code</emphasis></term>
1667 This directive introduces a new search index code.
1668 The argument is a one-character code to be used in the
1669 .abs files to select this particular index type. An index, roughly,
1670 corresponds to a particular structure attribute during search. Refer
1671 to <xref linkend="search"/>.
1673 </listitem></varlistentry>
1675 <term>sort <emphasis>field code type</emphasis></term>
1678 This directive introduces a
1679 sort index. The argument is a one-character code to be used in the
1680 .abs fie to select this particular index type. The corresponding
1681 use attribute must be used in the sort request to refer to this
1682 particular sort index. The corresponding character map (see below)
1683 is used in the sort process.
1685 </listitem></varlistentry>
1687 <term>completeness <emphasis>boolean</emphasis></term>
1690 This directive enables or disables complete field indexing.
1691 The value of the <emphasis>boolean</emphasis> should be 0
1692 (disable) or 1. If completeness is enabled, the index entry will
1693 contain the complete contents of the field (up to a limit), with words
1694 (non-space characters) separated by single space characters
1695 (normalized to " " on display). When completeness is
1696 disabled, each word is indexed as a separate entry. Complete subfield
1697 indexing is most useful for fields which are typically browsed (eg.
1698 titles, authors, or subjects), or instances where a match on a
1699 complete subfield is essential (eg. exact title searching). For fields
1700 where completeness is disabled, the search engine will interpret a
1701 search containing space characters as a word proximity search.
1703 </listitem></varlistentry>
1705 <term>charmap <emphasis>filename</emphasis></term>
1708 This is the filename of the character
1709 map to be used for this index for field type.
1711 </listitem></varlistentry>
1716 The contents of the character map files are structured as follows:
1723 <term>lowercase <emphasis>value-set</emphasis></term>
1726 This directive introduces the basic value set of the field type.
1727 The format is an ordered list (without spaces) of the
1728 characters which may occur in "words" of the given type.
1729 The order of the entries in the list determines the
1730 sort order of the index. In addition to single characters, the
1731 following combinations are legal:
1739 Backslashes may be used to introduce three-digit octal, or
1740 two-digit hex representations of single characters
1741 (preceded by <literal>x</literal>).
1742 In addition, the combinations
1743 \\, \\r, \\n, \\t, \\s (space — remember that real
1744 space-characters may not occur in the value definition), and
1745 \\ are recognized, with their usual interpretation.
1751 Curly braces {} may be used to enclose ranges of single
1752 characters (possibly using the escape convention described in the
1753 preceding point), eg. {a-z} to introduce the
1754 standard range of ASCII characters.
1755 Note that the interpretation of such a range depends on
1756 the concrete representation in your local, physical character set.
1762 paranthesises () may be used to enclose multi-byte characters -
1763 eg. diacritics or special national combinations (eg. Spanish
1764 "ll"). When found in the input stream (or a search term),
1765 these characters are viewed and sorted as a single character, with a
1766 sorting value depending on the position of the group in the value
1774 </listitem></varlistentry>
1776 <term>uppercase <emphasis>value-set</emphasis></term>
1779 This directive introduces the
1780 upper-case equivalencis to the value set (if any). The number and
1781 order of the entries in the list should be the same as in the
1782 <literal>lowercase</literal> directive.
1784 </listitem></varlistentry>
1786 <term>space <emphasis>value-set</emphasis></term>
1789 This directive introduces the character
1790 which separate words in the input stream. Depending on the
1791 completeness mode of the field in question, these characters either
1792 terminate an index entry, or delimit individual "words" in
1793 the input stream. The order of the elements is not significant —
1794 otherwise the representation is the same as for the
1795 <literal>uppercase</literal> and <literal>lowercase</literal>
1798 </listitem></varlistentry>
1800 <term>map <emphasis>value-set</emphasis>
1801 <emphasis>target</emphasis></term>
1804 This directive introduces a
1805 mapping between each of the members of the value-set on the left to
1806 the character on the right. The character on the right must occur in
1807 the value set (the <literal>lowercase</literal> directive) of
1808 the character set, but
1809 it may be a paranthesis-enclosed multi-octet character. This directive
1810 may be used to map diacritics to their base characters, or to map
1811 HTML-style character-representations to their natural form, etc.
1813 </listitem></varlistentry>
1821 <sect1 id="formats">
1822 <title>Exchange Formats</title>
1825 Converting records from the internal structure to en exchange format
1826 is largely an automatic process. Currently, the following exchange
1827 formats are supported:
1834 GRS-1. The internal representation is based on GRS-1/XML, so the
1835 conversion here is straightforward. The system will create
1836 applied variant and supported variant lists as required, if a record
1837 contains variant information.
1843 XML. The internal representation is based on GRS-1/XML so
1844 the mapping is trivial. Note that XML schemas, preprocessing
1845 instructions and comments are not part of the internal representation
1846 and therefore will never be part of a generated XML record.
1847 Future versions of the Zebra will support that.
1853 SUTRS. Again, the mapping is fairly straightforward. Indentation
1854 is used to show the hierarchical structure of the record. All
1855 "GRS" type records support both the GRS-1 and SUTRS
1857 <!-- FIXME - What is SUTRS - should be expanded here -->
1863 ISO2709-based formats (USMARC, etc.). Only records with a
1864 two-level structure (corresponding to fields and subfields) can be
1865 directly mapped to ISO2709. For records with a different structuring
1866 (eg., GILS), the representation in a structure like USMARC involves a
1867 schema-mapping (see <xref linkend="schema-mapping"/>), to an
1868 "implied" USMARC schema (implied,
1869 because there is no formal schema which specifies the use of the
1870 USMARC fields outside of ISO2709). The resultant, two-level record is
1871 then mapped directly from the internal representation to ISO2709. See
1872 the GILS schema definition files for a detailed example of this
1879 Explain. This representation is only available for records
1880 belonging to the Explain schema.
1886 Summary. This ASN-1 based structure is only available for records
1887 belonging to the Summary schema - or schema which provide a mapping
1888 to this schema (see the description of the schema mapping facility
1895 SOIF. Support for this syntax is experimental, and is currently
1896 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
1897 abstract syntaxes can be mapped to the SOIF format, although nested
1898 elements are represented by concatenation of the tag names at each
1900 <!-- FIXME - Is this used anywhere ? -H -->
1909 <!-- Keep this comment at the end of the file
1914 sgml-minimize-attributes:nil
1915 sgml-always-quote-attributes:t
1918 sgml-parent-document: "zebra.xml"
1919 sgml-local-catalogs: nil
1920 sgml-namecase-general:t
projects / idzebra-moved-to-github.git / blob