1 <chapter id="record-model">
2 <!-- $Id: recordmodel.xml,v 1.1 2002-04-09 13:26:26 adam Exp $ -->
3 <title>The Record Model</title>
6 The Zebra system is designed to support a wide range of data management
7 applications. The system can be configured to handle virtually any
8 kind of structured data. Each record in the system is associated with
9 a <emphasis>record schema</emphasis> which lends context to the data
10 elements of the record.
11 Any number of record schema can coexist in the system.
12 Although it may be wise to use only a single schema within
13 one database, the system poses no such restrictions.
17 The record model described in this chapter applies to the fundamental,
19 record type <literal>grs</literal> as introduced in
20 <xref linkend="record-types"/>.
24 Records pass through three different states during processing in the
34 When records are accessed by the system, they are represented
35 in their local, or native format. This might be SGML or HTML files,
36 News or Mail archives, MARC records. If the system doesn't already
37 know how to read the type of data you need to store, you can set up an
38 input filter by preparing conversion rules based on regular
39 expressions and possibly augmented by a flexible scripting language
41 The input filter produces as output an internal representation:
48 When records are processed by the system, they are represented
49 in a tree-structure, constructed by tagged data elements hanging off a
50 root node. The tagged elements may contain data or yet more tagged
51 elements in a recursive structure. The system performs various
52 actions on this tree structure (indexing, element selection, schema
60 Before transmitting records to the client, they are first
61 converted from the internal structure to a form suitable for exchange
62 over the network - according to the Z39.50 standard.
70 <sect1 id="local-representation">
71 <title>Local Representation</title>
74 As mentioned earlier, Zebra places few restrictions on the type of
75 data that you can index and manage. Generally, whatever the form of
76 the data, it is parsed by an input filter specific to that format, and
77 turned into an internal structure that Zebra knows how to handle. This
78 process takes place whenever the record is accessed - for indexing and
83 The RecordType parameter in the <literal>zebra.cfg</literal> file, or
84 the <literal>-t</literal> option to the indexer tells Zebra how to
85 process input records.
86 Two basic types of processing are available - raw text and structured
87 data. Raw text is just that, and it is selected by providing the
88 argument <emphasis>text</emphasis> to Zebra. Structured records are
89 all handled internally using the basic mechanisms described in the
91 Zebra can read structured records in many different formats.
92 How this is done is governed by additional parameters after the
93 "grs" keyboard, separated by "." characters.
97 Four basic subtypes to the <emphasis>grs</emphasis> type are
104 <term>grs.sgml</term>
107 This is the canonical input format —
108 described below. It is a simple SGML-like syntax.
113 <term>grs.regx.<emphasis>filter</emphasis></term>
116 This enables a user-supplied input
117 filter. The mechanisms of these filters are described below.
122 <term>grs.tcl.<emphasis>filter</emphasis></term>
125 Similar to grs.regx but using Tcl for rules.
130 <term>grs.marc.<emphasis>abstract syntax</emphasis></term>
133 This allows Zebra to read
134 records in the ISO2709 (MARC) encoding standard. In this case, the
135 last paramemeter <emphasis>abstract syntax</emphasis> names the
136 <literal>.abs</literal> file (see below)
137 which describes the specific MARC structure of the input record as
138 well as the indexing rules.
146 <title>Canonical Input Format</title>
149 Although input data can take any form, it is sometimes useful to
150 describe the record processing capabilities of the system in terms of
151 a single, canonical input format that gives access to the full
152 spectrum of structure and flexibility in the system. In Zebra, this
153 canonical format is an "SGML-like" syntax.
157 To use the canonical format specify <literal>grs.sgml</literal> as
162 Consider a record describing an information resource (such a record is
163 sometimes known as a <emphasis>locator record</emphasis>).
164 It might contain a field describing the distributor of the
165 information resource, which might in turn be partitioned into
166 various fields providing details about the distributor, like this:
172 <Distributor>
173 <Name> USGS/WRD </Name>
174 <Organization> USGS/WRD </Organization>
175 <Street-Address>
176 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
177 </Street-Address>
178 <City> ALBUQUERQUE </City>
179 <State> NM </State>
180 <Zip-Code> 87102 </Zip-Code>
181 <Country> USA </Country>
182 <Telephone> (505) 766-5560 </Telephone>
183 </Distributor>
190 The indentation used above is used to illustrate how Zebra
191 interprets the markup. The indentation, in itself, has no
192 significance to the parser for the canonical input format, which
193 discards superfluous whitespace.
197 The keywords surrounded by <...> are
198 <emphasis>tags</emphasis>, while the sections of text
199 in between are the <emphasis>data elements</emphasis>.
200 A data element is characterized by its location in the tree
201 that is made up by the nested elements.
202 Each element is terminated by a closing tag - beginning
203 with <literal><</literal>/, and containing the same symbolic
204 tag-name as the corresponding opening tag.
205 The general closing tag - <literal><</literal>>/ -
206 terminates the element started by the last opening tag. The
207 structuring of elements is significant.
208 The element <emphasis>Telephone</emphasis>,
209 for instance, may be indexed and presented to the client differently,
210 depending on whether it appears inside the
211 <emphasis>Distributor</emphasis> element, or some other,
212 structured data element such a <emphasis>Supplier</emphasis> element.
216 <title>Record Root</title>
219 The first tag in a record describes the root node of the tree that
220 makes up the total record. In the canonical input format, the root tag
221 should contain the name of the schema that lends context to the
222 elements of the record
223 (see <xref linkend="internal-representation"/>).
224 The following is a GILS record that
225 contains only a single element (strictly speaking, that makes it an
226 illegal GILS record, since the GILS profile includes several mandatory
227 elements - Zebra does not validate the contents of a record against
228 the Z39.50 profile, however - it merely attempts to match up elements
229 of a local representation with the given schema):
236 <title>Zen and the Art of Motorcycle Maintenance</title>
245 <title>Variants</title>
248 Zebra allows you to provide individual data elements in a number of
249 <emphasis>variant forms</emphasis>. Examples of variant forms are
250 textual data elements which might appear in different languages, and
251 images which may appear in different formats or layouts.
252 The variant system in Zebra is essentially a representation of
253 the variant mechanism of Z39.50-1995.
257 The following is an example of a title element which occurs in two
265 <var lang lang "eng">
266 Zen and the Art of Motorcycle Maintenance</>
267 <var lang lang "dan">
268 Zen og Kunsten at Vedligeholde en Motorcykel</>
275 The syntax of the <emphasis>variant element</emphasis> is
276 <literal><var class type value></literal>.
277 The available values for the <emphasis>class</emphasis> and
278 <emphasis>type</emphasis> fields are given by the variant set
279 that is associated with the current schema
280 (see <xref linkend="variant-set"/>).
284 Variant elements are terminated by the general end-tag </>, by
285 the variant end-tag </var>, by the appearance of another variant
286 tag with the same <emphasis>class</emphasis> and
287 <emphasis>value</emphasis> settings, or by the
288 appearance of another, normal tag. In other words, the end-tags for
289 the variants used in the example above could have been saved.
293 Variant elements can be nested. The element
300 <var lang lang "eng"><var body iana "text/plain">
301 Zen and the Art of Motorcycle Maintenance
308 Associates two variant components to the variant list for the title
313 Given the nesting rules described above, we could write
320 <var body iana "text/plain>
321 <var lang lang "eng">
322 Zen and the Art of Motorcycle Maintenance
323 <var lang lang "dan">
324 Zen og Kunsten at Vedligeholde en Motorcykel
331 The title element above comes in two variants. Both have the IANA body
332 type "text/plain", but one is in English, and the other in
333 Danish. The client, using the element selection mechanism of Z39.50,
334 can retrieve information about the available variant forms of data
335 elements, or it can select specific variants based on the requirements
344 <title>Input Filters</title>
347 In order to handle general input formats, Zebra allows the
348 operator to define filters which read individual records in their
349 native format and produce an internal representation that the system
354 Input filters are ASCII files, generally with the suffix
355 <literal>.flt</literal>.
356 The system looks for the files in the directories given in the
357 <emphasis>profilePath</emphasis> setting in the
358 <literal>zebra.cfg</literal> files.
359 The record type for the filter is
360 <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
361 (fundamental type <literal>grs</literal>, file read
362 type <literal>regx</literal>, argument
363 <emphasis>filter-filename</emphasis>).
367 Generally, an input filter consists of a sequence of rules, where each
368 rule consists of a sequence of expressions, followed by an action. The
369 expressions are evaluated against the contents of the input record,
370 and the actions normally contribute to the generation of an internal
371 representation of the record.
375 An expression can be either of the following:
385 The action associated with this expression is evaluated
386 exactly once in the lifetime of the application, before any records
387 are read. It can be used in conjunction with an action that
388 initializes tables or other resources that are used in the processing
397 Matches the beginning of the record. It can be used to
398 initialize variables, etc. Typically, the
399 <emphasis>BEGIN</emphasis> rule is also used
400 to establish the root node of the record.
408 Matches the end of the record - when all of the contents
409 of the record has been processed.
414 <term>/pattern/</term>
417 Matches a string of characters from the input record.
425 This keyword may only be used between two patterns.
426 It matches everything between (not including) those patterns.
434 The expression asssociated with this pattern is evaluated
435 once, before the application terminates. It can be used to release
436 system resources - typically ones allocated in the
437 <emphasis>INIT</emphasis> step.
445 An action is surrounded by curly braces ({...}), and
446 consists of a sequence of statements. Statements may be separated
447 by newlines or semicolons (;).
448 Within actions, the strings that matched the expressions
449 immediately preceding the action can be referred to as
450 $0, $1, $2, etc.
454 The available statements are:
461 <term>begin <emphasis>type [parameter ... ]</emphasis></term>
465 data element. The type is one of the following:
472 Begin a new record. The followingparameter should be the
473 name of the schema that describes the structure of the record, eg.
474 <literal>gils</literal> or <literal>wais</literal> (see below).
475 The <literal>begin record</literal> call should precede
476 any other use of the <emphasis>begin</emphasis> statement.
484 Begin a new tagged element. The parameter is the
485 name of the tag. If the tag is not matched anywhere in the tagsets
486 referenced by the current schema, it is treated as a local string
495 Begin a new node in a variant tree. The parameters are
496 <emphasis>class type value</emphasis>.
508 Create a data element. The concatenated arguments make
509 up the value of the data element.
510 The option <literal>-text</literal> signals that
511 the layout (whitespace) of the data should be retained for
513 The option <literal>-element</literal>
514 <emphasis>tag</emphasis> wraps the data up in
515 the <emphasis>tag</emphasis>.
516 The use of the <literal>-element</literal> option is equivalent to
517 preceding the command with a <emphasis>begin
518 element</emphasis> command, and following
519 it with the <emphasis>end</emphasis> command.
524 <term>end <emphasis>[type]</emphasis></term>
527 Close a tagged element. If no parameter is given,
528 the last element on the stack is terminated.
529 The first parameter, if any, is a type name, similar
530 to the <emphasis>begin</emphasis> statement.
531 For the <emphasis>element</emphasis> type, a tag
532 name can be provided to terminate a specific tag.
540 The following input filter reads a Usenet news file, producing a
541 record in the WAIS schema. Note that the body of a news posting is
542 separated from the list of headers by a blank line (or rather a
543 sequence of two newline characters.
549 BEGIN { begin record wais }
551 /^From:/ BODY /$/ { data -element name $1 }
552 /^Subject:/ BODY /$/ { data -element title $1 }
553 /^Date:/ BODY /$/ { data -element lastModified $1 }
555 begin element bodyOfDisplay
556 begin variant body iana "text/plain"
565 If Zebra is compiled with support for Tcl (Tool Command Language)
566 enabled, the statements described above are supplemented with a complete
567 scripting environment, including control structures (conditional
568 expressions and loop constructs), and powerful string manipulation
569 mechanisms for modifying the elements of a record. Tcl is a popular
570 scripting environment, with several tutorials available both online
578 <sect1 id="internal-representation">
579 <title>Internal Representation</title>
582 When records are manipulated by the system, they're represented in a
583 tree-structure, with data elements at the leaf nodes, and tags or
584 variant components at the non-leaf nodes. The root-node identifies the
585 schema that lends context to the tagging and structuring of the
586 record. Imagine a simple record, consisting of a 'title' element and
593 TITLE "Zen and the Art of Motorcycle Maintenance"
595 AUTHOR "Robert Pirsig"
601 A slightly more complex record would have the author element consist
602 of two elements, a surname and a first name:
608 TITLE "Zen and the Art of Motorcycle Maintenance"
618 The root of the record will refer to the record schema that describes
619 the structuring of this particular record. The schema defines the
620 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
621 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
622 addition, the schema establishes element set names that are used by
623 the client to request a subset of the elements of a given record. The
624 schema may also establish rules for converting the record to a
625 different schema, by stating, for each element, a mapping to a
630 <title>Tagged Elements</title>
633 A data element is characterized by its tag, and its position in the
634 structure of the record. For instance, while the tag "telephone
635 number" may be used different places in a record, we may need to
636 distinguish between these occurrences, both for searching and
637 presentation purposes. For instance, while the phone numbers for the
638 "customer" and the "service provider" are both
639 representatives for the same type of resource (a telephone number), it
640 is essential that they be kept separate. The record schema provides
641 the structure of the record, and names each data element (defined by
642 the sequence of tags - the tag path - by which the element can be
643 reached from the root of the record).
649 <title>Variants</title>
652 The children of a tag node may be either more tag nodes, a data node
653 (possibly accompanied by tag nodes),
654 or a tree of variant nodes. The children of variant nodes are either
655 more variant nodes or a data node (possibly accompanied by more
656 variant nodes). Each leaf node, which is normally a
657 data node, corresponds to a <emphasis>variant form</emphasis> of the
658 tagged element identified by the tag which parents the variant tree.
659 The following title element occurs in two different languages:
665 VARIANT LANG=ENG "War and Peace"
667 VARIANT LANG=DAN "Krig og Fred"
673 Which of the two elements are transmitted to the client by the server
674 depends on the specifications provided by the client, if any.
678 In practice, each variant node is associated with a triple of class,
679 type, value, corresponding to the variant mechanism of Z39.50.
685 <title>Data Elements</title>
688 Data nodes have no children (they are always leaf nodes in the record
694 Documentation needs extension here about types of nodes - numerical,
695 textual, etc., plus the various types of inclusion notes.
703 <sect1 id="data-model">
704 <title>Configuring Your Data Model</title>
707 The following sections describe the configuration files that govern
708 the internal management of data records. The system searches for the files
709 in the directories specified by the <emphasis>profilePath</emphasis>
710 setting in the <literal>zebra.cfg</literal> file.
714 <title>The Abstract Syntax</title>
717 The abstract syntax definition (also known as an Abstract Record
718 Structure, or ARS) is the focal point of the
719 record schema description. For a given schema, the ABS file may state any
720 or all of the following:
729 The object identifier of the Z39.50 schema associated
730 with the ARS, so that it can be referred to by the client.
736 The attribute set (which can possibly be a compound of multiple
737 sets) which applies in the profile. This is used when indexing and
738 searching the records belonging to the given profile.
744 The Tag set (again, this can consist of several different sets).
745 This is used when reading the records from a file, to recognize the
746 different tags, and when transmitting the record to the client -
747 mapping the tags to their numerical representation, if they are
754 The variant set which is used in the profile. This provides a
755 vocabulary for specifying the <emphasis>forms</emphasis> of data that appear inside
762 Element set names, which are a shorthand way for the client to
763 ask for a subset of the data elements contained in a record. Element
764 set names, in the retrieval module, are mapped to <emphasis>element
765 specifications</emphasis>, which contain information equivalent to the
766 <emphasis>Espec-1</emphasis> syntax of Z39.50.
772 Map tables, which may specify mappings to
773 <emphasis>other</emphasis> database profiles, if desired.
779 Possibly, a set of rules describing the mapping of elements to a
787 A list of element descriptions (this is the actual ARS of the
788 schema, in Z39.50 terms), which lists the ways in which the various
789 tags can be used and organized hierarchically.
798 Several of the entries above simply refer to other files, which
799 describe the given objects.
805 <title>The Configuration Files</title>
808 This section describes the syntax and use of the various tables which
809 are used by the retrieval module.
813 The number of different file types may appear daunting at first, but
814 each type corresponds fairly clearly to a single aspect of the Z39.50
815 retrieval facilities. Further, the average database administrator,
816 who is simply reusing an existing profile for which tables already
817 exist, shouldn't have to worry too much about the contents of these tables.
821 Generally, the files are simple ASCII files, which can be maintained
822 using any text editor. Blank lines, and lines beginning with a (#) are
823 ignored. Any characters on a line followed by a (#) are also ignored.
824 All other lines contain <emphasis>directives</emphasis>, which provide
825 some setting or value to the system.
826 Generally, settings are characterized by a single
827 keyword, identifying the setting, followed by a number of parameters.
828 Some settings are repeatable (r), while others may occur only once in a
829 file. Some settings are optional (o), whicle others again are
836 <title>The Abstract Syntax (.abs) Files</title>
839 The name of this file type is slightly misleading in Z39.50 terms,
840 since, apart from the actual abstract syntax of the profile, it also
841 includes most of the other definitions that go into a database
846 When a record in the canonical, SGML-like format is read from a file
847 or from the database, the first tag of the file should reference the
848 profile that governs the layout of the record. If the first tag of the
849 record is, say, <literal><gils></literal>, the system will look
850 for the profile definition in the file <literal>gils.abs</literal>.
851 Profile definitions are cached, so they only have to be read once
852 during the lifespan of the current process.
856 When writing your own input filters, the
857 <emphasis>record-begin</emphasis> command
858 introduces the profile, and should always be called first thing when
859 introducing a new record.
863 The file may contain the following directives:
870 <term>name <emphasis>symbolic-name</emphasis></term>
873 (m) This provides a shorthand name or
874 description for the profile. Mostly useful for diagnostic purposes.
879 <term>reference <emphasis>OID-name</emphasis></term>
882 (m) The reference name of the OID for the profile.
883 The reference names can be found in the <emphasis>util</emphasis>
884 module of <emphasis>YAZ</emphasis>.
889 <term>attset <emphasis>filename</emphasis></term>
892 (m) The attribute set that is used for
893 indexing and searching records belonging to this profile.
898 <term>tagset <emphasis>filename</emphasis></term>
901 (o) The tag set (if any) that describe
902 that fields of the records.
907 <term>varset <emphasis>filename</emphasis></term>
910 (o) The variant set used in the profile.
915 <term>maptab <emphasis>filename</emphasis></term>
918 (o,r) This points to a
919 conversion table that might be used if the client asks for the record
920 in a different schema from the native one.
922 </listitem></varlistentry>
924 <term>marc <emphasis>filename</emphasis></term>
927 (o) Points to a file containing parameters
928 for representing the record contents in the ISO2709 syntax. Read the
929 description of the MARC representation facility below.
931 </listitem></varlistentry>
933 <term>esetname <emphasis>name filename</emphasis></term>
937 given element set name with an element selection file. If an (@) is
938 given in place of the filename, this corresponds to a null mapping for
939 the given element set name.
941 </listitem></varlistentry>
943 <term>any <emphasis>tags</emphasis></term>
946 (o) This directive specifies a list of attributes
947 which should be appended to the attribute list given for each
948 element. The effect is to make every single element in the abstract
949 syntax searchable by way of the given attributes. This directive
950 provides an efficient way of supporting free-text searching across all
951 elements. However, it does increase the size of the index
952 significantly. The attributes can be qualified with a structure, as in
953 the <emphasis>elm</emphasis> directive below.
955 </listitem></varlistentry>
957 <term>elm <emphasis>path name attributes</emphasis></term>
960 (o,r) Adds an element to the abstract record syntax of the schema.
961 The <emphasis>path</emphasis> follows the
962 syntax which is suggested by the Z39.50 document - that is, a sequence
963 of tags separated by slashes (/). Each tag is given as a
964 comma-separated pair of tag type and -value surrounded by parenthesis.
965 The <emphasis>name</emphasis> is the name of the element, and
966 the <emphasis>attributes</emphasis>
967 specifies which attributes to use when indexing the element in a
968 comma-separated list.
969 A ! in place of the attribute name is equivalent to
970 specifying an attribute name identical to the element name.
971 A - in place of the attribute name
972 specifies that no indexing is to take place for the given element.
973 The attributes can be qualified with <emphasis>field
974 types</emphasis> to specify which
975 character set should govern the indexing procedure for that field.
976 The same data element may be indexed into several different
977 fields, using different character set definitions.
978 See the <xref linkend="field-structure-and-character-sets"/>.
979 The default field type is "w" for <emphasis>word</emphasis>.
981 </listitem></varlistentry>
987 The mechanism for controlling indexing is not adequate for
988 complex databases, and will probably be moved into a separate
989 configuration table eventually.
994 The following is an excerpt from the abstract syntax file for the GILS
1002 reference GILS-schema
1007 maptab gils-usmarc.map
1011 esetname VARIANT gils-variant.est # for WAIS-compliance
1012 esetname B gils-b.est
1013 esetname G gils-g.est
1018 elm (1,14) localControlNumber Local-number
1019 elm (1,16) dateOfLastModification Date/time-last-modified
1020 elm (2,1) title w:!,p:!
1021 elm (4,1) controlIdentifier Identifier-standard
1022 elm (2,6) abstract Abstract
1023 elm (4,51) purpose !
1024 elm (4,52) originator -
1025 elm (4,53) accessConstraints !
1026 elm (4,54) useConstraints !
1027 elm (4,70) availability -
1028 elm (4,70)/(4,90) distributor -
1029 elm (4,70)/(4,90)/(2,7) distributorName !
1030 elm (4,70)/(4,90)/(2,10 distributorOrganization !
1031 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1032 elm (4,70)/(4,90)/(4,3) distributorCity !
1039 <sect2 id="attset-files">
1040 <title>The Attribute Set (.att) Files</title>
1043 This file type describes the <emphasis>Use</emphasis> elements of
1045 It contains the following directives.
1051 <term>name <emphasis>symbolic-name</emphasis></term>
1054 (m) This provides a shorthand name or
1055 description for the attribute set.
1056 Mostly useful for diagnostic purposes.
1058 </listitem></varlistentry>
1060 <term>reference <emphasis>OID-name</emphasis></term>
1063 (m) The reference name of the OID for
1065 The reference names can be found in the <emphasis>util</emphasis>
1066 module of <emphasis>YAZ</emphasis>.
1068 </listitem></varlistentry>
1070 <term>include <emphasis>filename</emphasis></term>
1073 (o,r) This directive is used to
1074 include another attribute set as a part of the current one. This is
1075 used when a new attribute set is defined as an extension to another
1076 set. For instance, many new attribute sets are defined as extensions
1077 to the <emphasis>bib-1</emphasis> set.
1078 This is an important feature of the retrieval
1079 system of Z39.50, as it ensures the highest possible level of
1080 interoperability, as those access points of your database which are
1081 derived from the external set (say, bib-1) can be used even by clients
1082 who are unaware of the new set.
1084 </listitem></varlistentry>
1087 <emphasis>att-value att-name [local-value]</emphasis></term>
1091 repeatable directive introduces a new attribute to the set. The
1092 attribute value is stored in the index (unless a
1093 <emphasis>local-value</emphasis> is
1094 given, in which case this is stored). The name is used to refer to the
1095 attribute from the <emphasis>abstract syntax</emphasis>.
1097 </listitem></varlistentry>
1102 This is an excerpt from the GILS attribute set definition.
1103 Notice how the file describing the <emphasis>bib-1</emphasis>
1104 attribute set is referenced.
1111 reference GILS-attset
1114 att 2001 distributorName
1115 att 2002 indextermsControlled
1117 att 2004 accessConstraints
1118 att 2005 useConstraints
1126 <title>The Tag Set (.tag) Files</title>
1129 This file type defines the tagset of the profile, possibly by
1130 referencing other tag sets (most tag sets, for instance, will include
1131 tagsetG and tagsetM from the Z39.50 specification. The file may
1132 contain the following directives.
1139 <term>name <emphasis>symbolic-name</emphasis></term>
1142 (m) This provides a shorthand name or
1143 description for the tag set. Mostly useful for diagnostic purposes.
1145 </listitem></varlistentry>
1147 <term>reference <emphasis>OID-name</emphasis></term>
1150 (o) The reference name of the OID for the tag set.
1151 The reference names can be found in the <emphasis>util</emphasis>
1152 module of <emphasis>YAZ</emphasis>.
1153 The directive is optional, since not all tag sets
1154 are registered outside of their schema.
1156 </listitem></varlistentry>
1158 <term>type <emphasis>integer</emphasis></term>
1161 (m) The type number of the tagset within the schema
1162 profile (note: this specification really should belong to the .abs
1163 file. This will be fixed in a future release).
1165 </listitem></varlistentry>
1167 <term>include <emphasis>filename</emphasis></term>
1170 (o,r) This directive is used
1171 to include the definitions of other tag sets into the current one.
1173 </listitem></varlistentry>
1175 <term>tag <emphasis>number names type</emphasis></term>
1178 (o,r) Introduces a new tag to the set.
1179 The <emphasis>number</emphasis> is the tag number as used
1180 in the protocol (there is currently no mechanism for
1181 specifying string tags at this point, but this would be quick
1183 The <emphasis>names</emphasis> parameter is a list of names
1184 by which the tag should be recognized in the input file format.
1185 The names should be separated by slashes (/).
1186 The <emphasis>type</emphasis> is th recommended datatype of
1188 It should be one of the following:
1254 </listitem></varlistentry>
1259 The following is an excerpt from the TagsetG definition file.
1270 tag 3 publicationPlace string
1271 tag 4 publicationDate string
1272 tag 5 documentId string
1273 tag 6 abstract string
1275 tag 8 date generalizedtime
1276 tag 9 bodyOfDisplay string
1277 tag 10 organization string
1283 <sect2 id="variant-set">
1284 <title>The Variant Set (.var) Files</title>
1287 The variant set file is a straightforward representation of the
1288 variant set definitions associated with the protocol. At present, only
1289 the <emphasis>Variant-1</emphasis> set is known.
1293 These are the directives allowed in the file.
1300 <term>name <emphasis>symbolic-name</emphasis></term>
1303 (m) This provides a shorthand name or
1304 description for the variant set. Mostly useful for diagnostic purposes.
1306 </listitem></varlistentry>
1308 <term>reference <emphasis>OID-name</emphasis></term>
1311 (o) The reference name of the OID for
1312 the variant set, if one is required. The reference names can be found
1313 in the <emphasis>util</emphasis> module of <emphasis>YAZ</emphasis>.
1315 </listitem></varlistentry>
1317 <term>class <emphasis>integer class-name</emphasis></term>
1320 (m,r) Introduces a new
1321 class to the variant set.
1323 </listitem></varlistentry>
1325 <term>type <emphasis>integer type-name datatype</emphasis></term>
1329 new type to the current class (the one introduced by the most recent
1330 <emphasis>class</emphasis> directive).
1331 The type names belong to the same name space as the one used
1332 in the tag set definition file.
1334 </listitem></varlistentry>
1339 The following is an excerpt from the file describing the variant set
1340 <emphasis>Variant-1</emphasis>.
1351 type 1 variantId octetstring
1356 type 2 z39.50 string
1365 <title>The Element Set (.est) Files</title>
1368 The element set specification files describe a selection of a subset
1369 of the elements of a database record. The element selection mechanism
1370 is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
1371 syntax of the Z39.50 specification.
1372 In fact, the internal representation of an element set
1373 specification is identical to the <emphasis>Espec-1</emphasis> structure,
1374 and we'll refer you to the description of that structure for most of
1375 the detailed semantics of the directives below.
1380 Not all of the Espec-1 functionality has been implemented yet.
1381 The fields that are mentioned below all work as expected, unless
1387 The directives available in the element set file are as follows:
1393 <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
1396 (o) If variants are used in
1397 the following, this should provide the name of the variantset used
1398 (it's not currently possible to specify a different set in the
1399 individual variant request). In almost all cases (certainly all
1400 profiles known to us), the name
1401 <literal>Variant-1</literal> should be given here.
1403 </listitem></varlistentry>
1405 <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
1409 provides a default variant request for
1410 use when the individual element requests (see below) do not contain a
1411 variant request. Variant requests consist of a blank-separated list of
1412 variant components. A variant compont is a comma-separated,
1413 parenthesized triple of variant class, type, and value (the two former
1414 values being represented as integers). The value can currently only be
1415 entered as a string (this will change to depend on the definition of
1416 the variant in question). The special value (@) is interpreted as a
1417 null value, however.
1419 </listitem></varlistentry>
1422 <emphasis>path ['variant' variant-request]</emphasis></term>
1425 (o,r) This corresponds to a simple element request
1426 in <emphasis>Espec-1</emphasis>.
1427 The path consists of a sequence of tag-selectors, where each of
1428 these can consist of either:
1435 A simple tag, consisting of a comma-separated type-value pair in
1436 parenthesis, possibly followed by a colon (:) followed by an
1437 occurrences-specification (see below). The tag-value can be a number
1438 or a string. If the first character is an apostrophe ('), this
1439 forces the value to be interpreted as a string, even if it
1440 appears to be numerical.
1446 A WildThing, represented as a question mark (?), possibly
1447 followed by a colon (:) followed by an occurrences
1448 specification (see below).
1454 A WildPath, represented as an asterisk (*). Note that the last
1455 element of the path should not be a wildPath (wildpaths don't
1456 work in this version).
1465 The occurrences-specification can be either the string
1466 <literal>all</literal>, the string <literal>last</literal>, or
1467 an explicit value-range. The value-range is represented as
1468 an integer (the starting point), possibly followed by a
1469 plus (+) and a second integer (the number of elements, default
1474 The variant-request has the same syntax as the defaultVariantRequest
1475 above. Note that it may sometimes be useful to give an empty variant
1476 request, simply to disable the default for a specific set of fields
1477 (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
1478 but it works in this implementation).
1480 </listitem></varlistentry>
1485 The following is an example of an element specification belonging to
1492 simpleelement (1,10)
1493 simpleelement (1,12)
1495 simpleelement (1,14)
1497 simpleelement (4,52)
1504 <sect2 id="schema-mapping">
1505 <title>The Schema Mapping (.map) Files</title>
1508 Sometimes, the client might want to receive a database record in
1509 a schema that differs from the native schema of the record. For
1510 instance, a client might only know how to process WAIS records, while
1511 the database record is represented in a more specific schema, such as
1512 GILS. In this module, a mapping of data to one of the MARC formats is
1513 also thought of as a schema mapping (mapping the elements of the
1514 record into fields consistent with the given MARC specification, prior
1515 to actually converting the data to the ISO2709). This use of the
1516 object identifier for USMARC as a schema identifier represents an
1517 overloading of the OID which might not be entirely proper. However,
1518 it represents the dual role of schema and record syntax which
1519 is assumed by the MARC family in Z39.50.
1523 <emphasis>NOTE: The schema-mapping functions are so far limited to a
1524 straightforward mapping of elements. This should be extended with
1525 mechanisms for conversions of the element contents, and conditional
1526 mappings of elements based on the record contents.</emphasis>
1530 These are the directives of the schema mapping file format:
1537 <term>targetName <emphasis>name</emphasis></term>
1540 (m) A symbolic name for the target schema
1541 of the table. Useful mostly for diagnostic purposes.
1543 </listitem></varlistentry>
1545 <term>targetRef <emphasis>OID-name</emphasis></term>
1548 (m) An OID name for the target schema.
1549 This is used, for instance, by a server receiving a request to present
1550 a record in a different schema from the native one.
1551 The name, again, is found in the <emphasis>oid</emphasis>
1552 module of <emphasis>YAZ</emphasis>.
1554 </listitem></varlistentry>
1556 <term>map <emphasis>element-name target-path</emphasis></term>
1560 an element mapping rule to the table.
1562 </listitem></varlistentry>
1569 <title>The MARC (ISO2709) Representation (.mar) Files</title>
1572 This file provides rules for representing a record in the ISO2709
1573 format. The rules pertain mostly to the values of the constant-length
1574 header of the record.
1578 <emphasis>NOTE: This will be described better. We're in the process of
1579 re-evaluating and most likely changing the way that MARC records are
1580 handled by the system.</emphasis>
1585 <sect2 id="field-structure-and-character-sets">
1586 <title>Field Structure and Character Sets
1590 In order to provide a flexible approach to national character set
1591 handling, Zebra allows the administrator to configure the set up the
1592 system to handle any 8-bit character set — including sets that
1593 require multi-octet diacritics or other multi-octet characters. The
1594 definition of a character set includes a specification of the
1595 permissible values, their sort order (this affects the display in the
1596 SCAN function), and relationships between upper- and lowercase
1597 characters. Finally, the definition includes the specification of
1598 space characters for the set.
1602 The operator can define different character sets for different fields,
1603 typical examples being standard text fields, numerical fields, and
1604 special-purpose fields such as WWW-style linkages (URx).
1608 The field types, and hence character sets, are associated with data
1609 elements by the .abs files (see above).
1610 The file <literal>default.idx</literal>
1611 provides the association between field type codes (as used in the .abs
1612 files) and the character map files (with the .chr suffix). The format
1613 of the .idx file is as follows
1620 <term>index <emphasis>field type code</emphasis></term>
1623 This directive introduces a new search index code.
1624 The argument is a one-character code to be used in the
1625 .abs files to select this particular index type. An index, roughly,
1626 corresponds to a particular structure attribute during search. Refer
1627 to <xref linkend="search"/>.
1629 </listitem></varlistentry>
1631 <term>sort <emphasis>field code type</emphasis></term>
1634 This directive introduces a
1635 sort index. The argument is a one-character code to be used in the
1636 .abs fie to select this particular index type. The corresponding
1637 use attribute must be used in the sort request to refer to this
1638 particular sort index. The corresponding character map (see below)
1639 is used in the sort process.
1641 </listitem></varlistentry>
1643 <term>completeness <emphasis>boolean</emphasis></term>
1646 This directive enables or disables complete field indexing.
1647 The value of the <emphasis>boolean</emphasis> should be 0
1648 (disable) or 1. If completeness is enabled, the index entry will
1649 contain the complete contents of the field (up to a limit), with words
1650 (non-space characters) separated by single space characters
1651 (normalized to " " on display). When completeness is
1652 disabled, each word is indexed as a separate entry. Complete subfield
1653 indexing is most useful for fields which are typically browsed (eg.
1654 titles, authors, or subjects), or instances where a match on a
1655 complete subfield is essential (eg. exact title searching). For fields
1656 where completeness is disabled, the search engine will interpret a
1657 search containing space characters as a word proximity search.
1659 </listitem></varlistentry>
1661 <term>charmap <emphasis>filename</emphasis></term>
1664 This is the filename of the character
1665 map to be used for this index for field type.
1667 </listitem></varlistentry>
1672 The contents of the character map files are structured as follows:
1679 <term>lowercase <emphasis>value-set</emphasis></term>
1682 This directive introduces the basic value set of the field type.
1683 The format is an ordered list (without spaces) of the
1684 characters which may occur in "words" of the given type.
1685 The order of the entries in the list determines the
1686 sort order of the index. In addition to single characters, the
1687 following combinations are legal:
1695 Backslashes may be used to introduce three-digit octal, or
1696 two-digit hex representations of single characters
1697 (preceded by <literal>x</literal>).
1698 In addition, the combinations
1699 \\, \\r, \\n, \\t, \\s (space — remember that real
1700 space-characters may ot occur in the value definition), and
1701 \\ are recognised, with their usual interpretation.
1707 Curly braces {} may be used to enclose ranges of single
1708 characters (possibly using the escape convention described in the
1709 preceding point), eg. {a-z} to entroduce the
1710 standard range of ASCII characters.
1711 Note that the interpretation of such a range depends on
1712 the concrete representation in your local, physical character set.
1718 paranthesises () may be used to enclose multi-byte characters -
1719 eg. diacritics or special national combinations (eg. Spanish
1720 "ll"). When found in the input stream (or a search term),
1721 these characters are viewed and sorted as a single character, with a
1722 sorting value depending on the position of the group in the value
1730 </listitem></varlistentry>
1732 <term>uppercase <emphasis>value-set</emphasis></term>
1735 This directive introduces the
1736 upper-case equivalencis to the value set (if any). The number and
1737 order of the entries in the list should be the same as in the
1738 <literal>lowercase</literal> directive.
1740 </listitem></varlistentry>
1742 <term>space <emphasis>value-set</emphasis></term>
1745 This directive introduces the character
1746 which separate words in the input stream. Depending on the
1747 completeness mode of the field in question, these characters either
1748 terminate an index entry, or delimit individual "words" in
1749 the input stream. The order of the elements is not significant —
1750 otherwise the representation is the same as for the
1751 <literal>uppercase</literal> and <literal>lowercase</literal>
1754 </listitem></varlistentry>
1756 <term>map <emphasis>value-set</emphasis>
1757 <emphasis>target</emphasis></term>
1760 This directive introduces a
1761 mapping between each of the members of the value-set on the left to
1762 the character on the right. The character on the right must occur in
1763 the value set (the <literal>lowercase</literal> directive) of
1764 the character set, but
1765 it may be a paranthesis-enclosed multi-octet character. This directive
1766 may be used to map diacritics to their base characters, or to map
1767 HTML-style character-representations to their natural form, etc.
1769 </listitem></varlistentry>
1777 <sect1 id="formats">
1778 <title>Exchange Formats</title>
1781 Converting records from the internal structure to en exchange format
1782 is largely an automatic process. Currently, the following exchange
1783 formats are supported:
1790 GRS-1. The internal representation is based on GRS-1/XML, so the
1791 conversion here is straightforward. The system will create
1792 applied variant and supported variant lists as required, if a record
1793 contains variant information.
1799 XML. The internal representation is based on GRS-1/XML so
1800 the mapping is trivial. Note that XML schemas, preprocessing
1801 instructions and comments are not part of the internal representation
1802 and therefore will never be part of a generated XML record.
1803 Future versions of the Zebra will support that.
1809 SUTRS. Again, the mapping is fairly straighforward. Indentation
1810 is used to show the hierarchical structure of the record. All
1811 "GRS" type records support both the GRS-1 and SUTRS
1818 ISO2709-based formats (USMARC, etc.). Only records with a
1819 two-level structure (corresponding to fields and subfields) can be
1820 directly mapped to ISO2709. For records with a different structuring
1821 (eg., GILS), the representation in a structure like USMARC involves a
1822 schema-mapping (see <xref linkend="schema-mapping"/>), to an
1823 "implied" USMARC schema (implied,
1824 because there is no formal schema which specifies the use of the
1825 USMARC fields outside of ISO2709). The resultant, two-level record is
1826 then mapped directly from the internal representation to ISO2709. See
1827 the GILS schema definition files for a detailed example of this
1834 Explain. This representation is only available for records
1835 belonging to the Explain schema.
1841 Summary. This ASN-1 based structure is only available for records
1842 belonging to the Summary schema - or schema which provide a mapping
1843 to this schema (see the description of the schema mapping facility
1850 SOIF. Support for this syntax is experimental, and is currently
1851 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
1852 abstract syntaxes can be mapped to the SOIF format, although nested
1853 elements are represented by concatenation of the tag names at each
1863 <!-- Keep this comment at the end of the file
1868 sgml-minimize-attributes:nil
1869 sgml-always-quote-attributes:t
1872 sgml-parent-document: "zebra.xml"
1873 sgml-local-catalogs: nil
1874 sgml-namecase-general:t
projects / idzebra-moved-to-github.git / blob