--- /dev/null
+ <chapter id="record-model-grs">
+ <!-- $Id: recordmodel-grs.xml,v 1.1 2006-02-15 11:07:47 marc Exp $ -->
+ <title>GRS Record Model and Filter Modules</title>
+
+
+ <para>
+ The record model described in this chapter applies to the fundamental,
+ structured
+ record type <literal>grs</literal>, introduced in
+ <xref linkend="componentmodulesgrs"/>.
+ </para>
+
+
+ <sect1 id="grs-record-filters">
+ <title>GRS Record Filters</title>
+ <para>
+ Many basic subtypes of the <emphasis>grs</emphasis> type are
+ currently available:
+ </para>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term>grs.sgml</term>
+ <listitem>
+ <para>
+ This is the canonical input format
+ described <xref linkend="grs-canonical-format"/>. It is using
+ simple SGML-like syntax.
+ </para>
+ <!--
+ <para>
+ <literal>libidzebra1.4-mod-grs-sgml not packaged yet ??</literal>
+ </para>
+ -->
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>grs.marc<!--.<emphasis>abstract syntax</emphasis>--></term>
+ <listitem>
+ <para>
+ This allows Zebra to read
+ records in the ISO2709 (MARC) encoding standard.
+ <!-- In this case, the
+ last parameter <emphasis>abstract syntax</emphasis> names the
+ <literal>.abs</literal> file (see below)
+ which describes the specific MARC structure of the input record as
+ well as the indexing rules. -->
+ </para>
+ <para>
+ The loadable <literal>grs.marc</literal> filter module
+ is packaged in the GNU/Debian package
+ <literal>libidzebra1.4-mod-grs-marc</literal>
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>grs.marcxml<!--.<emphasis>abstract syntax</emphasis>--></term>
+ <listitem>
+ <para>
+ This allows Zebra to read
+ records in the ISO2709??? (MARCXML) encoding standard.
+ </para>
+ <para>
+ The loadable <literal>grs.marcxml</literal> filter module
+ is also contained in the GNU/Debian package
+ <literal>libidzebra1.4-mod-grs-marc</literal>
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>grs.danbib</term>
+ <listitem>
+ <para>
+ The <literal>grs.danbib</literal> filter parses DanBib
+ records, a danish MARC record variant called DANMARC.
+ DanBib is the Danish Union Catalogue hosted by the
+ Danish Bibliographic Centre (DBC).
+ </para>
+ <para>The loadable <literal>grs.danbib</literal> filter module
+ is packages in the GNU/Debian package
+ <literal>libidzebra1.4-mod-grs-danbib</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>grs.xml</term>
+ <listitem>
+ <para>
+ This filter reads XML records and uses <ulink url="http://expat.sourceforge.net/">Expat</ulink> to
+ parse them and convert them into IDZebra's internal
+ <literal>grs</literal> record model.
+ Only one record per file
+ is supported. The filter is only available if Zebra/YAZ
+ is compiled with EXPAT support.
+ </para>
+ <para>
+ The loadable <literal>grs.xml</literal> filter module
+ is packagged in the GNU/Debian package
+ <literal>libidzebra1.4-mod-grs-xml</literal>
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>grs.regx<!--.<emphasis>filter</emphasis>--></term>
+ <listitem>
+ <para>
+ This enables a user-supplied Regular Expressions input
+ filter described in
+ <xref linkend="grs-regx-tcl"/>.
+ </para>
+ <para>
+ The loadable <literal>grs.regx</literal> filter module
+ is packaged in the GNU/Debian package
+ <literal>libidzebra1.4-mod-grs-regx</literal>
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>grs.tcl<!--.<emphasis>filter</emphasis>--></term>
+ <listitem>
+ <para>
+ Similar to grs.regx but using Tcl for rules, described in
+ <xref linkend="grs-regx-tcl"/>.
+ </para>
+ <para>
+ The loadable <literal>grs.tcl</literal> filter module
+ is also packaged in the GNU/Debian package
+ <literal>libidzebra1.4-mod-grs-regx</literal>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <sect2 id="grs-canonical-format">
+ <title>GRS Canonical Input Format</title>
+
+ <para>
+ Although input data can take any form, it is sometimes useful to
+ describe the record processing capabilities of the system in terms of
+ a single, canonical input format that gives access to the full
+ spectrum of structure and flexibility in the system. In Zebra, this
+ canonical format is an "SGML-like" syntax.
+ </para>
+
+ <para>
+ To use the canonical format specify <literal>grs.sgml</literal> as
+ the record type.
+ </para>
+
+ <para>
+ Consider a record describing an information resource (such a record is
+ sometimes known as a <emphasis>locator record</emphasis>).
+ It might contain a field describing the distributor of the
+ information resource, which might in turn be partitioned into
+ various fields providing details about the distributor, like this:
+ </para>
+
+ <para>
+
+ <screen>
+ <Distributor>
+ <Name> USGS/WRD </Name>
+ <Organization> USGS/WRD </Organization>
+ <Street-Address>
+ U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
+ </Street-Address>
+ <City> ALBUQUERQUE </City>
+ <State> NM </State>
+ <Zip-Code> 87102 </Zip-Code>
+ <Country> USA </Country>
+ <Telephone> (505) 766-5560 </Telephone>
+ </Distributor>
+ </screen>
+
+ </para>
+
+ <!-- There is no indentation in the example above! -H
+ -note-
+ -para-
+ The indentation used above is used to illustrate how Zebra
+ interprets the mark-up. The indentation, in itself, has no
+ significance to the parser for the canonical input format, which
+ discards superfluous whitespace.
+ -/para-
+ -/note-
+ -->
+
+ <para>
+ The keywords surrounded by <...> are
+ <emphasis>tags</emphasis>, while the sections of text
+ in between are the <emphasis>data elements</emphasis>.
+ A data element is characterized by its location in the tree
+ that is made up by the nested elements.
+ Each element is terminated by a closing tag - beginning
+ with <literal><</literal>/, and containing the same symbolic
+ tag-name as the corresponding opening tag.
+ The general closing tag - <literal></></literal> -
+ terminates the element started by the last opening tag. The
+ structuring of elements is significant.
+ The element <emphasis>Telephone</emphasis>,
+ for instance, may be indexed and presented to the client differently,
+ depending on whether it appears inside the
+ <emphasis>Distributor</emphasis> element, or some other,
+ structured data element such a <emphasis>Supplier</emphasis> element.
+ </para>
+
+ <sect3>
+ <title>Record Root</title>
+
+ <para>
+ The first tag in a record describes the root node of the tree that
+ makes up the total record. In the canonical input format, the root tag
+ should contain the name of the schema that lends context to the
+ elements of the record
+ (see <xref linkend="grs-internal-representation"/>).
+ The following is a GILS record that
+ contains only a single element (strictly speaking, that makes it an
+ illegal GILS record, since the GILS profile includes several mandatory
+ elements - Zebra does not validate the contents of a record against
+ the Z39.50 profile, however - it merely attempts to match up elements
+ of a local representation with the given schema):
+ </para>
+
+ <para>
+
+ <screen>
+ <gils>
+ <title>Zen and the Art of Motorcycle Maintenance</title>
+ </gils>
+ </screen>
+
+ </para>
+
+ </sect3>
+
+ <sect3><!-- ### we shouldn't make such a big deal about this -->
+ <title>Variants</title>
+
+ <para>
+ Zebra allows you to provide individual data elements in a number of
+ <emphasis>variant forms</emphasis>. Examples of variant forms are
+ textual data elements which might appear in different languages, and
+ images which may appear in different formats or layouts.
+ The variant system in Zebra is essentially a representation of
+ the variant mechanism of Z39.50-1995.
+ </para>
+
+ <para>
+ The following is an example of a title element which occurs in two
+ different languages.
+ </para>
+
+ <para>
+
+ <screen>
+ <title>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance</>
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel</>
+ </title>
+ </screen>
+
+ </para>
+
+ <para>
+ The syntax of the <emphasis>variant element</emphasis> is
+ <literal><var class type value></literal>.
+ The available values for the <emphasis>class</emphasis> and
+ <emphasis>type</emphasis> fields are given by the variant set
+ that is associated with the current schema
+ (see <xref linkend="variant-set"/>).
+ </para>
+
+ <para>
+ Variant elements are terminated by the general end-tag </>, by
+ the variant end-tag </var>, by the appearance of another variant
+ tag with the same <emphasis>class</emphasis> and
+ <emphasis>value</emphasis> settings, or by the
+ appearance of another, normal tag. In other words, the end-tags for
+ the variants used in the example above could have been omitted.
+ </para>
+
+ <para>
+ Variant elements can be nested. The element
+ </para>
+
+ <para>
+
+ <screen>
+ <title>
+ <var lang lang "eng"><var body iana "text/plain">
+ Zen and the Art of Motorcycle Maintenance
+ </title>
+ </screen>
+
+ </para>
+
+ <para>
+ Associates two variant components to the variant list for the title
+ element.
+ </para>
+
+ <para>
+ Given the nesting rules described above, we could write
+ </para>
+
+ <para>
+
+ <screen>
+ <title>
+ <var body iana "text/plain>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel
+ </title>
+ </screen>
+
+ </para>
+
+ <para>
+ The title element above comes in two variants. Both have the IANA body
+ type "text/plain", but one is in English, and the other in
+ Danish. The client, using the element selection mechanism of Z39.50,
+ can retrieve information about the available variant forms of data
+ elements, or it can select specific variants based on the requirements
+ of the end-user.
+ </para>
+
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="grs-regx-tcl">
+ <title>GRS REGX And TCL Input Filters</title>
+
+ <para>
+ In order to handle general input formats, Zebra allows the
+ operator to define filters which read individual records in their
+ native format and produce an internal representation that the system
+ can work with.
+ </para>
+
+ <para>
+ Input filters are ASCII files, generally with the suffix
+ <literal>.flt</literal>.
+ The system looks for the files in the directories given in the
+ <emphasis>profilePath</emphasis> setting in the
+ <literal>zebra.cfg</literal> files.
+ The record type for the filter is
+ <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
+ (fundamental type <literal>grs</literal>, file read
+ type <literal>regx</literal>, argument
+ <emphasis>filter-filename</emphasis>).
+ </para>
+
+ <para>
+ Generally, an input filter consists of a sequence of rules, where each
+ rule consists of a sequence of expressions, followed by an action. The
+ expressions are evaluated against the contents of the input record,
+ and the actions normally contribute to the generation of an internal
+ representation of the record.
+ </para>
+
+ <para>
+ An expression can be either of the following:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>INIT</term>
+ <listitem>
+ <para>
+ The action associated with this expression is evaluated
+ exactly once in the lifetime of the application, before any records
+ are read. It can be used in conjunction with an action that
+ initializes tables or other resources that are used in the processing
+ of input records.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>BEGIN</term>
+ <listitem>
+ <para>
+ Matches the beginning of the record. It can be used to
+ initialize variables, etc. Typically, the
+ <emphasis>BEGIN</emphasis> rule is also used
+ to establish the root node of the record.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>END</term>
+ <listitem>
+ <para>
+ Matches the end of the record - when all of the contents
+ of the record has been processed.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>/pattern/</term>
+ <listitem>
+ <para>
+ Matches a string of characters from the input record.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>BODY</term>
+ <listitem>
+ <para>
+ This keyword may only be used between two patterns.
+ It matches everything between (not including) those patterns.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>FINISH</term>
+ <listitem>
+ <para>
+ The expression associated with this pattern is evaluated
+ once, before the application terminates. It can be used to release
+ system resources - typically ones allocated in the
+ <emphasis>INIT</emphasis> step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ An action is surrounded by curly braces ({...}), and
+ consists of a sequence of statements. Statements may be separated
+ by newlines or semicolons (;).
+ Within actions, the strings that matched the expressions
+ immediately preceding the action can be referred to as
+ $0, $1, $2, etc.
+ </para>
+
+ <para>
+ The available statements are:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>begin <replaceable>type [parameter ... ]</replaceable></term>
+ <listitem>
+ <para>
+ Begin a new
+ data element. The <replaceable>type</replaceable> is one of
+ the following:
+ <variablelist>
+
+ <varlistentry>
+ <term>record</term>
+ <listitem>
+ <para>
+ Begin a new record. The following parameter should be the
+ name of the schema that describes the structure of the record, eg.
+ <literal>gils</literal> or <literal>wais</literal> (see below).
+ The <literal>begin record</literal> call should precede
+ any other use of the <replaceable>begin</replaceable> statement.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>element</term>
+ <listitem>
+ <para>
+ Begin a new tagged element. The parameter is the
+ name of the tag. If the tag is not matched anywhere in the tagsets
+ referenced by the current schema, it is treated as a local string
+ tag.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>variant</term>
+ <listitem>
+ <para>
+ Begin a new node in a variant tree. The parameters are
+ <replaceable>class type value</replaceable>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>data <replaceable>parameter</replaceable></term>
+ <listitem>
+ <para>
+ Create a data element. The concatenated arguments make
+ up the value of the data element.
+ The option <literal>-text</literal> signals that
+ the layout (whitespace) of the data should be retained for
+ transmission.
+ The option <literal>-element</literal>
+ <replaceable>tag</replaceable> wraps the data up in
+ the <replaceable>tag</replaceable>.
+ The use of the <literal>-element</literal> option is equivalent to
+ preceding the command with a <replaceable>begin
+ element</replaceable> command, and following
+ it with the <replaceable>end</replaceable> command.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>end <replaceable>[type]</replaceable></term>
+ <listitem>
+ <para>
+ Close a tagged element. If no parameter is given,
+ the last element on the stack is terminated.
+ The first parameter, if any, is a type name, similar
+ to the <replaceable>begin</replaceable> statement.
+ For the <replaceable>element</replaceable> type, a tag
+ name can be provided to terminate a specific tag.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>unread <replaceable>no</replaceable></term>
+ <listitem>
+ <para>
+ Move the input pointer to the offset of first character that
+ match rule given by <replaceable>no</replaceable>.
+ The first rule from left-to-right is numbered zero,
+ the second rule is named 1 and so on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <para>
+ The following input filter reads a Usenet news file, producing a
+ record in the WAIS schema. Note that the body of a news posting is
+ separated from the list of headers by a blank line (or rather a
+ sequence of two newline characters.
+ </para>
+
+ <para>
+
+ <screen>
+ BEGIN { begin record wais }
+
+ /^From:/ BODY /$/ { data -element name $1 }
+ /^Subject:/ BODY /$/ { data -element title $1 }
+ /^Date:/ BODY /$/ { data -element lastModified $1 }
+ /\n\n/ BODY END {
+ begin element bodyOfDisplay
+ begin variant body iana "text/plain"
+ data -text $1
+ end record
+ }
+ </screen>
+
+ </para>
+
+ <para>
+ If Zebra is compiled with support for Tcl enabled, the statements
+ described above are supplemented with a complete
+ scripting environment, including control structures (conditional
+ expressions and loop constructs), and powerful string manipulation
+ mechanisms for modifying the elements of a record.
+ </para>
+
+ </sect2>
+
+ </sect1>
+
+ <sect1 id="grs-internal-representation">
+ <title>GRS Internal Record Representation</title>
+
+ <para>
+ When records are manipulated by the system, they're represented in a
+ tree-structure, with data elements at the leaf nodes, and tags or
+ variant components at the non-leaf nodes. The root-node identifies the
+ schema that lends context to the tagging and structuring of the
+ record. Imagine a simple record, consisting of a 'title' element and
+ an 'author' element:
+ </para>
+
+ <para>
+
+ <screen>
+ ROOT
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ AUTHOR "Robert Pirsig"
+ </screen>
+
+ </para>
+
+ <para>
+ A slightly more complex record would have the author element consist
+ of two elements, a surname and a first name:
+ </para>
+
+ <para>
+
+ <screen>
+ ROOT
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ AUTHOR
+ FIRST-NAME "Robert"
+ SURNAME "Pirsig"
+ </screen>
+
+ </para>
+
+ <para>
+ The root of the record will refer to the record schema that describes
+ the structuring of this particular record. The schema defines the
+ element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
+ well as the structuring (SURNAME should appear below AUTHOR, etc.). In
+ addition, the schema establishes element set names that are used by
+ the client to request a subset of the elements of a given record. The
+ schema may also establish rules for converting the record to a
+ different schema, by stating, for each element, a mapping to a
+ different tag path.
+ </para>
+
+ <sect2>
+ <title>Tagged Elements</title>
+
+ <para>
+ A data element is characterized by its tag, and its position in the
+ structure of the record. For instance, while the tag "telephone
+ number" may be used different places in a record, we may need to
+ distinguish between these occurrences, both for searching and
+ presentation purposes. For instance, while the phone numbers for the
+ "customer" and the "service provider" are both
+ representatives for the same type of resource (a telephone number), it
+ is essential that they be kept separate. The record schema provides
+ the structure of the record, and names each data element (defined by
+ the sequence of tags - the tag path - by which the element can be
+ reached from the root of the record).
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>Variants</title>
+
+ <para>
+ The children of a tag node may be either more tag nodes, a data node
+ (possibly accompanied by tag nodes),
+ or a tree of variant nodes. The children of variant nodes are either
+ more variant nodes or a data node (possibly accompanied by more
+ variant nodes). Each leaf node, which is normally a
+ data node, corresponds to a <emphasis>variant form</emphasis> of the
+ tagged element identified by the tag which parents the variant tree.
+ The following title element occurs in two different languages:
+ </para>
+
+ <para>
+
+ <screen>
+ VARIANT LANG=ENG "War and Peace"
+ TITLE
+ VARIANT LANG=DAN "Krig og Fred"
+ </screen>
+
+ </para>
+
+ <para>
+ Which of the two elements are transmitted to the client by the server
+ depends on the specifications provided by the client, if any.
+ </para>
+
+ <para>
+ In practice, each variant node is associated with a triple of class,
+ type, value, corresponding to the variant mechanism of Z39.50.
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>Data Elements</title>
+
+ <para>
+ Data nodes have no children (they are always leaf nodes in the record
+ tree).
+ </para>
+
+ <!--
+ FIXME! Documentation needs extension here about types of nodes - numerical,
+ textual, etc., plus the various types of inclusion notes.
+ </para>
+ -->
+
+ </sect2>
+
+ </sect1>
+
+ <sect1 id="record-model-grs-conf">
+ <title>GRS Record Model Configuration</title>
+
+ <para>
+ The following sections describe the configuration files that govern
+ the internal management of <literal>grs</literal> records.
+ The system searches for the files
+ in the directories specified by the <emphasis>profilePath</emphasis>
+ setting in the <literal>zebra.cfg</literal> file.
+ </para>
+
+ <sect2>
+ <title>The Abstract Syntax</title>
+
+ <para>
+ The abstract syntax definition (also known as an Abstract Record
+ Structure, or ARS) is the focal point of the
+ record schema description. For a given schema, the ABS file may state any
+ or all of the following:
+ </para>
+
+ <!--
+ FIXME - Need a diagram here, or a simple explanation how it all hangs together -H
+ -->
+
+ <para>
+
+ <itemizedlist>
+ <listitem>
+
+ <para>
+ The object identifier of the Z39.50 schema associated
+ with the ARS, so that it can be referred to by the client.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The attribute set (which can possibly be a compound of multiple
+ sets) which applies in the profile. This is used when indexing and
+ searching the records belonging to the given profile.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The tag set (again, this can consist of several different sets).
+ This is used when reading the records from a file, to recognize the
+ different tags, and when transmitting the record to the client -
+ mapping the tags to their numerical representation, if they are
+ known.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The variant set which is used in the profile. This provides a
+ vocabulary for specifying the <emphasis>forms</emphasis> of
+ data that appear inside the records.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Element set names, which are a shorthand way for the client to
+ ask for a subset of the data elements contained in a record. Element
+ set names, in the retrieval module, are mapped to <emphasis>element
+ specifications</emphasis>, which contain information equivalent to the
+ <emphasis>Espec-1</emphasis> syntax of Z39.50.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Map tables, which may specify mappings to
+ <emphasis>other</emphasis> database profiles, if desired.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Possibly, a set of rules describing the mapping of elements to a
+ MARC representation.
+
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ A list of element descriptions (this is the actual ARS of the
+ schema, in Z39.50 terms), which lists the ways in which the various
+ tags can be used and organized hierarchically.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+
+ <para>
+ Several of the entries above simply refer to other files, which
+ describe the given objects.
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>The Configuration Files</title>
+
+ <para>
+ This section describes the syntax and use of the various tables which
+ are used by the retrieval module.
+ </para>
+
+ <para>
+ The number of different file types may appear daunting at first, but
+ each type corresponds fairly clearly to a single aspect of the Z39.50
+ retrieval facilities. Further, the average database administrator,
+ who is simply reusing an existing profile for which tables already
+ exist, shouldn't have to worry too much about the contents of these tables.
+ </para>
+
+ <para>
+ Generally, the files are simple ASCII files, which can be maintained
+ using any text editor. Blank lines, and lines beginning with a (#) are
+ ignored. Any characters on a line followed by a (#) are also ignored.
+ All other lines contain <emphasis>directives</emphasis>, which provide
+ some setting or value to the system.
+ Generally, settings are characterized by a single
+ keyword, identifying the setting, followed by a number of parameters.
+ Some settings are repeatable (r), while others may occur only once in a
+ file. Some settings are optional (o), while others again are
+ mandatory (m).
+ </para>
+
+ </sect2>
+
+ <sect2 id="abs-file">
+ <title>The Abstract Syntax (.abs) Files</title>
+
+ <para>
+ The name of this file type is slightly misleading in Z39.50 terms,
+ since, apart from the actual abstract syntax of the profile, it also
+ includes most of the other definitions that go into a database
+ profile.
+ </para>
+
+ <para>
+ When a record in the canonical, SGML-like format is read from a file
+ or from the database, the first tag of the file should reference the
+ profile that governs the layout of the record. If the first tag of the
+ record is, say, <literal><gils></literal>, the system will look
+ for the profile definition in the file <literal>gils.abs</literal>.
+ Profile definitions are cached, so they only have to be read once
+ during the lifespan of the current process.
+ </para>
+
+ <para>
+ When writing your own input filters, the
+ <emphasis>record-begin</emphasis> command
+ introduces the profile, and should always be called first thing when
+ introducing a new record.
+ </para>
+
+ <para>
+ The file may contain the following directives:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>name <replaceable>symbolic-name</replaceable></term>
+ <listitem>
+ <para>
+ (m) This provides a shorthand name or
+ description for the profile. Mostly useful for diagnostic purposes.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>reference <replaceable>OID-name</replaceable></term>
+ <listitem>
+ <para>
+ (m) The reference name of the OID for the profile.
+ The reference names can be found in the <emphasis>util</emphasis>
+ module of YAZ.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>attset <replaceable>filename</replaceable></term>
+ <listitem>
+ <para>
+ (m) The attribute set that is used for
+ indexing and searching records belonging to this profile.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>tagset <replaceable>filename</replaceable></term>
+ <listitem>
+ <para>
+ (o) The tag set (if any) that describe
+ that fields of the records.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>varset <replaceable>filename</replaceable></term>
+ <listitem>
+ <para>
+ (o) The variant set used in the profile.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>maptab <replaceable>filename</replaceable></term>
+ <listitem>
+ <para>
+ (o,r) This points to a
+ conversion table that might be used if the client asks for the record
+ in a different schema from the native one.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>marc <replaceable>filename</replaceable></term>
+ <listitem>
+ <para>
+ (o) Points to a file containing parameters
+ for representing the record contents in the ISO2709 syntax.
+ Read the description of the MARC representation facility below.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>esetname <replaceable>name filename</replaceable></term>
+ <listitem>
+ <para>
+ (o,r) Associates the
+ given element set name with an element selection file. If an (@) is
+ given in place of the filename, this corresponds to a null mapping for
+ the given element set name.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>any <replaceable>tags</replaceable></term>
+ <listitem>
+ <para>
+ (o) This directive specifies a list of attributes
+ which should be appended to the attribute list given for each
+ element. The effect is to make every single element in the abstract
+ syntax searchable by way of the given attributes. This directive
+ provides an efficient way of supporting free-text searching across all
+ elements. However, it does increase the size of the index
+ significantly. The attributes can be qualified with a structure, as in
+ the <replaceable>elm</replaceable> directive below.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>elm <replaceable>path name attributes</replaceable></term>
+ <listitem>
+ <para>
+ (o,r) Adds an element to the abstract record syntax of the schema.
+ The <replaceable>path</replaceable> follows the
+ syntax which is suggested by the Z39.50 document - that is, a sequence
+ of tags separated by slashes (/). Each tag is given as a
+ comma-separated pair of tag type and -value surrounded by parenthesis.
+ The <replaceable>name</replaceable> is the name of the element, and
+ the <replaceable>attributes</replaceable>
+ specifies which attributes to use when indexing the element in a
+ comma-separated list.
+ A ! in place of the attribute name is equivalent to
+ specifying an attribute name identical to the element name.
+ A - in place of the attribute name
+ specifies that no indexing is to take place for the given element.
+ The attributes can be qualified with <replaceable>field
+ types</replaceable> to specify which
+ character set should govern the indexing procedure for that field.
+ The same data element may be indexed into several different
+ fields, using different character set definitions.
+ See the <xref linkend="field-structure-and-character-sets"/>.
+ The default field type is <literal>w</literal> for
+ <emphasis>word</emphasis>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>xelm <replaceable>xpath attributes</replaceable></term>
+ <listitem>
+ <para>
+ Specifies indexing for record nodes given by
+ <replaceable>xpath</replaceable>. Unlike directive
+ elm, this directive allows you to index attribute
+ contents. The <replaceable>xpath</replaceable> uses
+ a syntax similar to XPath. The <replaceable>attributes</replaceable>
+ have same syntax and meaning as directive elm, except that operator
+ ! refers to the nodes selected by <replaceable>xpath</replaceable>.
+ <!--
+ xelm / !:w default index
+ xelm // !:w additional index
+ xelm /gils/title/@att myatt:w index attribute @att in myatt
+ xelm title/@att myatt:w same meaning.
+ -->
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>melm <replaceable>field$subfield attributes</replaceable></term>
+ <listitem>
+ <para>
+ This directive is specifically for MARC-formatted records,
+ ingested either in the form of MARCXML documents, or in the
+ ISO2709/Z39.2 format using the grs.marcxml input filter. You can
+ specify indexing rules for any subfield, or you can leave off the
+ <replaceable>$subfield</replaceable> part and specify default rules
+ for all subfields of the given field (note: default rules should come
+ after any subfield-specific rules in the configuration file). The
+ <replaceable>attributes</replaceable> have the same syntax and meaning
+ as for the 'elm' directive above.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>encoding <replaceable>encodingname</replaceable></term>
+ <listitem>
+ <para>
+ This directive specifies character encoding for external records.
+ For records such as XML that specifies encoding within the
+ file via a header this directive is ignored.
+ If neither this directive is given, nor an encoding is set
+ within external records, ISO-8859-1 encoding is assumed.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>xpath <literal>enable</literal>/<literal>disable</literal></term>
+ <listitem>
+ <para>
+ If this directive is followed by <literal>enable</literal>,
+ then extra indexing is performed to allow for XPath-like queries.
+ If this directive is not specified - equivalent to
+ <literal>disable</literal> - no extra XPath-indexing is performed.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <!-- Adam's version
+ <varlistentry>
+ <term>systag <replaceable>systemtag</replaceable> <replaceable>element</replaceable></term>
+ <listitem>
+ <para>
+ This directive maps system information to an element during
+ retrieval. This information is dynamically created. The
+ following system tags are defined
+ <variablelist>
+ <varlistentry>
+ <term>size</term>
+ <listitem>
+ <para>
+ Size of record in bytes. By default this
+ is mapped to element <literal>size</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>rank</term>
+ <listitem>
+ <para>
+ Score/rank of record. By default this
+ is mapped to element <literal>rank</literal>.
+ If no score was calculated for the record (non-ranked
+ searched) search this directive is ignored.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>sysno</term>
+ <listitem>
+ <para>
+ Zebra's system number (record ID) for the
+ record. By default this is mapped to element
+ <literal>localControlNumber</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ If you do not want a particular system tag to be applied,
+ then set the resulting element to something undefined in the
+ abs file (such as <literal>none</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+ -->
+
+ <!-- Mike's version -->
+ <varlistentry>
+ <term>
+ systag
+ <replaceable>systemTag</replaceable>
+ <replaceable>actualTag</replaceable>
+ </term>
+ <listitem>
+ <para>
+ Specifies what information, if any, Zebra should
+ automatically include in retrieval records for the
+ ``system fields'' that it supports.
+ <replaceable>systemTag</replaceable> may
+ be any of the following:
+ <variablelist>
+ <varlistentry>
+ <term><literal>rank</literal></term>
+ <listitem><para>
+ An integer indicating the relevance-ranking score
+ assigned to the record.
+ </para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>sysno</literal></term>
+ <listitem><para>
+ An automatically generated identifier for the record,
+ unique within this database. It is represented by the
+ <literal><localControlNumber></literal> element in
+ XML and the <literal>(1,14)</literal> tag in GRS-1.
+ </para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>size</literal></term>
+ <listitem><para>
+ The size, in bytes, of the retrieved record.
+ </para></listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ <para>
+ The <replaceable>actualTag</replaceable> parameter may be
+ <literal>none</literal> to indicate that the named element
+ should be omitted from retrieval records.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <note>
+ <para>
+ The mechanism for controlling indexing is not adequate for
+ complex databases, and will probably be moved into a separate
+ configuration table eventually.
+ </para>
+ </note>
+
+ <para>
+ The following is an excerpt from the abstract syntax file for the GILS
+ profile.
+ </para>
+
+ <para>
+
+ <screen>
+ name gils
+ reference GILS-schema
+ attset gils.att
+ tagset gils.tag
+ varset var1.var
+
+ maptab gils-usmarc.map
+
+ # Element set names
+
+ esetname VARIANT gils-variant.est # for WAIS-compliance
+ esetname B gils-b.est
+ esetname G gils-g.est
+ esetname F @
+
+ elm (1,10) rank -
+ elm (1,12) url -
+ elm (1,14) localControlNumber Local-number
+ elm (1,16) dateOfLastModification Date/time-last-modified
+ elm (2,1) title w:!,p:!
+ elm (4,1) controlIdentifier Identifier-standard
+ elm (2,6) abstract Abstract
+ elm (4,51) purpose !
+ elm (4,52) originator -
+ elm (4,53) accessConstraints !
+ elm (4,54) useConstraints !
+ elm (4,70) availability -
+ elm (4,70)/(4,90) distributor -
+ elm (4,70)/(4,90)/(2,7) distributorName !
+ elm (4,70)/(4,90)/(2,10) distributorOrganization !
+ elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
+ elm (4,70)/(4,90)/(4,3) distributorCity !
+ </screen>
+
+ </para>
+
+ </sect2>
+
+ <sect2 id="attset-files">
+ <title>The Attribute Set (.att) Files</title>
+
+ <para>
+ This file type describes the <replaceable>Use</replaceable> elements of
+ an attribute set.
+ It contains the following directives.
+ </para>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term>name <replaceable>symbolic-name</replaceable></term>
+ <listitem>
+ <para>
+ (m) This provides a shorthand name or
+ description for the attribute set.
+ Mostly useful for diagnostic purposes.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>reference <replaceable>OID-name</replaceable></term>
+ <listitem>
+ <para>
+ (m) The reference name of the OID for
+ the attribute set.
+ The reference names can be found in the <replaceable>util</replaceable>
+ module of <replaceable>YAZ</replaceable>.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>include <replaceable>filename</replaceable></term>
+ <listitem>
+ <para>
+ (o,r) This directive is used to
+ include another attribute set as a part of the current one. This is
+ used when a new attribute set is defined as an extension to another
+ set. For instance, many new attribute sets are defined as extensions
+ to the <replaceable>bib-1</replaceable> set.
+ This is an important feature of the retrieval
+ system of Z39.50, as it ensures the highest possible level of
+ interoperability, as those access points of your database which are
+ derived from the external set (say, bib-1) can be used even by clients
+ who are unaware of the new set.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>att
+ <replaceable>att-value att-name [local-value]</replaceable></term>
+ <listitem>
+ <para>
+ (o,r) This
+ repeatable directive introduces a new attribute to the set. The
+ attribute value is stored in the index (unless a
+ <replaceable>local-value</replaceable> is
+ given, in which case this is stored). The name is used to refer to the
+ attribute from the <replaceable>abstract syntax</replaceable>.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ This is an excerpt from the GILS attribute set definition.
+ Notice how the file describing the <emphasis>bib-1</emphasis>
+ attribute set is referenced.
+ </para>
+
+ <para>
+
+ <screen>
+ name gils
+ reference GILS-attset
+ include bib1.att
+
+ att 2001 distributorName
+ att 2002 indextermsControlled
+ att 2003 purpose
+ att 2004 accessConstraints
+ att 2005 useConstraints
+ </screen>
+
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>The Tag Set (.tag) Files</title>
+
+ <para>
+ This file type defines the tagset of the profile, possibly by
+ referencing other tag sets (most tag sets, for instance, will include
+ tagsetG and tagsetM from the Z39.50 specification. The file may
+ contain the following directives.
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>name <emphasis>symbolic-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) This provides a shorthand name or
+ description for the tag set. Mostly useful for diagnostic purposes.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>reference <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (o) The reference name of the OID for the tag set.
+ The reference names can be found in the <emphasis>util</emphasis>
+ module of <emphasis>YAZ</emphasis>.
+ The directive is optional, since not all tag sets
+ are registered outside of their schema.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>type <emphasis>integer</emphasis></term>
+ <listitem>
+ <para>
+ (m) The type number of the tagset within the schema
+ profile (note: this specification really should belong to the .abs
+ file. This will be fixed in a future release).
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>include <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) This directive is used
+ to include the definitions of other tag sets into the current one.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>tag <emphasis>number names type</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) Introduces a new tag to the set.
+ The <emphasis>number</emphasis> is the tag number as used
+ in the protocol (there is currently no mechanism for
+ specifying string tags at this point, but this would be quick
+ work to add).
+ The <emphasis>names</emphasis> parameter is a list of names
+ by which the tag should be recognized in the input file format.
+ The names should be separated by slashes (/).
+ The <emphasis>type</emphasis> is the recommended data type of
+ the tag.
+ It should be one of the following:
+
+ <itemizedlist>
+ <listitem>
+ <para>
+ structured
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ string
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ numeric
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ bool
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ oid
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ generalizedtime
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ intunit
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ int
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ octetstring
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ null
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The following is an excerpt from the TagsetG definition file.
+ </para>
+
+ <para>
+ <screen>
+ name tagsetg
+ reference TagsetG
+ type 2
+
+ tag 1 title string
+ tag 2 author string
+ tag 3 publicationPlace string
+ tag 4 publicationDate string
+ tag 5 documentId string
+ tag 6 abstract string
+ tag 7 name string
+ tag 8 date generalizedtime
+ tag 9 bodyOfDisplay string
+ tag 10 organization string
+ </screen>
+ </para>
+
+ </sect2>
+
+ <sect2 id="variant-set">
+ <title>The Variant Set (.var) Files</title>
+
+ <para>
+ The variant set file is a straightforward representation of the
+ variant set definitions associated with the protocol. At present, only
+ the <emphasis>Variant-1</emphasis> set is known.
+ </para>
+
+ <para>
+ These are the directives allowed in the file.
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>name <emphasis>symbolic-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) This provides a shorthand name or
+ description for the variant set. Mostly useful for diagnostic purposes.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>reference <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (o) The reference name of the OID for
+ the variant set, if one is required. The reference names can be found
+ in the <emphasis>util</emphasis> module of <emphasis>YAZ</emphasis>.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>class <emphasis>integer class-name</emphasis></term>
+ <listitem>
+ <para>
+ (m,r) Introduces a new
+ class to the variant set.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>type <emphasis>integer type-name datatype</emphasis></term>
+ <listitem>
+ <para>
+ (m,r) Addes a
+ new type to the current class (the one introduced by the most recent
+ <emphasis>class</emphasis> directive).
+ The type names belong to the same name space as the one used
+ in the tag set definition file.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The following is an excerpt from the file describing the variant set
+ <emphasis>Variant-1</emphasis>.
+ </para>
+
+ <para>
+
+ <screen>
+ name variant-1
+ reference Variant-1
+
+ class 1 variantId
+
+ type 1 variantId octetstring
+
+ class 2 body
+
+ type 1 iana string
+ type 2 z39.50 string
+ type 3 other string
+ </screen>
+
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>The Element Set (.est) Files</title>
+
+ <para>
+ The element set specification files describe a selection of a subset
+ of the elements of a database record. The element selection mechanism
+ is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
+ syntax of the Z39.50 specification.
+ In fact, the internal representation of an element set
+ specification is identical to the <emphasis>Espec-1</emphasis> structure,
+ and we'll refer you to the description of that structure for most of
+ the detailed semantics of the directives below.
+ </para>
+
+ <note>
+ <para>
+ Not all of the Espec-1 functionality has been implemented yet.
+ The fields that are mentioned below all work as expected, unless
+ otherwise is noted.
+ </para>
+ </note>
+
+ <para>
+ The directives available in the element set file are as follows:
+ </para>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (o) If variants are used in
+ the following, this should provide the name of the variantset used
+ (it's not currently possible to specify a different set in the
+ individual variant request). In almost all cases (certainly all
+ profiles known to us), the name
+ <literal>Variant-1</literal> should be given here.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
+ <listitem>
+ <para>
+ (o) This directive
+ provides a default variant request for
+ use when the individual element requests (see below) do not contain a
+ variant request. Variant requests consist of a blank-separated list of
+ variant components. A variant compont is a comma-separated,
+ parenthesized triple of variant class, type, and value (the two former
+ values being represented as integers). The value can currently only be
+ entered as a string (this will change to depend on the definition of
+ the variant in question). The special value (@) is interpreted as a
+ null value, however.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>simpleElement
+ <emphasis>path ['variant' variant-request]</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) This corresponds to a simple element request
+ in <emphasis>Espec-1</emphasis>.
+ The path consists of a sequence of tag-selectors, where each of
+ these can consist of either:
+ </para>
+
+ <para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ A simple tag, consisting of a comma-separated type-value pair in
+ parenthesis, possibly followed by a colon (:) followed by an
+ occurrences-specification (see below). The tag-value can be a number
+ or a string. If the first character is an apostrophe ('), this
+ forces the value to be interpreted as a string, even if it
+ appears to be numerical.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ A WildThing, represented as a question mark (?), possibly
+ followed by a colon (:) followed by an occurrences
+ specification (see below).
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ A WildPath, represented as an asterisk (*). Note that the last
+ element of the path should not be a wildPath (wildpaths don't
+ work in this version).
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+
+ <para>
+ The occurrences-specification can be either the string
+ <literal>all</literal>, the string <literal>last</literal>, or
+ an explicit value-range. The value-range is represented as
+ an integer (the starting point), possibly followed by a
+ plus (+) and a second integer (the number of elements, default
+ being one).
+ </para>
+
+ <para>
+ The variant-request has the same syntax as the defaultVariantRequest
+ above. Note that it may sometimes be useful to give an empty variant
+ request, simply to disable the default for a specific set of fields
+ (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
+ but it works in this implementation).
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The following is an example of an element specification belonging to
+ the GILS profile.
+ </para>
+
+ <para>
+
+ <screen>
+ simpleelement (1,10)
+ simpleelement (1,12)
+ simpleelement (2,1)
+ simpleelement (1,14)
+ simpleelement (4,1)
+ simpleelement (4,52)
+ </screen>
+
+ </para>
+
+ </sect2>
+
+ <sect2 id="schema-mapping">
+ <title>The Schema Mapping (.map) Files</title>
+
+ <para>
+ Sometimes, the client might want to receive a database record in
+ a schema that differs from the native schema of the record. For
+ instance, a client might only know how to process WAIS records, while
+ the database record is represented in a more specific schema, such as
+ GILS. In this module, a mapping of data to one of the MARC formats is
+ also thought of as a schema mapping (mapping the elements of the
+ record into fields consistent with the given MARC specification, prior
+ to actually converting the data to the ISO2709). This use of the
+ object identifier for USMARC as a schema identifier represents an
+ overloading of the OID which might not be entirely proper. However,
+ it represents the dual role of schema and record syntax which
+ is assumed by the MARC family in Z39.50.
+ </para>
+
+ <!--
+ <emphasis>NOTE: FIXME! The schema-mapping functions are so far limited to a
+ straightforward mapping of elements. This should be extended with
+ mechanisms for conversions of the element contents, and conditional
+ mappings of elements based on the record contents.</emphasis>
+ -->
+
+ <para>
+ These are the directives of the schema mapping file format:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>targetName <emphasis>name</emphasis></term>
+ <listitem>
+ <para>
+ (m) A symbolic name for the target schema
+ of the table. Useful mostly for diagnostic purposes.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>targetRef <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) An OID name for the target schema.
+ This is used, for instance, by a server receiving a request to present
+ a record in a different schema from the native one.
+ The name, again, is found in the <emphasis>oid</emphasis>
+ module of <emphasis>YAZ</emphasis>.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>map <emphasis>element-name target-path</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) Adds
+ an element mapping rule to the table.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>The MARC (ISO2709) Representation (.mar) Files</title>
+
+ <para>
+ This file provides rules for representing a record in the ISO2709
+ format. The rules pertain mostly to the values of the constant-length
+ header of the record.
+ </para>
+
+ <!--
+ NOTE: FIXME! This will be described better. We're in the process of
+ re-evaluating and most likely changing the way that MARC records are
+ handled by the system.</emphasis>
+ -->
+
+ </sect2>
+
+ <sect2 id="field-structure-and-character-sets">
+ <title>Field Structure and Character Sets
+ </title>
+
+ <para>
+ In order to provide a flexible approach to national character set
+ handling, Zebra allows the administrator to configure the set up the
+ system to handle any 8-bit character set — including sets that
+ require multi-octet diacritics or other multi-octet characters. The
+ definition of a character set includes a specification of the
+ permissible values, their sort order (this affects the display in the
+ SCAN function), and relationships between upper- and lowercase
+ characters. Finally, the definition includes the specification of
+ space characters for the set.
+ </para>
+
+ <para>
+ The operator can define different character sets for different fields,
+ typical examples being standard text fields, numerical fields, and
+ special-purpose fields such as WWW-style linkages (URx).
+ </para>
+
+ <sect3 id="default-idx-file">
+ <title>The default.idx file</title>
+ <para>
+ The field types, and hence character sets, are associated with data
+ elements by the .abs files (see above).
+ The file <literal>default.idx</literal>
+ provides the association between field type codes (as used in the .abs
+ files) and the character map files (with the .chr suffix). The format
+ of the .idx file is as follows
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>index <emphasis>field type code</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces a new search index code.
+ The argument is a one-character code to be used in the
+ .abs files to select this particular index type. An index, roughly,
+ corresponds to a particular structure attribute during search. Refer
+ to <xref linkend="search"/>.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>sort <emphasis>field code type</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces a
+ sort index. The argument is a one-character code to be used in the
+ .abs fie to select this particular index type. The corresponding
+ use attribute must be used in the sort request to refer to this
+ particular sort index. The corresponding character map (see below)
+ is used in the sort process.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>completeness <emphasis>boolean</emphasis></term>
+ <listitem>
+ <para>
+ This directive enables or disables complete field indexing.
+ The value of the <emphasis>boolean</emphasis> should be 0
+ (disable) or 1. If completeness is enabled, the index entry will
+ contain the complete contents of the field (up to a limit), with words
+ (non-space characters) separated by single space characters
+ (normalized to " " on display). When completeness is
+ disabled, each word is indexed as a separate entry. Complete subfield
+ indexing is most useful for fields which are typically browsed (eg.
+ titles, authors, or subjects), or instances where a match on a
+ complete subfield is essential (eg. exact title searching). For fields
+ where completeness is disabled, the search engine will interpret a
+ search containing space characters as a word proximity search.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>charmap <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ This is the filename of the character
+ map to be used for this index for field type.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+ </sect3>
+
+ <sect3 id="character-map-files">
+ <title>The character map file format</title>
+ <para>
+ The contents of the character map files are structured as follows:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>lowercase <emphasis>value-set</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces the basic value set of the field type.
+ The format is an ordered list (without spaces) of the
+ characters which may occur in "words" of the given type.
+ The order of the entries in the list determines the
+ sort order of the index. In addition to single characters, the
+ following combinations are legal:
+ </para>
+
+ <para>
+
+ <itemizedlist>
+ <listitem>
+ <para>
+ Backslashes may be used to introduce three-digit octal, or
+ two-digit hex representations of single characters
+ (preceded by <literal>x</literal>).
+ In addition, the combinations
+ \\, \\r, \\n, \\t, \\s (space — remember that real
+ space-characters may not occur in the value definition), and
+ \\ are recognized, with their usual interpretation.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Curly braces {} may be used to enclose ranges of single
+ characters (possibly using the escape convention described in the
+ preceding point), eg. {a-z} to introduce the
+ standard range of ASCII characters.
+ Note that the interpretation of such a range depends on
+ the concrete representation in your local, physical character set.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ paranthesises () may be used to enclose multi-byte characters -
+ eg. diacritics or special national combinations (eg. Spanish
+ "ll"). When found in the input stream (or a search term),
+ these characters are viewed and sorted as a single character, with a
+ sorting value depending on the position of the group in the value
+ statement.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>uppercase <emphasis>value-set</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces the
+ upper-case equivalencis to the value set (if any). The number and
+ order of the entries in the list should be the same as in the
+ <literal>lowercase</literal> directive.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>space <emphasis>value-set</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces the character
+ which separate words in the input stream. Depending on the
+ completeness mode of the field in question, these characters either
+ terminate an index entry, or delimit individual "words" in
+ the input stream. The order of the elements is not significant —
+ otherwise the representation is the same as for the
+ <literal>uppercase</literal> and <literal>lowercase</literal>
+ directives.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>map <emphasis>value-set</emphasis>
+ <emphasis>target</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces a mapping between each of the
+ members of the value-set on the left to the character on the
+ right. The character on the right must occur in the value
+ set (the <literal>lowercase</literal> directive) of the
+ character set, but it may be a paranthesis-enclosed
+ multi-octet character. This directive may be used to map
+ diacritics to their base characters, or to map HTML-style
+ character-representations to their natural form, etc. The
+ map directive can also be used to ignore leading articles in
+ searching and/or sorting, and to perform other special
+ transformations. See section <xref
+ linkend="leading-articles"/>.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+ </sect3>
+ <sect3 id="leading-articles">
+ <title>Ignoring leading articles</title>
+ <para>
+ In addition to specifying sort orders, space (blank) handling,
+ and upper/lowercase folding, you can also use the character map
+ files to make Zebra ignore leading articles in sorting records,
+ or when doing complete field searching.
+ </para>
+ <para>
+ This is done using the <literal>map</literal> directive in the
+ character map file. In a nutshell, what you do is map certain
+ sequences of characters, when they occur <emphasis> in the
+ beginning of a field</emphasis>, to a space. Assuming that the
+ character "@" is defined as a space character in your file, you
+ can do:
+ <screen>
+ map (^The\s) @
+ map (^the\s) @
+ </screen>
+ The effect of these directives is to map either 'the' or 'The',
+ followed by a space character, to a space. The hat ^ character
+ denotes beginning-of-field only when complete-subfield indexing
+ or sort indexing is taking place; otherwise, it is treated just
+ as any other character.
+ </para>
+ <para>
+ Because the <literal>default.idx</literal> file can be used to
+ associate different character maps with different indexing types
+ -- and you can create additional indexing types, should the need
+ arise -- it is possible to specify that leading articles should
+ be ignored either in sorting, in complete-field searching, or
+ both.
+ </para>
+ <para>
+ If you ignore certain prefixes in sorting, then these will be
+ eliminated from the index, and sorting will take place as if
+ they weren't there. However, if you set the system up to ignore
+ certain prefixes in <emphasis>searching</emphasis>, then these
+ are deleted both from the indexes and from query terms, when the
+ client specifies complete-field searching. This has the effect
+ that a search for 'the science journal' and 'science journal'
+ would both produce the same results.
+ </para>
+ </sect3>
+ </sect2>
+ </sect1>
+
+ <sect1 id="grs-exchange-formats">
+ <title>GRS Exchange Formats</title>
+
+ <para>
+ Converting records from the internal structure to an exchange format
+ is largely an automatic process. Currently, the following exchange
+ formats are supported:
+ </para>
+
+ <para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ GRS-1. The internal representation is based on GRS-1/XML, so the
+ conversion here is straightforward. The system will create
+ applied variant and supported variant lists as required, if a record
+ contains variant information.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ XML. The internal representation is based on GRS-1/XML so
+ the mapping is trivial. Note that XML schemas, preprocessing
+ instructions and comments are not part of the internal representation
+ and therefore will never be part of a generated XML record.
+ Future versions of the Zebra will support that.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ SUTRS. Again, the mapping is fairly straightforward. Indentation
+ is used to show the hierarchical structure of the record. All
+ "GRS" type records support both the GRS-1 and SUTRS
+ representations.
+ <!-- FIXME - What is SUTRS - should be expanded here -->
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ ISO2709-based formats (USMARC, etc.). Only records with a
+ two-level structure (corresponding to fields and subfields) can be
+ directly mapped to ISO2709. For records with a different structuring
+ (eg., GILS), the representation in a structure like USMARC involves a
+ schema-mapping (see <xref linkend="schema-mapping"/>), to an
+ "implied" USMARC schema (implied,
+ because there is no formal schema which specifies the use of the
+ USMARC fields outside of ISO2709). The resultant, two-level record is
+ then mapped directly from the internal representation to ISO2709. See
+ the GILS schema definition files for a detailed example of this
+ approach.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Explain. This representation is only available for records
+ belonging to the Explain schema.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Summary. This ASN-1 based structure is only available for records
+ belonging to the Summary schema - or schema which provide a mapping
+ to this schema (see the description of the schema mapping facility
+ above).
+ </para>
+ </listitem>
+
+ <!-- FIXME - Is this used anywhere ? -H -->
+ <listitem>
+ <para>
+ SOIF. Support for this syntax is experimental, and is currently
+ keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
+ abstract syntaxes can be mapped to the SOIF format, although nested
+ elements are represented by concatenation of the tag names at each
+ level.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+ </para>
+ </sect1>
+
+ </chapter>
+ <!-- Keep this comment at the end of the file
+ Local variables:
+ mode: sgml
+ sgml-omittag:t
+ sgml-shorttag:t
+ sgml-minimize-attributes:nil
+ sgml-always-quote-attributes:t
+ sgml-indent-step:1
+ sgml-indent-data:t
+ sgml-parent-document: "zebra.xml"
+ sgml-local-catalogs: nil
+ sgml-namecase-general:t
+ End:
+ -->
projects / idzebra-moved-to-github.git / blobdiff