Use entity idcommon rather than common

[idzebra-moved-to-github.git] / doc / recordmodel-grs.xml

 <chapter id="grs">
  <!-- $Id: recordmodel-grs.xml,v 1.8 2007-02-20 14:28:31 marc Exp $ -->
  <title>&grs1; Record Model and Filter Modules</title>

     <note>
      <para>
        The functionality of this record model has been improved and
        replaced by the DOM &xml; record model. See 
        <xref linkend="record-model-domxml"/>.
      </para>
     </note>

  <para>
   The record model described in this chapter applies to the fundamental,
   structured
   record type <literal>grs</literal>, introduced in
   <xref linkend="componentmodulesgrs"/>.
  </para>


  <section id="grs-filters">
   <title>&grs1; Record Filters</title>
   <para>
    Many basic subtypes of the <emphasis>grs</emphasis> type are
    currently available:
   </para>

   <para>
    <variablelist>
     <varlistentry>
      <term><literal>grs.sgml</literal></term>
      <listitem>
       <para>
        This is the canonical input format
        described <xref linkend="grs-canonical-format"/>. It is using
        simple &sgml;-like syntax. 
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>grs.marc.</literal><replaceable>type</replaceable></term>
      <listitem>
       <para>
        This allows &zebra; to read
        records in the ISO2709 (&marc;) encoding standard. 
        Last parameter <replaceable>type</replaceable> names the
        <literal>.abs</literal> file (see below)
        which describes the specific &marc; structure of the input record as
        well as the indexing rules.
       </para>
       <para>The <literal>grs.marc</literal> uses an internal represtantion
        which is not &xml; conformant. In particular &marc; tags are
        presented as elements with the same name. And &xml; elements
        may not start with digits. Therefore this filter is only
        suitable for systems returning &grs1; and &marc; records. For &xml;
        use <literal>grs.marcxml</literal> filter instead (see below).
       </para>
       <para>
         The loadable <literal>grs.marc</literal> filter module
         is packaged in the GNU/Debian package
        <literal>libidzebra2.0-mod-grs-marc</literal>
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>grs.marcxml.</literal><replaceable>type</replaceable></term>
      <listitem>
       <para>
        This allows &zebra; to read ISO2709 encoded records.
        Last parameter <replaceable>type</replaceable> names the
        <literal>.abs</literal> file (see below)
        which describes the specific &marc; structure of the input record as
        well as the indexing rules.
       </para>
       <para>
        The internal representation for <literal>grs.marcxml</literal>
        is the same as for <ulink url="&url.marcxml;">&marcxml;</ulink>.
        It slightly more complicated to work with than 
        <literal>grs.marc</literal> but &xml; conformant.
       </para>
       <para>
        The loadable <literal>grs.marcxml</literal> filter module
        is also contained in the GNU/Debian package
        <literal>libidzebra2.0-mod-grs-marc</literal>
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>grs.xml</literal></term>
      <listitem>
       <para>
        This filter reads &xml; records and uses
        <ulink url="http://expat.sourceforge.net/">Expat</ulink> to
        parse them and convert them into ID&zebra;'s internal 
        <literal>grs</literal> record model.
        Only one record per file is supported, due to the fact &xml; does
        not allow two documents to "follow" each other (there is no way
        to know when a document is finished).
        This filter is only available if &zebra; is compiled with EXPAT support.
       </para>
       <para>
        The loadable <literal>grs.xml</literal> filter module
        is packagged in the GNU/Debian package
        <literal>libidzebra2.0-mod-grs-xml</literal>
        </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>grs.regx.</literal><replaceable>filter</replaceable></term>
      <listitem>
       <para>
        This enables a user-supplied Regular Expressions input
        filter described in <xref linkend="grs-regx-tcl"/>.
       </para>
       <para>
        The loadable <literal>grs.regx</literal> filter module
        is packaged in the GNU/Debian package
        <literal>libidzebra2.0-mod-grs-regx</literal>
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>grs.tcl.</literal><replaceable>filter</replaceable></term>
      <listitem>
       <para>
        Similar to grs.regx but using Tcl for rules, described in 
        <xref linkend="grs-regx-tcl"/>.
       </para>
       <para>
        The loadable <literal>grs.tcl</literal> filter module
        is also packaged in the GNU/Debian package
        <literal>libidzebra2.0-mod-grs-regx</literal>
       </para>
      </listitem>
     </varlistentry>

    </variablelist>
   </para>

   <section id="grs-canonical-format">
    <title>&grs1; Canonical Input Format</title>

    <para>
     Although input data can take any form, it is sometimes useful to
     describe the record processing capabilities of the system in terms of
     a single, canonical input format that gives access to the full
     spectrum of structure and flexibility in the system. In &zebra;, this
     canonical format is an "&sgml;-like" syntax.
    </para>

    <para>
     To use the canonical format specify <literal>grs.sgml</literal> as
     the record type.
    </para>

    <para>
     Consider a record describing an information resource (such a record is
     sometimes known as a <emphasis>locator record</emphasis>).
     It might contain a field describing the distributor of the
     information resource, which might in turn be partitioned into
     various fields providing details about the distributor, like this:
    </para>

    <para>

     <screen>
      &#60;Distributor&#62;
        &#60;Name&#62; USGS/WRD &#60;/Name&#62;
        &#60;Organization&#62; USGS/WRD &#60;/Organization&#62;
        &#60;Street-Address&#62;
          U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
        &#60;/Street-Address&#62;
        &#60;City&#62; ALBUQUERQUE &#60;/City&#62;
        &#60;State&#62; NM &#60;/State&#62;
        &#60;Zip-Code&#62; 87102 &#60;/Zip-Code&#62;
        &#60;Country&#62; USA &#60;/Country&#62;
        &#60;Telephone&#62; (505) 766-5560 &#60;/Telephone&#62;
      &#60;/Distributor&#62;
     </screen>

    </para>

    <!-- There is no indentation in the example above!  -H
    -note-
     -para-
      The indentation used above is used to illustrate how &zebra;
      interprets the mark-up. The indentation, in itself, has no
      significance to the parser for the canonical input format, which
      discards superfluous whitespace.
     -/para-
    -/note-
    -->

    <para>
     The keywords surrounded by &lt;...&gt; are
     <emphasis>tags</emphasis>, while the sections of text
     in between are the <emphasis>data elements</emphasis>.
     A data element is characterized by its location in the tree
     that is made up by the nested elements.
     Each element is terminated by a closing tag - beginning
     with <literal>&#60;</literal>/, and containing the same symbolic
     tag-name as the corresponding opening tag.
     The general closing tag - <literal>&lt;/&gt;</literal> -
     terminates the element started by the last opening tag. The
     structuring of elements is significant.
     The element <emphasis>Telephone</emphasis>,
     for instance, may be indexed and presented to the client differently,
     depending on whether it appears inside the
     <emphasis>Distributor</emphasis> element, or some other,
     structured data element such a <emphasis>Supplier</emphasis> element.
    </para>

    <section id="grs-record-root">
     <title>Record Root</title>

     <para>
      The first tag in a record describes the root node of the tree that
      makes up the total record. In the canonical input format, the root tag
      should contain the name of the schema that lends context to the
      elements of the record
      (see <xref linkend="grs-internal-representation"/>).
      The following is a GILS record that
      contains only a single element (strictly speaking, that makes it an
      illegal GILS record, since the GILS profile includes several mandatory
      elements - &zebra; does not validate the contents of a record against
      the &z3950; profile, however - it merely attempts to match up elements
      of a local representation with the given schema):
     </para>

     <para>

      <screen>
       &#60;gils&#62;
          &#60;title&#62;Zen and the Art of Motorcycle Maintenance&#60;/title&#62;
       &#60;/gils&#62;
      </screen>

     </para>

    </section>

    <section id="grs-variants">
     <title>Variants</title>

     <para>
      &zebra; allows you to provide individual data elements in a number of
      <emphasis>variant forms</emphasis>. Examples of variant forms are
      textual data elements which might appear in different languages, and
      images which may appear in different formats or layouts.
      The variant system in &zebra; is essentially a representation of
      the variant mechanism of &z3950;-1995.
     </para>

     <para>
      The following is an example of a title element which occurs in two
      different languages.
     </para>

     <para>

      <screen>
       &#60;title&#62;
       &#60;var lang lang "eng"&#62;
       Zen and the Art of Motorcycle Maintenance&#60;/&#62;
       &#60;var lang lang "dan"&#62;
       Zen og Kunsten at Vedligeholde en Motorcykel&#60;/&#62;
       &#60;/title&#62;
      </screen>

     </para>

     <para>
      The syntax of the <emphasis>variant element</emphasis> is
      <literal>&lt;var class type value&gt;</literal>.
      The available values for the <emphasis>class</emphasis> and
      <emphasis>type</emphasis> fields are given by the variant set
      that is associated with the current schema
      (see <xref linkend="grs-variants"/>).
     </para>

     <para>
      Variant elements are terminated by the general end-tag &#60;/&#62;, by
      the variant end-tag &#60;/var&#62;, by the appearance of another variant
      tag with the same <emphasis>class</emphasis> and
      <emphasis>value</emphasis> settings, or by the
      appearance of another, normal tag. In other words, the end-tags for
      the variants used in the example above could have been omitted.
     </para>

     <para>
      Variant elements can be nested. The element
     </para>

     <para>

      <screen>
       &#60;title&#62;
       &#60;var lang lang "eng"&#62;&#60;var body iana "text/plain"&#62;
       Zen and the Art of Motorcycle Maintenance
       &#60;/title&#62;
      </screen>

     </para>

     <para>
      Associates two variant components to the variant list for the title
      element.
     </para>

     <para>
      Given the nesting rules described above, we could write
     </para>

     <para>

      <screen>
       &#60;title&#62;
       &#60;var body iana "text/plain&#62;
       &#60;var lang lang "eng"&#62;
       Zen and the Art of Motorcycle Maintenance
       &#60;var lang lang "dan"&#62;
       Zen og Kunsten at Vedligeholde en Motorcykel
       &#60;/title&#62;
      </screen>

     </para>

     <para>
      The title element above comes in two variants. Both have the IANA body
      type "text/plain", but one is in English, and the other in
      Danish. The client, using the element selection mechanism of &z3950;,
      can retrieve information about the available variant forms of data
      elements, or it can select specific variants based on the requirements
      of the end-user.
     </para>

    </section>

   </section>

   <section id="grs-regx-tcl">
    <title>&grs1; REGX And TCL Input Filters</title>

    <para>
     In order to handle general input formats, &zebra; allows the
     operator to define filters which read individual records in their
     native format and produce an internal representation that the system
     can work with.
    </para>

    <para>
     Input filters are ASCII files, generally with the suffix
     <literal>.flt</literal>.
     The system looks for the files in the directories given in the
     <emphasis>profilePath</emphasis> setting in the
     <literal>zebra.cfg</literal> files.
     The record type for the filter is
     <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
     (fundamental type <literal>grs</literal>, file read
     type <literal>regx</literal>, argument
     <emphasis>filter-filename</emphasis>).
    </para>
    
    <para>
     Generally, an input filter consists of a sequence of rules, where each
     rule consists of a sequence of expressions, followed by an action. The
     expressions are evaluated against the contents of the input record,
     and the actions normally contribute to the generation of an internal
     representation of the record.
    </para>
    
    <para>
     An expression can be either of the following:
    </para>

    <para>
     <variablelist>

      <varlistentry>
       <term><literal>INIT</literal></term>
       <listitem>
        <para>
         The action associated with this expression is evaluated
         exactly once in the lifetime of the application, before any records
         are read. It can be used in conjunction with an action that
         initializes tables or other resources that are used in the processing
         of input records.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>BEGIN</literal></term>
       <listitem>
        <para>
         Matches the beginning of the record. It can be used to
         initialize variables, etc. Typically, the
         <emphasis>BEGIN</emphasis> rule is also used
         to establish the root node of the record.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>END</literal></term>
       <listitem>
        <para>
         Matches the end of the record - when all of the contents
         of the record has been processed.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>/</literal><replaceable>reg</replaceable><literal>/</literal>
       </term>
       <listitem>
        <para>
         Matches regular expression pattern <replaceable>reg</replaceable>
         from the input record. The operators supported are the same
         as for regular expression queries. Refer to 
         <xref linkend="querymodel-regular"/>.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>BODY</literal></term>
       <listitem>
        <para>
         This keyword may only be used between two patterns.
         It matches everything between (not including) those patterns.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>FINISH</literal></term>
       <listitem>
        <para>
         The expression associated with this pattern is evaluated
         once, before the application terminates. It can be used to release
         system resources - typically ones allocated in the
         <emphasis>INIT</emphasis> step.
        </para>
       </listitem>
      </varlistentry>
     </variablelist>
    </para>

    <para>
     An action is surrounded by curly braces ({...}), and
     consists of a sequence of statements. Statements may be separated
     by newlines or semicolons (;).
     Within actions, the strings that matched the expressions
     immediately preceding the action can be referred to as
     $0, $1, $2, etc.
    </para>

    <para>
     The available statements are:
    </para>

    <para>
     <variablelist>

      <varlistentry>
       <term>begin <replaceable>type [parameter ... ]</replaceable></term>
       <listitem>
        <para>
         Begin a new
         data element. The <replaceable>type</replaceable> is one of
         the following:
         <variablelist>
          
          <varlistentry>
           <term>record</term>
           <listitem>
            <para>
             Begin a new record. The following parameter should be the
             name of the schema that describes the structure of the record, eg.
             <literal>gils</literal> or <literal>wais</literal> (see below).
             The <literal>begin record</literal> call should precede
             any other use of the <replaceable>begin</replaceable> statement.
            </para>
           </listitem>
          </varlistentry>
          <varlistentry>
           <term>element</term>
           <listitem>
            <para>
             Begin a new tagged element. The parameter is the
             name of the tag. If the tag is not matched anywhere in the tagsets
             referenced by the current schema, it is treated as a local string
             tag.
            </para>
           </listitem>
          </varlistentry>
          <varlistentry>
           <term>variant</term>
           <listitem>
            <para>
             Begin a new node in a variant tree. The parameters are
             <replaceable>class type value</replaceable>.
            </para>
           </listitem>
          </varlistentry>
         </variablelist>
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>data <replaceable>parameter</replaceable></term>
       <listitem>
        <para>
         Create a data element. The concatenated arguments make
         up the value of the data element.
         The option <literal>-text</literal> signals that
         the layout (whitespace) of the data should be retained for
         transmission.
         The option <literal>-element</literal>
         <replaceable>tag</replaceable> wraps the data up in
         the <replaceable>tag</replaceable>.
         The use of the <literal>-element</literal> option is equivalent to
         preceding the command with a <replaceable>begin
          element</replaceable> command, and following
         it with the <replaceable>end</replaceable> command.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>end <replaceable>[type]</replaceable></term>
       <listitem>
        <para>
         Close a tagged element. If no parameter is given,
         the last element on the stack is terminated.
         The first parameter, if any, is a type name, similar
         to the <replaceable>begin</replaceable> statement.
         For the <replaceable>element</replaceable> type, a tag
         name can be provided to terminate a specific tag.
        </para>
       </listitem>
      </varlistentry>

      <varlistentry>
       <term>unread <replaceable>no</replaceable></term>
       <listitem>
        <para>
         Move the input pointer to the offset of first character that
         match rule given by <replaceable>no</replaceable>.
         The first rule from left-to-right is numbered zero,
         the second rule is named 1 and so on.
        </para>
       </listitem>
      </varlistentry>

     </variablelist>
    </para>

    <para>
     The following input filter reads a Usenet news file, producing a
     record in the WAIS schema. Note that the body of a news posting is
     separated from the list of headers by a blank line (or rather a
     sequence of two newline characters.
    </para>

    <para>

     <screen>
      BEGIN                { begin record wais }

      /^From:/ BODY /$/    { data -element name $1 }
      /^Subject:/ BODY /$/ { data -element title $1 }
      /^Date:/ BODY /$/    { data -element lastModified $1 }
      /\n\n/ BODY END      {
         begin element bodyOfDisplay
         begin variant body iana "text/plain"
         data -text $1
         end record
      }
     </screen>

    </para>

    <para>
     If &zebra; is compiled with support for Tcl enabled, the statements
     described above are supplemented with a complete
     scripting environment, including control structures (conditional
     expressions and loop constructs), and powerful string manipulation
     mechanisms for modifying the elements of a record.
    </para>

   </section>

  </section>

  <section id="grs-internal-representation">
   <title>&grs1; Internal Record Representation</title>

   <para>
    When records are manipulated by the system, they're represented in a
    tree-structure, with data elements at the leaf nodes, and tags or
    variant components at the non-leaf nodes. The root-node identifies the
    schema that lends context to the tagging and structuring of the
    record. Imagine a simple record, consisting of a 'title' element and
    an 'author' element:
   </para>

   <para>

    <screen>
     ROOT 
        TITLE     "Zen and the Art of Motorcycle Maintenance"
        AUTHOR    "Robert Pirsig"
    </screen>

   </para>

   <para>
    A slightly more complex record would have the author element consist
    of two elements, a surname and a first name:
   </para>

   <para>

    <screen>
     ROOT  
        TITLE  "Zen and the Art of Motorcycle Maintenance"
        AUTHOR
           FIRST-NAME "Robert"
           SURNAME    "Pirsig"
    </screen>

   </para>

   <para>
    The root of the record will refer to the record schema that describes
    the structuring of this particular record. The schema defines the
    element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
    well as the structuring (SURNAME should appear below AUTHOR, etc.). In
    addition, the schema establishes element set names that are used by
    the client to request a subset of the elements of a given record. The
    schema may also establish rules for converting the record to a
    different schema, by stating, for each element, a mapping to a
    different tag path.
   </para>

   <section id="grs-tagged-elements">
    <title>Tagged Elements</title>

    <para>
     A data element is characterized by its tag, and its position in the
     structure of the record. For instance, while the tag "telephone
     number" may be used different places in a record, we may need to
     distinguish between these occurrences, both for searching and
     presentation purposes. For instance, while the phone numbers for the
     "customer" and the "service provider" are both
     representatives for the same type of resource (a telephone number), it
     is essential that they be kept separate. The record schema provides
     the structure of the record, and names each data element (defined by
     the sequence of tags - the tag path - by which the element can be
     reached from the root of the record).
    </para>

   </section>

   <section id="grs-variant-details">
    <title>Variants</title>

    <para>
     The children of a tag node may be either more tag nodes, a data node
     (possibly accompanied by tag nodes),
     or a tree of variant nodes. The children of  variant nodes are either
     more variant nodes or a data node (possibly accompanied by more
     variant nodes). Each leaf node, which is normally a
     data node, corresponds to a <emphasis>variant form</emphasis> of the
     tagged element identified by the tag which parents the variant tree.
     The following title element occurs in two different languages:
    </para>

    <para>

     <screen>
      VARIANT LANG=ENG  "War and Peace"
      TITLE
      VARIANT LANG=DAN  "Krig og Fred"
     </screen>

    </para>

    <para>
     Which of the two elements are transmitted to the client by the server
     depends on the specifications provided by the client, if any.
    </para>
    
    <para>
     In practice, each variant node is associated with a triple of class,
     type, value, corresponding to the variant mechanism of &z3950;.
    </para>
    
   </section>
   
   <section id="grs-data-elements">
    <title>Data Elements</title>
    
    <para>
     Data nodes have no children (they are always leaf nodes in the record
     tree).
    </para>
    
    <!--
    FIXME! Documentation needs extension here about types of nodes - numerical,
    textual, etc., plus the various types of inclusion notes.
   </para>
    -->
    
   </section>
   
  </section>
  
  <section id="grs-conf">
   <title>&grs1; Record Model Configuration</title>
   
   <para>
    The following sections describe the configuration files that govern
    the internal management of <literal>grs</literal> records. 
    The system searches for the files
    in the directories specified by the <emphasis>profilePath</emphasis>
    setting in the <literal>zebra.cfg</literal> file.
   </para>

   <section id="grs-abstract-syntax">
    <title>The Abstract Syntax</title>

    <para>
     The abstract syntax definition (also known as an Abstract Record
     Structure, or ARS) is the focal point of the
     record schema description. For a given schema, the ABS file may state any
     or all of the following:
    </para>

    <!--
     FIXME - Need a diagram here, or a simple explanation how it all hangs together -H
    -->

    <para>

     <itemizedlist>
      <listitem>

       <para>
        The object identifier of the &z3950; schema associated
        with the ARS, so that it can be referred to by the client.
       </para>
      </listitem>

      <listitem>
       <para>
        The attribute set (which can possibly be a compound of multiple
        sets) which applies in the profile. This is used when indexing and
        searching the records belonging to the given profile.
       </para>
      </listitem>

      <listitem>
       <para>
        The tag set (again, this can consist of several different sets).
        This is used when reading the records from a file, to recognize the
        different tags, and when transmitting the record to the client -
        mapping the tags to their numerical representation, if they are
        known.
       </para>
      </listitem>
      
      <listitem>
       <para>
        The variant set which is used in the profile. This provides a
        vocabulary for specifying the <emphasis>forms</emphasis> of
        data that appear inside the records.
       </para>
      </listitem>

      <listitem>
       <para>
        Element set names, which are a shorthand way for the client to
        ask for a subset of the data elements contained in a record. Element
        set names, in the retrieval module, are mapped to <emphasis>element
         specifications</emphasis>, which contain information equivalent to the
        <emphasis>Espec-1</emphasis> syntax of &z3950;.
       </para>
      </listitem>

      <listitem>
       <para>
        Map tables, which may specify mappings to
        <emphasis>other</emphasis> database profiles, if desired.
       </para>
      </listitem>

      <listitem>
       <para>
        Possibly, a set of rules describing the mapping of elements to a
        &marc; representation.

       </para>
      </listitem>

      <listitem>      
       <para>
        A list of element descriptions (this is the actual ARS of the
        schema, in &z3950; terms), which lists the ways in which the various
        tags can be used and organized hierarchically.
       </para>
      </listitem>

     </itemizedlist>

    </para>

    <para>
     Several of the entries above simply refer to other files, which
     describe the given objects.
    </para>

   </section>

   <section id="grs-configuration-files">
    <title>The Configuration Files</title>

    <para>
     This section describes the syntax and use of the various tables which
     are used by the retrieval module.
    </para>

    <para>
     The number of different file types may appear daunting at first, but
     each type corresponds fairly clearly to a single aspect of the &z3950;
     retrieval facilities. Further, the average database administrator,
     who is simply reusing an existing profile for which tables already
     exist, shouldn't have to worry too much about the contents of these tables.
    </para>

    <para>
     Generally, the files are simple ASCII files, which can be maintained
     using any text editor. Blank lines, and lines beginning with a (#) are
     ignored. Any characters on a line followed by a (#) are also ignored.
     All other lines contain <emphasis>directives</emphasis>, which provide
     some setting or value to the system.
     Generally, settings are characterized by a single
     keyword, identifying the setting, followed by a number of parameters.
     Some settings are repeatable (r), while others may occur only once in a
     file. Some settings are optional (o), while others again are
     mandatory (m).
    </para>
    
   </section>
   
   <section id="abs-file">
    <title>The Abstract Syntax (.abs) Files</title>
    
    <para>
     The name of this file type is slightly misleading in &z3950; terms,
     since, apart from the actual abstract syntax of the profile, it also
     includes most of the other definitions that go into a database
     profile.
    </para>
    
    <para>
     When a record in the canonical, &sgml;-like format is read from a file
     or from the database, the first tag of the file should reference the
     profile that governs the layout of the record. If the first tag of the
     record is, say, <literal>&lt;gils&gt;</literal>, the system will look
     for the profile definition in the file <literal>gils.abs</literal>.
     Profile definitions are cached, so they only have to be read once
     during the lifespan of the current process. 
    </para>

    <para>
     When writing your own input filters, the
     <emphasis>record-begin</emphasis> command
     introduces the profile, and should always be called first thing when
     introducing a new record.
    </para>
    
    <para>
     The file may contain the following directives:
    </para>
    
    <para>
     <variablelist>
      
      <varlistentry>
       <term>name <replaceable>symbolic-name</replaceable></term>
       <listitem>
        <para>
         (m) This provides a shorthand name or
         description for the profile. Mostly useful for diagnostic purposes.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>reference <replaceable>OID-name</replaceable></term>
       <listitem>
        <para>
         (m) The reference name of the OID for the profile.
         The reference names can be found in the <emphasis>util</emphasis>
         module of &yaz;.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>attset <replaceable>filename</replaceable></term>
       <listitem>
        <para>
         (m) The attribute set that is used for
         indexing and searching records belonging to this profile.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>tagset <replaceable>filename</replaceable></term>
       <listitem>
        <para>
         (o) The tag set (if any) that describe
         that fields of the records.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>varset <replaceable>filename</replaceable></term>
       <listitem>
        <para>
         (o) The variant set used in the profile.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>maptab <replaceable>filename</replaceable></term>
       <listitem>
        <para>
         (o,r) This points to a
         conversion table that might be used if the client asks for the record
         in a different schema from the native one.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>marc <replaceable>filename</replaceable></term>
       <listitem>
        <para>
         (o) Points to a file containing parameters
         for representing the record contents in the ISO2709 syntax.
         Read the description of the &marc; representation facility below.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>esetname <replaceable>name filename</replaceable></term>
       <listitem>
        <para>
         (o,r) Associates the
         given element set name with an element selection file. If an (@) is
         given in place of the filename, this corresponds to a null mapping for
         the given element set name.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>all <replaceable>tags</replaceable></term>
       <listitem>
        <para>
         (o) This directive specifies a list of attributes
         which should be appended to the attribute list given for each
         element. The effect is to make every single element in the abstract
         syntax searchable by way of the given attributes. This directive
         provides an efficient way of supporting free-text searching across all
         elements. However, it does increase the size of the index
         significantly. The attributes can be qualified with a structure, as in
         the <replaceable>elm</replaceable> directive below.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>elm <replaceable>path name attributes</replaceable></term>
       <listitem>
        <para>
         (o,r) Adds an element to the abstract record syntax of the schema.
         The <replaceable>path</replaceable> follows the
         syntax which is suggested by the &z3950; document - that is, a sequence
         of tags separated by slashes (&#x2f;). Each tag is given as a
         comma-separated pair of tag type and -value surrounded by parenthesis.
         The <replaceable>name</replaceable> is the name of the element, and
         the <replaceable>attributes</replaceable>
         specifies which attributes to use when indexing the element in a
         comma-separated list.
         A <literal>!</literal> in place of the attribute name is equivalent
         to specifying an attribute name identical to the element name.
         A <literal>-</literal> in place of the attribute name
         specifies that no indexing is to take place for the given element.
         The attributes can be qualified with <replaceable>field
          types</replaceable> to specify which
         character set should govern the indexing procedure for that field.
         The same data element may be indexed into several different
         fields, using different character set definitions.
         See the <xref linkend="fields-and-charsets"/>.
         The default field type is <literal>w</literal> for
         <emphasis>word</emphasis>.
        </para>
       </listitem>
      </varlistentry>
      
      <varlistentry>
       <term>xelm <replaceable>xpath attributes</replaceable></term>
       <listitem>
        <para>
         Specifies indexing for record nodes given by
         <replaceable>xpath</replaceable>. Unlike directive
         elm, this directive allows you to index attribute
         contents. The <replaceable>xpath</replaceable> uses
         a syntax similar to XPath. The <replaceable>attributes</replaceable>
         have same syntax and meaning as directive elm, except that operator
         ! refers to the nodes selected by <replaceable>xpath</replaceable>.
         <!--
         xelm   /         !:w                 default index
         xelm   //        !:w                 additional index
         xelm   /gils/title/@att    myatt:w   index attribute @att in myatt
         xelm   title/@att          myatt:w   same meaning.
         -->
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>melm <replaceable>field$subfield attributes</replaceable></term>
       <listitem>
        <para>
         This directive is specifically for &marc;-formatted records,
         ingested either in the form of &marcxml; documents, or in the
         ISO2709/Z39.2 format using the grs.marcxml input filter. You can
         specify indexing rules for any subfield, or you can leave off the
         <replaceable>$subfield</replaceable> part and specify default rules
         for all subfields of the given field (note: default rules should come
         after any subfield-specific rules in the configuration file). The
         <replaceable>attributes</replaceable> have the same syntax and meaning
         as for the 'elm' directive above.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>encoding <replaceable>encodingname</replaceable></term>
       <listitem>
        <para>
         This directive specifies character encoding for external records.
         For records such as &xml; that specifies encoding within the
         file via a header this directive is ignored.
         If neither this directive is given, nor an encoding is set
         within external records, ISO-8859-1 encoding is assumed.
         </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>xpath <literal>enable</literal>/<literal>disable</literal></term>
       <listitem>
        <para>
         If this directive is followed by <literal>enable</literal>,
         then extra indexing is performed to allow for XPath-like queries.
         If this directive is not specified - equivalent to 
         <literal>disable</literal> - no extra XPath-indexing is performed.
        </para>
       </listitem>
      </varlistentry>

      <!-- Adam's version 
      <varlistentry>
       <term>systag <replaceable>systemtag</replaceable> <replaceable>element</replaceable></term>
       <listitem>
        <para>
         This directive maps system information to an element during
         retrieval. This information is dynamically created. The
         following system tags are defined
         <variablelist>
          <varlistentry>
           <term>size</term>
           <listitem>
            <para>
             Size of record in bytes. By default this
             is mapped to element <literal>size</literal>.
            </para>
           </listitem>
          </varlistentry>

          <varlistentry>
           <term>rank</term>
           <listitem>
            <para>
             Score/rank of record. By default this
             is mapped to element <literal>rank</literal>.
             If no score was calculated for the record (non-ranked
             searched) search this directive is ignored.
            </para>
           </listitem>
          </varlistentry>
          
          <varlistentry>
           <term>sysno</term>
           <listitem> 
            <para>
             &zebra;'s system number (record ID) for the
             record. By default this is mapped to element
             <literal>localControlNumber</literal>.
            </para>
           </listitem>
          </varlistentry>
         </variablelist>
         If you do not want a particular system tag to be applied,
         then set the resulting element to something undefined in the
         abs file (such as <literal>none</literal>).
        </para>
       </listitem>
      </varlistentry>
      -->

      <!-- Mike's version -->
      <varlistentry>
       <term>
        systag
        <replaceable>systemTag</replaceable>
        <replaceable>actualTag</replaceable>
       </term>
       <listitem>
        <para>
         Specifies what information, if any, &zebra; should
         automatically include in retrieval records for the 
         ``system fields'' that it supports.
         <replaceable>systemTag</replaceable> may
         be any of the following:
         <variablelist>
          <varlistentry>
           <term><literal>rank</literal></term>
           <listitem><para>
            An integer indicating the relevance-ranking score
            assigned to the record.
           </para></listitem>
          </varlistentry>
          <varlistentry>
           <term><literal>sysno</literal></term>
           <listitem><para>
            An automatically generated identifier for the record,
            unique within this database.  It is represented by the
            <literal>&lt;localControlNumber&gt;</literal> element in
            &xml; and the <literal>(1,14)</literal> tag in &grs1;.
           </para></listitem>
          </varlistentry>
          <varlistentry>
           <term><literal>size</literal></term>
           <listitem><para>
            The size, in bytes, of the retrieved record.
           </para></listitem>
          </varlistentry>
         </variablelist>
        </para>
        <para>
         The <replaceable>actualTag</replaceable> parameter may be
         <literal>none</literal> to indicate that the named element
         should be omitted from retrieval records.
        </para>
       </listitem>
      </varlistentry>
     </variablelist>
    </para>
    
    <note>
     <para>
      The mechanism for controlling indexing is not adequate for
      complex databases, and will probably be moved into a separate
      configuration table eventually.
     </para>
    </note>
    
    <para>
     The following is an excerpt from the abstract syntax file for the GILS
     profile.
    </para>

    <para>

     <screen>
      name gils
      reference GILS-schema
      attset gils.att
      tagset gils.tag
      varset var1.var

      maptab gils-usmarc.map

      # Element set names

      esetname VARIANT gils-variant.est  # for WAIS-compliance
      esetname B gils-b.est
      esetname G gils-g.est
      esetname F @

      elm (1,10)               rank                        -
      elm (1,12)               url                         -
      elm (1,14)               localControlNumber     Local-number
      elm (1,16)               dateOfLastModification Date/time-last-modified
      elm (2,1)                title                       w:!,p:!
      elm (4,1)                controlIdentifier      Identifier-standard
      elm (2,6)                abstract               Abstract
      elm (4,51)               purpose                     !
      elm (4,52)               originator                  - 
      elm (4,53)               accessConstraints           !
      elm (4,54)               useConstraints              !
      elm (4,70)               availability                -
      elm (4,70)/(4,90)        distributor                 -
      elm (4,70)/(4,90)/(2,7)  distributorName             !
      elm (4,70)/(4,90)/(2,10) distributorOrganization     !
      elm (4,70)/(4,90)/(4,2)  distributorStreetAddress    !
      elm (4,70)/(4,90)/(4,3)  distributorCity             !
     </screen>

    </para>

   </section>

   <section id="attset-files">
    <title>The Attribute Set (.att) Files</title>

    <para>
     This file type describes the <replaceable>Use</replaceable> elements of
     an attribute set. 
     It contains the following directives. 
    </para>
    
    <para>
     <variablelist>
      <varlistentry>
       <term>name <replaceable>symbolic-name</replaceable></term>
       <listitem>
        <para>
         (m) This provides a shorthand name or
         description for the attribute set.
         Mostly useful for diagnostic purposes.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>reference <replaceable>OID-name</replaceable></term>
       <listitem>
        <para>
         (m) The reference name of the OID for
         the attribute set.
         The reference names can be found in the <replaceable>util</replaceable>
         module of <replaceable>&yaz;</replaceable>.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>include <replaceable>filename</replaceable></term>
       <listitem>
        <para>
         (o,r) This directive is used to
         include another attribute set as a part of the current one. This is
         used when a new attribute set is defined as an extension to another
         set. For instance, many new attribute sets are defined as extensions
         to the <replaceable>bib-1</replaceable> set.
         This is an important feature of the retrieval
         system of &z3950;, as it ensures the highest possible level of
         interoperability, as those access points of your database which are
         derived from the external set (say, bib-1) can be used even by clients
         who are unaware of the new set.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>att
        <replaceable>att-value att-name [local-value]</replaceable></term>
       <listitem>
        <para>
         (o,r) This
         repeatable directive introduces a new attribute to the set. The
         attribute value is stored in the index (unless a
         <replaceable>local-value</replaceable> is
         given, in which case this is stored). The name is used to refer to the
         attribute from the <replaceable>abstract syntax</replaceable>. 
        </para>
       </listitem></varlistentry>
     </variablelist>
    </para>

    <para>
     This is an excerpt from the GILS attribute set definition.
     Notice how the file describing the <emphasis>bib-1</emphasis>
     attribute set is referenced.
    </para>

    <para>

     <screen>
      name gils
      reference GILS-attset
      include bib1.att

      att 2001          distributorName
      att 2002          indextermsControlled
      att 2003          purpose
      att 2004          accessConstraints
      att 2005          useConstraints
     </screen>

    </para>

   </section>

   <section id="grs-tag-files">
    <title>The Tag Set (.tag) Files</title>

    <para>
     This file type defines the tagset of the profile, possibly by
     referencing other tag sets (most tag sets, for instance, will include
     tagsetG and tagsetM from the &z3950; specification. The file may
     contain the following directives.
    </para>

    <para>
     <variablelist>

      <varlistentry>
       <term>name <emphasis>symbolic-name</emphasis></term>
       <listitem>
        <para>
         (m) This provides a shorthand name or
         description for the tag set. Mostly useful for diagnostic purposes.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>reference <emphasis>OID-name</emphasis></term>
       <listitem>
        <para>
         (o) The reference name of the OID for the tag set.
         The reference names can be found in the <emphasis>util</emphasis>
         module of <emphasis>&yaz;</emphasis>.
         The directive is optional, since not all tag sets
         are registered outside of their schema.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>type <emphasis>integer</emphasis></term>
       <listitem>
        <para>
         (m) The type number of the tagset within the schema
         profile (note: this specification really should belong to the .abs
         file. This will be fixed in a future release).
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>include <emphasis>filename</emphasis></term>
       <listitem>
        <para>
         (o,r) This directive is used
         to include the definitions of other tag sets into the current one.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>tag <emphasis>number names type</emphasis></term>
       <listitem>
        <para>
         (o,r) Introduces a new tag to the set.
         The <emphasis>number</emphasis> is the tag number as used
         in the protocol (there is currently no mechanism for
         specifying string tags at this point, but this would be quick
         work to add).
         The <emphasis>names</emphasis> parameter is a list of names
         by which the tag should be recognized in the input file format.
         The names should be separated by slashes (/).
         The <emphasis>type</emphasis> is the recommended data type of
         the tag.
         It should be one of the following:

         <itemizedlist>
          <listitem>
           <para>
            structured
           </para>
          </listitem>

          <listitem>
           <para>
            string
           </para>
          </listitem>

          <listitem>
           <para>
            numeric
           </para>
          </listitem>

          <listitem>
           <para>
            bool
           </para>
          </listitem>

          <listitem>
           <para>
            oid
           </para>
          </listitem>

          <listitem>
           <para>
            generalizedtime
           </para>
          </listitem>

          <listitem>
           <para>
            intunit
           </para>
          </listitem>

          <listitem>
           <para>
            int
           </para>
          </listitem>

          <listitem>
           <para>
            octetstring
           </para>
          </listitem>

          <listitem>
           <para>
            null
           </para>
          </listitem>

         </itemizedlist>

        </para>
       </listitem></varlistentry>
     </variablelist>
    </para>

    <para>
     The following is an excerpt from the TagsetG definition file.
    </para>

    <para>
     <screen>
      name tagsetg
      reference TagsetG
      type 2

      tag       1       title           string
      tag       2       author          string
      tag       3       publicationPlace string
      tag       4       publicationDate string
      tag       5       documentId      string
      tag       6       abstract        string
      tag       7       name            string
      tag       8       date            generalizedtime
      tag       9       bodyOfDisplay   string
      tag       10      organization    string
     </screen>
    </para>

   </section>

   <section id="grs-var-files">
    <title>The Variant Set (.var) Files</title>

    <para>
     The variant set file is a straightforward representation of the
     variant set definitions associated with the protocol. At present, only
     the <emphasis>Variant-1</emphasis> set is known.
    </para>

    <para>
     These are the directives allowed in the file.
    </para>

    <para>
     <variablelist>

      <varlistentry>
       <term>name <emphasis>symbolic-name</emphasis></term>
       <listitem>
        <para>
         (m) This provides a shorthand name or
         description for the variant set. Mostly useful for diagnostic purposes.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>reference <emphasis>OID-name</emphasis></term>
       <listitem>
        <para>
         (o) The reference name of the OID for
         the variant set, if one is required. The reference names can be found
         in the <emphasis>util</emphasis> module of <emphasis>&yaz;</emphasis>.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>class <emphasis>integer class-name</emphasis></term>
       <listitem>
        <para>
         (m,r) Introduces a new
         class to the variant set.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>type <emphasis>integer type-name datatype</emphasis></term>
       <listitem>
        <para>
         (m,r) Addes a
         new type to the current class (the one introduced by the most recent
         <emphasis>class</emphasis> directive).
         The type names belong to the same name space as the one used
         in the tag set definition file.
        </para>
       </listitem></varlistentry>
     </variablelist>
    </para>

    <para>
     The following is an excerpt from the file describing the variant set
     <emphasis>Variant-1</emphasis>.
    </para>

    <para>

     <screen>
      name variant-1
      reference Variant-1

      class 1 variantId

      type      1       variantId               octetstring

      class 2 body

      type      1       iana                    string
      type      2       z39.50                  string
      type      3       other                   string
     </screen>

    </para>

   </section>

   <section id="grs-est-files">
    <title>The Element Set (.est) Files</title>

    <para>
     The element set specification files describe a selection of a subset
     of the elements of a database record. The element selection mechanism
     is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
     syntax of the &z3950; specification.
     In fact, the internal representation of an element set
     specification is identical to the <emphasis>Espec-1</emphasis> structure,
     and we'll refer you to the description of that structure for most of
     the detailed semantics of the directives below.
    </para>

    <note>
     <para>
      Not all of the Espec-1 functionality has been implemented yet.
      The fields that are mentioned below all work as expected, unless
      otherwise is noted.
     </para>
    </note>
    
    <para>
     The directives available in the element set file are as follows:
    </para>

    <para>
     <variablelist>
      <varlistentry>
       <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
       <listitem>
        <para>
         (o) If variants are used in
         the following, this should provide the name of the variantset used
         (it's not currently possible to specify a different set in the
         individual variant request). In almost all cases (certainly all
         profiles known to us), the name
         <literal>Variant-1</literal> should be given here.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
       <listitem>
        <para>
         (o) This directive
         provides a default variant request for
         use when the individual element requests (see below) do not contain a
         variant request. Variant requests consist of a blank-separated list of
         variant components. A variant compont is a comma-separated,
         parenthesized triple of variant class, type, and value (the two former
         values being represented as integers). The value can currently only be
         entered as a string (this will change to depend on the definition of
         the variant in question). The special value (@) is interpreted as a
         null value, however.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>simpleElement
        <emphasis>path ['variant' variant-request]</emphasis></term>
       <listitem>
        <para>
         (o,r) This corresponds to a simple element request
         in <emphasis>Espec-1</emphasis>.
         The path consists of a sequence of tag-selectors, where each of
         these can consist of either:
        </para>

        <para>
         <itemizedlist>
          <listitem>
           <para>
            A simple tag, consisting of a comma-separated type-value pair in
            parenthesis, possibly followed by a colon (:) followed by an
            occurrences-specification (see below). The tag-value can be a number
            or a string. If the first character is an apostrophe ('), this
            forces the value to be interpreted as a string, even if it
            appears to be numerical.
           </para>
          </listitem>

          <listitem>
           <para>
            A WildThing, represented as a question mark (?), possibly
            followed by a colon (:) followed by an occurrences
            specification (see below).
           </para>
          </listitem>

          <listitem>
           <para>
            A WildPath, represented as an asterisk (*). Note that the last
            element of the path should not be a wildPath (wildpaths don't
            work in this version).
           </para>
          </listitem>

         </itemizedlist>

        </para>

        <para>
         The occurrences-specification can be either the string
         <literal>all</literal>, the string <literal>last</literal>, or
         an explicit value-range. The value-range is represented as
         an integer (the starting point), possibly followed by a
         plus (+) and a second integer (the number of elements, default
         being one).
        </para>

        <para>
         The variant-request has the same syntax as the defaultVariantRequest
         above. Note that it may sometimes be useful to give an empty variant
         request, simply to disable the default for a specific set of fields
         (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
         but it works in this implementation).
        </para>
       </listitem></varlistentry>
     </variablelist>
    </para>

    <para>
     The following is an example of an element specification belonging to
     the GILS profile.
    </para>

    <para>

     <screen>
      simpleelement (1,10)
      simpleelement (1,12)
      simpleelement (2,1)
      simpleelement (1,14)
      simpleelement (4,1)
      simpleelement (4,52)
     </screen>

    </para>

   </section>

   <section id="schema-mapping">
    <title>The Schema Mapping (.map) Files</title>

    <para>
     Sometimes, the client might want to receive a database record in
     a schema that differs from the native schema of the record. For
     instance, a client might only know how to process WAIS records, while
     the database record is represented in a more specific schema, such as
     GILS. In this module, a mapping of data to one of the &marc; formats is
     also thought of as a schema mapping (mapping the elements of the
     record into fields consistent with the given &marc; specification, prior
     to actually converting the data to the ISO2709). This use of the
     object identifier for &usmarc; as a schema identifier represents an
     overloading of the OID which might not be entirely proper. However,
     it represents the dual role of schema and record syntax which
     is assumed by the &marc; family in &z3950;.
    </para>

    <!--
     <emphasis>NOTE: FIXME! The schema-mapping functions are so far limited to a
      straightforward mapping of elements. This should be extended with
      mechanisms for conversions of the element contents, and conditional
      mappings of elements based on the record contents.</emphasis>
    -->

    <para>
     These are the directives of the schema mapping file format:
    </para>

    <para>
     <variablelist>

      <varlistentry>
       <term>targetName <emphasis>name</emphasis></term>
       <listitem>
        <para>
         (m) A symbolic name for the target schema
         of the table. Useful mostly for diagnostic purposes.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>targetRef <emphasis>OID-name</emphasis></term>
       <listitem>
        <para>
         (m) An OID name for the target schema.
         This is used, for instance, by a server receiving a request to present
         a record in a different schema from the native one.
         The name, again, is found in the <emphasis>oid</emphasis>
         module of <emphasis>&yaz;</emphasis>.
        </para>
       </listitem></varlistentry>
      <varlistentry>
       <term>map <emphasis>element-name target-path</emphasis></term>
       <listitem>
        <para>
         (o,r) Adds
         an element mapping rule to the table.
        </para>
       </listitem></varlistentry>
     </variablelist>
    </para>

   </section>

   <section id="grs-mar-files">
    <title>The &marc; (ISO2709) Representation (.mar) Files</title>

    <para>
     This file provides rules for representing a record in the ISO2709
     format. The rules pertain mostly to the values of the constant-length
     header of the record.
    </para>

    <!--
     NOTE: FIXME! This will be described better. We're in the process of
      re-evaluating and most likely changing the way that &marc; records are
      handled by the system.</emphasis>
    -->

   </section>
  </section>

  <section id="grs-exchange-formats">
   <title>&grs1; Exchange Formats</title>

   <para>
    Converting records from the internal structure to an exchange format
    is largely an automatic process. Currently, the following exchange
    formats are supported:
   </para>

   <para>
    <itemizedlist>
     <listitem>
      <para>
       &grs1;. The internal representation is based on &grs1;/&xml;, so the
       conversion here is straightforward. The system will create
       applied variant and supported variant lists as required, if a record
       contains variant information.
      </para>
     </listitem>

     <listitem>
      <para>
       &xml;. The internal representation is based on &grs1;/&xml; so
       the mapping is trivial. Note that &xml; schemas, preprocessing
       instructions and comments are not part of the internal representation
       and therefore will never be part of a generated &xml; record.
       Future versions of the &zebra; will support that.
      </para>
     </listitem>

     <listitem>
      <para>
       &sutrs;. Again, the mapping is fairly straightforward. Indentation
       is used to show the hierarchical structure of the record. All
       "&grs1;" type records support both the &grs1; and &sutrs;
       representations.
       <!-- FIXME - What is &sutrs; - should be expanded here -->
      </para>
     </listitem>

     <listitem>
      <para>
       ISO2709-based formats (&usmarc;, etc.). Only records with a
       two-level structure (corresponding to fields and subfields) can be
       directly mapped to ISO2709. For records with a different structuring
       (eg., GILS), the representation in a structure like &usmarc; involves a
       schema-mapping (see <xref linkend="schema-mapping"/>), to an
       "implied" &usmarc; schema (implied,
       because there is no formal schema which specifies the use of the
       &usmarc; fields outside of ISO2709). The resultant, two-level record is
       then mapped directly from the internal representation to ISO2709. See
       the GILS schema definition files for a detailed example of this
       approach.
      </para>
     </listitem>

     <listitem>
      <para>
       Explain. This representation is only available for records
       belonging to the Explain schema.
      </para>
     </listitem>

     <listitem>
      <para>
       Summary. This ASN-1 based structure is only available for records
       belonging to the Summary schema - or schema which provide a mapping
       to this schema (see the description of the schema mapping facility
       above).
      </para>
     </listitem>

     <!-- FIXME - Is this used anywhere ? -H -->  
     <listitem>
      <para>
       SOIF. Support for this syntax is experimental, and is currently
       keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
       abstract syntaxes can be mapped to the SOIF format, although nested
       elements are represented by concatenation of the tag names at each
       level.
      </para>
     </listitem>
   
    </itemizedlist>
   </para>
  </section>
  
  <section id="grs-extended-marc-indexing">
   <title>Extended indexing of &marc; records</title>
   
   <para>Extended indexing of &marc; records will help you if you need index a
    combination of subfields, or index only a part of the whole field,
    or use during indexing process embedded fields of &marc; record.
   </para>
   
   <para>Extended indexing of &marc; records additionally allows:
    <itemizedlist>
     
     <listitem>
      <para>to index data in LEADER of &marc; record</para>
     </listitem>
     
     <listitem>
      <para>to index data in control fields (with fixed length)</para>
     </listitem>
     
     <listitem>
      <para>to use during indexing the values of indicators</para>
     </listitem>
     
     <listitem>
      <para>to index linked fields for UNI&marc; based formats</para>
     </listitem>
     
    </itemizedlist>
   </para>
   
   <note><para>In compare with simple indexing process the extended indexing
     may increase (about 2-3 times) the time of indexing process for &marc;
     records.</para></note>
   
   <section id="formula">
    <title>The index-formula</title>
    
    <para>At the beginning, we have to define the term
     <emphasis>index-formula</emphasis> for &marc; records. This term helps
     to understand the notation of extended indexing of &marc; records by &zebra;.
     Our definition is based on the document
     <ulink url="http://www.rba.ru/rusmarc/soft/Z39-50.htm">"The table
      of conformity for &z3950; use attributes and R&usmarc; fields"</ulink>.
     The document is available only in russian language.</para>
    
    <para>
     The <emphasis>index-formula</emphasis> is the combination of
     subfields presented in such way:
    </para>
    
    <screen>
     71-00$a, $g, $h ($c){.$b ($c)} , (1)
    </screen>
    
    <para>
     We know that &zebra; supports a &bib1; attribute - right truncation.
     In this case, the <emphasis>index-formula</emphasis> (1) consists from 
     forms, defined in the same way as (1)</para>
    
    <screen>
     71-00$a, $g, $h
     71-00$a, $g
     71-00$a
    </screen>
    
    <note>
     <para>The original &marc; record may be without some elements, which included in <emphasis>index-formula</emphasis>.
     </para>
    </note>
    
    <para>This notation includes such operands as:
     <variablelist>
      
      <varlistentry>
       <term>#</term>
       <listitem><para>It means whitespace character.</para></listitem>
      </varlistentry>
      
      <varlistentry>
       <term>-</term>
       <listitem><para>The position may contain any value, defined by
         &marc; format.
         For example, <emphasis>index-formula</emphasis></para>
        
        <screen>
         70-#1$a, $g , (2)
        </screen>
        
        <para>includes</para> 
        
        <screen>
         700#1$a, $g
         701#1$a, $g
         702#1$a, $g
        </screen>
        
       </listitem>
      </varlistentry>
      
      <varlistentry>
       <term>{...}</term>
       <listitem>
        <para>The repeatable elements are defined in figure-brackets {}.
         For example,
         <emphasis>index-formula</emphasis></para>
        
        <screen>
         71-00$a, $g, $h ($c){.$b ($c)} , (3)
        </screen>
        
        <para>includes</para>
        
        <screen>
         71-00$a, $g, $h ($c). $b ($c)
         71-00$a, $g, $h ($c). $b ($c). $b ($c)
         71-00$a, $g, $h ($c). $b ($c). $b ($c). $b ($c)
        </screen>
        
       </listitem>
      </varlistentry>
     </variablelist>
     
     <note>
      <para>
       All another operands are the same as accepted in &marc; world.
      </para>
     </note>
    </para>
   </section>
   
   <section id="notation">
    <title>Notation of <emphasis>index-formula</emphasis> for &zebra;</title>
    
    
    <para>Extended indexing overloads <literal>path</literal> of
     <literal>elm</literal> definition in abstract syntax file of &zebra;
     (<literal>.abs</literal> file). It means that names beginning with
     <literal>"mc-"</literal> are interpreted by &zebra; as
     <emphasis>index-formula</emphasis>. The database index is created and
     linked with <emphasis>access point</emphasis> (&bib1; use attribute)
     according to this formula.</para>
    
    <para>For example, <emphasis>index-formula</emphasis></para>
    
    <screen>
     71-00$a, $g, $h ($c){.$b ($c)} , (4)
    </screen>
    
    <para>in <literal>.abs</literal> file looks like:</para>
    
    <screen>
     mc-71.00_$a,_$g,_$h_(_$c_){.$b_(_$c_)}
    </screen>
    
    
    <para>The notation of <emphasis>index-formula</emphasis> uses the operands:
     <variablelist>
      
      <varlistentry>
       <term>_</term>
       <listitem><para>It means whitespace character.</para></listitem>
      </varlistentry>
      
      <varlistentry>
       <term>.</term>
       <listitem><para>The position may contain any value, defined by
         &marc; format. For example,
         <emphasis>index-formula</emphasis></para>
        
        <screen>
         70-#1$a, $g , (5)
        </screen>
        
        <para>matches <literal>mc-70._1_$a,_$g_</literal> and includes</para>
        
        <screen>
         700_1_$a,_$g_
         701_1_$a,_$g_
         702_1_$a,_$g_
        </screen>
       </listitem>
      </varlistentry>
      
      <varlistentry>
       <term>{...}</term>
       <listitem><para>The repeatable elements are defined in
         figure-brackets {}. For example,
         <emphasis>index-formula</emphasis></para>
        
        <screen>
         71#00$a, $g, $h ($c) {.$b ($c)} , (6)
        </screen>
        
        <para>matches 
         <literal>mc-71.00_$a,_$g,_$h_(_$c_){.$b_(_$c_)}</literal> and
         includes</para>
        
        <screen>
         71.00_$a,_$g,_$h_(_$c_).$b_(_$c_)
         71.00_$a,_$g,_$h_(_$c_).$b_(_$c_).$b_(_$c_)
         71.00_$a,_$g,_$h_(_$c_).$b_(_$c_).$b_(_$c_).$b_(_$c_)
        </screen>
       </listitem>
      </varlistentry>
      
      <varlistentry>
       <term>&#60;...&#62;</term>
       <listitem><para>Embedded <emphasis>index-formula</emphasis> (for
         linked fields) is between &#60;&#62;. For example,
         <emphasis>index-formula</emphasis>
        </para>
        
        <screen>
         4--#-$170-#1$a, $g ($c) , (7)
        </screen>
        
        <para>matches
         <literal>mc-4.._._$1&#60;70._1_$a,_$g_(_$c_)&#62;_</literal> and
         includes</para>
        
        <screen>
         463_._$1&#60;70._1_$a,_$g_(_$c_)&#62;_
        </screen>
        
       </listitem>
      </varlistentry>
     </variablelist>
    </para>
    
    <note>
     <para>All another operands are the same as accepted in &marc; world.</para>
    </note>
    
    <section id="grs-examples">
     <title>Examples</title>
     
     <para>
      <orderedlist>
       
       <listitem>
        
        <para>indexing LEADER</para>
        
        <para>You need to use keyword "ldr" to index leader. For example,
         indexing data from 6th and 7th position of LEADER</para>
        
        <screen>
         elm mc-ldr[6] Record-type !
         elm mc-ldr[7] Bib-level   !
        </screen>
        
       </listitem>
       
       <listitem>
        
        <para>indexing data from control fields</para>
        
        <para>indexing date (the time added to database)</para>
        
        <screen>
         elm mc-008[0-5] Date/time-added-to-db !        
        </screen>
        
        <para>or for R&usmarc; (this data included in 100th field)</para>
        
        <screen>
         elm mc-100___$a[0-7]_ Date/time-added-to-db !
        </screen>
        
       </listitem>
       
       <listitem>
        
        <para>using indicators while indexing</para>

        <para>For R&usmarc; <emphasis>index-formula</emphasis>
         <literal>70-#1$a, $g</literal> matches</para>
        
        <screen>
         elm 70._1_$a,_$g_ Author !:w,!:p
        </screen>
        
        <para>When &zebra; finds a field according to 
         <literal>"70."</literal> pattern it checks the indicators. In this
         case the value of first indicator doesn't mater, but the value of
         second one must be whitespace, in another case a field is not 
         indexed.</para>
       </listitem>
       
       <listitem>
        
        <para>indexing embedded (linked) fields for UNI&marc; based
         formats</para>
        
        <para>For R&usmarc; <emphasis>index-formula</emphasis> 
         <literal>4--#-$170-#1$a, $g ($c)</literal> matches</para>
        
        <screen><![CDATA[
         elm mc-4.._._$1<70._1_$a,_$g_(_$c_)>_ Author !:w,!:p
         ]]></screen>
        
        <para>Data are extracted from record if the field matches to
         <literal>"4.._."</literal> pattern and data in linked field
         match to embedded
         <emphasis>index-formula</emphasis>
         <literal>70._1_$a,_$g_(_$c_)</literal>.</para>
        
       </listitem>
       
      </orderedlist>
     </para>
     
     
    </section>
   </section>

  </section>
  
 </chapter>
 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:t
 sgml-parent-document: "zebra.xml"
 sgml-local-catalogs: nil
 sgml-namecase-general:t
 End:
 -->