Version 2.0.59

[idzebra-moved-to-github.git] / doc / administration.xml

 <chapter id="administration">
  <title>Administrating &zebra;</title>
  <!-- ### It's a bit daft that this chapter (which describes half of
  the configuration-file formats) is separated from
  "recordmodel-grs.xml" (which describes the other half) by the
  instructions on running zebraidx and zebrasrv.  Some careful
  re-ordering is required here.
  -->

  <para>
   Unlike many simpler retrieval systems, &zebra; supports safe, incremental
   updates to an existing index.
  </para>

  <para>
   Normally, when &zebra; modifies the index it reads a number of records
   that you specify.
   Depending on your specifications and on the contents of each record
   one the following events take place for each record:
   <variablelist>

    <varlistentry>
     <term>Insert</term>
     <listitem>
      <para>
       The record is indexed as if it never occurred before.
       Either the &zebra; system doesn't know how to identify the record or
       &zebra; can identify the record but didn't find it to be already indexed.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>Modify</term>
     <listitem>
      <para>
       The record has already been indexed.
       In this case either the contents of the record or the location
       (file) of the record indicates that it has been indexed before.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>Delete</term>
     <listitem>
      <para>
       The record is deleted from the index. As in the
       update-case it must be able to identify the record.
      </para>
     </listitem>
    </varlistentry>
   </variablelist>
  </para>

  <para>
   Please note that in both the modify- and delete- case the &zebra;
   indexer must be able to generate a unique key that identifies the record
   in question (more on this below).
  </para>

  <para>
   To administrate the &zebra; retrieval system, you run the
   <literal>zebraidx</literal> program.
   This program supports a number of options which are preceded by a dash,
   and a few commands (not preceded by dash).
  </para>

  <para>
   Both the &zebra; administrative tool and the &acro.z3950; server share a
   set of index files and a global configuration file.
   The name of the configuration file defaults to
   <literal>zebra.cfg</literal>.
   The configuration file includes specifications on how to index
   various kinds of records and where the other configuration files
   are located. <literal>zebrasrv</literal> and <literal>zebraidx</literal>
   <emphasis>must</emphasis> be run in the directory where the
   configuration file lives unless you indicate the location of the
   configuration file by option <literal>-c</literal>.
  </para>

  <sect1 id="record-types">
   <title>Record Types</title>

   <para>
    Indexing is a per-record process, in which either insert/modify/delete
    will occur. Before a record is indexed search keys are extracted from
    whatever might be the layout the original record (sgml,html,text, etc..).
    The &zebra; system currently supports two fundamental types of records:
    structured and simple text.
    To specify a particular extraction process, use either the
    command line option <literal>-t</literal> or specify a
    <literal>recordType</literal> setting in the configuration file.
   </para>

  </sect1>

  <sect1 id="zebra-cfg">
   <title>The &zebra; Configuration File</title>

   <para>
    The &zebra; configuration file, read by <literal>zebraidx</literal> and
    <literal>zebrasrv</literal> defaults to <literal>zebra.cfg</literal>
    unless specified by <literal>-c</literal> option.
   </para>

   <para>
    You can edit the configuration file with a normal text editor.
    parameter names and values are separated by colons in the file. Lines
    starting with a hash sign (<literal>#</literal>) are
    treated as comments.
   </para>

   <para>
    If you manage different sets of records that share common
    characteristics, you can organize the configuration settings for each
    type into "groups".
    When <literal>zebraidx</literal> is run and you wish to address a
    given group you specify the group name with the <literal>-g</literal>
    option.
    In this case settings that have the group name as their prefix
    will be used by <literal>zebraidx</literal>.
    If no <literal>-g</literal> option is specified, the settings
    without prefix are used.
   </para>

   <para>
    In the configuration file, the group name is placed before the option
    name itself, separated by a dot (.). For instance, to set the record type
    for group <literal>public</literal> to <literal>grs.sgml</literal>
    (the &acro.sgml;-like format for structured records) you would write:
   </para>

   <para>
    <screen>
     public.recordType: grs.sgml
    </screen>
   </para>

   <para>
    To set the default value of the record type to <literal>text</literal>
    write:
   </para>

   <para>
    <screen>
     recordType: text
    </screen>
   </para>

   <para>
    The available configuration settings are summarized below. They will be
    explained further in the following sections.
   </para>

   <!--
   FIXME - Didn't Adam make something to have multiple databases in multiple dirs...
   -->

   <para>
    <variablelist>

     <varlistentry>
      <term>
       <emphasis>group</emphasis>
       .recordType[<emphasis>.name</emphasis>]:
       <replaceable>type</replaceable>
      </term>
      <listitem>
       <para>
        Specifies how records with the file extension
        <emphasis>name</emphasis> should be handled by the indexer.
        This option may also be specified as a command line option
        (<literal>-t</literal>). Note that if you do not specify a
        <emphasis>name</emphasis>, the setting applies to all files.
        In general, the record type specifier consists of the elements (each
        element separated by dot), <emphasis>fundamental-type</emphasis>,
        <emphasis>file-read-type</emphasis> and arguments. Currently, two
        fundamental types exist, <literal>text</literal> and
        <literal>grs</literal>.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term><emphasis>group</emphasis>.recordId:
       <replaceable>record-id-spec</replaceable></term>
      <listitem>
       <para>
        Specifies how the records are to be identified when updated. See
        <xref linkend="locating-records"/>.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term><emphasis>group</emphasis>.database:
       <replaceable>database</replaceable></term>
      <listitem>
       <para>
        Specifies the &acro.z3950; database name.
        <!-- FIXME - now we can have multiple databases in one server. -H -->
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term><emphasis>group</emphasis>.storeKeys:
       <replaceable>boolean</replaceable></term>
      <listitem>
       <para>
        Specifies whether key information should be saved for a given
        group of records. If you plan to update/delete this type of
        records later this should be specified as 1; otherwise it
        should be 0 (default), to save register space.
        <!-- ### this is the first mention of "register" -->
        See <xref linkend="file-ids"/>.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term><emphasis>group</emphasis>.storeData:
       <replaceable>boolean</replaceable></term>
      <listitem>
       <para>
        Specifies whether the records should be stored internally
        in the &zebra; system files.
        If you want to maintain the raw records yourself,
        this option should be false (0).
        If you want &zebra; to take care of the records for you, it
        should be true(1).
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <!-- ### probably a better place to define "register" -->
      <term>register: <replaceable>register-location</replaceable></term>
      <listitem>
       <para>
        Specifies the location of the various register files that &zebra; uses
        to represent your databases.
        See <xref linkend="register-location"/>.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>shadow: <replaceable>register-location</replaceable></term>
      <listitem>
       <para>
        Enables the <emphasis>safe update</emphasis> facility of &zebra;, and
        tells the system where to place the required, temporary files.
        See <xref linkend="shadow-registers"/>.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>lockDir: <replaceable>directory</replaceable></term>
      <listitem>
       <para>
        Directory in which various lock files are stored.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>keyTmpDir: <replaceable>directory</replaceable></term>
      <listitem>
       <para>
        Directory in which temporary files used during zebraidx's update
        phase are stored.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>setTmpDir: <replaceable>directory</replaceable></term>
      <listitem>
       <para>
        Specifies the directory that the server uses for temporary result sets.
        If not specified <literal>/tmp</literal> will be used.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>profilePath: <replaceable>path</replaceable></term>
      <listitem>
       <para>
        Specifies a path of profile specification files.
        The path is composed of one or more directories separated by
        colon. Similar to <literal>PATH</literal> for UNIX systems.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>modulePath: <replaceable>path</replaceable></term>
      <listitem>
       <para>
        Specifies a path of record filter modules.
        The path is composed of one or more directories separated by
        colon. Similar to <literal>PATH</literal> for UNIX systems.
        The 'make install' procedure typically puts modules in
        <filename>/usr/local/lib/idzebra-2.0/modules</filename>.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>index: <replaceable>filename</replaceable></term>
      <listitem>
       <para>
        Defines the filename which holds fields structure
        definitions. If omitted, the file <filename>default.idx</filename>
        is read.
        Refer to <xref linkend="default-idx-file"/> for
        more information.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>sortmax: <replaceable>integer</replaceable></term>
      <listitem>
       <para>
        Specifies the maximum number of records that will be sorted
        in a result set.  If the result set contains more than
        <replaceable>integer</replaceable> records, records after the
        limit will not be sorted.  If omitted, the default value is
        1,000.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>staticrank: <replaceable>integer</replaceable></term>
      <listitem>
       <para>
        Enables whether static ranking is to be enabled (1) or
        disabled (0). If omitted, it is disabled - corresponding
        to a value of 0.
        Refer to <xref linkend="administration-ranking-static"/> .
       </para>
      </listitem>
     </varlistentry>


     <varlistentry>
      <term>estimatehits: <replaceable>integer</replaceable></term>
      <listitem>
       <para>
        Controls whether &zebra; should calculate approximate hit counts and
        at which hit count it is to be enabled.
        A value of 0 disables approximate hit counts.
        For a positive value approximate hit count is enabled
        if it is known to be larger than <replaceable>integer</replaceable>.
       </para>
       <para>
        Approximate hit counts can also be triggered by a particular
        attribute in a query.
        Refer to <xref linkend="querymodel-zebra-global-attr-limit"/>.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>attset: <replaceable>filename</replaceable></term>
      <listitem>
       <para>
        Specifies the filename(s) of attribute set files for use in
        searching. In many configurations <filename>bib1.att</filename>
        is used, but that is not required. If Classic Explain
        attributes is to be used for searching,
        <filename>explain.att</filename> must be given.
        The path to att-files in general can be given using
        <literal>profilePath</literal> setting.
        See also <xref linkend="attset-files"/>.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>memMax: <replaceable>size</replaceable></term>
      <listitem>
       <para>
        Specifies <replaceable>size</replaceable> of internal memory
        to use for the zebraidx program.
        The amount is given in megabytes - default is 4 (4 MB).
        The more memory, the faster large updates happen, up to about
        half the free memory available on the computer.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>tempfiles: <replaceable>Yes/Auto/No</replaceable></term>
      <listitem>
       <para>
        Tells zebra if it should use temporary files when indexing. The
        default is Auto, in which case zebra uses temporary files only
        if it would need more that <replaceable>memMax</replaceable>
        megabytes of memory. This should be good for most uses.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>root: <replaceable>dir</replaceable></term>
      <listitem>
       <para>
        Specifies a directory base for &zebra;. All relative paths
        given (in profilePath, register, shadow) are based on this
        directory. This setting is useful if your &zebra; server
        is running in a different directory from where
        <literal>zebra.cfg</literal> is located.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>passwd: <replaceable>file</replaceable></term>
      <listitem>
       <para>
        Specifies a file with description of user accounts for &zebra;.
        The format is similar to that known to Apache's htpasswd files
        and UNIX' passwd files. Non-empty lines not beginning with
        # are considered account lines. There is one account per-line.
        A line consists of fields separate by a single colon character.
        First field is username, second is password.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>passwd.c: <replaceable>file</replaceable></term>
      <listitem>
       <para>
        Specifies a file with description of user accounts for &zebra;.
        File format is similar to that used by the passwd directive except
        that the password are encrypted. Use Apache's htpasswd or similar
        for maintenance.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>perm.<replaceable>user</replaceable>:
       <replaceable>permstring</replaceable></term>
      <listitem>
       <para>
        Specifies permissions (privilege) for a user that are allowed
        to access &zebra; via the passwd system. There are two kinds
        of permissions currently: read (r) and write(w). By default
        users not listed in a permission directive are given the read
        privilege. To specify permissions for a user with no
        username, or &acro.z3950; anonymous style use
        <literal>anonymous</literal>. The permstring consists of
        a sequence of characters. Include character <literal>w</literal>
        for write/update access, <literal>r</literal> for read access and
        <literal>a</literal> to allow anonymous access through this account.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>dbaccess: <replaceable>accessfile</replaceable></term>
      <listitem>
       <para>
        Names a file which lists database subscriptions for individual users.
        The access file should consists of lines of the form
        <literal>username: dbnames</literal>, where dbnames is a list of
        database names, separated by '+'. No whitespace is allowed in the
        database list.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>encoding: <replaceable>charsetname</replaceable></term>
      <listitem>
       <para>
        Tells &zebra; to interpret the terms in Z39.50 queries as
        having been encoded using the specified character
        encoding.  The default is <literal>ISO-8859-1</literal>; one
        useful alternative is <literal>UTF-8</literal>.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>storeKeys: <replaceable>value</replaceable></term>
      <listitem>
       <para>
        Specifies whether &zebra; keeps a copy of indexed keys.
        Use a value of 1 to enable; 0 to disable. If storeKeys setting is
        omitted, it is enabled. Enabled storeKeys
        are required for updating and deleting records.  Disable only
        storeKeys to save space and only plan to index data once.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term>storeData: <replaceable>value</replaceable></term>
      <listitem>
       <para>
        Specifies whether &zebra; keeps a copy of indexed records.
        Use a value of 1 to enable; 0 to disable. If storeData setting is
        omitted, it is enabled. A storeData setting of 0 (disabled) makes
        Zebra fetch records from the original locaction in the file
        system using filename, file offset and file length. For the
        DOM and ALVIS filter, the storeData setting is ignored.
       </para>
      </listitem>
     </varlistentry>

    </variablelist>
   </para>

  </sect1>

  <sect1 id="locating-records">
   <title>Locating Records</title>

   <para>
    The default behavior of the &zebra; system is to reference the
    records from their original location, i.e. where they were found when you
    run <literal>zebraidx</literal>.
    That is, when a client wishes to retrieve a record
    following a search operation, the files are accessed from the place
    where you originally put them - if you remove the files (without
    running <literal>zebraidx</literal> again, the server will return
    diagnostic number 14 (``System error in presenting records'') to
    the client.
   </para>

   <para>
    If your input files are not permanent - for example if you retrieve
    your records from an outside source, or if they were temporarily
    mounted on a CD-ROM drive,
    you may want &zebra; to make an internal copy of them. To do this,
    you specify 1 (true) in the <literal>storeData</literal> setting. When
    the &acro.z3950; server retrieves the records they will be read from the
    internal file structures of the system.
   </para>

  </sect1>

  <sect1 id="simple-indexing">
   <title>Indexing with no Record IDs (Simple Indexing)</title>

   <para>
    If you have a set of records that are not expected to change over time
    you may can build your database without record IDs.
    This indexing method uses less space than the other methods and
    is simple to use.
   </para>

   <para>
    To use this method, you simply omit the <literal>recordId</literal> entry
    for the group of files that you index. To add a set of records you use
    <literal>zebraidx</literal> with the <literal>update</literal> command. The
    <literal>update</literal> command will always add all of the records that it
    encounters to the index - whether they have already been indexed or
    not. If the set of indexed files change, you should delete all of the
    index files, and build a new index from scratch.
   </para>

   <para>
    Consider a system in which you have a group of text files called
    <literal>simple</literal>.
    That group of records should belong to a &acro.z3950; database called
    <literal>textbase</literal>.
    The following <literal>zebra.cfg</literal> file will suffice:
   </para>
   <para>

    <screen>
     profilePath: /usr/local/idzebra/tab
     attset: bib1.att
     simple.recordType: text
     simple.database: textbase
    </screen>

   </para>

   <para>
    Since the existing records in an index can not be addressed by their
    IDs, it is impossible to delete or modify records when using this method.
   </para>

  </sect1>

  <sect1 id="file-ids">
   <title>Indexing with File Record IDs</title>

   <para>
    If you have a set of files that regularly change over time: Old files
    are deleted, new ones are added, or existing files are modified, you
    can benefit from using the <emphasis>file ID</emphasis>
    indexing methodology.
    Examples of this type of database might include an index of WWW
    resources, or a USENET news spool area.
    Briefly speaking, the file key methodology uses the directory paths
    of the individual records as a unique identifier for each record.
    To perform indexing of a directory with file keys, again, you specify
    the top-level directory after the <literal>update</literal> command.
    The command will recursively traverse the directories and compare
    each one with whatever have been indexed before in that same directory.
    If a file is new (not in the previous version of the directory) it
    is inserted into the registers; if a file was already indexed and
    it has been modified since the last update, the index is also
    modified; if a file has been removed since the last
    visit, it is deleted from the index.
   </para>

   <para>
    The resulting system is easy to administrate. To delete a record you
    simply have to delete the corresponding file (say, with the
    <literal>rm</literal> command). And to add records you create new
    files (or directories with files). For your changes to take effect
    in the register you must run <literal>zebraidx update</literal> with
    the same directory root again. This mode of operation requires more
    disk space than simpler indexing methods, but it makes it easier for
    you to keep the index in sync with a frequently changing set of data.
    If you combine this system with the <emphasis>safe update</emphasis>
    facility (see below), you never have to take your server off-line for
    maintenance or register updating purposes.
   </para>

   <para>
    To enable indexing with pathname IDs, you must specify
    <literal>file</literal> as the value of <literal>recordId</literal>
    in the configuration file. In addition, you should set
    <literal>storeKeys</literal> to <literal>1</literal>, since the &zebra;
    indexer must save additional information about the contents of each record
    in order to modify the indexes correctly at a later time.
   </para>

   <!--
   FIXME - There must be a simpler way to do this with Adams string tags -H
   -->

   <para>
    For example, to update records of group <literal>esdd</literal>
    located below
    <literal>/data1/records/</literal> you should type:
    <screen>
     $ zebraidx -g esdd update /data1/records
    </screen>
   </para>

   <para>
    The corresponding configuration file includes:
    <screen>
     esdd.recordId: file
     esdd.recordType: grs.sgml
     esdd.storeKeys: 1
    </screen>
   </para>

   <note>
    <para>You cannot start out with a group of records with simple
     indexing (no record IDs as in the previous section) and then later
     enable file record Ids. &zebra; must know from the first time that you
     index the group that
     the files should be indexed with file record IDs.
    </para>
   </note>

   <para>
    You cannot explicitly delete records when using this method (using the
    <literal>delete</literal> command to <literal>zebraidx</literal>. Instead
    you have to delete the files from the file system (or move them to a
    different location)
    and then run <literal>zebraidx</literal> with the
    <literal>update</literal> command.
   </para>
   <!-- ### what happens if a file contains multiple records? -->
  </sect1>

  <sect1 id="generic-ids">
   <title>Indexing with General Record IDs</title>

   <para>
    When using this method you construct an (almost) arbitrary, internal
    record key based on the contents of the record itself and other system
    information. If you have a group of records that explicitly associates
    an ID with each record, this method is convenient. For example, the
    record format may contain a title or a ID-number - unique within the group.
    In either case you specify the &acro.z3950; attribute set and use-attribute
    location in which this information is stored, and the system looks at
    that field to determine the identity of the record.
   </para>

   <para>
    As before, the record ID is defined by the <literal>recordId</literal>
    setting in the configuration file. The value of the record ID specification
    consists of one or more tokens separated by whitespace. The resulting
    ID is represented in the index by concatenating the tokens and
    separating them by ASCII value (1).
   </para>

   <para>
    There are three kinds of tokens:
    <variablelist>

     <varlistentry>
      <term>Internal record info</term>
      <listitem>
       <para>
        The token refers to a key that is
        extracted from the record. The syntax of this token is
        <literal>(</literal> <emphasis>set</emphasis> <literal>,</literal>
        <emphasis>use</emphasis> <literal>)</literal>,
        where <emphasis>set</emphasis> is the
        attribute set name <emphasis>use</emphasis> is the
        name or value of the attribute.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>System variable</term>
      <listitem>
       <para>
        The system variables are preceded by

        <screen>
         $
        </screen>
        and immediately followed by the system variable name, which
        may one of
        <variablelist>

         <varlistentry>
          <term>group</term>
          <listitem>
           <para>
            Group name.
           </para>
          </listitem>
         </varlistentry>
         <varlistentry>
          <term>database</term>
          <listitem>
           <para>
            Current database specified.
           </para>
          </listitem>
         </varlistentry>
         <varlistentry>
          <term>type</term>
          <listitem>
           <para>
            Record type.
           </para>
          </listitem>
         </varlistentry>
        </variablelist>
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>Constant string</term>
      <listitem>
       <para>
        A string used as part of the ID &mdash; surrounded
        by single- or double quotes.
       </para>
      </listitem>
     </varlistentry>
    </variablelist>
   </para>

   <para>
    For instance, the sample GILS records that come with the &zebra;
    distribution contain a unique ID in the data tagged Control-Identifier.
    The data is mapped to the &acro.bib1; use attribute Identifier-standard
    (code 1007). To use this field as a record id, specify
    <literal>(bib1,Identifier-standard)</literal> as the value of the
    <literal>recordId</literal> in the configuration file.
    If you have other record types that uses the same field for a
    different purpose, you might add the record type
    (or group or database name) to the record id of the gils
    records as well, to prevent matches with other types of records.
    In this case the recordId might be set like this:

    <screen>
     gils.recordId: $type (bib1,Identifier-standard)
    </screen>

   </para>

   <para>
    (see <xref linkend="grs"/>
    for details of how the mapping between elements of your records and
    searchable attributes is established).
   </para>

   <para>
    As for the file record ID case described in the previous section,
    updating your system is simply a matter of running
    <literal>zebraidx</literal>
    with the <literal>update</literal> command. However, the update with general
    keys is considerably slower than with file record IDs, since all files
    visited must be (re)read to discover their IDs.
   </para>

   <para>
    As you might expect, when using the general record IDs
    method, you can only add or modify existing records with the
    <literal>update</literal> command.
    If you wish to delete records, you must use the,
    <literal>delete</literal> command, with a directory as a parameter.
    This will remove all records that match the files below that root
    directory.
   </para>

  </sect1>

  <sect1 id="register-location">
   <title>Register Location</title>

   <para>
    Normally, the index files that form dictionaries, inverted
    files, record info, etc., are stored in the directory where you run
    <literal>zebraidx</literal>. If you wish to store these, possibly large,
    files somewhere else, you must add the <literal>register</literal>
    entry to the <literal>zebra.cfg</literal> file.
    Furthermore, the &zebra; system allows its file
    structures to span multiple file systems, which is useful for
    managing very large databases.
   </para>

   <para>
    The value of the <literal>register</literal> setting is a sequence
    of tokens. Each token takes the form:

    <emphasis>dir</emphasis><literal>:</literal><emphasis>size</emphasis>

    The <emphasis>dir</emphasis> specifies a directory in which index files
    will be stored and the <emphasis>size</emphasis> specifies the maximum
    size of all files in that directory. The &zebra; indexer system fills
    each directory in the order specified and use the next specified
    directories as needed.
    The <emphasis>size</emphasis> is an integer followed by a qualifier
    code,
    <literal>b</literal> for bytes,
    <literal>k</literal> for kilobytes.
    <literal>M</literal> for megabytes,
    <literal>G</literal> for gigabytes.
    Specifying a negative value disables the checking (it still needs the unit,
    use <literal>-1b</literal>).
   </para>

   <para>
    For instance, if you have allocated three disks for your register, and
    the first disk is mounted
    on <literal>/d1</literal> and has 2GB of free space, the
    second, mounted on <literal>/d2</literal> has 3.6 GB, and the third,
    on which you have more space than you bother to worry about, mounted on
    <literal>/d3</literal> you could put this entry in your configuration file:

    <screen>
     register: /d1:2G /d2:3600M /d3:-1b
    </screen>
   </para>

   <para>
    Note that &zebra; does not verify that the amount of space specified is
    actually available on the directory (file system) specified - it is
    your responsibility to ensure that enough space is available, and that
    other applications do not attempt to use the free space. In a large
    production system, it is recommended that you allocate one or more
    file system exclusively to the &zebra; register files.
   </para>

  </sect1>

  <sect1 id="shadow-registers">
   <title>Safe Updating - Using Shadow Registers</title>

   <sect2 id="shadow-registers-description">
    <title>Description</title>

    <para>
     The &zebra; server supports <emphasis>updating</emphasis> of the index
     structures. That is, you can add, modify, or remove records from
     databases managed by &zebra; without rebuilding the entire index.
     Since this process involves modifying structured files with various
     references between blocks of data in the files, the update process
     is inherently sensitive to system crashes, or to process interruptions:
     Anything but a successfully completed update process will leave the
     register files in an unknown state, and you will essentially have no
     recourse but to re-index everything, or to restore the register files
     from a backup medium.
     Further, while the update process is active, users cannot be
     allowed to access the system, as the contents of the register files
     may change unpredictably.
    </para>

    <para>
     You can solve these problems by enabling the shadow register system in
     &zebra;.
     During the updating procedure, <literal>zebraidx</literal> will temporarily
     write changes to the involved files in a set of "shadow
     files", without modifying the files that are accessed by the
     active server processes. If the update procedure is interrupted by a
     system crash or a signal, you simply repeat the procedure - the
     register files have not been changed or damaged, and the partially
     written shadow files are automatically deleted before the new updating
     procedure commences.
    </para>

    <para>
     At the end of the updating procedure (or in a separate operation, if
     you so desire), the system enters a "commit mode". First,
     any active server processes are forced to access those blocks that
     have been changed from the shadow files rather than from the main
     register files; the unmodified blocks are still accessed at their
     normal location (the shadow files are not a complete copy of the
     register files - they only contain those parts that have actually been
     modified). If the commit process is interrupted at any point during the
     commit process, the server processes will continue to access the
     shadow files until you can repeat the commit procedure and complete
     the writing of data to the main register files. You can perform
     multiple update operations to the registers before you commit the
     changes to the system files, or you can execute the commit operation
     at the end of each update operation. When the commit phase has
     completed successfully, any running server processes are instructed to
     switch their operations to the new, operational register, and the
     temporary shadow files are deleted.
    </para>

   </sect2>

   <sect2 id="shadow-registers-how-to-use">
    <title>How to Use Shadow Register Files</title>

    <para>
     The first step is to allocate space on your system for the shadow
     files.
     You do this by adding a <literal>shadow</literal> entry to the
     <literal>zebra.cfg</literal> file.
     The syntax of the <literal>shadow</literal> entry is exactly the
     same as for the <literal>register</literal> entry
     (see <xref linkend="register-location"/>).
     The location of the shadow area should be
     <emphasis>different</emphasis> from the location of the main register
     area (if you have specified one - remember that if you provide no
     <literal>register</literal> setting, the default register area is the
     working directory of the server and indexing processes).
    </para>

    <para>
     The following excerpt from a <literal>zebra.cfg</literal> file shows
     one example of a setup that configures both the main register
     location and the shadow file area.
     Note that two directories or partitions have been set aside
     for the shadow file area. You can specify any number of directories
     for each of the file areas, but remember that there should be no
     overlaps between the directories used for the main registers and the
     shadow files, respectively.
    </para>
    <para>

     <screen>
      register: /d1:500M
      shadow: /scratch1:100M /scratch2:200M
     </screen>

    </para>

    <para>
     When shadow files are enabled, an extra command is available at the
     <literal>zebraidx</literal> command line.
     In order to make changes to the system take effect for the
     users, you'll have to submit a "commit" command after a
     (sequence of) update operation(s).
    </para>

    <para>

     <screen>
      $ zebraidx update /d1/records
      $ zebraidx commit
     </screen>

    </para>

    <para>
     Or you can execute multiple updates before committing the changes:
    </para>

    <para>

     <screen>
      $ zebraidx -g books update /d1/records  /d2/more-records
      $ zebraidx -g fun update /d3/fun-records
      $ zebraidx commit
     </screen>

    </para>

    <para>
     If one of the update operations above had been interrupted, the commit
     operation on the last line would fail: <literal>zebraidx</literal>
     will not let you commit changes that would destroy the running register.
     You'll have to rerun all of the update operations since your last
     commit operation, before you can commit the new changes.
    </para>

    <para>
     Similarly, if the commit operation fails, <literal>zebraidx</literal>
     will not let you start a new update operation before you have
     successfully repeated the commit operation.
     The server processes will keep accessing the shadow files rather
     than the (possibly damaged) blocks of the main register files
     until the commit operation has successfully completed.
    </para>

    <para>
     You should be aware that update operations may take slightly longer
     when the shadow register system is enabled, since more file access
     operations are involved. Further, while the disk space required for
     the shadow register data is modest for a small update operation, you
     may prefer to disable the system if you are adding a very large number
     of records to an already very large database (we use the terms
     <emphasis>large</emphasis> and <emphasis>modest</emphasis>
     very loosely here, since every application will have a
     different perception of size).
     To update the system without the use of the the shadow files,
     simply run <literal>zebraidx</literal> with the <literal>-n</literal>
     option (note that you do not have to execute the
     <emphasis>commit</emphasis> command of <literal>zebraidx</literal>
     when you temporarily disable the use of the shadow registers in
     this fashion.
     Note also that, just as when the shadow registers are not enabled,
     server processes will be barred from accessing the main register
     while the update procedure takes place.
    </para>

   </sect2>

  </sect1>


  <sect1 id="administration-ranking">
   <title>Relevance Ranking and Sorting of Result Sets</title>

   <sect2 id="administration-overview">
    <title>Overview</title>
    <para>
     The default ordering of a result set is left up to the server,
     which inside &zebra; means sorting in ascending document ID order.
     This is not always the order humans want to browse the sometimes
     quite large hit sets. Ranking and sorting comes to the rescue.
    </para>

    <para>
     In cases where a good presentation ordering can be computed at
     indexing time, we can use a fixed <literal>static ranking</literal>
     scheme, which is provided for the <literal>alvis</literal>
     indexing filter. This defines a fixed ordering of hit lists,
     independently of the query issued.
    </para>

    <para>
     There are cases, however, where relevance of hit set documents is
     highly dependent on the query processed.
     Simply put, <literal>dynamic relevance ranking</literal>
     sorts a set of retrieved records such that those most likely to be
     relevant to your request are retrieved first.
     Internally, &zebra; retrieves all documents that satisfy your
     query, and re-orders the hit list to arrange them based on
     a measurement of similarity between your query and the content of
     each record.
    </para>

    <para>
     Finally, there are situations where hit sets of documents should be
     <literal>sorted</literal> during query time according to the
     lexicographical ordering of certain sort indexes created at
     indexing time.
    </para>
   </sect2>


   <sect2 id="administration-ranking-static">
    <title>Static Ranking</title>

    <para>
     &zebra; uses internally inverted indexes to look up term frequencies
     in documents. Multiple queries from different indexes can be
     combined by the binary boolean operations <literal>AND</literal>,
     <literal>OR</literal> and/or <literal>NOT</literal> (which
     is in fact a binary <literal>AND NOT</literal> operation).
     To ensure fast query execution
     speed, all indexes have to be sorted in the same order.
    </para>
    <para>
     The indexes are normally sorted according to document
     <literal>ID</literal> in
     ascending order, and any query which does not invoke a special
     re-ranking function will therefore retrieve the result set in
     document
     <literal>ID</literal>
     order.
    </para>
    <para>
     If one defines the
     <screen>
      staticrank: 1
     </screen>
     directive in the main core &zebra; configuration file, the internal document
     keys used for ordering are augmented by a preceding integer, which
     contains the static rank of a given document, and the index lists
     are ordered
     first by ascending static rank,
     then by ascending document <literal>ID</literal>.
     Zero
     is the ``best'' rank, as it occurs at the
     beginning of the list; higher numbers represent worse scores.
    </para>
    <para>
     The experimental <literal>alvis</literal> filter provides a
     directive to fetch static rank information out of the indexed &acro.xml;
     records, thus making <emphasis>all</emphasis> hit sets ordered
     after <emphasis>ascending</emphasis> static
     rank, and for those doc's which have the same static rank, ordered
     after <emphasis>ascending</emphasis> doc <literal>ID</literal>.
     See <xref linkend="record-model-alvisxslt"/> for the gory details.
    </para>
   </sect2>


   <sect2 id="administration-ranking-dynamic">
    <title>Dynamic Ranking</title>
    <para>
     In order to fiddle with the static rank order, it is necessary to
     invoke additional re-ranking/re-ordering using dynamic
     ranking or score functions. These functions return positive
     integer scores, where <emphasis>highest</emphasis> score is
     ``best'';
     hit sets are sorted according to <emphasis>descending</emphasis>
     scores (in contrary
     to the index lists which are sorted according to
     ascending rank number and document ID).
    </para>
    <para>
     Dynamic ranking is enabled by a directive like one of the
     following in the zebra configuration file (use only one of these a time!):
     <screen>
      rank: rank-1        # default TDF-IDF like
      rank: rank-static   # dummy do-nothing
     </screen>
    </para>

    <para>
     Dynamic ranking is done at query time rather than
     indexing time (this is why we
     call it ``dynamic ranking'' in the first place ...)
     It is invoked by adding
     the &acro.bib1; relation attribute with
     value ``relevance'' to the &acro.pqf; query (that is,
     <literal>@attr&nbsp;2=102</literal>, see also
     <ulink url="&url.z39.50;bib1.html">
      The &acro.bib1; Attribute Set Semantics</ulink>, also in
     <ulink url="&url.z39.50.attset.bib1;">HTML</ulink>).
     To find all articles with the word <literal>Eoraptor</literal> in
     the title, and present them relevance ranked, issue the &acro.pqf; query:
     <screen>
      @attr 2=102 @attr 1=4 Eoraptor
     </screen>
    </para>

    <sect3 id="administration-ranking-dynamic-rank1">
     <title>Dynamically ranking using &acro.pqf; queries with the 'rank-1'
      algorithm</title>

     <para>
      The default <literal>rank-1</literal> ranking module implements a
      TF/IDF (Term Frequecy over Inverse Document Frequency) like
      algorithm. In contrast to the usual definition of TF/IDF
      algorithms, which only considers searching in one full-text
      index, this one works on multiple indexes at the same time.
      More precisely,
      &zebra; does boolean queries and searches in specific addressed
      indexes (there are inverted indexes pointing from terms in the
      dictionary to documents and term positions inside documents).
      It works like this:
      <variablelist>
       <varlistentry>
        <term>Query Components</term>
        <listitem>
         <para>
          First, the boolean query is dismantled into its principal components,
          i.e. atomic queries where one term is looked up in one index.
          For example, the query
          <screen>
           @attr 2=102 @and @attr 1=1010 Utah @attr 1=1018 Springer
          </screen>
          is a boolean AND between the atomic parts
          <screen>
           @attr 2=102 @attr 1=1010 Utah
          </screen>
          and
          <screen>
           @attr 2=102 @attr 1=1018 Springer
          </screen>
          which gets processed each for itself.
         </para>
        </listitem>
       </varlistentry>

       <varlistentry>
        <term>Atomic hit lists</term>
        <listitem>
         <para>
          Second, for each atomic query, the hit list of documents is
          computed.
         </para>
         <para>
          In this example, two hit lists for each index
          <literal>@attr 1=1010</literal>  and
          <literal>@attr 1=1018</literal> are computed.
         </para>
        </listitem>
       </varlistentry>

       <varlistentry>
        <term>Atomic scores</term>
        <listitem>
         <para>
          Third, each document in the hit list is assigned a score (_if_ ranking
          is enabled and requested in the query)  using a TF/IDF scheme.
         </para>
         <para>
          In this example, both atomic parts of the query assign the magic
          <literal>@attr 2=102</literal> relevance attribute, and are
          to be used in the relevance ranking functions.
         </para>
         <para>
          It is possible to apply dynamic ranking on only parts of the
          &acro.pqf; query:
          <screen>
           @and @attr 2=102 @attr 1=1010 Utah @attr 1=1018 Springer
          </screen>
          searches for all documents which have the term 'Utah' on the
          body of text, and which have the term 'Springer' in the publisher
          field, and sort them in the order of the relevance ranking made on
          the body-of-text index only.
         </para>
        </listitem>
       </varlistentry>

       <varlistentry>
        <term>Hit list merging</term>
        <listitem>
         <para>
          Fourth, the atomic hit lists are merged according to the boolean
          conditions to a final hit list of documents to be returned.
         </para>
         <para>
          This step is always performed, independently of the fact that
          dynamic ranking is enabled or not.
         </para>
        </listitem>
       </varlistentry>

       <varlistentry>
        <term>Document score computation</term>
        <listitem>
         <para>
          Fifth, the total score of a document is computed as a linear
          combination of the atomic scores of the atomic hit lists
         </para>
         <para>
          Ranking weights may be used to pass a value to a ranking
          algorithm, using the non-standard &acro.bib1; attribute type 9.
          This allows one branch of a query to use one value while
          another branch uses a different one.  For example, we can search
          for <literal>utah</literal> in the
          <literal>@attr 1=4</literal> index with weight 30, as
          well as in the <literal>@attr 1=1010</literal> index with weight 20:
          <screen>
           @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 @attr 1=1010 city
          </screen>
         </para>
         <para>
          The default weight is
          sqrt(1000) ~ 34 , as the &acro.z3950; standard prescribes that the top score
          is 1000 and the bottom score is 0, encoded in integers.
         </para>
         <warning>
          <para>
           The ranking-weight feature is experimental. It may change in future
           releases of zebra.
          </para>
         </warning>
        </listitem>
       </varlistentry>

       <varlistentry>
        <term>Re-sorting of hit list</term>
        <listitem>
         <para>
          Finally, the final hit list is re-ordered according to scores.
         </para>
        </listitem>
       </varlistentry>
      </variablelist>

     </para>


     <para>
      The <literal>rank-1</literal> algorithm
      does not use the static rank
      information in the list keys, and will produce the same ordering
      with or without static ranking enabled.
     </para>


     <!--
     <sect3 id="administration-ranking-dynamic-rank1">
     <title>Dynamically ranking &acro.pqf; queries with the 'rank-static'
     algorithm</title>
     <para>
     The dummy <literal>rank-static</literal> reranking/scoring
     function returns just
     <literal>score = max int - staticrank</literal>
     in order to preserve the static ordering of hit sets that would
     have been produced had it not been invoked.
     Obviously, to combine static and dynamic ranking usefully,
     it is necessary
     to make a new ranking
     function; this is left
     as an exercise for the reader.
    </para>
    </sect3>
     -->

     <warning>
      <para>
       <literal>Dynamic ranking</literal> is not compatible
       with <literal>estimated hit sizes</literal>, as all documents in
       a hit set must be accessed to compute the correct placing in a
       ranking sorted list. Therefore the use attribute setting
       <literal>@attr&nbsp;2=102</literal> clashes with
       <literal>@attr&nbsp;9=integer</literal>.
      </para>
     </warning>

     <!--
     we might want to add ranking like this:
     UNPUBLISHED:
     Simple BM25 Extension to Multiple Weighted Fields
     Stephen Robertson, Hugo Zaragoza and Michael Taylor
     Microsoft Research
     ser@microsoft.com
     hugoz@microsoft.com
     mitaylor2microsoft.com
     -->

    </sect3>

    <sect3 id="administration-ranking-dynamic-cql">
     <title>Dynamically ranking &acro.cql; queries</title>
     <para>
      Dynamic ranking can be enabled during sever side &acro.cql;
      query expansion by adding <literal>@attr&nbsp;2=102</literal>
      chunks to the &acro.cql; config file. For example
      <screen>
       relationModifier.relevant                = 2=102
      </screen>
      invokes dynamic ranking each time a &acro.cql; query of the form
      <screen>
       Z> querytype cql
       Z> f alvis.text =/relevant house
      </screen>
      is issued. Dynamic ranking can also be automatically used on
      specific &acro.cql; indexes by (for example) setting
      <screen>
       index.alvis.text                        = 1=text 2=102
      </screen>
      which then invokes dynamic ranking each time a &acro.cql; query of the form
      <screen>
       Z> querytype cql
       Z> f alvis.text = house
      </screen>
      is issued.
     </para>

    </sect3>

   </sect2>


   <sect2 id="administration-ranking-sorting">
    <title>Sorting</title>
    <para>
     &zebra; sorts efficiently using special sorting indexes
     (type=<literal>s</literal>; so each sortable index must be known
     at indexing time, specified in the configuration of record
     indexing.  For example, to enable sorting according to the &acro.bib1;
     <literal>Date/time-added-to-db</literal> field, one could add the line
     <screen>
      xelm /*/@created               Date/time-added-to-db:s
     </screen>
     to any <literal>.abs</literal> record-indexing configuration file.
     Similarly, one could add an indexing element of the form
     <screen><![CDATA[
      <z:index name="date-modified" type="s">
      <xsl:value-of select="some/xpath"/>
     </z:index>
      ]]></screen>
     to any <literal>alvis</literal>-filter indexing stylesheet.
    </para>
    <para>
     Indexing can be specified at searching time using a query term
     carrying the non-standard
     &acro.bib1; attribute-type <literal>7</literal>.  This removes the
     need to send a &acro.z3950; <literal>Sort Request</literal>
     separately, and can dramatically improve latency when the client
     and server are on separate networks.
     The sorting part of the query is separate from the rest of the
     query - the actual search specification - and must be combined
     with it using OR.
    </para>
    <para>
     A sorting subquery needs two attributes: an index (such as a
     &acro.bib1; type-1 attribute) specifying which index to sort on, and a
     type-7 attribute whose value is be <literal>1</literal> for
     ascending sorting, or <literal>2</literal> for descending.  The
     term associated with the sorting attribute is the priority of
     the sort key, where <literal>0</literal> specifies the primary
     sort key, <literal>1</literal> the secondary sort key, and so
     on.
    </para>
    <para>For example, a search for water, sort by title (ascending),
     is expressed by the &acro.pqf; query
     <screen>
      @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
     </screen>
     whereas a search for water, sort by title ascending,
     then date descending would be
     <screen>
      @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
     </screen>
    </para>
    <para>
     Notice the fundamental differences between <literal>dynamic
      ranking</literal> and <literal>sorting</literal>: there can be
     only one ranking function defined and configured; but multiple
     sorting indexes can be specified dynamically at search
     time. Ranking does not need to use specific indexes, so
     dynamic ranking can be enabled and disabled without
     re-indexing; whereas, sorting indexes need to be
     defined before indexing.
    </para>

   </sect2>


  </sect1>

  <sect1 id="administration-extended-services">
   <title>Extended Services: Remote Insert, Update and Delete</title>

   <note>
    <para>
     Extended services are only supported when accessing the &zebra;
     server using the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
     protocol. The <ulink url="&url.sru;">&acro.sru;</ulink> protocol does
     not support extended services.
    </para>
   </note>

   <para>
    The extended services are not enabled by default in zebra - due to the
    fact that they modify the system. &zebra; can be configured
    to allow anybody to
    search, and to allow only updates for a particular admin user
    in the main zebra configuration file <filename>zebra.cfg</filename>.
    For user <literal>admin</literal>, you could use:
    <screen>
     perm.anonymous: r
     perm.admin: rw
     passwd: passwordfile
    </screen>
    And in the password file
    <filename>passwordfile</filename>, you have to specify users and
    encrypted passwords as colon separated strings.
    Use a tool like <filename>htpasswd</filename>
    to maintain the encrypted passwords.
    <screen>
     admin:secret
    </screen>
    It is essential to configure  &zebra; to store records internally,
    and to support
    modifications and deletion of records:
    <screen>
     storeData: 1
     storeKeys: 1
    </screen>
    The general record type should be set to any record filter which
    is able to parse &acro.xml; records, you may use any of the two
    declarations (but not both simultaneously!)
    <screen>
     recordType: dom.filter_dom_conf.xml
     # recordType: grs.xml
    </screen>
    Notice the difference to the specific instructions
    <screen>
     recordType.xml: dom.filter_dom_conf.xml
     # recordType.xml: grs.xml
    </screen>
    which only work when indexing XML files from the filesystem using
    the <literal>*.xml</literal> naming convention.
   </para>
   <para>
    To enable transaction safe shadow indexing,
    which is extra important for this kind of operation, set
    <screen>
     shadow: directoryname: size (e.g. 1000M)
    </screen>
    See <xref linkend="zebra-cfg"/> for additional information on
    these configuration options.
   </para>
   <note>
    <para>
     It is not possible to carry information about record types or
     similar to &zebra; when using extended services, due to
     limitations of the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
     protocol. Therefore, indexing filters can not be chosen on a
     per-record basis. One and only one general &acro.xml; indexing filter
     must be defined.
     <!-- but because it is represented as an OID, we would need some
     form of proprietary mapping scheme between record type strings and
     OIDs. -->
     <!--
     However, as a minimum, it would be extremely useful to enable
     people to use &acro.marc21;, assuming grs.marcxml.marc21 as a record
     type.
     -->
    </para>
   </note>


   <sect2 id="administration-extended-services-z3950">
    <title>Extended services in the &acro.z3950; protocol</title>

    <para>
     The <ulink url="&url.z39.50;">&acro.z3950;</ulink> standard allows
     servers to accept special binary <emphasis>extended services</emphasis>
     protocol packages, which may be used to insert, update and delete
     records into servers. These carry  control and update
     information to the servers, which are encoded in seven package fields:
    </para>

    <table id="administration-extended-services-z3950-table" frame="top">
     <title>Extended services &acro.z3950; Package Fields</title>
     <tgroup cols="3">
      <thead>
       <row>
        <entry>Parameter</entry>
        <entry>Value</entry>
        <entry>Notes</entry>
       </row>
      </thead>
      <tbody>
       <row>
        <entry><literal>type</literal></entry>
        <entry><literal>'update'</literal></entry>
        <entry>Must be set to trigger extended services</entry>
       </row>
       <row>
        <entry><literal>action</literal></entry>
        <entry><literal>string</literal></entry>
        <entry>
         Extended service action type with
         one of four possible values: <literal>recordInsert</literal>,
         <literal>recordReplace</literal>,
         <literal>recordDelete</literal>,
         and <literal>specialUpdate</literal>
        </entry>
       </row>
       <row>
        <entry><literal>record</literal></entry>
        <entry><literal>&acro.xml; string</literal></entry>
        <entry>An &acro.xml; formatted string containing the record</entry>
       </row>
       <row>
        <entry><literal>syntax</literal></entry>
        <entry><literal>'xml'</literal></entry>
        <entry>XML/SUTRS/MARC. GRS-1 not supported.
         The default filter (record type) as given by recordType in
         zebra.cfg is used to parse the record.</entry>
       </row>
       <row>
        <entry><literal>recordIdOpaque</literal></entry>
        <entry><literal>string</literal></entry>
        <entry>
         Optional client-supplied, opaque record
         identifier used under insert operations.
        </entry>
       </row>
       <row>
        <entry><literal>recordIdNumber </literal></entry>
        <entry><literal>positive number</literal></entry>
        <entry>&zebra;'s internal system number,
         not allowed for  <literal>recordInsert</literal> or
         <literal>specialUpdate</literal> actions which result in fresh
         record inserts.
        </entry>
       </row>
       <row>
        <entry><literal>databaseName</literal></entry>
        <entry><literal>database identifier</literal></entry>
        <entry>
         The name of the database to which the extended services should be
         applied.
        </entry>
       </row>
      </tbody>
     </tgroup>
    </table>


    <para>
     The <literal>action</literal> parameter can be any of
     <literal>recordInsert</literal> (will fail if the record already exists),
     <literal>recordReplace</literal> (will fail if the record does not exist),
     <literal>recordDelete</literal> (will fail if the record does not
     exist), and
     <literal>specialUpdate</literal> (will insert or update the record
     as needed, record deletion is not possible).
    </para>

    <para>
     During all actions, the
     usual rules for internal record ID generation apply, unless an
     optional <literal>recordIdNumber</literal> &zebra; internal ID or a
     <literal>recordIdOpaque</literal> string identifier is assigned.
     The default ID generation is
     configured using the <literal>recordId:</literal> from
     <filename>zebra.cfg</filename>.
     See <xref linkend="zebra-cfg"/>.
    </para>

    <para>
     Setting of the <literal>recordIdNumber</literal> parameter,
     which must be an existing &zebra; internal system ID number, is not
     allowed during any  <literal>recordInsert</literal> or
     <literal>specialUpdate</literal> action resulting in fresh record
     inserts.
    </para>

    <para>
     When retrieving existing
     records indexed with &acro.grs1; indexing filters, the &zebra; internal
     ID number is returned in the field
     <literal>/*/id:idzebra/localnumber</literal> in the namespace
     <literal>xmlns:id="http://www.indexdata.dk/zebra/"</literal>,
     where it can be picked up for later record updates or deletes.
    </para>

    <para>
     A new element set for retrieval of internal record
     data has been added, which can be used to access minimal records
     containing only the <literal>recordIdNumber</literal> &zebra;
     internal ID, or the <literal>recordIdOpaque</literal> string
     identifier. This works for any indexing filter used.
     See <xref linkend="special-retrieval"/>.
    </para>

    <para>
     The <literal>recordIdOpaque</literal> string parameter
     is an client-supplied, opaque record
     identifier, which may be  used under
     insert, update and delete operations. The
     client software is responsible for assigning these to
     records.      This identifier will
     replace zebra's own automagic identifier generation with a unique
     mapping from <literal>recordIdOpaque</literal> to the
     &zebra; internal <literal>recordIdNumber</literal>.
     <emphasis>The opaque <literal>recordIdOpaque</literal> string
      identifiers
      are not visible in retrieval records, nor are
      searchable, so the value of this parameter is
      questionable. It serves mostly as a convenient mapping from
      application domain string identifiers to &zebra; internal ID's.
     </emphasis>
    </para>
   </sect2>


   <sect2 id="administration-extended-services-yaz-client">
    <title>Extended services from yaz-client</title>

    <para>
     We can now start a yaz-client admin session and create a database:
     <screen>
      <![CDATA[
      $ yaz-client localhost:9999 -u admin/secret
      Z> adm-create
      ]]>
     </screen>
     Now the <literal>Default</literal> database was created,
     we can insert an &acro.xml; file (esdd0006.grs
     from example/gils/records) and index it:
     <screen>
      <![CDATA[
      Z> update insert id1234 esdd0006.grs
      ]]>
     </screen>
     The 3rd parameter - <literal>id1234</literal> here -
     is the  <literal>recordIdOpaque</literal> package field.
    </para>
    <para>
     Actually, we should have a way to specify "no opaque record id" for
     yaz-client's update command.. We'll fix that.
    </para>
    <para>
     The newly inserted record can be searched as usual:
     <screen>
      <![CDATA[
      Z> f utah
      Sent searchRequest.
      Received SearchResponse.
      Search was a success.
      Number of hits: 1, setno 1
      SearchResult-1: term=utah cnt=1
      records returned: 0
      Elapsed: 0.014179
      ]]>
     </screen>
    </para>
    <para>
     Let's delete the beast, using the same
     <literal>recordIdOpaque</literal> string parameter:
     <screen>
      <![CDATA[
      Z> update delete id1234
      No last record (update ignored)
      Z> update delete 1 esdd0006.grs
      Got extended services response
      Status: done
      Elapsed: 0.072441
      Z> f utah
      Sent searchRequest.
      Received SearchResponse.
      Search was a success.
      Number of hits: 0, setno 2
      SearchResult-1: term=utah cnt=0
      records returned: 0
      Elapsed: 0.013610
      ]]>
     </screen>
    </para>
    <para>
     If shadow register is enabled in your
     <filename>zebra.cfg</filename>,
     you must run the adm-commit command
     <screen>
      <![CDATA[
      Z> adm-commit
      ]]>
     </screen>
     after each update session in order write your changes from the
     shadow to the life register space.
    </para>
   </sect2>


   <sect2 id="administration-extended-services-yaz-php">
    <title>Extended services from yaz-php</title>

    <para>
     Extended services are also available from the &yaz; &acro.php; client layer. An
     example of an &yaz;-&acro.php; extended service transaction is given here:
     <screen>
      <![CDATA[
      $record = '<record><title>A fine specimen of a record</title></record>';

      $options = array('action' => 'recordInsert',
      'syntax' => 'xml',
      'record' => $record,
      'databaseName' => 'mydatabase'
      );

      yaz_es($yaz, 'update', $options);
      yaz_es($yaz, 'commit', array());
      yaz_wait();

      if ($error = yaz_error($yaz))
      echo "$error";
      ]]>
     </screen>
    </para>
   </sect2>

   <sect2 id="administration-extended-services-debugging">
    <title>Extended services debugging guide</title>
    <para>
     When debugging ES over PHP we recommend the following order of tests:
    </para>

    <itemizedlist>
     <listitem>
      <para>
       Make sure you have a nice record on your filesystem, which you can
       index from the filesystem by use of the zebraidx command.
       Do it exactly as you planned, using one of the GRS-1 filters,
       or the DOMXML filter.
       When this works, proceed.
      </para>
     </listitem>
     <listitem>
      <para>
       Check that your server setup is OK before you even coded one single
       line PHP using ES.
       Take the same record form the file system, and send as ES via
       <literal>yaz-client</literal> like described in
       <xref linkend="administration-extended-services-yaz-client"/>,
       and
       remember the <literal>-a</literal> option which tells you what
       goes over the wire! Notice also the section on permissions:
       try
       <screen>
        perm.anonymous: rw
       </screen>
       in <literal>zebra.cfg</literal> to make sure you do not run into
       permission  problems (but never expose such an insecure setup on the
       internet!!!). Then, make sure to set the general
       <literal>recordType</literal> instruction, pointing correctly
       to the GRS-1 filters,
       or the DOMXML filters.
      </para>
     </listitem>
     <listitem>
      <para>
       If you insist on using the <literal>sysno</literal> in the
       <literal>recordIdNumber</literal> setting,
       please make sure you do only updates and deletes. Zebra's internal
       system number is not allowed for
       <literal>recordInsert</literal> or
       <literal>specialUpdate</literal> actions
       which result in fresh record inserts.
      </para>
     </listitem>
     <listitem>
      <para>
       If <literal>shadow register</literal> is enabled in your
       <literal>zebra.cfg</literal>, you must remember running the
       <screen>
        Z> adm-commit
       </screen>
       command as well.
      </para>
     </listitem>
     <listitem>
      <para>
       If this works, then proceed to do the same thing in your PHP script.
      </para>
     </listitem>
    </itemizedlist>


   </sect2>

  </sect1>

 </chapter>

 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:t
 sgml-parent-document: "idzebra.xml"
 sgml-local-catalogs: nil
 sgml-namecase-general:t
 End:
 -->