[idzebra-moved-to-github.git] / doc / server.xml

<chapter id="server">
 <!-- $Id: server.xml,v 1.9 2006-01-19 09:27:00 marc Exp $ -->
 <title>The Z39.50 Server</title>
 
 <sect1 id="zebrasrv">
  <title>Running the Z39.50 Server (zebrasrv)</title>

  <!--
   FIXME - We need to be consistent here, zebraidx had the options at the
           end, and lots of explaining text before them. Same for zebrasvr! -H
   FIXME - At least we need a small intro, what is zebrasvr, and how it
           can be run (inetd, nt service, stand-alone program, daemon...) -H
  -->

  <!-- re-write by MC, using the newly created input files for the 
   zebrasrv manpage -->


 <sect2><title>DESCRIPTION</title>
    <para>Zebra is a high-performance, general-purpose structured text indexing
   and retrieval engine. It reads structured records in a variety of input
   formats (eg. email, XML, MARC) and allows access to them through exact
   boolean search expressions and relevance-ranked free-text queries. 
   </para>
   <para>
    <command>zebrasrv</command> is the Z39.50 and  <ulink url="http://www.loc.gov/standards/sru/srw/">SRW</ulink>/U frontend
    server for the <command>Zebra</command> indexer.
   </para> 
   <para> 
    On Unix you can run the <command>zebrasrv</command>
    server from the command line - and put it
    in the background. It may also operate under the inet daemon.
    On WIN32 you can run the server as a console application or
    as a WIN32 Service.
   </para>
  </sect2>

 <sect2>
   <title>SYNOPSIS</title>
    &zebrasrv-synopsis;
 </sect2>

 <sect2>
   <title>OPTIONS</title>
    
   <para>
    The options for <command>zebrasrv</command> are the same
    as those for YAZ' <command>yaz-ztest</command>.
    Option <literal>-c</literal> specifies a Zebra configuration
    file - if omitted <filename>zebra.cfg</filename> is read.
   </para>
   
  &zebrasrv-options;
  </sect2>
  <sect2><title>VIRTUAL HOSTS</title>
    <para>
     <command>zebrasrv</command> uses the YAZ server frontend and does
     support multiple virtual servers behind multiple listening sockets.
    </para>
    &zebrasrv-virtual;
  </sect2>
  <sect2><title>FILES</title>
   <para>
    <filename>zebra.cfg</filename>
   </para>
  </sect2>
  <sect2><title>SEE ALSO</title>
   <para>
    <citerefentry>
     <refentrytitle>zebraidx</refentrytitle>
     <manvolnum>1</manvolnum>
    </citerefentry>,
    <citerefentry>
     <refentrytitle>yaz-ztest</refentrytitle>
     <manvolnum>8</manvolnum>
    </citerefentry>
   </para>
   <para>
    Section "The Z39.50 Server" in the Zebra manual.
    <filename>http://www.indexdata.dk/zebra/doc/server.tkl</filename>
   </para>
   <para>
    Section "Virtual Hosts" in the YAZ manual.
    <filename>http://www.indexdata.dk/yaz/doc/server.vhosts.tkl</filename>
   </para>
   <para>
    Section "Specification of <ulink url="http://www.loc.gov/standards/sru/cql/">CQL</ulink> to RPN mappings" in the YAZ manual.
    <filename>http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map</filename>
   </para>
   <para>
    The Zebra software is Copyright <command>Index Data</command> 
    <filename>http://www.indexdata.dk</filename>
    and distributed under the
    GPLv2 license.
   </para> 
  </sect2>

  <!--
  <para>
   <emphasis remap="bf">Syntax</emphasis>

   <screen>
    zebrasrv [options] [listener-address ...]
   </screen>

  </para>

  <para>
   <emphasis remap="bf">Options</emphasis>
   <variablelist>

    <varlistentry>
     <term>-a <replaceable>APDU file</replaceable></term>
     <listitem>
      <para>
       Specify a file for dumping PDUs (for diagnostic purposes).
       The special name "-" sends output to <literal>stderr</literal>.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-c <replaceable>config-file</replaceable></term>
     <listitem>
      <para>
       Read configuration information from
       <replaceable>config-file</replaceable>.
       The default configuration is <literal>./zebra.cfg</literal>.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-S</term>
     <listitem>
      <para>
       Don't fork on connection requests. This can be useful for
       symbolic-level debugging. The server can only accept a single
       connection in this mode.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-z</term>
     <listitem>
      <para>
       Use the Z39.50 protocol. Currently the only protocol supported.
       The option is retained for historical reasons, and for future
       extensions.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-l <replaceable>logfile</replaceable></term>
     <listitem>
      <para>
       Specify an output file for the diagnostic messages.
       The default is to write this information to <literal>stderr</literal>.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-v <replaceable>log-level</replaceable></term>
     <listitem>
      <para>
       The log level. Use a comma-separated list of members of the set
       {fatal,debug,warn,log,all,none}.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-u <replaceable>username</replaceable></term>
     <listitem>
      <para>
       Set user ID. Sets the real UID of the server process to that of the
       given <replaceable>username</replaceable>.
       It's useful if you aren't comfortable with having the
       server run as root, but you need to start it as such to bind a
       privileged port.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-w <replaceable>working-directory</replaceable></term>
     <listitem>
      <para>
       Change working directory.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-i</term>
     <listitem>
      <para>
       Run under the Internet superserver, <literal>inetd</literal>.
       Make sure you use the logfile option <literal>-l</literal> in
       conjunction with this mode and specify the <literal>-l</literal>
       option before any other options.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-t <replaceable>timeout</replaceable></term>
     <listitem>
      <para>
       Set the idle session timeout (default 60 minutes).
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term>-k <replaceable>kilobytes</replaceable></term>
     <listitem>
      <para>
       Set the (approximate) maximum size of
       present response messages. Default is 1024 KB (1 MB).
      </para>
     </listitem>
    </varlistentry>
   </variablelist>
  </para>
  -->
 </sect1>


 <sect1 id="protocol-support">
  <title>Z39.50 Protocol Support and Behavior</title>

  <sect2>
   <title>Initialization</title>

   <para>
    During initialization, the server will negotiate to version 3 of the
    Z39.50 protocol, and the option bits for Search, Present, Scan,
    NamedResultSets, and concurrentOperations will be set, if requested by
    the client. The maximum PDU size is negotiated down to a maximum of
    1 MB by default.
   </para>

  </sect2>

  <sect2 id="search">
   <title>Search</title>

   <!--
    FIXME - Need to explain the string tag stuff before people get bogged 
            down with all these attribute numbers. Perhaps in its own
            chapter? -H
   -->

   <para>
    The supported query type are 1 and 101. All operators are currently
    supported with the restriction that only proximity units of type "word"
    are supported for the proximity operator.
    Queries can be arbitrarily complex.
    Named result sets are supported, and result sets can be used as operands
    without limitations.
    Searches may span multiple databases.
   </para>

   <para>
    The server has full support for piggy-backed retrieval (see
    also the following section).
   </para>

   <para>
    <emphasis>Use</emphasis> attributes are interpreted according to the
    attribute sets which have been loaded in the
    <literal>zebra.cfg</literal> file, and are matched against specific
    fields as specified in the <literal>.abs</literal> file which
    describes the profile of the records which have been loaded.
    If no Use attribute is provided, a default of Bib-1 Any is assumed.
   </para>

   <para>
    If a <emphasis>Structure</emphasis> attribute of
    <emphasis>Phrase</emphasis> is used in conjunction with a
    <emphasis>Completeness</emphasis> attribute of
    <emphasis>Complete (Sub)field</emphasis>, the term is matched
    against the contents of the phrase (long word) register, if one
    exists for the given <emphasis>Use</emphasis> attribute.
    A phrase register is created for those fields in the
    <literal>.abs</literal> file that contains a
    <literal>p</literal>-specifier.
    <!-- ### whatever the hell _that_ is -->
   </para>

   <para>
    If <emphasis>Structure</emphasis>=<emphasis>Phrase</emphasis> is
    used in conjunction with <emphasis>Incomplete Field</emphasis> - the
    default value for <emphasis>Completeness</emphasis>, the
    search is directed against the normal word registers, but if the term
    contains multiple words, the term will only match if all of the words
    are found immediately adjacent, and in the given order.
    The word search is performed on those fields that are indexed as
    type <literal>w</literal> in the <literal>.abs</literal> file.
   </para>

   <para>
    If the <emphasis>Structure</emphasis> attribute is
    <emphasis>Word List</emphasis>,
    <emphasis>Free-form Text</emphasis>, or
    <emphasis>Document Text</emphasis>, the term is treated as a
    natural-language, relevance-ranked query.
    This search type uses the word register, i.e. those fields
    that are indexed as type <literal>w</literal> in the
    <literal>.abs</literal> file.
   </para>

   <para>
    If the <emphasis>Structure</emphasis> attribute is
    <emphasis>Numeric String</emphasis> the term is treated as an integer.
    The search is performed on those fields that are indexed
    as type <literal>n</literal> in the <literal>.abs</literal> file.
   </para>

   <para>
    If the <emphasis>Structure</emphasis> attribute is
    <emphasis>URx</emphasis> the term is treated as a URX (URL) entity.
    The search is performed on those fields that are indexed as type
    <literal>u</literal> in the <literal>.abs</literal> file.
   </para>

   <para>
    If the <emphasis>Structure</emphasis> attribute is
    <emphasis>Local Number</emphasis> the term is treated as
    native Zebra Record Identifier.
   </para>

   <para>
    If the <emphasis>Relation</emphasis> attribute is
    <emphasis>Equals</emphasis> (default), the term is matched
    in a normal fashion (modulo truncation and processing of
    individual words, if required).
    If <emphasis>Relation</emphasis> is <emphasis>Less Than</emphasis>,
    <emphasis>Less Than or Equal</emphasis>,
    <emphasis>Greater than</emphasis>, or <emphasis>Greater than or
     Equal</emphasis>, the term is assumed to be numerical, and a
    standard regular expression is constructed to match the given
    expression.
    If <emphasis>Relation</emphasis> is <emphasis>Relevance</emphasis>,
    the standard natural-language query processor is invoked.
   </para>

   <para>
    For the <emphasis>Truncation</emphasis> attribute,
    <emphasis>No Truncation</emphasis> is the default.
    <emphasis>Left Truncation</emphasis> is not supported.
    <emphasis>Process # in search term</emphasis> is supported, as is
    <emphasis>Regxp-1</emphasis>.
    <emphasis>Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
    search. As a default, a single error (deletion, insertion, 
    replacement) is accepted when terms are matched against the register
    contents.
   </para>

   <sect3>
    <title>Regular expressions</title>
    
    <para>
     Each term in a query is interpreted as a regular expression if
     the truncation value is either <emphasis>Regxp-1</emphasis> (102)
     or <emphasis>Regxp-2</emphasis> (103).
     Both query types follow the same syntax with the operands:
     <variablelist>

      <varlistentry>
       <term>x</term>
       <listitem>
        <para>
         Matches the character <emphasis>x</emphasis>.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>.</term>
       <listitem>
        <para>
         Matches any character.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>[</literal>..<literal>]</literal></term>
       <listitem>
        <para>
         Matches the set of characters specified;
         such as <literal>[abc]</literal> or <literal>[a-c]</literal>.
        </para>
       </listitem>
      </varlistentry>
     </variablelist>
     and the operators:
     <variablelist>
      
      <varlistentry>
       <term>x*</term>
       <listitem>
        <para>
         Matches <emphasis>x</emphasis> zero or more times. Priority: high.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>x+</term>
       <listitem>
        <para>
         Matches <emphasis>x</emphasis> one or more times. Priority: high.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>x?</term>
       <listitem>
        <para>
         Matches <emphasis>x</emphasis> zero or once. Priority: high.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>xy</term>
       <listitem>
        <para>
         Matches <emphasis>x</emphasis>, then <emphasis>y</emphasis>.
         Priority: medium.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>x|y</term>
       <listitem>
        <para>
         Matches either <emphasis>x</emphasis> or <emphasis>y</emphasis>.
         Priority: low.
        </para>
       </listitem>
      </varlistentry>
     </variablelist>
     The order of evaluation may be changed by using parentheses.
    </para>

    <para>
     If the first character of the <emphasis>Regxp-2</emphasis> query
     is a plus character (<literal>+</literal>) it marks the
     beginning of a section with non-standard specifiers.
     The next plus character marks the end of the section.
     Currently Zebra only supports one specifier, the error tolerance,
     which consists one digit. 
    </para>

    <para>
     Since the plus operator is normally a suffix operator the addition to
     the query syntax doesn't violate the syntax for standard regular
     expressions.
    </para>

   </sect3>

   <sect3>
    <title>Query examples</title>

    <para>
     Phrase search for <emphasis>information retrieval</emphasis> in
     the title-register:
     <screen>
      @attr 1=4 "information retrieval"
     </screen>
    </para>

    <para>
     Ranked search for the same thing:
     <screen>
      @attr 1=4 @attr 2=102 "Information retrieval"
     </screen>
    </para>

    <para>
     Phrase search with a regular expression:
     <screen>
      @attr 1=4 @attr 5=102 "informat.* retrieval"
     </screen>
    </para>

    <para>
     Ranked search with a regular expression:
     <screen>
      @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
     </screen>
    </para>

    <para>
     In the GILS schema (<literal>gils.abs</literal>), the
     west-bounding-coordinate is indexed as type <literal>n</literal>,
     and is therefore searched by specifying
     <emphasis>structure</emphasis>=<emphasis>Numeric String</emphasis>.
     To match all those records with west-bounding-coordinate greater
     than -114 we use the following query:
     <screen>
      @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
     </screen> 
    </para>
   </sect3>
  </sect2>

  <sect2>
   <title>Present</title>
   <para>
    The present facility is supported in a standard fashion. The requested
    record syntax is matched against the ones supported by the profile of
    each record retrieved. If no record syntax is given, SUTRS is the
    default. The requested element set name, again, is matched against any
    provided by the relevant record profiles.
   </para>
  </sect2>
  <sect2>
   <title>Scan</title>
   <para>
    The attribute combinations provided with the termListAndStartPoint are
    processed in the same way as operands in a query (see above).
    Currently, only the term and the globalOccurrences are returned with
    the termInfo structure.
   </para>
  </sect2>
  <sect2>
   <title>Sort</title>

   <para>
    Z39.50 specifies three different types of sort criteria.
    Of these Zebra supports the attribute specification type in which
    case the use attribute specifies the "Sort register".
    Sort registers are created for those fields that are of type "sort" in
    the default.idx file. 
    The corresponding character mapping file in default.idx specifies the
    ordinal of each character used in the actual sort.
   </para>

   <para>
    Z39.50 allows the client to specify sorting on one or more input
    result sets and one output result set.
    Zebra supports sorting on one result set only which may or may not
    be the same as the output result set.
   </para>
  </sect2>
  <sect2>
   <title>Close</title>
   <para>
    If a Close PDU is received, the server will respond with a Close PDU
    with reason=FINISHED, no matter which protocol version was negotiated
    during initialization. If the protocol version is 3 or more, the
    server will generate a Close PDU under certain circumstances,
    including a session timeout (60 minutes by default), and certain kinds of
    protocol errors. Once a Close PDU has been sent, the protocol
    association is considered broken, and the transport connection will be
    closed immediately upon receipt of further data, or following a short
    timeout.
   </para>
  </sect2>
 </sect1>
</chapter>
 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:t
 sgml-parent-document: "zebra.xml"
 sgml-local-catalogs: nil
 sgml-namecase-general:t
 End:
 -->