Use entity idcommon rather than common

[metaproxy-moved-to-github.git] / doc / book.xml

<?xml version="1.0" standalone="no"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
    "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd" 
[
     <!ENTITY % local SYSTEM "local.ent">
     %local;
     <!ENTITY manref SYSTEM "manref.xml">
     <!ENTITY progref SYSTEM "progref.xml">
     <!ENTITY % idcommon SYSTEM "common/common.ent">
     %idcommon;
     <!-- Next line allows imagedata/@format="PDF" and is taken from 
         http://lists.oasis-open.org/archives/docbook/200303/msg00163.html
     -->
     <!ENTITY % local.notation.class "| PDF">
     <!-- Next line is necessary for some XML parsers, for reasons I
          don't understand.  I got this from
         http://lists.oasis-open.org/archives/docbook/200303/msg00180.html
     -->
     <!NOTATION PDF SYSTEM "PDF">
]>
<!-- $Id: book.xml,v 1.60 2007-05-22 11:05:45 adam Exp $ -->
<book id="metaproxy">
 <bookinfo>
  <title>Metaproxy - User's Guide and Reference</title>
  <authorgroup>
   <author>
    <firstname>Adam</firstname><surname>Dickmeiss</surname>
   </author>
   <author>
    <firstname>Marc</firstname><surname>Cromme</surname>
   </author>
   <author>
    <firstname>Mike</firstname><surname>Taylor</surname>
   </author>
  </authorgroup>
  <releaseinfo>&version;</releaseinfo>
  <copyright>
   <year>2005-2007</year>
   <holder>Index Data ApS</holder>
  </copyright>
  <abstract>
   <simpara>
    This manual is part of Metaproxy version &version;.
    </simpara>
   <simpara>
    Metaproxy is a universal router, proxy and encapsulated
    metasearcher for information retrieval protocols.  It accepts,
    processes, interprets and redirects requests from IR clients using
    standard protocols such as the binary
    <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
    and  the information search and retireval 
    web services <ulink url="&url.sru;">SRU</ulink>
    and <ulink url="&url.srw;">SRW</ulink>, as
    well as functioning as a limited
    <ulink url="&url.http;">HTTP</ulink> server. 
   </simpara>
   <simpara>
    Metaproxy is configured by an XML file which
    specifies how the software should function in terms of routes that
    the request packets can take through the proxy, each step on a
    route being an instantiation of a filter.  Filters come in many
    types, one for each operation: accepting Z39.50 packets, logging,
    query transformation, multiplexing, etc.  Further filter-types can
    be added as loadable modules to extend Metaproxy functionality,
    using the filter API.
   </simpara>
   <simpara>
    Metaproxy is covered by the GNU General Public License version 2.
   </simpara>
   <simpara>
    <inlinemediaobject>
     <imageobject>
      <imagedata fileref="common/id.png" format="PNG"/>
     </imageobject>
     <imageobject>
      <imagedata fileref="common/id.eps" format="EPS"/>
     </imageobject>
    </inlinemediaobject>
   </simpara>
  </abstract>
 </bookinfo>

 <chapter id="introduction">
  <title>Introduction</title>
  
  <para>
   <ulink url="&url.metaproxy;">Metaproxy</ulink>
   is a stand alone program that acts as a universal router, proxy and
   encapsulated metasearcher for information retrieval protocols such
   as <ulink url="&url.z39.50;">Z39.50</ulink>, and in the future
   <ulink url="&url.sru;">SRU</ulink> and <ulink url="&url.srw;">SRW</ulink>.
   To clients, it acts as a server of these protocols: it can be searched,
   records can be retrieved from it, etc. 
   To servers, it acts as a client: it searches in them,
   retrieves records from them, etc.  it satisfies its clients'
   requests by transforming them, multiplexing them, forwarding them
   on to zero or more servers, merging the results, transforming
   them, and delivering them back to the client.  In addition, it
   acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
   for further protocols can be added in a modular fashion, through the
   creation of new filters.
  </para>
  <screen>
   Anything goes in!
   Anything goes out!
   Fish, bananas, cold pyjamas,
   Mutton, beef and trout!
        - attributed to Cole Porter.
  </screen>
  <para>
   Metaproxy is a more capable alternative to
   <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
   being more powerful, flexible, configurable and extensible.  Among
   its many advantages over the older, more pedestrian work are
   support for multiplexing (encapsulated metasearching), routing by
   database name, authentication and authorization and serving local
   files via HTTP.  Equally significant, its modular architecture
   facilitites the creation of pluggable modules implementing further
   functionality.
  </para>
  <para>
   This manual will describe how to install Metaproxy
   before giving an overview of its architecture, then discussing the
   key concept of a filter in some depth and giving an overview of
   the various filter types, then discussing the configuration file
   format.  After this come several optional chapters which may be
   freely skipped: a detailed discussion of virtual databases and
   multi-database searching, some notes on writing extensions
   (additional filter types) and a high-level description of the
   source code.  Finally comes the reference guide, which contains
   instructions for invoking the <command>metaproxy</command>
   program, and detailed information on each type of filter,
   including examples.
  </para>
 </chapter>
 
 <chapter id="installation">
  <title>Installation</title>
  <para>
   Metaproxy depends on the following tools/libraries:
   <variablelist>
    <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
     <listitem>
      <para>
       This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
     <listitem>
      <para>This is an XSLT processor - based on 
       <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
       Libxslt must be installed with the development components
       (header files, etc.) as well as the run-time libraries.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
     <listitem>
      <para>
       The popular C++ library. Initial versions of Metaproxy
       was built with 1.33.0. Version 1.33.1 works too.
      </para>
     </listitem>
    </varlistentry>
   </variablelist>
  </para>
  <para>
   In order to compile Metaproxy a modern C++ compiler is
   required. Boost, in particular, requires the C++ compiler
   to facilitate the newest features. Refer to Boost
   <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
   for more information.
  </para>
  <para>
   We have successfully built Metaproxy using the compilers
   <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
   <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
  </para>

  <section id="installation.unix">
   <title>Installation on Unix (from Source)</title>
   <para>
    Here is a quick step-by-step guide on how to compile all the
    tools that Metaproxy uses. Only few systems have none of the required
    tools binary packages. If, for example, Libxml2/libxslt are already
    installed as development packages use those (and omit compilation).
   </para>
   
   <para>
    Libxml2/libxslt:
   </para>
   <screen>
    gunzip -c libxml2-version.tar.gz|tar xf -
    cd libxml2-version
    ./configure
    make
    su
    make install
   </screen>
   <screen>
    gunzip -c libxslt-version.tar.gz|tar xf -
    cd libxslt-version
    ./configure
    make
    su
    make install
   </screen>
   <para>
    YAZ/YAZ++:
   </para>
   <screen>
    gunzip -c yaz-version.tar.gz|tar xf -
    cd yaz-version
    ./configure
    make
    su
    make install
   </screen>
   <screen>
    gunzip -c yazpp-version.tar.gz|tar xf -
    cd yazpp-version
    ./configure
    make
    su
    make install
   </screen>
   <para>
    Boost:
   </para>
   <screen>
    gunzip -c boost-version.tar.gz|tar xf -
    cd boost-version
    ./configure
    make
    su
    make install
   </screen>
   <para>
    Metaproxy:
   </para>
   <screen>
    gunzip -c metaproxy-version.tar.gz|tar xf -
    cd metaproxy-version
    ./configure
    make
    su
    make install
   </screen>
  </section>

  <section id="installation.debian">
   <title>Installation on Debian GNU/Linux</title>
   <para>
    All dependencies for Metaproxy are available as 
    <ulink url="&url.debian;">Debian</ulink>
    packages for the sarge (stable in 2005) and etch (testing in 2005)
    distributions.
   </para>
   <para>
    The procedures for Debian based systems, such as
    <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
   </para>
   <para>
    There is currently no official Debian package for YAZ++.
    And the Debian package for YAZ is probably too old.
    Update the <filename>/etc/apt/sources.list</filename>
    to include the Index Data repository.
    See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
    for more information.
   </para>
   <screen>
    apt-get install libxslt1-dev
    apt-get install libyazpp-dev
    apt-get install libboost-dev
    apt-get install libboost-thread-dev
    apt-get install libboost-date-time-dev
    apt-get install libboost-program-options-dev
    apt-get install libboost-test-dev
   </screen>
   <para>
    With these packages installed, the usual configure + make
    procedure can be used for Metaproxy as outlined in
    <xref linkend="installation.unix"/>.
   </para>
  </section>

  <section id="installation.rpm">
   <title>Installation on RPM based Linux Systems</title>
   <para>
    All external dependencies for Metaproxy are available as 
    RPM packages, either from your distribution site, or from the 
    <ulink url="http://fr.rpmfind.net/">RPMfind</ulink> site.
   </para>
   <para>
    For example, an installation of the requires Boost C++ development 
    libraries on RedHat Fedora C4 and C5 can be done like this:
    <screen> 
    wget ftp://fr.rpmfind.net/wlinux/fedora/core/updates/testing/4/SRPMS/boost-1.33.0-3.fc4.src.rpm
    sudo rpmbuild --buildroot src/ --rebuild -p fc4/boost-1.33.0-3.fc4.src.rpm
    sudo rpm -U /usr/src/redhat/RPMS/i386/boost-*rpm
    </screen>
   </para>
   <para>
    The  <ulink url="&url.yaz;">YAZ</ulink> library is needed to
    compile &metaproxy;, see there
    for more information on available RPM packages.
   </para>
   <para>
    There is currently no official RPM package for YAZ++.
    See the <ulink url="&url.yazplusplus;">YAZ++</ulink> pages 
    for more information on a Unix tarball install.
   </para>
   <para>
    With these packages installed, the usual configure + make
    procedure can be used for Metaproxy as outlined in
    <xref linkend="installation.unix"/>.
   </para>
  </section>

  <section id="installation.windows">
   <title>Installation on Windows</title>
   <para>
    Metaproxy can be compiled with Microsoft
    <ulink url="&url.vstudio;">Visual Studio</ulink>.
    Version 2003 (C 7.1) and 2005 (C 8.0) is known to work.
   </para>
   <section id="installation.windows.boost">
    <title>Boost</title>
    <para>
     Get Boost from its <ulink url="&url.boost;">home page</ulink>.
     You also need Boost Jam (an alternative to make).
     That's also available from the Boost home page.
     The files to be downloaded are called something like:
     <filename>boost_1_33-1.exe</filename>
     and
     <filename>boost-jam-3.1.12-1-ntx86.zip</filename>.
     Unpack Boost Jam first. Put <filename>bjam.exe</filename>
     in your system path. Make a command prompt and ensure
     it can be found automatically. If not check the PATH.
     The Boost .exe is a self-extracting exe with
     complete source for Boost. Compile that source with
     Boost Jam (An alternative to Make).
     The compilation takes a while.
     For Visual Studio 2003, use
     <screen>
      bjam "-sTOOLS=vc-7_1"
     </screen>
     Here <literal>vc-7_1</literal> refers to a "Toolset" (compiler system).
     For Visual Studio 2005, use
     <screen>
      bjam "-sTOOLS=vc-8_0"
     </screen>
     To install the libraries in a common place, use
     <screen>
      bjam "-sTOOLS=vc-7_1" install
     </screen>
     (or vc-8_0 for VS 2005).
    </para>
    <para>
     By default, the Boost build process installs the resulting
     libraries + header files in
     <literal>\boost\lib</literal>, <literal>\boost\include</literal>.
    </para>
    <para>
     For more information about installing Boost refer to the
     <ulink url="&url.boost.getting.started;">getting started</ulink>
     pages.
    </para>
   </section>

   <section id="installation.windows.libxslt">
    <title>Libxslt</title>
    <para>
     <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
     for Windows from
     <ulink url="&url.libxml2.download.win32;">here</ulink>.
    </para>
    <para>
     Libxslt has other dependencies, but these can all be downloaded
     from the same site. Get the following:
     iconv, zlib, libxml2, libxslt.
    </para>
   </section>

   <section id="installation.windows.yaz">
    <title>YAZ</title>
    <para>
     <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
     for Windows from
     <ulink url="&url.yaz.download.win32;">here</ulink>.
    </para>
   </section>

   <section id="installation.windows.yazplusplus">
    <title>YAZ++</title>
    <para>
     Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
     Version 1.0 or later is required. For now get it from
     Index Data's
     <ulink url="&url.snapshot.download;">Snapshot area</ulink>.
    </para>
    <para>
     YAZ++ includes NMAKE makefiles, similar to those found in the
     YAZ package.
    </para>
   </section>

   <section id="installation.windows.metaproxy">
    <title>Metaproxy</title>
    <para>
     Metaproxy is shipped with NMAKE makefiles as well - similar
     to those found in the YAZ++/YAZ packages. Adjust this Makefile
     to point to the proper locations of Boost, Libxslt, Libxml2,
     zlib, iconv, yaz and yazpp.
    </para>

    <variablelist>
     <varlistentry><term><literal>DEBUG</literal></term>
      <listitem><para>
        If set to 1, the software is
        compiled with debugging libraries (code generation is
        multi-threaded debug DLL).
        If set to 0, the software is compiled with release libraries
        (code generation is multi-threaded DLL).
       </para></listitem>
     </varlistentry>

     <varlistentry>
      <term><literal>BOOST</literal></term>
      <listitem>
       <para>
        Boost install location
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term><literal>BOOST_VERSION</literal></term>
      <listitem>
       <para>
        Boost version (replace . with _).
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term><literal>BOOST_TOOLSET</literal></term>
      <listitem>
       <para>
        Boost toolset.
       </para>
      </listitem>
     </varlistentry>

     <varlistentry>
      <term><literal>LIBXSLT_DIR</literal>,
       <literal>LIBXML2_DIR</literal> ..</term>
      <listitem>
       <para>
        Specify the locations of Libxslt, libiconv, libxml2 and
        libxslt.
       </para>
      </listitem>
     </varlistentry>
      
    </variablelist>
    
    <para>
     After successful compilation you'll find
     <literal>metaproxy.exe</literal> in the
     <literal>bin</literal> directory.
    </para>
   </section>


  </section>
 </chapter>
 
<chapter id="yazproxy-comparison">
 <title>YAZ Proxy Comparison</title>
 <para>
  The table below lists facilities either supported by either
   <ulink url="&url.yazproxy;">YAZ Proxy</ulink> or Metaproxy.
 </para>
<table id="yazproxy-comparison-table">
 <title>Metaproxy / YAZ Proxy comparison</title>
 <tgroup cols="3">
  <thead>
   <row>
    <entry>Facility</entry>
    <entry>Metaproxy</entry>
    <entry>YAZ Proxy</entry>
   </row>
  </thead>
  <tbody>
   <row>
    <entry>Z39.50 server</entry>
    <entry>Using filter <literal>frontend_net</literal></entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>SRU server</entry>
    <entry>Supported with filter <literal>sru_z3950</literal></entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>Z39.50 client</entry>
    <entry>Supported with filter <literal>z3950_client</literal></entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>SRU client</entry>
    <entry>Unsupported</entry>
    <entry>Unsupported</entry>
   </row>
   <row>
    <entry>Connection reuse</entry>
    <entry>Supported with filter <literal>session_shared</literal></entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>Connection share</entry>
    <entry>Supported with filter <literal>session_shared</literal></entry>
    <entry>Unsupported</entry>
   </row>
   <row>
    <entry>Result set reuse</entry>
    <entry>Supported with filter <literal>session_shared</literal></entry>
    <entry>Within one Z39.50 session / HTTP keep-alive</entry>
   </row>
   <row>
    <entry>Record cache</entry>
    <entry>Unsupported</entry>
    <entry>Supported for last result set within one Z39.50/HTTP-keep alive session</entry>
   </row>
   <row>
    <entry>Z39.50 Virtual database, i.e. select any Z39.50 target for database</entry>
    <entry>Supported with filter <literal>virt_db</literal></entry>
    <entry>Unsupported</entry>
   </row>
   <row>
    <entry>SRU Virtual database, i.e. select any Z39.50 target for path</entry>
    <entry>Supported with filter <literal>virt_db</literal>, 
     <literal>sru_z3950</literal></entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>Multi target search</entry>
    <entry>Supported with filter <literal>multi</literal> (round-robin)</entry>
    <entry>Unsupported</entry>
   </row>
   <row>
    <entry>Retrieval and search limits</entry>
    <entry>Unsupported</entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>Bandwidth limits</entry>
    <entry>Unsupported</entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>Connect limits</entry>
    <entry>Unsupported</entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>Retrieval sanity check and conversions</entry>
    <entry>Supported using filter <literal>record_transform</literal></entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>Query check</entry>
    <entry>
     Supported in a limited way using <literal>query_rewrite</literal>
    </entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>Query rewrite</entry>
    <entry>Supported with <literal>query_rewrite</literal></entry>
    <entry>Unsupported</entry>
   </row>
   <row>
    <entry>Session invalidate for -1 hits</entry>
    <entry>Unsupported</entry>
    <entry>Supported</entry>
   </row>
   <row>
    <entry>Architecture</entry>
    <entry>Multi-threaded + select for networked modules such as
      <literal>frontend_net</literal>)</entry>
    <entry>Single-threaded using select</entry>
   </row>

   <row>
    <entry>Extensability</entry>
    <entry>Most functionality implemented as loadable modules</entry>
    <entry>Unsupported and experimental</entry>
   </row>

   <row>
    <entry><ulink url="&url.usemarcon;">USEMARCON</ulink></entry>
    <entry>Unsupported</entry>
    <entry>Supported</entry>
   </row>

   <row>
    <entry>Portability</entry>
    <entry>
     Requires YAZ, YAZ++ and modern C++ compiler supporting
     <ulink url="&url.boost;">Boost</ulink>.
    </entry>
    <entry>
     Requires YAZ and YAZ++.
     STL is not required so pretty much any C++ compiler out there should work.
    </entry>
   </row>

  </tbody>
 </tgroup>
</table>
</chapter>

 <chapter id="architecture">
  <title>The Metaproxy Architecture</title>
  <para>
   The Metaproxy architecture is based on three concepts:
   the <emphasis>package</emphasis>,
   the <emphasis>route</emphasis>
   and the <emphasis>filter</emphasis>.
  </para>
  <variablelist>
   <varlistentry>
    <term>Packages</term>
    <listitem>
     <para>
      A package is request or response, encoded in some protocol,
      issued by a client, making its way through Metaproxy, send to or
      received from a server, or sent back to the client.
     </para>
     <para>
      The core of a package is the protocol unit - for example, a
      Z39.50 Init Request or Search Response, or an SRU searchRetrieve
      URL or Explain Response.  In addition to this core, a package
      also carries some extra information added and used by Metaproxy
      itself.
     </para>
     <para>
      In general, packages are doctored as they pass through
      Metaproxy.  For example, when the proxy performs authentication
      and authorization on a Z39.50 Init request, it removes the
      authentication credentials from the package so that they are not
      passed onto the back-end server; and when search-response
      packages are obtained from multiple servers, they are merged
      into a single unified package that makes its way back to the
      client.
     </para>
    </listitem>
   </varlistentry>
   <varlistentry>
    <term>Routes</term>
    <listitem>
     <para>
      Packages make their way through routes, which can be thought of
      as programs that operate on the package data-type.  Each
      incoming package initially makes its way through a default
      route, but may be switched to a different route based on various
      considerations.  Routes are made up of sequences of filters (see
      below).
     </para>
    </listitem>
   </varlistentry>
   <varlistentry>
    <term>Filters</term>
    <listitem>
     <para>
      Filters provide the individual instructions within a route, and
      effect the necessary transformations on packages.  A particular
      configuration of Metaproxy is essentially a set of filters,
      described by configuration details and arranged in order in one
      or more routes.  There are many kinds of filter - about a dozen
      at the time of writing with more appearing all the time - each
      performing a specific function and configured by different
      information.
     </para>
     <para>
      The word ``filter'' is sometimes used rather loosely, in two
      different ways: it may be used to mean a particular
      <emphasis>type</emphasis> of filter, as when we speak of ``the
      auth_simple filter'' or ``the multi filter''; or it may be used
      to be a specific <emphasis>instance</emphasis> of a filter
      within a Metaproxy configuration.  For example, a single
      configuration will often contain multiple instances of the
      <literal>z3950_client</literal> filter.  In
      operational terms, of these is a separate filter.  In practice,
      context always make it clear which sense of the word ``filter''
      is being used.
     </para>
     <para>
      Extensibility of Metaproxy is primarily through the creation of
      plugins that provide new filters.  The filter API is small and
      conceptually simple, but there are many details to master.  See
      the section below on
      <link linkend="filters">Filters</link>.
     </para>
    </listitem>
   </varlistentry>
  </variablelist>
  <para>
   Since packages are created and handled by the system itself, and
   routes are conceptually simple, most of the remainder of this
   document concentrates on filters.  After a brief overview of the
   filter types follows, along with some thoughts on possible future
   directions.
  </para>
 </chapter>



 <chapter id="filters">
  <title>Filters</title>
  
  
  <section id="filters-introductory-notes">
   <title>Introductory notes</title>
   <para>
    It's useful to think of Metaproxy as an interpreter providing a small
    number of primitives and operations, but operating on a very
    complex data type, namely the ``package''.
   </para>
   <para>
    A package represents a Z39.50 or SRU/W request (whether for Init,
    Search, Scan, etc.)  together with information about where it came
    from.  Packages are created by front-end filters such as
    <literal>frontend_net</literal> (see below), which reads them from
    the network; other front-end filters are possible.  They then pass
    along a route consisting of a sequence of filters, each of which
    transforms the package and may also have side-effects such as
    generating logging.  Eventually, the route will yield a response,
    which is sent back to the origin.
   </para>
   <para>
    There are many kinds of filter: some that are defined statically
    as part of Metaproxy, and others may be provided by third parties
    and dynamically loaded.  They all conform to the same simple API
    of essentially two methods: <function>configure()</function> is
    called at startup time, and is passed an XML DOM tree representing that
    part of the configuration file that pertains to this filter
    instance: it is expected to walk that tree extracting relevant
    information; and <function>process()</function> is called every
    time the filter has to processes a package.
   </para>
   <para>
    While all filters provide the same API, there are different modes
    of functionality.  Some filters are sources: they create
    packages
    (<literal>frontend_net</literal>);
    others are sinks: they consume packages and return a result
    (<literal>backend_test</literal>,
    <literal>bounce</literal>,
    <literal>http_file</literal>, 
    <literal>z3950_client</literal>);
    the others are true filters, that read, process and pass on the
    packages they are fed
    (<literal>auth_simple</literal>,
    <literal>log</literal>,
    <literal>multi</literal>,
    <literal>query_rewrite</literal>,
    <literal>record_transform</literal>,
    <literal>session_shared</literal>,
    <literal>sru_z3950</literal>,
    <literal>template</literal>,
    <literal>virt_db</literal>).
   </para>
 </section>
  
  
  <section id="overview.filter.types">
   <title>Overview of filter types</title>
   <para>
    We now briefly consider each of the types of filter supported by
    the core Metaproxy binary.  This overview is intended to give a
    flavor of the available functionality; more detailed information
    about each type of filter is included below in
    <xref linkend="reference"/>.
   </para>
   <para>
    The filters are here named by the string that is used as the
    <literal>type</literal> attribute of a
    <literal>&lt;filter&gt;</literal> element in the configuration
    file to request them, with the name of the class that implements
    them in parentheses.  (The classname is not needed for normal
    configuration and use of Metaproxy; it is useful only to
    developers.)
   </para>
   <para>
    The filters are here listed in alphabetical order:
   </para>
   
<!--

### New filters:

New virt_db-alike that does inteligent peer choice, explain merging,
adds FD&N to explain.  Keeps init responses (like "virt_db Classic"),
makes routing choices based on local explain knowledge.  Ref IDDI
paper.

Filter to convert Explain Classic to ZeeRex.

CQL2PQF (which needs augmented ZeeRex) - MARC for Talis.

SRU2Z39.50 (ditto).

Figure out what additional information we need in:
        ZeeRex (check against D3.1)
        Init request (e.g. loop detection)
        Query package (e.g. number of hops)
        Query response (e.g. record source)

-->

   <section id="auth_simple">
    <title><literal>auth_simple</literal>
     (mp::filter::AuthSimple)</title>
    <para>
     Simple authentication and authorization.  The configuration
     specifies the name of a file that is the user register, which
     lists <varname>username</varname>:<varname>password</varname>
     pairs, one per line, colon separated. When a session begins, it
     is rejected unless username and passsword are supplied, and match
     a pair in the register.  The configuration file may also specific
     the name of another file that is the target register: this lists
     lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
     sets, one per line, with multiple database names separated by
     commas.  When a search is processed, it is rejected unless the
     database to be searched is one of those listed as available to
     the user.
    </para>
   </section>
   
   <section id="backend_test">
    <title><literal>backend_test</literal>
    (mp::filter::Backend_test)</title>
    <para>
     A partial sink that provides dummy responses in the manner of the
     <literal>yaz-ztest</literal> Z39.50 server.  This is useful only
     for testing.  Seriously, you don't need this.  Pretend you didn't
     even read this section.
    </para>
   </section>
   
   <section id="bounce">
    <title><literal>bounce</literal>
    (mp::filter::Bounce)</title>
    <para>
     A sink that swallows <emphasis>all packages</emphasis>, 
     and returns them almost unprocessed.
     It never sends any package of any type further down the row, but
     sets Z39.50 packages to Z_Close, and HTTP_Request packages to
     HTTP_Response err code 400 packages, and adds a suitable bounce
     message. 
     The bounce filter is usually added at end of each filter chain route
     to prevent infinite hanging of for example HTTP
     requests packages when only the Z39.50 client partial sink 
     filter is found in the
     route.  
    </para>
   </section>
   
   <section id="cql_rpn">
    <title><literal>cql_rpn</literal>
    (mp::filter::CQLtoRPN)</title>
    <para>
     A query language transforming filter which catches Z39.50 
     <literal>searchRequest</literal>
     packages containing <literal>CQL</literal> queries, transforms
     those to <literal>RPN</literal> queries,
     and sends the <literal>searchRequests</literal> on to the next
     filters. It is among other things useful in a SRU context. 
    </para>
   </section>
   
   <section id="frontend_net">
    <title><literal>frontend_net</literal>
     (mp::filter::FrontendNet)</title>
    <para>
     A source that accepts Z39.50 connections from a port
     specified in the configuration, reads protocol units, and
     feeds them into the next filter in the route.  When the result is
     received, it is returned to the original origin.
    </para>
   </section>

   <section id="http_file">
    <title><literal>http_file</literal>
     (mp::filter::HttpFile)</title>
    <para>
     A partial sink which swallows only 
     <literal>HTTP_Request</literal> packages, and 
     returns the contents of files from the local
     filesystem in response to HTTP requests.  
     It lets Z39.50 packages and all other forthcoming package types
     pass untouched. 
     (Yes, Virginia, this
     does mean that Metaproxy is also a Web-server in its spare time.  So
     far it does not contain either an email-reader or a Lisp
     interpreter, but that day is surely coming.)
    </para>
   </section>
   
   <section id="load_balance">
    <title><literal>load_balance</literal>
     (mp::filter::LoadBalance)</title>
    <para>
     Performs load balancing for incoming Z39.50 init requests.
     It is used together with the <literal>virt_db</literal> filter,
     but unlike the <literal>multi</literal> filter it does send an
     entire session to only one of the virtual backends. The 
     <literal>load_balance</literal> filter is assuming that
     all backend targets have equal content, and chooses the backend
     with least load cost for a new session.
    <warning>
     <para>
      This filter is experimental and yet not mature for heavy load
      production sites.
     </para>
    </warning>
   </para>
   </section>
      
   <section id="log">
    <title><literal>log</literal>
     (mp::filter::Log)</title>
    <para>
     Writes logging information to standard output, and passes on
     the package unchanged. A log file name can be specified, as well
     as multiple different logging formats.
   </para>
   </section>

   <section id="multi">
   <title><literal>multi</literal>
     (mp::filter::Multi)</title>
    <para>
     Performs multi-database searching.
     See
     <link linkend="multidb">the extended discussion</link>
     of virtual databases and multi-database searching below.
    </para>
   </section>
   
   <section id="query_rewrite">
   <title><literal>query_rewrite</literal>
     (mp::filter::QueryRewrite)</title>
    <para>
     Rewrites Z39.50 <literal>Type-1</literal> 
     and <literal>Type-101</literal> (``<literal>RPN</literal>'') 
     queries by a
     three-step process: the query is transliterated from Z39.50
     packet structures into an XML representation; that XML
     representation is transformed by an XSLT stylesheet; and the
     resulting XML is transliterated back into the Z39.50 packet
     structure.
    </para>
   </section>
   
   
   <section id="record_transform">
    <title><literal>record_transform</literal>
    (mp::filter::RecordTransform)</title>
    <para>
     This filter acts only on Z3950 present requests, and let all
     other types of packages and requests pass untouched. It's use is
     twofold: blocking Z3950  present requests, which the backend
     server does not understand and can not honor, and transforming
     the present syntax and elementset name according to the rules
     specified, to fetch only existing record formats, and transform
     them on the fly to requested record syntaxes.
    </para>
   </section>

   <section id="session_shared">
    <title><literal>session_shared</literal>
     (mp::filter::SessionShared)</title>
    <para>
     This filter implements global sharing of
     result sets (i.e. between threads and therefore between
     clients), yielding performance improvements by clever resource
     pooling. 
    </para>
   </section>

   <section id="sru_z3950">
    <title><literal>sru_z3950</literal>
    (mp::filter::SRUtoZ3950)</title>
    <para>
     This filter transforms valid
     SRU GET/POST/SOAP searchRetrieve requests to Z3950 init, search,
     and present requests, and wraps the
     received hit counts and XML records into suitable SRU response
     messages.
     The <literal>sru_z3950</literal> filter  processes also  SRU
     GET/POST/SOAP explain requests, returning
     either the absolute minimum required by the standard, or a  full 
     pre-defined ZeeReX explain record.
     See the 
     <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
     standard pages and the 
     <ulink url="&url.sru.explain;">SRU Explain</ulink> pages
     for more information on the correct explain syntax.
     SRU scan requests are not supported yet.
    </para>
   </section>
   
   <section id="template">
    <title><literal>template</literal>
     (mp::filter::Template)</title>
    <para>
     Does nothing at all, merely passing the packet on.  (Maybe it
     should be called <literal>nop</literal> or
     <literal>passthrough</literal>?)  This exists not to be used, but
     to be copied - to become the skeleton of new filters as they are
     written.  As with <literal>backend_test</literal>, this is not
     intended for civilians.
    </para>
   </section>
   
   <section id="virt_db">
    <title><literal>virt_db</literal>
     (mp::filter::VirtualDB)</title>
    <para>
     Performs virtual database selection: based on the name of the
     database in the search request, a server is selected, and its
     address added to the request in a <literal>VAL_PROXY</literal>
     otherInfo packet.  It will subsequently be used by a
     <literal>z3950_client</literal> filter.
     See
     <link linkend="multidb">the extended discussion</link>
     of virtual databases and multi-database searching below.
    </para>
   </section>
   
   <section id="z3950_client">
    <title><literal>z3950_client</literal>
     (mp::filter::Z3950Client)</title>
    <para>
     A partial sink which swallows only Z39.50 packages.
     It performs Z39.50 searching and retrieval by proxying the
     packages that are passed to it.  Init requests are sent to the
     address specified in the <literal>VAL_PROXY</literal> otherInfo
     attached to the request: this may have been specified by client,
     or generated by a <literal>virt_db</literal> filter earlier in
     the route.  Subsequent requests are sent to the same address,
     which is remembered at Init time in a Session object.
     HTTP_Request packages and all other forthcoming package types
     are passed untouched. 
    </para>
  </section>


   <section id="zeerex_explain">
    <title><literal>zeerex_explain</literal>
     (mp::filter::ZeerexExplain)</title>
    <para>
     This filter acts as a sink for
     Z39.50 explain requests, returning a static ZeeReX
     Explain XML record from the config section. All other packages
     are passed through.
     See the 
     <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
     standard pages
     for more information on the correct explain syntax.
    </para>
    <warning>
     <para>
      This filter is not yet completed.
     </para>
    </warning>
   </section>
   

  </section>
  
  
  <section id="future.directions">
   <title>Future directions</title>
  <para>
    Some other filters that do not yet exist, but which would be
    useful, are briefly described.  These may be added in future
    releases (or may be created by third parties, as loadable
    modules).
   </para>

   <variablelist>
    <varlistentry>
     <term><literal>frontend_cli</literal> (source)</term>
    <listitem>
      <para>
       Command-line interface for generating requests.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term><literal>sru_client</literal> (sink)</term>
     <listitem>
      <para>
       SRU/GET and SRU/SOAP searching and retrieval.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term><literal>opensearch_client</literal> (sink)</term>
     <listitem>
      <para>
       A9 OpenSearch searching and retrieval.
      </para>
     </listitem>
    </varlistentry>
   </variablelist>
  </section>
 </chapter>
 
 
 
 <chapter id="configuration">
  <title>Configuration: the Metaproxy configuration file format</title>
  
  
  <section id="configuration-introductory-notes">
   <title>Introductory notes</title>
   <para>
    If Metaproxy is an interpreter providing operations on packages, then
    its configuration file can be thought of as a program for that
    interpreter.  Configuration is by means of a single XML file, the name
    of which is supplied as the sole command-line argument to the
    <command>metaproxy</command> program.  (See
    <xref linkend="reference"/> below for more information on invoking
    Metaproxy.)
   </para>
  </section>
  
  <section id="overview.xml.structure">
   <title>Overview of the config file XML structure</title>
   <para>
    All elements and attributes are in the namespace
    <ulink url="http://indexdata.com/metaproxy"/>.
     This is most easily achieved by setting the default namespace on
     the top-level element, as here:
   </para>
   <screen>
    &lt;metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0"&gt;
   </screen>
   <para>
    The top-level element is &lt;metaproxy&gt;.  This contains a
    &lt;start&gt; element, a &lt;filters&gt; element and a
    &lt;routes&gt; element, in that order.  &lt;filters&gt; is
    optional; the other two are mandatory.  All three are
    non-repeatable.
   </para>
  <para>
    The &lt;start&gt; element is empty, but carries a
    <literal>route</literal> attribute, whose value is the name of
    route at which to start running - analogous to the name of the
    start production in a formal grammar.
   </para>
  <para>
    If present, &lt;filters&gt; contains zero or more &lt;filter&gt;
    elements.  Each filter carries a <literal>type</literal> attribute
    which specifies what kind of filter is being defined
    (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
    and contain various elements that provide suitable configuration
    for a filter of its type.  The filter-specific elements are
    described in
    <xref linkend="reference"/>.
    Filters defined in this part of the file must carry an
    <literal>id</literal> attribute so that they can be referenced
    from elsewhere.
   </para>
   <para>
    &lt;routes&gt; contains one or more &lt;route&gt; elements, each
    of which must carry an <literal>id</literal> element.  One of the
    routes must have the ID value that was specified as the start
    route in the &lt;start&gt; element's <literal>route</literal>
    attribute.  Each route contains zero or more &lt;filter&gt;
    elements.  These are of two types.  They may be empty, but carry a
    <literal>refid</literal> attribute whose value is the same as the
    <literal>id</literal> of a filter previously defined in the
    &lt;filters&gt; section.  Alternatively, a route within a filter
    may omit the <literal>refid</literal> attribute, but contain
    configuration elements similar to those used for filters defined
    in the &lt;filters&gt; section.  (In other words, each filter in a
    route may be included either by reference or by physical
    inclusion.)
   </para>
  </section>


  <section id="example.configuration">
   <title>An example configuration</title>
   <para>
    The following is a small, but complete, Metaproxy configuration
    file (included in the distribution as
    <literal>metaproxy/etc/config1.xml</literal>).
    This file defines a very simple configuration that simply proxies
    to whatever back-end server the client requests, but logs each
    request and response.  This can be useful for debugging complex
    client-server dialogues.
   </para>
   <screen><![CDATA[<?xml version="1.0"?>
<metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
  <start route="start"/>
  <filters>
    <filter id="frontend" type="frontend_net">
      <port>@:9000</port>
    </filter>
    <filter id="backend" type="z3950_client">
    </filter>
  </filters>
  <routes>  
    <route id="start">
      <filter refid="frontend"/>
      <filter type="log"/>
      <filter refid="backend"/>
      <filter type="bounce"/>
    </route>
  </routes>
</metaproxy>
]]></screen>
   <para>
    It works by defining a single route, called
    <literal>start</literal>, which consists of a sequence of four
    filters.  The first and last of these are included by reference:
    their <literal>&lt;filter&gt;</literal> elements have
    <literal>refid</literal> attributes that refer to filters defined
    within the prior <literal>&lt;filters&gt;</literal> section.  The
    middle filter is included inline in the route.
   </para>
   <para>
    The four filters in the route are as follows: first, a
    <literal>frontend_net</literal> filter accepts Z39.50 requests
    from any host on port 9000; then these requests are passed through
    a <literal>log</literal> filter that emits a message for each
    request; they are then fed into a <literal>z3950_client</literal>
    filter, which forwards all Z39.50 requests to the client-specified
    back-end Z39.509 server. Those Z39.50 packages are returned by the
    <literal>z3950_client</literal> filter, with the response data
    filled by the external Z39.50 server targeted.
    All non-Z39.50 packages are passed through to the
    <literal>bounce</literal> filter, which definitely bounces
    everything, including fish, bananas, cold pyjamas,
    mutton, beef and trout packages.
    When the response arrives, it is handed
    back to the <literal>log</literal> filter, which emits another
    message; and then to the <literal>frontend_net</literal> filter, 
    which returns the response to the client.
   </para>
  </section>

  <section id="config-file-modularity">
   <title>Config file modularity</title>
   <para>
    Metaproxy XML configuration snippets can be reused by other
    filters using the <literal>XInclude</literal> standard, as seen in
    the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU 
    configuration.
   <screen><![CDATA[
    <filter id="sru" type="sru_z3950">
      <database name="Default">
       <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
                    href="explain.xml"/>
      </database>
    </filter>
]]></screen>
    </para>
  </section>

  <section id="config-file-syntax-check">
   <title>Config file syntax checking</title>
   <para>
    The distribution contains RelaxNG Compact and XML syntax checking
    files, as well as XML Schema files. These are found in the
    distribution paths 
   <screen>
    xml/schema/metaproxy.rnc
    xml/schema/metaproxy.rng
    xml/schema/metaproxy.xsd
   </screen>
    and can be used to verify or debug the XML structure of
    configuration files. For example, using the utility
    <filename>xmllint</filename>, syntax checking is done like this:
   <screen>
    xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml 
    xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml 
   </screen>
    (A recent version of <literal>libxml2</literal> is required, as
    support for XML Schemas is a relatively recent addition.)
   </para>
   <para>
    You can of course use any other RelaxNG or XML Schema compliant tool
    you wish.
   </para>
   </section>
 </chapter>



 <chapter id="multidb">
  <title>Virtual databases and multi-database searching</title>


  <section id="multidb-introductory-notes">
   <title>Introductory notes</title>
   <para>
    Two of Metaproxy's filters are concerned with multiple-database
    operations.  Of these, <literal>virt_db</literal> can work alone
    to control the routing of searches to one of a number of servers,
    while <literal>multi</literal> can work together with
    <literal>virt_db</literal> to perform multi-database searching, merging
    the results into a unified result-set - ``metasearch in a box''.
   </para>
   <para>
    The interaction between
    these two filters is necessarily complex: it reflects the real,
    irreducible complexity of multi-database searching in a protocol such
    as Z39.50 that separates initialization from searching, and in
    which the database to be searched is not known at initialization
    time.
   </para>
   <para>
    It's possible to use these filters without understanding the
    details of their functioning and the interaction between them; the
    next two sections of this chapter are ``HOW-TO'' guides for doing
    just that.  However, debugging complex configurations will require
    a deeper understanding, which the last two sections of this
    chapters attempt to provide.
   </para>
  </section>


  <section id="multidb.virt_db">
   <title>Virtual databases with the <literal>virt_db</literal> filter</title>
   <para>
    Working alone, the purpose of the
    <literal>virt_db</literal>
    filter is to route search requests to one of a selection of
    back-end databases.  In this way, a single Z39.50 endpoint
    (running Metaproxy) can provide access to several different
    underlying services, including those that would otherwise be
    inaccessible due to firewalls.  In many useful configurations, the
    back-end databases are local to the Metaproxy installation, but
    the software does not enforce this, and any valid Z39.50 servers
    may be used as back-ends.
   </para>
   <para>
    For example, a <literal>virt_db</literal>
    filter could be set up so that searches in the virtual database
    ``lc'' are forwarded to the Library of Congress bibliographic
    catalogue server, and searches in the virtual database ``marc''
    are forwarded to the toy database of MARC records that Index Data
    hosts for testing purposes.  A <literal>virt_db</literal>
    configuration to make this switch would look like this:
   </para>
   <screen><![CDATA[<filter type="virt_db">
  <virtual>
    <database>lc</database>
    <target>z3950.loc.gov:7090/voyager</target>
  </virtual>
  <virtual>
    <database>marc</database>
    <target>indexdata.com/marc</target>
  </virtual>
</filter>]]></screen>
   <para>
    As well as being useful in it own right, this filter also provides
    the foundation for multi-database searching.
   </para>
  </section>


  <section id="multidb.multi">
   <title>Multi-database search with the <literal>multi</literal> filter</title>
   <para>
    To arrange for Metaproxy to broadcast searches to multiple back-end
    servers, the configuration needs to include two components: a
    <literal>virt_db</literal>
    filter that specifies multiple
    <literal>&lt;target&gt;</literal>
    elements, and a subsequent
    <literal>multi</literal>
    filter.  Here, for example, is a complete configuration that
    broadcasts searches to both the Library of Congress catalogue and
    Index Data's tiny testing database of MARC records:
   </para>
   <screen><![CDATA[<?xml version="1.0"?>
<metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
  <start route="start"/>
  <routes>
    <route id="start">
      <filter type="frontend_net">
        <threads>10</threads>
        <port>@:9000</port>
      </filter>
      <filter type="virt_db">
        <virtual>
          <database>lc</database>
          <target>z3950.loc.gov:7090/voyager</target>
        </virtual>
        <virtual>
          <database>marc</database>
          <target>indexdata.com/marc</target>
        </virtual>
        <virtual>
          <database>all</database>
          <target>z3950.loc.gov:7090/voyager</target>
          <target>indexdata.com/marc</target>
        </virtual>
      </filter>
      <filter type="multi"/>
      <filter type="z3950_client">
        <timeout>30</timeout>
      </filter>
      <filter type="bounce"/>
    </route>
  </routes>
</metaproxy>]]></screen>
   <para>
    (Using a
    <literal>virt_db</literal>
    filter that specifies multiple
    <literal>&lt;target&gt;</literal>
    elements but without a subsequent
    <literal>multi</literal>
    filter yields surprising and undesirable results, as will be
    described below.  Don't do that.)
   </para>
   <para>
    Metaproxy can be invoked with this configuration as follows:
   </para>
   <screen>../src/metaproxy --config config-simple-multi.xml</screen>
   <para>
    And thereafter, Z39.50 clients can connect to the running server
    (on port 9000, as specified in the configuration) and search in
    any of the databases
    <literal>lc</literal> (the Library of Congress catalogue),
    <literal>marc</literal> (Index Data's test database of MARC records)
    or
    <literal>all</literal> (both of these).  As an example, a session
    using the YAZ command-line client <literal>yaz-client</literal> is
    here included (edited for brevity and clarity):
   </para>
   <screen><![CDATA[$ yaz-client @:9000
Connecting...OK.
Z> base lc
Z> find computer
Search was a success.
Number of hits: 10000, setno 1
Elapsed: 5.521070
Z> base marc
Z> find computer
Search was a success.
Number of hits: 10, setno 3
Elapsed: 0.060187
Z> base all
Z> find computer
Search was a success.
Number of hits: 10010, setno 4
Elapsed: 2.237648
Z> show 1
[marc]Record type: USmarc
001    11224466
003 DLC
005 00000000000000.0
008 910710c19910701nju           00010 eng
010    $a 11224466
040    $a DLC $c DLC
050 00 $a 123-xyz
100 10 $a Jack Collins
245 10 $a How to program a computer
260 1  $a Penguin
263    $a 8710
300    $a p. cm.
Elapsed: 0.119612
Z> show 2
[VOYAGER]Record type: USmarc
001 13339105
005 20041229102447.0
008 030910s2004    caua          000 0 eng
035    $a (DLC)  2003112666
906    $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
925 0  $a acquire $b 1 shelf copy $x policy default
955    $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
010    $a   2003112666
020    $a 0761542892
040    $a DLC $c DLC $d DLC
050 00 $a MLCM 2004/03312 (G)
245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
246 3  $a Double-O-seven, everything or nothing
246 30 $a Prima's official strategy guide
260    $a Roseville, CA : $b Prima Games, $c c2004.
300    $a 161 p. : $b col. ill. ; $c 28 cm.
500    $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
650  0 $a Video games.
710 2  $a Kaizen Media Group.
856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
Elapsed: 0.150623
Z>
]]></screen>
   <para>
    As can be seen, the first record in the result set is from the
    Index Data test database, and the second from the Library of
    Congress database.  The result-set continues alternating records
    round-robin style until the point where one of the databases'
    records are exhausted.
   </para>
   <para>
    This example uses only two back-end databases; more may be used.
    There is no limitation imposed on the number of databases that may
    be metasearched in this way: issues of resource usage and
    administrative complexity dictate the practical limits.
   </para>
   <para>
    What happens when one of the databases doesn't respond?  By default,
    the entire multi-database search fails, and the appropriate
    diagnostic is returned to the client.  This is usually appropriate
    during development, when technicians need maximum information, but
    can be inconvenient in deployment, when users typically don't want
    to be bothered with problems of this kind and prefer just to get
    the records from the databases that are available.  To obtain this
    latter behavior add an empty
    <literal>&lt;hideunavailable&gt;</literal>
    element inside the
    <literal>multi</literal> filter:
   </para>
   <screen><![CDATA[      <filter type="multi">
        <hideunavailable/>
      </filter>]]></screen>
   <para>
    Under this regime, an error is reported to the client only if
    <emphasis>all</emphasis> the databases in a multi-database search
    are unavailable.
   </para>
  </section>


  <section id="multidb.what">
   <title>What's going on?</title>
   <warning>
    <title>Lark's vomit</title>
    <para>
     This section goes into a level of technical detail that is
     probably not necessary in order to configure and use Metaproxy.
     It is provided only for those who like to know how things work.
     You should feel free to skip on to the next section if this one
     doesn't seem like fun.
    </para>
   </warning>
   <para>
    Hold on tight - this may get a little hairy.
   </para>
   <para>
    In the general course of things, a Z39.50 Init request may carry
    with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
    whose value indicates the address of a Z39.50 server to which the
    ultimate connection is to be made.  (This otherInfo packet is
    supported by YAZ-based Z39.50 clients and servers, but has not yet
    been ratified by the Maintenance Agency and so is not widely used
    in non-Index Data software.  We're working on it.)
    The <literal>VAL_PROXY</literal> packet functions
    analogously to the absoluteURI-style Request-URI used with the GET
    method when a web browser asks a proxy to forward its request: see
    the
    <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
           >Request-URI</ulink>
    section of
    <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
           >the HTTP 1.1 specification</ulink>.
   </para>
   <para>
    Within Metaproxy, Search requests that are part of the same
    session as an Init request that carries a
    <literal>VAL_PROXY</literal> otherInfo are also annotated with the
    same information.  The role of the <literal>virt_db</literal>
    filter is to rewrite this otherInfo packet dependent on the
    virtual database that the client wants to search.
   </para>
   <para>
    When Metaproxy receives a Z39.50 Init request from a client, it
    doesn't immediately forward that request to the back-end server.
    Why not?  Because it doesn't know <emphasis>which</emphasis>
    back-end server to forward it to until the client sends a Search
    request that specifies the database that it wants to search in.
    Instead, it just treasures the Init request up in its heart; and,
    later, the first time the client does a search on one of the
    specified virtual databases, a connection is forged to the
    appropriate server and the Init request is forwarded to it.  If,
    later in the session, the same client searches in a different
    virtual database, then a connection is forged to the server that
    hosts it, and the same cached Init request is forwarded there,
    too.
   </para>
   <para>
    All of this clever Init-delaying is done by the
    <literal>frontend_net</literal> filter.  The
    <literal>virt_db</literal> filter knows nothing about it; in
    fact, because the Init request that is received from the client
    doesn't get forwarded until a Search request is received, the
    <literal>virt_db</literal> filter (and the
    <literal>z3950_client</literal> filter behind it) doesn't even get
    invoked at Init time.  The <emphasis>only</emphasis> thing that a
    <literal>virt_db</literal> filter ever does is rewrite the
    <literal>VAL_PROXY</literal> otherInfo in the requests that pass
    through it.
   </para>
   <para>
    It is possible for a <literal>virt_db</literal> filter to contain
    multiple
    <literal>&lt;target&gt;</literal>
    elements.  What does this mean?  Only that the filter will add
    multiple <literal>VAL_PROXY</literal> otherInfo packets to the
    Search requests that pass through it.  That's because the virtual
    DB filter is dumb, and does exactly what it's told - no more, no
    less.
    If a Search request with multiple <literal>VAL_PROXY</literal>
    otherInfo packets reaches a <literal>z3950_client</literal>
    filter, this is an error.  That filter doesn't know how to deal
    with multiple targets, so it will either just pick one and search
    in it, or (better) fail with an error message.
   </para>
   <para>
    The <literal>multi</literal> filter comes to the rescue!  This is
    the only filter that knows how to deal with multiple
    <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
    making multiple copies of the entire Search request: one for each
    <literal>VAL_PROXY</literal>.  Each of these new copies is then
    passed down through the remaining filters in the route.  (The
    copies are handled in parallel though the
    spawning of new threads.)  Since the copies each have only one
    <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
    <literal>z3950_client</literal> filter, which happily deals with
    each one individually.  When the results of the individual
    searches come back up to the <literal>multi</literal> filter, it
    merges them into a single Search response, which is what
    eventually makes it back to the client.
   </para>
  </section>


  <section id="multidb.picture">
   <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
   <simpara>
    <inlinemediaobject>
     <imageobject>
      <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
     </imageobject>
     <imageobject>
      <imagedata fileref="multi.png" format="PNG"/>
     </imageobject>
     <textobject>
      <!-- Fall back if none of the images can be used -->
      <phrase>
       [Here there should be a diagram showing the progress of
       packages through the filters during a simple virtual-database
       search and a multi-database search, but is seems that your
       tool chain has not been able to include the diagram in this
       document.]
      </phrase>
     </textobject>
<!-- ### This used to work with an older version of DocBook
     <caption>
      <para>Caption: progress of packages through filters.</para>
     </caption>
-->
    </inlinemediaobject>
   </simpara>
  </section>
 </chapter>


 <chapter id="sru-server">
  <title>Combined SRU webservice and Z39.50 server configuration</title>
  <para>
   Metaproxy can act as 
   <ulink url="&url.sru;">SRU</ulink> and 
   <ulink url="&url.srw;">SRW</ulink> 
   web service server, which translates web service requests to 
   <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink> packages and
   sends them off to common available targets.
  </para>
  <para>
  A typical setup for this operation needs a filter route including the
  following modules: 
  </para>
   
  <table id="sru-server-table-config" frame="top">
   <title>SRU/Z39.50 Server Filter Route Configuration</title>
   <tgroup cols="3">
    <thead>
     <row>
      <entry>Filter</entry>
      <entry>Importance</entry>
      <entry>Purpose</entry>
     </row>
    </thead>
    
    <tbody>
     <row>
      <entry><literal>frontend_net</literal></entry>
      <entry>required</entry>
      <entry>Accepting HTTP connections and passing them to following
      filters. Since this filter also accepts Z39.50 connections, the
      server works as SRU and Z39.50 server on the same port.</entry>
     </row>
     <row>
      <entry><literal>sru_z3950</literal></entry>
      <entry>required</entry>
      <entry>Accepting SRU GET/POST/SOAP explain and
       searchRetrieve requests for the the configured databases.
       Explain requests are directly served from the static XML configuration.
       SearchRetrieve requests are
       transformed  to Z39.50 search and present packages.
       All other HTTP and Z39.50 packages are passed unaltered.</entry>
     </row>
     <row>
      <entry><literal>http_file</literal></entry>
      <entry>optional</entry>
      <entry>Serving HTTP requests from the filesystem. This is only
      needed if the server should serve XSLT stylesheets, static HTML
      files or Java Script for thin browser based clients.
       Z39.50 packages are passed unaltered.</entry>
     </row>
     <row>
      <entry><literal>cql_rpn</literal></entry>
      <entry>required</entry>
      <entry>Usually, Z39.50 servers do not talk CQL, hence the
      translation of the CQL query language to RPN is mandatory in
      most cases. Affects only  Z39.50 search packages.</entry>
     </row>
     <row>
      <entry><literal>record_transform</literal></entry>
      <entry>optional</entry>
      <entry>Some Z39.50 backend targets can not present XML record
      syntaxes in common wanted element sets. using this filter, one
      can transform binary MARC records to MARCXML records, and
      further transform those to any needed XML schema/format by XSLT
      transformations. Changes only  Z39.50 present packages.</entry>
     </row>
     <row>
      <entry><literal>session_shared</literal></entry>
      <entry>optional</entry>
      <entry>The stateless nature of web services requires frequent
      re-searching of the same targets for display of paged result set
      records. This might be an unacceptable burden for the accessed
      backend Z39.50 targets, and this mosule can be added for
      efficient backend target resource pooling.</entry>
     </row>
     <row>
      <entry><literal>z3950_client</literal></entry>
      <entry>required</entry>
      <entry>Finally, a Z39.50 package sink is needed in the filter
      chain to provide the response packages. The Z39.50 client module
      is used to access external targets over the network, but any
      coming local Z39.50 package sink could be used instead of.</entry>
     </row>
     <row>
      <entry><literal>bounce</literal></entry>
      <entry>required</entry>
      <entry>Any Metaproxy package arriving here did not do so by
      purpose, and is bounced back with connection closure. this
      prevents inifinite package hanging inside the SRU server.</entry>
     </row>
    </tbody>
   </tgroup>
  </table>
  <para> 
   A typical minimal example  <ulink url="&url.sru;">SRU</ulink> and 
   <ulink url="&url.srw;">SRW</ulink> server configuration file is found
   in the tarball distribution at 
   <literal>etc/config-sru-to-z3950.xml</literal>.
  </para> 
  <para>
   Off course, any other metaproxy modules can be integrated into a
   SRU server solution, including, but not limited to, load balancing, 
   multiple target querying 
   (see  <xref linkend="multidb"/>), and complex RPN query rewrites. 
  </para>


 </chapter>

 <!--
 <chapter id="extensions">
  <title>Writing extensions for Metaproxy</title>
  <para>### To be written</para>
 </chapter>
 -->



 <chapter id="classes">
  <title>Classes in the Metaproxy source code</title>


  <section id="classes-introductory-notes">
   <title>Introductory notes</title>
   <para>
    <emphasis>Stop!  Do not read this!</emphasis>
    You won't enjoy it at all.  You should just skip ahead to
    <xref linkend="reference"/>,
    which tells
    <!-- The remainder of this paragraph is lifted verbatim from
    Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
    you things you really need to know, like the fact that the
    fabulously beautiful planet Bethselamin is now so worried about
    the cumulative erosion by ten billion visiting tourists a year
    that any net imbalance between the amount you eat and the amount
    you excrete whilst on the planet is surgically removed from your
    bodyweight when you leave: so every time you go to the lavatory it
    is vitally important to get a receipt.
   </para>
   <para>
    This chapter contains documentation of the Metaproxy source code, and is
    of interest only to maintainers and developers.  If you need to
    change Metaproxy's behavior or write a new filter, then you will most
    likely find this chapter helpful.  Otherwise it's a waste of your
    good time.  Seriously: go and watch a film or something.
    <citetitle>This is Spinal Tap</citetitle> is particularly good.
   </para>
   <para>
    Still here?  OK, let's continue.
   </para>
   <para>
    In general, classes seem to be named big-endianly, so that
    <literal>FactoryFilter</literal> is not a filter that filters
    factories, but a factory that produces filters; and
    <literal>FactoryStatic</literal> is a factory for the statically
    registered filters (as opposed to those that are dynamically
    loaded).
   </para>
  </section>

  <section id="individual.classes">
   <title>Individual classes</title>
   <para>
    The classes making up the Metaproxy application are here listed by
    class-name, with the names of the source files that define them in
    parentheses.
   </para>

   <section id="class-FactoryFilter">
    <title><literal>mp::FactoryFilter</literal>
     (<filename>factory_filter.cpp</filename>)</title>
    <para>
     A factory class that exists primarily to provide the
     <literal>create()</literal> method, which takes the name of a
     filter class as its argument and returns a new filter of that
     type.  To enable this, the factory must first be populated by
     calling <literal>add_creator()</literal> for static filters (this
     is done by the <literal>FactoryStatic</literal> class, see below)
     and <literal>add_creator_dyn()</literal> for filters loaded
     dynamically.
    </para>
   </section>

   <section id="class-FactoryStatic">
    <title><literal>mp::FactoryStatic</literal>
     (<filename>factory_static.cpp</filename>)</title>
    <para>
     A subclass of <literal>FactoryFilter</literal> which is
     responsible for registering all the statically defined filter
     types.  It does this by knowing about all those filters'
     structures, which are listed in its constructor.  Merely
     instantiating this class registers all the static classes.  It is
     for the benefit of this class that <literal>struct
      metaproxy_1_filter_struct</literal> exists, and that all the filter
     classes provide a static object of that type.
    </para>
   </section>

   <section id="class-filter-Base">
    <title><literal>mp::filter::Base</literal>
     (<filename>filter.cpp</filename>)</title>
    <para>
     The virtual base class of all filters.  The filter API is, on the
     surface at least, extremely simple: two methods.
     <literal>configure()</literal> is passed an XML DOM tree representing
     that part of the configuration file that pertains to this filter
     instance, and is expected to walk that tree extracting relevant
     information.  And <literal>process()</literal> processes a
     package (see below).  That surface simplicity is a bit
     misleading, as <literal>process()</literal> needs to know a lot
     about the <literal>Package</literal> class in order to do
     anything useful.
    </para>
   </section>

   <section id="class-AuthSimple">
    <title><literal>mp::filter::AuthSimple</literal>,
     <literal>Backend_test</literal>, etc.
     (<filename>filter_auth_simple.cpp</filename>,
     <filename>filter_backend_test.cpp</filename>, etc.)</title>
    <para>
     Individual filters.  Each of these is implemented by a header and
     a source file, named <filename>filter_*.hpp</filename> and
     <filename>filter_*.cpp</filename> respectively.  All the header
     files should be pretty much identical, in that they declare the
     class, including a private <literal>Rep</literal> class and a
     member pointer to it, and the two public methods.
    </para>
    <para>
     The source file for each filter needs to supply:
    </para>
    <itemizedlist>
     <listitem>
      <para>
       A definition of the private <literal>Rep</literal> class.
      </para>
     </listitem>
     <listitem>
      <para>
       Some boilerplate constructors and destructors.
      </para>
     </listitem>
     <listitem>
      <para>
       A <literal>configure()</literal> method that uses the
       appropriate XML fragment.
      </para>
     </listitem>
     <listitem>
      <para>
       Most important, the <literal>process()</literal> method that
       does all the actual work.
      </para>
     </listitem>
    </itemizedlist>
   </section>

   <section id="class-Package">
    <title><literal>mp::Package</literal>
     (<filename>package.cpp</filename>)</title>
    <para>
     Represents a package on its way through the series of filters
     that make up a route.  This is essentially a Z39.50 or SRU APDU
     together with information about where it came from, which is
     modified as it passes through the various filters.
    </para>
   </section>

   <section id="class-Pipe">
    <title><literal>mp::Pipe</literal>
     (<filename>pipe.cpp</filename>)</title>
    <para>
     This class provides a compatibility layer so that we have an IPC
     mechanism that works the same under Unix and Windows.  It's not
     particularly exciting.
    </para>
   </section>

   <section id="class-RouterChain">
    <title><literal>mp::RouterChain</literal>
     (<filename>router_chain.cpp</filename>)</title>
    <para>
     ### to be written
    </para>
   </section>

   <section id="class-RouterFleXML">
    <title><literal>mp::RouterFleXML</literal>
     (<filename>router_flexml.cpp</filename>)</title>
    <para>
     ### to be written
    </para>
   </section>

   <section id="class-Session">
    <title><literal>mp::Session</literal>
     (<filename>session.cpp</filename>)</title>
    <para>
     ### to be written
    </para>
   </section>

   <section id="class-ThreadPoolSocketObserver">
    <title><literal>mp::ThreadPoolSocketObserver</literal>
     (<filename>thread_pool_observer.cpp</filename>)</title>
    <para>
     ### to be written
    </para>
   </section>

   <section id="class-util">
    <title><literal>mp::util</literal>
     (<filename>util.cpp</filename>)</title>
    <para>
     A namespace of various small utility functions and classes,
     collected together for convenience.  Most importantly, includes
     the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
     ODR facilities.
    </para>
   </section>

   <section id="class-xml">
    <title><literal>mp::xml</literal>
     (<filename>xmlutil.cpp</filename>)</title>
    <para>
     A namespace of various XML utility functions and classes,
     collected together for convenience.
    </para>
   </section>
  </section>


  <section id="other.source.files">
   <title>Other Source Files</title>
   <para>
    In addition to the Metaproxy source files that define the classes
    described above, there are a few additional files which are
    briefly described here:
   </para>
   <variablelist>
    <varlistentry>
     <term><literal>metaproxy_prog.cpp</literal></term>
     <listitem>
      <para>
       The main function of the <command>metaproxy</command> program.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term><literal>ex_router_flexml.cpp</literal></term>
     <listitem>
      <para>
       Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
      </para>
     </listitem>
    </varlistentry>
    <varlistentry>
     <term><literal>test_*.cpp</literal></term>
     <listitem>
      <para>
       Unit-tests for various modules.
      </para>
     </listitem>
    </varlistentry>
   </variablelist>
   <para>
    ### Still to be described:
    <literal>ex_filter_frontend_net.cpp</literal>,
    <literal>filter_dl.cpp</literal>,
    <literal>plainfile.cpp</literal>,
    <literal>tstdl.cpp</literal>.
   </para>
  </section>
 </chapter>
 
 
 <reference id="reference">
  <title>Reference</title>
   <partintro id="reference-introduction">
    <para>
     The material in this chapter is drawn directly from the individual
     manual entries.  In particular, the Metaproxy invocation section is
     available using <command>man metaproxy</command>, and the section
     on each individual filter is available using the name of the filter
     as the argument to the <command>man</command> command.
    </para>
   </partintro>
   &manref;
 </reference>

<appendix id="license">
 <title>License</title>

  <para>
   Metaproxy, Copyright &copy; 1995-2007 Index Data ApS.
 </para>

  <para>
   Metaproxy is free software; you can redistribute it and/or modify it under
   the terms of the GNU General Public License as published by the Free
   Software Foundation; either version 2, or (at your option) any later
   version.
   </para>

  <para>
   Metaproxy is distributed in the hope that it will be useful, but WITHOUT ANY
   WARRANTY; without even the implied warranty of MERCHANTABILITY or
   FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
   for more details.
  </para>
  
  <para>
   You should have received a copy of the GNU General Public License
   along with Metaproxy; see the file LICENSE.  If not, write to the
   Free Software Foundation, 
   51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
   </para>
  
  <sect1 id="gpl">
   <title>GNU General Public License</title>
   <screen>
                    GNU GENERAL PUBLIC LICENSE
                       Version 2, June 1991

 Copyright (C) 1989, 1991 Free Software Foundation, Inc.
                       51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
w Everyone is permitted to copy and distribute verbatim copies
 of this license document, but changing it is not allowed.

                            Preamble

  The licenses for most software are designed to take away your
freedom to share and change it.  By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users.  This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it.  (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.)  You can apply it to
your programs, too.

  When we speak of free software, we are referring to freedom, not
price.  Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.

  To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.

  For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have.  You must make sure that they, too, receive or can get the
source code.  And you must show them these terms so they know their
rights.

  We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.

  Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software.  If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.

  Finally, any free program is threatened constantly by software
patents.  We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary.  To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.

  The precise terms and conditions for copying, distribution and
modification follow.

                    GNU GENERAL PUBLIC LICENSE
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

  0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License.  The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language.  (Hereinafter, translation is included without limitation in
the term "modification".)  Each licensee is addressed as "you".

Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope.  The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.

  1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.

You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.

  2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:

    a) You must cause the modified files to carry prominent notices
    stating that you changed the files and the date of any change.

    b) You must cause any work that you distribute or publish, that in
    whole or in part contains or is derived from the Program or any
    part thereof, to be licensed as a whole at no charge to all third
    parties under the terms of this License.

    c) If the modified program normally reads commands interactively
    when run, you must cause it, when started running for such
    interactive use in the most ordinary way, to print or display an
    announcement including an appropriate copyright notice and a
    notice that there is no warranty (or else, saying that you provide
    a warranty) and that users may redistribute the program under
    these conditions, and telling the user how to view a copy of this
    License.  (Exception: if the Program itself is interactive but
    does not normally print such an announcement, your work based on
    the Program is not required to print an announcement.)

These requirements apply to the modified work as a whole.  If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works.  But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.

Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.

In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.

  3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:

    a) Accompany it with the complete corresponding machine-readable
    source code, which must be distributed under the terms of Sections
    1 and 2 above on a medium customarily used for software interchange; or,

    b) Accompany it with a written offer, valid for at least three
    years, to give any third party, for a charge no more than your
    cost of physically performing source distribution, a complete
    machine-readable copy of the corresponding source code, to be
    distributed under the terms of Sections 1 and 2 above on a medium
    customarily used for software interchange; or,

    c) Accompany it with the information you received as to the offer
    to distribute corresponding source code.  (This alternative is
    allowed only for noncommercial distribution and only if you
    received the program in object code or executable form with such
    an offer, in accord with Subsection b above.)

The source code for a work means the preferred form of the work for
making modifications to it.  For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable.  However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.

If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.

  4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License.  Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.

  5. You are not required to accept this License, since you have not
signed it.  However, nothing else grants you permission to modify or
distribute the Program or its derivative works.  These actions are
prohibited by law if you do not accept this License.  Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.

  6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions.  You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.

  7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License.  If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all.  For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.

If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.

It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices.  Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.

This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.

  8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded.  In such case, this License incorporates
the limitation as if written in the body of this License.

  9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time.  Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.

Each version is given a distinguishing version number.  If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation.  If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.

  10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission.  For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this.  Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.

                            NO WARRANTY

  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.

  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.

                     END OF TERMS AND CONDITIONS

   </screen> 
  </sect1>
 </appendix>
</book>

 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:t
 sgml-parent-document: nil
 sgml-local-catalogs: nil
 sgml-namecase-general:t
 End:
 -->