-<!-- $Id: book.xml,v 1.15 2006-04-24 06:40:50 adam Exp $ -->
+<?xml version="1.0" standalone="no"?>
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
+ "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
+[
+ <!ENTITY copyright SYSTEM "copyright.xml">
+ <!ENTITY % local SYSTEM "local.ent">
+ %local;
+ <!ENTITY manref SYSTEM "manref.xml">
+ <!ENTITY gpl2 SYSTEM "gpl-2.0.xml">
+ <!ENTITY % idcommon SYSTEM "common/common.ent">
+ %idcommon;
+]>
+<book>
<bookinfo>
<title>Metaproxy - User's Guide and Reference</title>
- <author>
- <firstname>Mike</firstname><surname>Taylor</surname>
- </author>
- <author>
- <firstname>Adam</firstname><surname>Dickmeiss</surname>
- </author>
+ <authorgroup>
+ <author>
+ <firstname>Adam</firstname><surname>Dickmeiss</surname>
+ </author>
+ <author>
+ <firstname>Marc</firstname><surname>Cromme</surname>
+ </author>
+ <author>
+ <firstname>Mike</firstname><surname>Taylor</surname>
+ </author>
+ </authorgroup>
+ <releaseinfo>&version;</releaseinfo>
<copyright>
- <year>2006</year>
- <holder>Index Data ApS</holder>
+ <year>2005-2015</year>
+ <holder>Index Data</holder>
</copyright>
<abstract>
<simpara>
+ This manual is part of Metaproxy version &version;.
+ </simpara>
+ <simpara>
Metaproxy is a universal router, proxy and encapsulated
metasearcher for information retrieval protocols. It accepts,
processes, interprets and redirects requests from IR clients using
- standard protocols such as
+ standard protocols such as the binary
<ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
- (and in the future <ulink url="&url.sru;">SRU</ulink>
- and <ulink url="&url.srw;">SRW</ulink>), as
- well as functioning as a limited
- <ulink url="&url.http;">HTTP</ulink> server.
+ and the information search and retrieval
+ web service <ulink url="&url.sru;">SRU</ulink>
+ as well as functioning as a limited
+ <ulink url="&url.http;">HTTP</ulink> server.
+ </simpara>
+ <simpara>
Metaproxy is configured by an XML file which
specifies how the software should function in terms of routes that
the request packets can take through the proxy, each step on a
using the filter API.
</simpara>
<simpara>
- The terms under which Metaproxy will be distributed have yet to be
- established, but it will not necessarily be open source; so users
- should not at this stage redistribute the code without explicit
- written permission from the copyright holders, Index Data ApS.
+ Metaproxy is covered by the GNU General Public License version 2.
</simpara>
<simpara>
<inlinemediaobject>
<chapter id="introduction">
<title>Introduction</title>
-
-
+
<para>
<ulink url="&url.metaproxy;">Metaproxy</ulink>
- is a standalone program that acts as a universal router, proxy and
+ is a stand alone program that acts as a universal router, proxy and
encapsulated metasearcher for information retrieval protocols such
- as <ulink url="&url.z39.50;">Z39.50</ulink>, and in the future
- <ulink url="&url.sru;">SRU</ulink> and <ulink url="&url.srw;">SRW</ulink>.
+ as <ulink url="&url.z39.50;">Z39.50</ulink> and
+ <ulink url="&url.sru;">SRU</ulink>.
To clients, it acts as a server of these protocols: it can be searched,
- records can be retrieved from it, etc.
+ records can be retrieved from it, etc.
To servers, it acts as a client: it searches in them,
retrieves records from them, etc. it satisfies its clients'
requests by transforming them, multiplexing them, forwarding them
<screen>
Anything goes in!
Anything goes out!
- Cold bananas, fish, pyjamas,
+ Fish, bananas, cold pyjamas,
Mutton, beef and trout!
- attributed to Cole Porter.
</screen>
being more powerful, flexible, configurable and extensible. Among
its many advantages over the older, more pedestrian work are
support for multiplexing (encapsulated metasearching), routing by
- database name, authentication and authorisation and serving local
+ database name, authentication and authorization and serving local
files via HTTP. Equally significant, its modular architecture
facilitites the creation of pluggable modules implementing further
functionality.
</para>
<para>
- This manual will briefly describe Metaproxy's licensing situation
+ This manual will describe how to install Metaproxy
before giving an overview of its architecture, then discussing the
key concept of a filter in some depth and giving an overview of
the various filter types, then discussing the configuration file
</para>
</chapter>
-
-
- <chapter id="licence">
- <title>The Metaproxy Licence</title>
- <para>
- <emphasis role="strong">
- No decision has yet been made on the terms under which
- Metaproxy will be distributed.
- </emphasis>
- It is possible that, unlike
- other Index Data products, metaproxy may not be released under a
- free-software licence such as the GNU GPL. Until a decision is
- made and a public statement made, then, and unless it has been
- delivered to you other specific terms, please treat Metaproxy as
- though it were proprietary software.
- The code should not be redistributed without explicit
- written permission from the copyright holders, Index Data ApS.
- </para>
- </chapter>
-
<chapter id="installation">
<title>Installation</title>
<para>
- Metaproxy depends on the folloing tools/libraries:
+ Metaproxy depends on the following tools/libraries:
<variablelist>
<varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
<listitem>
</varlistentry>
<varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
<listitem>
- <para>This is an XSLT processor - based on
+ <para>This is an XSLT processor - based on
<ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
- Libxslt must be installed with the development components.
+ Libxslt must be installed with the development components
+ (header files, etc.) as well as the run-time libraries.
</para>
</listitem>
</varlistentry>
<varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
<listitem>
<para>
- The popular C++ library.
+ The popular C++ library. Initial versions of Metaproxy
+ was built with 1.32 but this is no longer supported.
+ Metaproxy is known to work with Boost version 1.33 through 1.55.
</para>
</listitem>
</varlistentry>
for more information.
</para>
<para>
- We have succesfully used Metaproxy with Boost using the compilers
- <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
- <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
+ We have successfully built Metaproxy using the compilers
+ <ulink url="&url.gcc;">GCC</ulink> and
+ <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink>.
</para>
+ <para>
+ As an option, Metaproxy may also be compiled with
+ <ulink url="&url.usemarcon;">USEMARCON</ulink> support which allows for
+ MARC conversions for the <xref linkend="ref-record_transform"/> filter.
+ </para>
<section id="installation.unix">
<title>Installation on Unix (from Source)</title>
<para>
tools binary packages. If, for example, Libxml2/libxslt are already
installed as development packages use those (and omit compilation).
</para>
-
+
+ <note>
+ <para>
+ <ulink url="&url.usemarcon;">USEMARCON</ulink> is not available
+ as a package at the moment, so Metaproxy must be built from source
+ if that is to be used.
+ </para>
+ </note>
+
+ <section id="libxml2.fromsource">
+ <title>Libxml2/libxslt</title>
+ <para>
+ Libxml2/libxslt:
+ </para>
+ <screen>
+ gunzip -c libxml2-version.tar.gz|tar xf -
+ cd libxml2-version
+ ./configure
+ make
+ su
+ make install
+ </screen>
+ <screen>
+ gunzip -c libxslt-version.tar.gz|tar xf -
+ cd libxslt-version
+ ./configure
+ make
+ su
+ make install
+ </screen>
+ </section>
+ <section id="usemarcon">
+ <title>USEMARCON (optional)</title>
+ <screen>
+ gunzip -c usemarcon317.tar.gz|tar xf -
+ cd usemarcon317
+ ./configure
+ make
+ su
+ make install
+ </screen>
+ </section>
+
+ <section id="yaz.fromsource">
+ <title>YAZ/YAZ++</title>
+ <screen>
+ gunzip -c yaz-version.tar.gz|tar xf -
+ cd yaz-version
+ ./configure
+ make
+ su
+ make install
+ </screen>
+ <screen>
+ gunzip -c yazpp-version.tar.gz|tar xf -
+ cd yazpp-version
+ ./configure
+ make
+ su
+ make install
+ </screen>
+ </section>
+ <section>
+ <title id="boost.fromsource">Boost</title>
+ <para>
+ Metaproxy needs components thread and test from
+ Boost.
+ </para>
+ <screen>
+ gunzip -c boost-version.tar.gz|tar xf -
+ cd boost-version
+ ./configure --with-libraries=thread,test,regex --with-toolset=gcc
+ make
+ su
+ make install
+ </screen>
+ <para>
+ However, under the hood bjam is used. You can invoke that with
+ </para>
+ <screen>
+ ./bjam --toolset=gcc --with-thread --with-test --with-regex stage
+ </screen>
+ <para>
+ Replace <literal>stage</literal> with <literal>clean</literal> /
+ <literal>install</literal> to perform clean and install respectively.
+ </para>
+ <para>
+ Add <literal>--prefix=DIR</literal> to install Boost in other
+ prefix than <literal>/usr/local</literal>.
+ </para>
+ </section>
+ <section id="metaproxy.fromsource">
+ <title>Metaproxy</title>
+ <screen>
+ gunzip -c metaproxy-version.tar.gz|tar xf -
+ cd metaproxy-version
+ ./configure
+ make
+ su
+ make install
+ </screen>
+ <para>
+ You may have to tell configure where Boost is installed by supplying
+ options <literal>--with-boost</literal> and <literal>--with-boost-toolset</literal>.
+ The former sets the PREFIX for Boost (same as --prefix for Boost above).
+ The latter the compiler toolset (eg. gcc34).
+ </para>
+ <para>
+ Pass <literal>--help</literal> to configure to get a list of
+ available options.
+ </para>
+ </section>
+ </section>
+
+ <section id="installation.debian">
+ <title>Installation on Debian GNU/Linux</title>
<para>
- Libxml2/libxslt:
+ All dependencies for Metaproxy are available as
+ <ulink url="&url.debian;">Debian</ulink> packages.
</para>
- <screen>
- gunzip -c libxml2-version.tar.gz|tar xf -
- cd libxml2-version
- ./configure
- make
- su
- make install
- </screen>
- <screen>
- gunzip -c libxslt-version.tar.gz|tar xf -
- cd libxslt-version
- ./configure
- make
- su
- make install
- </screen>
<para>
- YAZ/YAZ++:
+ The procedures for Debian based systems, such as
+ <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
</para>
- <screen>
- gunzip -c yaz-version.tar.gz|tar xf -
- cd yaz-version
- ./configure
- make
- su
- make install
- </screen>
- <screen>
- gunzip -c yazpp-version.tar.gz|tar xf -
- cd yazpp-version
- ./configure
- make
- su
- make install
- </screen>
<para>
- Boost:
+ There is currently no official Debian package for YAZ++.
+ And the official Debian package for YAZ is probably too old.
+ But Index Data builds "new" versions of those for Debian (i386, amd64 only).
</para>
- <screen>
- gunzip -c boost-version.tar.gz|tar xf -
- cd boost-version
- ./configure
- make
- su
- make install
- </screen>
<para>
- Metaproxy:
+ Update the <filename>/etc/apt/sources.list</filename>
+ to include the Index Data repository.
+ See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
+ for more information.
</para>
<screen>
- gunzip -c metaproxy-version.tar.gz|tar xf -
- cd metaproxy-version
- ./configure
- make
- su
- make install
+ apt-get install libxslt1-dev
+ apt-get install libyazpp6-dev
+ apt-get install libboost-dev
+ apt-get install libboost-system-dev
+ apt-get install libboost-thread-dev
+ apt-get install libboost-test-dev
+ apt-get install libboost-regex-dev
</screen>
+ <para>
+ With these packages installed, the usual configure + make
+ procedure can be used for Metaproxy as outlined in
+ <xref linkend="installation.unix"/>.
+ </para>
</section>
- <section id="installation.debian">
- <title>Installation on Debian</title>
+ <section id="installation.rpm">
+ <title>Installation on RPM based Linux Systems</title>
<para>
- ### To be written
+ All external dependencies for Metaproxy are available as
+ RPM packages, either from your distribution site, or from the
+ <ulink url="http://fr.rpmfind.net/">RPMfind</ulink> site.
</para>
- </section>
+ <para>
+ For example, an installation of the requires Boost C++ development
+ libraries on RedHat Fedora C4 and C5 can be done like this:
+ <screen>
+ wget ftp://fr.rpmfind.net/wlinux/fedora/core/updates/testing/4/SRPMS/boost-1.33.0-3.fc4.src.rpm
+ sudo rpmbuild --buildroot src/ --rebuild -p fc4/boost-1.33.0-3.fc4.src.rpm
+ sudo rpm -U /usr/src/redhat/RPMS/i386/boost-*rpm
+ </screen>
+ </para>
+ <para>
+ The <ulink url="&url.yaz;">YAZ</ulink> library is needed to
+ compile &metaproxy;, see there
+ for more information on available RPM packages.
+ </para>
+ <para>
+ There is currently no official RPM package for YAZ++.
+ See the <ulink url="&url.yazplusplus;">YAZ++</ulink> pages
+ for more information on a Unix tarball install.
+ </para>
+ <para>
+ With these packages installed, the usual configure + make
+ procedure can be used for Metaproxy as outlined in
+ <xref linkend="installation.unix"/>.
+ </para>
+ </section>
<section id="installation.windows">
<title>Installation on Windows</title>
<para>
- ### To be written
+ Metaproxy has been tested Microsoft
+ <ulink url="&url.vstudio;">Visual Studio</ulink>.
+ 2013 (C 12.0).
</para>
+ <section id="installation.windows.boost">
+ <title>Boost</title>
+ <para>
+ For Windows, it's easiest to get the precompiled Boost
+ package from <ulink url="&url.boost.windows.download;">here</ulink>.
+ Several versions of the Boost libraries may be selected when
+ installing Boost for windows. Please choose at least the
+ <emphasis>multithreaded</emphasis> (non-DLL) version because
+ the Metaproxy makefile uses that.
+ </para>
+ <para>
+ For more information about installing Boost refer to the
+ <ulink url="&url.boost.getting.started;">getting started</ulink>
+ pages.
+ </para>
+ </section>
+
+ <section id="installation.windows.libxslt">
+ <title>Libxslt</title>
+ <para>
+ <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
+ for Windows from
+ <ulink url="&url.libxml2.download.windows;">here</ulink>.
+ </para>
+ <para>
+ Libxslt also requires libxml2 to operate.
+ </para>
+ </section>
+
+ <section id="installation.windows.yaz">
+ <title>YAZ</title>
+ <para>
+ <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
+ for Windows from
+ <ulink url="&url.yaz.download.win32;">here</ulink>.
+ </para>
</section>
+
+ <section id="installation.windows.yazplusplus">
+ <title>YAZ++</title>
+ <para>
+ Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
+ Version 1.6.0 or later is required.
+ </para>
+ <para>
+ YAZ++ includes NMAKE makefiles, similar to those found in the
+ YAZ package.
+ </para>
+ </section>
+
+ <section id="installation.windows.metaproxy">
+ <title>Metaproxy</title>
+ <para>
+ Metaproxy is shipped with NMAKE makefiles as well - similar
+ to those found in the YAZ++/YAZ packages. Adjust this Makefile
+ to point to the proper locations of Boost, Libxslt, Libxml2,
+ zlib, iconv, yaz and yazpp.
+ </para>
+
+ <variablelist>
+ <varlistentry><term><literal>DEBUG</literal></term>
+ <listitem><para>
+ If set to 1, the software is
+ compiled with debugging libraries (code generation is
+ multi-threaded debug DLL).
+ If set to 0, the software is compiled with release libraries
+ (code generation is multi-threaded DLL).
+ </para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>BOOST</literal></term>
+ <listitem>
+ <para>
+ Boost install location
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>BOOST_VERSION</literal></term>
+ <listitem>
+ <para>
+ Boost version (replace . with _).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>BOOST_TOOLSET</literal></term>
+ <listitem>
+ <para>
+ Boost toolset.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>LIBXSLT_DIR</literal>,
+ <literal>LIBXML2_DIR</literal> ..</term>
+ <listitem>
+ <para>
+ Specify the locations of Libxslt, libiconv, libxml2 and
+ libxslt.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ <para>
+ After successful compilation you'll find
+ <literal>metaproxy.exe</literal> in the
+ <literal>bin</literal> directory.
+ </para>
+ </section>
+
+
+ </section>
</chapter>
-
+
+<chapter id="yazproxy-comparison">
+ <title>YAZ Proxy Comparison</title>
+ <para>
+ The table below lists facilities either supported by either
+ <ulink url="&url.yazproxy;">YAZ Proxy</ulink> or Metaproxy.
+ </para>
+<table id="yazproxy-comparison-table">
+ <title>Metaproxy / YAZ Proxy comparison</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Facility</entry>
+ <entry>Metaproxy</entry>
+ <entry>YAZ Proxy</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Z39.50 server</entry>
+ <entry>Using filter <xref linkend="ref-frontend_net"/></entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>SRU server</entry>
+ <entry>Supported with filter <xref linkend="ref-sru_z3950"/></entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>Z39.50 client</entry>
+ <entry>Supported with filter <xref linkend="ref-z3950_client"/></entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>SRU client</entry>
+ <entry>Supported with filter <xref linkend="ref-zoom"/></entry>
+ <entry>Unsupported</entry>
+ </row>
+ <row>
+ <entry>Connection reuse</entry>
+ <entry>Supported with filter <literal>session_shared</literal></entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>Connection share</entry>
+ <entry>Supported with filter <literal>session_shared</literal></entry>
+ <entry>Unsupported</entry>
+ </row>
+ <row>
+ <entry>Result set reuse</entry>
+ <entry>Supported with filter <literal>session_shared</literal></entry>
+ <entry>Within one Z39.50 session / HTTP keep-alive</entry>
+ </row>
+ <row>
+ <entry>Record cache</entry>
+ <entry>Supported by filter <literal>session_shared</literal></entry>
+ <entry>Supported for last result set within one Z39.50/HTTP-keep alive session</entry>
+ </row>
+ <row>
+ <entry>Z39.50 Virtual database, i.e. select any Z39.50 target for database</entry>
+ <entry>Supported with filter <literal>virt_db</literal></entry>
+ <entry>Unsupported</entry>
+ </row>
+ <row>
+ <entry>SRU Virtual database, i.e. select any Z39.50 target for path</entry>
+ <entry>Supported with filter <literal>virt_db</literal>,
+ <literal>sru_z3950</literal></entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>Multi target search</entry>
+ <entry>Supported with filter <literal>multi</literal> (round-robin)</entry>
+ <entry>Unsupported</entry>
+ </row>
+ <row>
+ <entry>Retrieval and search limits</entry>
+ <entry>Supported using filter <literal>limit</literal></entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>Bandwidth limits</entry>
+ <entry>Supported using filter <literal>limit</literal></entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>Connect limits</entry>
+ <entry>Supported by filter <literal>frontend_net</literal> (connect-max)</entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>Retrieval sanity check and conversions</entry>
+ <entry>Supported using filter <literal>record_transform</literal></entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>Query check</entry>
+ <entry>
+ Supported by <literal>query_rewrite</literal> which may be check
+ a query and throw diagnostics (errors)
+ </entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>Query rewrite</entry>
+ <entry>Supported with <literal>query_rewrite</literal></entry>
+ <entry>Unsupported</entry>
+ </row>
+ <row>
+ <entry>Session invalidate for -1 hits</entry>
+ <entry>Unsupported</entry>
+ <entry>Supported</entry>
+ </row>
+ <row>
+ <entry>Architecture</entry>
+ <entry>Multi-threaded + select for networked modules such as
+ <literal>frontend_net</literal>)</entry>
+ <entry>Single-threaded using select</entry>
+ </row>
+
+ <row>
+ <entry>Extensability</entry>
+ <entry>Most functionality implemented as loadable modules</entry>
+ <entry>Unsupported and experimental</entry>
+ </row>
+
+ <row>
+ <entry><ulink url="&url.usemarcon;">USEMARCON</ulink></entry>
+ <entry>Supported with <literal>record_transform</literal></entry>
+ <entry>Supported</entry>
+ </row>
+
+ <row>
+ <entry>Portability</entry>
+ <entry>
+ Requires YAZ, YAZ++ and modern C++ compiler supporting
+ <ulink url="&url.boost;">Boost</ulink>.
+ </entry>
+ <entry>
+ Requires YAZ and YAZ++.
+ STL is not required so pretty much any C++ compiler out there should work.
+ </entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+</table>
+</chapter>
+
<chapter id="architecture">
<title>The Metaproxy Architecture</title>
<para>
<para>
In general, packages are doctored as they pass through
Metaproxy. For example, when the proxy performs authentication
- and authorisation on a Z39.50 Init request, it removes the
+ and authorization on a Z39.50 Init request, it removes the
authentication credentials from the package so that they are not
passed onto the back-end server; and when search-response
packages are obtained from multiple servers, they are merged
The word ``filter'' is sometimes used rather loosely, in two
different ways: it may be used to mean a particular
<emphasis>type</emphasis> of filter, as when we speak of ``the
- auth_simplefilter'' or ``the multi filter''; or it may be used
+ auth_simple filter'' or ``the multi filter''; or it may be used
to be a specific <emphasis>instance</emphasis> of a filter
within a Metaproxy configuration. For example, a single
configuration will often contain multiple instances of the
plugins that provide new filters. The filter API is small and
conceptually simple, but there are many details to master. See
the section below on
- <link linkend="extensions">extensions</link>.
+ <link linkend="filters">Filters</link>.
</para>
</listitem>
</varlistentry>
<chapter id="filters">
<title>Filters</title>
-
-
- <section>
+
+
+ <section id="filters-introductory-notes">
<title>Introductory notes</title>
<para>
It's useful to think of Metaproxy as an interpreter providing a small
as part of Metaproxy, and others may be provided by third parties
and dynamically loaded. They all conform to the same simple API
of essentially two methods: <function>configure()</function> is
- called at startup time, and is passed a DOM tree representing that
+ called at startup time, and is passed an XML DOM tree representing that
part of the configuration file that pertains to this filter
instance: it is expected to walk that tree extracting relevant
information; and <function>process()</function> is called every
packages
(<literal>frontend_net</literal>);
others are sinks: they consume packages and return a result
- (<literal>z3950_client</literal>,
- <literal>backend_test</literal>,
- <literal>http_file</literal>);
+ (<literal>backend_test</literal>,
+ <literal>bounce</literal>,
+ <literal>http_file</literal>,
+ <literal>z3950_client</literal>);
the others are true filters, that read, process and pass on the
packages they are fed
(<literal>auth_simple</literal>,
<literal>log</literal>,
<literal>multi</literal>,
<literal>query_rewrite</literal>,
+ <literal>record_transform</literal>,
<literal>session_shared</literal>,
+ <literal>sru_z3950</literal>,
<literal>template</literal>,
<literal>virt_db</literal>).
</para>
</section>
-
-
+
+
<section id="overview.filter.types">
<title>Overview of filter types</title>
<para>
We now briefly consider each of the types of filter supported by
the core Metaproxy binary. This overview is intended to give a
- flavour of the available functionality; more detailed information
+ flavor of the available functionality; more detailed information
about each type of filter is included below in
- <link linkend="filterref"
- >the reference guide to Metaproxy filters</link>.
+ <xref linkend="reference"/>.
</para>
<para>
The filters are here named by the string that is used as the
<para>
The filters are here listed in alphabetical order:
</para>
-
- <section>
+
+<!--
+
+### New filters:
+
+New virt_db-alike that does inteligent peer choice, explain merging,
+adds FD&N to explain. Keeps init responses (like "virt_db Classic"),
+makes routing choices based on local explain knowledge. Ref IDDI
+paper.
+
+Filter to convert Explain Classic to ZeeRex.
+
+CQL2PQF (which needs augmented ZeeRex) - MARC for Talis.
+
+SRU2Z39.50 (ditto).
+
+Figure out what additional information we need in:
+ ZeeRex (check against D3.1)
+ Init request (e.g. loop detection)
+ Query package (e.g. number of hops)
+ Query response (e.g. record source)
+
+-->
+
+ <section id="auth_simple">
<title><literal>auth_simple</literal>
(mp::filter::AuthSimple)</title>
<para>
- Simple authentication and authorisation. The configuration
+ Simple authentication and authorization. The configuration
specifies the name of a file that is the user register, which
lists <varname>username</varname>:<varname>password</varname>
pairs, one per line, colon separated. When a session begins, it
the user.
</para>
</section>
-
- <section>
+
+ <section id="backend_test">
<title><literal>backend_test</literal>
(mp::filter::Backend_test)</title>
<para>
- A sink that provides dummy responses in the manner of the
+ A partial sink that provides dummy responses in the manner of the
<literal>yaz-ztest</literal> Z39.50 server. This is useful only
for testing. Seriously, you don't need this. Pretend you didn't
even read this section.
</para>
</section>
-
- <section>
+
+ <section id="bounce">
+ <title><literal>bounce</literal>
+ (mp::filter::Bounce)</title>
+ <para>
+ A sink that swallows <emphasis>all packages</emphasis>,
+ and returns them almost unprocessed.
+ It never sends any package of any type further down the row, but
+ sets Z39.50 packages to Z_Close, and HTTP_Request packages to
+ HTTP_Response err code 400 packages, and adds a suitable bounce
+ message.
+ The bounce filter is usually added at end of each filter chain route
+ to prevent infinite hanging of for example HTTP
+ requests packages when only the Z39.50 client partial sink
+ filter is found in the
+ route.
+ </para>
+ </section>
+
+ <section id="cql_rpn">
+ <title><literal>cql_rpn</literal>
+ (mp::filter::CQLtoRPN)</title>
+ <para>
+ A query language transforming filter which catches Z39.50
+ <literal>searchRequest</literal>
+ packages containing <literal>CQL</literal> queries, transforms
+ those to <literal>RPN</literal> queries,
+ and sends the <literal>searchRequests</literal> on to the next
+ filters. It is among other things useful in a SRU context.
+ </para>
+ </section>
+
+ <section id="frontend_net">
<title><literal>frontend_net</literal>
(mp::filter::FrontendNet)</title>
<para>
A source that accepts Z39.50 connections from a port
specified in the configuration, reads protocol units, and
feeds them into the next filter in the route. When the result is
- revceived, it is returned to the original origin.
+ received, it is returned to the original origin.
</para>
</section>
- <section>
+ <section id="http_file">
<title><literal>http_file</literal>
(mp::filter::HttpFile)</title>
<para>
- A sink that returns the contents of files from the local
- filesystem in response to HTTP requests. (Yes, Virginia, this
+ A partial sink which swallows only
+ <literal>HTTP_Request</literal> packages, and
+ returns the contents of files from the local
+ filesystem in response to HTTP requests.
+ It lets Z39.50 packages and all other forthcoming package types
+ pass untouched.
+ (Yes, Virginia, this
does mean that Metaproxy is also a Web-server in its spare time. So
far it does not contain either an email-reader or a Lisp
interpreter, but that day is surely coming.)
</para>
</section>
-
- <section>
+
+ <section id="load_balance">
+ <title><literal>load_balance</literal>
+ (mp::filter::LoadBalance)</title>
+ <para>
+ Performs load balancing for incoming Z39.50 init requests.
+ It is used together with the <literal>virt_db</literal> filter,
+ but unlike the <literal>multi</literal> filter it does send an
+ entire session to only one of the virtual backends. The
+ <literal>load_balance</literal> filter is assuming that
+ all backend targets have equal content, and chooses the backend
+ with least load cost for a new session.
+ <warning>
+ <para>
+ This filter is experimental and yet not mature for heavy load
+ production sites.
+ </para>
+ </warning>
+ </para>
+ </section>
+
+ <section id="log">
<title><literal>log</literal>
(mp::filter::Log)</title>
<para>
Writes logging information to standard output, and passes on
- the package unchanged.
+ the package unchanged. A log file name can be specified, as well
+ as multiple different logging formats.
</para>
</section>
-
- <section>
+
+ <section id="multi">
<title><literal>multi</literal>
(mp::filter::Multi)</title>
<para>
- Performs multicast searching.
+ Performs multi-database searching.
See
<link linkend="multidb">the extended discussion</link>
of virtual databases and multi-database searching below.
</para>
</section>
-
- <section>
+
+ <section id="query_rewrite">
<title><literal>query_rewrite</literal>
(mp::filter::QueryRewrite)</title>
<para>
- Rewrites Z39.50 Type-1 and Type-101 (``RPN'') queries by a
+ Rewrites Z39.50 <literal>Type-1</literal>
+ and <literal>Type-101</literal> (``<literal>RPN</literal>'')
+ queries by a
three-step process: the query is transliterated from Z39.50
packet structures into an XML representation; that XML
representation is transformed by an XSLT stylesheet; and the
structure.
</para>
</section>
-
- <section>
+
+
+ <section id="record_transform">
+ <title><literal>record_transform</literal>
+ (mp::filter::RecordTransform)</title>
+ <para>
+ This filter acts only on Z3950 present requests, and let all
+ other types of packages and requests pass untouched. It's use is
+ twofold: blocking Z3950 present requests, which the backend
+ server does not understand and can not honor, and transforming
+ the present syntax and elementset name according to the rules
+ specified, to fetch only existing record formats, and transform
+ them on the fly to requested record syntaxes.
+ </para>
+ </section>
+
+ <section id="session_shared">
<title><literal>session_shared</literal>
(mp::filter::SessionShared)</title>
<para>
- When this is finished, it will implement global sharing of
+ This filter implements global sharing of
result sets (i.e. between threads and therefore between
- clients), yielding performance improvements especially when
- incoming requests are from a stateless environment such as a
- web-server, in which the client process representing a session
- might be any one of many. However:
+ clients), yielding performance improvements by clever resource
+ pooling.
</para>
- <warning>
- <para>
- This filter is not yet completed.
- </para>
- </warning>
</section>
-
- <section>
+
+ <section id="sru_z3950">
+ <title><literal>sru_z3950</literal>
+ (mp::filter::SRUtoZ3950)</title>
+ <para>
+ This filter transforms valid
+ SRU GET/POST/SOAP searchRetrieve requests to Z3950 init, search,
+ and present requests, and wraps the
+ received hit counts and XML records into suitable SRU response
+ messages.
+ The <literal>sru_z3950</literal> filter processes also SRU
+ GET/POST/SOAP explain requests, returning
+ either the absolute minimum required by the standard, or a full
+ pre-defined ZeeReX explain record.
+ See the
+ <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
+ standard pages and the
+ <ulink url="&url.sru.explain;">SRU Explain</ulink> pages
+ for more information on the correct explain syntax.
+ SRU scan requests are not supported yet.
+ </para>
+ </section>
+
+ <section id="template">
<title><literal>template</literal>
(mp::filter::Template)</title>
<para>
intended for civilians.
</para>
</section>
-
- <section>
+
+ <section id="virt_db">
<title><literal>virt_db</literal>
- (mp::filter::Virt_db)</title>
+ (mp::filter::VirtualDB)</title>
<para>
Performs virtual database selection: based on the name of the
database in the search request, a server is selected, and its
of virtual databases and multi-database searching below.
</para>
</section>
-
- <section>
+
+ <section id="z3950_client">
<title><literal>z3950_client</literal>
(mp::filter::Z3950Client)</title>
<para>
- Performs Z39.50 searching and retrieval by proxying the
+ A partial sink which swallows only Z39.50 packages.
+ It performs Z39.50 searching and retrieval by proxying the
packages that are passed to it. Init requests are sent to the
address specified in the <literal>VAL_PROXY</literal> otherInfo
attached to the request: this may have been specified by client,
or generated by a <literal>virt_db</literal> filter earlier in
the route. Subsequent requests are sent to the same address,
which is remembered at Init time in a Session object.
+ HTTP_Request packages and all other forthcoming package types
+ are passed untouched.
</para>
</section>
+
+
+ <section id="zeerex_explain">
+ <title><literal>zeerex_explain</literal>
+ (mp::filter::ZeerexExplain)</title>
+ <para>
+ This filter acts as a sink for
+ Z39.50 explain requests, returning a static ZeeReX
+ Explain XML record from the config section. All other packages
+ are passed through.
+ See the
+ <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
+ standard pages
+ for more information on the correct explain syntax.
+ </para>
+ <warning>
+ <para>
+ This filter is not yet completed.
+ </para>
+ </warning>
+ </section>
+
+
</section>
-
-
+
+
<section id="future.directions">
<title>Future directions</title>
<para>
</listitem>
</varlistentry>
<varlistentry>
- <term><literal>frontend_sru</literal> (source)</term>
- <listitem>
- <para>
- Receive SRU (and perhaps SRW) requests.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>sru2z3950</literal> (filter)</term>
- <listitem>
- <para>
- Translate SRU requests into Z39.50 requests.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
<term><literal>sru_client</literal> (sink)</term>
<listitem>
<para>
- SRU searching and retrieval.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>srw_client</literal> (sink)</term>
- <listitem>
- <para>
- SRW searching and retrieval.
+ SRU/GET and SRU/SOAP searching and retrieval.
</para>
</listitem>
</varlistentry>
</variablelist>
</section>
</chapter>
-
-
-
+
+
+
<chapter id="configuration">
<title>Configuration: the Metaproxy configuration file format</title>
-
-
- <section>
+
+
+ <section id="configuration-introductory-notes">
<title>Introductory notes</title>
<para>
If Metaproxy is an interpreter providing operations on packages, then
its configuration file can be thought of as a program for that
- interpreter. Configuration is by means of a single file, the name
+ interpreter. Configuration is by means of a single XML file, the name
of which is supplied as the sole command-line argument to the
<command>metaproxy</command> program. (See
- <link linkend="progref">the reference guide</link>
- below for more information on invoking Metaproxy.)
- </para>
- <para>
- The configuration files are written in XML. (But that's just an
- implementation detail - they could just as well have been written
- in YAML or Lisp-like S-expressions, or in a custom syntax.)
- </para>
- <para>
- Since XML has been chosen, an XML schema,
- <filename>config.xsd</filename>, is provided for validating
- configuration files. This file is supplied in the
- <filename>etc</filename> directory of the Metaproxy distribution. It
- can be used by (among other tools) the <command>xmllint</command>
- program supplied as part of the <literal>libxml2</literal>
- distribution:
- </para>
- <screen>
- xmllint --noout --schema etc/config.xsd my-config-file.xml
- </screen>
- <para>
- (A recent version of <literal>libxml2</literal> is required, as
- support for XML Schemas is a relatively recent addition.)
+ <xref linkend="reference"/> below for more information on invoking
+ Metaproxy.)
</para>
</section>
-
+
<section id="overview.xml.structure">
- <title>Overview of XML structure</title>
+ <title>Overview of the config file XML structure</title>
<para>
All elements and attributes are in the namespace
- <ulink url="http://indexdata.dk/yp2/config/1"/>.
+ <ulink url="http://indexdata.com/metaproxy"/>.
This is most easily achieved by setting the default namespace on
the top-level element, as here:
</para>
<screen>
- <yp2 xmlns="http://indexdata.dk/yp2/config/1">
+ <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
</screen>
<para>
- The top-level element is <yp2>. This contains a
- <start> element, a <filters> element and a
- <routes> element, in that order. <filters> is
- optional; the other two are mandatory. All three are
- non-repeatable.
+ The top-level element is <metaproxy>. This contains
+ a <dlpath> element,
+ a <start> element,
+ a <filters> element and
+ a <routes> element, in that order. <dlpath> and
+ <filters> are optional; the other two are mandatory.
+ All four are non-repeatable.
+ </para>
+ <para>
+ The <dlpath;> element contains a text element which
+ specifies the location of filter modules. This is only needed
+ if Metaproxy must load 3rd party filters (most filters with Metaproxy
+ are built into the Metaproxy application).
</para>
<para>
The <start> element is empty, but carries a
and contain various elements that provide suitable configuration
for a filter of its type. The filter-specific elements are
described in
- <link linkend="filterref">the reference guide below</link>.
+ <xref linkend="reference"/>.
Filters defined in this part of the file must carry an
<literal>id</literal> attribute so that they can be referenced
from elsewhere.
<para>
The following is a small, but complete, Metaproxy configuration
file (included in the distribution as
- <literal>metaproxy/etc/config0.xml</literal>).
+ <literal>metaproxy/etc/config1.xml</literal>).
This file defines a very simple configuration that simply proxies
- to whatever backend server the client requests, but logs each
+ to whatever back-end server the client requests, but logs each
request and response. This can be useful for debugging complex
client-server dialogues.
</para>
- <screen><![CDATA[
-<?xml version="1.0"?>
-<yp2 xmlns="http://indexdata.dk/yp2/config/1">
+ <screen><![CDATA[<?xml version="1.0"?>
+<metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
+ <dlpath>/usr/lib/metaproxy/modules</dlpath>
<start route="start"/>
<filters>
<filter id="frontend" type="frontend_net">
<filter id="backend" type="z3950_client">
</filter>
</filters>
- <routes>
+ <routes>
<route id="start">
<filter refid="frontend"/>
<filter type="log"/>
<filter refid="backend"/>
+ <filter type="bounce"/>
</route>
</routes>
-</yp2>
+</metaproxy>
]]></screen>
<para>
It works by defining a single route, called
- <literal>start</literal>, which consists of a sequence of three
+ <literal>start</literal>, which consists of a sequence of four
filters. The first and last of these are included by reference:
their <literal><filter></literal> elements have
<literal>refid</literal> attributes that refer to filters defined
middle filter is included inline in the route.
</para>
<para>
- The three filters in the route are as follows: first, a
+ The four filters in the route are as follows: first, a
<literal>frontend_net</literal> filter accepts Z39.50 requests
from any host on port 9000; then these requests are passed through
a <literal>log</literal> filter that emits a message for each
request; they are then fed into a <literal>z3950_client</literal>
- filter, which forwards the requests to the client-specified
- backend Z39.509 server. When the response arrives, it is handed
+ filter, which forwards all Z39.50 requests to the client-specified
+ back-end Z39.509 server. Those Z39.50 packages are returned by the
+ <literal>z3950_client</literal> filter, with the response data
+ filled by the external Z39.50 server targeted.
+ All non-Z39.50 packages are passed through to the
+ <literal>bounce</literal> filter, which definitely bounces
+ everything, including fish, bananas, cold pyjamas,
+ mutton, beef and trout packages.
+ When the response arrives, it is handed
back to the <literal>log</literal> filter, which emits another
- message; and then to the front-end filter, which returns the
- response to the client.
+ message; and then to the <literal>frontend_net</literal> filter,
+ which returns the response to the client.
</para>
</section>
+
+ <section id="config-file-modularity">
+ <title>Config file modularity</title>
+ <para>
+ Metaproxy XML configuration snippets can be reused by other
+ filters using the <literal>XInclude</literal> standard, as seen in
+ the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU
+ configuration.
+ <screen><![CDATA[
+ <filter id="sru" type="sru_z3950">
+ <database name="Default">
+ <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
+ href="explain.xml"/>
+ </database>
+ </filter>
+]]></screen>
+ </para>
+ </section>
+
+ <section id="config-file-syntax-check">
+ <title>Config file syntax checking</title>
+ <para>
+ The distribution contains RelaxNG Compact and XML syntax checking
+ files, as well as XML Schema files. These are found in the
+ distribution paths
+ <screen>
+ xml/schema/metaproxy.rnc
+ xml/schema/metaproxy.rng
+ xml/schema/metaproxy.xsd
+ </screen>
+ and can be used to verify or debug the XML structure of
+ configuration files. For example, using the utility
+ <filename>xmllint</filename>, syntax checking is done like this:
+ <screen>
+ xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml
+ xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml
+ </screen>
+ (A recent version of <literal>libxml2</literal> is required, as
+ support for XML Schemas is a relatively recent addition.)
+ </para>
+ <para>
+ You can of course use any other RelaxNG or XML Schema compliant tool
+ you wish.
+ </para>
+ </section>
</chapter>
<title>Virtual databases and multi-database searching</title>
- <section>
+ <section id="multidb-introductory-notes">
<title>Introductory notes</title>
+ <para>
+ Two of Metaproxy's filters are concerned with multiple-database
+ operations. Of these, <literal>virt_db</literal> can work alone
+ to control the routing of searches to one of a number of servers,
+ while <literal>multi</literal> can work together with
+ <literal>virt_db</literal> to perform multi-database searching, merging
+ the results into a unified result-set - ``metasearch in a box''.
+ </para>
+ <para>
+ The interaction between
+ these two filters is necessarily complex: it reflects the real,
+ irreducible complexity of multi-database searching in a protocol such
+ as Z39.50 that separates initialization from searching, and in
+ which the database to be searched is not known at initialization
+ time.
+ </para>
+ <para>
+ It's possible to use these filters without understanding the
+ details of their functioning and the interaction between them; the
+ next two sections of this chapter are ``HOW-TO'' guides for doing
+ just that. However, debugging complex configurations will require
+ a deeper understanding, which the last two sections of this
+ chapters attempt to provide.
+ </para>
+ </section>
+
+
+ <section id="multidb.virt_db">
+ <title>Virtual databases with the <literal>virt_db</literal> filter</title>
+ <para>
+ Working alone, the purpose of the
+ <literal>virt_db</literal>
+ filter is to route search requests to one of a selection of
+ back-end databases. In this way, a single Z39.50 endpoint
+ (running Metaproxy) can provide access to several different
+ underlying services, including those that would otherwise be
+ inaccessible due to firewalls. In many useful configurations, the
+ back-end databases are local to the Metaproxy installation, but
+ the software does not enforce this, and any valid Z39.50 servers
+ may be used as back-ends.
+ </para>
+ <para>
+ For example, a <literal>virt_db</literal>
+ filter could be set up so that searches in the virtual database
+ ``lc'' are forwarded to the Library of Congress bibliographic
+ catalogue server, and searches in the virtual database ``marc''
+ are forwarded to the toy database of MARC records that Index Data
+ hosts for testing purposes. A <literal>virt_db</literal>
+ configuration to make this switch would look like this:
+ </para>
+ <screen><![CDATA[<filter type="virt_db">
+ <virtual>
+ <database>lc</database>
+ <target>lx2.loc.gov:210/LCDB</target>
+ </virtual>
+ <virtual>
+ <database>marc</database>
+ <target>indexdata.com/marc</target>
+ </virtual>
+</filter>]]></screen>
+ <para>
+ As well as being useful in it own right, this filter also provides
+ the foundation for multi-database searching.
+ </para>
+ </section>
+
+
+ <section id="multidb.multi">
+ <title>Multi-database search with the <literal>multi</literal> filter</title>
+ <para>
+ To arrange for Metaproxy to broadcast searches to multiple back-end
+ servers, the configuration needs to include two components: a
+ <literal>virt_db</literal>
+ filter that specifies multiple
+ <literal><target></literal>
+ elements, and a subsequent
+ <literal>multi</literal>
+ filter. Here, for example, is a complete configuration that
+ broadcasts searches to both the Library of Congress catalogue and
+ Index Data's tiny testing database of MARC records:
+ </para>
+ <screen><![CDATA[<?xml version="1.0"?>
+<metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
+ <start route="start"/>
+ <routes>
+ <route id="start">
+ <filter type="frontend_net">
+ <threads>10</threads>
+ <port>@:9000</port>
+ </filter>
+ <filter type="virt_db">
+ <virtual>
+ <database>lc</database>
+ <target>lx2.loc.gov:210/LCDB</target>
+ </virtual>
+ <virtual>
+ <database>marc</database>
+ <target>indexdata.com/marc</target>
+ </virtual>
+ <virtual>
+ <database>all</database>
+ <target>lx2.loc.gov:210/LCDB</target>
+ <target>indexdata.com/marc</target>
+ </virtual>
+ </filter>
+ <filter type="multi"/>
+ <filter type="z3950_client">
+ <timeout>30</timeout>
+ </filter>
+ <filter type="bounce"/>
+ </route>
+ </routes>
+</metaproxy>]]></screen>
+ <para>
+ (Using a
+ <literal>virt_db</literal>
+ filter that specifies multiple
+ <literal><target></literal>
+ elements but without a subsequent
+ <literal>multi</literal>
+ filter yields surprising and undesirable results, as will be
+ described below. Don't do that.)
+ </para>
+ <para>
+ Metaproxy can be invoked with this configuration as follows:
+ </para>
+ <screen>../src/metaproxy --config config-simple-multi.xml</screen>
+ <para>
+ And thereafter, Z39.50 clients can connect to the running server
+ (on port 9000, as specified in the configuration) and search in
+ any of the databases
+ <literal>lc</literal> (the Library of Congress catalogue),
+ <literal>marc</literal> (Index Data's test database of MARC records)
+ or
+ <literal>all</literal> (both of these). As an example, a session
+ using the YAZ command-line client <literal>yaz-client</literal> is
+ here included (edited for brevity and clarity):
+ </para>
+ <screen><![CDATA[$ yaz-client @:9000
+Connecting...OK.
+Z> base lc
+Z> find computer
+Search was a success.
+Number of hits: 10000, setno 1
+Elapsed: 5.521070
+Z> base marc
+Z> find computer
+Search was a success.
+Number of hits: 10, setno 3
+Elapsed: 0.060187
+Z> base all
+Z> find computer
+Search was a success.
+Number of hits: 10010, setno 4
+Elapsed: 2.237648
+Z> show 1
+[marc]Record type: USmarc
+001 11224466
+003 DLC
+005 00000000000000.0
+008 910710c19910701nju 00010 eng
+010 $a 11224466
+040 $a DLC $c DLC
+050 00 $a 123-xyz
+100 10 $a Jack Collins
+245 10 $a How to program a computer
+260 1 $a Penguin
+263 $a 8710
+300 $a p. cm.
+Elapsed: 0.119612
+Z> show 2
+[VOYAGER]Record type: USmarc
+001 13339105
+005 20041229102447.0
+008 030910s2004 caua 000 0 eng
+035 $a (DLC) 2003112666
+906 $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
+925 0 $a acquire $b 1 shelf copy $x policy default
+955 $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
+010 $a 2003112666
+020 $a 0761542892
+040 $a DLC $c DLC $d DLC
+050 00 $a MLCM 2004/03312 (G)
+245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
+246 3 $a Double-O-seven, everything or nothing
+246 30 $a Prima's official strategy guide
+260 $a Roseville, CA : $b Prima Games, $c c2004.
+300 $a 161 p. : $b col. ill. ; $c 28 cm.
+500 $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
+650 0 $a Video games.
+710 2 $a Kaizen Media Group.
+856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
+Elapsed: 0.150623
+Z>
+]]></screen>
+ <para>
+ As can be seen, the first record in the result set is from the
+ Index Data test database, and the second from the Library of
+ Congress database. The result-set continues alternating records
+ round-robin style until the point where one of the databases'
+ records are exhausted.
+ </para>
+ <para>
+ This example uses only two back-end databases; more may be used.
+ There is no limitation imposed on the number of databases that may
+ be metasearched in this way: issues of resource usage and
+ administrative complexity dictate the practical limits.
+ </para>
+ <para>
+ What happens when one of the databases doesn't respond? By default,
+ the entire multi-database search fails, and the appropriate
+ diagnostic is returned to the client. This is usually appropriate
+ during development, when technicians need maximum information, but
+ can be inconvenient in deployment, when users typically don't want
+ to be bothered with problems of this kind and prefer just to get
+ the records from the databases that are available. To obtain this
+ latter behavior add an empty
+ <literal><hideunavailable></literal>
+ element inside the
+ <literal>multi</literal> filter:
+ </para>
+ <screen><![CDATA[ <filter type="multi">
+ <hideunavailable/>
+ </filter>]]></screen>
+ <para>
+ Under this regime, an error is reported to the client only if
+ <emphasis>all</emphasis> the databases in a multi-database search
+ are unavailable.
+ </para>
+ </section>
+
+
+ <section id="multidb.what">
+ <title>What's going on?</title>
<warning>
<title>Lark's vomit</title>
<para>
- This chapter goes into a level of technical detail that is
+ This section goes into a level of technical detail that is
probably not necessary in order to configure and use Metaproxy.
It is provided only for those who like to know how things work.
You should feel free to skip on to the next section if this one
</para>
</warning>
<para>
- Two of Metaproxy's filters are concerned with multiple-database
- operations. Of these, <literal>virt_db</literal> can work alone
- to control the routing of searches to one of a number of servers,
- while <literal>multi</literal> can work with the output of
- <literal>virt_db</literal> to perform multicast searching, merging
- the results into a unified result-set. The interaction between
- these two filters is necessarily complex: it reflecting the real,
- irreducible complexity of multicast searching in a protocol such
- as Z39.50 that separates initialisation from searching, and in
- which the database to be searched is not known at initialisation
- time.
- </para>
- <para>
Hold on tight - this may get a little hairy.
</para>
<para>
>the HTTP 1.1 specification</ulink>.
</para>
<para>
- The role of the <literal>virt_db</literal> filter is to rewrite
- this otherInfo packet dependent on the virtual database that the
- client wants to search. For example, a <literal>virt_db</literal>
- filter could be set up so that searches in the virtual database
- ``lc'' are forwarded to the Library of Congress server, and
- searches in the virtual database ``id'' are forwarded to the toy
- GILS database that Index Data hosts for testing purposes. A
- <literal>virt_db</literal> configuration to make this switch would
- look like this:
+ Within Metaproxy, Search requests that are part of the same
+ session as an Init request that carries a
+ <literal>VAL_PROXY</literal> otherInfo are also annotated with the
+ same information. The role of the <literal>virt_db</literal>
+ filter is to rewrite this otherInfo packet dependent on the
+ virtual database that the client wants to search.
</para>
- <screen><![CDATA[
- <filter type="virt_db">
- <virtual>
- <database>lc</database>
- <target>z3950.loc.gov:7090/Voyager</target>
- </virtual>
- <virtual>
- <database>id</database>
- <target>indexdata.dk/gils</target>
- </virtual>
- </filter>]]></screen>
<para>
When Metaproxy receives a Z39.50 Init request from a client, it
doesn't immediately forward that request to the back-end server.
Why not? Because it doesn't know <emphasis>which</emphasis>
- back-end server to forward it to until the client sends a search
+ back-end server to forward it to until the client sends a Search
request that specifies the database that it wants to search in.
Instead, it just treasures the Init request up in its heart; and,
later, the first time the client does a search on one of the
<literal>frontend_net</literal> filter. The
<literal>virt_db</literal> filter knows nothing about it; in
fact, because the Init request that is received from the client
- doesn't get forwarded until a Search reqeust is received, the
+ doesn't get forwarded until a Search request is received, the
<literal>virt_db</literal> filter (and the
<literal>z3950_client</literal> filter behind it) doesn't even get
invoked at Init time. The <emphasis>only</emphasis> thing that a
<literal>VAL_PROXY</literal> otherInfo in the requests that pass
through it.
</para>
+ <para>
+ It is possible for a <literal>virt_db</literal> filter to contain
+ multiple
+ <literal><target></literal>
+ elements. What does this mean? Only that the filter will add
+ multiple <literal>VAL_PROXY</literal> otherInfo packets to the
+ Search requests that pass through it. That's because the virtual
+ DB filter is dumb, and does exactly what it's told - no more, no
+ less.
+ If a Search request with multiple <literal>VAL_PROXY</literal>
+ otherInfo packets reaches a <literal>z3950_client</literal>
+ filter, this is an error. That filter doesn't know how to deal
+ with multiple targets, so it will either just pick one and search
+ in it, or (better) fail with an error message.
+ </para>
+ <para>
+ The <literal>multi</literal> filter comes to the rescue! This is
+ the only filter that knows how to deal with multiple
+ <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
+ making multiple copies of the entire Search request: one for each
+ <literal>VAL_PROXY</literal>. Each of these new copies is then
+ passed down through the remaining filters in the route. (The
+ copies are handled in parallel though the
+ spawning of new threads.) Since the copies each have only one
+ <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
+ <literal>z3950_client</literal> filter, which happily deals with
+ each one individually. When the results of the individual
+ searches come back up to the <literal>multi</literal> filter, it
+ merges them into a single Search response, which is what
+ eventually makes it back to the client.
+ </para>
+
+ <mediaobject>
+ <imageobject>
+ <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
+ </imageobject>
+ <imageobject>
+ <imagedata fileref="multi.png" format="PNG"/>
+ </imageobject>
+ <textobject>
+ <!-- Fall back if none of the images can be used -->
+ <phrase>
+ [Here there should be a diagram showing the progress of
+ packages through the filters during a simple virtual-database
+ search and a multi-database search, but is seems that your
+ tool chain has not been able to include the diagram in this
+ document.]
+ </phrase>
+ </textobject>
+ <caption>
+ <para>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</para>
+ </caption>
+ </mediaobject>
</section>
</chapter>
+ <chapter id="sru-server">
+ <title>Combined SRU webservice and Z39.50 server configuration</title>
+ <para>
+ Metaproxy can act as
+ <ulink url="&url.sru;">SRU</ulink> and
+ web service server, which translates web service requests to
+ <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink> packages and
+ sends them off to common available targets.
+ </para>
+ <para>
+ A typical setup for this operation needs a filter route including the
+ following modules:
+ </para>
+
+ <table id="sru-server-table-config" frame="top">
+ <title>SRU/Z39.50 Server Filter Route Configuration</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Filter</entry>
+ <entry>Importance</entry>
+ <entry>Purpose</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><literal>frontend_net</literal></entry>
+ <entry>required</entry>
+ <entry>Accepting HTTP connections and passing them to following
+ filters. Since this filter also accepts Z39.50 connections, the
+ server works as SRU and Z39.50 server on the same port.</entry>
+ </row>
+ <row>
+ <entry><literal>sru_z3950</literal></entry>
+ <entry>required</entry>
+ <entry>Accepting SRU GET/POST/SOAP explain and
+ searchRetrieve requests for the the configured databases.
+ Explain requests are directly served from the static XML configuration.
+ SearchRetrieve requests are
+ transformed to Z39.50 search and present packages.
+ All other HTTP and Z39.50 packages are passed unaltered.</entry>
+ </row>
+ <row>
+ <entry><literal>http_file</literal></entry>
+ <entry>optional</entry>
+ <entry>Serving HTTP requests from the filesystem. This is only
+ needed if the server should serve XSLT stylesheets, static HTML
+ files or Java Script for thin browser based clients.
+ Z39.50 packages are passed unaltered.</entry>
+ </row>
+ <row>
+ <entry><literal>cql_rpn</literal></entry>
+ <entry>required</entry>
+ <entry>Usually, Z39.50 servers do not talk CQL, hence the
+ translation of the CQL query language to RPN is mandatory in
+ most cases. Affects only Z39.50 search packages.</entry>
+ </row>
+ <row>
+ <entry><literal>record_transform</literal></entry>
+ <entry>optional</entry>
+ <entry>Some Z39.50 backend targets can not present XML record
+ syntaxes in common wanted element sets. using this filter, one
+ can transform binary MARC records to MARCXML records, and
+ further transform those to any needed XML schema/format by XSLT
+ transformations. Changes only Z39.50 present packages.</entry>
+ </row>
+ <row>
+ <entry><literal>session_shared</literal></entry>
+ <entry>optional</entry>
+ <entry>The stateless nature of web services requires frequent
+ re-searching of the same targets for display of paged result set
+ records. This might be an unacceptable burden for the accessed
+ backend Z39.50 targets, and this mosule can be added for
+ efficient backend target resource pooling.</entry>
+ </row>
+ <row>
+ <entry><literal>z3950_client</literal></entry>
+ <entry>required</entry>
+ <entry>Finally, a Z39.50 package sink is needed in the filter
+ chain to provide the response packages. The Z39.50 client module
+ is used to access external targets over the network, but any
+ coming local Z39.50 package sink could be used instead of.</entry>
+ </row>
+ <row>
+ <entry><literal>bounce</literal></entry>
+ <entry>required</entry>
+ <entry>Any Metaproxy package arriving here did not do so by
+ purpose, and is bounced back with connection closure. this
+ prevents inifinite package hanging inside the SRU server.</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para>
+ A typical minimal example <ulink url="&url.sru;">SRU</ulink>
+ server configuration file is found in the tarball distribution at
+ <literal>etc/config-sru-to-z3950.xml</literal>.
+ </para>
+ <para>
+ Off course, any other metaproxy modules can be integrated into a
+ SRU server solution, including, but not limited to, load balancing,
+ multiple target querying
+ (see <xref linkend="multidb"/>), and complex RPN query rewrites.
+ </para>
+
+ </chapter>
+
+ <!--
<chapter id="extensions">
<title>Writing extensions for Metaproxy</title>
<para>### To be written</para>
</chapter>
-
+ -->
<title>Classes in the Metaproxy source code</title>
- <section>
+ <section id="classes-introductory-notes">
<title>Introductory notes</title>
<para>
<emphasis>Stop! Do not read this!</emphasis>
You won't enjoy it at all. You should just skip ahead to
- <link linkend="refguide">the reference guide</link>,
+ <xref linkend="reference"/>,
which tells
<!-- The remainder of this paragraph is lifted verbatim from
Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
<para>
This chapter contains documentation of the Metaproxy source code, and is
of interest only to maintainers and developers. If you need to
- change Metaproxy's behaviour or write a new filter, then you will most
+ change Metaproxy's behavior or write a new filter, then you will most
likely find this chapter helpful. Otherwise it's a waste of your
good time. Seriously: go and watch a film or something.
<citetitle>This is Spinal Tap</citetitle> is particularly good.
parentheses.
</para>
- <section>
+ <section id="class-FactoryFilter">
<title><literal>mp::FactoryFilter</literal>
(<filename>factory_filter.cpp</filename>)</title>
<para>
</para>
</section>
- <section>
+ <section id="class-FactoryStatic">
<title><literal>mp::FactoryStatic</literal>
(<filename>factory_static.cpp</filename>)</title>
<para>
</para>
</section>
- <section>
+ <section id="class-filter-Base">
<title><literal>mp::filter::Base</literal>
(<filename>filter.cpp</filename>)</title>
<para>
The virtual base class of all filters. The filter API is, on the
surface at least, extremely simple: two methods.
- <literal>configure()</literal> is passed a DOM tree representing
+ <literal>configure()</literal> is passed an XML DOM tree representing
that part of the configuration file that pertains to this filter
instance, and is expected to walk that tree extracting relevant
information. And <literal>process()</literal> processes a
- package (see below). That surface simplicitly is a bit
+ package (see below). That surface simplicity is a bit
misleading, as <literal>process()</literal> needs to know a lot
about the <literal>Package</literal> class in order to do
anything useful.
</para>
</section>
- <section>
+ <section id="class-AuthSimple">
<title><literal>mp::filter::AuthSimple</literal>,
<literal>Backend_test</literal>, etc.
(<filename>filter_auth_simple.cpp</filename>,
<filename>filter_*.cpp</filename> respectively. All the header
files should be pretty much identical, in that they declare the
class, including a private <literal>Rep</literal> class and a
- member pointer to it, and the two public methods. The only extra
- information in any filter header is additional private types and
- members (which should really all be in the <literal>Rep</literal>
- anyway) and private methods (which should also remain known only
- to the source file, but C++'s brain-damaged design requires this
- dirty laundry to be exhibited in public. Thanks, Bjarne!)
+ member pointer to it, and the two public methods.
</para>
<para>
The source file for each filter needs to supply:
</itemizedlist>
</section>
- <section>
+ <section id="class-Package">
<title><literal>mp::Package</literal>
(<filename>package.cpp</filename>)</title>
<para>
</para>
</section>
- <section>
+ <section id="class-Pipe">
<title><literal>mp::Pipe</literal>
(<filename>pipe.cpp</filename>)</title>
<para>
</para>
</section>
- <section>
+ <section id="class-RouterChain">
<title><literal>mp::RouterChain</literal>
(<filename>router_chain.cpp</filename>)</title>
<para>
</para>
</section>
- <section>
+ <section id="class-RouterFleXML">
<title><literal>mp::RouterFleXML</literal>
(<filename>router_flexml.cpp</filename>)</title>
<para>
</para>
</section>
- <section>
+ <section id="class-Session">
<title><literal>mp::Session</literal>
(<filename>session.cpp</filename>)</title>
<para>
</para>
</section>
- <section>
+ <section id="class-ThreadPoolSocketObserver">
<title><literal>mp::ThreadPoolSocketObserver</literal>
(<filename>thread_pool_observer.cpp</filename>)</title>
<para>
</para>
</section>
- <section>
+ <section id="class-util">
<title><literal>mp::util</literal>
(<filename>util.cpp</filename>)</title>
<para>
</para>
</section>
- <section>
+ <section id="class-xml">
<title><literal>mp::xml</literal>
(<filename>xmlutil.cpp</filename>)</title>
<para>
</chapter>
+ <reference id="reference">
+ <title>Reference</title>
+ <partintro id="reference-introduction">
+ <para>
+ The material in this chapter is drawn directly from the individual
+ manual entries. In particular, the Metaproxy invocation section is
+ available using <command>man metaproxy</command>, and the section
+ on each individual filter is available using the name of the filter
+ as the argument to the <command>man</command> command.
+ </para>
+ </partintro>
+ &manref;
+ </reference>
- <chapter id="refguide">
- <title>Reference guide</title>
- <para>
- The material in this chapter is drawn directly from the individual
- manual entries. In particular, the Metaproxy invocation section is
- available using <command>man metaproxy</command>, and the section
- on each individual filter is available using the name of the filter
- as the argument to the <command>man</command> command.
- </para>
+<appendix id="license">
+ <title>License</title>
+ ©right;
- <section id="progref">
- <title>Metaproxy invocation</title>
- &progref;
- </section>
+ <para>
+ Metaproxy is free software; you can redistribute it and/or modify it under
+ the terms of the GNU General Public License as published by the Free
+ Software Foundation; either version 2, or (at your option) any later
+ version.
+ </para>
+ <para>
+ Metaproxy is distributed in the hope that it will be useful, but WITHOUT ANY
+ WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ for more details.
+ </para>
- <section id="filterref">
- <title>Reference guide to Metaproxy filters</title>
- &manref;
- </section>
- </chapter>
+ <para>
+ You should have received a copy of the GNU General Public License
+ along with Metaproxy; see the file LICENSE. If not, write to the
+ Free Software Foundation,
+ 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ </para>
+ </appendix>
+ &gpl2;
+</book>
- <!-- Keep this comment at the end of the file
- Local variables:
- mode: sgml
- sgml-omittag:t
- sgml-shorttag:t
- sgml-minimize-attributes:nil
- sgml-always-quote-attributes:t
- sgml-indent-step:1
- sgml-indent-data:t
- sgml-parent-document: "main.xml"
- sgml-local-catalogs: nil
- sgml-namecase-general:t
- nxml-child-indent: 1
- End:
- -->
+<!-- Keep this comment at the end of the file
+Local variables:
+mode: nxml
+nxml-child-indent: 1
+End:
+-->
projects / metaproxy-moved-to-github.git / blobdiff