<?xml version="1.0" standalone="no"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
- "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd"
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
+ "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
[
+ <!ENTITY copyright SYSTEM "copyright.xml">
<!ENTITY % local SYSTEM "local.ent">
%local;
<!ENTITY manref SYSTEM "manref.xml">
- <!ENTITY progref SYSTEM "progref.xml">
<!ENTITY gpl2 SYSTEM "gpl-2.0.xml">
<!ENTITY % idcommon SYSTEM "common/common.ent">
%idcommon;
- <!-- Next line allows imagedata/@format="PDF" and is taken from
- http://lists.oasis-open.org/archives/docbook/200303/msg00163.html
- -->
- <!ENTITY % local.notation.class "| PDF">
- <!-- Next line is necessary for some XML parsers, for reasons I
- don't understand. I got this from
- http://lists.oasis-open.org/archives/docbook/200303/msg00180.html
- -->
- <!NOTATION PDF SYSTEM "PDF">
]>
-<book id="metaproxy">
+<book>
<bookinfo>
<title>Metaproxy - User's Guide and Reference</title>
<authorgroup>
</authorgroup>
<releaseinfo>&version;</releaseinfo>
<copyright>
- <year>2005-2009</year>
- <holder>Index Data ApS</holder>
+ <year>2005-2015</year>
+ <holder>Index Data</holder>
</copyright>
<abstract>
<simpara>
processes, interprets and redirects requests from IR clients using
standard protocols such as the binary
<ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
- and the information search and retrieval
+ and the information search and retrieval
web service <ulink url="&url.sru;">SRU</ulink>
as well as functioning as a limited
- <ulink url="&url.http;">HTTP</ulink> server.
+ <ulink url="&url.http;">HTTP</ulink> server.
</simpara>
<simpara>
Metaproxy is configured by an XML file which
<chapter id="introduction">
<title>Introduction</title>
-
+
<para>
<ulink url="&url.metaproxy;">Metaproxy</ulink>
is a stand alone program that acts as a universal router, proxy and
as <ulink url="&url.z39.50;">Z39.50</ulink> and
<ulink url="&url.sru;">SRU</ulink>.
To clients, it acts as a server of these protocols: it can be searched,
- records can be retrieved from it, etc.
+ records can be retrieved from it, etc.
To servers, it acts as a client: it searches in them,
retrieves records from them, etc. it satisfies its clients'
requests by transforming them, multiplexing them, forwarding them
including examples.
</para>
</chapter>
-
+
<chapter id="installation">
<title>Installation</title>
<para>
</varlistentry>
<varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
<listitem>
- <para>This is an XSLT processor - based on
+ <para>This is an XSLT processor - based on
<ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
Libxslt must be installed with the development components
(header files, etc.) as well as the run-time libraries.
<para>
The popular C++ library. Initial versions of Metaproxy
was built with 1.32 but this is no longer supported.
- Metaproxy is known to work with Boost version 1.33 through 1.38.
+ Metaproxy is known to work with Boost version 1.33 through 1.55.
</para>
</listitem>
</varlistentry>
</para>
<para>
We have successfully built Metaproxy using the compilers
- <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
- <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
+ <ulink url="&url.gcc;">GCC</ulink> and
+ <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink>.
</para>
+ <para>
+ As an option, Metaproxy may also be compiled with
+ <ulink url="&url.usemarcon;">USEMARCON</ulink> support which allows for
+ MARC conversions for the <xref linkend="ref-record_transform"/> filter.
+ </para>
<section id="installation.unix">
<title>Installation on Unix (from Source)</title>
<para>
installed as development packages use those (and omit compilation).
</para>
+ <note>
+ <para>
+ <ulink url="&url.usemarcon;">USEMARCON</ulink> is not available
+ as a package at the moment, so Metaproxy must be built from source
+ if that is to be used.
+ </para>
+ </note>
+
<section id="libxml2.fromsource">
<title>Libxml2/libxslt</title>
<para>
make install
</screen>
</section>
+ <section id="usemarcon">
+ <title>USEMARCON (optional)</title>
+ <screen>
+ gunzip -c usemarcon317.tar.gz|tar xf -
+ cd usemarcon317
+ ./configure
+ make
+ su
+ make install
+ </screen>
+ </section>
+
<section id="yaz.fromsource">
<title>YAZ/YAZ++</title>
<screen>
<screen>
gunzip -c boost-version.tar.gz|tar xf -
cd boost-version
- ./configure --with-libraries=thread,test --with-toolset=gcc
+ ./configure --with-libraries=thread,test,regex --with-toolset=gcc
make
su
make install
However, under the hood bjam is used. You can invoke that with
</para>
<screen>
- ./bjam --toolset=gcc --with-thread --with-test stage
+ ./bjam --toolset=gcc --with-thread --with-test --with-regex stage
</screen>
<para>
Replace <literal>stage</literal> with <literal>clean</literal> /
The latter the compiler toolset (eg. gcc34).
</para>
<para>
- Pass <literal>--help</literal> to configure to get a list of
+ Pass <literal>--help</literal> to configure to get a list of
available options.
</para>
</section>
<section id="installation.debian">
<title>Installation on Debian GNU/Linux</title>
<para>
- All dependencies for Metaproxy are available as
- <ulink url="&url.debian;">Debian</ulink>
- packages for the sarge (stable in 2005) and etch (testing in 2005)
- distributions.
+ All dependencies for Metaproxy are available as
+ <ulink url="&url.debian;">Debian</ulink> packages.
</para>
<para>
The procedures for Debian based systems, such as
<para>
There is currently no official Debian package for YAZ++.
And the official Debian package for YAZ is probably too old.
- But Index Data bulds "new" versions of those for Debian (i386 only).
+ But Index Data builds "new" versions of those for Debian (i386, amd64 only).
</para>
<para>
Update the <filename>/etc/apt/sources.list</filename>
</para>
<screen>
apt-get install libxslt1-dev
- apt-get install libyazpp2-dev
+ apt-get install libyazpp6-dev
apt-get install libboost-dev
+ apt-get install libboost-system-dev
apt-get install libboost-thread-dev
apt-get install libboost-test-dev
+ apt-get install libboost-regex-dev
</screen>
<para>
With these packages installed, the usual configure + make
<section id="installation.rpm">
<title>Installation on RPM based Linux Systems</title>
<para>
- All external dependencies for Metaproxy are available as
- RPM packages, either from your distribution site, or from the
+ All external dependencies for Metaproxy are available as
+ RPM packages, either from your distribution site, or from the
<ulink url="http://fr.rpmfind.net/">RPMfind</ulink> site.
</para>
<para>
- For example, an installation of the requires Boost C++ development
+ For example, an installation of the requires Boost C++ development
libraries on RedHat Fedora C4 and C5 can be done like this:
- <screen>
+ <screen>
wget ftp://fr.rpmfind.net/wlinux/fedora/core/updates/testing/4/SRPMS/boost-1.33.0-3.fc4.src.rpm
sudo rpmbuild --buildroot src/ --rebuild -p fc4/boost-1.33.0-3.fc4.src.rpm
sudo rpm -U /usr/src/redhat/RPMS/i386/boost-*rpm
</para>
<para>
There is currently no official RPM package for YAZ++.
- See the <ulink url="&url.yazplusplus;">YAZ++</ulink> pages
+ See the <ulink url="&url.yazplusplus;">YAZ++</ulink> pages
for more information on a Unix tarball install.
</para>
<para>
<section id="installation.windows">
<title>Installation on Windows</title>
<para>
- Metaproxy can be compiled with Microsoft
+ Metaproxy has been tested Microsoft
<ulink url="&url.vstudio;">Visual Studio</ulink>.
- Versions 2003 (C 7.1), 2005 (C 8.0) and 2008 (C 9.0) is known to work.
+ 2013 (C 12.0).
</para>
<section id="installation.windows.boost">
<title>Boost</title>
<para>
<ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
for Windows from
- <ulink url="&url.libxml2.download.win32;">here</ulink>.
+ <ulink url="&url.libxml2.download.windows;">here</ulink>.
</para>
<para>
- Libxslt has other dependencies, but these can all be downloaded
- from the same site. Get the following package:
- iconv, zlib, libxml2, libxslt.
+ Libxslt also requires libxml2 to operate.
</para>
</section>
<title>YAZ++</title>
<para>
Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
- Version 1.1.0 or later is required.
+ Version 1.6.0 or later is required.
</para>
<para>
YAZ++ includes NMAKE makefiles, similar to those found in the
</para>
</listitem>
</varlistentry>
-
+
</variablelist>
-
+
<para>
After successful compilation you'll find
<literal>metaproxy.exe</literal> in the
</section>
</chapter>
-
+
<chapter id="yazproxy-comparison">
<title>YAZ Proxy Comparison</title>
<para>
<tbody>
<row>
<entry>Z39.50 server</entry>
- <entry>Using filter <literal>frontend_net</literal></entry>
+ <entry>Using filter <xref linkend="ref-frontend_net"/></entry>
<entry>Supported</entry>
</row>
<row>
<entry>SRU server</entry>
- <entry>Supported with filter <literal>sru_z3950</literal></entry>
+ <entry>Supported with filter <xref linkend="ref-sru_z3950"/></entry>
<entry>Supported</entry>
</row>
<row>
<entry>Z39.50 client</entry>
- <entry>Supported with filter <literal>z3950_client</literal></entry>
+ <entry>Supported with filter <xref linkend="ref-z3950_client"/></entry>
<entry>Supported</entry>
</row>
<row>
<entry>SRU client</entry>
- <entry>Unsupported</entry>
+ <entry>Supported with filter <xref linkend="ref-zoom"/></entry>
<entry>Unsupported</entry>
</row>
<row>
</row>
<row>
<entry>Record cache</entry>
- <entry>Unsupported</entry>
+ <entry>Supported by filter <literal>session_shared</literal></entry>
<entry>Supported for last result set within one Z39.50/HTTP-keep alive session</entry>
</row>
<row>
</row>
<row>
<entry>SRU Virtual database, i.e. select any Z39.50 target for path</entry>
- <entry>Supported with filter <literal>virt_db</literal>,
+ <entry>Supported with filter <literal>virt_db</literal>,
<literal>sru_z3950</literal></entry>
<entry>Supported</entry>
</row>
</row>
<row>
<entry>Retrieval and search limits</entry>
- <entry>Unsupported</entry>
+ <entry>Supported using filter <literal>limit</literal></entry>
<entry>Supported</entry>
</row>
<row>
<entry>Bandwidth limits</entry>
- <entry>Unsupported</entry>
+ <entry>Supported using filter <literal>limit</literal></entry>
<entry>Supported</entry>
</row>
<row>
<entry>Connect limits</entry>
- <entry>Unsupported</entry>
+ <entry>Supported by filter <literal>frontend_net</literal> (connect-max)</entry>
<entry>Supported</entry>
</row>
<row>
<row>
<entry>Query check</entry>
<entry>
- Supported in a limited way using <literal>query_rewrite</literal>
+ Supported by <literal>query_rewrite</literal> which may be check
+ a query and throw diagnostics (errors)
</entry>
<entry>Supported</entry>
</row>
<row>
<entry><ulink url="&url.usemarcon;">USEMARCON</ulink></entry>
- <entry>Unsupported</entry>
+ <entry>Supported with <literal>record_transform</literal></entry>
<entry>Supported</entry>
</row>
<chapter id="filters">
<title>Filters</title>
-
-
+
+
<section id="filters-introductory-notes">
<title>Introductory notes</title>
<para>
others are sinks: they consume packages and return a result
(<literal>backend_test</literal>,
<literal>bounce</literal>,
- <literal>http_file</literal>,
+ <literal>http_file</literal>,
<literal>z3950_client</literal>);
the others are true filters, that read, process and pass on the
packages they are fed
<literal>virt_db</literal>).
</para>
</section>
-
-
+
+
<section id="overview.filter.types">
<title>Overview of filter types</title>
<para>
<para>
The filters are here listed in alphabetical order:
</para>
-
+
<!--
### New filters:
the user.
</para>
</section>
-
+
<section id="backend_test">
<title><literal>backend_test</literal>
(mp::filter::Backend_test)</title>
even read this section.
</para>
</section>
-
+
<section id="bounce">
<title><literal>bounce</literal>
(mp::filter::Bounce)</title>
<para>
- A sink that swallows <emphasis>all packages</emphasis>,
+ A sink that swallows <emphasis>all packages</emphasis>,
and returns them almost unprocessed.
It never sends any package of any type further down the row, but
sets Z39.50 packages to Z_Close, and HTTP_Request packages to
HTTP_Response err code 400 packages, and adds a suitable bounce
- message.
+ message.
The bounce filter is usually added at end of each filter chain route
to prevent infinite hanging of for example HTTP
- requests packages when only the Z39.50 client partial sink
+ requests packages when only the Z39.50 client partial sink
filter is found in the
- route.
+ route.
</para>
</section>
-
+
<section id="cql_rpn">
<title><literal>cql_rpn</literal>
(mp::filter::CQLtoRPN)</title>
<para>
- A query language transforming filter which catches Z39.50
+ A query language transforming filter which catches Z39.50
<literal>searchRequest</literal>
packages containing <literal>CQL</literal> queries, transforms
those to <literal>RPN</literal> queries,
and sends the <literal>searchRequests</literal> on to the next
- filters. It is among other things useful in a SRU context.
+ filters. It is among other things useful in a SRU context.
</para>
</section>
-
+
<section id="frontend_net">
<title><literal>frontend_net</literal>
(mp::filter::FrontendNet)</title>
<title><literal>http_file</literal>
(mp::filter::HttpFile)</title>
<para>
- A partial sink which swallows only
- <literal>HTTP_Request</literal> packages, and
+ A partial sink which swallows only
+ <literal>HTTP_Request</literal> packages, and
returns the contents of files from the local
- filesystem in response to HTTP requests.
+ filesystem in response to HTTP requests.
It lets Z39.50 packages and all other forthcoming package types
- pass untouched.
+ pass untouched.
(Yes, Virginia, this
does mean that Metaproxy is also a Web-server in its spare time. So
far it does not contain either an email-reader or a Lisp
interpreter, but that day is surely coming.)
</para>
</section>
-
+
<section id="load_balance">
<title><literal>load_balance</literal>
(mp::filter::LoadBalance)</title>
Performs load balancing for incoming Z39.50 init requests.
It is used together with the <literal>virt_db</literal> filter,
but unlike the <literal>multi</literal> filter it does send an
- entire session to only one of the virtual backends. The
+ entire session to only one of the virtual backends. The
<literal>load_balance</literal> filter is assuming that
all backend targets have equal content, and chooses the backend
with least load cost for a new session.
</warning>
</para>
</section>
-
+
<section id="log">
<title><literal>log</literal>
(mp::filter::Log)</title>
of virtual databases and multi-database searching below.
</para>
</section>
-
+
<section id="query_rewrite">
<title><literal>query_rewrite</literal>
(mp::filter::QueryRewrite)</title>
<para>
- Rewrites Z39.50 <literal>Type-1</literal>
- and <literal>Type-101</literal> (``<literal>RPN</literal>'')
+ Rewrites Z39.50 <literal>Type-1</literal>
+ and <literal>Type-101</literal> (``<literal>RPN</literal>'')
queries by a
three-step process: the query is transliterated from Z39.50
packet structures into an XML representation; that XML
structure.
</para>
</section>
-
-
+
+
<section id="record_transform">
<title><literal>record_transform</literal>
(mp::filter::RecordTransform)</title>
This filter implements global sharing of
result sets (i.e. between threads and therefore between
clients), yielding performance improvements by clever resource
- pooling.
+ pooling.
</para>
</section>
messages.
The <literal>sru_z3950</literal> filter processes also SRU
GET/POST/SOAP explain requests, returning
- either the absolute minimum required by the standard, or a full
+ either the absolute minimum required by the standard, or a full
pre-defined ZeeReX explain record.
- See the
+ See the
<ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
- standard pages and the
+ standard pages and the
<ulink url="&url.sru.explain;">SRU Explain</ulink> pages
for more information on the correct explain syntax.
SRU scan requests are not supported yet.
</para>
</section>
-
+
<section id="template">
<title><literal>template</literal>
(mp::filter::Template)</title>
intended for civilians.
</para>
</section>
-
+
<section id="virt_db">
<title><literal>virt_db</literal>
(mp::filter::VirtualDB)</title>
of virtual databases and multi-database searching below.
</para>
</section>
-
+
<section id="z3950_client">
<title><literal>z3950_client</literal>
(mp::filter::Z3950Client)</title>
the route. Subsequent requests are sent to the same address,
which is remembered at Init time in a Session object.
HTTP_Request packages and all other forthcoming package types
- are passed untouched.
+ are passed untouched.
</para>
</section>
Z39.50 explain requests, returning a static ZeeReX
Explain XML record from the config section. All other packages
are passed through.
- See the
+ See the
<ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
standard pages
for more information on the correct explain syntax.
</para>
</warning>
</section>
-
+
</section>
-
-
+
+
<section id="future.directions">
<title>Future directions</title>
<para>
</variablelist>
</section>
</chapter>
-
-
-
+
+
+
<chapter id="configuration">
<title>Configuration: the Metaproxy configuration file format</title>
-
-
+
+
<section id="configuration-introductory-notes">
<title>Introductory notes</title>
<para>
Metaproxy.)
</para>
</section>
-
+
<section id="overview.xml.structure">
<title>Overview of the config file XML structure</title>
<para>
All four are non-repeatable.
</para>
<para>
- The <dlpath;> element contains a text element which
+ The <dlpath;> element contains a text element which
specifies the location of filter modules. This is only needed
if Metaproxy must load 3rd party filters (most filters with Metaproxy
are built into the Metaproxy application).
<filter id="backend" type="z3950_client">
</filter>
</filters>
- <routes>
+ <routes>
<route id="start">
<filter refid="frontend"/>
<filter type="log"/>
mutton, beef and trout packages.
When the response arrives, it is handed
back to the <literal>log</literal> filter, which emits another
- message; and then to the <literal>frontend_net</literal> filter,
+ message; and then to the <literal>frontend_net</literal> filter,
which returns the response to the client.
</para>
</section>
<para>
Metaproxy XML configuration snippets can be reused by other
filters using the <literal>XInclude</literal> standard, as seen in
- the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU
+ the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU
configuration.
<screen><![CDATA[
<filter id="sru" type="sru_z3950">
<para>
The distribution contains RelaxNG Compact and XML syntax checking
files, as well as XML Schema files. These are found in the
- distribution paths
+ distribution paths
<screen>
xml/schema/metaproxy.rnc
xml/schema/metaproxy.rng
configuration files. For example, using the utility
<filename>xmllint</filename>, syntax checking is done like this:
<screen>
- xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml
- xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml
+ xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml
+ xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml
</screen>
(A recent version of <literal>libxml2</literal> is required, as
support for XML Schemas is a relatively recent addition.)
merges them into a single Search response, which is what
eventually makes it back to the client.
</para>
- </section>
-
- <section id="multidb.picture">
- <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
- <simpara>
- <inlinemediaobject>
+ <mediaobject>
<imageobject>
<imagedata fileref="multi.pdf" format="PDF" scale="50"/>
</imageobject>
document.]
</phrase>
</textobject>
-<!-- ### This used to work with an older version of DocBook
<caption>
- <para>Caption: progress of packages through filters.</para>
+ <para>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</para>
</caption>
--->
- </inlinemediaobject>
- </simpara>
+ </mediaobject>
</section>
</chapter>
<chapter id="sru-server">
<title>Combined SRU webservice and Z39.50 server configuration</title>
<para>
- Metaproxy can act as
- <ulink url="&url.sru;">SRU</ulink> and
- web service server, which translates web service requests to
+ Metaproxy can act as
+ <ulink url="&url.sru;">SRU</ulink> and
+ web service server, which translates web service requests to
<ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink> packages and
sends them off to common available targets.
</para>
<para>
A typical setup for this operation needs a filter route including the
- following modules:
+ following modules:
</para>
-
+
<table id="sru-server-table-config" frame="top">
<title>SRU/Z39.50 Server Filter Route Configuration</title>
<tgroup cols="3">
<entry>Purpose</entry>
</row>
</thead>
-
+
<tbody>
<row>
<entry><literal>frontend_net</literal></entry>
</tbody>
</tgroup>
</table>
- <para>
+ <para>
A typical minimal example <ulink url="&url.sru;">SRU</ulink>
- server configuration file is found in the tarball distribution at
+ server configuration file is found in the tarball distribution at
<literal>etc/config-sru-to-z3950.xml</literal>.
- </para>
+ </para>
<para>
Off course, any other metaproxy modules can be integrated into a
- SRU server solution, including, but not limited to, load balancing,
- multiple target querying
- (see <xref linkend="multidb"/>), and complex RPN query rewrites.
+ SRU server solution, including, but not limited to, load balancing,
+ multiple target querying
+ (see <xref linkend="multidb"/>), and complex RPN query rewrites.
</para>
</para>
</section>
</chapter>
-
-
+
+
<reference id="reference">
<title>Reference</title>
<partintro id="reference-introduction">
<appendix id="license">
<title>License</title>
- <para>
- Metaproxy, Copyright © 1995-2009 Index Data ApS.
- </para>
+ ©right;
<para>
Metaproxy is free software; you can redistribute it and/or modify it under
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.
</para>
-
+
<para>
You should have received a copy of the GNU General Public License
along with Metaproxy; see the file LICENSE. If not, write to the
- Free Software Foundation,
+ Free Software Foundation,
51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
</para>
-
+
</appendix>
&gpl2;
</book>
- <!-- Keep this comment at the end of the file
- Local variables:
- mode: sgml
- sgml-omittag:t
- sgml-shorttag:t
- sgml-minimize-attributes:nil
- sgml-always-quote-attributes:t
- sgml-indent-step:1
- sgml-indent-data:t
- sgml-parent-document: nil
- sgml-local-catalogs: nil
- sgml-namecase-general:t
- End:
- -->
+<!-- Keep this comment at the end of the file
+Local variables:
+mode: nxml
+nxml-child-indent: 1
+End:
+-->
projects / metaproxy-moved-to-github.git / blobdiff