1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
3 "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd"
5 <!ENTITY % local SYSTEM "local.ent">
7 <!ENTITY manref SYSTEM "manref.xml">
8 <!ENTITY progref SYSTEM "progref.xml">
9 <!ENTITY % idcommon SYSTEM "common/common.ent">
11 <!-- Next line allows imagedata/@format="PDF" and is taken from
12 http://lists.oasis-open.org/archives/docbook/200303/msg00163.html
14 <!ENTITY % local.notation.class "| PDF">
15 <!-- Next line is necessary for some XML parsers, for reasons I
16 don't understand. I got this from
17 http://lists.oasis-open.org/archives/docbook/200303/msg00180.html
19 <!NOTATION PDF SYSTEM "PDF">
23 <title>Metaproxy - User's Guide and Reference</title>
26 <firstname>Adam</firstname><surname>Dickmeiss</surname>
29 <firstname>Marc</firstname><surname>Cromme</surname>
32 <firstname>Mike</firstname><surname>Taylor</surname>
35 <releaseinfo>&version;</releaseinfo>
37 <year>2005-2009</year>
38 <holder>Index Data ApS</holder>
42 This manual is part of Metaproxy version &version;.
45 Metaproxy is a universal router, proxy and encapsulated
46 metasearcher for information retrieval protocols. It accepts,
47 processes, interprets and redirects requests from IR clients using
48 standard protocols such as the binary
49 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
50 and the information search and retrieval
51 web service <ulink url="&url.sru;">SRU</ulink>
52 as well as functioning as a limited
53 <ulink url="&url.http;">HTTP</ulink> server.
56 Metaproxy is configured by an XML file which
57 specifies how the software should function in terms of routes that
58 the request packets can take through the proxy, each step on a
59 route being an instantiation of a filter. Filters come in many
60 types, one for each operation: accepting Z39.50 packets, logging,
61 query transformation, multiplexing, etc. Further filter-types can
62 be added as loadable modules to extend Metaproxy functionality,
66 Metaproxy is covered by the GNU General Public License version 2.
71 <imagedata fileref="common/id.png" format="PNG"/>
74 <imagedata fileref="common/id.eps" format="EPS"/>
81 <chapter id="introduction">
82 <title>Introduction</title>
85 <ulink url="&url.metaproxy;">Metaproxy</ulink>
86 is a stand alone program that acts as a universal router, proxy and
87 encapsulated metasearcher for information retrieval protocols such
88 as <ulink url="&url.z39.50;">Z39.50</ulink> and
89 <ulink url="&url.sru;">SRU</ulink>.
90 To clients, it acts as a server of these protocols: it can be searched,
91 records can be retrieved from it, etc.
92 To servers, it acts as a client: it searches in them,
93 retrieves records from them, etc. it satisfies its clients'
94 requests by transforming them, multiplexing them, forwarding them
95 on to zero or more servers, merging the results, transforming
96 them, and delivering them back to the client. In addition, it
97 acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
98 for further protocols can be added in a modular fashion, through the
99 creation of new filters.
104 Fish, bananas, cold pyjamas,
105 Mutton, beef and trout!
106 - attributed to Cole Porter.
109 Metaproxy is a more capable alternative to
110 <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
111 being more powerful, flexible, configurable and extensible. Among
112 its many advantages over the older, more pedestrian work are
113 support for multiplexing (encapsulated metasearching), routing by
114 database name, authentication and authorization and serving local
115 files via HTTP. Equally significant, its modular architecture
116 facilitites the creation of pluggable modules implementing further
120 This manual will describe how to install Metaproxy
121 before giving an overview of its architecture, then discussing the
122 key concept of a filter in some depth and giving an overview of
123 the various filter types, then discussing the configuration file
124 format. After this come several optional chapters which may be
125 freely skipped: a detailed discussion of virtual databases and
126 multi-database searching, some notes on writing extensions
127 (additional filter types) and a high-level description of the
128 source code. Finally comes the reference guide, which contains
129 instructions for invoking the <command>metaproxy</command>
130 program, and detailed information on each type of filter,
135 <chapter id="installation">
136 <title>Installation</title>
138 Metaproxy depends on the following tools/libraries:
140 <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
143 This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
147 <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
149 <para>This is an XSLT processor - based on
150 <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
151 Libxslt must be installed with the development components
152 (header files, etc.) as well as the run-time libraries.
156 <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
159 The popular C++ library. Initial versions of Metaproxy
160 was built with 1.32 but this is no longer supported.
161 Metaproxy is known to work with Boost version 1.33 through 1.38.
168 In order to compile Metaproxy a modern C++ compiler is
169 required. Boost, in particular, requires the C++ compiler
170 to facilitate the newest features. Refer to Boost
171 <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
172 for more information.
175 We have successfully built Metaproxy using the compilers
176 <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
177 <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
180 <section id="installation.unix">
181 <title>Installation on Unix (from Source)</title>
183 Here is a quick step-by-step guide on how to compile all the
184 tools that Metaproxy uses. Only few systems have none of the required
185 tools binary packages. If, for example, Libxml2/libxslt are already
186 installed as development packages use those (and omit compilation).
189 <section id="libxml2.fromsource">
190 <title>Libxml2/libxslt</title>
195 gunzip -c libxml2-version.tar.gz|tar xf -
203 gunzip -c libxslt-version.tar.gz|tar xf -
211 <section id="yaz.fromsource">
212 <title>YAZ/YAZ++</title>
214 gunzip -c yaz-version.tar.gz|tar xf -
222 gunzip -c yazpp-version.tar.gz|tar xf -
231 <title id="boost.fromsource">Boost</title>
233 Metaproxy needs components thread and test from
237 gunzip -c boost-version.tar.gz|tar xf -
239 ./configure --with-libraries=thread,test --with-toolset=gcc
245 However, under the hood bjam is used. You can invoke that with
248 ./bjam --toolset=gcc --with-thread --with-test stage
251 Replace <literal>stage</literal> with <literal>clean</literal> /
252 <literal>install</literal> to perform clean and install respectively.
255 Add <literal>--prefix=DIR</literal> to install Boost in other
256 prefix than <literal>/usr/local</literal>.
259 <section id="metaproxy.fromsource">
260 <title>Metaproxy</title>
262 gunzip -c metaproxy-version.tar.gz|tar xf -
270 You may have to tell configure where Boost is installed by supplying
271 options <literal>--with-boost</literal> and <literal>--with-boost-toolset</literal>.
272 The former sets the PREFIX for Boost (same as --prefix for Boost above).
273 The latter the compiler toolset (eg. gcc34).
276 Pass <literal>--help</literal> to configure to get a list of
282 <section id="installation.debian">
283 <title>Installation on Debian GNU/Linux</title>
285 All dependencies for Metaproxy are available as
286 <ulink url="&url.debian;">Debian</ulink>
287 packages for the sarge (stable in 2005) and etch (testing in 2005)
291 The procedures for Debian based systems, such as
292 <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
295 There is currently no official Debian package for YAZ++.
296 And the official Debian package for YAZ is probably too old.
297 But Index Data bulds "new" versions of those for Debian (i386 only).
300 Update the <filename>/etc/apt/sources.list</filename>
301 to include the Index Data repository.
302 See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
303 for more information.
306 apt-get install libxslt1-dev
307 apt-get install libyazpp2-dev
308 apt-get install libboost-dev
309 apt-get install libboost-thread-dev
310 apt-get install libboost-test-dev
313 With these packages installed, the usual configure + make
314 procedure can be used for Metaproxy as outlined in
315 <xref linkend="installation.unix"/>.
319 <section id="installation.rpm">
320 <title>Installation on RPM based Linux Systems</title>
322 All external dependencies for Metaproxy are available as
323 RPM packages, either from your distribution site, or from the
324 <ulink url="http://fr.rpmfind.net/">RPMfind</ulink> site.
327 For example, an installation of the requires Boost C++ development
328 libraries on RedHat Fedora C4 and C5 can be done like this:
330 wget ftp://fr.rpmfind.net/wlinux/fedora/core/updates/testing/4/SRPMS/boost-1.33.0-3.fc4.src.rpm
331 sudo rpmbuild --buildroot src/ --rebuild -p fc4/boost-1.33.0-3.fc4.src.rpm
332 sudo rpm -U /usr/src/redhat/RPMS/i386/boost-*rpm
336 The <ulink url="&url.yaz;">YAZ</ulink> library is needed to
337 compile &metaproxy;, see there
338 for more information on available RPM packages.
341 There is currently no official RPM package for YAZ++.
342 See the <ulink url="&url.yazplusplus;">YAZ++</ulink> pages
343 for more information on a Unix tarball install.
346 With these packages installed, the usual configure + make
347 procedure can be used for Metaproxy as outlined in
348 <xref linkend="installation.unix"/>.
352 <section id="installation.windows">
353 <title>Installation on Windows</title>
355 Metaproxy can be compiled with Microsoft
356 <ulink url="&url.vstudio;">Visual Studio</ulink>.
357 Versions 2003 (C 7.1), 2005 (C 8.0) and 2008 (C 9.0) is known to work.
359 <section id="installation.windows.boost">
362 For Windows, it's easiest to get the precompiled Boost
363 package from <ulink url="&url.boost.windows.download;">here</ulink>.
364 Several versions of the Boost libraries may be selected when
365 installing Boost for windows. Please choose at least the
366 <emphasis>multithreaded</emphasis> (non-DLL) version because
367 the Metaproxy makefile uses that.
370 For more information about installing Boost refer to the
371 <ulink url="&url.boost.getting.started;">getting started</ulink>
376 <section id="installation.windows.libxslt">
377 <title>Libxslt</title>
379 <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
381 <ulink url="&url.libxml2.download.win32;">here</ulink>.
384 Libxslt has other dependencies, but these can all be downloaded
385 from the same site. Get the following package:
386 iconv, zlib, libxml2, libxslt.
390 <section id="installation.windows.yaz">
393 <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
395 <ulink url="&url.yaz.download.win32;">here</ulink>.
399 <section id="installation.windows.yazplusplus">
402 Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
403 Version 1.1.0 or later is required.
406 YAZ++ includes NMAKE makefiles, similar to those found in the
411 <section id="installation.windows.metaproxy">
412 <title>Metaproxy</title>
414 Metaproxy is shipped with NMAKE makefiles as well - similar
415 to those found in the YAZ++/YAZ packages. Adjust this Makefile
416 to point to the proper locations of Boost, Libxslt, Libxml2,
417 zlib, iconv, yaz and yazpp.
421 <varlistentry><term><literal>DEBUG</literal></term>
423 If set to 1, the software is
424 compiled with debugging libraries (code generation is
425 multi-threaded debug DLL).
426 If set to 0, the software is compiled with release libraries
427 (code generation is multi-threaded DLL).
432 <term><literal>BOOST</literal></term>
435 Boost install location
441 <term><literal>BOOST_VERSION</literal></term>
444 Boost version (replace . with _).
450 <term><literal>BOOST_TOOLSET</literal></term>
459 <term><literal>LIBXSLT_DIR</literal>,
460 <literal>LIBXML2_DIR</literal> ..</term>
463 Specify the locations of Libxslt, libiconv, libxml2 and
472 After successful compilation you'll find
473 <literal>metaproxy.exe</literal> in the
474 <literal>bin</literal> directory.
482 <chapter id="yazproxy-comparison">
483 <title>YAZ Proxy Comparison</title>
485 The table below lists facilities either supported by either
486 <ulink url="&url.yazproxy;">YAZ Proxy</ulink> or Metaproxy.
488 <table id="yazproxy-comparison-table">
489 <title>Metaproxy / YAZ Proxy comparison</title>
493 <entry>Facility</entry>
494 <entry>Metaproxy</entry>
495 <entry>YAZ Proxy</entry>
500 <entry>Z39.50 server</entry>
501 <entry>Using filter <literal>frontend_net</literal></entry>
502 <entry>Supported</entry>
505 <entry>SRU server</entry>
506 <entry>Supported with filter <literal>sru_z3950</literal></entry>
507 <entry>Supported</entry>
510 <entry>Z39.50 client</entry>
511 <entry>Supported with filter <literal>z3950_client</literal></entry>
512 <entry>Supported</entry>
515 <entry>SRU client</entry>
516 <entry>Unsupported</entry>
517 <entry>Unsupported</entry>
520 <entry>Connection reuse</entry>
521 <entry>Supported with filter <literal>session_shared</literal></entry>
522 <entry>Supported</entry>
525 <entry>Connection share</entry>
526 <entry>Supported with filter <literal>session_shared</literal></entry>
527 <entry>Unsupported</entry>
530 <entry>Result set reuse</entry>
531 <entry>Supported with filter <literal>session_shared</literal></entry>
532 <entry>Within one Z39.50 session / HTTP keep-alive</entry>
535 <entry>Record cache</entry>
536 <entry>Unsupported</entry>
537 <entry>Supported for last result set within one Z39.50/HTTP-keep alive session</entry>
540 <entry>Z39.50 Virtual database, i.e. select any Z39.50 target for database</entry>
541 <entry>Supported with filter <literal>virt_db</literal></entry>
542 <entry>Unsupported</entry>
545 <entry>SRU Virtual database, i.e. select any Z39.50 target for path</entry>
546 <entry>Supported with filter <literal>virt_db</literal>,
547 <literal>sru_z3950</literal></entry>
548 <entry>Supported</entry>
551 <entry>Multi target search</entry>
552 <entry>Supported with filter <literal>multi</literal> (round-robin)</entry>
553 <entry>Unsupported</entry>
556 <entry>Retrieval and search limits</entry>
557 <entry>Unsupported</entry>
558 <entry>Supported</entry>
561 <entry>Bandwidth limits</entry>
562 <entry>Unsupported</entry>
563 <entry>Supported</entry>
566 <entry>Connect limits</entry>
567 <entry>Unsupported</entry>
568 <entry>Supported</entry>
571 <entry>Retrieval sanity check and conversions</entry>
572 <entry>Supported using filter <literal>record_transform</literal></entry>
573 <entry>Supported</entry>
576 <entry>Query check</entry>
578 Supported in a limited way using <literal>query_rewrite</literal>
580 <entry>Supported</entry>
583 <entry>Query rewrite</entry>
584 <entry>Supported with <literal>query_rewrite</literal></entry>
585 <entry>Unsupported</entry>
588 <entry>Session invalidate for -1 hits</entry>
589 <entry>Unsupported</entry>
590 <entry>Supported</entry>
593 <entry>Architecture</entry>
594 <entry>Multi-threaded + select for networked modules such as
595 <literal>frontend_net</literal>)</entry>
596 <entry>Single-threaded using select</entry>
600 <entry>Extensability</entry>
601 <entry>Most functionality implemented as loadable modules</entry>
602 <entry>Unsupported and experimental</entry>
606 <entry><ulink url="&url.usemarcon;">USEMARCON</ulink></entry>
607 <entry>Unsupported</entry>
608 <entry>Supported</entry>
612 <entry>Portability</entry>
614 Requires YAZ, YAZ++ and modern C++ compiler supporting
615 <ulink url="&url.boost;">Boost</ulink>.
618 Requires YAZ and YAZ++.
619 STL is not required so pretty much any C++ compiler out there should work.
628 <chapter id="architecture">
629 <title>The Metaproxy Architecture</title>
631 The Metaproxy architecture is based on three concepts:
632 the <emphasis>package</emphasis>,
633 the <emphasis>route</emphasis>
634 and the <emphasis>filter</emphasis>.
638 <term>Packages</term>
641 A package is request or response, encoded in some protocol,
642 issued by a client, making its way through Metaproxy, send to or
643 received from a server, or sent back to the client.
646 The core of a package is the protocol unit - for example, a
647 Z39.50 Init Request or Search Response, or an SRU searchRetrieve
648 URL or Explain Response. In addition to this core, a package
649 also carries some extra information added and used by Metaproxy
653 In general, packages are doctored as they pass through
654 Metaproxy. For example, when the proxy performs authentication
655 and authorization on a Z39.50 Init request, it removes the
656 authentication credentials from the package so that they are not
657 passed onto the back-end server; and when search-response
658 packages are obtained from multiple servers, they are merged
659 into a single unified package that makes its way back to the
668 Packages make their way through routes, which can be thought of
669 as programs that operate on the package data-type. Each
670 incoming package initially makes its way through a default
671 route, but may be switched to a different route based on various
672 considerations. Routes are made up of sequences of filters (see
681 Filters provide the individual instructions within a route, and
682 effect the necessary transformations on packages. A particular
683 configuration of Metaproxy is essentially a set of filters,
684 described by configuration details and arranged in order in one
685 or more routes. There are many kinds of filter - about a dozen
686 at the time of writing with more appearing all the time - each
687 performing a specific function and configured by different
691 The word ``filter'' is sometimes used rather loosely, in two
692 different ways: it may be used to mean a particular
693 <emphasis>type</emphasis> of filter, as when we speak of ``the
694 auth_simple filter'' or ``the multi filter''; or it may be used
695 to be a specific <emphasis>instance</emphasis> of a filter
696 within a Metaproxy configuration. For example, a single
697 configuration will often contain multiple instances of the
698 <literal>z3950_client</literal> filter. In
699 operational terms, of these is a separate filter. In practice,
700 context always make it clear which sense of the word ``filter''
704 Extensibility of Metaproxy is primarily through the creation of
705 plugins that provide new filters. The filter API is small and
706 conceptually simple, but there are many details to master. See
708 <link linkend="filters">Filters</link>.
714 Since packages are created and handled by the system itself, and
715 routes are conceptually simple, most of the remainder of this
716 document concentrates on filters. After a brief overview of the
717 filter types follows, along with some thoughts on possible future
724 <chapter id="filters">
725 <title>Filters</title>
728 <section id="filters-introductory-notes">
729 <title>Introductory notes</title>
731 It's useful to think of Metaproxy as an interpreter providing a small
732 number of primitives and operations, but operating on a very
733 complex data type, namely the ``package''.
736 A package represents a Z39.50 or SRU/W request (whether for Init,
737 Search, Scan, etc.) together with information about where it came
738 from. Packages are created by front-end filters such as
739 <literal>frontend_net</literal> (see below), which reads them from
740 the network; other front-end filters are possible. They then pass
741 along a route consisting of a sequence of filters, each of which
742 transforms the package and may also have side-effects such as
743 generating logging. Eventually, the route will yield a response,
744 which is sent back to the origin.
747 There are many kinds of filter: some that are defined statically
748 as part of Metaproxy, and others may be provided by third parties
749 and dynamically loaded. They all conform to the same simple API
750 of essentially two methods: <function>configure()</function> is
751 called at startup time, and is passed an XML DOM tree representing that
752 part of the configuration file that pertains to this filter
753 instance: it is expected to walk that tree extracting relevant
754 information; and <function>process()</function> is called every
755 time the filter has to processes a package.
758 While all filters provide the same API, there are different modes
759 of functionality. Some filters are sources: they create
761 (<literal>frontend_net</literal>);
762 others are sinks: they consume packages and return a result
763 (<literal>backend_test</literal>,
764 <literal>bounce</literal>,
765 <literal>http_file</literal>,
766 <literal>z3950_client</literal>);
767 the others are true filters, that read, process and pass on the
768 packages they are fed
769 (<literal>auth_simple</literal>,
770 <literal>log</literal>,
771 <literal>multi</literal>,
772 <literal>query_rewrite</literal>,
773 <literal>record_transform</literal>,
774 <literal>session_shared</literal>,
775 <literal>sru_z3950</literal>,
776 <literal>template</literal>,
777 <literal>virt_db</literal>).
782 <section id="overview.filter.types">
783 <title>Overview of filter types</title>
785 We now briefly consider each of the types of filter supported by
786 the core Metaproxy binary. This overview is intended to give a
787 flavor of the available functionality; more detailed information
788 about each type of filter is included below in
789 <xref linkend="reference"/>.
792 The filters are here named by the string that is used as the
793 <literal>type</literal> attribute of a
794 <literal><filter></literal> element in the configuration
795 file to request them, with the name of the class that implements
796 them in parentheses. (The classname is not needed for normal
797 configuration and use of Metaproxy; it is useful only to
801 The filters are here listed in alphabetical order:
808 New virt_db-alike that does inteligent peer choice, explain merging,
809 adds FD&N to explain. Keeps init responses (like "virt_db Classic"),
810 makes routing choices based on local explain knowledge. Ref IDDI
813 Filter to convert Explain Classic to ZeeRex.
815 CQL2PQF (which needs augmented ZeeRex) - MARC for Talis.
819 Figure out what additional information we need in:
820 ZeeRex (check against D3.1)
821 Init request (e.g. loop detection)
822 Query package (e.g. number of hops)
823 Query response (e.g. record source)
827 <section id="auth_simple">
828 <title><literal>auth_simple</literal>
829 (mp::filter::AuthSimple)</title>
831 Simple authentication and authorization. The configuration
832 specifies the name of a file that is the user register, which
833 lists <varname>username</varname>:<varname>password</varname>
834 pairs, one per line, colon separated. When a session begins, it
835 is rejected unless username and passsword are supplied, and match
836 a pair in the register. The configuration file may also specific
837 the name of another file that is the target register: this lists
838 lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
839 sets, one per line, with multiple database names separated by
840 commas. When a search is processed, it is rejected unless the
841 database to be searched is one of those listed as available to
846 <section id="backend_test">
847 <title><literal>backend_test</literal>
848 (mp::filter::Backend_test)</title>
850 A partial sink that provides dummy responses in the manner of the
851 <literal>yaz-ztest</literal> Z39.50 server. This is useful only
852 for testing. Seriously, you don't need this. Pretend you didn't
853 even read this section.
857 <section id="bounce">
858 <title><literal>bounce</literal>
859 (mp::filter::Bounce)</title>
861 A sink that swallows <emphasis>all packages</emphasis>,
862 and returns them almost unprocessed.
863 It never sends any package of any type further down the row, but
864 sets Z39.50 packages to Z_Close, and HTTP_Request packages to
865 HTTP_Response err code 400 packages, and adds a suitable bounce
867 The bounce filter is usually added at end of each filter chain route
868 to prevent infinite hanging of for example HTTP
869 requests packages when only the Z39.50 client partial sink
870 filter is found in the
875 <section id="cql_rpn">
876 <title><literal>cql_rpn</literal>
877 (mp::filter::CQLtoRPN)</title>
879 A query language transforming filter which catches Z39.50
880 <literal>searchRequest</literal>
881 packages containing <literal>CQL</literal> queries, transforms
882 those to <literal>RPN</literal> queries,
883 and sends the <literal>searchRequests</literal> on to the next
884 filters. It is among other things useful in a SRU context.
888 <section id="frontend_net">
889 <title><literal>frontend_net</literal>
890 (mp::filter::FrontendNet)</title>
892 A source that accepts Z39.50 connections from a port
893 specified in the configuration, reads protocol units, and
894 feeds them into the next filter in the route. When the result is
895 received, it is returned to the original origin.
899 <section id="http_file">
900 <title><literal>http_file</literal>
901 (mp::filter::HttpFile)</title>
903 A partial sink which swallows only
904 <literal>HTTP_Request</literal> packages, and
905 returns the contents of files from the local
906 filesystem in response to HTTP requests.
907 It lets Z39.50 packages and all other forthcoming package types
910 does mean that Metaproxy is also a Web-server in its spare time. So
911 far it does not contain either an email-reader or a Lisp
912 interpreter, but that day is surely coming.)
916 <section id="load_balance">
917 <title><literal>load_balance</literal>
918 (mp::filter::LoadBalance)</title>
920 Performs load balancing for incoming Z39.50 init requests.
921 It is used together with the <literal>virt_db</literal> filter,
922 but unlike the <literal>multi</literal> filter it does send an
923 entire session to only one of the virtual backends. The
924 <literal>load_balance</literal> filter is assuming that
925 all backend targets have equal content, and chooses the backend
926 with least load cost for a new session.
929 This filter is experimental and yet not mature for heavy load
937 <title><literal>log</literal>
938 (mp::filter::Log)</title>
940 Writes logging information to standard output, and passes on
941 the package unchanged. A log file name can be specified, as well
942 as multiple different logging formats.
947 <title><literal>multi</literal>
948 (mp::filter::Multi)</title>
950 Performs multi-database searching.
952 <link linkend="multidb">the extended discussion</link>
953 of virtual databases and multi-database searching below.
957 <section id="query_rewrite">
958 <title><literal>query_rewrite</literal>
959 (mp::filter::QueryRewrite)</title>
961 Rewrites Z39.50 <literal>Type-1</literal>
962 and <literal>Type-101</literal> (``<literal>RPN</literal>'')
964 three-step process: the query is transliterated from Z39.50
965 packet structures into an XML representation; that XML
966 representation is transformed by an XSLT stylesheet; and the
967 resulting XML is transliterated back into the Z39.50 packet
973 <section id="record_transform">
974 <title><literal>record_transform</literal>
975 (mp::filter::RecordTransform)</title>
977 This filter acts only on Z3950 present requests, and let all
978 other types of packages and requests pass untouched. It's use is
979 twofold: blocking Z3950 present requests, which the backend
980 server does not understand and can not honor, and transforming
981 the present syntax and elementset name according to the rules
982 specified, to fetch only existing record formats, and transform
983 them on the fly to requested record syntaxes.
987 <section id="session_shared">
988 <title><literal>session_shared</literal>
989 (mp::filter::SessionShared)</title>
991 This filter implements global sharing of
992 result sets (i.e. between threads and therefore between
993 clients), yielding performance improvements by clever resource
998 <section id="sru_z3950">
999 <title><literal>sru_z3950</literal>
1000 (mp::filter::SRUtoZ3950)</title>
1002 This filter transforms valid
1003 SRU GET/POST/SOAP searchRetrieve requests to Z3950 init, search,
1004 and present requests, and wraps the
1005 received hit counts and XML records into suitable SRU response
1007 The <literal>sru_z3950</literal> filter processes also SRU
1008 GET/POST/SOAP explain requests, returning
1009 either the absolute minimum required by the standard, or a full
1010 pre-defined ZeeReX explain record.
1012 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
1013 standard pages and the
1014 <ulink url="&url.sru.explain;">SRU Explain</ulink> pages
1015 for more information on the correct explain syntax.
1016 SRU scan requests are not supported yet.
1020 <section id="template">
1021 <title><literal>template</literal>
1022 (mp::filter::Template)</title>
1024 Does nothing at all, merely passing the packet on. (Maybe it
1025 should be called <literal>nop</literal> or
1026 <literal>passthrough</literal>?) This exists not to be used, but
1027 to be copied - to become the skeleton of new filters as they are
1028 written. As with <literal>backend_test</literal>, this is not
1029 intended for civilians.
1033 <section id="virt_db">
1034 <title><literal>virt_db</literal>
1035 (mp::filter::VirtualDB)</title>
1037 Performs virtual database selection: based on the name of the
1038 database in the search request, a server is selected, and its
1039 address added to the request in a <literal>VAL_PROXY</literal>
1040 otherInfo packet. It will subsequently be used by a
1041 <literal>z3950_client</literal> filter.
1043 <link linkend="multidb">the extended discussion</link>
1044 of virtual databases and multi-database searching below.
1048 <section id="z3950_client">
1049 <title><literal>z3950_client</literal>
1050 (mp::filter::Z3950Client)</title>
1052 A partial sink which swallows only Z39.50 packages.
1053 It performs Z39.50 searching and retrieval by proxying the
1054 packages that are passed to it. Init requests are sent to the
1055 address specified in the <literal>VAL_PROXY</literal> otherInfo
1056 attached to the request: this may have been specified by client,
1057 or generated by a <literal>virt_db</literal> filter earlier in
1058 the route. Subsequent requests are sent to the same address,
1059 which is remembered at Init time in a Session object.
1060 HTTP_Request packages and all other forthcoming package types
1061 are passed untouched.
1066 <section id="zeerex_explain">
1067 <title><literal>zeerex_explain</literal>
1068 (mp::filter::ZeerexExplain)</title>
1070 This filter acts as a sink for
1071 Z39.50 explain requests, returning a static ZeeReX
1072 Explain XML record from the config section. All other packages
1075 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
1077 for more information on the correct explain syntax.
1081 This filter is not yet completed.
1090 <section id="future.directions">
1091 <title>Future directions</title>
1093 Some other filters that do not yet exist, but which would be
1094 useful, are briefly described. These may be added in future
1095 releases (or may be created by third parties, as loadable
1101 <term><literal>frontend_cli</literal> (source)</term>
1104 Command-line interface for generating requests.
1109 <term><literal>sru_client</literal> (sink)</term>
1112 SRU/GET and SRU/SOAP searching and retrieval.
1117 <term><literal>opensearch_client</literal> (sink)</term>
1120 A9 OpenSearch searching and retrieval.
1130 <chapter id="configuration">
1131 <title>Configuration: the Metaproxy configuration file format</title>
1134 <section id="configuration-introductory-notes">
1135 <title>Introductory notes</title>
1137 If Metaproxy is an interpreter providing operations on packages, then
1138 its configuration file can be thought of as a program for that
1139 interpreter. Configuration is by means of a single XML file, the name
1140 of which is supplied as the sole command-line argument to the
1141 <command>metaproxy</command> program. (See
1142 <xref linkend="reference"/> below for more information on invoking
1147 <section id="overview.xml.structure">
1148 <title>Overview of the config file XML structure</title>
1150 All elements and attributes are in the namespace
1151 <ulink url="http://indexdata.com/metaproxy"/>.
1152 This is most easily achieved by setting the default namespace on
1153 the top-level element, as here:
1156 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1159 The top-level element is <metaproxy>. This contains
1160 a <dlpath> element,
1161 a <start> element,
1162 a <filters> element and
1163 a <routes> element, in that order. <dlpath> and
1164 <filters> are optional; the other two are mandatory.
1165 All four are non-repeatable.
1168 The <dlpath;> element contains a text element which
1169 specifies the location of filter modules. This is only needed
1170 if Metaproxy must load 3rd party filters (most filters with Metaproxy
1171 are built into the Metaproxy application).
1174 The <start> element is empty, but carries a
1175 <literal>route</literal> attribute, whose value is the name of
1176 route at which to start running - analogous to the name of the
1177 start production in a formal grammar.
1180 If present, <filters> contains zero or more <filter>
1181 elements. Each filter carries a <literal>type</literal> attribute
1182 which specifies what kind of filter is being defined
1183 (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
1184 and contain various elements that provide suitable configuration
1185 for a filter of its type. The filter-specific elements are
1187 <xref linkend="reference"/>.
1188 Filters defined in this part of the file must carry an
1189 <literal>id</literal> attribute so that they can be referenced
1193 <routes> contains one or more <route> elements, each
1194 of which must carry an <literal>id</literal> element. One of the
1195 routes must have the ID value that was specified as the start
1196 route in the <start> element's <literal>route</literal>
1197 attribute. Each route contains zero or more <filter>
1198 elements. These are of two types. They may be empty, but carry a
1199 <literal>refid</literal> attribute whose value is the same as the
1200 <literal>id</literal> of a filter previously defined in the
1201 <filters> section. Alternatively, a route within a filter
1202 may omit the <literal>refid</literal> attribute, but contain
1203 configuration elements similar to those used for filters defined
1204 in the <filters> section. (In other words, each filter in a
1205 route may be included either by reference or by physical
1211 <section id="example.configuration">
1212 <title>An example configuration</title>
1214 The following is a small, but complete, Metaproxy configuration
1215 file (included in the distribution as
1216 <literal>metaproxy/etc/config1.xml</literal>).
1217 This file defines a very simple configuration that simply proxies
1218 to whatever back-end server the client requests, but logs each
1219 request and response. This can be useful for debugging complex
1220 client-server dialogues.
1222 <screen><![CDATA[<?xml version="1.0"?>
1223 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1224 <dlpath>/usr/lib/metaproxy/modules</dlpath>
1225 <start route="start"/>
1227 <filter id="frontend" type="frontend_net">
1230 <filter id="backend" type="z3950_client">
1235 <filter refid="frontend"/>
1236 <filter type="log"/>
1237 <filter refid="backend"/>
1238 <filter type="bounce"/>
1244 It works by defining a single route, called
1245 <literal>start</literal>, which consists of a sequence of four
1246 filters. The first and last of these are included by reference:
1247 their <literal><filter></literal> elements have
1248 <literal>refid</literal> attributes that refer to filters defined
1249 within the prior <literal><filters></literal> section. The
1250 middle filter is included inline in the route.
1253 The four filters in the route are as follows: first, a
1254 <literal>frontend_net</literal> filter accepts Z39.50 requests
1255 from any host on port 9000; then these requests are passed through
1256 a <literal>log</literal> filter that emits a message for each
1257 request; they are then fed into a <literal>z3950_client</literal>
1258 filter, which forwards all Z39.50 requests to the client-specified
1259 back-end Z39.509 server. Those Z39.50 packages are returned by the
1260 <literal>z3950_client</literal> filter, with the response data
1261 filled by the external Z39.50 server targeted.
1262 All non-Z39.50 packages are passed through to the
1263 <literal>bounce</literal> filter, which definitely bounces
1264 everything, including fish, bananas, cold pyjamas,
1265 mutton, beef and trout packages.
1266 When the response arrives, it is handed
1267 back to the <literal>log</literal> filter, which emits another
1268 message; and then to the <literal>frontend_net</literal> filter,
1269 which returns the response to the client.
1273 <section id="config-file-modularity">
1274 <title>Config file modularity</title>
1276 Metaproxy XML configuration snippets can be reused by other
1277 filters using the <literal>XInclude</literal> standard, as seen in
1278 the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU
1281 <filter id="sru" type="sru_z3950">
1282 <database name="Default">
1283 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
1284 href="explain.xml"/>
1291 <section id="config-file-syntax-check">
1292 <title>Config file syntax checking</title>
1294 The distribution contains RelaxNG Compact and XML syntax checking
1295 files, as well as XML Schema files. These are found in the
1298 xml/schema/metaproxy.rnc
1299 xml/schema/metaproxy.rng
1300 xml/schema/metaproxy.xsd
1302 and can be used to verify or debug the XML structure of
1303 configuration files. For example, using the utility
1304 <filename>xmllint</filename>, syntax checking is done like this:
1306 xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml
1307 xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml
1309 (A recent version of <literal>libxml2</literal> is required, as
1310 support for XML Schemas is a relatively recent addition.)
1313 You can of course use any other RelaxNG or XML Schema compliant tool
1321 <chapter id="multidb">
1322 <title>Virtual databases and multi-database searching</title>
1325 <section id="multidb-introductory-notes">
1326 <title>Introductory notes</title>
1328 Two of Metaproxy's filters are concerned with multiple-database
1329 operations. Of these, <literal>virt_db</literal> can work alone
1330 to control the routing of searches to one of a number of servers,
1331 while <literal>multi</literal> can work together with
1332 <literal>virt_db</literal> to perform multi-database searching, merging
1333 the results into a unified result-set - ``metasearch in a box''.
1336 The interaction between
1337 these two filters is necessarily complex: it reflects the real,
1338 irreducible complexity of multi-database searching in a protocol such
1339 as Z39.50 that separates initialization from searching, and in
1340 which the database to be searched is not known at initialization
1344 It's possible to use these filters without understanding the
1345 details of their functioning and the interaction between them; the
1346 next two sections of this chapter are ``HOW-TO'' guides for doing
1347 just that. However, debugging complex configurations will require
1348 a deeper understanding, which the last two sections of this
1349 chapters attempt to provide.
1354 <section id="multidb.virt_db">
1355 <title>Virtual databases with the <literal>virt_db</literal> filter</title>
1357 Working alone, the purpose of the
1358 <literal>virt_db</literal>
1359 filter is to route search requests to one of a selection of
1360 back-end databases. In this way, a single Z39.50 endpoint
1361 (running Metaproxy) can provide access to several different
1362 underlying services, including those that would otherwise be
1363 inaccessible due to firewalls. In many useful configurations, the
1364 back-end databases are local to the Metaproxy installation, but
1365 the software does not enforce this, and any valid Z39.50 servers
1366 may be used as back-ends.
1369 For example, a <literal>virt_db</literal>
1370 filter could be set up so that searches in the virtual database
1371 ``lc'' are forwarded to the Library of Congress bibliographic
1372 catalogue server, and searches in the virtual database ``marc''
1373 are forwarded to the toy database of MARC records that Index Data
1374 hosts for testing purposes. A <literal>virt_db</literal>
1375 configuration to make this switch would look like this:
1377 <screen><![CDATA[<filter type="virt_db">
1379 <database>lc</database>
1380 <target>z3950.loc.gov:7090/voyager</target>
1383 <database>marc</database>
1384 <target>indexdata.com/marc</target>
1386 </filter>]]></screen>
1388 As well as being useful in it own right, this filter also provides
1389 the foundation for multi-database searching.
1394 <section id="multidb.multi">
1395 <title>Multi-database search with the <literal>multi</literal> filter</title>
1397 To arrange for Metaproxy to broadcast searches to multiple back-end
1398 servers, the configuration needs to include two components: a
1399 <literal>virt_db</literal>
1400 filter that specifies multiple
1401 <literal><target></literal>
1402 elements, and a subsequent
1403 <literal>multi</literal>
1404 filter. Here, for example, is a complete configuration that
1405 broadcasts searches to both the Library of Congress catalogue and
1406 Index Data's tiny testing database of MARC records:
1408 <screen><![CDATA[<?xml version="1.0"?>
1409 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1410 <start route="start"/>
1413 <filter type="frontend_net">
1414 <threads>10</threads>
1417 <filter type="virt_db">
1419 <database>lc</database>
1420 <target>z3950.loc.gov:7090/voyager</target>
1423 <database>marc</database>
1424 <target>indexdata.com/marc</target>
1427 <database>all</database>
1428 <target>z3950.loc.gov:7090/voyager</target>
1429 <target>indexdata.com/marc</target>
1432 <filter type="multi"/>
1433 <filter type="z3950_client">
1434 <timeout>30</timeout>
1436 <filter type="bounce"/>
1439 </metaproxy>]]></screen>
1442 <literal>virt_db</literal>
1443 filter that specifies multiple
1444 <literal><target></literal>
1445 elements but without a subsequent
1446 <literal>multi</literal>
1447 filter yields surprising and undesirable results, as will be
1448 described below. Don't do that.)
1451 Metaproxy can be invoked with this configuration as follows:
1453 <screen>../src/metaproxy --config config-simple-multi.xml</screen>
1455 And thereafter, Z39.50 clients can connect to the running server
1456 (on port 9000, as specified in the configuration) and search in
1457 any of the databases
1458 <literal>lc</literal> (the Library of Congress catalogue),
1459 <literal>marc</literal> (Index Data's test database of MARC records)
1461 <literal>all</literal> (both of these). As an example, a session
1462 using the YAZ command-line client <literal>yaz-client</literal> is
1463 here included (edited for brevity and clarity):
1465 <screen><![CDATA[$ yaz-client @:9000
1469 Search was a success.
1470 Number of hits: 10000, setno 1
1474 Search was a success.
1475 Number of hits: 10, setno 3
1479 Search was a success.
1480 Number of hits: 10010, setno 4
1483 [marc]Record type: USmarc
1486 005 00000000000000.0
1487 008 910710c19910701nju 00010 eng
1491 100 10 $a Jack Collins
1492 245 10 $a How to program a computer
1498 [VOYAGER]Record type: USmarc
1500 005 20041229102447.0
1501 008 030910s2004 caua 000 0 eng
1502 035 $a (DLC) 2003112666
1503 906 $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
1504 925 0 $a acquire $b 1 shelf copy $x policy default
1505 955 $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
1508 040 $a DLC $c DLC $d DLC
1509 050 00 $a MLCM 2004/03312 (G)
1510 245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
1511 246 3 $a Double-O-seven, everything or nothing
1512 246 30 $a Prima's official strategy guide
1513 260 $a Roseville, CA : $b Prima Games, $c c2004.
1514 300 $a 161 p. : $b col. ill. ; $c 28 cm.
1515 500 $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
1516 650 0 $a Video games.
1517 710 2 $a Kaizen Media Group.
1518 856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
1523 As can be seen, the first record in the result set is from the
1524 Index Data test database, and the second from the Library of
1525 Congress database. The result-set continues alternating records
1526 round-robin style until the point where one of the databases'
1527 records are exhausted.
1530 This example uses only two back-end databases; more may be used.
1531 There is no limitation imposed on the number of databases that may
1532 be metasearched in this way: issues of resource usage and
1533 administrative complexity dictate the practical limits.
1536 What happens when one of the databases doesn't respond? By default,
1537 the entire multi-database search fails, and the appropriate
1538 diagnostic is returned to the client. This is usually appropriate
1539 during development, when technicians need maximum information, but
1540 can be inconvenient in deployment, when users typically don't want
1541 to be bothered with problems of this kind and prefer just to get
1542 the records from the databases that are available. To obtain this
1543 latter behavior add an empty
1544 <literal><hideunavailable></literal>
1546 <literal>multi</literal> filter:
1548 <screen><![CDATA[ <filter type="multi">
1550 </filter>]]></screen>
1552 Under this regime, an error is reported to the client only if
1553 <emphasis>all</emphasis> the databases in a multi-database search
1559 <section id="multidb.what">
1560 <title>What's going on?</title>
1562 <title>Lark's vomit</title>
1564 This section goes into a level of technical detail that is
1565 probably not necessary in order to configure and use Metaproxy.
1566 It is provided only for those who like to know how things work.
1567 You should feel free to skip on to the next section if this one
1568 doesn't seem like fun.
1572 Hold on tight - this may get a little hairy.
1575 In the general course of things, a Z39.50 Init request may carry
1576 with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
1577 whose value indicates the address of a Z39.50 server to which the
1578 ultimate connection is to be made. (This otherInfo packet is
1579 supported by YAZ-based Z39.50 clients and servers, but has not yet
1580 been ratified by the Maintenance Agency and so is not widely used
1581 in non-Index Data software. We're working on it.)
1582 The <literal>VAL_PROXY</literal> packet functions
1583 analogously to the absoluteURI-style Request-URI used with the GET
1584 method when a web browser asks a proxy to forward its request: see
1586 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
1587 >Request-URI</ulink>
1589 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
1590 >the HTTP 1.1 specification</ulink>.
1593 Within Metaproxy, Search requests that are part of the same
1594 session as an Init request that carries a
1595 <literal>VAL_PROXY</literal> otherInfo are also annotated with the
1596 same information. The role of the <literal>virt_db</literal>
1597 filter is to rewrite this otherInfo packet dependent on the
1598 virtual database that the client wants to search.
1601 When Metaproxy receives a Z39.50 Init request from a client, it
1602 doesn't immediately forward that request to the back-end server.
1603 Why not? Because it doesn't know <emphasis>which</emphasis>
1604 back-end server to forward it to until the client sends a Search
1605 request that specifies the database that it wants to search in.
1606 Instead, it just treasures the Init request up in its heart; and,
1607 later, the first time the client does a search on one of the
1608 specified virtual databases, a connection is forged to the
1609 appropriate server and the Init request is forwarded to it. If,
1610 later in the session, the same client searches in a different
1611 virtual database, then a connection is forged to the server that
1612 hosts it, and the same cached Init request is forwarded there,
1616 All of this clever Init-delaying is done by the
1617 <literal>frontend_net</literal> filter. The
1618 <literal>virt_db</literal> filter knows nothing about it; in
1619 fact, because the Init request that is received from the client
1620 doesn't get forwarded until a Search request is received, the
1621 <literal>virt_db</literal> filter (and the
1622 <literal>z3950_client</literal> filter behind it) doesn't even get
1623 invoked at Init time. The <emphasis>only</emphasis> thing that a
1624 <literal>virt_db</literal> filter ever does is rewrite the
1625 <literal>VAL_PROXY</literal> otherInfo in the requests that pass
1629 It is possible for a <literal>virt_db</literal> filter to contain
1631 <literal><target></literal>
1632 elements. What does this mean? Only that the filter will add
1633 multiple <literal>VAL_PROXY</literal> otherInfo packets to the
1634 Search requests that pass through it. That's because the virtual
1635 DB filter is dumb, and does exactly what it's told - no more, no
1637 If a Search request with multiple <literal>VAL_PROXY</literal>
1638 otherInfo packets reaches a <literal>z3950_client</literal>
1639 filter, this is an error. That filter doesn't know how to deal
1640 with multiple targets, so it will either just pick one and search
1641 in it, or (better) fail with an error message.
1644 The <literal>multi</literal> filter comes to the rescue! This is
1645 the only filter that knows how to deal with multiple
1646 <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
1647 making multiple copies of the entire Search request: one for each
1648 <literal>VAL_PROXY</literal>. Each of these new copies is then
1649 passed down through the remaining filters in the route. (The
1650 copies are handled in parallel though the
1651 spawning of new threads.) Since the copies each have only one
1652 <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
1653 <literal>z3950_client</literal> filter, which happily deals with
1654 each one individually. When the results of the individual
1655 searches come back up to the <literal>multi</literal> filter, it
1656 merges them into a single Search response, which is what
1657 eventually makes it back to the client.
1662 <section id="multidb.picture">
1663 <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
1667 <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
1670 <imagedata fileref="multi.png" format="PNG"/>
1673 <!-- Fall back if none of the images can be used -->
1675 [Here there should be a diagram showing the progress of
1676 packages through the filters during a simple virtual-database
1677 search and a multi-database search, but is seems that your
1678 tool chain has not been able to include the diagram in this
1682 <!-- ### This used to work with an older version of DocBook
1684 <para>Caption: progress of packages through filters.</para>
1687 </inlinemediaobject>
1693 <chapter id="sru-server">
1694 <title>Combined SRU webservice and Z39.50 server configuration</title>
1696 Metaproxy can act as
1697 <ulink url="&url.sru;">SRU</ulink> and
1698 web service server, which translates web service requests to
1699 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink> packages and
1700 sends them off to common available targets.
1703 A typical setup for this operation needs a filter route including the
1707 <table id="sru-server-table-config" frame="top">
1708 <title>SRU/Z39.50 Server Filter Route Configuration</title>
1712 <entry>Filter</entry>
1713 <entry>Importance</entry>
1714 <entry>Purpose</entry>
1720 <entry><literal>frontend_net</literal></entry>
1721 <entry>required</entry>
1722 <entry>Accepting HTTP connections and passing them to following
1723 filters. Since this filter also accepts Z39.50 connections, the
1724 server works as SRU and Z39.50 server on the same port.</entry>
1727 <entry><literal>sru_z3950</literal></entry>
1728 <entry>required</entry>
1729 <entry>Accepting SRU GET/POST/SOAP explain and
1730 searchRetrieve requests for the the configured databases.
1731 Explain requests are directly served from the static XML configuration.
1732 SearchRetrieve requests are
1733 transformed to Z39.50 search and present packages.
1734 All other HTTP and Z39.50 packages are passed unaltered.</entry>
1737 <entry><literal>http_file</literal></entry>
1738 <entry>optional</entry>
1739 <entry>Serving HTTP requests from the filesystem. This is only
1740 needed if the server should serve XSLT stylesheets, static HTML
1741 files or Java Script for thin browser based clients.
1742 Z39.50 packages are passed unaltered.</entry>
1745 <entry><literal>cql_rpn</literal></entry>
1746 <entry>required</entry>
1747 <entry>Usually, Z39.50 servers do not talk CQL, hence the
1748 translation of the CQL query language to RPN is mandatory in
1749 most cases. Affects only Z39.50 search packages.</entry>
1752 <entry><literal>record_transform</literal></entry>
1753 <entry>optional</entry>
1754 <entry>Some Z39.50 backend targets can not present XML record
1755 syntaxes in common wanted element sets. using this filter, one
1756 can transform binary MARC records to MARCXML records, and
1757 further transform those to any needed XML schema/format by XSLT
1758 transformations. Changes only Z39.50 present packages.</entry>
1761 <entry><literal>session_shared</literal></entry>
1762 <entry>optional</entry>
1763 <entry>The stateless nature of web services requires frequent
1764 re-searching of the same targets for display of paged result set
1765 records. This might be an unacceptable burden for the accessed
1766 backend Z39.50 targets, and this mosule can be added for
1767 efficient backend target resource pooling.</entry>
1770 <entry><literal>z3950_client</literal></entry>
1771 <entry>required</entry>
1772 <entry>Finally, a Z39.50 package sink is needed in the filter
1773 chain to provide the response packages. The Z39.50 client module
1774 is used to access external targets over the network, but any
1775 coming local Z39.50 package sink could be used instead of.</entry>
1778 <entry><literal>bounce</literal></entry>
1779 <entry>required</entry>
1780 <entry>Any Metaproxy package arriving here did not do so by
1781 purpose, and is bounced back with connection closure. this
1782 prevents inifinite package hanging inside the SRU server.</entry>
1788 A typical minimal example <ulink url="&url.sru;">SRU</ulink>
1789 server configuration file is found in the tarball distribution at
1790 <literal>etc/config-sru-to-z3950.xml</literal>.
1793 Off course, any other metaproxy modules can be integrated into a
1794 SRU server solution, including, but not limited to, load balancing,
1795 multiple target querying
1796 (see <xref linkend="multidb"/>), and complex RPN query rewrites.
1803 <chapter id="extensions">
1804 <title>Writing extensions for Metaproxy</title>
1805 <para>### To be written</para>
1811 <chapter id="classes">
1812 <title>Classes in the Metaproxy source code</title>
1815 <section id="classes-introductory-notes">
1816 <title>Introductory notes</title>
1818 <emphasis>Stop! Do not read this!</emphasis>
1819 You won't enjoy it at all. You should just skip ahead to
1820 <xref linkend="reference"/>,
1822 <!-- The remainder of this paragraph is lifted verbatim from
1823 Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
1824 you things you really need to know, like the fact that the
1825 fabulously beautiful planet Bethselamin is now so worried about
1826 the cumulative erosion by ten billion visiting tourists a year
1827 that any net imbalance between the amount you eat and the amount
1828 you excrete whilst on the planet is surgically removed from your
1829 bodyweight when you leave: so every time you go to the lavatory it
1830 is vitally important to get a receipt.
1833 This chapter contains documentation of the Metaproxy source code, and is
1834 of interest only to maintainers and developers. If you need to
1835 change Metaproxy's behavior or write a new filter, then you will most
1836 likely find this chapter helpful. Otherwise it's a waste of your
1837 good time. Seriously: go and watch a film or something.
1838 <citetitle>This is Spinal Tap</citetitle> is particularly good.
1841 Still here? OK, let's continue.
1844 In general, classes seem to be named big-endianly, so that
1845 <literal>FactoryFilter</literal> is not a filter that filters
1846 factories, but a factory that produces filters; and
1847 <literal>FactoryStatic</literal> is a factory for the statically
1848 registered filters (as opposed to those that are dynamically
1853 <section id="individual.classes">
1854 <title>Individual classes</title>
1856 The classes making up the Metaproxy application are here listed by
1857 class-name, with the names of the source files that define them in
1861 <section id="class-FactoryFilter">
1862 <title><literal>mp::FactoryFilter</literal>
1863 (<filename>factory_filter.cpp</filename>)</title>
1865 A factory class that exists primarily to provide the
1866 <literal>create()</literal> method, which takes the name of a
1867 filter class as its argument and returns a new filter of that
1868 type. To enable this, the factory must first be populated by
1869 calling <literal>add_creator()</literal> for static filters (this
1870 is done by the <literal>FactoryStatic</literal> class, see below)
1871 and <literal>add_creator_dyn()</literal> for filters loaded
1876 <section id="class-FactoryStatic">
1877 <title><literal>mp::FactoryStatic</literal>
1878 (<filename>factory_static.cpp</filename>)</title>
1880 A subclass of <literal>FactoryFilter</literal> which is
1881 responsible for registering all the statically defined filter
1882 types. It does this by knowing about all those filters'
1883 structures, which are listed in its constructor. Merely
1884 instantiating this class registers all the static classes. It is
1885 for the benefit of this class that <literal>struct
1886 metaproxy_1_filter_struct</literal> exists, and that all the filter
1887 classes provide a static object of that type.
1891 <section id="class-filter-Base">
1892 <title><literal>mp::filter::Base</literal>
1893 (<filename>filter.cpp</filename>)</title>
1895 The virtual base class of all filters. The filter API is, on the
1896 surface at least, extremely simple: two methods.
1897 <literal>configure()</literal> is passed an XML DOM tree representing
1898 that part of the configuration file that pertains to this filter
1899 instance, and is expected to walk that tree extracting relevant
1900 information. And <literal>process()</literal> processes a
1901 package (see below). That surface simplicity is a bit
1902 misleading, as <literal>process()</literal> needs to know a lot
1903 about the <literal>Package</literal> class in order to do
1908 <section id="class-AuthSimple">
1909 <title><literal>mp::filter::AuthSimple</literal>,
1910 <literal>Backend_test</literal>, etc.
1911 (<filename>filter_auth_simple.cpp</filename>,
1912 <filename>filter_backend_test.cpp</filename>, etc.)</title>
1914 Individual filters. Each of these is implemented by a header and
1915 a source file, named <filename>filter_*.hpp</filename> and
1916 <filename>filter_*.cpp</filename> respectively. All the header
1917 files should be pretty much identical, in that they declare the
1918 class, including a private <literal>Rep</literal> class and a
1919 member pointer to it, and the two public methods.
1922 The source file for each filter needs to supply:
1927 A definition of the private <literal>Rep</literal> class.
1932 Some boilerplate constructors and destructors.
1937 A <literal>configure()</literal> method that uses the
1938 appropriate XML fragment.
1943 Most important, the <literal>process()</literal> method that
1944 does all the actual work.
1950 <section id="class-Package">
1951 <title><literal>mp::Package</literal>
1952 (<filename>package.cpp</filename>)</title>
1954 Represents a package on its way through the series of filters
1955 that make up a route. This is essentially a Z39.50 or SRU APDU
1956 together with information about where it came from, which is
1957 modified as it passes through the various filters.
1961 <section id="class-Pipe">
1962 <title><literal>mp::Pipe</literal>
1963 (<filename>pipe.cpp</filename>)</title>
1965 This class provides a compatibility layer so that we have an IPC
1966 mechanism that works the same under Unix and Windows. It's not
1967 particularly exciting.
1971 <section id="class-RouterChain">
1972 <title><literal>mp::RouterChain</literal>
1973 (<filename>router_chain.cpp</filename>)</title>
1979 <section id="class-RouterFleXML">
1980 <title><literal>mp::RouterFleXML</literal>
1981 (<filename>router_flexml.cpp</filename>)</title>
1987 <section id="class-Session">
1988 <title><literal>mp::Session</literal>
1989 (<filename>session.cpp</filename>)</title>
1995 <section id="class-ThreadPoolSocketObserver">
1996 <title><literal>mp::ThreadPoolSocketObserver</literal>
1997 (<filename>thread_pool_observer.cpp</filename>)</title>
2003 <section id="class-util">
2004 <title><literal>mp::util</literal>
2005 (<filename>util.cpp</filename>)</title>
2007 A namespace of various small utility functions and classes,
2008 collected together for convenience. Most importantly, includes
2009 the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
2014 <section id="class-xml">
2015 <title><literal>mp::xml</literal>
2016 (<filename>xmlutil.cpp</filename>)</title>
2018 A namespace of various XML utility functions and classes,
2019 collected together for convenience.
2025 <section id="other.source.files">
2026 <title>Other Source Files</title>
2028 In addition to the Metaproxy source files that define the classes
2029 described above, there are a few additional files which are
2030 briefly described here:
2034 <term><literal>metaproxy_prog.cpp</literal></term>
2037 The main function of the <command>metaproxy</command> program.
2042 <term><literal>ex_router_flexml.cpp</literal></term>
2045 Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
2050 <term><literal>test_*.cpp</literal></term>
2053 Unit-tests for various modules.
2059 ### Still to be described:
2060 <literal>ex_filter_frontend_net.cpp</literal>,
2061 <literal>filter_dl.cpp</literal>,
2062 <literal>plainfile.cpp</literal>,
2063 <literal>tstdl.cpp</literal>.
2069 <reference id="reference">
2070 <title>Reference</title>
2071 <partintro id="reference-introduction">
2073 The material in this chapter is drawn directly from the individual
2074 manual entries. In particular, the Metaproxy invocation section is
2075 available using <command>man metaproxy</command>, and the section
2076 on each individual filter is available using the name of the filter
2077 as the argument to the <command>man</command> command.
2083 <appendix id="license">
2084 <title>License</title>
2087 Metaproxy, Copyright © 1995-2009 Index Data ApS.
2091 Metaproxy is free software; you can redistribute it and/or modify it under
2092 the terms of the GNU General Public License as published by the Free
2093 Software Foundation; either version 2, or (at your option) any later
2098 Metaproxy is distributed in the hope that it will be useful, but WITHOUT ANY
2099 WARRANTY; without even the implied warranty of MERCHANTABILITY or
2100 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2105 You should have received a copy of the GNU General Public License
2106 along with Metaproxy; see the file LICENSE. If not, write to the
2107 Free Software Foundation,
2108 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2112 <title>GNU General Public License</title>
2114 GNU GENERAL PUBLIC LICENSE
2115 Version 2, June 1991
2117 Copyright (C) 1989, 1991 Free Software Foundation, Inc.
2118 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2119 w Everyone is permitted to copy and distribute verbatim copies
2120 of this license document, but changing it is not allowed.
2124 The licenses for most software are designed to take away your
2125 freedom to share and change it. By contrast, the GNU General Public
2126 License is intended to guarantee your freedom to share and change free
2127 software--to make sure the software is free for all its users. This
2128 General Public License applies to most of the Free Software
2129 Foundation's software and to any other program whose authors commit to
2130 using it. (Some other Free Software Foundation software is covered by
2131 the GNU Library General Public License instead.) You can apply it to
2134 When we speak of free software, we are referring to freedom, not
2135 price. Our General Public Licenses are designed to make sure that you
2136 have the freedom to distribute copies of free software (and charge for
2137 this service if you wish), that you receive source code or can get it
2138 if you want it, that you can change the software or use pieces of it
2139 in new free programs; and that you know you can do these things.
2141 To protect your rights, we need to make restrictions that forbid
2142 anyone to deny you these rights or to ask you to surrender the rights.
2143 These restrictions translate to certain responsibilities for you if you
2144 distribute copies of the software, or if you modify it.
2146 For example, if you distribute copies of such a program, whether
2147 gratis or for a fee, you must give the recipients all the rights that
2148 you have. You must make sure that they, too, receive or can get the
2149 source code. And you must show them these terms so they know their
2152 We protect your rights with two steps: (1) copyright the software, and
2153 (2) offer you this license which gives you legal permission to copy,
2154 distribute and/or modify the software.
2156 Also, for each author's protection and ours, we want to make certain
2157 that everyone understands that there is no warranty for this free
2158 software. If the software is modified by someone else and passed on, we
2159 want its recipients to know that what they have is not the original, so
2160 that any problems introduced by others will not reflect on the original
2161 authors' reputations.
2163 Finally, any free program is threatened constantly by software
2164 patents. We wish to avoid the danger that redistributors of a free
2165 program will individually obtain patent licenses, in effect making the
2166 program proprietary. To prevent this, we have made it clear that any
2167 patent must be licensed for everyone's free use or not licensed at all.
2169 The precise terms and conditions for copying, distribution and
2170 modification follow.
2172 GNU GENERAL PUBLIC LICENSE
2173 TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
2175 0. This License applies to any program or other work which contains
2176 a notice placed by the copyright holder saying it may be distributed
2177 under the terms of this General Public License. The "Program", below,
2178 refers to any such program or work, and a "work based on the Program"
2179 means either the Program or any derivative work under copyright law:
2180 that is to say, a work containing the Program or a portion of it,
2181 either verbatim or with modifications and/or translated into another
2182 language. (Hereinafter, translation is included without limitation in
2183 the term "modification".) Each licensee is addressed as "you".
2185 Activities other than copying, distribution and modification are not
2186 covered by this License; they are outside its scope. The act of
2187 running the Program is not restricted, and the output from the Program
2188 is covered only if its contents constitute a work based on the
2189 Program (independent of having been made by running the Program).
2190 Whether that is true depends on what the Program does.
2192 1. You may copy and distribute verbatim copies of the Program's
2193 source code as you receive it, in any medium, provided that you
2194 conspicuously and appropriately publish on each copy an appropriate
2195 copyright notice and disclaimer of warranty; keep intact all the
2196 notices that refer to this License and to the absence of any warranty;
2197 and give any other recipients of the Program a copy of this License
2198 along with the Program.
2200 You may charge a fee for the physical act of transferring a copy, and
2201 you may at your option offer warranty protection in exchange for a fee.
2203 2. You may modify your copy or copies of the Program or any portion
2204 of it, thus forming a work based on the Program, and copy and
2205 distribute such modifications or work under the terms of Section 1
2206 above, provided that you also meet all of these conditions:
2208 a) You must cause the modified files to carry prominent notices
2209 stating that you changed the files and the date of any change.
2211 b) You must cause any work that you distribute or publish, that in
2212 whole or in part contains or is derived from the Program or any
2213 part thereof, to be licensed as a whole at no charge to all third
2214 parties under the terms of this License.
2216 c) If the modified program normally reads commands interactively
2217 when run, you must cause it, when started running for such
2218 interactive use in the most ordinary way, to print or display an
2219 announcement including an appropriate copyright notice and a
2220 notice that there is no warranty (or else, saying that you provide
2221 a warranty) and that users may redistribute the program under
2222 these conditions, and telling the user how to view a copy of this
2223 License. (Exception: if the Program itself is interactive but
2224 does not normally print such an announcement, your work based on
2225 the Program is not required to print an announcement.)
2227 These requirements apply to the modified work as a whole. If
2228 identifiable sections of that work are not derived from the Program,
2229 and can be reasonably considered independent and separate works in
2230 themselves, then this License, and its terms, do not apply to those
2231 sections when you distribute them as separate works. But when you
2232 distribute the same sections as part of a whole which is a work based
2233 on the Program, the distribution of the whole must be on the terms of
2234 this License, whose permissions for other licensees extend to the
2235 entire whole, and thus to each and every part regardless of who wrote it.
2237 Thus, it is not the intent of this section to claim rights or contest
2238 your rights to work written entirely by you; rather, the intent is to
2239 exercise the right to control the distribution of derivative or
2240 collective works based on the Program.
2242 In addition, mere aggregation of another work not based on the Program
2243 with the Program (or with a work based on the Program) on a volume of
2244 a storage or distribution medium does not bring the other work under
2245 the scope of this License.
2247 3. You may copy and distribute the Program (or a work based on it,
2248 under Section 2) in object code or executable form under the terms of
2249 Sections 1 and 2 above provided that you also do one of the following:
2251 a) Accompany it with the complete corresponding machine-readable
2252 source code, which must be distributed under the terms of Sections
2253 1 and 2 above on a medium customarily used for software interchange; or,
2255 b) Accompany it with a written offer, valid for at least three
2256 years, to give any third party, for a charge no more than your
2257 cost of physically performing source distribution, a complete
2258 machine-readable copy of the corresponding source code, to be
2259 distributed under the terms of Sections 1 and 2 above on a medium
2260 customarily used for software interchange; or,
2262 c) Accompany it with the information you received as to the offer
2263 to distribute corresponding source code. (This alternative is
2264 allowed only for noncommercial distribution and only if you
2265 received the program in object code or executable form with such
2266 an offer, in accord with Subsection b above.)
2268 The source code for a work means the preferred form of the work for
2269 making modifications to it. For an executable work, complete source
2270 code means all the source code for all modules it contains, plus any
2271 associated interface definition files, plus the scripts used to
2272 control compilation and installation of the executable. However, as a
2273 special exception, the source code distributed need not include
2274 anything that is normally distributed (in either source or binary
2275 form) with the major components (compiler, kernel, and so on) of the
2276 operating system on which the executable runs, unless that component
2277 itself accompanies the executable.
2279 If distribution of executable or object code is made by offering
2280 access to copy from a designated place, then offering equivalent
2281 access to copy the source code from the same place counts as
2282 distribution of the source code, even though third parties are not
2283 compelled to copy the source along with the object code.
2285 4. You may not copy, modify, sublicense, or distribute the Program
2286 except as expressly provided under this License. Any attempt
2287 otherwise to copy, modify, sublicense or distribute the Program is
2288 void, and will automatically terminate your rights under this License.
2289 However, parties who have received copies, or rights, from you under
2290 this License will not have their licenses terminated so long as such
2291 parties remain in full compliance.
2293 5. You are not required to accept this License, since you have not
2294 signed it. However, nothing else grants you permission to modify or
2295 distribute the Program or its derivative works. These actions are
2296 prohibited by law if you do not accept this License. Therefore, by
2297 modifying or distributing the Program (or any work based on the
2298 Program), you indicate your acceptance of this License to do so, and
2299 all its terms and conditions for copying, distributing or modifying
2300 the Program or works based on it.
2302 6. Each time you redistribute the Program (or any work based on the
2303 Program), the recipient automatically receives a license from the
2304 original licensor to copy, distribute or modify the Program subject to
2305 these terms and conditions. You may not impose any further
2306 restrictions on the recipients' exercise of the rights granted herein.
2307 You are not responsible for enforcing compliance by third parties to
2310 7. If, as a consequence of a court judgment or allegation of patent
2311 infringement or for any other reason (not limited to patent issues),
2312 conditions are imposed on you (whether by court order, agreement or
2313 otherwise) that contradict the conditions of this License, they do not
2314 excuse you from the conditions of this License. If you cannot
2315 distribute so as to satisfy simultaneously your obligations under this
2316 License and any other pertinent obligations, then as a consequence you
2317 may not distribute the Program at all. For example, if a patent
2318 license would not permit royalty-free redistribution of the Program by
2319 all those who receive copies directly or indirectly through you, then
2320 the only way you could satisfy both it and this License would be to
2321 refrain entirely from distribution of the Program.
2323 If any portion of this section is held invalid or unenforceable under
2324 any particular circumstance, the balance of the section is intended to
2325 apply and the section as a whole is intended to apply in other
2328 It is not the purpose of this section to induce you to infringe any
2329 patents or other property right claims or to contest validity of any
2330 such claims; this section has the sole purpose of protecting the
2331 integrity of the free software distribution system, which is
2332 implemented by public license practices. Many people have made
2333 generous contributions to the wide range of software distributed
2334 through that system in reliance on consistent application of that
2335 system; it is up to the author/donor to decide if he or she is willing
2336 to distribute software through any other system and a licensee cannot
2339 This section is intended to make thoroughly clear what is believed to
2340 be a consequence of the rest of this License.
2342 8. If the distribution and/or use of the Program is restricted in
2343 certain countries either by patents or by copyrighted interfaces, the
2344 original copyright holder who places the Program under this License
2345 may add an explicit geographical distribution limitation excluding
2346 those countries, so that distribution is permitted only in or among
2347 countries not thus excluded. In such case, this License incorporates
2348 the limitation as if written in the body of this License.
2350 9. The Free Software Foundation may publish revised and/or new versions
2351 of the General Public License from time to time. Such new versions will
2352 be similar in spirit to the present version, but may differ in detail to
2353 address new problems or concerns.
2355 Each version is given a distinguishing version number. If the Program
2356 specifies a version number of this License which applies to it and "any
2357 later version", you have the option of following the terms and conditions
2358 either of that version or of any later version published by the Free
2359 Software Foundation. If the Program does not specify a version number of
2360 this License, you may choose any version ever published by the Free Software
2363 10. If you wish to incorporate parts of the Program into other free
2364 programs whose distribution conditions are different, write to the author
2365 to ask for permission. For software which is copyrighted by the Free
2366 Software Foundation, write to the Free Software Foundation; we sometimes
2367 make exceptions for this. Our decision will be guided by the two goals
2368 of preserving the free status of all derivatives of our free software and
2369 of promoting the sharing and reuse of software generally.
2373 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
2374 FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
2375 OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
2376 PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
2377 OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
2378 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
2379 TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
2380 PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
2381 REPAIR OR CORRECTION.
2383 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
2384 WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
2385 REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
2386 INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
2387 OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
2388 TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
2389 YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
2390 PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
2391 POSSIBILITY OF SUCH DAMAGES.
2393 END OF TERMS AND CONDITIONS
2400 <!-- Keep this comment at the end of the file
2405 sgml-minimize-attributes:nil
2406 sgml-always-quote-attributes:t
2409 sgml-parent-document: nil
2410 sgml-local-catalogs: nil
2411 sgml-namecase-general:t
projects / metaproxy-moved-to-github.git / blob