<?xml version="1.0" standalone="no"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
- "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd"
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
+ "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
[
<!ENTITY % local SYSTEM "local.ent">
%local;
<author>
<firstname>Jakub</firstname><surname>Skoczen</surname>
</author>
+ <author>
+ <firstname>Mike</firstname><surname>Taylor</surname>
+ </author>
+ <author>
+ <firstname>Dennis</firstname><surname>Schafroth</surname>
+ </author>
<releaseinfo>&version;</releaseinfo>
<copyright>
<year>©right-year;</year>
</copyright>
<abstract>
<simpara>
- Pazpar2 is a high-performance, user interface-independent, data
- model-independent metasearching
- middle-ware featuring merging, relevance ranking, record sorting,
+ Pazpar2 is a high-performance metasearch engine featuring
+ merging, relevance ranking, record sorting,
and faceted results.
+ It is middleware: it has no user interface of its own, but can be
+ configured and controlled by an XML-over-HTTP web-service to provide
+ metasearching functionality behind any user interface.
</simpara>
<simpara>
This document is a guide and reference to Pazpar2 version &version;.
<chapter id="introduction">
<title>Introduction</title>
+
+ <section id="what.pazpar2.is">
+ <title>What Pazpar2 is</title>
<para>
- Pazpar2 is a stand-alone metasearch client with a web-service API, designed
- to be used either from a browser-based client (JavaScript, Flash, Java,
+ Pazpar2 is a stand-alone metasearch engine with a web-service API, designed
+ to be used either from a browser-based client (JavaScript, Flash,
+ Java applet,
etc.), from server-side code, or any combination of the two.
Pazpar2 is a highly optimized client designed to
search many resources in parallel. It implements record merging,
relevance-ranking and sorting by arbitrary data content, and facet
- analysis for browsing purposes. It is designed to be data model
+ analysis for browsing purposes. It is designed to be data-model
independent, and is capable of working with MARC, DublinCore, or any
other <ulink url="&url.xml;">XML</ulink>-structured response format
-- <ulink url="&url.xslt;">XSLT</ulink> is used to normalize and extract
data from retrieval records for display and analysis. It can be used
against any server which supports the
- <ulink url="&url.z39.50;">Z39.50</ulink> and <ulink url="&url.sru;">SRU/SRW</ulink>
- protocol. Proprietary
- backend modules can be used to support a large number of other protocols
- (please contact Index Data for further information about this).
+ <ulink url="&url.z39.50;">Z39.50</ulink>, <ulink url="&url.sru;">SRU/SRW</ulink>
+ or <ulink url="&url.solr;">SOLR</ulink> protocol. Proprietary
+ backend modules can function as connectors between these standard
+ protocols and any non-standard API, including web-site scraping, to
+ support a large number of other protocols.
</para>
<para>
Additional functionality such as
- user management, attractive displays are expected to be implemented by
- applications that use Pazpar2. Pazpar2 is user interface independent.
- Its functionality is exposed through a simple REST-style web-service API,
- designed to be simple to use from an Ajax-enabled browser, Flash
+ user management and attractive displays are expected to be implemented by
+ applications that use Pazpar2. Pazpar2 itself is user-interface independent.
+ Its functionality is exposed through a simple XML-based web-service API,
+ designed to be easy to use from an Ajax-enabled browser, Flash
animation, Java applet, etc., or from a higher-level server-side language
- like PHP or Java. Because session information can be shared between
- browser-based logic and your server-side scripting, there is tremendous
- flexibility in how you implement your business logic on top of Pazpar2.
+ like PHP, Perl or Java. Because session information can be shared between
+ browser-based logic and server-side scripting, there is tremendous
+ flexibility in how you implement application-specific logic on top
+ of Pazpar2.
</para>
<para>
Once you launch a search in Pazpar2, the operation continues behind the
scenes. Pazpar2 connects to servers, carries out searches, and
retrieves, deduplicates, and stores results internally. Your application
code may periodically inquire about the status of an ongoing operation,
- and ask to see records or other result set facets. Result becomes
- available immediately, and it is easy to build end-user interfaces which
+ and ask to see records or result set facets. Results become
+ available immediately, and it is easy to build end-user interfaces than
feel extremely responsive, even when searching more than 100 servers
concurrently.
</para>
normalized to XML/UTF-8, and then further normalized using XSLT to a
simple internal representation that is suitable for analysis. By
providing XSLT stylesheets for different kinds of result records, you
- can tune Pazpar2 to work against different kinds of information
- retrieval servers. Finally, metadata is extracted, in a configurable
- way, from this internal record, to support display, merging, ranking,
+ can configure Pazpar2 to work against different kinds of information
+ retrieval servers. Finally, metadata is extracted in a configurable
+ way from this internal record, to support display, merging, ranking,
result set facets, and sorting. Pazpar2 is not bound to a specific model
- of metadata, such as DublinCore or MARC -- by providing the right
- configuration, it can work with a number of different kinds of data in
+ of metadata, such as DublinCore or MARC: by providing the right
+ configuration, it can work with any combination of different kinds of data in
support of many different applications.
</para>
<para>
search several hundred targets in parallel, or you can use it to support
hundreds of concurrent users. It is implemented with the same attention
to performance and economy that we use in our indexing engines, so that
- you can focus on building your application, without worrying about the
+ you can focus on building your application without worrying about the
details of metasearch logic. You can devote all of your attention to
usability and let Pazpar2 do what it does best -- metasearch.
</para>
<para>
- If you wish to connect to commercial or other databases which do not
- support open standards, please contact Index Data. We have a licensing
- agreement with a third party vendor which will enable Pazpar2 to access
- thousands of online databases, in addition to the vast number of catalogs
- and online services that support the Z39.50/SRU/SRW protocols.
- </para>
- <para>
Pazpar2 is our attempt to re-think the traditional paradigms for
implementing and deploying metasearch logic, with an uncompromising
approach to performance, and attempting to make maximum use of the
Enjoy!
</para>
<para>
- Pazpar2 is covered by the GNU license version 2.
+ Pazpar2 is covered by the GNU General Public License (GPL) version 2.
See <xref linkend="license"/> for further information.
</para>
+ </section>
+
+ <section id="connectors">
+ <title>Connectors to non-standard databases</title>
+ <para>
+ If you wish to connect to commercial or other databases which do not
+ support open standards, please contact Index Data on
+ <email>info@indexdata.com</email>. We have a
+ proprietary framework for building connectors that enable Pazpar2
+ to access
+ thousands of online databases, in addition to the vast number of catalogs
+ and online services that support the Z39.50/SRU/SRW/SOLR protocols.
+ </para>
+ </section>
+
+ <section id="name">
+ <title>A note on the name Pazpar2</title>
+ <para>
+ The name Pazpar2 derives from three sources. One one hand, it is
+ Index Data's second major piece of software that does parallel
+ searching of Z39.50 targets. On the other, it is a near-homophone
+ of Passpartout, the ever-helpful servant in Jules Verne's novel
+ Around the World in Eighty Days (who helpfully uses the language
+ of his master). Finally, "passe par tout" means something like
+ "passes through anything" in French -- on other words, a universal
+ solution, or if you like a MasterKey.
+ </para>
+ </section>
</chapter>
<chapter id="installation">
<title>Installation</title>
<para>
- The Pazpar2 package is very small. It includes documentation as well
+ The Pazpar2 package includes documentation as well
as the Pazpar2 server. The package also includes a simple user
- interface test1 which consists of a single HTML page and a single
+ interface called "test1", which consists of a single HTML page and a single
JavaScript file to illustrate the use of Pazpar2.
</para>
<para>
</para>
<section id="installation.unix">
- <title>Installation on Unix (from Source)</title>
+ <title>Installation from source on Unix (including Linux, MacOS, etc.)</title>
<para>
The latest source code for Pazpar2 is available from
<ulink url="&url.pazpar2.download;"/>.
- Only few systems have none of the required
- tools binary packages.
- If, for example, Libxml2/libXSLT libraries
- are already installed as development packages use these.
+ Most Unix-based operating systems have the required
+ tools available as binary packages.
+ For example, if Libxml2/libXSLT libraries
+ are already installed as development packages, use these.
</para>
<para>
- Ensure that the development libraries + header files are
+ Ensure that the development libraries and header files are
available on your system before compiling Pazpar2. For installation
- of YAZ, refer to the YAZ installation chapter.
+ of YAZ, refer to the Installation chapter of the YAZ manual at
+ <ulink url="&url.yaz.install;"/>.
+ </para>
+ <para>
+ Once the dependencies are in place, Pazpar2 can be unpacked and
+ installed as follows:
</para>
<screen>
- gunzip -c pazpar2-version.tar.gz|tar xf -
- cd pazpar2-version
+ tar xzf pazpar2-VERSION.tar.gz
+ cd pazpar2-VERSION
./configure
make
- su
- make install
+ sudo make install
</screen>
<para>
The <literal>make install</literal> will install manpages as well as the
</section>
<section id="installation.win32">
- <title>Installation on Windows (from Source)</title>
+ <title>Installation from source on Windows</title>
<para>
Pazpar2 can be built for Windows using
<ulink url="&url.vstudio;">Microsoft Visual Studio</ulink>.
processed by the NMAKE utility part of Visual Studio.
</para>
<para>
- Ensure that the development libraries + header files are
+ Ensure that the development libraries and header files are
available on your system before compiling Pazpar2. For installation
- of YAZ, refer to the YAZ installation chapter.
+ of YAZ, refer to
+ the Installation chapter of the YAZ manual at
+ <ulink url="&url.yaz.install;"/>.
It is easiest if YAZ and Pazpar2 are unpacked in the same
directory (side-by-side).
</para>
<para>
The compilation is tuned by editing the makefile of Pazpar2.
The process is similar to YAZ. Adjust the various directories
- <literal>YAZ_DIR</literal>, <literal>ZLIB_DIR</literal>, ..
+ <literal>YAZ_DIR</literal>, <literal>ZLIB_DIR</literal>, etc.,
+ as required.
</para>
<para>
Compile Pazpar2 by invoking <application>nmake</application> in
in the same invocation. For example:
<screen>
cd \MyPazpar2\etc
- ..\bin\pazpar2 -install -c pazpar2.cfg -l pazpar2.log
+ ..\bin\pazpar2 -install -f pazpar2.cfg -l pazpar2.log
</screen>
The Pazpar2 service may now be controlled via the Service Control
Panel. It may be unregistered by passing the <literal>-remove</literal>
</section>
<section id="installation.test1">
- <title>Installation of test1 interface</title>
+ <title>Installation of test interfaces</title>
<para>
- In this section we outline how to install a simple interface that
- is part of the Pazpar2 source package. Note that Debian users can
- save time by just installing package <literal>pazpar2-test1</literal>.
+ In this section we show how to make available the set of simple
+ interfaces that are part of the Pazpar2 source package, and which
+ demonstrate some ways to use Pazpar2. (Note that Debian users can
+ save time by just installing the package <literal>pazpar2-test1</literal>.)
</para>
<para>
- A web server must be installed and running on the system, such as Apache.
+ A web server, such as Apache, must be installed and running on the system.
</para>
<para>
..\bin\pazpar2 -f pazpar2.cfg
</screen>
This will start a Pazpar2 listener on port 9004. It will proxy
- HTTP requests to localhost - port 80, which we assume will be the regular
+ HTTP requests to port 80 on localhost, which we assume will be the regular
HTTP server on the system. Inspect and modify pazpar2.cfg as needed
- if this is to be changed. The pazpar2.cfg includes settings from the
+ if this is to be changed. The pazpar2.cfg file includes settings from the
file <filename>settings/edu.xml</filename>
to use for searches.
</para>
+
<para>
- Make a new console and move to the other stuff.
- For more information about pazpar2 options refer to the manpage.
+ The test UIs are located in <literal>www</literal>. Ensure that this
+ directory is available to the web server by copying
+ <literal>www</literal> to the document root,
+ using Apache's <literal>Alias</literal> directive, or
+ creating a symbolic link: for example, on a Debian or Ubuntu
+ system with Apache2 installed from the standard package, you might
+ make the link as follows:
+ <screen>
+ cd .../pazpar2
+ sudo ln -s `pwd`/www /var/www/pazpar2-demo
+ </screen>
</para>
-
+
<para>
- The test1 UI is located in <literal>www/test1</literal>. Ensure this
- directory is available to the web server by either copying
- <literal>test1</literal> to the document root, create a symlink or
- use Apache's <literal>Alias</literal> directive.
+ This makes the test applications visible at
+ <ulink url="http://localhost/pazpar2-demo/"/>
+ but they can not be run successfully from that URL, as they submit
+ search requests back to the server form which they were served,
+ and Apache2 doesn't know how to handle them. Instead, the test
+ applications must be accessed from Pazpar2 itself, acting as a
+ proxy to Apache2, at the URL
+ <ulink url="http://localhost:9004/pazpar2-demo/"/>
</para>
<para>
- The interface test1 interface should now be available on port 8004.
+ From here, the demo applications can be
+ accessed: <literal>test1</literal>, <literal>test2</literal> and
+ <literal>jsdemo</literal>
+ are pure HTML+JavaScript setups, needing no server-side
+ intelligence;
+ <literal>demo</literal>
+ requires PHP on the server.
</para>
<para>
- If you don't see the test1 interface. See if test1 is really available
- on the same URL but on port 80. If it's not, the Apache configuration
- (or other) is not correct.
+ If you don't see the test interfaces, check whether they are available
+ on port 80 (i.e. directly from the Apache2 server). If not, the
+ Apache configuration is incorrect.
</para>
<para>
In order to use Apache as frontend for the interface on port 80
</section>
<section id="installation.debian">
- <title>Installation on Debian GNU/Linux</title>
+ <title>Installation on Debian or Ubuntu GNU/Linux</title>
<para>
- Index Data provides Debian packages for Pazpar2. These are prepared
- for Debian versions Etch and Lenny (as of 2007).
- These packages are available at
- <ulink url="&url.pazpar2.download.debian;"/>.
+ Index Data provides Debian and Ubuntu packages for Pazpar2.
+ As of February 2010, these
+ are prepared for Debian versions Etch, Lenny and Squeeze; and for
+ Ubuntu versions 8.04 (hardy), 8.10 (intrepid), 9.04 (jaunty) and
+ 9.10 (karmic). These packages are available at
+ <ulink url="&url.pazpar2.download.debian;"/> and
+ <ulink url="&url.pazpar2.download.ubuntu;"/>.
</para>
</section>
On a Debian based Apache 2 system, the relevant modules can
be enabled with:
<screen>
- sudo a2enmod proxy_http
+ sudo a2enmod proxy_http proxy_balancer
</screen>
</para>
<title>Connecting to non-standard resources</title>
<para>
Pazpar2 uses Z39.50 as its switchboard language -- i.e. as far as it
- is concerned, all resources speak Z39.50, or its webservices derivatives,
- SRU/SRW. It is, however, equipped
+ is concerned, all resources speak Z39.50, its webservices derivatives,
+ SRU/SRW and SOLR servers exposing Lucene indexes. It is, however, equipped
to handle a broad range of different server behavior, through
configurable query mapping and record normalization. If you develop
configuration, stylesheets, etc., for a new type of resources, we
</para>
</section>
+ <section id="load_balancing">
+ <title>Load balancing</title>
+ <para>
+ Just like any web server, Pazpar2, can be load balanced by a standard hardware or software load balancer as long as the session stickiness is ensured. If you are already running the Apache2 web server in front of Pazpar2 and use the apache mod_proxy module to 'relay' client requests to Pazpar2, this set up can be easily extended to include load balancing capabilites. To do so you need to enable the <ulink url="http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html">
+ mod_proxy_balancer
+ </ulink> module in your Apache2 installation.
+ </para>
+
+ <para>
+ On a Debian based Apache 2 system, the relevant modules can
+ be enabled with:
+ <screen>
+ sudo a2enmod proxy_http
+ </screen>
+ </para>
+
+ <para>
+ The mod_proxy_balancer can pass all 'sessionsticky' requests to the same backend worker as long as the requests are marked with the originating worker's ID (called 'route'). If the Pazpar2 serverID is configured (by setting an 'id' attribute on the 'server' element in the Pazpar2 configuration file) Pazpar2 will append it to the 'session' element returned during the 'init' in a mod_proxy_balancer compatible manner. Since the 'session' is then re-sent by the client (for all pazpar2 request besides 'init'), the balancer can use the marker to pass the request to the right route. To do so the balancer needs to be configured to inspect the 'session' parameter.
+ </para>
+
+ <example id="load_balancing.example">
+ <title>Apache 2 load balancing configuration</title>
+ <para>
+ Having 4 Pazpar2 instances running on the same host, port range of 8004-8007 and serverIDs of: pz1, pz2, pz3 and pz4 respectively we could use the following Apache 2 configuration to expose a single pazpar2 'endpoint' on a standard (<filename>/pazpar2/search.pz2</filename>) location:
+
+ <screen><![CDATA[
+ <Proxy *>
+ AddDefaultCharset off
+ Order deny,allow
+ Allow from all
+ </Proxy>
+ ProxyVia Off
+
+ # 'route' has to match the configured pazpar2 server ID
+ <Proxy balancer://pz2cluster>
+ BalancerMember http://localhost:8004 route=pz1
+ BalancerMember http://localhost:8005 route=pz2
+ BalancerMember http://localhost:8006 route=pz3
+ BalancerMember http://localhost:8007 route=pz4
+ </Proxy>
+
+ # route is resent in the 'session' param which has the form:
+ # 'sessid.serverid', understandable by the mod_proxy_load_balancer
+ # this is not going to work if the client tampers with the 'session' param
+ ProxyPass /pazpar2/search.pz2 balancer://pz2cluster lbmethod=byrequests stickysession=session nofailover=On]]></screen>
+
+ The 'ProxyPass' line sets up a reverse proxy for request ‘/pazpar2/search.pz2’ and delegates all requests to the load balancer (virtual worker) with name ‘pz2cluster’. Sticky sessions are enabled and implemented using the ‘session’ parameter. The ‘Proxy’ section lists all the servers (real workers) which the load balancer can use.
+ </para>
+
+ </example>
+
+ </section>
+
+
</chapter> <!-- Using Pazpar2 -->
<reference id="reference">
projects / pazpar2-moved-to-github.git / blobdiff