-<chapter id="proxy">
- <title>The YAZ Proxy</title>
+ <chapter id="proxy">
+ <title>The YAZ Proxy</title>
<para>
The YAZ proxy is a transparent Z39.50-to-Z39.50 gateway. That is,
it is a Z39.50 server which has as its back-end a Z39.50 client
start it up. It will work exactly as usual, but all the packets
will be sent via the proxy, which will generate a log like this:
</para>
- <screen>
+ <screen><![CDATA[
decode choice
initRequest {
referenceId OCTETSTRING(len=4) 69 6E 69 74
}
}
}
+]]>
</screen>
</section>
-
+
<section id="proxy-target">
<title>Specifying the Backend Target</title>
<para>
<orderedlist>
<listitem>
<para> If the <literal>InitializeRequest</literal> PDU from the
- client includes an <literal>otherInfo</literal> element with OID
+ client includes an
+ <link linkend="otherinfo-encoding"><literal>otherInfo</literal></link>
+ element with OID
<literal>1.2.840.10003.10.1000.81.1</literal>, then the
contents of that element specify the target to be used, in the
usual YAZ address format (typically
<listitem>
<para> Otherwise, the Proxy uses the default target, if one was
specified on the command-line with the <literal>-t</literal>
- option.
+ option. A default target can also be specified in the
+ XML Config file.
</para>
</listitem>
<listitem>
</para>
</section>
<section id="proxy-keepalive">
- <title>Keep-alive Facility for Stateless Clients</title>
+ <title>Keep-alive Facility</title>
+ <para>
+ The keep-alive is a facility where the proxy keeps the connection to the
+ backend - even if the client closes the connection to the proxy.
+ </para>
<para>
- Stateless clients such as web gateways may generate a cookie for a Z39.50
- session which is sent to the proxy as part of PDUs.
- In this case, the proxy will keep alive its Z39.50 session
- to the backend target even when the connection from the client
- to the proxy is closed. When the client contacts the
- proxy again, and re-issues the same cookie, the proxy reuses the
- Z39.50 connection with the backend target.
+ If a new or another client connects to the proxy again and requests the
+ same backend it will be reassigned to this backend. In this case, the
+ proxy sends an initialize response directly to the client and an
+ initialize handshake with the backend is omitted.
</para>
<para>
- There is no
- guarantee that the Z39.50 connection to the backend
- target is kept forever: the proxy will shut it down after certain
- idle time.
- <!-- ### How long? Wot no command-line option? -->
- So in effect, the connection from the client's
- point of view should be considered stateless, and the keep-alive
- facility should be treated only as a performance booster.
+ When a client reconnects, query and record caching works better, if the
+ proxy assigns it to the same backend as before. And the result set
+ (if any) is re-used. To achieve this, Index Data defined a session
+ cookie which identifies the backend session.
</para>
<para>
- Cookies may be passed in an <literal>otherInfo</literal> element
- with OID <literal>1.2.840.10003.10.1000.81.2</literal>.
+ The cookie is defined by the client and is sent as part of the
+ Initialize Request and passed in an
+ <link linkend="otherinfo-encoding"><literal>otherInfo</literal></link>
+ element with OID <literal>1.2.840.10003.10.1000.81.2</literal>.
+ </para>
+ <para>
+ Clients that do not send a cookie as part of the initialize request
+ may still better performance, since the init handshake is saved.
</para>
</section>
-
- <section id="proxy-cache">
+
+ <section id="query-cache">
<title>Query Caching</title>
<para>
Simple stateless clients often send identical Z39.50 searches
backend target, so that if an identical query is received next, it
is turned into Present Requests rather than new Search Requests.
</para>
- <!-- ### should be generalised to an arbitrary-sized cache -->
+ <note>
+ <para>
+ In a future we release will will probably allows for
+ an arbitrary-sized cache for targets supporting named result sets.
+ </para>
+ </note>
<para>
- This optimization should work for any Z39.50 client and/or
- target. The target does not have to support named result sets.
+ You can enable/disable query caching using option -o.
</para>
- <!-- ### There should be an option to turn this off, as it will
- affect semantics for some searches on some databases:
- e.g. "ten most recent stories" in a newswire database.
- -->
</section>
-
- <section id="proxy-optimizations">
+
+ <section id="record-cache">
+ <title>Record Caching</title>
+ <para>
+ As an option, the proxy may also cache result set records for the
+ last search.
+ The proxy takes into account the Record Syntax and CompSpec.
+ The CompSpec includes simple element set names as well.
+ By default the cache is 200000 bytes per session.
+ </para>
+ </section>
+
+ <section id="query-validation">
+ <title>Query Validation</title>
+ <para>
+ The Proxy may also be configured to trap particular attributes in
+ Type-1 queries and send Bib-1 diagnostics back to the client without
+ even consulting the backend target. This facility may be useful if
+ a target does not properly issue diagnostics when unsupported attributes
+ are send to it.
+ </para>
+ </section>
+
+ <section id="record-validation">
+ <title>Record Syntax Validation</title>
+ <para>
+ The proxy may be configured to accept, reject or convert records.
+ When accepted, the target passes search/present requests to the
+ backend target under the assumption that the target can honor the
+ request (In fact it may not do that). When a record is rejected because
+ the record syntax is "unsupported" the proxy returns a diagnostic to the
+ client. Finally, the proxy may convert records.
+ </para>
+ <para>
+ In the current version the only supported conversion is
+ MARC21/USMARC in MARC-8 charset to MARCXML in UTF-8. Future version of
+ the proxy may do other record/charset conversions.
+ </para>
+ </section>
+
+ <section id="other-optimizations">
<title>Other Optimizations</title>
<para>
- We've had some plans to support caching of result set records,
+ We've had some plans to support global caching of result set records,
but this has not yet been implemented.
</para>
</section>
- <section id="proxy-usage">
- <title>Proxy Usage</title>
+ <section id="proxy-config-file">
+ <title>Proxy Configuration File</title>
<para>
+ The Proxy as an option may read a configuration file using option
+ <literal>-c</literal> followed by the filename of a config file.
+ </para>
+ <para>
+ The config file is in XML format. The YAZ proxy must be compiled
+ with <ulink url="http://www.xmlsoft.org/">libxml2</ulink> and
+ <ulink url="http://xmlsoft.org/XSLT/">libXSLT</ulink> support in
+ order for the config file facility to be enabled.
</para>
- <refentry id="yaz-proxy">
- <refmeta>
- <refentrytitle>yaz-proxy</refentrytitle>
- <manvolnum>8</manvolnum>
- </refmeta>
- <refnamediv>
- <refname>yaz-proxy</refname>
- <refpurpose>The YAZ toolkit's transparent Z39.50 proxy</refpurpose>
- </refnamediv>
- <refsynopsisdiv>
- <cmdsynopsis>
- <command>yaz-proxy</command>
- <arg choice="opt">-a <replaceable>filename</replaceable></arg>
- <arg choice="opt">-c <replaceable>num</replaceable></arg>
- <arg choice="opt">-v <replaceable>level</replaceable></arg>
- <arg choice="opt">-t <replaceable>target</replaceable></arg>
- <arg choice="opt">-u <replaceable>auth</replaceable></arg>
- <arg choice="opt">-o <replaceable>level</replaceable></arg>
- <arg choice="req"><replaceable>host</replaceable>:<replaceable>port</replaceable></arg>
- </cmdsynopsis>
- </refsynopsisdiv>
-
- <refsect1><title>DESCRIPTION</title>
- <para>
- The proxy runs stand-alone (not from
- <literal>inetd</literal>). The
- <replaceable>host</replaceable>:<replaceable>port</replaceable>
- argument specifies host address to listen to, and the port to
- listen on. Use the host <literal>@</literal>
- to listen for connections coming from any address.
- </para>
- </refsect1>
- <refsect1><title>OPTIONS</title>
- <variablelist>
- <varlistentry><term>-a <replaceable>filename</replaceable></term>
- <listitem><para>
- Specifies the name of a file to which to write a log of the
- APDUs (protocol packets) that pass through the proxy. The
- special filename <literal>-</literal> may be used to indicate
- standard output.
- </para></listitem>
- </varlistentry>
- <varlistentry><term>-c <replaceable>num</replaceable></term>
- <listitem><para>
- Specifies the maximum number of connections to be cached
- [default 50].
- </para></listitem>
- </varlistentry>
- <varlistentry><term>-v <replaceable>level</replaceable></term>
- <listitem><para>
- Sets the logging level. <replaceable>level</replaceable> is
- a comma-separated list of members of the set
- {<literal>fatal</literal>,<literal>debug</literal>,<literal>warn</literal>,<literal>log</literal>,<literal>malloc</literal>,<literal>all</literal>,<literal>none</literal>}.
- </para></listitem>
- </varlistentry>
- <varlistentry><term>-t <replaceable>target</replaceable></term>
- <listitem><para>
- Specifies the default backend target to use when a client
- connects that does not explicitly specify a target in its
- <literal>initRequest</literal>.
- </para></listitem>
- </varlistentry>
- <varlistentry><term>-u <replaceable>auth</replaceable></term>
- <listitem><para>
- Specifies authentication info to be sent to the backend target.
- This is useful if you happen to have an internal target that
- requires authentication, or if the client software does not allow
- you to set it.
- </para></listitem>
- </varlistentry>
- <varlistentry><term>-o <replaceable>level</replaceable></term>
- <listitem><para>
- Sets level for optimization. Use zero to disable; non-zero
- to enable. Handling for this is not fully implemented;
- we will probably use a bit mask to enable/disable specific
- features.
- </para></listitem>
- </varlistentry>
- </variablelist>
- </refsect1>
- <refsect1>
- <title>EXAMPLES</title>
- <para>
- The following command starts the proxy, listening on port
- 9000, with its default backend target set to the Library of
- Congress bibliographic server:
- </para>
+ <tip>
+ <para>To check for a config file to be well-formed, the yaz-proxy may
+ be invoked without specifying a listening port, i.e.
<screen>
- $ yaz-proxy -t z3950.loc.gov:7090 @:9000
+ yaz-proxy -c myconfig.xml
</screen>
+ If this does not produce errors, the file is well-formed.
+ </para>
+ </tip>
+ <section id="proxy-config-header">
+ <title>Proxy Configuration Header</title>
+ <para>
+ The proxy config file must have a root element called
+ <literal>proxy</literal>. All information except an optional XML
+ header must be stored within the <literal>proxy</literal> element.
+ </para>
+ <screen>
+ <?xml version="1.0"?>
+ <proxy>
+ <!-- content here .. -->
+ </proxy>
+ </screen>
+ </section>
+ <section id="proxy-config-target">
+ <title>Configuration: target</title>
+ <para>
+ The element <literal>target</literal> which may be repeated zero
+ or more times with parent element <literal>proxy</literal> contains
+ information about each backend target.
+ The <literal>target</literal> element have two attributes:
+ <literal>name</literal> which holds the logical name of the backend
+ target (required) and <literal>default</literal> (optional) which
+ (when given) specifies that the backend target is the default target -
+ equivalent to command line option <literal>-t</literal>.
+ </para>
+ <para>
+ <screen>
+ <?xml version="1.0"?>
+ <proxy>
+ <target name="server1" default="1">
+ <!-- description of server1 .. -->
+ </target>
+ <target name="server2">
+ <!-- description of server2 .. -->
+ </target>
+ </proxy>
+ </screen>
+ </para>
+ </section>
+ <section id="proxy-config-url">
+ <title>Configuration:url</title>
+ <para>
+ The <literal>url</literal> which may be repeated one or more times
+ should be the child of the <literal>target</literal> element.
+ The CDATA of <literal>url</literal> is the Z-URL of the backend.
+ </para>
+ <para>
+ Multiple <literal>url</literal> element may be used. In that case, then
+ a client initiates a session, the proxy chooses the URL with the lowest
+ number of active sessions, thereby distributing the load. It is
+ assumed that each URL represents the same database (data).
+ </para>
+ </section>
+ <section id="proxy-config-keepalive">
+ <title>Configuration: keepalive</title>
+ <para>The <literal>keepalive</literal> element holds information about
+ the keepalive Z39.50 sessions. Keepalive sessions are proxy-to-backend
+ sessions that is no longer associated with a client session.
+ </para>
+ <para>The <literal>keepalive</literal> element which is the child of
+ the <literal>target</literal>holds two elements:
+ <literal>bandwidth</literal> and <literal>pdu</literal>.
+ The <literal>bandwidth</literal> is the maximum total bytes
+ transferred to/from the target. If a target session exceeds this
+ limit, it is shut down (and no longer kept alive).
+ The <literal>pdu</literal> is the maximum number of requests sent
+ to the target. If a target session exceeds this limit, it is
+ shut down. The idea of these two limits is that avoid very long
+ sessions that use resources in a backend (that leaks!).
+ </para>
+ <para>
+ The following sets maximum number of bytes transferred in a
+ target session to 1 MB and maxinum of requests to 400.
+ <screen>
+ <keepalive>
+ <bandwidth>1048576</bandwidth>
+ <retrieve>400</retrieve>
+ </keepalive>
+ </screen>
+ </para>
+ </section>
+ <section id="proxy-config-limit">
+ <title>Configuration: limit</title>
+ <para>
+ The <literal>limit</literal> section specifies bandwidth/pdu requests
+ limits for an active session.
+ The proxy records bandwidth/pdu requests during the last 60 seconds
+ (1 minute). The <literal>limit</literal> may include the
+ elements <literal>bandwidth</literal>, <literal>pdu</literal>,
+ and <literal>retrieve</literal>. The <literal>bandwidth</literal>
+ measures the number of bytes transferred within the last minute.
+ The <literal>pdu</literal> is the number of requests in the last
+ minute. The <literal>retrieve</literal> holds the maximum records to
+ be retrieved in one Present Request.
+ </para>
+ <para>
+ If a bandwidth/pdu limit is reached the proxy will postpone the
+ requests to the target and wait one or more seconds. The idea of the
+ limit is to ensure that clients that downloads hundreds or thousands of
+ records do not hurt other users.
+ </para>
+ <para>
+ The following sets maximum number of bytes transferred per minute to
+ 500Kbytes and maximum number of requests to 40.
+ <screen>
+ <limit>
+ <bandwidth>524288</bandwidth>
+ <retrieve>40</retrieve>
+ </limit>
+ </screen>
+ </para>
+ <note>
<para>
- The LOC target is sometimes very slow. You can connect to
- it using yaz-client as follows:
+ Typically the limits for keepalive are much higher than
+ those for session minute average.
</para>
+ </note>
+ </section>
+
+ <section id="proxy-config-attribute">
+ <title>Configuration: attribute</title>
+ <para>
+ The <literal>attribute</literal> element specifies accept or reject
+ or a particular attribute type, value pair.
+ Well-behaving targets will reject unsupported attributes on their
+ own. This feature is useful for targets that do not gracefully
+ handle unsupported attributes.
+ </para>
+ <para>
+ Attribute elements may be repeated. The proxy inspects the attribute
+ specifications in the order as specified in the configuration file.
+ When a given attribute specification matches a given attribute list
+ in a query, the proxy takes appropriate action (reject, accept).
+ </para>
+ <para>
+ If no attribute specifications matches the attribute list in a query,
+ it is accepted.
+ </para>
+ <para>
+ The <literal>attribute</literal> element has two required attributes:
+ <literal>type</literal> which is the Attribute Type-1 type, and
+ <literal>value</literal> which is the Attribute Type-1 value.
+ The special value/type <literal>*</literal> matches any attribute
+ type/value. A value may also be specified as a list with each
+ value separated by comma, a value may also be specified as a
+ list: low value - dash - high value.
+ </para>
+ <para>
+ If attribute <literal>error</literal> is given, that holds a
+ Bib-1 diagnostic which is sent to the client if the particular
+ type, value is part of a query.
+ </para>
+ <para>
+ If attribute <literal>error</literal> is not given, the attribute
+ type, value is accepted and passed to the backend target.
+ </para>
+ <para>
+ A target that supports use attributes 1,4, 1000 through 1003 and
+ no other use attributes, could use the following rules:
<screen>
- $ yaz-client localhost:9000/voyager
- Connecting...Ok.
- Sent initrequest.
- Connection accepted by target.
- ID : 34
- Name : Voyager LMS - Z39.50 Server
- Version: 1.13
- Options: search present
- Elapsed: 7.131197
- Z> f computer
- Sent searchRequest.
- Received SearchResponse.
- Search was a success.
- Number of hits: 10000
- records returned: 0
- Elapsed: 6.695174
- Z> f computer
- Sent searchRequest.
- Received SearchResponse.
- Search was a success.
- Number of hits: 10000
- records returned: 0
- Elapsed: 0.001417
+ <attribute type="1" value="1,4,1000-1003">
+ <attribute type="1" value="*" error="114"/>
</screen>
- <para>
- In this test, the second search was more than 4000 times faster
- than the first, because the proxy cached the result of the first
- search and noticed that the second was the same.
+ </para>
+ </section>
+
+ <section id="proxy-config-syntax">
+ <title>Configuration: syntax</title>
+ <para>
+ The <literal>syntax</literal> element specifies accept or reject
+ or a particular record syntax request from the client.
+ </para>
+ <para>
+ The <literal>syntax</literal> has one required attribute:
+ <literal>type</literal> which is the Preferred Record Syntax.
+ </para>
+ <para>
+ If attribute <literal>error</literal> is given, that holds a
+ Bib-1 diagnostic which is sent to the client if the particular
+ record syntax is part of a present - or search request.
+ </para>
+ <para>
+ If attribute <literal>error</literal> is not given, the record syntax
+ is accepted and passed to the backend target.
+ </para>
+ <para>
+ If attribute <literal>marcxml</literal> is given, the proxy will
+ perform MARC21 to MARCXML conversion. In this case the
+ <literal>type</literal> should be XML. The proxy will use
+ preferred record syntax USMARC/MARC21 against the backend target.
+ </para>
+ <para>To accept USMARC and offer MARCXML XML records but reject
+ all other requests the following configuration could be used:
+ <screen>
+ <proxy>
+ <target name="mytarget">
+ <syntax type="usmarc"/>
+ <syntax type="xml" marcxml="1"/>
+ <syntax type="*" error="238"/>
+ </target>
+ </proxy>
+ </screen>
+ </para>
+ </section>
+
+ <section id="proxy-config-target-timeout">
+ <title>Configuration: target-timeout</title>
+ <para>
+ The element <literal>target-timeout</literal> is the child of element
+ <literal>target</literal> and specifies the amount in seconds before
+ a target session is shut down.
+ </para>
+ <para>
+ This can also be specified on the command line by using option
+ <literal>-T</literal>. Refer to <xref linkend="proxy-usage"/>.
+ </para>
+ </section>
+
+ <section id="proxy-config-client-timeout">
+ <title>Configuration: client-timeout</title>
+ <para>
+ The element <literal>client-timeout</literal> is the child of element
+ <literal>target</literal> and specifies the amount in seconds before
+ a client session is shut down.
</para>
+ <para>
+ This can also be specified on the command line by using option
+ <literal>-i</literal>. Refer to <xref linkend="proxy-usage"/>.
+ </para>
+ </section>
+
+ <section id="proxy-config-preinit">
+ <title>Configuration: preinit</title>
+ <para>
+ The element <literal>preinit</literal> is the child of element
+ <literal>target</literal> and specifies the number of spare
+ connection to a target. By default no spare connection are
+ created by the proxy. If the proxy uses a target exclusive or
+ a lot, the preinit session will ensure that target sessions
+ have been made before the client makes a connection and will therefore
+ reduce the connect-init handshake dramatically. Never set this to
+ more than 5.
+ </para>
+ </section>
+
+ <section id="proxy-config-max-clients">
+ <title>Configuration: max-clients</title>
+ <para>
+ The element <literal>max-clients</literal> is the child of element
+ <literal>proxy</literal> and specifies the total number of
+ allowed connections to targets (all targets). If this limit
+ is reached the proxy will close the least recently used connection.
+ </para>
+ <para>
+ Note, that many Unix systems impose a system on the number of
+ open files allowed in a single process, typically in the
+ range 256 (Solaris) to 1024 (Linux).
+ The proxy uses 2 sockets per session + a few files
+ for logging. As a rule of thumb, ensure that 2*max-clients + 5
+ can be opened by the proxy process.
+ </para>
+ <tip>
<para>
- The YAZ command-line client,
- <literal>yaz-client</literal>,
- allows you to set the backend target in
- the <literal>initRequest</literal> using the
- <literal>-p</literal> option. For example, to connect to
- Index Data's target you could use:
+ Using the <ulink url="http://www.gnu.org/software/bash/bash.html">
+ bash</ulink> shell, you can set the limit with
+ <literal>ulimit -n</literal><replaceable>no</replaceable>.
+ Use <literal>ulimit -a</literal> to display limits.
</para>
- <screen>
- yaz-client -p indexdata.dk localhost:9000/gils
+ </tip>
+ </section>
+
+ <section id="proxy-config-log">
+ <title>Configuration: log</title>
+ <para>
+ The element <literal>log</literal> is the child of element
+ <literal>proxy</literal> and specifies what to be logged by the
+ proxy.
+ </para>
+ <para>
+ Specify the log file with command-line option <literal>-l</literal>.
+ </para>
+ <para>
+ The text of the <literal>log</literal> element is a sequence of
+ options separated by white space. See the table below:
+ <table frame="top"><title>Logging options</title>
+ <tgroup cols="2">
+ <colspec colwidth="1*" colname="option"/>
+ <colspec colwidth="2*" colname="description"/>
+ <thead>
+ <row>
+ <entry>Option</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><literal>client-apdu</literal></entry>
+ <entry>
+ Log APDUs as reported by YAZ for the
+ communication between the client and the proxy.
+ This facility is equivalent to the APDU logging that
+ happens when using option <literal>-a</literal>, however
+ this tells the proxy to log in the same file as given
+ by <literal>-l</literal>.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>server-apdu</literal></entry>
+ <entry>
+ Log APDUs as reported by YAZ for the
+ communication between the proxy and the server (backend).
+ </entry>
+ </row>
+ <row>
+ <entry><literal>clients-requests</literal></entry>
+ <entry>
+ Log a brief description about requests transferred between
+ the client and the proxy. The name of the request and the size
+ of the APDU is logged.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>server-requests</literal></entry>
+ <entry>
+ Log a brief description about requests transferred between
+ the proxy and the server (backend). The name of the request
+ and the size of the APDU is logged.
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ </para>
+ <para>
+ To log communication in details between the proxy and the backend, th
+ following configuration could be used:
+ <screen><![CDATA[
+ <target name="mytarget">
+ <log>server-apdu server-requests</log>
+ </target>
+]]>
</screen>
- </refsect1>
+ </para>
+ </section>
+
+ </section>
+ <section id="proxy-usage">
+ <title>Proxy Usage</title>
+ <para>
+ </para>
+ <refentry id="yaz-proxy">
+ &yaz-proxy-ref;
</refentry>
</section>
+ <section id="otherinfo-encoding"><title>OtherInformation Encoding</title>
+ <para>
+ The proxy uses the OtherInformation definition to carry
+ information about the target address and cookie.
+ </para>
+ <screen>
+ OtherInformation ::= [201] IMPLICIT SEQUENCE OF SEQUENCE{
+ category [1] IMPLICIT InfoCategory OPTIONAL,
+ information CHOICE{
+ characterInfo [2] IMPLICIT InternationalString,
+ binaryInfo [3] IMPLICIT OCTET STRING,
+ externallyDefinedInfo [4] IMPLICIT EXTERNAL,
+ oid [5] IMPLICIT OBJECT IDENTIFIER}}
+--
+ InfoCategory ::= SEQUENCE{
+ categoryTypeId [1] IMPLICIT OBJECT IDENTIFIER OPTIONAL,
+ categoryValue [2] IMPLICIT INTEGER}
+ </screen>
+ <para>
+ The <literal>categoryTypeId</literal> is either
+ OID 1.2.840.10003.10.1000.81.1, 1.2.840.10003.10.1000.81.2
+ for proxy target and proxy cookie respectively. The
+ integer element <literal>category</literal> is set to 0.
+ The value proxy and cookie is stored in element
+ <literal>characterInfo</literal> of the <literal>information</literal>
+ choice.
+ </para>
+ </section>
</chapter>
<!-- Keep this comment at the end of the file
Local variables:
sgml-namecase-general:t
End:
-->
+
projects / yazpp-moved-to-github.git / blobdiff