[yaz-moved-to-github.git] / doc / book.xml

<?xml version="1.0" standalone="no"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
    "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
[
     <!ENTITY % local SYSTEM "local.ent">
     %local;
     <!ENTITY % entities SYSTEM "entities.ent">
     %entities;
     <!ENTITY % idcommon SYSTEM "common/common.ent">
     %idcommon;
]>
<book>
 <bookinfo>
  <title>YAZ User&apos;s Guide and Reference</title>
  <authorgroup>
   <author><firstname>Sebastian</firstname><surname>Hammer</surname></author>
   <author><firstname>Adam</firstname><surname>Dickmeiss</surname></author>
   <author><firstname>Mike</firstname><surname>Taylor</surname></author>
   <author><firstname>Heikki</firstname><surname>Levanto</surname></author>
   <author><firstname>Dennis</firstname><surname>Schafroth</surname></author>
  </authorgroup>
  <releaseinfo>&version;</releaseinfo>
  <copyright>
   <year>&copyright-year;</year>
   <holder>Index Data</holder>
  </copyright>
  <abstract>
   <simpara>
    This document is the programmer's guide and reference to the &yaz;
    package version &version;. &yaz; is a compact toolkit that provides
    access to the Z39.50 and SRU/Solr protocols, as well as a set of
    higher-level tools for implementing the server and client
    roles, respectively.
    The documentation can be used on its own, or as a reference when
    looking at the example applications provided with the package.
   </simpara>
   <simpara>
    <inlinemediaobject>
     <imageobject>
      <imagedata fileref="common/id.png" format="PNG"/>
     </imageobject>
     <imageobject>
      <imagedata fileref="common/id.eps" format="EPS"/>
     </imageobject>
    </inlinemediaobject>
   </simpara></abstract>
 </bookinfo>
 <chapter id="introduction">
  <title>Introduction</title>
  <para>
   &yaz; is a C/C++ library for information retrieval applications
   using the Z39.50/SRU/Solr protocols for information retrieval.
  </para>
  <para>
   Properties of &yaz;:
   <itemizedlist>
    <listitem>
     <para>
      Complete
      <ulink url="&url.z39.50;">Z39.50</ulink> version 3 support.
      Amendments and Z39.50-2002 revision is supported.
    </para>
    </listitem>
    <listitem>
     <para>
      Supports
      <ulink url="&url.sru;">SRU GET/POST/SOAP</ulink>
      version 1.1, 1.2 and 2.0 (over HTTP and HTTPS).
    </para>
    </listitem>
    <listitem>
     <para>
      Includes BER encoders/decoders for the
      <ulink url="&url.ill;">ISO ILL</ulink>
      protocol.
    </para>
    </listitem>
    <listitem>
     <para>
      Supports
      <ulink url="&url.solr;">Solr</ulink> Web Service version 1.4.x
      (client side only)
     </para>
    </listitem>
    <listitem>
     <para>
      Supports the following transports: BER over TCP/IP
      (<ulink url="&url.ber.over.tcpip;">RFC1729</ulink>),
      BER over unix local socket, and
      <ulink url="&url.http.1.1;">HTTP 1.1</ulink>.
    </para>
    </listitem>
    <listitem>
     <para>
      Secure Socket Layer support using
      <ulink url="&url.gnutls;">GnuTLS</ulink>.
      If enabled, &yaz; uses HTTPS transport (for SOAP) or
      "Secure BER" (for Z39.50).
     </para>
    </listitem>
    <listitem>
     <para>
      Offers
      <ulink url="&url.zoom;">ZOOM</ulink> C API implementing
      Z39.50, SRU and Solr Web Service.
    </para>
    </listitem>
    <listitem>
     <para>
      The &yaz; library offers a set of useful utilities
      related to the protocols, such as MARC (ISO2709) parser,
      CCL (ISO8777) parser,
      <ulink url="&url.cql;">CQL</ulink>
      parser, memory management routines, character set conversion.
     </para>
    </listitem>
    <listitem>
     <para>
      Portable code. &yaz; compiles out-of-the box on most Unixes and
      on Windows using Microsoft Visual C++.
     </para>
    </listitem>
    <listitem>
     <para>
      Fast operation. The C based BER encoders/decoders as well
      as the server component of &yaz; is very fast.
    </para>
    </listitem>
    <listitem>
     <para>
      Liberal license that allows for commercial use of &yaz;.
     </para>
    </listitem>
   </itemizedlist>
  </para>

  <sect1 id="introduction.reading">
   <title>Reading this Manual</title>
   <para>
    Most implementors only need to read a fraction of the
    material in thie manual, so a quick walkthrough of the chapters
    is in order.
   </para>
   <itemizedlist>
    <listitem>
     <para>
      <xref linkend="installation"/> contains installation
      instructions for &yaz;. You don't need reading this
      if you expect to download &yaz; binaries.
      However, the chapter contains information about how
      to make <emphasis>your</emphasis> application link
      with &yaz;.
     </para>
    </listitem>
    <listitem>
     <para>
      <xref linkend="zoom"/> describes the ZOOM API of &yaz;.
      This is definitely worth a read if you wish to develop a Z39.50/SRU
      client.
     </para>
    </listitem>
    <listitem>
     <para>
      <xref linkend="server"/> describes the generic frontend server
      and explains how to develop server Z39.50/SRU applications for &yaz;.
      Obviously worth reading if you're to develop a server.
    </para>
    </listitem>
    <listitem>
     <para>
      <xref linkend="yaz-client"/> describes how to use the &yaz; Z39.50
      client. If you're developer and wish to test your server
      or a server from another party, you might find this chapter
      useful.
    </para>
    </listitem>
    <listitem>
     <para>
      <xref linkend="asn"/> documents the most commonly used Z39.50
      C data structures offered by the &yaz; API. Client
      developers using ZOOM and non-Z39.50 implementors may skip this.
     </para>
    </listitem>
    <listitem>
     <para>
      <xref linkend="soap"/> describes how SRU and SOAP is used
      in &yaz;. Only if you're developing SRU applications
      this section is a must.
     </para>
    </listitem>
    <listitem>
     <para>
      <xref linkend="tools"/> contains sections for the various
      tools offered by &yaz;. Scan through the material quickly
      and see what's relevant to you! SRU implementors
      might find the <link linkend="cql">CQL</link> section
      particularly useful.
     </para>
    </listitem>
    <listitem>
     <para>
      <xref linkend="odr"/> goes through the details of the
      ODR module which is the work horse that encodes and decodes
      BER packages. Implementors using ZOOM only, do <emphasis>not</emphasis>
      need reading this.
      Most other Z39.50 implementors only need to read the first two
      sections (<xref linkend="odr.introduction"/> and
      <xref linkend="odr.use"/>).
     </para>
    </listitem>
    <listitem>
     <para>
      <xref linkend="comstack"/> describes the network layer module
      COMSTACK. Implementors using ZOOM or the generic frontend server
      may skip this. Others, presumably, handling client/server
     communication on their own should read this.
     </para>
    </listitem>
   </itemizedlist>
  </sect1>
  <sect1 id="introduction.api">
   <title>The API</title>
   <para>
    The <ulink url="&url.yaz;">&yaz;</ulink>
    toolkit offers several different levels of access to the
    <ulink url="&url.z39.50;">ISO23950/Z39.50</ulink>,
    <ulink url="&url.ill;">ILL</ulink> and
    <ulink url="&url.sru;">SRU</ulink>
    protocols.
    The level that you need to use depends on your requirements, and
    the role (server or client) that you want to implement.
    If you're developing a client application you should consider the
    <link linkend="zoom">ZOOM</link> API.
    It is, by far, the easiest way to develop clients in C.
    Server implementers should consider the
    <link linkend="server">generic frontend server</link>.
    None of those high-level APIs support the whole protocol, but
    they do include most facilities used in existing Z39.50 applications.
   </para>
   <para>
    If you're using 'exotic' functionality (meaning anything not included in
    the high-level APIs), developing non-standard extensions to Z39.50 or
    you're going to develop an ILL application you'll have to learn the lower
    level APIs of &yaz;.
   </para>
   <para>
    The YAZ toolkit modules is shown in figure <xref linkend="yaz.layer"/>.
   </para>
   <figure id="yaz.layer">
    <title>YAZ layers</title>
    <mediaobject>
     <imageobject>
      <imagedata fileref="apilayer.png" format="PNG"/>
     </imageobject>
     <imageobject>
      <imagedata fileref="apilayer.eps" format="EPS"/>
     </imageobject>
    </mediaobject>
   </figure>
   <para>
    There are four layers.
    <itemizedlist>
     <listitem>
      <para>A client or server application (or both).
       This layer includes ZOOM and the generic frontend server.
      </para>
     </listitem>
     <listitem>
      <para>
       The second layer provides a C represenation of the
       protocol units (packages) for Z39.50 ASN.1, ILL ASN.1,
       SRU.
      </para>
     </listitem>
     <listitem>
      <para>
       The third layer encodes and decodes protocol data units to
       simple packages (buffer with certain length). The &odr; module
       encodes and decodes BER whereas the HTTP modules encodes and
       decodes HTTP ruquests/responses.
      </para>
     </listitem>
     <listitem>
      <para>
       The lowest layer is &comstack; which exchanges the encoded packages
       with a peer process over a network.
      </para>
     </listitem>
    </itemizedlist>
   </para>
   <para>
    The &asn; module represents the ASN.1 definition of
    the Z39.50 protocol. It establishes a set of type and
    structure definitions, with one structure for each of the top-level
    PDUs, and one structure or type for each of the contained ASN.1 types.
    For primitive types, or other types that are defined by the ASN.1
    standard itself (such as the EXTERNAL type), the C representation is
    provided by the &odr; (Open Data Representation) subsystem.
  </para>
   <para>
     &odr; is a basic mechanism for representing an
    ASN.1 type in the C programming language, and for implementing BER
    encoders and decoders for values of that type. The types defined in
    the &asn; module generally have the prefix <literal>Z_</literal>, and
    a suffix corresponding to the name of the type in the ASN.1
    specification of the protocol (generally Z39.50-1995). In the case of
    base types (those originating in the ASN.1 standard itself), the prefix
    <literal>Odr_</literal> is sometimes seen. Either way, look for
    the actual definition in either <filename>z-core.h</filename> (for the types
    from the protocol), <filename>odr.h</filename> (for the primitive ASN.1
    types).
    The &asn; library also provides functions (which are, in turn,
    defined using &odr; primitives) for encoding and decoding data values.
    Their general form is
    <funcsynopsis>
     <funcprototype><funcdef>int <function>z_<replaceable>xxx</replaceable></function></funcdef>
      <paramdef>ODR <parameter>o</parameter></paramdef>
      <paramdef>Z_<replaceable>xxx</replaceable> **<parameter>p</parameter></paramdef>
      <paramdef>int <parameter>optional</parameter></paramdef>
      <paramdef>const char *<parameter>name</parameter></paramdef>
     </funcprototype>
    </funcsynopsis>
    (note the lower-case &quot;z&quot; in the function name)
   </para>
   <note>
    <para>
     If you are using the premade definitions of the &asn; module, and you
     are not adding new protocol of your own, the only parts of &odr; that you
     need to worry about are documented in
     <xref linkend="odr.use"/>.
    </para>
   </note>
   <para>
    When you have created a BER-encoded buffer, you can use the &comstack;
    subsystem to transmit (or receive) data over the network. The &comstack;
    module provides simple functions for establishing a connection
    (passively or actively, depending on the role of your application),
    and for exchanging BER-encoded PDUs over that connection. When you
    create a connection endpoint, you need to specify what transport to
    use (TCP/IP, SSL or UNIX sockets).
    For the remainder of the connection's lifetime, you don't have
    to worry about the underlying transport protocol at all - the &comstack;
    will ensure that the correct mechanism is used.
   </para>
   <para>
    We call the combined interfaces to &odr;, &asn;, and &comstack; the service
    level API. It's the API that most closely models the Z39.50
    service/protocol definition, and it provides unlimited access to all
    fields and facilities of the protocol definitions.
   </para>
   <para>
    The reason that the &yaz; service-level API is a conglomerate of the
    APIs from three different submodules is twofold. First, we wanted to allow
    the user a choice of different options for each major task. For instance,
    if you don't like the protocol API provided by &odr;/&asn;, you
    can use SNACC or BERUtils instead, and still have the benefits of the
    transparent transport approach of the &comstack; module. Secondly,
    we realize that you may have to fit the toolkit into an existing
    event-processing structure, in a way that is incompatible with
    the &comstack; interface or some other part of &yaz;.
   </para>
  </sect1>
 </chapter>
 <chapter id="installation">
  <title>Compilation and Installation</title>
  <sect1 id="installation-introduction">
   <title>Introduction</title>
   <para>
    The latest version of the software will generally be found at:
   </para>
   <para>
    <ulink url="&url.yaz.download;"/>
   </para>
   <para>
    We have tried our best to keep the software portable, and on many
    platforms, you should be able to compile everything with little or
    no changes.
   </para>
   <para>
    The software is regularly tested on
    <ulink url="&url.debian;">Debian GNU/Linux</ulink>,
    <ulink url="&url.centos;">CentOS</ulink>,
    <ulink url="&url.ubuntu;">Ubuntu Linux</ulink>,
    <ulink url="&url.freebsd;">FreeBSD (i386)</ulink>,
    <ulink url="&url.macosx;">MAC OSX</ulink>,
    <ulink url="&url.solaris;">Solaris</ulink>,
    Windows 7, Windows XP.
   </para>
   <para>
    Some versions have be known to work on HP/UX,
    DEC Unix, <ulink url="&url.netbsd;">NetBSD</ulink>,
    <ulink url="&url.openbsd;">OpenBSD</ulink>,
    IBM AIX,
    Data General DG/UX (with some CFLAGS tinkering),
    SGI/IRIX, DDE Supermax, Apple Macintosh (using the Codewarrior programming
    environment and the GUSI socket libraries),
    IBM AS/400 .
   </para>
   <para>
    If you move the software to other platforms, we'd be grateful if you'd
    let us know about it. If you run into difficulties, we will try to help
    if we can, and if you solve the problems, we would be happy to include
    your fixes in the next release. So far, we have mostly avoided
    <literal>#ifdefs</literal> for individual platforms, and we'd
    like to keep it that way as far as it makes sense.
   </para>
   <para>
    We maintain a mailing-list for the purpose of announcing new releases and
    bug-fixes, as well as general discussion. Subscribe by
    filling-in the form
    <ulink url="&url.yaz.mailinglist;">here</ulink>.
    General questions and problems can be directed at
    <ulink url="&url.yaz.mail;"/>, or the address given at the top of
     this document.
   </para>
  </sect1>
  <sect1 id="installation.unix"><title>UNIX</title>
   <para>
    We provide
    <ulink url="&url.debian;">Debian GNU/Linux</ulink> (i386 and amd64),
    <ulink url="&url.ubuntu;">Ubuntu</ulink> (i386 and amd64)
    and
    <ulink url="&url.centos;">CentOS</ulink> (amd64 only) packages for &yaz;.
    You should be able to create packages for other CPUs by building
    them from the source package.
   </para>
   <para>
    YAZ is also part of several packages repositories. Some of them are
   </para>
   <itemizedlist>
    <listitem>
     <para>
      Solaris CSW: <ulink url="http://www.opencsw.org/packages/yaz/"/>
     </para>
    </listitem>
    <listitem>
     <para>
      Solaris: <ulink url="http://unixpackages.com"/>
     </para>
    </listitem>
    <listitem>
     <para>
      FreeBSD: <ulink url="http://www.freshports.org/net/yaz"/>
     </para>
    </listitem>
    <listitem>
     <para>
      Debian: <ulink url="http://packages.debian.org/search?keywords=yaz"/>
     </para>
    </listitem>
    <listitem>
     <para>
      Ubuntu: <ulink url="https://launchpad.net/ubuntu/+source/yaz"/>
     </para>
    </listitem>
    <listitem>
     <para>
      NetBSD:
      <ulink url="http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/net/yaz/README.html"/>
     </para>
    </listitem>
   </itemizedlist>
   <sect2 id="installation.source.unix">
    <title>Compiling from source on Unix</title>
    <para>
     Note that if your system doesn't have a native ANSI C compiler, you may
     have to acquire one separately. We recommend
     <ulink url="&url.gcc;">GCC</ulink>.
    </para>
    <para>
     If you wish to use character set conversion facilities in &yaz; or if you
     are compiling &yaz; for use with Zebra it is a good idea to ensure that
     the iconv library is installed. Some Unixes today already have it
     - if not, we suggest
     <ulink url="&url.libiconv;">GNU libiconv</ulink>.
    </para>
    <para>
     YAZ 3.0.16 and later includes a wrapper for the
     <ulink url="&url.icu;">ICU</ulink>
     (International Components for Unicode).
     In order to use this, the developer version of the ICU library
     must be available. ICU support is recommended for applications
     such as Pazpar2 and Zebra.
    </para>
    <para>
     The <ulink url="&url.libxslt;">libxslt</ulink>,
     <ulink url="&url.libxml2;">libxml2</ulink> librararies are required
     if &yaz; is to support SRU/Solr.
     These libraries are very portable and should compile out-of-the
     box on virtually all Unix platforms. It is available in binary
     forms for Linux and others.
    </para>
    <para>
     The GNU tools
     <ulink url="&url.autoconf;">Autoconf</ulink>,
     <ulink url="&url.automake;">Automake</ulink> and
     <ulink url="&url.libtool;">Libtool</ulink>
     are used to generate Makefiles and configure &yaz; for the system.
     You do <emphasis>not</emphasis> these tools unless you're using the
     Git version of &yaz;.
    </para>
    <para>
     The CQL parser for &yaz; is built using
     GNU <ulink url="&url.bison;">Bison</ulink>.
     This tool is only needed if you're using the Git version of &yaz;.
    </para>
    <para>
     &yaz; includes a tiny ASN.1 compiler. This compiler is
     written in <ulink url="&url.tcl;">Tcl</ulink>.
     But as for Bison you do not need it unless you're using Git
     version of &yaz; or you're using the compiler to built own codecs
     for private ASN.1.
    </para>
    <para>
     Generally it should be sufficient to run configure without options,
     like this:
    </para>
    <screen>
     ./configure
    </screen>
    <para>
     The configure script attempts to use use the C compiler specified by
     the <literal>CC</literal> environment variable. If not set, GNU C will be
     used if it is available. The <literal>CFLAGS</literal> environment
     variable holds options to be passed to the C compiler. If you're using
     Bourne-compatible shell you may pass something like this to use a
     particular C compiler with optimization enabled:
    </para>
    <screen>
     CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
    </screen>
    <para>
     To customize &yaz;, the configure script also accepts a set of options.
     The most important are:
     <variablelist>
      <varlistentry>
       <term>
        <literal>--prefix</literal>=<replaceable>prefix</replaceable>
       </term>
       <listitem>
        <para>Specifies installation prefix for &yaz;. This is
        only needed if you run <literal>make install</literal> later to
        perform a "system" installation. The prefix is
        <literal>/usr/local</literal> if not specified.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--enable-tcpd</literal>
       </term>
       <listitem>
        <para>The front end server will be built using Wietse's
        <ulink url="&url.tcpwrapper;">TCP wrapper library</ulink>.
        It allows you to allow/deny clients depending on IP number.
        The TCP wrapper library is often used in GNU/Linux and
        BSD distributions.
        See
        <citerefentry>
         <refentrytitle>hosts_access</refentrytitle>
         <manvolnum>5</manvolnum>
        </citerefentry>
        and
        <citerefentry>
         <refentrytitle>tcpd</refentrytitle>
         <manvolnum>8</manvolnum>
         </citerefentry>.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--enable-threads</literal>
       </term>
       <listitem>
        <para>&yaz; will be built using POSIX threads.
        Specifically, <constant>_REENTRANT</constant> will be defined during
        compilation.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--disable-shared</literal>
       </term>
       <listitem>
        <para>The make process will not create shared
        libraries (also known as shared objects <filename>.so</filename>).
        By default, shared libraries are created -
        equivalent to <literal>--enable-shared</literal>.
       </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--disable-shared</literal>
       </term>
       <listitem>
        <para>The make process will not create
        static libraries (<filename>.a</filename>).
        By default, static libraries are created -
        equivalent to <literal>--enable-static</literal>.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--with-iconv</literal>[=<replaceable>prefix</replaceable>]
       </term>
       <listitem>
        <para>Compile &yaz; with iconv library in directory
        <replaceable>prefix</replaceable>. By default configure will
        search for iconv on the system. Use this option if it
        doesn't find iconv. Alternatively,
        <literal>--without-iconv</literal>, can be uset to force &yaz;
        not to use iconv.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--with-xslt</literal>[=<replaceable>prefix</replaceable>]
       </term>
       <listitem>
        <para>Compile &yaz; with
        <ulink url="&url.libxslt;">libxslt</ulink> in directory
        <replaceable>prefix</replaceable>.
        Use this option if you want XSLT and XML support.
        By default, configure will
        search for libxslt on the system. Use this option if it
        libxslt is not found automatically. Alternatively,
        <literal>--without-xslt</literal>, can be used to force &yaz;
        not to use libxslt.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--with-xml2</literal>[=<replaceable>prefix</replaceable>]
       </term>
       <listitem>
        <para>Compile &yaz; with
        <ulink url="&url.libxml2;">libxml2</ulink> in directory
        <replaceable>prefix</replaceable>.
        Use this option if you want &yaz; to use XML and support SRU/Solr.
        By default, configure will
        search for libxml2 on the system. Use this option if it
        libxml2 is not found automatically. Alternatively,
        <literal>--without-xml2</literal>, can be used to force &yaz;
        not to use libxml2.
        </para>
        <para>
         Note that option <literal>--with-xslt</literal>
         also enables libxml2.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--with-gnutls</literal>[=<replaceable>prefix</replaceable>]
       </term>
       <listitem>
        <para>&yaz; will be linked with the GNU TLS libraries and
        an SSL COMSTACK will be provided. By default configure enables
        SSL support for YAZ if the GNU TLS development libraries are found
        on the system.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--with-icu</literal>[=<replaceable>prefix</replaceable>]
       </term>
       <listitem>
        <para>&yaz; will be linked the
        <ulink url="&url.icu;">ICU</ulink> library in the prefix if given.
        If prefix is not given, the libraries exposed by the script
        <application>icu-config</application> will be used if found.
        </para>
       </listitem>
      </varlistentry>

      <varlistentry>
       <term>
        <literal>--with-libgcrypt</literal>[=<replaceable>prefix</replaceable>]
       </term>
       <listitem>
        <para>&yaz; will be linked with
        <ulink url="&url.libgcrypt;">Libgcrypt</ulink> in the prefix if given.
        If prefix is not given, the libraries exposed by the script
        <application>libgcrypt-config</application> will be used if found.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--with-memcached</literal>
       </term>
       <listitem>
        <para>&yaz; will be linked with
        <ulink url="&url.libmemcached;">libMemcached</ulink> to allow
        for result-set caching for ZOOM.
        The prefix can not be given. Note that YAZ will only search
        for libMemcached if Libgcrypt is also enabled.
        Note that 0.40 of libmemcached is required.
       </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term>
        <literal>--with-redis</literal>
       </term>
       <listitem>
        <para>&yaz; will be linked with the hiredis C library
        to allow for result-set caching for ZOOM on a
        <ulink url="&url.redis;">redis</ulink> server.
        The prefix can not be given. Note that YAZ will only search
        for hiredis if Libgcrypt is also enabled.
       </para>
       </listitem>
      </varlistentry>

     </variablelist>
    </para>
    <para>
     When configured, build the software by typing:
     <screen>
      make
     </screen>
    </para>
    <para>
     The following files are generated by the make process:
     <variablelist>
      <varlistentry>
       <term><filename>src/libyaz.la</filename></term>
       <listitem><para>
        Main &yaz; library. This is no ordinary library. It's
        a Libtool archive.
        By default, &yaz; creates a static library in
        <filename>lib/.libs/libyaz.a</filename>.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>src/libyaz_server.la</filename></term>
       <listitem><para>
         Generic Frontend server. This is an add-on for libyaz.la.
         Code in this library uses POSIX threads functions - if POSIX
         threads are available on the platform.
        </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>src/libyaz_icu.la</filename></term>
       <listitem><para>
        Functions that wrap the ICU library.
        </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>ztest/yaz-ztest</filename></term>
       <listitem><para>Test Z39.50 server.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>client/yaz-client</filename></term>
       <listitem><para>Z39.50 client for testing the protocol.
       See chapter <link linkend="yaz-client">
       YAZ client</link> for more information.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>util/yaz-config</filename></term>
       <listitem><para>A Bourne-shell script, generated by configure, that
       specifies how external applications should compile - and link with
       &yaz;.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>util/yaz-asncomp</filename></term>
       <listitem><para>The ASN.1 compiler for &yaz;. Requires the
       Tcl Shell, <application>tclsh</application>, in
       <literal>PATH</literal> to operate.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>util/yaz-iconv</filename></term>
       <listitem><para>This program converts data in one character set to
       another. This command exercises the YAZ character set
       conversion API.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>util/yaz-marcdump</filename></term>
       <listitem><para>This program parses ISO2709 encoded MARC records
       and prints them in line-format or XML.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>util/yaz-icu</filename></term>
       <listitem><para>This program exposes the ICU wrapper library if that
       is enabled for YAZ. Only if ICU is available this program is
       useful.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>util/yaz-url</filename></term>
       <listitem><para>This program is a simple HTTP page fetcher ala
       wget or curl.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>zoom/zoomsh</filename></term>
       <listitem><para>
        A simple shell implemented on top of the
        <link linkend="zoom">ZOOM</link> functions.
        The shell is a command line application that allows you to enter
        simple commands to perform ZOOM operations.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>zoom/zoomtst1</filename>,
       <filename>zoom/zoomtst2</filename>, ..</term>
       <listitem><para>
        Several small applications that demonstrates the ZOOM API.
       </para></listitem>
      </varlistentry>
     </variablelist>
    </para>
    <para>
     If you wish to install &yaz; in system directories
     <filename>/usr/local/bin</filename>,
     <filename>/usr/local/lib</filename> .. etc, you can type:
    </para>
    <screen>
     make install
    </screen>
    <para>
     You probably need to have root access in order to perform this.
     You must specify the <literal>--prefix</literal> option for configure if
     you wish to install &yaz; in other directories than the default
     <filename>/usr/local/</filename>.
    </para>
    <para>
     If you wish to perform an un-installation of &yaz;, use:
    </para>
    <screen>
     make uninstall
    </screen>
    <para>
     This will only work if you haven't reconfigured &yaz; (and therefore
     changed installation prefix). Note that uninstall will not
     remove directories created by make install, e.g.
     <filename>/usr/local/include/yaz</filename>.
    </para>
   </sect2>
   <sect2 id="installation-linking-yaz-unix">
    <title>How to make apps using YAZ on UNIX</title>
    <para>
     This section describes how to compile - and link your own
     applications using the &yaz; toolkit.
     If you're used to Makefiles this shouldn't be hard. As for
     other libraries you have used before, you have to set a proper include
     path for your C/C++ compiler and specify the location of
     &yaz; libraries. You can do it by hand, but generally we suggest
     you use the <filename>yaz-config</filename> that is generated
     by <filename>configure</filename>. This is especially
     important if you're using the threaded version of &yaz; which
     require you to pass more options to your linker/compiler.
    </para>
    <para>
     The <filename>yaz-config</filename> script accepts command line
     options that makes the <filename>yaz-config</filename> script print
     options that you should use in your make process.
     The most important ones are:
     <literal>--cflags</literal>, <literal>--libs</literal>
     which prints C compiler flags, and linker flags respectively.
    </para>
    <para>
     A small and complete <literal>Makefile</literal> for a C
     application consisting of one source file,
     <filename>myprog.c</filename>, may look like this:
     <screen>
      YAZCONFIG=/usr/local/bin/yaz-config
      CFLAGS=`$(YAZCONFIG) --cflags`
      LIBS=`$(YAZCONFIG) --libs`
      myprog: myprog.o
         $(CC) $(CFLAGS) -o myprog myprog.o $(LIBS)
      </screen>
    </para>
    <para>
     The CFLAGS variable consists of a C compiler directive that will set
     the include path to the <emphasis>parent</emphasis> directory
     of <filename>yaz</filename>. That is, if &yaz; header files were
     installed in <filename>/usr/local/include/yaz</filename>,
     then include path is set to <filename>/usr/local/include</filename>.
     Therefore, in your applications you should use
     <screen>
      #include &lt;yaz/proto.h>
     </screen>
     and <emphasis>not</emphasis>
     <screen>
      #include &lt;proto.h>
     </screen>
    </para>
    <para>
     For Libtool users, the <filename>yaz-config</filename> script provides
     a different variant of option <literal>--libs</literal>, called
     <literal>--lalibs</literal> that returns the name of the
     Libtool archive(s) for &yaz; rather than the ordinary ones.
    </para>
    <para>
     For applications using the threaded version of &yaz;,
     specify <literal>threads</literal> after the
     other options. When <literal>threads</literal> is given,
     more flags and linker flags will be printed by
     <filename>yaz-config</filename>. If our previous example was
      using threads, you'd have to modify the lines that set
     <literal>CFLAGS</literal> and <literal>LIBS</literal> as
     follows:
     <screen>
      CFLAGS=`$(YAZCONFIG) --cflags threads`
      LIBS=`$(YAZCONFIG) --libs threads`
     </screen>
     There is no need specify POSIX thread libraries in your Makefile.
     The <literal>LIBS</literal> variable includes that as well.
    </para>
   </sect2>
  </sect1>
  <sect1 id="installation.win32">
   <title>Windows</title>
   <para>The easiest way to install YAZ on Windows is by downloading
   an installer from
   <ulink url="&url.yaz.download.win32;">here</ulink>.
   The installer comes with source too - in case you wish to
   compile YAZ with different compiler options, etc.
   </para>

   <sect2 id="installation.win32.source">
    <title>Compiling from Source on Windows</title>
    <para>
     &yaz; is shipped with "makefiles" for the NMAKE tool that comes
     with <ulink url="&url.vstudio;">
     Microsoft Visual Studio</ulink>. It has been tested with
     Microsoft Visual Studio 2013.
    </para>
    <para>
     Start a command prompt and switch the sub directory
     <filename>WIN</filename> where the file <filename>makefile</filename>
     is located. Customize the installation by editing the
     <filename>makefile</filename> file (for example by using notepad).
     The following summarizes the most important settings in that file:
     <variablelist>
      <varlistentry>
       <term><literal>DEBUG</literal></term>
       <listitem><para>
        If set to 1, the software is
        compiled with debugging libraries (code generation is
        multi-threaded debug DLL).
        If set to 0, the software is compiled with release libraries
        (code generation is multi-threaded DLL).
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>HAVE_TCL</literal>, <literal>TCL</literal></term>
       <listitem><para>
        If <literal>HAVE_TCL</literal> is set to 1, nmake will
        use the ASN.1 compiler (<ulink url="&url.tcl;">Tcl</ulink> based).
        You must set <literal>TCL</literal> to the full path of the Tcl
        interpreter. A Windows version of Tcl is part of
        <ulink url="&url.gitwindows;">Git for Windows</ulink>.
       </para>
       <para>
        If you do not have Tcl installed, set
        <literal>HAVE_TCL</literal> to 0.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>HAVE_BISON</literal>,
       <literal>BISON</literal></term>
       <listitem><para>
        If GNU Bison is present, you might set <literal>HAVE_BISON</literal>
        to 1 and specify the Bison executable in <literal>BISON</literal>.
        Bison is only required if you use the Git version of
        YAZ or if you modify the grammar for CQL
        (<filename>cql.y</filename>).
       </para>
       <para>
        A Windows version of GNU Bison is part of
        <ulink url="&url.gitwindows;">Git for Windows</ulink>.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>HAVE_ICONV</literal>,
       <literal>ICONV_DIR</literal></term>
       <listitem><para>
        If <literal>HAVE_ICONV</literal> is set to 1, YAZ is compiled
        with iconv support. In this configuration, set
        <literal>ICONV_DIR</literal> to the iconv source directory.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>HAVE_LIBXML2</literal>,
       <literal>LIBXML2_DIR</literal></term>
       <listitem>
        <para>
         If <literal>HAVE_LIBXML2</literal> is set to 1, YAZ is compiled
         with SRU support. In this configuration, set
         <literal>LIBXML2_DIR</literal> to the
         <ulink url="&url.libxml2;">libxml2</ulink> source directory.
        </para>
        <para>
         You can get pre-compiled Libxml2+Libxslt DLLs and headers from
         <ulink url="&url.libxml2.download.windows;">here</ulink>.
         Should you with to compile those libraries yourself, refer to
         to <xref linkend="installation.windows.libxml2"/>
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>HAVE_LIBXSLT</literal>,
       <literal>LIBXSLT_DIR</literal></term>
       <listitem>
        <para>
         If <literal>HAVE_LIBXSLT</literal> is set to 1, YAZ is compiled
         with XSLT support. In this configuration, set
         <literal>LIBXSLT_DIR</literal> to the
         <ulink url="&url.libxslt;">libxslt</ulink> source directory.
        </para>
        <note>
         <para>
          libxslt depends libxml2.
         </para>
        </note>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>HAVE_ICU</literal>,
       <literal>ICU_DIR</literal></term>
       <listitem>
        <para>
         If <literal>HAVE_ICU</literal> is set to 1, YAZ is compiled
         with <ulink url="&url.icu;">ICU</ulink> support.
         In this configuration, set
         <literal>ICU_DIR</literal> to the
         <ulink url="&url.icu;">ICU</ulink> source directory.
        </para>
       </listitem>
      </varlistentry>
     </variablelist>
    </para>
    <para>
     When satisfied with the settings in the makefile, type
     <screen>
      nmake
     </screen>
    </para>
    <note>
     <para>
      If the <filename>nmake</filename> command is not found on your system
      you probably haven't defined the environment variables required to
      use that tool. To fix that, find and run the batch file
      <filename>vcvars32.bat</filename>. You need to run it from within
      the command prompt or set the environment variables "globally";
      otherwise it doesn't work.
     </para>
    </note>
    <para>
     If you wish to recompile &yaz; - for example if you modify
     settings in the <filename>makefile</filename> you can delete
     object files, etc by running.
     <screen>
      nmake clean
     </screen>
    </para>
    <para>
     The following files are generated upon successful compilation:
     <variablelist>
      <varlistentry>
       <term><filename>bin/yaz&soversion;.dll</filename> /
       <filename>bin/yaz&soversion;d.dll</filename></term>
       <listitem><para>
        &yaz; Release/Debug DLL.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>lib/yaz&soversion;.lib</filename> /
       <filename>lib/yaz&soversion;d.lib</filename></term>
       <listitem><para>
        Import library for <filename>yaz&soversion;.dll</filename> /
        <filename>yaz&soversion;d.dll</filename>.
      </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>bin/yaz_cond&soversion;.dll</filename> /
       <filename>bin/yaz_cond&soversion;d.dll</filename></term>
       <listitem><para>
        Release/Debug DLL for condition variable utilities (condvar.c).
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>lib/yaz_cond&soversion;.lib</filename> /
       <filename>lib/yaz_cond&soversion;d.lib</filename></term>
       <listitem><para>
        Import library for <filename>yaz_cond&soversion;.dll</filename> /
        <filename>yaz_cond&soversion;d.dll</filename>.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>bin/yaz_icu&soversion;.dll</filename> /
       <filename>bin/yaz_icu&soversion;d.dll</filename></term>
       <listitem><para>
        Release/Debug DLL for the ICU wrapper utility.
        Only build if HAVE_ICU is 1.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>lib/yaz_icu&soversion;.lib</filename> /
       <filename>lib/yaz_icu&soversion;d.lib</filename></term>
       <listitem><para>
        Import library for <filename>yaz_icu&soversion;.dll</filename> /
        <filename>yaz_icu&soversion;d.dll</filename>.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>bin/yaz-ztest.exe</filename></term>
       <listitem><para>
        Z39.50 multi-threaded test/example server. It's a WIN32
        console application.
      </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>bin/yaz-client.exe</filename></term>
       <listitem><para>
        &yaz; Z39.50 client application. It's a WIN32 console application.
        See chapter <link linkend="yaz-client">YAZ client</link> for more
        information.
      </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>bin/yaz-icu.exe</filename></term>
       <listitem><para>This program exposes the ICU wrapper library if that
       is enabled for YAZ. Only if ICU is available this program is
       build.
      </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>bin/zoomsh.exe</filename></term>
       <listitem><para>
        Simple console application implemented on top of the
        <link linkend="zoom">ZOOM</link> functions.
        The application is a command line shell that allows you to enter
        simple commands to perform ZOOM operations.
      </para></listitem>
      </varlistentry>
      <varlistentry>
       <term><filename>bin/zoomtst1.exe</filename>,
       <filename>bin/zoomtst2.exe</filename>, ..</term>
       <listitem><para>
        Several small applications that demonstrates the ZOOM API.
      </para></listitem>
      </varlistentry>
     </variablelist>
    </para>
   </sect2>

   <sect2 id="installation-linking-yaz-win32">
    <title>How to make apps using YAZ on Windows</title>
    <para>
     This section will go though the process of linking your Windows
     applications with &yaz;.
    </para>
    <para>
     Some people are confused by the fact that we use the nmake
     tool to build &yaz;. They think they have to do that too - in order
     to make their Windows applications work with &yaz;. The good news is that
     you don't have to. You can use the integrated environment of
     Visual Studio if desired for your own application.
    </para>
    <para>
     When setting up a project or Makefile you have to set the following:
     <variablelist>
      <varlistentry>
       <term>include path</term>
       <listitem><para>
        Set it to the <filename>include</filename> directory of &yaz;.
        </para></listitem>
      </varlistentry>
      <varlistentry>
       <term>import library <filename>yaz&soversion;.lib</filename></term>
       <listitem><para>
        You must link with this library. It's located in the
        sub directory <filename>lib</filename> of &yaz;.
        If you want to link with the debug version of &yaz;, you must
        link against <filename>yaz&soversion;d.lib</filename> instead.
       </para></listitem>
      </varlistentry>
      <varlistentry>
       <term>dynamic link library
       <filename>yaz&soversion;.dll</filename>
       </term>
       <listitem><para>
        This DLL must be in your execution path when you invoke
        your application. Specifically, you should distribute this
        DLL with your application.
       </para></listitem>
      </varlistentry>
     </variablelist>
    </para>
   </sect2>

   <sect2 id="installation.windows.libxml2">
    <title>Compiling Libxml2 and Libxslt on windows</title>
    <para>
     Download libxml2 and Libxslt source and unpack it.
     In the example below we install  Libxml2 2.9.2 and Libxslt 1.1.28
     for 32-bit, so we  use the destination directories
     libxml2.2.9.2.win32 and libxslt-1.1.28.win32 to reflect both
     version and architecture.
     <screen>
      cd win32
      cscript configure.js prefix=c:\libxml2-2.9.2.win32 iconv=no
      nmake
      nmake install
     </screen>
    </para>
    <para>
     For Libxslt it is similar. We must ensure that compilation of
     Libxslt links against the already installed libxml2.
     <screen>
      cd win32
      cscript configure.js prefix=c:\libxslt-1.1.28.win32 iconv=no \
          lib=c:\libxmlt-2.9.2.win32\lib \
          include=c:\libxmlt-2.9.2.win32\include\libxml2
      nmake
      nmake install
     </screen>
    </para>
   </sect2>

  </sect1>
 </chapter>
 <!--
     ### Still to document:
     ZOOM_connection_errcode(c)
     ZOOM_connection_errmsg(c)
     ZOOM_connection_addinfo(c)
     ZOOM_connection_addinfo(c)
     ZOOM_connection_diagset(c);
     ZOOM_connection_save_apdu_wrbuf
     ZOOM_diag_str(error)
     ZOOM_resultset_record_immediate(s, pos)
     ZOOM_resultset_cache_reset(r)
     ZOOM_options_set_callback(opt, function, handle)
     ZOOM_options_create_with_parent2(parent1, parent2)
     ZOOM_options_getl(opt, name, len)
     ZOOM_options_setl(opt, name, value, len)
     ZOOM_options_get_bool(opt, name, defa)
     ZOOM_options_get_int(opt, name, defa)
     ZOOM_options_set_int(opt, name, value)
 -->
 <chapter id="zoom">
  <title>ZOOM</title>
  <para>
   &zoom; is an acronym for 'Z39.50 Object-Orientation Model' and is
   an initiative started by Mike Taylor (Mike is from the UK, which
   explains the peculiar name of the model). The goal of &zoom; is to
   provide a common Z39.50 client API not bound to a particular
   programming language or toolkit.
  </para>
  <para>
   From YAZ version 2.1.12, <ulink url="&url.sru;">SRU</ulink> is supported.
   You can make SRU ZOOM connections by specifying scheme
   <literal>http://</literal> for the hostname for a connection.
   The dialect of SRU used is specified by the value of the
   connection's <literal>sru</literal> option, which may be SRU over
   HTTP GET (<literal>get</literal>),
   SRU over HTTP POST (<literal>post</literal>), (SRU over
   SOAP) (<literal>soap</literal>) or <literal>solr</literal>
   (<ulink url="&url.solr;">Solr</ulink> Web Service).
   Using the facility for embedding options in target strings, a
   connection can be forced to use SRU rather the SRW (the default) by
   prefixing the target string with <literal>sru=get,</literal>, like this:
   <literal>sru=get,http://sru.miketaylor.org.uk:80/sru.pl</literal>
  </para>
  <para>
   <ulink url="&url.solr;">Solr</ulink>  protocol support was added to
   YAZ in version 4.1.0, as a dialect of a SRU protocol, since both are
   HTTP based protocols.
  </para>
  <para>
   The lack of a simple Z39.50 client API for &yaz; has become more
   and more apparent over time. So when the first &zoom; specification
   became available,
   an implementation for &yaz; was quickly developed. For the first time, it is
   now as easy (or easier!) to develop clients than servers with &yaz;. This
   chapter describes the &zoom; C binding. Before going further, please
   reconsider whether C is the right programming language for the job.
   There are other language bindings available for &yaz;, and still
   more
   are in active development. See the
   <ulink url="&url.zoom;">ZOOM web-site</ulink> for
   more information.
  </para>
  <para>
   In order to fully understand this chapter you should read and
   try the example programs <literal>zoomtst1.c</literal>,
   <literal>zoomtst2.c</literal>, .. in the <literal>zoom</literal>
   directory.
  </para>
  <para>
   The C language misses features found in object oriented languages
   such as C++, Java, etc. For example, you'll have to manually,
   destroy all objects you create, even though you may think of them as
   temporary. Most objects has a <literal>_create</literal> - and a
   <literal>_destroy</literal> variant.
   All objects are in fact pointers to internal stuff, but you don't see
   that because of typedefs. All destroy methods should gracefully ignore a
   <literal>NULL</literal> pointer.
  </para>
  <para>
   In each of the sections below you'll find a sub section called
   protocol behavior, that describes how the API maps to the Z39.50
   protocol.
  </para>
  <sect1 id="zoom-connections">
   <title>Connections</title>
   <para>The Connection object is a session with a target.
   </para>
   <synopsis>
    #include &lt;yaz/zoom.h>

    ZOOM_connection ZOOM_connection_new(const char *host, int portnum);

    ZOOM_connection ZOOM_connection_create(ZOOM_options options);

    void ZOOM_connection_connect(ZOOM_connection c, const char *host,
                                 int portnum);
    void ZOOM_connection_destroy(ZOOM_connection c);
   </synopsis>
   <para>
    Connection objects are created with either function
    <function>ZOOM_connection_new</function> or
    <function>ZOOM_connection_create</function>.
    The former creates and automatically attempts to establish a network
    connection with the target. The latter doesn't establish
    a connection immediately, thus allowing you to specify options
    before establishing network connection using the function
    <function>ZOOM_connection_connect</function>.
    If the port number, <literal>portnum</literal>, is zero, the
    <literal>host</literal> is consulted for a port specification.
    If no port is given, 210 is used. A colon denotes the beginning of
    a port number in the host string. If the host string includes a
    slash, the following part specifies a database for the connection.
   </para>
   <para>
    You can prefix the host with a scheme followed by colon. The
    default scheme is <literal>tcp</literal> (Z39.50 protocol).
    The scheme <literal>http</literal> selects SRU/get over HTTP by default,
    but can overridded to use SRU/post, SRW and the Solr protocol.
   </para>
   <para>
    You can prefix the scheme-qualified host-string with one or more
    comma-separated
    <literal><parameter>key</parameter>=<parameter>value</parameter></literal>
    sequences, each of which represents an option to be set into the
    connection structure <emphasis>before</emphasis> the
    protocol-level connection is forged and the initialization
    handshake takes place.  This facility can be used to provide
    authentication credentials, as in host-strings such as:
    <literal>user=admin,password=halfAm4n,tcp:localhost:8017/db</literal>
   </para>
   <para>
    Connection objects should be destroyed using the function
    <function>ZOOM_connection_destroy</function>.
   </para>
   <synopsis>
    void ZOOM_connection_option_set(ZOOM_connection c,
                                    const char *key, const char *val);

    void ZOOM_connection_option_setl(ZOOM_connection c,
                                     const char *key,
                                     const char *val, int len);

    const char *ZOOM_connection_option_get(ZOOM_connection c,
                                           const char *key);
    const char *ZOOM_connection_option_getl(ZOOM_connection c,
                                            const char *key,
                                            int *lenp);
   </synopsis>
   <para>
    The functions <function>ZOOM_connection_option_set</function> and
    <function>ZOOM_connection_option_setl</function> allows you to
    set an option given by <parameter>key</parameter> to the value
    <parameter>value</parameter> for the connection.
    For <function>ZOOM_connection_option_set</function>, the
    value is assumed to be a 0-terminated string. Function
    <function>ZOOM_connection_option_setl</function> specifies a
    value of a certain size (len).
   </para>
   <para>
    Functions <function>ZOOM_connection_option_get</function> and
    <function>ZOOM_connection_option_getl</function> returns
    the value for an option given by <parameter>key</parameter>.
   </para>
   <table id="zoom-connection-options" frame="top">
    <title>ZOOM Connection Options</title>
    <tgroup cols="3">
     <colspec colwidth="4*" colname="name"></colspec>
     <colspec colwidth="7*" colname="description"></colspec>
     <colspec colwidth="3*" colname="default"></colspec>
     <thead>
      <row>
       <entry>Option</entry>
       <entry>Description</entry>
       <entry>Default</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
       implementationName</entry><entry>Name of Your client
      </entry><entry>none</entry></row>
      <row><entry>
       user</entry><entry>Authentication user name
      </entry><entry>none</entry></row>
      <row><entry>
       group</entry><entry>Authentication group name
      </entry><entry>none</entry></row>
      <row><entry>
       password</entry><entry>Authentication password.
      </entry><entry>none</entry></row>
      <row><entry>
       authenticationMode</entry><entry>How authentication is encoded.
      </entry><entry>basic</entry></row>
      <row><entry>
       host</entry><entry>Target host. This setting is "read-only".
       It's automatically set internally when connecting to a target.
      </entry><entry>none</entry></row>
      <row><entry>
       proxy</entry><entry>Proxy host. If set, the logical host
       is encoded in the otherInfo area of the Z39.50 Init PDU
       with OID 1.2.840.10003.10.1000.81.1.
      </entry><entry>none</entry></row>
      <row><entry>
       clientIP</entry><entry>Client IP. If set, is
       encoded in the otherInfo area of a Z39.50 PDU with OID
       1.2.840.10003.10.1000.81.3. Holds the original IP addreses
       of a client. Is used of ZOOM is used in a gateway of some sort.
      </entry><entry>none</entry></row>
      <row><entry>
       async</entry><entry>If true (1) the connection operates in
       asynchronous operation which means that all calls are non-blocking
       except
       <link linkend="zoom.events"><function>ZOOM_event</function></link>.
      </entry><entry>0</entry></row>
      <row><entry>
       maximumRecordSize</entry><entry> Maximum size of single record.
      </entry><entry>1 MB</entry></row>
      <row><entry>
       preferredMessageSize</entry><entry> Maximum size of multiple records.
      </entry><entry>1 MB</entry></row>
      <row><entry>
       lang</entry><entry> Language for negotiation.
      </entry><entry>none</entry></row>
      <row><entry>
       charset</entry><entry> Character set for negotiation.
      </entry><entry>none</entry></row>
      <row><entry>
       serverImplementationId</entry><entry>
       Implementation ID of server.  (The old targetImplementationId
       option is also supported for the benefit of old applications.)
      </entry><entry>none</entry></row>
      <row><entry>
       targetImplementationName</entry><entry>
       Implementation Name of server.  (The old
       targetImplementationName option is also supported for the
       benefit of old applications.)
      </entry><entry>none</entry></row>
      <row><entry>
       serverImplementationVersion</entry><entry>
       Implementation Version of server.  (the old
       targetImplementationVersion option is also supported for the
       benefit of old applications.)
      </entry><entry>none</entry></row>
      <row><entry>
       databaseName</entry><entry>One or more database names
       separated by character plus (<literal>+</literal>), which to
       be used by subsequent search requests on this Connection.
      </entry><entry>Default</entry></row>
      <row><entry>
       piggyback</entry><entry>True (1) if piggyback should be
       used in searches; false (0) if not.
      </entry><entry>1</entry></row>
      <row><entry>
       smallSetUpperBound</entry><entry>If hits is less than or equal to this
       value, then target will return all records using small element set name
      </entry><entry>0</entry></row>
      <row><entry>
       largeSetLowerBound</entry><entry>If hits is greater than this
       value, the target will return no records.
      </entry><entry>1</entry></row>
      <row><entry>
       mediumSetPresentNumber</entry><entry>This value represents
       the number of records to be returned as part of a search when when
       hits is less than or equal to large set lower bound and if hits
       is greater than small set upper bound.
      </entry><entry>0</entry></row>
      <row><entry>
       smallSetElementSetName</entry><entry>
       The element set name to be used for small result sets.
      </entry><entry>none</entry></row>
      <row><entry>
       mediumSetElementSetName</entry><entry>
       The element set name to be for medium-sized result sets.
      </entry><entry>none</entry></row>
      <row><entry>
       init_opt_search, init_opt_present, init_opt_delSet, etc.</entry><entry>
       After a successful Init, these options may be interrogated to
       discover whether the server claims to support the specified
       operations.
      </entry><entry>none</entry></row>
      <row>
       <entry>sru</entry><entry>
       SRU/Solr transport type. Must be either <literal>soap</literal>,
       <literal>get</literal>, <literal>post</literal>, or
       <literal>solr</literal>.
      </entry><entry>soap</entry></row>
      <row><entry>
       sru_version</entry><entry>
       SRU/SRW version. Should be <literal>1.1</literal>, or
       <literal>1.2</literal>. This is , prior to connect, the version
       to offer (highest version). And following connect (in fact
       first operation), holds the negotiated version with the server
       (same or lower version).
      </entry><entry>1.2</entry></row>
      <row id="zoom.extraArgs.option"><entry>
       extraArgs</entry><entry>
       Extra arguments for SRU/Solr URLs. The value must be
       URL encoded already.
      </entry><entry></entry></row>
      <row id="zoom.facets.option"><entry>
       facets</entry><entry>
       Requested or recommend facets may be given before a search is sent.
       The value of this setting is described in <xref linkend="facets"/>
       For inspection of the facets returned, refer to the functions
       described in <xref linkend="zoom.facets"/>.
      </entry><entry>none</entry></row>
      <row><entry>
       apdulog</entry><entry>
       If set to a true value such as "1", a log of low-level
       protocol packets is emitted on standard error stream.  This
       can be very useful for debugging.
      </entry><entry>0</entry></row>
      <row><entry>
       saveAPDU</entry><entry>
       If set to a true value such as "1", a log of low-level
       protocol packets is saved. The log can be retrieved by reading
       option APDU. Setting saveAPDU always has the side effect of
       resetting the currently saved log. This setting is
       <emphasis>write-only</emphasis>. If read, NULL will be returned.
       It is only recognized in
       <function>ZOOM_connection_option_set</function>.
      </entry><entry>0</entry></row>
      <row><entry>
       APDU</entry><entry>
       Returns the log of protocol packets. Will be empty if logging
       is not enabled (see saveAPDU above). This setting is
       <emphasis>read-only</emphasis>. It is only recognized if used
       in call to <function>ZOOM_connection_option_get</function> or
       <function>ZOOM_connection_option_getl</function>.
      </entry><entry></entry></row>
      <row><entry>
       memcached</entry><entry>
       If given and non-empty,
       <ulink url="&url.libmemcached;">libMemcached</ulink>
       will be configured for the connection.
       This option is inspected by ZOOM when a connection is established.
       If the <literal>memcached</literal> option is given
       and YAZ is compiled without libMemcached support, an internal
       diagnostic (10018) will be thrown.
       libMemcached support is available for YAZ 5.0.13 or later. If this
       option is supplied for an earlier version of YAZ, it is
       <emphasis>ignored</emphasis>.
       The value of this option is a list options - each is of the
       form <literal>--name=value</literal>.
       Option <literal>--server=</literal>host[:port] specifies a memcached
       server. It may be repeated for multiple memcached servers.
       Option <literal>--expire=</literal>seconds sets expiry time in seconds
       for how long result sets are to be cached.
      </entry><entry>none</entry></row>
      <row><entry>
       redis</entry><entry>
       If given and non-empty,
       a <ulink url="&url.redis;">redis</ulink> context will be created
       for the connection.
       This option is inspected by ZOOM when a connection is established.
       If the <literal>redis</literal> option is given
       and YAZ is compiled without redis support, an internal
       diagnostic (10018) will be thrown.
       redis support is available for YAZ 5.2.0 or later. If this
       option is supplied for an earlier version of YAZ, it is
       <emphasis>ignored</emphasis>.
       The value of this option is a set options, similar to that
       of the memcached setting. At this stage only --server=host[:port]
       and --expire=seconds is supported.
      </entry><entry>none</entry></row>
     </tbody>
    </tgroup>
   </table>
   <para>
    If either option <literal>lang</literal> or <literal>charset</literal>
    is set, then
    <ulink url="&url.z39.50.charneg;">
     Character Set and Language Negotiation</ulink> is in effect.
   </para>
   <synopsis>
     int ZOOM_connection_error(ZOOM_connection c, const char **cp,
                               const char **addinfo);
     int ZOOM_connection_error_x(ZOOM_connection c, const char **cp,
                                 const char **addinfo, const char **dset);
   </synopsis>
   <para>
    Function <function>ZOOM_connection_error</function> checks for
    errors for the last operation(s) performed. The function returns
    zero if no errors occurred; non-zero otherwise indicating the error.
    Pointers <parameter>cp</parameter> and <parameter>addinfo</parameter>
    holds messages for the error and additional-info if passed as
    non-<literal>NULL</literal>. Function
    <function>ZOOM_connection_error_x</function> is an extended version
    of <function>ZOOM_connection_error</function> that is capable of
    returning name of diagnostic set in <parameter>dset</parameter>.
   </para>
   <sect2 id="zoom-connection-z39.50">
    <title>Z39.50 Protocol behavior</title>
    <para>
     The calls <function>ZOOM_connection_new</function> and
     <function>ZOOM_connection_connect</function> establishes a TCP/IP
     connection and sends an Initialize Request to the target if
     possible. In addition, the calls waits for an Initialize Response
     from the target and the result is inspected (OK or rejected).
    </para>
    <para>
     If <literal>proxy</literal> is set then the client will establish
     a TCP/IP connection with the peer as specified by the
     <literal>proxy</literal> host and the hostname as part of the
     connect calls will be set as part of the Initialize Request.
     The proxy server will then "forward" the PDU's transparently
     to the target behind the proxy.
    </para>
    <para>
     For the authentication parameters, if option <literal>user</literal>
     is set and both options <literal>group</literal> and
     <literal>pass</literal> are unset, then Open style
     authentication is used (Version 2/3) in which case the username
     is usually followed by a slash, then by a password.
     If either <literal>group</literal>
     or <literal>pass</literal> is set then idPass authentication
     (Version 3 only) is used. If none of the options are set, no
     authentication parameters are set as part of the Initialize Request
     (obviously).
    </para>
    <para>
     When option <literal>async</literal> is 1, it really means that
     all network operations are postponed (and queued) until the
     function <literal>ZOOM_event</literal> is invoked. When doing so
     it doesn't make sense to check for errors after
     <literal>ZOOM_connection_new</literal> is called since that
     operation "connecting - and init" is still incomplete and the
     API cannot tell the outcome (yet).
    </para>
    </sect2>
   <sect2 id="zoom.sru.init.behavior">
    <title>SRU/Solr Protocol behavior</title>
    <para>
     The HTTP based protocols (SRU, SRW, Solr) doesn't feature an
     Inititialize Request, so  the connection phase merely establishes a
     TCP/IP connection with the HTTP server.
    </para>
    <para>Most of the ZOOM connection options do not
     affect SRU/Solr and they are ignored. However, future versions
     of &yaz; might honor <literal>implementationName</literal> and
     put that as part of User-Agent header for HTTP requests.
     </para>
    <para>
     The <literal>charset</literal> is used in the Content-Type header
     of HTTP requests.
    </para>
    <para>
     Setting <literal>authentcationMode</literal> specifies how
     authentication parameters are encoded for HTTP. The default is
     "<literal>basic</literal>" where <literal>user</literal> and
     <literal>password</literal> are encoded by using HTTP basic
     authentication.
     </para>
    <para>
     If <literal>authentcationMode</literal> is "<literal>url</literal>", then
     user and password are encoded in the URL by parameters
     <literal>x-username</literal> and <literal>x-password</literal> as
     given by the SRU standard.
    </para>
   </sect2>
  </sect1>
  <sect1 id="zoom.query">
   <title>Queries</title>
   <para>
    Query objects represents queries.
   </para>
   <synopsis>
     ZOOM_query ZOOM_query_create(void);

     void ZOOM_query_destroy(ZOOM_query q);

     int ZOOM_query_prefix(ZOOM_query q, const char *str);

     int ZOOM_query_cql(ZOOM_query s, const char *str);

     int ZOOM_query_sortby(ZOOM_query q, const char *criteria);

     int ZOOM_query_sortby2(ZOOM_query q, const char *strategy,
                            const char *criteria);
   </synopsis>
   <para>
    Create query objects using <function>ZOOM_query_create</function>
    and destroy them by calling <function>ZOOM_query_destroy</function>.
    RPN-queries can be specified in <link linkend="PQF">PQF</link>
    notation by using the
    function <function>ZOOM_query_prefix</function>.
    The <function>ZOOM_query_cql</function> specifies a CQL
    query to be sent to the server/target.
    More query types will be added in future versions of &yaz;, such as
    <link linkend="CCL">CCL</link> to RPN-mapping, native CCL query,
    etc. In addition to a search, a sort criteria may be set. Function
    <function>ZOOM_query_sortby</function> enables Z39.50 sorting and
    it takes sort criteria using the same string notation as
    yaz-client's <link linkend="sortspec">sort command</link>.
   </para>
   <para id="zoom.query.sortby2">
    <function>ZOOM_query_sortby2</function> is similar to
    <function>ZOOM_query_sortby</function> but allows a strategy for
    sorting. The reason for the strategy parameter is that some
    protocols offers multiple ways of performing sorting.
    For example, Z39.50 has the standard sort, which is performed after
    search on an existing result set.
    It's also possible to use CQL in Z39.50 as the query type and use
    CQL's SORTBY keyword. Finally, Index Data's
    Zebra server also allows sorting to be specified as part of RPN (Type 7).
   </para>
   <table id="zoom-sort-strategy" frame="top">
    <title>ZOOM sort strategy</title>
    <tgroup cols="2">
     <colspec colwidth="2*" colname="name"/>
     <colspec colwidth="5*" colname="description"/>
     <thead>
      <row>
       <entry>Name</entry>
       <entry>Description</entry>
      </row>
     </thead>
     <tbody>
      <row>
       <entry>z39.50</entry><entry>Z39.50 resultset sort</entry>
      </row>
      <row>
       <entry>type7</entry><entry>Sorting embedded in RPN(Type-7)</entry>
      </row>
      <row>
       <entry>cql</entry><entry>CQL SORTBY</entry>
      </row>
      <row>
       <entry>sru11</entry><entry>SRU sortKeys parameter</entry>
      </row>
      <row>
       <entry>solr</entry><entry>Solr sort</entry>
      </row>
      <row>
       <entry>embed</entry><entry>type7 for Z39.50, cql for SRU,
        solr for Solr protocol</entry>
      </row>
     </tbody>
    </tgroup>
   </table>
  </sect1>
  <sect1 id="zoom.resultsets"><title>Result sets</title>
   <para>
    The result set object is a container for records returned from
    a target.
   </para>
   <synopsis>
     ZOOM_resultset ZOOM_connection_search(ZOOM_connection, ZOOM_query q);

     ZOOM_resultset ZOOM_connection_search_pqf(ZOOM_connection c,
                                               const char *q);
     void ZOOM_resultset_destroy(ZOOM_resultset r);
   </synopsis>
   <para>
    Function <function>ZOOM_connection_search</function> creates
    a result set given a connection and query.
    Destroy a result set by calling
    <function>ZOOM_resultset_destroy</function>.
    Simple clients may using PQF only may use function
    <function>ZOOM_connection_search_pqf</function> in which case
    creating query objects is not necessary.
   </para>
   <synopsis>
     void ZOOM_resultset_option_set(ZOOM_resultset r,
                                    const char *key, const char *val);

     const char *ZOOM_resultset_option_get(ZOOM_resultset r, const char *key);

     size_t ZOOM_resultset_size(ZOOM_resultset r);
   </synopsis>
   <para>
    Functions <function>ZOOM_resultset_options_set</function> and
    <function>ZOOM_resultset_get</function> sets and gets an option
    for a result set similar to <function>ZOOM_connection_option_get</function>
    and <function>ZOOM_connection_option_set</function>.
   </para>
   <para>
    The number of hits also called result-count is returned by
    function <function>ZOOM_resultset_size</function>.
   </para>
   <table id="zoom.resultset.options"
    frame="top"><title>ZOOM Result set Options</title>
    <tgroup cols="3">
     <colspec colwidth="4*" colname="name"></colspec>
     <colspec colwidth="7*" colname="description"></colspec>
     <colspec colwidth="2*" colname="default"></colspec>
     <thead>
      <row>
       <entry>Option</entry>
       <entry>Description</entry>
       <entry>Default</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        start</entry><entry>Offset of first record to be
        retrieved from target. First record has offset 0 unlike the
        protocol specifications where first record has position 1.
        This option affects ZOOM_resultset_search and
        ZOOM_resultset_search_pqf and must be set before any of
        these functions are invoked. If a range of
        records must be fetched manually after search,
        function ZOOM_resultset_records should be used.
       </entry><entry>0</entry></row>
      <row><entry>
        count</entry><entry>Number of records to be retrieved.
        This option affects ZOOM_resultset_search and
        ZOOM_resultset_search_pqf and must be set before any of
        these functions are invoked.
       </entry><entry>0</entry></row>
      <row><entry>
        presentChunk</entry><entry>The number of records to be
        requested from the server in each chunk (present request). The
        value 0 means to request all the records in a single chunk.
        (The old <literal>step</literal>
        option is also supported for the benefit of old applications.)
       </entry><entry>0</entry></row>
      <row><entry>
        elementSetName</entry><entry>Element-Set name of records.
        Most targets should honor element set name <literal>B</literal>
        and <literal>F</literal> for brief and full respectively.
       </entry><entry>none</entry></row>
      <row><entry>
        preferredRecordSyntax</entry><entry>Preferred Syntax, such as
        <literal>USMARC</literal>, <literal>SUTRS</literal>, etc.
       </entry><entry>none</entry></row>
      <row><entry>
        schema</entry><entry>Schema for retrieval, such as
        <literal>Gils-schema</literal>, <literal>Geo-schema</literal>, etc.
       </entry><entry>none</entry></row>
      <row><entry>
        setname</entry><entry>Name of Result Set (Result Set ID).
        If this option isn't set, the ZOOM module will automatically
        allocate a result set name.
       </entry><entry>default</entry></row>
      <row><entry>
        rpnCharset</entry><entry>Character set for RPN terms.
        If this is set, ZOOM C will assume that the ZOOM application is
        running UTF-8. Terms in RPN queries are then converted to the
        rpnCharset. If this is unset, ZOOM C will not assume any encoding
        of RPN terms and no conversion is performed.
       </entry><entry>none</entry></row>
     </tbody>
    </tgroup>
   </table>
   <para>
    For servers that support Search Info report, the following
    options may be read using <function>ZOOM_resultset_get</function>.
    This detailed information is read after a successful search has
    completed.
   </para>
   <para>
    This information is a list of of items, where each item is
    information about a term or subquery. All items in the list
    are prefixed by
    <literal>SearchResult.</literal><replaceable>no</replaceable>
    where no presents the item number (0=first, 1=second).
    Read <literal>searchresult.size</literal> to determine the
    number of items.
   </para>
   <table id="zoom.search.info.report.options"
    frame="top"><title>Search Info Report Options</title>
    <tgroup cols="2">
     <colspec colwidth="4*" colname="name"></colspec>
     <colspec colwidth="7*" colname="description"></colspec>
     <thead>
      <row>
       <entry>Option</entry>
       <entry>Description</entry>
      </row>
     </thead>
     <tbody>
      <row>
       <entry>searchresult.size</entry>
       <entry>
        number of search result entries. This option is-nonexistant
        if no entries are returned by the server.
       </entry>
      </row>
      <row>
       <entry>searchresult.<replaceable>no</replaceable>.id</entry>
       <entry>sub query ID</entry>
      </row>
      <row>
       <entry>searchresult.<replaceable>no</replaceable>.count</entry>
       <entry>result count for item (number of hits)</entry>
      </row>
      <row>
       <entry>searchresult.<replaceable>no</replaceable>.subquery.term</entry>
       <entry>subquery term</entry>
      </row>
      <row>
       <entry>
        searchresult.<replaceable>no</replaceable>.interpretation.term
       </entry>
       <entry>interpretation term</entry>
      </row>
      <row>
       <entry>
        searchresult.<replaceable>no</replaceable>.recommendation.term
       </entry>
       <entry>recommendation term</entry>
      </row>
     </tbody>
    </tgroup>
   </table>
   <sect2 id="zoom.z3950.resultset.sort">
    <title>Z39.50 Result-set Sort</title>
    <synopsis>
     void ZOOM_resultset_sort(ZOOM_resultset r,
                              const char *sort_type, const char *sort_spec);

     int ZOOM_resultset_sort1(ZOOM_resultset r,
                              const char *sort_type, const char *sort_spec);
    </synopsis>
    <para>
     <function>ZOOM_resultset_sort</function> and
     <function>ZOOM_resultset_sort1</function> both sort an existing
     result-set. The sort_type parameter is not use. Set it to "yaz".
     The sort_spec is same notation as ZOOM_query_sortby and identical
     to that offered by yaz-client's
     <link linkend="sortspec">sort command</link>.
    </para>
    <para>
     These functions only work for Z39.50. Use the more generic utility
     <link linkend="zoom.query.sortby2">
      <function>ZOOM_query_sortby2</function></link>
     for other protocols (and even Z39.50).
    </para>
   </sect2>
   <sect2 id="zoom.z3950.resultset.behavior">
    <title>Z39.50 Protocol behavior</title>
    <para>
     The creation of a result set involves at least a SearchRequest
     - SearchResponse protocol handshake. Following that, if a sort
     criteria was specified as part of the query, a SortRequest -
     SortResponse handshake takes place. Note that it is necessary to
     perform sorting before any retrieval takes place, so no records will
     be returned from the target as part of the SearchResponse because these
     would be unsorted. Hence, piggyback is disabled when sort criteria
     are set. Following Search - and a possible sort - Retrieval takes
     place - as one or more Present Requests/Response pairs being
     transferred.
     </para>
    <para>
     The API allows for two different modes for retrieval. A high level
     mode which is somewhat more powerful and a low level one.
     The low level is enabled when searching on a Connection object
     for which the settings
     <literal>smallSetUpperBound</literal>,
     <literal>mediumSetPresentNumber</literal> and
     <literal>largeSetLowerBound</literal> are set. The low level mode
     thus allows you to precisely set how records are returned as part
     of a search response as offered by the Z39.50 protocol.
     Since the client may be retrieving records as part of the
     search response, this mode doesn't work well if sorting is used.
     </para>
    <para>
     The high-level mode allows you to fetch a range of records from
     the result set with a given start offset. When you use this mode
     the client will automatically use piggyback if that is possible
     with the target and perform one or more present requests as needed.
     Even if the target returns fewer records as part of a present response
     because of a record size limit, etc. the client will repeat sending
     present requests. As an example, if option <literal>start</literal>
     is 0 (default) and <literal>count</literal> is 4, and
     <literal>piggyback</literal> is 1 (default) and no sorting criteria
     is specified, then the client will attempt to retrieve the 4
     records as part the search response (using piggyback). On the other
     hand, if either <literal>start</literal> is positive or if
     a sorting criteria is set, or if <literal>piggyback</literal>
     is 0, then the client will not perform piggyback but send Present
     Requests instead.
    </para>
    <para>
     If either of the options <literal>mediumSetElementSetName</literal> and
     <literal>smallSetElementSetName</literal> are unset, the value
     of option <literal>elementSetName</literal> is used for piggyback
     searches. This means that for the high-level mode you only have
     to specify one elementSetName option rather than three.
     </para>
   </sect2>
   <sect2 id="zoom.sru.resultset.behavior">
    <title>SRU Protocol behavior</title>
    <para>
     Current version of &yaz; does not take advantage of a result set id
     returned by the SRU server. Future versions might do, however.
     Since, the ZOOM driver does not save result set IDs any
     present (retrieval) is transformed to a SRU SearchRetrieveRequest
     with same query but, possibly, different offsets.
    </para>
    <para>
     Option <literal>schema</literal> specifies SRU schema
     for retrieval. However, options <literal>elementSetName</literal> and
     <literal>preferredRecordSyntax</literal> are ignored.
    </para>
    <para>
     Options <literal>start</literal> and <literal>count</literal>
     are supported by SRU.
     The remaining options
     <literal>piggyback</literal>,
     <literal>smallSetUpperBound</literal>,
     <literal>largeSetLowerBound</literal>,
     <literal>mediumSetPresentNumber</literal>,
     <literal>mediumSetElementSetName</literal>,
      <literal>smallSetElementSetName</literal> are
     unsupported.
    </para>
    <para>
     SRU supports CQL queries, <emphasis>not</emphasis> PQF.
     If PQF is used, however, the PQF query is transferred anyway
     using non-standard element <literal>pQuery</literal> in
     SRU SearchRetrieveRequest.
    </para>
    <para>
     Solr queries has to be done in Solr query format.
    </para>
    <para>
     Unfortunately, SRU or Solr does not define a database setting. Hence,
     <literal>databaseName</literal> is unsupported and ignored.
     However, the path part in host parameter for functions
     <function>ZOOM_connecton_new</function> and
     <function>ZOOM_connection_connect</function> acts as a
     database (at least for the &yaz; SRU server).
    </para>
   </sect2>
  </sect1>
  <sect1 id="zoom.records">
   <title>Records</title>
   <para>
    A record object is a retrieval record on the client side -
    created from result sets.
   </para>
   <synopsis>
     void ZOOM_resultset_records(ZOOM_resultset r,
                                 ZOOM_record *recs,
                                 size_t start, size_t count);
     ZOOM_record ZOOM_resultset_record(ZOOM_resultset s, size_t pos);

     const char *ZOOM_record_get(ZOOM_record rec, const char *type,
                                 size_t *len);

     int ZOOM_record_error(ZOOM_record rec, const char **msg,
                           const char **addinfo, const char **diagset);

     ZOOM_record ZOOM_record_clone(ZOOM_record rec);

     void ZOOM_record_destroy(ZOOM_record rec);
   </synopsis>
   <para>
    References to temporary records are returned by functions
    <function>ZOOM_resultset_records</function> or
    <function>ZOOM_resultset_record</function>.
    </para>
   <para>
    If a persistent reference to a record is desired
    <function>ZOOM_record_clone</function> should be used.
    It returns a record reference that should be destroyed
    by a call to <function>ZOOM_record_destroy</function>.
   </para>
   <para>
    A single record is returned by function
    <function>ZOOM_resultset_record</function> that takes a
    position as argument. First record has position zero.
    If no record could be obtained <literal>NULL</literal> is returned.
   </para>
   <para>
    Error information for a record can be checked with
    <function>ZOOM_record_error</function> which returns non-zero
    (error code) if record is in error, called <emphasis>Surrogate
     Diagnostics</emphasis> in Z39.50.
   </para>
   <para>
    Function <function>ZOOM_resultset_records</function> retrieves
    a number of records from a result set. Parameter <literal>start</literal>
    and <literal>count</literal> specifies the range of records to
    be returned. Upon completion array
    <literal>recs[0], ..recs[count-1]</literal>
    holds record objects for the records. The array of records
     <literal>recs</literal> should be allocated prior the call
    <function>ZOOM_resultset_records</function>. Note that for those
    records that couldn't be retrieved from the target
    <literal>recs[ ..]</literal> is set to <literal>NULL</literal>.
   </para>
   <para id="zoom.record.get">
    In order to extract information about a single record,
    <function>ZOOM_record_get</function> is provided. The
    function returns a pointer to certain record information. The
    nature (type) of the pointer depends on the parameter,
    <parameter>type</parameter>.
   </para>
   <para>
    The <parameter>type</parameter> is a string of the format:
   </para>
   <para>
    <replaceable>format</replaceable>[;charset=<replaceable>from</replaceable>[/<replaceable>opacfrom</replaceable>][,<replaceable>to</replaceable>]][;format=<replaceable>v</replaceable>][;base64=<replaceable>xpath</replaceable>]
   </para>
   <para>
    If <literal>charset</literal> is given, then <replaceable>from</replaceable>
    specifies the character set of the record in its original form
    (as returned by the server), <replaceable>to</replaceable> specifies
    the output (returned) character set encoding.
    If <replaceable>to</replaceable> is omitted, then UTF-8 is assumed.
    If charset is not given, then no character set conversion takes place.
    OPAC records may be returned in a different
    set from the bibliographic MARC record. If this is this the case,
    <replaceable>opacfrom</replaceable> should be set to the character set
    of the OPAC record part.
   </para>

   <para>
    The <literal>format</literal> is generic but can only be used to
    specify XML indentation when the value <replaceable>v</replaceable>
    is 1 (<literal>format=1</literal>).
   </para>
   <para>
    The <literal>base64</literal> allows a full record to be extracted
    from base64-encoded string in an XML document.
   </para>
   <note>
     <para>
       Specifying the OPAC record character set requires YAZ 4.1.5 or later.
     </para>
     <para>
       Specifying the base64 parameter requires YAZ 4.2.35 or later.
     </para>
   </note>
   <para>
    The format argument controls whether record data should be XML
    pretty-printed (post process operation).
    It is enabled only if format value <replaceable>v</replaceable> is
    <literal>1</literal> and the record content is XML well-formed.
   </para>
   <para>
    In addition, for certain types, the length
    <literal>len</literal> passed will be set to the size in bytes of
    the returned information.
    </para>
   <para>
    The following are the supported values for <replaceable>form</replaceable>.
    <variablelist>
     <varlistentry><term><literal>database</literal></term>
      <listitem><para>Database of record is returned
        as a C null-terminated string. Return type
        <literal>const char *</literal>.
       </para></listitem>
     </varlistentry>
     <varlistentry><term><literal>syntax</literal></term>
      <listitem><para>The transfer syntax of the record is returned
        as a C null-terminated string containing the symbolic name of
        the record syntax, e.g. <literal>Usmarc</literal>. Return type
        is
        <literal>const char *</literal>.
       </para></listitem>
     </varlistentry>
     <varlistentry><term><literal>schema</literal></term>
      <listitem><para>The schema of the record is returned
        as a C null-terminated string. Return type is
        <literal>const char *</literal>.
       </para></listitem>
     </varlistentry>
     <varlistentry><term><literal>render</literal></term>
      <listitem><para>The record is returned in a display friendly
        format. Upon completion buffer is returned
        (type <literal>const char *</literal>) and length is stored in
        <literal>*len</literal>.
       </para></listitem>
     </varlistentry>
     <varlistentry><term><literal>raw</literal></term>
      <listitem><para>The record is returned in the internal
        YAZ specific format. For GRS-1, Explain, and others, the
        raw data is returned as type
        <literal>Z_External *</literal> which is just the type for
        the member <literal>retrievalRecord</literal> in
        type <literal>NamePlusRecord</literal>.
        For SUTRS and octet aligned record (including all MARCs) the
        octet buffer is returned and the length of the buffer.
       </para></listitem>
     </varlistentry>
     <varlistentry><term><literal>xml</literal></term>
      <listitem><para>The record is returned in XML if possible.
        SRU, Solr and Z39.50 records with transfer syntax XML are
        returned verbatim. MARC records are returned in
        <ulink url="&url.marcxml;">
         MARCXML
         </ulink>
        (converted from ISO2709 to MARCXML by YAZ).
        OPAC records are also converted to XML and the
        bibliographic record is converted to MARCXML (when possible).
        GRS-1 records are not supported for this form.
        Upon completion, the XML buffer is returned
        (type <literal>const char *</literal>) and length is stored in
        <literal>*len</literal>.
       </para></listitem>
     </varlistentry>
     <varlistentry><term><literal>opac</literal></term>
      <listitem><para>OPAC information for record is returned in XML
        if an OPAC record is present at the position given. If no
        OPAC record is present, a NULL pointer is returned.
       </para></listitem>
     </varlistentry>
     <varlistentry><term><literal>txml</literal></term>
      <listitem><para>The record is returned in TurboMARC if possible.
        SRU and Z39.50 records with transfer syntax XML are
        returned verbatim. MARC records are returned in
        <link linkend="tools.turbomarc">
         TurboMARC
        </link>
        (converted from ISO2709 to TurboMARC by YAZ).
        Upon completion, the XML buffer is returned
        (type <literal>const char *</literal>) and length is stored in
        <literal>*len</literal>.
       </para></listitem>
     </varlistentry>
     <varlistentry><term><literal>json</literal></term>
      <listitem><para>Like xml, but MARC records are converted to
        <ulink url="&url.marc_in_json;">MARC-in-JSON</ulink>.
       </para></listitem>
     </varlistentry>

    </variablelist>
   </para>
   <para>
    Most
    <ulink url="&url.marc21;">MARC21</ulink>
    records uses the
    <ulink url="&url.marc8;">MARC-8</ulink>
    character set encoding.
    An application that wishes to display in Latin-1 would use
    <screen>
     render; charset=marc8,iso-8859-1
    </screen>
   </para>
   <sect2 id="zoom.z3950.record.behavior">
    <title>Z39.50 Protocol behavior</title>
    <para>
     The functions <function>ZOOM_resultset_record</function> and
     <function>ZOOM_resultset_records</function> inspects the client-side
     record cache. Records not found in cache are fetched using
     Present.
     The functions may block (and perform network I/O)  - even though option
     <literal>async</literal> is 1, because they return records objects.
     (and there's no way to return records objects without retrieving them!).
     </para>
    <para>
     There is a trick, however, in the usage of function
     <function>ZOOM_resultset_records</function> that allows for
     delayed retrieval (and makes it non-blocking). By using
     a null pointer for <parameter>recs</parameter> you're indicating
     you're not interested in getting records objects
     <emphasis>now</emphasis>.
    </para>
   </sect2>
   <sect2 id="zoom.sru.record.behavior">
    <title>SRU/Solr Protocol behavior</title>
    <para>
     The ZOOM driver for SRU/Solr treats records returned by a SRU/Solr server
     as if they where Z39.50 records with transfer syntax XML and
     no element set name or database name.
    </para>
   </sect2>
  </sect1>
  <sect1 id="zoom.facets"><title>Facets</title>
   <para>
    Facet operations is not part of the official ZOOM specification, but
    is an Index Data extension for YAZ-based Z39.50 targets,
    <ulink url="&url.solr;">Solr</ulink> and SRU 2.0 targets.

    Facets may be requestd by the
     <link linkend="zoom.facets.option">facets</link> option before a
    search is sent.
    For inspection of the returned facets, the following functions are
    available:
   </para>
   <synopsis>
    ZOOM_facet_field *ZOOM_resultset_facets(ZOOM_resultset r);

    ZOOM_facet_field ZOOM_resultset_get_facet_field(ZOOM_resultset r,
                                                    const char *facet_name);

    ZOOM_facet_field ZOOM_resultset_get_facet_field_by_index(ZOOM_resultset r,
                                                             int pos);

    size_t ZOOM_resultset_facets_size(ZOOM_resultset r);

    const char *ZOOM_facet_field_name(ZOOM_facet_field facet_field);

    size_t ZOOM_facet_field_term_count(ZOOM_facet_field facet_field);

    const char *ZOOM_facet_field_get_term(ZOOM_facet_field facet_field,
                                          size_t idx, int *freq);
   </synopsis>
   <para>
    References to temporary structures are returned by all functions.
    They are only valid as long the Result set is valid.
    <function>ZOOM_resultset_get_facet_field</function> or
    <function>ZOOM_resultset_get_facet_field_by_index</function>.
    <function>ZOOM_resultset_facets</function>.
    <function>ZOOM_facet_field_name</function>.
    <function>ZOOM_facet_field_get_term</function>.
    </para>
   <para id="zoom.resultset.get_facet_field">
    A single Facet field  is returned by function
    <function>ZOOM_resultset_get_facet_field</function> or
    <function>ZOOM_resultset_get_facet_field_by_index</function> that takes
    a  result set and facet name or positive index respectively. First
    facet has position zero. If no facet could be obtained (invalid name
    or index out of bounds) <literal>NULL</literal> is returned.
   </para>
   <para id="zoom.resultset.facets">
    An array of facets field can be returned by
    <function>ZOOM_resultset_facets</function>. The length of the array is
    given by <function>ZOOM_resultset_facets_size</function>. The array is
    zero-based and last entry will be at
    <function>ZOOM_resultset_facets_size(result_set)</function>-1.
   </para>
   <para id="zoom.resultset.facets_names">
    It is possible to interate over facets by name, by calling
    <function>ZOOM_resultset_facets_names</function>.
    This will return an const array of char * where each string can be used
    as parameter for <function>ZOOM_resultset_get_facet_field</function>.
   </para>
   <para>
   Function <function>ZOOM_facet_field_name</function> gets the request
    facet name from a returned facet field.
   </para>
   <para>
    Function <function>ZOOM_facet_field_get_term</function> returns the
    idx'th term and term count for a facet field.
    Idx must between 0 and
    <function>ZOOM_facet_field_term_count</function>-1, otherwise the
    returned reference will be <literal>NULL</literal>. On a valid idx, the
    value of the freq reference will be the term count.
    The <literal>freq</literal> parameter must be valid pointer to integer.
   </para>
   </sect1>
  <sect1 id="zoom.scan"><title>Scan</title>
   <para>
    This section describes an interface for Scan. Scan is not an
    official part of the ZOOM model yet. The result of a scan operation
    is the <literal>ZOOM_scanset</literal> which is a set of terms
    returned by a target.
   </para>

   <para>
    The Scan interface is supported for both Z39.50, SRU and Solr.
   </para>

   <synopsis>
    ZOOM_scanset ZOOM_connection_scan(ZOOM_connection c,
                                      const char *startpqf);

    ZOOM_scanset ZOOM_connection_scan1(ZOOM_connection c,
                                       ZOOM_query q);

    size_t ZOOM_scanset_size(ZOOM_scanset scan);

    const char *ZOOM_scanset_term(ZOOM_scanset scan, size_t pos,
                                  size_t *occ, size_t *len);

    const char *ZOOM_scanset_display_term(ZOOM_scanset scan, size_t pos,
                                          size_t *occ, size_t *len);

    void ZOOM_scanset_destroy(ZOOM_scanset scan);

    const char *ZOOM_scanset_option_get(ZOOM_scanset scan,
                                        const char *key);

    void ZOOM_scanset_option_set(ZOOM_scanset scan, const char *key,
                                 const char *val);
    </synopsis>
   <para>
    The scan set is created by function
    <function>ZOOM_connection_scan</function> which performs a scan
    operation on the connection using the specified
    <parameter>startpqf</parameter>.
    If the operation was successful, the size of the scan set can be
    retrieved by a call to <function>ZOOM_scanset_size</function>.
    Like result sets, the items are numbered 0,..size-1.
    To obtain information about a particular scan term, call function
    <function>ZOOM_scanset_term</function>. This function takes
    a scan set offset <literal>pos</literal> and returns a pointer
    to a <emphasis>raw term</emphasis> or <literal>NULL</literal> if
    non-present.
    If present, the <literal>occ</literal> and <literal>len</literal>
    are set to the number of occurrences and the length
    of the actual term respectively.
    <function>ZOOM_scanset_display_term</function> is similar to
    <function>ZOOM_scanset_term</function> except that it returns
    the <emphasis>display term</emphasis> rather than the raw term.
    In a few cases, the term is different from display term. Always
    use the display term for display and the raw term for subsequent
    scan operations (to get more terms, next scan result, etc).
   </para>
   <para>
    A scan set may be freed by a call to function
    <function>ZOOM_scanset_destroy</function>.
    Functions <function>ZOOM_scanset_option_get</function> and
    <function>ZOOM_scanset_option_set</function> retrieves and sets
    an option respectively.
   </para>
   <para>
    The <parameter>startpqf</parameter> is a subset of PQF, namely
    the Attributes+Term part. Multiple <literal>@attr</literal> can
    be used. For example to scan in title (complete) phrases:
    <literallayout>
     @attr 1=4 @attr 6=2 "science o"
    </literallayout>
   </para>
   <para>
    The <function>ZOOM_connecton_scan1</function> is a newer and
    more generic alternative to <function>ZOOM_connection_scan</function>
    which allows to use both CQL and PQF for Scan.
   </para>
   <table frame="top" id="zoom.scanset.options">
    <title>ZOOM Scan Set Options</title>
    <tgroup cols="3">
     <colspec colwidth="4*" colname="name"></colspec>
     <colspec colwidth="7*" colname="description"></colspec>
     <colspec colwidth="2*" colname="default"></colspec>
     <thead>
      <row>
       <entry>Option</entry>
       <entry>Description</entry>
       <entry>Default</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        number</entry><entry>Number of Scan Terms requested in next scan.
        After scan it holds the actual number of terms returned.
       </entry><entry>20</entry></row>
      <row><entry>
        position</entry><entry>Preferred Position of term in response
        in next scan; actual position after completion of scan.
       </entry><entry>1</entry></row>
      <row><entry>
        stepSize</entry><entry>Step Size
       </entry><entry>0</entry></row>
      <row><entry>
        scanStatus</entry><entry>An integer indicating the Scan Status
        of last scan.
       </entry><entry>0</entry></row>
      <row><entry>
        rpnCharset</entry><entry>Character set for RPN terms.
        If this is set, ZOOM C will assume that the ZOOM application is
        running UTF-8. Terms in RPN queries are then converted to the
        rpnCharset. If this is unset, ZOOM C will not assume any encoding
        of RPN terms and no conversion is performed.
       </entry><entry>none</entry></row>
     </tbody>
    </tgroup>
   </table>
  </sect1>
  <sect1 id="zoom.extendedservices">
   <title>Extended Services</title>
   <para>
    ZOOM offers an interface to a subset of the Z39.50 extended services
    as well as a few privately defined ones:
   </para>
   <itemizedlist>
    <listitem>
     <para>
      Z39.50 Item Order (ILL).
      See <xref linkend="zoom.item.order"/>.
     </para>
    </listitem>
    <listitem>
     <para>
      Record Update. This allows a client to insert, modify or delete
      records.
      See <xref linkend="zoom.record.update"/>.
     </para>
    </listitem>
    <listitem>
     <para>
      Database Create. This a non-standard feature. Allows a client
      to create a database.
      See <xref linkend="zoom.database.create"/>.
     </para>
    </listitem>
    <listitem>
     <para>
      Database Drop. This a non-standard feature. Allows a client
      to delete/drop a database.
      See <xref linkend="zoom.database.drop"/>.
     </para>
     </listitem>
    <listitem>
     <para>
      Commit operation. This a non-standard feature. Allows a client
      to commit operations.
      See <xref linkend="zoom.commit"/>.
     </para>
    </listitem>
    <!-- all the ILL PDU options should go here too -->
   </itemizedlist>
   <para>
    To create an extended service operation a <literal>ZOOM_package</literal>
    must be created. The operation is a five step operation. The
    package is created, package is configured by means of options,
    the package is send, result is inspected (by means of options),
    the package is destroyed.
   </para>
   <synopsis>
    ZOOM_package ZOOM_connection_package(ZOOM_connection c,
                                         ZOOM_options options);

    const char *ZOOM_package_option_get(ZOOM_package p,
                                        const char *key);
    void ZOOM_package_option_set(ZOOM_package p, const char *key,
                                 const char *val);
    void ZOOM_package_send(ZOOM_package p, const char *type);

    void ZOOM_package_destroy(ZOOM_package p);
   </synopsis>
   <para>
    The <function>ZOOM_connection_package</function> creates a
    package for the connection given using the options specified.
   </para>
   <para>
    Functions <function>ZOOM_package_option_get</function> and
    <function>ZOOM_package_option_set</function> gets and sets
    options.
   </para>
   <para>
    <function>ZOOM_package_send</function> sends
    the package the via connection specified in
    <function>ZOOM_connection_package</function>.
    The <parameter>type</parameter> specifies the actual extended service
    package type to be sent.
   </para>
   <table frame="top" id="zoom.extendedservices.options">
    <title>Extended Service Common Options</title>
    <tgroup cols="3">
     <colspec colwidth="4*" colname="name"></colspec>
     <colspec colwidth="7*" colname="description"></colspec>
     <colspec colwidth="3*" colname="default"></colspec>
     <thead>
      <row>
       <entry>Option</entry>
       <entry>Description</entry>
       <entry>Default</entry>
      </row>
     </thead>
     <tbody>
      <row>
       <entry>package-name</entry>
       <entry>Extended Service Request package name. Must be specified
       as part of a request</entry>
       <entry>none</entry>
      </row>
      <row>
       <entry>user-id</entry>
       <entry>User ID of Extended Service Package. Is a request option</entry>
       <entry>none</entry>
      </row>
      <row>
       <entry>function</entry>
       <entry>
        Function of package - one of <literal>create</literal>,
        <literal>delete</literal>, <literal>modify</literal>. Is
        a request option.
       </entry>
       <entry><literal>create</literal></entry>
      </row>
      <row>
       <entry>waitAction</entry>
       <entry>
        Wait action for package. Possible values:
        <literal>wait</literal>, <literal>waitIfPossible</literal>,
        <literal>dontWait</literal> or <literal>dontReturnPackage</literal>.
       </entry>
       <entry><literal>waitIfPossible</literal></entry>
      </row>
      <row>
       <entry>targetReference</entry>
       <entry>
        Target Reference. This is part of the response as returned
        by the server. Read it after a successful operation.
       </entry>
       <entry><literal>none</literal></entry>
      </row>
     </tbody>
    </tgroup>
   </table>
   <sect2 id="zoom.item.order">
    <title>Item Order</title>
    <para>
     For Item Order, type must be set to <literal>itemorder</literal> in
     <function>ZOOM_package_send</function>.
    </para>

    <table frame="top" id="zoom.item.order.options">
     <title>Item Order Options</title>
     <tgroup cols="3">
      <colspec colwidth="4*" colname="name"></colspec>
      <colspec colwidth="7*" colname="description"></colspec>
      <colspec colwidth="3*" colname="default"></colspec>
      <thead>
       <row>
        <entry>Option</entry>
        <entry>Description</entry>
        <entry>Default</entry>
       </row>
      </thead>
      <tbody>
       <row>
        <entry>contact-name</entry>
        <entry>ILL contact name</entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>contact-phone</entry>
        <entry>ILL contact phone</entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>contact-email</entry>
        <entry>ILL contact email</entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>itemorder-setname</entry>
        <entry>Name of result set for record</entry>
        <entry>default</entry>
       </row>
       <row>
        <entry>itemorder-item</entry>
        <entry>Position for item (record) requested. An integer</entry>
        <entry>1</entry>
       </row>
      </tbody>
     </tgroup>
    </table>
    <para>
     There are two variants of item order: ILL-variant and
     XML document variant. In order to use the XML variant the setting
     <literal>doc</literal> must hold the XML item order document. If that
     setting is unset, the ILL-variant is used.
    </para>

    <table frame="top" id="zoom.illrequest.options">
     <title>ILL Request Options</title>
     <tgroup cols="1">
      <colspec colwidth="4*" colname="name"></colspec>
      <thead>
       <row>
        <entry>Option</entry>
       </row>
      </thead>
      <tbody>
       <row><entry>protocol-version-num</entry></row>
       <row><entry>transaction-id,initial-requester-id,person-or-institution-symbol,person</entry></row>
       <row><entry>transaction-id,initial-requester-id,person-or-institution-symbol,institution</entry></row>
       <row><entry>transaction-id,initial-requester-id,name-of-person-or-institution,name-of-person</entry></row>
       <row><entry>transaction-id,initial-requester-id,name-of-person-or-institution,name-of-institution</entry></row>
       <row><entry>transaction-id,transaction-group-qualifier</entry></row>
       <row><entry>transaction-id,transaction-qualifier</entry></row>
       <row><entry>transaction-id,sub-transaction-qualifier</entry></row>
       <row><entry>service-date-time,this,date</entry></row>
       <row><entry>service-date-time,this,time</entry></row>
       <row><entry>service-date-time,original,date</entry></row>
       <row><entry>service-date-time,original,time</entry></row>
       <row><entry>requester-id,person-or-institution-symbol,person</entry></row>
       <row><entry>requester-id,person-or-institution-symbol,institution</entry></row>
       <row><entry>requester-id,name-of-person-or-institution,name-of-person</entry></row>
       <row><entry>requester-id,name-of-person-or-institution,name-of-institution</entry></row>
       <row><entry>responder-id,person-or-institution-symbol,person</entry></row>
       <row><entry>responder-id,person-or-institution-symbol,institution</entry></row>
       <row><entry>responder-id,name-of-person-or-institution,name-of-person</entry></row>
       <row><entry>responder-id,name-of-person-or-institution,name-of-institution</entry></row>
       <row><entry>transaction-type</entry></row>
       <row><entry>delivery-address,postal-address,name-of-person-or-institution,name-of-person</entry></row>
       <row><entry>delivery-address,postal-address,name-of-person-or-institution,name-of-institution</entry></row>
       <row><entry>delivery-address,postal-address,extended-postal-delivery-address</entry></row>
       <row><entry>delivery-address,postal-address,street-and-number</entry></row>
       <row><entry>delivery-address,postal-address,post-office-box</entry></row>
       <row><entry>delivery-address,postal-address,city</entry></row>
       <row><entry>delivery-address,postal-address,region</entry></row>
       <row><entry>delivery-address,postal-address,country</entry></row>
       <row><entry>delivery-address,postal-address,postal-code</entry></row>
       <row><entry>delivery-address,electronic-address,telecom-service-identifier</entry></row>
       <row><entry>delivery-address,electronic-address,telecom-service-addreess</entry></row>
       <row><entry>billing-address,postal-address,name-of-person-or-institution,name-of-person</entry></row>
       <row><entry>billing-address,postal-address,name-of-person-or-institution,name-of-institution</entry></row>
       <row><entry>billing-address,postal-address,extended-postal-delivery-address</entry></row>
       <row><entry>billing-address,postal-address,street-and-number</entry></row>
       <row><entry>billing-address,postal-address,post-office-box</entry></row>
       <row><entry>billing-address,postal-address,city</entry></row>
       <row><entry>billing-address,postal-address,region</entry></row>
       <row><entry>billing-address,postal-address,country</entry></row>
       <row><entry>billing-address,postal-address,postal-code</entry></row>
       <row><entry>billing-address,electronic-address,telecom-service-identifier</entry></row>
       <row><entry>billing-address,electronic-address,telecom-service-addreess</entry></row>
       <row><entry>ill-service-type</entry></row>
       <row><entry>requester-optional-messages,can-send-RECEIVED</entry></row>
       <row><entry>requester-optional-messages,can-send-RETURNED</entry></row>
       <row><entry>requester-optional-messages,requester-SHIPPED</entry></row>
       <row><entry>requester-optional-messages,requester-CHECKED-IN</entry></row>
       <row><entry>search-type,level-of-service</entry></row>
       <row><entry>search-type,need-before-date</entry></row>
       <row><entry>search-type,expiry-date</entry></row>
       <row><entry>search-type,expiry-flag</entry></row>
       <row><entry>place-on-hold</entry></row>
       <row><entry>client-id,client-name</entry></row>
       <row><entry>client-id,client-status</entry></row>
       <row><entry>client-id,client-identifier</entry></row>
       <row><entry>item-id,item-type</entry></row>
       <row><entry>item-id,call-number</entry></row>
       <row><entry>item-id,author</entry></row>
       <row><entry>item-id,title</entry></row>
       <row><entry>item-id,sub-title</entry></row>
       <row><entry>item-id,sponsoring-body</entry></row>
       <row><entry>item-id,place-of-publication</entry></row>
       <row><entry>item-id,publisher</entry></row>
       <row><entry>item-id,series-title-number</entry></row>
       <row><entry>item-id,volume-issue</entry></row>
       <row><entry>item-id,edition</entry></row>
       <row><entry>item-id,publication-date</entry></row>
       <row><entry>item-id,publication-date-of-component</entry></row>
       <row><entry>item-id,author-of-article</entry></row>
       <row><entry>item-id,title-of-article</entry></row>
       <row><entry>item-id,pagination</entry></row>
       <row><entry>item-id,ISBN</entry></row>
       <row><entry>item-id,ISSN</entry></row>
       <row><entry>item-id,additional-no-letters</entry></row>
       <row><entry>item-id,verification-reference-source</entry></row>
       <row><entry>copyright-complicance</entry></row>
       <row><entry>retry-flag</entry></row>
       <row><entry>forward-flag</entry></row>
       <row><entry>requester-note</entry></row>
       <row><entry>forward-note</entry></row>
      </tbody>
     </tgroup>
    </table>
   </sect2>
   <sect2 id="zoom.record.update">
    <title>Record Update</title>
    <para>
     For Record Update, type must be set to <literal>update</literal> in
     <function>ZOOM_package_send</function>.
    </para>
    <table frame="top" id="zoom.record.update.options">
     <title>Record Update Options</title>
     <tgroup cols="3">
      <colspec colwidth="4*" colname="name"></colspec>
      <colspec colwidth="7*" colname="description"></colspec>
      <colspec colwidth="3*" colname="default"></colspec>
      <thead>
       <row>
        <entry>Option</entry>
        <entry>Description</entry>
        <entry>Default</entry>
       </row>
      </thead>
      <tbody>
       <row>
        <entry>action</entry>
        <entry>
         The update action. One of
         <literal>specialUpdate</literal>,
         <literal>recordInsert</literal>,
         <literal>recordReplace</literal>,
         <literal>recordDelete</literal>,
         <literal>elementUpdate</literal>.
        </entry>
        <entry><literal>specialUpdate (recordInsert for updateVersion=1 which does not support specialUpdate)</literal></entry>
       </row>
       <row>
        <entry>recordIdOpaque</entry>
        <entry>Opaque Record ID</entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>recordIdNumber</entry>
        <entry>Record ID number</entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>record</entry>
        <entry>The record itself</entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>recordOpaque</entry>
        <entry>Specifies an opaque record which is
          encoded as an ASN.1 ANY type with the OID as tiven by option
          <literal>syntax</literal> (see below).
          Option <literal>recordOpaque</literal> is an alternative
          to record - and <literal>record</literal> option (above) is
          ignored if recordOpaque is set. This option is only available in
          YAZ 3.0.35 and later and is meant to facilitate Updates with
          servers from OCLC.
        </entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>syntax</entry>
        <entry>The record syntax (transfer syntax). Is a string that
         is a known record syntax.
        </entry>
        <entry>no syntax</entry>
       </row>
       <row>
        <entry>databaseName</entry>
        <entry>Database from connection object</entry>
        <entry>Default</entry>
       </row>
       <row>
        <entry>correlationInfo.note</entry>
        <entry>Correlation Info Note (string)</entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>correlationInfo.id</entry>
        <entry>Correlation Info ID (integer)</entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>elementSetName</entry>
        <entry>Element Set for Record</entry>
        <entry>none</entry>
       </row>
       <row>
        <entry>updateVersion</entry>
        <entry>Record Update version which holds one of the values
         1, 2 or 3. Each version has a distinct OID:
         1.2.840.10003.9.5
         (<ulink url="&url.z39.50.extupdate1;">first version</ulink>) ,
         1.2.840.10003.9.5.1
         (second version) and
         1.2.840.10003.9.5.1.1
         (<ulink url="&url.z39.50.extupdate3;">third and
          newest version</ulink>).
        </entry>
        <entry>3</entry>
       </row>
      </tbody>
     </tgroup>
    </table>

   </sect2>

   <sect2 id="zoom.database.create"><title>Database Create</title>
    <para>
     For Database Create, type must be set to <literal>create</literal> in
     <function>ZOOM_package_send</function>.
    </para>

    <table frame="top" id="zoom.database.create.options">
     <title>Database Create Options</title>
     <tgroup cols="3">
      <colspec colwidth="4*" colname="name"></colspec>
      <colspec colwidth="7*" colname="description"></colspec>
      <colspec colwidth="3*" colname="default"></colspec>
      <thead>
       <row>
        <entry>Option</entry>
        <entry>Description</entry>
        <entry>Default</entry>
       </row>
      </thead>
      <tbody>
       <row>
        <entry>databaseName</entry>
        <entry>Database from connection object</entry>
        <entry>Default</entry>
       </row>
      </tbody>
     </tgroup>
    </table>
   </sect2>
   <sect2 id="zoom.database.drop">
    <title>Database Drop</title>
    <para>
     For Database Drop, type must be set to <literal>drop</literal> in
     <function>ZOOM_package_send</function>.
    </para>
    <table frame="top" id="zoom.database.drop.options">
     <title>Database Drop Options</title>
     <tgroup cols="3">
      <colspec colwidth="4*" colname="name"></colspec>
      <colspec colwidth="7*" colname="description"></colspec>
      <colspec colwidth="3*" colname="default"></colspec>
      <thead>
       <row>
        <entry>Option</entry>
        <entry>Description</entry>
        <entry>Default</entry>
       </row>
      </thead>
      <tbody>
       <row>
        <entry>databaseName</entry>
        <entry>Database from connection object</entry>
        <entry>Default</entry>
       </row>
      </tbody>
     </tgroup>
    </table>
   </sect2>
   <sect2 id="zoom.commit">
    <title>Commit Operation</title>
    <para>
     For Commit, type must be set to <literal>commit</literal> in
     <function>ZOOM_package_send</function>.
    </para>
   </sect2>
   <sect2 id="zoom.extended.services.behavior">
    <title>Protocol behavior</title>
    <para>
     All the extended services are Z39.50-only.
    </para>
    <note>
     <para>
      The database create, drop and commit services are privately defined
      operations.
      Refer to <filename>esadmin.asn</filename> in YAZ for the ASN.1
      definitions.
     </para>
    </note>
   </sect2>
  </sect1>
  <sect1 id="zoom.options">
   <title>Options</title>
   <para>
    Most &zoom; objects provide a way to specify options to change behavior.
    From an implementation point of view a set of options is just like
    an associative array / hash.
   </para>
   <synopsis>
     ZOOM_options ZOOM_options_create(void);

     ZOOM_options ZOOM_options_create_with_parent(ZOOM_options parent);

     void ZOOM_options_destroy(ZOOM_options opt);
   </synopsis>
   <synopsis>
     const char *ZOOM_options_get(ZOOM_options opt, const char *name);

     void ZOOM_options_set(ZOOM_options opt, const char *name,
                           const char *v);
   </synopsis>
   <synopsis>
     typedef const char *(*ZOOM_options_callback)
                            (void *handle, const char *name);

     ZOOM_options_callback
             ZOOM_options_set_callback(ZOOM_options opt,
                                       ZOOM_options_callback c,
                                       void *handle);
   </synopsis>
  </sect1>
  <sect1 id="zoom.queryconversions">
   <title>Query conversions</title>
   <synopsis>
    int ZOOM_query_cql2rpn(ZOOM_query s, const char *cql_str,
                           ZOOM_connection conn);

    int ZOOM_query_ccl2rpn(ZOOM_query s, const char *ccl_str,
                           const char *config,
                           int *ccl_error, const char **error_string,
                           int *error_pos);
   </synopsis>
   <para>
    <function>ZOOM_query_cql2rpn</function> translates the CQL string,
    client-side, into RPN which may be passed to the server.
    This is useful for server's that don't themselves
    support CQL, for which <function>ZOOM_query_cql</function> is useless.
    `conn' is used  only as a place to stash diagnostics if compilation
    fails; if this information is not needed, a null pointer may be used.
    The CQL conversion is driven by option <literal>cqlfile</literal> from
    connection conn. This specifies a conversion file (eg pqf.properties)
    which <emphasis>must</emphasis> be present.
   </para>
   <para>
    <function>ZOOM_query_ccl2rpn</function> translates the CCL string,
    client-side, into RPN which may be passed to the server.
    The conversion is driven by the specification given by
    <literal>config</literal>. Upon completion 0 is returned on success; -1
    is returned on on failure. Om failure <literal>error_string</literal> and
    <literal>error_pos</literal> holds error message and position of
    first error in original CCL string.
   </para>
  </sect1>
  <sect1 id="zoom.events"><title>Events</title>
   <para>
    If you're developing non-blocking applications, you have to deal
    with events.
   </para>
   <synopsis>
    int ZOOM_event(int no, ZOOM_connection *cs);
   </synopsis>
   <para>
    The <function>ZOOM_event</function> executes pending events for
    a number of connections. Supply the number of connections in
    <literal>no</literal> and an array of connections in
    <literal>cs</literal> (<literal>cs[0] ... cs[no-1]</literal>).
    A pending event could be a sending a search, receiving a response,
    etc.
    When an event has occurred for one of the connections, this function
    returns a positive integer <literal>n</literal> denoting that an event
    occurred for connection <literal>cs[n-1]</literal>.
    When no events are pending for the connections, a value of zero is
    returned.
    To ensure that all outstanding requests are performed call this function
    repeatedly until zero is returned.
   </para>
   <para>
    If <function>ZOOM_event</function> returns and returns non-zero, the
    last event that occurred can be expected.
   </para>
   <synopsis>
    int ZOOM_connection_last_event(ZOOM_connection cs);
   </synopsis>
   <para>
    <function>ZOOM_connection_last_event</function> returns an event type
    (integer) for the last event.
   </para>

   <table frame="top" id="zoom.event.ids">
    <title>ZOOM Event IDs</title>
    <tgroup cols="2">
     <colspec colwidth="4*" colname="name"></colspec>
     <colspec colwidth="7*" colname="description"></colspec>
     <thead>
      <row>
       <entry>Event</entry>
       <entry>Description</entry>
      </row>
     </thead>
     <tbody>
      <row>
       <entry>ZOOM_EVENT_NONE</entry>
       <entry>No event has occurred</entry>
      </row>
      <row>
       <entry>ZOOM_EVENT_CONNECT</entry>
       <entry>TCP/IP connect has initiated</entry>
      </row>
      <row>
       <entry>ZOOM_EVENT_SEND_DATA</entry>
       <entry>Data has been transmitted (sending)</entry>
      </row>
      <row>
       <entry>ZOOM_EVENT_RECV_DATA</entry>
       <entry>Data has been received)</entry>
      </row>
      <row>
       <entry>ZOOM_EVENT_TIMEOUT</entry>
       <entry>Timeout</entry>
      </row>
      <row>
       <entry>ZOOM_EVENT_UNKNOWN</entry>
       <entry>Unknown event</entry>
      </row>
      <row>
       <entry>ZOOM_EVENT_SEND_APDU</entry>
       <entry>An APDU has been transmitted (sending)</entry>
      </row>
      <row>
       <entry>ZOOM_EVENT_RECV_APDU</entry>
       <entry>An APDU has been received</entry>
      </row>
      <row>
       <entry>ZOOM_EVENT_RECV_RECORD</entry>
       <entry>A result-set record has been received</entry>
      </row>
      <row>
       <entry>ZOOM_EVENT_RECV_SEARCH</entry>
       <entry>A search result been received</entry>
      </row>
     </tbody>
    </tgroup>
   </table>
  </sect1>
 </chapter>
 <chapter id="server">
  <title>Generic server</title>
  <sect1 id="server.introduction"><title>Introduction</title>
   <para>
    If you aren't into documentation, a good way to learn how the
    back end interface works is to look at the <filename>backend.h</filename>
    file. Then, look at the small dummy-server in
    <filename>ztest/ztest.c</filename>. The <filename>backend.h</filename>
    file also makes a good reference, once you've chewed your way through
    the prose of this file.
   </para>
   <para>
    If you have a database system that you would like to make available by
    means of Z39.50 or SRU, &yaz; basically offers your two options. You
    can use the APIs provided by the &asn;, &odr;, and &comstack;
    modules to
    create and decode PDUs, and exchange them with a client.
    Using this low-level interface gives you access to all fields and
    options of the protocol, and you can construct your server as close
    to your existing database as you like.
    It is also a fairly involved process, requiring
    you to set up an event-handling mechanism, protocol state machine,
    etc. To simplify server implementation, we have implemented a compact
    and simple, but reasonably full-functioned server-frontend that will
    handle most of the protocol mechanics, while leaving you to
    concentrate on your database interface.
   </para>
   <note>
    <para>
     The backend interface was designed in anticipation of a specific
     integration task, while still attempting to achieve some degree of
     generality. We realize fully that there are points where the
     interface can be improved significantly. If you have specific
     functions or parameters that you think could be useful, send us a
     mail (or better, sign on to the mailing list referred to in the
     top-level README file). We will try to fit good suggestions into future
     releases, to the extent that it can be done without requiring
     too many structural changes in existing applications.
    </para>
   </note>
   <note>
    <para>
     The &yaz; server does not support XCQL.
     </para>
   </note>
  </sect1>
  <sect1 id="server.frontend">
   <title>The Database Frontend</title>
   <para>
    We refer to this software as a generic database frontend. Your
    database system is the <emphasis>backend database</emphasis>, and the
    interface between the two is called the <emphasis>backend API</emphasis>.
    The backend API consists of a small number of function handlers and
    structure definitions. You are required to provide the
    <function>main()</function> routine for the server (which can be
    quite simple), as well as a set of handlers to match each of the
    prototypes.
    The interface functions that you write can use any mechanism you like
    to communicate with your database system: You might link the whole
    thing together with your database application and access it by
    function calls; you might use IPC to talk to a database server
    somewhere; or you might link with third-party software that handles
    the communication for you (like a commercial database client library).
    At any rate, the handlers will perform the tasks of:
   </para>
   <itemizedlist>
    <listitem><para>
     Initialization.
    </para></listitem>
    <listitem><para>
     Searching.
    </para></listitem>
    <listitem><para>
     Fetching records.
    </para></listitem>
    <listitem><para>
     Scanning the database index (optional - if you wish to implement SCAN).
    </para></listitem>
    <listitem><para>
     Extended Services (optional).
    </para></listitem>
    <listitem><para>
     Result-Set Delete (optional).
    </para></listitem>
    <listitem><para>
     Result-Set Sort (optional).
    </para></listitem>
    <listitem><para>
     Return Explain for SRU (optional).
    </para></listitem>
   </itemizedlist>
   <para>
    (more functions will be added in time to support as much of
    Z39.50-1995 as possible).
   </para>
  </sect1>
  <sect1 id="server.backend">
   <title>The Backend API</title>
   <para>
    The header file that you need to use the interface are in the
    <filename>include/yaz</filename> directory. It's called
    <filename>backend.h</filename>. It will include other files from
    the <filename>include/yaz</filename> directory, so you'll
    probably want to use the -I option of your compiler to tell it
    where to find the files. When you run
    <literal>make</literal> in the top-level &yaz; directory,
    everything you need to create your server is to link with the
    <filename>lib/libyaz.la</filename> library.
   </para>
  </sect1>
  <sect1 id="server.main">
   <title>Your main() Routine</title>
   <para>
    As mentioned, your <function>main()</function> routine can be quite brief.
    If you want to initialize global parameters, or read global configuration
    tables, this is the place to do it. At the end of the routine, you should
    call the function
   </para>
   <synopsis>
int statserv_main(int argc, char **argv,
                  bend_initresult *(*bend_init)(bend_initrequest *r),
                  void (*bend_close)(void *handle));
   </synopsis>
   <para>
    The third and fourth arguments are pointers to handlers. Handler
    <function>bend_init</function> is called whenever the server receives
    an Initialize Request, so it serves as a Z39.50 session initializer. The
    <function>bend_close</function> handler is called when the session is
    closed.
   </para>
   <para>
    <function>statserv_main</function> will establish listening sockets
    according to the parameters given. When connection requests are received,
    the event handler will typically <function>fork()</function> and
    create a sub-process to handle a new connection.
    Alternatively the server may be setup to create threads for each
    connection.
    If you do use global variables and forking, you should be aware, then,
    that these cannot be shared between associations, unless you explicitly
    disable forking by command line parameters.
   </para>
   <para>
    The server provides a mechanism for controlling some of its behavior
    without using command-line options. The function
   </para>
   <synopsis>
    statserv_options_block *statserv_getcontrol(void);
   </synopsis>
   <para>
    will return a pointer to a <literal>struct statserv_options_block</literal>
    describing the current default settings of the server. The structure
    contains these elements:
    <variablelist>
     <varlistentry>
      <term><literal>int dynamic</literal></term>
      <listitem><para>
       A boolean value, which determines whether the server
       will fork on each incoming request (TRUE), or not (FALSE). Default is
       TRUE. This flag is only read by UNIX-based servers (WIN32 based servers
       doesn't fork).
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>int threads</literal></term>
      <listitem><para>
       A boolean value, which determines whether the server
       will create a thread on each incoming request (TRUE), or not (FALSE).
       Default is FALSE. This flag is only read by UNIX-based servers
       that offer POSIX Threads support.
       WIN32-based servers always operate in threaded mode.
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>int inetd</literal></term>
      <listitem><para>
       A boolean value, which determines whether the server
       will operates under a UNIX INET daemon (inetd). Default is FALSE.
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>char logfile[ODR_MAXNAME+1]</literal></term>
      <listitem><para>File for diagnostic output (&quot;&quot;: stderr).
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>char apdufile[ODR_MAXNAME+1]</literal></term>
      <listitem><para>
       Name of file for logging incoming and outgoing APDUs
       (&quot;&quot;: don't log APDUs, &quot;-&quot;:
       <literal>stderr</literal>).
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>char default_listen[1024]</literal></term>
      <listitem><para>Same form as the command-line specification of
      listener address. &quot;&quot;: no default listener address.
      Default is to listen at &quot;tcp:@:9999&quot;. You can only
      specify one default listener address in this fashion.
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>enum oid_proto default_proto;</literal></term>
      <listitem><para>Either <literal>PROTO_Z3950</literal> or
      <literal>PROTO_SR</literal>.
      Default is <literal>PROTO_Z39_50</literal>.
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>int idle_timeout;</literal></term>
      <listitem><para>Maximum session idle-time, in minutes. Zero indicates
      no (infinite) timeout. Default is 15 minutes.
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>int maxrecordsize;</literal></term>
      <listitem><para>Maximum permissible record (message) size. Default
      is 64 MB. This amount of memory will only be allocated if a
      client requests a very large amount of records in one operation
      (or a big record).
      Set it to a lower number if you are worried about resource
      consumption on your host system.
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>char configname[ODR_MAXNAME+1]</literal></term>
      <listitem><para>Passed to the backend when a new connection is received.
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>char setuid[ODR_MAXNAME+1]</literal></term>
      <listitem><para>Set user id to the user specified, after binding
      the listener addresses.
     </para></listitem>
     </varlistentry>
     <varlistentry>
      <term>
       <literal>void (*bend_start)(struct statserv_options_block *p)</literal>
      </term>
      <listitem><para>Pointer to function which is called after the
      command line options have been parsed - but before the server
      starts listening.
      For forked UNIX servers this handler is called in the mother
      process; for threaded servers this handler is called in the
      main thread.
      The default value of this pointer is NULL in which case it
      isn't invoked by the frontend server.
      When the server operates as an NT service this handler is called
      whenever the service is started.
      </para></listitem>
     </varlistentry>
     <varlistentry>
      <term>
       <literal>void (*bend_stop)(struct statserv_options_block *p)</literal>
      </term>
      <listitem><para>Pointer to function which is called whenever the server
      has stopped listening for incoming connections. This function pointer
      has a default value of NULL in which case it isn't called.
      When the server operates as an NT service this handler is called
      whenever the service is stopped.
      </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>void *handle</literal></term>
      <listitem><para>User defined pointer (default value NULL).
      This is a per-server handle that can be used to specify "user-data".
      Do not confuse this with the session-handle as returned by bend_init.
      </para></listitem>
     </varlistentry>
    </variablelist>
   </para>
   <para>
    The pointer returned by <literal>statserv_getcontrol</literal> points to
    a static area. You are allowed to change the contents of the structure,
    but the changes will not take effect before you call
   </para>
   <synopsis>
void statserv_setcontrol(statserv_options_block *block);
   </synopsis>
   <note>
    <para>
     that you should generally update this structure before calling
     <function>statserv_main()</function>.
    </para>
   </note>
  </sect1>
  <sect1 id="server.backendfunctions">
   <title>The Backend Functions</title>
   <para>
    For each service of the protocol, the backend interface declares one or
    two functions. You are required to provide implementations of the
    functions representing the services that you wish to implement.
   </para>
   <sect2 id="server.init">
    <title>Init</title>
    <synopsis>
bend_initresult (*bend_init)(bend_initrequest *r);
    </synopsis>
    <para>
     This handler is called once for each new connection request, after
     a new process/thread has been created, and an Initialize Request has
     been received from the client. The pointer to the
     <function>bend_init</function> handler is passed in the call to
     <function>statserv_start</function>.
    </para>
    <para>
     This handler is also called when operating in SRU mode - when
     a connection has been made (even though SRU does not offer
     this service).
    </para>
    <para>
     Unlike previous versions of YAZ, the <function>bend_init</function> also
     serves as a handler that defines the Z39.50 services that the backend
     wish to support. Pointers to <emphasis>all</emphasis> service handlers,
     including search - and fetch must be specified here in this handler.
    </para>
    <para>
     The request - and result structures are defined as
    </para>
    <synopsis>
typedef struct bend_initrequest
{
    /** \brief user/name/password to be read */
    Z_IdAuthentication *auth;
    /** \brief encoding stream (for results) */
    ODR stream;
    /** \brief printing stream */
    ODR print;
    /** \brief decoding stream (use stream for results) */
    ODR decode;
    /** \brief reference ID */
    Z_ReferenceId *referenceId;
    /** \brief peer address of client */
    char *peer_name;

    /** \brief character set and language negotiation

    see include/yaz/z-charneg.h
    */
    Z_CharSetandLanguageNegotiation *charneg_request;

    /** \brief character negotiation response */
    Z_External *charneg_response;

    /** \brief character set (encoding) for query terms

    This is NULL by default. It should be set to the native character
    set that the backend assumes for query terms */
    char *query_charset;

    /** \brief whehter query_charset also applies to recors

    Is 0 (No) by default. Set to 1 (yes) if records is in the same
    character set as queries. If in doubt, use 0 (No).
    */
    int records_in_same_charset;

    char *implementation_id;
    char *implementation_name;
    char *implementation_version;

    /** \brief Z39.50 sort handler */
    int (*bend_sort)(void *handle, bend_sort_rr *rr);
    /** \brief SRU/Z39.50 search handler */
    int (*bend_search)(void *handle, bend_search_rr *rr);
    /** \brief SRU/Z39.50 fetch handler */
    int (*bend_fetch)(void *handle, bend_fetch_rr *rr);
    /** \brief SRU/Z39.50 present handler */
    int (*bend_present)(void *handle, bend_present_rr *rr);
    /** \brief Z39.50 extended services handler */
    int (*bend_esrequest) (void *handle, bend_esrequest_rr *rr);
    /** \brief Z39.50 delete result set handler */
    int (*bend_delete)(void *handle, bend_delete_rr *rr);
    /** \brief Z39.50 scan handler */
    int (*bend_scan)(void *handle, bend_scan_rr *rr);
    /** \brief Z39.50 segment facility handler */
    int (*bend_segment)(void *handle, bend_segment_rr *rr);
    /** \brief SRU explain handler */
    int (*bend_explain)(void *handle, bend_explain_rr *rr);
    /** \brief SRU scan handler */
    int (*bend_srw_scan)(void *handle, bend_scan_rr *rr);
    /** \brief SRU record update handler */
    int (*bend_srw_update)(void *handle, bend_update_rr *rr);

    /** \brief whether named result sets are supported (0=disable, 1=enable) */
    int named_result_sets;
} bend_initrequest;

typedef struct bend_initresult
{
    int errcode;               /* 0==OK */
    char *errstring;           /* system error string or NULL */
    void *handle;              /* private handle to the backend module */
} bend_initresult;
    </synopsis>
    <para>
     In general, the server frontend expects that the
     <literal>bend_*result</literal> pointer that you return is valid at
     least until the next call to a <literal>bend_* function</literal>.
     This applies to all of the functions described herein. The parameter
     structure passed to you in the call belongs to the server frontend, and
     you should not make assumptions about its contents after the current
     function call has completed. In other words, if you want to retain any
     of the contents of a request structure, you should copy them.
    </para>
    <para>
     The <literal>errcode</literal> should be zero if the initialization of
     the backend went well. Any other value will be interpreted as an error.
     The <literal>errstring</literal> isn't used in the current version, but
     one option would be to stick it in the initResponse as a VisibleString.
     The <literal>handle</literal> is the most important parameter. It should
     be set to some value that uniquely identifies the current session to
     the backend implementation. It is used by the frontend server in any
     future calls to a backend function.
     The typical use is to set it to point to a dynamically allocated state
     structure that is private to your backend module.
    </para>
    <para>
     The <literal>auth</literal> member holds the authentication information
     part of the Z39.50 Initialize Request. Interpret this if your serves
     requires authentication.
    </para>
    <para>
     The members <literal>peer_name</literal>,
     <literal>implementation_id</literal>,
     <literal>implementation_name</literal> and
     <literal>implementation_version</literal> holds
     DNS of client, ID of implementor, name
     of client (Z39.50) implementation - and version.
    </para>
    <para>
     The <literal>bend_</literal> - members are set to NULL when
     <function>bend_init</function> is called. Modify the pointers by
     setting them to point to backend functions.
    </para>
   </sect2>
   <sect2 id="server.search.retrieve">
    <title>Search and Retrieve</title>
    <para>
     We now describe the handlers that are required to support search -
     and retrieve. You must support two functions - one for search - and one
     for fetch (retrieval of one record). If desirable you can provide a
     third handler which is called when a present request is received which
     allows you to optimize retrieval of multiple-records.
    </para>
    <synopsis>
int (*bend_search) (void *handle, bend_search_rr *rr);

typedef struct {
    char *setname;             /* name to give to this set */
    int replace_set;           /* replace set, if it already exists */
    int num_bases;             /* number of databases in list */
    char **basenames;          /* databases to search */
    Z_ReferenceId *referenceId;/* reference ID */
    Z_Query *query;            /* query structure */
    ODR stream;                /* encode stream */
    ODR decode;                /* decode stream */
    ODR print;                 /* print stream */

    bend_request request;
    bend_association association;
    int *fd;
    int hits;                  /* number of hits */
    int errcode;               /* 0==OK */
    char *errstring;           /* system error string or NULL */
    Z_OtherInformation *search_info; /* additional search info */
    char *srw_sortKeys;        /* holds SRU/SRW sortKeys info */
    char *srw_setname;         /* holds SRU/SRW generated resultsetID */
    int *srw_setnameIdleTime;  /* holds SRU/SRW life-time */
    int estimated_hit_count;   /* if hit count is estimated */
    int partial_resultset;     /* if result set is partial */
} bend_search_rr;
    </synopsis>
    <para>
     The <function>bend_search</function> handler is a fairly close
     approximation of a protocol Z39.50 Search Request - and Response PDUs
     The <literal>setname</literal> is the resultSetName from the protocol.
     You are required to establish a mapping between the set name and whatever
     your backend database likes to use.
     Similarly, the <literal>replace_set</literal> is a boolean value
     corresponding to the resultSetIndicator field in the protocol.
     <literal>num_bases/basenames</literal> is a length of/array of character
     pointers to the database names provided by the client.
     The <literal>query</literal> is the full query structure as defined in
     the protocol ASN.1 specification.
     It can be either of the possible query types, and it's up to you to
     determine if you can handle the provided query type.
     Rather than reproduce the C interface here, we'll refer you to the
     structure definitions in the file
     <filename>include/yaz/z-core.h</filename>. If you want to look at the
     attributeSetId OID of the RPN query, you can either match it against
     your own internal tables, or you can use the <link linkend="tools.oid">
     OID tools</link>.
    </para>
    <para>
     The structure contains a number of hits, and an
     <literal>errcode/errstring</literal> pair. If an error occurs
     during the search, or if you're unhappy with the request, you should
     set the errcode to a value from the BIB-1 diagnostic set. The value
     will then be returned to the user in a nonsurrogate diagnostic record
     in the response. The <literal>errstring</literal>, if provided, will
     go in the addinfo field. Look at the protocol definition for the
     defined error codes, and the suggested uses of the addinfo field.
    </para>
    <para>
     The <function>bend_search</function> handler is also called when
     the frontend server receives a SRU SearchRetrieveRequest.
     For SRU, a CQL query is usually provided by the client.
     The CQL query is available as part of <literal>Z_Query</literal>
     structure (note that CQL is now part of Z39.50 via an external).
     To support CQL in existing implementations that only do Type-1,
     we refer to the CQL-to-PQF tool described
     <link linkend="cql.to.pqf">here</link>.
    </para>
    <para>
     To maintain backwards compatibility, the frontend server
     of yaz always assume that error codes are BIB-1 diagnostics.
     For SRU operation, a Bib-1 diagnostic code is mapped to
     SRU diagnostic.
    </para>
    <synopsis>
int (*bend_fetch) (void *handle, bend_fetch_rr *rr);

typedef struct bend_fetch_rr {
    char *setname;             /* set name */
    int number;                /* record number */
    Z_ReferenceId *referenceId;/* reference ID */
    Odr_oid *request_format;        /* format, transfer syntax (OID) */
    Z_RecordComposition *comp; /* Formatting instructions */
    ODR stream;                /* encoding stream - memory source if req */
    ODR print;                 /* printing stream */

    char *basename;            /* name of database that provided record */
    int len;                   /* length of record or -1 if structured */
    char *record;              /* record */
    int last_in_set;           /* is it?  */
    Odr_oid *output_format;        /* response format/syntax (OID) */
    int errcode;               /* 0==success */
    char *errstring;           /* system error string or NULL */
    int surrogate_flag;        /* surrogate diagnostic */
    char *schema;              /* string record schema input/output */
} bend_fetch_rr;
    </synopsis>
    <para>
     The frontend server calls the <function>bend_fetch</function> handler
     when it needs database records to fulfill a Z39.50 Search Request, a
     Z39.50 Present Request or a SRU SearchRetrieveRequest.
     The <literal>setname</literal> is simply the name of the result set
     that holds the reference to the desired record.
     The <literal>number</literal> is the offset into the set (with 1
     being the first record in the set). The <literal>format</literal> field
     is the record format requested by the client (See
     <xref linkend="tools.oid"/>).
     A value of NULL for <literal>format</literal> indicates that the
     client did not request a specific format.
     The <literal>stream</literal> argument is an &odr; stream which
     should be used for allocating space for structured data records.
     The stream will be reset when all records have been assembled, and
     the response package has been transmitted.
     For unstructured data, the backend is responsible for maintaining a
     static or dynamic buffer for the record between calls.
    </para>
    <para>
     If a SRU SearchRetrieveRequest is received by the frontend server,
     the <literal>referenceId</literal> is NULL and the
     <literal>format</literal> (transfer syntax) is the OID for XML.
     The schema for SRU is stored in both the
     <literal>Z_RecordComposition</literal>
     structure and <literal>schema</literal> (simple string).
    </para>
    <para>
     In the structure, the <literal>basename</literal> is the name of the
     database that holds the
     record. <literal>len</literal> is the length of the record returned, in
     bytes, and <literal>record</literal> is a pointer to the record.
     <literal>last_in_set</literal> should be nonzero only if the record
     returned is the last one in the given result set.
     <literal>errcode</literal> and <literal>errstring</literal>, if
     given, will be interpreted as a global error pertaining to the
     set, and will be returned in a non-surrogate-diagnostic.
     If you wish to return the error as a surrogate-diagnostic
     (local error) you can do this by setting
     <literal>surrogate_flag</literal> to 1 also.
    </para>
    <para>
     If the <literal>len</literal> field has the value -1, then
     <literal>record</literal> is assumed to point to a constructed data
     type. The <literal>format</literal> field will be used to determine
     which encoder should be used to serialize the data.
    </para>
    <note>
     <para>
      If your backend generates structured records, it should use
      <function>odr_malloc()</function> on the provided stream for allocating
      data: This allows the frontend server to keep track of the record sizes.
     </para>
    </note>
    <para>
     The <literal>format</literal> field is mapped to an object identifier
     in the direct reference of the resulting EXTERNAL representation
     of the record.
    </para>
    <note>
     <para>
      The current version of &yaz; only supports the direct reference mode.
     </para>
    </note>
    <synopsis>
int (*bend_present) (void *handle, bend_present_rr *rr);

typedef struct {
    char *setname;             /* set name */
    int start;
    int number;                /* record number */
    Odr_oid *format;           /* format, transfer syntax (OID) */
    Z_ReferenceId *referenceId;/* reference ID */
    Z_RecordComposition *comp; /* Formatting instructions */
    ODR stream;                /* encoding stream - memory source if required */
    ODR print;                 /* printing stream */
    bend_request request;
    bend_association association;

    int hits;                  /* number of hits */
    int errcode;               /* 0==OK */
    char *errstring;           /* system error string or NULL */
} bend_present_rr;
    </synopsis>
    <para>
     The <function>bend_present</function> handler is called when
     the server receives a Z39.50 Present Request.
     The <literal>setname</literal>,
     <literal>start</literal> and <literal>number</literal> is the
     name of the result set - start position - and number of records to
     be retrieved respectively. <literal>format</literal> and
     <literal>comp</literal> is the preferred transfer syntax and element
     specifications of the present request.
    </para>
    <para>
     Note that this is handler serves as a supplement for
     <function>bend_fetch</function> and need not to be defined in order to
     support search - and retrieve.
    </para>
   </sect2>
   <sect2 id="server.delete">
    <title>Delete</title>
    <para>
     For back-ends that supports delete of a result set only one handler
     must be defined.
    </para>
    <synopsis>
int (*bend_delete)(void *handle, bend_delete_rr *rr);

typedef struct bend_delete_rr {
    int function;
    int num_setnames;
    char **setnames;
    Z_ReferenceId *referenceId;
    int delete_status;      /* status for the whole operation */
    int *statuses;          /* status each set - indexed as setnames */
    ODR stream;
    ODR print;
} bend_delete_rr;
    </synopsis>
    <note>
     <para>
      The delete set function definition is rather primitive, mostly because
      we have had no practical need for it as of yet. If someone wants
      to provide a full delete service, we'd be happy to add the
      extra parameters that are required. Are there clients out there
      that will actually delete sets they no longer need?
     </para>
    </note>
   </sect2>
   <sect2 id="server.scan">
    <title>Scan</title>
    <para>
     For servers that wish to offer the scan service one handler
     must be defined.
    </para>
    <synopsis>
int (*bend_scan)(void *handle, bend_scan_rr *rr);

typedef enum {
    BEND_SCAN_SUCCESS,  /* ok */
    BEND_SCAN_PARTIAL   /* not all entries could be found */
} bend_scan_status;

typedef struct bend_scan_rr {
    int num_bases;      /* number of elements in databaselist */
    char **basenames;   /* databases to search */
    Odr_oid *attributeset;
    Z_ReferenceId *referenceId; /* reference ID */
    Z_AttributesPlusTerm *term;
    ODR stream;         /* encoding stream - memory source if required */
    ODR print;          /* printing stream */

    int *step_size;     /* step size */
    int term_position;  /* desired index of term in result list/returned */
    int num_entries;    /* number of entries requested/returned */

    /* scan term entries. The called handler does not have
       to allocate this. Size of entries is num_entries (see above) */
    struct scan_entry *entries;
    bend_scan_status status;
    int errcode;
    char *errstring;
    char *scanClause;   /* CQL scan clause */
    char *setname;      /* Scan in result set (NULL if omitted) */
} bend_scan_rr;
    </synopsis>
   <para>
    This backend server handles both Z39.50 scan
    and SRU scan. In order for a handler to distinguish between SRU (CQL) scan
    Z39.50 Scan , it must check for a non-NULL value of
    <literal>scanClause</literal>.
   </para>
   <note>
    <para>
     if designed today, it would be a choice using a union or similar,
     but that would break binary compatibility with existing servers.
    </para>
    </note>
   </sect2>
  </sect1>
  <sect1 id="server.invocation">
   <title>Application Invocation</title>
   <para>
    The finished application has the following
    invocation syntax (by way of <function>statserv_main()</function>):
   </para>
   &gfs-synopsis;
   <para>
    The options are:
    &gfs-options;
   </para>
   <para>
    A listener specification consists of a transport mode followed by a
    colon (:) followed by a listener address. The transport mode is
    either <literal>tcp</literal>, <literal>unix:</literal> or
    <literal>ssl</literal>.
   </para>
   <para>
    For TCP and SSL, an address has the form
   </para>
   <synopsis>
    hostname | IP-number [: portnumber]
   </synopsis>
   <para>
    The port number defaults to 210 (standard Z39.50 port).
   </para>
   <para>
    For UNIX, the address is the filename of socket.
   </para>
   <para>
    For TCP/IP and SSL, the special hostnames <literal>@</literal>,
    maps to <literal>IN6ADDR_ANY_INIT</literal> with
    IPV4 binding as well (bindv6only=0),
    The special hostname <literal>@4</literal> binds to
    <literal>INADDR_ANY</literal> (IPV4 only listener).
    The special hostname <literal>@6</literal> binds to
    <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
   </para>
   <example id="server.example.running.unix">
    <title>Running the GFS on Unix</title>
    <para>
     Assuming the server application <replaceable>appname</replaceable> is
     started as root, the following will make it listen on port 210.
     The server will change identity to <literal>nobody</literal>
     and write its log to <filename>/var/log/app.log</filename>.
     <screen>
      application -l /var/log/app.log -u nobody tcp:@:210
     </screen>
    </para>
    <para>
     The server will accept Z39.50 requests and offer SRU service on port 210.
    </para>
   </example>
   <example id="server.example.apache.sru">
    <title>Setting up Apache as SRU Frontend</title>
    <para>
     If you use <ulink url="&url.apache;">Apache</ulink>
     as your public web server and want to offer HTTP port 80
     access to the YAZ server on 210, you can use the
     <ulink url="&url.apache.directive.proxypass;">
      <literal>ProxyPass</literal></ulink>
     directive.
     If you have virtual host
     <literal>srw.mydomain</literal> you can use the following directives
     in Apache's httpd.conf:
     <screen>
      &lt;VirtualHost *>
       ErrorLog /home/srw/logs/error_log
       TransferLog /home/srw/logs/access_log
       ProxyPass / http://srw.mydomain:210/
      &lt;/VirtualHost>
     </screen>
    </para>
    <para>
     The above for the Apache 1.3 series.
    </para>
   </example>
   <example id="server.example.local.access">
    <title>Running a server with local access only</title>
    <para>
     Servers that is only being accessed from the local host should listen
     on UNIX file socket rather than a Internet socket. To listen on
     <filename>/tmp/mysocket</filename> start the server as follows:
     <screen>
      application unix:/tmp/mysocket
     </screen>
    </para>
   </example>
  </sect1>
  <sect1 id="server.vhosts">
   <title>GFS Configuration and Virtual Hosts</title>
   &gfs-virtual;
  </sect1>
 </chapter>
 <chapter id="asn">
  <title>The Z39.50 ASN.1 Module</title>
  <sect1 id="asn.introduction">
   <title>Introduction</title>
   <para>
    The &asn; module provides you with a set of C struct definitions for the
    various PDUs of the Z39.50 protocol, as well as for the complex types
    appearing within the PDUs. For the primitive data types, the C
    representation often takes the form of an ordinary C language type,
    such as <literal>Odr_int</literal> which is equivalent to an integral
    C integer. For ASN.1 constructs that have no direct
    representation in C, such as general octet strings and bit strings,
    the &odr; module (see section <link linkend="odr">The ODR Module</link>)
    provides auxiliary definitions.
   </para>
   <para>
    The &asn; module is located in sub directory <filename>z39.50</filename>.
    There you'll find C files that implements encoders and decoders for the
    Z39.50 types. You'll also find the protocol definitions:
    <filename>z3950v3.asn</filename>, <filename>esupdate.asn</filename>,
    and others.
   </para>
  </sect1>
  <sect1 id="asn.preparing">
   <title>Preparing PDUs</title>
   <para>
    A structure representing a complex ASN.1 type doesn't in itself contain the
    members of that type. Instead, the structure contains
    <emphasis>pointers</emphasis> to the members of the type.
    This is necessary, in part, to allow a mechanism for specifying which
    of the optional structure (SEQUENCE) members are present, and which
    are not. It follows that you will need to somehow provide space for
    the individual members of the structure, and set the pointers to
    refer to the members.
   </para>
   <para>
    The conversion routines don't care how you allocate and maintain your
    C structures - they just follow the pointers that you provide.
    Depending on the complexity of your application, and your personal
    taste, there are at least three different approaches that you may take
    when you allocate the structures.
   </para>
   <para>
    You can use static or automatic local variables in the function that
    prepares the PDU. This is a simple approach, and it provides the most
    efficient form of memory management. While it works well for flat
    PDUs like the InitReqest, it will generally not be sufficient for say,
    the generation of an arbitrarily complex RPN query structure.
   </para>
   <para>
    You can individually create the structure and its members using the
    <function>malloc(2)</function> function. If you want to ensure that
    the data is freed when it is no longer needed, you will have to
    define a function that individually releases each member of a
    structure before freeing the structure itself.
   </para>
   <para>
    You can use the <function>odr_malloc()</function> function (see
    <xref linkend="odr.use"/> for details). When you use
    <function>odr_malloc()</function>, you can release all of the
    allocated data in a single operation, independent of any pointers and
    relations between the data. <function>odr_malloc()</function> is based on a
    &quot;nibble-memory&quot;
    scheme, in which large portions of memory are allocated, and then
    gradually handed out with each call to <function>odr_malloc()</function>.
    The next time you call <function>odr_reset()</function>, all of the
    memory allocated since the last call is recycled for future use (actually,
    it is placed on a free-list).
   </para>
   <para>
    You can combine all of the methods described here. This will often be
    the most practical approach. For instance, you might use
    <function>odr_malloc()</function> to allocate an entire structure and
    some of its elements, while you leave other elements pointing to global
    or per-session default variables.
   </para>
   <para>
    The &asn; module provides an important aid in creating new PDUs. For
    each of the PDU types (say, <function>Z_InitRequest</function>), a
    function is provided that allocates and initializes an instance of
    that PDU type for you. In the case of the InitRequest, the function is
    simply named <function>zget_InitRequest()</function>, and it sets up
    reasonable default value for all of the mandatory members. The optional
    members are generally initialized to null pointers. This last aspect
    is very important: it ensures that if the PDU definitions are
    extended after you finish your implementation (to accommodate
    new versions of the protocol, say), you won't get into trouble with
    uninitialized pointers in your structures. The functions use
    <function>odr_malloc()</function> to
    allocate the PDUs and its members, so you can free everything again with a
    single call to <function>odr_reset()</function>. We strongly recommend
    that you use the <literal>zget_*</literal>
    functions whenever you are preparing a PDU (in a C++ API, the
    <literal>zget_</literal>
    functions would probably be promoted to constructors for the
    individual types).
   </para>
   <para>
   The prototype for the individual PDU types generally look like this:
   </para>
   <synopsis>
    Z_&lt;type> *zget_&lt;type>(ODR o);
   </synopsis>
   <para>
    eg.:
   </para>
   <synopsis>
    Z_InitRequest *zget_InitRequest(ODR o);
   </synopsis>
   <para>
   The &odr; handle should generally be your encoding stream, but it
    needn't be.
   </para>
   <para>
    As well as the individual PDU functions, a function
    <function>zget_APDU()</function> is provided, which allocates
    a top-level Z-APDU of the type requested:
   </para>
   <synopsis>
    Z_APDU *zget_APDU(ODR o, int which);
   </synopsis>
   <para>
    The <varname>which</varname> parameter is (of course) the discriminator
    belonging to the <varname>Z_APDU</varname> <literal>CHOICE</literal> type.
    All of the interface described here is provided by the &asn; module, and
    you access it through the <filename>proto.h</filename> header file.
   </para>
  </sect1>
  <sect1 id="asn.external">
   <title>EXTERNAL Data</title>
   <para>
    In order to achieve extensibility and adaptability to different
    application domains, the new version of the protocol defines many
    structures outside of the main ASN.1 specification, referencing them
    through ASN.1 EXTERNAL constructs. To simplify the construction and
    access to the externally referenced data, the &asn; module defines a
    specialized version of the EXTERNAL construct, called
    <literal>Z_External</literal>.It is defined thus:
   </para>
   <screen>
typedef struct Z_External
{
    Odr_oid *direct_reference;
    int *indirect_reference;
    char *descriptor;
    enum
    {
        /* Generic types */
        Z_External_single = 0,
        Z_External_octet,
        Z_External_arbitrary,

        /* Specific types */
        Z_External_SUTRS,
        Z_External_explainRecord,
        Z_External_resourceReport1,
        Z_External_resourceReport2

    ...

    } which;
    union
    {
        /* Generic types */
        Odr_any *single_ASN1_type;
        Odr_oct *octet_aligned;
        Odr_bitmask *arbitrary;

        /* Specific types */
        Z_SUTRS *sutrs;
        Z_ExplainRecord *explainRecord;
        Z_ResourceReport1 *resourceReport1;
        Z_ResourceReport2 *resourceReport2;

        ...

    } u;
} Z_External;
   </screen>
   <para>
    When decoding, the &asn; module will attempt to determine which
    syntax describes the data by looking at the reference fields
    (currently only the direct-reference). For ASN.1 structured data, you
    need only consult the <literal>which</literal> field to determine the
    type of data. You can the access  the data directly through the union.
    When constructing data for encoding, you set the union pointer to point
    to the data, and set the <literal>which</literal> field accordingly.
    Remember also to set the direct (or indirect) reference to the correct
    OID for the data type.
    For non-ASN.1 data such as MARC records, use the
    <literal>octet_aligned</literal> arm of the union.
   </para>
   <para>
    Some servers return ASN.1 structured data values (eg. database
    records) as BER-encoded records placed in the
    <literal>octet-aligned</literal> branch of the EXTERNAL CHOICE.
    The ASN-module will <emphasis>not</emphasis> automatically decode
    these records. To help you decode the records in the application, the
    function
   </para>
   <screen>
   Z_ext_typeent *z_ext_gettypebyref(const oid *oid);
   </screen>
   <para>
    Can be used to retrieve information about the known, external data
    types. The function return a pointer to a static area, or NULL, if no
    match for the given direct reference is found. The
    <literal>Z_ext_typeent</literal>
    is defined as:
   </para>
   <screen>
typedef struct Z_ext_typeent
{
    int oid[OID_SIZE]; /* the direct-reference OID. */
    int what;          /* discriminator value for the external CHOICE */
    Odr_fun fun;       /* decoder function */
} Z_ext_typeent;
   </screen>
   <para>
    The <literal>what</literal> member contains the
    <literal>Z_External</literal> union discriminator value for the
    given type: For the SUTRS record syntax, the value would be
    <literal>Z_External_sutrs</literal>.
    The <literal>fun</literal> member contains a pointer to the
    function which encodes/decodes the given type. Again, for the SUTRS
    record syntax, the value of <literal>fun</literal> would be
    <literal>z_SUTRS</literal> (a function pointer).
   </para>
   <para>
    If you receive an EXTERNAL which contains an octet-string value that
    you suspect of being an ASN.1-structured data value, you can use
    <literal>z_ext_gettypebyref</literal> to look for the provided
    direct-reference.
    If the return value is different from NULL, you can use the provided
    function to decode the BER string (see <xref linkend="odr.use"/>
    ).
   </para>
   <para>
    If you want to <emphasis>send</emphasis> EXTERNALs containing
    ASN.1-structured values in the occtet-aligned branch of the CHOICE, this
    is possible too. However, on the encoding phase, it requires a somewhat
    involved juggling around of the various buffers involved.
   </para>
   <para>
    If you need to add new, externally defined data types, you must update
    the struct above, in the source file <filename>prt-ext.h</filename>, as
    well as the encoder/decoder in the file <filename>prt-ext.c</filename>.
    When changing the latter, remember to update both the
    <literal>arm</literal> arrary and the list
    <literal>type_table</literal>, which drives the CHOICE biasing that
    is necessary to tell the different, structured types apart
    on decoding.
   </para>
   <note>
    <para>
     Eventually, the EXTERNAL processing will most likely
     automatically insert the correct OIDs or indirect-refs. First,
     however, we need to determine how application-context management
     (specifically the presentation-context-list) should fit into the
     various modules.
    </para>
   </note>
  </sect1>
  <sect1 id="asn.pdu">
   <title>PDU Contents Table</title>
   <para>
    We include, for reference, a listing of the fields of each top-level
    PDU, as well as their default settings.
   </para>
   <table frame="top" id="asn.default.initialize.request">
    <title>Default settings for PDU Initialize Request</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
    <thead>
     <row>
      <entry>Field</entry>
      <entry>Type</entry>
      <entry>Default Value</entry>
     </row>
    </thead>
    <tbody>
     <row><entry>
      referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
     </entry></row>
     <row><entry>
      protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
     </entry></row>
     <row><entry>
      options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
     </entry></row>
     <row><entry>
      preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
     </entry></row>
     <row><entry>
      maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
     </entry></row>
     <row><entry>
      idAuthentication</entry><entry>Z_IdAuthentication</entry><entry>NULL
     </entry></row>
     <row><entry>
      implementationId</entry><entry>char*</entry><entry>"81"
     </entry></row>
     <row><entry>
      implementationName</entry><entry>char*</entry><entry>"YAZ"
     </entry></row>
     <row><entry>
      implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
     </entry></row>
     <row><entry>
      userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
     </entry></row>
     <row><entry>
      otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
     </entry></row>
    </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.initialize.response">
    <title>Default settings for PDU Initialize Response</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
       referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
      </entry></row>
      <row><entry>
       protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
      </entry></row>
      <row><entry>
       options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
      </entry></row>
      <row><entry>
       preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
      </entry></row>
      <row><entry>
       maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
      </entry></row>
      <row><entry>
       result</entry><entry>Odr_bool</entry><entry>TRUE
      </entry></row>
      <row><entry>
       implementationId</entry><entry>char*</entry><entry>"id)"
      </entry></row>
      <row><entry>
       implementationName</entry><entry>char*</entry><entry>"YAZ"
      </entry></row>
      <row><entry>
       implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
      </entry></row>
      <row><entry>
       userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
      </entry></row>
      <row><entry>
       otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
      </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.search.request">
    <title>Default settings for PDU Search Request</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        smallSetUpperBound</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        largeSetLowerBound</entry><entry>Odr_int</entry><entry>1
       </entry></row>
      <row><entry>
        mediumSetPresentNumber</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        replaceIndicator</entry><entry>Odr_bool</entry><entry>TRUE
       </entry></row>
      <row><entry>
        resultSetName</entry><entry>char *</entry><entry>"default"
       </entry></row>
      <row><entry>
        num_databaseNames</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        databaseNames</entry><entry>char **</entry><entry>NULL
       </entry></row>
      <row><entry>
        smallSetElementSetNames</entry><entry>Z_ElementSetNames
       </entry><entry>NULL
       </entry></row>
      <row><entry>
        mediumSetElementSetNames</entry><entry>Z_ElementSetNames
       </entry><entry>NULL
       </entry></row>
      <row><entry>
        preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
       </entry></row>
      <row><entry>
        query</entry><entry>Z_Query</entry><entry>NULL
       </entry></row>
      <row><entry>
        additionalSearchInfo</entry><entry>Z_OtherInformation
       </entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.search.response">
    <title>Default settings for PDU Search Response</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
       referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
      </entry></row>
      <row><entry>
       resultCount</entry><entry>Odr_int</entry><entry>0
      </entry></row>
      <row><entry>
       numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
      </entry></row>
      <row><entry>
       nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
      </entry></row>
      <row><entry>
       searchStatus</entry><entry>Odr_bool</entry><entry>TRUE
      </entry></row>
      <row><entry>
       resultSetStatus</entry><entry>Odr_int</entry><entry>NULL
      </entry></row>
      <row><entry>
       presentStatus</entry><entry>Odr_int</entry><entry>NULL
      </entry></row>
      <row><entry>
       records</entry><entry>Z_Records</entry><entry>NULL
      </entry></row>
      <row><entry>
       additionalSearchInfo</entry>
       <entry>Z_OtherInformation</entry><entry>NULL
      </entry></row>
      <row><entry>
       otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
      </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.present.request">
    <title>Default settings for PDU Present Request</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        resultSetId</entry><entry>char*</entry><entry>"default"
       </entry></row>
      <row><entry>
        resultSetStartPoint</entry><entry>Odr_int</entry><entry>1
       </entry></row>
      <row><entry>
        numberOfRecordsRequested</entry><entry>Odr_int</entry><entry>10
       </entry></row>
      <row><entry>
        num_ranges</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        additionalRanges</entry><entry>Z_Range</entry><entry>NULL
       </entry></row>
      <row><entry>
        recordComposition</entry><entry>Z_RecordComposition</entry><entry>NULL
       </entry></row>
      <row><entry>
        preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
       </entry></row>
      <row><entry>
        maxSegmentCount</entry><entry>Odr_int</entry><entry>NULL
       </entry></row>
      <row><entry>
        maxRecordSize</entry><entry>Odr_int</entry><entry>NULL
       </entry></row>
      <row><entry>
        maxSegmentSize</entry><entry>Odr_int</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.present.response">
    <title>Default settings for PDU Present Response</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        presentStatus</entry><entry>Odr_int</entry><entry>Z_PresentStatus_success
       </entry></row>
      <row><entry>
        records</entry><entry>Z_Records</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.delete.result.set.request">
    <title>Default settings for Delete Result Set Request</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>referenceId
       </entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        deleteFunction</entry><entry>Odr_int</entry><entry>Z_DeleteResultSetRequest_list
       </entry></row>
      <row><entry>
        num_ids</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        resultSetList</entry><entry>char**</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.delete.result.set.response">
    <title>Default settings for Delete Result Set Response</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        deleteOperationStatus</entry><entry>Odr_int</entry>
       <entry>Z_DeleteStatus_success</entry></row>
      <row><entry>
        num_statuses</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        deleteListStatuses</entry><entry>Z_ListStatus**</entry><entry>NULL
       </entry></row>
      <row><entry>
        numberNotDeleted</entry><entry>Odr_int</entry><entry>NULL
       </entry></row>
      <row><entry>
        num_bulkStatuses</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        bulkStatuses</entry><entry>Z_ListStatus</entry><entry>NUL
        L</entry></row>
      <row><entry>
        deleteMessage</entry><entry>char*</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.scan.request">
    <title>Default settings for Scan Request</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        num_databaseNames</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        databaseNames</entry><entry>char**</entry><entry>NULL
       </entry></row>
      <row><entry>
        attributeSet</entry><entry>Odr_oid</entry><entry>NULL
       </entry></row>
      <row><entry>
        termListAndStartPoint</entry><entry>Z_AttributesPlus...
       </entry><entry>NULL</entry></row>
      <row><entry>
        stepSize</entry><entry>Odr_int</entry><entry>NULL
       </entry></row>
      <row><entry>
        numberOfTermsRequested</entry><entry>Odr_int</entry><entry>20
       </entry></row>
      <row><entry>
        preferredPositionInResponse</entry><entry>Odr_int</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.scan.response">
    <title>Default settings for Scan Response</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
       referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
      </entry></row>
      <row><entry>
       stepSize</entry><entry>Odr_int</entry><entry>NULL
      </entry></row>
      <row><entry>
       scanStatus</entry><entry>Odr_int</entry><entry>Z_Scan_success
      </entry></row>
      <row><entry>
       numberOfEntriesReturned</entry><entry>Odr_int</entry><entry>0
      </entry></row>
      <row><entry>
       positionOfTerm</entry><entry>Odr_int</entry><entry>NULL
      </entry></row>
      <row><entry>
       entries</entry><entry>Z_ListEntris</entry><entry>NULL
      </entry></row>
      <row><entry>
       attributeSet</entry><entry>Odr_oid</entry><entry>NULL
      </entry></row>
      <row><entry>
       otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
      </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.trigger.resource.control.request">
    <title>Default settings for Trigger Resource Control Request</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        requestedAction</entry><entry>Odr_int</entry><entry>
        Z_TriggerResourceCtrl_resou..
       </entry></row>
      <row><entry>
        prefResourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
       </entry></row>
      <row><entry>
        resultSetWanted</entry><entry>Odr_bool</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.resource.control.request">
    <title>Default settings for Resource Control Request</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        suspendedFlag</entry><entry>Odr_bool</entry><entry>NULL
       </entry></row>
      <row><entry>
        resourceReport</entry><entry>Z_External</entry><entry>NULL
       </entry></row>
      <row><entry>
        partialResultsAvailable</entry><entry>Odr_int</entry><entry>NULL
       </entry></row>
      <row><entry>
        responseRequired</entry><entry>Odr_bool</entry><entry>FALSE
       </entry></row>
      <row><entry>
        triggeredRequestFlag</entry><entry>Odr_bool</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.resource.control.response">
    <title>Default settings for Resource Control Response</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        continueFlag</entry><entry>bool_t</entry><entry>TRUE
       </entry></row>
      <row><entry>
        resultSetWanted</entry><entry>bool_t</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.access.control.request">
    <title>Default settings for Access Control Request</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        which</entry><entry>enum</entry><entry>Z_AccessRequest_simpleForm;
       </entry></row>
      <row><entry>
        u</entry><entry>union</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.access.control.response">
    <title>Default settings for Access Control Response</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        which</entry><entry>enum</entry><entry>Z_AccessResponse_simpleForm
       </entry></row>
      <row><entry>
        u</entry><entry>union</entry><entry>NULL
       </entry></row>
      <row><entry>
        diagnostic</entry><entry>Z_DiagRec</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.segment">
    <title>Default settings for Segment</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>value=0
       </entry></row>
      <row><entry>
        num_segmentRecords</entry><entry>Odr_int</entry><entry>0
       </entry></row>
      <row><entry>
        segmentRecords</entry><entry>Z_NamePlusRecord</entry><entry>NULL
       </entry></row>
      <row><entry>otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
   <table frame="top" id="asn.default.close">
    <title>Default settings for Close</title>
    <tgroup cols="3">
     <colspec colwidth="7*" colname="field"></colspec>
     <colspec colwidth="5*" colname="type"></colspec>
     <colspec colwidth="7*" colname="value"></colspec>
     <thead>
      <row>
       <entry>Field</entry>
       <entry>Type</entry>
       <entry>Default Value</entry>
      </row>
     </thead>
     <tbody>
      <row><entry>
        referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
       </entry></row>
      <row><entry>
        closeReason</entry><entry>Odr_int</entry><entry>Z_Close_finished
       </entry></row>
      <row><entry>
        diagnosticInformation</entry><entry>char*</entry><entry>NULL
       </entry></row>
      <row><entry>
        resourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
       </entry></row>
      <row><entry>
        resourceFormat</entry><entry>Z_External</entry><entry>NULL
       </entry></row>
      <row><entry>
        otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
       </entry></row>
     </tbody>
    </tgroup>
   </table>
  </sect1>
 </chapter>
 <chapter id="soap">
  <title>SOAP and SRU</title>
  <sect1 id="soap.introduction">
   <title>Introduction</title>
   <para>
    &yaz; uses a very simple implementation of
    <ulink url="&url.soap;">SOAP</ulink> that only,
    currenly, supports what is sufficient to offer SRU SOAP functionality.
    The implementation uses the
    <ulink url="&url.libxml2.api.tree;">tree API</ulink> of
    libxml2 to encode and decode SOAP packages.
   </para>
   <para>
    Like the Z39.50 ASN.1 module, the &yaz; SRU implementation uses
    simple C structs to represent SOAP packages as well as
    HTTP packages.
   </para>
  </sect1>
  <sect1 id="soap.http">
   <title>HTTP</title>
   <para>
    &yaz; only offers HTTP as transport carrier for SOAP, but it is
    relatively easy to change that.
   </para>
   <para>
    The following definition of <literal>Z_GDU</literal> (Generic Data
    Unit) allows for both HTTP and Z39.50 in one packet.
   </para>
   <synopsis>
#include &lt;yaz/zgdu.h&gt;

#define Z_GDU_Z3950         1
#define Z_GDU_HTTP_Request  2
#define Z_GDU_HTTP_Response 3
typedef struct {
  int which;
  union {
    Z_APDU *z3950;
    Z_HTTP_Request *HTTP_Request;
    Z_HTTP_Response *HTTP_Response;
  } u;
} Z_GDU ;
   </synopsis>
   <para>
    The corresponding Z_GDU encoder/decoder is <function>z_GDU</function>.
    The <literal>z3950</literal> is any of the known BER encoded Z39.50
    APDUs.
    <literal>HTTP_Request</literal> and <literal>HTTP_Response</literal>
    is the HTTP Request and Response respectively.
   </para>
  </sect1>
  <sect1 id="soap.xml">
   <title>SOAP Packages</title>
   <para>
    Every SOAP package in &yaz; is represented as follows:
    <synopsis>
#include &lt;yaz/soap.h&gt;

typedef struct {
    char *fault_code;
    char *fault_string;
    char *details;
} Z_SOAP_Fault;

typedef struct {
    int no;
    char *ns;
    void *p;
} Z_SOAP_Generic;

#define Z_SOAP_fault 1
#define Z_SOAP_generic 2
#define Z_SOAP_error 3
typedef struct {
    int which;
    union {
        Z_SOAP_Fault   *fault;
        Z_SOAP_Generic *generic;
        Z_SOAP_Fault   *soap_error;
    } u;
    const char *ns;
} Z_SOAP;
    </synopsis>
   </para>
   <para>
    The <literal>fault</literal> and <literal>soap_error</literal>
    arms represent both a SOAP fault - struct
    <literal>Z_SOAP_Fault</literal>. Any other generic
    (valid) package is represented by <literal>Z_SOAP_Generic</literal>.
   </para>
   <para>
    The <literal>ns</literal> as part of <literal>Z_SOAP</literal>
    is the namespace for SOAP itself and reflects the SOAP
    version. For version 1.1 it is
    <literal>http://schemas.xmlsoap.org/soap/envelope/</literal>,
    for version 1.2 it is
    <literal>http://www.w3.org/2001/06/soap-envelope</literal>.
   </para>
   <synopsis>
int z_soap_codec(ODR o, Z_SOAP **pp,
                 char **content_buf, int *content_len,
                 Z_SOAP_Handler *handlers);
   </synopsis>
   <para>
    The <literal>content_buf</literal> and <literal>content_len</literal>
    is XML buffer and length of buffer respectively.
   </para>
   <para>
    The <literal>handlers</literal> is a list of SOAP codec
    handlers - one handler for each service namespace. For SRU SOAP, the
    namespace would be <literal>http://www.loc.gov/zing/srw/v1.0/</literal>.
   </para>
   <para>
    When decoding, the <function>z_soap_codec</function>
    inspects the XML content
    and tries to match one of the services namespaces of the
    supplied handlers. If there is a match a handler function
    is invoked which decodes that particular SOAP package.
    If successful, the returned <literal>Z_SOAP</literal> package will be
    of type <literal>Z_SOAP_Generic</literal>.
    Member <literal>no</literal> is
    set the offset of handler that matched; <literal>ns</literal>
    is set to namespace of matching handler; the void pointer
    <literal>p</literal> is set to the C data structure assocatiated
    with the handler.
   </para>
   <para>
    When a NULL namespace is met (member <literal>ns</literal> bwlow),
    that specifies end-of-list.
   </para>
   <para>
    Each handler is defined as follows:
    <synopsis>
typedef struct {
    char *ns;
    void *client_data;
    Z_SOAP_fun f;
} Z_SOAP_Handler;
    </synopsis>
    The <literal>ns</literal> is namespace of service associated with
    handler <literal>f</literal>. <literal>client_data</literal>
    is user-defined data which is passed to handler.
   </para>
   <para>
    The prototype for a SOAP service handler is:
    <synopsis>
int handler(ODR o, void * ptr, void **handler_data,
            void *client_data, const char *ns);
    </synopsis>
    The <parameter>o</parameter> specifies the mode (decode/encode)
    as usual. The second argument, <parameter>ptr</parameter>,
    is a libxml2 tree node pointer (<literal>xmlNodePtr</literal>)
    and is a pointer to the <literal>Body</literal> element
    of the SOAP package. The <parameter>handler_data</parameter>
    is an opaque pointer to a C definitions associated with the
    SOAP service. <parameter>client_data</parameter> is the pointer
    which was set as part of the <literal>Z_SOAP_handler</literal>.
    Finally, <parameter>ns</parameter> the service namespace.
   </para>
  </sect1>
  <sect1 id="soap.srw">
   <title>SRU</title>
   <para>
    SRU SOAP is just one implementation of a SOAP handler as described
    in the previous section.
    The encoder/decoder handler for SRU is defined as
    follows:
    <synopsis>
#include &lt;yaz/srw.h&gt;

int yaz_srw_codec(ODR o, void * pptr,
                  Z_SRW_GDU **handler_data,
                  void *client_data, const char *ns);
    </synopsis>
    Here, <literal>Z_SRW_GDU</literal> is either
    searchRetrieveRequest or a searchRetrieveResponse.
   </para>
   <note>
    <para>
     The xQuery and xSortKeys are not handled yet by
     the SRW implementation of &yaz;. Explain is also missing.
     Future versions of &yaz; will include these features.
    </para>
   </note>
   <para>
    The definition of searchRetrieveRequest is:
    <synopsis>
typedef struct {

#define Z_SRW_query_type_cql  1
#define Z_SRW_query_type_xcql 2
#define Z_SRW_query_type_pqf  3
    int query_type;
    union {
        char *cql;
        char *xcql;
        char *pqf;
    } query;

#define Z_SRW_sort_type_none 1
#define Z_SRW_sort_type_sort 2
#define Z_SRW_sort_type_xSort 3
    int sort_type;
    union {
        char *none;
        char *sortKeys;
        char *xSortKeys;
    } sort;
    int  *startRecord;
    int  *maximumRecords;
    char *recordSchema;
    char *recordPacking;
    char *database;
} Z_SRW_searchRetrieveRequest;
    </synopsis>
    Please observe that data of type xsd:string is represented
    as a char pointer (<literal>char *</literal>). A null pointer
    means that the element is absent.
    Data of type xsd:integer is representd as a pointer to
    an int (<literal>int *</literal>). Again, a null pointer
    us used for absent elements.
   </para>
   <para>
    The SearchRetrieveResponse has the following definition.
    <synopsis>
typedef struct {
    int * numberOfRecords;
    char * resultSetId;
    int * resultSetIdleTime;

    Z_SRW_record *records;
    int num_records;

    Z_SRW_diagnostic *diagnostics;
    int num_diagnostics;
    int *nextRecordPosition;
} Z_SRW_searchRetrieveResponse;
    </synopsis>
    The <literal>num_records</literal> and <literal>num_diagnostics</literal>
    is number of returned records and diagnostics respectively and also
    correspond to the "size of" arrays <literal>records</literal>
    and <literal>diagnostics</literal>.
   </para>
   <para>
    A retrieval record is defined as follows:
    <synopsis>
typedef struct {
    char *recordSchema;
    char *recordData_buf;
    int recordData_len;
    int *recordPosition;
} Z_SRW_record;
    </synopsis>
    The record data is defined as a buffer of some length so that
    data can be of any type. SRW 1.0 currenly doesn't allow for this
    (only XML), but future versions might do.
   </para>
   <para>
    And, a diagnostic as:
    <synopsis>
typedef struct {
    int  *code;
    char *details;
} Z_SRW_diagnostic;
    </synopsis>
   </para>
  </sect1>
 </chapter>
 <chapter id="tools">
  <title>Supporting Tools</title>
  <para>
   In support of the service API - primarily the ASN module, which
   provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
   a collection of tools that support the development of applications.
  </para>
  <sect1 id="tools.query">
   <title>Query Syntax Parsers</title>
   <para>
    Since the type-1 (RPN) query structure has no direct, useful string
    representation, every origin application needs to provide some form of
    mapping from a local query notation or representation to a
    <token>Z_RPNQuery</token> structure. Some programmers will prefer to
    construct the query manually, perhaps using
    <function>odr_malloc()</function> to simplify memory management.
    The &yaz; distribution includes three separate, query-generating tools
    that may be of use to you.
   </para>
   <sect2 id="PQF">
    <title>Prefix Query Format</title>
    <para>
     Since RPN or reverse polish notation is really just a fancy way of
     describing a suffix notation format (operator follows operands), it
     would seem that the confusion is total when we now introduce a prefix
     notation for RPN. The reason is one of simple laziness - it's somewhat
     simpler to interpret a prefix format, and this utility was designed
     for maximum simplicity, to provide a baseline representation for use
     in simple test applications and scripting environments (like Tcl). The
     demonstration client included with YAZ uses the PQF.
    </para>
    <note>
     <para>
      The PQF have been adopted by other parties developing Z39.50
      software. It is often referred to as Prefix Query Notation
      - PQN.
     </para>
    </note>
    <para>
     The PQF is defined by the pquery module in the YAZ library.
     There are two sets of function that have similar behavior. First
     set operates on a PQF parser handle, second set doesn't. First set
     set of functions are more flexible than the second set. Second set
     is obsolete and is only provided to ensure backwards compatibility.
    </para>
    <para>
     First set of functions all operate on a PQF parser handle:
    </para>
    <synopsis>
     #include &lt;yaz/pquery.h&gt;

     YAZ_PQF_Parser yaz_pqf_create(void);

     void yaz_pqf_destroy(YAZ_PQF_Parser p);

     Z_RPNQuery *yaz_pqf_parse(YAZ_PQF_Parser p, ODR o, const char *qbuf);

     Z_AttributesPlusTerm *yaz_pqf_scan(YAZ_PQF_Parser p, ODR o,
                          Odr_oid **attributeSetId, const char *qbuf);

     int yaz_pqf_error(YAZ_PQF_Parser p, const char **msg, size_t *off);
    </synopsis>
    <para>
     A PQF parser is created and destructed by functions
     <function>yaz_pqf_create</function> and
     <function>yaz_pqf_destroy</function> respectively.
     Function <function>yaz_pqf_parse</function> parses query given
     by string <literal>qbuf</literal>. If parsing was successful,
     a Z39.50 RPN Query is returned which is created using ODR stream
     <literal>o</literal>. If parsing failed, a NULL pointer is
     returned.
     Function <function>yaz_pqf_scan</function> takes a scan query in
     <literal>qbuf</literal>. If parsing was successful, the function
     returns attributes plus term pointer and modifies
     <literal>attributeSetId</literal> to hold attribute set for the
     scan request - both allocated using ODR stream <literal>o</literal>.
     If parsing failed, yaz_pqf_scan returns a NULL pointer.
     Error information for bad queries can be obtained by a call to
     <function>yaz_pqf_error</function> which returns an error code and
     modifies <literal>*msg</literal> to point to an error description,
     and modifies <literal>*off</literal> to the offset within last
     query were parsing failed.
    </para>
    <para>
     The second set of functions are declared as follows:
    </para>
    <synopsis>
     #include &lt;yaz/pquery.h&gt;

     Z_RPNQuery *p_query_rpn(ODR o, oid_proto proto, const char *qbuf);

     Z_AttributesPlusTerm *p_query_scan(ODR o, oid_proto proto,
                             Odr_oid **attributeSetP, const char *qbuf);

     int p_query_attset(const char *arg);
    </synopsis>
    <para>
     The function <function>p_query_rpn()</function> takes as arguments an
     &odr; stream (see section <link linkend="odr">The ODR Module</link>)
     to provide a memory source (the structure created is released on
     the next call to <function>odr_reset()</function> on the stream), a
     protocol identifier (one of the constants <token>PROTO_Z3950</token> and
     <token>PROTO_SR</token>), an attribute set reference, and
     finally a null-terminated string holding the query string.
    </para>
    <para>
     If the parse went well, <function>p_query_rpn()</function> returns a
     pointer to a <literal>Z_RPNQuery</literal> structure which can be
     placed directly into a <literal>Z_SearchRequest</literal>.
     If parsing failed, due to syntax error, a NULL pointer is returned.
    </para>
    <para>
     The <literal>p_query_attset</literal> specifies which attribute set
     to use if the query doesn't specify one by the
     <literal>@attrset</literal> operator.
     The <literal>p_query_attset</literal> returns 0 if the argument is a
     valid attribute set specifier; otherwise the function returns -1.
    </para>
    <para>
     The grammar of the PQF is as follows:
    </para>
    <literallayout>
     query ::= top-set query-struct.

     top-set ::= [ '@attrset' string ]

     query-struct ::= attr-spec | simple | complex | '@term' term-type query

     attr-spec ::= '@attr' [ string ] string query-struct

     complex ::= operator query-struct query-struct.

     operator ::= '@and' | '@or' | '@not' | '@prox' proximity.

     simple ::= result-set | term.

     result-set ::= '@set' string.

     term ::= string.

     proximity ::= exclusion distance ordered relation which-code unit-code.

     exclusion ::= '1' | '0' | 'void'.

     distance ::= integer.

     ordered ::= '1' | '0'.

     relation ::= integer.

     which-code ::= 'known' | 'private' | integer.

     unit-code ::= integer.

     term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
    </literallayout>
    <para>
     You will note that the syntax above is a fairly faithful
     representation of RPN, except for the Attribute, which has been
     moved a step away from the term, allowing you to associate one or more
     attributes with an entire query structure. The parser will
     automatically apply the given attributes to each term as required.
    </para>
    <para>
     The @attr operator is followed by an attribute specification
     (<literal>attr-spec</literal> above). The specification consists
     of an optional attribute set, an attribute type-value pair and
     a sub-query. The attribute type-value pair is packed in one string:
     an attribute type, an equals sign, and an attribute value, like this:
     <literal>@attr 1=1003</literal>.
     The type is always an integer but the value may be either an
     integer or a string (if it doesn't start with a digit character).
     A string attribute-value is encoded as a Type-1 ``complex''
     attribute with the list of values containing the single string
     specified, and including no semantic indicators.
    </para>
    <para>
     Version 3 of the Z39.50 specification defines various encoding of terms.
     Use <literal>@term </literal> <replaceable>type</replaceable>
     <replaceable>string</replaceable>,
     where type is one of: <literal>general</literal>,
     <literal>numeric</literal> or <literal>string</literal>
     (for InternationalString).
     If no term type has been given, the <literal>general</literal> form
     is used.  This is the only encoding allowed in both versions 2 and 3
     of the Z39.50 standard.
    </para>
    <sect3 id="PQF-prox">
     <title>Using Proximity Operators with PQF</title>
     <note>
      <para>
       This is an advanced topic, describing how to construct
       queries that make very specific requirements on the
       relative location of their operands.
       You may wish to skip this section and go straight to
       <link linkend="pqf-examples">the example PQF queries</link>.
      </para>
      <para>
       <warning>
        <para>
         Most Z39.50 servers do not support proximity searching, or
         support only a small subset of the full functionality that
         can be expressed using the PQF proximity operator.  Be
         aware that the ability to <emphasis>express</emphasis> a
         query in PQF is no guarantee that any given server will
         be able to <emphasis>execute</emphasis> it.
        </para>
       </warning>
      </para>
     </note>
     <para>
      The proximity operator <literal>@prox</literal> is a special
      and more restrictive version of the conjunction operator
      <literal>@and</literal>.  Its semantics are described in
      section 3.7.2 (Proximity) of Z39.50 the standard itself, which
      can be read on-line at
      <ulink url="&url.z39.50.proximity;"/>
     </para>
     <para>
      In PQF, the proximity operation is represented by a sequence
      of the form
      <screen>
       @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
      </screen>
      in which the meanings of the parameters are as described in in
      the standard, and they can take the following values:
      <itemizedlist>
       <listitem>
        <formalpara><title>exclusion</title>
        <para>
         0 = false (i.e. the proximity condition specified by the
         remaining parameters must be satisfied) or
         1 = true (the proximity condition specified by the
         remaining parameters must <emphasis>not</emphasis> be
         satisifed).
        </para>
       </formalpara>
       </listitem>
       <listitem>
        <formalpara><title>distance</title><para>
        An integer specifying the difference between the locations
        of the operands: e.g. two adjacent words would have
        distance=1 since their locations differ by one unit.
       </para>
       </formalpara></listitem>
       <listitem>
        <formalpara><title>ordered</title><para>
        1 = ordered (the operands must occur in the order the
        query specifies them) or
        0 = unordered (they may appear in either order).
       </para>
       </formalpara>
       </listitem>
       <listitem>
        <formalpara><title>relation</title><para>
        Recognised values are
        1 (lessThan),
        2 (lessThanOrEqual),
        3 (equal),
        4 (greaterThanOrEqual),
        5 (greaterThan) and
        6 (notEqual).
       </para>
       </formalpara>
       </listitem>
       <listitem>
        <formalpara><title>which-code</title><para>
        <literal>known</literal>
        or
        <literal>k</literal>
        (the unit-code parameter is taken from the well-known list
        of alternatives described in below) or
        <literal>private</literal>
        or
        <literal>p</literal>
        (the unit-code paramater has semantics specific to an
        out-of-band agreement such as a profile).
       </para>
       </formalpara>
       </listitem>
       <listitem>
        <formalpara><title>unit-code</title><para>
        If the which-code parameter is <literal>known</literal>
        then the recognised values are
        1 (character),
        2 (word),
        3 (sentence),
        4 (paragraph),
        5 (section),
        6 (chapter),
        7 (document),
        8 (element),
        9 (subelement),
        10 (elementType) and
        11 (byte).
        If which-code is <literal>private</literal> then the
        acceptable values are determined by the profile.
       </para>
        </formalpara>
       </listitem>
      </itemizedlist>
      (The numeric values of the relation and well-known unit-code
      parameters are taken straight from
      <ulink url="&url.z39.50.proximity.asn1;"
             >the ASN.1</ulink> of the proximity structure in the standard.)
     </para>
    </sect3>
    <sect3 id="pqf-examples">
     <title>PQF queries</title>
     <example id="example.pqf.simple.terms">
      <title>PQF queries using simple terms</title>
      <para>
       <screen>
        dylan

        "bob dylan"
       </screen>
      </para>
     </example>
     <example id="pqf.example.pqf.boolean.operators">
      <title>PQF boolean operators</title>
      <para>
       <screen>
        @or "dylan" "zimmerman"

        @and @or dylan zimmerman when

        @and when @or dylan zimmerman
       </screen>
      </para>
     </example>
     <example id="example.pqf.result.sets">
      <title>PQF references to result sets</title>
      <para>
       <screen>
        @set Result-1

        @and @set seta @set setb
       </screen>
      </para>
     </example>
     <example id="example.pqf.attributes">
      <title>Attributes for terms</title>
      <para>
       <screen>
        @attr 1=4 computer

        @attr 1=4 @attr 4=1 "self portrait"

        @attrset exp1 @attr 1=1 CategoryList

        @attr gils 1=2008 Copenhagen

        @attr 1=/book/title computer
       </screen>
      </para>
     </example>
     <example id="example.pqf.proximity">
      <title>PQF Proximity queries</title>
      <para>
       <screen>
        @prox 0 3 1 2 k 2 dylan zimmerman
       </screen>
       Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
       distance, ordered, relation, which-code and unit-code, in that
       order.  So:
       <itemizedlist>
        <listitem>
         <para>exclusion = 0: the proximity condition must hold</para>
        </listitem>
        <listitem>
         <para>distance = 3: the terms must be three units apart</para>
        </listitem>
        <listitem>
         <para>
          ordered = 1: they must occur in the order they are specified
        </para>
        </listitem>
        <listitem>
         <para>
        relation = 2: lessThanOrEqual (to the distance of 3 units)
        </para>
        </listitem>
        <listitem>
         <para>
          which-code is ``known'', so the standard unit-codes are used
         </para>
        </listitem>
        <listitem>
         <para>unit-code = 2: word.</para>
        </listitem>
       </itemizedlist>
       So the whole proximity query means that the words
       <literal>dylan</literal> and <literal>zimmerman</literal> must
       both occur in the record, in that order, differing in position
       by three or fewer words (i.e. with two or fewer words between
       them.)  The query would find ``Bob Dylan, aka. Robert
       Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
       since the distance in this case is four.
      </para>
     </example>
     <example id="example.pqf.search.term.type">
      <title>PQF specification of search term type</title>
      <para>
       <screen>
        @term string "a UTF-8 string, maybe?"
       </screen>
      </para>
     </example>
     <example id="example.pqf.mixed.queries">
      <title>PQF mixed queries</title>
      <para>
       <screen>
        @or @and bob dylan @set Result-1

        @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"

        @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
       </screen>
       The last of these examples is a spatial search: in
       <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
              >the GILS attribute set</ulink>,
       access point
       2038 indicates West Bounding Coordinate and
       2030 indicates East Bounding Coordinate,
       so the query is for areas extending from -114 degrees
       to no more than -109 degrees.
      </para>
     </example>
    </sect3>
   </sect2>
   <sect2 id="CCL"><title>CCL</title>
    <para>
     Not all users enjoy typing in prefix query structures and numerical
     attribute values, even in a minimalistic test client. In the library
     world, the more intuitive Common Command Language - CCL (ISO 8777)
     has enjoyed some popularity - especially before the widespread
     availability of graphical interfaces. It is still useful in
     applications where you for some reason or other need to provide a
     symbolic language for expressing boolean query structures.
    </para>
    <sect3 id="ccl.syntax">
     <title>CCL Syntax</title>
     <para>
      The CCL parser obeys the following grammar for the FIND argument.
      The syntax is annotated by in the lines prefixed by
      <literal>--</literal>.
     </para>
     <screen>
      CCL-Find ::= CCL-Find Op Elements
                | Elements.

      Op ::= "and" | "or" | "not"
      -- The above means that Elements are separated by boolean operators.

      Elements ::= '(' CCL-Find ')'
                | Set
                | Terms
                | Qualifiers Relation Terms
                | Qualifiers Relation '(' CCL-Find ')'
                | Qualifiers '=' string '-' string
      -- Elements is either a recursive definition, a result set reference, a
      -- list of terms, qualifiers followed by terms, qualifiers followed
      -- by a recursive definition or qualifiers in a range (lower - upper).

      Set ::= 'set' = string
      -- Reference to a result set

      Terms ::= Terms Prox Term
             | Term
      -- Proximity of terms.

      Term ::= Term string
            | string
      -- This basically means that a term may include a blank

      Qualifiers ::= Qualifiers ',' string
                  | string
      -- Qualifiers is a list of strings separated by comma

      Relation ::= '=' | '>=' | '&lt;=' | '&lt;>' | '>' | '&lt;'
      -- Relational operators. This really doesn't follow the ISO8777
      -- standard.

      Prox ::= '%' | '!'
      -- Proximity operator

     </screen>
     <example id="example.ccl.queries">
      <title>CCL queries</title>
      <para>
       The following queries are all valid:
      </para>
      <screen>
       dylan

       "bob dylan"

       dylan or zimmerman

       set=1

       (dylan and bob) or set=1

       righttrunc?

       "notrunc?"

       singlechar#mask
      </screen>
      <para>
       Assuming that the qualifiers <literal>ti</literal>,
       <literal>au</literal>
       and <literal>date</literal> are defined we may use:
      </para>
      <screen>
       ti=self portrait

       au=(bob dylan and slow train coming)

       date>1980 and (ti=((self portrait)))
      </screen>
     </example>
    </sect3>
    <sect3 id="ccl.qualifiers">
     <title>CCL Qualifiers</title>
     <para>
      Qualifiers are used to direct the search to a particular searchable
      index, such as title (ti) and author indexes (au). The CCL standard
      itself doesn't specify a particular set of qualifiers, but it does
      suggest a few short-hand notations. You can customize the CCL parser
      to support a particular set of qualifiers to reflect the current target
      profile. Traditionally, a qualifier would map to a particular
      use-attribute within the BIB-1 attribute set. It is also
      possible to set other attributes, such as the structure
      attribute.
     </para>
     <para>
      A  CCL profile is a set of predefined CCL qualifiers that may be
      read from a file or set in the CCL API.
      The YAZ client reads its CCL qualifiers from a file named
      <filename>default.bib</filename>. There are four types of
      lines in a CCL profile: qualifier specification,
      qualifier alias, comments and directives.
     </para>
     <sect4 id="ccl.qualifier.specification">
      <title>Qualifier specification</title>
      <para>
       A qualifier specification is of the form:
      </para>
      <para>
       <replaceable>qualifier-name</replaceable>
       [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
       [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
      </para>
      <para>
       where <replaceable>qualifier-name</replaceable> is the name of the
       qualifier to be used (eg. <literal>ti</literal>),
       <replaceable>type</replaceable> is attribute type in the attribute
       set (Bib-1 is used if no attribute set is given) and
       <replaceable>val</replaceable> is attribute value.
       The <replaceable>type</replaceable> can be specified as an
       integer or as it be specified either as a single-letter:
       <literal>u</literal> for use,
       <literal>r</literal> for relation,<literal>p</literal> for position,
       <literal>s</literal> for structure,<literal>t</literal> for truncation
       or <literal>c</literal> for completeness.
       The attributes for the special qualifier name <literal>term</literal>
       are used when no CCL qualifier is given in a query.
       <table id="ccl.common.bib1.attributes">
        <title>Common Bib-1 attributes</title>
        <tgroup cols="2">
         <colspec colwidth="2*" colname="type"></colspec>
         <colspec colwidth="9*" colname="description"></colspec>
         <thead>
          <row>
           <entry>Type</entry>
           <entry>Description</entry>
          </row>
         </thead>
         <tbody>
          <row>
           <entry><literal>u=</literal><replaceable>value</replaceable></entry>
           <entry>
            Use attribute (1). Common use attributes are
            1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
            62 Subject, 1003 Author), 1016 Any. Specify value
            as an integer.
           </entry>
          </row>
          <row>
           <entry><literal>r=</literal><replaceable>value</replaceable></entry>
           <entry>
            Relation attribute (2). Common values are
            1 &lt;, 2 &lt;=, 3 =, 4 &gt;=, 5 &gt;, 6 &lt;&gt;,
            100 phonetic, 101 stem, 102 relevance, 103 always matches.
           </entry>
          </row>
          <row>
           <entry><literal>p=</literal><replaceable>value</replaceable></entry>
           <entry>
            Position attribute (3). Values: 1 first in field, 2
            first in any subfield, 3 any position in field.
           </entry>
          </row>
          <row>
           <entry><literal>s=</literal><replaceable>value</replaceable></entry>
           <entry>
            Structure attribute (4). Values: 1 phrase, 2 word,
            3 key, 4 year, 5 date, 6 word list, 100 date (un),
            101 name (norm), 102 name (un), 103 structure, 104 urx,
            105 free-form-text, 106 document-text, 107 local-number,
            108 string, 109 numeric string.
           </entry>
          </row>
          <row>
           <entry><literal>t=</literal><replaceable>value</replaceable></entry>
           <entry>
            Truncation attribute (5). Values: 1 right, 2 left,
            3 left&amp; right, 100 none, 101 process #, 102 regular-1,
            103 regular-2, 104 CCL.
           </entry>
          </row>
          <row>
           <entry><literal>c=</literal><replaceable>value</replaceable></entry>
           <entry>
            Completeness attribute (6). Values: 1 incomplete subfield,
            2 complete subfield, 3 complete field.
           </entry>
          </row>
         </tbody>
        </tgroup>
       </table>
      </para>
      <para>
       Refer to <xref linkend="bib1"/> or the complete
       <ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
      </para>
      <para>
       It is also possible to specify non-numeric attribute values,
       which are used in combination with certain types.
       The special combinations are:
       <table id="ccl.special.attribute.combos">
        <title>Special attribute combos</title>
        <tgroup cols="2">
         <colspec colwidth="2*" colname="name"></colspec>
         <colspec colwidth="9*" colname="description"></colspec>
         <thead>
          <row>
           <entry>Name</entry>
           <entry>Description</entry>
          </row>
         </thead>
         <tbody>
          <row>
           <entry><literal>s=pw</literal></entry>
           <entry>
            The structure is set to either word or phrase depending
            on the number of tokens in a term (phrase-word).
           </entry>
          </row>
          <row>
           <entry><literal>s=al</literal></entry>
           <entry>
            Each token in the term is ANDed. (and-list).
            This does not set the structure at all.
           </entry>
          </row>
          <row><entry><literal>s=ol</literal></entry>
          <entry>
           Each token in the term is ORed. (or-list).
           This does not set the structure at all.
          </entry>
          </row>
          <row><entry><literal>s=ag</literal></entry>
          <entry>
           Tokens that appears as phrases (with blank in them) gets
           structure phrase attached (4=1). Tokens that appear to be words
           gets structure word attached (4=2). Phrases and words are
           ANDed. This is a variant of s=al and s=pw, with the main
           difference that words are not split (with operator AND)
           but instead kept in one RPN token. This facility appeared
           in YAZ 4.2.38.
          </entry>
          </row>
          <row><entry><literal>s=sl</literal></entry>
          <entry>
           Tokens are split into sub-phrases of all combinations - in order.
           This facility appeared in YAZ 5.14.0.
          </entry>
          </row>
          <row><entry><literal>r=o</literal></entry>
          <entry>
           Allows ranges and the operators greather-than, less-than, ...
           equals.
           This sets Bib-1 relation attribute accordingly (relation
           ordered). A query construct is only treated as a range if
           dash is used and that is surrounded by white-space. So
           <literal>-1980</literal> is treated as term
           <literal>"-1980"</literal> not <literal>&lt;= 1980</literal>.
           If <literal>- 1980</literal> is used, however, that is
           treated as a range.
          </entry>
          </row>
          <row><entry><literal>r=r</literal></entry>
          <entry>
           Similar to <literal>r=o</literal> but assumes that terms
           are non-negative (not prefixed with <literal>-</literal>).
           Thus, a dash will always be treated as a range.
           The construct <literal>1980-1990</literal> is
           treated as a range with <literal>r=r</literal> but as a
           single term <literal>"1980-1990"</literal> with
           <literal>r=o</literal>. The special attribute
           <literal>r=r</literal> is available in YAZ 2.0.24 or later.
          </entry>
          </row>
          <row><entry><literal>r=omiteq</literal></entry>
          <entry>
           This will omit relation=equals (@attr 2=3) when r=o / r=r
           is used. This is useful for servers that somehow breaks
           when an explicit relation=equals is used. Omitting the
           relation is usually safe because "equals" is the default
           behavior. This tweak was added in YAZ version 5.1.2.
          </entry>
          </row>
          <row><entry><literal>t=l</literal></entry>
          <entry>
           Allows term to be left-truncated.
           If term is of the form <literal>?x</literal>, the resulting
           Type-1 term is <literal>x</literal> and truncation is left.
          </entry>
          </row>
          <row><entry><literal>t=r</literal></entry>
          <entry>
           Allows term to be right-truncated.
           If term is of the form <literal>x?</literal>, the resulting
           Type-1 term is <literal>x</literal> and truncation is right.
          </entry>
          </row>
          <row><entry><literal>t=n</literal></entry>
          <entry>
           If term is does not include <literal>?</literal>, the
           truncation attribute is set to none (100).
          </entry>
          </row>
          <row><entry><literal>t=b</literal></entry>
          <entry>
           Allows term to be both left&amp;right truncated.
           If term is of the form <literal>?x?</literal>, the
           resulting term is <literal>x</literal> and trunctation is
           set to both left&amp;right.
          </entry>
          </row>
          <row><entry><literal>t=x</literal></entry>
          <entry>
           Allows masking anywhere in a term, thus fully supporting
           # (mask one character) and ? (zero or more of any).
           If masking is used, trunction is set to 102 (regexp-1 in term)
           and the term is converted accordingly to a regular expression.
          </entry>
          </row>
          <row><entry><literal>t=z</literal></entry>
          <entry>
           Allows masking anywhere in a term, thus fully supporting
           # (mask one character) and ? (zero or more of any).
           If masking is used, trunction is set to 104 (Z39.58 in term)
           and the term is converted accordingly to Z39.58 masking term -
           actually the same truncation as CCL itself.
          </entry>
          </row>
         </tbody>
        </tgroup>
       </table>
      </para>
      <example id="example.ccl.profile">
       <title>CCL profile</title>
       <para>
        Consider the following definition:
       </para>
       <screen>
        ti       u=4 s=1
        au       u=1 s=1
        term     s=105
        ranked   r=102
        date     u=30 r=o
       </screen>
       <para>
        <literal>ti</literal> and <literal>au</literal> both set
        structure attribute to phrase (s=1).
        <literal>ti</literal>
        sets the use-attribute to 4. <literal>au</literal> sets the
        use-attribute to 1.
        When no qualifiers are used in the query the structure-attribute is
        set to free-form-text (105) (rule for <literal>term</literal>).
        The <literal>date</literal> sets the relation attribute to
        the relation used in the CCL query and sets the use attribute
        to 30 (Bib-1 Date).
       </para>
       <para>
        You can combine attributes. To Search for "ranked title" you
        can do
        <screen>
         ti,ranked=knuth computer
        </screen>
        which will set relation=ranked, use=title, structure=phrase.
       </para>
       <para>
        Query
        <screen>
         date > 1980
        </screen>
        is a valid query. But
        <screen>
         ti > 1980
        </screen>
        is invalid.
       </para>
      </example>
     </sect4>
     <sect4 id="ccl.qualifier.alias">
      <title>Qualifier alias</title>
      <para>
       A qualifier alias is of the form:
      </para>
      <para>
       <replaceable>q</replaceable>
       <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
      </para>
      <para>
       which declares <replaceable>q</replaceable> to
       be an alias for <replaceable>q1</replaceable>,
       <replaceable>q2</replaceable>... such that the CCL
       query <replaceable>q=x</replaceable> is equivalent to
       <replaceable>q1=x or q2=x or ...</replaceable>.
      </para>
     </sect4>
     <sect4 id="ccl.comments">
      <title>Comments</title>
      <para>
       Lines with white space or lines that begin with
       character <literal>#</literal> are treated as comments.
      </para>
     </sect4>
     <sect4 id="ccl.directives">
      <title>Directives</title>
      <para>
       Directive specifications takes the form
      </para>
      <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
      </para>
      <table id="ccl.directives.table">
       <title>CCL directives</title>
       <tgroup cols="3">
        <colspec colwidth="2*" colname="name"></colspec>
        <colspec colwidth="8*" colname="description"></colspec>
        <colspec colwidth="1*" colname="default"></colspec>
        <thead>
         <row>
          <entry>Name</entry>
          <entry>Description</entry>
          <entry>Default</entry>
         </row>
        </thead>
        <tbody>
         <row>
          <entry>truncation</entry>
          <entry>Truncation character</entry>
          <entry><literal>?</literal></entry>
         </row>
         <row>
          <entry>mask</entry>
          <entry>Masking character. Requires YAZ 4.2.58 or later</entry>
          <entry><literal>#</literal></entry>
         </row>
         <row>
          <entry>field</entry>
          <entry>Specifies how multiple fields are to be
           combined. There are two modes: <literal>or</literal>:
           multiple qualifier fields are ORed,
           <literal>merge</literal>: attributes for the qualifier
           fields are merged and assigned to one term.
           </entry>
          <entry><literal>merge</literal></entry>
         </row>
         <row>
          <entry>case</entry>
          <entry>Specifies if CCL operators and qualifiers should be
           compared with case sensitivity or not. Specify 1 for
           case sensitive; 0 for case insensitive.</entry>
          <entry><literal>1</literal></entry>
         </row>
         <row>
          <entry>and</entry>
          <entry>Specifies token for CCL operator AND.</entry>
          <entry><literal>and</literal></entry>
         </row>
         <row>
          <entry>or</entry>
          <entry>Specifies token for CCL operator OR.</entry>
          <entry><literal>or</literal></entry>
         </row>
         <row>
          <entry>not</entry>
          <entry>Specifies token for CCL operator NOT.</entry>
          <entry><literal>not</literal></entry>
         </row>
         <row>
          <entry>set</entry>
          <entry>Specifies token for CCL operator SET.</entry>
          <entry><literal>set</literal></entry>
         </row>
        </tbody>
       </tgroup>
      </table>
     </sect4>
    </sect3>
    <sect3 id="ccl.api">
     <title>CCL API</title>
     <para>
      All public definitions can be found in the header file
      <filename>ccl.h</filename>. A profile identifier is of type
      <literal>CCL_bibset</literal>. A profile must be created with the call
      to the function <function>ccl_qual_mk</function> which returns a profile
      handle of type <literal>CCL_bibset</literal>.
     </para>
     <para>
      To read a file containing qualifier definitions the function
      <function>ccl_qual_file</function> may be convenient. This function
      takes an already opened <literal>FILE</literal> handle pointer as
      argument along with a <literal>CCL_bibset</literal> handle.
     </para>
     <para>
      To parse a simple string with a FIND query use the function
     </para>
     <screen>
struct ccl_rpn_node *ccl_find_str(CCL_bibset bibset, const char *str,
                                  int *error, int *pos);
     </screen>
     <para>
      which takes the CCL profile (<literal>bibset</literal>) and query
      (<literal>str</literal>) as input. Upon successful completion the RPN
      tree is returned. If an error occur, such as a syntax error, the integer
      pointed to by <literal>error</literal> holds the error code and
      <literal>pos</literal> holds the offset inside query string in which
      the parsing failed.
     </para>
     <para>
      An English representation of the error may be obtained by calling
      the <literal>ccl_err_msg</literal> function. The error codes are
      listed in <filename>ccl.h</filename>.
     </para>
     <para>
      To convert the CCL RPN tree (type
      <literal>struct ccl_rpn_node *</literal>)
      to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
      must be used. This function which is part of YAZ is implemented in
      <filename>yaz-ccl.c</filename>.
      After calling this function the CCL RPN tree is probably no longer
      needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
     </para>
     <para>
      A CCL profile may be destroyed by calling the
      <function>ccl_qual_rm</function> function.
     </para>
     <para>
      The token names for the CCL operators may be changed by setting the
      globals (all type <literal>char *</literal>)
      <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
      <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
      An operator may have aliases, i.e. there may be more than one name for
      the operator. To do this, separate each alias with a space character.
     </para>
    </sect3>
   </sect2>
   <sect2 id="cql">
    <title>CQL</title>
    <para>
     <ulink url="&url.cql;">CQL</ulink>
     - Common Query Language - was defined for the
     <ulink url="&url.sru;">SRU</ulink> protocol.
     In many ways CQL has a similar syntax to CCL.
     The objective of CQL is different. Where CCL aims to be
     an end-user language, CQL is <emphasis>the</emphasis> protocol
     query language for SRU.
    </para>
    <tip>
     <para>
      If you are new to CQL, read the
      <ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
     </para>
    </tip>
    <para>
     The CQL parser in &yaz; provides the following:
     <itemizedlist>
      <listitem>
       <para>
        It parses and validates a CQL query.
       </para>
      </listitem>
      <listitem>
       <para>
        It generates a C structure that allows you to convert
        a CQL query to some other query language, such as SQL.
       </para>
      </listitem>
      <listitem>
       <para>
        The parser converts a valid CQL query to PQF, thus providing a
        way to use CQL for both SRU servers and Z39.50 targets at the
        same time.
       </para>
      </listitem>
      <listitem>
       <para>
        The parser converts CQL to XCQL.
        XCQL is an XML representation of CQL.
        XCQL is part of the SRU specification. However, since SRU
        supports CQL only, we don't expect XCQL to be widely used.
        Furthermore, CQL has the advantage over XCQL that it is
        easy to read.
       </para>
      </listitem>
     </itemizedlist>
    </para>
    <sect3 id="cql.parsing">
     <title>CQL parsing</title>
     <para>
      A CQL parser is represented by the <literal>CQL_parser</literal>
      handle. Its contents should be considered &yaz; internal (private).
      <synopsis>
#include &lt;yaz/cql.h&gt;

typedef struct cql_parser *CQL_parser;

CQL_parser cql_parser_create(void);
void cql_parser_destroy(CQL_parser cp);
      </synopsis>
     A parser is created by <function>cql_parser_create</function> and
     is destroyed by <function>cql_parser_destroy</function>.
     </para>
     <para>
      To parse a CQL query string, the following function
      is provided:
      <synopsis>
int cql_parser_string(CQL_parser cp, const char *str);
      </synopsis>
      A CQL query is parsed by the <function>cql_parser_string</function>
      which takes a query <parameter>str</parameter>.
      If the query was valid (no syntax errors), then zero is returned;
      otherwise -1 is returned to indicate a syntax error.
     </para>
     <para>
      <synopsis>
int cql_parser_stream(CQL_parser cp,
                      int (*getbyte)(void *client_data),
                      void (*ungetbyte)(int b, void *client_data),
                      void *client_data);

int cql_parser_stdio(CQL_parser cp, FILE *f);
      </synopsis>
      The functions <function>cql_parser_stream</function> and
      <function>cql_parser_stdio</function> parses a CQL query
      - just like <function>cql_parser_string</function>.
      The only difference is that the CQL query can be
      fed to the parser in different ways.
      The <function>cql_parser_stream</function> uses a generic
      byte stream as input. The <function>cql_parser_stdio</function>
      uses a <literal>FILE</literal> handle which is opened for reading.
     </para>
    </sect3>
    <sect3 id="cql.tree">
     <title>CQL tree</title>
     <para>
      The the query string is valid, the CQL parser
      generates a tree representing the structure of the
      CQL query.
     </para>
     <para>
      <synopsis>
struct cql_node *cql_parser_result(CQL_parser cp);
      </synopsis>
      <function>cql_parser_result</function> returns the
      a pointer to the root node of the resulting tree.
     </para>
     <para>
      Each node in a CQL tree is represented by a
      <literal>struct cql_node</literal>.
      It is defined as follows:
      <synopsis>
#define CQL_NODE_ST 1
#define CQL_NODE_BOOL 2
#define CQL_NODE_SORT 3
struct cql_node {
    int which;
    union {
        struct {
            char *index;
            char *index_uri;
            char *term;
            char *relation;
            char *relation_uri;
            struct cql_node *modifiers;
        } st;
        struct {
            char *value;
            struct cql_node *left;
            struct cql_node *right;
            struct cql_node *modifiers;
        } boolean;
        struct {
            char *index;
            struct cql_node *next;
            struct cql_node *modifiers;
            struct cql_node *search;
        } sort;
    } u;
};
      </synopsis>
      There are three node types: search term (ST), boolean (BOOL)
      and sortby (SORT).
      A modifier is treated as a search term too.
     </para>
     <para>
      The search term node has five members:
      <itemizedlist>
       <listitem>
        <para>
         <literal>index</literal>: index for search term.
         If an index is unspecified for a search term,
         <literal>index</literal> will be NULL.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>index_uri</literal>: index URi for search term
         or NULL if none could be resolved for the index.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>term</literal>: the search term itself.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>relation</literal>: relation for search term.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>relation_uri</literal>: relation URI for search term.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>modifiers</literal>: relation modifiers for search
         term. The <literal>modifiers</literal> list itself of cql_nodes
         each of type <literal>ST</literal>.
        </para>
       </listitem>
      </itemizedlist>
     </para>
     <para>
      The boolean node represents <literal>and</literal>,
      <literal>or</literal>, <literal>not</literal> +
      proximity.
      <itemizedlist>
       <listitem>
        <para>
         <literal>left</literal> and <literal>right</literal>: left
         - and right operand respectively.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>modifiers</literal>: proximity arguments.
        </para>
       </listitem>
      </itemizedlist>
     </para>
     <para>
      The sort node represents both the SORTBY clause.
     </para>
    </sect3>
    <sect3 id="cql.to.pqf">
     <title>CQL to PQF conversion</title>
     <para>
      Conversion to PQF (and Z39.50 RPN) is tricky by the fact
      that the resulting RPN depends on the Z39.50 target
      capabilities (combinations of supported attributes).
      In addition, the CQL and SRU operates on index prefixes
      (URI or strings), whereas the RPN uses Object Identifiers
      for attribute sets.
     </para>
     <para>
      The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
      type. It represents a particular mapping between CQL and RPN.
      This handle is created and destroyed by the functions:
     <synopsis>
cql_transform_t cql_transform_open_FILE (FILE *f);
cql_transform_t cql_transform_open_fname(const char *fname);
void cql_transform_close(cql_transform_t ct);
     </synopsis>
     The first two functions create a tranformation handle from
     either an already open FILE or from a filename respectively.
     </para>
     <para>
      The handle is destroyed by <function>cql_transform_close</function>
      in which case no further reference of the handle is allowed.
     </para>
     <para>
      When a <literal>cql_transform_t</literal> handle has been created
      you can convert to RPN.
      <synopsis>
int cql_transform_buf(cql_transform_t ct,
                      struct cql_node *cn, char *out, int max);
      </synopsis>
      This function converts the CQL tree <literal>cn</literal>
      using handle <literal>ct</literal>.
      For the resulting PQF, you supply a buffer <literal>out</literal>
      which must be able to hold at at least <literal>max</literal>
      characters.
     </para>
     <para>
      If conversion failed, <function>cql_transform_buf</function>
      returns a non-zero SRU error code; otherwise zero is returned
      (conversion successful).  The meanings of the numeric error
      codes are listed in the SRU specification somewhere (no
      direct link anymore).
     </para>
     <para>
      If conversion fails, more information can be obtained by calling
      <synopsis>
int cql_transform_error(cql_transform_t ct, char **addinfop);
      </synopsis>
      This function returns the most recently returned numeric
      error-code and sets the string-pointer at
      <literal>*addinfop</literal> to point to a string containing
      additional information about the error that occurred: for
      example, if the error code is 15 (``Illegal or unsupported context
      set''), the additional information is the name of the requested
      context set that was not recognised.
     </para>
     <para>
      The SRU error-codes may be translated into brief human-readable
      error messages using
      <synopsis>
const char *cql_strerror(int code);
      </synopsis>
     </para>
     <para>
      If you wish to be able to produce a PQF result in a different
      way, there are two alternatives.
      <synopsis>
void cql_transform_pr(cql_transform_t ct,
                      struct cql_node *cn,
                      void (*pr)(const char *buf, void *client_data),
                      void *client_data);

int cql_transform_FILE(cql_transform_t ct,
                       struct cql_node *cn, FILE *f);
      </synopsis>
      The former function produces output to a user-defined
      output stream. The latter writes the result to an already
      open <literal>FILE</literal>.
     </para>
    </sect3>
    <sect3 id="cql.to.rpn">
     <title>Specification of CQL to RPN mappings</title>
     <para>
      The file supplied to functions
      <function>cql_transform_open_FILE</function>,
      <function>cql_transform_open_fname</function> follows
      a structure found in many Unix utilities.
      It consists of mapping specifications - one per line.
      Lines starting with <literal>#</literal> are ignored (comments).
     </para>
     <para>
      Each line is of the form
      <literallayout>
       <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
      </literallayout>
     </para>
     <para>
      An RPN pattern is a simple attribute list. Each attribute pair
      takes the form:
      <literallayout>
       [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
      </literallayout>
      The attribute <replaceable>set</replaceable> is optional.
      The <replaceable>type</replaceable> is the attribute type,
      <replaceable>value</replaceable> the attribute value.
     </para>
     <para>
      The character <literal>*</literal> (asterisk) has special meaning
      when used in the RPN pattern.
      Each occurrence of <literal>*</literal> is substituted with the
      CQL matching name (index, relation, qualifier etc).
      This facility can be used to copy a CQL name verbatim to the RPN result.
     </para>
     <para>
      The following CQL patterns are recognized:
      <variablelist>
       <varlistentry>
        <term>
         <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
        </term>
        <listitem>
         <para>
          This pattern is invoked when a CQL index, such as
          dc.title is converted. <replaceable>set</replaceable>
          and <replaceable>name</replaceable> are the context set and index
          name respectively.
          Typically, the RPN specifies an equivalent use attribute.
         </para>
         <para>
          For terms not bound by an index the pattern
          <literal>index.cql.serverChoice</literal> is used.
          Here, the prefix <literal>cql</literal> is defined as
          <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
          If this pattern is not defined, the mapping will fail.
         </para>
         <para>
          The pattern,
          <literal>index.</literal><replaceable>set</replaceable><literal>.*</literal>
          is used when no other index pattern is matched.
         </para>
        </listitem>
       </varlistentry>
       <varlistentry>
        <term>
         <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
         (DEPRECATED)
        </term>
        <listitem>
         <para>
          For backwards compatibility, this is recognised as a synonym of
          <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
         </para>
        </listitem>
       </varlistentry>
       <varlistentry>
        <term>
         <literal>relation.</literal><replaceable>relation</replaceable>
        </term>
        <listitem>
         <para>
          This pattern specifies how a CQL relation is mapped to RPN.
          <replaceable>pattern</replaceable> is name of relation
          operator. Since <literal>=</literal> is used as
          separator between CQL pattern and RPN, CQL relations
          including <literal>=</literal> cannot be
          used directly. To avoid a conflict, the names
          <literal>ge</literal>,
          <literal>eq</literal>,
          <literal>le</literal>,
          must be used for CQL operators, greater-than-or-equal,
          equal, less-than-or-equal respectively.
          The RPN pattern is supposed to include a relation attribute.
         </para>
         <para>
          For terms not bound by a relation, the pattern
          <literal>relation.scr</literal> is used. If the pattern
          is not defined, the mapping will fail.
         </para>
         <para>
          The special pattern, <literal>relation.*</literal> is used
          when no other relation pattern is matched.
         </para>
        </listitem>
       </varlistentry>
       <varlistentry>
        <term>
         <literal>relationModifier.</literal><replaceable>mod</replaceable>
        </term>
        <listitem>
         <para>
          This pattern specifies how a CQL relation modifier is mapped to RPN.
          The RPN pattern is usually a relation attribute.
         </para>
        </listitem>
       </varlistentry>
       <varlistentry>
        <term>
         <literal>structure.</literal><replaceable>type</replaceable>
        </term>
        <listitem>
         <para>
          This pattern specifies how a CQL structure is mapped to RPN.
          Note that this CQL pattern is somewhat to similar to
          CQL pattern <literal>relation</literal>.
          The <replaceable>type</replaceable> is a CQL relation.
         </para>
         <para>
          The pattern, <literal>structure.*</literal> is used
          when no other structure pattern is matched.
          Usually, the RPN equivalent specifies a structure attribute.
         </para>
        </listitem>
       </varlistentry>
       <varlistentry>
        <term>
         <literal>position.</literal><replaceable>type</replaceable>
        </term>
        <listitem>
         <para>
          This pattern specifies how the anchor (position) of
          CQL is mapped to RPN.
          The <replaceable>type</replaceable> is one
          of <literal>first</literal>, <literal>any</literal>,
          <literal>last</literal>, <literal>firstAndLast</literal>.
         </para>
         <para>
          The pattern, <literal>position.*</literal> is used
          when no other position pattern is matched.
         </para>
        </listitem>
       </varlistentry>
       <varlistentry>
        <term>
         <literal>set.</literal><replaceable>prefix</replaceable>
        </term>
        <listitem>
         <para>
          This specification defines a CQL context set for a given prefix.
          The value on the right hand side is the URI for the set -
          <emphasis>not</emphasis> RPN. All prefixes used in
          index patterns must be defined this way.
         </para>
        </listitem>
       </varlistentry>
       <varlistentry>
        <term>
         <literal>set</literal>
        </term>
        <listitem>
         <para>
          This specification defines a default CQL context set for index names.
          The value on the right hand side is the URI for the set.
         </para>
        </listitem>
       </varlistentry>
      </variablelist>
     </para>
     <example id="example.cql.to.rpn.mapping">
      <title>CQL to RPN mapping file</title>
      <para>
       This simple file defines two context sets, three indexes and three
       relations, a position pattern and a default structure.
      </para>
      <programlisting><![CDATA[
       set.cql  = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
       set.dc   = http://www.loc.gov/zing/cql/dc-indexes/v1.0/

       index.cql.serverChoice = 1=1016
       index.dc.title         = 1=4
       index.dc.subject       = 1=21

       relation.<             = 2=1
       relation.eq            = 2=3
       relation.scr           = 2=3

       position.any           = 3=3 6=1

       structure.*            = 4=1
]]>
      </programlisting>
      <para>
       With the mappings above, the CQL query
       <screen>
        computer
       </screen>
       is converted to the PQF:
       <screen>
        @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
       </screen>
       by rules <literal>index.cql.serverChoice</literal>,
       <literal>relation.scr</literal>, <literal>structure.*</literal>,
       <literal>position.any</literal>.
      </para>
      <para>
       CQL query
       <screen>
        computer^
       </screen>
       is rejected, since <literal>position.right</literal> is
       undefined.
      </para>
      <para>
       CQL query
       <screen>
        >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
       </screen>
       is converted to
       <screen>
        @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
       </screen>
      </para>
     </example>
     <example id="example.cql.to.rpn.string">
      <title>CQL to RPN string attributes</title>
      <para>
       In this example we allow any index to be passed to RPN as
       a use attribute.
      </para>
      <programlisting><![CDATA[
       # Identifiers for prefixes used in this file. (index.*)
       set.cql  = info:srw/cql-context-set/1/cql-v1.1
       set.rpn  = http://bogus/rpn
       set      = http://bogus/rpn

       # The default index when none is specified by the query
       index.cql.serverChoice     = 1=any

       index.rpn.*                = 1=*
       relation.eq                = 2=3
       structure.*                = 4=1
       position.any               = 3=3
]]>
      </programlisting>
      <para>
       The <literal>http://bogus/rpn</literal> context set is also the default
       so we can make queries such as
       <screen>
        title = a
       </screen>
       which is converted to
       <screen>
        @attr 2=3 @attr 4=1 @attr 3=3 @attr 1=title "a"
       </screen>
      </para>
     </example>
     <example id="example.cql.to.rpn.bathprofile">
      <title>CQL to RPN using Bath Profile</title>
      <para>
       The file <filename>etc/pqf.properties</filename> has mappings from
       the Bath Profile and Dublin Core to RPN.
       If YAZ is installed as a package it's usually located
       in <filename>/usr/share/yaz/etc</filename> and part of the
       development package, such as <literal>libyaz-dev</literal>.
      </para>
     </example>
    </sect3>
    <sect3 id="cql.xcql">
     <title>CQL to XCQL conversion</title>
     <para>
      Conversion from CQL to XCQL is trivial and does not
      require a mapping to be defined.
      There three functions to choose from depending on the
      way you wish to store the resulting output (XML buffer
      containing XCQL).
      <synopsis>
int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
void cql_to_xml(struct cql_node *cn,
                void (*pr)(const char *buf, void *client_data),
                void *client_data);
void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
      </synopsis>
      Function <function>cql_to_xml_buf</function> converts
      to XCQL and stores result in a user supplied buffer of a given
      max size.
     </para>
     <para>
      <function>cql_to_xml</function> writes the result in
      a user defined output stream.
      <function>cql_to_xml_stdio</function> writes to a
      a file.
     </para>
    </sect3>
    <sect3 id="rpn.to.cql">
     <title>PQF to CQL conversion</title>
     <para>
      Conversion from PQF to CQL is offered by the two functions shown
      below. The former uses a generic stream for result. The latter
      puts result in a WRBUF (string container).
      <synopsis>
#include &lt;yaz/rpn2cql.h>

int cql_transform_rpn2cql_stream(cql_transform_t ct,
                                 void (*pr)(const char *buf, void *client_data),
                                 void *client_data,
                                 Z_RPNQuery *q);

int cql_transform_rpn2cql_wrbuf(cql_transform_t ct,
                                WRBUF w,
                                Z_RPNQuery *q);
      </synopsis>
      The configuration is the same as used in CQL to PQF conversions.
     </para>
    </sect3>
   </sect2>
  </sect1>
  <sect1 id="tools.oid">
   <title>Object Identifiers</title>
   <para>
    The basic YAZ representation of an OID is an array of integers,
    terminated with the value -1. This integer is of type
    <literal>Odr_oid</literal>.
   </para>
   <para>
    Fundamental OID operations and the type <literal>Odr_oid</literal>
    are defined in <filename>yaz/oid_util.h</filename>.
   </para>
   <para>
    An OID can either be declared as a automatic variable or it can
    allocated using the memory utilities or ODR/NMEM. It's
    guaranteed that an OID can fit in <literal>OID_SIZE</literal> integers.
   </para>
   <example id="tools.oid.bib1.1"><title>Create OID on stack</title>
    <para>
     We can create an OID for the Bib-1 attribute set with:
     <screen>
      Odr_oid bib1[OID_SIZE];
      bib1[0] = 1;
      bib1[1] = 2;
      bib1[2] = 840;
      bib1[3] = 10003;
      bib1[4] = 3;
      bib1[5] = 1;
      bib1[6] = -1;
     </screen>
    </para>
   </example>
   <para>
    And OID may also be filled from a string-based representation using
    dots (.). This is achieved by function
    <screen>
     int oid_dotstring_to_oid(const char *name, Odr_oid *oid);
    </screen>
    This functions returns 0 if name could be converted; -1 otherwise.
   </para>
   <example id="tools.oid.bib1.2"><title>Using oid_oiddotstring_to_oid</title>
    <para>
     We can fill the Bib-1 attribute set OID easier with:
     <screen>
      Odr_oid bib1[OID_SIZE];
      oid_oiddotstring_to_oid("1.2.840.10003.3.1", bib1);
     </screen>
   </para>
   </example>
   <para>
    We can also allocate an OID dynamically on a ODR stream with:
    <screen>
    Odr_oid *odr_getoidbystr(ODR o, const char *str);
    </screen>
    This creates an OID from string-based representation using dots.
    This function take an &odr; stream as parameter. This stream is used to
    allocate memory for the data elements, which is released on a
    subsequent call to <function>odr_reset()</function> on that stream.
   </para>
   <example id="tools.oid.bib1.3">
    <title>Using odr_getoidbystr</title>
    <para>
     We can create a OID for the Bib-1 attribute set with:
     <screen>
      Odr_oid *bib1 = odr_getoidbystr(odr, "1.2.840.10003.3.1");
     </screen>
    </para>
   </example>
   <para>
    The function
    <screen>
     char *oid_oid_to_dotstring(const Odr_oid *oid, char *oidbuf)
    </screen>
    does the reverse of <function>oid_oiddotstring_to_oid</function>. It
    converts an OID to the string-based representation using dots.
    The supplied char buffer <literal>oidbuf</literal> holds the resulting
    string and must be at least <literal>OID_STR_MAX</literal> in size.
   </para>
   <para>
    OIDs can be copied with <function>oid_oidcpy</function> which takes
    two OID lists as arguments. Alternativly, an OID copy can be allocated
    on a ODR stream with:
    <screen>
     Odr_oid *odr_oiddup(ODR odr, const Odr_oid *o);
    </screen>
   </para>
   <para>
    OIDs can be compared with <function>oid_oidcmp</function> which returns
    zero if the two OIDs provided are identical; non-zero otherwise.
   </para>
   <sect2 id="tools.oid.database">
    <title>OID database</title>
    <para>
     From YAZ version 3 and later, the oident system has been replaced
     by an OID database. OID database is a misnomer .. the old odient
     system was also a database.
    </para>
    <para>
     The OID database is really just a map between named Object Identifiers
     (string) and their OID raw equivalents. Most operations either
     convert from string to OID or other way around.
    </para>
    <para>
     Unfortunately, whenever we supply a string we must also specify the
     <emphasis>OID class</emphasis>. The class is necessary because some
     strings correspond to multiple OIDs. An example of such a string is
     <literal>Bib-1</literal> which may either be an attribute-set
     or a diagnostic-set.
    </para>
    <para>
     Applications using the YAZ database should include
     <filename>yaz/oid_db.h</filename>.
    </para>
    <para>
     A YAZ database handle is of type <literal>yaz_oid_db_t</literal>.
     Actually that's a pointer. You need not think deal with that.
     YAZ has a built-in database which can be considered "constant" for
     most purposes.
     We can get hold that by using function <function>yaz_oid_std</function>.
    </para>
    <para>
     All functions with prefix <function>yaz_string_to_oid</function>
     converts from class + string to OID. We have variants of this
     operation due to different memory allocation strategies.
    </para>
    <para>
     All functions with prefix
     <function>yaz_oid_to_string</function> converts from OID to string
     + class.
    </para>
    <example id="tools.oid.bib1.4">
     <title>Create OID with YAZ DB</title>
     <para>
      We can create an OID for the Bib-1 attribute set on the ODR stream
      odr with:
      <screen>
        Odr_oid *bib1 =
        yaz_string_to_oid_odr(yaz_oid_std(), CLASS_ATTSET, "Bib-1", odr);
      </screen>
      This is more complex than using <function>odr_getoidbystr</function>.
      You would only use <function>yaz_string_to_oid_odr</function> when the
      string (here Bib-1) is supplied by a user or configuration.
     </para>
    </example>
   </sect2>
   <sect2 id="tools.oid.std">
    <title>Standard OIDs</title>
    <para>
     All the object identifers in the standard OID database as returned
     by <function>yaz_oid_std</function> can referenced directly in a
     program as a constant OID.
     Each constant OID is prefixed with <literal>yaz_oid_</literal> -
     followed by OID class (lowercase) - then by OID name (normalized and
     lowercase).
    </para>
    <para>
     See <xref linkend="list-oids"/> for list of all object identifiers
     built into YAZ.
     These are declared in <filename>yaz/oid_std.h</filename> but are
     included by <filename>yaz/oid_db.h</filename> as well.
    </para>
    <example id="tools.oid.bib1.5">
     <title>Use a built-in OID</title>
     <para>
      We can allocate our own OID filled with the constant OID for
      Bib-1 with:
      <screen>
       Odr_oid *bib1 = odr_oiddup(o, yaz_oid_attset_bib1);
      </screen>
     </para>
    </example>
   </sect2>
  </sect1>
  <sect1 id="tools.nmem">
   <title>Nibble Memory</title>
   <para>
    Sometimes when you need to allocate and construct a large,
    interconnected complex of structures, it can be a bit of a pain to
    release the associated memory again. For the structures describing the
    Z39.50 PDUs and related structures, it is convenient to use the
    memory-management system of the &odr; subsystem (see
    <xref linkend="odr.use"/>). However, in some circumstances
    where you might otherwise benefit from using a simple nibble memory
    management system, it may be impractical to use
    <function>odr_malloc()</function> and <function>odr_reset()</function>.
    For this purpose, the memory manager which also supports the &odr;
    streams is made available in the NMEM module. The external interface
    to this module is given in the <filename>nmem.h</filename> file.
   </para>
   <para>
    The following prototypes are given:
   </para>
   <screen>
    NMEM nmem_create(void);
    void nmem_destroy(NMEM n);
    void *nmem_malloc(NMEM n, size_t size);
    void nmem_reset(NMEM n);
    size_t nmem_total(NMEM n);
    void nmem_init(void);
    void nmem_exit(void);
   </screen>
   <para>
    The <function>nmem_create()</function> function returns a pointer to a
    memory control handle, which can be released again by
    <function>nmem_destroy()</function> when no longer needed.
    The function <function>nmem_malloc()</function> allocates a block of
    memory of the requested size. A call to <function>nmem_reset()</function>
    or <function>nmem_destroy()</function> will release all memory allocated
    on the handle since it was created (or since the last call to
    <function>nmem_reset()</function>. The function
    <function>nmem_total()</function> returns the number of bytes currently
    allocated on the handle.
   </para>
   <para>
    The nibble memory pool is shared amongst threads. POSIX
    mutex'es and WIN32 Critical sections are introduced to keep the
    module thread safe. Function <function>nmem_init()</function>
    initializes the nibble memory library and it is called automatically
    the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
    function <function>DllMain</function> to achieve this. You should
    <emphasis>not</emphasis> call <function>nmem_init</function> or
    <function>nmem_exit</function> unless you're absolute sure what
    you're doing. Note that in previous &yaz; versions you'd have to call
    <function>nmem_init</function> yourself.
   </para>
  </sect1>
  <sect1 id="tools.log">
   <title>Log</title>
   <para>
    &yaz; has evolved a fairly complex log system which should be useful both
    for debugging &yaz; itself, debugging applications that use &yaz;, and for
    production use of those applications.
   </para>
   <para>
    The log functions are declared in header <filename>yaz/log.h</filename>
    and implemented in <filename>src/log.c</filename>.
    Due to name clash with syslog and some math utilities the logging
    interface has been modified as of YAZ 2.0.29. The obsolete interface
    is still available if in header file <filename>yaz/log.h</filename>.
    The key points of the interface are:
   </para>
   <screen>
    void yaz_log(int level, const char *fmt, ...)
    void yaz_log_init(int level, const char *prefix, const char *name);
    void yaz_log_init_file(const char *fname);
    void yaz_log_init_level(int level);
    void yaz_log_init_prefix(const char *prefix);
    void yaz_log_time_format(const char *fmt);
    void yaz_log_init_max_size(int mx);

    int yaz_log_mask_str(const char *str);
    int yaz_log_module_level(const char *name);
   </screen>
   <para>
    The reason for the whole log module is the <function>yaz_log</function>
    function. It takes a bitmask indicating the log levels, a
    <literal>printf</literal>-like format string, and a variable number of
    arguments to log.
   </para>
   <para>
    The <literal>log level</literal> is a bit mask, that says on which level(s)
    the log entry should be made, and optionally set some behaviour of the
    logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
    YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
    that modify the way the log entry is written:<literal>YLOG_ERRNO,
    YLOG_NOTIME, YLOG_FLUSH</literal>.
    Most of the rest of the bits are deprecated, and should not be used. Use
    the dynamic log levels instead.
   </para>
   <para>
    Applications that use &yaz;, should not use the LOG_LOG for ordinary
    messages, but should make use of the dynamic loglevel system. This consists
    of two parts, defining the loglevel and checking it.
   </para>
   <para>
    To define the log levels, the (main) program should pass a string to
    <function>yaz_log_mask_str</function> to define which log levels are to be
    logged. This string should be a comma-separated list of log level names,
    and can contain both hard-coded names and dynamic ones. The log level
    calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
    for each word it meets, unless the word starts with a '-', in which case it
    clears the bit. If the string <literal>'none'</literal> is found,
    all bits are cleared. Typically this string comes from the command-line,
    often identified by <literal>-v</literal>. The
    <function>yaz_log_mask_str</function> returns a log level that should be
    passed to <function>yaz_log_init_level</function> for it to take effect.
   </para>
   <para>
    Each module should check what log bits it should be used, by calling
    <function>yaz_log_module_level</function> with a suitable name for the
    module. The name is cleared from a preceding path and an extension, if any,
    so it is quite possible to use <literal>__FILE__</literal> for it. If the
    name has been passed to <function>yaz_log_mask_str</function>, the routine
    returns a non-zero bitmask, which should then be used in consequent calls
    to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
    yaz_log, in time-critical places, or when the log entry would take time
    to construct.)
   </para>
   <para>
    Yaz uses the following dynamic log levels:
    <literal>server, session, request, requestdetail</literal> for the server
    functionality.
    <literal>zoom</literal> for the zoom client api.
    <literal>ztest</literal> for the simple test server.
    <literal>malloc, nmem, odr, eventl</literal> for internal
    debugging of yaz itself.
    Of course, any program using yaz is welcome to define as many new
    ones, as it needs.
   </para>
   <para>
    By default the log is written to stderr, but this can be changed by a call
    to <function>yaz_log_init_file</function> or
    <function>yaz_log_init</function>. If the log is directed to a file, the
    file size is checked at every write, and if it exceeds the limit given in
    <function>yaz_log_init_max_size</function>, the log is rotated. The
    rotation keeps one old version (with a <literal>.1</literal> appended to
    the name). The size defaults to 1GB. Setting it to zero will disable the
    rotation feature.
   </para>
   <screen>
    A typical yaz-log looks like this
  13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
  13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
  13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo  OK:7 hits
  13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2  OK 2 records returned
  13:24:13-23/11 yaz-ztest(1) [request] Close OK
   </screen>
   <para>
    The log entries start with a time stamp. This can be omitted by setting the
    <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
    can be hoped to produce identical log files, that are easy to diff. The
    format of the time stamp can be set with
    <function>yaz_log_time_format</function>, which takes a format string just
    like <function>strftime</function>.
   </para>
   <para>
    Next in a log line comes the prefix, often the name of the program. For
    yaz-based servers, it can also contain the session number. Then
    comes one or more logbits in square brackets, depending on the logging
    level set by <function>yaz_log_init_level</function> and the loglevel
    passed to <function>yaz_log_init_level</function>. Finally comes the format
    string and additional values passed to <function>yaz_log</function>
   </para>
   <para>
    The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
    <literal>loglevel</literal>, will log all the log-level affecting
    operations. This can come in handy if you need to know what other log
    levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
   </para>
   <para>
    The log system is almost independent of the rest of &yaz;, the only
    important dependence is of <filename>nmem</filename>, and that only for
    using the semaphore definition there.
   </para>
   <para>
    The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
    the same time, the log bit names were changed from
    <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
    to avoid collision with <filename>syslog.h</filename>.
   </para>
  </sect1>
  <sect1 id="marc">
   <title>MARC</title>
   <para>
    YAZ provides a fast utility for working with MARC records.
    Early versions of the MARC utility only allowed decoding of ISO2709.
    Today the utility may both encode - and decode to a varity of formats.
   </para>
   <synopsis><![CDATA[
    #include <yaz/marcdisp.h>

    /* create handler */
    yaz_marc_t yaz_marc_create(void);
    /* destroy */
    void yaz_marc_destroy(yaz_marc_t mt);

    /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
    void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
    #define YAZ_MARC_LINE      0
    #define YAZ_MARC_SIMPLEXML 1
    #define YAZ_MARC_OAIMARC   2
    #define YAZ_MARC_MARCXML   3
    #define YAZ_MARC_ISO2709   4
    #define YAZ_MARC_XCHANGE   5
    #define YAZ_MARC_CHECK     6
    #define YAZ_MARC_TURBOMARC 7
    #define YAZ_MARC_JSON      8

    /* supply iconv handle for character set conversion .. */
    void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);

    /* set debug level, 0=none, 1=more, 2=even more, .. */
    void yaz_marc_debug(yaz_marc_t mt, int level);

    /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
    On success, result in *result with size *rsize. */
    int yaz_marc_decode_buf(yaz_marc_t mt, const char *buf, int bsize,
                            const char **result, size_t *rsize);

    /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
       On success, result in WRBUF */
    int yaz_marc_decode_wrbuf(yaz_marc_t mt, const char *buf,
                              int bsize, WRBUF wrbuf);
]]>
   </synopsis>
   <note>
    <para>
     The synopsis is just a basic subset of all functionality. Refer
     to the actual header file <filename>marcdisp.h</filename> for
     details.
    </para>
   </note>
   <para>
    A MARC conversion handle must be created by using
    <function>yaz_marc_create</function> and destroyed
    by calling <function>yaz_marc_destroy</function>.
   </para>
   <para>
    All other function operate on a <literal>yaz_marc_t</literal> handle.
    The output is specified by a call to <function>yaz_marc_xml</function>.
    The <literal>xmlmode</literal> must be one of
    <variablelist>
     <varlistentry>
      <term>YAZ_MARC_LINE</term>
      <listitem>
       <para>
        A simple line-by-line format suitable for display but not
        recommend for further (machine) processing.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>YAZ_MARC_MARCXML</term>
      <listitem>
       <para>
        <ulink url="&url.marcxml;">MARCXML</ulink>.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>YAZ_MARC_ISO2709</term>
      <listitem>
       <para>
        ISO2709 (sometimes just referred to as "MARC").
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>YAZ_MARC_XCHANGE</term>
      <listitem>
       <para>
        <ulink url="&url.marcxchange;">MarcXchange</ulink>.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>YAZ_MARC_CHECK</term>
      <listitem>
       <para>
        Pseudo format for validation only. Does not generate
        any real output except diagnostics.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>YAZ_MARC_TURBOMARC</term>
      <listitem>
       <para>
        XML format with same semantics as MARCXML but more compact
        and geared towards fast processing with XSLT. Refer to
        <xref linkend="tools.turbomarc"/> for more information.
       </para>
      </listitem>
     </varlistentry>
     <varlistentry>
      <term>YAZ_MARC_JSON</term>
      <listitem>
       <para>
        <ulink url="&url.marc_in_json;">MARC-in_JSON</ulink> format.
       </para>
      </listitem>
     </varlistentry>
    </variablelist>
   </para>
   <para>
    The actual conversion functions are
    <function>yaz_marc_decode_buf</function> and
    <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
    a MARC record. The former function operates on simple buffers, the
    stores the resulting record in a WRBUF handle (WRBUF is a simple string
    type).
   </para>
   <example id="example.marc.display">
    <title>Display of MARC record</title>
    <para>
     The following program snippet illustrates how the MARC API may
     be used to convert a MARC record to the line-by-line format:
     <programlisting><![CDATA[
      void print_marc(const char *marc_buf, int marc_buf_size)
      {
         char *result;      /* for result buf */
         size_t result_len;    /* for size of result */
         yaz_marc_t mt = yaz_marc_create();
         yaz_marc_xml(mt, YAZ_MARC_LINE);
         yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
                             &result, &result_len);
         fwrite(result, result_len, 1, stdout);
         yaz_marc_destroy(mt);  /* note that result is now freed... */
      }
]]>
     </programlisting>
    </para>
   </example>
   <sect2 id="tools.turbomarc">
    <title>TurboMARC</title>
    <para>
     TurboMARC is yet another XML encoding of a MARC record. The format
     was designed for fast processing with XSLT.
    </para>
    <para>
     Applications like
     Pazpar2 uses XSLT to convert an XML encoded MARC record to an internal
     representation. This conversion mostly check the tag of a MARC field
     to determine the basic rules in the conversion. This check is
     costly when that is tag is encoded as an attribute in MARCXML.
     By having the tag value as the element instead, makes processing
     many times faster (at least for Libxslt).
    </para>
    <para>
     TurboMARC is encoded as follows:
     <itemizedlist>
      <listitem>
       <para>
        Record elements is part of namespace
        "<literal>http://www.indexdata.com/turbomarc</literal>".
      </para>
      </listitem>
      <listitem>
       <para>
        A record is enclosed in element <literal>r</literal>.
      </para>
      </listitem>
      <listitem>
       <para>
        A collection of records is enclosed in element
        <literal>collection</literal>.
       </para>
      </listitem>
      <listitem>
       <para>
        The leader is encoded as element <literal>l</literal> with the
        leader content as its (text) value.
      </para>
      </listitem>
      <listitem>
       <para>
        A control field is encoded as element <literal>c</literal> concatenated
        with the tag value of the control field if the tag value
        matches the regular expression <literal>[a-zA-Z0-9]*</literal>.
        If the tag value do not match the regular expression
        <literal>[a-zA-Z0-9]*</literal> the control field is encoded
        as element <literal>c</literal> and attribute <literal>code</literal>
        will hold the tag value.
        This rule ensure that in the rare cases where a tag value might
        result in a non-wellformed XML YAZ encode it as a coded attribute
        (as in MARCXML).
       </para>
       <para>
        The control field content is the the text value of this element.
        Indicators are encoded as attribute names
        <literal>i1</literal>, <literal>i2</literal>, etc.. and
        corresponding values for each indicator.
       </para>
      </listitem>
      <listitem>
       <para>
        A data field is encoded as element <literal>d</literal> concatenated
        with the tag value of the data field or using the attribute
        <literal>code</literal> as described in the rules for control fields.
        The children of the data field element is subfield elements.
        Each subfield element is encoded as <literal>s</literal>
        concatenated with the sub field code.
        The text of the subfield element is the contents of the subfield.
        Indicators are encoded as attributes for the data field element similar
        to the encoding for control fields.
      </para>
      </listitem>
     </itemizedlist>
    </para>
   </sect2>
  </sect1>
  <sect1 id="tools.retrieval">
   <title>Retrieval Facility</title>
   <para>
    YAZ version 2.1.20 or later includes a Retrieval facility tool
    which allows a SRU/Z39.50 to describe itself and perform record
    conversions. The idea is the following:
    <itemizedlist>
     <listitem>
      <para>
       An SRU/Z39.50 client sends a retrieval request which includes
       a combination of the following parameters: syntax (format),
       schema (or element set name).
      </para>
     </listitem>
     <listitem>
      <para>
       The retrieval facility is invoked with parameters in a
       server/proxy. The retrieval facility matches the parameters a set of
       "supported" retrieval types.
       If there is no match, the retrieval signals an error
       (syntax and / or schema not supported).
      </para>
     </listitem>
     <listitem>
      <para>
       For a successful match, the backend is invoked with the same
       or altered retrieval parameters (syntax, schema). If
       a record is received from the backend, it is converted to the
       frontend name / syntax.
      </para>
     </listitem>
     <listitem>
      <para>
       The resulting record is sent back the client and tagged with
       the frontend syntax / schema.
      </para>
     </listitem>
    </itemizedlist>
   </para>
   <para>
    The Retrieval facility is driven by an XML configuration. The
    configuration is neither Z39.50 ZeeRex or SRU ZeeRex. But it
    should be easy to generate both of them from the XML configuration.
    (unfortunately the two versions
    of ZeeRex differ substantially in this regard).
   </para>
   <sect2 id="tools.retrieval.format">
    <title>Retrieval XML format</title>
    <para>
     All elements should be covered by namespace
     <literal>http://indexdata.com/yaz</literal> .
     The root element node must be <literal>retrievalinfo</literal>.
    </para>
    <para>
     The <literal>retrievalinfo</literal> must include one or
     more <literal>retrieval</literal> elements. Each
    <literal>retrieval</literal> defines specific combination of
     syntax, name and identifier supported by this retrieval service.
    </para>
    <para>
     The <literal>retrieval</literal> element may include any of the
     following attributes:
     <variablelist>
      <varlistentry><term><literal>syntax</literal> (REQUIRED)</term>
       <listitem>
        <para>
         Defines the record syntax. Possible values is any
         of the names defined in YAZ' OID database or a raw
         OID in (n.n ... n).
        </para>
       </listitem>
      </varlistentry>
      <varlistentry><term><literal>name</literal> (OPTIONAL)</term>
       <listitem>
        <para>
         Defines the name of the retrieval format. This can be
         any string. For SRU, the value, is equivalent to schema (short-hand);
         for Z39.50 it's equivalent to simple element set name.
         For YAZ 3.0.24 and later this name may be specified as a glob
         expression with operators
         <literal>*</literal> and <literal>?</literal>.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry><term><literal>identifier</literal> (OPTIONAL)</term>
       <listitem>
        <para>
         Defines the URI schema name of the retrieval format. This can be
         any string. For SRU, the value, is equivalent to URI schema.
         For Z39.50, there is no equivalent.
        </para>
       </listitem>
      </varlistentry>
     </variablelist>
    </para>
    <para>
     The <literal>retrieval</literal> may include one
     <literal>backend</literal> element. If a <literal>backend</literal>
     element is given, it specifies how the records are retrieved by
     some backend and how the records are converted from the backend to
     the "frontend".
    </para>
    <para>
     The attributes, <literal>name</literal> and <literal>syntax</literal>
     may be specified for the <literal>backend</literal> element. These
     semantics of these attributes is equivalent to those for the
     <literal>retrieval</literal>. However, these values are passed to
     the "backend".
    </para>
    <para>
     The <literal>backend</literal> element may includes one or more
     conversion instructions (as children elements). The supported
     conversions are:
     <variablelist>
      <varlistentry><term><literal>marc</literal></term>
       <listitem>
        <para>
         The <literal>marc</literal> element specifies a conversion
         to - and from ISO2709 encoded MARC and
         <ulink url="&url.marcxml;">&acro.marcxml;</ulink>/MarcXchange.
         The following attributes may be specified:
         <variablelist>
          <varlistentry>
           <term><literal>inputformat</literal> (REQUIRED)</term>
           <listitem>
            <para>
             Format of input. Supported values are
             <literal>marc</literal> (for ISO2709), <literal>xml</literal>
             (MARCXML/MarcXchange) and <literal>json</literal>
             (<ulink url="&url.marc_in_json;">MARC-in_JSON</ulink>).
            </para>
           </listitem>
          </varlistentry>
          <varlistentry>
           <term><literal>outputformat</literal> (REQUIRED)</term>
           <listitem>
            <para>
             Format of output. Supported values are
             <literal>line</literal> (MARC line format);
             <literal>marcxml</literal> (for MARCXML),
             <literal>marc</literal> (ISO2709),
             <literal>marcxhcange</literal> (for MarcXchange),
             or <literal>json</literal>
             (<ulink url="&url.marc_in_json;">MARC-in_JSON </ulink>).
            </para>
           </listitem>
          </varlistentry>
          <varlistentry>
           <term><literal>inputcharset</literal> (OPTIONAL)</term>
           <listitem>
            <para>
             Encoding of input. For XML input formats, this need not
             be given, but for ISO2709 based inputformats, this should
             be set to the encoding used. For MARC21 records, a common
             inputcharset value  would be <literal>marc-8</literal>.
            </para>
           </listitem>
          </varlistentry>
          <varlistentry>
           <term><literal>outputcharset</literal> (OPTIONAL)</term>
           <listitem>
            <para>
             Encoding of output. If outputformat is XML based, it is
             strongly recommened to use <literal>utf-8</literal>.
            </para>
           </listitem>
          </varlistentry>
         </variablelist>
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>select</literal></term>
       <listitem>
        <para>
         The <literal>select</literal> selects one or more text nodes
         and decodes them as XML.
         The following attributes may be specified:
         <variablelist>
          <varlistentry><term><literal>path</literal> (REQUIRED)</term>
           <listitem>
            <para>
             X-Path expression for selecting text nodes.
            </para>
           </listitem>
          </varlistentry>
         </variablelist>
        </para>
        <para>
         This conversion is available in YAZ 5.8.0 and later.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>solrmarc</literal></term>
       <listitem>
        <para>
         The <literal>solrmarc</literal> decodes solrmarc records.
         It assumes that the input is pure solrmarc text (no escaping)
         and will convert all sequences of the form #XX; to a single
         character of the hexadecimal value as given by XX. The output,
         presumably, is a valid ISO2709 buffer.
        </para>
        <para>
         This conversion is available in YAZ 5.0.21 and later.
        </para>
       </listitem>
      </varlistentry>
      <varlistentry>
       <term><literal>xslt</literal></term>
       <listitem>
        <para>
         The <literal>xslt</literal> element specifies a conversion
         via &acro.xslt;. The following attributes may be specified:
         <variablelist>
          <varlistentry><term><literal>stylesheet</literal> (REQUIRED)</term>
           <listitem>
            <para>
             Stylesheet file.
            </para>
           </listitem>
          </varlistentry>
         </variablelist>
        </para>
       </listitem>
      </varlistentry>
     </variablelist>
    </para>
   </sect2>
   <sect2 id="tools.retrieval.examples">
    <title>Retrieval Facility Examples</title>
    <example id="tools.retrieval.marc21">
     <title>MARC21 backend</title>
     <para>
      A typical way to use the retrieval facility is to enable XML
      for servers that only supports ISO2709 encoded MARC21 records.
     </para>
     <programlisting><![CDATA[
     <retrievalinfo>
       <retrieval syntax="usmarc" name="F"/>
       <retrieval syntax="usmarc" name="B"/>
       <retrieval syntax="xml" name="marcxml"
                  identifier="info:srw/schema/1/marcxml-v1.1">
         <backend syntax="usmarc" name="F">
           <marc inputformat="marc" outputformat="marcxml"
                 inputcharset="marc-8"/>
         </backend>
       </retrieval>
       <retrieval syntax="xml" name="dc">
         <backend syntax="usmarc" name="F">
           <marc inputformat="marc" outputformat="marcxml"
                 inputcharset="marc-8"/>
           <xslt stylesheet="MARC21slim2DC.xsl"/>
         </backend>
       </retrieval>
     </retrievalinfo>
]]>
     </programlisting>
     <para>
      This means that our frontend supports:
      <itemizedlist>
       <listitem>
        <para>
         MARC21 F(ull) records.
        </para>
       </listitem>
       <listitem>
        <para>
         MARC21 B(rief) records.
        </para>
       </listitem>
       <listitem>
        <para>
         MARCXML records.
        </para>
       </listitem>
       <listitem>
        <para>
         Dublin core records.
        </para>
       </listitem>
      </itemizedlist>
     </para>
    </example>
    <example id="tools.retrieval.marcxml">
     <title>MARCXML backend</title>
     <para>
      SRW/SRU and Solr backends returns records in XML.
      If they return MARCXML or MarcXchange, the retrieval module
      can convert those into ISO2709 formats, most commonly USMARC
      (AKA MARC21).
      In this example, the backend returns MARCXML for schema="marcxml".
     </para>
     <programlisting><![CDATA[
     <retrievalinfo>
       <retrieval syntax="usmarc">
         <backend syntax="xml" name="marcxml">
           <marc inputformat="xml" outputformat="marc"
                 outputcharset="marc-8"/>
         </backend>
       </retrieval>
       <retrieval syntax="xml" name="marcxml"
                  identifier="info:srw/schema/1/marcxml-v1.1"/>
       <retrieval syntax="xml" name="dc">
         <backend syntax="xml" name="marcxml">
           <xslt stylesheet="MARC21slim2DC.xsl"/>
         </backend>
       </retrieval>
     </retrievalinfo>
]]>
     </programlisting>
     <para>
      This means that our frontend supports:
      <itemizedlist>
       <listitem>
        <para>
         MARC21 records (any element set name) in MARC-8 encoding.
        </para>
       </listitem>
       <listitem>
        <para>
         MARCXML records for element-set=marcxml
        </para>
       </listitem>
       <listitem>
        <para>
         Dublin core records for element-set=dc.
        </para>
       </listitem>
      </itemizedlist>
     </para>
    </example>
   </sect2>
   <sect2 id="tools.retrieval.api">
    <title>API</title>
    <para>
     It should be easy to use the retrieval systems from applications. Refer
     to the headers
     <filename>yaz/retrieval.h</filename> and
     <filename>yaz/record_conv.h</filename>.
    </para>
   </sect2>
  </sect1>
  <sect1 id="sorting">
   <title>Sorting</title>
   <para>
    This chapter describes sorting and how it is supported in YAZ.
    Sorting applies to a result-set.
    The
    <ulink url="http://www.loc.gov/z3950/agency/markup/05.html#3.2.7">
     Z39.50 sorting facility
    </ulink>
    takes one or more input result-sets
    and one result-set as output. The most simple case is that
    the input-set is the same as the output-set.
   </para>
   <para>
    Z39.50 sorting has a separate APDU (service) that is, thus, performed
    following a search (two phases).
   </para>
   <para>
    In SRU/Solr, however, the model is different. Here, sorting is specified
    during the the search operation. Note, however, that SRU might
    perform sort as separate search, by referring to an existing result-set
    in the query (result-set reference).
   </para>
   <sect2>
    <title>Using the Z39.50 sort service</title>
    <para>
     yaz-client and the ZOOM API supports the Z39.50 sort facility. In any
     case the sort sequence or sort critiera is using a string notation.
     This notation is a one-line notation suitable for being manually
     entered or generated and allows for easy logging (one liner).
     For the ZOOM API, the sort is specified in the call to ZOOM_query_sortby
     function. For yaz-client the sort is performed and specified using
     the sort and sort+ commands. For description of the sort criteria notation
     refer to the <link linkend="sortspec">sort command</link> in the
     yaz-client manual.
    </para>
    <para>
     The ZOOM API might choose one of several sort strategies for
     sorting. Refer to <xref linkend="zoom-sort-strategy"/>.
    </para>
   </sect2>
   <sect2>
    <title>Type-7 sort</title>
    <para>
     Type-7 sort is an extension to the Bib-1 based RPN query where the
     sort specification is embedded as an Attribute-Plus-Term.
    </para>
    <para>
     The objectives for introducing Type-7 sorting is that it allows
     a client to perform sorting even if it does not implement/support
     Z39.50 sort. Virtually all Z39.50 client software supports
     RPN queries. It also may improve performance because the sort
     critieria is specified along with the search query.
    </para>
    <para>
     The sort is triggered by the presence of type 7 and the value of type 7
     specifies the
     <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortKeySpec">
      sortRelation
     </ulink>
     The value for type 7 is 1 for ascending and 2 for descending.
     For the
     <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortElement">
      sortElement
     </ulink>
     only the generic part is handled. If generic sortKey is of type
     sortField, then attribute type 1 is present and the value is
     sortField (InternationalString). If generic sortKey is of type
     sortAttributes, then the attributes in list is used . generic sortKey
     of type elementSpec is not supported.
    </para>
    <para>
     The term in the sorting Attribute-Plus-Term combo should hold
     an integer. The value is 0 for primary sorting criteria, 1 for second
     criteria, etc.
    </para>
   </sect2>
  </sect1>
  <sect1 id="facets">
   <title>Facets</title>
   <para>
    YAZ supports facets for in Solr, SRU 2.0 and Z39.50 protocols.
   </para>
   <para>
    Like Type-1/RPN, YAZ supports a string notation for specifying
    facets. For the API this is performed by
    <function>yaz_pqf_parse_facet_list</function>.
   </para>
   <para>
    For ZOOM C the facets are given by option "facets"
    For yaz-client it is used for the facets command.
   </para>
   <para>
    The grammar of this specification is as follows:
    <literallayout>
   facet-spec ::= facet-list

   facet-list ::= facet-list ',' attr-spec | attr-spec

   attr-spec ::= attr-spec '@attr' string | '@attr' string

    </literallayout>
    The notation is inspired by PQF. The string following '@attr'
    may not include blanks and is of the form
    <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>,
    where <replaceable>type</replaceable> is an integer and
    <replaceable>value</replaceable> is a string or an integer.
   </para>
   <para>
    The Facets specification is not Bib-1. The following types apply:
   </para>
   <table id="facet.attributes">
    <title>Facet attributes</title>
    <tgroup cols="2">
     <colspec colwidth="2*" colname="type"></colspec>
     <colspec colwidth="9*" colname="description"></colspec>
     <thead>
      <row>
       <entry>Type</entry>
       <entry>Description</entry>
      </row>
     </thead>
     <tbody>
      <row>
       <entry>1</entry>
       <entry>
        Field-name. This is often a string, eg "Author", "Year", etc.
       </entry>
      </row>
      <row>
       <entry>2</entry>
       <entry>
        Sort order. Value should be an integer.
        Value 0: count descending (frequency). Value 1: alpha ascending.
       </entry>
      </row>
      <row>
       <entry>3</entry>
       <entry>
        Number of terms requested.
       </entry>
      </row>
      <row>
       <entry>4</entry>
       <entry>
        Start offset.
       </entry>
      </row>
     </tbody>
    </tgroup>
   </table>
  </sect1>
 </chapter>
 <chapter id="odr">
  <title>The ODR Module</title>
  <sect1 id="odr.introduction">
   <title>Introduction</title>
   <para>
    &odr; is the BER-encoding/decoding subsystem of &yaz;. Care as been taken
    to isolate &odr; from the rest of the package - specifically from the
    transport interface. &odr; may be used in any context where basic
    ASN.1/BER representations are used.
   </para>
   <para>
    If you are only interested in writing a Z39.50 implementation based on
    the PDUs that are already provided with &yaz;, you only need to concern
    yourself with the section on managing ODR streams
    (<xref linkend="odr.use"/>). Only if you need to
    implement ASN.1 beyond that which has been provided, should you
    worry about the second half of the documentation
    (<xref linkend="odr.programming"/>).
    If you use one of the higher-level interfaces, you can skip this
    section entirely.
   </para>
   <para>
    This is important, so we'll repeat it for emphasis: <emphasis>You do
    not need to read <xref linkend="odr.programming"/>
    to implement Z39.50 with &yaz;.</emphasis>
   </para>
   <para>
    If you need a part of the protocol that isn't already in &yaz;, you
    should contact the authors before going to work on it yourself: We
    might already be working on it. Conversely, if you implement a useful
    part of the protocol before us, we'd be happy to include it in a
    future release.
   </para>
  </sect1>
  <sect1 id="odr.use">
   <title>Using ODR</title>
   <sect2 id="odr.streams">
    <title>ODR Streams</title>
    <para>
     Conceptually, the ODR stream is the source of encoded data in the
     decoding mode; when encoding, it is the receptacle for the encoded
     data. Before you can use an ODR stream it must be allocated. This is
     done with the function
    </para>
    <synopsis>
     ODR odr_createmem(int direction);
    </synopsis>
    <para>
     The <function>odr_createmem()</function> function takes as argument one
     of three manifest constants: <literal>ODR_ENCODE</literal>,
     <literal>ODR_DECODE</literal>, or <literal>ODR_PRINT</literal>.
     An &odr; stream can be in only one mode - it is not possible to change
     its mode once it's selected. Typically, your program will allocate
     at least two ODR streams - one for decoding, and one for encoding.
    </para>
    <para>
     When you're done with the stream, you can use
    </para>
    <synopsis>
     void odr_destroy(ODR o);
    </synopsis>
    <para>
     to release the resources allocated for the stream.
    </para>
   </sect2>
   <sect2 id="odr.memory.management">
    <title id="memory">Memory Management</title>
    <para>
     Two forms of memory management take place in the &odr; system. The first
     one, which has to do with allocating little bits of memory (sometimes
     quite large bits of memory, actually) when a protocol package is
     decoded, and turned into a complex of interlinked structures. This
     section deals with this system, and how you can use it for your own
     purposes. The next section deals with the memory management which is
     required when encoding data - to make sure that a large enough buffer is
     available to hold the fully encoded PDU.
    </para>
    <para>
     The &odr; module has its own memory management system, which is
     used whenever memory is required. Specifically, it is used to allocate
     space for data when decoding incoming PDUs. You can use the memory
     system for your own purposes, by using the function
    </para>
    <synopsis>
     void *odr_malloc(ODR o, size_t size);
    </synopsis>
    <para>
     You can't use the normal <function>free(2)</function> routine to free
     memory allocated by this function, and &odr; doesn't provide a parallel
     function. Instead, you can call
    </para>
    <synopsis>
     void odr_reset(ODR o);
    </synopsis>
    <para>
     when you are done with the
     memory: Everything allocated since the last call to
     <function>odr_reset()</function> is released.
     The <function>odr_reset()</function> call is also required to clear
     up an error condition on a stream.
    </para>
    <para>
     The function
    </para>
    <synopsis>
     size_t odr_total(ODR o);
    </synopsis>
    <para>
     returns the number of bytes allocated on the stream since the last call to
     <function>odr_reset()</function>.
    </para>
    <para>
     The memory subsystem of &odr; is fairly efficient at allocating and
     releasing little bits of memory. Rather than managing the individual,
     small bits of space, the system maintains a free-list of larger chunks
     of memory, which are handed out in small bits. This scheme is
     generally known as a <emphasis>nibble memory</emphasis> system.
     It is very useful for maintaining short-lived constructions such
     as protocol PDUs.
    </para>
    <para>
     If you want to retain a bit of memory beyond the next call to
     <function>odr_reset()</function>, you can use the function
    </para>
    <synopsis>
     ODR_MEM odr_extract_mem(ODR o);
    </synopsis>
    <para>
     This function will give you control of the memory recently allocated
     on the ODR stream. The memory will live (past calls to
     <function>odr_reset()</function>), until you call the function
    </para>
    <synopsis>
     void odr_release_mem(ODR_MEM p);
    </synopsis>
    <para>
     The opaque <literal>ODR_MEM</literal> handle has no other purpose than
     referencing the memory block for you until you want to release it.
    </para>
    <para>
     You can use <function>odr_extract_mem()</function> repeatedly between
     allocating data, to retain individual control of separate chunks of data.
    </para>
   </sect2>
   <sect2 id="odr.encoding.and.decoding">
    <title>Encoding and Decoding Data</title>
    <para>
     When encoding data, the ODR stream will write the encoded octet string
     in an internal buffer. To retrieve the data, use the function
    </para>
    <synopsis>
     char *odr_getbuf(ODR o, int *len, int *size);
    </synopsis>
    <para>
     The integer pointed to by len is set to the length of the encoded
     data, and a pointer to that data is returned. <literal>*size</literal>
     is set to the size of the buffer (unless <literal>size</literal> is null,
     signaling that you are not interested in the size). The next call to
     a primitive function using the same &odr; stream will overwrite the
     data, unless a different buffer has been supplied using the call
    </para>
    <synopsis>
     void odr_setbuf(ODR o, char *buf, int len, int can_grow);
    </synopsis>
    <para>
     which sets the encoding (or decoding) buffer used by
     <literal>o</literal> to <literal>buf</literal>, using the length
     <literal>len</literal>.
     Before a call to an encoding function, you can use
     <function>odr_setbuf()</function> to provide the stream with an encoding
     buffer of sufficient size (length). The <literal>can_grow</literal>
     parameter tells the encoding &odr; stream whether it is allowed to use
     <function>realloc(2)</function> to increase the size of the buffer when
     necessary. The default condition of a new encoding stream is equivalent
     to the results of calling
    </para>
    <synopsis>
     odr_setbuf(stream, 0, 0, 1);
    </synopsis>
    <para>
     In this case, the stream will allocate and reallocate memory as
     necessary. The stream reallocates memory by repeatedly doubling the
     size of the buffer - the result is that the buffer will typically
     reach its maximum, working size with only a small number of reallocation
     operations. The memory is freed by the stream when the latter is destroyed,
     unless it was assigned by the user with the <literal>can_grow</literal>
     parameter set to zero (in this case, you are expected to retain
     control of the memory yourself).
    </para>
    <para>
     To assume full control of an encoded buffer, you must first call
     <function>odr_getbuf()</function> to fetch the buffer and its length.
     Next, you should call <function>odr_setbuf()</function> to provide a
     different buffer (or a null pointer) to the stream. In the simplest
     case, you will reuse the same buffer over and over again, and you
     will just need to call <function>odr_getbuf()</function> after each
     encoding operation to get the length and address of the buffer.
     Note that the stream may reallocate the buffer during an encoding
     operation, so it is necessary to retrieve the correct address after
     each encoding operation.
    </para>
    <para>
     It is important to realize that the ODR stream will not release this
     memory when you call <function>odr_reset()</function>: It will
     merely update its internal pointers to prepare for the encoding of a
     new data value.
     When the stream is released by the <function>odr_destroy()</function>
     function, the memory given to it by <function>odr_setbuf</function> will
     be released <emphasis>only</emphasis> if the <literal>can_grow</literal>
     parameter to <function>odr_setbuf()</function> was nonzero. The
     <literal>can_grow</literal> parameter, in other words, is a way of
     signaling who is to own the buffer, you or the ODR stream. If you never call
     <function>odr_setbuf()</function> on your encoding stream, which is
     typically the case, the buffer allocated by the stream will belong to
     the stream by default.
    </para>
    <para>
     When you wish to decode data, you should first call
     <function>odr_setbuf()</function>, to tell the decoding stream
     where to find the encoded data, and how long the buffer is
     (the <literal>can_grow</literal> parameter is ignored by a decoding
     stream). After this, you can call the function corresponding to the
     data you wish to decode (eg, <function>odr_integer()</function> odr
     <function>z_APDU()</function>).
    </para>
    <example id="example.odr.encoding.and.decoding.functions">
     <title>Encoding and decoding functions</title>
     <synopsis>
      int odr_integer(ODR o, Odr_int **p, int optional, const char *name);

      int z_APDU(ODR o, Z_APDU **p, int optional, const char *name);
     </synopsis>
    </example>
    <para>
     If the data is absent (or doesn't match the tag corresponding to
     the type), the return value will be either 0 or 1 depending on the
     <literal>optional</literal> flag. If <literal>optional</literal>
     is 0 and the data is absent, an error flag will be raised in the
     stream, and you'll need to call <function>odr_reset()</function> before
     you can use the stream again. If <literal>optional</literal> is
     nonzero, the pointer <emphasis>pointed</emphasis> to/ by
     <literal>p</literal> will be set to the null value, and the function
     will return 1.
     The <literal>name</literal> argument is used to pretty-print the
     tag in question. It may be set to <literal>NULL</literal> if
     pretty-printing is not desired.
    </para>
    <para>
     If the data value is found where it's expected, the pointer
     <emphasis>pointed to</emphasis> by the <literal>p</literal> argument
     will be set to point to the decoded type.
     The space for the type will be allocated and owned by the &odr;
     stream, and it will live until you call
     <function>odr_reset()</function> on the stream. You cannot use
     <function>free(2)</function> to release the memory.
     You can decode several data elements (by repeated calls to
     <function>odr_setbuf()</function> and your decoding function), and
     new memory will be allocated each time. When you do call
     <function>odr_reset()</function>, everything decoded since the
     last call to <function>odr_reset()</function> will be released.
    </para>
    <example id="example.odr.encoding.of.integer">
     <title>Encoding and decoding of an integer</title>
     <para>
      The use of the double indirection can be a little confusing at first
      (its purpose will become clear later on, hopefully),
      so an example is in order. We'll encode an integer value, and
      immediately decode it again using a different stream. A useless, but
      informative operation.
     </para>
     <programlisting><![CDATA[
void do_nothing_useful(Odr_int value)
{
    ODR encode, decode;
    Odr_int *valp, *resvalp;
    char *bufferp;
    int len;

    /* allocate streams */
    if (!(encode = odr_createmem(ODR_ENCODE)))
        return;
    if (!(decode = odr_createmem(ODR_DECODE)))
        return;

    valp = &value;
    if (odr_integer(encode, &valp, 0, 0) == 0)
    {
        printf("encoding went bad\n");
        return;
    }
    bufferp = odr_getbuf(encode, &len, 0);
    printf("length of encoded data is %d\n", len);

    /* now let's decode the thing again */
    odr_setbuf(decode, bufferp, len, 0);
    if (odr_integer(decode, &resvalp, 0, 0) == 0)
    {
        printf("decoding went bad\n");
        return;
    }
    /* ODR_INT_PRINTF format for printf (such as %d) */
    printf("the value is " ODR_INT_PRINTF "\n", *resvalp);

    /* clean up */
    odr_destroy(encode);
    odr_destroy(decode);
}
]]>
     </programlisting>
     <para>
      This looks like a lot of work, offhand. In practice, the &odr; streams
      will typically be allocated once, in the beginning of your program
      (or at the beginning of a new network session), and the encoding
      and decoding will only take place in a few, isolated places in your
      program, so the overhead is quite manageable.
     </para>
    </example>
   </sect2>
   <sect2 id="odr.printing">
    <title>Printing</title>
    <para>
     When an ODR stream is created of type <literal>ODR_PRINT</literal>
     the ODR module will print the contents of a PDU in a readable format.
     By default output is written to the <literal>stderr</literal> stream.
     This behavior can be changed, however, by calling the function
     <synopsis>
      odr_setprint(ODR o, FILE *file);
     </synopsis>
     before encoders or decoders are being invoked.
     It is also possible to direct the output to a buffer (of indeed
     another file), by using the more generic mechanism:
     <synopsis>
      void odr_set_stream(ODR o, void *handle,
                         void (*stream_write)(ODR o, void *handle, int type,
                                              const char *buf, int len),
                         void (*stream_close)(void *handle));
     </synopsis>
     Here the user provides an opaque handle and two handlers,
     <replaceable>stream_write</replaceable> for writing,
     and <replaceable>stream_close</replaceable> which is supposed
     to close/free resources associated with handle.
     The <replaceable>stream_close</replaceable> handler is optional and
     if NULL for the function is provided, it will not be invoked.
     The <replaceable>stream_write</replaceable> takes the ODR handle
     as parameter, the user defined handle, a type
     <literal>ODR_OCTETSTRING</literal>, <literal>ODR_VISIBLESTRING</literal>
     which indicates the type of contents is being written.
    </para>
    <para>
     Another utility useful for diagnostics (error handling) or as
     part of the printing facilities is:
     <synopsis>
      const char **odr_get_element_path(ODR o);
     </synopsis>
     which returns a list of current elements that ODR deals with at the
     moment. For the returned array, say <literal>ar</literal>,
     <literal>ar[0]</literal> is the top level element,
     <literal>ar[n]</literal> is the last. The last element has the
     property that <literal>ar[n+1] == NULL</literal>.
    </para>
    <example id="example.odr.element.path.record">
     <title>Element Path for record</title>
     <para>
      For a database record part of a PresentResponse the
      array returned by <function>odr_get_element</function>
      is <literal>presentResponse</literal>, <literal>databaseOrSurDiagnostics</literal>, <literal>?</literal>, <literal>record</literal>, <literal>?</literal>, <literal>databaseRecord</literal> . The question mark appears due to
      unnamed constructions.
     </para>
    </example>
   </sect2>
   <sect2 id="odr.diagnostics">
    <title>Diagnostics</title>
    <para>
     The encoding/decoding functions all return 0 when an error occurs.
     Until you call <function>odr_reset()</function>, you cannot use the
     stream again, and any function called will immediately return 0.
    </para>
    <para>
     To provide information to the programmer or administrator, the function
    </para>
    <synopsis>
     void odr_perror(ODR o, char *message);
    </synopsis>
    <para>
     is provided, which prints the <literal>message</literal> argument to
     <literal>stderr</literal> along with an error message from the stream.
    </para>
    <para>
     You can also use the function
    </para>
    <synopsis>
     int odr_geterror(ODR o);
    </synopsis>
    <para>
     to get the current error number from the screen. The number will be
     one of these constants:
    </para>
    <table frame="top" id="odr.error.codes">
     <title>ODR Error codes</title>
     <tgroup cols="2">
      <thead>
       <row>
        <entry>code</entry>
        <entry>Description</entry>
       </row>
      </thead>
      <tbody>
       <row>
        <entry>OMEMORY</entry><entry>Memory allocation failed.</entry>
       </row>
       <row>
        <entry>OSYSERR</entry><entry>A system- or library call has failed.
         The standard diagnostic variable <literal>errno</literal> should be
         examined to determine the actual error.</entry>
       </row>
       <row>
        <entry>OSPACE</entry><entry>No more space for encoding.
         This will only occur when the user has explicitly provided a
         buffer for an encoding stream without allowing the system to
         allocate more space.</entry>
       </row>
       <row>
        <entry>OREQUIRED</entry><entry>This is a common protocol error; A
         required data element was missing during encoding or decoding.</entry>
       </row>
       <row>
        <entry>OUNEXPECTED</entry><entry>An unexpected data element was
        found during decoding.</entry>
       </row>
       <row>
        <entry>OOTHER</entry><entry>Other error. This is typically an
        indication of misuse of the &odr; system by the programmer, and also
        that the diagnostic system isn't as good as it should be, yet.</entry>
       </row>
      </tbody>
     </tgroup>
    </table>
    <para>
     The character string array
    </para>
    <synopsis>
     char *odr_errlist[]
    </synopsis>
    <para>
     can be indexed by the error code to obtain a human-readable
     representation of the problem.
    </para>
   </sect2>
   <sect2 id="odr.summary.and.synopsis">
    <title>Summary and Synopsis</title>
    <synopsis>
     #include &lt;yaz/odr.h>

     ODR odr_createmem(int direction);

     void odr_destroy(ODR o);

     void odr_reset(ODR o);

     char *odr_getbuf(ODR o, int *len, int *size);

     void odr_setbuf(ODR o, char *buf, int len, int can_grow);

     void *odr_malloc(ODR o, int size);

     NMEM odr_extract_mem(ODR o);

     int odr_geterror(ODR o);

     void odr_perror(ODR o, const char *message);

     extern char *odr_errlist[];
    </synopsis>
   </sect2>
  </sect1>
  <sect1 id="odr.programming">
   <title>Programming with ODR</title>
   <para>
    The API of &odr; is designed to reflect the structure of ASN.1, rather
    than BER itself. Future releases may be able to represent data in
    other external forms.
   </para>
   <tip>
    <para>
     There is an ASN.1 tutorial available at
     <ulink url="&url.asn.1.tutorial;">this site</ulink>.
     This site also has standards for ASN.1 (X.680) and BER (X.690)
     <ulink url="&url.asn.1.standards;">online</ulink>.
    </para>
   </tip>
   <para>
    The ODR interface is based loosely on that of the Sun Microsystems
    XDR routines.
    Specifically, each function which corresponds to an ASN.1 primitive
    type has a dual function. Depending on the settings of the ODR
    stream which is supplied as a parameter, the function may be used
    either to encode or decode data. The functions that can be built
    using these primitive functions, to represent more complex data types,
    share this quality. The result is that you only have to enter the
    definition for a type once - and you have the functionality of encoding,
    decoding (and pretty-printing) all in one unit.
    The resulting C source code is quite compact, and is a pretty
    straightforward representation of the source ASN.1 specification.
   </para>
   <para>
    In many cases, the model of the XDR functions works quite well in this
    role.
    In others, it is less elegant. Most of the hassle comes from the optional
    SEQUENCE members which don't exist in XDR.
   </para>
   <sect2 id="odr.primitive.asn1.types">
    <title>The Primitive ASN.1 Types</title>
    <para>
     ASN.1 defines a number of primitive types (many of which correspond
     roughly to primitive types in structured programming languages, such as C).
    </para>
    <sect3 id="odr.integer">
     <title>INTEGER</title>
     <para>
      The &odr; function for encoding or decoding (or printing) the ASN.1
      INTEGER type looks like this:
     </para>
     <synopsis>
      int odr_integer(ODR o, Odr_int **p, int optional, const char *name);
     </synopsis>
     <para>
      The <literal>Odr_int</literal> is just a simple integer.
     </para>
     <para>
      This form is typical of the primitive &odr; functions. They are named
      after the type of data that they encode or decode. They take an &odr;
      stream, an indirect reference to the type in question, and an
      <literal>optional</literal> flag (corresponding to the OPTIONAL keyword
      of ASN.1) as parameters. They all return an integer value of either one
      or zero.
      When you use the primitive functions to construct encoders for complex
      types of your own, you should follow this model as well. This
      ensures that your new types can be reused as elements in yet more
      complex types.
     </para>
     <para>
      The <literal>o</literal> parameter should obviously refer to a properly
      initialized &odr; stream of the right type (encoding/decoding/printing)
      for the operation that you wish to perform.
     </para>
     <para>
      When encoding or printing, the function first looks at
      <literal>* p</literal>. If <literal>* p</literal> (the pointer pointed
      to by <literal>p</literal>) is a null pointer, this is taken to mean that
      the data element is absent. If the <literal>optional</literal> parameter
      is nonzero, the function will return one (signifying success) without
      any further processing. If the <literal>optional</literal> is zero, an
      internal error flag is set in the &odr; stream, and the function will
      return 0. No further operations can be carried out on the stream without
      a call to the function <function>odr_reset()</function>.
     </para>
     <para>
      If <literal>*p</literal> is not a null pointer, it is expected to
      point to an instance of the data type. The data will be subjected to
      the encoding rules, and the result will be placed in the buffer held
      by the &odr; stream.
     </para>
     <para>
      The other ASN.1 primitives have similar functions that operate in
      similar manners:
     </para>
    </sect3>
    <sect3 id="odr.boolean">
     <title>BOOLEAN</title>
     <synopsis>
int odr_bool(ODR o, Odr_bool **p, int optional, const char *name);
     </synopsis>
    </sect3>
    <sect3 id="odr.real">
     <title>REAL</title>
     <para>
      Not defined.
     </para>
    </sect3>
    <sect3 id="odr.null">
     <title>NULL</title>
     <synopsis>
int odr_null(ODR o, Odr_null **p, int optional, const char *name);
     </synopsis>
     <para>
      In this case, the value of **p is not important. If <literal>*p</literal>
      is different from the null pointer, the null value is present, otherwise
      it's absent.
     </para>
    </sect3>
    <sect3 id="odr.octet.string">
     <title>OCTET STRING</title>
     <synopsis>
typedef struct odr_oct
{
    unsigned char *buf;
    int len;
} Odr_oct;

int odr_octetstring(ODR o, Odr_oct **p, int optional,
                    const char *name);
     </synopsis>
     <para>
      The <literal>buf</literal> field should point to the character array
      that holds the octetstring. The <literal>len</literal> field holds the
      actual length.
      The character array need not be null terminated.
     </para>
     <para>
      To make things a little easier, an alternative is given for string
      types that are not expected to contain embedded NULL characters (eg.
      VisibleString):
     </para>
     <synopsis>
      int odr_cstring(ODR o, char **p, int optional, const char *name);
     </synopsis>
     <para>
      Which encoded or decodes between OCTETSTRING representations and
      null-terminates C strings.
     </para>
     <para>
      Functions are provided for the derived string types, eg:
     </para>
     <synopsis>
int odr_visiblestring(ODR o, char **p, int optional,
                      const char *name);
     </synopsis>
    </sect3>
    <sect3 id="odr.bit.string">
     <title>BIT STRING</title>
     <synopsis>
int odr_bitstring(ODR o, Odr_bitmask **p, int optional,
                  const char *name);
     </synopsis>
     <para>
      The opaque type <literal>Odr_bitmask</literal> is only suitable for
      holding relatively brief bit strings, eg. for options fields, etc.
      The constant <literal>ODR_BITMASK_SIZE</literal> multiplied by 8
      gives the maximum possible number of bits.
     </para>
     <para>
      A set of macros are provided for manipulating the
      <literal>Odr_bitmask</literal> type:
     </para>
     <synopsis>
void ODR_MASK_ZERO(Odr_bitmask *b);

void ODR_MASK_SET(Odr_bitmask *b, int bitno);

void ODR_MASK_CLEAR(Odr_bitmask *b, int bitno);

int ODR_MASK_GET(Odr_bitmask *b, int bitno);
     </synopsis>
     <para>
      The functions are modeled after the manipulation functions that
      accompany the <literal>fd_set</literal> type used by the
      <function>select(2)</function> call.
      <literal>ODR_MASK_ZERO</literal> should always be called first on a
      new bitmask, to initialize the bits to zero.
     </para>
    </sect3>
    <sect3 id="odr.object.identifier">
     <title>OBJECT IDENTIFIER</title>
     <synopsis>
int odr_oid(ODR o, Odr_oid **p, int optional, const char *name);
     </synopsis>
     <para>
      The C OID representation is simply an array of integers, terminated by
      the value -1 (the <literal>Odr_oid</literal> type is synonymous with
      the <literal>short</literal> type).
      We suggest that you use the OID database module (see
      <xref linkend="tools.oid.database"/>) to handle object identifiers
      in your application.
     </para>
    </sect3>
   </sect2>
   <sect2 id="odr.tagging.primitive.types">
    <title>Tagging Primitive Types</title>
    <para>
     The simplest way of tagging a type is to use the
     <function>odr_implicit_tag()</function> or
     <function>odr_explicit_tag()</function> macros:
    </para>
    <synopsis>
int odr_implicit_tag(ODR o, Odr_fun fun, int class, int tag,
                     int optional, const char *name);

int odr_explicit_tag(ODR o, Odr_fun fun, int class, int tag,
                     int optional, const char *name);
    </synopsis>
    <para>
     To create a type derived from the integer type by implicit tagging, you
     might write:
    </para>
    <screen>
     MyInt ::= [210] IMPLICIT INTEGER
    </screen>
    <para>
     In the &odr; system, this would be written like:
    </para>
    <screen>
int myInt(ODR o, Odr_int **p, int optional, const char *name)
{
    return odr_implicit_tag(o, odr_integer, p,
                            ODR_CONTEXT, 210, optional, name);
}
    </screen>
    <para>
     The function <function>myInt()</function> can then be used like any of
     the primitive functions provided by &odr;. Note that the behavior of
     <function>odr_explicit_tag()</function>
     and <function>odr_implicit_tag()</function> macros
     act exactly the same as the functions they are applied to - they
     respond to error conditions, etc, in the same manner - they
     simply have three extra parameters. The class parameter may
     take one of the values: <literal>ODR_CONTEXT</literal>,
     <literal>ODR_PRIVATE</literal>, <literal>ODR_UNIVERSAL</literal>, or
     <literal>/ODR_APPLICATION</literal>.
    </para>
   </sect2>
   <sect2 id="odr.constructed.types">
    <title>Constructed Types</title>
    <para>
     Constructed types are created by combining primitive types. The
      &odr; system only implements the SEQUENCE and SEQUENCE OF constructions
     (although adding the rest of the container types should be simple
     enough, if the need arises).
    </para>
    <para>
     For implementing SEQUENCEs, the functions
    </para>
    <synopsis>
int odr_sequence_begin(ODR o, void *p, int size, const char *name);
int odr_sequence_end(ODR o);
    </synopsis>
    <para>
     are provided.
    </para>
    <para>
     The <function>odr_sequence_begin()</function> function should be
     called in the beginning of a function that implements a SEQUENCE type.
     Its parameters are the &odr; stream, a pointer (to a pointer to the type
     you're implementing), and the <literal>size</literal> of the type
     (typically a C structure). On encoding, it returns 1 if
     <literal>* p</literal> is a null pointer. The <literal>size</literal>
     parameter is ignored. On decoding, it returns 1 if the type is found in
     the data stream. <literal>size</literal> bytes of memory are allocated,
     and <literal>*p</literal> is set to point to this space.
     <function>odr_sequence_end()</function> is called at the end of the
     complex function. Assume that a type is defined like this:
    </para>
    <screen>
MySequence ::= SEQUENCE {
     intval INTEGER,
     boolval BOOLEAN OPTIONAL
}
    </screen>
    <para>
     The corresponding &odr; encoder/decoder function and the associated data
     structures could be written like this:
    </para>
    <screen>
typedef struct MySequence
{
    Odr_int *intval;
    Odr_bool *boolval;
} MySequence;

int mySequence(ODR o, MySequence **p, int optional, const char *name)
{
    if (odr_sequence_begin(o, p, sizeof(**p), name) == 0)
        return optional &amp;&amp; odr_ok(o);
    return
        odr_integer(o, &amp;(*p)->intval, 0, "intval") &amp;&amp;
        odr_bool(o, &amp;(*p)->boolval, 1, "boolval") &amp;&amp;
        odr_sequence_end(o);
}
    </screen>
    <para>
     Note the 1 in the call to <function>odr_bool()</function>, to mark
     that the sequence member is optional.
     If either of the member types had been tagged, the macros
     <function>odr_implicit_tag()</function> or
     <function>odr_explicit_tag()</function>
     could have been used.
     The new function can be used exactly like the standard functions provided
     with &odr;. It will encode, decode or pretty-print a data value of the
     <literal>MySequence</literal> type. We like to name types with an
     initial capital, as done in ASN.1 definitions, and to name the
     corresponding function with the first character of the name in lower case.
     You could, of course, name your structures, types, and functions any way
     you please - as long as you're consistent, and your code is easily readable.
     <literal>odr_ok</literal> is just that - a predicate that returns the
     state of the stream. It is used to ensure that the behavior of the new
     type is compatible with the interface of the primitive types.
    </para>
   </sect2>
   <sect2 id="odr.tagging.constructed.types">
    <title>Tagging Constructed Types</title>
    <note>
     <para>
      See <xref linkend="odr.tagging.primitive.types"/> for information
      on how to tag the primitive types, as well as types that are
      already defined.
     </para>
    </note>
    <sect3 id="odr.implicit.tagging">
     <title>Implicit Tagging</title>
     <para>
      Assume the type above had been defined as
     </para>
     <screen>
MySequence ::= [10] IMPLICIT SEQUENCE {
      intval INTEGER,
      boolval BOOLEAN OPTIONAL
}
     </screen>
     <para>
      You would implement this in &odr; by calling the function
     </para>
     <synopsis>
int odr_implicit_settag(ODR o, int class, int tag);
     </synopsis>
     <para>
      which overrides the tag of the type immediately following it. The
      macro <function>odr_implicit_tag()</function> works by calling
      <function>odr_implicit_settag()</function> immediately
      before calling the function pointer argument.
      Your type function could look like this:
     </para>
     <screen>
int mySequence(ODR o, MySequence **p, int optional, const char *name)
{
    if (odr_implicit_settag(o, ODR_CONTEXT, 10) == 0 ||
        odr_sequence_begin(o, p, sizeof(**p), name) == 0)
        return optional &amp;&amp; odr_ok(o);
    return
        odr_integer(o, &amp;(*p)->intval, 0, "intval") &amp;&amp;
        odr_bool(o, &amp;(*p)->boolval, 1, "boolval") &amp;&amp;
        odr_sequence_end(o);
}
     </screen>
     <para>
      The definition of the structure <literal>MySequence</literal> would be
      the same.
     </para>
    </sect3>
    <sect3 id="odr.explicit.tagging">
     <title>Explicit Tagging</title>
     <para>
      Explicit tagging of constructed types is a little more complicated,
      since you are in effect adding a level of construction to the data.
     </para>
     <para>
      Assume the definition:
     </para>
     <screen>
MySequence ::= [10] IMPLICIT SEQUENCE {
   intval INTEGER,
   boolval BOOLEAN OPTIONAL
}
     </screen>
     <para>
      Since the new type has an extra level of construction, two new functions
      are needed to encapsulate the base type:
     </para>
     <synopsis>
int odr_constructed_begin(ODR o, void *p, int class, int tag,
                          const char *name);

int odr_constructed_end(ODR o);
     </synopsis>
     <para>
      Assume that the IMPLICIT in the type definition above were replaced
      with EXPLICIT (or that the IMPLICIT keyword were simply deleted, which
      would be equivalent). The structure definition would look the same,
      but the function would look like this:
     </para>
     <screen>
int mySequence(ODR o, MySequence **p, int optional, const char *name)
{
    if (odr_constructed_begin(o, p, ODR_CONTEXT, 10, name) == 0)
        return optional &amp;&amp; odr_ok(o);
    if (o->direction == ODR_DECODE)
        *p = odr_malloc(o, sizeof(**p));
    if (odr_sequence_begin(o, p, sizeof(**p), 0) == 0)
    {
        *p = 0; /* this is almost certainly a protocol error */
        return 0;
    }
    return
        odr_integer(o, &amp;(*p)->intval, 0, "intval") &amp;&amp;
        odr_bool(o, &amp;(*p)->boolval, 1, "boolval") &amp;&amp;
        odr_sequence_end(o) &amp;&amp;
        odr_constructed_end(o);
}
     </screen>
     <para>
      Notice that the interface here gets kind of nasty. The reason is
      simple: Explicitly tagged, constructed types are fairly rare in
      the protocols that we care about, so the
      esthetic annoyance (not to mention the dangers of a cluttered
      interface) is less than the time that would be required to develop a
      better interface. Nevertheless, it is far from satisfying, and it's a
      point that will be worked on in the future. One option for you would
      be to simply apply the <function>odr_explicit_tag()</function> macro to
      the first function, and not
      have to worry about <function>odr_constructed_*</function> yourself.
      Incidentally, as you might have guessed, the
      <function>odr_sequence_</function> functions are themselves
      implemented using the <function>/odr_constructed_</function> functions.
     </para>
    </sect3>
   </sect2>
   <sect2 id="odr.sequence.of">
    <title>SEQUENCE OF</title>
    <para>
     To handle sequences (arrays) of a specific type, the function
    </para>
    <synopsis>
int odr_sequence_of(ODR o, int (*fun)(ODR o, void *p, int optional),
                    void *p, int *num, const char *name);
    </synopsis>
    <para>
     The <literal>fun</literal> parameter is a pointer to the decoder/encoder
     function of the type. <literal>p</literal> is a pointer to an array of
     pointers to your type. <literal>num</literal> is the number of elements
     in the array.
    </para>
    <para>
     Assume a type
    </para>
    <screen>
MyArray ::= SEQUENCE OF INTEGER
    </screen>
    <para>
     The C representation might be
    </para>
    <screen>
typedef struct MyArray
{
    int num_elements;
    Odr_int **elements;
} MyArray;
    </screen>
    <para>
     And the function might look like
    </para>
    <screen>
int myArray(ODR o, MyArray **p, int optional, const char *name)
{
    if (o->direction == ODR_DECODE)
        *p = odr_malloc(o, sizeof(**p));
    if (odr_sequence_of(o, odr_integer, &amp;(*p)->elements,
        &amp;(*p)->num_elements, name))
        return 1;
    *p = 0;
        return optional &amp;&amp; odr_ok(o);
}
    </screen>
   </sect2>
   <sect2 id="odr.choice.types">
    <title>CHOICE Types</title>
    <para>
     The choice type is used fairly often in some ASN.1 definitions, so
     some work has gone into streamlining its interface.
    </para>
    <para>
     CHOICE types are handled by the function:
    </para>
    <synopsis>
int odr_choice(ODR o, Odr_arm arm[], void *p, void *whichp,
               const char *name);
    </synopsis>
    <para>
     The <literal>arm</literal> array is used to describe each of the possible
     types that the CHOICE type may assume. Internally in your application,
     the CHOICE type is represented as a discriminated union. That is, a
     C union accompanied by an integer (or enum) identifying the active
     'arm' of the union.
     <literal>whichp</literal> is a pointer to the union discriminator.
     When encoding, it is examined to determine the current type.
     When decoding, it is set to reference the type that was found in
     the input stream.
    </para>
    <para>
     The Odr_arm type is defined thus:
    </para>
    <screen>
typedef struct odr_arm
{
    int tagmode;
    int class;
    int tag;
    int which;
    Odr_fun fun;
    char *name;
} Odr_arm;
    </screen>
    <para>
     The interpretation of the fields are:
    </para>
    <variablelist>
     <varlistentry>
      <term>tagmode</term>
      <listitem><para>Either <literal>ODR_IMPLICIT</literal>,
      <literal>ODR_EXPLICIT</literal>, or <literal>ODR_NONE</literal> (-1)
      to mark   no tagging.</para></listitem>
     </varlistentry>
     <varlistentry>
      <term>which</term>
      <listitem><para>The value of the discriminator that corresponds to
      this CHOICE element. Typically, it will be a #defined constant, or
      an enum member.</para></listitem>
     </varlistentry>
     <varlistentry>
      <term>fun</term>
      <listitem><para>A pointer to a function that implements the type of
      the CHOICE member. It may be either a standard &odr; type or a type
      defined by yourself.</para></listitem>
     </varlistentry>
     <varlistentry>
      <term>name</term>
      <listitem><para>Name of tag.</para></listitem>
     </varlistentry>
    </variablelist>
    <para>
     A handy way to prepare the array for use by the
     <function>odr_choice()</function> function is to
     define it as a static, initialized array in the beginning of your
     decoding/encoding function. Assume the type definition:
    </para>
    <screen>
MyChoice ::= CHOICE {
    untagged INTEGER,
    tagged   [99] IMPLICIT INTEGER,
    other    BOOLEAN
}
    </screen>
    <para>
     Your C type might look like
    </para>
    <screen>
typedef struct MyChoice
{
    enum
    {
        MyChoice_untagged,
        MyChoice_tagged,
        MyChoice_other
    } which;
    union
    {
        Odr_int *untagged;
        Odr_int *tagged;
        Odr_bool *other;
    } u;
};
    </screen>
    <para>
     And your function could look like this:
    </para>
    <screen>
int myChoice(ODR o, MyChoice **p, int optional, const char *name)
{
    static Odr_arm arm[] =
    {
      {-1, -1, -1, MyChoice_untagged, odr_integer, "untagged"},
      {ODR_IMPLICIT, ODR_CONTEXT, 99, MyChoice_tagged, odr_integer,
      "tagged"},
      {-1, -1, -1, MyChoice_other, odr_boolean, "other"},
      {-1, -1, -1, -1, 0}
    };

    if (o->direction == ODR_DECODE)
        *p = odr_malloc(o, sizeof(**p);
    else if (!*p)
        return optional &amp;&amp; odr_ok(o);

    if (odr_choice(o, arm, &amp;(*p)->u, &amp;(*p)->which), name)
        return 1;
    *p = 0;
        return optional &amp;&amp; odr_ok(o);
}
    </screen>
    <para>
     In some cases (say, a non-optional choice which is a member of a
     sequence), you can "embed" the union and its discriminator in the
     structure belonging to the enclosing type, and you won't need to
     fiddle with memory allocation to create a separate structure to
     wrap the discriminator and union.
    </para>
    <para>
     The corresponding function is somewhat nicer in the Sun XDR interface.
     Most of the complexity of this interface comes from the possibility of
     declaring sequence elements (including CHOICEs) optional.
    </para>
    <para>
     The ASN.1 specifications naturally requires that each member of a
     CHOICE have a distinct tag, so they can be told apart on decoding.
     Sometimes it can be useful to define a CHOICE that has multiple types
     that share the same tag. You'll need some other mechanism, perhaps
     keyed to the context of the CHOICE type. In effect, we would like to
     introduce a level of context-sensitiveness to our ASN.1 specification.
     When encoding an internal representation, we have no problem, as long
     as each CHOICE member has a distinct discriminator value. For
     decoding, we need a way to tell the choice function to look for a
     specific arm of the table. The function
    </para>
    <synopsis>
void odr_choice_bias(ODR o, int what);
    </synopsis>
    <para>
     provides this functionality. When called, it leaves a notice for the next
     call to <function>odr_choice()</function> to be called on the decoding
     stream <literal>o</literal> that only the <literal>arm</literal> entry with
     a <literal>which</literal> field equal to <literal>what</literal>
     should be tried.
    </para>
    <para>
     The most important application (perhaps the only one, really) is in
     the definition of application-specific EXTERNAL encoders/decoders
     which will automatically decode an ANY member given the direct or
     indirect reference.
    </para>
   </sect2>
  </sect1>
  <sect1 id="odr.debugging">
   <title>Debugging</title>
   <para>
    The protocol modules are suffering somewhat from a lack of diagnostic
    tools at the moment. Specifically ways to pretty-print PDUs that
    aren't recognized by the system. We'll include something to this end
    in a not-too-distant release. In the meantime, what we do when we get
    packages we don't understand is to compile the ODR module with
    <literal>ODR_DEBUG</literal> defined. This causes the module to dump tracing
    information as it processes data units. With this output and the
    protocol specification (Z39.50), it is generally fairly easy to see
    what goes wrong.
   </para>
  </sect1>
 </chapter>
 <chapter id="comstack">
  <title>The COMSTACK Module</title>
  <sect1 id="comstack.synopsis">
   <title>Synopsis (blocking mode)</title>
   <programlisting><![CDATA[
    COMSTACK stack;
    char *buf = 0;
    int size = 0, length_incoming;
    char server_address_str[] = "localhost:9999";
    void *server_address_ip;
    int status;

    char *protocol_package = "GET / HTTP/1.0\r\n\r\n";
    int protocol_package_length = strlen(protocol_package);

    stack = cs_create(tcpip_type, 1, PROTO_HTTP);
    if (!stack) {
        perror("cs_create");  /* use perror() here since we have no stack yet */
        return -1;
    }

    server_address_ip = cs_straddr(stack, server_address_str);
    if (!server_address_ip) {
        fprintf(stderr, "cs_straddr: address could not be resolved\n");
        return -1;
    }

    status = cs_connect(stack, server_address_ip);
    if (status) {
        fprintf(stderr, "cs_connect: %s\n", cs_strerror(stack));
        return -1;
    }

    status = cs_rcvconnect(stack);
    if (status) {
        fprintf(stderr, "cs_rcvconnect: %s\n", cs_strerror(stack));
        return -1;
    }

    status = cs_put(stack, protocol_package, protocol_package_length);
    if (status) {
        fprintf(stderr, "cs_put: %s\n", cs_strerror(stack));
        return -1;
    }

    /* Now get a response */
    length_incoming = cs_get(stack, &buf, &size);
    if (!length_incoming) {
        fprintf(stderr, "Connection closed\n");
        return -1;
    } else if (length_incoming < 0) {
        fprintf(stderr, "cs_get: %s\n", cs_strerror(stack));
        return -1;
    }

    /* Print result */
    fwrite(buf, length_incoming, 1, stdout);

    /* clean up */
    cs_close(stack);
    if (buf)
        xfree(buf);
    return 0;
]]>
   </programlisting>

  </sect1>
  <sect1 id="comstack.introduction">
   <title>Introduction</title>
   <para>
    The &comstack;
    subsystem provides a transparent interface to different types of transport
    stacks for the exchange of BER-encoded data and HTTP packets.
    At present, the RFC1729 method (BER over TCP/IP), local UNIX socket and an
    experimental SSL stack are supported, but others may be added in time.
    The philosophy of the
    module is to provide a simple interface by hiding unused options and
    facilities of the underlying libraries. This is always done at the risk
    of losing generality, and it may prove that the interface will need
    extension later on.
   </para>
   <note>
    <para>
     There hasn't been interest in the XTImOSI stack for some years.
     Therefore, it is no longer supported.
     </para>
   </note>
   <para>
    The interface is implemented in such a fashion that only the
    sub-layers constructed to the transport methods that you wish to
    use in your application are linked in.
   </para>
   <para>
    You will note that even though simplicity was a goal in the design,
    the interface is still orders of magnitudes more complex than the
    transport systems found in many other packages. One reason is that
    the interface needs to support the somewhat different requirements of
    the different lower-layer communications stacks; another important
    reason is that the interface seeks to provide a more or less
    industrial-strength approach to asynchronous event-handling.
    When no function is allowed to block, things get more complex -
    particularly on the server side.
    We urge you to have a look at the demonstration client and server
    provided with the package. They are meant to be easily readable and
    instructive, while still being at least moderately useful.
   </para>
  </sect1>
  <sect1 id="comstack.common">
   <title>Common Functions</title>
   <sect2 id="comstack.managing.endpoints">
    <title>Managing Endpoints</title>
    <synopsis>
     COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
    </synopsis>
    <para>
     Creates an instance of the protocol stack - a communications endpoint.
     The <literal>type</literal> parameter determines the mode
     of communication. At present the following values are supported:
    </para>
    <variablelist>
     <varlistentry>
      <term><literal>tcpip_type</literal></term>
      <listitem><para>TCP/IP (BER over TCP/IP or HTTP over TCP/IP)
      </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>ssl_type</literal></term>
      <listitem><para>Secure Socket Layer (SSL). This COMSTACK
      is experimental and is not fully implemented. If
      HTTP is used, this effectively is HTTPS.
      </para></listitem>
     </varlistentry>
     <varlistentry>
      <term><literal>unix_type</literal></term>
      <listitem><para>Unix socket (unix only). Local Transfer via
      file socket. See <citerefentry><refentrytitle>unix</refentrytitle>
      <manvolnum>7</manvolnum></citerefentry>.
      </para></listitem>
     </varlistentry>
    </variablelist>
    <para>
     The <function>cs_create</function> function returns a null-pointer
     if a system error occurs.
     The <literal>blocking</literal> parameter should be one if
     you wish the association to operate in blocking mode, zero otherwise.
     The <literal>protocol</literal> field should be
     <literal>PROTO_Z3950</literal> or <literal>PROTO_HTTP</literal>.
     Protocol <literal>PROTO_SR</literal> is no longer supported.
    </para>
    <synopsis>
     void cs_close(COMSTACK handle);
    </synopsis>
    <para>
     Closes the connection (as elegantly as the lower layers will permit),
     and releases the resources pointed to by the
     <literal>handle</literal>
     parameter. The
     <literal>handle</literal>
     should not be referenced again after this call.
    </para>
    <note>
     <para>
      We really need a soft disconnect, don't we?
     </para>
    </note>
   </sect2>
   <sect2 id="comstack.data.exchange">
    <title>Data Exchange</title>
    <synopsis>
     int cs_put(COMSTACK handle, char *buf, int len);
    </synopsis>
    <para>
     Sends <literal>buf</literal> down the wire.
     In blocking mode, this function will return only when a full buffer has
     been written, or an error has occurred. In nonblocking mode, it's
     possible that the function will be unable to send the full buffer
     at once, which will be indicated by a return value of 1.
     The function will keep track of the number of octets already written; you
     should call it repeatedly with the same values of <literal>buf</literal>
     and <literal>len</literal>, until the buffer has been transmitted.
     When a full buffer has been sent, the function will return 0 for
     success. -1 indicates an error condition (see below).
    </para>
    <synopsis>
     int cs_get(COMSTACK handle, char **buf, int *size);
    </synopsis>
    <para>
     Receives a PDU or HTTP Response from the peer. Returns the number of
     bytes read.
     In nonblocking mode, it is possible that not all of the packet can be
     read at once. In this case, the function returns 1. To simplify the
     interface, the function is
     responsible for managing the size of the buffer. It will be reallocated
     if necessary to contain large packages, and will sometimes be moved
     around internally by the subsystem when partial packages are read. Before
     calling
     <function>cs_get</function>
     for the fist time, the buffer can be initialized to the null pointer,
     and the length should also be set to 0 - cs_get will perform a
     <function>malloc(2)</function>
     on the buffer for you. When a full buffer has been read, the size of
     the package is returned (which will always be greater than 1). -1
     indicates an error condition.
    </para>
    <para>
     See also the <function>cs_more()</function> function below.
    </para>
    <synopsis>
     int cs_more(COMSTACK handle);
    </synopsis>
    <para>
     The <function>cs_more()</function> function should be used in conjunction
     with <function>cs_get</function> and
     <function>select(2)</function>.
     The <function>cs_get()</function> function will sometimes
     (notably in the TCP/IP mode) read more than a single protocol package
     off the network. When this happens, the extra package is stored
     by the subsystem. After calling <function>cs_get()</function>, and before
     waiting for more input, You should always call
     <function>cs_more()</function>
     to check if there's a full protocol package already read. If
     <function>cs_more()</function>
     returns 1,
     <function>cs_get()</function>
     can be used to immediately fetch the new package. For the
     mOSI
     subsystem, the function should always return 0, but if you want your
     stuff to be protocol independent, you should use it.
    </para>
    <note>
     <para>
      The <function>cs_more()</function>
      function is required because the RFC1729-method
      does not provide a way of separating individual PDUs, short of
      partially decoding the BER. Some other implementations will carefully
      nibble at the packet by calling
      <function>read(2)</function>
      several times. This was felt to be too inefficient (or at least
      clumsy) - hence the call for this extra function.
     </para>
    </note>
    <synopsis>
     int cs_look(COMSTACK handle);
    </synopsis>
    <para>
     This function is useful when you're operating in nonblocking
     mode. Call it when
     <function>select(2)</function>
     tells you there's something happening on the line. It returns one of
     the following values:
    </para>
    <variablelist>
     <varlistentry>
      <term>CS_NONE</term>
      <listitem><para>
       No event is pending. The data found on the line was not a
       complete package.
      </para></listitem>
     </varlistentry>
     <varlistentry>
      <term>CS_CONNECT</term>
      <listitem><para>
       A response to your connect request has been received. Call
       <function>cs_rcvconnect</function>
       to process the event and to finalize the connection establishment.
      </para></listitem>
     </varlistentry>
     <varlistentry>
      <term>CS_DISCON</term>
      <listitem><para>
       The other side has closed the connection (or maybe sent a disconnect
       request - but do we care? Maybe later). Call
       <function>cs_close</function> to close your end of the association
       as well.
      </para></listitem>
     </varlistentry>
     <varlistentry>
      <term>CS_LISTEN</term>
      <listitem><para>
       A connect request has been received.
       Call <function>cs_listen</function> to process the event.
      </para></listitem>
     </varlistentry>
     <varlistentry>
      <term>CS_DATA</term>
      <listitem><para>
       There's data to be found on the line.
       Call <function>cs_get</function> to get it.
      </para></listitem>
     </varlistentry>
    </variablelist>
    <note>
     <para>
      You should be aware that even if
      <function>cs_look()</function>
      tells you that there's an event event pending, the corresponding
      function may still return and tell you there was nothing to be found.
      This means that only part of a package was available for reading. The
      same event will show up again, when more data has arrived.
     </para>
    </note>
    <synopsis>
     int cs_fileno(COMSTACK h);
    </synopsis>
    <para>
     Returns the file descriptor of the association. Use this when
     file-level operations on the endpoint are required
     (<function>select(2)</function> operations, specifically).
    </para>
   </sect2>
  </sect1>
  <sect1 id="comstack.client">
   <title>Client Side</title>
   <synopsis>
    int cs_connect(COMSTACK handle, void *address);
   </synopsis>
   <para>
    Initiate a connection with the target at <literal>address</literal>
    (more on addresses below). The function will return 0 on success, and 1 if
    the operation does not complete immediately (this will only
    happen on a nonblocking endpoint). In this case, use
    <function>cs_rcvconnect</function> to complete the operation,
    when <function>select(2)</function> or <function>poll(2)</function>
    reports input pending on the association.
   </para>
   <synopsis>
    int cs_rcvconnect(COMSTACK handle);
   </synopsis>
   <para>
    Complete a connect operation initiated by <function>cs_connect()</function>.
    It will return 0 on success; 1 if the operation has not yet completed (in
    this case, call the function again later); -1 if an error has occurred.
   </para>
  </sect1>
  <sect1 id="comstack.server">
   <title>Server Side</title>
   <para>
    To establish a server under the <application>inetd</application>
    server, you can use
   </para>
   <synopsis>
    COMSTACK cs_createbysocket(int socket, CS_TYPE type, int blocking,
                               int protocol);
   </synopsis>
   <para>
    The <literal>socket</literal> parameter is an established socket (when
    your application is invoked from <application>inetd</application>, the
    socket will typically be 0.
    The following parameters are identical to the ones for
    <function>cs_create</function>.
   </para>
   <synopsis>
    int cs_bind(COMSTACK handle, void *address, int mode)
   </synopsis>
   <para>
    Binds a local address to the endpoint. Read about addresses below. The
    <literal>mode</literal> parameter should be either
    <literal>CS_CLIENT</literal> or <literal>CS_SERVER</literal>.
   </para>
   <synopsis>
    int cs_listen(COMSTACK handle, char *addr, int *addrlen);
   </synopsis>
   <para>
    Call this to process incoming events on an endpoint that has been
    bound in listening mode. It will return 0 to indicate that the connect
    request has been received, 1 to signal a partial reception, and -1 to
    indicate an error condition.
   </para>
   <synopsis>
    COMSTACK cs_accept(COMSTACK handle);
   </synopsis>
   <para>
    This finalizes the server-side association establishment, after
    cs_listen has completed successfully. It returns a new connection
    endpoint, which represents the new association. The application will
    typically wish to fork off a process to handle the association at this
    point, and continue listen for new connections on the old
    <literal>handle</literal>.
   </para>
   <para>
    You can use the call
   </para>
   <synopsis>
    const char *cs_addrstr(COMSTACK);
   </synopsis>
   <para>
    on an established connection to retrieve the host-name of the remote host.
   </para>
   <note>
    <para>
     You may need to use this function with some care if your
     name server service is slow or unreliable
    </para>
   </note>
  </sect1>
  <sect1 id="comstack.addresses">
   <title>Addresses</title>
   <para>
    The low-level format of the addresses are different depending on the
    mode of communication you have chosen. A function is provided by each
    of the lower layers to map a user-friendly string-form address to the
    binary form required by the lower layers.
   </para>
   <synopsis>
    void *cs_straddr(COMSTACK handle, const char *str);
   </synopsis>
   <para>
    The format for TCP/IP and SSL addresses is:
   </para>
   <synopsis>
    &lt;host> [ ':' &lt;portnum> ]
   </synopsis>
   <para>
    The <literal>hostname</literal> can be either a domain name or an
    IP address. The port number, if omitted, defaults to 210.
   </para>
   <para>
    For TCP/IP and SSL, the special hostnames <literal>@</literal>,
    maps to <literal>IN6ADDR_ANY_INIT</literal> with
    IPV4 binding as well (bindv6only=0),
    The special hostname <literal>@4</literal> binds to
    <literal>INADDR_ANY</literal> (IPV4 only listener).
    The special hostname <literal>@6</literal> binds to
    <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
   </para>
   <para>
    For UNIX sockets, the format of an address is the socket filename.
   </para>
   <para>
    When a connection has been established, you can use
   </para>
   <synopsis>
    const char *cs_addrstr(COMSTACK h);
   </synopsis>
   <para>
    to retrieve the host name of the peer system. The function returns
    a pointer to a static area, which is overwritten on the next call
    to the function.
   </para>
   <para>
    A fairly recent addition to the &comstack; module is the utility
    function
   </para>
   <synopsis>
    COMSTACK cs_create_host (const char *str, int blocking, void **vp);
   </synopsis>
   <para>
    which is just a wrapper for <function>cs_create</function> and
    <function>cs_straddr</function>. The <parameter>str</parameter>
    is similar to that described for <function>cs_straddr</function>
    but with a prefix denoting the &comstack; type. Prefixes supported
    are <literal>tcp:</literal>, <literal>unix:</literal> and
    <literal>ssl:</literal> for TCP/IP, UNIX and SSL respectively.
    If no prefix is given, then TCP/IP is used.
    The <parameter>blocking</parameter> is passed to
    function <function>cs_create</function>. The third parameter
    <parameter>vp</parameter> is a pointer to &comstack; stack type
    specific values.
    Parameter <parameter>vp</parameter> is reserved for future use.
    Set it to <literal>NULL</literal>.
   </para>
  </sect1>
  <sect1 id="comstack.ssl">
   <title>SSL</title>
   <para>
    <synopsis>
     void *cs_get_ssl(COMSTACK cs);
    </synopsis>
    Returns the SSL handle, <literal>SSL *</literal> for comstack. If comstack
    is not of type SSL, NULL is returned.
   </para>
   <para>
    <synopsis>
     int cs_set_ssl_ctx(COMSTACK cs, void *ctx);
    </synopsis>
    Sets SSL context for comstack. The parameter is expected to be of type
    <literal>SSL_CTX *</literal>. This function should be called just
    after comstack has been created (before connect, bind, etc).
    This function returns 1 for success; 0 for failure.
   </para>
   <para>
    <synopsis>
     int cs_set_ssl_certificate_file(COMSTACK cs, const char *fname);
    </synopsis>
    Sets SSL certificate for comstack as a PEM file. This function
    returns 1 for success; 0 for failure.
   </para>
   <para>
    <synopsis>
     int cs_get_ssl_peer_certificate_x509(COMSTACK cs, char **buf, int *len);
    </synopsis>
    This function returns the peer certificate. If successful,
    <literal>*buf</literal> and <literal>*len</literal> holds
    X509 buffer and length respectively. Buffer should be freed
    with <literal>xfree</literal>. This function returns 1 for success;
    0 for failure.
   </para>
  </sect1>
  <sect1 id="comstack.diagnostics">
   <title>Diagnostics</title>
   <para>
    All functions return -1 if an error occurs. Typically, the functions
    will return 0 on success, but the data exchange functions
    (<function>cs_get</function>, <function>cs_put</function>,
    <function>cs_more</function>) follow special rules. Consult their
    descriptions.
   </para>
   <para>
    The error code for the COMSTACK can be retrieved using C macro
    <function>cs_errno</function> which will return one
    of the error codes <literal>CSYSERR</literal>,
    <literal>CSOUTSTATE</literal>,
    <literal>CSNODATA</literal>, ...
   </para>
   <synopsis>
    int cs_errno(COMSTACK handle);
   </synopsis>
   <para>
    You can the textual representation of the error code
    by using <function>cs_errmsg</function> - which
    works like <function>strerror(3)</function>
   </para>
   <synopsis>
    const char *cs_errmsg(int n);
   </synopsis>
   <para>
    It is also possible to get straight to the textual represenataion
    without the error code by using
    <function>cs_strerror</function>.
   </para>
   <synopsis>
    const char *cs_strerror(COMSTACK h);
   </synopsis>
  </sect1>
  <sect1 id="comstack.summary">
   <title>Summary and Synopsis</title>
   <synopsis><![CDATA[
    #include <yaz/comstack.h>

    #include <yaz/tcpip.h>  /* this is for TCP/IP and SSL support */
    #include <yaz/unix.h>   /* this is for UNIX socket support */

    COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);

    COMSTACK cs_createbysocket(int s, CS_TYPE type, int blocking,
                               int protocol);
    COMSTACK cs_create_host(const char *str, int blocking,
                            void **vp);

    int cs_bind(COMSTACK handle, int mode);

    int cs_connect(COMSTACK handle, void *address);

    int cs_rcvconnect(COMSTACK handle);

    int cs_listen(COMSTACK handle);

    COMSTACK cs_accept(COMSTACK handle);

    int cs_put(COMSTACK handle, char *buf, int len);

    int cs_get(COMSTACK handle, char **buf, int *size);

    int cs_more(COMSTACK handle);

    void cs_close(COMSTACK handle);

    int cs_look(COMSTACK handle);

    void *cs_straddr(COMSTACK handle, const char *str);

    const char *cs_addrstr(COMSTACK h);
]]>
   </synopsis>
  </sect1>
 </chapter>
 <chapter id="future">
  <title>Future Directions</title>
  <para>
   We have a new and better version of the front-end server on the drawing
   board. Resources and external commitments will govern when we'll be
   able to do something real with it. Features should include greater
   flexibility, greater support for access/resource control, and easy
   support for Explain (possibly with Zebra as an extra database engine).
  </para>
  <para>
   &yaz; is a BER toolkit and as such should support all protocols
   out there based on that. We'd like to see running ILL applications.
   It shouldn't be that hard. Another thing that would be interesting is
   LDAP. Maybe a generic framework for doing IR using both LDAP and
   Z39.50 transparently.
  </para>
  <para>
   The SOAP implementation is incomplete. In the future we hope
   to add more features to it. Perhaps make a WSDL/XML Schema compiler.
   The authors of libxml2 are already working on XML Schema / RelaxNG
   compilers so this may not be too hard.
  </para>
  <para>
   It would be neat to have a proper module mechanism for the Generic
   Frontend Server so that backend would be dynamically
   loaded (as shared objects / DLLs).
  </para>
  <para>
   Other than that, &yaz; generally moves in the directions which appear to
   make the most people happy (including ourselves, as prime users of the
   software). If there's something you'd like to see in here, then drop
   us a note and let's see what we can come up with.
  </para>
 </chapter>
 <reference id="reference">
  <title>Reference</title>
   <partintro id="reference-introduction">
    <para>
     The material in this chapter is drawn directly from the individual
     manual entries.
    </para>
   </partintro>
   &manref;
 </reference>
 <appendix id="list-oids">
  <title>List of Object Identifiers</title>
  <para>
   These is a list of object identifiers that are built into YAZ.
  </para>
  &std-oid-table;
 </appendix>
 <appendix id="bib1-diagnostics">
  <title>Bib-1 diagnostics</title>
  <para>
   List of Bib-1 diagnostics that are known to YAZ.
  </para>
  &bib1-diag-table;
 </appendix>
 <appendix id="sru-diagnostics">
  <title>SRU diagnostics</title>
  <para>
   List of SRU diagnostics that are known to YAZ.
  </para>
  &srw-diag-table;
 </appendix>
 <appendix id="license">
  <title>License</title>
  <sect1 id="license.indexdata">
   <title>Index Data Copyright</title>
   <para>
    Copyright &#xa9; &copyright-year; Index Data.
   </para>
   <para>
    All rights reserved.
   </para>
   <para>
    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions are met:
   </para>
   <itemizedlist>
    <listitem>
     <para>
      Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
     </para>
    </listitem>
    <listitem>
     <para>
      Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
     </para>
    </listitem>
    <listitem>
     <para>
      Neither the name of Index Data nor the names of its contributors
      may be used to endorse or promote products derived from this
      software without specific prior written permission.
      </para>
    </listitem>
   </itemizedlist>
   <para>
    THIS SOFTWARE IS PROVIDED BY INDEX DATA ``AS IS'' AND ANY
    EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
    WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
    DISCLAIMED. IN NO EVENT SHALL INDEX DATA BE LIABLE FOR
    ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
    SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
    CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
    LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
    OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGE.
   </para>
  </sect1>
 </appendix>
 <appendix id="indexdata">
  <title>About Index Data</title>
  <para>
   Index Data is a consulting and software-development enterprise that
   specializes in library and information management systems. Our
   interests and expertise span a broad range of related fields, and one
   of our primary, long-term objectives is the development of a powerful
   information management
   system with open network interfaces and hyper-media capabilities.
  </para><para>
   We make this software available free of charge, on a fairly unrestrictive
   license; as a service to the networking community, and to further the
   development of quality software for open network communication.
  </para><para>
   We'll be happy to answer questions about the software, and about ourselves
   in general.
  </para>
  <para>
   <address>
    Index Data ApS
    <street>Amagerf&#xe6;lledvej 56</street>
    <postcode>2300 Copenhagen S</postcode>
    <country>Denmark</country>
    Email <email>info@indexdata.dk</email>
   </address>
  </para>
  <para>
   The Hacker's Jargon File has the following to say about the
   use of the
   prefix &quot;YA&quot; in the name of a software product.
  </para>
  <para>
   <citation>
    Yet Another. adj. 1. Of your own work: A
    humorous allusion often used in titles to acknowledge that the
    topic is not original, though the content is.  As in &quot;Yet Another
    AI Group&quot; or &quot;Yet Another Simulated Annealing Algorithm&quot;.
    2. Of
    others' work: Describes something of which there are already far
    too many.
   </citation>
  </para>
 </appendix>
 <appendix id="credits">
  <title>Credits</title>
  <para>
   This appendix lists individuals that have contributed in the development
   of &yaz;. Some have contributed with code, while others have provided bug
   fixes or suggestions. If we're missing somebody, of if you, for
   whatever reason, don't like to be listed here, let us know.
  </para>
  <itemizedlist>
   <listitem><para>
    Gary Anderson
   </para></listitem>
   <listitem><para>
    Dimitrios Andreadis
    </para></listitem>
   <listitem><para>
    Morten B&#xf8;geskov
   </para></listitem>
   <listitem><para>
    Rocco Carbone
   </para></listitem>
   <listitem><para>
    Matthew Carey
   </para></listitem>
   <listitem><para>
    Hans van Dalen
   </para></listitem>
   <listitem><para>
    Irina Dijour
   </para></listitem>
   <listitem><para>
    Larry E. Dixson
   </para></listitem>
   <listitem><para>
    Hans van den Dool
   </para></listitem>
   <listitem><para>
    Mads Bondo Dydensborg
   </para></listitem>
   <listitem><para>
    Franck Falcoz
   </para></listitem>
   <listitem><para>
    Kevin Gamiel
   </para></listitem>
   <listitem><para>
    Morten Garkier Hendriksen
   </para></listitem>
   <listitem><para>
    Morten Holmqvist
   </para></listitem>
   <listitem><para>
    Ian Ibbotson
   </para></listitem>
   <listitem><para>
    Shigeru Ishida
   </para></listitem>
   <listitem><para>
    Heiko Jansen
   </para></listitem>
   <listitem><para>
    David Johnson
   </para></listitem>
   <listitem><para>
    Oleg Kolobov
   </para></listitem>
   <listitem><para>
    Giannis Kosmas
   </para></listitem>
   <listitem><para>
    Kang-Jin Lee
   </para></listitem>
   <listitem><para>
    Pieter Van Lierop
   </para></listitem>
   <listitem><para>
    Stefan Lohrum
   </para></listitem>
   <listitem><para>
    Ronald van der Meer
   </para></listitem>
   <listitem><para>
    Thomas W. Place
   </para></listitem>
   <listitem><para>
    Peter Popovics
   </para></listitem>
   <listitem><para>
    Jacob Chr. Poulsen
   </para></listitem>
   <listitem><para>
    Ko van der Sloot
   </para></listitem>
   <listitem><para>
    Mike Taylor
   </para></listitem>
   <listitem><para>
    Rustam T. Usmanov
   </para></listitem>
   <listitem><para>
    Charles Woodfield
   </para></listitem>
   <listitem><para>
    Tom Andr&#xe9; &#xd8;verland
   </para></listitem>
  </itemizedlist>
 </appendix>
</book>

<!-- Keep this comment at the end of the file
Local variables:
mode: nxml
nxml-child-indent: 1
End:
-->