Add documentation for CQL->RPN transformation's error-reporting

[yaz-moved-to-github.git] / doc / tools.xml

<!-- $Id: tools.xml,v 1.23 2003-05-22 16:57:28 mike Exp $ -->
 <chapter id="tools"><title>Supporting Tools</title>
  
  <para>
   In support of the service API - primarily the ASN module, which
   provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
   a collection of tools that support the development of applications.
  </para>

  <sect1 id="tools.query"><title>Query Syntax Parsers</title>

   <para>
    Since the type-1 (RPN) query structure has no direct, useful string
    representation, every origin application needs to provide some form of
    mapping from a local query notation or representation to a
    <token>Z_RPNQuery</token> structure. Some programmers will prefer to
    construct the query manually, perhaps using
    <function>odr_malloc()</function> to simplify memory management.
    The &yaz; distribution includes two separate, query-generating tools
    that may be of use to you.
   </para>

   <sect2 id="PQF"><title>Prefix Query Format</title>

    <para>
     Since RPN or reverse polish notation is really just a fancy way of
     describing a suffix notation format (operator follows operands), it
     would seem that the confusion is total when we now introduce a prefix
     notation for RPN. The reason is one of simple laziness - it's somewhat
     simpler to interpret a prefix format, and this utility was designed
     for maximum simplicity, to provide a baseline representation for use
     in simple test applications and scripting environments (like Tcl). The
     demonstration client included with YAZ uses the PQF.
    </para>

    <note>
     <para>
      The PQF have been adopted by other parties developing Z39.50
      software. It is often referred to as Prefix Query Notation
      - PQN.
     </para>
    </note>
    <para>
     The PQF is defined by the pquery module in the YAZ library. 
     There are two sets of function that have similar behavior. First
     set operates on a PQF parser handle, second set doesn't. First set
     set of functions are more flexible than the second set. Second set
     is obsolete and is only provided to ensure backwards compatibility.
    </para>
    <para>
     First set of functions all operate on a PQF parser handle:
    </para>
    <synopsis>
     #include &lt;yaz/pquery.h&gt;

     YAZ_PQF_Parser yaz_pqf_create (void);

     void yaz_pqf_destroy (YAZ_PQF_Parser p);

     Z_RPNQuery *yaz_pqf_parse (YAZ_PQF_Parser p, ODR o, const char *qbuf);

     Z_AttributesPlusTerm *yaz_pqf_scan (YAZ_PQF_Parser p, ODR o,
                          Odr_oid **attributeSetId, const char *qbuf);


     int yaz_pqf_error (YAZ_PQF_Parser p, const char **msg, size_t *off);
    </synopsis>
    <para>
     A PQF parser is created and destructed by functions
     <function>yaz_pqf_create</function> and
     <function>yaz_pqf_destroy</function> respectively.
     Function <function>yaz_pqf_parse</function> parses query given
     by string <literal>qbuf</literal>. If parsing was successful,
     a Z39.50 RPN Query is returned which is created using ODR stream
     <literal>o</literal>. If parsing failed, a NULL pointer is
     returned.
     Function <function>yaz_pqf_scan</function> takes a scan query in 
     <literal>qbuf</literal>. If parsing was successful, the function
     returns attributes plus term pointer and modifies
     <literal>attributeSetId</literal> to hold attribute set for the
     scan request - both allocated using ODR stream <literal>o</literal>.
     If parsing failed, yaz_pqf_scan returns a NULL pointer.
     Error information for bad queries can be obtained by a call to
     <function>yaz_pqf_error</function> which returns an error code and
     modifies <literal>*msg</literal> to point to an error description,
     and modifies <literal>*off</literal> to the offset within last
     query were parsing failed.
    </para>
    <para>
     The second set of functions are declared as follows:
    </para>
    <synopsis>
     #include &lt;yaz/pquery.h&gt;

     Z_RPNQuery *p_query_rpn (ODR o, oid_proto proto, const char *qbuf);

     Z_AttributesPlusTerm *p_query_scan (ODR o, oid_proto proto,
                             Odr_oid **attributeSetP, const char *qbuf);

     int p_query_attset (const char *arg);
    </synopsis>
    <para>
     The function <function>p_query_rpn()</function> takes as arguments an
      &odr; stream (see section <link linkend="odr">The ODR Module</link>)
     to provide a memory source (the structure created is released on
     the next call to <function>odr_reset()</function> on the stream), a
     protocol identifier (one of the constants <token>PROTO_Z3950</token> and
     <token>PROTO_SR</token>), an attribute set reference, and
     finally a null-terminated string holding the query string.
    </para>
    <para>
     If the parse went well, <function>p_query_rpn()</function> returns a
     pointer to a <literal>Z_RPNQuery</literal> structure which can be
     placed directly into a <literal>Z_SearchRequest</literal>. 
     If parsing failed, due to syntax error, a NULL pointer is returned.
    </para>
    <para>
     The <literal>p_query_attset</literal> specifies which attribute set
     to use if the query doesn't specify one by the
     <literal>@attrset</literal> operator.
     The <literal>p_query_attset</literal> returns 0 if the argument is a
     valid attribute set specifier; otherwise the function returns -1.
    </para>

    <para>
     The grammar of the PQF is as follows:
    </para>

    <literallayout>
     query ::= top-set query-struct.

     top-set ::= &lsqb; '@attrset' string &rsqb;

     query-struct ::= attr-spec | simple | complex | '@term' term-type

     attr-spec ::= '@attr' &lsqb; string &rsqb; string query-struct

     complex ::= operator query-struct query-struct.

     operator ::= '@and' | '@or' | '@not' | '@prox' proximity.

     simple ::= result-set | term.

     result-set ::= '@set' string.

     term ::= string.

     proximity ::= exclusion distance ordered relation which-code unit-code.

     exclusion ::= '1' | '0' | 'void'.

     distance ::= integer.

     ordered ::= '1' | '0'.

     relation ::= integer.

     which-code ::= 'known' | 'private' | integer.

     unit-code ::= integer.

     term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
    </literallayout>

    <para>
     You will note that the syntax above is a fairly faithful
     representation of RPN, except for the Attribute, which has been
     moved a step away from the term, allowing you to associate one or more
     attributes with an entire query structure. The parser will
     automatically apply the given attributes to each term as required.
    </para>

    <para>
     The @attr operator is followed by an attribute specification 
     (<literal>attr-spec</literal> above). The specification consists
     of optional an attribute set, an attribute type-value pair and
     a sub query. The attribute type-value pair is packed in one string:
     an attribute type, a dash, followed by an attribute value. 
     The type is always an integer but the value may be either an
     integer or a string (if it doesn't start with a digit character).
    </para>

    <para>
     Version 3 of the Z39.50 specification defines various encoding of terms.
     Use <literal>@term </literal> <replaceable>type</replaceable>
     <replaceable>string</replaceable>,
     where type is one of: <literal>general</literal>,
     <literal>numeric</literal> or <literal>string</literal>
     (for InternationalString).
     If no term type has been given, the <literal>general</literal> form
     is used.  This is the only encoding allowed in both versions 2 and 3
     of the Z39.50 standard.
    </para>
    
    <sect3 id="PQF-prox">
      <title>Using Proximity Operators with PQF</title>
      <note>
        <para>
          This is an advanced topic, describing how to construct
          queries that make very specific requirements on the
          relative location of their operands.
          You may wish to skip this section and go straight to
          <link linkend="pqf-examples">the example PQF queries</link>.
        </para>
        <para>
          <warning>
            <para>
              Most Z39.50 servers do not support proximity searching, or
              support only a small subset of the full functionality that
              can be expressed using the PQF proximity operator.  Be
              aware that the ability to <emphasis>express</emphasis> a
              query in PQF is no guarantee that any given server will
              be able to <emphasis>execute</emphasis> it.
            </para>
          </warning>
        </para>
      </note>
      <para>
        The proximity operator <literal>@prox</literal> is a special
        and more restrictive version of the conjunction operator
        <literal>@and</literal>.  Its semantics are described in 
        section 3.7.2 (Proximity) of Z39.50 the standard itself, which
        can be read on-line at
        <ulink url="http://lcweb.loc.gov/z3950/agency/markup/09.html"/>
      </para>
      <para>
        In PQF, the proximity operation is represented by a sequence
        of the form
        <screen>
@prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
        </screen>
        in which the meanings of the parameters are as described in in
        the standard, and they can take the following values:
        <itemizedlist>
          <listitem><formalpara><title>exclusion</title><para>
            0 = false (i.e. the proximity condition specified by the
            remaining parameters must be satisfied) or
            1 = true (the proximity condition specified by the
            remaining parameters must <emphasis>not</emphasis> be
            satisifed).
          </para></formalpara></listitem>
          <listitem><formalpara><title>distance</title><para>
            An integer specifying the difference between the locations
            of the operands: e.g. two adjacent words would have
            distance=1 since their locations differ by one unit.
          </para></formalpara></listitem>
          <listitem><formalpara><title>ordered</title><para>
            1 = ordered (the operands must occur in the order the
            query specifies them) or
            0 = unordered (they may appear in either order).
          </para></formalpara></listitem>
          <listitem><formalpara><title>relation</title><para>
            Recognised values are
            1 (lessThan),
            2 (lessThanOrEqual),
            3 (equal),
            4 (greaterThanOrEqual),
            5 (greaterThan) and
            6 (notEqual).
          </para></formalpara></listitem>
          <listitem><formalpara><title>which-code</title><para>
            <literal>known</literal>
            or
            <literal>k</literal>
            (the unit-code parameter is taken from the well-known list
            of alternatives described in below) or
            <literal>private</literal>
            or
            <literal>p</literal>
            (the unit-code paramater has semantics specific to an
            out-of-band agreement such as a profile).
          </para></formalpara></listitem>
          <listitem><formalpara><title>unit-code</title><para>
            If the which-code parameter is <literal>known</literal>
            then the recognised values are
            1 (character),
            2 (word),
            3 (sentence),
            4 (paragraph),
            5 (section),
            6 (chapter),
            7 (document),
            8 (element),
            9 (subelement),
            10 (elementType) and
            11 (byte).
            If which-code is <literal>private</literal> then the
            acceptable values are determined by the profile.
          </para></formalpara></listitem>
        </itemizedlist>
        (The numeric values of the relation and well-known unit-code
        parameters are taken straight from
        <ulink url="http://lcweb.loc.gov/z3950/agency/asn1.html#ProximityOperator"
        >the ASN.1</ulink> of the proximity structure in the standard.)
      </para>
    </sect3>

    <sect3 id="pqf-examples"><title>PQF queries</title>

     <para>Queries using simple terms.
      <screen>
      dylan
      "bob dylan"
      </screen>
     </para>
     <para>Boolean operators.
      <screen>
       @or "dylan" "zimmerman"
       @and @or dylan zimmerman when
       @and when @or dylan zimmerman
      </screen>
     </para>
     <para>
      Reference to result sets.
      <screen>
       @set Result-1
       @and @set seta setb
      </screen>
     </para>
     <para>
      Attributes for terms.
      <screen>
       @attr 1=4 computer
       @attr 1=4 @attr 4=1 "self portrait"
       @attr exp1 @attr 1=1 CategoryList
       @attr gils 1=2008 Copenhagen
       @attr 1=/book/title computer
      </screen>
     </para>
     <para>
      Proximity.
      <screen>
       @prox 0 3 1 2 k 2 dylan zimmerman
      </screen>
      <note><para>
      Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
      distance, ordered, relation, which-code and unit-code, in that
      order.  So:
      <itemizedlist>
        <listitem><para>
          exclusion = 0: the proximity condition must hold
        </para></listitem>
        <listitem><para>
          distance = 3: the terms must be three units apart
        </para></listitem>
        <listitem><para>
          ordered = 1: they must occur in the order they are specified
        </para></listitem>
        <listitem><para>
          relation = 2: lessThanOrEqual (to the distance of 3 units)
        </para></listitem>
        <listitem><para>
          which-code is ``known'', so the standard unit-codes are used
        </para></listitem>
        <listitem><para>
          unit-code = 2: word.
        </para></listitem>
      </itemizedlist>
      So the whole proximity query means that the words
      <literal>dylan</literal> and <literal>zimmerman</literal> must
      both occur in the record, in that order, differing in position
      by three or fewer words (i.e. with two or fewer words between
      them.)  The query would find ``Bob Dylan, aka. Robert
      Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
      since the distance in this case is four.
      </para></note>
     </para>
     <para>
      Specifying term type.
      <screen>
       @term string "a UTF-8 string, maybe?"
      </screen>
     </para>
     <para>Mixed queries
      <screen>
       @or @and bob dylan @set Result-1
       
       @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
       
       @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
      </screen>
      <note>
        <para>
          The last of these examples is a spatial search: in
          <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
          >the GILS attribute set</ulink>,
          access point
          2038 indicates West Bounding Coordinate and
          2030 indicates East Bounding Coordinate,
          so the query is for areas extending from -114 degrees
          to no more than -109 degrees.
        </para>
      </note>
     </para>
    </sect3>
   </sect2>
   <sect2 id="CCL"><title>Common Command Language</title>

    <para>
     Not all users enjoy typing in prefix query structures and numerical
     attribute values, even in a minimalistic test client. In the library
     world, the more intuitive Common Command Language (or ISO 8777) has
     enjoyed some popularity - especially before the widespread
     availability of graphical interfaces. It is still useful in
     applications where you for some reason or other need to provide a
     symbolic language for expressing boolean query structures.
    </para>

    <para>
     The <ulink url="http://europagate.dtv.dk/">EUROPAGATE</ulink>
     research project working under the Libraries programme
     of the European Commission's DG XIII has, amongst other useful tools,
     implemented a general-purpose CCL parser which produces an output
     structure that can be trivially converted to the internal RPN
     representation of &yaz; (The <literal>Z_RPNQuery</literal> structure).
     Since the CCL utility - along with the rest of the software
     produced by EUROPAGATE - is made freely available on a liberal
     license, it is included as a supplement to &yaz;.
    </para>

    <sect3><title>CCL Syntax</title>

     <para>
      The CCL parser obeys the following grammar for the FIND argument.
      The syntax is annotated by in the lines prefixed by
      <literal>&dash;&dash;</literal>.
     </para>

     <screen>
      CCL-Find ::= CCL-Find Op Elements
                | Elements.

      Op ::= "and" | "or" | "not"
      -- The above means that Elements are separated by boolean operators.

      Elements ::= '(' CCL-Find ')'
                | Set
                | Terms
                | Qualifiers Relation Terms
                | Qualifiers Relation '(' CCL-Find ')'
                | Qualifiers '=' string '-' string
      -- Elements is either a recursive definition, a result set reference, a
      -- list of terms, qualifiers followed by terms, qualifiers followed
      -- by a recursive definition or qualifiers in a range (lower - upper).

      Set ::= 'set' = string
      -- Reference to a result set

      Terms ::= Terms Prox Term
             | Term
      -- Proximity of terms.

      Term ::= Term string
            | string
      -- This basically means that a term may include a blank

      Qualifiers ::= Qualifiers ',' string
                  | string
      -- Qualifiers is a list of strings separated by comma

      Relation ::= '=' | '>=' | '&lt;=' | '&lt;>' | '>' | '&lt;'
      -- Relational operators. This really doesn't follow the ISO8777
      -- standard.

      Prox ::= '%' | '!'
      -- Proximity operator

     </screen>
     
     <example><title>CCL queries</title>
      <para>
       The following queries are all valid:
      </para>
      
      <screen>
       dylan
       
       "bob dylan"
       
       dylan or zimmerman
       
       set=1
       
       (dylan and bob) or set=1
       
      </screen>
      <para>
       Assuming that the qualifiers <literal>ti</literal>,
       <literal>au</literal>
       and <literal>date</literal> are defined we may use:
      </para>
      
      <screen>
       ti=self portrait
       
       au=(bob dylan and slow train coming)

       date>1980 and (ti=((self portrait)))
       
      </screen>
     </example>
     
    </sect3>
    <sect3><title>CCL Qualifiers</title>
     
     <para>
      Qualifiers are used to direct the search to a particular searchable
      index, such as title (ti) and author indexes (au). The CCL standard
      itself doesn't specify a particular set of qualifiers, but it does
      suggest a few short-hand notations. You can customize the CCL parser
      to support a particular set of qualifiers to reflect the current target
      profile. Traditionally, a qualifier would map to a particular
      use-attribute within the BIB-1 attribute set. However, you could also
      define qualifiers that would set, for example, the
      structure-attribute.
     </para>

     <para>
      A  CCL profile is a set of predefined CCL qualifiers that may be
      read from a file.
      The YAZ client reads its CCL qualifiers from a file named
      <filename>default.bib</filename>. Each line in the file has the form:
     </para>

     <para>
      <replaceable>qualifier-name</replaceable>  
      [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
      [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...      
     </para>

     <para>
      where <replaceable>qualifier-name</replaceable> is the name of the
      qualifier to be used (eg. <literal>ti</literal>),
      <replaceable>type</replaceable> is attribute type in the attribute
      set (Bib-1 is used if no attribute set is given) and
      <replaceable>val</replaceable> is attribute value.
      The <replaceable>type</replaceable> can be specified as an
      integer or as it be specified either as a single-letter:
      <literal>u</literal> for use, 
      <literal>r</literal> for relation,<literal>p</literal> for position,
      <literal>s</literal> for structure,<literal>t</literal> for truncation
      or <literal>c</literal> for completeness.
      The attributes for the special qualifier name <literal>term</literal>
      are used when no CCL qualifier is given in a query.
     </para>

     <example><title>CCL profile</title>
      <para>
       Consider the following definition:
      </para>
      
      <screen>
       ti       u=4 s=1
       au       u=1 s=1
       term     s=105
       ranked   r=102
      </screen>
      <para>
       Three qualifiers are defined, <literal>ti</literal>, 
       <literal>au</literal> and <literal>ranked</literal>.
       <literal>ti</literal> and <literal>au</literal> both set 
       structure attribute to phrase (s=1).
       <literal>ti</literal>
       sets the use-attribute to 4. <literal>au</literal> sets the
       use-attribute to 1.
       When no qualifiers are used in the query the structure-attribute is
       set to free-form-text (105).
      </para>
      <para>
       You can combine attributes. To Search for "ranked title" you
       can do 
       <screen>
        ti,ranked=knuth computer
       </screen>
       which will use "relation is ranked", "use is title", "structure is
       phrase".
      </para>
     </example>
     
    </sect3>
    <sect3><title>CCL API</title>
     <para>
      All public definitions can be found in the header file
      <filename>ccl.h</filename>. A profile identifier is of type
      <literal>CCL_bibset</literal>. A profile must be created with the call
      to the function <function>ccl_qual_mk</function> which returns a profile
      handle of type <literal>CCL_bibset</literal>.
     </para>

     <para>
      To read a file containing qualifier definitions the function
      <function>ccl_qual_file</function> may be convenient. This function
      takes an already opened <literal>FILE</literal> handle pointer as
      argument along with a <literal>CCL_bibset</literal> handle.
     </para>

     <para>
      To parse a simple string with a FIND query use the function
     </para>
     <screen>
struct ccl_rpn_node *ccl_find_str (CCL_bibset bibset, const char *str,
                                   int *error, int *pos);
     </screen>
     <para>
      which takes the CCL profile (<literal>bibset</literal>) and query
      (<literal>str</literal>) as input. Upon successful completion the RPN
      tree is returned. If an error occur, such as a syntax error, the integer
      pointed to by <literal>error</literal> holds the error code and
      <literal>pos</literal> holds the offset inside query string in which
      the parsing failed.
     </para>

     <para>
      An English representation of the error may be obtained by calling
      the <literal>ccl_err_msg</literal> function. The error codes are
      listed in <filename>ccl.h</filename>.
     </para>

     <para>
      To convert the CCL RPN tree (type
      <literal>struct ccl_rpn_node *</literal>)
      to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
      must be used. This function which is part of YAZ is implemented in
      <filename>yaz-ccl.c</filename>.
      After calling this function the CCL RPN tree is probably no longer
      needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
     </para>

     <para>
      A CCL profile may be destroyed by calling the
      <function>ccl_qual_rm</function> function.
     </para>

     <para>
      The token names for the CCL operators may be changed by setting the
      globals (all type <literal>char *</literal>)
      <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
      <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
      An operator may have aliases, i.e. there may be more than one name for
      the operator. To do this, separate each alias with a space character.
     </para>
    </sect3>
   </sect2>
   <sect2 id="tools.cql"><title>CQL</title>
    <para>
     <ulink url="http://www.loc.gov/z3950/agency/zing/cql/">CQL</ulink>
      - Common Query Language - was defined for the
     <ulink url="http://www.loc.gov/z3950/agency/zing/srw/">SRW</ulink>
     protocol.
     In many ways CQL has a similar syntax to CCL.
     The objective of CQL is different. Where CCL aims to be
     an end-user language, CQL is <emphasis>the</emphasis> protocol
     query language for SRW.
    </para>
    <tip>
     <para>
      If you are new to CQL, read the 
      <ulink url="http://zing.z3950.org/cql/intro.html">Gentle
       Introduction</ulink>.
     </para>
    </tip>
    <para>
     The CQL parser in &yaz; provides the following:
     <itemizedlist>
      <listitem>
       <para>
        It parses and validates a CQL query.
       </para>
      </listitem>
      <listitem>
       <para>
        It generates a C structure that allows you to convert
        a CQL query to some other query language, such as SQL.
       </para>
      </listitem>
      <listitem>
       <para>
        The parser converts a valid CQL query to PQF, thus providing a
        way to use CQL for both SRW/SRU servers and Z39.50 targets at the
        same time.
       </para>
      </listitem>
      <listitem>
       <para>
        The parser converts CQL to
        <ulink url="http://www.loc.gov/z3950/agency/zing/cql/xcql.html">
         XCQL</ulink>.
        XCQL is an XML representation of CQL.
        XCQL is part of the SRW specification. However, since SRU
        supports CQL only, we don't expect XCQL to be widely used.
        Furthermore, CQL has the advantage over XCQL that it is
        easy to read.
       </para>
      </listitem>
     </itemizedlist>
    </para>
    <sect3 id="tools.cql.parsing"><title>CQL parsing</title>
     <para>
      A CQL parser is represented by the <literal>CQL_parser</literal>
      handle. Its contents should be considered &yaz; internal (private).
      <synopsis>
#include &lt;yaz/cql.h&gt;

typedef struct cql_parser *CQL_parser;

CQL_parser cql_parser_create(void);
void cql_parser_destroy(CQL_parser cp);
      </synopsis>
     A parser is created by <function>cql_parser_create</function> and
     is destroyed by <function>cql_parser_destroy</function>.
     </para>
     <para>
      To parse a CQL query string, the following function
      is provided:
      <synopsis>
int cql_parser_string(CQL_parser cp, const char *str);
      </synopsis>
      A CQL query is parsed by the <function>cql_parser_string</function>
      which takes a query <parameter>str</parameter>.
      If the query was valid (no syntax errors), then zero is returned;
      otherwise a non-zero error code is returned.
     </para>
     <para>
      <synopsis>
int cql_parser_stream(CQL_parser cp,
                      int (*getbyte)(void *client_data),
                      void (*ungetbyte)(int b, void *client_data),
                      void *client_data);

int cql_parser_stdio(CQL_parser cp, FILE *f);
      </synopsis>
      The functions <function>cql_parser_stream</function> and
      <function>cql_parser_stdio</function> parses a CQL query
      - just like <function>cql_parser_string</function>.
      The only difference is that the CQL query can be
      fed to the parser in different ways.
      The <function>cql_parser_stream</function> uses a generic
      byte stream as input. The <function>cql_parser_stdio</function>
      uses a <literal>FILE</literal> handle which is opened for reading.
     </para>
    </sect3>
    
    <sect3 id="tools.cql.tree"><title>CQL tree</title>
     <para>
      The the query string is validl, the CQL parser
      generates a tree representing the structure of the
      CQL query.
     </para>
     <para>
      <synopsis>
struct cql_node *cql_parser_result(CQL_parser cp);
      </synopsis>
      <function>cql_parser_result</function> returns the
      a pointer to the root node of the resulting tree.
     </para>
     <para>
      Each node in a CQL tree is represented by a 
      <literal>struct cql_node</literal>.
      It is defined as follows:
      <synopsis>
#define CQL_NODE_ST 1
#define CQL_NODE_BOOL 2
#define CQL_NODE_MOD 3
struct cql_node {
    int which;
    union {
        struct {
            char *index;
            char *term;
            char *relation;
            struct cql_node *modifiers;
            struct cql_node *prefixes;
        } st;
        struct {
            char *value;
            struct cql_node *left;
            struct cql_node *right;
            struct cql_node *modifiers;
            struct cql_node *prefixes;
        } boolean;
        struct {
            char *name;
            char *value;
            struct cql_node *next;
        } mod;
    } u;
};
      </synopsis>
      There are three kinds of nodes, search term (ST), boolean (BOOL),
      and modifier (MOD).
     </para>
     <para>
      The search term node has five members:
      <itemizedlist>
       <listitem>
        <para>
         <literal>index</literal>: index for search term.
         If an index is unspecified for a search term,
         <literal>index</literal> will be NULL.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>term</literal>: the search term itself.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>relation</literal>: relation for search term.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>modifiers</literal>: relation modifiers for search
         term. The <literal>modifiers</literal> is a simple linked
         list (NULL for last entry). Each relation modifier node
         is of type <literal>MOD</literal>.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>prefixes</literal>: index prefixes for search
         term. The <literal>prefixes</literal> is a simple linked
         list (NULL for last entry). Each prefix node
         is of type <literal>MOD</literal>.
        </para>
       </listitem>
      </itemizedlist>
     </para>

     <para>
      The boolean node represents both <literal>and</literal>,
      <literal>or</literal>, not as well as
      proximity.
      <itemizedlist>
       <listitem>
        <para>
         <literal>left</literal> and <literal>right</literal>: left
         - and right operand respectively.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>modifiers</literal>: proximity arguments.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>prefixes</literal>: index prefixes.
         The <literal>prefixes</literal> is a simple linked
         list (NULL for last entry). Each prefix node
         is of type <literal>MOD</literal>.
        </para>
       </listitem>
      </itemizedlist>
     </para>

     <para>
      The modifier node is a "utility" node used for name-value pairs,
      such as prefixes, proximity arguements, etc.
      <itemizedlist>
       <listitem>
        <para>
         <literal>name</literal> name of mod node.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>value</literal> value of mod node.
        </para>
       </listitem>
       <listitem>
        <para>
         <literal>next</literal>: pointer to next node which is
         always a mod node (NULL for last entry).
        </para>
       </listitem>
      </itemizedlist>
     </para>

    </sect3>
    <sect3 id="tools.cql.pqf"><title>CQL to PQF conversion</title>
     <para>
      Conversion to PQF (and Z39.50 RPN) is tricky by the fact
      that the resulting RPN depends on the Z39.50 target
      capabilities (combinations of supported attributes). 
      In addition, the CQL and SRW operates on index prefixes
      (URI or strings), whereas the RPN uses Object Identifiers
      for attribute sets.
     </para>
     <para>
      The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
      type. It represents a particular mapping between CQL and RPN.
      This handle is created and destroyed by the functions:
     <synopsis>
cql_transform_t cql_transform_open_FILE (FILE *f);
cql_transform_t cql_transform_open_fname(const char *fname);
void cql_transform_close(cql_transform_t ct);
      </synopsis>
      The first two functions create a tranformation handle from
      either an already open FILE or from a filename respectively.
     </para>
     <para>
      The handle is destroyed by <function>cql_transform_close</function> 
      in which case no further reference of the handle is allowed.
     </para>
     <para>
      When a <literal>cql_transform_t</literal> handle has been created
      you can convert to RPN.
      <synopsis>
int cql_transform_buf(cql_transform_t ct,
                      struct cql_node *cn, char *out, int max);
      </synopsis>
      This function converts the CQL tree <literal>cn</literal> 
      using handle <literal>ct</literal>.
      For the resulting PQF, you supply a buffer <literal>out</literal>
      which must be able to hold at at least <literal>max</literal>
      characters.
     </para>
     <para>
      If conversion failed, <function>cql_transform_buf</function>
      returns a non-zero SRW error code; otherwise zero is returned
      (conversion successful).  The meanings of the numeric error
      codes are listed in the SRW specifications at
      <ulink url="http://www.loc.gov/srw/diagnostic-list.html"/>
     </para>
     <para>
      If conversion fails, more information can be obtained by calling
      <synopsis>
int cql_transform_error(cql_transform_t ct, char **addinfop);
      </synopsis>
      This function returns the most recently returned numeric
      error-code and sets the string-pointer at
      <literal>*addinfop</literal> to point to a string containing
      additional information about the error that occurred: for
      example, if the error code is 15 (``Illegal or unsupported index
      set''), the additional information is the name of the requested
      index set that was not recognised.
     </para>
     <para>
      If you wish to be able to produce a PQF result in a different
      way, there are two alternatives.
      <synopsis>
void cql_transform_pr(cql_transform_t ct,
                      struct cql_node *cn,
                      void (*pr)(const char *buf, void *client_data),
                      void *client_data);

int cql_transform_FILE(cql_transform_t ct,
                       struct cql_node *cn, FILE *f);
      </synopsis>
      The former function produces output to a user-defined
      output stream. The latter writes the result to an already
      open <literal>FILE</literal>.
     </para>
    </sect3>
    <sect3 id="tools.cql.map">
     <title>Specification of CQL to RPN mapping</title>
     <para>
      The file supplied to functions 
      <function>cql_transform_open_FILE</function>,
      <function>cql_transform_open_fname</function> follows
      a structure found in many Unix utilities.
      It consists of mapping specifications - one per line.
      Lines starting with <literal>#</literal> are ignored (comments).
     </para>
     <para>
      Each line is of the form
      <literallayout>
       <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
      </literallayout>
     </para>
     <para>
      An RPN pattern is a simple attribute list. Each attribute pair
      takes the form:
      <literallayout>
       [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
      </literallayout>
      The attribute <replaceable>set</replaceable> is optional.
      The <replaceable>type</replaceable> is the attribute type,
      <replaceable>value</replaceable> the attribute value.
     </para>
     <para>
      The following CQL patterns are recognized:
      <variablelist>
       <varlistentry><term>
         <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
        </term>
        <listitem>
         <para>
          This pattern is invoked when a CQL qualifier, such as 
          dc.title is converted. <replaceable>set</replaceable>
          and <replaceable>name</replaceable> is the index set and qualifier
          name respectively.
          Typically, the RPN specifies an equivalent use attribute.
         </para>
         <para>
          For terms not bound by a qualifier the pattern
          <literal>qualifier.srw.serverChoice</literal> is used.
          Here, the prefix <literal>srw</literal> is defined as
          <literal>http://www.loc.gov/zing/cql/srw-indexes/v1.0/</literal>.
          If this pattern is not defined, the mapping will fail.
         </para>
        </listitem>
       </varlistentry>
       <varlistentry><term>
         <literal>relation.</literal><replaceable>relation</replaceable>
        </term>
        <listitem>
         <para>
          This pattern specifies how a CQL relation is mapped to RPN.
          <replaceable>pattern</replaceable> is name of relation
          operator. Since <literal>=</literal> is used as
          separator between CQL pattern and RPN, CQL relations
          including <literal>=</literal> cannot be
          used directly. To avoid a conflict, the names
          <literal>ge</literal>,
          <literal>eq</literal>,
          <literal>le</literal>,
          must be used for CQL operators, greater-than-or-equal,
          equal, less-than-or-equal respectively.
          The RPN pattern is supposed to include a relation attribute.
         </para>
         <para>
          For terms not bound by a relation, the pattern
          <literal>relation.scr</literal> is used. If the pattern
          is not defined, the mapping will fail.
         </para>
         <para>
          The special pattern, <literal>relation.*</literal> is used
          when no other relation pattern is matched.
         </para>
        </listitem>
       </varlistentry>

       <varlistentry><term>
         <literal>relationModifier.</literal><replaceable>mod</replaceable>
        </term>
        <listitem>
         <para>
          This pattern specifies how a CQL relation modifier is mapped to RPN.
          The RPN pattern is usually a relation attribute.
         </para>
        </listitem>
       </varlistentry>

       <varlistentry><term>
         <literal>structure.</literal><replaceable>type</replaceable>
        </term>
        <listitem>
         <para>
          This pattern specifies how a CQL structure is mapped to RPN.
          Note that this CQL pattern is somewhat to similar to
          CQL pattern <literal>relation</literal>. 
          The <replaceable>type</replaceable> is a CQL relation.
         </para>
         <para>
          The pattern, <literal>structure.*</literal> is used
          when no other structure pattern is matched.
          Usually, the RPN equivalent specifies a structure attribute.
         </para>
        </listitem>
       </varlistentry>

       <varlistentry><term>
         <literal>position.</literal><replaceable>type</replaceable>
        </term>
        <listitem>
         <para>
          This pattern specifies how the anchor (position) of
          CQL is mapped to RPN.
          The <replaceable>type</replaceable> is one
          of <literal>first</literal>, <literal>any</literal>,
          <literal>last</literal>, <literal>firstAndLast</literal>.
         </para>
         <para>
          The pattern, <literal>position.*</literal> is used
          when no other position pattern is matched.
         </para>
        </listitem>
       </varlistentry>

       <varlistentry><term>
         <literal>set.</literal><replaceable>prefix</replaceable>
        </term>
        <listitem>
         <para>
          This specification defines a CQL index set for a given prefix.
          The value on the right hand side is the URI for the set - 
          <emphasis>not</emphasis> RPN. All prefixes used in
          qualifier patterns must be defined this way.
         </para>
        </listitem>
       </varlistentry>
      </variablelist>
     </para>
     <example><title>CQL to RPN mapping file</title>
      <para>
       This simple file defines two index sets, three qualifiers and three
       relations, a position pattern and a default structure.
      </para>
      <programlisting><![CDATA[
       set.srw    = http://www.loc.gov/zing/cql/srw-indexes/v1.0/
       set.dc     = http://www.loc.gov/zing/cql/dc-indexes/v1.0/

       qualifier.srw.serverChoice = 1=1016
       qualifier.dc.title         = 1=4
       qualifier.dc.subject       = 1=21
  
       relation.<                 = 2=1
       relation.eq                = 2=3
       relation.scr               = 2=3

       position.any               = 3=3 6=1

       structure.*                = 4=1
]]>
      </programlisting>
      <para>
       With the mappings above, the CQL query
       <screen>
        computer
       </screen>
       is converted to the PQF:
       <screen>
        @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
       </screen>
       by rules <literal>qualifier.srw.serverChoice</literal>,
       <literal>relation.scr</literal>, <literal>structure.*</literal>,
       <literal>position.any</literal>.
      </para>
      <para>
       CQL query
       <screen>
        computer^
       </screen>
       is rejected, since <literal>position.right</literal> is
       undefined.
      </para>
      <para>
       CQL query
       <screen>
        >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
       </screen>
       is converted to
       <screen>
        @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
       </screen>
      </para>
     </example>
    </sect3>
    <sect3 id="tools.cql.xcql"><title>CQL to XCQL conversion</title>
     <para>
      Conversion from CQL to XCQL is trivial and does not
      require a mapping to be defined.
      There three functions to choose from depending on the
      way you wish to store the resulting output (XML buffer
      containing XCQL).
      <synopsis>
int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
void cql_to_xml(struct cql_node *cn, 
                void (*pr)(const char *buf, void *client_data),
                void *client_data);
void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
      </synopsis>
      Function <function>cql_to_xml_buf</function> converts
      to XCQL and stores result in a user supplied buffer of a given
      max size.
     </para>
     <para>
      <function>cql_to_xml</function> writes the result in
      a user defined output stream.
      <function>cql_to_xml_stdio</function> writes to a
      a file.
     </para>
    </sect3>
   </sect2>
  </sect1>
  <sect1 id="tools.oid"><title>Object Identifiers</title>

   <para>
    The basic YAZ representation of an OID is an array of integers,
    terminated with the value -1. The &odr; module provides two
    utility-functions to create and copy this type of data elements:
   </para>

   <screen>
    Odr_oid *odr_getoidbystr(ODR o, char *str);
   </screen>

   <para>
    Creates an OID based on a string-based representation using dots (.)
    to separate elements in the OID.
   </para>

   <screen>
    Odr_oid *odr_oiddup(ODR odr, Odr_oid *o);
   </screen>

   <para>
    Creates a copy of the OID referenced by the <emphasis>o</emphasis>
    parameter.
    Both functions take an &odr; stream as parameter. This stream is used to
    allocate memory for the data elements, which is released on a
    subsequent call to <function>odr_reset()</function> on that stream.
   </para>

   <para>
    The OID module provides a higher-level representation of the
    family of object identifiers which describe the Z39.50 protocol and its
    related objects. The definition of the module interface is given in
    the <filename>oid.h</filename> file.
   </para>

   <para>
    The interface is mainly based on the <literal>oident</literal> structure.
    The definition of this structure looks like this:
   </para>

   <screen>
typedef struct oident
{
    oid_proto proto;
    oid_class oclass;
    oid_value value;
    int oidsuffix[OID_SIZE];
    char *desc;
} oident;
   </screen>

   <para>
    The proto field takes one of the values
   </para>

   <screen>
    PROTO_Z3950
    PROTO_SR
   </screen>

   <para>
    If you don't care about talking to SR-based implementations (few
    exist, and they may become fewer still if and when the ISO SR and ANSI
    Z39.50 documents are merged into a single standard), you can ignore
    this field on incoming packages, and always set it to PROTO_Z3950
    for outgoing packages.
   </para>
   <para>

    The oclass field takes one of the values
   </para>

   <screen>
    CLASS_APPCTX
    CLASS_ABSYN
    CLASS_ATTSET
    CLASS_TRANSYN
    CLASS_DIAGSET
    CLASS_RECSYN
    CLASS_RESFORM
    CLASS_ACCFORM
    CLASS_EXTSERV
    CLASS_USERINFO
    CLASS_ELEMSPEC
    CLASS_VARSET
    CLASS_SCHEMA
    CLASS_TAGSET
    CLASS_GENERAL
   </screen>

   <para>
    corresponding to the OID classes defined by the Z39.50 standard.

    Finally, the value field takes one of the values
   </para>

   <screen>
    VAL_APDU
    VAL_BER
    VAL_BASIC_CTX
    VAL_BIB1
    VAL_EXP1
    VAL_EXT1
    VAL_CCL1
    VAL_GILS
    VAL_WAIS
    VAL_STAS
    VAL_DIAG1
    VAL_ISO2709
    VAL_UNIMARC
    VAL_INTERMARC
    VAL_CCF
    VAL_USMARC
    VAL_UKMARC
    VAL_NORMARC
    VAL_LIBRISMARC
    VAL_DANMARC
    VAL_FINMARC
    VAL_MAB
    VAL_CANMARC
    VAL_SBN
    VAL_PICAMARC
    VAL_AUSMARC
    VAL_IBERMARC
    VAL_EXPLAIN
    VAL_SUTRS
    VAL_OPAC
    VAL_SUMMARY
    VAL_GRS0
    VAL_GRS1
    VAL_EXTENDED
    VAL_RESOURCE1
    VAL_RESOURCE2
    VAL_PROMPT1
    VAL_DES1
    VAL_KRB1
    VAL_PRESSET
    VAL_PQUERY
    VAL_PCQUERY
    VAL_ITEMORDER
    VAL_DBUPDATE
    VAL_EXPORTSPEC
    VAL_EXPORTINV
    VAL_NONE
    VAL_SETM
    VAL_SETG
    VAL_VAR1
    VAL_ESPEC1
   </screen>

   <para>
    again, corresponding to the specific OIDs defined by the standard.
   </para>

   <para>
    The desc field contains a brief, mnemonic name for the OID in question.
   </para>

   <para>
    The function
   </para>

   <screen>
    struct oident *oid_getentbyoid(int *o);
   </screen>

   <para>
    takes as argument an OID, and returns a pointer to a static area
    containing an <literal>oident</literal> structure. You typically use
    this function when you receive a PDU containing an OID, and you wish
    to branch out depending on the specific OID value.
   </para>

   <para>
    The function
   </para>

   <screen>
    int *oid_ent_to_oid(struct oident *ent, int *dst);
   </screen>

   <para>
    Takes as argument an <literal>oident</literal> structure - in which
    the <literal>proto</literal>, <literal>oclass</literal>/, and
    <literal>value</literal> fields are assumed to be set correctly -
    and returns a pointer to a the buffer as given by <literal>dst</literal>
    containing the base
    representation of the corresponding OID. The function returns
    NULL and the array dst is unchanged if a mapping couldn't place.
    The array <literal>dst</literal> should be at least of size
    <literal>OID_SIZE</literal>.
   </para>
   <para>

    The <function>oid_ent_to_oid()</function> function can be used whenever
    you need to prepare a PDU containing one or more OIDs. The separation of
    the <literal>protocol</literal> element from the remainder of the
    OID-description makes it simple to write applications that can
    communicate with either Z39.50 or OSI SR-based applications.
   </para>

   <para>
    The function
   </para>

   <screen>
    oid_value oid_getvalbyname(const char *name);
   </screen>

   <para>
    takes as argument a mnemonic OID name, and returns the
    <literal>/value</literal> field of the first entry in the database that 
    contains the given name in its <literal>desc</literal> field.
   </para>

   <para>
    Finally, the module provides the following utility functions, whose
    meaning should be obvious:
   </para>

   <screen>
    void oid_oidcpy(int *t, int *s);
    void oid_oidcat(int *t, int *s);
    int oid_oidcmp(int *o1, int *o2);
    int oid_oidlen(int *o);
   </screen>

   <note>
    <para>
     The OID module has been criticized - and perhaps rightly so
     - for needlessly abstracting the
     representation of OIDs. Other toolkits use a simple
     string-representation of OIDs with good results. In practice, we have
     found the interface comfortable and quick to work with, and it is a
     simple matter (for what it's worth) to create applications compatible
     with both ISO SR and Z39.50. Finally, the use of the
     <literal>/oident</literal> database is by no means mandatory.
     You can easily create your own system for representing OIDs, as long
     as it is compatible with the low-level integer-array representation
     of the ODR module.
    </para>
   </note>

  </sect1>

  <sect1 id="tools.nmem"><title>Nibble Memory</title>

   <para>
    Sometimes when you need to allocate and construct a large,
    interconnected complex of structures, it can be a bit of a pain to
    release the associated memory again. For the structures describing the
    Z39.50 PDUs and related structures, it is convenient to use the
    memory-management system of the &odr; subsystem (see
    <link linkend="odr-use">Using ODR</link>). However, in some circumstances
    where you might otherwise benefit from using a simple nibble memory
    management system, it may be impractical to use
    <function>odr_malloc()</function> and <function>odr_reset()</function>.
    For this purpose, the memory manager which also supports the &odr;
    streams is made available in the NMEM module. The external interface
    to this module is given in the <filename>nmem.h</filename> file.
   </para>

   <para>
    The following prototypes are given:
   </para>

   <screen>
    NMEM nmem_create(void);
    void nmem_destroy(NMEM n);
    void *nmem_malloc(NMEM n, int size);
    void nmem_reset(NMEM n);
    int nmem_total(NMEM n);
    void nmem_init(void);
    void nmem_exit(void);
   </screen>

   <para>
    The <function>nmem_create()</function> function returns a pointer to a
    memory control handle, which can be released again by
    <function>nmem_destroy()</function> when no longer needed.
    The function <function>nmem_malloc()</function> allocates a block of
    memory of the requested size. A call to <function>nmem_reset()</function>
    or <function>nmem_destroy()</function> will release all memory allocated
    on the handle since it was created (or since the last call to
    <function>nmem_reset()</function>. The function
    <function>nmem_total()</function> returns the number of bytes currently
    allocated on the handle.
   </para>

   <para>
    The nibble memory pool is shared amongst threads. POSIX
    mutex'es and WIN32 Critical sections are introduced to keep the
    module thread safe. Function <function>nmem_init()</function>
    initializes the nibble memory library and it is called automatically
    the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
    function <function>DllMain</function> to achieve this. You should
    <emphasis>not</emphasis> call <function>nmem_init</function> or
    <function>nmem_exit</function> unless you're absolute sure what
    you're doing. Note that in previous &yaz; versions you'd have to call
    <function>nmem_init</function> yourself. 
   </para>

  </sect1>
 </chapter>
 
 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:t
 sgml-parent-document: "yaz.xml"
 sgml-local-catalogs: nil
 sgml-namecase-general:t
 End:
 -->