-<!-- $Id: tools.xml,v 1.4 2001-08-08 19:33:21 adam Exp $ -->
- <chapter><title>Supporting Tools</title>
+<!-- $Id: tools.xml,v 1.15 2003-01-22 09:43:32 adam Exp $ -->
+ <chapter id="tools"><title>Supporting Tools</title>
<para>
In support of the service API - primarily the ASN module, which
- provides the programmatic interface to the Z39.50 APDUs, &yaz; contains
+ provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
a collection of tools that support the development of applications.
</para>
- <sect1><title>Query Syntax Parsers</title>
+ <sect1 id="tools.query"><title>Query Syntax Parsers</title>
<para>
Since the type-1 (RPN) query structure has no direct, useful string
that may be of use to you.
</para>
- <sect2><title id="PQF">Prefix Query Format</title>
+ <sect2 id="PQF"><title>Prefix Query Format</title>
<para>
Since RPN or reverse polish notation is really just a fancy way of
in simple test applications and scripting environments (like Tcl). The
demonstration client included with YAZ uses the PQF.
</para>
+
+ <note>
+ <para>
+ The PQF have been adopted by other parties developing Z39.50
+ software. It is often referred to as Prefix Query Notation
+ - PQN.
+ </para>
+ </note>
<para>
- The PQF is defined by the pquery module in the YAZ library. The
- <filename>pquery.h</filename> file provides the declaration of the
- functions
+ The PQF is defined by the pquery module in the YAZ library.
+ There are two sets of function that have similar behavior. First
+ set operates on a PQF parser handle, second set doesn't. First set
+ set of functions are more flexible than the second set. Second set
+ is obsolete and is only provided to ensure backwards compatibility.
</para>
- <screen>
-Z_RPNQuery *p_query_rpn (ODR o, oid_proto proto, const char *qbuf);
+ <para>
+ First set of functions all operate on a PQF parser handle:
+ </para>
+ <synopsis>
+ #include <yaz/pquery.h>
-Z_AttributesPlusTerm *p_query_scan (ODR o, oid_proto proto,
- Odr_oid **attributeSetP, const char *qbuf);
+ YAZ_PQF_Parser yaz_pqf_create (void);
-int p_query_attset (const char *arg);
- </screen>
+ void yaz_pqf_destroy (YAZ_PQF_Parser p);
+
+ Z_RPNQuery *yaz_pqf_parse (YAZ_PQF_Parser p, ODR o, const char *qbuf);
+
+ Z_AttributesPlusTerm *yaz_pqf_scan (YAZ_PQF_Parser p, ODR o,
+ Odr_oid **attributeSetId, const char *qbuf);
+
+
+ int yaz_pqf_error (YAZ_PQF_Parser p, const char **msg, size_t *off);
+ </synopsis>
+ <para>
+ A PQF parser is created and destructed by functions
+ <function>yaz_pqf_create</function> and
+ <function>yaz_pqf_destroy</function> respectively.
+ Function <function>yaz_pqf_parse</function> parses query given
+ by string <literal>qbuf</literal>. If parsing was successful,
+ a Z39.50 RPN Query is returned which is created using ODR stream
+ <literal>o</literal>. If parsing failed, a NULL pointer is
+ returned.
+ Function <function>yaz_pqf_scan</function> takes a scan query in
+ <literal>qbuf</literal>. If parsing was successful, the function
+ returns attributes plus term pointer and modifies
+ <literal>attributeSetId</literal> to hold attribute set for the
+ scan request - both allocated using ODR stream <literal>o</literal>.
+ If parsing failed, yaz_pqf_scan returns a NULL pointer.
+ Error information for bad queries can be obtained by a call to
+ <function>yaz_pqf_error</function> which returns an error code and
+ modifies <literal>*msg</literal> to point to an error description,
+ and modifies <literal>*off</literal> to the offset within last
+ query were parsing failed.
+ </para>
+ <para>
+ The second set of functions are declared as follows:
+ </para>
+ <synopsis>
+ #include <yaz/pquery.h>
+
+ Z_RPNQuery *p_query_rpn (ODR o, oid_proto proto, const char *qbuf);
+
+ Z_AttributesPlusTerm *p_query_scan (ODR o, oid_proto proto,
+ Odr_oid **attributeSetP, const char *qbuf);
+
+ int p_query_attset (const char *arg);
+ </synopsis>
<para>
The function <function>p_query_rpn()</function> takes as arguments an
&odr; stream (see section <link linkend="odr">The ODR Module</link>)
<para>
If the parse went well, <function>p_query_rpn()</function> returns a
pointer to a <literal>Z_RPNQuery</literal> structure which can be
- placed directly into a <literal>Z_SearchRequest</literal>.
+ placed directly into a <literal>Z_SearchRequest</literal>.
+ If parsing failed, due to syntax error, a NULL pointer is returned.
</para>
<para>
-
The <literal>p_query_attset</literal> specifies which attribute set
to use if the query doesn't specify one by the
<literal>@attrset</literal> operator.
The grammar of the PQF is as follows:
</para>
- <screen>
- Query ::= [ '@attrset' AttSet ] QueryStruct.
-
- AttSet ::= string.
+ <literallayout>
+ query ::= top-set query-struct.
- QueryStruct ::= [ Attribute ] Simple | Complex.
+ top-set ::= [ '@attrset' string ]
- Attribute ::= '@attr' [ AttSet ] AttributeType '=' AttributeValue.
+ query-struct ::= attr-spec | simple | complex | '@term' term-type
- AttributeType ::= integer.
+ attr-spec ::= '@attr' [ string ] string query-struct
- AttributeValue ::= integer.
+ complex ::= operator query-struct query-struct.
- Complex ::= Operator QueryStruct QueryStruct.
+ operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
- Operator ::= '@and' | '@or' | '@not' | '@prox' Proximity.
+ simple ::= result-set | term.
- Simple ::= ResultSet | Term.
+ result-set ::= '@set' string.
- ResultSet ::= '@set' string.
+ term ::= string.
- Term ::= string | '"' string '"'.
+ proximity ::= exclusion distance ordered relation which-code unit-code.
- Proximity ::= Exclusion Distance Ordered Relation WhichCode UnitCode.
+ exclusion ::= '1' | '0' | 'void'.
- Exclusion ::= '1' | '0' | 'void'.
+ distance ::= integer.
- Distance ::= integer.
+ ordered ::= '1' | '0'.
- Ordered ::= '1' | '0'.
+ relation ::= integer.
- Relation ::= integer.
+ which-code ::= 'known' | 'private' | integer.
- WhichCode ::= 'known' | 'private' | integer.
+ unit-code ::= integer.
- UnitCode ::= integer.
- </screen>
+ term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
+ </literallayout>
<para>
You will note that the syntax above is a fairly faithful
- representation of RPN, except for the Attibute, which has been
+ representation of RPN, except for the Attribute, which has been
moved a step away from the term, allowing you to associate one or more
attributes with an entire query structure. The parser will
automatically apply the given attributes to each term as required.
</para>
<para>
+ The @attr operator is followed by an attribute specification
+ (<literal>attr-spec</literal> above). The specification consists
+ of optional an attribute set, an attribute type-value pair and
+ a sub query. The attribute type-value pair is packed in one string:
+ an attribute type, a dash, followed by an attribute value.
+ The type is always an integer but the value may be either an
+ integer or a string (if it doesn't start with a digit character).
+ </para>
+
+ <para>
+ Z39.50 version 3 defines various encoding of terms.
+ Use the @term operator to indicate the encoding type:
+ <literal>general</literal>, <literal>numeric</literal>,
+ <literal>string</literal> (for InternationalString), ..
+ If no term type has been given, the <literal>general</literal> form
+ is used which is the only encoding allowed in both version 2 - and 3
+ of the Z39.50 standard.
+ </para>
+
+ <para>
The following are all examples of valid queries in the PQF.
</para>
@or @and bob dylan @set Result-1
+ @attr 1=4 computer
+
@attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
@attr 4=1 @attr 1=4 "self portrait"
@prox 0 3 1 2 k 2 dylan zimmerman
@and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
+
+ @term string "a UTF-8 string, maybe?"
+
+ @attr 1=/book/title computer
</screen>
</sect2>
- <sect2><title id="CCL">Common Command Language</title>
+ <sect2 id="CCL"><title>Common Command Language</title>
<para>
Not all users enjoy typing in prefix query structures and numerical
</para>
<para>
- The EUROPAGATE research project working under the Libraries programme
+ The <ulink url="http://europagate.dtv.dk/">EUROPAGATE</ulink>
+ research project working under the Libraries programme
of the European Commission's DG XIII has, amongst other useful tools,
implemented a general-purpose CCL parser which produces an output
structure that can be trivially converted to the internal RPN
- representation of YAZ (The <literal>Z_RPNQuery</literal> structure).
+ representation of &yaz; (The <literal>Z_RPNQuery</literal> structure).
Since the CCL utility - along with the rest of the software
- produced by EUROPAGATE - is made freely available on a liberal license, it
- is included as a supplement to YAZ.
+ produced by EUROPAGATE - is made freely available on a liberal
+ license, it is included as a supplement to &yaz;.
</para>
<sect3><title>CCL Syntax</title>
| string
-- Qualifiers is a list of strings separated by comma
- Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
+ Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
-- Relational operators. This really doesn't follow the ISO8777
-- standard.
index, such as title (ti) and author indexes (au). The CCL standard
itself doesn't specify a particular set of qualifiers, but it does
suggest a few short-hand notations. You can customize the CCL parser
- to support a particular set of qualifiers to relect the current target
+ to support a particular set of qualifiers to reflect the current target
profile. Traditionally, a qualifier would map to a particular
use-attribute within the BIB-1 attribute set. However, you could also
define qualifiers that would set, for example, the
<para>
which takes the CCL profile (<literal>bibset</literal>) and query
(<literal>str</literal>) as input. Upon successful completion the RPN
- tree is returned. If an error eccur, such as a syntax error, the integer
+ tree is returned. If an error occur, such as a syntax error, the integer
pointed to by <literal>error</literal> holds the error code and
<literal>pos</literal> holds the offset inside query string in which
the parsing failed.
</para>
<para>
- An english representation of the error may be obtained by calling
+ An English representation of the error may be obtained by calling
the <literal>ccl_err_msg</literal> function. The error codes are
listed in <filename>ccl.h</filename>.
</para>
</para>
</sect3>
</sect2>
+ <sect2 id="tools.cql"><title>CQL</title>
+ <para>
+ <ulink url="http://www.loc.gov/z3950/agency/zing/cql/">CQL</ulink>
+ - Common Query Language - was defined for the
+ <ulink url="http://www.loc.gov/z3950/agency/zing/srw/">SRW</ulink>
+ protocol.
+ In many ways CQL has a similar syntax to CCL.
+ The objective of CQL is different. Where CCL aims to be
+ an end-user language, CQL is <emphasis>the</emphasis> protocol
+ query language for SRW. Unlike PQF (Z39.50 Type-1), CQL is easy
+ to read.
+ </para>
+ <tip>
+ <para>
+ If you are new to CQL, read the
+ <ulink url="http://zing.z3950.org/cql/intro.html">Gentle
+ Introduction</ulink>.
+ </para>
+ </tip>
+ <para>
+ The CQL parser in &yaz; provides the following:
+ <itemizedlist>
+ <listitem>
+ <para>
+ It parses and validates a CQL query.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ It generates a C structure that allows you to convert
+ a CQL query to some other query language, such as SQL.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The parser converts a valid CQL query to PQF, thus providing a
+ way to use CQL for both SRW/SRU servers and Z39.50 targets at the
+ same time.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The parser converts CQL to
+ <ulink url="http://www.loc.gov/z3950/agency/zing/cql/xcql.html">
+ XCQL</ulink>.
+ XCQL is an XML representation of CQL.
+ XCQL is part of the SRW specification. However, since SRU
+ supports CQL only, we don't expect XCQL to be widely used.
+ Furthermore, CQL has the advantage over XCQL that it is
+ easy to read.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <sect3 id="tools.cql.parsing"><title>CQL parsing</title>
+ <para>
+ A CQL parser is represented by the <literal>CQL_parser</literal>
+ handle. Its contents should be considered &yaz; internal (private).
+ <synopsis>
+#include <yaz/cql.h>
+
+typedef struct cql_parser *CQL_parser;
+
+CQL_parser cql_parser_create(void);
+void cql_parser_destroy(CQL_parser cp);
+
+int cql_parser_string(CQL_parser cp, const char *str);
+ </synopsis>
+ A parser is created by <function>cql_parser_create</function> and
+ is destroyed by <function>cql_parser_destroy</function>.
+ </para>
+ <para>
+ A CQL query is parsed by the <function>cql_parser_string</function>
+ which takes a query <parameter>str</parameter>.
+ If the query was valid (no syntax errors), then zero is returned;
+ otherwise a non-zero error code is returned.
+ </para>
+ <para>
+ <synopsis>
+int cql_parser_stream(CQL_parser cp,
+ int (*getbyte)(void *client_data),
+ void (*ungetbyte)(int b, void *client_data),
+ void *client_data);
+
+int cql_parser_stdio(CQL_parser cp, FILE *f);
+ </synopsis>
+ The functions <function>cql_parser_stream</function> and
+ <function>cql_parser_stdio</function> parses a CQL query
+ - just like <function>cql_parser_string</function>.
+ The only difference is that the CQL query can be
+ fed to the parser in different ways.
+ The <function>cql_parser_stream</function> uses a generic
+ byte stream as input. The <function>cql_parser_stdio</function>
+ uses a <literal>FILE</literal> handle which is opened for reading.
+ </para>
+ </sect3>
+ <sect3 id="tools.cql.tree"><title>CQL tree</title>
+ <para>
+ We now turn to the tree representation of a valid CQL query.
+ <synopsis>
+#define CQL_NODE_ST 1
+#define CQL_NODE_BOOL 2
+#define CQL_NODE_MOD 3
+struct cql_node {
+ int which;
+ union {
+ struct {
+ char *index;
+ char *term;
+ char *relation;
+ struct cql_node *modifiers;
+ struct cql_node *prefixes;
+ } st;
+ struct {
+ char *value;
+ struct cql_node *left;
+ struct cql_node *right;
+ struct cql_node *modifiers;
+ struct cql_node *prefixes;
+ } bool;
+ struct {
+ char *name;
+ char *value;
+ struct cql_node *next;
+ } mod;
+ } u;
+};
+ </synopsis>
+ There are three kinds of nodes, search term (ST), boolean (BOOL),
+ and modifier (MOD).
+ </para>
+ <para>
+ The search term node has five members:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>index</literal>: index for search term.
+ If an index is unspecified for a search term,
+ <literal>index</literal> will be NULL.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>term</literal>: the search term itself.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>relation</literal>: relation for search term.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>modifiers</literal>: relation modifiers for search
+ term. The <literal>modifiers</literal> is a simple linked
+ list (NULL for last entry). Each relation modifier node
+ is of type <literal>MOD</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>prefixes</literal>: index prefixes for search
+ term. The <literal>prefixes</literal> is a simple linked
+ list (NULL for last entry). Each prefix node
+ is of type <literal>MOD</literal>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The boolean node represents both <literal>and</literal>,
+ <literal>or</literal>, not as well as
+ proximity.
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>left</literal> and <literal>right</literal>: left
+ - and right operand respectively.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>modifiers</literal>: proximity arguments.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>prefixes</literal>: index prefixes.
+ The <literal>prefixes</literal> is a simple linked
+ list (NULL for last entry). Each prefix node
+ is of type <literal>MOD</literal>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The modifier node is a "utility" node used for name-value pairs,
+ such as prefixes, proximity arguements, etc.
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>name</literal> name of mod node.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>value</literal> value of mod node.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>next</literal>: pointer to next node which is
+ always a mod node (NULL for last entry).
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ </sect3>
+ </sect2>
</sect1>
- <sect1><title>Object Identifiers</title>
+ <sect1 id="tools.oid"><title>Object Identifiers</title>
<para>
The basic YAZ representation of an OID is an array of integers,
<para>
The OID module provides a higher-level representation of the
- family of object identifers which describe the Z39.50 protocol and its
+ family of object identifiers which describe the Z39.50 protocol and its
related objects. The definition of the module interface is given in
the <filename>oid.h</filename> file.
</para>
The <function>oid_ent_to_oid()</function> function can be used whenever
you need to prepare a PDU containing one or more OIDs. The separation of
- the <literal>protocol</literal> element from the remainer of the
+ the <literal>protocol</literal> element from the remainder of the
OID-description makes it simple to write applications that can
communicate with either Z39.50 or OSI SR-based applications.
</para>
</sect1>
- <sect1><title>Nibble Memory</title>
+ <sect1 id="tools.nmem"><title>Nibble Memory</title>
<para>
Sometimes when you need to allocate and construct a large,
void nmem_reset(NMEM n);
int nmem_total(NMEM n);
void nmem_init(void);
+ void nmem_exit(void);
</screen>
<para>
allocated on the handle.
</para>
- <note>
- <para>
- The nibble memory pool is shared amonst threads. POSIX
- mutex'es and WIN32 Critical sections are introduced to keep the
- module thread safe. On WIN32 function <function>nmem_init()</function>
- initialises the Critical Section handle and should be called once
- before any other nmem function is used.
- </para>
- </note>
+ <para>
+ The nibble memory pool is shared amongst threads. POSIX
+ mutex'es and WIN32 Critical sections are introduced to keep the
+ module thread safe. Function <function>nmem_init()</function>
+ initializes the nibble memory library and it is called automatically
+ the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
+ function <function>DllMain</function> to achieve this. You should
+ <emphasis>not</emphasis> call <function>nmem_init</function> or
+ <function>nmem_exit</function> unless you're absolute sure what
+ you're doing. Note that in previous &yaz; versions you'd have to call
+ <function>nmem_init</function> yourself.
+ </para>
</sect1>
</chapter>
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document: "yaz.xml"
- sgml-local-catalogs: "../../docbook/docbook.cat"
+ sgml-local-catalogs: nil
sgml-namecase-general:t
End:
-->
projects / yaz-moved-to-github.git / blobdiff