1 <!doctype linuxdoc system>
3 <title>YAZ User's Guide and Reference
4 <author><htmlurl url="http://www.indexdata.dk/" name="Index Data">, <tt><htmlurl url="mailto:info@indexdata.dk" name="info@indexdata.dk"></>
5 <date>$Revision: 1.10 $
7 This document is the programmer's guide and reference to the YAZ
8 package. YAZ is a compact toolkit that provides access to the
9 Z39.50/SR protocol, as well as a set of higher-level tools for
10 implementing the server and client roles, respectively.
11 The documentation can be used on its own, or as a reference when looking
12 at the example applications provided with the package.
20 The <bf/YAZ/ toolkit offers several different levels of access to the
21 Z39.50 and SR protocols. The level that you need to use depends on
22 your requirements, and the role (server or client) that you
25 The basic level, which is independent of the role, consists of three
29 <item><bf/ASN/, which provides a C representation of the Z39.50/SR protocol
31 <item><bf/ODR/, which encodes and decodes the packages according to the BER
33 <item><bf/COMSTACK/, which exchanges the encoded packages with a peer process
37 The ASN module represents the ASN.1 definition of the
38 SR/Z39.50 protocol. It establishes a set of type and
39 structure definitions, with one structure for each of the top-level
40 PDUs, and one structure or type for each of the contained ASN.1 types.
41 For primitive types, or other types that are defined by the ASN.1
42 standard itself (such as the EXTERNAL type), the C representation is provided
43 by the <bf/ODR/ (Open Data Representation) subsystem.
45 <bf/ODR/ is a basic mechanism for representing an
46 ASN.1 type in the C programming language, and for implementing BER
47 encoders and decoders for values of that type. The types defined in
49 module generally have the prefix <tt/Z_/, and a suffix corresponding to
50 the name of the type in the ASN.1
51 specification of the protocol (generally Z39.50-1995). In the case of
52 base types (those originating in the ASN.1 standard itself), the prefix
53 <tt/Odr_/ is sometimes seen. Either way, look for
54 the actual definition in either <tt/proto.h/ (for the types from the
55 protocol), <tt/odr.h/ (for the primitive ASN.1 types, or <tt/odr_use.h/ (for the
56 ASN.1 <it/useful/ types). The <bf/ASN/ library also provides functions (which
57 are, in turn, defined using <bf/ODR/ primitives) for encoding and decoding data
58 values. Their general form is
61 int z_xxx(ODR o, Z_xxx **p, int optional, const char *name);
64 (note the lower-case &dquot;z&dquot; in the function name)
67 NOTE: If you are using the premade definitions of the <bf/ASN/ module, and
68 you are not adding new protocol of your own, the only parts of ODR
69 that you need to worry about are documented in section
70 <ref id="odr-use" name="Using ODR">.
73 When you have created a BER-encoded buffer, you can use the <bf/COMSTACK/
74 subsystem to transmit (or receive) data over the network. The <bf/COMSTACK/
75 module provides simple functions for establishing a connection
76 (passively or actively, depending on the role of your application),
77 and for exchanging BER-encoded PDUs over that connection. When you
78 create a connection endpoint, you need to specify what transport to
79 use (OSI or TCP/IP), and which protocol you want to use (SR or
80 Z39.50). For the remainer of the connection's lifetime, you don't have
81 to worry about the underlying transport protocol at all - the <bf/COMSTACK/
82 will ensure that the correct mechanism is used.
84 We call the combined interfaces to <bf/ODR/, <bf/ASN/, and <bf/COMSTACK/ the service
85 level API. It's the API that most closely models the Z39.50/SR
86 service/protocol definition, and it provides unlimited access to all
87 fields and facilities of the protocol definitions.
89 The reason that the <bf/YAZ/ service-level API is a conglomerate of the
90 APIs from three different submodules is twofold. First, we wanted to allow the
91 user a choice of different options for each major task. For instance,
92 if you don't like the protocol API provided by <bf/ODR//<bf/ASN/, you
93 can use SNACC or BERUtils instead, and still have the benefits of the
94 transparent transport approach of the <bf/COMSTACK/ module. Secondly,
95 we realise that you may have to fit the toolkit into an existing
96 event-processing structure, in a way that
97 is incompatible with the <bf/COMSTACK/ interface or some other part of <bf/YAZ/.
99 <sect>Compilation and Installation
102 The latest version of the software will generally be found at
105 <htmlurl url="http://ftp.indexdata.dk/pub/yaz/" name="http://ftp.indexdata.dk/pub/yaz/">
108 We have tried our best to keep the software portable, and on many
109 platforms, you should be able to compile everything with little or no changes.
110 So far, the software has been ported
111 to the following platforms with little or no difficulties.
121 <item>Data General DG/UX (with some CFLAGS tinkering)
125 <item>Non-unix systems
127 <item>Apple Macintosh (using the Codewarrior programming environment and the
128 GUSI socket libraries)
129 <item>MS Windows 95/NT (Win32)
134 If you move the software to other platforms, we'd be grateful if you'd
135 let us know about it. If you run into difficulties, we will try to help if we
136 can, and if you solve the problems, we would be happy to
137 include your fixes in the next release. So far, we have mostly avoided
138 #ifdefs for individual platforms, and we'd like to keep it that
139 way as far as it makes sense.
141 We maintain a mailing-list for the purpose of announcing new releases and
142 bug-fixes, as well as general discussion. Subscribe by sending mail to
143 <tt/yaz-request@indexdata.dk/. General questions and problems can be
144 directed at <tt/yaz-help@indexdata.dk/, or the address given at the top
150 Note that if your system doesn't have a native ANSI C compiler, you may
151 have to acquire one separately. We recommend gcc.
153 For UNIX we use GNU configure to create Makefiles for YAZ.
154 Generally it should be sufficient to run configure without options:
160 The configure script attempts to use use the C compiler specified by
161 the <tt/CC/ environment variable. If not set, GNU C will be used
162 if it is available. The <tt/CFLAGS/ environment variable holds options
163 to be passed to the C compiler. If you're using Bourne-compatible shell
164 you may pass something like this to use a particular C compiler with
165 optimization enabled:
167 CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
170 To customize <bf/YAZ/ the configure script also accepts a set of options.
171 The most important are:
174 <tag><tt>-</tt><tt>-prefix </tt>path</tag> Specifies installation prefix. This is
175 only needed if you run <tt>make install</tt> later to perform a
176 "system" installation. The prefix is <tt>/usr/local</tt> if not
179 <tag><tt>-</tt><tt>-enable-comp </tt></tag> YAZ will be built using
180 the ASN.1 compiler for YAZ (default). If you wish to use the
181 old decoders (in sub directory asn) use <tt>--disable-comp</tt> instead.
183 <tag><tt>-</tt><tt>-enable-threads</tt></tag> YAZ will be built using
184 POSIX threads. Specifically, <tt>_REENTRANT</tt> will be defined
188 When configured, build the software by typing:
193 The following files are generated by the make process:
195 <tag><tt>lib/libyaz.a</tt></tag> The <bf/YAZ/ programmers' library.
197 <tag><tt>ztest/yaz-ztest</tt></tag> A test Z39.50 server.
199 <tag><tt>client/yaz-client</tt></tag> A command mode Z39.50 client.
201 <tag><tt>yaz-config</tt></tag> A Bourne-shell script that holds build
202 settings for <bf/YAZ/.
204 <tag><tt>yaz-comp</tt></tag> The ASN.1 compiler for YAZ. Requires the
205 Tcl Shell, tclsh, in current path to work.
209 If you wish to install <bf/YAZ/ in system directories such as /usr/local/bin,
210 /usr/local/lib) you can type:
216 You probably need to have root access in order to perform this.
217 You must specify the <tt>--prefix</tt> option for configure if you
218 going to install in anything but /usr/local/.
220 If you wish to perform an un-installation of YAZ you use:
226 This will only work if you haven't reconfigured YAZ (and therefore
227 changed installation prefix). Note that uninstall will not
228 remove directories created by make install, e.g.
229 <tt>/usr/local/include/yaz</tt>.
234 <bf/YAZ/ is shipped with "makefiles" for the NMAKE tool that comes
237 Start an MS-DOS prompt and switch the sub directory <tt>WIN</tt> where
238 the file <tt>makefile</tt> is located. Customize the installation
239 by editing the <tt>makefile</tt> file (for example by using notepad).
241 The following summarises the most important settings in that file:
244 <tag><tt>NEW_Z3950</tt></tag> If 1, the auto-generated decoder/encoders
245 for Z39.50 as written by the ASN.1 compiler will be used. If 0, the old
246 decoders for Z39.50 will be used. Note, when 1, the setting TCL should
247 point to the Tcl shell on your system.
248 <tag><tt>DEBUG</tt></tag> If set to 1, the software is
249 compiled with debugging libraries. If set to 0, the software
250 is compiled with release (non-debugging) libraries.
253 When satisfied with the settings in the makefile type
258 The following is generated upon successful compilation:
260 <tag><tt>bin/yaz.dll</tt></tag> A multithreaded DLL with everything
261 except the frontend server library.
262 <tag><tt>lib/yaz.lib</tt></tag> An import library for <tt>yaz.dll</tt>.
263 <tag><tt>lib/server.lib</tt></tag> The frontend server library.
264 <tag><tt>bin/yaz-ztest.exe</tt></tag> A test Z39.50 server.
265 <tag><tt>bin/yaz-client.exe</tt></tag> A command mode Z39.50 client.
268 <sect>Using the Yaz-client
271 yaz-client is a linemode Z39.50 client. It supports a fair amount of the
272 functionality of Z39.50-1995 standard, but some things you need to enable or
273 disable by recompilation. Its primary purpose is to exercise the
274 package, and verify that the protocol works OK.
276 It can be started by typing
278 yaz-client [-m <marclog>] [ -a <apdulog>] tcp:<hostname>:<port>[/<database>]
280 at the UNIX prompt, to connect to a Z39.50 server.
284 <item><bf/-m/ Turns dumping of the raw MARC records on in ISO 2709 format.
285 Marclog is the filename to write to.
286 <item><bf/-a/ Turns dumping of the APDU on. Apdulog is the filename to
287 write to. If apdulog is "-" the APDU is written to the screen.
290 In order to connect to Index Data's test Z39.50 server on
291 bagel.indexdata.dk, port 210 and with the database name marc, one would
294 yaz-client tcp:bagel.indexdata.dk:210/marc
296 In order to also dump the APDU to the screen you would have to write
298 yaz-client -a - tcp:bagel.indexdata.dk:210/marc
300 Use '?' to get a list of the available commands.
302 The commands are (the letters in parenthesis are short names for the commands):
304 <tag/open (o)/Opens a connection to a server. The syntax is the same as described above for connecting from the command line.
307 open ('tcp'|'osi')':'[<tsel>'/']<host>[':'<port>]
316 Sends the RPN query to the server.
322 Deletes a result set on the server.
325 delete <setname>
328 Sets the name of the database to search in if it wasn't already set in the
329 connect string. More than one database can be searched in parallel by writing
330 several databases (hosted on the same server) separated by white space.
333 base <base-name>
336 Shows a record. If no record number is specified the next record in the result set is shown.
339 show <rec#>['+'<#recs>['+'<setname>]]
342 Scans the database index for a term. The syntax resembles the syntax for <bf/find/.
343 If you want to scan for the word <it/water/ you would write
347 but if you want to scan only in, say the title field, you would write
356 Sorts a result set. The sort command takes a sequence of sort specifications. A sort
357 specification holds a field (sort criteria) and is followed by flags.
358 If the sort criteria includes = it is assumed that the sort SortKey
359 is of type sortAttributes using Bib-1. The integer before the = is
360 the attribute type and the integer following the = is the attribute
361 value. If no = is in the SortKey it is treated as a sortfield-type
362 of type InternationalString. Flags observed are
364 <item><bf/s/ (sort case sensitive)
365 <item><bf/i/ (sort case insensitive), < (ascending), > (descending).
369 1=4 i< (use is title, insensitive, ascending).
370 Title s> (String Title, sensitive, descending).
374 sort <sortkey> <flag> <sortkey> <flag> ...
377 Same as <bf/sort/ but stores the sorted result set in a new result set.
380 sort+ <sortkey> <flag> <sortkey> <flag> ...
383 Sets up a authentication string if a server requires authentication. The authentication string is first
384 sent to the server when the <bf/open/ command is issued.
387 authentication <acctstring>
390 Sets the limit for when no records should be returned together with the search result.
391 See the <htmlurl url="http://lcweb.loc.gov/z3950/agency/markup/04.html#3.2.2.1.6" name="Z39.50 standard">
395 lslb <largeSetLowerBound>
398 Sets the limit for when all records should be returned with the search result.
399 See the <htmlurl url="http://lcweb.loc.gov/z3950/agency/markup/04.html#3.2.2.1.6" name="Z39.50 standard">
403 ssub <smallSetUpperBound>
406 Sets the number of records should be returned if the number of records in the
407 result set is between the values of <bf/lslb/ and <bf/ssub/.
408 See the <htmlurl url="http://lcweb.loc.gov/z3950/agency/markup/04.html#3.2.2.1.6" name="Z39.50 standard">
412 mspn <mediumSetPresentNumber>
415 Displays the values of <bf/lslb/, <bf/ssub/ and <bf/mspn/.
421 Switches named result sets on and off. Default is on.
427 Sends a trigger resource ctrl cancel to the server.
433 Sets the preferred transfer syntax the records should be returned in.
434 yaz-client supports all the record syntaxes that corrently are registered
435 with the (see the <htmlurl url="http://lcweb.loc.gov/z3950/agency/defns/oids.html#5" name="Z39.50 Agency"> for more details) including usmarc, sutrs, GRS1 and XML
438 format <recordsyntax>
441 Sets the preferred schema the records should be returned in.
443 <tscreen><verb>schema <schema>
446 Sets the element set name for the records.
450 <item>Any other element set name the server may support
454 elements <elementSetName>
457 Sends a close request.
462 Sets the query type. Default is prefix.
464 <item><bf/prefix/ RPN query.
465 <item><bf/CCL/ CCL (Common Command Language) query.
466 <item><bf/ CCL2RPN/A CCL variant that is interpreted on the client side
467 which allows you to specify field names. See section <ref id="CCL" name="Common Command Language"> for more details.
471 querytype <type>
474 Specifies the default attribute set in the query. Default is Bib1
475 (see the <htmlurl url="http://lcweb.loc.gov/z3950/agency/defns/bib1.html" name="Z39.50 Agency">). This command applies only to the prefix querytype. Note that
476 you can also specify the attribute set directly in the RPN query.
479 attributeset <attrset>
482 Sets a reference id for a request to be send to the server.
483 See the <htmlurl url="http://lcweb.loc.gov/z3950/agency/markup/08.html#3.4" name="Z39.50 standard">
490 Sends an itemorder to the server. Only the required fields are implemented.
493 itemorder 1|2 <item>
496 Sends an item update to the server. This feature is not fully implemented yet.
502 <sect1>Searching with yaz-client
504 The simplest form of a RPM query in yaz-client would be something like
512 This leaves it up to the server what fields to search but most servers will search
513 in all fields. Some servers does not support this feature though, and require that
514 a search attribute is defined. This would look something like this in yaz-client:
518 where we search in the title field. If we want to search in the author field <bf/and/
519 in the title field, and in the title field using right truncation it could look something like this:
521 f @and @attr 1=1003 knuth @attr 1=4 @attr 5=1 computer
523 Finally using a mix of Bib-1 and GILS attributes could look something like this:
525 f @attrset Bib-1 @and @attr GILS 2=2008 Washington @attr 1=21 weather
527 For the full spacifiction of the RPN query language please see the section <ref id="PQF" name="Prefix Query Format">.
534 The <bf/ASN/ module provides you with a set of C struct definitions for the
535 various PDUs of the protocol, as well as for the complex types
536 appearing within the PDUs. For the primitive data types, the C
537 representation often takes the form of an ordinary C language type,
538 such as <tt/int/. For ASN.1 constructs that have no direct
539 representation in C, such as general octet strings and bit strings,
540 the <bf/ODR/ module (see section <ref id="odr" name="ODR">) provides auxiliary
543 <sect1>Preparing PDUs
546 A structure representing a complex ASN.1 type doesn't in itself contain the
547 members of that type. Instead, the structure contains <it/pointers/ to
548 the members of the type. This is necessary, in part, to allow a mechanism for
549 specifying which of the optional structure (SEQUENCE) members are
550 present, and which are not. It follows that you will need to somehow
551 provide space for the individual members of the structure, and
552 set the pointers to refer to the members.
554 The conversion routines don't care how you allocate and maintain your
555 C structures - they just follow the pointers that you provide.
556 Depending on the complexity of your application, and your personal
557 taste, there are at least three different approaches that you may take
558 when you allocate the structures.
562 You can use static or automatic local variables in the function that
563 prepares the PDU. This is a simple approach, and it provides the most
564 efficient form of memory management. While it works well for flat
565 PDUs like the InitReqest, it will generally not be sufficient for say,
566 the generation of an arbitrarily complex RPN query structure.
568 You can individually create the structure and its members using the
569 <tt/malloc/(2) function. If you want to ensure that the data is freed when
570 it is no longer needed, you will have to define a function that
571 individually releases each member of a structure before freeing the
574 You can use the <tt/odr_malloc()/ function (see section <ref
576 name="Using ODR"> for details). When you use <tt/odr_malloc()/, you can release
578 allocated data in a single operation, independent of any pointers and
579 relations between the data. <tt/odr_malloc()/ is based on a
580 &dquot;nibble-memory&dquot;
581 scheme, in which large portions of memory are allocated, and then
582 gradually handed out with each call to <tt/odr_malloc()/. The next time you
583 call <tt/odr_reset()/, all of the memory allocated since the last call is
584 recycled for future use (actually, it is placed on a free-list).
586 You can combine all of the methods described here. This will often be
587 the most practical approach. For instance, you might use <tt/odr_malloc()/ to
588 allocate an entire structure and some of its elements, while you leave
589 other elements pointing to global or per-session default variables.
592 The <bf/ASN/ module provides an important aid in creating new PDUs. For
593 each of the PDU types (say, <tt/Z_InitRequest/), a function is provided
594 that allocates and initializes an instance of that PDU type for you.
595 In the case of the InitRequest, the function is simply named
596 <tt/zget_InitRequest()/, and it sets up reasonable default value for all of
597 the mandatory members. The optional members are generally initialized to null
598 pointers. This last aspect is very important: it ensures that if the
599 PDU definitions are extended after you finish your implementation
601 new versions of the protocol, say), you won't get into trouble with
602 uninitialized pointers in your structures. The functions use
604 allocate the PDUs and its members, so you can free everything again with a
605 single call to <tt/odr_reset()/. We strongly recommend that you use the
607 functions whenever you are preparing a PDU (in a C++ API, the
609 functions would probably be promoted to constructors for the
612 The prototype for the individual PDU types generally look like this:
615 Z_<type> *zget_<type>(ODR o);
621 Z_InitRequest *zget_InitRequest(ODR o);
624 The <bf/ODR/ handle should generally be your encoding stream, but it
627 As well as the individual PDU functions, a function <tt/zget_APDU()/ is
628 provided, which allocates a toplevel Z-APDU of the type requested:
631 Z_APDU *zget_APDU(ODR o, int which);
634 The <tt/which/ parameter is (of course) the discriminator belonging to the
635 <tt/Z_APDU/ CHOICE type. All of the interface described here is provided by
636 the <bf/ASN/ module, and you access it through the <tt/proto.h/ header file.
638 <sect1>Object Identifiers<label id="oid">
641 When you refer to object identifiers in your application, you need to
642 be aware that SR and Z39.50 use two different set of OIDs to refer to
643 the same objects. To handle this easily, <bf/YAZ/ provides a utility module
644 to <bf/ASN/ which provides an internal representation of the OIDs used in
645 both protocols. Each oid is described by a structure:
648 typedef struct oident
650 enum oid_proto proto;
651 enum oid_class class;
652 enum oid_value value;
653 int oidsuffix[OID_SIZE];
658 The <tt/proto/ field can be set to either <tt/PROTO_SR/ or
659 <tt/PROTO_Z3950/. The <tt/class/ might be, say, <tt/CLASS_RECSYN/, and the
660 <tt/value/ might be <tt/VAL_USMARC/ for the USMARC record format. Functions
663 int *oid_ent_to_oid(struct oident *ent, int *dst);
664 struct oident *oid_getentbyoid(int *o);
667 are provided to map between object identifiers and database entries.
668 If you store a member of the <tt/oid_proto/ type in your association
669 state information, it's a simple matter, at runtime, to generate the
670 correct OID when you need it. For decoding, you can simply ignore the
671 proto field, or if you're strict, you can verify that your peer is
672 using the OID family from the correct protocol. The <tt/desc/ field is
673 a short, human-readable name for the PDU, useful mainly for diagnostic
677 NOTE: The old function oid_getoidbyent still exists but is
678 not thread safe. Use oid_ent_to_oid instead and pass an array of
683 NOTE: Plans are underway to merge the two protocols into a single
684 definition, with one set of object identifiers. When this happens, the
685 oid module will no longer be required to support protocol
686 independence, but it should still be useful as a simple OID database.
692 In order to achieve extensibility and adaptability to different
693 application domains, the new version of the protocol defines many
694 structures outside of the main ASN.1 specification, referencing them
695 through ASN.1 EXTERNAL constructs. To simplify the construction and access
696 to the externally referenced data, the <bf/ASN/ module defines a
697 specialized version of the EXTERNAL construct, called <tt/Z_External/.
701 typedef struct Z_External
703 Odr_oid *direct_reference;
704 int *indirect_reference;
709 Z_External_single = 0,
711 Z_External_arbitrary,
715 Z_External_explainRecord,
716 Z_External_resourceReport1,
717 Z_External_resourceReport2
725 Odr_any *single_ASN1_type;
726 Odr_oct *octet_aligned;
727 Odr_bitmask *arbitrary;
731 Z_ExplainRecord *explainRecord;
732 Z_ResourceReport1 *resourceReport1;
733 Z_ResourceReport2 *resourceReport2;
741 When decoding, the <bf/ASN/ module will attempt to determine which
742 syntax describes the data by looking at the reference fields
743 (currently only the direct-reference). For ASN.1 structured data, you
744 need only consult the <tt/which/ field to determine the type of data.
745 You can the access the data directly through the union. When
746 constructing data for encoding, you set the union pointer to point to
747 the data, and set the <tt/which/ field accordingly. Remember also to
748 set the direct (or indirect) reference to the correct OID for the data
749 type. For non-ASN.1 data such as MARC records, use the
750 <tt/octet_aligned/ arm of the union.
752 Some servers return ASN.1 structured data values (eg. database
753 records) as BER-encoded records placed in the <tt/octet-aligned/
754 branch of the EXTERNAL CHOICE. The ASN-module will <it/not/
755 automatically decode these records. To help you decode the records in
756 the application, the function
759 Z_ext_typeent *z_ext_gettypebyref(oid_value ref);
762 Can be used to retrieve information about the known, external data
763 types. The function return a pointer to a static area, or NULL, if no
764 match for the given direct reference is found. The <tt/Z_ext_typeent/
768 typedef struct Z_ext_typeent
770 oid_value dref; /* the direct-reference OID value. */
771 int what; /* discriminator value for the external CHOICE */
772 Odr_fun fun; /* decoder function */
776 The <tt/what/ member contains the Z_External union discriminator value
777 for the given type: For the SUTRS record syntax, the value would be
778 <tt/Z_External_sutrs/. The <tt/fun/ member contains a pointer to the
779 function which encodes/decodes the given type. Again, for the SUTRS
780 record syntax, the value of <tt/fun/ would be <tt/z_SUTRS/ (a function
783 If you receive an EXTERNAL which contains an octet-string value that
784 you suspect of being an ASN.1-structured data value, you can use
785 <tt/z_ext_gettypebyref/ to look for the provided direct-reference. If
786 the return value is different from NULL, you can use the provided
787 function to decode the BER string (see section <ref id="odr-use"
790 If you want to <it/send/ EXTERNALs containing ASN.1-structured values
791 in the occtet-aligned branch of the CHOICE, this is possible too.
792 However, on the encoding phase, it requires a somewhat involved
793 juggling around of the various buffers involved.
795 If you need to add new, externally defined data types, you must update
796 the struct above, in the source file <tt/prt-ext.h/, as well as the
797 encoder/decoder in the file <tt/prt-ext.c/. When changing the latter,
798 remember to update both the <tt/arm/ arrary and the list <tt/type_table/,
799 which drives the CHOICE biasing that is necessary to tell the
800 different, structured types apart on decoding.
803 NOTE: Eventually, the EXTERNAL processing will most likely
804 automatically insert the correct OIDs or indirect-refs. First,
805 however, we need to determine how application-context management
806 (specifically the presentation-context-list) should fit into the
810 <sect1>PDU Contents Table
813 We include, for reference, a listing of the fields of each top-level
814 PDU, as well as their default settings.
819 Field Type Default value
821 referenceId Z_ReferenceId NULL
822 protocolVersion Odr_bitmask Empty bitmask
823 options Odr_bitmask Empty bitmask
824 preferredMessageSize int 30*1024
825 maximumRecordSize int 30*1024
826 idAuthentication Z_IdAuthentication NULL
827 implementationId char* "YAZ (id=81)"
828 implementationName char* "Index Data/YAZ"
829 implementationVersion char* YAZ_VERSION
830 userInformationField Z_UserInformation NULL
831 otherInfo Z_OtherInformation NULL
835 Field Type Default value
837 referenceId Z_ReferenceId NULL
838 protocolVersion Odr_bitmask Empty bitmask
839 options Odr_bitmask Empty bitmask
840 preferredMessageSize int 30*1024
841 maximumRecordSize int 30*1024
843 implementationId char* "YAZ (id=81)"
844 implementationName char* "Index Data/YAZ"
845 implementationVersion char* YAZ_VERSION
846 userInformationField Z_UserInformat.. NULL
847 otherInfo Z_OtherInformation NULL
851 Field Type Default value
853 referenceId Z_ReferenceId NULL
854 smallSetUpperBound int 0
855 largeSetLowerBound int 1
856 mediumSetPresentNumber int 0
857 replaceIndicator bool_t TRUE
858 resultSetName char* "default"
859 num_databaseNames int 0
860 databaseNames char** NULL
861 smallSetElementSetNames Z_ElementSetNames NULL
862 mediumSetElementSetNames Z_ElementSetNames NULL
863 preferredRecordSyntax Odr_oid NULL
865 additionalSearchInfo Z_OtherInformation NULL
866 otherInfo Z_OtherInformation NULL
870 Field Type Default value
872 referenceId Z_ReferenceId NULL
874 numberOfRecordsReturned int 0
875 nextResultSetPosition int 0
876 searchStatus bool_t TRUE
877 resultSetStatus int NULL
878 presentStatus int NULL
879 records Z_Records NULL
880 additionalSearchInfo Z_OtherInformation NULL
881 otherInfo Z_OtherInformation NULL
885 Field Type Default value
887 referenceId Z_ReferenceId NULL
888 resultSetId char* "default"
889 resultSetStartPoint int 1
890 numberOfRecordsRequested int 10
892 additionalRanges Z_Range NULL
893 recordComposition Z_RecordComposition NULL
894 preferredRecordSyntax Odr_oid NULL
895 maxSegmentCount int NULL
896 maxRecordSize int NULL
897 maxSegmentSize int NULL
898 otherInfo Z_OtherInformation NULL
902 Field Type Default value
904 referenceId Z_ReferenceId NULL
905 numberOfRecordsReturned int 0
906 nextResultSetPosition int 0
907 presentStatus int Z_PRES_SUCCESS
908 records Z_Records NULL
909 otherInfo Z_OtherInformation NULL
911 Z_DeleteResultSetRequest
912 ------------------------
913 Field Type Default value
915 referenceId Z_ReferenceId NULL
916 deleteFunction int Z_DeleteRequest_list
918 resultSetList char** NULL
919 otherInfo Z_OtherInformation NULL
921 Z_DeleteResultSetResponse
922 -------------------------
923 Field Type Default value
925 referenceId Z_ReferenceId NULL
926 deleteOperationStatus int Z_DeleteStatus_success
928 deleteListStatuses Z_ListStatus** NULL
929 numberNotDeleted int NULL
930 num_bulkStatuses int 0
931 bulkStatuses Z_ListStatus NULL
932 deleteMessage char* NULL
933 otherInfo Z_OtherInformation NULL
937 Field Type Default value
939 referenceId Z_ReferenceId NULL
940 num_databaseNames int 0
941 databaseNames char** NULL
942 attributeSet Odr_oid NULL
943 termListAndStartPoint Z_AttributesPlus... NULL
945 numberOfTermsRequested int 20
946 preferredPositionInResponse int NULL
947 otherInfo Z_OtherInformation NULL
951 Field Type Default value
953 referenceId Z_ReferenceId NULL
955 scanStatus int Z_Scan_success
956 numberOfEntriesReturned int 0
957 positionOfTerm int NULL
958 entries Z_ListEntris NULL
959 attributeSet Odr_oid NULL
960 otherInfo Z_OtherInformation NULL
962 Z_TriggerResourceControlRequest
963 -------------------------------
964 Field Type Default value
966 referenceId Z_ReferenceId NULL
967 requestedAction int Z_TriggerResourceCtrl_resou..
968 prefResourceReportFormat Odr_oid NULL
969 resultSetWanted bool_t NULL
970 otherInfo Z_OtherInformation NULL
972 Z_ResourceControlRequest
973 ------------------------
974 Field Type Default value
976 referenceId Z_ReferenceId NULL
977 suspendedFlag bool_t NULL
978 resourceReport Z_External NULL
979 partialResultsAvailable int NULL
980 responseRequired bool_t FALSE
981 triggeredRequestFlag bool_t NULL
982 otherInfo Z_OtherInformation NULL
984 Z_ResourceControlResponse
985 -------------------------
986 Field Type Default value
988 referenceId Z_ReferenceId NULL
989 continueFlag bool_t TRUE
990 resultSetWanted bool_t NULL
991 otherInfo Z_OtherInformation NULL
993 Z_AccessControlRequest
994 ----------------------
995 Field Type Default value
997 referenceId Z_ReferenceId NULL
998 which enum Z_AccessRequest_simpleForm;
1000 otherInfo Z_OtherInformation NULL
1002 Z_AccessControlResponse
1003 -----------------------
1004 Field Type Default value
1006 referenceId Z_ReferenceId NULL
1007 which enum Z_AccessResponse_simpleForm
1009 diagnostic Z_DiagRec NULL
1010 otherInfo Z_OtherInformation NULL
1014 Field Type Default value
1016 referenceId Z_ReferenceId NULL
1017 numberOfRecordsReturned int value=0
1018 num_segmentRecords int 0
1019 segmentRecords Z_NamePlusRecord NULL
1020 otherInfo Z_OtherInformation NULL
1024 Field Type Default value
1026 referenceId Z_ReferenceId NULL
1027 closeReason int Z_Close_finished
1028 diagnosticInformation char* NULL
1029 resourceReportFormat Odr_oid NULL
1030 resourceFormat Z_External NULL
1031 otherInfo Z_OtherInformation NULL
1034 <sect>Supporting Tools
1037 In support of the service API - primarily the ASN module, which
1038 provides the programmatic interface to the Z39.50 APDUs, YAZ contains
1039 a collection of tools that support the development of applications.
1041 <sect1>Query Syntax Parsers
1044 Since the type-1 (RPN) query structure has no direct, useful string
1045 representation, every origin application needs to provide some form of
1046 mapping from a local query notation or representation to a
1047 <tt/Z_RPNQuery/ structure. Some programmers will prefer to construct
1048 the query manually, perhaps using <tt/odr_malloc()/ to simplify memory
1049 management. The YAZ distribution includes two separate,
1050 query-generating tools that may be of use to you.
1052 <sect2>Prefix Query Format<label id="PQF">
1055 Since RPN or reverse polish notation is really just a fancy way of
1056 describing a suffix notation format (operator follows operands), it
1057 would seem that the confusion is total when we now introduce a prefix
1058 notation for RPN. The reason is one of simple laziness - it's somewhat
1059 simpler to interpret a prefix format, and this utility was designed
1060 for maximum simplicity, to provide a baseline representation for use
1061 in simple test applications and scripting environments (like Tcl). The
1062 demonstration client included with YAZ uses the PQF.
1064 The PQF is defined by the pquery module in the YAZ library. The
1065 <tt/pquery.h/ file provides the declaration of the functions
1068 Z_RPNQuery *p_query_rpn (ODR o, oid_proto proto, const char *qbuf);
1070 Z_AttributesPlusTerm *p_query_scan (ODR o, oid_proto proto,
1071 Odr_oid **attributeSetP, const char *qbuf);
1073 int p_query_attset (const char *arg);
1076 The function <tt/p_query_rpn()/ takes as arguments an <bf/ODR/ stream
1077 (see section <ref id="odr" name="The ODR Module">) to provide a memory
1078 source (the structure created is released on the next call to
1079 <tt/odr_reset()/ on the stream/), a protocol identifier (one of the
1080 constants <tt/PROTO_Z3950/ and <tt/PROTO_SR/), an attribute set
1081 reference, and finally a null-terminated string holding the query
1084 If the parse went well, <tt/p_query_rpn()/ returns a pointer to a
1085 <tt/Z_RPNQuery/ structure which can be placed directly into a
1086 <tt/Z_SearchRequest/.
1088 The <tt/p_query_attset/ specifies which attribute set to use if
1089 the query doesn't specify one by the <tt/@attrset/ operator. The
1090 <tt/p_query_attset/ returns 0 if the argument is a valid attribute
1091 set specifier; otherwise the function returns -1.
1093 The grammar of the PQF is as follows:
1096 Query ::= [ AttSet ] QueryStruct.
1100 QueryStruct ::= { Attribute } Simple | Complex.
1102 Attribute ::= '@attr' AttributeType '=' AttributeValue.
1104 AttributeType ::= integer.
1106 AttributeValue ::= integer.
1108 Complex ::= Operator QueryStruct QueryStruct.
1110 Operator ::= '@and' | '@or' | '@not' | '@prox' Proximity.
1112 Simple ::= ResultSet | Term.
1114 ResultSet ::= '@set' string.
1116 Term ::= string | '"' string '"'.
1118 Proximity ::= Exclusion Distance Ordered Relation WhichCode UnitCode.
1120 Exclusion ::= '1' | '0' | 'void'.
1122 Distance ::= integer.
1124 Ordered ::= '1' | '0'.
1126 Relation ::= integer.
1128 WhichCode ::= 'known' | 'private' | integer.
1130 UnitCode ::= integer.
1133 You will note that the syntax above is a fairly faithful
1134 representation of RPN, except for the <tt/Attibute/, which has been
1135 moved a step away from the term, allowing you to associate one or more
1136 attributes with an entire query structure. The parser will
1137 automatically apply the given attributes to each term as required.
1139 The following are all examples of valid queries in the PQF.
1146 @or "dylan" "zimmerman"
1150 @or @and bob dylan @set Result-1
1152 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
1154 @attr 4=1 @attr 1=4 "self portrait"
1156 @prox 0 3 1 2 k 2 dylan zimmerman
1159 <sect2>Common Command Language<label id="CCL">
1162 Not all users enjoy typing in prefix query structures and numerical
1163 attribute values, even in a minimalistic test client. In the library
1164 world, the more intuitive Common Command Language (or ISO 8777) has
1165 enjoyed some popularity - especially before the widespread
1166 availability of graphical interfaces. It is still useful in
1167 applications where you for some reason or other need to provide a
1168 symbolic language for expressing boolean query structures.
1170 The EUROPAGATE research project working under the Libraries programme
1171 of the European Commission's DG XIII has, amongst other useful tools,
1172 implemented a general-purpose CCL parser which produces an output
1173 structure that can be trivially converted to the internal RPN
1174 representation of YAZ (The <tt/Z_RPNQuery/ structure). Since the CCL
1175 utility - along with the rest of the software produced by EUROPAGATE -
1176 is made freely available on a liberal license, it is included as a
1181 The CCL parser obeys the following grammar for the FIND argument.
1182 The syntax is annotated by in the lines prefixed by <tt/--/.
1185 CCL-Find ::= CCL-Find Op Elements
1188 Op ::= "and" | "or" | "not"
1189 -- The above means that Elements are separated by boolean operators.
1191 Elements ::= '(' CCL-Find ')'
1194 | Qualifiers Relation Terms
1195 | Qualifiers Relation '(' CCL-Find ')'
1196 | Qualifiers '=' string '-' string
1197 -- Elements is either a recursive definition, a result set reference, a
1198 -- list of terms, qualifiers followed by terms, qualifiers followed
1199 -- by a recursive definition or qualifiers in a range (lower - upper).
1201 Set ::= 'set' = string
1202 -- Reference to a result set
1204 Terms ::= Terms Prox Term
1206 -- Proximity of terms.
1208 Term ::= Term string
1210 -- This basically means that a term may include a blank
1212 Qualifiers ::= Qualifiers ',' string
1214 -- Qualifiers is a list of strings separated by comma
1216 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
1217 -- Relational operators. This really doesn't follow the ISO8777
1221 -- Proximity operator
1225 The following queries are all valid:
1236 (dylan and bob) or set=1
1239 Assuming that the qualifiers <tt/ti/, <tt/au/ and <tt/date/
1240 are defined we may use:
1244 au=(bob dylan and slow train coming)
1246 date>1980 and (ti=((self portrait)))
1249 <sect3>CCL Qualifiers
1252 Qualifiers are used to direct the search to a particular searchable
1253 index, such as title (ti) and author indexes (au). The CCL standard
1254 itself doesn't specify a particular set of qualifiers, but it does
1255 suggest a few short-hand notations. You can customize the CCL parser
1256 to support a particular set of qualifiers to relect the current target
1257 profile. Traditionally, a qualifier would map to a particular
1258 use-attribute within the BIB-1 attribute set. However, you could also
1259 define qualifiers that would set, for example, the
1260 structure-attribute.
1262 Consider a scenario where the target support ranked searches in the
1263 title-index. In this case, the user could specify
1265 ti,ranked=knuth computer
1267 and the <tt/ranked/ would map to structure=free-form-text
1268 (4=105) and the <tt/ti/ would map to title (1=4).
1270 A "profile" with a set predefined CCL qualifiers can be read from a
1271 file. The YAZ client reads its CCL qualifiers from a file named
1272 <tt/default.bib/. Each line in the file has the form:
1274 <it/qualifier-name/ <it/type/=<it/val/ <it/type/=<it/val/ ...
1276 where <it/qualifier-name/ is the name of the qualifier to be used
1277 (eg. <tt/ti/), <it/type/ is a BIB-1 category type and <it/val/ is the
1278 corresponding BIB-1 attribute value. The <it/type/ can be either
1279 numeric or it may be either <tt/u/ (use), <tt/r/ (relation), <tt/p/
1280 (position), <tt/s/ (structure), <tt/t/ (truncation) or <tt/c/
1281 (completeness). The <it/qualifier-name/ <tt/term/ has a special
1282 meaning. The types and values for this definition is used when <it/no/
1283 qualifier is present.
1285 Consider the following definition:
1291 Two qualifiers are defined, <tt/ti/ and <tt/au/. They both set the
1292 structure-attribute to phrase (1). <tt/ti/ sets the use-attribute to
1293 4. <tt/au/ sets the use-attribute to 1. When no qualifiers are used
1294 in the query the structure-attribute is set to free-form-text (105).
1299 All public definitions can be found in the header file <tt/ccl.h/.
1300 A profile identifier is of type <tt/CCL_bibset/. A profile must be
1301 created with the call to the function <tt/ccl_qual_mk/ which returns
1302 a profile handle of type <tt/CCL_bibset/.
1304 To read a file containing qualifier definitions the function
1305 <tt/ccl_qual_file/ may be convenient. This function takes an already
1306 opened <tt/FILE/ handle pointer as argument along with a
1307 <tt/CCL_bibset/ handle.
1309 To parse a simple string with a FIND query use the function
1311 struct ccl_rpn_node *ccl_find_str (CCL_bibset bibset, const char *str,
1312 int *error, int *pos);
1314 which takes the CCL profile (<tt/bibset/) and query (<tt/str/) as
1315 input. Upon successful completion the RPN tree is returned. If an
1316 error eccur, such as a syntax error, the integer pointed to by
1317 <tt/error/ holds the error code and <tt/pos/ holds the offset inside
1318 query string in which the parsing failed.
1320 An english representation of the error may be obtained by calling
1321 the <tt/ccl_err_msg/ function. The error codes are listed in
1324 To convert the CCL RPN tree (type <tt/struct ccl_rpn_node */) to the
1325 Z_RPNQuery of YAZ the function <tt/ccl_rpn_query/ must be used. This
1326 function which is part of YAZ is implemented in <tt/yaz-ccl.c/.
1327 After calling this function the CCL RPN tree is probably no longer
1328 needed. The <tt/ccl_rpn_delete/ destroys the CCL RPN tree.
1330 A CCL profile may be destroyed by calling the <tt/ccl_qual_rm/
1333 The token names for the CCL operators may be changed by setting the
1334 globals (all type <tt/char */) <tt/ccl_token_and/, <tt/ccl_token_or/,
1335 <tt/ccl_token_not/ and <tt/ccl_token_set/.
1336 An operator may have aliases, i.e. there may be more than one name for
1337 the operator. To do this, separate each alias with a space character.
1339 <sect1>Object Identifiers
1342 The basic YAZ representation of an OID is an array of integers,
1343 terminated with the value -1. The <bf/ODR/ module provides two
1344 utility-functions to create and copy this type of data elements:
1347 Odr_oid *odr_getoidbystr(ODR o, char *str);
1350 Creates an OID based on a string-based representation using dots (.)
1351 to separate elements in the OID.
1354 Odr_oid *odr_oiddup(ODR odr, Odr_oid *o);
1357 Creates a copy of the OID referenced by the <it/o/ parameter. Both
1358 functions take an <bf/ODR/ stream as parameter. This stream is used to
1359 allocate memory for the data elements, which is released on a
1360 subsequent call to <tt/odr_reset()/ on that stream.
1362 The <bf/OID/ module provides a higher-level representation of the
1363 family of object identifers which describe the Z39.50 protocol and its
1364 related objects. The definition of the module interface is given in
1365 the <tt/oid.h/ file.
1367 The interface is mainly based on the <tt/oident/ structure. The
1368 definition of this structure looks like this:
1371 typedef struct oident
1376 int oidsuffix[OID_SIZE];
1381 The <it/proto/ field takes one of the values
1388 If you don't care about talking to SR-based implementations (few
1389 exist, and they may become fewer still if and when the ISO SR and ANSI
1390 Z39.50 documents are merged into a single standard), you can ignore
1391 this field on incoming packages, and always set it to PROTO_Z3950
1392 for outgoing packages.
1394 The <it/oclass/ field takes one of the values
1414 corresponding to the OID classes defined by the Z39.50 standard.
1416 Finally, the <it/value/ field takes one of the values
1472 again, corresponding to the specific OIDs defined by the standard.
1474 The <it/desc/ field contains a brief, mnemonic name for the OID in
1480 struct oident *oid_getentbyoid(int *o);
1483 takes as argument an OID, and returns a pointer to a static area
1484 containing an <tt/oident/ structure. You typically use this function
1485 when you receive a PDU containing an OID, and you wish to branch out
1486 depending on the specific OID value.
1491 int *oid_ent_to_oid(struct oident *ent, int *dst);
1494 Takes as argument an <tt/oident/ structure - in which the <it/proto/,
1495 <it/oclass/, and <it/value/ fields are assumed to be set correctly -
1496 and returns a pointer to a the buffer as given by <it/dst/
1498 representation of the corresponding OID. The function returns
1499 NULL and the array dst is unchanged if a mapping couldn't place.
1500 The array <it/dst/ should be at least of size <tt>OID_SIZE</tt>.
1502 The <tt/oid_ent_to_oid()/ function can be used whenever you need to
1503 prepare a PDU containing one or more OIDs. The separation of the
1504 <it/protocol/ element from the remainer of the OID-description makes
1505 it simple to write applications that can communicate with either
1506 Z39.50 or OSI SR-based applications.
1511 oid_value oid_getvalbyname(const char *name);
1514 takes as argument a mnemonic OID name, and returns the <it/value/
1515 field of the first entry in the database that contains the given name
1516 in its <it/desc/ field.
1518 Finally, the module provides the following utility functions, whose
1519 meaning should be obvious:
1522 void oid_oidcpy(int *t, int *s);
1523 void oid_oidcat(int *t, int *s);
1524 int oid_oidcmp(int *o1, int *o2);
1525 int oid_oidlen(int *o);
1529 NOTE: The <bf/OID/ module has been criticized - and perhaps rightly so
1530 - for needlessly abstracting the
1531 representation of OIDs. Other toolkits use a simple
1532 string-representation of OIDs with good results. In practice, we have
1533 found the interface comfortable and quick to work with, and it is a
1534 simple matter (for what it's worth) to create applications compatible
1535 with both ISO SR and Z39.50. Finally, the use of the <tt/oident/
1536 database is by no means mandatory. You can easily create your
1537 own system for representing OIDs, as long as it is compatible with the
1538 low-level integer-array representation of the ODR module.
1541 <sect1>Nibble Memory
1544 Sometimes when you need to allocate and construct a large,
1545 interconnected complex of structures, it can be a bit of a pain to
1546 release the associated memory again. For the structures describing the
1547 Z39.50 PDUs and related structures, it is convenient to use the
1548 memory-management system of the <bf/ODR/ subsystem (see
1549 <ref id="odr-use" name="Using ODR">). However, in some circumstances
1550 where you might otherwise benefit from using a simple nibble memory
1551 management system, it may be impractical to use <tt/odr_malloc()/ and
1552 <bf/odr_reset()/. For this purpose, the memory manager which also
1553 supports the <bf/ODR/ streams is made available in the <bf/NMEM/
1554 module. The external interface to this module is given in the
1557 The following prototypes are given:
1560 NMEM nmem_create(void);
1561 void nmem_destroy(NMEM n);
1562 void *nmem_malloc(NMEM n, int size);
1563 void nmem_reset(NMEM n);
1564 int nmem_total(NMEM n);
1565 void nmem_init(void);
1568 The <tt/nmem_create()/ function returns a pointer to a memory control
1569 handle, which can be released again by <tt/nmem_destroy()/ when no
1570 longer needed. The function <tt/nmem_malloc()/ allocates a block of
1571 memory of the requested size. A call to <tt/nmem_reset()/ or
1572 <tt/nmem_destroy()/ will release all memory allocated on the handle
1573 since it was created (or since the last call to
1574 <tt/nmem_reset()/. The function <tt/nmem_total()/ returns the number
1575 of bytes currently allocated on the handle.
1577 Note that the nibble memory pool is shared amonst threads. Posix
1578 mutex'es and WIN32 Critical sections are introduced to keep the
1579 module thread safe. On WIN32 function <tt/nmem_init()/ initialises
1580 the Critical Section handle and should be called once before any
1581 other nmem function is used.
1583 <sect>The ODR Module<label id="odr">
1588 <bf/ODR/ is the BER-encoding/decoding subsystem of <bf/YAZ/. Care as been taken
1589 to isolate <bf/ODR/ from the rest of the package - specifically from the
1590 transport interface. <bf/ODR/ may be used in any context where basic
1591 ASN.1/BER representations are used.
1593 If you are only interested in writing a Z39.50 implementation based on
1594 the PDUs that are already provided with <bf/YAZ/, you only need to concern
1595 yourself with the section on managing ODR streams (section
1596 <ref id="odr-use" name="Using ODR">). Only if you need to
1597 implement ASN.1 beyond that which has been provided, should you
1598 worry about the second half of the documentation
1599 (section <ref id="odr-prog" name="Programming with ODR">). If you use
1600 one of the higher-level interfaces, you can skip this section entirely.
1602 This is important, so we'll repeat it for emphasis: <it>You do not
1603 need to read section <ref id="odr-prog" name="Programming with ODR"> to
1604 implement Z39.50 with <bf/YAZ/.</it>
1606 If you need a part of the protocol that isn't already in <bf/YAZ/, you
1608 contact the authors before going to work on it yourself: We
1609 might already be working on it. Conversely, if you implement a useful
1610 part of the protocol before us, we'd be happy to include it in a
1613 <sect1>Using ODR<label id="odr-use">
1619 Conceptually, the ODR stream is the source of encoded data in the
1620 decoding mode; when encoding, it is the receptacle for the encoded
1621 data. Before you can use an ODR stream it must be allocated. This is
1622 done with the function
1625 ODR odr_createmem(int direction);
1628 The <tt/odr_createmem()/ function takes as argument one of three manifest
1629 constants: <tt/ODR_ENCODE/, <tt/ODR_DECODE/, or <tt/ODR_PRINT/. An
1631 in only one mode - it is not possible to change its mode once it's
1632 selected. Typically, your program will allocate at least two ODR
1633 streams - one for decoding, and one for encoding.
1635 When you're done with the stream, you can use
1638 void odr_destroy(ODR o);
1641 to release the resources allocated for the stream.
1643 <sect2>Memory Management<label id="memory">
1646 Two forms of memory management take place in the ODR system. The first
1647 one, which has to do with allocating little bits of memory (sometimes
1648 quite large bits of memory, actually) when a protocol package is
1649 decoded, and turned into a complex of interlinked structures. This
1650 section deals with this system, and how you can use it for your own
1651 purposes. The next section deals with the memory management which is
1652 required when encoding data - to make sure that a large enough buffer is
1653 available to hold the fully encoded PDU.
1655 The <bf/ODR/ module has its own memory management system, which is
1656 used whenever memory is required. Specifically, it is used to allocate
1657 space for data when decoding incoming PDUs. You can use the memory
1658 system for your own purposes, by using the function
1661 void *odr_malloc(ODR o, int size);
1664 You can't use the normal <tt/free/(2) routine to free memory allocated
1665 by this function, and <bf/ODR/ doesn't provide a parallel function. Instead,
1669 void odr_reset(ODR o, int size);
1672 when you are done with the
1673 memory: Everything allocated since the last call to <tt/odr_reset()/ is
1674 released. The <tt/odr_reset()/ call is also required to clear up an
1675 error condition on a stream.
1680 int odr_total(ODR o);
1683 returns the number of bytes allocated on the stream since the last call to
1686 The memory subsystem of <bf/ODR/ is fairly efficient at allocating and
1687 releasing little bits of memory. Rather than managing the individual,
1688 small bits of space, the system maintains a freelist of larger chunks
1689 of memory, which are handed out in small bits. This scheme is
1690 generally known as a <it/nibble memory/ system. It is very useful for
1691 maintaing short-lived constructions such as protocol PDUs.
1693 If you want to retain a bit of memory beyond the next call to
1694 <tt/odr_reset()/, you can use the function
1697 ODR_MEM odr_extract_mem(ODR o);
1700 This function will give you control of the memory recently allocated
1701 on the ODR stream. The memory will live (past calls to <tt/odr_reset()/),
1702 until you call the function
1705 void odr_release_mem(ODR_MEM p);
1708 The opaque <tt/ODR_MEM/ handle has no other purpose than referencing
1709 the memory block for you until you want to release it.
1711 You can use <tt/odr_extract_mem()/ repeatedly between allocating data,
1712 to retain individual control of separate chunks of data.
1714 <sect2>Encoding and Decoding Data
1717 When encoding data, the ODR stream will write the encoded octet string
1718 in an internal buffer. To retrieve the data, use the function
1721 char *odr_getbuf(ODR o, int *len, int *size);
1724 The integer pointed to by len is set to the length of the encoded
1725 data, and a pointer to that data is returned. *<tt/size/ is set to the size
1726 of the buffer (unless <tt/size/ is null, signalling that you are
1727 not interested in the size). The next call to a primitive function
1728 using the same ODR stream will overwrite the data, unless a different
1729 buffer has been supplied using the call
1732 void odr_setbuf(ODR o, char *buf, int len, int can_grow);
1735 which sets the encoding (or decoding) buffer used by <tt/o/ to
1737 using the length <tt/len/. Before a call to an encoding function,
1738 you can use <tt/odr_setbuf()/ to provide the stream with an encoding buffer
1739 of sufficient size (length). The <tt/can_grow/ parameter tells the encoding
1740 ODR stream whether it is allowed to use <tt/realloc/(2) to increase the size
1741 of the buffer when necessary. The default condition of a new encoding
1742 stream is equivalent to the results of calling
1745 odr_setbuf(stream, 0, 0, 1);
1748 In this case, the stream will allocate and reallocate memory as
1749 necessary. The stream reallocates memory by repeatedly doubling the
1750 size of the buffer - the result is that the buffer will typically
1751 reach its maximum, working size with only a small number of reallocation
1752 operations. The memory is freed by the stream when the latter is destroyed,
1753 unless it was assigned by the user with the <tt/can_grow/
1754 parameter set to zero (in this case, you are expected to retain
1755 control of the memory yourself).
1757 To assume full control of an encoded buffer, you must first call
1758 <tt/odr_getbuf()/ to fetch the buffer and its length. Next, you should
1759 call <tt/odr_setbuf()/ to provide a different buffer (or a null
1760 pointer) to the stream. In the simplest case, you will reuse the same
1761 buffer over and over again, and you will just need to call
1762 <tt/odr_getbuf()/ after each encoding operation to get the length and
1763 address of the buffer. Note that the stream may reallocate the buffer
1764 during an encoding operation, so it is necessary to retrieve the
1765 correct address after each encoding operation.
1767 It is important to realise that the ODR stream will not release this
1768 memory when you call <tt/odr_reset()/: It will merely update its
1769 internal pointers to prepare for the encoding of a new data value.
1770 When the stream is released by the <tt/odr_destroy()/ function, the
1771 memory given to it by odr_setbuf will be released <it/only/ if the
1772 <tt/can_grow/ parameter to <tt/odr_setbuf()/ was nonzero. The
1773 <tt/can_grow/ parameter, in other words, is a way of signalling who
1774 is to own the buffer, you or the ODR stream. If you never call
1775 <tt/odr_setbuf()/ on your encoding stream, which is typically the
1776 case, the buffer allocated by the stream will belong to the stream by
1779 When you wish to decode data, you should first call <tt/odr_setbuf()/, to
1780 tell the decoding stream where to find the encoded data, and how long
1781 the buffer is (the <tt/can_grow/ parameter is ignored by a decoding
1782 stream). After this, you can call the function corresponding to the
1783 data you wish to decode (eg, <tt/odr_integer()/ odr <tt/z_APDU()/).
1785 Examples of encoding/decoding functions:
1788 int odr_integer(ODR o, int **p, int optional, const char *name);
1790 int z_APDU(ODR o, Z_APDU **p, int optional, const char *name);
1793 If the data is absent (or
1794 doesn't match the tag corresponding to the type), the return value
1795 will be either 0 or 1 depending on the <tt/optional/ flag. If
1797 is 0 and the data is absent, an error flag will be raised in the
1798 stream, and you'll need to call <tt/odr_reset()/ before you can use the
1799 stream again. If <tt/optional/ is nonzero, the pointer <it/pointed to/ by
1800 <tt/p/ will be set to the null value, and the function will return 1.
1801 The <tt/name/ argument is used to pretty-print the tag in question.
1802 It may be set to <tt/NULL/ if pretty-printing is not desired.
1804 If the data value is found where it's expected, the pointer
1805 <it/pointed to/ by the <tt/p/ argument will be set to point to the
1807 space for the type will be allocated and owned by the ODR stream, and
1808 it will live until you call <tt/odr_reset()/ on the stream. You cannot use
1809 <tt/free/(2) to release the memory. You can decode several data elements
1810 (by repeated calls to <tt/odr_setbuf()/ and your decoding function), and
1811 new memory will be allocated each time. When you do call
1813 everything decoded since the last call to <tt/odr_reset()/ will be
1816 The use of the double indirection can be a little confusing at first
1817 (its purpose will become clear later on, hopefully),
1818 so an example is in order. We'll encode an integer value, and
1819 immediately decode it again using a different stream. A useless, but
1820 informative operation.
1823 void do_nothing_useful(int value)
1826 int *valp, *resvalp;
1830 /* allocate streams */
1831 if (!(encode = odr_createmem(ODR_ENCODE)))
1833 if (!(decode = odr_createmem(ODR_DECODE)))
1837 if (odr_integer(encode, &ero;valp, 0, 0) == 0)
1839 printf("encoding went bad\n");
1842 bufferp = odr_getbuf(encode, &ero;len);
1843 printf("length of encoded data is %d\n", len);
1845 /* now let's decode the thing again */
1846 odr_setbuf(decode, bufferp, len);
1847 if (odr_integer(decode, &ero;resvalp, 0, 0) == 0)
1849 printf("decoding went bad\n");
1852 printf("the value is %d\n", *resvalp);
1855 odr_destroy(encode);
1856 odr_destroy(decode);
1860 This looks like a lot of work, offhand. In practice, the ODR streams
1861 will typically be allocated once, in the beginning of your program (or at the
1862 beginning of a new network session), and the encoding and decoding
1863 will only take place in a few, isolated places in your program, so the
1864 overhead is quite manageable.
1869 The encoding/decoding functions all return 0 when an error occurs.
1870 Until you call <tt/odr_reset()/, you cannot use the stream again, and
1871 any function called will immediately return 0.
1873 To provide information to the programmer or administrator, the
1877 void odr_perror(ODR o, char *message);
1880 is provided, which prints the <tt/message/ argument to <tt/stderr/
1881 along with an error message from the stream.
1883 You can also use the function
1886 int odr_geterror(ODR o);
1889 to get the current error number from the screen. The number will be
1890 one of these constants:
1893 <tag/OMEMORY/Memory allocation failed.
1894 <tag/OSYSERR/A system- or library call has failed. The standard
1895 diagnostic variable <tt/errno/
1896 should be examined to determine the actual error.
1897 <tag/OSPACE/No more space for encoding. This will only occur when the user has
1898 explicitly provided a buffer for an encoding stream without allowing
1899 the system to allocate more space.
1900 <tag/OREQUIRED/This is a common protocol error; A required data element was
1902 encoding or decoding.
1903 <tag/OUNEXPECTED/An unexpected data element was found during decoding.
1904 <tag/OOTHER/Other error. This is typically an indication of misuse of
1905 the <bf/ODR/ system by the programmer, and also that the diagnostic
1906 system isn't as good as it should be, yet.
1909 The character string array
1912 char *odr_errlist[]
1915 can be indexed by the error code to obtain a human-readable
1916 representation of the problem.
1918 <sect2>Summary and Synopsis
1924 ODR odr_createmem(int direction);
1926 void odr_destroy(ODR o);
1928 void odr_reset(ODR o);
1930 char *odr_getbuf(ODR o, int *len);
1932 void odr_setbuf(ODR o, char *buf, int len);
1934 void *odr_malloc(ODR o, int size);
1936 ODR_MEM odr_extract_mem(ODR o);
1938 void odr_release_mem(ODR_MEM r);
1940 int odr_geterror(ODR o);
1942 void odr_perror(char *message);
1944 extern char *odr_errlist[];
1947 <sect1>Programming with ODR<label id="odr-prog">
1950 The API of <bf/ODR/ is designed to reflect the structure of ASN.1, rather
1951 than BER itself. Future releases may be able to represent data in
1952 other external forms.
1954 The interface is based loosely on that of the Sun Microsystems XDR routines.
1955 Specifically, each function which corresponds to an ASN.1 primitive
1956 type has a dual function. Depending on the settings of the ODR
1957 stream which is supplied as a parameter, the function may be used
1958 either to encode or decode data. The functions that can be built
1959 using these primitive functions, to represent more complex datatypes, share
1960 this quality. The result is that you only have to enter the definition
1961 for a type once - and you have the functionality of encoding, decoding
1962 (and pretty-printing) all in one unit. The resulting C source code is
1963 quite compact, and is a pretty straightforward representation of the
1964 source ASN.1 specification. Although no ASN.1 compiler is supplied
1965 with <bf/ODR/ at this time, it shouldn't be too difficult to write one, or
1966 perhaps even to adapt an existing compiler to output <bf/ODR/ routines
1967 (not surprisingly, writing encoders/decoders using <bf/ODR/ turns out
1970 In many cases, the model of the XDR functions works quite well in this
1972 others, it is less elegant. Most of the hassle comes from the optional
1973 SEQUENCE memebers which don't exist in XDR.
1975 <sect2>The Primitive ASN.1 Types
1978 ASN.1 defines a number of primitive types (many of which correspond
1980 primitive types in structured programming languages, such as C).
1985 The <bf/ODR/ function for encoding or decoding (or printing) the ASN.1
1986 INTEGER type looks like this:
1989 int odr_integer(ODR o, int **p, int optional, const char *name);
1992 (we don't allow values that can't be contained in a C integer.)
1994 This form is typical of the primitive <bf/ODR/ functions. They are named
1995 after the type of data that they encode or decode. They take an
1997 stream, an indirect reference to the type in question, and an
1998 <tt/optional/ flag (corresponding to the OPTIONAL keyword of ASN.1) as
1999 parameters. They all return an integer value of either one or zero.
2000 When you use the primitive functions to construct encoders for complex
2001 types of your own, you should follow this model as well. This
2002 ensures that your new types can be reused as elements in yet more
2005 The <tt/o/ parameter should obviously refer to a properly
2007 stream of the right type (encoding/decoding/printing) for the
2008 operation that you wish to perform.
2010 When encoding or printing, the function first looks at *<tt/p/. If
2012 pointer pointed to by <tt/p/) is a null pointer, this is taken to mean that
2013 the data element is absent. If the <tt/optional/ parameter is nonzero,
2014 the function will return one (signifying success) without any further
2015 processing. If the <tt/optional/ is zero, an internal error flag is
2016 set in the ODR stream, and the function will return 0. No further
2017 operations can be carried out on the stream without a call to
2018 the function <tt/odr_reset()/.
2020 If *<tt/p/ is not a null pointer, it is expected to point to an instance of
2021 the data type. The data will be subjected to the encoding rules, and
2022 the result will be placed in the buffer held by the ODR stream.
2024 The other ASN.1 primitives have similar functions that operate in
2031 int odr_bool(ODR o, bool_t **p, int optional, const char *name);
2043 int odr_null(ODR o, bool_t **p, int optional, const char *name);
2046 In this case, the value of **p is not important. If *p is different
2047 from the null pointer, the null value is present, otherwise it's
2054 typedef struct odr_oct
2061 int odr_octetstring(ODR o, Odr_oct **p, int optional, const char *name);
2064 The <tt/buf/ field should point to the character array that holds the
2065 octetstring. The <tt/len/ field holds the actual length, while the
2067 field gives the size of the allocated array (not of interest to you,
2068 in most cases). The character array need not be null terminated.
2070 To make things a little easier, an alternative is given for string
2071 types that are not expected to contain embedded NULL characters (eg.
2075 int odr_cstring(ODR o, char **p, int optional, const char *name);
2078 Which encoded or decodes between OCTETSTRING representations and
2079 null-terminates C strings.
2081 Functions are provided for the derived string types, eg:
2084 int odr_visiblestring(ODR o, char **p, int optional, const char *name);
2091 int odr_bitstring(ODR o, Odr_bitmask **p, int optional, const char *name);
2094 The opaque type <tt/Odr_bitmask/ is only suitable for holding relatively brief bit
2095 strings, eg. for options fields, etc. The constant
2096 <tt/ODR_BITMASK_SIZE/
2097 multiplied by 8 gives the maximum possible number of bits.
2099 A set of macros are provided for manipulating the
2104 void ODR_MASK_ZERO(Odr_bitmask *b);
2106 void ODR_MASK_SET(Odr_bitmask *b, int bitno);
2108 void ODR_MASK_CLEAR(Odr_bitmask *b, int bitno);
2110 int ODR_MASK_GET(Odr_bitmask *b, int bitno);
2113 The functions are modelled after the manipulation functions that
2114 accompany the <tt/fd_set/ type used by the <tt/select/(2) call.
2115 <tt/ODR_MASK_ZERO/ should always be called first on a new bitmask, to
2116 initialize the bits to zero.
2118 <sect3>OBJECT IDENTIFIER
2122 int odr_oid(ODR o, Odr_oid **p, int optional, const char *name);
2125 The C OID represenation is simply an array of integers, terminated by
2126 the value -1 (the <tt/Odr_oid/ type is synonymous with the <tt/int/
2127 type). We suggest that you use the OID database module (see section
2128 <ref id="oid" name="Object Identifiers">) to handle object identifiers
2129 in your application.
2131 <sect2>Tagging Primitive Types<label id="tag-prim">
2134 The simplest way of tagging a type is to use the <tt/odr_implicit_tag()/ or
2135 <tt/odr_explicit_tag()/ macros:
2138 int odr_implicit_tag(ODR o, Odr_fun fun, int class, int tag, int
2139 optional, const char *name);
2141 int odr_explicit_tag(ODR o, Odr_fun fun, int class, int tag,
2142 int optional, const char *name);
2145 To create a type derived from the integer type by implicit tagging, you
2149 MyInt ::= [210] IMPLICIT INTEGER
2152 In the <bf/ODR/ system, this would be written like:
2155 int myInt(ODR o, int **p, int optional, const char *name)
2157 return odr_implicit_tag(o, odr_integer, p,
2158 ODR_CONTEXT, 210, optional, name);
2162 The function <tt/myInt()/ can then be used like any of the primitive
2163 functions provided by ODR. Note that the behavior of
2165 and <tt/odr_implicit()/ macros act exactly the same as the functions they
2166 are applied to - they respond to error conditions, etc, in the same
2167 manner - they simply have three extra parameters. The class parameter may
2168 take one of the values: <tt/ODR_CONTEXT/, <tt/ODR_PRIVATE/,
2169 <tt/ODR_UNIVERSAL/, or
2170 <tt/ODR_APPLICATION/.
2172 <sect2>Constructed Types
2175 Constructed types are created by combining primitive types. The
2177 system only implements the SEQUENCE and SEQUENCE OF constructions
2178 (although adding the rest of the container types should be simple
2179 enough, if the need arises).
2181 For implementing SEQUENCEs, the functions
2184 int odr_sequence_begin(ODR o, void *p, int size, const char *name);
2185 int odr_sequence_end(ODR o);
2190 The <tt/odr_sequence_begin()/ function should be called in the beginning of a
2191 function that implements a SEQUENCE type. Its parameters are the
2193 stream, a pointer (to a pointer to the type you're implementing), and
2194 the <tt/size/ of the type (typically a C structure). On encoding, it
2195 returns 1 if *<tt/p/ is a null pointer. The <tt/size/ parameter is ignored. On
2196 decoding, it returns 1 if the type is found in the data stream.
2198 bytes of memory are allocated, and *<tt/p/ is set to point to this space.
2199 <tt/odr_sequence_end()/ is called at the end of the complex function. Assume
2200 that a type is defined like this:
2203 MySequence ::= SEQUENCE {
2205 boolval BOOLEAN OPTIONAL }
2208 The corresponding ODR encoder/decoder function and the associated data
2209 structures could be written like this:
2212 typedef struct MySequence
2218 int mySequence(ODR o, MySequence **p, int optional, const char *name)
2220 if (odr_sequence_begin(o, p, sizeof(**p), name) == 0)
2221 return optional &ero;&ero; odr_ok(o);
2223 odr_integer(o, &ero;(*p)->intval, 0, "intval") &ero;&ero;
2224 odr_bool(o, &ero;(*p)->boolval, 1, "boolval") &ero;&ero;
2225 odr_sequence_end(o);
2229 Note the 1 in the call to <tt/odr_bool()/, to mark that the sequence
2230 member is optional. If either of the member types had been tagged, the
2231 macros <tt/odr_implicit()/ or <tt/odr_explicit()/ could have been used.
2233 function can be used exactly like the standard functions provided with
2234 <bf/ODR/. It will encode, decode or pretty-print a data value of the
2235 <tt/MySequence/ type. We like to name types with an initial capital, as
2236 done in ASN.1 definitions, and to name the corresponding function with
2237 the first character of the name in lower case. You could, of course,
2238 name your structures, types, and functions any way you please - as
2239 long as you're consistent, and your code is easily readable.
2241 just that - a predicate that returns the state of the stream. It is
2242 used to ensure that the behaviour of the new type is compatible with
2243 the interface of the primitive types.
2245 <sect2>Tagging Constructed Types
2249 NOTE: See section <ref id="tag-prim" name="Tagging Primitive types">
2250 for information on how to tag the primitive types, as well as types
2251 that are already defined.
2254 <sect3>Implicit Tagging
2257 Assume the type above had been defined as
2260 MySequence ::= [10] IMPLICIT SEQUENCE {
2262 boolval BOOLEAN OPTIONAL }
2265 You would implement this in <bf/ODR/ by calling the function
2268 int odr_implicit_settag(ODR o, int class, int tag);
2271 which overrides the tag of the type immediately following it. The
2272 macro <tt/odr_implicit()/ works by calling <tt/odr_implicit_settag()/
2274 before calling the function pointer argument. Your type function could
2278 int mySequence(ODR o, MySequence **p, int optional, const char *name)
2280 if (odr_implicit_settag(o, ODR_CONTEXT, 10) == 0 ||
2281 odr_sequence_begin(o, p, sizeof(**p), name) == 0)
2282 return optional &ero;&ero; odr_ok(o);
2284 odr_integer(o, &ero;(*p)->intval, 0, "intval") &ero;&ero;
2285 odr_bool(o, &ero;(*p)->boolval, 1, "boolval") &ero;&ero;
2286 odr_sequence_end(o);
2290 The definition of the structure <tt/MySequence/ would be the same.
2292 <sect3>Explicit Tagging
2295 Explicit tagging of constructed types is a little more complicated,
2296 since you are in effect adding a level of construction to the data.
2298 Assume the definition:
2301 MySequence ::= [10] IMPLICIT SEQUENCE {
2303 boolval BOOLEAN OPTIONAL }
2306 Since the new type has an extra level of construction, two new functions
2307 are needed to encapsulate the base type:
2310 int odr_constructed_begin(ODR o, void *p, int class, int tag,
2313 int odr_constructed_end(ODR o);
2316 Assume that the IMPLICIT in the type definition above were replaced
2317 with EXPLICIT (or that the IMPLICIT keyword were simply deleted, which
2318 would be equivalent). The structure definition would look the same,
2319 but the function would look like this:
2322 int mySequence(ODR o, MySequence **p, int optional, const char *name)
2324 if (odr_constructed_begin(o, p, ODR_CONTEXT, 10, name) == 0)
2325 return optional &ero;&ero; odr_ok(o);
2326 if (o->direction == ODR_DECODE)
2327 *p = odr_malloc(o, sizeof(**p));
2328 if (odr_sequence_begin(o, p, sizeof(**p), 0) == 0)
2330 *p = 0; /* this is almost certainly a protocol error */
2334 odr_integer(o, &ero;(*p)->intval, 0, "intval") &ero;&ero;
2335 odr_bool(o, &ero;(*p)->boolval, 1, "boolval") &ero;&ero;
2336 odr_sequence_end(o) &ero;&ero;
2337 odr_constructed_end(o);
2341 Notice that the interface here gets kind of nasty. The reason is
2342 simple: Explicitly tagged, constructed types are fairly rare in
2343 the protocols that we care about, so the
2344 aesthetic annoyance (not to mention the dangers of a cluttered
2345 interface) is less than the time that would be required to develop a
2346 better interface. Nevertheless, it is far from satisfying, and it's a
2347 point that will be worked on in the future. One option for you would
2348 be to simply apply the <tt/odr_explicit()/ macro to the first function,
2350 have to worry about <tt/odr_constructed_*/ yourself. Incidentally, as you
2351 might have guessed, the <tt/odr_sequence_/ functions are themselves
2352 implemented using the <tt/odr_constructed_/ functions.
2357 To handle sequences (arrays) of a apecific type, the function
2360 int odr_sequence_of(ODR o, int (*fun)(ODR o, void *p, int optional),
2361 void *p, int *num, const char *name);
2364 The <tt/fun/ parameter is a pointer to the decoder/encoder
2365 function of the type. <tt/p/ is a pointer to an array of pointers to your
2366 type. <tt/num/ is the number of elements in the array.
2371 MyArray ::= SEQUENCE OF INTEGER
2374 The C representation might be
2377 typedef struct MyArray
2384 And the function might look like
2387 int myArray(ODR o, MyArray **p, int optional, const char *name)
2389 if (o->direction == ODR_DECODE)
2390 *p = odr_malloc(o, sizeof(**p));
2391 if (odr_sequence_of(o, odr_integer, &ero;(*p)->elements,
2392 &ero;(*p)->num_elements, name))
2395 return optional &ero;&ero; odr_ok(o);
2402 The choice type is used fairly often in some ASN.1 definitions, so
2403 some work has gone into streamlining its interface.
2405 CHOICE types are handled by the function:
2408 int odr_choice(ODR o, Odr_arm arm[], void *p, void *whichp,
2412 The <tt/arm/ array is used to describe each of the possible types that the
2413 CHOICE type may assume. Internally in your application, the CHOICE
2414 type is represented as a discriminated union. That is, a C union
2415 accompanied by an integer (or enum) identifying the active 'arm' of
2416 the union. <tt/whichp/ is a pointer to the union discriminator. When
2417 encoding, it is examined to determine the current type. When decoding,
2418 it is set to reference the type that was found in the input stream.
2420 The Odr_arm type is defined thus:
2423 typedef struct odr_arm
2434 The interpretation of the fields are:
2437 <tag/tagmode/Either <tt/ODR_IMPLICIT/, <tt/ODR_EXPLICIT/, or <tt/ODR_NONE/
2440 <tag/class, tag/The class and tag of the type (-1 if no tagging is
2442 <tag/which/The value of the discriminator that corresponds to
2443 this CHOICE element. Typically, it will be a #defined constant, or
2445 <tag/fun/A pointer to a function that implements the type of
2446 the CHOICE member. It may be either a standard <bf/ODR/ type or a type
2447 defined by yourself.
2448 <tag/name/Name of tag.
2451 A handy way to prepare the array for use by the <tt/odr_choice()/ function
2453 define it as a static, initialized array in the beginning of your
2454 decoding/encoding function. Assume the type definition:
2457 MyChoice ::= CHOICE {
2459 tagged [99] IMPLICIT INTEGER,
2464 Your C type might look like
2467 typedef struct MyChoice
2484 And your function could look like this:
2487 int myChoice(ODR o, MyChoice **p, int optional, const char *name)
2489 static Odr_arm arm[] =
2491 {-1, -1, -1, MyChoice_untagged, odr_integer, "untagged"},
2492 {ODR_IMPLICIT, ODR_CONTEXT, 99, MyChoice_tagged, odr_integer,
2494 {-1, -1, -1, MyChoice_other, odr_boolean, "other"},
2498 if (o->direction == ODR_DECODE)
2499 *p = odr_malloc(o, sizeof(**p);
2501 return optional &ero;&ero; odr_ok(o);
2503 if (odr_choice(o, arm, &ero;(*p)->u, &ero;(*p)->which), name)
2506 return optional &ero;&ero; odr_ok(o);
2510 In some cases (say, a non-optional choice which is a member of a sequence),
2511 you can &dquot;embed&dquot; the union and its discriminator in the structure
2512 belonging to the enclosing type, and you won't need to fiddle with
2513 memory allocation to create a separate structure to wrap the
2514 discriminator and union.
2516 The corresponding function is somewhat nicer in the Sun XDR interface.
2517 Most of the complexity of this interface comes from the possibility of
2518 declaring sequence elements (including CHOICEs) optional.
2520 The ASN.1 specifictions naturally requires that each member of a
2521 CHOICE have a distinct tag, so they can be told apart on decoding.
2522 Sometimes it can be useful to define a CHOICE that has multiple types
2523 that share the same tag. You'll need some other mechanism, perhaps
2524 keyed to the context of the CHOICE type. In effect, we would like to
2525 introduce a level of context-sensitiveness to our ASN.1 specification.
2526 When encoding an internal representation, we have no problem, as long
2527 as each CHOICE member has a distinct discriminator value. For
2528 decoding, we need a way to tell the choice function to look for a
2529 specific arm of the table. The function
2532 void odr_choice_bias(ODR o, int what);
2535 provides this functionality. When called, it leaves a notice for the
2536 next call to <tt/odr_choice()/ to be called on the decoding
2537 stream <tt/o/ that only the <tt/arm/ entry with a <tt/which/ field
2538 equal to <tt/what/ should be tried.
2540 The most important application (perhaps the only one, really) is in
2541 the definition of application-specific EXTERNAL encoders/decoders
2542 which will automatically decode an ANY member given the direct or
2548 The protocol modules are suffering somewhat from a lack of diagnostic
2549 tools at the moment. Specifically ways to pretty-print PDUs that
2550 aren't recognized by the system. We'll include something to this end
2551 in a not-too-distant release. In the meantime, what we do when we get
2552 packages we don't understand is to compile the ODR module with
2553 <tt/ODR_DEBUG/ defined. This causes the module to dump tracing
2554 information as it processes data units. With this output and the
2555 protocol specification (Z39.50), it is generally fairly easy to see
2558 <sect>The COMSTACK Module<label id="comstack">
2560 <sect1>Synopsis (blocking mode)
2568 int size = 0, length_incoming;
2569 char *protocol_package;
2570 int protocol_package_length;
2571 char server_address[] = "myserver.com:2100";
2574 stack = cs_create(tcpip_type, 1, PROTO_Z3950);
2576 perror("cs_create"); /* note use of perror() here since we have no stack yet */
2580 status = cs_connect(stack, server_address);
2582 cs_perror(stack, "cs_connect");
2586 status = cs_put(stack, protocol_package, protocol_package_length);
2588 cs_perror(stack, "cs_put");
2592 /* Now get a response */
2594 length_incoming = cs_get(stack, &buf, &size);
2595 if (!length_incoming) {
2596 fprintf(stderr, "Connection closed\n");
2598 } else if (length_incoming < 0) {
2599 cs_perror(stack, "cs_get");
2603 /* Do stuff with buf here */
2616 subsystem provides a transparent interface to different types of transport
2617 stacks for the exchange of BER-encoded data. At present, the
2618 RFC1729 method (BER over TCP/IP), and Peter Furniss' XTImOSI
2619 stack are supported, but others may be added in time. The philosophy of the
2620 module is to provide a simple interface by hiding unused options and
2621 facilities of the underlying libraries. This is always done at the risk
2622 of losing generality, and it may prove that the interface will need
2625 The interface is implemented in such a fashion that only the
2626 sub-layers constructed to the transport methods that you wish to
2627 use in your application are linked in.
2629 You will note that even though simplicity was a goal in the design,
2630 the interface is still orders of magnitudes more complex than the
2631 transport systems found in many other packages. One reason is that
2632 the interface needs to support the somewhat different requirements of
2633 the different lower-layer communications stacks; another important reason is
2634 that the interface seeks to provide a more or less industrial-strength
2635 approach to asynchronous event-handling. When no function is allowed
2636 to block, things get more complex - particularly on the server
2637 side. We urge you to have a look at the demonstration client and server
2638 provided with the package. They are meant to be easily readable and
2639 instructive, while still being at least moderately useful.
2641 <sect1>Common Functions
2643 <sect2>Managing Endpoints
2647 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
2650 Creates an instance of the protocol stack - a communications endpoint.
2653 parameter determines the mode of communication. At present, the
2658 are recognized. The function returns a null-pointer if a system error
2659 occurs. The <tt/blocking/ parameter should be one if you wish the
2660 association to operate in blocking mode, zero otherwise. The
2661 <tt/protocol/ field should be one of <tt/PROTO_SR/ or <tt/PROTO_Z3950/.
2664 int cs_close(COMSTACK handle);
2667 Closes the connection (as elegantly as the lower layers will permit),
2668 and releases the resouces pointed to by the
2672 should not be referenced again after this call.
2676 We really need a soft disconnect, don't we?
2679 <sect2>Data Exchange
2683 int cs_put(COMSTACK handle, char *buf, int len);
2688 down the wire. In blocking mode, this function will return only when a
2689 full buffer has been written, or an error has occurred. In nonblocking
2690 mode, it's possible that the function will be unable to send the full
2691 buffer at once, which will be indicated by a return value of 1. The
2692 function will keep track of the number of octets already written; you
2693 should call it repeatedly with the same values of <tt/buf/ and
2694 <tt/len/, until the buffer has been transmitted. When a full buffer
2695 has been sent, the function will return 0 for success. -1 indicates
2696 an error condition (see below).
2699 int cs_get(COMSTACK handle, char **buf, int *size);
2702 Receives a PDU from the peer. Returns the number of bytes
2703 read. In nonblocking mode, it is possible that not all of the packet can be
2704 read at once. In this case, the function returns 1. To simplify the
2705 interface, the function is
2706 responsible for managing the size of the buffer. It will be reallocated
2707 if necessary to contain large packages, and will sometimes be moved
2708 around internally by the subsystem when partial packages are read. Before
2711 for the fist time, the buffer can be initialized to the null pointer,
2712 and the length should also be set to 0 - cs_get will perform a
2714 on the buffer for you. When a full buffer has been read, the size of
2715 the package is returned (which will always be greater than 1). -1
2716 indicates an error condition.
2723 int cs_more(COMSTACK handle);
2726 The <tt/cs_more()/ function should be used in conjunction with
2730 The <tt/cs_get()/ function will sometimes (notably in the TCP/IP mode) read
2731 more than a single protocol package off the network. When this
2732 happens, the extra package is stored by the subsystem. After calling
2733 <tt/cs_get()/, and before waiting for more input,
2734 You should always call
2736 to check if there's a full protocol package already read. If
2740 can be used to immediately fetch the new package. For the
2742 subsystem, the function should always return 0, but if you want your
2743 stuff to be protocol independent, you should use it.
2746 NOTE: The <tt/cs_more()/
2747 function is required because the
2749 does not provide a way of separating individual PDUs, short of
2750 partially decoding the BER. Some other implementations will carefully
2751 nibble at the packet by calling
2753 several times. This was felt to be too inefficient (or at least
2754 clumsy) - hence the call for this extra function.
2758 int cs_look(COMSTACK handle);
2761 This function is useful when you're operating in nonblocking
2764 tells you there's something happening on the line. It returns one of
2765 the following values:
2768 <tag/CS_NONE/No event is pending. The data found on the line was
2769 not a complete package.
2770 <tag/CS_CONNECT/A response to your connect request has been received. Call
2772 to process the event and to finalize the connection establishment.
2773 <tag/CS_DISCON/The other side has closed the connection (or maybe
2775 request - but do we care? Maybe later). Call
2776 <tt/cs_close/ To close your end of the association as well.
2777 <tag/CS_LISTEN/A connect request has been received. Call
2779 to process the event.
2780 <tag/CS_DATA/There's data to be found on the line. Call
2787 You should be aware that even if
2789 tells you that there's an event event pending, the corresponding
2790 function may still return and tell you there was nothing to be found.
2791 This means that only part of a package was available for reading. The
2792 same event will show up again, when more data has arrived.
2796 int cs_fileno(COMSTACK h);
2799 Returns the file descriptor of the association. Use this when
2800 file-level operations on the endpoint are required (<tt/select/(2)
2801 operations, specifically).
2807 int cs_connect(COMSTACK handle, void *address);
2810 Initiate a connection with the target at <tt/address/ (more on
2811 addresses below). The function will return 0 on success, and 1 if
2812 the operation does not complete immediately (this will only
2813 happen on a nonblocking endpoint). In this case, use
2814 <tt/cs_rcvconnect/ to complete the operation, when <tt/select/(2)
2815 reports input pending on the association.
2818 int cs_rcvconnect(COMSTACK handle);
2821 Complete a connect operation initiated by <tt/cs_connect()/. It will
2822 return 0 on success; 1 if the operation has not yet completed (in this
2823 case, call the function again later); -1 if an error has occured.
2828 To establish a server under the <tt/inetd/ server, you can use
2831 COMSTACK cs_createbysocket(int socket, CS_TYPE type, int blocking,
2835 The <it/socket/ parameter is an established socket (when your
2836 application is invoked from <tt/inetd/, the socket will typically be
2837 0. The following parameters are identical to the ones for
2841 int cs_bind(COMSTACK handle, void *address, int mode)
2844 Binds a local address to the endpoint. Read about addresses below. The
2845 <tt/mode/ parameter should be either <tt/CS_CLIENT/ or <tt/CS_SERVER/.
2848 int cs_listen(COMSTACK handle, char *addr, int *addrlen);
2851 Call this to process incoming events on an endpoint that has been
2852 bound in listening mode. It will return 0 to indicate that the connect
2853 request has been received, 1 to signal a partial reception, and -1 to
2854 indicate an error condition.
2857 COMSTACK cs_accept(COMSTACK handle);
2860 This finalises the server-side association establishment, after
2861 cs_listen has completed successfully. It returns a new connection
2862 endpoint, which represents the new association. The application will
2863 typically wish to fork off a process to handle the association at this
2864 point, and continue listen for new connections on the old <tt/handle/.
2866 You can use the call
2869 char *cs_addrstr(COMSTACK);
2872 on an established connection to retrieve the hostname of the remote host.
2874 <it>NOTE: You may need to use this function with some care if your
2875 name server service is slow or unreliable</it>
2880 The low-level format of the addresses are different depending on the
2881 mode of communication you have chosen. A function is provided by each
2882 of the lower layers to map a user-friendly string-form address to the
2883 binary form required by the lower layers.
2886 struct sockaddr_in *tcpip_strtoaddr(char *str);
2888 struct netbuf *mosi_strtoaddr(char *str);
2891 The format for TCP/IP addresses is straightforward:
2894 <host> [ ':' <portnum> ]
2897 The <tt/hostname/ can be either a domain name or an IP address. The
2898 port number, if omitted, defaults to 210.
2900 For OSI, the format is
2903 [ <t-selector> '/' ] <host> [ ':' <port> ]
2906 The transport selector is given as an even number of hex digits.
2908 You'll note that the address format for the OSI mode are just a subset
2909 of full presentation addresses. We use presentation addresses because
2910 xtimosi doesn't, in itself, allow access to the X.500 Directory
2911 service. We use a limited form, because we haven't yet come across an
2912 implementation that used more of the elements of a full p-address. It
2913 is a fairly simple matter to add the rest of the elements to the
2914 address format as needed, however: Xtimosi <it/does/ support the full
2915 P-address structure.
2917 In both transport modes, the special hostname &dquot;@&dquot; is mapped
2918 to any local address (the manifest constant INADDR_ANY). It is used
2919 to establish local listening endpoints in the server role.
2921 When a connection has been established, you can use
2924 char cs_addrstr(COMSTACK h);
2927 to retrieve the host name of the peer system. The function returns a pointer
2928 to a static area, which is overwritten on the next call to the function.
2931 NOTE: We have left the issue of X.500 name-to-address mapping open, for the
2932 moment. It would be a simple matter to provide a table-based mapping,
2933 if desired. Alternately, we could use the X.500 client-function that
2934 is provided with the ISODE (although this would defeat some of the
2935 purpose of using ThinOSI in the first place. We have been told that it
2936 should be within the realm of the possible to implement a lightweight
2937 implementation of the necessary X.500 client capabilities on top of
2938 ThinOSI. This would be the ideal solution, we feel. On the other hand, it
2939 still remains to be seen just what role the Directory will play in a world
2940 populated by ThinOSI and other pragmatic solutions.
2946 All functions return -1 if an error occurs. Typically, the functions
2947 will return 0 on success, but the data exchange functions
2948 (<tt/cs_get/, <tt/cs_put/, <tt/cs_more/)
2949 follow special rules. Consult their descriptions.
2951 When a function (including the data exchange functions) reports an
2955 to determine the cause of the
2956 problem. The function
2959 void cs_perror(COMSTACK handle char *message);
2966 argument, along with a system message, to
2968 Use the character array
2971 extern const char *cs_errlist[];
2974 to get hold of the message, if you want to process it differently.
2978 const char *cs_stackerr(COMSTACK handle);
2981 Returns an error message from the lower layer, if one has been
2984 <sect1>Enabling OSI Communication
2986 <sect2>Installing Xtimosi
2988 Although you will have to download Peter Furniss' XTI/mOSI
2989 implementation for yourself, we've tried to make the integration as
2992 The latest version of xtimosi will generally be under
2995 ftp://pluto.ulcc.ac.uk/ulcc/thinosi/xtimosi/
2998 When you have downloaded and unpacked the archive, it will (we assume)
2999 have created a directory called <tt/xtimosi/. We suggest that you place
3000 this directory <it/in the same directory/ where you unpacked the
3002 distribution. This way, you shouldn't have to fiddle with the
3003 makefiles of <tt/YAZ/ beyond uncommenting a few lines.
3005 Go to <tt>xtimosi/src</tt>, and type &dquot;<tt/make libmosi.a/&dquot;.
3006 This should generally
3007 create the library, ready to use.
3012 The currently available release of xtimosi has some inherent
3013 problems that make it disfunction on certain platforms - eg. the
3014 Digital OSF/1 workstations. It is supposedly primarily a
3015 compiler problem, and we hope to see a release that is generally
3016 portable. While we can't guarantee that it can be brought to work
3017 on your platform, we'll be happy to talk to you about problems
3018 that you might see, and relay information to the author of the
3019 software. There are some signs that the <bf/gcc/ compiler is more
3020 likely to produce a fully functional library, but this hasn't
3021 been verified (we think that the problem is limited to the use
3022 of hexadecimal escape-codes used in strings, which are silently
3023 ignored by some compilers).
3027 A problem has been encountered in the communication with
3028 ISODE-based applications. If the ISODE presentation-user calls
3029 <tt/PReadRequest()/ with a timeout value different from <tt/OK/ or <tt/NOTOK/,
3030 he will get an immediate TIMEOUT abort when receiving large (>2041
3031 bytes, which is the SPDU-size that the ISODE likes to work with) packages
3032 from an xtimosi-based implementation (probably most
3033 other implementations as well, in fact). It seems to be a flaw in the
3034 ISODE API, and the workaround (for ISODE users) is to either not
3035 use an explicit timeout (switching to either blocking or
3036 nonblocking mode), or to check that the timer really has expired
3037 before closing the connection.
3040 The next step in the installation is to modify the makefile in the toplevel
3042 directory. The place to change is in the top of the file, and is
3043 clearly marked with a comment.
3045 Now run <tt/make/ in the <bf/YAZ/ toplevel directory (do a
3046 &dquot;<tt/make clean/&dquot; first, if the
3047 system has been previously made without OSI support). Use the <bf/YAZ/
3049 and <bf/client/ demo programs to verify that OSI communication works OK. Then,
3050 you can go ahead and try to talk to other implementations.
3053 NOTE: Our interoperability experience is limited to version
3054 7 of the Nordic SR-Nett package, which has had several
3055 protocol errors fixed from the earlier releases. If you have
3056 problems or successes in interoperating with other
3057 implementations, we'd be glad to hear about it, or to help
3058 you make things work, as our resources allow.
3061 If you write your own applications based on <bf/YAZ/, and you wish to
3062 include OSI support, the procedure is equally simple. You should
3063 include the <tt/xmosi.h/ header file in addition to <tt/comstack.h/.
3065 will define the manifest constant <tt/mosi_type/, which you should pass
3066 to the <tt/cs_create()/ function. In
3067 addition, you should use the function <tt/mosi_strtoaddr()/ rather than
3068 <tt/tcpip_strtoaddr()/ when you need to prepare an address.
3070 When you link your application, you should include (after the
3072 library) the <tt/libmosi.a/ library, and the <tt/librfc.a/ library
3074 <bf/YAZ/ (for OSI transport).
3076 As always, it can be very useful, if not essential, to have a look at the
3077 example applications
3078 to see how things are done.
3080 <sect2>OSI Transport
3083 Xtimosi requires an implementation of the OSI transport service under
3084 the X/OPEN XTI API. We provide an implementation of the RFC1006
3085 encapsulation of OSI/TP0 in TCP/IP (through the Berkeley Sockets API),
3086 as an independent part of <bf/YAZ/ (it's found under the <tt/rfc1006/
3087 directory). If you have access to an OSI transport provider under XTI,
3088 you should be able to make that work too, although it may require
3089 tinkering with the <tt/mosi_strtoaddr()/ function.
3091 <sect2>Presentation Context Management
3094 To simplify the implementation, we use Peter Furniss' alternative (PRF)
3096 for the Control of the presentation negotiation phase. This format
3097 is enabled by default when you
3100 The current version of <bf/YAZ/ does <it/not/ support presentation-layer
3101 negotiation of response record formats. The primary reason is that we
3102 have had access to no other SR or Z39.50 implementations over OSI that
3104 method. Secondarily, we believe that the EXPLAIN facility is a superior
3105 mechanism for relaying target capabilities in this respect. This is not to
3106 say that we have no intentions of supporting presentation context
3107 negotiation - we have just hitherto given it a lower priority than other
3108 aspects of the protocol.
3110 One thing is certain: The addition of this capability to <bf/YAZ/ should
3111 have only a minimal impact on existing applications, and on the
3112 interface to the software in general. Most likely, we will add an extra
3113 layer of interface to the processing of EXPLAIN records, which will
3114 convert back and forth between <tt/oident/ records (see section
3115 <ref id="oid" name="Object Identifiers">) and direct or indirect
3116 references, given the current association setup. Implementations based
3117 on any of the higher-level interfaces will most likely not have to be
3120 <sect1>Summary and Synopsis
3124 #include <comstack.h>
3126 #include <tcpip.h> /* this is for TCP/IP support */
3127 #include <xmosi.h> /* and this is for mOSI support */
3129 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
3131 COMSTACK cs_createbysocket(int s, CS_TYPE type, int blocking,
3134 int cs_bind(COMSTACK handle, int mode);
3136 int cs_connect(COMSTACK handle, void *address);
3138 int cs_rcvconnect(COMSTACK handle);
3140 int cs_listen(COMSTACK handle);
3142 COMSTACK cs_accept(COMSTACK handle);
3144 int cs_put(COMSTACK handle, char *buf, int len);
3146 int cs_get(COMSTACK handle, char **buf, int *size);
3148 int cs_more(COMSTACK handle);
3150 int cs_close(COMSTACK handle);
3152 int cs_look(COMSTACK handle);
3154 struct sockaddr_in *tcpip_strtoaddr(char *str);
3156 struct netbuf *mosi_strtoaddr(char *str);
3158 extern int cs_errno;
3160 void cs_perror(COMSTACK handle char *message);
3162 const char *cs_stackerr(COMSTACK handle);
3164 extern const char *cs_errlist[];
3167 <sect>Making an IR Interface for Your Database with YAZ<label id="server">
3173 NOTE: If you aren't into documentation, a good way to learn how the
3174 backend interface works is to look at the backend.h file. Then,
3175 look at the small dummy-server in server/ztest.c. Finally, you can
3176 have a look at the seshigh.c file, which is where most of the
3177 logic of the frontend server is located. The backend.h file also
3178 makes a good reference, once you've chewed your way through
3179 the prose of this file.
3182 If you have a database system that you would like to make available by
3183 means of Z39.50/SR, <bf/YAZ/ basically offers your two options. You
3184 can use the APIs provided by the <bf/ASN/, <bf/ODR/, and <bf/COMSTACK/
3186 create and decode PDUs, and exchange them with a client. Using this
3187 low-level interface gives you access to all fields and options of the
3188 protocol, and you can construct your server as close to your existing
3189 database as you like. It is also a fairly involved process, requiring
3190 you to set up an event-handling mechanism, protocol state machine,
3191 etc. To simplify server implementation, we have implemented a compact
3192 and simple, but reasonably full-functioned server-frontend that will
3193 handle most of the protocol mechanics, while leaving you to
3194 concentrate on your database interface.
3197 NOTE: The backend interface was designed in anticipation of a specific
3198 integration task, while still attempting to achieve some degree of
3199 generality. We realise fully that there are points where the
3200 interface can be improved significantly. If you have specific
3201 functions or parameters that you think could be useful, send us a
3202 mail (or better, sign on to the mailing list referred to in the
3203 toplevel README file). We will try to fit good suggestions into future
3204 releases, to the extent that it can be done without requiring
3205 too many structural changes in existing applications.
3208 <sect1>The Database Frontend
3211 We refer to this software as a generic database frontend. Your
3212 database system is the <it/backend database/, and the interface between
3213 the two is called the <it/backend API/. The backend API consists of a
3214 small number of function prototypes and structure definitions. You are
3215 required to provide the <bf/main()/ routine for the server (which can be
3216 quite simple), as well as functions to match each of the prototypes.
3217 The interface functions that you write can use any mechanism you like
3218 to communicate with your database system: You might link the whole
3219 thing together with your database application and access it by
3220 function calls; you might use IPC to talk to a database server
3221 somewhere; or you might link with third-party software that handles
3222 the communication for you (like a commercial database client library).
3223 At any rate, the functions will perform the tasks of:
3226 <item>Initialization.
3228 <item>Fetching records.
3229 <item>Scanning the database index (if you wish to implement SCAN).
3232 (more functions will be added in time to support as much of
3233 Z39.50-1995 as possible).
3235 Because the model where pipes or sockets are used to access the backend
3236 database is a fairly common one, we have added a mechanism that allows this
3237 communication to take place asynchronously. In this mode, the frontend
3238 server doesn't have to block while the backend database is processing
3239 a request, but can wait for additional PDUs from the client.
3241 <sect1>The Backend API
3244 The headers files that you need to use the interface are in the
3245 include/ directory. They are called <tt/statserv.h/ and <tt/backend.h/. They
3246 will include other files from the <tt/include/ directory, so you'll
3247 probably want to use the -I option of your compiler to tell it where
3248 to find the files. When you run <tt/make/ in the toplevel <bf/YAZ/ directory,
3249 everything you need to create your server is put the lib/libyaz.a
3250 library. If you want OSI as well, you'll also need to link in the
3251 <tt/libmosi.a/ library from the xtimosi distribution (see the mosi.txt
3252 file), a well as the <tt>lib/librfc.a</tt> library (to provide OSI transport
3255 <sect1>Your main() Routine
3258 As mentioned, your <bf/main()/ routine can be quite brief. If you want to
3259 initialize global parameters, or read global configuration tables,
3260 this is the place to do it. At the end of the routine, you should call
3263 <tscreen><verb>int statserv_main(int argc, char **argv);</verb></tscreen>
3265 <bf/Statserv_main/ will establish listening sockets according to the
3266 parameters given. When connection requests are received, the event
3267 handler will typically <bf/fork()/ to handle the new request. If you do use
3268 global variables, you should be aware, then, that these cannot be
3269 shared between associations, unless you explicitly disallow forking by
3270 command line parameters (we advise against this for any purposes
3271 except debugging, as a crash or hang in the server process will affect
3272 all users currently signed on to the server).
3274 The server provides a mechanism for controlling some of its behavior
3275 without using command-line options. The function
3278 statserv_options_block *statserv_getcontrol(void);
3281 Will return a pointer to a <tt/struct statserv_options_block/ describing
3282 the current default settings of the server. The structure contains
3286 <tag/int dynamic/A boolean value, which determines whether the server
3287 will fork on each incoming request (TRUE), or not (FALSE). Default is
3289 <tag/int loglevel/Set this by ORing the constants defined in include/log.h.
3290 <tag/char logfile[ODR_MAXNAME+1]/File for diagnostic output
3291 (&dquot;&dquot;: stderr).
3292 <tag/char apdufile[ODR_MAXNAME+1]/Name of file for logging incoming and
3293 outgoing APDUs (&dquot;&dquot;: don't log APDUs, &dquot;-&dquot;: <tt/stderr/).
3294 <tag/char default_listen[1024]/Same form as the command-line
3295 specification of listener address. &dquot;&dquot;: no default listener address.
3296 Default is to listen at &dquot;tcp:@:9999&dquot;. You can only
3297 specify one default listener address in this fashion.
3298 <tag/enum oid_proto default_proto;/Either <tt/PROTO_SR/ or <tt/PROTO_Z3950/.
3299 Default is <tt/PROTO_Z39_50/.
3300 <tag/int idle_timeout;/Maximum session idletime, in minutes. Zero indicates
3301 no (infinite) timeout. Default is 120 minutes.
3302 <tag/int maxrecordsize;/Maximum permissible record (message) size. Default
3303 is 1Mb. This amount of memory will only be allocated if a client requests a
3304 very large amount of records in one operation (or a big record). Set it
3306 if you are worried about resource consumption on your host system.
3307 <tag/char configname[ODR_MAXNAME+1]/Passed to the backend when a
3308 new connection is received.
3309 <tag/char setuid[ODR_MAXNAME+1]/Set user id to the user specified,
3310 after binding the listener addresses.
3313 The pointer returned by <tt/statserv_getcontrol/ points to a static area.
3314 You are allowed to change the contents of the structure, but the
3315 changes will not take effect before you call
3318 void statserv_setcontrol(statserv_options_block *block);
3321 Note that you should generally update this structure <it/before/ calling
3322 <tt/statserv_main()/.
3324 <sect1>The Backend Functions
3327 For each service of the protocol, the backend interface declares one or
3328 two functions. You are required to provide implementations of the
3329 functions representing the services that you wish to implement.
3331 <tscreen><verb>bend_initresult *bend_init(bend_initrequest *r);</verb></tscreen>
3333 This function is called once for each new connection request, after
3334 a new process has been forked, and an initRequest has been received
3335 from the client. The parameter and result structures are defined as
3339 typedef struct bend_initrequest
3344 typedef struct bend_initresult
3346 int errcode; /* 0==OK */
3347 char *errstring; /* system error string or NULL */
3348 void *handle; /* private handle to the backend module */
3353 The <tt/configname/ of <tt/bend_initrequest/ is currently always set to
3354 &dquot;default-config&dquot;. We haven't had use for putting anything special in
3355 the initrequest yet, but something might go there if the need arises
3356 (account/password info would be obvious).
3358 In general, the server frontend expects that the <tt/bend_*result/
3359 pointer that you return is valid at least until the next call to a
3360 <tt/bend_* function/. This applies to all of the functions described
3361 herein. The parameter structure passed to you in the call belongs to
3362 the server frontend, and you should not make assumptions about its
3363 contents after the current function call has completed. In other
3364 words, if you want to retain any of the contents of a request
3365 structure, you should copy them.
3367 The <tt/errcode/ should be zero if the initialization of the backend went
3368 well. Any other value will be interpreted as an error. The
3369 <tt/errstring/ isn't used in the current version, but one option
3370 would be to stick it
3371 in the initResponse as a VisibleString. The <tt/handle/ is the most
3372 important parameter. It should be set to some value that uniquely
3373 identifies the current session to the backend implementation. It is
3374 used by the frontend server in any future calls to a backend function.
3375 The typical use is to set it to point to a dynamically allocated state
3376 structure that is private to your backend module.
3380 bend_searchresult *bend_search(void *handle, bend_searchrequest *r,
3382 bend_searchresult *bend_searchresponse(void *handle);
3384 typedef struct bend_searchrequest
3386 char *setname; /* name to give to this set */
3387 int replace_set; /* replace set, if it already exists */
3388 int num_bases; /* number of databases in list */
3389 char **basenames; /* databases to search */
3390 Z_Query *query; /* query structure */
3391 } bend_searchrequest;
3393 typedef struct bend_searchresult
3395 int hits; /* number of hits */
3396 int errcode; /* 0==OK */
3397 char *errstring; /* system error string or NULL */
3398 } bend_searchresult;
3402 The first thing to notice about the search request interface (as well
3403 as all of the following requests), is that it consists of two separate
3404 functions. The idea is to provide a simple facility for
3405 asynchronous communication with the backend server. When a
3406 searchrequest comes in, the server frontend will fill out the
3407 <tt/bend_searchrequest/ tructure, and call the <tt/bend_search
3408 function/. The <tt/fd/
3409 argument will point to an integer variable. If you are able to do
3410 asynchronous I/O with your database server, you should set *<tt/fd/ to the
3411 file descriptor you use for the communication, and return a null
3412 pointer. The server frontend will then <tt/select()/ on the *<tt/fd/,
3414 <tt/bend_searchresult/ when it sees that data is available. If you don't
3415 support asynchronous I/O, you should return a pointer to the
3416 <tt/bend_searchresult/ immediately, and leave *<tt/fd/ untouched. This
3417 construction is common to all of the <tt/bend_/ functions (except
3418 <tt/bend_init/). Note that you can choose to support this facility in none,
3419 any, or all of the <tt/bend_/ functions, and you can respond
3420 differently on each request at run-time. The server frontend will
3423 The <tt/bend_searchrequest/ is a fairly close approximation of a protocol
3424 searchRequest PDU. The <tt/setname/ is the resultSetName from the protocol. You
3425 are required to establish a mapping between the set name and whatever
3426 your backend database likes to use. Similarly, the <tt/replace_set/ is a
3427 boolean value corresponding to the resultSetIndicator field in the
3428 protocol. <tt>Num_bases/basenames</tt> is a length of/array of character
3429 pointers to the database names provided by the client. The <tt/query/ is the
3430 full query structure as defined in the protocol ASN.1 specification.
3431 It can be either of the possible query types, and it's up to you to
3432 determine if you can handle the provided query type. Rather than
3433 reproduce the C interface here, we'll refer you to the structure
3434 definitions in the file <tt>include/proto.h</tt>. If you want to look at the
3435 attributeSetId OID of the RPN query, you can either match it against
3436 your own internal tables, or you can use the <tt/oid_getentbyoid/ function
3437 provided by <bf/YAZ/.
3439 The result structure contains a number of hits, and an
3440 <tt>errcode/errstring</tt> pair. If an error occurs during the search, or if
3441 you're unhappy with the request, you should set the errcode to a value
3442 from the BIB-1 diagnostic set. The value will then be returned to the
3443 user in a nonsurrogate diagnostic record in the response. The
3444 <tt/errstring/, if provided, will go in the addinfo field. Look at the
3445 protocol definition for the defined error codes, and the suggested
3446 uses of the addinfo field.
3450 bend_fetchresult *bend_fetch(void *handle, bend_fetchrequest *r,
3452 bend_fetchresult *bend_fetchresponse(void *handle);
3454 typedef struct bend_fetchrequest
3456 char *setname; /* set name */
3457 int number; /* record number */
3459 } bend_fetchrequest;
3461 typedef struct bend_fetchresult
3463 char *basename; /* name of database that provided record */
3464 int len; /* length of record */
3465 char *record; /* record */
3466 int last_in_set; /* is it? */
3468 int errcode; /* 0==success */
3469 char *errstring; /* system error string or NULL */
3475 NOTE: The <tt/bend_fetchresponse()/ function is not yet supported
3476 in this version of the software. Your implementation of <tt/bend_fetch()/
3477 should always return a pointer to a <tt/bend_fetchresult/.
3480 The frontend server calls <tt/bend_fetch/ when it needs database records to
3481 fulfill a searchRequest or a presentRequest. The <tt/setname/ is simply the
3482 name of the result set that holds the reference to the desired record.
3483 The <tt/number/ is the offset into the set (with 1 being the first record
3484 in the set). The <tt/format/ field is the record format requested by the
3485 client (See section <ref id="oid" name="Object Identifiers">). The value
3486 <tt/VAL_NONE/ indicates that the client did not request a specific format.
3487 The <tt/stream/ argument is an <bf/ODR/ stream which should be used for
3488 allocating space for structured data records. The stream will be reset when
3489 all records have been assembled, and the response package has been transmitted.
3490 For unstructured data, the backend is responsible for maintaining a static
3491 or dynamic buffer for the record between calls.
3493 In the result structure, the <tt/basename/ is the name of the
3494 database that holds the
3495 record. <tt/Len/ is the length of the record returned, in bytes, and
3497 is a pointer to the record. <tt/Last_in_set/ should be nonzero only if the
3498 record returned is the last one in the given result set. <tt/Errcode/ and
3499 <tt/errstring/, if given, will currently be interpreted as a global error
3500 pertaining to the set, and will be returned in a
3501 nonSurrogateDiagnostic.
3503 <it>NOTE: This is silly. Add a flag to say which is which.</it>
3505 If the <tt/len/ field has the value -1, then <tt/record/ is assumed to point
3506 to a constructed data type. The <tt/format/ field will be used to determine
3507 which encoder should be used to serialize the data.
3510 NOTE: If your backend generates structured records, it should use
3511 <tt/odr_malloc()/ on the provided stream for allocating data: This allows
3512 the frontend server to keep track of the record sizes.
3515 The <tt/format/ field is mapped to an object identifier in the direct
3516 reference of the resulting EXTERNAL representation of the record.
3518 <it>NOTE: The current version of <bf/YAZ/ only supports the direct reference
3523 bend_deleteresult *bend_delete(void *handle, bend_deleterequest *r,
3525 bend_deleteresult *bend_deleteresponse(void *handle);
3527 typedef struct bend_deleterequest
3530 } bend_deleterequest;
3532 typedef struct bend_deleteresult
3534 int errcode; /* 0==success */
3535 char *errstring; /* system error string or NULL */
3536 } bend_deleteresult;
3541 NOTE: The &dquot;delete&dquot; function is not yet supported
3542 in this version of the software.
3546 NOTE: The delete set function definition is rather primitive, mostly
3548 have had no practical need for it as of yet. If someone wants
3549 to provide a full delete service, we'd be happy to add the
3550 extra parameters that are required. Are there clients out there
3551 that will actually delete sets they no longer need?
3556 bend_scanresult *bend_scan(void *handle, bend_scanrequest *r,
3558 bend_scanresult *bend_scanresponse(void *handle);
3560 typedef struct bend_scanrequest
3562 int num_bases; /* number of elements in databaselist */
3563 char **basenames; /* databases to search */
3564 Z_AttributesPlusTerm *term;
3565 int term_position; /* desired index of term in result list */
3566 int num_entries; /* number of entries requested */
3569 typedef struct bend_scanresult
3590 NOTE: The <tt/bend_scanresponse()/ function is not yet supported
3591 in this version of the software. Your implementation of <tt/bend_scan()/
3592 should always return a pointer to a <tt/bend_scanresult/.
3595 <sect1>Application Invocation
3598 The finished application has the following
3599 invocation syntax (by way of <tt/statserv_main()/):
3602 appname [-szSu -a apdufile -l logfile -v loglevel]
3603 [listener ...]
3609 <tag/-a/APDU file. Specify a file for dumping PDUs (for diagnostic purposes).
3610 The special name &dquot;-&dquot; sends output to <tt/stderr/.
3612 <tag/-S/Don't fork on connection requests. This is good for debugging, but
3613 not recommended for real operation: Although the server is
3614 asynchronous and non-blocking, it can be nice to keep a software
3615 malfunction (okay then, a crash) from affecting all current users.
3617 <tag/-s/Use the SR protocol.
3619 <tag/-z/Use the Z39.50 protocol (default). These two options complement
3620 eachother. You can use both multiple times on the same command
3621 line, between listener-specifications (see below). This way, you
3622 can set up the server to listen for connections in both protocols
3623 concurrently, on different local ports.
3625 <tag/-l/The logfile.
3627 <tag/-v/The log level. Use a comma-separated list of members of the set
3628 {fatal,debug,warn,log,all,none}.
3629 <tag/-u/Set user ID. Sets the real UID of the server process to that of the
3630 given user. It's useful if you aren't comfortable with having the
3631 server run as root, but you need to start it as such to bind a
3634 <tag/-w/Working directory.
3635 <tag/-i/Use this when running from the <tt/inetd/ server.
3637 <tag/-t/Idle session timeout, in minutes.
3639 <tag/-k/Maximum record size/message size, in kilobytes.
3643 A listener specification consists of a transport mode followed by a
3644 colon (:) followed by a listener address. The transport mode is
3645 either <tt/osi/ or <tt/tcp/.
3647 For TCP, an address has the form
3650 hostname | IP-number [: portnumber]
3653 The port number defaults to 210 (standard Z39.50 port).
3655 For osi, the address form is
3658 [t-selector /] hostname | IP-number [: portnumber]
3661 The transport selector is given as a string of hex digits (with an even
3662 number of digits). The default port number is 102 (RFC1006 port).
3670 osi:0402/dbserver.osiworld.com:3000
3674 In both cases, the special hostname &dquot;@&dquot; is mapped to
3675 the address INADDR_ANY, which causes the server to listen on any local
3676 interface. To start the server listening on the registered ports for
3677 Z39.50 and SR over OSI/RFC1006, and to drop root privileges once the
3678 ports are bound, execute the server like this (from a root shell):
3681 my-server -u daemon tcp:@ -s osi:@
3684 You can replace <tt/daemon/ with another user, eg. your own account, or
3685 a dedicated IR server account. <tt/my-server/ should be the name of your
3686 server application. You can test the procedure with the <tt/ztest/
3689 <sect1>Summary and Synopsis
3693 #include <backend.h>
3695 bend_initresult *bend_init(bend_initrequest *r);
3697 bend_searchresult *bend_search(void *handle, bend_searchrequest *r,
3700 bend_searchresult *bend_searchresponse(void *handle);
3702 bend_fetchresult *bend_fetch(void *handle, bend_fetchrequest *r,
3705 bend_fetchresult *bend_fetchresponse(void *handle);
3707 bend_scanresult *bend_scan(void *handle, bend_scanrequest *r, int *fd);
3709 bend_scanresult *bend_scanresponse(void *handle);
3711 bend_deleteresult *bend_delete(void *handle, bend_deleterequest *r,
3714 bend_deleteresult *bend_deleteresponse(void *handle);
3716 void bend_close(void *handle);
3719 <sect>Future Directions
3722 The software has been
3723 successfully ported to the Mac as well as Windows NT/95 - we'd like to
3724 test those ports better and make sure they work as they should.
3726 We have a new and better version of the frontend server on the drawing
3727 board. Resources and external commitments will govern when we'll be
3728 able to do something real with it. Fetures should include greater
3729 flexibility, greter support for access/resource control, and easy
3730 support for Explain (possibly with Zebra as an extra database engine).
3732 We now support all PDUs of Z39.50-1995. If there is one of the
3733 supporting structures that you need but can't find in the prt*.h
3734 files, send us a note; it may be on its way.
3736 The 'retrieval' module needs to be finalized and documented. We think
3737 it can form a useful resource for people dealing with complex record
3738 structures, but for now, you'll mostly have to chew through the code
3739 yourself to make use of it. Not acceptable.
3741 Other than that, YAZ generally moves in the directions which appear to
3742 make the most people happy (including ourselves, as prime users of the
3743 software). If there's something you'd like to see in here, then drop
3744 us a note and let's see what we can come up with.
3748 <sect1>Index Data Copyright
3751 Copyright © 1995-2000 Index Data.
3753 Permission to use, copy, modify, distribute, and sell this software and
3754 its documentation, in whole or in part, for any purpose, is hereby granted,
3757 1. This copyright and permission notice appear in all copies of the
3758 software and its documentation. Notices of copyright or attribution
3759 which appear at the beginning of any file must remain unchanged.
3761 2. The names of Index Data or the individual authors may not be used to
3762 endorse or promote products derived from this software without specific
3763 prior written permission.
3765 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
3766 EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
3767 WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
3768 IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
3769 INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES
3770 WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR
3771 NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF
3772 LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
3775 <sect1>Additional Copyright Statements
3778 The optional CCL query language interpreter is covered by the following
3781 Copyright © 1995, the EUROPAGATE consortium (see below).
3783 The EUROPAGATE consortium members are:
3785 University College Dublin
3786 Danmarks Teknologiske Videnscenter
3787 An Chomhairle Leabharlanna
3788 Consejo Superior de Investigaciones Cientificas
3790 Permission to use, copy, modify, distribute, and sell this software and
3791 its documentation, in whole or in part, for any purpose, is hereby granted,
3794 1. This copyright and permission notice appear in all copies of the
3795 software and its documentation. Notices of copyright or attribution
3796 which appear at the beginning of any file must remain unchanged.
3798 2. The names of EUROPAGATE or the project partners may not be used to
3799 endorse or promote products derived from this software without specific
3800 prior written permission.
3802 3. Users of this software (implementors and gateway operators) agree to
3803 inform the EUROPAGATE consortium of their use of the software. This
3804 information will be used to evaluate the EUROPAGATE project and the
3805 software, and to plan further developments. The consortium may use
3806 the information in later publications.
3808 4. Users of this software agree to make their best efforts, when
3809 documenting their use of the software, to acknowledge the EUROPAGATE
3810 consortium, and the role played by the software in their work.
3812 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
3813 EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
3814 WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
3815 IN NO EVENT SHALL THE EUROPAGATE CONSORTIUM OR ITS MEMBERS BE LIABLE
3816 FOR ANY SPECIAL, INCIDENTAL, INDIRECT OR CONSEQUENTIAL DAMAGES OF
3817 ANY KIND, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA
3818 OR PROFITS, WHETHER OR NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND
3819 ON ANY THEORY OF LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE
3820 USE OR PERFORMANCE OF THIS SOFTWARE.
3822 <sect>About Index Data
3825 Index Data is a consulting and software-development enterprise that
3826 specialises in library and information management systems. Our
3827 interests and expertise span a broad range of related fields, and one
3828 of our primary, long-term objectives is the development of a powerful
3829 information management
3830 system with open network interfaces and hypermedia capabilities.
3832 We make this software available free of charge, on a fairly unrestrictive
3833 license; as a service to the networking community, and to further the
3834 development of quality software for open network communication.
3836 We'll be happy to answer questions about the software, and about ourselves
3841 Købmagergade 43
3842 DK-1150 Copenhagen K
3847 Phone: +45 3341 0100
3849 Email: info@indexdata.dk
3852 The <it>Hacker's Jargon File</it> has the following to say about the
3854 prefix &dquot;YA&dquot; in the name of a software product.
3857 Yet Another. adj. 1. Of your own work: A
3858 humorous allusion often used in titles to acknowledge that the
3859 topic is not original, though the content is. As in &dquot;Yet Another
3860 AI Group&dquot; or &dquot;Yet Another Simulated Annealing Algorithm&dquot;.
3862 others' work: Describes something of which there are already far
projects / yaz-moved-to-github.git / blob