It seems wrong that in IRSpy we have a database of 3000 Z39.50 targets, but no easy way to make them available to MasterKey applications. Since Jason built the IRSpy toroid, which exposes IRSpy's ZeeRex database in Torus format, it seems that have had the pieces we need to present those 3000 databases to MasterKey administrators; but in reality there are several significant barriers to using the toroid output. 1. DATA QUALITY Of the 3000 or so records in the IRSpy database, some large proportion represent servers that no longer exist, or never existed, or for which we do not have access credentials. At present we can not even identify which records fall into this category, because the records in the database do not contain a reliability score (and therefore searching by the score is not possible): the score is calculated when records are displayed in the web UI, and so is "display-only" TASK: calculate reliability as part of the existing record-transformation process, store the score and index is, make it available as a search criterion. Once bad servers have been identified, we will need to decide what to do with them. One option of course is just to throw the records away, but I am always reluctant to discard information. It would probably be better to introduce a notion of status into the IRSpy database, mark the relevant records Dormant or similar, and arrange that the IRSpy toroid ignores all such records. TASK: add a notion of record status to the IRSpy database, supported in the Web UI and in Z39.50/SRU searches, and arrange for regular record maintenance to modify this setting for records as required. 2. OVERNIGHT TEST RUNS Every night, the IRSpy host runs a large series of tests on registered servers to determine which are still alive, whether their capabilities have changed, etc. One seventh of the server are tested each night, so that the database is traversed every week. For some time now, though, the overnight tests are failing -- sometimes quickly, sometimes after running for a long and just occasionally not at all -- due to an XML/XSLT problem: runtime error: file ../lib/ZOOM/../../xsl/irspy2zeerex.xsl line 174 element param xsltApplyXSLTTemplate: A potential infinite template recursion was detected. This error masks another that was starting to manifest with increasing frequency, to do with invalid characters in XML. These obviously need fixing, as the overnight runs are crucial for maintaining the quality of the data. TASK: fix the "infinite recursion" XSTL problem in the overnight run. TASK: fix the invalid XML character problem as soon as it is once more visible. 3. DATA RICHNESS The overlap between the set of data generated by IRSpy and what is required by MasterKey is surprisingly small: at present, the IRSpy Toroid provides only two fields in the records that it propagates: ZURL and displayName. Other MasterKey target description fields fall into four categories: * Some, such as authentication, could be propagated simply by extending the toroid's zeerex2torus.xsl transformation * Others pertain to information that is known to IRSpy, but which require additional business logic to extract: for example, cclmap_au could be set to @attr 1=1003 for targets that have been determined to support that access point, and @attr 1=1 for that that support this but not 1003, and left blank for others. Similarly, requestSyntax could be chosen from among those syntaxes that are supported. * Other fields specify information which could in principle be determined by IRSpy, but for which there are presently no tests -- for example, a carefully design test could probably determine what query encoding is in use in a given Z39.50 server, and what record encoding is used in returned records -- but no such tests have been created for IRSpy. * Finally, there are yet other MasterKey field that IRSpy could not even in principle hope to determine: for example, URL Recipe seems to be a lost cause, as it relates to web-sites that are "parallel" Z39.50/SRU servers as well as to those servers themselves. Such fields will need to be maintained by hand. Of the as-yet unsupported MasterKey fields, some have no equivalent in ZeeRex and are therefore not representable in ZeeRex records. IRSpy uses an extended ZeeRex scheme for its database, so this is not an immediate problem. In general, not only are more tests needed within IRSpy, but more intelligence is needed in transforming the data that IRSpy does discover into information that MasterKey can use (and that intelligence would perhaps not be best expressed in XSLT). 4. ADMIN CONSOLE'S HANDLING OF LARGE NUMBERS OF TARGETS Up till now, the MasterKey Admin Console has been used exclusively with small sets of targets, not exceeding 30 or perhaps 40. It has therefore been simplest to present all targets together on a single page -- a strategy that will certainly not work well when we start using the Admin Console to choose targets from IRSpy's much larger selection. (At a rough guess, perhaps half to two thirds of the registered servers in IRSpy are active and functional, so we are looking at a list of 1500-2000 targets.) To handle this, we would need to re-tool the Admin Console so that, as well as running in its current mode (which is still appropriate in many situations) it can also run in a mode where it does not show complete lists of targets, but invites administrators to search for specific targets; and in which it limits the number of results on each page and provides a means of stepping back and forth through the pages representing the full list. In short, it needs to present its data in a way that more closely resembles the way the IRSpy web UI works. Happily, some of the substrate code for this already exists, as I imagined a search model rather than a browse model when building the earliest Admin Console. The big missing area is paging through long result-lists. 5. MISCELLANEOUS BUGS Finally, Bugzilla shows 18 open IRSpy bugs at http://tinyurl.com/irspybugs These are of very variable importance and difficulty, but unfortunately all currently appear as P4s, as they have been downgraded through time to reflect their lack of urgency. (In retrospect, this demonstrates that we need to tweak our Bugzilla practices so as to distinguish between urgency and importance.) These bugs should be reviewed, and we should determine which of them ought to be solved as part of a MasterKey integration project. On top of this is the wishlist: http://twiki.indexdata.dk/cgi-bin/twiki/view/ID/IRSpyWishList