Archives of the New Millenium: Exploring the archival issues of the early twenty-first century

Barbara Reed

Archives of the New Millenium: Exploring the archival issues of the early twenty-first century

Barbara Reed

I'd like to briefly introduce myself. I am the mook, avatar or intelligent agent of Barbara Reed who unfortunately couldn't be physically present at this conference. I have chosen to visualise before you as a hologram of a more youthful Barbara and hope that you will forgive any flickering or fading in my holographic presentation. Before I start, I'd also like to let you know that Barbara is connected to the presentation today and I'll be teleposting any comments or queries to her for her direct response as we go along.

Barbara has authorised me to deliver the following text:

Thank you for inviting me to address the 65th annual conference of ARANZ. It was a delight to be invited to present the keynote address and it's an additional pleasure to re-visit a country with family associations: one which I haven't visited since last century. At that stage I was a physical person, not like the virtual representation of my 80-something years, that I present before you today. As I approach my topic of a review of progress in archives and recordkeeping (a quaint old-world distinction I know), I laud your collective endeavours to get into the theme of 'sailing (the) past' by wearing your distinctive turn of the century clothes!

So, what has the past decade shown us to be our professional image and achievements? This is of course, a subjective analysis. Each of you will no doubt feel that I have overlooked a key project or outcome and I'd appreciate hearing your feelings at the end of the paper. In that sense, looking backwards is almost as dangerous as looking forwards - but perhaps not quite: in that the past, while subject to endless interpretations, has already happened.

Today, let me highlight three things particularly for attention:

• The comnet web storage project with its emphasis on personal recordkeeping
• The work of Ian Vestigator and the interesting questions of identity, authentication and transactionality raised by his revelations.
• The palimpsest archival exhibition/interface project named of course in an ironic reference to the parchment days where scarcity of recording media encouraged the rubbing out of one text and its replacement by a subsequent one.

I've worked with each of these projects in an advisory capacity and while I don't speak on behalf of the project directors, I find that each of them raises questions that are fascinating for our profession. Each of these projects I believe reflects some different facets of how we are deploying our professional skills. These projects all highlight our professional concerns, reveal attitudes and characteristics that have informed our professional development and ethical attitudes and also point to future developments or opportunities.

1. Appraisal of web-storage

The Comnet business was one of many reappropriated into public ownership. Comnet started offering on-line web storage in the first years of this century - an alternative to personal and small business storage options. The success of web storage venture took everyone by surprise at the time. At its recent nationalisation there was in excess of 75,000 petabytes accumulated. Now that the government has control, the archives has sponsored a United Humanities-funded pilot project to appraise 1% of the accumulation and assess the significance, or otherwise, of the material stored there.

Initial findings from the project show that this material is, as was suspected, quite a remarkable resource. Initial attempts to scope the project found, stored amongst the corporate data, a surprising wealth of information relating to the everyday lives of individuals. This formed the basis of the pilot selection. A conscious decision was made not to focus on the corporate material. The rationale used was that this was the responsibility of the businesses themselves, wheras the opportunities to work with private and personal records were far rarer. The material appraised, selected almost at random from across the years of the first decade of the century, reveals some quite remarkable demographic shifts going on in our society during that time.

We already knew that the enthusiastic uptake of this form of public storage in the early decades was extraordinary and unexpected at that time. One hypothesis for the pilot is that we should be able to trace the growth of comfort, levels of trust and digital literacy of the ordinary person. The material should allow a representation of the increasing levels of trust that people were prepared to invest in the technology and that the patterns of use of the web-storage facility are directly proportional to that trust. Knowing that the encryption mechanisms were as stable as claimed, finding no invasions of privacy as was feared and demonstrated confidentiality ensured by the generally sustainable security, fuelled adoption of the technology beyond expectations.

The initial findings appear to bear out the hypotheses put forward. The project expectations are on track. What we've found is an amazing outpouring of unexpected personal communications, family correspondence, work related material, school and educational notes and reports, multi-media playing and a range of other creative endeavours. Even the single percent of the material included in the pilot project shows the evolution of documentary form between the paper literate world of the C20 and the digital world that we know today. We can trace the intrusion of multi-media, shifts in technology, trends in technological uptake as well as sociological and demographic indicators, all aggregate outcomes of what is also fantastic individually flavoured social documentation.

Of course, from a recordkeeping perspective, the project demonstrates all the known and horror characteristics of non-records compliant data sets. The project bears out all the issues of appraisal of electronic records that we know, and knew even back at the turn of the century. Retrospectively dealing with this material is a nightmare. Software dependencies of the data formats causes horrendous problems. Reconstructing the contextual metadata to enable understanding of the originators as provenance entities has been lengthy and a resource intensive process. Forensic recreation of links, even where the decision was taken to treat the material within one creator's space as the 'record', is not straightforward.

Some innovative steps have been taken in this international, high profile and well funded project. Alliances with the 'software archive' company have been consolidated and some of the project funds allocated to supporting the continuing operations of defunct and antiquated software along with documentation of software protocols. Similarly, there have been new commercial ventures identified and supported in start-up mode.

Two streams of work have been supported. The first is a set of component programs (called 'fix' bots) which seek to migrate the software dependent remnants. The original and the newly migrated bits are then bound together to reside in storage. The second is a set of component programs nicknamed 'undeading'. The undeading proglets crawl over data and extract relevant metatags, recreating as much of the contextual data and transaction history as can be patched together from what remains. As part of the international collaborative and philanthropic nature of the project funded by the United Humanities grant, the component programs developed have to be lodged or registered within the public domains of netspace.

The social, cultural and historical significance of this resource is undeniable even by those who critique the project's operation. Similarly for the recordkeeping profession, long terms spin offs of this project are immense, being relevant for every individual archives and recordkeeping program - public, community or private. But, even with all the great things:

• the multi-million dollar pilot project funding
• the collaborative nature of the archival project staff involving representatives from every nation of the world
• the 'fix' and 'undeading' bots available for general use
• the commitment of the resurgent Microhard corp the chances of the pilot being extended to cover the remaining 99% of the accumulation are not great.

What the project has done by deploying clever technology aligned with archival know-how has, for the first time, demonstrated that this retrospective approach is just possible and that the rewards for rescuing such a rich resource may be much greater than were originally thought possible. With this high profile project, its successes, its alliances and its universal collaborative base, the stage is set for a re-positioning of the nerd archivist image still held by a disturbingly large proportion of our elite working classes.

2. Recordkeeping scandals

The second area I've chosen to focus on is a particular scam and its implications being revealed in the work of Ian Vestigator. You will all be familiar with the high profile investigative work into data scams, many of which have recordkeeping implications, that Ian has become known for. You may even have been visited by him or his avatar, Sniff (his intelligent agent who somewhat disconcertingly visualises as a twitching nose). Ian's work helps us to focus on the dark side, a side we prefer not to dwell upon but which we must constantly keep to the forefront of our professional understandings and analyses.

For those who don't know Ian, his Vestigator Report broadsheet is vital reading for all recordkeeping professionals. One recent example of his work, now widely picked up by the mainstream media, is the spate of prosecutions resulting from Ian's investigations into the corrupt dealings of the security services in the years prior to the universal adoption of the encryption treaties. This work revealed a systematic abuse of the transaction data generated by all business transactions. Shonky practices included data hijacking, fraud and the syphoning of the smallest units of international currency into the hands of the cyber-mafia, where it accumulated to form large and untraceable sums of money. Working with forensic and regulatory recordkeeping specialists, Ian has built incontrovertible cases of corruption utilising the techniques developed in recordkeeping analysis of transaction history.

The specific investigation I'd like to highlight is more than usually close to home for our profession and our positioning. It is a complex scam which involves our authentication agencies and unravels the edges of what looks like a particularly nasty excursion into information warfare and controls during the 2030s. The genesis of the scam appears to be the use of netbots in the early part of the century to create false persona profiles which bear all the virtual characteristics of individuals on the net. The origins of the mechanisms used were designed as spamming or spoogeing agents to jam up communications capacities of news groups on the old Internet 1. The instigators of the scam set up cheap and disposable email accounts. The email protocols then carried the activated netbots into the specified location. The netbots then crawled across newgroups, grabbing headers from messages and filling the contents with pre-set written pieces or by just grabbing the contents and reassigning new headers. The object of the exercise was to build up a profile which looked and seemed to behave in ways compatible with an individual operating on the net. With this profile, the spoogers then registered as authenticated transactors with the private authentication agencies. They then conducted totally fictitious transactions and financial dealings. It seems that the transactions conducted reach into all areas of corporate and government operations.

Ian and Sniff have uncovered widespread and quite systematic problems with these false identities. Working out what is an authentic transaction and what is a fraudulent one is a recordkeeping problem. Since the recordkeeping profession has finally achieved the much lobbied for recognition as the transaction authenticator, I foresee a considerable amount of professional time being spent in retrospectively re-authenticating transactions which are in question as having originated from a false virtual identity. To do this, our forensic archivists will need to make innovative and unprecedented use of the transaction data generated in the transmission, process and management metadata. Indeed the lynchpin of our capacity to do this successfully will be the full exploitation of the use history and event history recordkeeping metadata, once derided by our information colleagues as being over the top. Proving the authenticity and reliability of transactions and the consequences of any breach of basic protocols will remain with us for a long while. The careers and time of regulatory and forensic recordkeepers will be occupied for some period to come as expert witnesses and investigators for the streams of legal cases already slated for hearing.

But there are other issues that should concern us in this case. This recordkeeping dimension of authentic transactions is but one implication of false virtual identities. The broader question of cyber identity needs to concern us also. Since we've assumed the social responsibility as transaction (or records) authentication agents, we must keep abreast of the possibilities of falsification and deliberate rule evasion.

Ian's work has revealed that this particular scam was known to internet regulators and the myriad of commercial authentication agencies around in the 2030s. A covert but concerted strategy of 'killing' avatars and mooks took place during that time. It seems that this was sanctioned and even encouraged by the regulators during the 2030s. During the height of this problem, it seems that organisations of all kinds invested in what amounts to 'killing agents' whose express functionality was to destroy mooks and avatars which were suspected of having no physical manifestation behind them. The regulators even had a set of malware infiltrating transmission streams as a virus, which was designed to covertly eliminate these cyber-representatives.

I suspect that this is just one of the covert information warfare operations that will gradually be revealed as Ian purses his investigations further. And this one appears to have been operating under the apparent authority of the internet regulators. This specific actions that Ian has uncovered here were undertaken with what we can assume to be appropriate motives - to protect the authenticity of transactions undertaken by authorised avatars and mooks. But the unleashing of this technology, its sanctioning by authorities and the implications of covert information warfare are more disturbing.

The wholesale termination of millions of virtual identities during the 2030's by international government's Identifight program is now being uncovered. The extent of social disenfranchisement of those 'undone' during those times has never been fully revealed. Where is the line between reality and representations of reality? What constitutes sentience in virtual transactions? Is it the capacity to learn? When does an agent, avatar or mook become more real than the thing it purports to represent? What happens when these objects are turned into agents of destruction or destabilisation? Who judges? Should such extermination software be outlawed? The digiterati are discussing the potential implications of these revelations furiously. The recordkeeping voice and angle needs to be clearly and persuasively advocated. But what is the recordkeeping community's opinion?

Some of the big questions we must ask are:

• Who gets to decide the rights of virtual existence?
• Who is able to actually destroy or kill virtual representations?
• Are corporations responsible for transacting business able to make these decisions, as they did during that time, or merely to bar entrance and interaction?
• What implications or consequences are felt by the person whose on-line representation is summarily terminated?

As recordkeepers we have more than general interest in the resolution of the issues. We now have specific professional responsibility to ensure that the transactions we authenticate are reliable. With hindsight we can see that the formation of the Registrar of Cyber-Identity (a direct functional descendant of the Registrar of Births Deaths and Marriages) was a direct response to this problem. For the last 3 years, mooks or avatars enabled to transact business for individuals, have been required to be formally registration of with the Registrar of Cyber-Identity. The establishment of the transaction authentication role at the same time as the Identity registration role is again a logical outcome of this scary set of information warfare actions. Part of the framework of transaction authentication that is now standard is certification of identity from the Registrar of Identity.

But as this case is so clearly demonstrating, our professional responsibilities are bound up with these issues of cyber existence. As the professional group tasked with authenticating transactions, our recordkeeping professional domain extends into the heart of trustworthy and sustainable business. The stakes in this game are much higher and more critical than any professional role we've undertaken to date and we had best be wary not to be too naive about what is entailed in the administration and management of this role.

3. Visualising archives and records

The third project that I wish to highlight is much pleasanter to contemplate. The palimpsest archival exhibition of the international archives uses the combined talents of the leading edge recordkeepers, intergraphic designers and data architects. This project has utilised software and thinking which is setting a standard for all to emulate.

The palimpsest exhibition is essentially a user interface to the archives of the world. Conceived of to permit a self directed exploration and display of items from the records selected for exhibit and under intellectual control of archives, the major technology deployed is what used to be called search engines or portals. By fully exploiting the metadata associated with records, which has been available in standard formats ever since, the adoption of the recordkeeping metadata set and XML way back in the early years of this century, the exhibition finally shows the potential of visualising the relationships and records metadata.

The user interface allows people to access the exhibition from any medium. It is totally transparent to media (digital media, of course). From a uniform front entrance point, the software invites the user to explore. What is different about this technology is that the software uses highly complex artificial intelligence nodes which 'learn' about the person entering the exhibition. Tailored views, presentations of specific records, individualised ways of approaching the material available all make for a totally unique experience each and every time someone approaches the archives exhibition - hence the name palimpsest which has lovely parchment world overtones. Users are able to roam across all the material selected for inclusion in the exhibition.

The initial interface is based on data visualisation techniques which colours the interfaces and paths available. Each relationship is a potential pathway. Each pathway is indicated by a thread. Different coloured threads and different textures connote different and alternative ways of approaching material. At times the threads interweave and what appears almost to be a woven cloth like appearance emerges. The path a user chooses to follow is lit up. Backtracking at any time is possible using this method. Previously explored paths and those which lead to pre-defined places can be used to get to particular points. Indeed the exhibition idea is just that - following like on a guided tour, a particular pathway.

The technology builds on geographic and spatial metaphors - we can delve, drilling right down to a particular item or document, or scan a large horizon. Inevitably, I suppose, the graphics are drawn from the paper world and tangible things. I find it fascinating that the designers have drawn on the concepts of labyrinths and crypts building on descriptions of the British chancellery offices from the C15th. To give you a sense of this, let me quote from the design document by the intergraphic designer which explains it:

'I saw that the crypt was a vast labyrinth extending far beyond the chancel to encompass the area beneath the nave and then, for all I could tell, Chancery Lane and perhaps a good part of London too. Narrow corridors barely two feet in width and overhung by rolls of parchment - some as big around as saucers, others thin as pipe stems - slithered away into darkness on either side, then divided into other, equally cramped tributaries.'

With acceptance of the final proof of concept, the project now goes into full production mode. Access to the paper collections that have been digitised can be made available. Here, too, the technologies have been revolutionised. Instead of the old ways of digitising, new techniques specifically linked to this project have found ways of emulating the experience of reading paper files. Virtual reality interfaces allow page turning, and even the experience of riffling through files. There is a sense of the tactile nature of these objects, which if not the real thing, is the nearest we've ever come to projecting the pleasure of exploring paper files.

Developed initially to serve this exhibiting purpose, those of us who've been lucky enough to have experience with the technology, can see that this approach to archival exhibitions has much greater application for recordkeeping work in general. I think that this technology or extensions from it will form not only an exhibition entry point, but be the major entry point for all users of the archives - rather than thinking of a specific exhibition, the individual experience and tailoring will allow every person coming to the archives to structure their explorations through this interface. Of course, the role of the reference archivist will not be removed. Should such an expansion of the concept be realised, there will be way stations and rest points where the skills of experienced staff will be available to point out further tracks or extract searchers from dead ends. Initially these will be reference mooks (artificial constructs) but the capacity to interactively link with the human experts will also be there. It will also be possible to follow someone else's pathway - in the footsteps of their discoveries. The technology allows annotations and references from the various travellers and explorers to be accessed at signposts which can be left along the way. If, for example, we want to recreate the archival experience of Ian Vestigator using the archives, we should be able to do this (subject, of course, to the usual privacy protections).

Of course, all this capacity wouldn't have been possible without the rigorous and uniform adoption of metadata standards for records. The relationships and contextual links, so dear to our archival hearts are there as the pathways for exploration - for the first time visualised as threads between individual records or sets of records. The threads are multi-dimensional, allowing vertical and horizontal exploration of all of our resource base. Where the track ends, or comes to a stop, there is an explanation from the metadata remnants of destroyed records, or the equivalent of the old paper forms of series registrations, to let explorers know why they reach a dead end, and where and how to get to any existing non-digitised hardcopy originals.

Further stages of development of this exciting technology will allow other archival institutions to link in their collections. Of course, the records managed by some of our colleagues won't be suited, as the systems and standard ways of documenting the records world weren't globally linked to the relational records documentation models derived originally from Peter Scott's CRS system, until about 2020. Records in some major institutions won't be able to be linked until significant data scrubbing and clean up are done. Interestingly enough, even those that have been persisting in isolationist views of archival and records methodologies are coming to see the potential power of these technologies and the spin off industry of exporting recordkeeping consulting expertise is also growing apace.

The interdesigners are also really excited about the possibilities of what we've uncovered here. They are actively pursuing projects to make this technology the standard interface for all records use - even for those records not in the public domain. This is the true potential power of the technology and finally vindicates our professional view - that records are a limitless and always renewing information resource.

Indeed one can conceive of a new paradigm for information searching and that was the thinking (initially regarded with amazement by the designers) to build authenticity stamps around those resources which are verifiable records. Of course, this is less a problem now that the standard softwares automatically attribute metadata as transactions occur - but it provides some sense of security about what we're looking at from the dreadfully muddy days of the last 3 decades of the C20 and the first one or two of this century.

Concluding musings

At last, we can see tangible results with huge social and cultural spin-offs from so much of the sheer grind we all put into establishing recordkeeping infrastructure and professional understandings over the past half century. When we reflect back, we can see that much was done on the smell of an oily rag, insufficiently resourced, against significant odds, and amidst, what at the time seemed like constant, internecine war amongst information professions jostling for prominence and pre-eminence in the digital world we now inhabit. Looking back, we wasted so much energy having to constantly assert the unique perspectives of recordkeeping which make it such a distinct information viewpoint.

While my paper has deliberately focused on the up-side of the technology and recordkeeping, that we can do so with optimism seems in contrast to the doom sayers of late last century. Then, I recall dire mutterings about the division of the profession, the inability to keep the digital and the paper recordkeeping traditions and professional specialisations in harmony. It is a matter of professional congratulations for all of us that we can now happily cross these media boundaries by concentrating on the principles of recordkeeping which unite all recordkeepers from Ancient Mesopotamia to today, with each society and culture adding value to our rich recordkeeping traditions.

Ah, but do you remember the problems that we encountered along the way in order to work out at the end of the paper age what we needed to fiercely hang on to as our professional rationale and what we could safely discard as a peculiarly physical way of doing something. The things that I think were vital in contributing to today's professional placement are:

• That our educators and theoreticians could articulate the continuum of recordkeeping traditions
• That we were able to separate concepts from practices and jetison paper practices where we needed to
• Encouraging and enthusing other professions to engage with us
• reskilling ourselves
• being able to contribute positively in an information management framework while still articulating - consistently and strongly - the distinct role of records as evidence of activity.
• Incorporating an understanding recordkeeping into school based literacy programs
• Protecting our educational standards, even though this at times meant advocating non-conformist and non-traditional education delivery and accreditation methods

While I do think we've got something to be proud of in the last 50 years of our professional development, I don't think we can rest on these laurels. The dynamism of the projects I've outlined to you today argues for even more innovative and flexible utilisation of our collective skills. The speed of technological change is just as fast now as it has been any time in the last 50 years. I look forward with curiosity and intrigue to hearing a similar assessment of where we are at from the turn of the next century.

Thank you

Acknowledgements

Science Fiction influences
Dennis Danvers, Circuit of Heaven, Avon, 1999
Greg Egan, Permutation City, Millenium, 1998
William Gibson, Neuromancer, Harper Collins, 1995

Neal Stephenson, The Diamond Age, Viking, 1995
Neal Stephenson, Snow Crash, Roc, 1993
Bruce Sterling, Heavy Weather, Phoenix, 1995

Social commentators
John Seely Brown, Chief Research Scientist, Xerox Parc
Paul Saffo, Institute for the Future Archives in the New Millenium. ARANZ July 1999 11

© 1998 . All Rights Reserved. Licence: Limited to on-line viewing and the making of one (1) printout for off-line reading purposes only.

About     Research     Publications     Consulting     Links     Sitemap

Authorised by Head, School of Information Management and Systems. Caution.
Maintained by Records Continuum Research Group.

Last updated 5 January 2000.