Backtalk: The Long, Long Road to the 19th Century, June 1, 2012

When Gale’s editors approached the creation of the recently released Nineteenth Century Collections Online, which offers unprecedented access to global archives, they needed smelling salts. Here’s how they tackled the job (see the review).‚ Ed.

In 2004, Gale’s Eighteenth Century Collections Online (ECCO) changed the face of digital scholarship. The database’s wide range of primary-source content allows researchers to approach projects that were previously technically infeasible as well as to use materials that were at one time unavailable online and housed at very remote sites.

Customers soon asked when we would do the same thing for the 19th century. We saw the logic and the opportunity, but the technology available in 2005 made the endeavor too large. The era’s amount of publishing dwarfs that of the 18th century (hat tip to the steam-driven printing press and increased literacy rates, for example). Plus, there is no comprehensive bibliography. How could we approach the mountain of books, manuscripts, images, newspapers, pamphlets, and more?

The journey began with products such as Gale World Scholar, which represented our first major foray in two complementary, synchronous methodologies of product creation: user-driven product design and agile development.

User-driven product design goes way beyond market research. Extensive, in-person interviews result in the creation of personae who represent the needs, workflows, and goals of the variety of users who will rely on our products. A constant loop of user testing and feedback helps our development team create advanced tools and features that fit into our customers’ workflows while remaining intuitive and user-friendly.

Agile‚ a flexible, cutting-edge software development methodology‚ allowed us to embrace this user-driven product design. In the old days, development teams were given product specifications and sent back to their cubicles to start coding. Gale knocked down the cubicles and emphasized teamwork and communication. The result is a faster development cycle that centers on the specific, evolving needs and goals of our user personae and gives us a working product very early in the process. The product is never done‚ it continues to change and incorporate new tools that meet user needs.

By 2010, we felt ready to tackle NCCO and decided to take a modular approach, assembling a program of archives that covers the period through a variety of topics and themes.

We also formed a still-growing global advisory board made up of scholars and librarians of different backgrounds and specializations who were brought together to advise on topics, themes, concepts, and regions to be covered. They also recommend institutions, associations, and scholars to provide the evaluation of concepts; consult on product features, design, and user experience; identify scholars, librarians, and other specialists to define editorial criteria and content; and conceptualize and commission items such as introductory essays.

Of key importance, too, are the partnering libraries and archives. Curators at these institutions know their collections inside out, and this makes them uniquely qualified to advise on editorial matters. Gale has collaborated with more than 150 institutions to assemble more than 150 million pages of primary-source content. Recognizing that it’s virtually impossible to publish everything from the time, we aim to be provisionally comprehensive, meaning that the program focuses on the major issues, events, and topics of the long, long 19th century. The traditionally defined long 19th century is from 1789 to 1914, but NCCO includes a good deal of content from well before and after those dates, as we include entire collections.

Four major archives will be published annually, with each devoted to a particular theme or field of research. The first four archives for 2012 will be British Politics and Society; Asia and the West: Diplomacy and Cultural Exchange; British Theatre, Music, and Literature: High and Popular Culture; and European Literature, 1790‚ 1840: The Corvey Collection.

User experience matters

Having decided on coverage, Gale set about making the user experience far richer than ever. First, we committed to capturing images at a resolution of 400 dots per inch, striking a balance between image quality and retrieval speed. We can capture documents ranging from small pamphlets to large drawings and even enormous maps. Where the original material could be damaged by handling, we work directly with the source institution to ensure preservation. Gale funds extensive conservation efforts, and the creation of a digital surrogate saves the originals from wear.

NCCO is packed with handwritten manuscripts. Since no technology exists to read them electronically and thus make them searchable, manuscripts get lost in databases when they compete for attention with printed works. Still, our customers value the materials highly, so we hand-key selected handwritten text (places, names, and dates). We’ve also added two million searchable terms to the documents to increase searchability.

Another breakthrough involves new textual-analysis tools. Term clusters help users discover related content, and our graphing tool exposes the occurrence of words and concepts over user-defined periods. NCCO also introduces comprehensive subject indexing, using a taxonomy we developed for the 19th century that contains tens of thousands of common terms along with countless place and personal names.

We continue to upgrade the user experience and source new archives. For 2013, we’re exploring collections related to science, women, photography, and colonialism in Africa. Stay tuned!