Fantastic Futures 2024 Papers

Abbey Potter. Innovating Responsibly: Pathways for AI Transformation in GLAMs

This workshop will introduce participants to the LC Labs AI Planning Framework and key activities to evaluate ways libraries could use AI responsibly. Participants will collaborate in small groups to test out or generate use cases, document the potential risks and benefits of AI in libraries, assess data readiness, explore testing scenarios, and evaluate outcomes.

The LC Labs’ Artificial Intelligence Planning Framework helps staff assess the potential risks, benefits and readiness of AI use cases and provides a roadmap for understanding, experimenting, and implementing AI at the Library. It is based on several years of experimentation, demonstrations, and research into how machine learning (ML) and AI could be applied responsibly at the Library. The AIPF results in quality benchmarks, documentation, or evidence for supporting the business justification (or the opposite) for enhancing products or services with AI tools or approaches.  The AI planning framework and activities will demonstrate how to protect organizations, staff and users while encouraging innovation.

Structured workshop: 2 hours, Abbey Potter, Library of Congress & Mike Trizna, Smithsonian

Benjamin Lee. Reanimating and Reinterpreting the Archive with AI: Unifying Scholarship and Practice

This presentation brings together different disciplinary perspectives to explore how AI helps us to reanimate and reinterpret digital collections held by GLAM institutions. With two case studies, the presenters will demonstrate how academic projects not only further scholarly inquiry but also advance GLAM practice itself. Andrew Dean (Assistant Professor, Writing & Literature, Deakin University) and Benjamin Lee (Assistant Professor, University of Washington Information School) will speak about their collaboration to reimplement Nobel Prize-winning novelist J.M. Coetzee’s AI-generated poetry – one of the earliest examples of generative language produced by a literary figure. With the preservation of Coetzee’s FORTRAN code at the Harry Ransom Center at UT Austin, it is possible to recover the generative algorithms that Coetzee wrote during his time as a computer programmer in the mid-1960’s, before his rise as a novelist. Significantly, the re-implementation of this code makes it possible not only to properly understand his computer-generated poetry but also reanimate the algorithms themselves, thereby enabling the generation of new Coetzee poems. In his presentation, Dean will talk about this ongoing work, its implications within literary studies, and how this work serves as a roadmap for GLAM institutions attempting to provide access to born-digital code files.  Katherine Bode (Professor, Literary and Textual Studies, Australian National University) will discuss a project exploring the potential of word embedding techniques for archival research. Contrary to the prevailing belief that machine-learning or Artificial Intelligence (AI) technologies discover or uncover patterns in their training datasets, this project posits them as participants in the evolving knowledge apparatus of the historical record. They do not uncover so much as become part of relations that constitute the possibility of historical knowledge; and like traditional infrastructural and institutional arrangements, they are infused by history, politics, and power. We explore this issue with a case study devoted to investigating Irishness in an expansive archive of digitised Australian newspaper fiction. Formed from complex intersections of sectarianism, race, geography, migration, and diaspora, literary traditions, and religions, Irishness in Australian literature is not a findable thing so much as an array of intersecting cultural forms and practices. By moving between the textual technologies (the newspaper novels) we’re accustomed to reading and writing with and those we’re not (a word embedding of a large digital archive) we consider the values that might - and should - guide archival research in changing techno-textual conditions.  

Presentation: 15 mins, Benjamin Lee, University of Washington, Kath Bode, Australian National University & Andrew Dean, Deakin University

Emily Pugh. AI-Generated Metadata and the Culture of Image Search

GLAM patrons both /are/ and /are not/ prepared for the kinds of image searching that AI-generated metadata will facilitate. This is one takeaway from a four-year Getty Research Institute (GRI) project entitled PhotoTech, which focused on using AI for both art-historical research and metadata generation in relation to digital image collections. Based on the outcomes of this project, this presentation will provide an overview of the current approaches to image searching as they exist in the field of cultural heritage, explicating the disconnects between these approaches and those required by an environment of AI-enhanced search. The fifteen-minute presentation will consist of two parts. I will begin by sharing the results of a qualitative survey on image searching practices we conducted as part of PhotoTech, which received over 500 responses from professionals in cultural heritage and art and architectural history. The survey exposed the extent to which the current search practices of such professionals are tied to traditional approaches to cataloguing, relying for example on standardized subject headings and vocabularies. However, based on responses to a series of questions about computationally generated metadata, the survey also revealed that these users are by no means resistant to searches powered by AI-generated metadata. 
Next, I will explain how the results of the survey became useful in relation to the research activities of PhotoTech, and specifically our exploration a collection of 30,000 digitized images of photographs of cultural heritage sites in Europe and South Asia. We focused in analyzing these images on a basic, well-established computer vision (CV) task: figure detection. Starting from the simple question of whether photographs depicted or did not depict people, we applied the output of the CV algorithm to research how photographic aesthetics of archaeological and architectural photography developed in the context of colonialism and nationalism. We initially focused on figure detection to support research questions; however, noting hesitancy among the survey respondents in regard to tasking computers with complex, interpretive tasks, we also considered how number of figures might provide an alternate approach to collections discovery. Could a focus on figures provide a way to subvert conventional standards and categories of archival description that assume a European positionality (e.g., labels based on Christian iconography like “Annunciation” or descriptions like “native village”) or that make determinations about the age or gender of the subjects depicted? The insights of our application of AI to both researching and searching in the context of PhotoTech will, I hope, provide a clearer picture of how we might begin to facilitate users’ uptake of new ways of image searching and, as part of this, leverage AI to make collections discovery more representative and inclusive of diverse populations of subjects and scholars. 

Presentation: 15 mins, Emily Pugh, Getty Research Institute 

Francis Crimmins. Evaluation of techniques that improve findability of historic images in a large and diverse corpus using AI vision models and embeddings.

The NLA has a large number of images available via its online catalogue. The vast majority of these do not have any detailed textual description, and so keyword searches have to rely on matches against the title and author fields. We have implemented a prototype system which uses an AI vision model to generate a detailed textual description of a representative sample of 5000 images from the catalogue. In a sense this can be considered as a “transcription” of each image, in the same way transcriptions can be automatically generated for audio data. This allows us to provide a standard keyword search over these descriptions, giving improved precision and recall when searching for pictures. A spoken version of these descriptions can also be generated using a Text-to-Speech model. This could be used to provide better accessibility for some users e.g. they could listen to the description of the item. (We have only done this for a small number of descriptions, as a proof of concept). We have also generated numeric embeddings (vectors) for each of these descriptions, and used this to provide a "similar images" feature, by performing a k-nearest neighbour search across the vectors. As well as a traditional list display the similar images can also be displayed in a grid. Clicking on an image automatically reorders the grid, so that the “seed” image will be in the top left hand side position, with the next most similar beside it, and so on, in left to right reading order. As you get further away from the seed image you will see the image content starting to diverge, and by clicking on other images it is possible to browse quite diverse "neighbourhoods" within the dataset. This supports a "search and browse" capability for interacting with the collection. The numeric embeddings can also be clustered, with a representative image (closest point to the mean embedding) used for each cluster. This implementation generates a static set of clusters, allowing this overview of image groupings in the collection to be displayed, without needing any initial text query. No dynamic clustering of search results is currently supported using this approach. If an image is clicked on then a k-nearest neighbors query is run as described earlier. The initial set of images displayed for that request should also be within the representative image’s cluster, but eventually images from outside the grouping will start to be displayed. The cluster display allows a user to get a sense of what type of images are available in the collection being searched. For example, it can be used by a curator to identify clusters of images they may want to exclude. Finally, we have taken the images and text descriptions and used OpenAI CLIP to generate image embeddings, and also used the text embedding capability of CLIP to generate text embeddings for both the NLA catalogue metadata for the image and the OpenAI description. The overall number of images processed is 201k (although only the original 5000 images have the OpenAI text description). We plan to present: (1) A comparison of image descriptions generated by proprietary (e.g. OpenAI gpt-4-vision) and open-source (e.g. LLaVA) models. (2) A comparison of the effectiveness of embeddings generated by open-source (many available) and proprietary (OpenAI) text embeddings from these generated image descriptions against the open-source CLIP native image embedding for: text-based image searching; image similarity searching; automatic image clustering. (3) A comparison of text/keyword, image embedding and "hybrid" (combination) searches. 

Presentation: 30 mins, Francis Crimmins, National Library of Australia

Honiana Love. Join us on a journey to the future

Ngā Taonga Sound & Vision is Aotearoa New Zealand’s audiovisual archive with a collection spanning more than 100 years of rich history. We hold a unique position as the only dedicated audiovisual archive capturing the history of New Zealand, told in sound and moving image. There are over 800,000 items in our care, dating back to 1895, and include film and television, sound and radio recordings. Ngā Taonga is a small organisation, of less than 120 kaimahi, operating as an independent charitable trust, governed by a Board of Trustees and funded predominantly by Manatū Taonga, the Ministry for Culture and Heritage; the Lottery Grants Board; and Te Māngai Pāho. Over the last five years, the organisation has transformed from having an at-risk and largely inaccessible collection stored on physical formats to an archive that now has a significant proportion of its collection digitally preserved – with strong pathways for sharing collection material. We are, in essence, a digital archive that supports analogue preservation. Our rapid change was only possible because of our small size and our willingness to innovate. While we have limited financial resources, we can be agile and develop different ways of responding to the changing needs of the sector, supported by our skilled kaimahi. This presentation will offer an overview of the innovations we’ve adopted, to date, including our fundamental commitment to mātauranga Māori (a concept that essentially translates to ‘traditional Māori knowledge’). We care for the largest body of historical recordings of te reo Māori (‘the Māori language’) and mātauranga Māori in the world; taonga (‘treasured possessions’) which are unique to Aotearoa New Zealand. We are committed to being a kaupapa-centred organisation, a concept where we recognise and deeply embrace that, in our work, there is more than one world view. We have created a relationship framework for working with kaitiaki (those people who have a connection to items in our collection as caretakers or guardians), recognising the importance of their role in approving the use of items they have a connection with. We have mātauranga Māori specialist roles across the organisation to ensure that we have that we have a level of expertise to be able to engage with taonga Māori in a genuine Māori way when required. We developed an unprecedented programme (‘Rokirokitia’ – ‘to preserve or care for’) of community-based magnetic media digitisation which saw us working with iwi/Māori all over Aotearoa New Zealand. We supplied portable purpose-built digitisation kits and training so that communities could rescue near-obsolete media and store it locally in digital formats. We are running a digital preservation project (called Utaina) for at-risk audiovisual material at a scale not previously undertaken in Aotearoa. In 2023 it was the largest digitisation project in the world and represents a new approach to digital preservation. We have worked with the developer (Te Hiku Media) to test Artificial Intelligence transcription of te reo Māori to support our commitment to create catalogue entries in the language of the taonga we hold. We do not own any of the material in our collections and the nuances of upholding our commitment to respecting the rights of numerous different groups places us in a unique position in the sector - copyright holders (those who hold rights in the content of the material), depositors (those who own the physical item deposited with us), and kaitiaki (those holding cultural ownership rights). We have developed a preservation prioritisation framework that considers access to have the same weighting as preservation needs and cultural / national significance. Now we turn our energies to becoming a digital-first organisation and the challenge of integrating technology into our organisation within an environment where technology is evolving exponentially and shows no signs of slowing. We face the challenge so many others are also dealing with. How do we avoid the Betamax trap, investing in technology that has a limited lifespan before being discontinued? How do we manage our limited funds to make the best investments for the future? What does adoption of emerging technology look like for small organisations or not-for-profits? What does a collaborative investment in technology look like – what could we offer partners to engage with us? How do we juggle the investment in staff needed to engage with nascent technology when our funding barely allows us to invest in the technology itself? How can we engage with communities to support their move to digital? What are the barriers to championing an indigenous language in a world where English remains the language of the internet? How do we navigate the opportunities and challenges of our position at the crossroads of the corporate world, the GLAM sector and government? Can we use modern technology to review our historic records and update with modern approaches to access? 

Presentation: 30 mins, Honiana Love, Ngā Taonga Sound & Vision 

Javier de la Rosa. The Mímir Project: Evaluating the Impact of Copyrighted Materials on Generative Large Language Models for Norwegian Languages

The Mímir Project, an initiative by the Norwegian government, aims to assess the significance and influence of copyrighted materials in the development and performance of generative large language models (LLMs) specifically tailored for the Norwegian languages. This collaborative effort involves three leading institutions from different regions of the country, each contributing unique expertise in language technology, corpora curation, model training, copyright law, and computational linguistics. The ultimate goal of the project is to gather empirical evidence that will guide the formulation of a compensation scheme for authors whose works are utilized by these advanced AI systems, ensuring that intellectual property rights are respected and adequately compensated. The advent of LLMs has revolutionized natural language processing (NLP), enabling machines to generate, understand, and interact using human languages with unprecedented accuracy and fluency. However, these models require vast amounts of textual data to train, often sourced from a wide array of digital repositories that include copyrighted materials. This raises critical legal and ethical questions regarding the use of such content without explicit permission from authors, as well as the economic implications for the creative industry. The Mímir Project addresses these concerns by systematically evaluating the role of copyrighted texts in enhancing the capabilities of LLMs for Norwegian. The project's methodology involves a comprehensive analysis that spans several stages. Initially, a diverse corpus of Norwegian language data is assembled, incorporating both copyrighted and non-copyrighted materials, plus materials commonly found in the Internet. This corpus serves as the foundation for training various LLMs, each with different configurations and access levels to copyrighted content. By comparing the performance of these models across a range of linguistic tasks, such as text generation, translation, and sentiment analysis, the project seeks to quantify the specific contributions of copyrighted materials to the overall model efficacy. To ensure robustness and reliability, the evaluation framework focuses on both foundational model generation and linguistically-inspired metrics. Quantitative measures include traditional NLP benchmarks like perplexity, BLEU scores, and ROUGE scores, which provide objective assessments of model accuracy and fluency. Linguistic analysis, on the other hand, involves assessing the coherence, creativity, and contextual relevance of the generated outputs. This dual approach allows for a nuanced understanding of how copyrighted materials impact the performance and utility of LLMs. One of the central hypotheses of the Mímir Project is that copyrighted materials, due to their high quality and curated nature, significantly enhance the linguistic richness and diversity captured by LLMs. By systematically isolating the effects of these materials, the project aims to identify specific areas where their inclusion leads to substantial improvements. More specifically, many ablations are conducted on the different aspects of the materials such as their source (books vs newspapers), factuality (fiction, nonfiction), or source language (translated vs original works in Norwegian). This insight is crucial for developing a balanced and fair compensation framework that recognizes the value contributed by authors while fostering innovation in AI and NLP. Moreover, the Mímir Project explores the broader socio-economic implications of its findings. By engaging with stakeholders from the publishing industry, legal experts, and policymakers, the project facilitates a multi-faceted dialogue on how best to harmonize the interests of content creators and technology developers. This dialogue is essential for crafting policies that not only safeguard intellectual property rights but also encourage the ethical and sustainable development of AI technologies. The outcomes of the Mímir Project are expected to have significant ramifications for both national and international contexts. For Norway, the project provides a critical evidence base that can inform national copyright policies and support the creation of a compensation scheme tailored to the digital age. Internationally, the findings contribute to the ongoing discourse on AI ethics and copyright law, offering a model that other countries can adapt and implement. In conclusion, the Mímir Project represents a pioneering effort to empirically assess the impact of copyrighted materials on generative LLMs for Norwegian languages. By bridging the gap between technology and intellectual property rights, the project aims to foster a more equitable and innovative future for AI development. The insights gained from this initiative will not only enhance our understanding of LLMs but also pave the way for more sustainable and fair practices in the use of copyrighted content in AI training.  

Presentation: 30 mins, Javier de la Rosa, National Library of Norway

Jon Dunn. Whisper applied to digitized historical audiovisual materials

Beginning in 2015, Indiana University (IU) undertook a massive project, known as the IU Media Digitization and Preservation Initiative (MDPI, https://mdpi.iu.edu) to digitize rare and unique audio, video, and film assets from across IU’s collections that resulted in over 350,000 digitized items to be preserved, including collections from the IU Libraries Moving Image Archive, Cook Music Library, Black Film Center & Archive, Archives of Traditional Music, and many other campus and university units. The MDPI digitized corpus contains significant diversity in terms of physical media formats, quality of audio and video in the original recording, content genre, language, and amount of existing metadata. From 2017-2023, with support from the Mellon Foundation, IU Libraries collaborated with AVP and other partners to develop the Audiovisual Metadata Platform (AMP, https://go.iu.edu), an experimental open source software system for building and executing workflows that combine AI/ML and human steps to help create metadata for AV resources, which was demonstrated at Fantastic Futures 2023. We are beginning to collaborate with collection managers at IU to use AMP with portions of the MDPI corpus. In the meantime, accessibility by users with disabilities has become increasingly important to libraries and universities in order to broaden access and ensure compliance with government regulations and institutional policy. With accessibility needs in mind, IU Libraries is exploring the feasibility of using the open source Whisper automated speech recognition tool from OpenAI (https://openai.com/research/whisper) in conjunction with IU’s HPE Cray EX supercomputer, known as Big Red 200 (https://kb.iu.edu/d/brcc), to generate captions or transcripts for this enormous collection of digitized materials. Testing is being performed using small, medium, and large (v2 and v3) Whisper ML models against a selection of content that samples this diversity of physical media, audio quality, and content format. Outputs produced by Whisper will be compared to ground truth for each media item, created through use of a professional human transcription service. Content to be tested includes files digitized from wax cylinders, Betacam videotapes, open reel audio tapes, audiocassettes, motion picture film, VHS, and more. Content will include both audio-only and video materials ranging from field recordings and home video to lecture capture, oral histories, educational films, student events, and broadcast radio and TV programs. Because of its distinct nature, we intend to set aside recordings that primarily feature music in order to focus on spoken word materials. In this presentation, we will demonstrate trends in scores such as word error rate and word information processed across Whisper models and types of source content, highlight interesting examples of Whisper successes and failures, share our experiences with processing files using a shared research-oriented high-performance computing environment, and discuss our future plans for using automated speech recognition output to support accessibility and discovery. We will also discuss our experiences with Whisper in comparison to commercially-available automated speech recognition models. This presentation will build on the growing body of work testing how well and in what configurations Whisper can be used effectively to create captions and transcripts for audiovisual materials, particularly for digitized historical materials.  

Authors: Emily Lynema, Jon W. Dunn, Brian Wheeler, and Jon Cameron, Indiana University Libraries 

Presentation: 15 mins, Jon Dunn, Indiana University Libraries

Kara Kennedy. AI Literacy for Librarians in Low-Resource, Multicultural Communities

Emerging Generative AI tools such as ChatGPT offer some of the greatest challenges but also some of the greatest benefits to people served by libraries housed within universities and polytechnic institutions. On the one hand, the tools can be used to shortcut learning, and students are at risk of getting in trouble for cheating. The tools can be used by teaching staff to shortcut grading, leading to potential issues with data privacy breaches and biased results. Despite so-called AI detection tools having low accuracy rates, they are still being used and are more likely to flag ESOL students. This has an outsized effect on multicultural and migrant communities, who already face less access to devices and the internet due to the digital divide. On the other hand, the tools can be used to help students overcome socio-economic and language barriers by enabling them to have on-demand access to virtual tutors and style their communication more formally. The tools can be used by teaching staff to create more engaging lessons, more accurate rubrics, and more customisable resources. To be able to take advantage of these powerful new tools safely and responsibly, though, people need to develop AI literacy, the newest part of digital and information literacy. Librarians remain trusted sources for information literacy for both staff, students, and members of the public. They assist with finding good sources, avoiding copyright violations, and referencing appropriately. Librarians are also often on the front line of assisting people with technology because they have a physical presence that is more accessible than a phone hotline or online helpdesk. But helping others with AI technology requires more knowledge and skills than showing someone how to access an electronic journal or use the printer. AI tools have a wide range of capabilities, and the effectiveness of their outputs is still heavily influenced by the prompts used. There are significant issues around biases, hallucinations, copyrights, and ethical considerations. AI does not rely on databases but predictive, mathematical models. It produces original content that has not author and no copyright. It offers a major challenge to fundamental questions about information sources and data that librarians need to be prepared for in their engagement with the public. With an aging librarian workforce, continuing lack of representation from marginalised communities, and limited professional development budgets, it becomes challenging to know what staff need to learn to best serve their increasingly diverse communities. The institutional libraries servicing Auckland have a customer base that is incredibly multicultural and multilingual, with many students and staff identifying as Māori, Pasifika, Chinese, or Indian, and having strong connections to other countries and non-English languages. They have a wide variety of socioeconomic backgrounds and technological proficiencies, and the pandemic lockdowns revealed how many lack access to devices and internet at home. For some, then, the library is their main connection to the digital world and will also become their gateway to AI tools. It is a trusted source of knowledge they will turn to in order to learn about new technology and how to use it responsibly and safely. It is clear that the role of librarian now needs to include AI literacy to enable staff to keep up with the shifts in how information is retrieved, relied upon, and sourced, and the threat that AI poses to people’s relationship with textual, audio, and visual information. This presentation will offer a vision of what AI literacy looks like for librarians and their professional colleagues working in a low-resource educational environment servicing a high number of low-socioeconomic, multicultural customers. 

Lightning talk: 5 mins, Kara Kennedy, Manukau Institute of Technology 

Kate Follington. C19th Handwriting and Model Making with Transkribus

Government archives from the 19th century have been traditionally difficult to transcribe due to the variety of cursive handwriting found within the records. The scale of records held within state collections adds an additional complexity because a digital team must consider which records aligned with an AI tool will facilitate better public access to popular records. Public Record Office Victoria has embarked on a journey of experimentation using European tool Transkribus to build different machine learning models, alongside public models, to work out which transcription approach will yield the best return on investment in time and outcome. The questions we are asking include: One hand models, or multi hand models? Can we have ten hands in one model and still see clean transcription? Should we start with indexes rather than correspondence records? How long did it take to build the model to get decent results, and what error rate can we tolerate? This presentation will share the results of this experimentation process using the Ned Kelly series along with other popular record series. 

Lightning talk: 5 mins

Kate Follington, Public Record Office Victoria, State Archives Victoria

Laura McGuiness. Just One More Access Point: LLM Assistance with Authority Control

For decades, librarians and archivists have created authority records to disambiguate author names. Frequently created using Machine Readable Cataloging (MARC21), authority records are highly structured data that authorize a preferred name format and then cross-reference variant or related names to the correct author. These authority records ensure standardized access points for users, creating a linking framework for related or identical names in a database. Los Alamos National Laboratory (LANL) is a United States national laboratory responsible for ensuring a safe, secure, and effective nuclear stockpile. LANL Librarians have been creating and maintaining authority records manually within an Integrated Library System (ILS). While many records are linked to LCNAMES, the Library of Congress name authority vocabulary, most name authorities were manually created. Like many other manual processes, however, this did not scale well and was time consuming. LANL’s National Security Research Center (NSRC) is the Lab’s classified library that houses tens of millions of materials related to the development, testing, and production of nuclear weapons. With the NSRC’s recent digitization of legacy archival materials dating back to the Manhattan Project, additional authority control has become a necessity. Despite the work of past librarians and the modern use of persistent identifiers such as ORCiDs, older publications (including born-analog and newly digitized publications) lack any mechanism of name disambiguation. Name ambiguity decreases the number of standardized access points for a resource, reducing the reliability of a database's search results. Because a researcher’s work at LANL is often critical to ensuring national security, it is imperative that all possible access points are available for use. Accurate authority records are especially important for traditionally underrepresented groups who have been disproportionately affected by name ambiguity in databases. This includes individuals who are more likely to change their name over time, names that have been transliterated from a foreign language into English or remain in non-Latin script, as well as double-barreled surnames (e.g. Maria Goeppert Mayer). Name disambiguation contributes to reliable bibliometrics and helps ensure scientific discoveries are correctly attributed. In a field where gender and racial minorities often experience lower authorship attributions, authority control gives recognition to scientists whose contributions have been overlooked in scientific databases. While the benefits to name authority records are many, there are multiple ethical concerns to consider in this project surrounding both AI use and authority control. For the former, we are mindful of ensuring transparency of the training set, outputs, and decision-making processes. Our team will be documenting the criteria chosen for inclusion or exclusion of certain data sets to provide a clearer understanding of the LLM’s learning processes. This includes ethical concerns with the training data, such as the incorporation of Resource Description & Access (RDA) Rule 9.7 or the MARC21 375 field, which require assigning gender to authority records. Attributed by scholars responsible for “outing” gender identities through authority records, this is one piece of training data that we may consider excluding. While LANL librarians did not abide by these cataloging rules, the possible use of external authority records may introduce this concept. To mitigate these concerns, our project team will be establishing internal accountability measures that will assess potential impact and biases. Currently, LANL has several large institutional databases lacking name authority records. This lack of access control contributes to information noise, producing overwhelming and ambiguous search results. Recently, we have begun investigating various Machine Learning (ML) approaches to support the creation of valid name authority records. We are utilizing Large Language Models (LLMs) to create accurate LANL name authority records. We will begin with the existing 17,033 records currently in LANL’s ILS, Alma, to train Llama2 on the proper format of a record and what components it must contain. We will then use an existing local database to generate the authority records. Once the LLM is able to properly produce a record, we will feed it the contents of the local database and evaluate the generated authority records. Author disambiguation is inherent in the process of authority control. Names are generally not unique, frequently change over a person’s lifetime, and are prone to multiple permutations in spelling and format. When using an LLM, there are key pieces of information in a database that would aid in matching, including: the author’s publication titles, dates active, and recurring co-authors. A combination of string matching via metadata filtering and vector or graph searching will aid in disambiguation by clustering linking features such as semantically related phrases or other document elements. The project team is also considering templatizing and populating a prompt that will ensure consistency in the generated response’s structure and semantics. The long-term goal of this project is to implement automatic authority control within LANL databases. As additional legacy materials are digitized and cataloged, the LLM will continuously locate new author names and either create an authority record or map to an existing one. These authority records will then be implemented in a semantic platform that will assist our cataloging and search processes across multiple databases. Ultimately, this will both improve cataloging efficiency and ensure findability of data critical to nuclear deterrence and national security. 

Lightning talk: 5 mins, Laura McGuiness, Los Alamos National Laboratory

Lizabeth Johnson. Enhanced stewardship and data sovereignty through the implementation of an ontology-enhanced large language model (LLM)

Many archivists have experienced the conundrum of how to manage collections and make them discoverable to users.  It is far from uncommon for an archive to hold collections that have never been properly indexed because of the lack of staff or resources, whether physical or computational.  A further issue for some archives is the existence of legacy software systems that were used to catalog collections, but which are no longer supported and cannot communicate with more modern software.  Now add in the complicating factor that users rely on the documents in your collection to be discoverable for the purpose of meeting national security needs.  This is a nested set of problems that Los Alamos National Laboratory’s (LANL) National Security Research Center (NSRC) is attempting to address by adapting a large language model (LLM) that can be used to make documents discoverable to staff who need them for their research. The NSRC’s archive holds multiple collections dating from 1943 to the present, and the documents, photos, and films held by the archive number in the tens of millions.  However, only a fraction of these items is easily accessible to LANL staff, while the rest are in physical collections or are only known to archival and library staff who have access to the antiquated software systems that continue to house the items.  This creates a roadblock in terms of the NSRC’s stewardship of its materials because, while it maintains them, it cannot make them easily discoverable to researchers.  In some cases, the byzantine nature of the collections can frustrate even the most dedicated librarian. This difficulty regarding stewardship has also led to concerns over data sovereignty at LANL.  To date, several divisions have chosen to maintain their own collections, separate from the NSRC, because of the issue of accessibility and management of that material.  Essentially, scientists in those divisions want to ensure that they will be able to continue to have access to their data without fearing that it might become hard to find in the labyrinthine NSRC physical and digital collection space.  This has created situations where researchers have reached out to the NSRC for documents that exist outside of the NSRC and are therefore unavailable to those researchers unless they have a personal connection to the author(s) of the documents in question.  Within a scientific setting, this can have the unfortunate effect of forcing researchers to repeat work that has already been done, particularly in situations where the work was unsuccessful and the researchers would have benefitted from that knowledge in order to modify their own approach or simply not pursue that approach in the first place.  While some progress has been made in terms of the NSRC’s outreach to these other divisions in an effort to encourage them to make their data more accessible, the problem of discoverability remains. In order to address these issues with stewardship and data sovereignty, members of the NSRC’s staff, along with staff from other divisions at LANL, are working on a project to adapt an LLM to make the NSRC’s collections more accessible.  We plan to train the LLM on our collections and enhance its accuracy through the use of an ontology and retrieval augmented generation (RAG).  Ontologies are an important tool used with LLMs to create a concrete vocabulary and clear relationships between entities, and because of the specialized nature of the collections in the NSRC, our ontology will be developed in-house.  When integrated into a RAG approach, the ontology will enhance the accuracy of the LLM and reduce hallucinations, which are a significant concern for any organization attempting to adapt generative AI but even more significant within a national security setting.  Once we have such a system in place, it is our hope that this will help to create a cultural shift in institutional practice, such that other divisions that have maintained their own collections because of concerns over data sovereignty and discoverability will add their data to the LLM as well.  That, in turn, will make the material accessible to other researchers who need it for their work.  Our goal is to have a working prototype of this system in place within three years so that LANL staff can more effectively meet the national security needs of the United States. For much of its history, LANL’s staff has been faced with the task of generating cutting edge scientific and engineering knowledge.  However, stewardship of that knowledge and data sovereignty were afterthoughts.  Now, the NSRC has the task of undertaking stewardship of decades of knowledge and making it discoverable as well as preserving data sovereignty.  My presentation will focus on how this LLM tool will help the NSRC meet this longstanding challenge. 

Lightning talk: 5 mins, Lizabeth Johnson, Los Alamos National Laboratory

Miguel Escobar Varela. AI-Assisted Analysis of Malay-Language Periodicals in Singapore

This presentation will offer an overview of the early stages of a collaborative project that brings historians, digital humanists and computer scientists together to explore the historical significance and cultural connections of Malay-language periodicals published in Singapore from the late 19th to mid-20th century. The study will utilize a unique methodology that combines optical character recognition (OCR) and large language models (LLMs) to convert newspapers and magazines written in Jawi (the Arabic script adapted to Malay phonemes) to Rumi (Malay in Roman characters), enabling the processing and analysis of a vast corpus of historical periodicals at an unprecedented scale. The project's innovative approach involves the use of AI techniques for transcription, segmentation, and document-flow identification, allowing for the efficient and accurate processing of the historical materials. By leveraging LLM models fine-tuned on historical Malay-language corpora, the research team aims to develop a range of specialized models for specific tasks, such as narrative modelling, nuanced named entity recognition (NER), and theme identification. These smaller, task-specific models are expected to outperform state-of-the-art commercial solutions in terms of accuracy when dealing with culturally and historically specific content, while also having a reduced carbon footprint due to their lower computational resource requirements. The AI-assisted analysis of the Malay-language periodicals will facilitate comprehensive research on a wide array of topics, including natural disasters, economic history, and the study of cultural performances as documented in these historical sources. By examining these themes through a comparative lens, the project seeks to shed light on the interconnectedness of Singapore's history with the broader Malay-speaking world during the late colonial period. The project's significance lies in its potential to unlock valuable information from historical sources that have previously been inaccessible or difficult to analyse due to the sheer volume of the material, and declining levels of familiarity with the Jawi script amongst Malay speakers in Singapore. By harnessing cutting-edge AI technologies, the research team aims to contribute to a deeper understanding of Singapore's rich cultural heritage and its place within the region during a pivotal period in its history. Moreover, the project serves as a demonstration of the potential for AI-assisted analysis in the field of historical research, particularly when dealing with non-Western languages and scripts. The methodology developed through this study could be adapted and applied to other historical contexts and languages, opening up new avenues for scholarly inquiry and cross-cultural understanding. We seek to also contribute to the burgeoning open source community currently experimenting with best practices for fine-tuning LLMs for important analytical tasks in the humanities. This research project represents a significant step forward in the study of Singapore's history and its cultural connections to the Malay-speaking world. By combining innovative AI technologies with a comparative perspective, the project aims to provide new insights into the historical significance of Malay-language periodicals and their role in shaping Singapore's cultural heritage. The project's findings are expected to contribute to a more comprehensive and nuanced understanding of Singapore's past, while also showcasing the potential of AI-assisted analysis in unlocking the wealth of information contained within historical sources. 

Presentation: 15 mins, Miguel Escobar Varela, National University of Singapore

Neil Fitzgerald. Digitisation was only the start

For around the last thirty years the British Library has digitised its collections. Historically this work was undertaken not from core grant-in-aid funding, but via external sources including research, philanthropic, and commercial. Starting with boutique digitisation of its Treasures, unique manuscripts, often visually impressive, with the end goal presentation online. Encouraged by the prevailing government ethos of the day, the Library’s 2001 New Strategic Directions plan included an enabling strategy of, ‘Using partnerships to achieve more by working together than they could do on their own,’ which drove digitisation of materials for other purposes, including to service the growing commercial family history market. Later, public/private partnerships drove mass digitisation initiatives, such as the Microsoft Digitisation Project , whose output underpinned much of the early BL Labs work. What lessons learned can we apply to the current AI/machine learning situation? Institutional adoption of new technologies and methods can be disjointed. There are many reasons for this, including required organisational and practise change, which in the UK has been compounded by the tightening fiscal environment over many years. Even when evidence for action has been justified and accepted, implementation requires sustained effort. For fields such as AI/ML that are going through such rapid change, this can create higher perceived organisational risks that need to be managed in an appropriate way. We can draw on other relevant organisational experiences where we have built up knowledge over time to inform approach. The British Library’s Digital Research Team was established in 2012 in response to a strategic review of the Library’s then Scholarship & Collections Department, with one of the key aims to better support digital scholarship. The British Library’s collections and scholarship are international. We would not be able to operate effectively without our global connections across academia, other GLAM organisations, and participation in consortia and communities of interest, across a range of mutually beneficial shared interests, and with users. One example of this is in the field of automatic transcriptions, initially through optical character recognition (OCR) and more recently handwritten text recognition (HTR). The Library’s interest in this area started following two large digitisation newspaper projects, funded by JISC , undertaken in the early to mid-2000s. These showed the importance of usable OCR for text and data mining research methods, and led directly to the pan-European IMPACT Project , who’s aim was to make digitisation, ‘Better, Faster & Cheaper’, especially for the type of historical materials found in memory institutions , and also demonstrate that the approaches developed had the potential to apply to other less supported scripts/languages. We have continued to engage with different OCR/HTR initiatives, especially for low resource languages present in the Library’s collections. One strategy to accomplish this has been OCR/HTR competitions that invite researchers to tackle the challenge of generating accurate and scalable transcription of historical texts. As a collection holding and Independent Research Organisation , we are interested in both advancing the state of the art, but also translating this research into operational and production workflows. I will explore potential approaches available to overcome barriers to applying state of the art research, including collaboration via open frameworks and mutuality - an alternative path to reliance on purely commercial options that enables everyone to contribute to, and benefit from, the value chain. Consideration will be given to the implications of these approaches and the necessity for collection holding institutions to play an active role in shaping AI/ML across this landscape for wider benefit. 

Presentation: 30 mins, Neil Fitzgerald, The British Library

Peter Leonard. Beyond ChatGPT: Transformers Models for Collections

Advances in generative AI have occurred so quickly that their ultimate impact on GLAM institutions remains hard to ascertain. Nevertheless, Transformers-based neural networks have already made an indelible impact on both popular and academic conversations. Cultural heritage work stands to benefit from the new possibilities engendered by the implications of the mathematical concept of “self-attention” inherent in these models. Transformers-based models have shown that many complicated problems in culture beyond digital text — a human voice, a handwritten word, even a scene from a motion picture –– are now tractable to computation. Crucially, these developments are happening at the same moment that the “conversational interface” of ChatGPT has occasioned the most fundamental re-conceptualization of human-computer interaction since Douglas Englebart’s mouse. The implications for memory institutions are thus dual: that more and more complex human culture will be subject to computation, and that chat-based interfaces may prove to be too popular to ignore as ways of interacting with this cultural material. The first of these may seem to be an unalloyed good. Transformers hold great potential towards creating description or transcription for accessibility, helping libraries meet their legal and ethical requirements for online collections. They may also provide baseline description for backlogs of uncataloged digital images and media.  One of the most recent developments is the availability of "multi-modal" large language models, which are capable of processing images and motion pictures. Importantly, all of this work can be done locally, with open-source networks running on hardware disconnected from the infrastructure (and business plans) of commercial companies.  This talk will examine work with several of these non-commercial networks, drawn from experience on Stanford’s collections, including digitized photographic negatives, manuscript collections from the 19th century, oral histories of minoritized campus populations, and other archives. The second of these implications is more complicated. All of these networks, and their various representations of culture through mechanisms such as embeddings, are potential sites of archival engagement through chat.  Although many are skeptical of the precision, determinism, and honesty of “instructable” large language models, GLAM institutions should nevertheless experiment and discover what conversational modalities can offer their patrons.  Enabling researchers to query a collection or “chat with an archive” can provide a compelling and familiar entry point for massive digital collections, whether for novice users or scholars with knowledge of the data. This talk will show some preliminary work on “chatting with archives” in the domain of Silicon Valley history and 1970s performance art videos. 

Presentation: 30 mins

Peter Leonard & Lindsay King, Stanford University Libraries

Rasa Bočytė. Against Al Narratives

It is too easy to forget the pervasive impact of the language, visuals and metaphors that we use to describe our relation to AI. Mainstream narratives are often oversimplications of AI's capabilities and potential impact and depict AI’s “transformative potential” and “inevitable adoption”. They use imagery that exclusively depict white collar jobs, anthropomorphised robots and context-deprived scenes reminiscent of science fiction films [1]. Not only does this distort the public's understanding of AI capabilities which often results in bewilderment and fear, it also pushes AI developers and researchers towards imagining a specific future scenario that they should be building. In fact, big tech companies have harnessed such narratives to justify large-scale investments in AI capabilities over addressing much more pressing issues concerns related to social justice, (mis)representation and user agency [2]. By using these narratives and tropes, we risk subconsciously perpetuating and reinforcing such AI development in libraries, archives and museums (LAMs) rather than advocating for more nuanced and balanced approaches to discussing both the potential benefits and challenges of AI. LAMs can be places where the meaning of AI is defined, where reflection on its role and significance take place and where public values are promoted [3]. This raises important questions for our sector: If we refuse to use mainstream narratives, how can we build alternative visions of our relation to AI? What kind of futures could the AI4LAM community conjure by using new metaphors and tropes? And what role could these narratives help to define for LAMs in AI discourse? This hands-on workshop invites the AI4LAM community to start collectively creating alternative AI narratives and actively shape the discourse about AI in LAMs. During the session, participants will work in groups on a narrative format chosen by the groups - for instance, a manifesto, map, poem, collage, etc. The session moderators will guide the groups during the creative process to address four key narrative elements: Perspective - participants will reflect on the relation between the LAM community and AI. Who is influencing who in this relationship? Specifically, we will discuss the perspective changes when thinking about “AI for culture” and “culture for AI” [4]; Actors - who should sit at the table when AI adoption is discussed? We will consider how the presence of different experiences and knowledge can shape engagement with AI in LAMs; Language - in their narratives, we will critically reflect on phrases, adjectives and verbs commonly-used to describe AI and create their own vocabularies; Visuals - instead of falling into stereotypical tropes of futuristic robots, participants will be inspired by alternative visuals that aim to be much more inclusive and realistic. At the end of the session, we will collectively discuss the narratives that emerged and how they could impact the work with AI in LAMs. The exercise will help participants develop a more critical eye towards AI narratives and equip them with concrete strategies to create alternative ones. The workshop serves as an invitation for the AI4LAM community to consider the importance of this topic in its future activities. If accepted, the authors would like to coordinate with the conference organisers on how the narratives produced during the workshop could be further discussed in the community. For instance, during the conference we could do a lightning talk to present a snapshot of these narratives or discuss them during a future AI4LAM community call. References: [1] Vrabič Dežman, D. Promising the future, encoding the past: AI hype and public media imagery. AI Ethics (2024). https://doi.org/10.1007/s43681-024-00474-x. [2] https://www.dair-institute.org/tescreal/ 
[3] Thiel S, Bernhardt JC, eds. AI in Museums: Reflections, Perspectives and Applications. Transcript Verlag; 2024. 4]  Netherlands AI Coalition. The Art of AI for All: The connective power of culture and media (2021) https://nlaic.com/wp-content/uploads/2022/02/The-Art-of-AI-for-All.pdf; 

Hands on (hack): 2 hours, Rasa Bočytė & Johan Oomen, Netherlands Institute for Sound & Vision

Workshop: Rasa Bočytė, Netherlands Institute for Sound & Vision & Basil Dewhurst, National Film and Sound Archive of Australia 

Terhi Nurmikko-Fuller. Mapping Indigeneity in Institutional Repositories

Since the blockbuster release of Chat-GPT, AI has gone mainstream, and industry partners are increasingly eager to feed their large language models with structured datasets capturing domain-expertise and real-world data. In this paper we will argue that what is needed in this age of AI is not just more data, but smarter data. What we need is knowledge that has been represented in ways that capture various perspectives and nuances. In doing so we can start to look at ways to reduce historical and social bias, and to maximise the power and potential of bringing thoughtfully and CAREfully mapped data and increasingly powerful algorithms together. With aspirations to embed the United Nations Declaration of the Rights of Indigenous Peoples (UNDRIP) in its business processes, the Australian National University (ANU) has set upon a challenge of identifying all content relating to Indigenous research across its myriad (and of course completely disparate) repositories, collections, libraries, and other information custodians. This already formidable task is complicated by the different schemas used to represent heterogeneous collections. It is made even more difficult by the lack of any metadata level record of Indigeneity. Prior attempts to map Indigenous research or research that can be considered to be “Indigenous” highlight the complexity of the situation. For example, at one extreme, some data custodians may hold the view that all research carried out in Australia would be considered to have connections, through Country, to Indigeneity; scientists studying frogs are on Indigenous land; engineers build drones that fly in Indigenous skies. At the other end of the spectrum are those who may struggle to see that connection especially in fields that do not readily pertain to Indigenous knowledge, art, culture, or lived experience as a domain of study. With the absence of explicit recording of “Indigeneity” as a specific field in the metadata records, and within a void created by the lack of an existing, comprehensive, university-wide policy on recognising and representing research in terms of engagement with Indigenous data, researchers, and locations, we adopted a preliminary, pragmatic approach of identifying relevant content in three vast datasets: ANU ARIES, which contains all the metadata from all the publications written by all ANU academics; ANU Theses, which contains all the metadata from all the theses completed at the ANU; and, ANU Collections. Here, we report on a Linked Data project of institutional proportions. It is a critical evaluation of the challenges and opportunities the process of converting the metadata from these three internally complex knowledge repositories to the machine-readable format of RDF using repository-specific structures that combine top-level ontologies such as Schema.org with bespoke elements to represent this data in a way that adheres to international standards but is expanded to capture implicit connections and tacit knowledge. We already know of examples in industry where knowledge graphs (KG) are used to inform large language models (LLMS): here is our opportunity to provide LLMs with not just large datasets, but with CAREfully curated KGs that provide not just data points, but contextualised information capturing knowledge about and from Indigenous contexts. In the first instance, we have sought to explicitly represent publications, theses, and objects that have an explicit connection to Indigenous languages, to Indigenous places, or were created, analysed, or otherwise engaged with in projects where the supervisor has been an Indigenous researcher. Going forward, our aspiration is to expand the underlying KG to incorporate relationships that represent Indigenous ways of knowing. The existing data would be enriched with additional properties that show non-Western knowledge structures as different but equally valid ways of meaning-making and semantic capture of the world. In this way, we seek to take preliminary steps towards realising a project that can help tackle issues of intellectual colonialism, diversify the dominance of Western knowledge structures, and indeed help us towards a fantastic future. What we seek to provide are reproducible steps and methodological guidelines that can be utilised by other institutions to tentatively start to tackle the task of identifying Indigenous content in their heterogenous collections.;  

Authors: Terhi Nurmikko-Fuller, Adegboyega Adeniran, Janet McDougall, Lawson Lewis, Nicholas Car, Sam Provost, Helen Wright. 

Presentation: 15 mins, Terhi Nurmikko-Fuller, Australian National University