Searching Large Audiovisual Archives

Efficient search for information in large volumes of oral history audiovisual data came into prominence roughly at the turn of the century when substantial amounts of archive materials (stored so far on films, video tapes and various analogue audio storage devices) started to be digitized and, at the same time, digital personal recording devices became so affordable that also the amount of newly generated content grew with a geometric rate. One of the first digital archives that was in need for efficient search capabilities were the recordings of testimonies given by the Holocaust survivors, collected by the Survivors of the Shoah Visual History Foundation (now USC Shoah Foundation). The consortium of research teams from the US and the Czech Republic was established in 2001 and started to build a system that used automatic speech recognition and information retrieval techniques to give users an effective and user-friendly way of accessing the information contained in the archive.

The aim of the talk is to provide a detailed overview of the core methods that our team has been using in both automatic speech recognition (from HMM-based systems used at the beginning to modern end-to-end neural models that are used today) and information retrieval (from the heuristic approach that was implemented about 10 years ago as a proof-of-concept to the current transformer-based solution). We will also show the progress in the system's performance over the two decades of work on this task, as well as the evolution of the graphical user interface used for the actual access to the collection.