2020 IASA - FIAT/IFTA Joint Conference

Machine learning based technologies have spawned a number of technical pilots, prototypes and proof-of-concepts in recent years, but how to measure which ideas and visions should be further refined into everyday tools and workflows for media archives and media companies? This session presents findings from the MeMAD project (memad.eu) end user evaluations where archivists and other media professionals tried out different combinations of automatically created metadata in their daily tasks.

We present and discuss alternative approaches and strategies for evaluating the usefulness and quality of tools like automated speech recognition (ASR), face recognition, machine translation and named entity recognition (NER), supported by case examples from the project evaluation sessions. This session also describes potential winning combinations of these technologies for different user groups and tasks

Instead of technical performance metrics of different algorithms, we focus on topics such as end user experience and overall productivity in the media industry context. This session also discusses different requirements and preconditions that our archive tools and data need to meet to be able to perform these types of evaluations, as well as observed differences between human based and machine based content description practices.

MeMAD is an Horizon 2020 research and innovation project. This session is co-presented by two of the project partners: Yle, the Finnish Broadcasting Company representing the industry needs in the project and Limecraft providing and developing the project prototype platform that has been used in the project evaluations.