Information Society Technologies

Search In Audio Visual Content Using Peer-to-peer IR

IST FP6 Project


Demos

Content-based image and video search in large audio-visual collections

An online content based image and video search in a collection of 100M images and several hours of BBC news videos is now available at http://sapir.isti.cnr.it/index. The engine can search through images or videos based on similarity to an existing image, text metadata, spoken content, and GPS coordinates. This demo is a result of the FP6 EU SAPIR project (http://www.sapir.eu) for large scale search in audio-visual content using peer-to-peer information retrieval (P2P IR) and was developed through the collaboration of IBM, CNR, MU-BRNO, MPII and Eurix.

The search engine’s web interface developed by CNR allows users to conduct multi-modal queries using a combination of query-by-example image, GPS coordinates and text. The queries are executed by a multi-layer P2P overlay network where the image and GPS part are sent to MUFIN (Multi-Feature Indexing Network, http://mufin.fi.muni.cz), the text part as well as tag-value conditions are sent to MINERVA (http://www.mpi-inf.mpg.de/departments/d5/software/minerva/index.html) and the video and spoken part are sent to IBM’s indices (http://www.haifa.ibm.com/dept/imt/mst.html). The results are then fused by IBM middleware to find the best matches, taking into account both the image context and text. Video streaming and delivery is done by Eurix.

The SAPIR demo image collection includes 100 million images extracted from Flickr by CoPhir - (Content-based Photo Image Retrieval, http://cophir.isti.cnr.it/). Each image contains five MPEG-7 global visual descriptors for: color structure, color layout, scalable color, edge histogram, and homogeneous texture. The images also contain any tags or comments manually added by Flickr users, which, it should be noted, may not always describe the image accurately.

The demo is available thanks to the IBM Shared University Research (SUR) Award - a worldwide equipment award program designed to promote research in areas of mutual value and interest to IBM and universities.

For more information, contact Yosi Mass, yosimass@il.ibm.com




 
 

Partners

  
IBM Haifa Research Laboratory
IBM Haifa Research Laboratory
 
 
ISTI - CNR
ISTI - CNR
 
 
Max-Planck Institute for Informatics
Max-Planck Institute for Informatics
 
 
University of Padova
University of Padova
 
 
Eurix
Eurix
 
 
Xerox Research Centre Europe
Xerox Research Centre Europe
 
 
Masaryk University
Masaryk University
 
 
Telefonica Investigacion y Desarrollo
Telefonica Investigacion y Desarrollo
 
 
Telenor
Telenor