Abstract

Driven by a larger and more diverse user-base and datasets, modern Information Retrieval techniques are
striving to become contextually-aware in order to provide users with a more satisfactory search experience.
While text-only retrieval methods are significantly more accurate and faster to render results than purely
visual retrieval methods, these latter provide a rich complementary medium which can be used to obtain
relevant and different results from those obtained using text-only retrieval. Moreover, the visual retrieval
methods can be used to learn the user’s context and preferences, in particular the user’s relevance feedback,
and exploit them to narrow down the search to more accurate results. Despite the overall deficiency in
precision of visual retrieval result, the top results are accurate enough to be used for query expansion, when
expanded in a controlled manner.
The method we propose overcomes the usual pitfalls of visual retrieval:
1. The hardware barrier giving rise to prohibitively slow systems.
2. Results dominated by noise.
3. A significant gap between the low-level features and the semantics of the query.
In our thesis, the first barrier is overcome by employing a simple block-based visual features which
outperforms a method based on MPEG-7 features specially at early precision (precision of the top results).
For the second obstacle, lists from words semantically weighted according to their degree of relation to
the original query or to relevance feedback from example images are formed. These lists provide filters
through which the confidence in the candidate results is assessed for inclusion in the results. This allows
for more reliable Pseudo-Relevance Feedback (PRF). This technique is then used to bridge the third barrier;
the semantic gap. It consists of a second step query, re-querying the data set with an query expanded with
weighted words obtained from the initial query, and semantically filtered (SF) without human intervention.
We developed our PRF-SF method on the IAPR TC-12 benchmark dataset of 20,000 tourist images, obtaining
promising results, and tested it on the different and much larger Belga benchmark dataset of approximately
500,000 news images originating from a different source. Our experiments confirmed the potential of
the method in improving the overall Mean Average Precision, recall, as well as the level of diversity of the
results measured using cluster recall.