Login | Register

Tracking eye movements to uncover the nature of visual-linguistic interaction in static and dynamic scenes


Tracking eye movements to uncover the nature of visual-linguistic interaction in static and dynamic scenes

Van de Velde, Caroline (2008) Tracking eye movements to uncover the nature of visual-linguistic interaction in static and dynamic scenes. PhD thesis, Concordia University.

Text (application/pdf)
NR37735.pdf - Accepted Version


These studies examined the role of sentence and visual context in the access to verb-complement information, using a new eye tracker and change blindness paradigm. Participants' eye movements were monitored as they viewed pictures of objects (Experiment 1) or dynamic scenes (Experiment 2), and listened to related sentences. In Experiment 1, two sets of sentences were contrasted, a highly constraining causative construction in which there was a close conceptual relation between the verb and its direct object (e.g., "The woman burned the candle ") and a neutral construction with a transitive perception verb (e.g., "The woman admired the candle "). Starting at three different points within the presentation of the verb (onset, middle, offset) and noun (onset, offset, offset+200 ms), participants' task was to look and indicate whether the target objects mentioned in the sentences (e.g., " candle ") were present in the visual displays. Results indicate that semantic information extracted at the verb can be used to constrain the domain of reference in the scene and in some cases predict the referent of the grammatical complement of the verb, depending on tasks demands, conceptual consolidation, of the scene, and the presence of competitor objects. In Experiment 2, two different classes of verbs were contrasted, denominal and non-denominal verbs, which either implicitly (e.g., "The woman will iron the shirt ") or explicitly (e.g., "The woman will chop the vegetables with the knife ") named the instrument nouns. These movies were edited, unbeknownst to the participants, so that the real referents of the verb's grammatical object (e.g., " shirt/vegetables ") or instrument (e.g., " iron/knife ") gradually dissolved. Participants' task was to provide a detailed description of each scene, and later perform a recognition task. Although results indicate that information extracted from the sentence helped identify, describe and remember scene details, visual context seems to take precedence over linguistic input properties in guiding eye movements. In conclusion, the processor appears to incrementally integrate all available knowledge, linguistic and non-linguistic, with the aim to rapidly interpret the linguistic description of what we see in the world and how we may interact with it.

Divisions:Concordia University > Faculty of Arts and Science > Psychology
Item Type:Thesis (PhD)
Authors:Van de Velde, Caroline
Pagination:xv, 300 leaves : ill. ; 29 cm.
Institution:Concordia University
Degree Name:Ph. D.
Thesis Supervisor(s):Von Grünau, Michael
ID Code:975644
Deposited By: Concordia University Library
Deposited On:22 Jan 2013 16:12
Last Modified:18 Jan 2018 17:40
Related URLs:
All items in Spectrum are protected by copyright, with all rights reserved. The use of items is governed by Spectrum's terms of access.

Repository Staff Only: item control page

Downloads per month over past year

Back to top Back to top