Visual selection processes in real-world scenes are guided by exogenous and endogenous factors. Endogenous factors correspond to top-down influences like the cognitive tasks or prior knowledge. Object processing is affected by the gist of the scene within which it is embedded and the prior knowledge about the objects. On one hand, it has been shown that incongruent objects result in prolonged and more frequent eye-fixations than congruent objects (Underwood et al, 2008; Võ & Henderson, 2009). On the other hand, previous event-related potential (ERP) research has suggested that manipulating the semantic congruency between an object and the surrounding scene affects the high level representation of that object (Ganis & Kutas, 2003). The congruency effect is reflected by a late ERP resembling the N300/N400 effect previously associated with semantic integration (Mudrik, Lamy, & Deouell, 2010). The present study investigates the effect of semantic congruency on scene processing using eye-fixation related potentials (EFRPs). We simultaneously registered electroencephalographic (EEG) and eye-tracking signals of participants exploring natural scenes during 4 sec in preparation for a recognition memory test. We compared EFRPs evoked by congruent vs. incongruent eye-fixations (e.g., a fork in a kitchen vs. the same fork in a bathroom). First, we replicated previous eye movement results, showing that incongruent objects were more fixated and for a longer duration than congruent objects. Second, the EFRP analysis revealed that both early and late EFRPs were influenced by the congruency effect. The P1 EFRP and a late EFRP emerging around 260 ms after the fixation onset were modulated by semantic congruency. The top-down encoding of the scene was built during the first eye fixations; a mismatch between the semantic knowledge of objects and the features of the scene affected the scene exploration. These results suggest that top-down information influences early object processing during natural viewing.