Object processing is affected by the gist of the scene within which it is embedded. Incongruent objects result in prolonged and more frequent eye-fixations than congruent objects. In parallel, previous event-related potential (ERP) research has suggested that the congruency effect is reflected by a late ERP resembling the N300/N400 effect. The present study investigates the effect of semantic congruency on scene processing using eye-fixation related potentials (EFRPs). We simultaneously registered electroencephalographic (EEG) and eye-tracking signals of participants exploring natural scenes in preparation for a recognition memory test. We compared EFRPs evoked by congruent vs. incongruent eye-fixations (e.g., a fork in a kitchen vs. the same fork in a bathroom). First, we replicated previous eye movement results, showing that incongruent objects were fixated more and longer than congruent objects. Second, the EFRP analysis revealed that the P1 EFRP and a later EFRP emerging around 260 ms after the fixation onset were modulated by semantic congruency. The top-down encoding of the scene was built during the first eye fixations; a mismatch between the semantic knowledge of objects and the features of the scene affected scene exploration. These results suggest that top-down information influences early object processing during natural viewing.