Tôt and tard are irregular adverbs.
Tôt means early in a general, non-measurable sense: early in the day, or in one's life, one's career, relationship, in a film, book, etc.
Tard means late in the sense of late in the day (or in one's life, career, etc.).
En avance (early), en retard (late), à l'heure (on time) are invariable phrases: they don’t change. They always refer to a schedule: en avance means early in the sense of ahead of schedule and en retard means late, behind schedule. They are usually used with 'être' and verbs of movement like 'arriver', 'partir', 'rentrer', 'revenir': the verbs that take in other words, the verbs from the Maison d’être. They are placed at the end of the sentence.