When using "processing time" in Apache Flink, it means that events are processed based on the system clock time of the machine running the Flink job. While this approach is simple and efficient, it can lead to incorrect results in certain scenarios. Here's a simple explanation of how using "processing time" can give wrong results:
Event Arrival Order: In a distributed system, events can arrive out of order due to various factors like network delays or parallel processing. With "processing time," events are processed as soon as they arrive without considering their actual order. This can lead to incorrect results if the order of events is crucial for your application's correctness.
Event Lateness: "Processing time" doesn't consider the timestamps associated with events. If events are delayed or arrive late due to factors like buffering or system load, "processing time" cannot account for this lateness. As a result, time-based operations like windowing or aggregations might include late events or exclude events that should have been processed.
Event Time Skew: Events generated by different sources or devices may have different clocks or time zones, causing event time skew. "Processing time" doesn't account for these time differences, leading to inaccurate results when events from different sources are processed together.
System Clock Drift: System clocks on different machines may not be perfectly synchronized, resulting in clock drift. "Processing time" relies on system clocks, so if the clocks drift significantly, it can lead to inconsistent timestamps and incorrect event ordering.
To address these issues, Flink provides the concept of "event time." With "event time," events are processed based on their inherent timestamps, which are often provided as part of the event data. By correctly assigning timestamps and using watermarks to track event time progress, Flink can handle out-of-order events, event lateness, and event time skew.
Using "event time" ensures that events are processed in the order they occurred, accounts for event lateness, and allows for correct handling of time-based operations. However, working with "event time" introduces additional complexities, such as defining custom timestamp assigners and handling watermarks correctly, to ensure accurate results.