Title: Bridging Fault Lines in Public Data: From Data Engineering to Evidence-Based Decision Making
Abstract: Public institutions in India generate vast volumes of administrative and financial data across systems that are often designed for reporting and administration rather than analysis. These systems are usually fragmented across departments, platforms and databases, with differences in definitions, formats, reporting structures and levels of data quality. Therefore, the fault lines in the underlying data propagate through systems and ultimately distort the evidence used for policy decisions. This fragmentation is not incidental but structural, persists across organisations, and only a minority of public organisations report having a comprehensive operational view of their own data. Further, these fault lines do not stay contained within the systems that produce them, they propagate outward and distort the evidence base on which policy decisions are made. Systematic reviews of evidence-informed policymaking consistently identify such institutional deficits. This is further compounded by siloed decision-making and misaligned incentives across agencies, which often prevent the same data from being reused or reconciled even where the technical means exist. Lastly, analytical failure in evidence-based decision-making is rarely just the data problem; it is also a state capacity and capability issue. Layered on top of data design failure is the capacity failure. The individual-level competencies and organisational resources needed to convert raw data into usable evidence remain unevenly distributed within public institutions. These three fault lines, i.e., system design, analytical capability, and state capacity, are not independent failures to be addressed piecemeal but rather a single, connected chain running from system design to policy evidence.