Embedded Files
The data you need might be available but not in a way that you can use it in your current systems. You will need to consider how it can be accessed and what level of processing is required.
It could be that the data is currently in a system or format that does not interact with the exploration or analysis systems. It could also be that the data comes from different systems and formats and will need unifying to a single source of truth that can be accessed through a single system. Both of these cases would require setting up a data engineering pipeline considering all the systems architecture in place.
It could be that the data is accessible and can interact with systems already in place but still needs cleaning before it can be used. This would normally involve tasks such as removing incomplete data, removing duplicate data, finding and correcting or removing errors.
While these jobs should be carried out by those with the relevant skills, the domain expert should stay in the loop to advise when needed.
Organisations may already have systems in place that collect data, or they may have new data requirements that mean new systems or devices are needed to collect data. This often leads to multiple data formats that do not interact in a single system. Even if they do interact in a single system, it may not be the system currently available to the data users. In these cases, the data will need to be sent through an engineering pipeline that will end up with a single source of truth from which the user can extract the data.
A typical example of this would be the desire to connect your MES (Manufacturing Execution System) to your ERP (Enterprise Resource Planning) system for optimised scheduling. The systems don't typically interact unless you have bought them together but new ways of making them interact are being developed.
It could simply be that for more oversight in operations and to identify root cause of failures you might need to compare machining data to image data, or some other sensors and quality measures.
Another, more complex task might be the sharing of data through the supply chain (or even within different departments of the same organisation). The additional difficulty here is gathering the correct information and securely transporting data across locations.
This processing of the data can be thought of as a part of building the digital thread.
Once the data is in a format that can be read into your exploration and analysis system it is likely that it will require a good clean up. There are multiple reasons that data can be in need of cleaning:
Data items are duplicated,
Data is incomplete and can't be used,
Errors are present,
Outliers need investigating,
Units need changing,
Data needs standardising
Data schema/structure are to be implemented.
The reasons for cleaning the data may be practical to ensure that codes can run efficiently and effectively, but it may be determined by the data collection method and the problem being addressed.
Cleaning is not the same as feature transformation where one might want/need to use features of the data instead of the raw data to analyse or visualise particular features. Cleaning is required to be confident in the features that are derived.
The Medallion Structure
The Medallion Structure
We use the Medallion Structure to describe the level of processing of the data as Bronze, Silver and Gold. The medallion structure has been taken from the Databricks data engineering platform. A similar structure is described by CookieCutter using the terms raw, interim and processed data to describe the data engineering workflow.
Bronze/Raw Data
Bronze/Raw Data
The bronze level denotes the raw data and has had no processing. The raw data can be a very large file and should be kept in its original format to ensure reproducibility and traceability.
An efficient way of processing data can be to keep a copy of the processing steps but only keep the raw data in storage. This avoids the storage of many data files.
Silver/Interim Data
Silver/Interim Data
Silver data has been subject to some processing and exploration but may not contain the precise features of the data in a format that is required for visualisation or analysis.
Silver data may be processed and cleaned through a data engineering pipeline, or in the case of very large data files have some processing via an edge device. At this stage, schema and structure may be added to the data.
Gold/Processed Data
Gold/Processed Data
The Gold datasets contain the final features that will be used for a particular analysis. The data will have been processed and checked and may have undergone feature engineering or dimension reduction. If the data is being used for multiple insights there may be multiple Gold datasets.
Page updated
Report abuse