Data processing notes

What is DATA INFORMATION  

The word "data" is the plural of datum, which means fact, observation, assumption or occurrence. More precisely, data are representations of facts pertaining to people, things, ideas and events. Data are represented by symbols such as letters of the alphabets, numerals or other special symbols.  Information can be defined as “data that has been transformed into a meaningful and useful form for specific purposes”. 

What is DATA PROCESSING  

Data processing is the act of handling or manipulating data in some fashion. Regardless of the activities involved in it, processing tries to assign meaning to data. Data processing is the process through which facts and figures are collected, assigned meaning, communicated to others and retained for future use.  Hence we can define data processing as a series of actions or operations that converts data into useful information.

STEPS IN DATA PROCESSING 

1.  Collection  

2. Conversion  

3. Manipulation  

4. Storage  

5. Communication 

COLLECTION  

Data originates in the form of events transaction or some observations.  This data is then recorded in some usable form. Data may be initially recorded on paper source documents and then converted into a machine usable form for processing.  Alternatively, they may be recorded by a direct input device in a paperless, machine-readable form. Data collection is also termed as data capture. 

CONVERSION  

Once the data is collected, it is converted from its source documents to a form that is more suitable for processing. The data is first codified by assigning identification codes.  A code comprises of numbers, letters, special characters, or a combination of these It is useful to codify data, when data requires classification.  To classify means to categorize, i.e., data with similar characteristics are placed in similar categories or groups. For example, one may like to arrange accounts data according to account number or date. Hence a balance sheet can easily be prepared. MANIPULATION  Once data is collected and converted, it is ready for the manipulation function which converts data into information. Manipulation consists of following activities: 1. Sorting 2. Calculating 3. Summarizing 4. Comparing 

MANAGING THE OUTPUT RESULTS 

Once data has been captured and manipulated following activities may be carried out :  Storing To store is to hold data for continued or later use.  Retrieving To retrieve means to recover or find again the stored data or information. Retrieval techniques use data storage devices. 

COMMUNICATION  

Communication is the process of sharing information. Thus, communication involves the transfer of data and information produced by the data processing system to the prospective users of such information or to another data processing system.  As a result, reports and documents are prepared and delivered to the users. In electronic data processing, results are communicated through display units or terminals. 

REPRODUCTION  

To reproduce is to copy or duplicate data or information.  This reproduction activity may be done by hand or by machine. 

DATA ORGANISATION 

Data can be arranged in a variety of ways, but a hierarchical approach to organization is generally recommended.  Data Item A data item is the smallest unit of information stored in computer file.  Field Data items are physically arranged as fields in a computer file. Their length may be fixed or variable.  Record A record is a collection of related data items or fields. Each record normally corresponds to a specific unit of information.  File The collection of records is called a file. A file contains all the related records for an application.  Database The collection of related files is called a database. A database contains all the related files for a particular application.

 Difference between data and information

• Data processing converts raw data into useful information

•  Data storage hierarchy commonly used to facilitate data processing

• Standard methods of organizing data

• Basic concepts of database systems

Without data processing, companies limit their access to the very data that can hone their competitive edge and deliver critical business insights. That's why it's crucial for all companies to understand the necessity of processing all their data, and how to go about it.

• Data is a collection of facts – unorganized but able to be organized into useful information

• Information is data arranged in an order and form that is useful to the people who receive it

• Data processing is a series of actions or operations that converts data into useful information

• A data processing system includes resources such as people, procedures, and devices used to process input data for producing desirable output

What is data processing ?

Data processing occurs when data is collected and translated into usable information. Usually performed by a data scientist or team of data scientists, it is important for data processing to be done correctly as not to negatively affect the end product, or data output.

Data processing starts with data in its raw form and converts it into a more readable format (graphs, documents, etc.), giving it the form and context necessary to be interpreted by computers and utilized by employees throughout an organization.

Six stages of data processing

1. Data collection

Collecting data is the first step in data processing. Data is pulled from available sources, including data lakes and data warehouses. It is important that the data sources available are trustworthy and well-built so the data collected (and later used as information) is of the highest possible quality.

2. Data preparation

Once the data is collected, it then enters the data preparation stage. Data preparation, often referred to as “pre-processing” is the stage at which raw data is cleaned up and organized for the following stage of data processing. During preparation, raw data is diligently checked for any errors. The purpose of this step is to eliminate bad data (redundant, incomplete, or incorrect data) and begin to create high-quality data for the best business intelligence.

3. Data input

The clean data is then entered into its destination (perhaps a CRM like Salesforce or a data warehouse like Redshift), and translated into a language that it can understand. Data input is the first stage in which raw data begins to take the form of usable information.

4. Processing

During this stage, the data inputted to the computer in the previous stage is actually processed for interpretation. Processing is done using machine learning algorithms, though the process itself may vary slightly depending on the source of data being processed (data lakes, social networks, connected devices etc.) and its intended use (examining advertising patterns, medical diagnosis from connected devices, determining customer needs, etc.).

5. Data output/interpretation

The output/interpretation stage is the stage at which data is finally usable to non-data scientists. It is translated, readable, and often in the form of graphs, videos, images, plain text, etc.). Members of the company or institution can now begin to self-serve the data for their own data analytics projects.

6. Data storage

The final stage of data processing is storage. After all of the data is processed, it is then stored for future use. While some information may be put to use immediately, much of it will serve a purpose later on. Plus, properly stored data is a necessity for compliance with data protection legislation like GDPR. When data is properly stored, it can be quickly and easily accessed by members of the organization when needed.