Basic Concepts of Data Analysis 

• In order to clarify the preceding generalities, a few examples are provided: 

1. Socioeconomic surveys: In the interdisciplinary areas of sociology, economics, and political science, such aspects are taken as the economic well-being of different ethnic groups, consumer expenditure patterns of different income levels, and attitudes toward pending legislation. Such studies are typically based on data oriented by interviewing or contacting a representative sample of person selected by statistical process from a large population that forms the domain of study. The data are then analyzed and interpretations of the issue in questions are made. 

2. Clinical diagnosis: Early detection is of paramount importance for the successful surgical treatment of many types of fatal diseases, say, for example, cancer or AIDS. Because frequent in-hospital checkups are expensive or inconvenient, doctors are searching for effective diagnosis process that patients can administer themselves. To determine the merits of a new process in terms of its rates of success in detecting true cases avoiding false detection, the process must be field tested on a large number of persons, who must then undergo in-hospital diagnostic test for comparison. Therefore, proper planning (designing the experiments) and data collection are required, which then need to be analyzed for final conclusions. 

3. Plant breeding: Experiments involving the cross fertilization of different genetic types of plant species to produce high-yielding hybrids are of considerable interest to agricultural scientists. As a simple example, suppose that the yield of two hybrid varieties are to be compared under specific climatic conditions. The only way to learn about the relative performance of these two varieties is to grow them at a number of sites, collect data on their yield, and then analyze the data. 

• In recent years, attempts have been made to treat all these problems within the framework of a unified theory called decision theory. Whether or not statistical inference is viewed within the broader framework of decision theory depends heavily on the theory of probability. This is a mathematical theory, but the question of subjectivity versus objectivity arises in its applications and in its interpretations. We shall approach the subject of statistics as a science, developing each statistical idea as far as possible from its probabilistic foundation and applying each idea to different real-life problems as soon as it has been developed. 

• Statistical data obtained from surveys, experiments, or any series of measurements are often so numerous that they are virtually useless, unless they are condensed or reduced into a more suitable form. Sometimes, it may be satisfactory to present data just as they are, and let them speak for  themselves; on other occasions, it may be necessary only to group the data and present results in the form of tables or in a graphical form. The summarization and exposition of the different important aspects of the data is commonly called descriptive statistics. This idea includes the condensation of the data in the form of tables, their graphical presentation, and computation of numerical indicators of the central tendency and variability. 

There are mainly two main aspects of describing a data set: 

1. Summarization and description of the overall pattern of the data by 

• Presentation of tables and graphs 

• Examination of the overall shape of the graphical data for important features, including symmetry or departure from it 

• Scanning graphical data for any unusual observations, which seems to stick out from the major mass of the data 

2. Computation of the numerical measures for 

• A typical or representative value that indicates the center of the data 

• The amount of spread or variation present in the data 

• Summarization and description of the data can be done in different ways. For a univariate data, the most popular methods are histogram, bar chart, frequency tables, box plot, or the stem and leaf plots. For bivariate or multivariate data, the useful methods are scatter plots or Chernoff faces. A wonderful exposition of the different exploratory data analysis techniques can be found in Tukey (1977), and for some recent development, see Theus and Urbanek (2008). 

• A typical or representative value that indicates the center of the data is the average value or the mean of the data. But since the mean is not a very robust estimate and is very much susceptible to the outliers, often, median can be used to represent the center of the data. In case of a symmetric distribution, both mean and median are the same, but in general, they are different. Other than mean or median, trimmed mean or the Windsorized mean can also be used to represent the central value of a data set. The amount of spread or the variation present in a data set can be measured using the standard deviation or the interquartile range. 

What is qualitative data?

• Qualitative data is non-statistical and is typically unstructured or semi-structured. This data isn't necessarily measured using hard numbers you use to develop graphs and charts. Instead, it is categorized based on properties, attributes, labels, and other identifiers. 

Qualitative data can be used to ask the question, 'why'. It is investigative and asks open-ended questions to conduct the research. Generating this data from qualitative research is used for theorizations, interpretations, developing hypotheses, and initial understandings.

Real-world examples of qualitative data: 

•  Product reviews

• Interview transcripts

• Texts and documents

• Customer testimonials

• Focus group responses

• Notes and observations

• Audio and video recordings

• Survey and questionnaire labels and categories

What is quantitative data?

• Contrary to qualitative data, quantitative data is statistical and typically structured – meaning it is more rigid and defined. This data type is measured using numbers and values, making it a more suitable candidate for data analysis. 

Whereas qualitative is open for exploration, quantitative data is much more concise and close-ended. It can be used to ask 'how much' or 'how many,' followed by conclusive information. 

Real-world examples of quantitative data: 

• Calculations (annual revenue)

• Measurements (height, width, and weight)

• Counts (the number of people who signed up for the webinar)

• Projections (predicted revenue increase as a percentage during a fiscal year)

• Quantification of qualitative data (customer satisfaction score calculation based on ratings on a scale of 1 to 5)

Attributes and Variables

• An attribute is a quality of an object (person, thing, etc.). Attributes are closely related to variables. A variable is a logical set of attributes.Variables can "vary" – for example, be high or low. How high, or how low, is determined by the value of the attribute (and in fact, an attribute could be just the word "low" or "high"). 

Examples:

• Age is an attribute that can be operationalized in many ways. It can be dichotomized so that only two values – "old" and "young" – are allowed for further data processing. In this case the attribute "age" is operationalized as a binary variable. If more than two values are possible and they can be ordered, the attribute is represented by ordinal variable, such as "young", "middle age", and "old".  

Levels of Measurement or Scales of Measurement:

• Levels of measurement, also called scales of measurement, tell you how precisely variables are recorded. In scientific research, a variable is anything that can take on different values across your data set (e.g., height or test scores). 

There are 4 levels of measurement: 

1. Nominal: the data can only be categorized

2. Ordinal: the data can be categorized and ranked

3. Interval: the data can be categorized, ranked, and evenly spaced

4. Ratio: the data can be categorized, ranked, evenly spaced, and has a natural zero.

• Depending on the level of measurement of the variable, what you can do to analyze your data may be limited. There is a hierarchy in the complexity and precision of the level of measurement, from low (nominal) to high (ratio). 

Nominal, ordinal, interval, and ratio data:

• Going from lowest to highest, the 4 levels of measurement are cumulative. This means that they each take on the properties of lower levels and add new properties. 

1. Nominal level: You can categorize your data by labelling them in mutually exclusive groups, but there is no order between the categories. 

• Examples: City of birth, Gender, Ethinicity, Car brands, Marital status, etc.

2. Ordinal level: You can categorize and rank your data in an order, but you cannot say anything about the intervals between the rankings. Although you can rank the top 5 Olympic medallists, this scale does not tell you how close or far apart they are in number of wins. 

• Examples: Top 5 Olympic medallists, Language ability (e.g., beginner, intermediate, fluent), Likert-type questions  (e.g., very dissatisfied to very satisfied),etc.

3. Interval level: You can categorize, rank, and infer equal intervals between neighboring data points, but there is no true zero point.The difference between any two adjacent temperatures is the same: one degree. But  zero degrees is defined differently depending on the scale – it doesn’t mean an absolute absence of temperature.The same is true for test scores and personality inventories. A zero on a test is arbitrary; it does not mean that the test-taker has an absolute lack of the trait being measured.

• Examples: Test scores (e.g., IQ or exams), Personality inventories, Temperature in Fahrenheit or Celsius, etc.

4. Ratio level: ou can categorize, rank, and infer equal intervals between neighboring data points, and there is a true zero point. A true zero means there is an absence of the variable of interest. In ratio scales, zero does mean an absolute lack of the variable. For example, in the Kelvin temperature scale, there are no negative degrees of temperature – zero means an absolute lack of thermal energy.

• Examples: Height, Age, Weight, Temperature in Kelvin, etc.

Why are levels of measurement important?

• The level at which you measure a variable determines how you can analyze your data.

• The different levels limit which descriptive statistics you can use to get an overall summary of your data, and which type of inferential statistics you can perform on your data to support or refute your hypothesis.

• In many cases, your variables can be measured at different levels, so you have to choose the level of measurement you will use before data collection begins.

Example of a variable at 2 levels of measurement: 

• You can measure the variable of income at an ordinal or ratio level. 

1. Ordinal level: You create brackets of income ranges: $0–$19,999, $20,000–$39,999, and $40,000–$59,999. You ask participants to select the bracket that represents their annual income. The brackets are coded with numbers from 1–3.

2. Ratio level: You collect data on the exact annual incomes of your participants.

• At a ratio level, you can see that the difference between A and B’s incomes is far greater than the difference between B and C’s incomes. 

At an ordinal level, however, you only know the income bracket for each participant, not their exact income. Since you cannot say exactly how much each income differs from the others in your data set, you can only order the income levels and group the participants. 

Which descriptive statistics can I apply on my data?

• Descriptive statistics help you get an idea of the “middle” and “spread” of your data through measures of central tendency and variability.

When measuring the central tendency or variability of your data set, your level of measurement decides which methods you can use based on the mathematical operations that are appropriate for each level.

The methods you can apply are cumulative; at higher levels, you can apply all mathematical operations and measures used at lower levels.

• A random variable, or simply variable, is a characteristic of a population or sample. 

Examples: Student grades, which varies from student to student; and stock prices, which varies from stock to stock as well as over time. 

Typically denoted by a capital letter: X, Y , Z. . . T

he values of a variable are possible observations or realizations of that variable. The possible values of a variable usually land in a specified range. Examples: Student Grades: the interval [0, 100]. Stock Prices: nonnegative real numbers. 

• Data are the observed values of a variable. Examples: Grades of a sample of students: {34, 78, 64, 90, 76} Prices of stocks in a portfolio: {$54.25, $42.50, $48.75} 

Data fall into three main groups: 

1. Interval Data: 

• Real numbers, e.g., heights, weights, prices, etc. Also referred to as quantitative or numerical data. 

Arithmetic operations can be performed on interval data, 

2. Nominal Data:

• Names or categories, e.g., {Male, Female} and {single, Married, Divorced, Widowed}. Also referred to as qualitative or categorical data. 

3. Ordinal Data:

• Ordinal Data are also categorical in nature, but their values have an order.

Example: Course Ratings: Poor, Fair, Good, Very Good, Excellent. Student Grades: F, D, C, B, A. 

Taste Preferences: First Choice, Second Choice, Last Choice.  

Tabular Methods

• Tabular method of data presentation is wide spread in all spheres of human life. These methods are used to summarize data from a sample or population into table format. Data is grouped into categories and the number (or frequency) of observations in each category is obtained.

• Frequency distribution is a type of tabular method. A frequency distribution is a tabular summary of data showing the frequency of items in each of several non-overlapping classes. The objective is to provide insights about the data that cannot be quickly obtained by looking only at the original data. 

Graphical Methods

• These methods are applied to visually describe data from a sample or population. The shape of sample data can indicate the shape of the population from which it is taken. Graphs provide visual summaries of data which is more quickly and completely describe essential information than tables of numbers.

• Graphs are essential as these provide insight for the analyst into the data under scrutiny, and illustrate important concepts when presenting the results to others. A graphical method is developed which signifies the accuracy of test results. The graphs can be constructed from Producer's scores and Consumer's scores on each of the scales of test score, antigen dose and probability of protection against disease.

General Rules for Drawing Graphs, Diagrams and Maps 

1. Selection of a Suitable Method:

• Data represent various themes such as temperature, rainfall, growth and distribution of the population, production, distribution and trade of different commodities, etc. These characteristics of the data need to be suitably represented by an appropriate graphical method. For example, data related to the temperature or growth of population between different periods in time and for different countries/states may best be represented using line graphs. Similarly, bar diagrams are suited best for showing rainfall or the production of commodities. The population distribution, both human and livestock, or the distribution of the crop producing areas may suitably be represented on dot maps and the population density using choropleth maps. 

2. Selection of Suitable Scale: 

• The scale is used as measure of the data for representation over diagrams and maps. Hence, the selection of suitable scale for the given data sets should be carefully made and must take into consideration entire data that is to be represented. The scale should neither be too large nor too small. 

3. Design:

• We know that the design is an important cartographic task . The following components of the cartographic designs are important. Hence, these should be carefully shown on the final diagram/map. 

1. Title: The title of the diagram/map indicates the name of the area, reference year of the data used and the caption of the diagram. These components are represented using letters and numbers of different font sizes and thickness. Besides, their placing also matters. Normally, title, subtitle and the corresponding year are shown in the centre at the top of the map/diagram. 

2. Legend: A legend or index is an important component of any diagram/map. It explains the colours, shades, symbols and signs used in the map and diagram. It should also be carefully drawn and must correspond to the contents of the map/diagram. Besides, it also needs to be properly positioned. Normally, a legend is shown either at the lower left or lower right side of the map sheet. 

3. Direction: The maps, being a representation of the part of the earth’s surface, need be oriented to the directions. Hence, the direction symbol, i. e. North, should also be drawn and properly placed on the final map. 

Construction of Diagrams 

• The data possess measurable characteristics such as length, width and volume. The diagrams and the maps that are drawn to represent these data related characteristics may be grouped into the following types: 

1. One-dimensional diagrams such as line graph, poly graph, bar diagram, histogram, age, sex, pyramid, etc.;

2.  Two-dimensional diagram such as pie diagram and rectangular diagram;

3. Three-dimensional diagrams such as cube and spherical diagrams. 

• It would not be possible to discuss the methods of construction of these many types of diagrams and maps primarily due to the time constraint. We will, therefore, describe the most commonly drawn diagrams and maps and the way they are constructed. These are : 

1. Line graphs 

• The line graphs are usually drawn to represent the time series data related to the temperature, rainfall, population growth, birth rates and the death rates. Table 3.1 provides the data used for the construction of Fig 3.2.  Construction of a Line Graph:

1. Simplify the data by converting it into round numbers such as the growth rate of population as shown in Table 3.1 for the years 1961 and 1981 may be rounded to 2.0 and 2.2 respectively.

2. Draw X and Y-axis. Mark the time series variables (years/months) on the X axis and the data quantity/value to be plotted (growth of population in per cent or the temperature in 0C) on Y axis.

3. Choose an appropriate scale and label it on Y-axis. If the data involves a negative figure then the selected scale should also show it as shown in Fig. 3.1 

4. Plot the data to depict year/month-wise values according to the selected scale on Y-axis, mark the location of the plotted values by a dot and join these dots by a free hand drawn line. 

1.Bar diagrams:

• The bar diagrams are drawn through columns of equal width. It is also called a columnar diagram. Following rules should be observed while constructing a bar diagram: 

(a) The width of all the bars or columns should be similar. 

(b) All the bars should be placed on equal intervals/distance. 

(c) Bars may be shaded with colours or patterns to make them distinct and attractive. 

The simple, compound or polybar diagram may be constructed to suit the data characteristics. 

Simple Bar Diagram:

• A simple bar diagram is constructed for an immediate comparison. It is advisable to arrange the given data set in an ascending or descending order and plot the data variables accordingly. However, time series data are represented according to the sequencing of the time period. 

Example: Construct a simple bar diagram to represent the rainfall data of Thiruvananthapuram as given in Table 3.3 : 

2. Pie diagram:

• Pie diagram is another graphical method of the representation of data. It is drawn to depict the total value of the given attribute using a circle. Dividing the circle into corresponding degrees of angle then represent the sub– sets of the data. Hence, it is also called as Divided Circle Diagram.

• The angle of each variable is calculated using the following formulae. 

(Value of given State/Region *360)/(Total Value of All States/Regions )  

If data is given in percentage form, the angles are calculated using the given formulae. 

(Percentage of x *360 )/100

For example, a pie diagram may be drawn to show total population of India along with the proportion of the rural and urban population. In this case the circle of an appropriate radius is drawn to represent the total population and its sub-divisions into rural and urban population are shown by corresponding degrees of angle. 

Example: Represent the data as given in Table 3.7 (a) with a suitable diagram.