Embedded Files
Big data refers to extremely large, complex, or rapidly generated datasets that require specialised software and hardware to collect, store, process, and analyse.
Key Concepts
Key Concepts
Volume: The large amount of data being generated and stored.
Velocity: The speed at which data is created, collected, and processed.
Variety: The different forms of data, such as text, images, video, location data, sensor readings, and transaction records.
Machine learning and data analytics: Systems identify patterns, trends, relationships, and predictions from large datasets.
Algorithms/Mechanisms:
Algorithms/Mechanisms:
Distributed computing: Processing data across multiple computers or servers.
Data mining: Searching large datasets to identify useful patterns or relationships.
Clustering and classification: Grouping similar data or placing data into defined categories.
Predictive analytics: Using historical data to predict future events or behaviour.
Databases and cloud storage: Storing and managing large amounts of structured and unstructured data.
Examples of Use:
Examples of Use:
Online retailers recommending products based on customer behaviour.
Healthcare organisations identifying disease patterns and predicting health risks.
Banks detecting unusual transactions and possible fraud.
Social media platforms analysing user interests and engagement.
Transport systems analysing traffic and travel patterns.
Key Issues:
Key Issues:
Accuracy and bias—incorrect, incomplete, or unrepresentative data may produce misleading results.
Privacy—large datasets may contain personal, sensitive, or location information.
Security—stored data may be stolen, leaked, altered, or misused.
Dataset quality—poor-quality data can lead to unreliable conclusions and decisions.
Ownership and consent—people may not know how their data is collected, shared, or analysed.
Human Impact & Perspectives:
Human Impact & Perspectives:
Big data can improve decision-making, increase efficiency, identify important trends, and support services such as healthcare, transport, and disaster response.
Concerns include loss of privacy, unfair profiling, automated discrimination, surveillance, and over-reliance on data-driven decisions.
Different views: Businesses and governments may value efficiency, prediction, and improved services, while privacy advocates and affected communities may focus on consent, fairness, security, and how personal data is used.
Teacher Help
Teacher Help
2026 Assessment specifications are advising: For Big data, questions may cover: characteristics of big data (volume, variety, velocity, etc.), generation, processing and analysing data in different formats, interpretation and representation (bias and display), tools and technologies used in big data, and big data considerations (privacy, ethics, and data governance).
Understanding Big Data
Understanding Big Data
First you need to understand what Computer Vision entails.
The GOAT: The CSField Guide: Big Data Read through the material. If you learn this stuff inside out and you'll smash it.
Understanding Volume, Velocity, Variety and Veracity (The fundamentals).
Understanding Volume, Velocity, Variety and Veracity (The fundamentals).
The Problems with Big Data
The Problems with Big Data
Integrating Different Data Sources
Integrating Different Data Sources
Records from different systems must be matched correctly. One system might identify a person using a customer number, while another uses an email address or device identifier.
Incorrectly linking records can produce false patterns. Failing to link relevant records can hide genuine patterns.
Scalability and Reliability
Scalability and Reliability
A system that works with 10,000 records may not work with 10 billion records.
Big-data platforms must be able to:
Add additional storage and processing capacity.
Distribute work across multiple machines.
Continue operating when a machine fails.
Prevent duplicate or lost processing.
Produce results within a useful time.
Privacy
Privacy
Big-data systems may collect location, health, financial, educational, biometric, behavioural, or communication data.
Even when names are removed, combining several datasets may make it possible to identify an individual indirectly.
Bias and Data Invisibility
Bias and Data Invisibility
Some groups generate more digital data than others. People with smartphones, reliable internet access, bank accounts, transport cards, or active social-media profiles may be more visible to a data system.
People who produce less digital data may appear to have less need, even when their actual need is greater.
Security
Security
Large centralised datasets are valuable targets. A breach may expose information about thousands or millions of people.
Security measures may include:
Encryption
Authentication
Role-based access
Audit logs
Network security
Backups
Monitoring
Data minimisation
Ownership, Consent, and Purpose
Ownership, Consent, and Purpose
People may not understand that their information is being collected, combined, or reused.
A dataset originally collected for transport planning might later be used for policing, commercial advertising, or monitoring individuals. This is sometimes called purpose creep.
Correlation and Causation
Correlation and Causation
Big-data analysis may reveal that two things occur together. This does not necessarily prove that one causes the other.
For example, a suburb with more social-media posts about crime may not have more crime. It may simply have more active social-media users.
Representation
Representation
Dashboards, heat maps, rankings, and summary scores can make complex information easier to understand. However, they can also hide uncertainty, missing data, variation within groups, or the assumptions used to calculate a score.
A dashboard can appear objective and authoritative even when the data underneath it is incomplete.
Why Big Data Is a Computer Science Problem & Real-World Applications and Impact
Why Big Data Is a Computer Science Problem & Real-World Applications and Impact
Its a Computer Science problem because...
Its a Computer Science problem because...
Big data requires knowledge from several areas of computer science:
Databases: Storing, querying, updating, and linking data.
Distributed systems: Coordinating storage and processing across multiple computers.
Algorithms and data structures: Searching, sorting, grouping, aggregating, and analysing information efficiently.
Networking: Moving data between sensors, devices, servers, and cloud platforms.
Machine learning: Identifying patterns and making predictions from large datasets.
Cybersecurity: Protecting data against unauthorised access, alteration, or loss.
Human-computer interaction: Designing dashboards and visualisations that people can interpret accurately.
Software engineering: Building reliable data pipelines that can be tested, monitored, maintained, and scaled.
It is a real world thing because ...
It is a real world thing because ...
Healthcare
Healthcare
Large datasets can be used to examine disease patterns, hospital demand, treatment outcomes, and population health.
Potential benefits include earlier intervention and better allocation of resources. Risks include privacy breaches, incorrect predictions, and groups being underrepresented in health data.
Transport and Smart Cities
Transport and Smart Cities
Traffic sensors, bus-card scans, GPS data, weather information, cameras, and event schedules can be combined to manage transport systems.
Environmental Monitoring
Environmental Monitoring
Sensors can continuously record river levels, rainfall, air quality, soil moisture, water quality, temperature, and pollution.
Retail and Online Services
Retail and Online Services
Businesses analyse transactions, searches, browsing behaviour, and purchases to forecast demand, detect fraud, and recommend products.
Science
Science
Astronomy, climate modelling, genetics, and particle physics produce datasets that may be too large for one computer to process effectively.
Government and Public Policy
Government and Public Policy
Governments may combine administrative datasets to study employment, education, health, housing, and social outcomes.
Big Data Pipelines
Big Data Pipelines
A data pipeline is a connected sequence of steps that moves data from where it is created to where it can be used.
A data pipeline is a connected sequence of steps that moves data from where it is created to where it can be used.
Data generation: Information is created by sensors, cameras, phones, websites, transactions, and public records.
Data ingestion: Data enters the system either in batches or continuously in real time.
Cleaning and validation: Errors, duplicates, missing values, and inconsistent formats are checked and corrected.
Storage: Data is saved in databases, data warehouses, data lakes, or cloud systems.
Processing: Large tasks are divided across several computers to improve speed and capacity.
Analysis: The data is examined for patterns, trends, groups, unusual results, and predictions.
Representation: Findings are displayed using graphs, maps, tables, alerts, and dashboards.
TASK 1: Big Data in Action- Student Activity: Data Sort
TASK 1: Big Data in Action- Student Activity: Data Sort
Classify each source as structured, semi-structured, or unstructured. Then decide whether it is likely to arrive as a batch or stream.
Data source Likely format Batch or stream
Ambulance call-out records
Hospital waiting-time feed
Weather sensor messages
Traffic camera video
Public event schedule
Social-media posts
Which source is likely to have the greatest volume?
Which requires the fastest processing?
Which is most difficult to interpret accurately?
Which groups may be underrepresented?
What cleaning or validation would be needed?
Data source Likely format Batch or stream
Ambulance call-out records Structured Both
Hospital waiting-time feed Structured or semi-structured Stream
Weather sensor messages Semi-structured Stream
Traffic camera video Unstructured Stream
Public event schedule Structured Batch
Social-media posts Semi-structured or unstructured Stream
Which source is likely to have the greatest volume?
Which source is likely to have the greatest volume?
Traffic camera video, because continuous video produces very large files.
Which requires the fastest processing?
Which requires the fastest processing?
Ambulance call-out records and hospital waiting-time data, because delays could affect emergency care.
Which is most difficult to interpret accurately?
Which is most difficult to interpret accurately?
Social-media posts, because they may contain slang, sarcasm, false information, and unclear context.
Which groups may be underrepresented?
Which groups may be underrepresented?
People without smartphones, internet access, social-media accounts, transport access, or confidence using digital services.
What cleaning or validation would be needed?
What cleaning or validation would be needed?
The system may need to:
Remove duplicate records.
Check timestamps and locations.
Correct inconsistent formats.
Identify missing values.
Filter faulty sensor readings.
Confirm social-media information using reliable sources.
Check that camera or sensor data is current.
How does Big Data get Processed?
How does Big Data get Processed?
A single spreadsheet, database, or computer may be able to handle a large amount of data, but it is often not enough to manage the volume, velocity, variety, and veracity of Big Data.
Big-data processing is therefore usually distributed across multiple computers. Distributed processing divides a large task into smaller parts, which are processed at the same time by different computers. The partial results are then combined into a final output.
For example, to count millions of transport-card scans:
The dataset is divided into smaller sections.
Each computer counts the scans in its assigned section.
The separate totals are combined.
The final result is stored or displayed.
This allows the system to process much larger datasets more quickly and reliably than a single computer could.
Examples of Big Data Processing Tools
Examples of Big Data Processing Tools
Apache Hadoop
Apache Hadoop
Apache Hadoop is a collection of tools for storing and processing large datasets across a computer cluster.
HDFS
HDFS
The Hadoop Distributed File System, or HDFS, divides files into blocks and stores them on different computers called DataNodes. A NameNode keeps track of where those blocks are stored. HDFS is designed for high-throughput access to large datasets and can replicate blocks to protect against machine failure.
Apache Spark is a tool for processing very large datasets across multiple computers. It divides data into smaller parts that can be analysed at the same time. Spark supports batch processing, live data streams, SQL, and machine learning, and it can keep frequently used data in memory to speed up repeated calculations.
Apache Spark
Apache Spark
Apache Spark is a distributed processing engine used for batch processing, data analysis, SQL queries, machine learning, and streaming data. It divides data into partitions and sends the partitions to different computers for processing.
Apache Spark can keep frequently reused data in memory instead of repeatedly reading it from storage. This often makes it more suitable than traditional MapReduce for repeated calculations, interactive analysis, and machine-learning tasks.
Spark’s Structured Streaming can process continuously arriving data, such as transport scans, financial transactions, or sensor readings. It treats the incoming data as new rows being added to a continuously updated table.
Hadoop and Spark Compared
Hadoop and Spark Compared
Hadoop MapReduce Apache Spark
Often used for large batch jobs Used for batch and streaming data
Frequently writes intermediate results to storage Can keep reused data in memory
Uses map, shuffle, and reduce stages Supports more flexible processing operations
Suitable for long-running processing jobs Suitable for rapid analysis and machine learning
Often works with HDFS and YARN Can use HDFS, cloud storage, and other systems
Hadoop and Spark can also work together. Data may be stored in Hadoop’s HDFS, managed using YARN, and processed using Spark.
TASK 2: Big Data in Action- SmartView
TASK 2: Big Data in Action- SmartView
Big Data in Action
Big Data in Action
SmartView brings together live or frequently updated information about transport, cycle counters, parking, weather, air quality, traffic cameras, incidents, and other city services
Open the Big Data portal by clicking on the image. Explore. Then answer the questions below.
Student Questions (Click on the chevron to see the answers)
Student Questions (Click on the chevron to see the answers)
Student Questions
Student Questions
Identify five different data sources.
Explain how volume, velocity, and variety apply.
Identify structured, semi-structured, and unstructured information.
Which information appears to be live, and which may be historical?
What decisions could a city planner make from this information?
What important information is not shown?
Which communities might be less visible in the data?
How could the interface create a misleading impression of certainty?
Smartview Student Questions — Model Answers
Smartview Student Questions — Model Answers
1. Identify five different data sources.
1. Identify five different data sources.
Smartview collects data from:
Traffic sensors and cameras
Public-transport card scans
Weather stations
Mobile-phone location data
Social-media posts relating to city events
These sources provide information about traffic, public-transport use, environmental conditions, people’s movements, and public activity around events.
2. Explain how volume, velocity, and variety apply.
2. Explain how volume, velocity, and variety apply.
Volume: Smartview collects huge amounts of data from lots of people and devices. These systems could generate lots of records and large quantities of video and location data.
Velocity: Traffic readings, transport-card scans, location updates, weather readings, and social-media posts arrive continuously. Some of this information must be analysed in real time or near real time so that planners can respond to congestion, emergencies, or changing transport demand.
Variety: Smartview combines numerical sensor readings, database records, location coordinates, JSON messages, text posts, images, and video. These formats must be cleaned, standardised, and linked before they can be analysed together.
3. Identify structured, semi-structured, and unstructured information.
3. Identify structured, semi-structured, and unstructured information.
Structured information includes:
Public-transport card scans
Traffic-sensor counts
Weather readings
GPS coordinates
These records usually contain defined fields such as date, time, location, route number, temperature, or passenger count.
Semi-structured information includes:
JSON messages from sensors
Mobile-phone location logs
Social-media posts containing usernames, timestamps, hashtags, and location tags
This data has some identifiable fields or metadata but may not fit into a fixed relational database table.
Unstructured information includes:
Traffic-camera images and video
The written content of social-media posts
Photographs, audio, or video uploaded to social media
This information does not have a simple row-and-column structure and may require computer vision, natural-language processing, or other specialised analysis.
4. Which information appears to be live, and which may be historical?
4. Which information appears to be live, and which may be historical?
Traffic sensors, cameras, weather stations, public-transport scans, mobile-phone locations, and social-media posts could therefore all provide live or near-live information.
However, once this data is stored, it becomes historical data. Smartview could compare current conditions with information collected over previous hours, days, months, or years.
Current traffic-sensor readings could show congestion now.
Historical traffic data could show that congestion regularly occurs at 8:00 am.
Live transport-card scans could show current passenger numbers.
Historical scans could help identify long-term changes in demand.
5. What decisions could a city planner make from this information?
5. What decisions could a city planner make from this information?
A planner could adjust public transport, traffic lights, road routes, emergency responses, and future infrastructure.
Add buses to routes experiencing high demand.
Adjust traffic-light timing.
Decide where new bus stops, cycleways, or transport hubs are needed.
Plan transport for concerts, sports events, or festivals.
Plan road maintenance or infrastructure upgrades.
The information could support better decisions, but it should not make decisions automatically without checking data quality and considering local knowledge.
6. What important information is not shown?
6. What important information is not shown?
The system may not show why people are travelling, whether services are affordable, people’s accessibility needs, or the experiences of residents.
Why people are travelling.
Whether a person is a resident, worker, tourist, or someone passing through.
People who do not use smartphones, transport cards, or social media.
Whether people have disabilities or accessibility requirements.
Whether existing services are affordable.
The system shows what can be measured digitally. It does not necessarily show the complete needs of the community.
7. Which communities might be less visible in the data?
7. Which communities might be less visible in the data?
People without smartphones, internet access, transport cards, or social-media accounts may produce less data and be underrepresented.
People without smartphones.
People with limited internet access.
Older people who use fewer digital services.
Young children without personal devices.
People who use cash rather than transport cards.
Low digital visibility should not be interpreted as low demand.
8. How could the interface create a misleading impression of certainty?
8. How could the interface create a misleading impression of certainty?
Heat maps and exact scores may appear completely accurate even when data is missing, delayed, biased, or based on assumptions.
The interface may also hide:
Which sources contributed to the result.
How each source was weighted.
How missing data was handled.
When each source was last updated.
Differences within a large suburb or map area.
Groups that are not represented in the data.
For example, an area with many social-media posts and smartphones may appear to have high transport demand. Another area with fewer connected residents may appear to have low demand, even though its residents have a greater need for additional buses.
TASK 3: Big Data in Action Gapminder
TASK 3: Big Data in Action Gapminder
Gapminder Tools
Gapminder Tools
Gapminder is an interactive data-visualisation tool that allows students to compare countries using indicators such as population, income, health, education, and life expectancy. In the bubble-chart view, each bubble represents a country, while its position, size, and colour show different data values. Students can change the indicators and animate the graph over time to identify patterns, trends, relationships, and differences between countries. It is a useful example of how large datasets can be combined and presented visually, while also encouraging students to question data sources, missing information, estimates, and whether a graph shows correlation or causation.
Instructions:
Instructions:
Open the Gapminder bubble-chart tool.
This is set up for you so:
The x-axis shows income per person.
The y-axis shows life expectancy.
Bubble size represents population.
Bubble colour represents geographic region.
The timeline begins around 1950 and runs to the latest available year.
Select New Zealand and at least two contrasting countries. Play the animation and observe how their bubbles change over time.
Note: Gapminder combines information from multiple sources into consistent time-series data. Some missing values are estimated or interpolated, so the visualisation should be treated as a useful representation rather than a perfectly complete record.
Student Questions (Click on the chevron to see the answers)
Student Questions (Click on the chevron to see the answers)
Data generation:
How would the original population, income, and life-expectancy data have been generated or collected? Identify a likely source for each type of data.Data formats:
What different types of information are needed to create the visualisation? Consider numerical measurements, country names, years, regions, and source information.Data processing:
Describe two processing steps Gapminder may need to perform before information from different organisations can be combined into one dataset.Missing data:
Why might some countries or years have missing information? Explain one benefit and one risk of estimating values to fill these gaps.Data representation:
Explain what the bubble’s position, size, colour, and movement represent. Identify one strength and one limitation of presenting the data in this way.Interpretation and certainty:
Explain two ways the chart could create a misleading impression. Consider country averages, estimated values, bubble size, axis scales, missing data, or differences within countries.
Gapminder — Model Answers
Gapminder — Model Answers
Data generation
Data generation
Population data comes from censuses and birth and death records. Income data comes from national economic records. Life expectancy is calculated using health and mortality data.
Data formats
Data formats
The visualisation uses numerical measurements, country and region names, years, population figures, and information about data sources. These must use consistent labels and identifiers.
Data processing
Data processing
Standardise the data, such as using the same country names, currencies, units, and date formats.
Clean and combine the data by removing errors, matching records, and resolving differences between sources.
Missing data
Missing data
Data may be missing because of poor recordkeeping, conflict, changing borders, or limited resources. Estimating values creates a more complete timeline, but the estimates may be inaccurate.
Data representation
Data representation
The bubble’s horizontal and vertical positions show the selected indicators. Its size represents population, its colour represents region, and its movement shows change over time.
A strength is that several variables can be compared at once. A limitation is that bubbles may overlap or be difficult to compare accurately.
Interpretation and certainty
Interpretation and certainty
Country averages can hide differences between regions and groups within a country. Estimated values, smooth animations, bubble sizes, and axis scales may also make the data appear more precise and certain than it really is.
TASK 4: Google Trends
TASK 4: Google Trends
Big Data in Action
Big Data in Action
Students can compare search interest for terms across time and geographic regions.
Google Trends is particularly useful for teaching velocity, sampling, aggregation, representation, and interpretation. Its figures are normalised to a 0–100 scale rather than displaying raw search totals. Google also states that Trends uses a sample of searches and should not be treated as a scientific poll or a perfect representation of public opinion.
Student Questions (Click on the chevron to see the answers)
Student Questions (Click on the chevron to see the answers)
Compare searches in New Zealand for:
"FIFA"
“flood”
"flu"
Try some of your own...
What event might explain each spike?
Does a spike show approval, concern, curiosity, or something else?
What population groups might Google Trends underrepresent?
Why would using this data alone to allocate emergency resources be unreliable?
Google Trends — Model Answers
Google Trends — Model Answers
Google Trends — Model Answers
Google Trends — Model Answers
What event might explain each spike?
What event might explain each spike?
FIFA: Major World Cup matches, team announcements, results, or controversial incidents. The current 2026 FIFA World Cup would explain a strong recent spike.
Flood: Severe weather events such as the January 2023 Auckland floods or Cyclone Gabrielle in February 2023.
Flu: The beginning of New Zealand’s winter flu season, increases in infections, health warnings, or vaccination campaigns. Flu is most common from May to October.
What does a spike show?
What does a spike show?
A spike shows increased search interest, not necessarily approval. People may be curious, concerned, seeking help, following news, or trying to understand an event.
Which groups might be underrepresented?
Which groups might be underrepresented?
People without reliable internet or digital devices
People who use other search engines
People with limited digital literacy
People searching in different languages or using different terms
People directly affected by an emergency who have lost power or internet access
Why is it unreliable for allocating emergency resources?
Why is it unreliable for allocating emergency resources?
Search activity measures public attention rather than the severity of an emergency or the number of people needing assistance. Media coverage can increase searches, while badly affected communities may produce little search data because they have lost power, internet access, or devices. Emergency decisions should also use official reports, sensor data, emergency calls, community information, and direct assessments.
Task 5: Putting it all together!
1. Google Trends — Processing Search Data
Task 5: Putting it all together!
1. Google Trends — Processing Search Data
Google Trends processes very large numbers of searches, then anonymises, aggregates, samples, and normalises the results before displaying them as values from 0 to 100.
Explain why this data would need to be processed across many computers. Describe how Apache Hadoop MapReduce could divide historical search records into smaller sections and combine the results. Then explain why Apache Spark might be more suitable for updating current search trends quickly.
2. Gapminder — Combining Global Datasets
2. Gapminder — Combining Global Datasets
Gapminder combines population, income, health, and other information from multiple countries and organisations into interactive visualisations.
Describe how a distributed system could store and process this data. Explain how Hadoop HDFS could store the large dataset and how Spark could clean, standardise, join, and analyse records from different sources. Identify one data-quality problem that could affect the final bubble chart.
3. SmartView — Processing Live City Data
3. SmartView — Processing Live City Data
Christchurch SmartView combines live or regularly updated information, including weather, transport, environmental conditions, cycle counters, incidents, and public services.
Explain how distributed processing could divide incoming city data between several computers. Compare using Spark streaming for live sensor updates with using Hadoop MapReduce for analysing several years of historical data. Recommend which tool should be used for updating the live dashboard and justify your choice.
Big-Data Processing — Model Answers
Big-Data Processing — Model Answers
1. Google Trends
1. Google Trends
Google Trends must process billions of search records, so one computer would not be powerful enough. Distributed processing divides the search data across many computers, allowing separate parts to be processed at the same time.
Hadoop MapReduce could divide historical search records by date, region, or search term. Each computer would count searches in its section, and the results would then be combined and normalised into the 0–100 values shown by Google Trends.
Apache Spark would be more suitable for current trends because it can process continuously arriving data and keep frequently used information in memory. This would allow the trends display to update more quickly.
2. Gapminder
2. Gapminder
Gapminder combines large datasets about population, income, health, countries, and years. A distributed system could divide this information across several computers for storage and processing.
Hadoop HDFS could split the datasets into blocks and store copies across multiple computers. Apache Spark could then clean country names, standardise units and dates, join records from different organisations, and calculate the values needed for the bubble chart.
One possible data-quality problem is missing or estimated information. Some countries may not have reliable records for every year, which could make the visualisation appear more accurate than it really is.
3. SmartView
3. SmartView
SmartView receives information from transport systems, weather stations, traffic sensors, cycle counters, and other city services. Distributed processing could send different data sources or time periods to separate computers for simultaneous processing.
Spark streaming would be suitable for live data because it can process new sensor readings as they arrive and quickly update the dashboard.
Hadoop MapReduce would be more suitable for analysing several years of stored data, such as identifying long-term traffic or cycling patterns.
Spark should be used for the live dashboard because it can handle continuously arriving information with less delay. Hadoop could still be used for slower, large-scale historical analysis.
Effects on People: The positives and negatives - Data Analysis, Bias, Dashboards, and Misinterpretation
Effects on People: The positives and negatives - Data Analysis, Bias, Dashboards, and Misinterpretation
Big-data systems often summarise millions of records into a simple display.
This is useful because a decision-maker cannot inspect every individual record. However, every summary involves choices about:
Which data is included or excluded
How categories are defined
How missing values are handled
How different sources are weighted
Which time period is displayed
Which colours and scales are used
Whether uncertainty is shown
Data Availability Bias
Data Availability Bias
A suburb with more smartphones, transport cards, and social-media users produces more records. It may therefore appear to have greater demand simply because more information is available.
Aggregation
Aggregation
A whole suburb may be given one score. This can hide differences between:
Streets
Times of day
Age groups
Transport users
Communities within the suburb
Colour and Scale
Colour and Scale
Bright red may suggest danger, urgency, or extreme demand even when the numerical difference between suburbs is small.
Changing the scale or category boundaries can make the same data appear more or less significant.
Missing Context
Missing Context
A large number of transport scans could mean:
The suburb has a larger population.
A major event is taking place.
Many people travel through the suburb.
Public transport is already operating effectively.
Outdated Data
Outdated Data
A dashboard may update every five minutes but still contain data sources that are several hours old.
The interface should show when each source was last updated
False Precision
False Precision
A score such as 78.4 appears highly accurate. However, it may be based on:
Estimates
Samples
Missing records
Inaccurate locations
Subjective weightings
Displaying decimal places does not make the underlying data more reliable.
Misleading Visual Representation
Misleading Visual Representation
Graphs can mislead when area, colour, scale, or category size is not proportional to the underlying values. The misleading-graphs teaching resource provides examples of how visual choices can alter a viewer’s interpretation.
Can you identify How Big Data can be misinterpreted and why?
Can you identify How Big Data can be misinterpreted and why?
Look at the pictures below and answer these questions :
Look at the pictures below and answer these questions :
Student Questions
Student Questions
What data has been included and excluded?
Does the map show raw totals or values adjusted for population?
How are the three data sources weighted?
When was each source last updated?
How are missing records handled?
Does the dashboard show uncertainty?
Which communities might be hidden?
What other evidence should a planner examine?
Answers
Answers
Model Answers – Dashboard 1: LearnWell
Included: LMS activity, video watch data, assessment attempts.
Excluded: offline study, phone calls, in-person help, outcomes.Not adjusted for population. Larger regions may naturally have more students.
LMS activity 50%, Video platform 30%, Assessment system 20%.
LMS: 9:25 AM, Video platform: 8:55 AM, Assessment: 6:15 AM.
Missing records are ignored in the calculation.
No. Uncertainty is not displayed.
Students without internet access, those not logging in, ESOL students, part-time learners.
Population size, student demographics, internet access data, surveys, achievement results, attendance.
Answers
Answers
Model Answers – Dashboard 3: ShopGauge
Included: in-store sales, loyalty activity, online orders.
Excluded: competitor data, tourism/events, returns, out-of-stock items.Not adjusted for population. Densely populated catchments may look higher.
POS transactions 60%, Loyalty programme 25%, Online orders 15%.
POS: 10:00 AM, Loyalty: 9:45 AM, Online orders: 9:50 AM.
Missing records are ignored in the calculation.
No. Uncertainty is not shown.
People who do not shop at these stores, cash-only customers, tourists, and people without loyalty accounts or online access.
Census data, competitor activity, local events, economic conditions, and customer surveys.
Ethics, Privacy, and Data Governance
Ethics, Privacy, and Data Governance
Privacy
Privacy
Students should consider:
Was the information collected for a lawful purpose?
Did people know it was being collected?
Is all the collected information necessary?
Could the same goal be achieved using less personal data?
Who can access the information?
How securely is it stored?
How long will it be retained?
Can individuals access or correct their information?
Could it later be used for a different purpose?
New Zealand’s Privacy Act 2020 contains principles governing how organisations collect, store, use, share, retain, and correct personal information.
Bias and Fairness
Bias and Fairness
Bias may enter through:
Who is included in the dataset
Who is missing
Where sensors are placed
Unequal access to devices
Incorrect labels
Historical inequalities
Data-cleaning rules
Weightings and algorithms
Visualisation choices
Human interpretation
A technically accurate system can still produce an unfair result when the data does not represent the whole population.
Transparency
Transparency
Users should be told:
What data was used
Where it came from
When it was collected
What information was missing
How records were cleaned
How the result was calculated
How confident the system is
Whether a person can challenge the result
Accountability
Accountability
A responsible system should include:
A named person or organisation responsible for it
Audit trails showing how decisions were made
Human oversight
A complaints process
Regular accuracy and bias testing
Procedures for correcting errors
Clear limits on automated decisions
Māori Data Sovereignty
Māori Data Sovereignty
In Aotearoa, data governance should also consider who has authority over information about Māori people, iwi, communities, language, culture, and environments.
Te Mana Raraunga defines Māori data sovereignty as recognising that Māori data should be subject to Māori governance and used to support Māori and iwi aspirations. It treats Māori data as a living taonga with cultural and strategic value.
This means responsible governance involves more than individual privacy or legal compliance. It may require:
Māori participation in decision-making
Collective as well as individual consent
Clear benefits for Māori communities
Protection against harmful or misleading uses
Respect for tikanga and relationships with data
Māori authority over access, interpretation, and reuse
Final Examination Preparation Task
Final Examination Preparation Task
Use the Your own example scenario to write four connected responses.
A. Technical Foundations
A. Technical Foundations
Explain:
Volume
Velocity
Variety
Structured data
Semi-structured data
Unstructured data
Connect each concept to specific data sources in the scenario.
B. Storage and Processing
B. Storage and Processing
Explain:
Why one conventional database or server may struggle
How distributed storage works
How distributed processing works
The difference between batch and stream processing
One relevant technology such as Hadoop, MapReduce, Spark, a data lake, or cloud processing
A strong response should explain how the technology works rather than only naming it.
C. Reliability and Human Impact
C. Reliability and Human Impact
Analyse:
How data is collected
Cleaning and validation
Missing or duplicate records
Data availability bias
Aggregation
Dashboard design
How technical choices affect communities and decision-makers
Students should connect the technical issue to a human consequence.
For example:
Because smartphone-location data is used, people without smartphones may be less visible. This could cause planners to underestimate transport demand in lower-connectivity communities.
D. Perspectives and Critical Conclusion
D. Perspectives and Critical Conclusion
Compare two perspectives, such as:
Emergency planners and residents
Researchers and data subjects
City councils and privacy advocates
Technology companies and affected communities
Government agencies and iwi
Recommend improvements such as:
Including additional data sources
Adjusting values for population
Showing uncertainty ranges
Displaying source-update times
Auditing results for bias
Consulting affected communities
Providing human review
Establishing clear data-governance rules
Finish with a justified conclusion about when the system should and should not be used.
Strong Concluding Idea
Strong Concluding Idea
Big data is not objective simply because it is large. It can reveal patterns that people could not identify manually, but it can also make existing inequalities harder to see when data availability is mistaken for actual need.
Big-data systems are most valuable when their technical limitations are visible, their data is carefully governed, their results can be challenged, and human judgement can question or override their conclusions.
Page updated
Google Sites
Report abuse