Data sets can be viewed in a range of different files and structures. The most common are:

1. CSV - commas and new lines

2. JSON - Bracket notations like objects

3. Text - new lines and spaces

Data structures are placed in a specific structure using commas, new lines, arrays etc. to separate and organise the data. Using different programming languages and libraries we can load this data and structure in a way to conduct some processing to print, make graphs or search data in our applications. Most modern day applications connect to external sources to access, edit and remove data.

API (Application Programming Interface)

An API provides control to access and retrieve data from a data source online. Using custom commands and queries data is returned as JSON.  Generally you need to have access to their source using a unique token which gives you limited calls to their service. Some services are open access. Some simple API service examples include:

• Weather API - https://openweathermap.org/api

• Sport scores API - https://api.squiggle.com.au/

Data sets that we can use to get started are local files (CSV or Json)

• National Public Toilet Database - https://data.gov.au/dataset/ds-dga-553b3049-2b8b-46a2-95e6-640d7986a8c1/details

• CFS Current Incidents - https://data.sa.gov.au/data/dataset/south-australian-country-fire-service-current-incidents-rss-feed

• SA Parks Database - https://data.sa.gov.au/data/dataset/parks-listings/resource/4aec8622-c685-4484-bb83-18cda3f30959

• Open Library - https://openlibrary.org/subjects/love.json

Quick Code Reference

Reading CSV files - Sample Function

This example code uses a async function to read the CSV file and store the data as text. It then splits the text blob file by newline characters into an array called rows. When then loop over each row, splitting the string by commas into an array called row. Finally we access each field in turn depending on the data source.

async function getData() {

    const response = await fetch("data.csv");

    const data = await response.text();

    console.log(data);

    const rows = data.split('\n').slice(1);

    rows.forEach(element => {

        const row = elementt.split(',');

const data1 = row[0];

const data2 = row[2];

    }

}

Reading JSON from an online source

Using a jQuery built in function we can load a JSON file into a local variable. We just need to pass the url into source. 

• JSON.stringify enables us to to turn the JSON structure to plain text.

• JSON.parse will plain text into JSON form.

const sourceURL ='test.com.au';

var data;

$.getJSON(sourceURL, function (result) {

console.log(result);

data = result;

});

What do we do with the data now its stored in a global variable?

Once you have received and organised the data into a structure that you can iterate (loop) over, you can sort and organise the data. 

• Sort the data

• Search and filter data

Sorting Data from a JSON Format

Source: https://www.javascripttutorial.net/array/javascript-sort-an-array-of-objects/#:~:text=To%20sort%20an%20array%20of%20objects%2C%20you%20use%20the%20sort,determines%20the%20order%20of%20objects.

If data is stored as JSON we can can use an array sort function to sort by specific properties in ascending or descending order. Note that these sort functions use binary. So when sorting by alphabetical order we need to make sure that all text is in lowercase. See the example below.

This function compares two values at a time and moves them up or down in a temporary array. We still need to understand our dataset to run these sort functions.

Sorting by a text field

function sortList() {

    dataset.sort((a, b) => {

        let fa = a.lastname.toLowerCase(),

        let fb = b.lastname.toLowerCase();

        if (fa < fb) {

            return -1;

        }

        if (fa > fb) {

            return 1;

        }

        return 0;

    });

}

If we would like to sort other types of data like dates, times, floats etc. You will need to do some more reading to convert the data into a readable format for searching and comparison.

Filtering (Searching) Data

Source: https://stackabuse.com/guide-to-javascripts-filter-method/

JavaScript has a built in function called filter that can return objects of data matching your specific criteria.

Here is a simple example using a basic array or objects

const students = [

    { firstName: "John", lastName: "Doe", graduationYear : 2022 },

    { firstName: "Stephen", lastName: "Matt", graduationYear : 2023 },

    { firstName: "Abigail", lastName: "Susu", graduationYear : 2022 }

];

const currentYear = new Date().getFullYear();

let graduatingStudents = students.filter((element) => {

    if (element.graduationYear === currentYear) {

        return element;

    }

});

console.log(graduatingStudents);