WHAT IS BIG DATA?

"Big data is a generic term given to datasets that are so large or complicated that it is difficult to store, manipulate and analyse them. The difficulty partly comes from the fact that the dataset is so massive that you would need multiple servers to physically store and provide access to it wihtin a timescale that is useful. Another complication is that standard database software would not be able to cope wih the quanity of data generated so ti would be difficult to structure the data and produce any meaningful analysis of them. A further complexity is the speed at which the data are changing as many big data projects use data that are being updated in real-time.

Data are usually collected for a specific purpose. Data are given structure to turn them into information and information then has to be used for it to become knowledge. If the quantity of dat cannot be structured and analysed, then they cannot produce any useful results. This is the challenge of working with big data - if it can't be turned into useful information, there is little point collecting it in the first place.

The three main features of big data are:

• volume: the sheer amount of data is on a very large scale

• variety: the type of data being collected is wide-ranging, varied and may be difficult to classify

• velocity: the data changes quickly and may include constantly changing data sources."

Data - are structured to produce - information - is used to produce - knowledge

https://www.zdnet.com/article/volume-velocity-and-variety-understanding-the-three-vs-of-big-data/

"Cloud" and "big data"? Systems engineers used to draw network diagrams of local area networks. Between the diagrams of LANs, we'd draw a cloud-like jumble meant to refer to, pretty much, "the undefined stuff in between." Of course, the Internet became the ultimate undefined stuff in between, and the cloud became The Cloud.

To Uncle Steve, Aunt Becky, and Janice in Accounting, "The Cloud" means the place where you store your photos and other stuff. Many people don't really know that "cloud" is a shorthand, and the reality of the cloud is the growth of almost unimaginably huge data centers holding vast quantities of information.

Big data is another one of those shorthand words, but this is one that Janice in Accounting, Jack in Marketing, and Bob on the board really do need to understand. Not only can big data answer big questions and open new doors to opportunity, your competitors are almost undoubtedly using big data for their own competitive advantage.

So what is big data? The answer, like most in tech, depends on your perspective. Here's a good way to think of it. Big data is data that's too big for traditional data management to handle.

Big, of course, is also subjective. That's why we'll describe it according to three vectors: volume, velocity, and variety -- the three V's

Big data is used for different purposes. In some cases, it is used to record factual data such as banking transactions. However, it is increasingly being used to analyse trends and try to make predictions based on relationships and correlations within the data eg. scientists use data to predict the impact of climate change; business analysts use big data to predict sales; healthcare specialists use it to predict the spread of diseases; meteorologists use it to predict major weather events.

Examples of big data

• Scientific research - scientists generate large volumes of data that could be measure din terms of petabytes or exabytes eg readings from weather sensors, data collected from telescopic observations, biological results of experiments or global statistics on health issues. In all these cases the data are being collected and analysed for scientific purposes, usually to improve the quality of people's lives eg human genome project uses masses of data to try to find the causes of genetic illnesses, with a view to eradicate them.

• Retail - All large businesses make use of data. Online retailers in particular can have millions of customers generating billions of sales. These data are used to improve the performance of the business. Sales data can be collected and analysed to help spot trends in consumer behaviour enabling businesses to become more profitable.

• Banking - The banking sector has to handle billions of transactions on an annual basis. They need to keep these data secure and have an audit trail of every single transactions to prevent against loss and fraud.

• Government - most government departments and agencies have massive datasets. eg the NHS records every single patient appointment and operation . These data are critical to the successful treatment of patients and in many cases are a matter of life and death.

• Mobile networks - There were an estimated 4.6 billion mobile phone contracts around the world. All of the customer and call data are recorded to enable bills to be generated.

• Security - Legislation allows for mobile phone calls, texts, email messages and other online communications to be recorded. this represents billions of items of data every day and can bge used by the security services to spot terrorist threats

• Real-time applications - Many applications, particularly online and mobile, make use of real-time data eg weather apps take data readings from sensors, city traders use software that enables trading based on second-by-second share price fluctuations.

• Internet - A log of big data gets created through everyday use of the Internet. Eg. data from social media websites could be analysed to understand social attitudes and trends. Data from search queries can be used to understand how people use the web.