Embedded Files
GENERAL KNOWLEDGE /
1. DATASETS
Data stored in persistent storage format is called dataset.
Text files
stores characters
if tabular data then comma-separated values - CSV (but it can be tabulator-separated too)
a single file, size is limited, takes higher amount of time to open larger files
the program is necessary to process and manage the contents
Spreadsheets
the referenced application is a tool for storing, organizing, analysing
CSV, XLS, TSV
the application is for humans, not for programs
Databases
data can be accessed, updated, queried, deleted
query language is necessary (like Structured Query Language - SQL)
typically stores many multiple tables and different types of data
2. ANALYSIS
Statistics: mathematical analysis of mass of data; see functions (Google Sheet) and SQL phrases in brackets:
number of rows (COUNTA / COUNT)
average of data (AVERAGE / AVG)
largest and smallest data (MAX, MIN)
total value (SUM)
filtering (COUNTIF, AVERAGEIF, MAXIFS, MINIFS, SUMIF / WHERE)
logical operators (AND, OR as functions / AND, OR)
relational operators (<, <=, =, >=, >)
summarizing as group (pivot table / GROUP BY)
Patterns: if exists, usually spotable from the data or from the visualized version of it.
Trends:
upward
steady
downward
The trend can describe the whole analysed data, or it can be divided to more trends.
Statistical fluctuations: on a specific scale data can seem to be random or volatile. By choosing a different scale the trend - if exists - can pop-up.
Predictions: based on detected trends, calculates data on yet-or-already unknown points.
Prediction strategies (strongly depend on the data):
using the most recent change in the data
using the average of the choosen last amount of data
using the whole data
dividing the whole data to trends and using the frequency of changing these trends
transforming data to changing rates in % or using other equation, then using another strategy on these
any other strategy based on statistical methods
Linear regression: find the lines fitting the best to the data or the last part of it.
Statistical correlation: compute correlation between two sets of data:
positive: the increasing trend in data exists together with increasing trend in the other, and vica versa
negative: the trend directions are the opposites to each other
non-existing
The correlation can be visualized if the two datasets are forming the coordinate axies of a figure (scatter plot). The positive correlation appears as an increasing set of points (the more of this meets the more of that), the negative as a descreasing set of points (the more of this meets the less of that), while in the non-existing case there will not be trend in the points.
The rate of correlation is expressed in a closed [-1, +1] interval: +1 is a strong positive, -1 is a string negative correcation, 0 shows no correlation.
A variation of the scatter plot is the bubble plot where instead of points bubbles show the points, the bubble size shows the correlation with a third dataset (3D correlation). Such plot can be created from income and life expectancy, where every point represents a country, the bubble size show the country population.
Other added dimension can be coloring (every continent has different colour).
If linear scale on an axis is problematic or does not show correlation, logarithm scale is available to use as well.
In case the datasets have the same field (e.g. time), another technique is to plot the datasets together but the other axis will have more respective names (e.g. population from the first dataset, income from the other dataset; in such case it is possible to choose the second axis scale independently, and the correlation can pop-up visually).
Correlation does not imply causation. It shows connection but one will not cause the other by having correlation, it does not show the why.
In other cases, a correlation appears by coincidence.
3. BIG DATA
Big data comes from
huge databases or
huge amount of small databases
Sources:
scientific researches (e.g. high-resolution experiments)
digital libraries (e.g. archives)
medical records (e.g. MRI images)
user-facing applications (e.g. with millions of users)
Challenges:
storage (disk arrays, disk enclosures, data centers)
processing (scalable, parallel and distribited)
cooling and electricity (for storage and processing)
responsibility (sensitive data, PII: Personal Identifiable Information)
4. MACHINE LEARNING, AI
Machine learning (ML) is a type of algorithm that automatically improves itself based on experience, not by a programmer writing a better algorithm. The algorithm gains experience by processing more and more data and then modifying itself based on the properties of the data.
Three general approaches:
reinforcement learning: the algorithm performs actions that will be rewarded the most, often used by game-playing AI or navigational robots
unsupervised machine learning: the algorithm finds patterns in unlabeled data by clustering and identifying similarities, popular uses include recommendation systems and targeted advertising
supervised machine learning
Supervised machine learning: the algorithm analyzes labeled data and learns how to map input data to an output label, often used for classification and prediction.
The neural networks are the popular approach.
The input flows through many layers of neurons in hidden layers, and ultimately output is produced.
Training: e.g. if the aim is to identify contents of images, the network is fed with a lot of images which has a label of what it shows. The process will give weight to the root from the starting point to the ending point between the neurons, which results in the given image content (e.g. plane, flower, dog). As the time goes by, based on the feedback on the success of classification, the best root will have the highest total score of probability. One root with the highest total score per identified object type.
Accuracy: depends on the training data in amount and diversity. If the accuracy is low, the ML will have a huge bias.
The AIs are Large Language Models (LLMs), based on ML techniques. These models use natural language processing, utilizing text parsing algorithms based on libraries of word vectors which rank the similarity of words to other words based on how often they typically co-occur in digitized texts.
Predictions: ML systems it can make predictions based on historical data used for training. In case the training data is full of biases, the predictions will suffer from biases as well. Using biased prediction systems cause biased results and decisions afterwards.
Criminal justice:
risk assessment, renders defendents to high-, medium- and low-risk categories where high-risk means the defendent will commit another crime with high probability
used for determining bonds amounts and sentences
depending on the training data, it can have e.g. racial bias (lower risk for white defendents, higher risk for black defendents) coming from the fact that black people are more likely are under arrest on a suspicion than white people even when the factual crime rate is around the same
Hiring decisions:
usage started to spread due to higher amount of applications and the need of getting rid of human biases in HR decisions
the automated screening approach (Applicant Tracking Systems - ATS) categorizes into great, good and not-a-fit
low amount of resumes scored by HR as training material leads to huge bias
sex-based bias can appear (males are more lightly in favor over females) based on words in resumes connected to the word "woman"; the basic of such bias is that males are usually work with males and have less experience with female employees, coming from the fact that women in the past were less available or had less rights on the job market and in positions than males
another common bias is coming from the libraries used for natural language processing, coming from that terms related to science and math were more closely associated with males while terms related to the arts were more closely associated with females
similar bias is when positive associations are rendered to Euro-American names and negative to Afro-American names
there are software companies which build HR analysing systems, fighting against biases
nowadays many if not all bigger companies use a kind of ML or AI to analyse resumes and make predictions and decisions
Facial recognition:
makes predictions on gender, race, age, emotions, identity
it can work by detecting symmetry, contours of the lower head, lips, bone length and wideness of the ending of the nose, contrours of the eyes, eyebrows, position of the pupils within the eyes
other systems detect the surface of the face and create 3D mesh network from it
they can use optical or LIDAR technology
the biases mainly come from the defects of the utilized detection techniques. These systems work better on white male faces
Language translation:
the Rule-Based Machine Translation (RBMT) contained vocabulary and language rules; parsed the given sentence to words and categorized the words by their positions (adjective, noun etc.), translated the words, then used the target language rules to create the sentence
after the RBMT, as the digital archives appeared, the Statistical Machine Translation (SMT) appeared; broke down the text into smaller pieces and searched for texts by similarity in digitalized multi-language materials, then looked it up in the target language counterpart; with small training data SMT had far worse results than RBMT had, but with big data, was able to create fluent phrases within sentences at minimum
in the ML era the Neural Machine Translation (NMT) came, uses neural networks on enormous amounts of data; when trained well and with enough data, those algorithms can learn how to produce sentences that are fluent from start to finish
biases come due to the base materials which are highly biased: novels, essays, poems, articles, tweets, posts, comments; typical bias appears upon translating from non-gendered to gendered languages
5. SIMULATION
A simulation is an abstraction of an infinitely complex natural phenomena. It removes details that aren't necessary or are too difficult to simulate. The level of abstraction in a simulation depends on the aim of the simulation.
Simulators are based on a simulation, they can be physical, virtual (digital) and hybrid.
Simulations are used for
research, science
engineering
education
training
games
Simulations can be used for emulating events in the past, present, and make predictions as well since the simulated rules are able to point to the future, or point to the past in inverted mode.
EXAMPLE FOR SIMULATION OF A RACE BETWEEN THE TURTLE AND THE HARE IN EUROPEAN KVASI PSEUDO CODE:
Google Sites
Report abuse