• Mobile payments: The widespread adoption of mobile payments, such as Apple Pay and Google Wallet, in the 2010s greatly increased the convenience and accessibility of digital payments.
• Online lending platforms: The emergence of online lending platforms, such as Lending Club and Prosper, in the 2010s greatly expanded the reach and accessibility of personal and small business loans.
• Digital currencies and blockchain technology: The development of digital currencies such as Bitcoin and blockchain technology in the 2010s greatly increased the security and speed of digital transactions.
• Roboadvisers: The emergence of robo-advisers, which use algorithms to provide investment advice, in the 2010s greatly expanded the accessibility of investment management services.
• Digital banking: The popularization of digital banking in the form of online and mobile banking apps greatly increased the convenience and accessibility of banking services in the 2010s.
• Open banking: The concept of open banking, which allows customers to share their financial data with third-party providers, became more prevalent in the 2010s.
• Biometrics: The use of biometrics such as fingerprint and facial recognition for identity verification in fintech became more prevalent in the 2010s.
• Insurtech: The emergence of insurtech, which uses technology to make insurance more efficient and accessible, became more prevalent in the 2010s.
• Digital identities: The development of digital identities, which allow individuals to prove their identity online, became more prevalent in the 2010s.
• Artificial intelligence and machine learning: The use of artificial intelligence and machine learning in fintech, such as for fraud detection and risk management, became more prevalent in the 2010s.

• Legal research platforms, which use artificial intelligence to quickly search and analyze large volumes of legal information.
• Contract review software, which uses natural language processing to automatically review and identify key terms and clauses in legal documents.
• E-discovery tools, which automate the process of identifying, collecting, and reviewing electronic documents in the context of legal proceedings.
• Online dispute resolution (ODR), which uses technology to facilitate the resolution of disputes outside of traditional court proceedings.
• Legal chatbots, which use natural language processing to provide automated legal advice to individuals and businesses.

• CRISPR-Cas9: The development of CRISPR-Cas9 gene editing technology in the 2010s greatly expanded the capabilities of genetic engineering and opened up new opportunities for treating genetic diseases.
• Personalized medicine: The use of biotechnology in the form of personalized medicine, which tailors medical treatment to an individual's genetic makeup, became more prevalent in the 2010s.
• Synthetic biology: The development of synthetic biology in the 2010s allowed for the design and construction of new biological parts and systems.
• Bioprinting: The use of bioprinting technology to print living tissue and organs became an active area of research in the 2010s.
• Biodegradable plastics: The development of biodegradable plastics made from natural materials such as corn starch and sugarcane became more prevalent in the 2010s.
• Immunotherapy: The use of immunotherapy, which harnesses the body's immune system to fight cancer and other diseases, became an active area of research in the 2010s.
• Microbial biotechnology: The use of microbial biotechnology in the form of probiotics and fermented foods became more prevalent in the 2010s.
• Biopesticides: The development of biopesticides, which use natural materials such as bacteria and viruses to control pests, became more prevalent in the 2010s.
• Biorefineries: The use of biorefineries to convert biomass into a range of products including biofuels, chemicals, and materials became more prevalent in the 2010s.
• Biodegradable packaging: The development of biodegradable packaging made from natural materials such as plant fibers and starch became more prevalent in the 2010s
• In the last decade, DNA has been increasingly investigated as an alternative medium for cold data storage, presenting several advantages over standard hard drives such as a higher density, longer lifespan and lower energy consumption. However, such coding methods are limited by biochemical constraints that elevate the probability of errors being added to the coded nucleotides during synthesis, storage, and sequencing. Although such errors can be limited by carefully designing the produced strands, it is unfeasible to avoid them completely.

• Imaging technology such as MRI, CT and PET scans, which allow for non-invasive examination of internal organs and body structures.
• Robotic surgery, which uses robots to assist with or perform complex surgical procedures.
• Telemedicine, which allows for remote consultations and monitoring of patients using technology such as video conferencing and wearable devices.
• Biomedical devices such as pacemakers, artificial joints and insulin pumps, which improve the quality of life of patients with chronic conditions.
• Digital health tools, such as electronic health records (EHRs) and personal health apps, which allow for improved data sharing, monitoring and analysis of patients' health.

• Pangu-Weather by Huawei: Pangu-Weather is an AI model developed by Huawei that can predict weekly weather patterns worldwide much faster than traditional forecasting methods, while maintaining comparable accuracy. It employs a deep neural network trained on 39 years of reanalysis data, which combines historical weather observations with modern models. Unlike conventional methods that analyze weather variables one by one, Pangu-Weather can analyze all variables simultaneously within seconds. The AI system has demonstrated the ability to accurately predict extreme weather events, such as the path of a tropical cyclone, even without specific training data on such events. Pangu-Weather's success indicates that AI models can generalize weather patterns and handle previously unseen scenarios effectively.
• FourcastNet by Nvidia: Nvidia's FourcastNet is another AI-based weather forecasting model that has shown comparable accuracy to conventional methods while significantly reducing the time needed to produce forecasts. As one of the recent contributions to the AI-powered weather forecasting domain, FourcastNet aims to challenge traditional approaches with its efficient and rapid prediction capabilities. This technology, along with others like Pangu-Weather and GraphCast, is encouraging meteorologists to rethink how machine learning can complement existing weather prediction methods to achieve more reliable forecasts[2].
• GraphCast by Google DeepMind: GraphCast is another AI model developed by Google DeepMind that contributes to the advancement of weather forecasting. Though specific details about GraphCast's performance are not provided in the reference, it is mentioned alongside Pangu-Weather and FourcastNet as one of the AI systems that is pushing the boundaries of weather forecasting using machine learning techniques.

• Electric vehicles: The development and popularization of electric vehicles in the 2010s greatly increased the availability and accessibility of emissions-free transportation options.
• Autonomous vehicles: The development of autonomous vehicle technology in the 2010s greatly increased the capabilities and safety of vehicles, with companies such as Waymo, Tesla, Uber, and GM Cruise working on it.
• Connected cars: The integration of internet connectivity and advanced communication technology into vehicles in the 2010s greatly increased the capabilities and functionality of cars.
• Advanced driver assistance systems (ADAS): The widespread adoption of advanced driver assistance systems (ADAS) such as lane departure warning, adaptive cruise control, and automatic emergency braking in the 2010s greatly increased the safety of vehicles.
• Lightweight materials: The use of lightweight materials such as aluminum, carbon fiber, and composites in the 2010s greatly improved the fuel efficiency and performance of vehicles.
• Hybrid and plug-in hybrid vehicles: The development and popularization of hybrid and plug-in hybrid vehicles in the 2010s greatly increased the fuel efficiency and reduced emissions of vehicles.
• Fuel cell vehicles: The development of fuel cell vehicles, which use hydrogen fuel cells to generate electricity, became an active area of research in the 2010s.
• 3D printing: The use of 3D printing in the automotive industry in the 2010s greatly increased the efficiency and flexibility of manufacturing processes.

• Machine learning: The development and popularization of machine learning techniques such as deep learning and neural networks in the 2010s greatly expanded the capabilities of artificial intelligence. Machine learning innovation refers to the development and application of new techniques and algorithms in the field of machine learning. This can include advancements in areas such as deep learning, reinforcement learning, natural language processing, and computer vision. These innovations have led to significant improvements in the ability of machines to perform tasks such as image and speech recognition, language translation, and decision-making. Additionally, the increased availability of large amounts of data and computational power have also been key drivers of machine learning innovation.
• Natural Language Processing (NLP): The development of natural language processing (NLP) technologies in the 2010s greatly improved the ability of machines to understand and generate human language.
• Computer Vision: The development of computer vision technologies in the 2010s greatly improved the ability of machines to understand and interpret visual information.
• Robotics: The development of advanced robotics and autonomous systems in the 2010s greatly expanded the capabilities and applications of artificial intelligence.
• Virtual assistants: The emergence of virtual assistants such as Siri, Alexa, and Google Assistant in the 2010s greatly increased the accessibility and convenience of artificial intelligence.
• Recommendation Systems: The popularization of recommendation systems in the 2010s greatly improved the ability of machines to personalize and optimize experiences for users.
• Chatbots: The development and deployment of chatbots for customer service and other applications in the 2010s greatly increased the accessibility and convenience of artificial intelligence.
• Generative Adversarial Networks (GANs): The development of Generative Adversarial Networks (GANs) in the 2010s greatly improved the ability of machines to generate new data, such as images and text.
• Reinforcement Learning: The development of reinforcement learning in the 2010s greatly improved the ability of machines to learn and make decisions in complex, dynamic environments.
• Explainable AI (XAI): The development of explainable AI (XAI) in the 2010s greatly improved the ability of machines to provide understandable and transparent explanations for their decisions.

• Internet of Things (IoT): 5G networks are expected to play a major role in enabling the IoT by providing the necessary connectivity and low-latency required for IoT devices.
• Industry 4.0: 5G networks will also enable Industry 4.0, by providing fast, low-latency, and reliable connections for industrial automation and robotics, and allowing for real-time monitoring and control of industrial processes.
• Autonomous vehicles: With their low latency and high-bandwidth capabilities, 5G networks will enable the development of autonomous vehicles and smart transportation systems.
• Augmented Reality and Virtual Reality (AR/VR): 5G networks will provide the necessary connectivity and low-latency required for immersive AR and VR experiences, making it possible to use them in a variety of applications such as remote collaboration, education, and entertainment.
• Edge computing: 5G networks will enable the deployment of edge computing, which will bring computation and data storage closer to the devices that need it, reducing latency and providing faster response times.
• Network slicing: 5G networks will allow for the creation of multiple "slices" of the network, each with different characteristics and capabilities, to support the specific needs of different use cases and applications.
• Security: 5G networks will provide new security capabilities, such as network slicing, that will allow for more secure and reliable connections between devices and systems.

• Metaverse virtual world contains digital representations of individuals and objects in the physical world, allowing interaction and engagement in various contexts.
• Extended Reality (XR) is an umbrella term that refers to computer-generated environments that merge the physical and virtual worlds or create an entirely virtual experience for users. XR encompasses three technologies: Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). XR takes the human-to-PC screen interface and modifies it by either immersing users in a virtual environment (VR) or adding elements to the user's surroundings (AR) or both (MR). 
• Digital twins bridge the gap between the physical and digital realms by creating rich digital models of physical assets, objects, or environments of anything that is important. 

• Virtual and Augmented Reality: The use of VR and AR technologies is crucial for creating fully immersive and believable virtual worlds.
• Blockchain: Blockchain technology is being explored as a way to create secure and decentralized virtual worlds, where users have full control over their data and virtual assets.
• 5G and Edge computing: High-speed internet and edge computing are necessary to support real-time interactions and large numbers of users within the metaverse.
• Artificial Intelligence: AI is being used to create more realistic and responsive virtual characters and environments.
• Game engines: Game engines such as Unity and Unreal Engine are being used to create metaverse experiences.
• Interoperability: Interoperability between different virtual worlds, platforms and devices is important for creating a seamless metaverse experience.

• Identification of Use Cases, either existing or possible (automotive, defence, education, enterprise, eSports, events, finance, food, gaming, healthcare, hospitality, professional training,real estate, remote work, retail, social media, travel, virtual spaces, workflows,.
• Identification of External Services that a Metaverse can use (content creation /Roblox, MagicaVoxel, OmniVerse/, marketplace /OpenSea/, crypto wallets, cryptocurrency exchanges, development services /LandVault, InfiniteReality/, DigiSomni/, platforms /Metaverse.network, Vircadia/).
• Identification of Functionalities from use cases and external services (instance, environment, content representation, perception, user, interaction, information search, economy support)
• Analysis of the state and characteristics of the Technologies required to support the functionalities (sensory information, data processing, user devices, network, energy).
• Analysis of the issues regarding the management of the Metaverse Ecosystem and legal issues (governance /regulation, property, trademark, authorship, contract, tort, defamation, privacy, taxation, mental health/, stakeholders /manager, operator, user/)

• Real-time data integration: Digital twins are built with real-time data from sensors and other sources, providing a complete and accurate representation of the physical asset.
• Simulation and modeling: Digital twins can be used to simulate the behavior and performance of physical assets, allowing for analysis and testing of different scenarios.
• Predictive maintenance: Digital twins can analyze data to detect potential issues and predict when maintenance is required, reducing downtime and improving efficiency.
• Remote monitoring: Digital twins allow for remote monitoring and control of physical assets, reducing the need for on-site visits and increasing accessibility.
• Collaboration and decision making: Digital twins enable stakeholders to collaborate and make informed decisions based on a shared understanding of the physical asset's state and performance.
• Contextual information: Digital twins can include a range of contextual information, such as design specifications, historical data, and environmental conditions.

• Engine parameter specifies the AI model employed to generate predictions.
• Max tokens parameter specifies the maximum number of tokens that can be generated by the model. A token can be seen as a piece of word. As a rule of thumb, 1 token is around 4 characters.
• Temperature parameter close to 1 would mean that the logits are passed through the softmax function without modification. If the temperature is close to zero, the highest probable tokens will become very likely compared to the other tokens, i.e. the model becomes more deterministic and will always output the same set of tokens after a given sequence of words.
• Top p  parameter specifies a sampling threshold during inference time. Top p sampling (sometimes called nucleus sampling) is a technique used to sample possible outcomes of the model.
• Frequency penalty parameter controls the model’s tendency to repeat predictions.  It reduces the probability of words that have already been generated. The penalty depends on how many times a word has already occurred in the prediction.
• Presence penalty parameter encourages the model to make novel predictions. The presence penalty lowers the probability of a word if it already appeared in the predicted text. Unlike the frequency penalty, the presence penalty does not depend on the frequency at which words appear in past predictions.

• Bias and discrimination: GPT-3 and other language models are trained on data that reflects societal biases and prejudices. This can result in generated text that perpetuates harmful stereotypes and reinforces existing inequalities.
• Misinformation and propaganda: The ability of GPT-3 to generate coherent and context-aware text makes it susceptible to use for spreading misinformation and propaganda.
• Responsibility and accountability: As GPT-3 and other language models become more widely used, questions arise about who is responsible for their actions and who should be held accountable for any harm they cause.
• Privacy and security: GPT-3 and other language models process large amounts of sensitive data, raising questions about data privacy and security.
• Job displacement: The increasing capabilities of GPT-3 and other language models raise concerns about their potential impact on employment and the displacement of workers in language-related jobs.

• ChatGPT capabilities include answering questions, generating text, and translating text. GPT-3 demonstrate remarkable text generation ability. Given words, phrases or simple sentences as prefix, they can continue to generate text that are syntactically correct and semantically smooth conditioning on the given text. ChatGPT was fine-tuned on top of GPT-3.5 using supervised learning as well as reinforcement learning. In addition, OpenAI continues to gather data from users that could be used to further train and fine-tune ChatGPT. Unlike most chatbots, ChatGPT remembers previous prompts given to it in the same conversation. 
• OpenAI plug-ins are tools designed specifically for language models that help ChatGPT access up-to-date information, run computations, or use third-party services. The plug-ins enables ChatGPT Plus to access live web data versus information the large language model was already trained on. ChatGPT uses the Bing Application Programming Interface (API) to do searches and a text-based web browser to explore and interact with page results. It may combine data from different sources into a coherent response. 
• DALL-E is deep learning model based on Markov chains trained using variational inference. The goal of latent diffusion models (LDM) is to learn the latent structure of a dataset by modeling the way in which data points diffuse through the latent space. Feature space or embedding space is an embedding of a set of items within a manifold in which items resembling each other are positioned closer to one another in the latent space. DALL-E's model is a multimodal implementation of GPT-3 trained on text-image pairs from the Internet to generate digital images from natural language descriptions (prompts) which combines concepts, attributes, and styles. The model can generate imagery in multiple styles, including photorealistic imagery, paintings, and emoji. It can manipulate and rearrange objects in its images, and can correctly place design elements in novel compositions without explicit instruction.
• Stable Diffusion is a deep learning text-to-image model primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt. 

1. Write clear instructions.  These models can’t read your mind. If outputs are too long, ask for brief replies. If outputs are too simple, ask for expert-level writing. If you dislike the format, demonstrate the format you’d like to see. The less the model has to guess at what you want, the more likely you’ll get it.

2. Provide reference text.  Language models can confidently invent fake answers, especially when asked about esoteric topics or for citations and URLs. In the same way that a sheet of notes can help a student do better on a test, providing reference text to these models can help in answering with fewer fabrications.

3. Split complex tasks into simpler subtasks.  Just as it is good practice in software engineering to decompose a complex system into a set of modular components, the same is true of tasks submitted to a language model. Complex tasks tend to have higher error rates than simpler tasks. Furthermore, complex tasks can often be re-defined as a workflow of simpler tasks in which the outputs of earlier tasks are used to construct the inputs to later tasks.

4. Give the model time to "think".  If asked to multiply 17 by 28, you might not know it instantly, but can still work it out with time. Similarly, models make more reasoning errors when trying to answer right away, rather than taking time to work out an answer. Asking for a "chain of thought" before an answer can help the model reason its way toward correct answers more reliably.

5. Use external tools.  Compensate for the weaknesses of the model by feeding it the outputs of other tools. For example, a text retrieval system (sometimes called RAG or retrieval augmented generation) can tell the model about relevant documents. A code execution engine like OpenAI's Code Interpreter can help the model do math and run code. If a task can be done more reliably or efficiently by a tool rather than by a language model, offload it to get the best of both.

6. Test changes systematically.  Improving performance is easier if you can measure it. In some cases a modification to a prompt will achieve better performance on a few isolated examples but lead to worse overall performance on a more representative set of examples. Therefore to be sure that a change is net positive to performance it may be necessary to define a comprehensive test suite (also known an as an "eval").

• ChatGPT capabilities include answering questions, generating text, and translating text. GPT-3 demonstrate remarkable text generation ability. Given words, phrases or simple sentences as prefix, they can continue to generate text that are syntactically correct and semantically smooth conditioning on the given text. ChatGPT was fine-tuned on top of GPT-3.5 using supervised learning as well as reinforcement learning. In addition, OpenAI continues to gather data from users that could be used to further train and fine-tune ChatGPT. Unlike most chatbots, ChatGPT remembers previous prompts given to it in the same conversation. 
• OpenAI plug-ins are tools designed specifically for language models that help ChatGPT access up-to-date information, run computations, or use third-party services. The plug-ins enables ChatGPT Plus to access live web data versus information the large language model was already trained on. ChatGPT uses the Bing Application Programming Interface (API) to do searches and a text-based web browser to explore and interact with page results. It may combine data from different sources into a coherent response. 
• DALL-E is deep learning model based on Markov chains trained using variational inference. The goal of latent diffusion models (LDM) is to learn the latent structure of a dataset by modeling the way in which data points diffuse through the latent space. Feature space or embedding space is an embedding of a set of items within a manifold in which items resembling each other are positioned closer to one another in the latent space. DALL-E's model is a multimodal implementation of GPT-3 trained on text-image pairs from the Internet to generate digital images from natural language descriptions (prompts) which combines concepts, attributes, and styles. The model can generate imagery in multiple styles, including photorealistic imagery, paintings, and emoji. It can manipulate and rearrange objects in its images, and can correctly place design elements in novel compositions without explicit instruction.
• Stable Diffusion is a deep learning text-to-image model primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt. 

Model GPT-4

• GPT-4 is more than 10x the size of GPT-3 (175 B). We believe it has a total of ~1.8 trillion parameters across 120 layers. Mixture Of Experts (16 experts, each ~111B). Not a dense transformer like e.g. PaLM (or GPT-3). They use MQA instead of MHA (classic at this point).
• Each forward pass (generation of 1 token) only utilizes ~280B parameters and ~560 TFLOPs. This contrasts with the ~1.8 trillion parameters and ~3,700 TFLOP that would be required per forward pass of a purely dense model.

• To parallelize across all their A100s GPUs They utilized 8-way tensor parallelism. Beyond that, they are using 15-way pipeline parallelism. Also apparently they used DeepSpeed ZeRo Stage 1 or block-level FSDP.
• (You can check out my video on all of these strategies here: https://lnkd.in/e8ammDs5 3D parallelism is what you're looking for & ZeRo)

• They have a separate vision encoder from the text encoder, with cross-attention. The architecture is similar to Google DeepMind's Flamingo (I used to work on this project :) ). This adds more parameters on top of the 1.8T of GPT-4. It is fine-tuned with another ~2 trillion tokens, after the text-only pre-training.

• Trained on ~13T tokens (multiple epochs, not unique). Plus millions of rows of instruction fine-tuning data from ScaleAI & internally (I guess acquired through ChatGPT + their API before they changed the policy).
• 8k context length for the pre-training phase. The 32k seqlen version of GPT-4 is based on fine-tuning of the 8k after the pre-training. See e.g. MosaicML's blog on how to achieve this: https://lnkd.in/dagmDAZb)

• OpenAI’s training FLOPS for GPT-4 is ~2.15e25, on ~25,000 A100s for 90 to 100 days at about 32% to 36% MFU. Part of this extremely low utilization is due to an absurd number of failures requiring checkpoints that needed to be restarted from.
• If their cost in the cloud was about $1 per A100 hour, the training costs for this run alone would be about $63 million.
• Today, the pre-training could be done with ~8,192 H100 in ~55 days for $21.5 million at $2 per H100 hour

• OpenAI might be using speculative decoding on GPT-4's inference. See this paper: https://lnkd.in/d-C-QpwW
• The inference runs on a cluster of 128 GPUs. There are multiple of these clusters in multiple datacenters in different locations (it'll be hard for Elizier to nuke these xD). 8-way tensor parallelism and 16-way pipeline parallelism.

• Stable Diffusion is a deep learning text-to-image model primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt. 
• Models (checkpoint files) are pre-trained Stable Diffusion weights intended for generating general or a particular genre of images. What images a model can generate depends on the data used to train them. A model won't be able to generate a cat's image if there's never a cat in the training data. HuggingFace is the go-to platform for creators who build AI models for Stable Diffusion. There are currently over 270 models related to Stable Diffusion on the platform. You can use Stable Diffusion online through DreamStudio, without any kind of hardware requirements at all.
• Stable Diffusion is in a class of models called latent diffusion models. Latent diffusion models encode images and their text captions into vectors (basically a unique numerical representation for each image). During training time, the model adds random values (noise) to the vectors. And then you train a model to go from a slightly more noisy vector to a slightly less noisy vector. In other words, the model tries to reproduce the original numerical representation of every image in its training set, based on that image’s accompanying text caption. So a generated image wants to be similar to the images that most influenced it, by having a similar numerical representation.
• Stable Diffusion consists of 3 parts: the variational autoencoder (VAE), U-Net, and an optional text encoder. The VAE encoder compresses the image from pixel space to a smaller dimensional latent space, capturing a more fundamental semantic meaning of the image. Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion. The U-Net block, composed of a ResNet backbone, denoises the output from forward diffusion backwards to obtain a latent representation. Finally, the VAE decoder generates the final image by converting the representation back into pixel space. Stable Diffusion makes its source code available https://github.com/CompVis/stable-diffusion
• Stable Diffusion was trained on pairs of images and captions taken from LAION-5B, a publicly available dataset derived from Common Crawl data scraped from the web, where 5 billion image-text pairs were classified based on language and filtered into separate datasets by resolution, a predicted likelihood of containing a watermark, and predicted "aesthetic" score (e.g. subjective visual quality).
• To address the limitations of the model's initial training, end-users may opt to implement additional training to fine-tune generation outputs to match more specific use-cases. As visual styles and compositions are not subject to copyright, it is often interpreted that users of Stable Diffusion who generate images of artworks should not be considered to be infringing upon the copyright of visually similar works. However, individuals depicted in generated images may be protected by personality rights if their likeness is used,[48] and intellectual property such as recognizable brand logos still remain protected by copyright. 

New progress on key tech for making AR

ImageBind: Holistic AI learning across six modalities

Glossary

• Artificial intelligence (AI) is the ability of software to perform tasks that traditionally require human intelligence.
• Artificial neural networks (ANNs) are composed of interconnected layers of software-based calculators known as “neurons.” These networks can absorb vast amounts of input data and process that data through multiple layers that extract and learn the data’s features.
• Deep learning is a subset of machine learning that uses deep neural networks, which are layers of connected “neurons” whose connections have parameters or weights that can be trained. It is especially effective at learning from unstructured data such as images, text, and audio.
• Early and late scenarios are the extreme scenarios of our work-automation model. The “earliest” scenario flexes all parameters to the extremes of plausible assumptions, resulting in faster automation development and adoption, and the “latest” scenario flexes all parameters in the opposite direction. The reality is likely to fall somewhere between the two.
• Fine-tuning is the process of adapting a pretrained foundation model to perform better in a specific task. This entails a relatively short period of training on a labeled data set, which is much smaller than the data set the model was initially trained on. This additional training allows the model to learn and adapt to the nuances, terminology, and specific patterns found in the smaller data set.
• Foundation models (FM) are deep learning models trained on vast quantities of unstructured, unlabeled data that can be used for a wide range of tasks out of the box or adapted to specific tasks through fine-tuning. Examples of these models are GPT-4, PaLM, DALL·E 2, and Stable Diffusion.
• Generative AI is AI that is typically built using foundation models and has capabilities that earlier AI did not have, such as the ability to generate content. Foundation models can also be used for nongenerative purposes (for example, classifying user sentiment as negative or positive based on call transcripts) while offering significant improvement over earlier models. For simplicity, when we refer to generative AI in this article, we include all foundation model use cases.
• Graphics processing units (GPUs) are computer chips that were originally developed for producing computer graphics (such as for video games) and are also useful for deep learning applications. In contrast, traditional machine learning and other analyses usually run on central processing units (CPUs), normally referred to as a computer’s “processor.”  Unfortunately, many explanations and books about programming GPUs (NVIDIA, AMD) mix up techniques suitable for CPUs, such as spinlocks, mutexes, traditional locks, and thread sleeping/wake-up mechanisms, which are totally at odds with the way GPUs should be programmed. The reason for this, as many tech-bros around here know, lies in how CPU and GPU architectures differ. CPU architecture is built on time-slicing and MIMD instructions (Multiple Instruction, Multiple Data), while GPU groups its super-stellar type of instruction, SIMD (Single Instruction, Multiple Data ), into warps (NVIDIA-like) or wavefronts (AMD-like). From an easy abstract algebra point of view, the reason that GPU and CPU screw up each other lies in the impossible perfect mapping (morphisms) of CPU code into GPU code. Why? Take the easy explanation for category theory: CPU deals with objects (processing units) and arrows (operations) on different data streams in MIMD. Each processing unit (object) can execute different operations (arrows) on independent data streams. So, you have here, instead of a massive amount of threads (GPU), a massive amount of independent categories. From this mathematical perspective, a CPU is nothing more than a lot of independent operations, each with a diverse composition of morphisms, generating a vast number of independent execution paths (and saying 'a lot' is still an understatement).  In contrast, GPU with its SIMD involves only a single operation (arrow in category terms) applied simultaneously across multiple data elements (in category terms: objects). This can be seen as a single category where the same morphism is applied across various objects in parallel (for the math-savvy tech-bros, in reality, we have here an endomorphism operating in the same  category, kinda casting the same object into different types in traditional checked-type programming). Ok, now, after this brief excursion into the realm of category theory, we are ready to understand the core of the problem of why Parallelism used in CPU sucks when transferred to GPU: MIMD architectures support complex, branching operations where each processor can follow a different execution path. Translating this to SIMD would require aligning these diverse paths into a single, unified operation ( kinda a catamorphism from the general morphism of MIMD to the endomorphism within SIMD, not a trivial task, I tell you). The morphisms with MIMD allow processors to operate independently without waiting for others. SIMD requires data and operation synchronization across all processing elements, introducing complexity in aligning independent MIMD operations into a SIMD framework. And thus, my tech-bros, we lose the flexibility and independence in MIMD lost in SIMD.
• Large language models (LLMs) make up a class of foundation models that can process massive amounts of unstructured text and learn the relationships between words or portions of words, known as tokens. This enables LLMs to generate natural-language text, performing tasks such as summarization or knowledge extraction. GPT-4 (which underlies ChatGPT) and LaMDA (the model behind Bard) are examples of LLMs.
• Machine learning (ML) is a subset of AI in which a model gains capabilities after it is trained on, or shown, many example data points. Machine learning algorithms detect patterns and learn how to make predictions and recommendations by processing data and experiences, rather than by receiving explicit programming instruction. The algorithms also adapt and can become more effective in response to new data and experiences.
• Modality is a high-level data category such as numbers, text, images, video, and audio.
• Prompt engineering refers to the process of designing, refining, and optimizing input prompts to guide a generative AI model toward producing desired (that is, accurate) outputs.
• Self-attention, sometimes called intra-attention, is a mechanism that aims to mimic cognitive attention, relating different positions of a single sequence to compute a representation of the sequence.
• Structured data are tabular data (for example, organized in tables, databases, or spreadsheets) that can be used to train some machine learning models effectively.
• Transformers are a relatively new neural network architecture that relies on self-attention mechanisms to transform a sequence of inputs into a sequence of outputs while focusing its attention on important parts of the context around the inputs. Transformers do not rely on convolutions or recurrent neural networks.
• Technical automation potential refers to the share of the worktime that could be automated. We assessed the technical potential for automation across the global economy through an analysis of the component activities of each occupation. We used databases published by institutions including the World Bank and the US Bureau of Labor Statistics to break down about 850 occupations into approximately 2,100 activities, and we determined the performance capabilities needed for each activity based on how humans currently perform them.
• Use cases are targeted applications to a specific business challenge that produces one or more measurable outcomes. For example, in marketing, generative AI could be used to generate creative content such as personalized emails.
• Unstructured data lack a consistent format or structure (for example, text, images, and audio files) and typically require more advanced techniques to extract insights.

• Machine learning. Most AI systems today are powered by machine learning, where predictive models are trained on historical data and used to make future predictions. The rise of machine learning within AI started in the 1990s, representing a marked shift from the way AI systems were built previously: rather than specifying how to solve a task, a learning algorithm would induce it based on data — i.e., the how emerges from the dynamics of learning. Machine learning also represented a step towards homogenization: a wide range of applications could now be powered by a single generic learning algorithm such as logistic regression. Despite the ubiquity of machine learning within AI, semantically complex tasks in natural language processing (NLP) and computer vision such as question answering or object recognition, where the inputs are sentences or images, still required domain experts to perform “feature engineering” — that is, writing domain-specific logic to convert raw data into higher-level features (e.g., SIFT) in computer vision) that were more suitable for popular machine learning methods.
• Deep learning. Around 2010, a revival of deep neural networks under the moniker of deep learning [LeCun et al. 2015] started gaining traction in the field of machine learning. Deep learning was fueled by larger datasets, more computation (notably, the availability of GPUs), and greater audacity. Deep neural networks would be trained on the raw inputs (e.g., pixels), and higher-level features would emerge through training (a process dubbed “representation learning”). This led to massive performance gains on standard benchmarks, for example, in the seminal work of AlexNet [Krizhevsky et al. 2012] on the ImageNet dataset [Deng et al. 2009]. Deep learning also reflected a further shift towards homogenization: rather than having bespoke feature engineering pipelines for each application, the same deep neural network architecture could be used for many applications.
• Foundation models. Foundation models have taken shape most strongly in NLP, so we focus our story there for the moment. That said, much as deep learning was popularized in computer vision but exists beyond it, we understand foundation models as a general paradigm of AI, rather than specific to NLP in any way. By the end of 2018, the field of NLP was about to undergo another seismic change, marking the beginning of the era of foundation models. On a technical level, foundation models are enabled by transfer learning [Thrun 1998] and scale. The idea of transfer learning is to take the “knowledge” learned from one task (e.g., object recognition in images) and apply it to another task (e.g., activity recognition in videos). Within deep learning, pretraining is the dominant approach to transfer learning: a model is trained on a surrogate task (often just as a means to an end) and then adapted to the downstream task of interest via fine-tuning. Transfer learning is what makes foundation models possible, but scale is what makes but scale is what makes them powerful. Scale required three ingredients: 

1. improvements in computer hardware — e.g., GPU throughput and memory have increased 10× over the last four years; 
2. the development of the Transformer model architecture [Vaswani et al. 2017] that leverages the parallelism of the hardware to train much more expressive models than before; and 
3. the availability of much more training data. 

• Manipulative and deceptive practices.  AI systems that use subliminal techniques to materially distort a person’s decision-making capacity, leading to significant harm, are banned. This includes systems that manipulate behaviour or decisions in a way that the individual would not have otherwise made.
• Exploitation of vulnerabilities.  The Act prohibits AI systems that target individuals or groups based on age, disability, or socio-economic status to distort behaviour in harmful ways.
• Biometric categorisation.  It outlaws AI systems that categorise individuals based on biometric data to infer sensitive information like race, political opinions, or sexual orientation. This prohibition does not cover any labelling or filtering of lawfully acquired biometric datasets, such as images. There are also exceptions for law enforcement.
• Social scoring.  AI systems designed to evaluate individuals or groups over time based on their social behaviour or predicted personal characteristics, leading to detrimental treatment, are banned.
• Real-time biometric identification.  The use of real-time remote biometric identification systems in publicly accessible spaces for law enforcement is heavily restricted, with allowances only under narrowly defined circumstances that require judicial or independent administrative approval.
• Risk assessment in criminal offences.  The Act forbids AI systems that assess the risk of individuals committing criminal offences based solely on profiling, except when supporting human assessment already based on factual evidence.
• Facial recognition databases.  AI systems that create or expand facial recognition databases through untargeted scraping of images are prohibited.
• Emotion inference in workplaces and educational institutions.  The use of AI to infer emotions in sensitive environments like workplaces and schools is banned, barring exceptions for medical or safety reasons.

Explainable XAI

• Fairness and debiasing: Manage and monitor fairness. Scan your deployment for potential biases. 
• Model drift mitigation: Analyze your model and make recommendations based on the most logical outcome. Alert when models deviate from the intended outcomes.
• Model risk management: Quantify and mitigate model risk. Get alerted when a model performs inadequately. Understand what happened when deviations persist.
• Lifecycle automation: Build, run and manage models as part of integrated data and AI services. Unify the tools and processes on a platform to monitor models and share outcomes. Explain the dependencies of machine learning models.
• Multicloud-ready: Deploy AI projects across hybrid clouds including public clouds, private clouds and on premises. Promote trust and confidence with explainable AI.

Responsible AI

• Accountability referes to the need to explain and justify one's decision and actions to its partners, users and others with whom the system interacts.
• Responsibility referes to the role of people themselves and to the capability of AI systems answer for one's decision and identify errors or unexpected results.
• Transparency referes to the need to describe, inspect and reproduce the mechanisms through governance of the data used created.
• Fairness refers to the equitable treatment of individuals, or groups of individuals, by an AI system. Bias occurs when an AI system has been designed, intentionally or not, in a way that may make the system's output unfair. Bias can be present both in the algorithm of the AI system and in the data used to train and test it. It can emerge as a result of cultural, social, or institutional expectations; because of technical limitations of its design; or when the system is used in unanticipated contexts or to make decisions about communities that are not considered in the initial design.
• In order to ensure that systems will uphold human values, design methods are needed that incorporate ethical principles and address societal concerns.  Moreover, implementing ethical actions in machines will help us better understand ethics overall. Responsible AI implies the need for mechanisms that enable AI systems themselves to reason about, and act according to, ethics and human values. This requires models and algorithms to represent and reason about, and take decisions based on, human values, and to justify their decisions according to their effect on those values.

Ethically Aligned Design (EAD1)

• EAD Pillars of Conceptual Framework fall broadly into three areas, reflecting anthropological, political, and technical aspects.: universal human values, political self-determination and data agency, and technical dependability. 
• EAD General Principles have emerged through the continuous work of dedicated, open communities in a multi-year, creative, consensus-building process. They articulate high-level principles that should apply to all types of autonomous and intelligent systems (A/IS). Created to guide behavior and inform standards and policy making, the General Principles define imperatives for the ethical design, development, deployment, adoption, and decommissioning of autonomous and intelligent systems. The Principles consider the role of A/IS creators, i.e., those who design and manufacture, of operators, i.e., those with expertise specific to use of A/IS, other users, and any other stakeholders or affected parties.

• Human rights.  A/IS shall be created and operated to respect, promote, and protect internationally recognized human rights.
• Well-being.  A/IS creators shall adopt increased human well-being as a primary success criterion for development.
• Data agency.  A/IS creators shall empower individuals with the ability to access and securely share their data, to maintain people’s capacity to have control over their identity.
• Effectiveness.  A/IS creators and operators shall provide evidence of the effectiveness and fitness for purpose of A/IS.
• Transparency.  The basis of a particular A/IS decision should always be discoverable.
• Accountability.  A/IS shall be created and operated to provide an unambiguous rationale for all decisions made.
• Awareness of misuse.  A/IS creators shall guard against all potential misuses and risks of A/IS in operation.
• Competence.  A/IS creators shall specify and operators shall adhere to the knowledge and skill required for safe and effective operation. 

• maintaining human autonomy
• implications of cultural migration in A/IS
• applying goal-directed behavior (Virtue Ethics) to A/IS
• eequirement for rule-based ethics in practical programming

Arts and AI?

AI art generators:

• The most straightforward idea is to just use the training data itself as the targets. 
• Then, people tried hiding some parts of the training data and using those missing parts as the targets. This is called masking, which is how LLMs are trained these days.
• Then, people tried pairing up text and images and using each other as targets. This is called constrative learning. This is the C in the famous CLIP model from OpenAI, which is the basis of all the multimodal foundational models.

• real-time vision (images and videos), 
• memory, 
• GPTs, 
• browse, and 
• advanced data analysis.

• Contrastive learning has been widely explored in VLM pretraining, which designs contrastive objectives for learning discriminative image-text features
• Generative VLM pre-training learns semantic knowledge by learning to generate images or texts via masked image modelling, masked language modelling, masked cross-modal modelling and image-to-text generation. Generative Adversarial Networks (GANs) have been a significant breakthrough in generative modeling, with the unique ability to generate realistic and diverse samples. GANs consist of two components, the generator and the discriminator, which engage in a continuous adversarial process. The generator produces synthetic images from random noise, while the discriminator aims to distinguish between these generated images and real images from the training dataset. Through backpropagation and optimization, the generator continually refines its outputs in response to the feedback from the discriminator and generates more realistic images. Diffusion-based methods model the image generation process as a series of diffusion steps, progressively refining the generated image. Diffusion models (DMs), also commonly known as diffusion probabilistic models, represent a class of generative models founded on Markov chains and trained through weighted variational inference. The primary objective of is to learn the impact of noise on the available information in the sample or the degree to which the diffusion process reduces the information available. The two-step process in consists of forward and reverse diffusion. In the forward diffusion process, Gaussian noise is successively introduced, representing the diffusion process until the data becomes total noise. The reverse diffusion process trains a neural network to learn the conditional distribution probabilities, allowing the model to reverse the noise and reconstruct the original data effectively.  Inspired by the success of autoregressive Transformers, autoregressive methods focus on sequentially predicting individual pixels or regions in an image, generating the output based on a learned probability distribution.
• Alignment objectives enforce VLMs to align paired images and texts by learning to predict whether the given text describes the given image correctly. It can be broadly categorized into global image-text matching and local region-word matching for VLM pre-training.

• Representation studies how to represent and summarize multimodal data to reflect the heterogeneity and interconnections between individual modality elements.
• Alignment aims to identify the connections and interactions across all elements. 
• Reasoning aims to compose knowledge from multimodal evidence usually through multiple inferential steps for a task.
• Generation involves learning a generative process to produce raw modalities that reflect cross-modal interactions, structure, and coherence. 
• Transference aims to transfer knowledge between modalities and their representations. 
• Quantification involves empirical and theoretical studies to better understand the multimodal learning process.

• AI-enabled IoT applications and use cases: Explore innovative applications and use cases where AI enhances IoT functionalities and capabilities, spanning smart healthcare, intelligent transportation, precision agriculture, industrial automation, environmental monitoring, and more.
• AI-driven data analytics and decision-making: Investigate AI techniques for processing, analyzing, and deriving actionable insights from IoT-generated data streams, enabling predictive maintenance, anomaly detection, personalized recommendations, and intelligent automation.
• Generative AI and Large Language Models (LLMs) for IoT: Study the applications of generative AI in IoT environments; explore how LLMs, such as generative pre-trained transformer (GPT) models, can be utilized to generate synthetic data, enhance natural language understanding, and support human-machine interaction in IoT systems.
• Edge AI and distributed intelligence: Discuss the integration of AI algorithms and models at the edge of IoT networks, enabling real-time inference, adaptive learning, and autonomous decision-making closer to data sources, and minimizing latency and bandwidth requirements.
• AI-empowered robotics and sensing for IoT: Explore the integration of AI with robotics and sensing technologies in IoT systems. Introduce advancements in sensor technologies and data fusion techniques that enable intelligent data collection, processing, and analysis in dynamic environments.
• Privacy, security, and trustworthiness: Address the privacy and security implications of AI-enabled IoT systems, including data privacy, confidentiality, integrity, and authenticity. Investigate trustworthiness in AI-enabled IoT systems, emphasizing the need for reliability, transparency, explainability, accountability, and fairness.
• Standardization and interoperability: Discuss the challenges and opportunities in standardizing AI-enabled IoT technologies to ensure interoperability, compatibility, and seamless integration across heterogeneous IoT ecosystems. Explore emerging standards, protocols, and frameworks for facilitating collaboration and interoperability among AI and IoT technologies.
• Demonstrations, Proof-of-Concepts, and deployments: Present innovative demonstrations and proof-of-concepts showcasing the integration of AI technologies with IoT systems. Share insights and best practices for deploying AI-powered IoT solutions in diverse environments, including smart cities, healthcare, agriculture, manufacturing, transportation, and energy.
• Regulation and policy: Explore the regulatory and policy landscape surrounding AI and IoT technologies, including data governance, consumer protection, liability, accountability, and ethical considerations. Discuss the role of regulatory bodies, industry consortia, and international organizations in shaping ethical, legal, and policy frameworks to ensure the responsible development, deployment, and use of AI-powered IoT systems.