Horizon Smart

• Smart home devices: products such as smart thermostats, smart lighting, and smart appliances that can be controlled remotely through a smartphone app or voice commands.
• Smart wearables: products such as smartwatches, fitness trackers, and smart glasses that can track various health and fitness metrics, and provide notifications and other information.
• Smart transportation: services such as ride-sharing apps and self-driving cars, which use technology such as GPS, real-time data, and traffic management systems to optimize transportation.
• Smart healthcare: products such as wearable devices and telemedicine systems that can monitor patients' health and provide remote medical care.
• Smart energy management: use of technology such as IoT and analytics to optimize energy consumption and reduce waste.
• Smart industry: using technology such as IoT and automation to improve efficiency and productivity in manufacturing and other industries.
• Smart cities: using technology such as IoT, sensors and data analytics to improve urban infrastructure and services.

• Mobile payments: The widespread adoption of mobile payments, such as Apple Pay and Google Wallet, in the 2010s greatly increased the convenience and accessibility of digital payments.
• Online lending platforms: The emergence of online lending platforms, such as Lending Club and Prosper, in the 2010s greatly expanded the reach and accessibility of personal and small business loans.
• Digital currencies and blockchain technology: The development of digital currencies such as Bitcoin and blockchain technology in the 2010s greatly increased the security and speed of digital transactions.
• Roboadvisers: The emergence of robo-advisers, which use algorithms to provide investment advice, in the 2010s greatly expanded the accessibility of investment management services.
• Digital banking: The popularization of digital banking in the form of online and mobile banking apps greatly increased the convenience and accessibility of banking services in the 2010s.
• Open banking: The concept of open banking, which allows customers to share their financial data with third-party providers, became more prevalent in the 2010s.
• Biometrics: The use of biometrics such as fingerprint and facial recognition for identity verification in fintech became more prevalent in the 2010s.
• Insurtech: The emergence of insurtech, which uses technology to make insurance more efficient and accessible, became more prevalent in the 2010s.
• Digital identities: The development of digital identities, which allow individuals to prove their identity online, became more prevalent in the 2010s.
• Artificial intelligence and machine learning: The use of artificial intelligence and machine learning in fintech, such as for fraud detection and risk management, became more prevalent in the 2010s.

• Legal research platforms, which use artificial intelligence to quickly search and analyze large volumes of legal information.
• Contract review software, which uses natural language processing to automatically review and identify key terms and clauses in legal documents.
• E-discovery tools, which automate the process of identifying, collecting, and reviewing electronic documents in the context of legal proceedings.
• Online dispute resolution (ODR), which uses technology to facilitate the resolution of disputes outside of traditional court proceedings.
• Legal chatbots, which use natural language processing to provide automated legal advice to individuals and businesses.

• CRISPR-Cas9: The development of CRISPR-Cas9 gene editing technology in the 2010s greatly expanded the capabilities of genetic engineering and opened up new opportunities for treating genetic diseases.
• Personalized medicine: The use of biotechnology in the form of personalized medicine, which tailors medical treatment to an individual's genetic makeup, became more prevalent in the 2010s.
• Synthetic biology: The development of synthetic biology in the 2010s allowed for the design and construction of new biological parts and systems.
• Bioprinting: The use of bioprinting technology to print living tissue and organs became an active area of research in the 2010s.
• Biodegradable plastics: The development of biodegradable plastics made from natural materials such as corn starch and sugarcane became more prevalent in the 2010s.
• Immunotherapy: The use of immunotherapy, which harnesses the body's immune system to fight cancer and other diseases, became an active area of research in the 2010s.
• Microbial biotechnology: The use of microbial biotechnology in the form of probiotics and fermented foods became more prevalent in the 2010s.
• Biopesticides: The development of biopesticides, which use natural materials such as bacteria and viruses to control pests, became more prevalent in the 2010s.
• Biorefineries: The use of biorefineries to convert biomass into a range of products including biofuels, chemicals, and materials became more prevalent in the 2010s.
• Biodegradable packaging: The development of biodegradable packaging made from natural materials such as plant fibers and starch became more prevalent in the 2010s
• In the last decade, DNA has been increasingly investigated as an alternative medium for cold data storage, presenting several advantages over standard hard drives such as a higher density, longer lifespan and lower energy consumption. However, such coding methods are limited by biochemical constraints that elevate the probability of errors being added to the coded nucleotides during synthesis, storage, and sequencing. Although such errors can be limited by carefully designing the produced strands, it is unfeasible to avoid them completely.

• Imaging technology such as MRI, CT and PET scans, which allow for non-invasive examination of internal organs and body structures.
• Robotic surgery, which uses robots to assist with or perform complex surgical procedures.
• Telemedicine, which allows for remote consultations and monitoring of patients using technology such as video conferencing and wearable devices.
• Biomedical devices such as pacemakers, artificial joints and insulin pumps, which improve the quality of life of patients with chronic conditions.
• Digital health tools, such as electronic health records (EHRs) and personal health apps, which allow for improved data sharing, monitoring and analysis of patients' health.

• Pangu-Weather by Huawei: Pangu-Weather is an AI model developed by Huawei that can predict weekly weather patterns worldwide much faster than traditional forecasting methods, while maintaining comparable accuracy. It employs a deep neural network trained on 39 years of reanalysis data, which combines historical weather observations with modern models. Unlike conventional methods that analyze weather variables one by one, Pangu-Weather can analyze all variables simultaneously within seconds. The AI system has demonstrated the ability to accurately predict extreme weather events, such as the path of a tropical cyclone, even without specific training data on such events. Pangu-Weather's success indicates that AI models can generalize weather patterns and handle previously unseen scenarios effectively.
• FourcastNet by Nvidia: Nvidia's FourcastNet is another AI-based weather forecasting model that has shown comparable accuracy to conventional methods while significantly reducing the time needed to produce forecasts. As one of the recent contributions to the AI-powered weather forecasting domain, FourcastNet aims to challenge traditional approaches with its efficient and rapid prediction capabilities. This technology, along with others like Pangu-Weather and GraphCast, is encouraging meteorologists to rethink how machine learning can complement existing weather prediction methods to achieve more reliable forecasts[2].
• GraphCast by Google DeepMind: GraphCast is another AI model developed by Google DeepMind that contributes to the advancement of weather forecasting. Though specific details about GraphCast's performance are not provided in the reference, it is mentioned alongside Pangu-Weather and FourcastNet as one of the AI systems that is pushing the boundaries of weather forecasting using machine learning techniques.

• Electric vehicles: The development and popularization of electric vehicles in the 2010s greatly increased the availability and accessibility of emissions-free transportation options.
• Autonomous vehicles: The development of autonomous vehicle technology in the 2010s greatly increased the capabilities and safety of vehicles, with companies such as Waymo, Tesla, Uber, and GM Cruise working on it.
• Connected cars: The integration of internet connectivity and advanced communication technology into vehicles in the 2010s greatly increased the capabilities and functionality of cars.
• Advanced driver assistance systems (ADAS): The widespread adoption of advanced driver assistance systems (ADAS) such as lane departure warning, adaptive cruise control, and automatic emergency braking in the 2010s greatly increased the safety of vehicles.
• Lightweight materials: The use of lightweight materials such as aluminum, carbon fiber, and composites in the 2010s greatly improved the fuel efficiency and performance of vehicles.
• Hybrid and plug-in hybrid vehicles: The development and popularization of hybrid and plug-in hybrid vehicles in the 2010s greatly increased the fuel efficiency and reduced emissions of vehicles.
• Fuel cell vehicles: The development of fuel cell vehicles, which use hydrogen fuel cells to generate electricity, became an active area of research in the 2010s.
• 3D printing: The use of 3D printing in the automotive industry in the 2010s greatly increased the efficiency and flexibility of manufacturing processes.

Global trends

• Remote Healthcare & Wearables (B+): Remote healthcare with advanced wearables will enable patients to obtain remote medical assistance and physicians to perform procedures, consult with remote experts, and have access to vital health information
• Augmented Reality (B): Seamless integration between the real world and cyberspace will increasingly materialize.
• Software for the Edge2Cloud Continuum (B): This includes new software for the development and deployment of next-generation computing components, systems, and platforms that enable a transition to a compute continuum with strong capacities at the edge and far edge in an energy-efficient and trustworthy manner
• Open Hardware (B): From open system (OCP) to ISAs (RISC-V) and interconnects (CXL, UCIe) the open-source movement has expanded into hardware.
• AI-Assisted DevOps (B): The traditional DevOps approach will be improved to address the increasing complexity of software systems.
• 3D Printing in Personalized Healthcare (B-): 3D printing in healthcare will evolve towards customized additive manufacturing for individuals.
• Generative AI (B-): In the next few years generative AI will be used more and more, increasing effectiveness and enabling new services. It is also bound to raise ethical and societal issues. Expect strong impact on business (short term), education (long term), and society (medium to long term). Text-to-Image: Illustration and design demand exponentially increasing due to increase of digital channels and social-media outlets, TV and visual ads globally. Brands and entities (across industry) are struggling to have the perfect illustrations and designs on their digital channels. The time to illustrate and design a storyline within an app, a website or an ad is high, impacting time to market. Business requirements and briefs is challenging when done manually.
• IT for Sustainability (B-): Technology will evolve from sustainable IT to novel uses of IT for sustainability, clean energy, and a green economy.
• Autonomous Driving (B/C): Self-driving vehicles in controlled environments are starting to gain adoption at scale, backed by strong business cases.
• Digital Distributed Manufacturing (B/C): Digital Distributed Manufacturing will reduce energy and environmental footprints and increase the resilience of supply chains.
• Trusted Computing (B/C): There will be increased public awareness and attention to trusted/assured computation across all industry sectors. Governments will increase focus on legislative actions to ensure that public facing systems can be trusted.
• Huge Graph Neural Networks (B/C): Applications that use huge models, such as chatGPT, have demonstrated a real impact on a substantial set of problems. Graph Neural Networks can represent complex, “real-world” structures. We predict that huge GNN models will widely be used in machine learning.
• Adaptive, Generative Pharmaceuticals (C+): Advances in nanotechnology and AI could shorten the time to vaccine development and broaden their efficacy.
• Autonomous Robots & Brain-Machine I/F (C+): Pervasive uptake of robotic platforms will take place, including as extensions of the human body.
• Artificial General Intelligence (AGI) (C+): Advances in AI will lead to AGI systems that can understand or learn any intellectual task that a human being can perform.
• Global Digitalization of Monetary Transactions (C+): Digital transformation of monetary transactions will open new disruptive opportunities in global markets.
• Space ITC (C): As more companies send technology to space, the barriers to entry are decreasing rapidly.
• Sustainable Space Manufacturing (C/D): Space manufacturing and recycling technologies and services will improve the sustainability, resilience, and cost of the space ecosystem.
• Disinformation Detection/Correction (C/D): Improving the reliability of the information in public health, politics, and science will improve public information required for sound decisions from personal to societal levels.

• Machine learning: The development and popularization of machine learning techniques such as deep learning and neural networks in the 2010s greatly expanded the capabilities of artificial intelligence. Machine learning innovation refers to the development and application of new techniques and algorithms in the field of machine learning. This can include advancements in areas such as deep learning, reinforcement learning, natural language processing, and computer vision. These innovations have led to significant improvements in the ability of machines to perform tasks such as image and speech recognition, language translation, and decision-making. Additionally, the increased availability of large amounts of data and computational power have also been key drivers of machine learning innovation.
• Natural Language Processing (NLP): The development of natural language processing (NLP) technologies in the 2010s greatly improved the ability of machines to understand and generate human language.
• Computer Vision: The development of computer vision technologies in the 2010s greatly improved the ability of machines to understand and interpret visual information.
• Robotics: The development of advanced robotics and autonomous systems in the 2010s greatly expanded the capabilities and applications of artificial intelligence.
• Virtual assistants: The emergence of virtual assistants such as Siri, Alexa, and Google Assistant in the 2010s greatly increased the accessibility and convenience of artificial intelligence.
• Recommendation Systems: The popularization of recommendation systems in the 2010s greatly improved the ability of machines to personalize and optimize experiences for users.
• Chatbots: The development and deployment of chatbots for customer service and other applications in the 2010s greatly increased the accessibility and convenience of artificial intelligence.
• Generative Adversarial Networks (GANs): The development of Generative Adversarial Networks (GANs) in the 2010s greatly improved the ability of machines to generate new data, such as images and text.
• Reinforcement Learning: The development of reinforcement learning in the 2010s greatly improved the ability of machines to learn and make decisions in complex, dynamic environments.
• Explainable AI (XAI): The development of explainable AI (XAI) in the 2010s greatly improved the ability of machines to provide understandable and transparent explanations for their decisions.

• Internet of Things (IoT): 5G networks are expected to play a major role in enabling the IoT by providing the necessary connectivity and low-latency required for IoT devices.
• Industry 4.0: 5G networks will also enable Industry 4.0, by providing fast, low-latency, and reliable connections for industrial automation and robotics, and allowing for real-time monitoring and control of industrial processes.
• Autonomous vehicles: With their low latency and high-bandwidth capabilities, 5G networks will enable the development of autonomous vehicles and smart transportation systems.
• Augmented Reality and Virtual Reality (AR/VR): 5G networks will provide the necessary connectivity and low-latency required for immersive AR and VR experiences, making it possible to use them in a variety of applications such as remote collaboration, education, and entertainment.
• Edge computing: 5G networks will enable the deployment of edge computing, which will bring computation and data storage closer to the devices that need it, reducing latency and providing faster response times.
• Network slicing: 5G networks will allow for the creation of multiple "slices" of the network, each with different characteristics and capabilities, to support the specific needs of different use cases and applications.
• Security: 5G networks will provide new security capabilities, such as network slicing, that will allow for more secure and reliable connections between devices and systems.

• Metaverse virtual world contains digital representations of individuals and objects in the physical world, allowing interaction and engagement in various contexts.
• Extended Reality (XR) is an umbrella term that refers to computer-generated environments that merge the physical and virtual worlds or create an entirely virtual experience for users. XR encompasses three technologies: Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). XR takes the human-to-PC screen interface and modifies it by either immersing users in a virtual environment (VR) or adding elements to the user's surroundings (AR) or both (MR). 
• Digital twins bridge the gap between the physical and digital realms by creating rich digital models of physical assets, objects, or environments of anything that is important. 

• Virtual and Augmented Reality: The use of VR and AR technologies is crucial for creating fully immersive and believable virtual worlds.
• Blockchain: Blockchain technology is being explored as a way to create secure and decentralized virtual worlds, where users have full control over their data and virtual assets.
• 5G and Edge computing: High-speed internet and edge computing are necessary to support real-time interactions and large numbers of users within the metaverse.
• Artificial Intelligence: AI is being used to create more realistic and responsive virtual characters and environments.
• Game engines: Game engines such as Unity and Unreal Engine are being used to create metaverse experiences.
• Interoperability: Interoperability between different virtual worlds, platforms and devices is important for creating a seamless metaverse experience.

• Identification of Use Cases, either existing or possible (automotive, defence, education, enterprise, eSports, events, finance, food, gaming, healthcare, hospitality, professional training,real estate, remote work, retail, social media, travel, virtual spaces, workflows,.
• Identification of External Services that a Metaverse can use (content creation /Roblox, MagicaVoxel, OmniVerse/, marketplace /OpenSea/, crypto wallets, cryptocurrency exchanges, development services /LandVault, InfiniteReality/, DigiSomni/, platforms /Metaverse.network, Vircadia/).
• Identification of Functionalities from use cases and external services (instance, environment, content representation, perception, user, interaction, information search, economy support)
• Analysis of the state and characteristics of the Technologies required to support the functionalities (sensory information, data processing, user devices, network, energy).
• Analysis of the issues regarding the management of the Metaverse Ecosystem and legal issues (governance /regulation, property, trademark, authorship, contract, tort, defamation, privacy, taxation, mental health/, stakeholders /manager, operator, user/)

• Real-time data integration: Digital twins are built with real-time data from sensors and other sources, providing a complete and accurate representation of the physical asset.
• Simulation and modeling: Digital twins can be used to simulate the behavior and performance of physical assets, allowing for analysis and testing of different scenarios.
• Predictive maintenance: Digital twins can analyze data to detect potential issues and predict when maintenance is required, reducing downtime and improving efficiency.
• Remote monitoring: Digital twins allow for remote monitoring and control of physical assets, reducing the need for on-site visits and increasing accessibility.
• Collaboration and decision making: Digital twins enable stakeholders to collaborate and make informed decisions based on a shared understanding of the physical asset's state and performance.
• Contextual information: Digital twins can include a range of contextual information, such as design specifications, historical data, and environmental conditions.

• statistical language models (SLM).
• neural language models (NLM)
• pre-trained language models (PLM)
• large language models (LLM)

• Theory and principle. To understand the underlying working mechanism of LLMs, one of the greatest mysteries is how information is distributed, organized, and utilized through the very large, deep neural network.
• Model architecture. Due to the scalability and effectiveness, Transformer, consisting of stacked multi-head self-attention layers, has become the de facto architecture for building LLMs.
• Model training. In practice, it is very difficult to pretrain capable LLMs, due to the huge computation consumption and the sensitivity to data quality and training tricks.
• Model utilization. Since fine-tuning is very costly in real applications, prompting has become the prominent approach to using LLMs.
• Safety and alignment. Despite their capacities, LLMs pose similar safety challenges as small language models.
• Application and ecosystem. As LLMs have shown a strong capacity in solving various tasks, they can be applied in a broad range of real-world applications (i.e., following taskspecific natural language instructions). As a remarkable progress, ChatGPT has potentially changed the way how humans access information.

• Engine parameter specifies the AI model employed to generate predictions.
• Max tokens parameter specifies the maximum number of tokens that can be generated by the model. A token can be seen as a piece of word. As a rule of thumb, 1 token is around 4 characters.
• Temperature parameter close to 1 would mean that the logits are passed through the softmax function without modification. If the temperature is close to zero, the highest probable tokens will become very likely compared to the other tokens, i.e. the model becomes more deterministic and will always output the same set of tokens after a given sequence of words.
• Top p  parameter specifies a sampling threshold during inference time. Top p sampling (sometimes called nucleus sampling) is a technique used to sample possible outcomes of the model.
• Frequency penalty parameter controls the model’s tendency to repeat predictions.  It reduces the probability of words that have already been generated. The penalty depends on how many times a word has already occurred in the prediction.
• Presence penalty parameter encourages the model to make novel predictions. The presence penalty lowers the probability of a word if it already appeared in the predicted text. Unlike the frequency penalty, the presence penalty does not depend on the frequency at which words appear in past predictions.

• Bias and discrimination: GPT-3 and other language models are trained on data that reflects societal biases and prejudices. This can result in generated text that perpetuates harmful stereotypes and reinforces existing inequalities.
• Misinformation and propaganda: The ability of GPT-3 to generate coherent and context-aware text makes it susceptible to use for spreading misinformation and propaganda.
• Responsibility and accountability: As GPT-3 and other language models become more widely used, questions arise about who is responsible for their actions and who should be held accountable for any harm they cause.
• Privacy and security: GPT-3 and other language models process large amounts of sensitive data, raising questions about data privacy and security.
• Job displacement: The increasing capabilities of GPT-3 and other language models raise concerns about their potential impact on employment and the displacement of workers in language-related jobs.

• ChatGPT capabilities include answering questions, generating text, and translating text. GPT-3 demonstrate remarkable text generation ability. Given words, phrases or simple sentences as prefix, they can continue to generate text that are syntactically correct and semantically smooth conditioning on the given text. ChatGPT was fine-tuned on top of GPT-3.5 using supervised learning as well as reinforcement learning. In addition, OpenAI continues to gather data from users that could be used to further train and fine-tune ChatGPT. Unlike most chatbots, ChatGPT remembers previous prompts given to it in the same conversation. 
• OpenAI plug-ins are tools designed specifically for language models that help ChatGPT access up-to-date information, run computations, or use third-party services. The plug-ins enables ChatGPT Plus to access live web data versus information the large language model was already trained on. ChatGPT uses the Bing Application Programming Interface (API) to do searches and a text-based web browser to explore and interact with page results. It may combine data from different sources into a coherent response. 
• DALL-E is deep learning model based on Markov chains trained using variational inference. The goal of latent diffusion models (LDM) is to learn the latent structure of a dataset by modeling the way in which data points diffuse through the latent space. Feature space or embedding space is an embedding of a set of items within a manifold in which items resembling each other are positioned closer to one another in the latent space. DALL-E's model is a multimodal implementation of GPT-3 trained on text-image pairs from the Internet to generate digital images from natural language descriptions (prompts) which combines concepts, attributes, and styles. The model can generate imagery in multiple styles, including photorealistic imagery, paintings, and emoji. It can manipulate and rearrange objects in its images, and can correctly place design elements in novel compositions without explicit instruction.
• Stable Diffusion is a deep learning text-to-image model primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt. 

1. Write clear instructions.  These models can’t read your mind. If outputs are too long, ask for brief replies. If outputs are too simple, ask for expert-level writing. If you dislike the format, demonstrate the format you’d like to see. The less the model has to guess at what you want, the more likely you’ll get it.

2. Provide reference text.  Language models can confidently invent fake answers, especially when asked about esoteric topics or for citations and URLs. In the same way that a sheet of notes can help a student do better on a test, providing reference text to these models can help in answering with fewer fabrications.

3. Split complex tasks into simpler subtasks.  Just as it is good practice in software engineering to decompose a complex system into a set of modular components, the same is true of tasks submitted to a language model. Complex tasks tend to have higher error rates than simpler tasks. Furthermore, complex tasks can often be re-defined as a workflow of simpler tasks in which the outputs of earlier tasks are used to construct the inputs to later tasks.

4. Give the model time to "think".  If asked to multiply 17 by 28, you might not know it instantly, but can still work it out with time. Similarly, models make more reasoning errors when trying to answer right away, rather than taking time to work out an answer. Asking for a "chain of thought" before an answer can help the model reason its way toward correct answers more reliably.

5. Use external tools.  Compensate for the weaknesses of the model by feeding it the outputs of other tools. For example, a text retrieval system (sometimes called RAG or retrieval augmented generation) can tell the model about relevant documents. A code execution engine like OpenAI's Code Interpreter can help the model do math and run code. If a task can be done more reliably or efficiently by a tool rather than by a language model, offload it to get the best of both.

6. Test changes systematically.  Improving performance is easier if you can measure it. In some cases a modification to a prompt will achieve better performance on a few isolated examples but lead to worse overall performance on a more representative set of examples. Therefore to be sure that a change is net positive to performance it may be necessary to define a comprehensive test suite (also known an as an "eval").

• ChatGPT capabilities include answering questions, generating text, and translating text. GPT-3 demonstrate remarkable text generation ability. Given words, phrases or simple sentences as prefix, they can continue to generate text that are syntactically correct and semantically smooth conditioning on the given text. ChatGPT was fine-tuned on top of GPT-3.5 using supervised learning as well as reinforcement learning. In addition, OpenAI continues to gather data from users that could be used to further train and fine-tune ChatGPT. Unlike most chatbots, ChatGPT remembers previous prompts given to it in the same conversation. 
• OpenAI plug-ins are tools designed specifically for language models that help ChatGPT access up-to-date information, run computations, or use third-party services. The plug-ins enables ChatGPT Plus to access live web data versus information the large language model was already trained on. ChatGPT uses the Bing Application Programming Interface (API) to do searches and a text-based web browser to explore and interact with page results. It may combine data from different sources into a coherent response. 
• DALL-E is deep learning model based on Markov chains trained using variational inference. The goal of latent diffusion models (LDM) is to learn the latent structure of a dataset by modeling the way in which data points diffuse through the latent space. Feature space or embedding space is an embedding of a set of items within a manifold in which items resembling each other are positioned closer to one another in the latent space. DALL-E's model is a multimodal implementation of GPT-3 trained on text-image pairs from the Internet to generate digital images from natural language descriptions (prompts) which combines concepts, attributes, and styles. The model can generate imagery in multiple styles, including photorealistic imagery, paintings, and emoji. It can manipulate and rearrange objects in its images, and can correctly place design elements in novel compositions without explicit instruction.
• Stable Diffusion is a deep learning text-to-image model primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt. 

• First, OpenAI charges for access to its platform and services. Companies pay for access to its proprietary technology and the ability to integrate their own data and algorithms into OpenAI's platform. 
• Second, OpenAI makes money by licensing its technology to other organizations and companies. Over 300 applications are delivering GPT-3–powered search, conversation, text completion, and other advanced AI features through our API. 

• ChatGPT summarise and generate written content quickly and coherently for research, marketing and customer service scenarios. Capabilities include answering questions, generating text, and translating text. ChatGPT was fine-tuned on top of GPT-3.5 using supervised learning as well as reinforcement learning. In addition, OpenAI continues to gather data from users that could be used to further train and fine-tune ChatGPT. Unlike most chatbots, ChatGPT remembers previous prompts given to it in the same conversation.
• GPT-4 LLM model is a large multimodal model that can accept both image and text inputs and generate text outputs. GPT-4 and GPT-3 differ significantly because GPT-4 includes more data than GPT-3. Compared to GPT-3's 17 gigabytes of data, GPT-4, the most recent iteration of OpenAI, has 45 gigabytes of training data. As a result, GPT-4 can deliver significantly more accurate results than GPT-3. The new GPT-4 model has a long-form mode that offers a context window of 32,000 tokens (52 pages of text). That's more than an order of magnitude larger than the previous GPT-3 API that offered only 2,049 tokens (three pages of text).
• ClaudeAI context window was about 9,000 tokens, but the company has now increased it to 100,000 tokens (75,000 words). An average human can read 100,000 tokens in about five hours' time. However, this is only the time taken to read the tokens, and more time might be needed if one has to remember and analyze this information. OpenAI's GPT-4 LLM has a context window of 4,096 tokens (~3,000 words) when used with ChatGPT, but this can increase to 32,768 tokens while using GPT-4 API. 
• Codex LLM model generate code and documentation to help developers build apps faster. GitHub Copilot is a cloud-based artificial intelligence tool developed by GitHub on top of OpenAI Codex, a system that translates natural language to code make recommendations in different programming languages based on the appropriate prompts , to assist users of integrated development environments (IDEs) by autocompleting code.
• Embeddings LLM model embeddings measure the relatedness of text strings commonly used for: search (where results are ranked by relevance to a query string), clustering (where text strings are grouped by similarity), recommendations (where items with related text strings are recommended), anomaly detection (where outliers with little relatedness are identified), diversity measurement (where similarity distributions are analyzed), classification (where text strings are classified by their most similar label).
• MLLM model with multimodal perception will be better equipped to acquire commonsense knowledge beyond the information they glean from text alone; and that this perception enrichment will facilitate LLM applications in new domains such as robotics and document intelligence. Multimodal perception also has the benefit of unifying multiple APIs to form a single general graphical user interface. Microsoft Introduces Kosmos-1 that can perceive general modalities, follow instructions, and perform In-iontext learning.  KOSMOS-1 follows the MetaLM training process, where a transformer-based LLM acts as a general-purpose interface and is augmented with various perception modules. Consistent with the MetaLM philosophy, the team treats language models as a universal task layer, enabling KOSMOS-1 to unify various task predictions as texts and capably handle natural-language instructions and action sequences. Given a previous context, KOSMOS-1 learns to generate texts in an autoregressive manner. All non-text input modalities are embedded and then fed into its backbone transformer-based causal language model, with the transformer decoder serving as a general-purpose interface for all modalities. By interacting with natural language and the other modalities, KOSMOS-1 naturally inherits the capabilities of in-context learning and instruction following; and can thus handle both language and perception-intensive tasks.
• MS 365 Copilot combines the power of large language models LLMs with your data in the Microsoft Graph and the Microsoft 365 apps to turn your words into the powerful productivity tool.

• GPT-4 is more than 10x the size of GPT-3 (175 B). We believe it has a total of ~1.8 trillion parameters across 120 layers. Mixture Of Experts (16 experts, each ~111B). Not a dense transformer like e.g. PaLM (or GPT-3). They use MQA instead of MHA (classic at this point).
• Each forward pass (generation of 1 token) only utilizes ~280B parameters and ~560 TFLOPs. This contrasts with the ~1.8 trillion parameters and ~3,700 TFLOP that would be required per forward pass of a purely dense model.

• To parallelize across all their A100s GPUs They utilized 8-way tensor parallelism. Beyond that, they are using 15-way pipeline parallelism. Also apparently they used DeepSpeed ZeRo Stage 1 or block-level FSDP.
• (You can check out my video on all of these strategies here: https://lnkd.in/e8ammDs5 3D parallelism is what you're looking for & ZeRo)

• They have a separate vision encoder from the text encoder, with cross-attention. The architecture is similar to Google DeepMind's Flamingo (I used to work on this project :) ). This adds more parameters on top of the 1.8T of GPT-4. It is fine-tuned with another ~2 trillion tokens, after the text-only pre-training.

• Trained on ~13T tokens (multiple epochs, not unique). Plus millions of rows of instruction fine-tuning data from ScaleAI & internally (I guess acquired through ChatGPT + their API before they changed the policy).
• 8k context length for the pre-training phase. The 32k seqlen version of GPT-4 is based on fine-tuning of the 8k after the pre-training. See e.g. MosaicML's blog on how to achieve this: https://lnkd.in/dagmDAZb)

• OpenAI’s training FLOPS for GPT-4 is ~2.15e25, on ~25,000 A100s for 90 to 100 days at about 32% to 36% MFU. Part of this extremely low utilization is due to an absurd number of failures requiring checkpoints that needed to be restarted from.
• If their cost in the cloud was about $1 per A100 hour, the training costs for this run alone would be about $63 million.
• Today, the pre-training could be done with ~8,192 H100 in ~55 days for $21.5 million at $2 per H100 hour

• OpenAI might be using speculative decoding on GPT-4's inference. See this paper: https://lnkd.in/d-C-QpwW
• The inference runs on a cluster of 128 GPUs. There are multiple of these clusters in multiple datacenters in different locations (it'll be hard for Elizier to nuke these xD). 8-way tensor parallelism and 16-way pipeline parallelism.

• Stable Diffusion is a deep learning text-to-image model primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt. 
• Models (checkpoint files) are pre-trained Stable Diffusion weights intended for generating general or a particular genre of images. What images a model can generate depends on the data used to train them. A model won't be able to generate a cat's image if there's never a cat in the training data. HuggingFace is the go-to platform for creators who build AI models for Stable Diffusion. There are currently over 270 models related to Stable Diffusion on the platform. You can use Stable Diffusion online through DreamStudio, without any kind of hardware requirements at all.
• Stable Diffusion is in a class of models called latent diffusion models. Latent diffusion models encode images and their text captions into vectors (basically a unique numerical representation for each image). During training time, the model adds random values (noise) to the vectors. And then you train a model to go from a slightly more noisy vector to a slightly less noisy vector. In other words, the model tries to reproduce the original numerical representation of every image in its training set, based on that image’s accompanying text caption. So a generated image wants to be similar to the images that most influenced it, by having a similar numerical representation.
• Stable Diffusion consists of 3 parts: the variational autoencoder (VAE), U-Net, and an optional text encoder. The VAE encoder compresses the image from pixel space to a smaller dimensional latent space, capturing a more fundamental semantic meaning of the image. Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion. The U-Net block, composed of a ResNet backbone, denoises the output from forward diffusion backwards to obtain a latent representation. Finally, the VAE decoder generates the final image by converting the representation back into pixel space. Stable Diffusion makes its source code available https://github.com/CompVis/stable-diffusion
• Stable Diffusion was trained on pairs of images and captions taken from LAION-5B, a publicly available dataset derived from Common Crawl data scraped from the web, where 5 billion image-text pairs were classified based on language and filtered into separate datasets by resolution, a predicted likelihood of containing a watermark, and predicted "aesthetic" score (e.g. subjective visual quality).
• To address the limitations of the model's initial training, end-users may opt to implement additional training to fine-tune generation outputs to match more specific use-cases. As visual styles and compositions are not subject to copyright, it is often interpreted that users of Stable Diffusion who generate images of artworks should not be considered to be infringing upon the copyright of visually similar works. However, individuals depicted in generated images may be protected by personality rights if their likeness is used,[48] and intellectual property such as recognizable brand logos still remain protected by copyright. 

Glossary

• Artificial intelligence (AI) is the ability of software to perform tasks that traditionally require human intelligence.
• Artificial neural networks (ANNs) are composed of interconnected layers of software-based calculators known as “neurons.” These networks can absorb vast amounts of input data and process that data through multiple layers that extract and learn the data’s features.
• Deep learning is a subset of machine learning that uses deep neural networks, which are layers of connected “neurons” whose connections have parameters or weights that can be trained. It is especially effective at learning from unstructured data such as images, text, and audio.
• Early and late scenarios are the extreme scenarios of our work-automation model. The “earliest” scenario flexes all parameters to the extremes of plausible assumptions, resulting in faster automation development and adoption, and the “latest” scenario flexes all parameters in the opposite direction. The reality is likely to fall somewhere between the two.
• Fine-tuning is the process of adapting a pretrained foundation model to perform better in a specific task. This entails a relatively short period of training on a labeled data set, which is much smaller than the data set the model was initially trained on. This additional training allows the model to learn and adapt to the nuances, terminology, and specific patterns found in the smaller data set.
• Foundation models (FM) are deep learning models trained on vast quantities of unstructured, unlabeled data that can be used for a wide range of tasks out of the box or adapted to specific tasks through fine-tuning. Examples of these models are GPT-4, PaLM, DALL·E 2, and Stable Diffusion.
• Generative AI is AI that is typically built using foundation models and has capabilities that earlier AI did not have, such as the ability to generate content. Foundation models can also be used for nongenerative purposes (for example, classifying user sentiment as negative or positive based on call transcripts) while offering significant improvement over earlier models. For simplicity, when we refer to generative AI in this article, we include all foundation model use cases.
• Graphics processing units (GPUs) are computer chips that were originally developed for producing computer graphics (such as for video games) and are also useful for deep learning applications. In contrast, traditional machine learning and other analyses usually run on central processing units (CPUs), normally referred to as a computer’s “processor.”  Unfortunately, many explanations and books about programming GPUs (NVIDIA, AMD) mix up techniques suitable for CPUs, such as spinlocks, mutexes, traditional locks, and thread sleeping/wake-up mechanisms, which are totally at odds with the way GPUs should be programmed. The reason for this, as many tech-bros around here know, lies in how CPU and GPU architectures differ. CPU architecture is built on time-slicing and MIMD instructions (Multiple Instruction, Multiple Data), while GPU groups its super-stellar type of instruction, SIMD (Single Instruction, Multiple Data ), into warps (NVIDIA-like) or wavefronts (AMD-like). From an easy abstract algebra point of view, the reason that GPU and CPU screw up each other lies in the impossible perfect mapping (morphisms) of CPU code into GPU code. Why? Take the easy explanation for category theory: CPU deals with objects (processing units) and arrows (operations) on different data streams in MIMD. Each processing unit (object) can execute different operations (arrows) on independent data streams. So, you have here, instead of a massive amount of threads (GPU), a massive amount of independent categories. From this mathematical perspective, a CPU is nothing more than a lot of independent operations, each with a diverse composition of morphisms, generating a vast number of independent execution paths (and saying 'a lot' is still an understatement).  In contrast, GPU with its SIMD involves only a single operation (arrow in category terms) applied simultaneously across multiple data elements (in category terms: objects). This can be seen as a single category where the same morphism is applied across various objects in parallel (for the math-savvy tech-bros, in reality, we have here an endomorphism operating in the same  category, kinda casting the same object into different types in traditional checked-type programming). Ok, now, after this brief excursion into the realm of category theory, we are ready to understand the core of the problem of why Parallelism used in CPU sucks when transferred to GPU: MIMD architectures support complex, branching operations where each processor can follow a different execution path. Translating this to SIMD would require aligning these diverse paths into a single, unified operation ( kinda a catamorphism from the general morphism of MIMD to the endomorphism within SIMD, not a trivial task, I tell you). The morphisms with MIMD allow processors to operate independently without waiting for others. SIMD requires data and operation synchronization across all processing elements, introducing complexity in aligning independent MIMD operations into a SIMD framework. And thus, my tech-bros, we lose the flexibility and independence in MIMD lost in SIMD.
• Large language models (LLMs) make up a class of foundation models that can process massive amounts of unstructured text and learn the relationships between words or portions of words, known as tokens. This enables LLMs to generate natural-language text, performing tasks such as summarization or knowledge extraction. GPT-4 (which underlies ChatGPT) and LaMDA (the model behind Bard) are examples of LLMs.
• Machine learning (ML) is a subset of AI in which a model gains capabilities after it is trained on, or shown, many example data points. Machine learning algorithms detect patterns and learn how to make predictions and recommendations by processing data and experiences, rather than by receiving explicit programming instruction. The algorithms also adapt and can become more effective in response to new data and experiences.
• Modality is a high-level data category such as numbers, text, images, video, and audio.
• Prompt engineering refers to the process of designing, refining, and optimizing input prompts to guide a generative AI model toward producing desired (that is, accurate) outputs.
• Self-attention, sometimes called intra-attention, is a mechanism that aims to mimic cognitive attention, relating different positions of a single sequence to compute a representation of the sequence.
• Structured data are tabular data (for example, organized in tables, databases, or spreadsheets) that can be used to train some machine learning models effectively.
• Transformers are a relatively new neural network architecture that relies on self-attention mechanisms to transform a sequence of inputs into a sequence of outputs while focusing its attention on important parts of the context around the inputs. Transformers do not rely on convolutions or recurrent neural networks.
• Technical automation potential refers to the share of the worktime that could be automated. We assessed the technical potential for automation across the global economy through an analysis of the component activities of each occupation. We used databases published by institutions including the World Bank and the US Bureau of Labor Statistics to break down about 850 occupations into approximately 2,100 activities, and we determined the performance capabilities needed for each activity based on how humans currently perform them.
• Use cases are targeted applications to a specific business challenge that produces one or more measurable outcomes. For example, in marketing, generative AI could be used to generate creative content such as personalized emails.
• Unstructured data lack a consistent format or structure (for example, text, images, and audio files) and typically require more advanced techniques to extract insights.

• Machine learning. Most AI systems today are powered by machine learning, where predictive models are trained on historical data and used to make future predictions. The rise of machine learning within AI started in the 1990s, representing a marked shift from the way AI systems were built previously: rather than specifying how to solve a task, a learning algorithm would induce it based on data — i.e., the how emerges from the dynamics of learning. Machine learning also represented a step towards homogenization: a wide range of applications could now be powered by a single generic learning algorithm such as logistic regression. Despite the ubiquity of machine learning within AI, semantically complex tasks in natural language processing (NLP) and computer vision such as question answering or object recognition, where the inputs are sentences or images, still required domain experts to perform “feature engineering” — that is, writing domain-specific logic to convert raw data into higher-level features (e.g., SIFT) in computer vision) that were more suitable for popular machine learning methods.
• Deep learning. Around 2010, a revival of deep neural networks under the moniker of deep learning [LeCun et al. 2015] started gaining traction in the field of machine learning. Deep learning was fueled by larger datasets, more computation (notably, the availability of GPUs), and greater audacity. Deep neural networks would be trained on the raw inputs (e.g., pixels), and higher-level features would emerge through training (a process dubbed “representation learning”). This led to massive performance gains on standard benchmarks, for example, in the seminal work of AlexNet [Krizhevsky et al. 2012] on the ImageNet dataset [Deng et al. 2009]. Deep learning also reflected a further shift towards homogenization: rather than having bespoke feature engineering pipelines for each application, the same deep neural network architecture could be used for many applications.
• Foundation models. Foundation models have taken shape most strongly in NLP, so we focus our story there for the moment. That said, much as deep learning was popularized in computer vision but exists beyond it, we understand foundation models as a general paradigm of AI, rather than specific to NLP in any way. By the end of 2018, the field of NLP was about to undergo another seismic change, marking the beginning of the era of foundation models. On a technical level, foundation models are enabled by transfer learning [Thrun 1998] and scale. The idea of transfer learning is to take the “knowledge” learned from one task (e.g., object recognition in images) and apply it to another task (e.g., activity recognition in videos). Within deep learning, pretraining is the dominant approach to transfer learning: a model is trained on a surrogate task (often just as a means to an end) and then adapted to the downstream task of interest via fine-tuning. Transfer learning is what makes foundation models possible, but scale is what makes but scale is what makes them powerful. Scale required three ingredients: 

1. improvements in computer hardware — e.g., GPU throughput and memory have increased 10× over the last four years; 
2. the development of the Transformer model architecture [Vaswani et al. 2017] that leverages the parallelism of the hardware to train much more expressive models than before; and 
3. the availability of much more training data. 

EU AI Act

• Parliament and Council of the European Union, have been proposing Amendments (Regulations EC No 300/2008, EU No 167/2013, EU No 168/2013, EU 2018/858, EU 2018/1139 and EU 2019/2144 and Directives 2014/90/EU) to the Act since its 2021 inception. The AI Act establishes various new bodies in Article 64 and the following articles. These bodies (AI Office, European Artificial Intelligence Board, Advisory Forum, Scientific Panel of Independent Experts) are tasked with implementing and enforcing the Act. The approach combines EU-level coordination with national implementation, involving both public authorities and private sector participation. The Act mandates that member states establish their own notifying bodies. Conformity assessments are conducted to verify whether AI systems comply with the standards set out in the AI Act. This assessment can be done in two ways: either through self-assessment, where the AI system provider checks conformity, or through third-party conformity assessment, where the notifying body conducts the assessment. Notifying bodies also have the authority to carry out audits to ensure proper conformity assessments.
• The Act regulates entry to the EU internal market using the new legislative framework. It contains essential requirements that all AI systems must meet to access the EU market. These essential requirements are passed on to European Standardisation Organisations, which develop technical standards that further detail these requirements. These standards are developed by CEN/CENELEC JTC 21. The process for developing harmonised standards is complex, bringing together elements from the ESOs typical standard-development processes and measures unique to harmonised standards intended to support EU legislation. In the meantime, European officials have suggested that companies worldwide could sign on to a voluntary AI code of conduct, drafting in July 2023 (publishing 14 November 2024) set of rules outlining the norms, rules, and responsibilities or proper practices of an individual party or an organization. It requires organizations to ensure transparency, data governance, and human oversight in their AI solutions. The EU AI Act's Code of Practice, as detailed in Article 56, provides guidance for providers of general-purpose AI models (GPAI) on how to comply with the Act's obligations, particularly regarding Articles 53 and 55. It outlines how to address systemic risks associated with these models and includes objectives, measures, and key performance indicators (KPIs). The Code of Practice is designed to be a practical tool for demonstrating compliance, especially during the interim period before harmonized standards are established.  It will likely consist of two parts: one for all GPAI providers and another specifically addressing those with systemic risk.  The Code of Practice will include: objectives and measures for compliance; Key performance indicators (KPIs) for assessing the effectiveness of the measures; reporting obligations to the AI Office on the implementation and results of the measures. While not legally binding, the Code of Practice provides a framework for demonstrating compliance until European harmonized standards are developed and adopted, which is expected by August 2027 or later. 
• In the US, in October 2022, the White House issued a nonbinding Blueprint for an AI Bill of Rights, which framed AI governance as a civil rights issue, stating that citizens should be protected from algorithmic discrimination, privacy intrusion, and other harms. Blueprint suggests a civil rights approach in hopes of creating flexible rules that could keep up with fast-changing technologies. In April 2023 that he was circulating the draft of a “high level framework” for AI regulations. 
• Chinese regulations have already been put in force, starting with rules for recommendation algorithms that went into effect in March 2022, requiring transparency from the service providers and a way for citizens to opt out. Next, in January 2023, the Chinese government issued early rules governing generative AI, and further draft rules were proposed in April 2023. China’s initial set of rules for generative AI required websites to label AI-generated content, banned the production of fake news, and required companies to register their algorithms and disclose information about training data and performance. The draft rules go even further, requiring that AI companies verify the veracity of all the data used to train their models. 

• Ban on emotion-recognition AI. The European Parliament’s draft text bans the use of AI that attempts to recognize people’s emotions in policing, schools, and workplaces. Makers of emotion-recognition software claim that AI is able to determine when a student is not understanding certain material, or when a driver of a car might be falling asleep. The use of AI to conduct facial detection and analysis has been criticized for inaccuracy and bias, but it has not been banned in the draft text from the other two institutions, suggesting there’s a political fight to come.
• Ban on real-time biometrics and predictive policing in public spaces. This will be a major legislative battle, because the various EU bodies will have to sort out whether, and how, the ban is enforced in law. Policing groups are not in favor of a ban on real-time biometric technologies, which they say are necessary for modern policing. Some countries, like France, are actually planning to increase their use of facial recognition.
• Ban on social scoring. Social scoring by public agencies, or the practice of using data about people's social behavior to make generalizations and profiles, would be outlawed. That said, the outlook on social scoring, commonly associated with China and other authoritarian governments, isn’t really as simple as it may seem. The practice of using social behavior data to evaluate people is common in doling out mortgages and setting insurance rates, as well as in hiring and advertising. 
• New restrictions for gen AI. This draft is the first to propose ways to regulate generative AI, and ban the use of any copyrighted material in the training set of large language models like OpenAI’s GPT-4. OpenAI has already come under the scrutiny of European lawmakers for concerns about data privacy and copyright. The draft bill also requires that AI generated content be labeled as such. That said, the European Parliament now has to sell its policy to the European Commission and individual countries, which are likely to face lobbying pressure from the tech industry.
• New restrictions on recommendation algorithms on social media. The new draft assigns recommender systems to a “high risk” category, which is an escalation from the other proposed bills. This means that if it passes, recommender systems on social media platforms will be subject to much more scrutiny about how they work, and tech companies could be more liable for the impact of user-generated content.

• rules on high-impact general-purpose AI models that can cause systemic risk in the future, as well as on high-risk AI systems
• a revised system of governance with some enforcement powers at EU level
• extension of the list of prohibitions but with the possibility to use remote biometric identification by law enforcement authorities in public spaces, subject to safeguards
• better protection of rights through the obligation for deployers of high-risk AI systems to conduct a fundamental rights impact assessment prior to putting an AI system into use.

• Cognitive behavioural manipulation of people or specific vulnerable groups: for example voice-activated toys that encourage dangerous behaviour in children
• Social scoring: classifying people based on behaviour, socio-economic status or personal characteristics
• Real-time and remote biometric identification systems, such as facial recognition

• Biometric identification and categorisation of natural persons
• Management and operation of critical infrastructure
• Education and vocational training
• Employment, worker management and access to self-employment
• Access to and enjoyment of essential private services and public services and benefits
• Law enforcement
• Migration, asylum and border control management
• Assistance in legal interpretation and application of the law.

• Disclosing that the content was generated by AI
• Designing the model to prevent it from generating illegal content
• Publishing summaries of copyrighted data used for training

13 March 2024.  Following EU Parliament's Plenary Session vote (523/46/49), AI Act will undergo final linguistic approval by lawyer-linguists in April, a step considered a formality, before being published in the Official Journal. It will then come into effect after 21 days, with prohibited systems provisions in force six months later, by the end of 2024. Other provisions will come in over the next 2-3 years.

The AI Act imposes significant penalties for non-compliance with the prohibited systems provisions, with fines up to €35 million or 7% of global turnover. What's the message? Make sure you aren't providing or using prohibited AI systems in the EU by the end of the year.

The following will be prohibited AI systems under the AI Act by year end:

• Manipulative and deceptive practices.  AI systems that use subliminal techniques to materially distort a person’s decision-making capacity, leading to significant harm, are banned. This includes systems that manipulate behaviour or decisions in a way that the individual would not have otherwise made.

• Exploitation of vulnerabilities.  The Act prohibits AI systems that target individuals or groups based on age, disability, or socio-economic status to distort behaviour in harmful ways.

• Biometric categorisation.  It outlaws AI systems that categorise individuals based on biometric data to infer sensitive information like race, political opinions, or sexual orientation. This prohibition does not cover any labelling or filtering of lawfully acquired biometric datasets, such as images. There are also exceptions for law enforcement.

• Social scoring.  AI systems designed to evaluate individuals or groups over time based on their social behaviour or predicted personal characteristics, leading to detrimental treatment, are banned.

• Real-time biometric identification.  The use of real-time remote biometric identification systems in publicly accessible spaces for law enforcement is heavily restricted, with allowances only under narrowly defined circumstances that require judicial or independent administrative approval.

• Risk assessment in criminal offences.  The Act forbids AI systems that assess the risk of individuals committing criminal offences based solely on profiling, except when supporting human assessment already based on factual evidence.

• Facial recognition databases.  AI systems that create or expand facial recognition databases through untargeted scraping of images are prohibited.

• Emotion inference in workplaces and educational institutions.  The use of AI to infer emotions in sensitive environments like workplaces and schools is banned, barring exceptions for medical or safety reasons.

The Act mandates prior authorisation for the use of 'real-time' remote biometric identification systems by law enforcement.

Maj 2024.  AI Act stupa na snagu (usvojen u EU 13.03.2024.) kao prvi pravni akt u svetu koji na obavezujući način reguliše oblast veštačke inteligencije, a sa ciljem da obezbedi napredak društva, i istovremeno poštovanje prava i osiguravanje bezbednosti pojedinca.

• Preambula

• Sadržaj: I Opšte odredbe; II Zabranjene AI prakse; III AI sistemi visokog rizika; IV Obaveze transparentnosti za provajdere AI sistema; VIIIa AI modeli opšte namene; V Mere u cilju podrške inovacijama; VI Upravljanje; VII Baza podataka EU za visokorizične sisteme AI iz aneksa III; VIII Post-tržišni monitoring, deljenje informacija i nadzor tržišta; IX Kodeks ponašanja; X Poverljivost i kazne; XI Delegiranje ovlašćenja i postupanje organa; XII Završne odredbe

• Aneksi

Nakon zauzetog stava o etičkim principima kojima se tehnologija mora voditi, AI Act pokreće fazu regulacije, pretvaranja etičkih principa u načela i stvaranje konkretnih pravila.

AI Act temelji digitalni prostor AI ekosistema na tri oslonca:

• inovativni razvoj

• transparentnost i odgovornost

• sistem procene nivoa rizika po društvo i prava pojedinca

Akt klasifikuje aplikacije VI na osnovu procenjenog nivoa rizika, a zatim po funkcionalnosti svrstava različite pojavne oblike četiri kategorije:

• Neprihvatljiv rizik – zabranjene su prakse VI koje se smatraju previše štetnima da bi bile dozvoljene, poput manipulacije ponašanjem ljudi na njihovu štetu (prevarne tehnike za iskrivljavanje ponašanja i ometanje informisanog donošenja odluka) ili poput sistema koji nepravedno kategoriziraju pojedince (social scoring), pa i softvera za prepoznavanje lica u stvarnom vremenu na javnim mestima.

• Visok rizik – deli se na dve podkategorije: sistemi sigurnosnih komponenti (medicinski uređaji) i sistemi za osetljivu upotrebu (biometrija, kritična infrastruktura, obrazovanje, zapošljavanje, osnovne usluge, sprovođenje zakona, migracije, pravosuđe). Kategorija obuhvata sisteme VI koji bi mogli značajno uticati na sigurnost ljudi ili na fundamentalna prava. Neophodna je stroga usklađenost i procena sigurnosti ovih sistema, nadgledanje, upravljanje podacima, čuvanje zapisa, sajber-bezbednost, sistem prijavljivanja ozbiljnijih incidenata...

• Ograničeni rizik – sistemi VI koji direktno komuniciraju s korisnicima, poput chatbota. – Neophodno je da su transparentne u pogledu toga da njima upravlja VI kako bi korisnici bili svesni da ne komuniciraju s ljudima.

• Minimalni rizik – većina aplikacija VI spada u ovu kategoriju, gde se sloboda inovacija održava uz minimalnu regulatornu intervenciju. To su sistemi koji predstavljaju zanemariv rizik za prava ili sigurnost pojedinaca.  Koriste se uz minimalan nadzor.

Neophodno je da organizacije i kompanije koje razvijaju i upotrebljavaju VI sisteme i softver, u potpunosti operacionalizuju odgovornu VI (Responsible AI) koja podrazumeva preduzimanje planiranih radnji dizajniranja, primene i upotrebe VI za stvaranje vrednosti i izgradnju poverenja štiteći korisnike, sugrađane, društvo, od potencijalnih rizika VI.

Fair Artificial Intelligence.  Neophodno je da kompanije koje razvijaju sisteme zasnovane na VI primenjuju FairAI (MIM5 Minimal Interoperability Mechanisms 5) kao i sve tehničke mere sajberbezbednosti NIS2 direktive o sajber-bezbednosti.

Council of Europe's AI Convention (2023–2024) (Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law) aims to protect human rights against the harms of AI. The AI Convention may become the first legally-binding international treaty on AI.

• Fairness and debiasing: Manage and monitor fairness. Scan your deployment for potential biases. 
• Model drift mitigation: Analyze your model and make recommendations based on the most logical outcome. Alert when models deviate from the intended outcomes.
• Model risk management: Quantify and mitigate model risk. Get alerted when a model performs inadequately. Understand what happened when deviations persist.
• Lifecycle automation: Build, run and manage models as part of integrated data and AI services. Unify the tools and processes on a platform to monitor models and share outcomes. Explain the dependencies of machine learning models.
• Multicloud-ready: Deploy AI projects across hybrid clouds including public clouds, private clouds and on premises. Promote trust and confidence with explainable AI.

• Accountability referes to the need to explain and justify one's decision and actions to its partners, users and others with whom the system interacts.
• Responsibility referes to the role of people themselves and to the capability of AI systems answer for one's decision and identify errors or unexpected results.
• Transparency referes to the need to describe, inspect and reproduce the mechanisms through governance of the data used created.
• Fairness refers to the equitable treatment of individuals, or groups of individuals, by an AI system. Bias occurs when an AI system has been designed, intentionally or not, in a way that may make the system's output unfair. Bias can be present both in the algorithm of the AI system and in the data used to train and test it. It can emerge as a result of cultural, social, or institutional expectations; because of technical limitations of its design; or when the system is used in unanticipated contexts or to make decisions about communities that are not considered in the initial design.
• In order to ensure that systems will uphold human values, design methods are needed that incorporate ethical principles and address societal concerns.  Moreover, implementing ethical actions in machines will help us better understand ethics overall. Responsible AI implies the need for mechanisms that enable AI systems themselves to reason about, and act according to, ethics and human values. This requires models and algorithms to represent and reason about, and take decisions based on, human values, and to justify their decisions according to their effect on those values.

• EAD Pillars of Conceptual Framework fall broadly into three areas, reflecting anthropological, political, and technical aspects.: universal human values, political self-determination and data agency, and technical dependability. 
• EAD General Principles have emerged through the continuous work of dedicated, open communities in a multi-year, creative, consensus-building process. They articulate high-level principles that should apply to all types of autonomous and intelligent systems (A/IS). Created to guide behavior and inform standards and policy making, the General Principles define imperatives for the ethical design, development, deployment, adoption, and decommissioning of autonomous and intelligent systems. The Principles consider the role of A/IS creators, i.e., those who design and manufacture, of operators, i.e., those with expertise specific to use of A/IS, other users, and any other stakeholders or affected parties.

• Human rights.  A/IS shall be created and operated to respect, promote, and protect internationally recognized human rights.
• Well-being.  A/IS creators shall adopt increased human well-being as a primary success criterion for development.
• Data agency.  A/IS creators shall empower individuals with the ability to access and securely share their data, to maintain people’s capacity to have control over their identity.
• Effectiveness.  A/IS creators and operators shall provide evidence of the effectiveness and fitness for purpose of A/IS.
• Transparency.  The basis of a particular A/IS decision should always be discoverable.
• Accountability.  A/IS shall be created and operated to provide an unambiguous rationale for all decisions made.
• Awareness of misuse.  A/IS creators shall guard against all potential misuses and risks of A/IS in operation.
• Competence.  A/IS creators shall specify and operators shall adhere to the knowledge and skill required for safe and effective operation. 

• maintaining human autonomy
• implications of cultural migration in A/IS
• applying goal-directed behavior (Virtue Ethics) to A/IS
• eequirement for rule-based ethics in practical programming

•  Foundation Model Transparency Index (specifies 100 fine-grained indicators that comprehensively codify transparency for foundation models, spanning the upstream resources used to build a foundation model (e.g. data, labor, compute), details about the model itself (e.g. size, capabilities, risks), and the downstream use (e.g. distribution channels, usage policies, affected geographies)
• benefits of using human feedback to guide model behavior (Proximal Policy Optimization PPO) 
• control and refine generated content by influencing token probabilities (prevent the model from generating harmful, offensive, or biased content)
• prompt engineering (identify a limited set of high-priority keywords or topics; use regular expressions or natural language processing techniques to identify patterns in user inputs
• bias detection and correction after training (post-processing tools and techniques)

• During adaptation, a foundation model is converted into an adapted model (bottom row) in order to reflect updated information, desired behaviors, or deployment constraints.
• Evaluation gives context to machine learning models: it serves as a means for (1) tracking progress — how do we we measure the performance of models and how do we design improved models; (2) understanding — what behaviors do models exhibit and how do they perform on different slices of data; and (3) documentation — how do we efficiently summarize model behavior and communicate this to diverse stakeholders.
• Foundation models signal a paradigm shift where increasingly massive quantities of data are being “fed” to these models for improved adaptation performance with the overarching rule-of-thumb being "the more data the better" As previous sections have mentioned, this focus on data curation has raised concerns around the foundation model data lifecycle including (1) managing the data at such a large scale, (2) integrating data across new modalities, (3) reasoning over licensing and governance regulations — especially when considering the massing web-crawls used in foundation models training, and (4) understanding the data quality.
• As central components in critical data-driven decision-making systems, machine learning models must address a variety of security and privacy threats. These threats can be characterized using the traditional CIA triad of computer security. ML systems should protect the Confidentiality of user data against inference and reconstruction attacks. Moreover, the secrecy of trained models themselves can be at risk of model stealing attacks. The Integrity of ML systems can be compromised by adversarial examplesand data poisoning attacks. Finally, resource-depletion attacks can threaten the Availability of ML systems.

• The intrinsic bias present within foundation models is the byproduct of various training bias sources (left) which, alongside biases introduced during adaptation, determines the extrinsic harms (right) experienced by users in the context of specific downstream applications. We emphasize that the same foundation model is the shared foundation for many different applications; its biases propagate to these many applications as a result. Further, since the harms experienced by users are the result of specific adapted models, attributing these harms to the various processes and sources depicted in this diagram is both crucial and challenging.
• In this section,we consider misuse of foundation models—situations where people use foundation models as they are intended to be used (e.g., to generate language), but where their capabilities are intentionally leveraged to cause harm to populations or individuals. This definition positions misuse concerns between those of inequity (where models can cause harm without bad intentions; and security (where bad actors exploit unintentional abilities or vulnerabilities in models to cause harm.
• In this section, we describe how US law may influence, constrain, or foster the creation and use of foundation models.We note that the legal landscape surrounding algorithmic tools remains uncertain. We highlight issues pertaining to (1) model training, (2) liability for model predictions, and (3) protections for model outputs. Though understanding how the law affects foundation models is crucial, it is important to recognize that the law cannot be the only lens through which we evaluate the construction, maintenance, and use of foundation models. Ethical frameworks are necessary to understand where legally permissible applications of foundation models may still be ill-advised for the harms they inflict and are discussed in more depth in ethics and fairness. Studying the potential for misuse and possible security concerns is critical for preventing harmful outcomes ex ante, as opposed to the ex post treatment that legal mechanisms often provide.
• Foundation models have the potential to substantially improve overall living standards by increasing productivity and innovation. These models can be deployed to substitute for human labor, augment humans, or help in the discovery of new tasks and opportunities, which can lead to increased concentration of ownership and power, or more decentralization. On a broader level, the result can be either increased inequality due to potential centralization, or more broadly shared prosperity due to the easier adaptation of foundation models for a wide range of applications. The ultimate outcomes on all these dimensions are not dictated solely by technology or economics, but by the choices and actions of technologists, policymakers, managers, workers, and other members of society.

• The most straightforward idea is to just use the training data itself as the targets. 
• Then, people tried hiding some parts of the training data and using those missing parts as the targets. This is called masking, which is how LLMs are trained these days.
• Then, people tried pairing up text and images and using each other as targets. This is called constrative learning. This is the C in the famous CLIP model from OpenAI, which is the basis of all the multimodal foundational models.

• Scale: GPT-4 has ~1.8 trillion parameters across 120 layers, which is over 10 times larger than GPT-3.
• Mixture of experts (MoE): OpenAI utilizes 16 experts within their model, each with ~111B parameters for MLP.
• Dataset: GPT-4 is trained on ~13T tokens, including both text-based and code-based data, with some fine-tuning data from ScaleAI and internally.
• Dataset mixture: The training data included CommonCrawl & RefinedWeb, totalling 13T tokens. Speculation suggests additional sources like Twitter, Reddit, YouTube, and a large collection of textbooks.
• Training cost: The training costs for GPT-4 were around $63 million, taking into account the computational power required and the time of training.
• Inference cost: GPT-4 costs 3 times more than the 175B parameter Davinci, due to the larger clusters required and lower utilization rates.
• Inference architecture: The inference runs on a cluster of 128 GPUs, using 8-way tensor parallelism and 16-way pipeline parallelism.
• Vision multi-modal: GPT-4 includes a vision encoder for autonomous agents to read web pages and transcribe images and videos. This adds more parameters on top and it is fine-tuned with another ~2 trillion tokens.

• real-time vision (images and videos), 
• memory, 
• GPTs, 
• browse, and 
• advanced data analysis.

• Contrastive learning has been widely explored in VLM pretraining, which designs contrastive objectives for learning discriminative image-text features
• Generative VLM pre-training learns semantic knowledge by learning to generate images or texts via masked image modelling, masked language modelling, masked cross-modal modelling and image-to-text generation. Generative Adversarial Networks (GANs) have been a significant breakthrough in generative modeling, with the unique ability to generate realistic and diverse samples. GANs consist of two components, the generator and the discriminator, which engage in a continuous adversarial process. The generator produces synthetic images from random noise, while the discriminator aims to distinguish between these generated images and real images from the training dataset. Through backpropagation and optimization, the generator continually refines its outputs in response to the feedback from the discriminator and generates more realistic images. Diffusion-based methods model the image generation process as a series of diffusion steps, progressively refining the generated image. Diffusion models (DMs), also commonly known as diffusion probabilistic models, represent a class of generative models founded on Markov chains and trained through weighted variational inference. The primary objective of is to learn the impact of noise on the available information in the sample or the degree to which the diffusion process reduces the information available. The two-step process in consists of forward and reverse diffusion. In the forward diffusion process, Gaussian noise is successively introduced, representing the diffusion process until the data becomes total noise. The reverse diffusion process trains a neural network to learn the conditional distribution probabilities, allowing the model to reverse the noise and reconstruct the original data effectively.  Inspired by the success of autoregressive Transformers, autoregressive methods focus on sequentially predicting individual pixels or regions in an image, generating the output based on a learned probability distribution.
• Alignment objectives enforce VLMs to align paired images and texts by learning to predict whether the given text describes the given image correctly. It can be broadly categorized into global image-text matching and local region-word matching for VLM pre-training.

• AI-enabled IoT applications and use cases: Explore innovative applications and use cases where AI enhances IoT functionalities and capabilities, spanning smart healthcare, intelligent transportation, precision agriculture, industrial automation, environmental monitoring, and more.
• AI-driven data analytics and decision-making: Investigate AI techniques for processing, analyzing, and deriving actionable insights from IoT-generated data streams, enabling predictive maintenance, anomaly detection, personalized recommendations, and intelligent automation.
• Generative AI and Large Language Models (LLMs) for IoT: Study the applications of generative AI in IoT environments; explore how LLMs, such as generative pre-trained transformer (GPT) models, can be utilized to generate synthetic data, enhance natural language understanding, and support human-machine interaction in IoT systems.
• Edge AI and distributed intelligence: Discuss the integration of AI algorithms and models at the edge of IoT networks, enabling real-time inference, adaptive learning, and autonomous decision-making closer to data sources, and minimizing latency and bandwidth requirements.
• AI-empowered robotics and sensing for IoT: Explore the integration of AI with robotics and sensing technologies in IoT systems. Introduce advancements in sensor technologies and data fusion techniques that enable intelligent data collection, processing, and analysis in dynamic environments.
• Privacy, security, and trustworthiness: Address the privacy and security implications of AI-enabled IoT systems, including data privacy, confidentiality, integrity, and authenticity. Investigate trustworthiness in AI-enabled IoT systems, emphasizing the need for reliability, transparency, explainability, accountability, and fairness.
• Standardization and interoperability: Discuss the challenges and opportunities in standardizing AI-enabled IoT technologies to ensure interoperability, compatibility, and seamless integration across heterogeneous IoT ecosystems. Explore emerging standards, protocols, and frameworks for facilitating collaboration and interoperability among AI and IoT technologies.
• Demonstrations, Proof-of-Concepts, and deployments: Present innovative demonstrations and proof-of-concepts showcasing the integration of AI technologies with IoT systems. Share insights and best practices for deploying AI-powered IoT solutions in diverse environments, including smart cities, healthcare, agriculture, manufacturing, transportation, and energy.
• Regulation and policy: Explore the regulatory and policy landscape surrounding AI and IoT technologies, including data governance, consumer protection, liability, accountability, and ethical considerations. Discuss the role of regulatory bodies, industry consortia, and international organizations in shaping ethical, legal, and policy frameworks to ensure the responsible development, deployment, and use of AI-powered IoT systems.

• statistical language models
• neural language models
• pre-trained language models
• LLMs

• in-context learning, where LLMs learn a new task from a small set of examples presented in the prompt at inference time 
• instruction following, where LLMs, after instruction tuning, can follow the instructions for new types of tasks without using explicit examples
• multi-step reasoning, where LLMs can solve a complex task by breaking down that task into intermediate reasoning steps as demonstrated in the chain-of-thought prompt

• pre-training MLLMs intends to support different modalities using unified end-to-end models
• derived from instruction tuning  for NLP tasks, researchers are fine-tune pre-trained LLMs using multimodal instructions
• prompting technique provides certain context, examples, or instructions to the model, fulfilling specialized tasks without changing the model parameters
• visual reasoning application: recent visual reasoning systems tend to apply LLMs for better visual information analysis and visual-language integration

• Advancements in Multimodal AI Systems
• Enhanced Learning Efficiency
• Democratizing AI
• AI-driven personalization (Extending to Lifestyle and Daily Interactions)
• The Convergence of AI and AR: Reshaping Interaction with the World
• Ethical AI and Governance (Building Robust Ethical Frameworks, Governance Models for AI, Preventing Misuse of AI, Regulatory Landscape, Adaptive Legal Frameworks, Standardization of Practices, Privacy Regulations,Intellectual Property Rights, Combating Misuse, Global Coordination)
• Future Research Directions (Enhancing Creativity and Diversity of AI Outputs, Improving Multimodal Capabilities, Advancing Personalization Techniques, Ethical AI Development, Interdisciplinary Collaboration, Quantum AI Integration, AI in Climate Change and Sustainability, AI in Environmental Monitoring and Conservation, AI for Social Good, Augmented Reality and Virtual Reality Integration, Generative AI in Healthcare Diagnostics and Treatment, Cognitive and Emotional Intelligence in AI, AI for Creative Industries, Ethical AI Deployment in Diverse Cultural Contexts)

• Chatbot: able to have a conversation and remember previous interactions using the LLM language model.
• Conversational RAG (Retrieval-Augmented Generation): enable a chatbot experience over an external source of data. Chatbots combine retrieval-based and generative models to provide advanced customer service solutions. They use external knowledge sources to enhance accuracy and contextuality.
• Agents: build a chatbot that can take actions.

• Development: Build your applications using LangChain's open-source building blocks, components, and third-party integrations. Use LangGraph to build stateful agents with first-class streaming and human-in-the-loop support.
• Productionization: Use LangSmith to inspect, monitor and evaluate your chains, so that you can continuously optimize and deploy with confidence.
• Deployment: Turn your LangGraph applications into production-ready APIs and Assistants with LangGraph Cloud.

• Availability of open data.  The availability of data is crucial in training, modeling, and applications of AI irrespective of the domain. Data availability for analysis may be private or public. Metadata for private data may be published (e.g. data types and structures). However, public data, open for analysis by anyone, requires cleaning and anonymization to remove confidential or personal information.
• Access to research.  Balancing the two main aspects of research, namely advancements in domain-specific research and advancements in AI research requires collaboration between domain experts and AI researchers. Providing a platform for collaboration with experts from different realms of knowledge, facilitating cooperation, and exchange of information among them is key to creating a sustainable ecosystem for AI-based innovation.
• Deployment capability along with Infrastructure.  Two major categories of infrastructure are studied – physical infrastructure and communication infrastructure. Physical infrastructure elements play an important role in the integration and application of AI in data collection, aggregation - at the edge or core, training – federated or centralized, and in the application of Artificial Intelligence and Machine Learning (AI/ML) inference using actuators. In addition, there is backend infrastructure, such as compute availability, storage availability, fiber/wireless availability for the last mile, and high-speed wide area network capabilities, which would democratize AI/ML solutions and create scalability for innovations.
• Stakeholders buy-in enabled by Standards – trust, interoperability, security.  Interoperability and compliance with standards build trust. Secure standards lead to AI Readiness, as global participation and consensus decide whether pre-standard research could be adopted into the real world. Vendor ecosystems, including open source, are diverse in different domains of use cases. Going back to transportation use cases, for example, pedestrian safety and driver safety are important considerations. Adoption of AI-based solutions that involve humans such as pedestrians and drivers require their trust and perception of using AI-based solutions.
• Developer ecosystem created via opensource.  An energized third-party developer ecosystem not only fast-tracks adoption but also enables revenue generation. Developer ecosystem bootstraps reference implementations of algorithms, with baseline and open-source toolsets. Third-party applications, Application Programmer Interfaces (API), and Software Development Kits (SDK) along with crowd-sourced solutions increase the generalizability of AI/ML solutions across regions and domains via transfer learning. Hardware implementations, especially open-source IoT boards are evolving to host the edge data processing.
• Data collection and model validation via Sandbox pilot experimental setups.  Many use cases require an experimental sandbox, create experimental solutions, and validate them using experimental setups. While real-world data would imply a more reliable source of data and a realistic testing environment, not all scenarios could be encountered in the real world, especially when catastrophic events and related data are rare.