5.   Privacy Protection & Federated Learning

How does an AI PC or AI cellphone keep its private data while accessing GPT in the cloud? Can retrieval augmented generation (RAG) achieve such privacy goal on these edge devices? Or other measures / technologies need to play together with RAG to achieve privacy? Also, is it correct to say that a browser-server system has no way to achieve privacy?

5.1    Protecting Private Data in Edge and Cloud Environments

The concerns about data privacy in the context of AI systems, particularly when interacting with cloud-based services like GPT, are valid. Let's delve into how edge computing, RAG, and other technologies can be leveraged to protect sensitive information.

5.2    The Role of Edge Computing

Edge computing plays a crucial role in safeguarding privacy by processing data closer to its source. By minimizing the amount of data transmitted to the cloud, the risk of data breaches is reduced.

• Data Minimization: Only essential data is sent to the cloud, limiting potential exposure.

• Local Processing: Sensitive computations can be performed on the device, reducing reliance on cloud services.

• Faster Response Times: Edge computing often leads to lower latency, enhancing user experience and potentially reducing the need to send data to the cloud.

5.2.1      The Power of RAG

Retrieval Augmented Generation (RAG) can be a valuable tool for preserving privacy in AI systems. By combining the strengths of traditional search with generative AI, RAG can:

• Contextual Understanding: RAG models can better understand the context of a query by accessing relevant information from a local knowledge base.

• Reduced Reliance on Cloud: By leveraging local data, RAG can minimize the need to send sensitive information to the cloud.

• Enhanced Privacy: By processing data locally, RAG can help protect user privacy.

5.2.2      Additional Privacy Measures

To further enhance privacy, consider these additional strategies:

• Homomorphic Encryption: This advanced cryptographic technique allows computations to be performed on encrypted data without decryption, providing strong privacy guarantees.

• Federated Learning: Train AI models collaboratively across multiple devices without sharing raw data, preserving privacy.

• Differential Privacy: Add noise to data before sharing it to protect individual privacy while enabling statistical analysis.

• Zero-Knowledge Proofs: Verify information without revealing the underlying data, ensuring privacy and security.

5.2.3      Browser-Server Systems and Privacy Challenges

Traditional browser-server systems inherently pose greater privacy risks due to the centralized nature of data processing. User data is often transmitted to servers for processing, increasing the potential for data breaches and misuse.

To mitigate these risks, consider:

• Client-Side Processing: Offload non-sensitive computations to the client device to reduce data transmission.

• Secure Communication: Use encryption protocols (HTTPS) to protect data in transit.

• Data Minimization: Only send essential data to the server.

• User Control: Provide users with granular control over data sharing and privacy settings.

By combining edge computing, RAG, and advanced privacy technologies, it's possible to build AI systems that effectively protect user data while delivering valuable services.

5.3    Federated Learning: Preserving Privacy Through Collaborative Learning

How can federated learning preserve privacy by collaboratively learn across multiple devices without sharing raw data. Can you give an example to explain this?

Federated learning is a machine learning technique that allows multiple devices or servers to collaboratively train a shared model without sharing raw data. This approach effectively preserves privacy by keeping sensitive information decentralized.

5.4    A Simple Example

Imagine a scenario where a smartphone app wants to improve its handwriting recognition without compromising user privacy.

1. Local Training: Each user's device trains a local model on their handwritten data.

2. Model Updates: Instead of sending raw handwriting data to a central server, each device sends only the model updates (weights and biases) to a central server.

3. Model Aggregation: The central server aggregates the model updates from multiple devices to create a global model, ensuring that no individual data is revealed.

4. Model Distribution: The updated global model is sent back to the devices, where it's combined with the local models to improve handwriting recognition.

Key Points:

• Data Privacy: No raw data leaves the device. Only model updates are shared.

• Collaboration: Multiple devices contribute to improving the shared model.

• Security: Encryption and other security measures can be implemented to protect model updates during transmission.

Additional Considerations

• Differential Privacy: To further enhance privacy, differential privacy can be applied to add noise to the model updates, making it difficult to infer individual data points.

• Homomorphic Encryption: This advanced cryptographic technique allows computations to be performed on encrypted data, providing an additional layer of security.

• Secure Aggregation: Techniques like secure multi-party computation can be used to aggregate model updates without revealing individual contributions.

By combining these techniques, federated learning can create a robust framework for privacy-preserving AI model development.

5.5    Federated Learning and Blockchain

some of the standards and initiatives mentioned in relation to Federated Learning (FL) also consider the role of AI agents, particularly in the context of autonomous systems, privacy, security, and interoperability. AI agents in federated learning environments often play critical roles in orchestrating the training process, managing data privacy, and ensuring secure communication between nodes (clients and servers). Here’s how AI agents are associated with the standards:

5.5.1      Federated AI Technology Enabler (FATE)

• AI Agents: FATE includes AI agents as part of its architecture to manage federated learning tasks. These agents can autonomously handle model training, updates, and communication between different parties in a federated learning environment.

• Standardization: As FATE is involved in various standardization efforts, the role of AI agents in orchestrating federated learning is an area of interest, particularly in ensuring they adhere to privacy and security protocols.

5.5.2      IEEE Standards Association (IEEE-SA)

• P3652.1-2020: This guide covers the architectural framework for federated learning, which can include AI agents responsible for coordinating distributed training tasks. The standard also emphasizes the importance of AI agents in maintaining data privacy and secure communication.

• AI Agents in Security: AI agents are often part of the security framework in federated learning, ensuring that data is processed in a way that complies with privacy standards and regulations.

5.5.3      Linux Foundation's Federated Learning Community

• AI Agents and Interoperability: In the context of the Linux Foundation’s initiatives, AI agents are key to achieving interoperability between different federated learning systems. These agents need to operate under standardized protocols to communicate effectively across diverse systems.

• LF AI & Data: AI agents are also part of the broader AI ecosystem under the Linux Foundation, where they contribute to standardizing AI workflows, including federated learning.

5.5.4      ETSI (European Telecommunications Standards Institute)

• ISG SAI: In ETSI’s efforts to secure AI systems, AI agents are considered crucial for implementing security and privacy measures in federated learning. AI agents are expected to adhere to the standards developed for secure data handling, encryption, and communication in AI systems.

• Autonomous Agents: ETSI’s focus on AI in telecommunication includes the role of autonomous agents in federated learning, ensuring that they comply with emerging security and privacy standards.

5.5.5      Privacy and Security Protocols

• AI Agents in Secure Computation: AI agents are often embedded in the secure computation frameworks (like homomorphic encryption and secure multi-party computation) used in federated learning. These agents ensure that computation on encrypted data is done according to established standards, maintaining data privacy across federated nodes.

• AI Agents in Emerging Standards

• Autonomy: AI agents in federated learning are increasingly autonomous, handling tasks such as model training, aggregation, and optimization without human intervention. Standards related to federated learning consider the autonomy of these agents and how they must operate within the defined protocols to ensure consistent, secure, and privacy-preserving operations.

• Inter-Agent Communication: Another aspect of AI agent standardization involves the protocols for communication between agents in different federated learning systems. These protocols ensure that agents can collaborate or share insights without compromising security or privacy.

5.5.6      Conclusion

AI agents are indeed associated with the standards and frameworks related to Federated Learning, particularly in the areas of orchestration, privacy, security, and interoperability. These agents are crucial for automating the complex processes involved in federated learning, and the standards aim to ensure that they operate in a secure, privacy-preserving, and interoperable manner. As federated learning continues to evolve, the role of AI agents will likely become even more central, with standards continuing to adapt to these advancements.

5.6    Is Agent Communication a kind of Federated Learning?

The agent communication, e.g. the interest/position matching between a jobseeker agent and a headhunter agent could incorporate some principles of federated learning, but it's not necessarily federated learning by itself. Here’s why:

5.6.1      Federated Learning:

• Definition: Federated learning involves training machine learning models across multiple decentralized devices or servers (nodes) that hold local data samples, without sharing the data itself. Instead, each node trains a model locally and then only shares model updates (gradients or weights) with a central server or with other nodes.

• Privacy and Security: Federated learning is particularly focused on preserving data privacy, as the raw data never leaves the local devices, and only model updates are shared.

5.6.2      Jobseeker and Headhunter Agents:

• Communication: The jobseeker agent and headhunter agent communicate to match job positions with the skills, experience, and preferences of the job seeker. This communication might involve sharing information about job openings, candidate qualifications, preferences, and possibly even negotiation or recommendation processes.

• Interest/Position Matching: The matching process could involve machine learning models that help in ranking or recommending job positions to job seekers or candidates to recruiters. These models could be pre-trained or dynamically updated based on the interaction data between the agents.

5.6.3      Comparison with Federated Learning:

• Model Training vs. Matching: In federated learning, the main activity is the training of a machine learning model across multiple decentralized data sources. In contrast, the jobseeker and headhunter agents are primarily focused on matching interests (skills and job positions), which might involve applying pre-trained models but doesn’t necessarily involve training models across decentralized data sources.

• Data Privacy: While federated learning inherently involves privacy-preserving techniques, the communication between the jobseeker and headhunter agents may or may not be privacy-preserving, depending on how the system is designed. If privacy-preserving techniques are employed (e.g., encrypted communication, differential privacy), it might resemble some aspects of federated learning but still isn't the same as federated learning.

5.6.4      Potential Use of Federated Learning:

• Enhanced Matching: If the system were designed such that multiple jobseeker agents and headhunter agents across different platforms or companies collaboratively improve their matching algorithms by sharing model updates rather than raw data, this would be closer to federated learning. For example, each agent could improve its own recommendation or matching algorithm by participating in a federated learning process where they share updates to a common model without exposing personal data.

5.6.5      Conclusion:

The communication and matching process between a jobseeker agent and a headhunter agent is not inherently federated learning. However, if these agents were to engage in a process where they collaboratively improve their algorithms through shared updates while keeping their data private, then it would incorporate elements of federated learning.

So, while the interaction itself isn't federated learning, federated learning principles could be applied to enhance the privacy and effectiveness of the matching process.

5.7    Can Federated Learning Replace Large Job Marketplaces?

Does our job website compete against Linkedin or collaborate with Linkedin (e.g., pay for their job database)?  First, we need to find out whether the large Job Marketplaces like Linkedin, the intermediators, will be eliminated in future AI era because of the federated learning between jobseeker and hiring company directly. Because jobseeker always wants to get the best job among multiple hiring companies, and a hiring company always wants to get the best person among multiple jobseekers, the objectives of the two parties are different. So, the job websites, or intermediators, may always have their value of existence as centralized entities that oversee all resumes and all job postings to match. However, I am not sure whether the technology of federated learning can play an effective centralized, or "sees all" role, meaning that federated learning can replace the role of current large job marketplaces. One of the difficulties I can see is that if federated learning can protect privacy by decentralization, then how can resumes and job postings be matched globally and optimally?

Here is the reasoning to resolve the confusion:

The idea of federated learning replacing job websites is intriguing but complex, especially when considering the objectives of job seekers and hiring companies.

Federated learning is designed to protect privacy by enabling decentralized data processing where individual datasets remain local, and only model updates are shared. This privacy-centric approach makes it challenging to centralize data in the same way current job websites do, which have full visibility of all resumes and job postings to facilitate matching.

If federated learning were applied to job matching, each party (job seeker and hiring company) would retain control over their data, sharing only necessary information to improve matching algorithms. However, this decentralized nature could limit the ability to "see all" as job websites currently do, potentially reducing the effectiveness of the matching process.

For federated learning to replace large job marketplaces like Linkedin, it would need mechanisms to aggregate and balance the diverse objectives of both job seekers and hiring companies without violating privacy. This is a significant technical and conceptual challenge. Furthermore, if resumes and job postings were treated as public data, it could undermine the privacy protections that federated learning aims to provide.

In conclusion, while federated learning could contribute to more personalized and private job matching, it's unlikely to fully replace large job marketplaces like Linkedin unless these challenges are addressed. Centralized platforms might still be necessary to oversee and optimize the overall matching process, especially when different objectives and privacy concerns are involved.

5.8    Federated Learning and Blockchain for Job Marketplace

Our original idea regarding job website is to build a blockchain for this job business, such that jobseekers and hiring companies are all in one community, the privacy is protected, the trust is built via the transactional blockchain, and this can be decentralized. Now the federated learning (FL) technology is newer, and it may improve over blockchain's drawbacks, like easy to be hacked, and many governments felt threatened by the crypto coins. Can FL be a better technology than blockchain for my business aiming jobseeker, job transferring people, entrepreneurs, and industry hiring companies? We can now build agents to prove math theorems, develop algorithms to solve company's business transformation plan (BTP) to suit AI era, and generate code for them such that I can build FL subnet community (similar to blockchain) so that companies in this community can hire interested jobseekers who are trained to manipulate these agents via my training programs.

Our idea of leveraging federated learning (FL) for a decentralized job marketplace is intriguing, particularly when compared to blockchain technology. Here’s a breakdown of how FL could be a viable alternative, and some considerations:

5.8.1      Advantages of Federated Learning Over Blockchain:

5.8.1.1     Privacy Preservation:

1. • FL naturally aligns with privacy preservation by allowing job seekers and companies to keep sensitive data (like resumes and job postings) local while still participating in the training of a global model. This could address privacy concerns more effectively than blockchain, where data (even if encrypted) is distributed across nodes.

5.8.1.2     Security:

1. • FL can offer a more secure alternative to blockchain, especially if combined with techniques like differential privacy or secure multi-party computation (SMPC). These techniques ensure that individual data points are not exposed, reducing the risk of hacking or data breaches.

5.8.1.3     Regulatory Compliance:

1. • Given the concerns governments have with blockchain (especially in relation to cryptocurrencies), FL might face fewer regulatory hurdles, making it easier to deploy across different jurisdictions.

5.8.1.4     Decentralization with Coordination:

1. • FL provides a decentralized approach, yet with the ability to coordinate through a central model aggregator or coordinator. This could be an advantage in creating a balance between decentralization and the need for a coordinating entity, which could play a similar role to job websites but without holding all the data.

5.8.2      Challenges and Considerations:

5.8.2.1     Objective Alignment:

1. • You correctly noted that the objectives of job seekers and hiring companies are different. In FL, aligning these objectives within a single model may be complex. It may require advanced techniques to ensure that both parties benefit equally from the model without compromising privacy or fairness.

5.8.2.2     Interoperability:

1. • Building a FL-based system where different companies and job seekers can seamlessly interact would require standardized protocols for agent communication, model training, and data privacy. Ensuring that these standards are adhered to across all participants might be challenging.

5.8.2.3     Scalability:

1. • FL’s effectiveness in a large-scale, highly dynamic environment like a job marketplace is still an area of active research. You’d need to consider how to handle potentially thousands or millions of participants efficiently.

5.8.2.4     Centralization vs. Decentralization:

1. • Although FL offers a decentralized approach, it typically involves a central model aggregator. Deciding whether this should be fully decentralized (as in blockchain) or semi-centralized (with a trusted entity) will impact the design and governance of your system.

5.8.3      Combining FL with Blockchain:

5.8.3.1     Hybrid Approach:

• • One potential solution could be a hybrid approach where FL is used for training models and making predictions, while blockchain is used to maintain a decentralized ledger of transactions (such as job applications, hires, and training certifications). This could combine the strengths of both technologies.

5.8.3.2     Smart Contracts for FL Coordination:

• • Blockchain could also be used to manage smart contracts that govern the interactions between agents in the FL network, ensuring transparency and trust in how models are trained and used.

5.8.4      Next Steps and Suggestions:

5.8.4.1     Prototype Development:

1. • Start by developing a prototype of the FL-based job marketplace, focusing on a specific aspect (e.g., job matching). Test it in a controlled environment to understand the challenges and opportunities.

5.8.4.2     Agent Development:

1. • Develop the agents that can prove theorems, develop algorithms, and generate code. These agents could be a key differentiator in your platform, providing specialized services to companies within the FL network.

5.8.4.3     Community Building:

1. • Engage with potential participants (job seekers, companies) early on to gather feedback and ensure that your system meets their needs. This will also help in establishing trust and adoption.

5.8.4.4     Explore Partnerships:

1. • Consider partnerships with academic institutions or research labs to further explore the integration of FL and blockchain, particularly in areas like security, privacy, and scalability.

Overall, federated learning holds promise as a foundation for your vision, potentially offering a more flexible, secure, and privacy-preserving alternative to blockchain. However, a hybrid approach might be the best of both worlds, leveraging the strengths of each technology while mitigating their weaknesses.

5.9   Differential Privacy: Add Noise to Model Updates in Federated Learning

5.9.1      Differential Privacy

Differential privacy is a technique used to add noise to data to protect individual privacy while allowing for statistical analysis. In the context of federated learning, we can apply differential privacy to the model updates that are shared between devices.

5.9.2      Example: Handwriting Recognition

Let's consider the handwriting recognition example we discussed earlier. Each user trains a local model on their handwritten data. When the model updates are sent to the central server, differential privacy can be applied as follows:

1. Generate Noise: For each model update parameter, a random noise value is drawn from a specific noise distribution (e.g., Laplace or Gaussian distribution).

2. Add Noise: The noise value is added to the corresponding model update parameter.

3. Share Noisy Update: The modified (noisy) model update is sent to the central server.

The central server aggregates the noisy model updates from multiple devices to create a global model. While the noise added to individual updates might slightly degrade the model's accuracy, it significantly increases privacy by making it difficult to infer information about any specific user's data from the aggregated model.

5.9.3      Key Considerations

• Noise Level: The amount of noise added should be carefully calibrated. Too much noise can significantly degrade model performance, while too little noise might not provide sufficient privacy protection.

• Privacy Budget: Differential privacy often uses a privacy budget to limit the amount of information that can be inferred from the data.

• Computational Overhead: Adding noise and performing computations on noisy data can increase computational costs.

By carefully considering these factors and implementing differential privacy techniques, federated learning can be a powerful tool for preserving privacy while enabling collaborative model training.

5.9.4       Privacy Budget: A Measure of Information Disclosure

A privacy budget is not a monetary budget; it's a conceptual metric that represents the maximum amount of information that can be revealed about an individual through a computation or system.

Think of it as a token system. You have a limited number of tokens (the privacy budget) to spend on revealing information about yourself. Each piece of information you disclose consumes a portion of your budget. Once the budget is exhausted, no further information can be revealed.

5.9.5      Key Characteristics of a Privacy Budget

• Quantifiable: The privacy budget is a numerical value representing the maximum allowable information leakage.

• Flexible: The specific value of the privacy budget can be adjusted based on the sensitivity of the data and the desired level of privacy protection.

• Cumulative: The budget is consumed gradually with each computation or data disclosure.

• Trade-off: A higher privacy budget allows for more information to be revealed but comes at the cost of reduced privacy protection.

5.9.6      Example: Privacy Budget in Differential Privacy

In differential privacy, the privacy budget is used to control the amount of noise added to data. A higher privacy budget allows for less noise, potentially improving the accuracy of the model but also increasing the risk of privacy breaches.

By carefully managing the privacy budget, system designers can balance the need for data utility with the importance of protecting individual privacy.

5.10  Privacy of Client-Side Processing

While client-side processing can offer certain privacy advantages, it's essential to understand its limitations and potential vulnerabilities.

5.10.1  Advantages of Client-Side Processing

• Reduced Data Transmission: By processing data locally, sensitive information is less likely to be exposed to external servers.

• Faster Response Times: Client-side processing can often lead to quicker interactions as there's no need to wait for server-side responses.

• Offline Functionality: Some applications can operate offline, providing continued access to features without relying on a network connection.

5.10.2  Challenges and Considerations

• Limited Processing Power: Client devices, especially mobile devices, have less computational power compared to servers, which can restrict the complexity of tasks performed locally.

• Inconsistent Hardware: Different devices have varying processing capabilities, which can impact application performance and consistency.

• Security Risks: Malicious code can potentially access and manipulate data on the client side, compromising privacy.

• User Error: Users might inadvertently share sensitive data through unintentional actions or by installing malicious software.

5.10.3  Example: Password Hashing

A common example of client-side processing for privacy is password hashing. Instead of sending plain-text passwords to a server, the browser hashes the password locally before transmission. This makes it significantly more difficult for attackers to obtain user credentials, even if the server is compromised.

5.10.4  Additional Privacy Measures

To further enhance privacy in client-side processing, consider the following:

• Local Storage: Use browser storage options like Local Storage or IndexedDB to store data locally, reducing reliance on server-side storage.

• Encryption: Encrypt sensitive data before storing it locally to protect against unauthorized access.

• Regular Updates: Keep software and operating systems up-to-date to address security vulnerabilities.

• User Education: Educate users about best practices for online security, such as avoiding suspicious links and protecting their devices.

In conclusion, while client-side processing can offer privacy benefits, it's essential to be aware of its limitations and implement additional security measures to protect sensitive data. A balanced approach that combines client-side and server-side processing often provides the best outcome.

5.11 Browser / Server Model: Manipulating Client-Side Data with JavaScript

Let us say we use JavaScript to do browser programming. How can we manipulate the client-side data? For instance, we have a local (client-side) Python program to process some data, but how these data can automatically feed the JavaScript program on the webpage? Or I need manually copy these data to the text boxes programmed by JavaScript?

5.11.1  Understanding the Challenge

We're aiming to bridge the gap between a Python script processing data and a JavaScript application on a webpage. This involves transferring data from the Python environment to the browser.

5.11.2  Potential Solutions

Here are a few methods to achieve this:

5.11.2.1  Server-Side Data Transfer:

• API Endpoint: Create a REST API endpoint in your Python application to serve the processed data.

• JavaScript Fetch: Use JavaScript's fetch or XMLHttpRequest to make a request to the API endpoint and retrieve the data.

• Data Format: Choose a suitable data format (JSON, CSV, XML) for efficient data transfer.

Example:

JavaScript

5.11.2.2     File-Based Data Transfer

• Data Export: Export the processed data from Python into a file format (e.g., CSV, JSON). (This is not a browser/server model, explained in next subsection)

• File Upload: Provide a file upload mechanism in your JavaScript application for users to select and upload the file.

• Data Parsing: Use JavaScript libraries like Papa Parse to parse the uploaded file and process the data.

5.11.2.3     Manual Data Entry

• Copy and Paste: This method is less efficient but can be used for smaller datasets.

• User Interface: Create input fields in your JavaScript application where users can manually enter the data.

5.11.2.4     Considerations and Best Practices

• Data Security: If the data is sensitive, consider encrypting it before transferring or storing it.

• Data Format: Choose a data format that is easy to parse and process in both Python and JavaScript.

• Error Handling: Implement proper error handling to handle potential issues during data transfer and processing.

• User Experience: Design the user interface to facilitate easy data input and display.

• Performance: Optimize data transfer and processing for efficient performance.

By carefully considering these factors and selecting the appropriate method based on your specific requirements, you can effectively transfer data from your Python script to your JavaScript application.

5.11.3      Clarifying the fetch('/api/data') Endpoint

Let us question the specificity of the /api/data endpoint.

In the context of server-side data transfer, the fetch('/api/data') call indeed refers to a server-side endpoint. This endpoint is typically a URL that points to a specific resource or function on your server.

Here's a breakdown:

• Client-side: The JavaScript code on the web page initiates an HTTP request to the specified endpoint.

• Server-side: The server receives the request, processes the data, and sends a response back to the client.

Example:

If you have a Python Flask application running on your server, you might define an API route like this:

Python

In this case, the fetch('/api/data') call in JavaScript would send a request to this endpoint, and the server would return the JSON data.

Key Points:

• The /api/data part is a convention often used to indicate an API endpoint.

• The actual endpoint URL depends on your server configuration.

• Proper authentication and authorization mechanisms should be implemented to protect the data.

By understanding this distinction, you can effectively structure your web application to retrieve data from the server and process it on the client side.

5.11.4      How JavaScript Keeps Client’s Privacy

JavaScript, and by extension other browser-based languages like TypeScript, do not have direct access to the client-side file system. This restriction is primarily for security reasons. Allowing unrestricted file access would pose significant risks to user data and system integrity.

5.11.4.1     Key Points:

• No Direct File Access: JavaScript cannot directly read or write files from a user's computer without explicit user interaction (e.g., file uploads).

• Security Implications: Unrestricted file access could lead to malicious activities such as data theft, malware installation, or system compromise.

• User Consent: When a user uploads a file, they are granting explicit permission for the website to access and process that specific file.

5.11.5      Alternatives to Direct File Access

While direct file access is limited, there are alternative approaches:

• File Uploads: Users can manually upload files to the server for processing.

• Web Storage (localStorage, sessionStorage): These APIs allow for storing data locally on the client, but they are limited in size and scope.

• IndexedDB: For larger amounts of structured data, IndexedDB provides a more robust storage solution.

• Server-Side Processing: For complex file operations or sensitive data, delegate the processing to a server-side script.

By understanding these limitations and exploring alternative methods, you can effectively manage data within your web applications while prioritizing user privacy and security.