ai.txt is a domain-specific language (DSL) similar to robots.txt. Website owners use the /ai.txt file to give instructions about their site to AI agents.Â
Here's how it works:
A website owner creates an ai.txt file and puts it under this path of their website: /ai.txt
AI agents that access/interact with the website will first look for the /ai.txt file
If the /ai.txt file exists, the AI agents will parse it and follow the instructions inside when interacting with the website
TO BE EXPANDED
robots.txt is not enough !
Blocklist circumvention
Fine-grained control over AI user-agent actions
TO BE EXPANDED
Simple
Avoid verbosity
Avoid repetition
Actions mean what they say
Easy to use
DSL
Naturalistic language
Avoid logical impositions on the syntax i.e.: too many interactive behaviours between keywords
Customisable
Accommodate different use-cases
Accommodate different levels of security requirements
Accommodate different levels of technical understanding
Extensible
Accommodate rapidity of progress in AI space
# ai.txt for an online shopping website
User-agent: *
    Path: /product html
        Element: pricing
            Disallow: Extract
        Element: p.customer_review
            Disallow: Analyze
            Guide: Summarize
                Lang: en
                Guideline: Focus only on product functionality mentions
        Element: img.product_images
            Disallow: Manipulate
            Disallow: Train
User-agent: *
    Path: /cart html
        Element: checkout_process
            Disallow: Return
            Disallow: Extract
IDE, Agent Examples, ...
Coming soon ...