Embedded Files
Introduction
Introduction
Consider a network solutions architect or engineer trying to design and engineer a BGP [wiki, RFC4271] network deployment use case. Ideally, what the architect would like to do is to input the design requirements as prompt in a natural language and let the LLM response output the BGP network design that satisfies the requirements. The design output will consist of network topology and configuration steps based on BGP protocol and routing (design phase). In addition, the LLM may also output specific BGP router commands or CLI or API (engineering phase). A general introduction on designing a system or solution with LLM is provided here. Note, this is not like having an LLM answer a question from existing knowledge it has (like typical AI Assistant or Agent), rather have the LLM output a design based on (abstract) requirements.
In this article I discuss results of my experiments on BGP network design and engineering with a number of LLM systems that include nvidia/Llama3-ChatQA-1.5-8B, Google Gemini, and Meta AI. In my experiment all of these LLMs yielded incorrect or hallucinated results for the BGP network design described below. All these LLMs may have been trained with contents that include various networking related corpus from the Internet, including BGP. They can answer simple BGP related questions though hallucinated at times. We can also assume that services, such as Gemini and Meta AI are supported by powerful LLMs and resources. While the selected LLMs failed to yield correct results, we can apply advanced prompt engineering techniques on an opensource LLM to generate correct BGP network design and engineering output for the BGP network design requirement described below. The techniques described in this article can only be performed with opensource LLMs, which in my experiment is nvidia/Llama3-ChatQA-1.5-8B. The reason for choosing nvidia/Llama3-ChatQA-1.5-8B is that it allows augmenting the LLM with context and instructions. I call this advanced prompt engineering supported LLM AI-LLM Solution Design assistant, for BGP network solution design it is AI-LLM BGP network design assistant.
BGP Network Design Requirement
BGP Network Design Requirement
Here is the BGP network design requirement prompt: Autonomous Systems AS1 and AS2 have two network paths with each other connected via two BGP routers in each AS. AS2 has two sets of IP prefixes, for example 10.10.10.0/24 (IP1) and 10.10.20.0/24 (IP2). How do we configure BGP so that traffic from AS1 towards IP1 destinations in AS2 exits over one path and towards IP2 destinations over the other path, thus load sharing the two connections? Describe the network topology and then describe all the configuration steps in BGP routers in AS1 and AS2.
This requirement can be satisfied with multiple different ways based on BGP protocol, attributes, routing and route selection algorithm. For example, a) BGP attribute local-preference configurations in AS1, b) combination of community string (in AS2) and local-preference (in AS1) configurations, c) AS Path prepending configurations (in AS2).
BGP network based on this particular requirement may be deployed in Cloud networks, where a customer may connect to a public Cloud provider (AWS, GCP, OCI, Azure, etc.) with dedicated (private) network connections (such as AWS Direct Connect or OCI FastConnect). BGP is then run over these connections. In the above requirement there are two traffic paths, which, for example, may be assigned to traffic for two different departments, dept1 and dept2 of an organization (AS1). The IP prefixes IP1 and IP2 may be assigned to resources that belong to dept1 and dept2 respectively and hosted in a public Cloud (AS2). The BGP routers in AS1 has to be configured in a way so that above path requirement is satisfied. We consider BGP configuration for selecting paths AS1-AS2-IP1 and AS1-AS2-IP2. We do not consider BGP configuration for reverse path selection. Note, this particular use case is relatively simpler than other general BGP network deployment use cases or topologies.
Advanced Prompt Engineering for AI-LLM BGP Network Design Assistant
Advanced Prompt Engineering for AI-LLM BGP Network Design Assistant
Given above BGP network requirement all selected LLMs output network topology and configuration steps with the given (implicit) topology in the requirement. Some LLMs may also output (random) BGP configuration CLI commands together with the topology and configuration steps. As mentioned above all LLMs yielded incorrect or hallucinated answers for both configuration steps and CLI.
Prompts consist of user question, and optionally, system, instruction and context (you will notice in result sections below, for example, "Number of Tokens in Input (system+instruction+context+question): 1691" or "Number of Tokens in Input (system+instruction+question): 316"). Here is an outline of advanced prompt engineering for AI-LLM based solution design assistant:
Craft system, instruction and context parts of the prompt properly to increase the success of correct answers by the LLM.
Divide AI-LLM solution design process into Design (architecture) and engineering phases. Trying to get the answer with a single phase may yield hallucinated results (as all LLMs in the experiment did). Rather results from previous phase is input into next phase in steps in a way that is domain specific.
Provide domain specific constructs in instruction and context.
Context:
Provide in context high level design steps.
High level description of network topology only in a natural language (so that any topology format can be output).
Instruction:
Provide proper and domain specific JSON instructions crafted in a way so that programmatic and parallel inference of phases (mentioned above) is possible.
Instructions for generation of network topology graph format. The concept here is to generate topology graphs from English description only (no coding).
May 2024
Design Phase
Design Phase
By relying on the content in prompt context the LLM (nvidia/Llama3-ChatQA-1.5-8B) was able to output the BGP network design correctly. The LLM also followed correctly the JSON output requirements stated in the instruction part of the prompt.
Network Topology
Network Topology
Notice the networkTopology output in DOT format, which was guided by properly crafting JSON instruction in instruction together with (English) content in context parts of prompt. The DOT file then can be input to GraphViz to output network topology graph (see below). Note work is in progress to guide LLMs for more sophisticated topology graph format output from description in natural language prompt, context and instruction (no coding: for example, I generated these graphs with coding, coded in Go). See below.
Result: Design Phase
Result: Design Phase
Note on the configuration steps. Notice that community string assignments for IP1 and IP2 in R21_AS2 (IP1-->100:200, IP2-->100.150) are flipped in R22_AS2 (IP1-->100:150, IP2-->100.200). When R11_AS1 receives BGP updates from R21_AS2, based on the route-map policy, IP1 is associated with local-preference weight 200 and IP2 with 150 in R11_AS1. Similarly, in R12_AS1 IP1 is associated with local-preference weight 500 and IP2 with 200. As a result routers in AS1 will have following route entries:
An upstream router (R10_AS1) in AS1 will have following route entries:
IP1-->100:200, IP2-->100.150, nexthop: R11_AS1
IP1-->100:150, IP2-->100.200, nexthop: R12_AS1
R11_AS1 route entries:
IP1-->100:200, IP2-->100.150, nexthop: R21_AS2 (eBGP)
IP1-->100:150, IP2-->100.200, nexthop: R12_AS1 (iBGP)
R12_AS1 route entries:
IP1-->100:150, IP2-->100.200, nexthop: R22_AS2 (eBGP)
IP1-->100:200, IP2-->100.150, nexthop: R11_AS1 (iBGP)
Given all other BGP attribute values are equal or not present, BGP route selection algorithm prefers route with the highest local-preference. Hence traffic for IP1 entering into R10_AS1 will be sent to R11_AS1, then to R21_AS2 (Path1), and for IP2 to R12_AS1, then to R22_AS2 (Path2). Hence it is obvious that two different paths are selected because of community string and then local-preference flipping algorithm. None of the LLMs could figure this out on its own.
Number of Tokens in Input (system+instruction+context+question): 1680
"bgpPathLoadSharing": {
"question": "How do we configure BGP so that traffic from AS1 towards IP1 destinations in AS2 exits over one path and towards IP2 destinations over the other path, thus load sharing the two connections?",
"networkTopology": "R11_AS1->R12_AS1[dir=none,color=green]; R11_AS1->R21_AS2[dir=none,color=blue]; R21_AS2->R22_AS2[dir=none,color=green]; R22_AS2->R12_AS1[dir=none,color=blue];",
"configuration": {
"R21_AS2": [
"configure a IP prefix statement IPP1 that permits IP prefix, for example, 10.10.10.0/24 (IP1) and another statement IPP2 that permits IP prefix, for example, 10.10.20.0/24 (IP2)",
"configure a route-map policy RMP1 that matches IPP1 and sets the community string 100:200 and another route-map policy with the same name that matches IPP2 and sets the community string 100:150",
"configure an eBGP session with directly connected router R11 in AS1 and associate the route-map policy outbound",
"configure iBGP session with R22_AS2"
],
"R22_AS2": [
"configure a IP prefix statement IPP1 that permits IP prefix, for example, 10.10.10.0/24 and configure another statement IPP2 that permits IP prefix, for example, 10.10.20.0/24",
"configure a route-map policy RMP1 that matches IPP1 and sets the community string 100:150 and another route-map policy with the same name that matches IPP2 and sets the community string 100:200",
"configure an eBGP session with directly connected router R12 in AS1 and associate the route-map policy outbound",
"configure iBGP session with R21_AS2"
],
"R11_AS1": [
"configure a community list statement CML1 that permits 100:200 and another statement CML2 that permits 100:150",
"configure a route-map policy RMP1 that matches CML1 and sets a local-preference weight of 200 and another route-map policy with the same name that matches CML2 and sets the local-preference weight of 150",
"configure an eBGP session with directly connected router R21 in AS2 and associate the route-map policy inbound",
"configure iBGP session with R12_AS1"
],
"R12_AS1": [
"configure a community list statement CML1 that permits 100:200 and another statement CML2 that permits 100:150",
"configure a route-map policy RMP1 that matches CML1 and sets a local-preference weight of 200 and another route-map policy with the same name that matches CML2 and sets the local-preference weight of 150",
"configure an eBGP session with directly connected router R22 in AS2 and associate the route-map policy inbound",
"configure iBGP session with R11_AS1"
]
},
"conclusion": "By configuring the community strings and local-preference weights as described, traffic from AS1 towards IP1 destinations in AS2 will exit over the path R11_AS1->R21_AS2 and towards IP2 destinations over the path R12_AS1->R22_AS2, thus load sharing the two connections."
}
Result: Engineering Phase
Result: Engineering Phase
Inference for this phase can be parallelized. The result from this phase can then be merged with the design phase result to create a document, including the network topology graph.
Following is the BGP configuration command or CLI output corresponding to the configuration steps (in English) from the design phase for one of the routers (R11_AS1). CLIs for the rest of the steps from the design phase are not shown. No context, but JSON instruction was provided in this phase. Note, prompt did not include specific vendor or vendor OS version (but it picked Cisco CLI for a Cisco OS version).
Number of Tokens in Input (system+question): 281
{
"bgpConfigCommands": {
"commands": [
{
"description": "Configure a community list statement CML1 that permits 100:200",
"cli": [
"ip community-list standard CML1 permit 100:200"
]
},
{
"description": "Configure a community list statement CML2 that permits 100:150",
"cli": [
"ip community-list standard CML2 permit 100:150"
]
},
{
"description": "Configure a route-map policy RMP1 that matches CML1 and sets a local-preference weight of 200",
"cli": [
"route-map RMP1 permit 10",
" match community CML1",
" set local-preference 200"
]
},
{
"description": "Configure a route-map policy RMP1 that matches CML2 and sets the local-preference weight of 150",
"cli": [
"route-map RMP1 permit 20",
" match community CML2",
" set local-preference 150"
]
},
{
"description": "Configure an eBGP session with directly connected router R21_AS2 in AS2 and associate the route-map policy inbound",
"cli": [
"router bgp 1",
" neighbor 192.168.12.2 remote-as 2",
" address-family ipv4",
" neighbor 192.168.12.2 route-map RMP1 in"
]
},
{
"description": "Configure an iBGP session with R12_AS1",
"cli": [
"router bgp 1",
" neighbor 192.168.13.1 remote-as 1",
" address-family ipv4"
]
}
]
}
}
Network Topology
Network Topology
networkTopology": "R11_AS1->R12_AS1[dir=none,color=green]; R11_AS1->R21_AS2[dir=none,color=blue]; R21_AS2->R22_AS2[dir=none,color=green]; R22_AS2->R12_AS1[dir=none,color=blue];" (Note, color=green to indicate intra-AS and color=blue inter-AS connections).
May 2024
Wrong/hallucinated answer. It has got the JSON structure correct based on instructions provided.
Number of Tokens in Input (system+instruction+question): 316
{
"bgpPathLoadSharing": {
"question": "How do we configure BGP so that traffic from AS1 towards IP1 destinations in AS2 exits over one path and towards IP2 destinations over the other path, thus load sharing the two connections?",
"networkTopology": "Autonomous Systems AS1 and AS2 have two network paths with each other connected via two BGP routers in each AS.",
"configuration": {
"R11_AS1": [
"Configure a BGP session between R11_AS1 and R12_AS2",
"Advertise the IP1 prefix (10.10.10.0/24) to R12_AS2",
"Configure a BGP session between R11_AS1 and R13_AS2",
"Advertise the IP2 prefix (10.10.20.0/24) to R13_AS2"
],
"R12_AS2": [
"Configure a BGP session between R12_AS2 and R11_AS1",
"Advertise the IP1 prefix (10.10.10.0/24) to R11_AS1",
"Configure a BGP session between R12_AS2 and R13_AS2",
"Advertise the IP2 prefix (10.10.20.0/24) to R13_AS2"
],
"R13_AS2": [
"Configure a BGP session between R13_AS2 and R11_AS1",
"Advertise the IP1 prefix (10.10.10.0/24) to R11_AS1",
"Configure a BGP session between R13_AS2 and R12_AS2",
"Advertise the IP2 prefix (10.10.20.0/24) to R12_AS2"
]
},
"conclusion": "By configuring BGP sessions and advertising the appropriate IP prefixes, we can achieve path load sharing between AS1 and AS2, ensuring that traffic to IP1 destinations exits over one path and traffic to IP2 destinations exits over the other path."
}
}
Result: with Meta AI
Result: with Meta AI
May 2024
Network topology is OK to certain extent. But the BGP network design is wrong/hallucinated.
It repeated the network topology.
You
Autonomous Systems AS1 and AS2 have two network paths with each other connected via two BGP routers in eaach AS. AS2 has two sets of IP prefixes IP1 and IP2. How do we configure BGP so that traffic from AS1 towards IP1 destinations exits over one path and towards IP2 destinations over the other path, thus load sharing the two connections? Describe the network topology and then describe all the configuration steps in BGP routers in AS1 and AS2
Meta AI
Network Topology:
+---------------+
| AS1 |
+---------------+
|
|
v
+---------------+ +---------------+
| BGP Router | | BGP Router |
| (AS1) | | (AS1) |
+---------------+ +---------------+
| |
| |
v v
+---------------+ +---------------+
| AS2 | | AS2 |
| (Path 1) | | (Path 2) |
+---------------+ +---------------+
| |
| |
v v
+---------------+ +---------------+
| IP1 Prefix | | IP2 Prefix |
| (Destination)| | (Destination)|
+---------------+ +---------------+
Network Topology:
+---------------+
| AS1 |
+---------------+
|
|
v
+---------------+ +---------------+
| BGP Router | | BGP Router |
| (AS1) | | (AS1) |
+---------------+ +---------------+
| |
| |
v v
+---------------+ +---------------+
| AS2 | | AS2 |
| (Path 1) | | (Path 2) |
+---------------+ +---------------+
| |
| |
v v
+---------------+ +---------------+
| IP1 Prefix | | IP2 Prefix |
| (Destination)| | (Destination)|
+---------------+ +---------------+
Configuration Steps:
AS1 BGP Routers:
Configure BGP: Enable BGP on both routers with the AS number of AS1.
Define IP Prefixes: Define IP1 and IP2 prefixes as route maps or prefix lists.
Set up Route Reflection: Configure route reflection to allow the routers to exchange routes with each other.
Configure Path 1: On one router, configure a BGP neighbor relationship with AS2's Path 1 router, and set the route map or prefix list for IP1 prefixes.
Configure Path 2: On the other router, configure a BGP neighbor relationship with AS2's Path 2 router, and set the route map or prefix list for IP2 prefixes.
AS2 BGP Routers:
Configure BGP: Enable BGP on both routers with the AS number of AS2.
Define IP Prefixes: Define IP1 and IP2 prefixes as route maps or prefix lists.
Set up Route Reflection: Configure route reflection to allow the routers to exchange routes with each other.
Configure Path 1: On one router, configure a BGP neighbor relationship with AS1's router connected to Path 1, and set the route map or prefix list for IP1 prefixes.
Configure Path 2: On the other router, configure a BGP neighbor relationship with AS1's router connected to Path 2, and set the route map or prefix list for IP2 prefixes.
Set up AS-PATH: Configure the AS-PATH attribute to prepend the AS number of AS2 to the IP1 and IP2 prefixes advertised to AS1.
Set up MED: Configure the Multi-Exit Discriminator (MED) attribute to set the preference for IP1 and IP2 prefixes advertised to AS1.
By following these steps, traffic from AS1 towards IP1 destinations will exit over Path 1, and traffic towards IP2 destinations will exit over Path 2, achieving load sharing between the two connections.
Result: with Google Gemini
Result: with Google Gemini
May 2024
This output is more comprehensive compared to other outputs (except the one with the advanced prompt engineering). But it is still wrong/hallucinated. First, it picked the IP prefixes in the wrong AS and there is no correct path selection algorithm in configuration steps (as discussed above).
## Network Topology
**AS1:**
* Router1 (R1) - Connects to AS2 Router A (RA) via path 1.
* Router2 (R2) - Connects to AS2 Router B (RB) via path 2.
**AS2:**
* Router A (RA) - Connects to AS1 Router1 (R1).
* Router B (RB) - Connects to AS1 Router2 (R2).
## BGP Configuration
**Important Note:** Specific commands may vary depending on the vendor (e.g., Cisco, Juniper). The following outlines the general steps with placeholder commands. Consult your specific router's documentation for exact commands.
**1. Configure BGP on all Routers:**
* Enable BGP routing protocol on all routers (R1, R2, RA, RB).
* Configure the local Autonomous System (AS) number for each router within its respective AS.
**2. Configure BGP Neighbor Relationships:**
* On R1 and R2:
* Configure RA and RB as BGP neighbors with their respective IP addresses.
* Define neighbor attributes like timers and authentication (if needed).
* On RA and RB:
* Configure R1 and R2 as BGP neighbors with their respective IP addresses.
* Define neighbor attributes like timers and authentication (if needed).
**3. Configure Route Advertisement and Policy (AS1):**
* On R1:
* Configure a route map to differentiate between IP1 and IP2 prefixes.
* Match IP1 prefix (10.10.10.0/24) and set a community attribute (e.g., "IP1-Community").
* Match IP2 prefix (10.10.20.0/24) and set a different community attribute (e.g., "IP2-Community").
* Advertise all routes to AS2 neighbors (RA and RB).
* Apply the route map to the BGP advertisements towards AS2 neighbors to include the community attributes with the respective prefixes.
* On R2:
* Configure the same route map as R1 to differentiate between IP prefixes and assign community attributes.
* Advertise all routes to AS2 neighbors (RA and RB).
* Apply the route map to the BGP advertisements towards AS2 neighbors.
**4. Configure BGP Route Selection Policy (AS2):**
* On RA:
* Configure a route policy to select the path based on the community attribute associated with the prefix.
* When a route with "IP1-Community" is received, prefer the path connected to R1 (path 1).
* When a route with "IP2-Community" is received, prefer the path connected to R2 (path 2).
* Apply the route policy to inbound BGP routes from AS1 neighbors (R1 and R2).
* On RB:
* Configure the same route policy as RA to select the path based on the received community attribute.
* Apply the route policy to inbound BGP routes from AS1 neighbors (R1 and R2).
**5. Verification:**
* Verify BGP neighbor establishment on all routers.
* Check BGP routing table entries on each router to see the received prefixes with community attributes.
* On AS2 routers (RA and RB), verify the selected next hop for IP1 and IP2 prefixes based on the configured policy.
This configuration achieves traffic load balancing across the two paths in AS1 towards IP1 and IP2 destinations in AS2. Packets destined for IP1 will be routed through path 1 (connected to R1) and packets destined for IP2 will be routed through path 2 (connected to R2) based on the community attribute assigned by AS1 and the policy implemented by AS2.
Result: Llama 3 70B on HuggingFace
Result: Llama 3 70B on HuggingFace
May 2024
https://huggingface.co/chat/conversation/6637278c864117b8c28339e9
BGP network design is wrong/hallucinated.
BGP Router Configuration Steps:
AS1 BGP Routers:
Router1 (connected to AS2 RouterA):
Enable BGP routing: "router bgp <AS1_BGP_ASN>".
Configure the local BGP router ID: "bgp router-id <Router1_ID>".
Establish a BGP peering session with AS2 RouterA: "neighbor <AS2_RouterA_IP> remote-as <AS2_BGP_ASN>".
Advertise a default route to AS2: "network <AS1_IP_PREFIX> mask <AS1_IP_PREFIX_MASK>".
Configure outbound traffic policy:
Create a prefix-list that matches IP1 destinations: "ip prefix-list IP1_PREFIXES permit <IP1_PREFIX>".
Set the next-hop for IP1 traffic to AS2 RouterA: "set ip next-hop <AS2_RouterA_IP>".
Apply the outbound policy to traffic exiting towards AS2: "neighbor <AS2_RouterA_IP> out prefix-list IP1_PREFIXES out".
Ensure that the BGP routing table has valid routes to IP1 destinations via AS2 RouterA.
Router2 (connected to AS2 RouterB):
Repeat steps 1-3 from Router1, replacing values for Router2 and AS2 RouterB.
Do not advertise any networks or prefixes to AS2 RouterB.
Ensure that Router2 learns IP2 prefixes from AS2 RouterB via BGP updates.
AS2 BGP Routers:
RouterA (connected to AS1 Router1):
Enable BGP routing and configure the local BGP router ID as mentioned in AS1 routers.
Establish a BGP peering session with AS1 Router1.
Advertise IP1 prefixes to AS1: "network <IP1_PREFIX> mask <IP1_PREFIX_MASK>".
Ensure that RouterA also has routes to IP2 prefixes, either through static routes or IGP.
Configure outbound traffic policy:
Create a prefix-list that matches IP2 destinations: "ip prefix-list IP2_PREFIXES permit <IP2_PREFIX>".
Set the next-hop for IP2 traffic to AS1 Router2 (AS2 RouterB's IP): "set ip next-hop <AS1_Router2_IP>".
Apply the outbound policy to traffic exiting towards AS1: "neighbor <AS1_Router1_IP> out prefix-list IP2_PREFIXES out".
RouterB (connected to AS1 Router2):
Repeat steps 1-3 from RouterA, replacing values for RouterB and AS1 Router2.
Advertise IP2 prefixes to AS1: "network <IP2_PREFIX> mask <IP2_PREFIX_MASK>".
Ensure that RouterB also has routes to IP1 prefixes.
No outbound policy configuration is needed on RouterB.
With these configurations, BGP will direct traffic from AS1 towards IP1 destinations over the path through Router1 and AS2 RouterA. Similarly, traffic towards IP2 destinations will exit over the path through Router2 and AS2 RouterB, effectively load-sharing the two connections between AS1 and AS2.
Page updated
Google Sites
Report abuse