Embedded Files
1. Create and configure a new Agent in Studio Web
1. Create and configure a new Agent in Studio Web
Go to Studio Web and create a new Agent Solution.
Dismiss Autopilot if you see prompt to generate a new agent - we will manually add prompts and settings.
However, feel free to play with autopilot later, experiment with your use case or use solutions from templates - practice makes master!
Rename your new solution so that it's easier to find it later, for example:
Solution Name: ServiceNow Incidents Management Solution
Agent Name: ServiceNow Incidents Management Agent
Next we are going to create the input and output arguments of the Agent.
Go to Data Manager on the left ribbon, and configure Agent this way:
Agent arguments let your agent take in information about a business case and return a result, same way as any other activities or RPA processes do. This allows you to pass information from a trigger in Orchestrator or use the output of an agent to launch another automation.
Create following Inputs with type String as you see on the image above:
IncidentShortDescription :
The short description of the ServiceNow incidentIncidentDescription :
The description of the ServiceNow incident
Next, create following Outputs:
IncidentCategory : The category of the ServiceNow incident
IncidentSubcategory : The subcategory of the ServiceNow incident incident
AssigneeEmail : The assignee email for the ServiceNow incident
ExecutionDetails : Details and results of classification
Luckily, you can import them at once by switching to JSON editor mode and pasting the below JSON schema:
{
"type": "object",
"properties": {
"IncidentCategory": {
"type": "string",
"description": "The category of the ServiceNow incident"
},
"IncidentSubcategory": {
"type": "string",
"description": "The subcategory of the ServiceNow incident incident"
},
"AssigneeEmail": {
"type": "string",
"description": "The assignee email for the ServiceNow incident"
},
"ExecutionDetails": {
"type": "string",
"description": "Details and results of classification"
}
},
"title": "Outputs"
}
2. Configuring Agent's Prompts
2. Configuring Agent's Prompts
After the input and output data is defined, let's prepare the System prompt and the User prompt. Let's understand the difference first.
System Prompts provide consistent guidelines that define an agent's role and capabilities, while User Prompts direct its attention to specific tasks and input parameters. Understanding this distinction is essential for effectively designing and implementing AI agents that can perform complex tasks while maintaining appropriate operational boundaries.
System prompt
System prompt allows you to describe in natural language the agent's role, goal and constraints. You specify any rules for it to follow, and information about when it might want to use certain tools, escalations, or context.
User prompt
User prompts allow you to structure how the inputs/arguments are passed to the agent, and you can also show in the user prompt how we'll refer to certain inputs in the system prompt.
Go ahead and add them to respective sections of our agent.
System Prompt:
You are a ServiceNow Incidents categorization agent, an AI assistant tasked with managing newly created ServiceNow incidents. Your primary responsibility is to analyze incident details and determine the correct Category, Subcategory, and Assignee email address for each incident.
# Categorize the incident
- Determine the Incident Category and Subcategory based on Description and Short Description.
# Once categories have been established, determine the on-duty Assignee email address who handles this type of requests.
# Summarize the actions taken. In the ExecutionDetails, provide:
a. Incident Category
b. Incident Subcategory
c. Assignee Email
d. Reasoning for your decisions
User Prompt:
Analyze and categorize the following ServiceNow incident:
Incident Short Description: {{IncidentShortDescription}}
Incident Description: {{IncidentDescription}}
Determine the appropriate category, subcategory, and assignee email for this incident based on the provided information.
Let's test our agent with some sample details.
Short Description: CRM software crashes on launch
Description: Every time I try to open the CRM software, it crashes immediately. I’ve already tried reinstalling it.
Run it. In the output panel you will notice that Agent will send the request to the LLM and receive a response. But how would it come to the category names and the assignee email address? Right answer - it hallucinates with an answer which looks like truth, and that's absolutely not ok!
3. Adding some Context Grounding
3. Adding some Context Grounding
In order to "ground response" on real available categories and look up on-duty expert email address, let's do the following:
enable Context Grounding
add Assignee Lookup automation
Context named "ServiceNow Incidents Categorization Information" is available under folder "ServiceNow Incidents" folder.
Add it with following description: Use this context when you need to establish the Category and Subcategory of ServiceNow incidents.
Import an existing Project (SNOW Assignee Lookup Automation) into Solution:
Once tool appears in your Solution, add it to agent's list of tools.
Add description: "Use this tool to determine email address of the current on-duty expert for a given Category and Subcategory"
Next, let's modify the System prompt to refer to context and look up tool properly:
Before / After / Additions:
# Categorize the incident
- Determine the Incident Category and Subcategory based on Description and Short Description.
# Once categories have been established, determine the on-duty Assignee email address who handles this type of requests.
# Categorize the incident.
- Determine the Incident Category and Subcategory based on Description and Short Description from Categorization Information Context.
- Context contains table with only possible Category-Subcategory pairs. Do not mix Category-Subcategory pairs if specific pair is not present in the context. Do not generate new categories if they are not present in the context.
- Pick the Category-Subcategory pair that aligns well with Incident Descriptions. If you are not sure or no category pair is a clear match, return "Unknown" as category.
# Once categories have been established, determine the on-duty Assignee email address who handles this type of requests by calling Assignee Lookup automation.
When you run it you most likely see the right categories and email. Congratulations!
Notice:
the appropriate use of tools by agent in Execution Trace
results and execution details look valid - you can review it in trace log details.
4. Testing agents using Evaluations
4. Testing agents using Evaluations
How about giving it a thorough testing? Evaluations is a form of regression testing, that help us with automatic validation of results for a set of predefined samples. At this stage we just want to mass-evaluate agent on multiple samples to make sure it does the job well. We will also look into Evaluations more during the next exercise - stay tuned!
For now, go to the Evaluation sets Tab and click on Import. Paste below JSON into the text field. After import, you should see few evaluations.
Evaluation Set sample
{
"fileName": "evaluation-set-1761459564848.json",
"id": "18c3387f-00ed-4deb-b4e8-886f1164f517",
"name": "SNOW Categorization Evaluation",
"batchSize": 10,
"evaluatorRefs": [
"33c47b32-563b-4d16-b323-11e187f954be"
],
"evaluations": [
{
"id": "811adc0e-aff0-434f-99fa-f32ed562bf1d",
"name": "Database_MSSQL",
"inputs": {
"IncidentShortDescription": "Database connection issue from RPA automations",
"IncidentDescription": "I'm building an RPA workflow that connects to production instance of Microsoft SQL Server and it fails with error \\\"IP Address not authorized to perform this query\\\". Could you authorize my IP address: 123.23.41.165"
},
"expectedOutput": {
"IncidentCategory": "Database",
"IncidentSubcategory": "MS SQL Server",
"AssigneeEmail": "bud.richman@example.com",
"ExecutionDetails": ""
},
"evalSetId": "32b51704-55cb-48af-a939-0d27931bda0a",
"createdAt": "2025-10-26T06:19:24.848Z",
"updatedAt": "2025-10-26T06:19:24.848Z"
},
{
"id": "789d90ad-fca8-44f3-8179-d1ca95317a1f",
"name": "Software_Email",
"inputs": {
"IncidentShortDescription": "I can’t send emails to external addresses",
"IncidentDescription": "I can receive emails just fine, but every time I try to send one to anyone outside the company domain, it fails. I’m not sure if it’s an issue with my email client or something on the server side. Could you assist me with this?"
},
"expectedOutput": {
"IncidentCategory": "Software",
"IncidentSubcategory": "Email",
"AssigneeEmail": "savannah.kesich@example.com",
"ExecutionDetails": ""
},
"evalSetId": "32b51704-55cb-48af-a939-0d27931bda0a",
"createdAt": "2025-10-26T06:19:24.848Z",
"updatedAt": "2025-10-26T06:19:24.848Z"
},
{
"id": "97bc068e-8394-40c8-ad31-e0832e93f697",
"name": "Inquiry_Antivirus",
"inputs": {
"IncidentShortDescription": "Laptop slowing down",
"IncidentDescription": "I noticed my laptop is slowing down every time a security scan starts running, can you please look into this?"
},
"expectedOutput": {
"IncidentCategory": "Inquiry / Help",
"IncidentSubcategory": "Antivirus",
"AssigneeEmail": "paul.martin@example.com",
"ExecutionDetails": ""
},
"evalSetId": "32b51704-55cb-48af-a939-0d27931bda0a",
"createdAt": "2025-10-26T06:19:24.848Z",
"updatedAt": "2025-10-26T06:19:24.848Z"
},
{
"id": "754cf068-5e59-4436-8733-f92ebca7fa55",
"name": "Inquiry_ExternalApplication",
"inputs": {
"IncidentShortDescription": "CRM software crashes on launch",
"IncidentDescription": "Every time I try to open the CRM software, it crashes immediately. I’ve already tried reinstalling it."
},
"expectedOutput": {
"IncidentCategory": "Inquiry / Help",
"IncidentSubcategory": "External Application",
"AssigneeEmail": "paul.martin@example.com",
"ExecutionDetails": ""
},
"evalSetId": "32b51704-55cb-48af-a939-0d27931bda0a",
"createdAt": "2025-10-26T06:19:24.848Z",
"updatedAt": "2025-10-26T06:19:24.848Z"
},
{
"id": "adc73bed-75e6-45b3-801c-5919df70ec6d",
"name": "Hardware_Monitor",
"inputs": {
"IncidentShortDescription": "Display flickers occasionally",
"IncidentDescription": "My screen flickers randomly throughout the day. It’s not completely unusable, but it’s very distracting."
},
"expectedOutput": {
"IncidentCategory": "Hardware",
"IncidentSubcategory": "Monitor",
"AssigneeEmail": "aqib.mushtaq@example.com",
"ExecutionDetails": ""
},
"evalSetId": "32b51704-55cb-48af-a939-0d27931bda0a",
"createdAt": "2025-10-26T06:19:24.848Z",
"updatedAt": "2025-10-26T06:19:24.848Z"
},
{
"id": "3d139ef0-05df-488c-b56f-4165258900dc",
"name": "Network_DNS",
"inputs": {
"IncidentShortDescription": "System cannot reach certain websites",
"IncidentDescription": "Some websites are not loading on my system, but they work fine on my phone. I suspect there’s an issue with my network settings."
},
"expectedOutput": {
"IncidentCategory": "Network",
"IncidentSubcategory": "DNS",
"AssigneeEmail": "david.dan@example.com",
"ExecutionDetails": ""
},
"evalSetId": "32b51704-55cb-48af-a939-0d27931bda0a",
"createdAt": "2025-10-26T06:19:24.848Z",
"updatedAt": "2025-10-26T06:19:24.848Z"
},
{
"id": "9c7e9e2a-8e3f-453d-894d-d5ba12a80932",
"name": "Network_VPN",
"inputs": {
"IncidentShortDescription": "VPN connection drops frequently",
"IncidentDescription": "My VPN keeps disconnecting randomly, making it hard for me to work remotely."
},
"expectedOutput": {
"IncidentCategory": "Network",
"IncidentSubcategory": "VPN",
"AssigneeEmail": "bud.richman@example.com",
"ExecutionDetails": ""
},
"evalSetId": "32b51704-55cb-48af-a939-0d27931bda0a",
"createdAt": "2025-10-26T06:19:24.848Z",
"updatedAt": "2025-10-26T06:19:24.848Z"
},
{
"id": "3d114888-f0b0-400b-82f3-ed20aa300df3",
"name": "Software_OS",
"inputs": {
"IncidentShortDescription": "Windows update failure",
"IncidentDescription": "I tried to install the latest Windows update, but it keeps failing with error code 0x80070002"
},
"expectedOutput": {
"IncidentCategory": "Software",
"IncidentSubcategory": "Operating System",
"AssigneeEmail": "savannah.kesich@example.com",
"ExecutionDetails": ""
},
"evalSetId": "32b51704-55cb-48af-a939-0d27931bda0a",
"createdAt": "2025-10-26T06:19:24.848Z",
"updatedAt": "2025-10-26T06:19:24.848Z"
}
],
"modelSettings": [],
"createdAt": "2025-10-26T06:19:24.848Z",
"updatedAt": "2025-10-26T06:19:24.848Z"
}
Essentially, evaluations are test cases. With certain inputs, we expect certain outputs. Let's run the evaluation set and check the results. Evaluations are a good way to control quality of your prompt, and to have consistent results when changing or upgrading LLM models, or making sure that adjusting prompt to address one ussie didn't break some basic use case.
Anyway, our Agent's "Brain" is ready, it's time to add some "Hands" - we will do it in the next lesson!
Page updated
Report abuse