In this section, we primarily analyze the external manifestations of bug reports in Agent Frameworks, namely symptoms.
In our study, we classified the symptoms of 5,669 Agent Framework bug reports and ultimately summarized seven categories of symptoms to describe the specific behaviors of these bugs during execution, construction, interaction, or output.
In addition, we also compare the symptom of the Agent Framework with that of the DL framework (Compared with DL Framework).
S1. Crash
This symptom refers to a visible crash of an agent framework during operation, where the program fails to start, terminates unexpectedly at runtime, or becomes unable to execute its core functionality.
For example:
In AutoGen5979, when MultimodalWebSurfer calls Ollama, it passes an unsupported `tool_choice` parameter, triggering a `TypeError`.
S2. Functional Error
This symptom refers to observable inconsistencies between the expected behavior and the actual runtime behavior of an agent framework.
For example:
In CrewAI4347, as reported in this issue, Crew streaming can output chunks correctly; however, the `task_name` and `task_id` fields are missing.
S3. Build Failure
This symptom refers to situations where an agent framework or application cannot successfully enter a runnable state during deployment, installation, or startup.
For example:
In MetaGPT1132, when installing the release version of MetaGPT on Windows, a dependency `pwd` that is only supported on Linux is introduced, causing the framework to become unable to enter a runnable state before installation or startup.
S4. Poor Performance
This symptom refers to observable declines in runtime efficiency during agent execution.
For example:
In LangChain8391, `chain.arun()` is expected to execute summarization in parallel, but in practice it makes sequential `await` calls one by one, resulting in the asynchronous version being no faster than `chain()`.
S5. Hang
This symptom refers to a state in which an agent or execution graph becomes stalled without triggering explicit errors or termination signals.
For example:
In LangGraph5150, when `store.search()` is invoked before the database has been `setup()`, it calls `_cursor()`, which holds `self.lock` and subsequently calls `setup()`, where `setup()` attempts to acquire the same lock again, resulting in a deadlock.
S6. Unreported
This symptom refers to situations where an agent framework accepts inputs or configuration changes but does not react as expected, resulting in unresponsiveness or ignored functionality without explicit error reports.
For example:
In LangChain21293, after ChatOllama invokes the local Ollama, it fails to return the expected content, and no explicit error message is provided.
S7. Display Anomaly
This symptom refers to observable inconsistencies or errors in user-facing displays within an agent framework, including interface components, visualization panels, log outputs, or documentation pages.
For example:
In AutoGen6513, when the component library contains excessive content, the page scrolling region is incorrect, causing the entire content area to be scrolled.