I want to implement Gen AI observability for the Agent project ZeroClaw.
Things to consider when building observability:
- Configuring the data processing pipeline: where to initialize the SIGNAL provider, how to export, and where to export to
- Where to instrument
- How to send telemetry out
I believe the most needed requirement for ZeroClaw in terms of observability is to be able to intuitively see how the agent calls tools and interacts with LLM Providers, similar to the effect of LangGraph's LangSmith platform. I found a platform called Langfuse, which is open-source and able to structurally view all the operations that occurred during a single pass.
Determining where to instrument
I have looked at the Monkey Patching instrumentation library implemented by OTel for OpenAI. The instrumentation point is the request method of the OpenAI SDK. So does ZeroClaw have such a method?
ZeroClaw has a directory called provider, which contains methods for making requests to some vendors, including OpenAI. Thanks to OpenAI compatibility, ZeroClaw can use domestic LLM Providers.
The instrumentation point is initially determined to be here at Provider.
Previously, I thought instrumenting an Agent project would involve many aspects. Now it seems that we only need to instrument the methods related to LLM requests.
If the Agent project uses a popular LLM request proxy library such as LiteLLM, which already has built-in instrumentation, you can directly use its observability capabilities.
In short, we just need to observe at the points of LLM interaction.