Skip to content

Commit 28a6d0a

Browse files
authored
Support Virtual-GenAI monitoring (#13745)
Adds a VIRTUAL_GENAI observability layer for monitoring Generative AI service calls detected by agent plugins (e.g., Spring AI). Tracks traffic, latency, success rate, token usage, time-to-first-token, and estimated cost. Architecture - Provider = Service, Model = Instance — reuses ServiceMeta and ServiceInstance sources - Two new scopes: GenAIProviderAccess (service-level) and GenAIModelAccess (instance-level) for GenAI-specific metrics - Cost stored as long amplified by 10^6 to work with SumMetrics New Module: gen-ai-analyzer - Extracts GenAI metrics from agent spans via semantic convention tags (gen_ai.*) - Prefix-based model→provider matching (e.g., gpt* → openai) - Estimated cost calculation from configurable per-model pricing Configuration - gen-ai-config.yml — provider/model pricing for OpenAI, Anthropic, Gemini, Mistral, DeepSeek, ByteDance, Zhipu AI, Alibaba, Tencent, Moonshot, MiniMax, and Ollama (template) OAL - virtual-gen-ai.oal with new grammar tokens for provider and model metrics aggregation UI - Dashboard templates for provider-level and model-level views - "GenAI" menu section placed before "Self Observability" in both docs/menu.yml and ui-initialized-templates/menu.yaml Testing - Unit tests covering config loading, provider matching, and cost estimation - E2E test with mock LLM controller Documentation - docs/en/setup/service-agent/virtual-genai.md — span contract, provider config, available metrics
1 parent 86b8c45 commit 28a6d0a

53 files changed

Lines changed: 3383 additions & 10 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/skywalking.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -627,6 +627,8 @@ jobs:
627627
config: test/e2e-v2/cases/zipkin/kafka/e2e.yaml
628628
- name: Zipkin BanyanDB
629629
config: test/e2e-v2/cases/zipkin/banyandb/e2e.yaml
630+
- name: Virtual GenAI
631+
config: test/e2e-v2/cases/virtual-genai/e2e.yaml
630632

631633
- name: Nginx
632634
config: test/e2e-v2/cases/nginx/e2e.yaml

apm-dist/src/main/assembly/binary.xml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@
7575
<include>log-mal-rules/**</include>
7676
<include>telegraf-rules/*</include>
7777
<include>cilium-rules/*</include>
78+
<include>gen-ai-config.yml</include>
7879
</includes>
7980
<outputDirectory>config</outputDirectory>
8081
</fileSet>

docs/en/changes/changes.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,7 @@
166166
* Update hierarchy rule documentation: `auto-matching-rules` in `hierarchy-definition.yml` no longer use Groovy scripts. Rules now use a dedicated expression grammar supporting property access, String methods, if/else, comparisons, and logical operators. All shipped rules are fully compatible.
167167
* Activate `otlp-traces` handler in `receiver-otel` by default.
168168
* Update Istio E2E test versions: remove EOL 1.20.0, add 1.25.0–1.29.0 for ALS/Metrics/Ambient tests. Update Rover with Istio Process test from 1.15.0 to 1.28.0 with Kubernetes 1.28.
169+
* Support Virtual-GenAI monitoring.
169170

170171
#### UI
171172
* Fix the missing icon in new native trace view.
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Virtual GenAI
2+
3+
Virtual GenAI represents the Generative AI service nodes detected by [server agents' plugins](server-agents.md). The performance
4+
metrics of the GenAI operations are from the GenAI client-side perspective.
5+
6+
For example, a Spring AI plugin in the Java agent could detect the latency of a chat completion request.
7+
As a result, SkyWalking would show traffic, latency, success rate, token usage (input/output), and estimated cost in the GenAI dashboard.
8+
9+
## Span Contract
10+
11+
The GenAI operation span should have the following properties:
12+
- It is an **Exit** span
13+
- **Span's layer == GENAI**
14+
- Tag key = `gen_ai.provider.name`, value = The Generative AI provider, e.g. openai, anthropic, ollama
15+
- Tag key = `gen_ai.response.model`, value = The name of the GenAI model, e.g. gpt-4o, claude-3-5-sonnet
16+
- Tag key = `gen_ai.usage.input_tokens`, value = The number of tokens used in the GenAI input (prompt)
17+
- Tag key = `gen_ai.usage.output_tokens`, value = The number of tokens used in the GenAI response (completion)
18+
- Tag key = `gen_ai.server.time_to_first_token`, value = The duration in milliseconds until the first token is received (streaming requests only)
19+
- If the GenAI service is a remote API (e.g. OpenAI), the span's peer should be the network address (IP or domain) of the GenAI server.
20+
21+
## Provider Configuration
22+
23+
SkyWalking uses `gen-ai-config.yml` to map model names to providers and configure cost estimation.
24+
25+
When the `gen_ai.provider.name` tag is present in the span, it is used directly. Otherwise, SkyWalking matches the model name
26+
against `prefix-match` rules to identify the provider. For example, a model name starting with `gpt` is mapped to `openai`.
27+
28+
To configure cost estimation, add `models` with pricing under the provider:
29+
30+
31+
```yaml
32+
providers:
33+
- provider: openai
34+
prefix-match:
35+
- gpt
36+
models:
37+
- name: gpt-4o
38+
input-estimated-cost-per-m: 2.5 # estimated cost per 1,000,000 input tokens
39+
output-estimated-cost-per-m: 10 # estimated cost per 1,000,000 output tokens
40+
```
41+
42+
## Metrics
43+
44+
The following metrics are available at the **provider** (service) level:
45+
- `gen_ai_provider_cpm` - Calls per minute
46+
- `gen_ai_provider_sla` - Success rate
47+
- `gen_ai_provider_resp_time` - Average response time
48+
- `gen_ai_provider_latency_percentile` - Latency percentiles
49+
- `gen_ai_provider_input_tokens_sum / avg` - Input token usage
50+
- `gen_ai_provider_output_tokens_sum / avg` - Output token usage
51+
- `gen_ai_provider_total_estimated_cost / avg_estimated_cost` - Estimated cost
52+
53+
The following metrics are available at the **model** (service instance) level:
54+
- `gen_ai_model_call_cpm` - Calls per minute
55+
- `gen_ai_model_sla` - Success rate
56+
- `gen_ai_model_latency_avg / percentile` - Latency
57+
- `gen_ai_model_ttft_avg / percentile` - Time to first token (streaming only)
58+
- `gen_ai_model_input_tokens_sum / avg` - Input token usage
59+
- `gen_ai_model_output_tokens_sum / avg` - Output token usage
60+
- `gen_ai_model_total_estimated_cost / avg_estimated_cost` - Estimated cost
61+
62+
## Requirement
63+
`SkyWalking Java Agent` version >= 9.7

docs/menu.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,10 @@ catalog:
148148
catalog:
149149
- name: "Flink"
150150
path: "/en/setup/backend/backend-flink-monitoring"
151+
- name: "GenAI"
152+
catalog:
153+
- name: "Virtual GenAI"
154+
path: "/en/setup/service-agent/virtual-genai"
151155
- name: "Self Observability"
152156
catalog:
153157
- name: "OAP self telemetry"

oap-server/analyzer/agent-analyzer/pom.xml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,11 @@
4343
<artifactId>meter-analyzer</artifactId>
4444
<version>${project.version}</version>
4545
</dependency>
46+
<dependency>
47+
<groupId>org.apache.skywalking</groupId>
48+
<artifactId>gen-ai-analyzer</artifactId>
49+
<version>${project.version}</version>
50+
</dependency>
4651
<dependency>
4752
<groupId>org.apache.skywalking</groupId>
4853
<artifactId>server-testing</artifactId>

oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/VirtualServiceAnalysisListener.java

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,16 @@
2020

2121
import java.util.Arrays;
2222
import java.util.List;
23+
2324
import lombok.RequiredArgsConstructor;
2425
import org.apache.skywalking.apm.network.language.agent.v3.SegmentObject;
2526
import org.apache.skywalking.apm.network.language.agent.v3.SpanObject;
27+
import org.apache.skywalking.oap.analyzer.genai.module.GenAIAnalyzerModule;
28+
import org.apache.skywalking.oap.analyzer.genai.service.IGenAIMeterAnalyzerService;
2629
import org.apache.skywalking.oap.server.analyzer.provider.AnalyzerModuleConfig;
2730
import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.vservice.VirtualCacheProcessor;
2831
import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.vservice.VirtualDatabaseProcessor;
32+
import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.vservice.VirtualGenAIProcessor;
2933
import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.vservice.VirtualMQProcessor;
3034
import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.vservice.VirtualServiceProcessor;
3135
import org.apache.skywalking.oap.server.core.CoreModule;
@@ -71,23 +75,28 @@ public void parseEntry(final SpanObject span, final SegmentObject segmentObject)
7175
public static class Factory implements AnalysisListenerFactory {
7276
private final SourceReceiver sourceReceiver;
7377
private final NamingControl namingControl;
78+
private final IGenAIMeterAnalyzerService genAIMeterAnalyzerService;
7479

7580
public Factory(ModuleManager moduleManager) {
7681
this.sourceReceiver = moduleManager.find(CoreModule.NAME).provider().getService(SourceReceiver.class);
7782
this.namingControl = moduleManager.find(CoreModule.NAME)
7883
.provider()
7984
.getService(NamingControl.class);
85+
this.genAIMeterAnalyzerService = moduleManager.find(GenAIAnalyzerModule.NAME)
86+
.provider()
87+
.getService(IGenAIMeterAnalyzerService.class);
8088
}
8189

8290
@Override
8391
public AnalysisListener create(ModuleManager moduleManager, AnalyzerModuleConfig config) {
8492
return new VirtualServiceAnalysisListener(
85-
sourceReceiver,
86-
Arrays.asList(
87-
new VirtualCacheProcessor(namingControl, config),
88-
new VirtualDatabaseProcessor(namingControl, config),
89-
new VirtualMQProcessor(namingControl)
90-
)
93+
sourceReceiver,
94+
Arrays.asList(
95+
new VirtualCacheProcessor(namingControl, config),
96+
new VirtualDatabaseProcessor(namingControl, config),
97+
new VirtualMQProcessor(namingControl),
98+
new VirtualGenAIProcessor(namingControl, genAIMeterAnalyzerService)
99+
)
91100
);
92101
}
93102
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.vservice;
19+
20+
import lombok.RequiredArgsConstructor;
21+
import org.apache.skywalking.apm.network.language.agent.v3.SegmentObject;
22+
import org.apache.skywalking.apm.network.language.agent.v3.SpanLayer;
23+
import org.apache.skywalking.apm.network.language.agent.v3.SpanObject;
24+
import org.apache.skywalking.oap.analyzer.genai.service.IGenAIMeterAnalyzerService;
25+
import org.apache.skywalking.oap.server.core.analysis.Layer;
26+
import org.apache.skywalking.oap.server.core.config.NamingControl;
27+
import org.apache.skywalking.oap.server.core.source.GenAIMetrics;
28+
import org.apache.skywalking.oap.server.core.source.GenAIModelAccess;
29+
import org.apache.skywalking.oap.server.core.source.GenAIProviderAccess;
30+
import org.apache.skywalking.oap.server.core.source.ServiceInstance;
31+
import org.apache.skywalking.oap.server.core.source.ServiceMeta;
32+
import org.apache.skywalking.oap.server.core.source.Source;
33+
34+
import java.util.ArrayList;
35+
import java.util.List;
36+
import java.util.function.Consumer;
37+
38+
@RequiredArgsConstructor
39+
public class VirtualGenAIProcessor implements VirtualServiceProcessor {
40+
41+
private final NamingControl namingControl;
42+
43+
private final IGenAIMeterAnalyzerService meterAnalyzerService;
44+
45+
private final List<Source> recordList = new ArrayList<>();
46+
47+
@Override
48+
public void prepareVSIfNecessary(SpanObject span, SegmentObject segmentObject) {
49+
if (span.getSpanLayer() != SpanLayer.GenAI) {
50+
return;
51+
}
52+
53+
GenAIMetrics metrics = meterAnalyzerService.extractMetricsFromSWSpan(span, segmentObject);
54+
if (metrics == null) {
55+
return;
56+
}
57+
58+
recordList.add(toServiceMeta(metrics));
59+
recordList.add(toInstance(metrics));
60+
recordList.add(toProviderAccess(metrics));
61+
recordList.add(toModelAccess(metrics));
62+
}
63+
64+
private ServiceMeta toServiceMeta(GenAIMetrics metrics) {
65+
ServiceMeta service = new ServiceMeta();
66+
service.setName(namingControl.formatServiceName(metrics.getProviderName()));
67+
service.setLayer(Layer.VIRTUAL_GENAI);
68+
service.setTimeBucket(metrics.getTimeBucket());
69+
return service;
70+
}
71+
72+
private Source toInstance(GenAIMetrics metrics) {
73+
ServiceInstance instance = new ServiceInstance();
74+
instance.setTimeBucket(metrics.getTimeBucket());
75+
instance.setName(namingControl.formatInstanceName(metrics.getModelName()));
76+
instance.setServiceLayer(Layer.VIRTUAL_GENAI);
77+
instance.setServiceName(metrics.getProviderName());
78+
return instance;
79+
}
80+
81+
private GenAIProviderAccess toProviderAccess(GenAIMetrics metrics) {
82+
GenAIProviderAccess source = new GenAIProviderAccess();
83+
source.setName(namingControl.formatServiceName(metrics.getProviderName()));
84+
source.setInputTokens(metrics.getInputTokens());
85+
source.setOutputTokens(metrics.getOutputTokens());
86+
source.setTotalEstimatedCost(metrics.getTotalEstimatedCost());
87+
source.setLatency(metrics.getLatency());
88+
source.setStatus(metrics.isStatus());
89+
source.setTimeBucket(metrics.getTimeBucket());
90+
return source;
91+
}
92+
93+
private GenAIModelAccess toModelAccess(GenAIMetrics metrics) {
94+
GenAIModelAccess source = new GenAIModelAccess();
95+
source.setServiceName(namingControl.formatServiceName(metrics.getProviderName()));
96+
source.setModelName(namingControl.formatInstanceName(metrics.getModelName()));
97+
source.setInputTokens(metrics.getInputTokens());
98+
source.setOutputTokens(metrics.getOutputTokens());
99+
source.setTotalEstimatedCost(metrics.getTotalEstimatedCost());
100+
source.setTimeToFirstToken(metrics.getTimeToFirstToken());
101+
source.setLatency(metrics.getLatency());
102+
source.setStatus(metrics.isStatus());
103+
source.setTimeBucket(metrics.getTimeBucket());
104+
return source;
105+
}
106+
107+
@Override
108+
public void emitTo(Consumer<Source> consumer) {
109+
for (Source source : recordList) {
110+
if (source != null) {
111+
consumer.accept(source);
112+
}
113+
}
114+
}
115+
}
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Licensed to the Apache Software Foundation (ASF) under one or more
4+
~ contributor license agreements. See the NOTICE file distributed with
5+
~ this work for additional information regarding copyright ownership.
6+
~ The ASF licenses this file to You under the Apache License, Version 2.0
7+
~ (the "License"); you may not use this file except in compliance with
8+
~ the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing, software
13+
~ distributed under the License is distributed on an "AS IS" BASIS,
14+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
~ See the License for the specific language governing permissions and
16+
~ limitations under the License.
17+
~
18+
-->
19+
20+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
21+
<parent>
22+
<artifactId>analyzer</artifactId>
23+
<groupId>org.apache.skywalking</groupId>
24+
<version>${revision}</version>
25+
</parent>
26+
<modelVersion>4.0.0</modelVersion>
27+
28+
<artifactId>gen-ai-analyzer</artifactId>
29+
30+
<dependencies>
31+
<dependency>
32+
<groupId>org.apache.skywalking</groupId>
33+
<artifactId>server-core</artifactId>
34+
<version>${project.version}</version>
35+
</dependency>
36+
</dependencies>
37+
</project>

0 commit comments

Comments
 (0)