Skip to content

Reducing memory allocation from client-side stats#10705

Open
dougqh wants to merge 1 commit intomasterfrom
dougqh/client-side-stats-overhead
Open

Reducing memory allocation from client-side stats#10705
dougqh wants to merge 1 commit intomasterfrom
dougqh/client-side-stats-overhead

Conversation

@dougqh
Copy link
Contributor

@dougqh dougqh commented Feb 27, 2026 •
edited
Loading

What Does This Do

Introduces DDCache-s around UTF8BytesString construction

Motivation

UTF8BytesString are advantageous for serialization, but that only applies to the MetricKey instance that is actually serialized

Most MetricKey instances are only created to do a look-up into the map. so the UTF8BytesString creation was just extra work

This change reduces GC time by 7% in span creation stress test.
This change reduces impact on application throughput by 5-20% depending on heap size.

Additional Notes

This change is intended as a quick fix. I think there's still a lot of room for improvement by restructuring the code further, but that is left for a future PR.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

@dougqh
Reducing memory allocation from client-side stats
b453f26 
Introduced DDCache-s around UTF8BytesString construction

UTF8BytesString are advantageous for serialization, but that only applies to the key instance that is actually serialized

Most key instances here are being created to do a look-up into the map. so the UTF8BytesString creation was extra work.

This change is intended as a quick fix.  I think there's still a lot of room for improvement by restructuring the code further, but that is left for a future PR.

As is this changer reduces the impact on application throughput by 5-20% depending on the heap size.
@dougqh dougqh requested a review from a team as a code owner February 27, 2026 21:19
@dougqh dougqh requested a review from amarziali February 27, 2026 21:19
@dougqh dougqh added type: enhancement Enhancements and improvements tag: performance Performance related changes comp: metrics Metrics labels Feb 27, 2026
@pr-commenter
Copy link

pr-commenter bot commented Feb 27, 2026

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/client-side-stats-overhead
git_commit_date 1772200579 1772226956
git_commit_sha c6896b7 b453f26
release_version 1.60.0-SNAPSHOT~c6896b7cf7 1.60.0-SNAPSHOT~b453f26cf9
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1772228811 1772228811
ci_job_id 1464390125 1464390125
ci_pipeline_id 99550991 99550991
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-5e90c6kf 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-5e90c6kf 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 11 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~b453f26cf9, baseline=1.60.0-SNAPSHOT~c6896b7cf7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.069 s) : 0, 1068901
Total [baseline] (8.771 s) : 0, 8771020
Agent [candidate] (1.072 s) : 0, 1072189
Total [candidate] (8.735 s) : 0, 8735481
section iast
Agent [baseline] (1.229 s) : 0, 1229295
Total [baseline] (9.366 s) : 0, 9366341
Agent [candidate] (1.236 s) : 0, 1235941
Total [candidate] (9.366 s) : 0, 9365570
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.069 s -
Agent iast 1.229 s 160.394 ms (15.0%)
Total tracing 8.771 s -
Total iast 9.366 s 595.321 ms (6.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.072 s -
Agent iast 1.236 s 163.752 ms (15.3%)
Total tracing 8.735 s -
Total iast 9.366 s 630.089 ms (7.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~b453f26cf9, baseline=1.60.0-SNAPSHOT~c6896b7cf7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.194 ms) : 0, 1194
crashtracking [candidate] (1.196 ms) : 0, 1196
BytebuddyAgent [baseline] (630.203 ms) : 0, 630203
BytebuddyAgent [candidate] (632.468 ms) : 0, 632468
AgentMeter [baseline] (29.23 ms) : 0, 29230
AgentMeter [candidate] (29.199 ms) : 0, 29199
GlobalTracer [baseline] (258.186 ms) : 0, 258186
GlobalTracer [candidate] (258.826 ms) : 0, 258826
AppSec [baseline] (33.201 ms) : 0, 33201
AppSec [candidate] (33.336 ms) : 0, 33336
Debugger [baseline] (61.558 ms) : 0, 61558
Debugger [candidate] (63.47 ms) : 0, 63470
Remote Config [baseline] (621.228 µs) : 0, 621
Remote Config [candidate] (638.796 µs) : 0, 639
Telemetry [baseline] (14.645 ms) : 0, 14645
Telemetry [candidate] (11.537 ms) : 0, 11537
Flare Poller [baseline] (3.799 ms) : 0, 3799
Flare Poller [candidate] (5.354 ms) : 0, 5354
section iast
crashtracking [baseline] (1.189 ms) : 0, 1189
crashtracking [candidate] (1.193 ms) : 0, 1193
BytebuddyAgent [baseline] (793.813 ms) : 0, 793813
BytebuddyAgent [candidate] (799.872 ms) : 0, 799872
AgentMeter [baseline] (11.317 ms) : 0, 11317
AgentMeter [candidate] (11.527 ms) : 0, 11527
GlobalTracer [baseline] (247.132 ms) : 0, 247132
GlobalTracer [candidate] (248.01 ms) : 0, 248010
IAST [baseline] (27.0 ms) : 0, 27000
IAST [candidate] (27.036 ms) : 0, 27036
AppSec [baseline] (34.279 ms) : 0, 34279
AppSec [candidate] (32.504 ms) : 0, 32504
Debugger [baseline] (65.918 ms) : 0, 65918
Debugger [candidate] (67.191 ms) : 0, 67191
Remote Config [baseline] (543.116 µs) : 0, 543
Remote Config [candidate] (534.628 µs) : 0, 535
Telemetry [baseline] (8.656 ms) : 0, 8656
Telemetry [candidate] (8.552 ms) : 0, 8552
Flare Poller [baseline] (3.474 ms) : 0, 3474
Flare Poller [candidate] (3.449 ms) : 0, 3449
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~b453f26cf9, baseline=1.60.0-SNAPSHOT~c6896b7cf7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.062 s) : 0, 1062072
Total [baseline] (10.814 s) : 0, 10814169
Agent [candidate] (1.065 s) : 0, 1064579
Total [candidate] (10.915 s) : 0, 10915488
section appsec
Agent [baseline] (1.244 s) : 0, 1244444
Total [baseline] (10.993 s) : 0, 10992653
Agent [candidate] (1.254 s) : 0, 1254101
Total [candidate] (11.043 s) : 0, 11042757
section iast
Agent [baseline] (1.233 s) : 0, 1232788
Total [baseline] (11.202 s) : 0, 11201876
Agent [candidate] (1.232 s) : 0, 1232383
Total [candidate] (11.159 s) : 0, 11158830
section profiling
Agent [baseline] (1.196 s) : 0, 1196209
Total [baseline] (11.0 s) : 0, 11000129
Agent [candidate] (1.192 s) : 0, 1192168
Total [candidate] (11.002 s) : 0, 11001923
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.062 s -
Agent appsec 1.244 s 182.372 ms (17.2%)
Agent iast 1.233 s 170.716 ms (16.1%)
Agent profiling 1.196 s 134.137 ms (12.6%)
Total tracing 10.814 s -
Total appsec 10.993 s 178.483 ms (1.7%)
Total iast 11.202 s 387.706 ms (3.6%)
Total profiling 11.0 s 185.96 ms (1.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.065 s -
Agent appsec 1.254 s 189.523 ms (17.8%)
Agent iast 1.232 s 167.804 ms (15.8%)
Agent profiling 1.192 s 127.589 ms (12.0%)
Total tracing 10.915 s -
Total appsec 11.043 s 127.269 ms (1.2%)
Total iast 11.159 s 243.342 ms (2.2%)
Total profiling 11.002 s 86.435 ms (0.8%) 
gantt
    title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~b453f26cf9, baseline=1.60.0-SNAPSHOT~c6896b7cf7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.198 ms) : 0, 1198
crashtracking [candidate] (1.189 ms) : 0, 1189
BytebuddyAgent [baseline] (626.891 ms) : 0, 626891
BytebuddyAgent [candidate] (628.988 ms) : 0, 628988
AgentMeter [baseline] (28.977 ms) : 0, 28977
AgentMeter [candidate] (29.025 ms) : 0, 29025
GlobalTracer [baseline] (256.583 ms) : 0, 256583
GlobalTracer [candidate] (257.236 ms) : 0, 257236
AppSec [baseline] (32.853 ms) : 0, 32853
AppSec [candidate] (33.05 ms) : 0, 33050
Debugger [baseline] (62.891 ms) : 0, 62891
Debugger [candidate] (63.731 ms) : 0, 63731
Remote Config [baseline] (609.206 µs) : 0, 609
Remote Config [candidate] (618.118 µs) : 0, 618
Telemetry [baseline] (10.678 ms) : 0, 10678
Telemetry [candidate] (9.195 ms) : 0, 9195
Flare Poller [baseline] (5.307 ms) : 0, 5307
Flare Poller [candidate] (5.394 ms) : 0, 5394
section appsec
crashtracking [baseline] (1.186 ms) : 0, 1186
crashtracking [candidate] (1.187 ms) : 0, 1187
BytebuddyAgent [baseline] (660.739 ms) : 0, 660739
BytebuddyAgent [candidate] (666.077 ms) : 0, 666077
AgentMeter [baseline] (12.0 ms) : 0, 12000
AgentMeter [candidate] (12.1 ms) : 0, 12100
GlobalTracer [baseline] (259.492 ms) : 0, 259492
GlobalTracer [candidate] (260.972 ms) : 0, 260972
IAST [baseline] (25.682 ms) : 0, 25682
IAST [candidate] (25.92 ms) : 0, 25920
AppSec [baseline] (168.462 ms) : 0, 168462
AppSec [candidate] (169.994 ms) : 0, 169994
Debugger [baseline] (66.975 ms) : 0, 66975
Debugger [candidate] (67.554 ms) : 0, 67554
Remote Config [baseline] (686.685 µs) : 0, 687
Remote Config [candidate] (692.498 µs) : 0, 692
Telemetry [baseline] (9.501 ms) : 0, 9501
Telemetry [candidate] (9.671 ms) : 0, 9671
Flare Poller [baseline] (3.621 ms) : 0, 3621
Flare Poller [candidate] (3.733 ms) : 0, 3733
section iast
crashtracking [baseline] (1.189 ms) : 0, 1189
crashtracking [candidate] (1.183 ms) : 0, 1183
BytebuddyAgent [baseline] (796.706 ms) : 0, 796706
BytebuddyAgent [candidate] (795.473 ms) : 0, 795473
AgentMeter [baseline] (11.319 ms) : 0, 11319
AgentMeter [candidate] (11.353 ms) : 0, 11353
GlobalTracer [baseline] (247.612 ms) : 0, 247612
GlobalTracer [candidate] (247.666 ms) : 0, 247666
IAST [baseline] (27.03 ms) : 0, 27030
IAST [candidate] (27.102 ms) : 0, 27102
AppSec [baseline] (32.955 ms) : 0, 32955
AppSec [candidate] (32.199 ms) : 0, 32199
Debugger [baseline] (67.424 ms) : 0, 67424
Debugger [candidate] (68.763 ms) : 0, 68763
Remote Config [baseline] (551.498 µs) : 0, 551
Remote Config [candidate] (534.903 µs) : 0, 535
Telemetry [baseline] (8.563 ms) : 0, 8563
Telemetry [candidate] (8.641 ms) : 0, 8641
Flare Poller [baseline] (3.453 ms) : 0, 3453
Flare Poller [candidate] (3.461 ms) : 0, 3461
section profiling
crashtracking [baseline] (1.169 ms) : 0, 1169
crashtracking [candidate] (1.17 ms) : 0, 1170
BytebuddyAgent [baseline] (685.941 ms) : 0, 685941
BytebuddyAgent [candidate] (683.172 ms) : 0, 683172
AgentMeter [baseline] (8.574 ms) : 0, 8574
AgentMeter [candidate] (8.611 ms) : 0, 8611
GlobalTracer [baseline] (216.917 ms) : 0, 216917
GlobalTracer [candidate] (216.457 ms) : 0, 216457
AppSec [baseline] (32.767 ms) : 0, 32767
AppSec [candidate] (32.522 ms) : 0, 32522
Debugger [baseline] (67.373 ms) : 0, 67373
Debugger [candidate] (66.576 ms) : 0, 66576
Remote Config [baseline] (632.891 µs) : 0, 633
Remote Config [candidate] (625.958 µs) : 0, 626
Telemetry [baseline] (8.847 ms) : 0, 8847
Telemetry [candidate] (8.836 ms) : 0, 8836
Flare Poller [baseline] (3.794 ms) : 0, 3794
Flare Poller [candidate] (3.804 ms) : 0, 3804
ProfilingAgent [baseline] (99.233 ms) : 0, 99233
ProfilingAgent [candidate] (99.391 ms) : 0, 99391
Profiling [baseline] (99.813 ms) : 0, 99813
Profiling [candidate] (99.967 ms) : 0, 99967
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/client-side-stats-overhead
git_commit_date 1772200579 1772226956
git_commit_sha c6896b7 b453f26
release_version 1.60.0-SNAPSHOT~c6896b7cf7 1.60.0-SNAPSHOT~b453f26cf9
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1772229304 1772229304
ci_job_id 1464390128 1464390128
ci_pipeline_id 99550991 99550991
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-zk5s7akg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-zk5s7akg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 1 performance regressions! Performance is the same for 17 metrics, 16 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:profiling:high_load worse
[+54.904µs; +162.543µs] or [+3.514%; +10.403%]		 unstable
[+274.502µs; +1014.035µs] or [+6.534%; +24.138%]		 unstable
[-441.735op/s; +26.423op/s] or [-18.458%; +1.104%]		 1.671ms 4.845ms 2185.500op/s 1.563ms 4.201ms 2393.156op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load better
[-316.957µs; -210.668µs] or [-10.782%; -7.167%]		 better
[-575.764µs; -181.438µs] or [-7.108%; -2.240%]		 unstable
[-31.737op/s; +210.799op/s] or [-2.545%; +16.904%]		 2.676ms 7.722ms 1336.594op/s 2.940ms 8.100ms 1247.062op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~b453f26cf9, baseline=1.60.0-SNAPSHOT~c6896b7cf7
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.205 ms) : 18018, 18392
.   : milestone, 18205,
appsec (18.824 ms) : 18633, 19015
.   : milestone, 18824,
code_origins (17.503 ms) : 17329, 17677
.   : milestone, 17503,
iast (17.616 ms) : 17439, 17792
.   : milestone, 17616,
profiling (18.683 ms) : 18495, 18871
.   : milestone, 18683,
tracing (17.622 ms) : 17448, 17796
.   : milestone, 17622,
section candidate
no_agent (17.467 ms) : 17291, 17643
.   : milestone, 17467,
appsec (18.583 ms) : 18392, 18773
.   : milestone, 18583,
code_origins (17.392 ms) : 17219, 17564
.   : milestone, 17392,
iast (17.434 ms) : 17259, 17608
.   : milestone, 17434,
profiling (18.489 ms) : 18306, 18672
.   : milestone, 18489,
tracing (17.407 ms) : 17234, 17579
.   : milestone, 17407,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.205 ms [18.018 ms, 18.392 ms] -
appsec 18.824 ms [18.633 ms, 19.015 ms] 619.539 µs (3.4%)
code_origins 17.503 ms [17.329 ms, 17.677 ms] -701.827 µs (-3.9%)
iast 17.616 ms [17.439 ms, 17.792 ms] -589.044 µs (-3.2%)
profiling 18.683 ms [18.495 ms, 18.871 ms] 477.948 µs (2.6%)
tracing 17.622 ms [17.448 ms, 17.796 ms] -582.985 µs (-3.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 17.467 ms [17.291 ms, 17.643 ms] -
appsec 18.583 ms [18.392 ms, 18.773 ms] 1.116 ms (6.4%)
code_origins 17.392 ms [17.219 ms, 17.564 ms] -75.253 µs (-0.4%)
iast 17.434 ms [17.259 ms, 17.608 ms] -33.42 µs (-0.2%)
profiling 18.489 ms [18.306 ms, 18.672 ms] 1.022 ms (5.9%)
tracing 17.407 ms [17.234 ms, 17.579 ms] -60.076 µs (-0.3%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~b453f26cf9, baseline=1.60.0-SNAPSHOT~c6896b7cf7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.185 ms) : 1174, 1197
.   : milestone, 1185,
iast (3.179 ms) : 3137, 3220
.   : milestone, 3179,
iast_FULL (5.717 ms) : 5660, 5774
.   : milestone, 5717,
iast_GLOBAL (3.679 ms) : 3618, 3740
.   : milestone, 3679,
profiling (1.882 ms) : 1865, 1899
.   : milestone, 1882,
tracing (1.812 ms) : 1796, 1828
.   : milestone, 1812,
section candidate
no_agent (1.186 ms) : 1175, 1198
.   : milestone, 1186,
iast (3.163 ms) : 3121, 3204
.   : milestone, 3163,
iast_FULL (5.769 ms) : 5712, 5827
.   : milestone, 5769,
iast_GLOBAL (3.428 ms) : 3379, 3477
.   : milestone, 3428,
profiling (2.068 ms) : 2050, 2086
.   : milestone, 2068,
tracing (1.844 ms) : 1828, 1860
.   : milestone, 1844,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.185 ms [1.174 ms, 1.197 ms] -
iast 3.179 ms [3.137 ms, 3.22 ms] 1.993 ms (168.1%)
iast_FULL 5.717 ms [5.66 ms, 5.774 ms] 4.532 ms (382.3%)
iast_GLOBAL 3.679 ms [3.618 ms, 3.74 ms] 2.493 ms (210.3%)
profiling 1.882 ms [1.865 ms, 1.899 ms] 696.475 µs (58.8%)
tracing 1.812 ms [1.796 ms, 1.828 ms] 626.52 µs (52.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.186 ms [1.175 ms, 1.198 ms] -
iast 3.163 ms [3.121 ms, 3.204 ms] 1.976 ms (166.6%)
iast_FULL 5.769 ms [5.712 ms, 5.827 ms] 4.583 ms (386.3%)
iast_GLOBAL 3.428 ms [3.379 ms, 3.477 ms] 2.241 ms (188.9%)
profiling 2.068 ms [2.05 ms, 2.086 ms] 881.658 µs (74.3%)
tracing 1.844 ms [1.828 ms, 1.86 ms] 657.344 µs (55.4%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/client-side-stats-overhead
git_commit_date 1772200579 1772226956
git_commit_sha c6896b7 b453f26
release_version 1.60.0-SNAPSHOT~c6896b7cf7 1.60.0-SNAPSHOT~b453f26cf9
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1772228955 1772228955
ci_job_id 1464390130 1464390130
ci_pipeline_id 99550991 99550991
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-5hqri8jq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-5hqri8jq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~b453f26cf9, baseline=1.60.0-SNAPSHOT~c6896b7cf7
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.885 s) : 14885000, 14885000
.   : milestone, 14885000,
appsec (15.018 s) : 15018000, 15018000
.   : milestone, 15018000,
iast (18.242 s) : 18242000, 18242000
.   : milestone, 18242000,
iast_GLOBAL (17.705 s) : 17705000, 17705000
.   : milestone, 17705000,
profiling (14.935 s) : 14935000, 14935000
.   : milestone, 14935000,
tracing (14.678 s) : 14678000, 14678000
.   : milestone, 14678000,
section candidate
no_agent (15.518 s) : 15518000, 15518000
.   : milestone, 15518000,
appsec (14.57 s) : 14570000, 14570000
.   : milestone, 14570000,
iast (18.569 s) : 18569000, 18569000
.   : milestone, 18569000,
iast_GLOBAL (17.843 s) : 17843000, 17843000
.   : milestone, 17843000,
profiling (14.854 s) : 14854000, 14854000
.   : milestone, 14854000,
tracing (14.996 s) : 14996000, 14996000
.   : milestone, 14996000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.885 s [14.885 s, 14.885 s] -
appsec 15.018 s [15.018 s, 15.018 s] 133.0 ms (0.9%)
iast 18.242 s [18.242 s, 18.242 s] 3.357 s (22.6%)
iast_GLOBAL 17.705 s [17.705 s, 17.705 s] 2.82 s (18.9%)
profiling 14.935 s [14.935 s, 14.935 s] 50.0 ms (0.3%)
tracing 14.678 s [14.678 s, 14.678 s] -207.0 ms (-1.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.518 s [15.518 s, 15.518 s] -
appsec 14.57 s [14.57 s, 14.57 s] -948.0 ms (-6.1%)
iast 18.569 s [18.569 s, 18.569 s] 3.051 s (19.7%)
iast_GLOBAL 17.843 s [17.843 s, 17.843 s] 2.325 s (15.0%)
profiling 14.854 s [14.854 s, 14.854 s] -664.0 ms (-4.3%)
tracing 14.996 s [14.996 s, 14.996 s] -522.0 ms (-3.4%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~b453f26cf9, baseline=1.60.0-SNAPSHOT~c6896b7cf7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.467 ms) : 1456, 1479
.   : milestone, 1467,
appsec (3.785 ms) : 3563, 4006
.   : milestone, 3785,
iast (2.248 ms) : 2179, 2317
.   : milestone, 2248,
iast_GLOBAL (2.29 ms) : 2221, 2360
.   : milestone, 2290,
profiling (2.093 ms) : 2037, 2150
.   : milestone, 2093,
tracing (2.055 ms) : 2002, 2109
.   : milestone, 2055,
section candidate
no_agent (1.467 ms) : 1455, 1478
.   : milestone, 1467,
appsec (3.719 ms) : 3502, 3936
.   : milestone, 3719,
iast (2.246 ms) : 2177, 2315
.   : milestone, 2246,
iast_GLOBAL (2.287 ms) : 2217, 2357
.   : milestone, 2287,
profiling (2.066 ms) : 2012, 2121
.   : milestone, 2066,
tracing (2.047 ms) : 1993, 2100
.   : milestone, 2047,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.467 ms [1.456 ms, 1.479 ms] -
appsec 3.785 ms [3.563 ms, 4.006 ms] 2.317 ms (157.9%)
iast 2.248 ms [2.179 ms, 2.317 ms] 780.403 µs (53.2%)
iast_GLOBAL 2.29 ms [2.221 ms, 2.36 ms] 823.049 µs (56.1%)
profiling 2.093 ms [2.037 ms, 2.15 ms] 626.089 µs (42.7%)
tracing 2.055 ms [2.002 ms, 2.109 ms] 588.062 µs (40.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.467 ms [1.455 ms, 1.478 ms] -
appsec 3.719 ms [3.502 ms, 3.936 ms] 2.252 ms (153.5%)
iast 2.246 ms [2.177 ms, 2.315 ms] 778.975 µs (53.1%)
iast_GLOBAL 2.287 ms [2.217 ms, 2.357 ms] 820.302 µs (55.9%)
profiling 2.066 ms [2.012 ms, 2.121 ms] 599.338 µs (40.9%)
tracing 2.047 ms [1.993 ms, 2.1 ms] 579.821 µs (39.5%)

amarziali
amarziali reviewed Mar 2, 2026
View reviewed changes
dd-trace-core/src/main/java/datadog/trace/common/metrics/MetricKey.java
public final class MetricKey {
static final DDCache<String, UTF8BytesString> RESOURCE_CACHE = DDCaches.newFixedSizeCache(32);
static final DDCache<String, UTF8BytesString> SERVICE_CACHE = DDCaches.newFixedSizeCache(8);
static final DDCache<String, UTF8BytesString> SERVICE_SOURCE_CACHE = DDCaches.newFixedSizeCache(4);
Copy link
Contributor

@amarziali amarziali Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be already filled with UTF8BytesString. It can take values of the instrumentation name (that's should also be UTF8BytesString) so I suggest removing caching for this specific one. 4, is also too small

amarziali
amarziali reviewed Mar 2, 2026
View reviewed changes
dd-trace-core/src/main/java/datadog/trace/common/metrics/MetricKey.java
Comment on lines +18 to +21
static final DDCache<String, UTF8BytesString> TYPE_CACHE = DDCaches.newFixedSizeCache(8);
static final DDCache<String, UTF8BytesString> KIND_CACHE = DDCaches.newFixedSizeCache(8);
static final DDCache<String, UTF8BytesString> HTTP_METHOD_CACHE = DDCaches.newFixedSizeCache(8);
static final DDCache<String, UTF8BytesString> HTTP_ENDPOINT_CACHE = DDCaches.newFixedSizeCache(32);
Copy link
Contributor

@amarziali amarziali Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to widen those a bit? Also, for the http endpoint, I'm wondering if we should just do that caching earlier to make other serialisation benefitting of this (i.e. in EndpointResolver)

amarziali
amarziali reviewed Mar 2, 2026
View reviewed changes
dd-trace-core/src/main/java/datadog/trace/common/metrics/MetricKey.java
}

static UTF8BytesString utf8(DDCache<String, UTF8BytesString> cache, CharSequence charSeq) {
if ( charSeq == null ) {
Copy link
Contributor

@amarziali amarziali Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are things that are supposing to check nullity of this. Now if we replace with empty is not more the same semantic and this will break some code. I suggest to let the caller return the default value so it won't break the existing logic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@amarziali amarziali amarziali left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@amarziali amarziali

Labels

comp: metrics Metrics tag: performance Performance related changes type: enhancement Enhancements and improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dougqh @amarziali