Now the SW query is slow, should I adjust the indexShardsNumber? How much is appropriate?

[xjt2016]

The currently used Skywalking version is 8.0.1. Currently, there are 5 main segments of the segment index, and the size has reached about 26GB. Now the SW query is slow, should I adjust the indexShardsNumber? How much is appropriate?

[wu-sheng]

(1) First of all, consider to upgrade to latest 8.7.0. There are a lot of perf improvements in the whole year, including persistent performance and query.

(2) Then, make sure your ElasticSearch has enough CPU/Memory, and uses SSD. They are extremely important for your cluster performance.

The last step should consider indexShardsNumber. How many shards depend on your resources. If (2) doesn't get resolved, the increasement of shard number make very limited improvement.

[xjt2016]

I see. Based on the above answers, I tried arthas again to troubleshoot the problem of slow query, and found that WebApp was still slow, mainly caused by GraphQL. Help to see。

`---ts=2021-09-07 15:21:45;thread_name=qtp518167684-1293;id=50d;is_daemon=false;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@36baf30c
    `---[8531.138715ms] graphql.execution.AsyncExecutionStrategy:execute()
        +---[0.001649ms] graphql.execution.ExecutionContext:getInstrumentation() #41
        +---[0.004668ms] graphql.execution.instrumentation.parameters.InstrumentationExecutionStrategyParameters:<init>() #42
        +---[0.00483ms] graphql.execution.instrumentation.Instrumentation:beginExecutionStrategy() #44
        +---[0.001166ms] graphql.execution.ExecutionStrategyParameters:getFields() #46
        +---[0.001522ms] graphql.execution.ExecutionStrategyParameters:getPath() #52
        +---[0.015818ms] graphql.execution.ExecutionPath:segment() #52
        +---[0.036675ms] graphql.execution.ExecutionStrategyParameters:transform() #54
        +---[0.007117ms] graphql.execution.AsyncExecutionStrategy:isDeferred() #56
        +---[8531.041332ms] graphql.execution.AsyncExecutionStrategy:resolveField() #59
        |   `---[1476.563496ms] graphql.execution.AsyncExecutionStrategy:execute()
        |       +---[0.001379ms] graphql.execution.ExecutionContext:getInstrumentation() #41
        |       +---[0.004589ms] graphql.execution.instrumentation.parameters.InstrumentationExecutionStrategyParameters:<init>() #42
        |       +---[0.005765ms] graphql.execution.instrumentation.Instrumentation:beginExecutionStrategy() #44
        |       +---[0.001264ms] graphql.execution.ExecutionStrategyParameters:getFields() #46
        |       +---[0.001208ms] graphql.execution.ExecutionStrategyParameters:getPath() #52
        |       +---[0.019109ms] graphql.execution.ExecutionPath:segment() #52
        |       +---[0.037645ms] graphql.execution.ExecutionStrategyParameters:transform() #54
        |       +---[0.009562ms] graphql.execution.AsyncExecutionStrategy:isDeferred() #56
        |       +---[1476.460416ms] graphql.execution.AsyncExecutionStrategy:resolveField() #59
        |       |   `---[min=3.470517ms,max=85.757013ms,total=1371.283347ms,count=302] graphql.execution.AsyncExecutionStrategy:execute()
        |       |       +---[min=0.001052ms,max=0.006772ms,total=0.387901ms,count=302] graphql.execution.ExecutionContext:getInstrumentation() #41
        |       |       +---[min=0.003141ms,max=0.012894ms,total=1.109954ms,count=302] graphql.execution.instrumentation.parameters.InstrumentationExecutionStrategyParameters:<init>() #42
        |       |       +---[min=0.0038ms,max=0.012499ms,total=1.401739ms,count=302] graphql.execution.instrumentation.Instrumentation:beginExecutionStrategy() #44
        |       |       +---[min=9.16E-4ms,max=0.002102ms,total=0.347902ms,count=302] graphql.execution.ExecutionStrategyParameters:getFields() #46
        |       |       +---[min=8.68E-4ms,max=0.019478ms,total=5.442425ms,count=4832] graphql.execution.ExecutionStrategyParameters:getPath() #52
        |       |       +---[min=0.015501ms,max=0.055792ms,total=90.863949ms,count=4832] graphql.execution.ExecutionPath:segment() #52
        |       |       +---[min=0.026279ms,max=0.095561ms,total=147.619252ms,count=4832] graphql.execution.ExecutionStrategyParameters:transform() #54
        |       |       +---[min=0.003304ms,max=0.054186ms,total=21.168195ms,count=4832] graphql.execution.AsyncExecutionStrategy:isDeferred() #56
        |       |       +---[min=0.100997ms,max=81.70394ms,total=1077.691802ms,count=4832] graphql.execution.AsyncExecutionStrategy:resolveField() #59
        |       |       |   `---[min=0.359673ms,max=80.505573ms,total=329.789047ms,count=603] graphql.execution.AsyncExecutionStrategy:execute()
        |       |       |       +---[min=0.001021ms,max=0.009038ms,total=0.741976ms,count=603] graphql.execution.ExecutionContext:getInstrumentation() #41
        |       |       |       +---[min=0.003056ms,max=0.013751ms,total=2.194411ms,count=603] graphql.execution.instrumentation.parameters.InstrumentationExecutionStrategyParameters:<init>() #42
        |       |       |       +---[min=0.003726ms,max=0.013773ms,total=2.708063ms,count=603] graphql.execution.instrumentation.Instrumentation:beginExecutionStrategy() #44
        |       |       |       +---[min=9.23E-4ms,max=0.00651ms,total=0.665093ms,count=603] graphql.execution.ExecutionStrategyParameters:getFields() #46
        |       |       |       +---[min=8.98E-4ms,max=0.011892ms,total=1.434622ms,count=1210] graphql.execution.ExecutionStrategyParameters:getPath() #52
        |       |       |       +---[min=0.019946ms,max=79.921883ms,total=108.555649ms,count=1210] graphql.execution.ExecutionPath:segment() #52
        |       |       |       +---[min=0.02691ms,max=0.062627ms,total=37.24199ms,count=1210] graphql.execution.ExecutionStrategyParameters:transform() #54
        |       |       |       +---[min=0.00362ms,max=0.01879ms,total=5.682445ms,count=1210] graphql.execution.AsyncExecutionStrategy:isDeferred() #56
        |       |       |       +---[min=0.104218ms,max=0.551989ms,total=155.886275ms,count=1210] graphql.execution.AsyncExecutionStrategy:resolveField() #59
        |       |       |       +---[min=0.001048ms,max=0.012078ms,total=0.781403ms,count=603] graphql.execution.instrumentation.InstrumentationContext:onDispatched() #64
        |       |       |       +---[min=0.002049ms,max=0.011137ms,total=1.507488ms,count=603] graphql.execution.Async:each() #66
        |       |       |       `---[min=5.83E-4ms,max=0.007312ms,total=0.492892ms,count=603] graphql.execution.AsyncExecutionStrategy:handleResults() #66
        |       |       +---[min=0.001012ms,max=0.006617ms,total=0.377873ms,count=302] graphql.execution.instrumentation.InstrumentationContext:onDispatched() #64
        |       |       +---[min=0.002612ms,max=0.015269ms,total=0.984116ms,count=302] graphql.execution.Async:each() #66
        |       |       `---[min=5.69E-4ms,max=0.015178ms,total=0.23333ms,count=302] graphql.execution.AsyncExecutionStrategy:handleResults() #66
        |       +---[0.00104ms] graphql.execution.instrumentation.InstrumentationContext:onDispatched() #64
        |       +---[0.001664ms] graphql.execution.Async:each() #66
        |       `---[5.57E-4ms] graphql.execution.AsyncExecutionStrategy:handleResults() #66
        +---[0.001196ms] graphql.execution.instrumentation.InstrumentationContext:onDispatched() #64
        +---[0.001492ms] graphql.execution.Async:each() #66
        `---[5.72E-4ms] graphql.execution.AsyncExecutionStrategy:handleResults() #66

[wu-sheng]

I am not sure why arthas puts the stack like this, but be to honest, I don't think GraphQL would be a performance issue.
Just simply read(I am not sure) the stack AsyncExecutionStrategy:execute shows you, this profile doesn't identify the root cause. Don't focus on this, this is a false alarm from my understanding.

[xjt2016]

OK ，thank you

[xjt2016]

[wu-sheng]

I can't see any relationship with SkyWalking. Notice, this may cause the security leak to your company. Don't paste all private exception stack in the public OSS project.