Crashed backend with fatal error: concurrent map iteration and map write

[carry2web]

Backend crashed with this line in the logs and then several thousand lines of stacktraces of goroutines.
What causes this condition?

Memory and disk seems fine.

[lazynina]

@carry2web - do you have the top few lines of the panic? Usually the first few goroutines are helpful in identifying these issues.

I hit a similar panic recently and I believe it will be addressed in the next release, but the stack traces of the first few goroutines would be helpful to confirm it is the same issue.

[carry2web]

@lazynina
This is what I found in the logs, hope it helps you:

{"log":"E0315 15:59:17.431221 1 pos_consensus.go:184] FastHotStuffConsensus.HandleLocalBlockProposalEvent: Error proposing block: Error processing block locally: Error proces>{"log":"I0315 15:59:17.721866 1 server.go:2338] Server._handleBlock: Processing block \u003cHeader: \u003c 14002479, 4b8db0c965992067df94e021f7e4df6b644b7ed6b369749d0a3f14177>{"log":"I0315 15:59:18.897748 1 txindex.go:289] Update: Updating txindex tip (height: 14002476, hash: 80603b87a59133755ec9ebbfb0154ed3ef8d729334b3999f7eceb45906c6f171) to blo>{"log":"I0315 15:59:18.897809 1 txindex.go:392] Update: Txindex progress: block 14002477 / 14002477\n","stream":"stderr","time":"2025-03-15T15:59:18.897950476Z"}
{"log":"I0315 15:59:18.901641 1 txindex.go:458] Update: Txindex update complete. New tip: (height: 14002477, hash: 0663496e998cd5c86861a6a8797a4849c295ffe7bbfd3a56e48e128f2a0>{"log":"I0315 15:59:19.323052 1 server.go:2338] Server._handleBlock: Processing block \u003cHeader: \u003c 14002480, 7c9410d4b70b86022271ae2eb21598796439cb061c27e2b45247239f5>{"log":"fatal error: concurrent map iteration and map write\n","stream":"stderr","time":"2025-03-15T15:59:19.335546687Z"}
{"log":"\n","stream":"stderr","time":"2025-03-15T15:59:19.338501621Z"}
{"log":"goroutine 13905856 [running]:\n","stream":"stderr","time":"2025-03-15T15:59:19.338508731Z"}
{"log":"github.com/deso-protocol/core/lib.(*TransactionSpendingLimit).Copy(0xcdb171ee70)\n","stream":"stderr","time":"2025-03-15T15:59:19.338514481Z"}
{"log":"\u0009/deso/src/core/lib/network.go:7129 +0x4a9\n","stream":"stderr","time":"2025-03-15T15:59:19.338520262Z"}
{"log":"github.com/deso-protocol/core/lib.(*DerivedKeyEntry).Copy(0xc5fffccd80)\n","stream":"stderr","time":"2025-03-15T15:59:19.338526232Z"}
{"log":"\u0009/deso/src/core/lib/block_view_types.go:3855 +0xc5\n","stream":"stderr","time":"2025-03-15T15:59:19.338533622Z"}
{"log":"github.com/deso-protocol/core/lib.(*UtxoView)._checkAndUpdateDerivedKeySpendingLimit(0xc4a8eab880, 0xca5fb35b90, {0xc293af6240, 0x21, 0x21}, 0x28f, {0xca542f0678, 0x1, 0x1}>{"log":"\u0009/deso/src/core/lib/block_view.go:2364 +0x1df\n","stream":"stderr","time":"2025-03-15T15:59:19.338560442Z"}
{"log":"github.com/deso-protocol/core/lib.(*UtxoView)._connectBasicTransferWithExtraSpend(0xc4a8eab880, 0xca5fb35b90, 0xcb81450420, 0xd5a930, 0x0, 0x1)\n","stream":"stderr","time":>{"log":"\u0009/deso/src/core/lib/block_view.go:2311 +0x386b\n","stream":"stderr","time":"2025-03-15T15:59:19.338593593Z"}
{"log":"github.com/deso-protocol/core/lib.(*UtxoView)._connectBasicTransfer(0xc4a8eab880, 0xca5fb35b90, 0xcb81450420, 0xd5a930, 0x1)\n","stream":"stderr","time":"2025-03-15T15:59:1>{"log":"\u0009/deso/src/core/lib/block_view.go:1976 +0xb5\n","stream":"stderr","time":"2025-03-15T15:59:19.338630674Z"}
{"log":"github.com/deso-protocol/core/lib.(*UtxoView)._connectAuthorizeDerivedKey(0xc4a8eab880, 0xca5fb35b90, 0xcb81450420, 0xd5a930, 0x1)\n","stream":"stderr","time":"2025-03-15T1>