Skip to content

feat: P2P Resource manager with predefined profiles via CLI option #4629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alexmylonas wants to merge 14 commits into
base: develop
Choose a base branch
from
Open

feat: P2P Resource manager with predefined profiles via CLI option #4629

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
7c1e767
feat: Added resource profile cli option
alexmylonas Mar 12, 2026
3b135bd
Added unit tests for profile limits
alexmylonas Mar 12, 2026
84c4437
Updated README
alexmylonas Mar 12, 2026
19c402e
Merge branch 'develop' into feat/resource-profiles
alexmylonas Mar 13, 2026
4e34772
Remove empty string as valid opt for buildResourceManager
alexmylonas Mar 16, 2026
616fced
Updated rcmgr limits, inbound + outbound = total
alexmylonas Mar 16, 2026
fccba40
Addressed coderabbit feedback
alexmylonas Mar 16, 2026
0b66ee2
Updated limits for server profile
alexmylonas Mar 17, 2026
8a5481e
Merge branch 'develop' into feat/resource-profiles
alexmylonas Mar 20, 2026
6a67dc0
Refactor ResourceManager as ResourceProfile
alexmylonas Mar 20, 2026
a739fbf
Addressed feedback
alexmylonas Mar 20, 2026
975a18b
updated code comment
alexmylonas Mar 20, 2026
5948497
Updated outdated test
alexmylonas Mar 20, 2026
6be8b8f
Updated code comment
alexmylonas Mar 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Read the documentation on [docs.source.network](https://docs.source.network/).
* [Securing the HTTP API with TLS](#securing-the-http-api-with-tls)
* [Access Control System](#access-control-system)
* [Supporting CORS](#supporting-cors)
* [P2P Resource Management](#p2p-resource-management)
* [Backing up and restoring](#backing-up-and-restoring)
* [Telemetry](#telemetry)
* [Community](#community)
Expand Down Expand Up @@ -502,6 +503,20 @@ defradb start --allowed-origins=http://localhost:3000

The catch-all `*` is also a valid origin.

## P2P Resource Management
Comment thread
coderabbitai[bot] marked this conversation as resolved.

By default, DefraDB autoscales the resource limits for p2p connections based on system availability. For deployments with specific resource constraints or performance requirements, the `--resource-profile` flag selects a predefined limit profile:

```shell
defradb start --resource-profile limited
```

| Profile | Use case | Description |
|---------|----------|-------------|
| *(unset)* | General | autoscaled defaults |
| `limited` | Edge nodes, constrained hardware | Conservative system-wide caps on connections, streams, and memory |
| `server` | High-throughput nodes | Per-peer connections, stream and memory caps; system limits remain autoscaled |

## Backing up and restoring

It is currently not possible to do a full backup of DefraDB that includes the history of changes through the Merkle DAG. However, DefraDB currently supports a simple backup of the current data state in JSON format that can be used to seed a database or help with transitioning from one DefraDB version to another.
Expand Down
2 changes: 2 additions & 0 deletions cli/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ var ConfigFlags = map[string]string{
"peers": "net.peers",
"p2paddr": "net.p2paddresses",
"no-p2p": "net.p2pdisabled",
"resource-profile": "net.resourceprofile",
Copy link
Copy Markdown

@reecepbcups reecepbcups Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UX nit: I do not see any input validation at the CLI level (i.e. --resource-profile=typo would pass al the way to buildResourceManager at startup before erroring.)

Copy link
Copy Markdown
Author

@alexmylonas alexmylonas Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's correct.

In general, I can see there is limited content validation for input namely, peers or document-acp-type which is closer to the input format, both fail downstream.

I would be happy to add it if there is strong preference.

Copy link
Copy Markdown
Collaborator

@fredcarle fredcarle Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what you did is fine given the state of the start function. Although I don't think you should try to add validation specifically for the resource-profile flag, I would be curious to know what approach you would take to validate all the possible inputs to the start function generally.

Copy link
Copy Markdown
Author

@alexmylonas alexmylonas Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo, letting variables failing downstream at the package level instead of having a central validation isn't a bad direction and provides separation of concerns.

However, if we were to pursue that, I would add a Validate method to the NodeOptions which would run inside New() to do validation for all values related to node.

In addition, there are some cfg values not related to node like log.level which right now are not validated, so probably a Validate(cfg) for those to be run at the end of PersistentPreRunE

"allowed-origins": "api.allowed-origins",
"pubkeypath": "api.pubkeypath",
"privkeypath": "api.privkeypath",
Expand Down Expand Up @@ -91,6 +92,7 @@ var ConfigDefaults = map[string]any{
"net.peers": []string{},
"net.pubSubEnabled": true,
"net.relay": false,
"net.resourceprofile": "",
"keyring.backend": "file",
"keyring.disabled": false,
"keyring.namespace": "defradb",
Expand Down
1 change: 1 addition & 0 deletions cli/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ func TestLoadConfigNotExist(t *testing.T) {
assert.Equal(t, true, cfg.GetBool("net.pubsubenabled"))
assert.Equal(t, false, cfg.GetBool("net.relay"))
assert.Equal(t, []string{}, cfg.GetStringSlice("net.peers"))
assert.Equal(t, "", cfg.GetString("net.resourceprofile"))

assert.Equal(t, "info", cfg.GetString("log.level"))
assert.Equal(t, "stderr", cfg.GetString("log.output"))
Expand Down
8 changes: 7 additions & 1 deletion cli/start.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ func MakeStartCommand(ctx context.Context) *cobra.Command {
SetListenAddresses(cfg.GetStringSlice("net.p2pAddresses")...).
SetEnablePubSub(cfg.GetBool("net.pubSubEnabled")).
SetEnableRelay(cfg.GetBool("net.relayEnabled")).
SetBootstrapPeers(cfg.GetStringSlice("net.peers")...)
SetBootstrapPeers(cfg.GetStringSlice("net.peers")...).
SetResourceProfile(cfg.GetString("net.resourceprofile"))
opts.HTTP().
SetAddress(cfg.GetString("api.address")).
SetAllowedOrigins(cfg.GetStringSlice("api.allowed-origins")...).
Expand Down Expand Up @@ -284,6 +285,11 @@ func MakeStartCommand(ctx context.Context) *cobra.Command {
cfg.GetBool(config.ConfigFlags["no-p2p"]),
"Disable the peer-to-peer network synchronization system",
)
cmd.PersistentFlags().String(
"resource-profile",
cfg.GetString(config.ConfigFlags["resource-profile"]),
"Set the resource manager profile for the p2p network (supported: limited, server)",
)
cmd.PersistentFlags().StringArray(
"allowed-origins",
cfg.GetStringSlice(config.ConfigFlags["allowed-origins"]),
Expand Down
10 changes: 10 additions & 0 deletions client/options/node.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,9 @@ type NodeP2POptions struct {
EnableClearBackoffOnRetry bool
// PrivateKey is the private key for the P2P node.
PrivateKey []byte
// ResourceProfile sets predefined resource limits for the P2P host.
// Valid values are "limited" and "server".
ResourceProfile string
}

// NodeHTTPOptions contains HTTP API server configuration values.
Expand Down Expand Up @@ -508,6 +511,13 @@ func (sb *NodeP2POptionsBuilder) SetPrivateKey(key []byte) *NodeP2POptionsBuilde
return sb
}

// SetResourceProfile sets predefined resource manager limits for the P2P host.
// Valid values are "limited", "server", or "" (default autoscaling).
func (sb *NodeP2POptionsBuilder) SetResourceProfile(profile string) *NodeP2POptionsBuilder {
sb.append(func(opts *NodeP2POptions) { opts.ResourceProfile = profile })
return sb
}

// SetAll sets all P2P options from a plain data struct.
func (sb *NodeP2POptionsBuilder) SetAll(p2pOpts NodeP2POptions) *NodeP2POptionsBuilder {
sb.append(func(opts *NodeP2POptions) { *opts = p2pOpts })
Expand Down
19 changes: 10 additions & 9 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/sourcenetwork/defradb

go 1.25.5
go 1.25.7

require (
github.com/bits-and-blooms/bitset v1.24.4
Expand Down Expand Up @@ -31,7 +31,7 @@ require (
github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20250821084354-a425e60cd714
github.com/joho/godotenv v1.5.1
github.com/lestrrat-go/jwx/v2 v2.1.6
github.com/libp2p/go-libp2p v0.47.0
github.com/libp2p/go-libp2p v0.48.0
github.com/multiformats/go-multiaddr v0.16.1
github.com/multiformats/go-multibase v0.2.0
github.com/multiformats/go-multicodec v0.10.0
Expand Down Expand Up @@ -102,7 +102,9 @@ require (
cosmossdk.io/x/tx v0.14.0 // indirect
cosmossdk.io/x/upgrade v0.1.4 // indirect
dario.cat/mergo v1.0.2 // indirect
filippo.io/bigmod v0.1.1-0.20260103110540-f8a47775ebe5 // indirect
filippo.io/edwards25519 v1.1.1 // indirect
filippo.io/keygen v0.0.0-20260114151900-8e2790ea4c5b // indirect
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 // indirect
github.com/99designs/keyring v1.2.2 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
Expand Down Expand Up @@ -347,21 +349,18 @@ require (
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/petermattis/goid v0.0.0-20240813172612-4fcff4a6cae7 // indirect
github.com/pion/datachannel v1.5.10 // indirect
github.com/pion/dtls/v2 v2.2.12 // indirect
github.com/pion/dtls/v3 v3.1.1 // indirect
github.com/pion/dtls/v3 v3.1.2 // indirect
github.com/pion/ice/v4 v4.0.10 // indirect
github.com/pion/interceptor v0.1.41 // indirect
github.com/pion/logging v0.2.4 // indirect
github.com/pion/mdns/v2 v2.0.7 // indirect
github.com/pion/randutil v0.1.0 // indirect
github.com/pion/rtcp v1.2.15 // indirect
github.com/pion/rtcp v1.2.16 // indirect
github.com/pion/rtp v1.8.22 // indirect
github.com/pion/sctp v1.8.39 // indirect
github.com/pion/sdp/v3 v3.0.16 // indirect
github.com/pion/sdp/v3 v3.0.18 // indirect
github.com/pion/srtp/v3 v3.0.8 // indirect
github.com/pion/stun v0.6.1 // indirect
github.com/pion/stun/v3 v3.0.0 // indirect
github.com/pion/transport/v2 v2.2.10 // indirect
github.com/pion/stun/v3 v3.1.1 // indirect
github.com/pion/transport/v3 v3.0.8 // indirect
github.com/pion/transport/v4 v4.0.1 // indirect
github.com/pion/turn/v4 v4.1.1 // indirect
Expand Down Expand Up @@ -466,3 +465,5 @@ require (
)

replace github.com/wlynxg/anet => github.com/sourcenetwork/anet v0.0.0-20250417190629-7c87cba7799e

replace github.com/sourcenetwork/go-p2p => github.com/alexmylonas/go-p2p v0.0.0-20260320165929-6c5461abc636
Loading