redpanda: fine grain accounting

[emaxerrno]

Goal

To enable fine grain accounting inside redpanda for multi-tenant cloud

Redpanda Interface Impact

redpanda.yaml setting:

redpanda:
  - fine_grained_accounting: true

Thoughts on design

controller can create a topic (assigned to core0) called redpanda.<node-id>.accounting to every node that joins the cluster, with replication factor 1.

at depending on the ammount of keys:

• custom binary format - for diskspace efficiency/cpu usage.
• key=value pairs
• self describing schema message that evolves the format?
• ticks every 3 (could be 1) seconds. enables data usage /sec
• namespaced level statistics
• can we leverage the namespaces.

[emaxerrno]

cc: @twmb @dotnwat @dswang

[dotnwat]

Are there some benefits to this approach compared to scraping the data using an external service? Not necessarily Prometheus, but same concept.

One benefit is that we don't have to build such a service, but we would end up having to build something anyway that consumes all this data and merges it across all the topics used to store accounting data. If this is for accounting / billing it might make sense to use replication=3, but at the same time it seems like a service that might not be generally applicable to users outside a cloud / billing context and so might make sense as an external service.

+1 namespaces

[emaxerrno]

@twmb was able to distill some of his prev experience building large billing pipelinens into this. so wanted to jot it down.

[emaxerrno]

@dotnwat - telemetry data is useful outside of billing, if it comes unified as a kafka topic for example. i.e.: consuming it allows us to build dashboards w/ information only stored in redpanda - crdb does this for example very successfully

[dotnwat]

that makes sense im glad its useful outside billing.

there really isn't much to it on the implementation side. deciding on what metrics to collect will probably be the most debated topic.

so far i haven't heard of any plans to use namespaces. that probably doesn't need to block any development since it would primarily be another attribute on whatever data we end up storing.