baidu resource enrich#111
Conversation
Adds two new resource types to the baidu collector:
- ENI: region-scoped collector that enumerates VPCs in the region and
then calls ListEni per vpcId (the baidu SDK requires a vpcId on the
list call). Captures unattached ENIs and ENIs attached to non-BCC
resources (CCE workers, etc.). The Detail is a flat alias of eni.Eni
so the persisted instance JSON keeps the API's top-level field names
($.eniId, $.name, $.privateIpSet, ...), matching the JSONPath shape
downstream consumers already expect.
- ACL: region-scoped collector that enumerates VPCs and emits one ACL
record per VPC carrying the full set of subnet ACL entries returned
by ListAclEntrys. The Detail embeds *vpc.ListAclEntrysResult to keep
its fields (vpcId, vpcName, vpcCidr, aclEntrys) at the top level,
with sibling resourceId/resourceName fields holding the
"acl-{vpcId}" prefix used as the canonical resource identifier.
Registered in platform_config and wired into InitServices so both
resource types pick up clients from the existing baidu Services bag.
The baidu SDK's BLBModel / AppBLBModel / AppAllListenerModel /
AppServerGroupPort types under-model the actual /v1/blb, /v1/appblb,
/v1/{blb,appblb}/{id}/listener and /v1/appblb/{id}/appservergroup API
responses — fields like type, billingMethod, paymentTiming,
performanceLevel, allowModify, modificationProtectionReason,
underlayVip, expireTime on the load balancer; description on the
listener; healthCheckValid on AppServerGroupPort and on BLB listener
are silently dropped by the SDK structs.
Each affected Detail field switches to an enriched struct that embeds
the SDK type and appends the missing fields. Existing JSONPaths
($.Blb.blbId, $.AppBLB.name, $.AppServerGroupList[].portList[].port,
etc.) are preserved bit-for-bit; new fields appear as parallel siblings
under the same wrapper. The SDK-typed DescribeLoadBalancers /
DescribeAllListeners / DescribeAppServerGroup calls are replaced with
direct bce.RequestBuilder calls bound to the same URI and pagination
contract so a single API call still feeds both the existing and new
fields.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for collecting ENI and ACL resources from Baidu Cloud and enhances the BLB and AppBLB collectors by introducing 'enriched' structs and custom request builders to capture API fields missing from the standard SDK. The review feedback identifies a high-severity issue with an incorrect JSONPath for ENI addresses, suggests refactoring duplicated client initialization logic in services.go, and points out several instances in the BLB collectors where API errors are swallowed, which could result in incomplete resource data.
| RowField: schema.RowField{ | ||
| ResourceId: "$.eniId", | ||
| ResourceName: "$.name", | ||
| Address: "$.privateIpSet[0].publicIpAddress", |
There was a problem hiding this comment.
The JSONPath $.privateIpSet[0].publicIpAddress is likely incorrect. The eni.Eni struct in the Baidu SDK contains privateIpAddress within the privateIpSet elements, but does not include a publicIpAddress field. Since the collector uses the SDK struct, this field will be missing in the serialized JSON, and the JSONPath will fail to resolve. If the intention is to capture the primary private IP, please use $.privateIpSet[0].privateIpAddress.
| Address: "$.privateIpSet[0].publicIpAddress", | |
| Address: "$.privateIpSet[0].privateIpAddress", |
| case VPC, ACL: | ||
| vpcClient, err := vpc.NewClient(param.AK, param.SK, param.Region) | ||
| if err != nil { | ||
| log.GetWLogger().Warn(fmt.Sprintf("init vpc client failed, err: %s", err)) | ||
| } | ||
| s.VPCClient = vpcClient | ||
| case ENI: | ||
| vpcClient, err := vpc.NewClient(param.AK, param.SK, param.Region) | ||
| if err != nil { | ||
| log.GetWLogger().Warn(fmt.Sprintf("init vpc client failed, err: %s", err)) | ||
| } | ||
| s.VPCClient = vpcClient | ||
| eniClient, err := eni.NewClient(param.AK, param.SK, param.Region) | ||
| if err != nil { | ||
| log.GetWLogger().Warn(fmt.Sprintf("init eni client failed, err: %s", err)) | ||
| } | ||
| s.ENIClient = eniClient |
There was a problem hiding this comment.
The initialization logic for VPCClient is duplicated across the VPC, ACL, and ENI resource types. This can be refactored to reduce redundancy and improve maintainability.
| case VPC, ACL: | |
| vpcClient, err := vpc.NewClient(param.AK, param.SK, param.Region) | |
| if err != nil { | |
| log.GetWLogger().Warn(fmt.Sprintf("init vpc client failed, err: %s", err)) | |
| } | |
| s.VPCClient = vpcClient | |
| case ENI: | |
| vpcClient, err := vpc.NewClient(param.AK, param.SK, param.Region) | |
| if err != nil { | |
| log.GetWLogger().Warn(fmt.Sprintf("init vpc client failed, err: %s", err)) | |
| } | |
| s.VPCClient = vpcClient | |
| eniClient, err := eni.NewClient(param.AK, param.SK, param.Region) | |
| if err != nil { | |
| log.GetWLogger().Warn(fmt.Sprintf("init eni client failed, err: %s", err)) | |
| } | |
| s.ENIClient = eniClient | |
| case VPC, ACL, ENI: | |
| vpcClient, err := vpc.NewClient(param.AK, param.SK, param.Region) | |
| if err != nil { | |
| log.GetWLogger().Warn(fmt.Sprintf("init vpc client failed, err: %s", err)) | |
| } | |
| s.VPCClient = vpcClient | |
| if cloudAccountParam.ResourceType == ENI { | |
| eniClient, err := eni.NewClient(param.AK, param.SK, param.Region) | |
| if err != nil { | |
| log.GetWLogger().Warn(fmt.Sprintf("init eni client failed, err: %s", err)) | |
| } | |
| s.ENIClient = eniClient | |
| } |
| func describeAppAllListeners(ctx context.Context, client *appblb.Client, blbId string) (listenerList []enrichedAppListener) { | ||
| marker := "" | ||
| for { | ||
| response, err := client.DescribeAppAllListeners(blbId, args) | ||
| if err != nil { | ||
| result := &enrichedDescribeAppAllListenersResult{} | ||
| rb := bce.NewRequestBuilder(client). | ||
| WithMethod("GET"). | ||
| WithURL("/v1/appblb/" + blbId + "/listener"). | ||
| WithQueryParam("maxKeys", strconv.Itoa(50)). | ||
| WithResult(result) | ||
| if marker != "" { | ||
| rb = rb.WithQueryParam("marker", marker) | ||
| } | ||
| if err := rb.Do(); err != nil { | ||
| log.CtxLogger(ctx).Warn("DescribeAppAllListeners error", zap.Error(err)) | ||
| return | ||
| } | ||
| listenerList = append(listenerList, response.ListenerList...) | ||
| if response.NextMarker == "" { | ||
| listenerList = append(listenerList, result.ListenerList...) | ||
| if result.NextMarker == "" { | ||
| break | ||
| } | ||
| args.Marker = response.NextMarker | ||
| marker = result.NextMarker | ||
| } | ||
|
|
||
| return listenerList | ||
| } |
There was a problem hiding this comment.
The function describeAppAllListeners (and similarly describeAppServerGroup at line 207) swallows errors from the API call, returning a potentially empty or partial list without notifying the caller. This can lead to incomplete resource data being collected silently. Consider updating these functions to return an error along with the results, allowing the caller to handle API failures more robustly.
| func describeAllListeners(ctx context.Context, client *blb.Client, blbId string) (listenerList []enrichedBLBListener) { | ||
| marker := "" | ||
| for { | ||
| response, err := client.DescribeAllListeners(blbId, args) | ||
| if err != nil { | ||
| result := &enrichedDescribeAllListenersResult{} | ||
| rb := bce.NewRequestBuilder(client). | ||
| WithMethod("GET"). | ||
| WithURL("/v1/blb/" + blbId + "/listener"). | ||
| WithQueryParam("maxKeys", strconv.Itoa(50)). | ||
| WithResult(result) | ||
| if marker != "" { | ||
| rb = rb.WithQueryParam("marker", marker) | ||
| } | ||
| if err := rb.Do(); err != nil { | ||
| log.CtxLogger(ctx).Warn("DescribeAllListeners error", zap.Error(err)) | ||
| return | ||
| } | ||
| listenerList = append(listenerList, response.AllListenerList...) | ||
| if response.NextMarker == "" { | ||
| listenerList = append(listenerList, result.ListenerList...) | ||
| if result.NextMarker == "" { | ||
| break | ||
| } | ||
| args.Marker = response.NextMarker | ||
| marker = result.NextMarker | ||
| } | ||
|
|
||
| return listenerList | ||
| } |
There was a problem hiding this comment.
The function describeAllListeners swallows errors from the rb.Do() call. If the API request fails, it returns an empty or partial list, which might result in incomplete data collection. It is recommended to return the error to the caller for better error handling and visibility.
1 participant
Thank you for your contribution to CloudRec!
What About:
java)go)opa)Description:
Two related commits to expand baidu cloud coverage:
1.
feat(collector): add baidu ENI and ACL resources(d8c626d)Adds two previously-uncollected resource types under
BAIDU_CLOUD:ENI—bcc.Client.ListEnis()across all baidu BCC regionsACL—bcc.Client.ListAclEntrys()keyed by VPCBoth follow the flat-instance convention (struct embedding
*sdk.Foo+ sibling fields), so paths read naturally as$.eniId,$.vpcId, etc.2.
feat(collector): enrich baidu BLB and APPBLB with SDK-omitted fields(8371686)The baidu Go SDK's
BLBModel/AppBLBModel/AppAllListenerModel/AppServerGroupPortstructs under-model the actual API responses. Fields the SDK silently dropsare now captured:
GET /v1/blb,GET /v1/appblbtype,underlayVip,expireTime,billingMethod,paymentTiming,performanceLevel,allowModify,modificationProtectionReasonGET /v1/blb/{id}/listenerbackendPortType,healthCheckValidGET /v1/appblb/{id}/listenerdescriptionGET /v1/appblb/{id}/appservergroupportList[].healthCheckValidEach affected
Detailfield switches to a struct that embeds the SDK type and appends the missing fields, so existing JSONPaths ($.Blb.blbId,$.AppBLB.name,$.AppServerGroupList[].portList[].port, etc.) are preserved bit-for-bit. New fields appear as parallel siblings under the same wrapper.The SDK's typed
DescribeLoadBalancers/DescribeAllListeners/DescribeAppServerGroupcalls are replaced with directbce.RequestBuildercalls bound to the sameURI and pagination contract, so the request count is unchanged — a single API call now feeds both the SDK-modeled and new fields.
Validation
End-to-end run against two production-style baidu cloud accounts:
cloud_resource_instance_v1$.Blb.*and$.AppBLB.*instance has exactly 23 keys = 15 SDK-modeled + 8 newly preserveddescriptionpopulated on 608/608 APPBLBs with non-empty listeners (1 CCE-managed BLB withlistenerList=[]returnsnull— same behavior as pre-change SDK call)portList[].healthCheckValidpopulated on 15/15 APPBLBs with non-emptyportList$.Blb.*/$.AppBLB.*resolve unchanged