Support exclude_entity_ids in request body per query

[OrriMandarin]

Hi,

For the /match endpoint, excluding entities currently requires using a query parameter that has a strict maximum number of elements (50). Additionally, the exclusion list is applied globally across all queries in a single request, rather than being scoped per query.

Proposal:

• Allow specifying exclude_entity_ids within the request body for each individual query.
• This change would enable per-query exclusion and remove the current limitation on the number of IDs that can be excluded via query params.
• It would make multi-query requests more precise and easier to manage, especially when each query needs a different exclusion set.

What do you think about this suggestion?

[leonhandreke]

Thanks for reaching out! When I added exclude_entity_ids, I thought about this a bit. The problem is that the individual queries are just EntityExamples, so there isn't really a place to put per-query request parameters.

Currently, I'm thinking that the ability to batch multiple queries in one request has maybe turned out to be one of the less successful decisions in the yente API design. For a few reasons:

1. it complicates the semantics, because now you have parameters that are per-request but your really want them per-query
2. we're running on Cloud Run and there the model is to run many small instances and scale up only if needed (which I understand is not what you're doing). So really, it would be better to have more requests of roughly equal weight come in instead of some that are really fat.
3. should be easy to do on the client, and that's less complexity on the server (but maybe I'm living too much in async unicorn land an in practice, people are working with much less nice language support on their clients?)

Would it be an option for you to just do one request per query or is the batching something that is very valuable to you?

Back when we set the 50 limit, we were mostly worried what would happen to the search index if we passed in more, but I'd definitely be open to raising this limit (or making it configurable). If we'd be hitting the boundaries of what fits in a GET request, I'd also be open to just moving it to the POST body.

Just out of interest, what are you doing where you need to pass in more than 50? Our thinking when designing the feature was that it could be used to weed out confirmed false-positives - is that what's happening here or are you building for some other usecase?

Curious to hear your thoughts on all this! (and enjoy your weekend!)

[OrriMandarin]

Makes sense to keep one request per query, that’s what we do today. We’re not batching yet; it was just an open question after seeing the multi‑query example in your API docs.

As for needing more than 50 exclusions: we may not hit it often, but it’s theoretically possible. After a match, potentially many, depending on entity attributes and the configured threshold, we record false positives by saving their entity IDs. We reuse this list for future matches on the same entity, e.g., when OpenSanctions is updated or the entity changes on our side.

If we accumulate more than 50 false‑positive IDs, we won’t be able to pass the full list, and some won’t be excluded from the result.
We often say the case will never happen until a user proves us wrong 😅

[leonhandreke]

Hey, sorry for the late reply, long vacation :)

Would raising the the max make you a bit more comfortable with this? The only reason we put it in was because we don't know the ES performance profile of the query (but hopefully fine? we've never had ES performance issues, really), and we couldn't think of anyone who'd want more than 50. But a user will always prove us engineers wrong :)

[OrriMandarin]

Hey :-)

I’d misunderstood how exclude_entity_ids works. It seems we don’t need to provide every ID for an entity (see the referents field). I thought that if we exclude an entity with ID id_1, and the entity later changes to id_2 (with id_1 in referents), we’d need to pass both id_1 and id_2. But someone told me that’s not necessary because Yente checks both the current ID and the referents field when excluding an entity.

This should reduce the number of entity IDs we need to send in our queries.

We’ll keep monitoring our whitelist to see whether we really need to increase the max length for exclusions. For now, I think we’re OK.

[leonhandreke]

Yes, that's how it's supposed to work! There is the whole thing about updating IDs, and our story around that right now is not amazing (basically, call /entities to get a current ID and update it in your database), see https://www.opensanctions.org/docs/identifiers/.

I'll keep this issue open and added a new api-v2 label so that we don't forget about it when we do eventually make a breaking change to the API (see #1001 (comment), this might eventually happen)

[OrriMandarin]

Perfect 👍