gc cron job to clean up dangling policy_bindings entries #3065
Conversation
michaeljguarino
left a comment
michaeljguarino
left a comment
There was a problem hiding this comment.
we definitely need to make sure there are indices on all these columns as well, the exists queries are going to be quite slow if not.
|
|
||
| # Create a project with write_bindings - these should be kept | ||
| project = insert(:project, write_bindings: [%{user_id: user.id}]) | ||
| %{write_bindings: [kept_binding]} = Console.Repo.preload(project, [:write_bindings]) |
There was a problem hiding this comment.
i actually don't think this preload should be needed
There was a problem hiding this comment.
i think it's needed if we want a kept_binding that we can run assertions against after the cronjob
lib/console/deployments/cron.ex
Outdated
| |> PolicyBinding.ordered(asc: :id) | ||
| |> Repo.stream(method: :keyset) | ||
| |> Console.throttle(count: 100, pause: 1) | ||
| |> Task.async_stream(fn binding -> |
There was a problem hiding this comment.
we would want to chunk these in batches of 100 and delete each chunk in one query, to save db network round trips. Pretty sure there are some other examples of doing that here.
| timestamps() | ||
| end | ||
|
|
||
| def dangling(query \\ __MODULE__) do |
There was a problem hiding this comment.
The slightly scary thing about this is if another table is added that sets bindings, it's pretty likely it'll be forgotten that a clause is needed here to validate. Ideally we have some static check that confirms this, i just don't know how to organize it.
kinjalh
force-pushed
the
policy-bindings-gc-cron
branch
from
641a8d3 to
ed1ea75
Compare
gc cron job for dangling policy_bindings entries
Test Plan
unit test
Checklist
Plural Flow: console