Skip to content

[meta] Free RPC server allocations before throwing on error - #2378

Merged
kcudnik merged 2 commits into
opencomputeproject:masterfrom
MoriLevinzon:fix/rpc-server-error-path-leak
Nov 10, 2026
Merged

kcudnik merged 2 commits into
opencomputeproject:masterfrom
MoriLevinzon:fix/rpc-server-error-path-leak

Conversation

@MoriLevinzon

Copy link
Copy Markdown
Contributor

Fixes #2377

The generated RPC server functions free their list arguments (attribute lists, stat counter arrays) only after a successful SAI call. On a failed call, or a NULL API function pointer, they throw sai_thrift_exception before freeing, so every failed call leaks.

  • meta/templates/sai_rpc_server_functions.tt: new free_malloced_args block, called before each throw (NULL API pointer, failed status, failed stats call). It uses its own variable names because PROCESS shares the caller's stash, where indentation and arg are in use.
  • meta/checkrpcleaks.pl: checks the generated sai_rpc_server.cpp and fails if a function throws while still holding an allocation.
  • meta/Makefile: make rpc runs the check.

Testing:

  • On current master, checkrpcleaks.pl fails: 670 generated functions throw without freeing.
  • With this change it passes, and the regenerated sai_rpc_server.cpp differs from master only by the added free() calls (1472 lines, nothing removed).
  • make, make rpc, make sai_rpc_server.skeleton and make sai_rpc_frontend pass.

The generated RPC server functions allocate their list arguments
(attribute lists, stat counter arrays) and free them only after a
successful SAI call. When the call fails, or the API function pointer
is NULL, they throw sai_thrift_exception first, so every failed call
leaks those buffers. Tests that expect a call to be rejected leak on
every expected failure, which LeakSanitizer reports when the RPC
server exits.

Free the allocated arguments before each throw, and run the new
checkrpcleaks.pl from 'make rpc' so that a generated function which
throws while still holding an allocation fails the build.

Signed-off-by: Mori Levinzon <mlevinzon@nvidia.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Comment thread meta/checkrpcleaks.pl Outdated
if ($line =~ /^ \S.*?\b(sai_thrift_\w+)\(.*\) \{$/)
{
$function = $1;
@allocated = ();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are making empty allocated table here, would it be possible that this if() will happen twice? so first you will allocate (line 55), but not free, then this line happens ad 47, and you have allocated, and then empty ?

woulid it be better to do something like this:

$allocations{X} +=1; // if allocated, -=1 if deallocated, later on at the end, loop throug all keys and check if all are zeros, since currently you can have allocated empty, and freed containing something, which is not desired, and it will be not catched, since error code is set only when allocated is not empty

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, thanks. The check now does that, adapted to functions having several exits:

  • Function boundaries are tracked by brace depth; a function start seen before the current one ends is an error instead of a silent reset.

  • Per-variable counters: +1 per malloc, +1 per free on the success path, and at the function's return every variable must be freed as many times as it was allocated. Freeing something never allocated is an error too.

  • A plain per-function counter would see 1 malloc against 3 frees in e.g. create_tunnel (two error branches plus the success path), so each error branch is checked as its own path: every throw, and every early return inside a branch, must free each allocation made so far exactly once.

  • Branch layouts it can't follow (} else, else if) are reported rather than checked wrongly.

No errors on the regenerated server, 1472 errors in 670 functions on current master, and each of those cases is reported when injected.

Track function boundaries by brace depth, and report a function start
seen before the current function ends instead of resetting the state.

Count allocations and frees per variable. The success path must free
each variable as many times as it was allocated, and every throw or
early return inside a branch must free each allocation made so far
exactly once. Freeing a variable that was never allocated is an error,
and so is a branch layout the check does not follow.

Signed-off-by: Mori Levinzon <mlevinzon@nvidia.com>
@MoriLevinzon
MoriLevinzon requested a review from kcudnik November 3, 2026 10:59
@MoriLevinzon

Copy link
Copy Markdown
Contributor Author

@kcudnik The checker now tracks allocations per path as you suggested, details in the thread above. Could you take another look when you get a chance, and kick off /azp run? Thanks!

@kcudnik

kcudnik commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@MoriLevinzon

Copy link
Copy Markdown
Contributor Author

@kcudnik @tjchadaga this is approved and all checks passed. Could one of you merge it when you get a chance? Thanks!

@MoriLevinzon
MoriLevinzon requested a review from kcudnik November 10, 2026 13:39
@kcudnik
kcudnik merged commit a19d0a4 into opencomputeproject:master Nov 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RPC server leaks its list arguments when a SAI call fails

2 participants