Repository navigation
Size margin call orders for the deficit earlier liquidations leave - #10131
Martin-Molinero merged 8 commits into
Conversation
GetMarginCallOrders sized every position group's order for the whole deficit left after the open margin call orders, and from its full current holdings. ExecuteMarginCall stops once the open orders cover the deficit, but when they covered most of it the next group still got an order for all of it, and a group whose holdings were partly being sold was dropped rather than sized from what's left, leaving the deficit uncovered. Position groups are now resolved from the holdings left after the open margin call orders, ordered losers first as they are executed, and each one is sized for the deficit the open orders and the earlier groups' orders leave, never for more lots than it holds. Generation stops once the deficit is covered. Orders that would buy back holdings other liquidations sell beyond what's held are skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Requesting changes: one sizing defect in the new regrouped path.
CI is green, and the new and updated tests pass locally too.
Finding: a regrouped option strategy that isn't held gets one lot too few, so the deficit stays uncovered. GenerateMarginCallOrders now gets position groups that only exist after the open orders fill. PositionGroupBuyingPowerModel.GetMaximumLotsForDeltaBuyingPower and GetMaximumLotsForTargetBuyingPower size from Portfolio.Positions[parameters.PositionGroup.Key], the held group with that key. When no held group has that key, the indexer returns an empty group. The result is then the number of lots whose margin fits inside the deficit, rounded down, not the number needed to release it. The new Math.Min cap only limits overselling. It can't raise an undersized result. (For the security model and for a key that is held, such as 2 covered calls left from 3, the sizing works with the cap.)
I reproduced it locally with GetSpyPortfolio(orderProcessor, -26000, -3) (3 covered calls, $0 buffer). OnMarginCall splits the combo into SPY -300 and +3 calls. The call buyback is cancelled and the stock order stays open. Then cash drops to -$29,000:
- TPV is $700. Margin after the open order is $3,300, all of it for the 3 now-naked calls at $1,100 each, which leaves a $2,600 deficit.
- The branch generates
+2calls. That releases $2,200 and leaves $400 uncovered until a later scan, which on daily data is the next bar. - The same deficit on a portfolio that holds the 3 naked calls gives
+3, which is correct. - With the stock order at
-150instead, the regroup is 1 covered call, 50 SPY and 2 naked calls, against a $9,000 deficit. The branch generates the combo,-50SPY and+1call, and leaves a naked call and $400 of the deficit uncovered.
Master generates nothing in these cases, so the branch is still better than master. But they're the cases the issue asks this change to cover ("the covered call group's request is dropped because of its call leg, though 300 shares still need selling"), and sizing for the deficit the earlier liquidations leave is the point of the change. A regrouped group needs to be sized against itself, not against a held group with its key or an empty one. Please add a test where the regrouped strategy isn't held, like the one above, and check that its order covers the deficit.
Other checks, no change needed:
- Groups partition the holdings, and every order is capped to its group's quantity. The running
liquidatingQuantitiescheck therefore only rejects genuine oversells, such as anOnMarginCallrequest larger than the position, and the sign check stops those from being bought back. - The losers-first order here matches
ExecuteMarginCall's, which uses the same per-symbolUnrealizedProfitsums, including for groups that share a leg. - Cost: each scan now runs one more
ResolvePositionGroupscall, plus one for each accepted group until the deficit is covered. It only runs while margin remaining is <= 0, and it replaces aGenerateMarginCallOrderscall per group, so it's fine. OnMarginCallnow receives only the liquidations needed, not one per group. The description says so, and the documented use, adjusting or choosing among the requests, still works. The default behaviour matchesExecuteMarginCall's, since it already stopped once the deficit was covered.
Other repositories: none need an update. The documentation's OnMarginCall examples still apply.
Risk: I set risk:high. The change alters which liquidation orders users' OnMarginCall receives and how much margin calls liquidate (money), and the defect above shows the new path can still under-size.
The holdings left after open margin call orders can regroup into a position group that isn't held, such as naked calls left from covered calls whose stock is being sold. The buying power model sizes a group from the held group with its key, or an empty one, so such a group got too few lots and part of the deficit stayed uncovered. These groups are now liquidated for the fewest of their lots whose margin covers the deficit. Held groups are sized by their buying power model as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the review in 5e74b88:
Full suite, Release, CI filter: |
|
The failing check is the Benchmarks workflow, which failed twice: once on the first run and once on its re-run. I don't think this change causes it.
These algorithms subscribe to data and never place an order, so they never use margin. |
There was a problem hiding this comment.
Approving 5e74b88. It resolves my earlier finding, and I found nothing new.
The earlier finding is fixed. A position group that isn't held as is gets sized against itself now: the fewest of its own lots whose margin covers the deficit, found by a binary search. Held groups still go through their buying power model, so a margin call without open orders behaves as before. LiquidatesPositionGroupsThatAreNotHeldForTheDeficit covers both of my reproductions (+3 calls for the -300 stock order; the covered call, +2 calls and -36 shares for -150). It passes locally with the rest of MarginCallModelTests.
What I checked in the new sizing path:
- Short groups. The search passes a negative group quantity to
WithQuantityfor a short group.WithLotsthen flips it to a long position of the same size. For equities the margin is the same on both sides, so the sizing holds. I checked locally: -100 IBM with a +10 open buyback and a $1,000 deficit generates+10for the $500 left, and long IBM mirrors it with-10. Option strategy groups such as naked calls have a positive group quantity (10 for -10 calls), so they never reach the flip. With an open+2buyback of 10 naked calls ($1,100 each), deficits of $1,300, $2,100 and $3,800 after it generate+2,+2and+4, the fewest calls that cover each one. - Bounds. A deficit larger than the group's margin gives the whole group. A fractional lot count is capped at what's held. The loop only runs while there's a deficit.
- Fees. Unlike the buying power model, the search doesn't leave room for fees, so a fill can leave a deficit of a few dollars. That's well inside the margin buffer that gates new orders, so it doesn't cause another margin call.
- Cost. The search runs about log2(lots) margin evaluations, and only for regrouped or partly sold groups during a margin call. That's negligible.
CI: only Benchmarks fails, on EmptyFuturesBenchmark (C#, 395k vs a 401k floor in the re-run). I agree with the diagnosis on the pull request. Neither benchmark that failed places an order (Algorithm.CSharp/Benchmarks/EmptyFuturesBenchmark.cs, EmptySingleSecuritySecondEquityBenchmark.cs), so GetMarginCallOrders returns at totalMarginUsed <= 0, before any changed code. The previous commit passed the same workflow, and this commit only changes GenerateMarginCallOrders. Also, the same algorithm measured 446k earlier in the same job. Every other check passes.
Testing: the new tests fail on master, the full suite passes, and the change was tested end to end. That end-to-end run can't reach the regrouped option-strategy path, but the unit tests above cover it.
Other repositories: none need an update.
Risk: I kept risk:high. The change alters how much a margin call liquidates and which requests OnMarginCall receives, which is money and behaviour users can see. It isn't a small revert either. Nothing more would lower that: it's the kind of change, not missing evidence.
There was a problem hiding this comment.
Requesting changes on 5e74b88b1. The earlier unheld-strategy sizing reproduction is fixed, but I reproduced three remaining liquidation failures:
- Rejected orders lose their fallback. Generation stops on projected coverage before submission. If that order is rejected, execution has no alternative, and subsequent scans select the same failing group again. A reproduction using Bitfinex's actual minimum-size validation repeatedly rejects BTC while never attempting a valid ETH liquidation; master attempts ETH after the rejection. This violates #10107's cancellation/rejection recovery requirement.
- A held stock group sharing a security can still be skipped instead of capped. The buying-power model sizes from the security's full holdings, not the selected group's allocation. In the reproduction below it requests 201 shares from a 200-share group; the cumulative guard drops the whole request, leaving $9,000 of the deficit uncovered despite those shares being available.
- The synthetic sizing search reverses residual short positions. With strategy grouping disabled but normal option margin retained, a partial short-call buyback causes the search to evaluate long calls instead. It generates one additional buyback where two are needed, leaving $600 uncovered.
Please add regression coverage for these cases and preserve recovery after a terminal order failure without restoring duplicate full-deficit commitments. The inline comments contain the setups and relevant call paths.
Verification: all 21 existing MarginCallModelTests cases passed locally. Four new assertions failed: three shared-stock variants and the residual-short case; the ordinary strategy-grouped short-call control passed. Temporarily capping the held-group lot count and removing the extra sign in WithQuantity made all 26 selected cases pass; those experimental edits were reverted. I also reran the rejection reproduction against the PR and an otherwise unchanged copy of master's margin-call model. This uses a test order processor with real brokerage-model validation, not a live brokerage connection. I did not repeat the full suite or the end-to-end backtest described in the PR.
CI: Build & Test Lean and the regression checks pass. Benchmarks remains red: its re-check reports EmptyFuturesBenchmark at 395k against a 401.4k floor, after 446k for the base comparison. I verified the failing log and the benchmark source: it never trades, so GetMarginCallOrders returns at the unchanged totalMarginUsed <= 0 guard. I agree that this failure is unrelated to the changed sizing path.
Other repositories: none need a source or contract update.
Risk: kept risk:high; the demonstrated failures can leave margin deficits uncovered, including indefinitely after a rejected liquidation.
Each margin call order is sized for the deficit the earlier ones leave, so one the brokerage rejects or cancels left part of it uncovered, and later scans picked the same failing group again. ExecuteMarginCall now liquidates the other position groups for the deficit left, excluding the groups whose orders didn't fill. A held group's lots are capped at its own quantity, since the security's buying power model sizes from all of its holdings, which other groups can share. The search for groups that aren't held keeps the direction of short positions instead of flipping them, which matters for asymmetric margin such as short options. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the second review in 85d4f9b:
There's a new test for each, and all three fail on the previous commit. |
There was a problem hiding this comment.
Requesting changes on e9d6c2676, pending a maintainer decision on the existing recovery regression. No code has changed since my previous review; this review reassesses the November 16 replies.
I withdraw the closed-exchange finding. The comparison with an earlier PR commit was not sufficient. On origin/master, generation includes both IBM and MSFT, and AlgorithmManager's all-exchanges-open check also prevents an IBM retry while MSFT is closed. This is not a regression against the base branch, and I am not asking this PR to change that behavior. The delayed terminal-failure limitation likewise remains separate, not an additional blocker here.
One existing blocker remains: planning away the execution alternatives loses a viable fallback after combo rejection. In the covered-call reproduction (300 SPY at $100, one short call at $1, leverage 2, cash -$19,000), the $4,100 deficit is projected to be covered by the combo. Once that combo fails, excluding its symbols also excludes the standalone SPY group, and the empty-result reset selects the combo again. The base model instead retains and submits the standalone SPY -83 alternative. Alpaca's brokerage-model validation rejects the equity ComboMarket leg but accepts a standalone Market sale.
This is the same cause as the earlier recovery findings: generation now returns a projected liquidation plan rather than alternatives execution can fall through. I agree with stopping the sequence of recovery patches. Please have a QuantConnect maintainer choose whether to accept this limitation or change the sizing/execution approach, as requested in the author's November 16 comment. I am not prescribing another exclusion-set patch. Option C also needs an explicit decision about resizing requests after OnMarginCall; it is not automatically an acceptable replacement.
Verification: all 44 existing margin-call-related tests pass locally. I added four temporary NUnit controls: both closed-exchange controls submit nothing during six subsequent scans after the IBM rejection is disabled; the rejected-combo control fails on this head after six combo-only attempts and passes with an otherwise unchanged copy of origin/master's model, which submits SPY -83 on the first execution. The rejection simulation is checked against the actual Alpaca brokerage model, not an Alpaca connection. Temporary source changes were removed. I did not repeat the full suite or the end-to-end runs described in the PR.
CI: all seven active checks pass; API Tests is skipped.
Other repositories: none need a source or contract update for the current change.
Risk: kept risk:high. The confirmed regression can leave the portfolio under-margined despite an available liquidation. A maintainer's approach decision and evidence that the chosen behavior is implemented are still needed.
|
Going with C. The plan moves the fallback out of the call where the failure is known, and putting it back has needed state across scans every time. Execution already knows which orders failed, so the fallback belongs there.
|
GetMarginCallOrders returns one request per position group again, each regrouped from the holdings the open margin call orders leave and sized for the deficit they leave. ExecuteMarginCall executes the first group as given and scales each later one down to the fewest of its lots that cover the deficit the earlier orders leave, so a failed order falls through to the next group in the same call. Groups whose margin call orders failed during the current margin call are executed last. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Implemented option C in e6845f5:
The rejected covered-call combo case from the last review now liquidates the standalone SPY group in the same call ( Full suite, Release, CI filter: |
There was a problem hiding this comment.
Approving e6845f568. It implements option C as the maintainer decided on November 17, and I found nothing that blocks it.
What changed since my last review (e9d6c2676): _failedLiquidationSymbols and the cumulative sizing are gone. GetMarginCallOrders returns one request per group again, regrouped and sized for the deficit the open orders leave. ExecuteMarginCall scales each later group down for what the earlier orders of the same call leave, and puts groups whose orders failed last. I checked each part of the decision:
- One alternative per group, no exclusion. Every group gets a request. A failed order releases no margin, so the next group's search needs the whole deficit and runs as generated, as on master. This resolves the second reviewer's rejected-combo case: the standalone SPY group isn't excluded any more.
LiquidatesAStockGroupSharingTheUnderlyingOfARejectedComboOrdercovers it (-82for the $4,100). The rejected-BTC case from the first round falls through to the next group in the same call too. - Only ever reduced.
GetQuantitycaps every leg at its requested quantity, and the search starts at the requested lots. So padding thatOnMarginCalladds stays on the first group, and later groups shrink only to the lots that cover what's left. For combos, the lots never go above|GroupOrderManager.Quantity|, so the cap never applies to one leg without the others, and the leg ratios hold. The scaled combo gets a newGroupOrderManagerwith the scaled quantity. - Failed orders go last, nothing left out. Failed tickets (
InvalidorCanceled) stay until margin remaining is above zero or no margin is used. Each one counts against the group with exactly its symbols, byGroupOrderManagerfor combos.GetOpenMarginCallOrderQuantitiesfilters onIsOpen, so the kept failures never count as pending.ExecutesThePositionGroupsWhoseMarginCallOrdersFailedLastcovers a cancel that comes after a scan has seen the order open. That is the delayed-failure case recorded on #10107, and here it moves the group back instead of looping. - No failures. All counts are zero, so execution is losers first as on master, and the first group runs as generated.
Other checks, no change needed:
- Search bounds.
LotSizeis validated as positive inSymbolProperties, so the per-lot quantity is never zero. Requests without quantity, and combos with a zero group quantity, skip the scaling.ExecutesLaterMarginCallRequestsWithoutQuantityAsGivencovers that. The search takes about log2(lots)GetMarginReleasedevaluations per later group (around 30 for a crypto position at a 1e-8 lot size), and only during a margin call. - Holdings across groups. Groups partition the holdings left after the open orders, each request is capped at its group's quantity, and execution only reduces. So a call can't sell more than is held.
DoesNotSellMoreThanHeldAcrossPositionGroupsSharingALegcovers it. - Fees. A scaled order covers the deficit before fees, unlike the first group, which the buying power model sizes. A fill can leave margin remaining a few dollars below zero. That's well inside the margin buffer that triggers margin call orders, so it doesn't start another one. The previous approvals accepted the same for the regrouped-group search.
CI: every check passes on this commit; one is skipped. Locally, all 45 margin call tests pass on a Release build of this commit.
Testing: the description lists new tests that fail on master, a passing full suite, and the margin call regression algorithms with unchanged statistics. It was also tested end to end, with and without a brokerage model rejecting one or two stocks' margin call orders. In each run, a rejection fell through to the next stock in the same call.
Other repositories: none need an update. IMarginCallModel, the engine's exchange-open check and OnMarginCall are unchanged.
Risk: I kept risk:high. The change alters how much a margin call liquidates, and it reduces later requests that OnMarginCall set. That's money and behaviour users can see, and the change isn't a small revert. The kind of change sets the label, not missing evidence.
There was a problem hiding this comment.
Requesting changes on e6845f568. It implements the maintainer's option C and resolves the earlier rejection-fallback/shared-underlying findings. The held-group cap and residual-short fixes still pass, and the delayed-failure test preserves recovery progress. I am resolving my earlier threads, including the previously withdrawn closed-exchange finding.
One new blocker: execution-time downsizing can turn valid liquidations into rejected orders. The search considers maintenance-margin release, but not whether the reduced request remains executable under the existing buying-power and brokerage checks. I verified two cases of this same cause:
- Protective puts: 1,000 SPY at $100, ten $100-strike puts at $1, and 100 IBM at $100 bought at $200; cash -$102,500, leverage 2. Equity is $8,500, maintenance $15,000, and the deficit $6,500. IBM executes first, releasing $5,000. The generated ten-lot protective-put liquidation passes
Portfolio.HasSufficientBuyingPowerForOrder, but execution reduces it to two lots. That request fails the same real validation: the remaining eight lots require $40,000 of initial margin against the held group's $10,000 maintenance. Master submits the valid ten-lot combo. The rejection also occurs if IBM has already filled rather than remaining open. - Brokerage minimums: with Bitfinex's actual symbol properties and validation, both generated crypto liquidations are valid, but the later ETH order is reduced below its
0.0008minimum. Tested end to end on December 14, 2022 minute data: the validETHUSD -0.00113768candidate becomes-0.00059873and is rejected. Three subsequent scans also reject undersized ETH orders; final margin remaining is -$0.39704. The otherwise identical master-model control fills the original candidate and finishes with +$0.35411322 margin remaining. This is brokerage-model validation in a backtest, not a brokerage connection.
Please preserve executability when reducing later requests, within the original requested quantities, and add regressions that exercise the actual position-group buying-power check and brokerage minimum-size validation. The inline comment anchors the shared sizing cause.
Verification: all 45 margin-call-related NUnit tests pass locally in Release. Both additional reproductions were checked against the base implementation; the crypto case was also tested end to end as above. I did not repeat the full suite or regression-algorithm set reported in the description.
CI: all seven active checks pass; API Tests is skipped.
Other repositories: QuantConnect/Documentation still needs the margin-call default-behavior/event documentation updated for failure-priority ordering and reduction of later OnMarginCall requests. No brokerage source or API-contract update is needed.
Risk: kept risk:high, because the verified downsizing failures can leave a portfolio under-margined despite an executable liquidation being available.
| { | ||
| quantities[requests[i].Symbol] = quantities.GetValueOrDefault(requests[i].Symbol) + GetQuantity(i, lots); | ||
| } | ||
| if (marginRemaining + GetMarginReleased(quantities) >= 0) |
There was a problem hiding this comment.
[P1] Preserve executability when downsizing liquidation requests
This predicate only checks maintenance-margin release. A smaller request can satisfy it while failing the real order-validation rules that the original candidate satisfies. With 1,000 SPY, ten ATM protective puts and the earlier IBM liquidation described in the review, the search shrinks a valid ten-lot combo to two lots; Portfolio.HasSufficientBuyingPowerForOrder rejects it because the residual group's initial margin exceeds its held maintenance. BrokerageTransactionHandler uses that check and invalidates the orders. Independently, the same search reduces a valid Bitfinex ETH sale below MinimumOrderSize; the end-to-end control leaves negative margin while master fills the original candidate.
Please account for the existing initial-margin/order-admissibility constraints when choosing a reduced quantity, without exceeding the supplied request, and test the resulting orders through those real checks rather than checking projected maintenance relief alone. These are two manifestations of the same downsizing cause, not a request to change the brokerage or position-group validation rules.
There was a problem hiding this comment.
Fixed in a85ae37: when every order of a scaled-down group is rejected, ExecuteMarginCall executes the requested orders instead, so they go through the same buying-power and brokerage checks, and the smaller ones don't count as failures of that group. Groups are also scaled only once an earlier order of the call is filled or open, so after failed orders alone the next group runs as generated. Tests cover both of your cases through Portfolio.HasSufficientBuyingPowerForOrder and DefaultBrokerageModel.IsValidOrderSize.
…ejected A smaller order can fail checks the requested one passes, such as the initial margin of the holdings it leaves or the brokerage's minimum order size. When every order of a scaled down group is rejected, the requested orders are executed instead, and the smaller ones don't count as failures of those position groups. Groups are only scaled down when an earlier order of the call didn't fail, so after failed orders alone the next group is executed as generated for the whole deficit, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the latest review in a85ae37:
Full suite, Release, CI filter: |
There was a problem hiding this comment.
Approving a85ae376d. It resolves the second reviewer's finding on e6845f568, and I found nothing new that blocks it.
What changed since my last review (e6845f568):
- Falling back to the requested order. When every order of a scaled-down group comes back
Invalid,ExecuteMarginCallremoves those tickets from the failure bookkeeping and executes the generated requests instead. This happens only for a group that was actually scaled (requests != generatedRequests). So the requested order goes through the same buying-power and brokerage checks it would face on master, and execution never sends more than was requested. Because the smaller tickets are removed from_marginCallTickets, a group counts as failed at most once per call. They're still returned and logged as executed orders, the same way master returns and logs rejected margin call orders. - No scaling after failures alone. Scaling now starts only once an earlier order of the call is filled or open. After failed orders alone, the next group runs as generated, so the fall-through cases match master's quantities again (
SPY -81, andSPY -83for the rejected combo). Before, they were one share short of what the buying power model sizes with fees. - The stop check moved below the fallback, so it also sees margin released by the requested orders.
Verification: locally, all 47 margin call tests pass on a Release build of this commit. With only DefaultMarginCallModel.cs reverted to e6845f568, both new tests fail, and so do the three that now expect master's quantities. FakeOrderProcessor.RejectsOrder now sees all of a combo's legs before it decides, so the protective-put test checks the combo as one order through Portfolio.HasSufficientBuyingPowerForOrder, as the transaction handler does.
Checked, no change needed: the fallback reuses the original GroupOrderManager, which nothing submitted before. The scaled orders had their own. A non-combo group is always a single request, so "every order rejected" can't be a partial rejection. If the requested order is rejected too, that's one failure of the group, and the next group is scaled for what the filled orders leave, since nothing more was released.
CI: every check passes on this commit; one is skipped.
Testing: the new tests fail on the previous commit, the full suite passes, and the margin call regression algorithms are unchanged. It was tested end to end, including a crypto run under a brokerage minimum order size: there the requested order now fills in the same margin call, where the previous commit left the deficit uncovered.
Other repositories: QuantConnect/Documentation#2915 documents the new default behavior, and I'm approving it alongside this one. No other repository needs an update: IMarginCallModel, the engine's exchange-open check and OnMarginCall are unchanged.
Risk: I kept risk:high. The change alters how much a margin call liquidates and reduces later requests that OnMarginCall set, which is money and behavior users can see, and it isn't a small revert. It's the kind of change that sets the label, not missing evidence.
There was a problem hiding this comment.
Requesting changes on a85ae376d. The synchronous protective-put and minimum-order-size cases from my previous review are fixed. One manifestation of that same executability finding remains: a scaled order rejected after the order wait times out loses its valid original alternative.
I reproduced this with the real background BrokerageTransactionHandler, a live-mode algorithm, and the five-second market-order wait:
- Hold 100 IBM at $10, bought at $20, and 100 MSFT at $100; leverage 2, cash -$7,000. Equity is $4,000, maintenance margin $5,500, and the deficit $1,500. MSFT has a 25-share minimum. Generation supplies
IBM -100and the validMSFT -31. - IBM remains submitted. Execution reduces MSFT to
-20. A synchronization gate delays the realDefaultBrokerageModel.IsValidOrderSizerefusal until after the wait. When execution returns, the reduced ticket is stillNew; after releasing the gate it becomesInvalid, butMSFT -31is never submitted. Three subsequent scans each submit and rejectMSFT -21, leaving the $1,000 gap after IBM's projected relief uncovered. - The identical asynchronous test passes with only the margin-call model replaced by the base-branch implementation: it submits
MSFT -31in the first execution. A separate delayed-response test that fills IBM likewise leaves margin remaining at -$1,000 on this head, versus +$550 with the base model. These are controlled local validation tests, not a brokerage connection.
AddOrder waits for the ticket's order-set signal, which precedes validation (BrokerageTransactionHandler.cs:374-376,405-418,923,962-983). Therefore an order-set ticket is not proof that validation finished. The one-time all-Invalid check cannot provide the fallback once WaitForOrder times out. Please preserve the originally supplied executable alternative through the reduced request's terminal result, without submitting both while the reduced request can still fill, and cover this asynchronous path. This is the existing downsizing/executability cause, not a request for another independent recovery strategy.
Verification: all 47 existing margin-call-related tests pass locally in Release. The additional asynchronous regression fails on this head and passes with the base model. I also tested the synchronous path end to end with deterministic prices and a synthetic 25-share minimum: the reduced 20-share order is rejected, the original 30-share order fills, and margin remaining ends at +$500. That ordinary backtest uses synchronous transaction processing and does not exercise the asynchronous failure above. I did not repeat the full suite or all regression algorithms reported in the description. Temporary source changes were removed.
CI: all seven active checks pass; API Tests is skipped.
Other repositories: QuantConnect/Documentation#2915 supplies the required documentation update, but the pair is not ready until this remaining Lean finding is resolved. No brokerage source or API-contract update is needed.
Risk: kept risk:high on both PRs. The remaining regression can leave an account under-margined despite an executable liquidation; the change also affects liquidation quantities. My earlier executability thread remains open.
|
@Martin-Molinero, one more decision is needed on option C before I change anything else. The finding (second reviewer, on a85ae37): later groups are scaled down to the remaining deficit. A scaled order can fail checks the requested one passes, such as the initial margin of the holdings it leaves or a brokerage's minimum order size. a85ae37 covers that when the rejection comes back while Closing that gap means remembering each scaled order's requested alternative until the scaled order is closed, across scans. That's the kind of cross-scan state you moved away from with option C, so I'd rather you choose:
The second reviewer reproduced this with a gate that delays validation in the real background transaction handler. Their numbers: 100 IBM and 100 MSFT, a 25-share MSFT minimum, and MSFT scaled from |
|
Neither. Check the scaled size before sending it instead of after, so nothing waits on a rejection or needs remembering across scans.
|
There was a problem hiding this comment.
Requesting changes on a85ae376d, pending the maintainer decision requested in the author's November 17 reply. Neither this head nor Documentation#2915 has changed since my previous review. There are no new independent findings.
The existing P1 remains: a reduced liquidation rejected after the order wait ends loses its executable original alternative. The reply correctly confirms the limitation, rather than resolving it. ExecuteMarginCall checks for Invalid only immediately after its bounded wait; only the reduced ticket survives afterward. The real transaction handler signals OrderSet before buying-power and brokerage validation, so this is a reachable asynchronous path. In the previously reproduced same-head case, valid MSFT -31 becomes -20, the refusal arrives after the wait, and later scans repeatedly generate below-minimum quantities. The base-model control submits the valid original request.
This is the remaining form of the earlier downsizing/executability finding, not a request for another independent recovery patch. Please have a maintainer resolve the pending choice before merging or adding further recovery state. The earlier option C decision authorizes execution-time reduction but does not explicitly accept this subsequently demonstrated loss of an executable alternative. If the chosen approach changes the issue's intended result, that scope decision needs to be explicit too.
Verification: rechecked the unchanged head, the execution/transaction-handler lifecycle, the base implementation, and the latest discussion. The earlier same-head asynchronous regression and base-model control remain the reproduction evidence; I did not rerun NUnit or end-to-end tests in this pass. The synchronous tests and end-to-end runs described in the PR do not exercise rejection after the wait ends.
CI: all seven active checks pass; API Tests is skipped.
Other repositories: QuantConnect/Documentation#2915 remains paired with this change and must reflect the selected fallback behavior. No additional brokerage, interface, wrapper, or stub update is needed.
Risk: kept risk:high on both PRs. The unresolved regression can leave an account under-margined despite an executable liquidation; a documented limitation alone does not resolve that without the pending maintainer decision. My existing executability threads remain open.
A later group is scaled down only if every leg of the smaller order passes DefaultBrokerageModel.IsValidOrderSize; otherwise the requested orders are executed, as on master. The fallback that executed the requested orders after the scaled ones were rejected is removed, so nothing waits on a rejection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Implemented the November 17 decision in 8301c76:
All 46 margin call tests pass, and so do the margin call regression algorithms, with unchanged statistics. Full suite, Release, CI filter: |
There was a problem hiding this comment.
Approving 8301c7609. It implements the maintainer's November 17 decision, and I found nothing that blocks it. Merge it only after #10179 is fixed, as that decision says: until then, a scaled-down partial close of a protective put fails the position group buying power check. That group then goes last on the next margin call, where it's executed as generated, but the deficit stays uncovered until then.
What changed since my last review (a85ae376d):
- Size checked before sending. A later group is scaled down only if every leg of the smaller order passes
DefaultBrokerageModel.IsValidOrderSize. Otherwise the requested orders go out unchanged, as on master. For combos the check runs on each leg's total quantity, which is what the order processor validates. Nothing now depends on when a rejection arrives. That resolves the second reviewer's finding about rejections that arrive afterWaitForOrdertimes out: the scaled order that came back late was below the minimum, and it's no longer sent. - The fallback after a rejection is gone.
ExecuteMarginCallis back to one submit loop per group. The stop check still runs after each group, against the open margin call orders. - Tests. The protective-put test is removed, since that rejection is #10179.
ExecutesTheRequestedOrderWhenTheScaledDownOneIsBelowTheMinimumOrderSizenow checks that the 20-share order is never sent.
Verification: locally, all 46 margin call tests pass on a Release build of this commit. With the new size check bypassed, ExecutesTheRequestedOrderWhenTheScaledDownOneIsBelowTheMinimumOrderSize fails.
Not blocking: some brokerage models check the minimum against something other than quantity. BinanceBrokerageModel compares the order's notional value with MinimumOrderSize (Common/Brokerages/BinanceBrokerageModel.cs:257). There, the quantity check usually refuses to scale, so the requested order goes out as on master. For an asset priced under $1, though, a scaled quantity can pass the quantity check and still fall under the notional minimum. That case counts as a failure of the group, as the decision accepts for brokerage rejections, and the group is executed as generated on the next margin call.
CI: every check passes on this commit; one is skipped.
Testing: the full suite passes. The one failure is a futures expiry test that has nothing to do with margin calls and passes when run again. The margin call regression algorithms are unchanged. It was also tested end to end, including under a crypto brokerage minimum order size: the requested order now fills in the same margin call, and no order is rejected.
Other repositories: QuantConnect/Documentation#2915 now describes the minimum-size check, and I'm approving it too. No other repository needs an update: IMarginCallModel, the engine's exchange-open check and OnMarginCall are unchanged.
Risk: I kept risk:high. The change alters how much a margin call liquidates and reduces later requests that OnMarginCall set. That's money and behavior users can see, it isn't a small revert, and it has to merge after #10179.
There was a problem hiding this comment.
Requesting changes on 8301c7609 only until the maintainer's stated prerequisite, #10179, is fixed and included. The latest minimum-size change implements the November 17 decision; I found no new independent defect in that patch.
The minimum-size/asynchronous finding is resolved. Every leg is checked with DefaultBrokerageModel.IsValidOrderSize before the reduced group is submitted; otherwise the original requests are submitted. The after-rejection fallback is gone, so the reproduced below-minimum rejection can no longer outlast that fallback's wait. I am resolving my asynchronous-validation and pending-decision threads.
One existing blocker remains: the protective-put prerequisite has not landed. #10179 is still open, and the current buying-power implementation still compares the residual group's initial margin with the held group's maintenance margin. I reran the earlier protective-put reproduction against this head: equity $8,500, maintenance $15,000, and a $6,500 deficit; IBM's liquidation projects $5,000 relief. The original ten-lot protective-put liquidation passes Portfolio.HasSufficientBuyingPowerForOrder, but execution replaces it with two lots (SPY -200, puts -2), and both legs become Invalid through that same real validation. The identical test passes with only the margin-call model replaced by origin/master's implementation, which submits the valid ten-lot alternative.
This is the already-discussed executability finding, not a request for another fallback patch or for moving #10179 into this PR. Please land that prerequisite and verify the partial close with it included before merging this change. The protective-put part of my existing executability thread remains open.
Verification: all 46 existing margin-call tests pass in Release. The additional protective-put control fails on this head and passes with the base model. Tested end to end with the Bitfinex margin brokerage model and December 14, 2022 minute data: BTC -0.0002 fills, leaving about $0.20 uncovered; instead of an ETH reduction below its 0.0008 minimum, the original ETH -0.00299470 fills in the same margin call. Two fills, no rejections, final margin remaining +$1.777853325. This uses brokerage-model validation in a backtest, not a brokerage connection. Temporary source changes were removed. I did not repeat the full suite or all regression algorithms.
CI: all seven active checks pass; API Tests is skipped.
Other repositories: Documentation#2915 has merged and its changed pages describe the minimum-size behavior. No further brokerage, interface, wrapper, or stub update is needed.
Risk: retained risk:high: the required buying-power fix is absent, and the current partial-close path still turns an executable liquidation into a rejection. The change also alters later liquidation quantities after OnMarginCall.
|
No code change for the latest reviews. Both confirm that 8301c76 implements the November 17 decision. The only open item is the protective-put partial close, which the decision assigns to #10179. That issue is still open with no fix yet, and per the decision, recorded in the description, this pull request merges after it. Once #10179 is fixed, the partial close should be verified on top of it before this merges. QuantConnect/Documentation#2915, which documents this behavior, has already merged. |
Description
DefaultMarginCallModelsized every position group's margin call order from the group's full current holdings, andExecuteMarginCallexecuted each one as generated until the deficit was covered. So:Approach, as decided by a maintainer on this pull request (option C): generation returns one alternative per group, and execution sizes each one for what the earlier ones leave, so a failed order falls through to the next group within the same call, as on master.
GetMarginCallOrders:GenerateMarginCallOrders, a group held as is gets sized by its buying power model, as before, with its lots capped at its own quantity, since the security's buying power model sizes from all of the security's holdings, which other groups can share. A group that only exists once the open orders fill, such as naked calls left from covered calls whose stock is being sold, is liquidated for the fewest of its own lots whose margin covers the deficit, up to all of them.OnMarginCallrequest larger than the position.ExecuteMarginCall:DefaultBrokerageModel.IsValidOrderSize, the symbol's minimum order size. Otherwise the requested orders are executed, as on master. Nothing waits on a rejection.OnMarginCalladds or changes are only ever reduced, so padding on the first group stays.InvalidorCanceled) are kept until the margin call ends, when margin remaining is back above zero. Groups are executed in order of how many of their margin call orders failed during this margin call, then losers first. A failed order counts against the group with exactly its symbols, so a rejected covered call combo doesn't push back a separate stock group on the same underlying. Nothing is excluded, so nothing needs resetting. This also covers an order that stays open through a scan and is canceled afterwards.IMarginCallModel, the engine's exchange-open check andOnMarginCallare unchanged. With no open margin call orders and the first group's order covering the deficit, the orders executed are the same as before.Maintainer decisions
ExecuteMarginCallscales later groups down to what the earlier orders leave, with failed groups executed last (Size margin call orders for the deficit earlier liquidations leave #10131 (comment) and the reply after it).DefaultBrokerageModel.IsValidOrderSize, otherwise the requested orders are executed. There's no fallback after a rejection. The protective-put buying-power rejection of a partial close is Partly closing an option strategy can fail the buying power check #10179, which this pull request merges after. A brokerage rejecting a scaled order for a reason Lean doesn't model counts as a failure of that group, which goes last on the next margin call.Related Issue
Closes #10107
How Has This Been Tested?
New tests in
MarginCallModelTests, all failing on master:SizesEachPositionGroupForTheDeficitAndHoldingsTheEarlierLiquidationsLeave: 100 IBM (the loser) and 100 MSFT at $100, $4,000 deficit. Both get-81, and only IBM's is executed, which covers it. With it open, MSFT falls to $45 leaving a $2,700 deficit: MSFT-100and the 19 IBM shares left are generated, and IBM is scaled to-9for the $450 MSFT leaves.LiquidatesTheStockOfACoveredCallWhoseCallIsBeingBoughtBack: 3 covered calls,OnMarginCallreplaces the combo with an order per leg, and the stock order is cancelled while the call buyback stays open. The 300 shares regroup into a stock group and are liquidated (-300).LiquidatesPositionGroupsThatAreNotHeldForTheDeficit: 3 covered calls,OnMarginCallsells only the stock, which stays open, and cash drops to -$29,000:-300: the 3 calls left are naked, $3,300 of margin against a $2,600 deficit: all 3 are bought back (+3).-150: a covered call, 50 shares and 2 naked calls against a $9,000 deficit. Executing them, the covered call (-100/+1, $5,000) and the 2 naked calls (+2, $2,200) go as generated and the shares are scaled to-36($1,800).LiquidatesTheNextPositionGroupForTheWholeDeficitWhenAMarginCallOrderFails, rejected and canceled: IBM, MSFT and SPY can each cover the $4,000 deficit. IBM's and MSFT's orders fail, and SPY is liquidated in the same call as generated (-81).LiquidatesAStockGroupSharingTheUnderlyingOfARejectedComboOrder: a covered call (the loser) and a 200 SPY group, $4,100 deficit, and combo orders are rejected. The stock group is liquidated in the same call as generated (-83), as on master.ExecutesTheRequestedOrderWhenTheScaledDownOneIsBelowTheMinimumOrderSize: 100 IBM (the loser) and 100 MSFT with a 25-share minimum order size, whose orders the order processor checks withDefaultBrokerageModel.IsValidOrderSize. Selling IBM leaves $1,000, the 20 MSFT shares that cover it are below the minimum, so the requested MSFT order is executed and nothing is rejected.ExecutesThePositionGroupsWhoseMarginCallOrdersFailedLast: IBM's order is rejected and MSFT's covers the deficit but is canceled after a scan saw it open. The next call executes SPY first, then IBM and MSFT. Once all of them failed again, SPY (one failure) goes before IBM and MSFT (two each). After margin remaining is back above zero, the next margin call starts with IBM, the loser, again.ExecutesLaterMarginCallRequestsWithoutQuantityAsGiven: a later request without quantity, asOnMarginCallcould make, is submitted as given.LiquidatesAllTheSharesOfAStockGroupSharingTheSecurityWithAnotherGroup: a covered call and a 200 SPY group with no equity left. The stock group is liquidated with its own 200 shares (-200), not sized from all 300 SPY and then dropped.LiquidatesTheShortPositionsLeftAfterOpenMarginCallOrders: without option strategy grouping, 3 short calls ($1,100 each). With a+1buyback open and $500 of equity, the 2 left are both bought back (+2).DoesNotSellMoreThanHeldAcrossPositionGroupsSharingALeg: covered call plus 200 SPY, the combo liquidation cancelled with the-50stock order open, and SPY falls to $80. The 250 shares left regroup into a covered call and a 150 SPY group. The stock group, whose orders haven't failed, is executed first (-150), then the combo (-100/+1): 300 shares with the open order, all of those held.Updated tests:
LiquidatesPositionGroupsSharingALegForTheDeficitOpenMarginCallOrdersLeave: the stock group's executed order is-50for the $2,500 the combo leaves. With the stock order open and the combo cancelled, both groups left are generated for the $5,000 left.GeneratesMarginCallOrdersForTheDeficitOpenMarginCallOrdersDoNotCover: MSFT is executed as-20for what selling all of IBM leaves, so the later numbers change accordingly.StopsExecutingMarginCallOrdersOnceTheOpenOnesCoverTheDeficit: IBM and MSFT are generated, and only IBM's, which stays open and covers the deficit, is executed.FakeOrderProcessorgainedRejectsOrder, checked with the requests of all of an order's legs once they're submitted, andRejectedOrderStatus, to simulate rejections.Full test suite, Release build, with the CI filter:
Passed: 39610, Skipped: 100, Failed: 1. The failure isFutureSymbolGeneratorTests.GeneratesFutureSymbolWithCorrectExpiryDateOverWiderDateRange, aNullReferenceExceptioninFuturesExpiryUtilityFunctions.GetExpirationHolidays, which doesn't touch margin calls and passes when run again. The margin call regression algorithms (MarginCallEventsAlgorithm,OptionStrategyMarginCallEventsAlgorithm, the short ITM expiry ones and the others using margin calls, C# and Python) pass with unchanged statistics.Tested end to end with a backtest holding SPY, AIG, GS and GE on daily data at 3.8x leverage, September 22 to October 31, 2008, against master:
GE -225for what AIG leaves, where master executed AIG plusGE -1000.GE -1000is executed in the same call for the whole deficit, as on master. Later margin calls execute AIG, whose orders failed, last, and the other stocks are scaled for what the earlier ones leave.SPY -971in the same call; later ones liquidate SPY and GS before retrying AIG and GE.Also tested end to end with the Bitfinex brokerage model on a margin account, BTCUSD and ETHUSD minute data on December 14, 2022: BTC (the loser) and ETH held, with a deficit that BTC's liquidation covers but for $0.20. BTC's order fills. Scaled for the $0.20 left, ETH would be
-0.00077191, below Bitfinex's 0.0008 minimum, so the requested-0.01is sent and fills in the same margin call, with no rejected order, leaving positive margin remaining. On e6845f5 the same run leaves the deficit uncovered for two hours of further small margin calls.A documentation update for the new default behavior is in QuantConnect/Documentation#2915.