vaultFactory contract not upgradable #8079

turadg · 2023-07-21T15:18:32Z

Describe the bug

The vaultFactory contract in master fails upgrading the vaultFactory contract deployed to mainnet (simulated in docker upgrade-test scripts). There are multiple hurdles. (TO INCLUDE)

To Reproduce

Steps to reproduce the behavior:

check out 555b517 (from WIP: in VaultFactory, move auctioneer public facet from terms to private args #8049)
in packages/deployment/upgrade-tests: make build run
bundle the vault factory outside the container: bundle-source --cache-json /tmp packages/inter-protocol/src/vaultFactory/vaultFactory.js vaultFactory and then docker cp /tmp/bundle-vaultFactory.json $CONTAINERID:/tmp where $CONTAINERID is from docker ps.
inside the container, use upgrade scripts from outside the container: mv upgrade-test-scripts xx-scripts; ln -s /workspace/upgrade-test-scripts
mint some IST to cover the cost of installing the bundles on chain: . ./upgrade-test-scripts/agoric-upgrade-11/mint-ist.sh
run the upgrade script from within docker at /usr/src/agoric-sdk: ./upgrade-test-scripts/agoric-upgrade-11/auctioneer-upgrade-driver.sh
see various errors (details in comments below), though the kernel reports the contract vat is upgraded and the core eval script does report successful completion.

Expected behavior

vault factory upgrade does not produce errors; contract works after upgrade

Platform Environment

what OS are you using? linux, macos. what version of Node.js? whatever's in the upgrade-test docker stuff
is there anything special/unusual about your platform? upgrade-test docker stuff
what version of the Agoric-SDK are you using? 555b517

Additional context

work on upgrading vault factory to connect to a different auctioneer: #8049

Screenshots

We have a more complete log file internally.

The text was updated successfully, but these errors were encountered:

dckc · 2023-07-21T16:09:12Z

problem: `stabilityFeeSnapshot` vs `interestSnapshot` (fix available)

symptoms: Error: durable Kind stateShape mismatch (body) (full stacktrace below)
diagnosis: while renaming interest rate to stability fee (IOU issue ref), we didn't account for migration. The migration code would include the term interestSnapshot anyway, so the benefit of doing the migration is small. And changing exo state shapes isn't supported yet (IOU issue ref).
treatment: rename stabilityFeeSnapshot back to interestSnapshot (2a447fc + 3f2fb04)

full stacktrace

2023-07-20T18:12:25.898Z SwingSet: xsnap: v48: Error#1: durable Kind stateShape mismatch (body)
2023-07-20T18:12:25.898Z SwingSet: xsnap: v48: Error: durable Kind stateShape mismatch (body)
 at construct ()
 at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:56)
 at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:243)
 at fail (/bundled-source/.../node_modules/ses/src/error/assert.js:357)
 at insistSameCapData (/bundled-source/.../packages/swingset-liveslots/src/virtualObjectManager.js:287)
 at defineKindInternal (/bundled-source/.../packages/swingset-liveslots/src/virtualObjectManager.js:807)
 at defineDurableKindMulti (/bundled-source/.../packages/swingset-liveslots/src/virtualObjectManager.js:1186)
 at prepareVault (.../inter-protocol/src/vaultFactory/vault.js:870)
 at prepareVaultManagerKit (.../inter-protocol/src/vaultFactory/vaultManager.js:184)
 at prepareVaultDirector (.../inter-protocol/src/vaultFactory/vaultDirector.js:235)
 at provideDirector (.../inter-protocol/src/vaultFactory/vaultDirector.js:533)
 at prepare (.../inter-protocol/src/vaultFactory/vaultFactory.js:131)
 at ()
 at ()

dckc · 2023-07-21T16:15:23Z

problem(s): timestamp / oracle price, invitation (undiagnosed)

full timestamp / oracle price stacktraces

2023-07-20T19:35:16.818Z SwingSet: vat: v48: Error: timestamp: (an undefined) - Must be a copyRecord to match a copyRecord pattern: (an object)                                                          
 at construct ()                                                                                    
 at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:56)                                                                                                                  
 at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:243)                        
 at throwLabeled (.../patterns/src/utils.js:132)                                                    
 at applyLabelingError (.../patterns/src/utils.js:154)                                              
 at mustMatch (.../patterns/src/patterns/patternMatchers.js:528)                                    
 at absValue (.../time/src/timeMath.js:110)                                                         
 at subtractAbsAbs (.../time/src/timeMath.js:164)                                                   
 at calculateReportingPeriod (.../inter-protocol/src/interest.js:98)                                
 at chargeInterest (.../inter-protocol/src/interest.js:181)                                         
 at chargeAllVaults (.../inter-protocol/src/vaultFactory/vaultManager.js:449)                       
 at chargeAllVaults (.../inter-protocol/src/vaultFactory/vaultManager.js:415)                       
 at apply ()                                      
 at apply ()                                      
 at In "chargeAllVaults" method of (VaultManagerKit helper) (/bundled-source/.../node_modules/@endo/exo/src/exo-tools.js:42)                                                                             
 at updateState (.../inter-protocol/src/vaultFactory/vaultManager.js:371)                           
 at (.../notifier/src/asyncIterableAdaptor.js:63)                                                   
 at ()

and

2023-07-20T19:35:17.227Z SwingSet: ls: v48: Error: lockOraclePrices called before a collateral quote was available                                                                                       
 at construct ()                                  
 at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:56)                                                                                                                  
 at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:243)                        
 at fail (/bundled-source/.../node_modules/ses/src/error/assert.js:357)                             
 at lockOraclePrices (.../inter-protocol/src/vaultFactory/vaultManager.js:1037)                     
 at apply ()                                      
 at apply ()                                      
 at In "lockOraclePrices" method of (VaultManagerKit self) (/bundled-source/.../node_modules/@endo/exo/src/exo-tools.js:42)                                                                              
 at allManagersDo (.../inter-protocol/src/vaultFactory/vaultDirector.js:257)                        
 at (.../inter-protocol/src/vaultFactory/vaultDirector.js:440)                                      
 at apply ()                                      
 at apply ()                                      
 at dispatchToHandler (/bundled-source/.../node_modules/@endo/eventual-send/src/handled-promise.js:163)                                                                                                  
 at win (/bundled-source/.../node_modules/@endo/eventual-send/src/handled-promise.js:511)                                                                                                                
 at ()

invitation stacktrace

2023-07-20T19:35:16.253Z SwingSet: ls: v9: Logging sent error stack (Error#6)                                                                                                                            
2023-07-20T19:35:16.253Z SwingSet: ls: v9: Error#6: A Zoe invitation is required, not Object [Alleged: Zoe Invitation payment] {}                                                                        
2023-07-20T19:35:16.253Z SwingSet: ls: v9: Error: A Zoe invitation is required, not (an object)                                                                                                          
 at construct ()                                                                                                                                                                                         
 at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:56)                                                                                                                  
 at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:243)                                                                                                                             
 at onRejected (.../zoe/src/zoeService/invitationQueries.js:11)                                                                                                                                          
 at ()

SHORTFALL_INVITATION_KEY is another suspect (in addition to electorate):

2023-07-20T21:55:58.022Z SwingSet: vat: v48: 🛠️ updateShortfallReporter failed during start(); repair by updating governance (RemoteError(error:liveSlots:v9#70017)#1)

this seems odd; it doesn't make a new invitation:

    const newInvitation = await directorParamManager.getInternalParamValue(
      SHORTFALL_INVITATION_KEY,
    );

Chris-Hibbert · 2024-04-29T19:03:09Z

#9283 demonstrates that vaults are upgradeable.

closes: #8049 closes: #8740 closes: #8868 closes: #8918 closes: #8981 closes: #8079 refs: #8400 closes: #8735 closes: #7873 closes: #8726 closes: #7954 closes: #8757 closes: #8728 closes: #8789 ## Description Upgrade **VaultFactory** in A3P, relying on the new PriceFeeds, and auctions. The actual upgrade waits for the priceFeeds to start supplying before doing the upgrade, so there won't be any gap in priceUpdates. When the upgrade is finished, we also update the auctioneerKit and Auction instance in the bootstrap environment. This PR demonstrates that VaultFactory can be upgraded even though governance is not persistent (#8123). ### Security Considerations N/A ### Scaling Considerations This is largely in service of #8400, which reports that priceFeed vats are accumulating garbage. This PR switches to new priceFeeds, which won't have that problem, though cleaning up the existing vats is a task for the future. ### Documentation Considerations No changes to user-visible behavior. ### Testing Considerations A3P tests that verify that vaultFactory has been upgraded, that a new Auctioneer is running and is receiving prices. Verify that when prices drop, assets are sold via the auction, the bidder gets the proceeds, and the vaults are liquidated or reconstituted appropriately. ### Upgrade Considerations Upgrade all the vats!

turadg added bug Something isn't working Vaults VaultFactor (née Treasury) labels Jul 21, 2023

turadg mentioned this issue Jul 21, 2023

make Vault Factory configurable with new Auctioneer contract #7954

Closed

turadg self-assigned this Jul 21, 2023

This was referenced Jul 21, 2023

feat(auction): auction publish all bids off-chain #7618

Closed

revert StabilityFee schema change #8084

Merged

dckc mentioned this issue Sep 7, 2023

auctioneer doesn't recover when collateral is added without a price #8296

Closed

Jorge-Lopes mentioned this issue Dec 28, 2023

Expose All Bids in Vstorage Jorge-Lopes/liquidation-visibility#1

Open

3 tasks

warner mentioned this issue Jan 9, 2024

vat-vaultFactory may believe governance is broken when vat-auctioneer is upgraded #8726

Closed

turadg mentioned this issue Jan 13, 2024

WIP: in VaultFactory, move auctioneer public facet from terms to private args #8049

Closed

Chris-Hibbert assigned Chris-Hibbert and unassigned turadg Jan 23, 2024

Chris-Hibbert mentioned this issue Feb 7, 2024

prepare updates for Upgrade 15 #8868

Closed

14 tasks

Chris-Hibbert mentioned this issue Mar 27, 2024

Upgrade VaultFactory in A3P, and use new auctions and price feeds #8757

Closed

Chris-Hibbert mentioned this issue Apr 24, 2024

8981 upgrade vaults #9283

Merged

Chris-Hibbert closed this as completed Apr 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vaultFactory contract not upgradable #8079

vaultFactory contract not upgradable #8079

turadg commented Jul 21, 2023 •

edited

Loading

dckc commented Jul 21, 2023 •

edited

Loading

dckc commented Jul 21, 2023 •

edited

Loading

Chris-Hibbert commented Apr 29, 2024

vaultFactory contract not upgradable #8079

vaultFactory contract not upgradable #8079

Comments

turadg commented Jul 21, 2023 • edited Loading

Describe the bug

To Reproduce

Expected behavior

Platform Environment

Additional context

Screenshots

dckc commented Jul 21, 2023 • edited Loading

problem: stabilityFeeSnapshot vs interestSnapshot (fix available)

dckc commented Jul 21, 2023 • edited Loading

problem(s): timestamp / oracle price, invitation (undiagnosed)

Chris-Hibbert commented Apr 29, 2024

turadg commented Jul 21, 2023 •

edited

Loading

dckc commented Jul 21, 2023 •

edited

Loading

problem: `stabilityFeeSnapshot` vs `interestSnapshot` (fix available)

dckc commented Jul 21, 2023 •

edited

Loading