Could not get valid signature because Phantom sorts accountKeys before signing

[juchiast]

Summary

Our dApp need to request a signature from users and then submit it on the back-end. However, we are unable to get a correct signature because Phantom changes the transaction before it signs.

We have already found and resolved https://github.com/orgs/phantom/discussions/118 . However, we encounter more problems.

Example transaction message (base64):

AwEIEscL8A+E+o5v47gTGeVxaFuRx3Dh/g077V9U72FZUV3MCKbIaYjWft+Ul1aWoh2MnbdNskOO9QiCwnnDHmA/iNDOAq0G+/qWg2KKmXfDQF1HIuw9Op9LiSZK5iA7jcVQ9yBkxX67mIDHgN4N890Nk4NXxhDkyS10RWiGc3UwWt5wTrIt/idKg9sbYgCGDql8kWFMcnyEGmQ0oMzoYvKh3gdZACCi7YyQSduSWnvYjt13te0xWp3204LRG4CgCeZ1lWIC8z9aWv3VdIz2Fo2bsOxdKtUoM26+giSmAOexxGKFjG7osEWb4I6/+rLRH/snrb1oaiPFyg1/impvKgRQsXyN/rdGGtaFZVTTZSnifmwtp5dCB09f27r2Mem7qA9tFZJ2G/H45I1aQHlNCyDsRQY0g3srfB25bih2cUNMA1y7AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADBkZv5SEXMv/srbpyw5vnvIzlu8X3EmssQ5s6QAAAAAan1RcYe9FmNdrUBFX9wsDBJMaPIVZ1pdu6y18IAAAABt324ddloZPZy+FGzut5rBy0he1fWzeROoz1hX7/AKkLcGWx49F8RTidUn9rBMPNWLhscxqg/bVJttG8A/gpRjh6LiG/oY3jo3fV7G39jzFkGGYhu7if5Sm0GzmRbjx5jJclj04kifG7PRApFI4NgwtaE5na/xCEBI572Nvp+FnGm3o3UVUXfNjXWv3Q14ogyaX4ywQ01ZUBxwTnfaFPJ3ZVvfTUw3OgeEwm3zAFSnGCAFPJrMwpRSpIOwTEkdnRBwsACQMAAAAAAAAAAA4JBAgBAgACCgwN2gEqABgAAABTcGFjZSBPcGVyYXRvciBDaGFtZWxlb24EAAAAU1BPQ1MAAABodHRwczovL2Fzc2V0cy5zcGFjZW9wZXJhdG9yLmNvbS9tZXRhZGF0YS8yOTNlMDRiNi00MzNlLTRhY2QtYmRhMS04YWMzM2E1ZDllYzAuanNvbvoAAQEAAADOAq0G+/qWg2KKmXfDQF1HIuw9Op9LiSZK5iA7jcVQ9wBkAAEEAQA4ei4hv6GN46N31ext/Y8xZBhmIbu4n+UptBs5kW48eQAAAAEAAQFkAAAAAAAAAA4IAg4EDwkRCgwCNAEOCAIOBA4ODgoMAjQADg4HBQQIBgEDAgAKDA0ODgMsAwAODwMABAgGAQIOAAoMDRAODgsrAAEAAAAAAAAAAAoCAAUMAgAAAAC77qAAAAAA

accountKeys before signing:

[
  "EPzgzyt7Nwwrd3edXmdLPsx6KKYAH5QzVLCaL9uYoR6j",
  "ammP6c6U13tyPcwpkDDYM1RhcDi8VCNVBQjze7h2yyR",
  "EsBMSASEDM8n72k4N1QBHhWkW5Zu9cfAr1BkUnYqkiLE",
  "3BTAovuVzn1YbwN8Gx7z5Au5MrB82CeSyMQe3BJTfgnK",
  "6JCPEpiUYj3sHECRQqHD3DamMf4a9GDvBAXUKitKGMxi",
  "6zRQ5NKwbWerK92qrvDC2kr2KmsifaMUCjoJxPGAMNMA",
  "7bbZgkLBYpFLg5tTshSgQmvZqkgpZPUdVCUVMHWc7riQ",
  "AriyrSUt8Jne1HU2tXB19UJkf7vDJuTt4qJaUemZES58",
  "ATCAdp3euatJ9iK3rjWcQ7tYu3h6cXsTw8tcdUrwaXFq",
  "AZHkvzaQnWy4L6y1sCdBaPsSmpDssYMfQZaUGKnjc6rG",
  "11111111111111111111111111111111",
  "4oTtVn8TfdjMu4hsTBGL7xvuFN3YbUBbPn7xqatxBxjS",
  "ATokenGPvbdGVxr1b2hvZbsiqW5xWH25efTNsLJA8knL",
  "ComputeBudget111111111111111111111111111111",
  "ENHEHuWMr96W3ANfTyL9B95dqwCSDPu99mji8ZKyfo7c",
  "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s",
  "Sysvar1nstructions1111111111111111111111111",
  "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"
]

accountKeys after signing:

[
  "EPzgzyt7Nwwrd3edXmdLPsx6KKYAH5QzVLCaL9uYoR6j",
  "ammP6c6U13tyPcwpkDDYM1RhcDi8VCNVBQjze7h2yyR",
  "EsBMSASEDM8n72k4N1QBHhWkW5Zu9cfAr1BkUnYqkiLE",
  "3BTAovuVzn1YbwN8Gx7z5Au5MrB82CeSyMQe3BJTfgnK",
  "6JCPEpiUYj3sHECRQqHD3DamMf4a9GDvBAXUKitKGMxi",
  "6zRQ5NKwbWerK92qrvDC2kr2KmsifaMUCjoJxPGAMNMA",
  "7bbZgkLBYpFLg5tTshSgQmvZqkgpZPUdVCUVMHWc7riQ",
  "ATCAdp3euatJ9iK3rjWcQ7tYu3h6cXsTw8tcdUrwaXFq",
  "AZHkvzaQnWy4L6y1sCdBaPsSmpDssYMfQZaUGKnjc6rG",
  "AriyrSUt8Jne1HU2tXB19UJkf7vDJuTt4qJaUemZES58",
  "11111111111111111111111111111111",
  "ComputeBudget111111111111111111111111111111",
  "Sysvar1nstructions1111111111111111111111111",
  "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",
  "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s",
  "4oTtVn8TfdjMu4hsTBGL7xvuFN3YbUBbPn7xqatxBxjS",
  "ATokenGPvbdGVxr1b2hvZbsiqW5xWH25efTNsLJA8knL",
  "ENHEHuWMr96W3ANfTyL9B95dqwCSDPu99mji8ZKyfo7c"
]

Why does this happen and how can we resolve this issue?
Can we tell Phantom to just sign the message, don't do anything extra?

Example

No response

Steps to Reproduce

Example code:

const Sign: FC = () => {
  const { publicKey, signTransaction } = useWallet();
  const [transaction, setTransaction] = useState(
    "AwEIEscL8A+E+o5v47gTGeVxaFuRx3Dh/g077V9U72FZUV3MCKbIaYjWft+Ul1aWoh2MnbdNskOO9QiCwnnDHmA/iNDOAq0G+/qWg2KKmXfDQF1HIuw9Op9LiSZK5iA7jcVQ9yBkxX67mIDHgN4N890Nk4NXxhDkyS10RWiGc3UwWt5wTrIt/idKg9sbYgCGDql8kWFMcnyEGmQ0oMzoYvKh3gdZACCi7YyQSduSWnvYjt13te0xWp3204LRG4CgCeZ1lWIC8z9aWv3VdIz2Fo2bsOxdKtUoM26+giSmAOexxGKFjG7osEWb4I6/+rLRH/snrb1oaiPFyg1/impvKgRQsXyN/rdGGtaFZVTTZSnifmwtp5dCB09f27r2Mem7qA9tFZJ2G/H45I1aQHlNCyDsRQY0g3srfB25bih2cUNMA1y7AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADBkZv5SEXMv/srbpyw5vnvIzlu8X3EmssQ5s6QAAAAAan1RcYe9FmNdrUBFX9wsDBJMaPIVZ1pdu6y18IAAAABt324ddloZPZy+FGzut5rBy0he1fWzeROoz1hX7/AKkLcGWx49F8RTidUn9rBMPNWLhscxqg/bVJttG8A/gpRjh6LiG/oY3jo3fV7G39jzFkGGYhu7if5Sm0GzmRbjx5jJclj04kifG7PRApFI4NgwtaE5na/xCEBI572Nvp+FnGm3o3UVUXfNjXWv3Q14ogyaX4ywQ01ZUBxwTnfaFPJ3ZVvfTUw3OgeEwm3zAFSnGCAFPJrMwpRSpIOwTEkdnRBwsACQMAAAAAAAAAAA4JBAgBAgACCgwN2gEqABgAAABTcGFjZSBPcGVyYXRvciBDaGFtZWxlb24EAAAAU1BPQ1MAAABodHRwczovL2Fzc2V0cy5zcGFjZW9wZXJhdG9yLmNvbS9tZXRhZGF0YS8yOTNlMDRiNi00MzNlLTRhY2QtYmRhMS04YWMzM2E1ZDllYzAuanNvbvoAAQEAAADOAq0G+/qWg2KKmXfDQF1HIuw9Op9LiSZK5iA7jcVQ9wBkAAEEAQA4ei4hv6GN46N31ext/Y8xZBhmIbu4n+UptBs5kW48eQAAAAEAAQFkAAAAAAAAAA4IAg4EDwkRCgwCNAEOCAIOBA4ODgoMAjQADg4HBQQIBgEDAgAKDA0ODgMsAwAODwMABAgGAQIOAAoMDRAODgsrAAEAAAAAAAAAAAoCAAUMAgAAAAC77qAAAAAA"
  );
  const [before, setBefore] = useState("");
  const [after, setAfter] = useState("");
  const sign = useCallback(async () => {
    const buffer = Buffer.from(transaction, "base64");
    const solMsg = Message.from(buffer);
    const tx = Transaction.populate(solMsg);
    const signedTx = await signTransaction(tx);
    const before = tx.serializeMessage();
    const after = signedTx.serializeMessage();
    if (!before.equals(after)) {
      console.log(tx, signedTx);
      setBefore(JSON.stringify(tx.compileMessage().accountKeys, null, 2));
      setAfter(JSON.stringify(signedTx.compileMessage().accountKeys, null, 2));
      alert("tx changed");
      return;
    }
    const signature = signedTx.signatures.find((ele) =>
      ele.publicKey.equals(publicKey)
    ).signature;
    console.log(signature);
  }, [publicKey, signTransaction, setBefore, setAfter]);
  return (
    <>
      <textarea
        cols={30}
        rows={20}
        value={transaction}
        onChange={(e) => setTransaction(e.target.value)}
      ></textarea>
      <button onClick={sign}>Sign</button>
      <textarea cols={50} rows={20} value={before}></textarea>
      <textarea cols={50} rows={20} value={after}></textarea>
    </>
  );
};

Phantom Version

No response

Is there an existing discussion for this?

• I have searched the existing discussions

[juchiast]

Also, are there other changes made to the transaction before signing?

[juchiast]

could be related?

solana-labs/solana#21722

[juchiast]

anza-xyz/wallet-adapter#806

[adamdelphantom]

@juchiast has resolved this issue on his own. They will share their findings when possible.