Normalize salt, iv, uuid params of .toV3() before encrypting #95

michaelsbradleyjr · 2019-07-17T18:17:10Z

Previously, if salt, iv and/or uuid options were supplied as strings to .toV3() they would be passed to pbkdf2Sync/scrypt as strings. That could result in errors during encryption. Also, during decryption these options were always converted to Buffer instances such that supplying strings during encryption could result in output that could not be decrypted.

This commit fixes the inconsistencies, guards against bad inputs, and also makes encrypted output match up with the output of other wallet libraries, e.g. ethers, whenever the equivalent encryption options are used consistently across libraries.

The problem was discovered when working on #2938 of web3. The original idea was to remove the scrypt package as a strict dependency: for Node 12 compatibility (owing to ABI changes) and as part of an effort to remove modules that require compilation, i.e. require dev tools to be installed. That involved swapping in ethers for ethereumjs-wallet as a dev dependency. ethers is a distinct implementation when it comes to the to/from v3 code, unlike web3 and ethereumjs-wallet which share much of the same logic. I couldn't get the json outputs to match unless no custom scrypt options were provided when encrypting on the web3 side, and finally I realized what was going wrong with the options when they're strings.

coveralls · 2019-07-17T18:21:18Z

Coverage remained the same at 89.199% when pulling 09786cc on michaelsbradleyjr:fix/tov3-salt-buffer into c04b80e on ethereumjs:master.

alcuadrado · 2019-07-18T17:48:09Z

@michaelsbradleyjr thanks for submitting this PR! It looks good.

Just a few questions:

Can you point me to the wallet v3 spec/definition?
How are salts encoded? Hexa strings with or without the 0x prefix? Is this consistent across different implementations?

Thanks!

michaelsbradleyjr · 2019-07-21T20:31:20Z

@alcuadrado sorry, I didn't see your reply earlier.

The closest thing to a spec I could find is this: https://github.com/ethereum/wiki/wiki/Web3-Secret-Storage-Definition.

salt strings are not given with a 0x prefix in that document, and from what I've seen in sample/test data for geth and parity that's consistently the case.

See:

https://github.com/ethereum/go-ethereum/blob/master/accounts/keystore/testdata/v3_test_vector.json

https://github.com/paritytech/parity-ethereum/blob/7c335e87642b848f61ab3ee7bbff5964b53d6f11/accounts/ethstore/src/json/key_file.rs#L201-L208

Now, the ethers library requires users optionally providing their own salt when encryptying a wallet to prefix the string with 0x. But I think that's confusing since the JSON output doesn't have 0x prefixed strings — I feel like string inputs should match up with string outputs.

While I don't think it should hold up this PR, I do think it would be a good idea to do a bit more options checking for uuid, iv, and salt options, i.e. to make sure that they're the expected length and valid hex (per the discussion in Discord last Thursday). Can we introduce that improvement in another PR?

alcuadrado · 2019-07-31T20:29:25Z

I've reviewed this PR and think it's correct. We should implement these changes after #93.

Thanks @michaelsbradleyjr for taking the time to investigate and fixing this!

holgerd77 · 2019-08-01T11:05:26Z

@michaelsbradleyjr this fall into this heavy typescript transition work, sorry for the inconveniences, can you update the PR here on top?

michaelsbradleyjr · 2019-08-05T16:11:47Z

@holgerd77 I'll give it a shot. TS isn't my expertise, but I'll see what I can do.

holgerd77 · 2019-08-07T10:38:01Z

@michaelsbradleyjr Thanks, should be relatively straight-forward and nothing really deeply TS-specific.

holgerd77 · 2019-08-07T14:08:52Z

@michaelsbradleyjr We would like to do a follow-up release after this has been merged, please let us know if you won't make it to update here in the next 3-5 days, then we might want to find another solution. Thanks!

holgerd77 · 2019-08-13T11:18:14Z

@michaelsbradleyjr @alcuadrado Just had a longer look at this since I was planning to manually do the changes within a new PR and then ask for permission from @michaelsbradleyjr to merge.

TypeScript is actually now enforcing this to be a buffer (see here), so this should be implicitly fixed. Generally this is a good thing and I think this library will really hugely profit from the TypeScript transition.

On the same time this makes me feel uncomfortable since this is some kind of implicit API change (strings now being disallowed), and I wonder how many other such implicit changes/corrections are in with the TypeScript merge, eventually @the-jackalope can also give some additional comment here.

When I looked at the docs in README I realized that this library is very seriously under-documented, you can't even explicitly find out about the types accepted for parameters without looking at the code.

I think we shouldn't do this release without also having the docs automated. This is generally super important, that people can easily look up the current API and accepted types and especially have some guidance for people doing this specific version update to the TypeScript release. @alcuadrado do you eventually have some time/capacity to do the TSDoc integration here?

Cheers
Holger

alcuadrado · 2019-08-13T13:46:21Z

@michaelsbradleyjr @alcuadrado Just had a longer look at this since I was planning to manually do the changes within a new PR and then ask for permission from @michaelsbradleyjr to merge.

TypeScript is actually now enforcing this to be a buffer (see here), so this should be implicitly fixed. Generally this is a good thing and I think this library will really hugely profit from the TypeScript transition.

I think you are right. @michaelsbradleyjr can you confirm this? I'm not that familiar with this lib.

I think we shouldn't do this release without also having the docs automated. This is generally super important, that people can easily look up the current API and accepted types and especially have some guidance for people doing this specific version update to the TypeScript release. @alcuadrado do you eventually have some time/capacity to do the TSDoc integration here?

Sure, I'll do it later this week.

holgerd77 · 2019-08-13T14:11:55Z

@alcuadrado Great, thanks! 😄

michaelsbradleyjr · 2019-08-13T15:12:48Z

I'll get it done today, the bug is still there but it's subtle. It's still possible for folks using JS instead of TS to pass a string for the salt, iv, etc. The inputs need to be normalized into buffers before being passed to scrypt. I started work on the changes to the sources and the tests last week but didn't have time to finish.

michaelsbradleyjr · 2019-08-13T23:54:48Z

Progress is here: master...michaelsbradleyjr:fix/TS-REWRITE-tov3-salt-buffer

See L239 in test/index.ts. Tomorrow I'll make some small adjustments to src/index.ts and it should be good to go.

Bonus round: implement guards for salt, iv, uuid strings/buffers to check whether they're valid hex (in the case of strings) and the correct length (strings and buffers).

The reason for pulling in the ethers pkg as a devDep is to test in concert with a wholly independent implementation.

Some of these improvements need to be adapted into PRs for the web3 package. When I was working several weeks ago on the test suites for web3 (1.x and 2.x branches) I convinced myself only the salt argument needed to be normalized; but that's incorrect. Normalization needs to be done consistently for the iv and uuid arguments in addition to salt (cc: @nivida).

holgerd77 · 2019-08-14T09:22:34Z

👍

michaelsbradleyjr · 2019-08-14T23:14:12Z

@alcuadrado @holgerd77 my refactor is finished. As I mentioned previously, TypeScript isn't one of my strengths so I'm very open to suggestions re: improving what I changed and authored.

Note: the tests take longer to run now because they're doing more encryption and decryption work.

michaelsbradleyjr · 2019-08-14T23:19:21Z

Looks like Node v6.x would need a polyfill for padStart, not sure how you'd want that to be handled... though maybe it's time to drop support for Node v6.x? Node v8.x will reach EOL on 1 Jan 2020 and v6.x reached EOL in May/Jun 2019, so v6.x is definitely starting to be "old and creaky".

holgerd77 · 2019-08-15T09:42:56Z

@michaelsbradleyjr that's great!

Yes, we've dropped Node 6 support some time ago, you can just remove it from Travis.

holgerd77 · 2019-08-15T10:36:27Z

@s1na since this is falling very much into this area of type conversion, hex checking and the like you've been working on along the util library updates, can you do a review here?

michaelsbradleyjr · 2019-08-15T16:56:25Z

@holgerd77 okay, I removed Node v6.x from Travis — all checks now passing. I also made a few small tweaks I thought of later last night.

s1na

Thanks for the PR and the extensive tests. Made some minor comments here and there. I didn't go through the tests in detail.

As I've mentioned in other issues/PRs, I think using strings to represent binary data has its challenges and I'd rather we restrict the API to Buffer. So what I'd to would be to throw if e.g. salt is not a buffer. But I understand that's not very user-friendly...So if everyone else is aboard with these changes, I'm okay too.

I also noticed you're testing this library's outputs against ethers outputs. While I have lots of respect for ethers, it might hide a bug in both implementations.

Some note for future: The types have gotten quite complex and there are a lot of duplication. If something changes in one type, it's easy to forget modifying another one... I think it's in need of some refactoring/simplification.

s1na · 2019-08-20T09:16:09Z

src/index.ts

  return {
-    cipher: params.cipher || 'aes-128-ctr',


This can be simplified as: return { ...params, ...v3Defaults }

Done, but it needed to be: return { ...v3Defaults, ...(params as V3ParamsStrict) } in order for caller supplied params to replace the defaults.

s1na · 2019-08-20T09:19:55Z

src/index.ts

@@ -361,6 +443,7 @@ export default class Wallet {

  // public instance methods

+  // tslint:disable-next-line


Why is it necessary to ignore tslint errors here and in getPublicKey()?

Without disabling those lines for tslint I get the following when doing npm run tslint:

ERROR: /Users/michael/repos/ethereumjs-wallet/src/index.ts:439:10 - Refactor this getter so that it actually refers to the property 'privateKey' ERROR: /Users/michael/repos/ethereumjs-wallet/src/index.ts:448:10 - Refactor this getter so that it actually refers to the property 'publicKey'

I felt that refactoring that code from the TS rewrite was beyond the scope of this PR, especially since the README documents those get... methods as part of the API. So I chose to stop the linter from failing on those lines.

s1na · 2019-08-20T09:29:59Z

src/index.ts


-    let kdfParams: PBKDFParams | ScryptKDFParams
+    let kdfParams: KDFParams
    let derivedKey: Buffer
    switch (v3Params.kdf) {
      case KDFFunctions.PBKDF:


I couldn't comment on the code below, but now that KDFParams.salt is a buffer and not converted to string, kdfparams.salt could be passed to pbkdf2Sync and scryptsy instead of v3Params.

Nice catch, I've made the change.

s1na · 2019-08-20T09:42:54Z

src/index.ts

@@ -38,17 +83,38 @@ function mergeToV3ParamsWithDefaults(params?: Partial<V3Params>): V3Params {
  if (!params) {
    return v3Defaults
  }
+
+  if (typeof params.salt === 'string') {
+    params.salt = Buffer.from(validateHexString('salt', params.salt), 'hex')


I just noticed that in ethereumjs-util we pad the string to even length, so that if implicitly e.g. a 0x01 byte was converted to 0x1, it would be covered:

https://github.com/ethereumjs/ethereumjs-util/blob/3b1085059194b02354177d334f89cd82a5187883/src/bytes.ts#L76

Not sure whether that applies here.

It's a possibility, for sure, though I wonder if it would end up masking legit user errors more than providing shorthand/convenience.

s1na · 2019-08-20T09:45:42Z

src/index.ts

+  if (str.toLowerCase().startsWith('0x')) {
+    str = str.slice(2)
+  }
+  if (!/^[0-9a-f]{2}([0-9a-f]{2})*$/i.test(str)) {


This is implicitly testing for length >= 2.

Right: ff or 0xff would work but not f or 0xf. But I think I'm missing the point you have in mind, can you explain further?

Just that if validateHexString is called without a length argument (e.g. for salt), inputs with length < 2 will still be rejected. I don't expect anyone to provide a salt with length < 2, so it should be fine.

s1na · 2019-08-20T09:51:01Z

test/index.ts

+    uuid: Buffer.from(uuid, 'hex'),
+  }
+
+  const makePermutations = (...objs: Array<object>): Array<object> => {


Can you add some comments to clarify this part?

Comments added, let me know if additional clarity is needed.

michaelsbradleyjr · 2019-08-20T23:06:07Z

@s1na I've added a commit that can be squashed prior to merge, so it's easier to see the changes made.

One additional change I made is slicing away leading 0x as the first step in validating the hex strings because that's the same thing ethers does and for users the identical behavior may make it easier when going from one to other.

Regarding use of ethers. You're right, it's not a bullet-proof solution; but combined with testing random wallets in addition to fixtures it offers some reasonable assurance that encrypted wallets are portable across different implementations. It's already borne some fruit: ethers-io/ethers.js#582 (comment). Also, making use of ethers in the test suite of web3 was what originally tipped me off to the problem with the string/buffer parameters and I realized ethereumjs-wallet had the same problem.

s1na

Looks good

Previously, if `salt`, `iv` and/or `uuid` options were supplied as strings to `.toV3()` they would be passed to `pbkdf2Sync`/`scrypt` as strings. That could result in errors during encryption. Also, during decryption these options were always converted to Buffer instances such that supplying strings during encryption could result in output that could not be decrypted. This commit fixes the inconsistencies, guards against bad inputs, and also makes encrypted output match up with the output of other wallet libraries, e.g. `ethers`, whenever the equivalent encryption options are used consistently across libraries.

michaelsbradleyjr · 2019-08-26T19:15:22Z

The PR review commit has been squashed, so I think this one's ready to go. @holgerd77 @alcuadrado

holgerd77 · 2019-08-28T18:19:13Z

Build was failing here, have restarted to see if it was something temporary.

holgerd77

Tests are passing now, reapprove here after the commit squashing.

holgerd77 · 2019-08-28T18:40:26Z

Ok, would follow up with a v1.0.0 release here now (respectively in the coming days). Let me know if you have any comments.

holgerd77 · 2019-08-29T19:36:49Z

For the record: forgot that we wanted to do the docs automation (see thread above), will postpone the release for a few days until we are through with that.

This was referenced Jul 17, 2019

convert salt to Buffer instance before encrypting web3/web3.js#2950

Merged

Node >= v8.0.0 support for web3 1.0 web3/web3.js#2938

Merged

alcuadrado mentioned this pull request Jul 31, 2019

Validate fromV3 params #96

Open

alcuadrado mentioned this pull request Aug 7, 2019

Typescript rewrite #93

Merged

michaelsbradleyjr force-pushed the fix/tov3-salt-buffer branch from 09786cc to fb7bde9 Compare August 14, 2019 23:05

michaelsbradleyjr changed the title ~~Convert salt to Buffer instance before encrypting~~ Normalize salt, iv, uuid params of .toV3() before encrypting Aug 14, 2019

michaelsbradleyjr force-pushed the fix/tov3-salt-buffer branch from fb7bde9 to ea99042 Compare August 14, 2019 23:11

michaelsbradleyjr force-pushed the fix/tov3-salt-buffer branch from ea99042 to 7904ac8 Compare August 15, 2019 16:49

s1na reviewed Aug 20, 2019

View reviewed changes

s1na approved these changes Aug 23, 2019

View reviewed changes

michaelsbradleyjr force-pushed the fix/tov3-salt-buffer branch from b575950 to 693ee8b Compare August 26, 2019 19:08

michaelsbradleyjr force-pushed the fix/tov3-salt-buffer branch from 693ee8b to debbd4f Compare August 26, 2019 19:14

holgerd77 approved these changes Aug 28, 2019

View reviewed changes

holgerd77 merged commit ac90675 into ethereumjs:master Aug 28, 2019

		@@ -361,6 +443,7 @@ export default class Wallet {

		// public instance methods

		// tslint:disable-next-line

Normalize salt, iv, uuid params of .toV3() before encrypting #95

Normalize salt, iv, uuid params of .toV3() before encrypting #95

Conversation

michaelsbradleyjr commented Jul 17, 2019 • edited Loading

coveralls commented Jul 17, 2019

alcuadrado commented Jul 18, 2019

michaelsbradleyjr commented Jul 21, 2019 • edited Loading

alcuadrado commented Jul 31, 2019

holgerd77 commented Aug 1, 2019

michaelsbradleyjr commented Aug 5, 2019

holgerd77 commented Aug 7, 2019

holgerd77 commented Aug 7, 2019

holgerd77 commented Aug 13, 2019

alcuadrado commented Aug 13, 2019

holgerd77 commented Aug 13, 2019

michaelsbradleyjr commented Aug 13, 2019 • edited Loading

michaelsbradleyjr commented Aug 13, 2019 • edited Loading

holgerd77 commented Aug 14, 2019

michaelsbradleyjr commented Aug 14, 2019 • edited Loading

michaelsbradleyjr commented Aug 14, 2019 • edited Loading

holgerd77 commented Aug 15, 2019

holgerd77 commented Aug 15, 2019

michaelsbradleyjr commented Aug 15, 2019

s1na left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelsbradleyjr Aug 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelsbradleyjr commented Aug 20, 2019 • edited Loading

s1na left a comment

Choose a reason for hiding this comment

michaelsbradleyjr commented Aug 26, 2019

holgerd77 commented Aug 28, 2019

holgerd77 left a comment

Choose a reason for hiding this comment

holgerd77 commented Aug 28, 2019

holgerd77 commented Aug 29, 2019

michaelsbradleyjr commented Jul 17, 2019 •

edited

Loading

michaelsbradleyjr commented Jul 21, 2019 •

edited

Loading

michaelsbradleyjr commented Aug 13, 2019 •

edited

Loading

michaelsbradleyjr commented Aug 13, 2019 •

edited

Loading

michaelsbradleyjr commented Aug 14, 2019 •

edited

Loading

michaelsbradleyjr commented Aug 14, 2019 •

edited

Loading

michaelsbradleyjr Aug 20, 2019 •

edited

Loading

michaelsbradleyjr commented Aug 20, 2019 •

edited

Loading