Separate Keyword Arguments from Positional Arguments #2395

jeremyevans · 2019-08-20T19:35:23Z

This implements Ruby Feature 14183, see lengthy discussion in https://bugs.ruby-lang.org/issues/14183.

And, allow non-symbol keys as a keyword arugment

…words Treat the ** syntax as passing a copy of the hash as the last positional argument. If the hash being double splatted is empty, do not add a positional argument. Remove rb_no_keyword_hash, no longer needed.

Previously, it only applied if the call had more positional arguments than the method it was calling accepted.

Mostly requires adding ** in either calls or method definitions.

If all keys are not symbols, then the non-symbol keys would not be treated as keywords in previous versions. It doesn't make sense to treat these hashes as keywords to break compatibility and warn about behavior changes in Ruby 2.7 when the Ruby 3.0 behavior will be the same as the Ruby 2.6 for these hashes.

Now that keyword splats accept non-Symbols, the inspect value of the keyword is used instead of the string value.

Now that keyword splats accept non-Symbols, the type of exception changes. Previously, a TypeError (hash key "k1" is not a Symbol) was raised for this test, but now an ArgumentError (unknown keyword: "k1") is raised.

…d warning This restores compatibility with Ruby 2.6, splitting the last positional hash into positional and keyword arguments if it contains both symbol and non-symbol keys. However, in this case it will warn, as the behavior in Ruby 3 will be to not split the hash and keep it as a positional argument. This does not affect the handling of mixed symbol and non-symbol keys in bare keywords. Those are still treated as keywords now, as they were before this patch. This results in different behavior than Ruby 2.6, which would split the bare keywords and use the non-Symbol keys as a positional arguments.

This fixes MJIT after rb_hash_stlike_foreach used vm_args.c.

…arguments This syntax means the method should be treated as a method that uses keyword arguments, but no specific keyword arguments are supported, and therefore calling the method with keyword arguments will raise an ArgumentError. It is still allowed to double splat an empty hash when calling the method, as that does not pass any keyword arguments.

The on_params hook will use :nil as the keyword rest argument. There is a new on_nokw_param hook as well. This fixes a type issue in the previous code, where an ID was passed where a VALUE was the declared type. The symbol :nil is passed instead of the id.

Use false instead of nil for the keyword and keyword rest values in that case.

Use a [:nokey] entry in this case.

For methods that accept keyword arguments but do not accept a keyword splat, if a keyword splat is passed, or keywords are used with a non-symbol key, check the hash. If the hash contains all symbols, keep the same behavior as before. If the hash contains all non-symbols, move the hash to the last positional hash and warn. If the hash contains symbols and non-Symbols, split the hash and use the symbol keys for the keyword hash and non-symbol keys for the positional hash and warn.

Also handle some warnings for behavior that will change in 3.0.

This reverts the changes to parse.y in a5b3726 as they are not actually needed and cause the warning for duplicate hash keys to not be emitted.

…aration

jeremyevans · 2019-08-30T08:22:48Z

This was accepted at the developer meeting yesterday. I'm going to push a rebased version that fixes the conflict to get a final CI test before committing.

jeremyevans · 2019-08-30T15:53:04Z

Appveyvor CI failure appears unrelated: C:/projects/build/.ext/x64-mingw32/zlib.so: [BUG] Segmentation fault.

The test-bundler failures are actually caused by this, because bundler or bundler's tests have not been modified to support keyword argument separation, and the warnings generated are breaking tests checking for no warnings. I'll see if I can fix that.

There are more issues than this, but hopefully this is enough to get make test-bundler passing in CI.

jeremyevans · 2019-08-30T19:39:10Z

AppVeyor is still failing, but is has been continually failing since b3c8745. Everything else looks good, I'm going to merge this.

amatsuda · 2019-08-30T22:00:38Z

🎉

ioquatix · 2019-08-30T23:05:06Z

In the release window before 2.7 can we provide real world feedback?

mame · 2019-08-31T00:10:18Z

🎉 Thank you for your hard work, Jeremy!

I think 2.7-preview2 will be released before the final release.

jeremyevans · 2019-08-31T00:18:03Z

Sure, you can provide real world feedback. As much as possible, other than the change for keyword splats to accept non-Symbol keys, we want the 2.7 behavior to be the same as the 2.6 behavior, except for warnings. In Ruby 3, you will have to pass keyword parameters either literally or via the double splat, and Ruby will not automatically convert keyword parameters to positional hashes or vice versa if the method accepts keyword parameters. If the method does not accept keyword parameters, any keyword parameters will be treated as a positional hash argument.

ioquatix · 2019-08-31T00:19:14Z

Do you think we will also consider some kind of generic forwarding syntax, e.g.

def foo(...)
    bar(...)

... means same as *args, **opts, &block but without implicit copy/dup.

jeremyevans · 2019-08-31T00:22:32Z

Unrelated to this feature. I think it has been suggested before, but I can't find the feature request right now.

ioquatix · 2019-08-31T00:24:37Z

It's not completely unrelated, because this feature changes the requirements in order to do perfect forwarding. No longer *args, &block is sufficient. But it's used in many places. It became slightly more complicated. So, I wonder if we can simplify it again.

def foo(**)
    bar(**)

Could be another approach to reuse existing token.

jeremyevans · 2019-08-31T00:25:59Z

I believe def foo(**) end that already means accept no positional arguments, but any keyword arguments.

ruby/ruby#2395

Non-kwargs parameters should be to be braced for ruby/ruby#2395. See https://bugs.ruby-lang.org/issues/14183 for details. `Style/BracesAroundHashParameters` cop conflicts with that. This removes `Style/BracesAroundHashParameters` cop and auto-correct to following changes. rails/rails@d94263f...5665fb5

Follow ruby/ruby#2395. RuboCop's CI of Ruby 2.7 matrix is failing. This is due to a deprecation warning in Ruby 2.7. ```console 1) RuboCop::CLI when BlockDelimiters has braces_for_chaining style corrects SpaceBeforeBlockBraces, SpaceInsideBlockBraces offenses Failure/Error: expect($stderr.string).to eq('') expected: "" got: "/usr/local/bundle/gems/parser-2.6.4.0/lib/parser/source/tree_rewriter.rb:284: warning: The last argu...parser-2.6.4.0/lib/parser/source/tree_rewriter.rb:288: warning: for `trigger_policy' defined here\n" ``` https://circleci.com/gh/rubocop-hq/rubocop/67195 This PR suppress the above Ruby 2.7's warning.

Follow ruby/ruby#2395. RuboCop's CI of Ruby 2.7 matrix is failing. This is due to a deprecation warning in Ruby 2.7. ```console 1) RuboCop::CLI when BlockDelimiters has braces_for_chaining style corrects SpaceBeforeBlockBraces, SpaceInsideBlockBraces offenses Failure/Error: expect($stderr.string).to eq('') expected: "" got: "/usr/local/bundle/gems/parser-2.6.4.0/lib/parser/source/tree_rewriter.rb:284: warning: The last argu...parser-2.6.4.0/lib/parser/source/tree_rewriter.rb:288: warning: for `trigger_policy' defined here\n" ``` https://circleci.com/gh/rubocop-hq/rubocop/67195 This PR suppress the following Ruby 2.7's warning. ```console % cd path/to/parser % bundle exec rake (snip) /Users/koic/src/github.com/whitequark/parser/lib/parser/source/tree_rewriter.rb:284: warning: The last argument for `trigger_policy' (defined at /Users/koic/src/github.com/whitequark/parser/lib/parser/source/tree_rewriter.rb:288) is used as the keyword parameter ```

# Description: Follow ruby/ruby#2395. This PR suppress the following Ruby 2.7's real kwargs warning. ```console% ruby -v % ruby -v ruby 2.7.0dev (2019-09-11T21:58:51Z master 515b198) [x86_64-darwin17] % gem i bundler (snip) /Users/koic/.rbenv/versions/2.7.0-dev/lib/ruby/site_ruby/2.7.0/rubygems/package.rb:489: warning: The last argument is used as the keyword parameter /Users/koic/.rbenv/versions/2.7.0-dev/lib/ruby/2.7.0/fileutils.rb:171: warning: for `mkdir' defined here % cd path/to/rubygems % rake test (snip) ./Users/koic/src/github.com/rubygems/rubygems/lib/rubygems.rb:472: warning: The last argument is used as the keyword parameter /Users/koic/.rbenv/versions/2.7.0-dev/lib/ruby/2.7.0/fileutils.rb:197: warning: for `mkdir_p' defined here ```

2912: Suppress Ruby 2.7's real kwargs warning r=hsbt a=koic # Description: Follow ruby/ruby#2395. This PR suppress the following Ruby 2.7's real kwargs warning. ```console% ruby -v % ruby -v ruby 2.7.0dev (2019-09-11T21:58:51Z master 515b198) [x86_64-darwin17] % gem i bundler (snip) /Users/koic/.rbenv/versions/2.7.0-dev/lib/ruby/site_ruby/2.7.0/rubygems/package.rb:489: warning: The last argument is used as the keyword parameter /Users/koic/.rbenv/versions/2.7.0-dev/lib/ruby/2.7.0/fileutils.rb:171: warning: for `mkdir' defined here % cd path/to/rubygems % rake test (snip) ./Users/koic/src/github.com/rubygems/rubygems/lib/rubygems.rb:472: warning: The last argument is used as the keyword parameter /Users/koic/.rbenv/versions/2.7.0-dev/lib/ruby/2.7.0/fileutils.rb:197: warning: for `mkdir_p' defined here ``` ______________ # Tasks: - [x] Describe the problem / feature - [ ] Write tests - [x] Write code to solve the problem - [ ] Get code review from coworkers / friends I will abide by the [code of conduct](https://github.com/rubygems/rubygems/blob/master/CODE_OF_CONDUCT.md). Co-authored-by: Koichi ITO <[email protected]>

The Ruby core team decided to introduce a slight incompatibility to keyword arguments from Ruby 3.0, i.e. complete separation between keyword arguments literal and Hash literal. https://bugs.ruby-lang.org/issues/14183 ruby/ruby#2395 With that, current Ruby master warns when a Hash object was passed in as keyword arguments: ``` $ ruby -ve 'def f(x: nil) p x; end; hash = {x: 1}; f(hash)' ruby 2.7.0dev (2019-09-02T05:20:05Z master 83498854eb) [x86_64-darwin18] warning: The last argument is used as the keyword parameter warning: for `f' defined here 1 ``` To eliminate this warning, we need to prefix a "double splat" (**) to avoid ambiguity: ``` $ ruby -ve 'def f(x: nil) p x;end; hash = {x: 1}; f(**hash)' ruby 2.7.0dev (2019-09-02T05:20:05Z master 83498854eb) [x86_64-darwin18] 1 ``` See also: * ruby-i18n/i18n#486 * https://buildkite.com/rails/rails/builds/63974#7fb9ad05-a745-4022-b634-aa3eb9042b11/6-1865 ``` /usr/local/lib/ruby/gems/2.7.0/gems/mysql2-0.5.2/lib/mysql2/error.rb:55: warning: The last argument is used as the keyword parameter /usr/local/lib/ruby/gems/2.7.0/gems/mysql2-0.5.2/lib/mysql2/error.rb:94: warning: The last argument is used as the keyword parameter ```

mame and others added 20 commits August 30, 2019 01:16

Separate keyword arguments from positional arguments

9736e49

And, allow non-symbol keys as a keyword arugment

Fix hash to keyword warning to apply in all cases

bb16d33

Previously, it only applied if the call had more positional arguments than the method it was calling accepted.

Fix keyword argument separation issues in lib

de7753e

Mostly requires adding ** in either calls or method definitions.

Fix keyword argument separation warnings in test

bab4e8d

Update tests to fix warning message changes

eb2abdd

Now that keyword splats accept non-Symbols, the inspect value of the keyword is used instead of the string value.

Fix test after keyword argument separation

eda9533

Now that keyword splats accept non-Symbols, the type of exception changes. Previously, a TypeError (hash key "k1" is not a Symbol) was raised for this test, but now an ArgumentError (unknown keyword: "k1") is raised.

Set symbol export for rb_hash_stlike_foreach

b0fab6e

This fixes MJIT after rb_hash_stlike_foreach used vm_args.c.

Make RubyVM::AbstractSyntaxTree handle **nil syntax

41947b3

Use false instead of nil for the keyword and keyword rest values in that case.

Make Method/Proc#parameters handle **nil syntax

7b1bcc1

Use a [:nokey] entry in this case.

Make keyword_hash_split static

ecb754a

When splitting a keyword hash, dup it first to not mutate it

4105a60

Update specs to handle non-Symbols for keyword splats in 2.7

ae44d97

Also handle some warnings for behavior that will change in 3.0.

Add back missing warning for duplicate keys in splatted hashes

de4d29d

This reverts the changes to parse.y in a5b3726 as they are not actually needed and cause the warning for duplicate hash keys to not be emitted.

Fix remaining warning issues in the tests due to keyword argument sep…

13b9c6d

…aration

jeremyevans force-pushed the keyword-argument-separation branch from f27bb8a to 13b9c6d Compare August 30, 2019 08:23

Fix a couple of bundler issues with keyword argument separation

fbc82e7

There are more issues than this, but hopefully this is enough to get make test-bundler passing in CI.

jeremyevans merged commit b5b3afa into ruby:master Aug 30, 2019

jeremyevans deleted the keyword-argument-separation branch August 30, 2019 19:39

k0kubun added a commit to benchmark-driver/benchmark-driver that referenced this pull request Sep 1, 2019

Suppress warnings for keyword arguments in Ruby 2.7

353efa7

ruby/ruby#2395

koic mentioned this pull request Sep 3, 2019

Fix new warnings from Ruby rubocop/rubocop#7329

Merged

kamipo mentioned this pull request Sep 4, 2019

Do not enforce no braces for hash argument rails/rails#37121

Merged

koic mentioned this pull request Sep 4, 2019

Follow "Separate Keyword Arguments from Positional Arguments" whitequark/parser#607

Merged

koic mentioned this pull request Sep 12, 2019

Suppress Ruby 2.7's real kwargs warning rubygems/rubygems#2912

Merged

4 tasks

hachi8833 mentioned this pull request Sep 18, 2019

Discussion: Goby's parameter syntax in method definitions goby-lang/goby#751

Open

kamipo mentioned this pull request Nov 21, 2019

Keyword arguments have to be explicitly double-splatted in Ruby 2.7+ brianmario/mysql2#1084

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Separate Keyword Arguments from Positional Arguments #2395

Separate Keyword Arguments from Positional Arguments #2395

jeremyevans commented Aug 20, 2019

jeremyevans commented Aug 30, 2019

jeremyevans commented Aug 30, 2019

jeremyevans commented Aug 30, 2019

amatsuda commented Aug 30, 2019

ioquatix commented Aug 30, 2019

mame commented Aug 31, 2019

jeremyevans commented Aug 31, 2019

ioquatix commented Aug 31, 2019

jeremyevans commented Aug 31, 2019

ioquatix commented Aug 31, 2019

jeremyevans commented Aug 31, 2019

Separate Keyword Arguments from Positional Arguments #2395

Separate Keyword Arguments from Positional Arguments #2395

Conversation

jeremyevans commented Aug 20, 2019

jeremyevans commented Aug 30, 2019

jeremyevans commented Aug 30, 2019

jeremyevans commented Aug 30, 2019

amatsuda commented Aug 30, 2019

ioquatix commented Aug 30, 2019

mame commented Aug 31, 2019

jeremyevans commented Aug 31, 2019

ioquatix commented Aug 31, 2019

jeremyevans commented Aug 31, 2019

ioquatix commented Aug 31, 2019

jeremyevans commented Aug 31, 2019