Lock simplecov between 0.10 and 0.13 version #181

bliof · 2017-03-18T16:17:15Z

Done because the 0.14.0 version introduces some changes that break the
CodeClimate::TestReporter::Formatter#merge_results
simplecov-ruby/simplecov@a7747c1

Check also:

SimpleCov::Result - https://github.com/colszowka/simplecov/blob/v0.14.0/lib/simplecov/result.rb
SimpleCov::RawCoverage - https://github.com/colszowka/simplecov/blob/v0.14.0/lib/simplecov/raw_coverage.rb

bliof · 2017-03-18T16:22:24Z

@gordondiggs follow-up from #180

Done because the 0.14.0 version introduces some changes that break the `CodeClimate::TestReporter::Formatter#merge_results` simplecov-ruby/simplecov@a7747c1 Check also: * SimpleCov::Result - https://github.com/colszowka/simplecov/blob/v0.14.0/lib/simplecov/result.rb * SimpleCov::RawCoverage - https://github.com/colszowka/simplecov/blob/v0.14.0/lib/simplecov/raw_coverage.rb

maxjacobson

Hi @bliof, thanks for this PR and very good catch.

maxjacobson · 2017-03-20T15:16:30Z

lib/code_climate/test_reporter/version.rb

@@ -1,5 +1,5 @@
 module CodeClimate
  module TestReporter
-    VERSION = "1.0.7".freeze
+    VERSION = "1.0.8".freeze


No need to update this file in this PR. Will you instead update the changelog?

Feel free to edit the PR as you wish. There is some date in the changelog that is most probably the one for when the version is released.

Lock simplecov to supported versions `>= 0.10` and `<= 0.13`

maxjacobson · 2017-03-20T15:16:58Z

codeclimate-test-reporter.gemspec

@@ -14,7 +14,7 @@ Gem::Specification.new do |spec|
  spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }.reject { |f| f == "ci" }

  spec.required_ruby_version = ">= 1.9"
-  spec.add_runtime_dependency "simplecov"
+  spec.add_runtime_dependency "simplecov", ">= 0.10", "<= 0.13"


This looks good. I just want to confirm: is the minimum version number also necessary because that's when this method was introduced?

No idea. I just tried - commit with locked version - with 0.10, 0.11, 0.12, and 0.13.

OK - I'm going to update to remove the minimum check, but keep the maximum check. I think that's the smallest change to address the current issue.

Our process is to make changes like this while preparing a new release.

FranklinYu · 2017-03-27T23:42:25Z

Will v1.1.x branch in the future support SimpleCov v0.14.x?

colszowka · 2017-04-06T09:00:16Z

Hey there, It'd be very nice if this bug could be actually fixed instead of pinning to an outdated version of SimpleCov ❤️

The underlying change in SimpleCov is the replacement of Hash and Array core extensions with utility methods that do this, see the related PR that introduced the change It should be fairly easy to change this gem's code to use the new API, swapping the old call to original_result.merge_resultset (although I am not quite sure why this gem needs to deal with resultset merging in the first place, as this is a low-level functionality of simplecov and I'd expect code climate would only need to consume the final (merged) result)

As a side note, this issue only arises when running multiple test suites (i.e. rspec and cucumber). On an app I am currently working on, we had no problems running codeclimate with SimpleCov 0.14.

ale7714 · 2017-04-10T16:50:55Z

Hi @colszowka, thank you for your feedback! Currently we're working on a new unified test reporter that will support multiple coverage report formats and parallelized reporting. This new reporter don't rely on low-level functionality of SimpleCov and instead consumes the final result.

We're focusing most of our development efforts on this new tool. It's currently on alpha and we're currently using it with the last version of SimpleCov successfully. I recommend users of SimpleCov v0.14.x to review and try our new test reporter.

FranklinYu · 2017-04-10T19:23:05Z

@ale7714 The manual page indicates that the unified reporter gets the data from ./coverage/.resultset.json in Ruby projects. Would that JSON format change over time? Should we pin the SimpleCov version? We don't need to worry about that if we're using a gem.

ale7714 · 2017-04-10T20:09:21Z

@FranklinYu that's correct. We're parsing .resulset.json from SimpleCov to a Code Climate payload. For now, it's not needed to pin SimpleCov to a particular version. If we find some issue it require to use a particular version, we will update our docs.
Let me know if you have any other question, also feel free to open issues on the test reporter repo if you find any because we're still on an alpha stage.

colszowka · 2017-04-11T07:37:10Z

Thanks for the pointer @ale7714, I will try out the new reporter.

I would not advise towards using the resultset.json file since it's an internal cache only meant for simplecov's merging mechanism and you're therefore betting yourselves on an internal data structure. I also would really not like pinning to a particular version of simplecov again later down the road when the approach of using an internal cache file turns out to fail.

I would strongly recommend that you build your own simplecov formatter (it's easy). I don't know if it's fully functional, but https://github.com/vicentllongo/simplecov-json might be a good starting point. This way you can vendorize your dependency on json output from simplecov and guard your customers against pinning potentially buggy and outdated versions.

Disclaimer: I'm the author of simplecov.

FranklinYu · 2017-04-11T08:02:38Z

I agree with @colszowka. Depending on private interface is always fragile, and I don't think customers would be happy about that. There may be a time gap between SimpleCov release a new version and your team update the documentation; if a customer updates his SimpleCov during this gap, he/she is under risk that he/she can't be aware of. Since the unified reporter is still in early stage (there's only a single alpha release), it's time to make the switch before it's too late.

ale7714 · 2017-04-11T12:29:34Z

@colszowka thank you for your input! I appreciate it. I will review more about implementing our own simplecov formatter. I will bring your recommendations to the team.

FranklinYu · 2017-04-12T17:56:27Z

The current (low-level) workflow of the unified reporter, for non-parallel build, is

./cc-test-reporter format-coverage
./cc-test-reporter upload-coverage

It seems easy to just replace the first step with a Rake task provided by cc-simplecov-json (the imaginary formatter name):

bundle exec rake cc:coverage:format
./cc-test-reporter upload-coverage

as per codeclimate#181 (comment), the issue only occurs when using multiple test suites

They depend on internal APIs, see: codeclimate/ruby-test-reporter#180 codeclimate/ruby-test-reporter#181 This is a second rewrite of their test reporter but they don't plan on fixing this bug because they are working on the third rewrite... I'm tired of having to deal with CodeClimate breakages on all my projects so I'll be moving away from it. If you know a good alternative, **well-written**, with support for aggregating multiple test suites, please comment on this commit message.

codeclimate-hermes assigned maxjacobson Mar 18, 2017

maxjacobson suggested changes Mar 20, 2017

View reviewed changes

maxjacobson added 3 commits March 20, 2017 16:18

Undo version change

34bc2db

Our process is to make changes like this while preparing a new release.

Update codeclimate-test-reporter.gemspec

56159e5

Update CHANGELOG.md

5c1c9a2

maxjacobson approved these changes Mar 20, 2017

View reviewed changes

maxjacobson merged commit 9971c7e into codeclimate:master Mar 20, 2017

bliof deleted the lock-simplecov branch March 20, 2017 21:50

dblandin mentioned this pull request Mar 21, 2017

v1.0.7 incompatible with simplecov 0.14.0 #180

Closed

jgonera mentioned this pull request Apr 17, 2017

Allow Overall Statistics Calculation without loading the files that are covered simplecov-ruby/simplecov#573

Open

timbru31 added a commit to timbru31/ruby-test-reporter that referenced this pull request Apr 18, 2017

chore: remove simplecov version pinning

1b3b6c1

as per codeclimate#181 (comment), the issue only occurs when using multiple test suites

FranklinYu mentioned this pull request Jun 30, 2017

Include executable in release codeclimate/test-reporter#98

Closed

dependabot-preview bot mentioned this pull request Mar 14, 2018

Bump codeclimate-test-reporter from 1.0.3 to 1.0.8 fsek/web#737

Merged

werebus mentioned this pull request Sep 16, 2020

Update ruby to 2.6.6 umts/st-pax#240

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lock simplecov between 0.10 and 0.13 version #181

Lock simplecov between 0.10 and 0.13 version #181

bliof commented Mar 18, 2017

bliof commented Mar 18, 2017

maxjacobson left a comment

maxjacobson Mar 20, 2017

bliof Mar 20, 2017 •

edited

Loading

maxjacobson Mar 20, 2017

bliof Mar 20, 2017

maxjacobson Mar 20, 2017

FranklinYu commented Mar 27, 2017

colszowka commented Apr 6, 2017

ale7714 commented Apr 10, 2017

FranklinYu commented Apr 10, 2017

ale7714 commented Apr 10, 2017

colszowka commented Apr 11, 2017 •

edited

Loading

FranklinYu commented Apr 11, 2017

ale7714 commented Apr 11, 2017 •

edited

Loading

FranklinYu commented Apr 12, 2017

Lock simplecov between 0.10 and 0.13 version #181

Lock simplecov between 0.10 and 0.13 version #181

Conversation

bliof commented Mar 18, 2017

bliof commented Mar 18, 2017

maxjacobson left a comment

Choose a reason for hiding this comment

maxjacobson Mar 20, 2017

Choose a reason for hiding this comment

bliof Mar 20, 2017 • edited Loading

Choose a reason for hiding this comment

maxjacobson Mar 20, 2017

Choose a reason for hiding this comment

bliof Mar 20, 2017

Choose a reason for hiding this comment

maxjacobson Mar 20, 2017

Choose a reason for hiding this comment

FranklinYu commented Mar 27, 2017

colszowka commented Apr 6, 2017

ale7714 commented Apr 10, 2017

FranklinYu commented Apr 10, 2017

ale7714 commented Apr 10, 2017

colszowka commented Apr 11, 2017 • edited Loading

FranklinYu commented Apr 11, 2017

ale7714 commented Apr 11, 2017 • edited Loading

FranklinYu commented Apr 12, 2017

bliof Mar 20, 2017 •

edited

Loading

colszowka commented Apr 11, 2017 •

edited

Loading

ale7714 commented Apr 11, 2017 •

edited

Loading