Replace Monoid constraint by CommutativeSemigroup in the reduce syntax #203

alonsodomin · 2017-11-01T13:10:09Z

After the release of Cats 1.0.0-RC1, CommutativeSemigroup instances for tuples can be derived providing there is a CommutativeSemigroup for the members of the tuples, which unblocks #117.

Now, by changing the restriction from Monoid to CommutativeSemigroup one of the tests is no longer correct, the one reducing an RDD[Map[Int, Int]] into a Map[Int, Int] because there is not a CommutativeSemigroup for Maps.

The test has been commented out and I'm submitting this for review and discussion.

Fixes #117

iravid · 2017-11-01T14:33:23Z

Hi @alonsodomin, could you bump cats-effect to 0.5 and cats-mtl to the next version (after typelevel/cats-mtl#15 is merged and released)?

iravid · 2017-11-01T14:57:52Z

cats/src/test/scala/frameless/cats/test.scala

-    rdd.csum shouldBe 1.to(20).zip(1.to(20)).toMap
-  }
+  // property("rdd Map[Int,Int] monoid example") {
+  //   import frameless.cats.implicits._


The Map instance was moved to alleycats in typelevel/cats@1f0cba0

Hi @alonsodomin, can we bring this back based on @iravid suggestion?

Hi @imarios, sorry for leavin this unattended. I just pushed an upgrade of the cats-mtl package.

There is still the problem with the Map test as there is no CommutativeSemigroup instance possible of an arbitrary Map, there could be one for SortedMap but cats(and alleycats) do not provide with one.

You can see @iravid comment below in which he acknowledges that he was thinking of the Traversable instance, not the CommutativeSemigroup one. So, in this case, if frameless is interested on enforcing correctness for those methods, probably it's better to drop the test for the Map... either that, or provide to alleycats a CommutativeSemigroup instance for SortedMap and wait until it's merged...

What if we replace Map with SortedMap? Will we be able to have this test back?

Hi @imarios, just made a PR to cats adding a CommutativeMonoid instance for SortedMap typelevel/cats#2047. Once it gets merged I can recover the commented out tests in here.

alonsodomin · 2017-11-02T09:51:59Z

Hi @iravid, I'll bump cats-mtl version once that PR gets merged (cats-effect is already 0.5).

In the other hand alleycats provides with an instance of a traversable functor for Map but not instances of CommutativeSemigroup or CommutativeMonoid. I in fact wonder whether it is possible to have a CommutativeSemigroup for a map ... wouldn't swapping the ordering of operands affect the order of the keys in the map? Because in such a case merging to arbitrary maps is defo not commutative. If that is the reason for not having a CommutativeSemigroup for a Map, then a SortedMap should be commutative providing there is a CommutativeSemigroup for the value type, right?

Besides all of this. The tests do not seem to exercise the csumByKey, cminByKey, cmaxByKey methods in the pairRddOps syntax enrichment class. I will add some additional ones to verify them.

codecov-io · 2017-11-02T10:00:21Z

Codecov Report

Merging #203 into master will increase coverage by 0.47%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #203      +/-   ##
==========================================
+ Coverage   95.71%   96.18%   +0.47%     
==========================================
  Files          39       52      +13     
  Lines         793      944     +151     
  Branches       11       18       +7     
==========================================
+ Hits          759      908     +149     
- Misses         34       36       +2

Impacted Files	Coverage Δ
cats/src/main/scala/frameless/cats/implicits.scala	`75% <100%> (+13.46%)`	⬆️
...ore/src/main/scala/frameless/CatalystOrdered.scala	`100% <0%> (ø)`	⬆️
...ataset/src/main/scala/frameless/TypedEncoder.scala	`100% <0%> (ø)`	⬆️
...scala/frameless/functions/AggregateFunctions.scala	`100% <0%> (ø)`	⬆️
dataset/src/main/scala/frameless/implicits.scala	`100% <0%> (ø)`	⬆️
dataset/src/main/scala/frameless/TypedColumn.scala	`100% <0%> (ø)`	⬆️
...aset/src/main/scala/frameless/ops/GroupByOps.scala	`98.14% <0%> (ø)`	⬆️
...t/src/main/scala/frameless/functions/package.scala	`100% <0%> (ø)`	⬆️
...et/src/main/scala/frameless/ops/SmartProject.scala	`100% <0%> (ø)`	⬆️
...l/classification/TypedRandomForestClassifier.scala	`100% <0%> (ø)`
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update daca214...067fbbd. Read the comment docs.

iravid · 2017-11-17T08:51:13Z

@alonsodomin - entirely right re CommutativeSemigroup for Map! What I linked to was irrelevant; I was indeed thinking of the Traverse instance.

imarios · 2017-11-27T01:06:17Z

@iravid anything else you want to add for this PR? If not I can merge. Thank you @alonsodomin

iravid · 2017-11-27T06:23:30Z

Looks great to me @imarios, thanks @alonsodomin!

OlivierBlanvillain · 2017-11-27T09:31:09Z

Something is failing in sbt tut

alonsodomin · 2017-11-27T11:37:07Z

@OlivierBlanvillain yeah, it's an example about reducing an RDD with a Map. Since the PR with the CommutativeMonoid for SortedMap has already been merged in cats, I guess that this will need to wait for a new release in their side and then I can update the tests and tut docs to use a SortedMap instead

iravid · 2017-11-27T11:38:31Z

Ah, cool, so this PR will bump us to cats 1.0.0 :-)

ceedubs · 2017-12-04T16:19:57Z

I definitely like adding a Commutative constraint.

However, moving from Monoid to Semigroup seems like a change in a questionable direction to me, since an RDD can potentially be empty. What would people think about having something like the following two methods?

def csum(implicit m: CommutativeMonoid[A]): A = lhs.fold(m.empty)(_ |+| _)

// we may get better performance out of using `reduce` and catching
// `UnsupportedOperation` exceptions for empty RDDs
def csumOption(implicit m: CommutativeSemigroup[A]): Option[A] =
  lhs.fold[Option[A]](None)((acc, a) => Some(acc.fold(a)(_ |+| a)))

alonsodomin · 2017-12-04T18:33:01Z

I do like the idea of avoiding exceptions, not sure about significantly performance improvements since most of the overhead is comming from the Spark runtime. But if the other members are onboard the change is quite straightforward.

Would we also be interested in avoiding exceptions in the other cmin, cmax methods and making them return an Option[A] too?

ceedubs · 2017-12-04T18:49:36Z

@alonsodomin I'd love to see those cmin and cmax changes too.

I should probably add the disclaimer that everything that I say should be taken with a grain of salt, because I currently use neither Spark nor Frameless, but I think that I'll be using both in the near future :)

ceedubs · 2017-12-20T03:53:12Z

@alonsodomin I believe that RC2 should be released now. Are you still available to give it a try?

alonsodomin · 2017-12-20T09:03:43Z

@ceedubs yeah, will give another try.

alonsodomin · 2017-12-20T10:04:13Z

The build finally passes now. This is still using the CommutativeSemigroup and RDD.reduce method as in the original PR.

I do like @ceedubs proposal of having csum/csumOption (and friends) using CommutativeMonoid and CommutativeSemigroup to avoid unexpected exceptions at runtime. Is there anyone else that would like give his/her opinion on this matter?

ceedubs · 2018-01-02T16:36:59Z

cats/src/main/scala/frameless/cats/implicits.scala

+    def csumOption(implicit m: CommutativeSemigroup[A]): Option[A] =
+      lhs.aggregate[Option[A]](None)(
+        (acc, a) => Some(acc.fold(a)(_ |+| a)),
+        (l, r) => l.fold(r)(x => r.map(_ |+| x) <+> Some(x))


I may be reading this wrong, but this seems incorrect to me. Isn't x getting added in twice in the case where both l and r are Some?

If so, it's concerning that unit tests didn't catch this. Maybe some places that are calling toRDD on a List should be calling parallelize?

I should probably replace the <+> operator by orElse since I'm basically relying on the fact that the implementation of <+> for Option is that one.

In essence, I just want to express the Alternative between the two Option (<|>) but in cats this is implemented based on MonoidK, and therefore <|> becomes <+>...

Ah right that makes sense. I think that I'm one of the people who advocated for doing this in Cats, so I probably should have realized what was going on here :P

It may be more straightforward to just use orElse, but if it works the right way then my main concern is gone. Thanks for the explanation!

ceedubs · 2018-01-02T18:25:57Z

I'm not a frameless maintainer, but for what it's worth, I've reviewed this and it looks like good stuff to me!

imarios · 2018-01-15T05:21:34Z

@alonsodomin LGTM! Sorry for this taking so long. Did you cover everything you wanted from your side? If not, then I can merge.

alonsodomin · 2018-01-15T08:20:18Z

@imarios yeah, this is complete as far as I'm concerned.

Antonio Alonso Dominguez added 2 commits November 1, 2017 12:50

Use a CommutativeMonoid in the reduce operations

8363460

Use a commutative semigroup when reducing

b4dae3d

iravid reviewed Nov 1, 2017

View reviewed changes

Add tests for RDD reductions by key

8756a59

Verify the contents regardless of the order

9b5c8dd

Upgrade Cats MTL

f76c246

Antonio Alonso Dominguez added 2 commits December 20, 2017 10:06

Update cats

08010dc

Use sorted maps as they are commutative

a136d69

Antonio Alonso Dominguez added 3 commits December 21, 2017 19:59

empty-safe RDD reductions

c92df2f

relax constraint on csumByKey

2e43003

Upgrade Cats and related deps

606282f

ceedubs reviewed Jan 2, 2018

View reviewed changes

Replace <+> by orElse

067fbbd

imarios merged commit b99be3f into typelevel:master Jan 16, 2018

alonsodomin deleted the commutative_monoid branch January 16, 2018 18:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace Monoid constraint by CommutativeSemigroup in the reduce syntax #203

Replace Monoid constraint by CommutativeSemigroup in the reduce syntax #203

alonsodomin commented Nov 1, 2017

iravid commented Nov 1, 2017

iravid Nov 1, 2017

imarios Nov 25, 2017

alonsodomin Nov 25, 2017

imarios Nov 25, 2017

alonsodomin Nov 26, 2017

alonsodomin commented Nov 2, 2017

codecov-io commented Nov 2, 2017 •

edited

Loading

iravid commented Nov 17, 2017

imarios commented Nov 27, 2017

iravid commented Nov 27, 2017

OlivierBlanvillain commented Nov 27, 2017

alonsodomin commented Nov 27, 2017

iravid commented Nov 27, 2017

ceedubs commented Dec 4, 2017

alonsodomin commented Dec 4, 2017

ceedubs commented Dec 4, 2017

ceedubs commented Dec 20, 2017

alonsodomin commented Dec 20, 2017

alonsodomin commented Dec 20, 2017

ceedubs Jan 2, 2018

alonsodomin Jan 2, 2018

ceedubs Jan 2, 2018

ceedubs commented Jan 2, 2018

imarios commented Jan 15, 2018

alonsodomin commented Jan 15, 2018

Replace Monoid constraint by CommutativeSemigroup in the reduce syntax #203

Replace Monoid constraint by CommutativeSemigroup in the reduce syntax #203

Conversation

alonsodomin commented Nov 1, 2017

iravid commented Nov 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alonsodomin commented Nov 2, 2017

codecov-io commented Nov 2, 2017 • edited Loading

Codecov Report

iravid commented Nov 17, 2017

imarios commented Nov 27, 2017

iravid commented Nov 27, 2017

OlivierBlanvillain commented Nov 27, 2017

alonsodomin commented Nov 27, 2017

iravid commented Nov 27, 2017

ceedubs commented Dec 4, 2017

alonsodomin commented Dec 4, 2017

ceedubs commented Dec 4, 2017

ceedubs commented Dec 20, 2017

alonsodomin commented Dec 20, 2017

alonsodomin commented Dec 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ceedubs commented Jan 2, 2018

imarios commented Jan 15, 2018

alonsodomin commented Jan 15, 2018

codecov-io commented Nov 2, 2017 •

edited

Loading