Add index related helpers to Traverse #1761

andyscott · 2017-07-07T07:12:34Z

This adds traverseWithIndex, mapWithIndex and zipWithIndex to Traverse.

traverseWithIndexM is akin to traverse, but includes the index of the value in the structure F. This requires the effect to be monadic.
mapWithIndex is akin to map, but includes the index of the value in the structure F.
zipWithIndex returns F, with values tupled with their indices, just like Scala collection's zipWithIndex.

Instances for List, Vector, and Stream have overridden mapWithIndex and zipWithIndex that delegate to related underlying collections methods. mapWithIndex on List and Vector uses .iterator to avoid building an intermediate collection when first zipping with indices.

peterneyens · 2017-07-07T08:04:10Z

core/src/main/scala/cats/Traverse.scala

+   */
+  def stateTraverse[S, A, B](fa: F[A])(f: A => State[S, B]): State[S, F[B]] =
+    State[S, F[B]](s =>
+      traverse[State[S, ?] , A, B](fa)(f).run(s).value)


We don't need the additional State wrapping of run(s).value, right?

This is equal to traverse[State[S, ?] , A, B](fa)(f) and with SI 2712 fixed, we don't even need to specify the types, so it is just traverse(fa)(f).

To follow up on @peterneyens comment, since this is just equivalent to traverse, is there any reason to have a separate stateTraverse?

yeah, I'm confused why this is needed.

It isn't!

PSA: Don't code while deliriously tired.

codecov-io · 2017-07-07T08:16:28Z

Codecov Report

Merging #1761 into master will increase coverage by 0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1761      +/-   ##
==========================================
+ Coverage   94.17%   94.21%   +0.03%     
==========================================
  Files         256      256              
  Lines        4208     4218      +10     
  Branches       93       97       +4     
==========================================
+ Hits         3963     3974      +11     
+ Misses        245      244       -1

Impacted Files	Coverage Δ
core/src/main/scala/cats/instances/list.scala	`100% <100%> (ø)`	⬆️
core/src/main/scala/cats/instances/stream.scala	`94.82% <100%> (+0.18%)`	⬆️
core/src/main/scala/cats/Traverse.scala	`100% <100%> (ø)`	⬆️
core/src/main/scala/cats/instances/vector.scala	`97.87% <100%> (+0.09%)`	⬆️
core/src/main/scala/cats/Foldable.scala	`100% <0%> (+1.29%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb84fd4...9789c2d. Read the comment docs.

johnynek · 2017-07-07T16:42:09Z

core/src/main/scala/cats/Traverse.scala

+   */
+  def stateTraverse[S, A, B](fa: F[A])(f: A => State[S, B]): State[S, F[B]] =
+    State[S, F[B]](s =>
+      traverse[State[S, ?] , A, B](fa)(f).run(s).value)


yeah, I'm confused why this is needed.

johnynek · 2017-07-07T16:44:05Z

tests/src/test/scala/cats/tests/ListTests.scala

+
+  test("indexed consistent with zipWithIndex") {
+    forAll { (fa: List[String]) =>
+      Traverse[List].indexed(fa) should === (fa.zipWithIndex)


can we try this for a very long list (say 500k elements)? I'm concerned State with traverse can fail for us. I think I have seen issues like that before. I don't want to leave a landmine for someone.

I think we may be able to work around by using tailRecM on State though, but let's see if we need to.

I'll add a test for this

Added. Went ahead and created a dedicated file for testing Traverse instead of piggybacking on the tests in ListTests.scala.

johnynek · 2017-07-07T17:36:26Z

👍

kailuowang · 2017-07-07T18:58:56Z

seems problematic on scalaJS

Security context: 0x1784390b4629
1: flatMap__F1__Lcats_Eval [/home/travis/build/typelevel/cats/tests/.js/target/scala-2.10/cats-tests-test-fastopt.js:~49091] [pc=0x323fb4f2641] (this=0x3b070c22ee61 <a $c_Lcats_Eval$$anon$5 with map 0x14937c2e6199>,f=0xe6fa70d8d09 <a $c_sjsr_AnonFunction1 with map 0x14937c2a97e1>)
2: map__F1__Lcats_Eval [/home/travis/build/typelevel/cats/tests/.js/target/scala-2.10/cats-tests-test-fasto...
[error] FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

andyscott · 2017-07-07T20:48:32Z

I did some quick testing. On the JVM, everything is fine. However, on JS the performance of .indexed is nonlinear. It's only apparent once the collections get really big.

For testing, I used collections from 50,000 to 500,000 entries by increments of 50,000. The tests run reasonably quick for < 100,000 items in the collection. After 300,000 items, the tests get extremely slow or fail.

Currently I haven't looked for an underlying problem. For the time being, I have adjusted the tests so that JS is uses a smaller collection. It passed locally-- hopefully it does the same on the build server.

kailuowang · 2017-07-07T22:04:59Z

Do we know that 100,000 isn't enough to test against stack safety for JVM? Now the 3 tests take more than 30 seconds, not sure if that's necessary

[info] - Traverse[List].indexed stack safety (12 seconds, 943 milliseconds)
[info] - Traverse[Stream].indexed stack safety (15 seconds, 11 milliseconds)
[info] - Traverse[Vector].indexed stack safety (12 seconds, 585 milliseconds)

edmundnoble · 2017-07-07T23:16:09Z

@andyscott @kailuowang Afaik you actually need at most 70000 (see my latest merged PR).

andyscott · 2017-07-07T23:25:50Z

@edmundnoble Aah! Right now it's been switched to 100000. I'll switch it to 70000 :).

ase enter the commit message for your changes. Lines starting

edmundnoble · 2017-07-07T23:34:05Z

Can you also give me something like indexed which takes a function (A, Int) => B and produces an F[B]?

andyscott · 2017-07-07T23:36:31Z

something like...

// untested
def mapWithIndex[A, B](fa: F[A])(f: (A, Int) => B): F[B] = 
  traverse(fa)(a => State((s: Int) => (s + 1, f(a, s)))).runA(0).value

edmundnoble · 2017-07-07T23:43:06Z

Yeah sure.

andyscott · 2017-07-07T23:46:33Z

Done. I also updated indexed to be implemented in terms of mapWithIndex.

andyscott · 2017-07-10T01:23:43Z

Any objections to also adding traverseWithIndex?

def traverseWithIndex[G[_]: Monad, A, B](fa: F[A])(f: (A, Int) => G[B]): G[F[B]] =
  traverse(fa)(a => StateT((s: Int) => f(a, s).map(b => (s + 1, b)))).runA(0)

edmundnoble · 2017-07-10T01:51:02Z

That's fine with me but I think in the long run it could be better, as soon as we have a Sum new type for that Monoid you can change the constraint to G[_]: Applicative and use WriterT.

johnynek · 2017-07-10T18:16:15Z

core/src/main/scala/cats/Traverse.scala

+   * The behavior is consistent with the Scala collection library's
+   * `zipWithIndex` for collections such as `List`.
+   */
+  def indexed[A](fa: F[A]): F[(A, Int)] =


any reason not to call this zipWithIndex?

can we override this for List Vector and Stream?

Yep, will do.

johnynek · 2017-07-10T18:19:14Z

tests/src/test/scala/cats/tests/TraverseTests.scala

+  }
+
+  test(s"Traverse[$name].mapWithIndex") {
+    forAll { (fa: F[Int]) =>


can we take ((Int, Int)) => Int as a function and do:

fa.mapWithIndex((a, i) => fn((a, i))).toList should === (fa.toList.zipWithIndex.map(fn))

johnynek · 2017-07-10T18:23:35Z

tests/src/test/scala/cats/tests/TraverseTests.scala

+
+  test(s"Traverse[$name].traverseWithIndex") {
+    forAll { (fa: F[Int]) =>
+      fa.traverseWithIndex((a, i) => Option((a, i))).map(_.toList) should === (Option(fa.toList.zipWithIndex))


could we have a stronger test, like take fn: ((Int, Int)) => Option[Int] and then

val left = fa.traverseWithIndex((a, i) => fn((a, i))).map(_.toList) val right = (fa.toList.zipWithIndex match { case Nil => Some(Nil) case h :: tail => tail.foldM(h :: Nil) { case (l, ai) => fn(ai).map(_ :: l) }.map(_.reverse) }) left should === right

Update incoming. I switched from Option to Tuple2 to make the stronger test more straightforward.

johnynek · 2017-07-10T18:24:21Z

core/src/main/scala/cats/Traverse.scala

+   * Akin to [[map]], but also provides the value's index in structure
+   * F when calling the function.
+   */
+  def mapWithIndex[A, B](fa: F[A])(f: (A, Int) => B): F[B] =


can we override this for List, Vector and Stream?

andyscott · 2017-07-11T04:37:55Z

@johnynek I think I've addressed everything.

Using views seemed to be the best way to override mapWithIndex, as to avoid creating an intermediate collection for List and Vector traverse instances. I can switch it to something else, if desired.

Also: I added an "underlying" proxy so I could get coverage of the default method implementations. This also has the very minor benefit of being a really bad benchmark for comparing performance of the default implementation vs the overridden one. I consistently get slightly faster results for the overridden methods:

[info] TraverseListCheck:
[info] - Traverse[List].zipWithIndex (93 milliseconds)
[info] - Traverse[List].mapWithIndex (25 milliseconds)
[info] - Traverse[List].traverseWithIndex (38 milliseconds)

[info] TraverseListCheckUnderlying:
[info] - Traverse[List (underlying)].zipWithIndex (129 milliseconds)
[info] - Traverse[List (underlying)].mapWithIndex (38 milliseconds)
[info] - Traverse[List (underlying)].traverseWithIndex (32 milliseconds)

My benchmarking warmup consists of repeatedly running the one test over and over until the values settle ¯\(ツ)/¯.

…nstances

johnynek

👍

Just one minor comment about naming, but I don't have a better suggestion at this moment.

johnynek · 2017-07-11T22:00:54Z

tests/src/test/scala/cats/tests/TraverseTests.scala

+
+  // proxies a traverse instance so we can test default implementations
+  // to achieve coverage using default datatype instances
+  private def proxyTraverse[F[_]: Traverse]: Traverse[F] = new Traverse[F] {


this is nice.

johnynek · 2017-07-11T22:03:30Z

core/src/main/scala/cats/Traverse.scala

+   * Akin to [[traverse]], but also provides the value's index in
+   * structure F when calling the function.
+   */
+  def traverseWithIndex[G[_], A, B](fa: F[A])(f: (A, Int) => G[B])(implicit G: Monad[G]): G[F[B]] =


my only slight reservation with the name here is that traverse takes an Applicative[G] and implies often some parallelism, where this is sequential and requires Monad[G] since the applicative for StateT still requires Monad[G] (I think).

I don't really like that I had to require Monad. I suppose this could also be implemented with just Applicative if it's done with two passes on the underlying structure-- traverse(zipWithIndex(f))(ai => fa(ai._1, ai._2)). But I also don't like having to make two passes.

Disregard my previous comment that you can't do this. You can use the Group[Int] and WriterT instead of State.

if we can relax to Applicative that would be ideal.

I can't sort out how to make it work with WriterT. What did you have in mind @edmundnoble?

My bad, I can't find a way to do this. I can calculate the length though :P.

johnynek · 2017-07-12T18:25:55Z

core/src/main/scala/cats/instances/vector.scala

@@ -82,6 +82,12 @@ trait VectorInstances extends cats.kernel.instances.VectorInstances {
          G.map2Eval(f(a), lgvb)(_ +: _)
        }.value

+      override def mapWithIndex[A, B](fa: Vector[A])(f: (A, Int) => B): Vector[B] =
+        fa.view.zipWithIndex.map(ai => f(ai._1, ai._2)).toVector


can we benchmark with fa.iterator.zipWithIndex.map(..., I think .iterator may be faster than .view

An official benchmark in the benchmark submodule? Or will a quick local benchmark work, updating the code according?

…ator for performance

andyscott · 2017-07-14T15:49:28Z

@johnynek I renamed traverseWithIndex to traverseWithIndexM and added a note about traversing with index using applicative. I also switched .view to .iterator after a bit of investigation and more trivial benchmarking locally.

kailuowang · 2017-07-19T13:45:09Z

merged with 3 sign-offs

peterneyens reviewed Jul 7, 2017

View reviewed changes

andyscott changed the title ~~Add stateTraverse and indexed to Traverse~~ Add indexed to Traverse Jul 7, 2017

andyscott force-pushed the feature/stateTraverse-and-indexed branch from e3aa1ff to 98b6808 Compare July 7, 2017 16:45

johnynek reviewed Jul 7, 2017

View reviewed changes

andyscott force-pushed the feature/stateTraverse-and-indexed branch 2 times, most recently from edaaf4b to eef6979 Compare July 7, 2017 17:31

Add indexed to Traverse

50ba29c

andyscott force-pushed the feature/stateTraverse-and-indexed branch from eef6979 to 50ba29c Compare July 7, 2017 17:34

andyscott force-pushed the feature/stateTraverse-and-indexed branch from b0122c9 to 0eee566 Compare July 7, 2017 23:06

Test .indexed with a smaller collection so builds aren't terribly slow

4413058

ase enter the commit message for your changes. Lines starting

andyscott force-pushed the feature/stateTraverse-and-indexed branch from 0eee566 to 4413058 Compare July 7, 2017 23:26

andyscott changed the title ~~Add indexed to Traverse~~ Add mapWithIndex and indexed to Traverse Jul 7, 2017

Add mapWithIndex to Traverse; implement indexed in terms of mapWithIndex

83a0883

andyscott force-pushed the feature/stateTraverse-and-indexed branch from 8c6e3e6 to 83a0883 Compare July 7, 2017 23:54

kailuowang approved these changes Jul 9, 2017

View reviewed changes

Add .traverseWithIndex to Traverse

df74985

andyscott changed the title ~~Add mapWithIndex and indexed to Traverse~~ Add index related helpers to Traverse Jul 10, 2017

kailuowang approved these changes Jul 10, 2017

View reviewed changes

johnynek reviewed Jul 10, 2017

View reviewed changes

Rename indexed to zipWithIndex; Improve tests; add faster overrides

f6d9ade

Force testing of overridden traverse methods for list/vector/stream i…

e0109a7

…nstances

johnynek reviewed Jul 11, 2017

View reviewed changes

johnynek reviewed Jul 12, 2017

View reviewed changes

Rename traverseWithIndex to traverseWithIndexM; Change .view to .iter…

9789c2d

…ator for performance

DavidGregory084 mentioned this pull request Jul 14, 2017

Add operations for validating Traversables using the indices of their elements DavidGregory084/robots#1

Closed

edmundnoble approved these changes Jul 19, 2017

View reviewed changes

kailuowang merged commit 27a7399 into typelevel:master Jul 19, 2017

kailuowang modified the milestone: 1.0.0-MF Jul 24, 2017

Add index related helpers to Traverse #1761

Add index related helpers to Traverse #1761

Conversation

andyscott commented Jul 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jul 7, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnynek commented Jul 7, 2017

kailuowang commented Jul 7, 2017

andyscott commented Jul 7, 2017 • edited Loading

kailuowang commented Jul 7, 2017 • edited Loading

edmundnoble commented Jul 7, 2017

andyscott commented Jul 7, 2017

edmundnoble commented Jul 7, 2017

andyscott commented Jul 7, 2017 • edited Loading

edmundnoble commented Jul 7, 2017

andyscott commented Jul 7, 2017 • edited Loading

andyscott commented Jul 10, 2017 • edited Loading

edmundnoble commented Jul 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyscott commented Jul 11, 2017 • edited Loading

johnynek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyscott Jul 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyscott Jul 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyscott commented Jul 14, 2017 • edited Loading

kailuowang commented Jul 19, 2017

andyscott commented Jul 7, 2017 •

edited

Loading

codecov-io commented Jul 7, 2017 •

edited

Loading

andyscott commented Jul 7, 2017 •

edited

Loading

kailuowang commented Jul 7, 2017 •

edited

Loading

andyscott commented Jul 7, 2017 •

edited

Loading

andyscott commented Jul 7, 2017 •

edited

Loading

andyscott commented Jul 10, 2017 •

edited

Loading

andyscott commented Jul 11, 2017 •

edited

Loading

andyscott Jul 11, 2017 •

edited

Loading

andyscott Jul 12, 2017 •

edited

Loading

andyscott commented Jul 14, 2017 •

edited

Loading