Add or-tools #2377

SHaaD94 · 2020-02-26T17:08:01Z

This change is

https://github.com/LBNL-UCB-STI/or-tools-wrapper

REASY

Good stuff!

REASY · 2020-02-27T15:18:09Z

I can see some failed tests, https://beam-ci.tk/job/beam-4ci/7541/consoleFull:

java.io.IOError: java.io.IOException: Wrong index checksum, store was not closed properly and could be corrupted.
00:14:18   at org.mapdb.StoreDirect.checkHeaders(StoreDirect.java:269)
00:14:18   at org.mapdb.StoreDirect.<init>(StoreDirect.java:207)
00:14:18   at org.mapdb.DBMaker.extendStoreDirect(DBMaker.java:971)
00:14:18   at org.mapdb.DBMaker.makeEngine(DBMaker.java:758)
00:14:18   at org.mapdb.DBMaker.make(DBMaker.java:701)
00:14:18   at com.conveyal.osmlib.OSM.<init>(OSM.java:106)
00:14:18   at com.conveyal.r5.transit.TransportNetwork.fromFiles(TransportNetwork.java:198)
00:14:18   at com.conveyal.r5.transit.TransportNetwork.fromFiles(TransportNetwork.java:178)
00:14:18   at com.conveyal.r5.transit.TransportNetwork.fromDirectory(TransportNetwork.java:306)
00:14:18   at beam.router.r5.NetworkCoordinator.loadNetwork(NetworkCoordinator.scala:77)
00:14:18   ...
00:14:18   Cause: java.io.IOException: Wrong index checksum, store was not closed properly and could be corrupted.
00:14:18   at org.mapdb.StoreDirect.checkHeaders(StoreDirect.java:269)
00:14:18   at org.mapdb.StoreDirect.<init>(StoreDirect.java:207)
00:14:18   at org.mapdb.DBMaker.extendStoreDirect(DBMaker.java:971)
00:14:18   at org.mapdb.DBMaker.makeEngine(DBMaker.java:758)
00:14:18   at org.mapdb.DBMaker.make(DBMaker.java:701)
00:14:18   at com.conveyal.osmlib.OSM.<init>(OSM.java:106)
00:14:18   at com.conveyal.r5.transit.TransportNetwork.fromFiles(TransportNetwork.java:198)
00:14:18   at com.conveyal.r5.transit.TransportNetwork.fromFiles(TransportNetwork.java:178)
00:14:18   at com.conveyal.r5.transit.TransportNetwork.fromDirectory(TransportNetwork.java:306)
00:14:18   at beam.router.r5.NetworkCoordinator.loadNetwork(NetworkCoordinator.scala:77)

REASY · 2020-02-28T14:13:07Z

test!

SHaaD94 · 2020-02-28T14:47:09Z

test!

SHaaD94 · 2020-02-28T16:01:25Z

test!

haitamlaarabi · 2020-02-29T02:13:14Z

On Mac, i'm getting:
Exception in thread "main" java.lang.VerifyError: class org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader overrides final method getParserForType.()Lcom/google/protobuf/Parser; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.openstreetmap.osmosis.osmbinary.file.FileBlockHead.readHead(FileBlockHead.java:56) at org.openstreetmap.osmosis.osmbinary.file.FileBlock.process(FileBlock.java:130) at org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:34) at com.conveyal.osmlib.PBFInput.copyTo(PBFInput.java:237) at com.conveyal.osmlib.OSM.readFromFile(OSM.java:202) at com.conveyal.r5.transit.TransportNetwork.fromFiles(TransportNetwork.java:200) at com.conveyal.r5.transit.TransportNetwork.fromFiles(TransportNetwork.java:178) at com.conveyal.r5.transit.TransportNetwork.fromDirectory(TransportNetwork.java:306) at beam.router.r5.NetworkCoordinator.loadNetwork(NetworkCoordinator.scala:77) at beam.router.r5.NetworkCoordinator.loadNetwork$(NetworkCoordinator.scala:56)

this is probably due to protobuf-java, coming with or-tools. What about adding a gradle exception that excludes that jar from loading!?

JustinPihony

Is this PR ready for review? I only see it loading and not doing anything with it right now. Also, is there a reason for using com.github? It's not a huge jump to figure out at this early stage that it is something in our control, but later it could pose more of a challenge for newcomers to understand that this is an lbnl tool - not GitHub.

SHaaD94 · 2020-03-01T05:30:04Z

Is this PR ready for review? I only see it loading and not doing anything with it right now. Also, is there a reason for using com.github? It's not a huge jump to figure out at this early stage that it is something in our control, but later it could pose more of a challenge for newcomers to understand that this is an lbnl tool - not GitHub.

This PR is for replace existing 3rd party library for OR-Tools, it is being used in AlonsoMora pooling algorithm.
About com.github - we can rename it, just tell me preferred library package name.

JustinPihony · 2020-03-01T17:54:33Z

@wrashid @colinsheppard Any preference? Since it is for beam I'd say we stick with the beam namespace: beam.utilities.ortoolwrapper? And then it begs the question if this needs to be its own dependency or if it is small enough to be part of utilities itself?

SHaaD94 · 2020-03-02T14:48:06Z

test!

SHaaD94 · 2020-03-02T15:07:45Z

test!

REASY · 2020-03-02T18:31:01Z

test!

REASY · 2020-03-02T20:31:34Z

test!

REASY · 2020-03-02T20:44:05Z

Actual exception somehow is not visible. Here is what I see when I run ./gradlew specificTest -Psuite="beam.router.TollRoutingSpec" (make sure you removed files test\input\beamville\r5\osm.mapdb* and test/input/beamville/r5/network.dat)

*** RUN ABORTED ***
  java.lang.VerifyError: class org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader overrides final method getParserForType.()Lcom/google/protobuf/Parser;
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
  ...
:specificTest FAILED

FAILURE: Build failed with an exception.

haitamlaarabi · 2020-03-02T23:42:50Z

It is still failing. the exclude doesn't work as protobuf is encapsulated inside the wrapper jar.

REASY · 2020-03-04T18:22:53Z

@JustinPihony I think it fine that we leave the name as is. For example, beam-utilities has the following naming:

group: 'com.github.LBNL-UCB-STI', name: 'beam-utilities', version: 'v0.2.4'

SHaaD94 requested review from REASY and wrashid February 26, 2020 17:08

REASY requested changes Feb 27, 2020

View reviewed changes

REASY approved these changes Feb 27, 2020

View reviewed changes

SHaaD94 force-pushed the ez/or-tools-4ci branch from 016082c to 93ae585 Compare February 28, 2020 14:45

SHaaD94 mentioned this pull request Feb 28, 2020

Replace io.github.oliviercailloux:google-or-tools by our wrapper #2379

Closed

Add or-tools

4ebeb16

SHaaD94 force-pushed the ez/or-tools-4ci branch from 93ae585 to 4ebeb16 Compare February 28, 2020 14:55

1

5de9800

1

b9fb80a

SHaaD94 added 2 commits February 29, 2020 20:07

1

32d0b5c

return default pooling algo

f06f7c4

JustinPihony reviewed Feb 29, 2020

View reviewed changes

SHaaD94 added 2 commits March 4, 2020 21:22

bump version

6d551cc

1

ba4668c

REASY merged commit a4fb0b1 into develop Mar 4, 2020

REASY deleted the ez/or-tools-4ci branch March 4, 2020 18:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add or-tools #2377

Add or-tools #2377

SHaaD94 commented Feb 26, 2020 •

edited

Loading

REASY left a comment

REASY commented Feb 27, 2020

REASY commented Feb 28, 2020

SHaaD94 commented Feb 28, 2020

SHaaD94 commented Feb 28, 2020

haitamlaarabi commented Feb 29, 2020

JustinPihony left a comment

SHaaD94 commented Mar 1, 2020

JustinPihony commented Mar 1, 2020

SHaaD94 commented Mar 2, 2020

SHaaD94 commented Mar 2, 2020

REASY commented Mar 2, 2020

REASY commented Mar 2, 2020

REASY commented Mar 2, 2020 •

edited

Loading

haitamlaarabi commented Mar 2, 2020

REASY commented Mar 4, 2020

Add or-tools #2377

Add or-tools #2377

Conversation

SHaaD94 commented Feb 26, 2020 • edited Loading

REASY left a comment

Choose a reason for hiding this comment

REASY commented Feb 27, 2020

REASY commented Feb 28, 2020

SHaaD94 commented Feb 28, 2020

SHaaD94 commented Feb 28, 2020

haitamlaarabi commented Feb 29, 2020

JustinPihony left a comment

Choose a reason for hiding this comment

SHaaD94 commented Mar 1, 2020

JustinPihony commented Mar 1, 2020

SHaaD94 commented Mar 2, 2020

SHaaD94 commented Mar 2, 2020

REASY commented Mar 2, 2020

REASY commented Mar 2, 2020

REASY commented Mar 2, 2020 • edited Loading

haitamlaarabi commented Mar 2, 2020

REASY commented Mar 4, 2020

SHaaD94 commented Feb 26, 2020 •

edited

Loading

REASY commented Mar 2, 2020 •

edited

Loading