Update TinkerPop 3.6.1 [tp-tests] #3208

farodin91 · 2022-09-15T16:57:43Z

Fixes #3069

Thank you for contributing to JanusGraph!

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

Is there an issue associated with this PR? Is it referenced in the commit message?
Does your PR body contain #xyz where xyz is the issue number you are trying to resolve?
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?

For code changes:

Have you written and/or updated unit tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE.txt file, including the main LICENSE.txt file in the root of this repository?
If applicable, have you updated the NOTICE.txt file, including the main NOTICE.txt file found in the root of this repository?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered?

porunov · 2022-09-15T18:58:30Z

pom.xml

        <slf4j.version>1.7.36</slf4j.version>
        <logback.version>1.2.11</logback.version>
        <httpcomponents.httpclient.version>4.5.13</httpcomponents.httpclient.version>
        <httpcomponents.httpcore.version>4.4.15</httpcomponents.httpcore.version>
-        <hadoop2.version>2.8.5</hadoop2.version>
+        <hadoop2.version>3.3.4</hadoop2.version>


(nitpick) I think it makes sense to call this version hadoop3 now instead of hadoop2. I.e.:

<hadoop3.version>3.3.4</hadoop3.version>

That's a nitpick, so not necessary to resolve, we can refactor it afterwards

I will clean before marking the pr ready.

farodin91 · 2022-09-19T17:07:11Z

I will split this update into multiple PRs.

farodin91 · 2022-10-01T12:17:59Z

@porunov Hbase upgrade is a pain.

porunov · 2022-10-01T12:19:58Z

@porunov Hbase upgrade is a pain.

I imagine it is ...

farodin91 · 2022-10-03T07:54:44Z

@porunov @FlorianHockmann @li-boxuan @rngcntr Does any have a bit time to look into hbase hadoop tests?

farodin91 · 2022-10-09T14:18:08Z

It looks like this one of the bugs which stoping me to get this running fine. https://github.com/apache/hbase/pull/4819/files

Other tests in apache projects were deactivated in the combination to test hbase 2 with hadoop 3. https://github.com/apache/ranger

mad · 2022-10-12T13:37:37Z

@farodin91

TP tests are skipped

It would be nice to run it by adding [tp-tests] into commit message

farodin91 · 2022-10-12T17:37:19Z

@mad I would like to find a way to fix the hbase test before hand.

mad · 2022-10-13T10:56:49Z

@farodin91

Some info about hbase issue

Scan for janusgraph hbase return some data, but metadata say no data exists

Scan response

hbase:003:0> scan 'janusgraph', {LIMIT => 10}
ROW                                                 COLUMN+CELL                                                                                                                                            
 \x00\x00\x00\x00\x00\x00\x00\x03                   column=i:\xFF\xFF\xFF\xFF\xFF\xFE\xC7\x7F\x00\x00\x01\x83\xD0\xEB\xE0\x807f000101691244-bic-pc1, timestamp=2022-10-13T10:37:42.913, value=             
 \x00\x00\x00\x00\x00\x00\x00\x04                   column=i:\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x9B\x00\x00\x01\x83\xD0\xEB\xDF(7f000101691244-bic-pc1, timestamp=2022-10-13T10:37:42.569, value=                
 \x00\x00\x00\x00\x00\x00\x00\x04                   column=i:\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xCD\x00\x00\x01\x83\xD0\xEB\xDD\xCA7f000101691244-bic-pc1, timestamp=2022-10-13T10:37:42.219, value=             
 \x00\x00\x00\x00\x00\x00\x02\x0D                   column=e:\x02, timestamp=2022-10-13T10:37:44.127, value=\x00\x01\x08\x80                                                                               
 \x00\x00\x00\x00\x00\x00\x02\x0D                   column=e:\x10\xC0, timestamp=2022-10-13T10:37:44.127, value=\xA0vl\x1EvertexKe\xF9\x04\x80                                                             
 \x00\x00\x00\x00\x00\x00\x02\x0D                   column=e:\x10\xC2\x80\x14\x00, timestamp=2022-10-13T10:37:44.127, value=\x8F\x00\x01\x8E\x00\x8F\x80                                                   
 \x00\x00\x00\x00\x00\x00\x02\x0D                   column=e:\x10\xC2\x80\x18\x00, timestamp=2022-10-13T10:37:44.127, value=\x8F\x00\x01\x8E\x00\x90\x80                                                   
 \x00\x00\x00\x00\x00\x00\x02\x0D                   column=e:\x10\xC4, timestamp=2022-10-13T10:37:44.127, value=\x00\x82\x0C\x80                                                                           
 \x00\x00\x00\x00\x00\x00\x02\x0D                   column=e:\x10\xC8, timestamp=2022-10-13T10:37:44.127, value=\x00\x80\x00\x01\x83\xD0\xEB\xE1\xFF\x10\x80                                               
 \x18\xD4{\x96\x10\xA5\xA0vl\x1EvertexKe\xF9        column=g:\x00, timestamp=2022-10-13T10:37:44.127, value=\x04\x8D                                                                                       
 configuration                                      column=s:graph.janusgraph-version, timestamp=2022-10-13T10:37:42.043, value=\x92\xA01.0.0-SNAPSHO\xD4                                                  
 configuration                                      column=s:graph.storage-version, timestamp=2022-10-13T10:37:42.047, value=\x92\xA0\xB2                                                                  
 configuration                                      column=s:graph.timestamps, timestamp=2022-10-13T10:37:42.033, value=\xB6\x82                                                                           
 configuration                                      column=s:hidden.frozen, timestamp=2022-10-13T10:37:42.049, value=\x8F\x01                                                                              
 configuration                                      column=s:ids.num-partitions, timestamp=2022-10-13T10:37:42.035, value=\x8C\x82                                                                         
 configuration                                      column=s:storage.drop-on-clear, timestamp=2022-10-13T10:37:42.039, value=\x8F\x00                                                                      
 configuration                                      column=s:system-registration.7f000101691244-bic-pc1.startup-time, timestamp=2022-10-13T10:37:42.153, value=\xC1\x80\x00\x00\x00cG\xEAv\x01\x11ta\x80   
 \x88\x00\x00\x00\x00\x00\x00\x00                   column=i:\xFF\xFF\xFF\xFF\xFF\xFF\xD8\xEF\x00\x00\x01\x83\xD0\xEB\xE2\x077f000101691244-bic-pc1, timestamp=2022-10-13T10:37:43.303, value=             
 \x88\x00\x00\x00\x00\x00\x00\x03                   column=i:\xFF\xFF\xFF\xFF\xFF\xFE\xC7\x7F\x00\x00\x01\x83\xD0\xEB\xE3c7f000101691244-bic-pc1, timestamp=2022-10-13T10:37:43.651, value=                
 \x88\x00\x00\x00\x00\x00\x00\x80                   column=e:\x02, timestamp=2022-10-13T10:37:44.001, value=\x00\x01\x04\x91                                                                               
 \x88\x00\x00\x00\x00\x00\x00\x80                   column=e:$, timestamp=2022-10-13T10:37:44.001, value=\x04\x8D\x08\x91\xFF                                                                              
 \xFA<[T\x11\xA5\x82                                column=g:\x00\x04\x8D\x0C\x80, timestamp=2022-10-13T10:37:44.127, value=\x04\x8D                                                                       
9 row(s)
Took 0.0498 seconds

Metadata response

hbase:004:0> scan 'hbase:meta', {FILTER=>"PrefixFilter('janusgraph')", COLUMNS=>['info:regioninfo']}
ROW                                                 COLUMN+CELL                                                                                                                                            
 janusgraph,,1665657444679.32f32af13dd119840a3d0b3c column=info:regioninfo, timestamp=2022-10-13T10:37:41.600, value={ENCODED => 32f32af13dd119840a3d0b3c75e01e99, NAME => 'janusgraph,,1665657444679.32f32
 75e01e99.                                          af13dd119840a3d0b3c75e01e99.', STARTKEY => '', ENDKEY => ''}                                                                                           
1 row(s)

STARTKEY and ENDKEY are empties.

So, that lead to empty inputSplits here org.janusgraph.hadoop.formats.hbase.HBaseBinaryInputFormat#getSplits

farodin91 · 2022-10-13T11:17:46Z

@mad any idea why metadata is empty?

mad · 2022-10-13T12:23:45Z

@farodin91

Actually root cause is spark https://spark.apache.org/docs/3.2.0/core-migration-guide.html#upgrading-from-core-31-to-32

Since Spark 3.2, spark.hadoopRDD.ignoreEmptySplits is set to true by default which means Spark will not create empty partitions for empty input splits. To restore the behavior before Spark 3.2, you can set spark.hadoopRDD.ignoreEmptySplits to false.

So, just put spark.hadoopRDD.ignoreEmptySplits=false to hbase-read.properties and hbase-read-snapshot.properties

farodin91 · 2022-10-14T06:11:51Z

@mad Thank you.

...src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/strategy/JanusGraphStepStrategy.java

janusgraph-core/src/main/java/org/janusgraph/core/JanusGraph.java

docs/changelog.md

farodin91 · 2022-10-14T14:17:08Z

@mad Updated

Fixes JanusGraph#3069 * Upgrade hadoop 3.x * Remove Jackson 1 Signed-off-by: Jan Jansen <[email protected]>

mad · 2022-10-17T08:29:28Z

@farodin91

3.6 has some breaking changes https://github.com/apache/tinkerpop/blob/master/CHANGELOG.asciidoc - (breaking) mark

Do we need to take this into account?

farodin91 · 2022-10-18T07:30:21Z

@mad All breaking changes, i've handled multiple breaking changes. https://issues.apache.org/jira/browse/TINKERPOP-2507 and gryo removal.

farodin91 · 2022-10-20T16:00:28Z

@porunov Would you like to review it again?

porunov

LGTM. Thank you @farodin91 !

janusgraph-bot added the cla: external Externally-managed CLA label Sep 15, 2022

farodin91 force-pushed the tinkerpop branch from 692d637 to 5a46f65 Compare September 15, 2022 18:46

porunov reviewed Sep 15, 2022

View reviewed changes

farodin91 force-pushed the tinkerpop branch 2 times, most recently from abdd217 to 74a2cd3 Compare September 15, 2022 20:35

farodin91 added this to the Release v1.0.0 milestone Sep 15, 2022

farodin91 force-pushed the tinkerpop branch 2 times, most recently from 9387232 to a83351d Compare September 16, 2022 08:39

farodin91 force-pushed the tinkerpop branch 6 times, most recently from 4ed1715 to 5f1c4a6 Compare September 26, 2022 07:46

farodin91 force-pushed the tinkerpop branch from 5f1c4a6 to 0108190 Compare October 1, 2022 10:28

farodin91 force-pushed the tinkerpop branch from 0108190 to 04ac4f3 Compare October 2, 2022 13:04

li-boxuan linked an issue Oct 4, 2022 that may be closed by this pull request

Update to TinkerPop 3.6.x #3069

Closed

farodin91 force-pushed the tinkerpop branch from 04ac4f3 to c6d6316 Compare October 9, 2022 14:13

farodin91 changed the title ~~Update TinkerPop 3.6.1~~ Update TinkerPop 3.6.1 [tp-tests] Oct 12, 2022

farodin91 force-pushed the tinkerpop branch from c6d6316 to 1141fe7 Compare October 13, 2022 14:36

farodin91 marked this pull request as ready for review October 13, 2022 21:20

farodin91 requested a review from a team October 14, 2022 04:29

rngcntr reviewed Oct 14, 2022

View reviewed changes

...src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/strategy/JanusGraphStepStrategy.java Show resolved Hide resolved

janusgraph-core/src/main/java/org/janusgraph/core/JanusGraph.java Show resolved Hide resolved

mad reviewed Oct 14, 2022

View reviewed changes

docs/changelog.md Show resolved Hide resolved

farodin91 force-pushed the tinkerpop branch from 1141fe7 to 63247a3 Compare October 14, 2022 14:16

farodin91 requested a review from rngcntr October 14, 2022 14:17

Update TinkerPop 3.6.1 [tp-tests]

14e8ee2

Fixes JanusGraph#3069 * Upgrade hadoop 3.x * Remove Jackson 1 Signed-off-by: Jan Jansen <[email protected]>

farodin91 force-pushed the tinkerpop branch from 63247a3 to 14e8ee2 Compare October 15, 2022 09:05

mad approved these changes Oct 20, 2022

View reviewed changes

porunov approved these changes Oct 21, 2022

View reviewed changes

rngcntr approved these changes Oct 24, 2022

View reviewed changes

farodin91 merged commit ae07bbd into JanusGraph:master Oct 24, 2022

farodin91 deleted the tinkerpop branch October 24, 2022 07:19

lpalashevski mentioned this pull request Nov 10, 2022

OpenLineage FVT fails with gremlin 3.5.2->3.6.0 update odpi/egeria#6602

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update TinkerPop 3.6.1 [tp-tests] #3208

Update TinkerPop 3.6.1 [tp-tests] #3208

farodin91 commented Sep 15, 2022 •

edited

Loading

porunov Sep 15, 2022

porunov Sep 15, 2022

farodin91 Sep 15, 2022

farodin91 commented Sep 19, 2022 •

edited

Loading

farodin91 commented Oct 1, 2022

porunov commented Oct 1, 2022

farodin91 commented Oct 3, 2022

farodin91 commented Oct 9, 2022

mad commented Oct 12, 2022

farodin91 commented Oct 12, 2022

mad commented Oct 13, 2022

farodin91 commented Oct 13, 2022

mad commented Oct 13, 2022

farodin91 commented Oct 14, 2022

farodin91 commented Oct 14, 2022

mad commented Oct 17, 2022

farodin91 commented Oct 18, 2022

farodin91 commented Oct 20, 2022

porunov left a comment

Update TinkerPop 3.6.1 [tp-tests] #3208

Update TinkerPop 3.6.1 [tp-tests] #3208

Conversation

farodin91 commented Sep 15, 2022 • edited Loading

For all changes:

For code changes:

For documentation related changes:

porunov Sep 15, 2022

Choose a reason for hiding this comment

porunov Sep 15, 2022

Choose a reason for hiding this comment

farodin91 Sep 15, 2022

Choose a reason for hiding this comment

farodin91 commented Sep 19, 2022 • edited Loading

farodin91 commented Oct 1, 2022

porunov commented Oct 1, 2022

farodin91 commented Oct 3, 2022

farodin91 commented Oct 9, 2022

mad commented Oct 12, 2022

farodin91 commented Oct 12, 2022

mad commented Oct 13, 2022

farodin91 commented Oct 13, 2022

mad commented Oct 13, 2022

farodin91 commented Oct 14, 2022

farodin91 commented Oct 14, 2022

mad commented Oct 17, 2022

farodin91 commented Oct 18, 2022

farodin91 commented Oct 20, 2022

porunov left a comment

Choose a reason for hiding this comment

farodin91 commented Sep 15, 2022 •

edited

Loading

farodin91 commented Sep 19, 2022 •

edited

Loading