Integration Tests Failing on Develop #6731

djbrooke · 2020-03-09T14:22:31Z

We're seeing integration tests fail over the past few runs in develop:

edu.harvard.iq.dataverse.api.HarvestingServerIT.testOaiFunctionality
edu.harvard.iq.dataverse.api.DatasetsIT.testCreatePublishDestroyDataset
edu.harvard.iq.dataverse.api.DatasetsIT.testExcludeEmail

Some details about the failures:

https://jenkins.dataverse.org/job/IQSS-dataverse-develop/382/testReport/

donsizemore · 2020-03-09T21:49:35Z

@djbrooke FWIW I just ran this through in a VM on my new SSD. 8 cores and 8GB of RAM made the tests run cleanly, and the API test suite kept most of the cores pegged for the duration of the tests.

djbrooke · 2020-03-10T01:48:15Z

Thanks @donsizemore, that's good to hear. Is there an easy way to give the jenkins.dataverse.org instance some more juice so it can get through the tests?

cc @landreev because you mentioned you were going to look into the Harvesting test

donsizemore · 2020-03-10T14:41:21Z

@djbrooke tweaking the JVM heap didn't help... there are 181 occurrences of the following error, which I haven't seen before:

server.log_2020-03-10T14-18-55: java.lang.IllegalAccessException: Class edu.harvard.iq.dataverse.search.SearchServiceBean can not access a member of class edu.harvard.iq.dataverse.search.SearchFields with modifiers "private static transient" at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102) at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296) at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288) at java.lang.reflect.Field.get(Field.java:390) at edu.harvard.iq.dataverse.search.SearchServiceBean.search(SearchServiceBean.java:684)

landreev · 2020-03-19T19:28:20Z

OK, it appears that the 3 tests that have been failing consistently are the result of #6739.
That in turn was introduced in PR #6670; merged on Feb. 27,and that's when the RestAssured tests started failing.

landreev · 2020-03-19T19:28:50Z

So let's maybe put this one on hold, until #6739 is fixed.

landreev · 2020-03-24T14:57:35Z

OK, fixing #6739 appears to have fixed the 3 tests that were failing previously:

edu.harvard.iq.dataverse.api.HarvestingServerIT.testOaiFunctionality
edu.harvard.iq.dataverse.api.DatasetsIT.testCreatePublishDestroyDataset
edu.harvard.iq.dataverse.api.DatasetsIT.testExcludeEmail

However, it also appears that #6490 (also merged yesterday) has introduced 3 new failures:

edu.harvard.iq.dataverse.api.FilesIT.testReplaceFileBadJson
edu.harvard.iq.dataverse.api.FilesIT.test_006_ReplaceFileGood
edu.harvard.iq.dataverse.api.FilesIT.testForceReplaceAndUpdate

kcondon · 2020-03-24T19:06:29Z

I did a quick replace file test on develop and it failed silently in UI but saw this in log:

[2020-03-24T15:00:11.622-0400] [glassfish 4.1] [WARNING] [] [javax.enterprise.resource.webcontainer.jsf.lifecycle] [tid: _ThreadID=209 _ThreadName=jk-connector(4)] [timeMillis: 1585076411622] [levelValue: 900] [[
  /editFilesFragment.xhtml @144,65 fileUploadListener="#{EditDatafilesPage.handleFileUpload}": java.lang.NullPointerException
javax.el.ELException: /editFilesFragment.xhtml @144,65 fileUploadListener="#{EditDatafilesPage.handleFileUpload}": java.lang.NullPointerException
        at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:111)
        at org.primefaces.component.fileupload.FileUpload.broadcast(FileUpload.java:63)
        at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:755)
        at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:931)
        at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:78)
        at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
        at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646)
        at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:295)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
        at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
        at edu.harvard.iq.dataverse.datasetutility.AddReplaceFileHelper.runAddReplacePhase1(AddReplaceFileHelper.java:506)
        at edu.harvard.iq.dataverse.datasetutility.AddReplaceFileHelper.runReplaceFromUI_Phase1(AddReplaceFileHelper.java:459)
        at edu.harvard.iq.dataverse.datasetutility.FileReplacePageHelper.handleNativeFileUpload(FileReplacePageHelper.java:117)
        at edu.harvard.iq.dataverse.EditDatafilesPage.handleReplaceFileUpload(EditDatafilesPage.java:1874)
        at edu.harvard.iq.dataverse.EditDatafilesPage.handleFileUpload(EditDatafilesPage.java:1984)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.el.parser.AstValue.invoke(AstValue.java:289)
        at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:304)
        at org.jboss.weld.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40)
        at org.jboss.weld.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50)
        at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
        ... 45 more
]]

djbrooke · 2020-03-24T19:15:09Z

Thanks @kcondon.

@qqmyers based on the log entry is it likely that this was introduced by the work in #6490? If so, could you take a look?

qqmyers · 2020-03-24T21:58:33Z

I looked briefly this afternoon. It's going to be very hard to debug without seeing the log for the tests to find out where the problem starts. Is there a way to get that? (I didn't see anything obvious in jenkins).
My guesses were either that the test uses a default/dummy value that is no longer valid, or that it doesn't set something that exists for normal datasets but was left out to keep the test simple. As I said, hard to tell without being able to see the details prior to the test failure.

landreev · 2020-03-24T22:03:06Z

@qqmeyers The server logs from the node that's running the tests are available through jenkins; hold on, I'll find the links.
But, from what @kcondon is saying above, it sounds like it's not just Jenkins - more like file replace is just broken in the develop branch as of now - ?
(and what happens when you just try these 3 RestAssured tests in your own environment?)

qqmyers · 2020-03-24T22:04:34Z

Ah - didn't look up. His log will help.

landreev · 2020-03-24T22:05:56Z

@qqmyers The server logs and such from the last Jenkins run:
https://jenkins.dataverse.org/job/IQSS-dataverse-develop/ws/target/

(and of course the full console output log of the jenkins test job is here: https://jenkins.dataverse.org/job/IQSS-dataverse-develop/396/consoleFull)

landreev · 2020-03-24T22:08:55Z

(also, the EC2 instance that ran the test is still up - they are kept alive for a couple of days - so if you need shell access to it, Don should be able to send the key)

qqmyers · 2020-03-24T22:22:08Z

OK - hopefully an easy fix in the PR above

landreev self-assigned this Mar 17, 2020

qqmyers mentioned this issue Mar 24, 2020

Fix replace bug #6765

Merged

kcondon closed this as completed in #6765 Mar 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration Tests Failing on Develop #6731

Integration Tests Failing on Develop #6731

djbrooke commented Mar 9, 2020

donsizemore commented Mar 9, 2020

djbrooke commented Mar 10, 2020

donsizemore commented Mar 10, 2020

landreev commented Mar 19, 2020

landreev commented Mar 19, 2020

landreev commented Mar 24, 2020

kcondon commented Mar 24, 2020

djbrooke commented Mar 24, 2020

qqmyers commented Mar 24, 2020

landreev commented Mar 24, 2020

qqmyers commented Mar 24, 2020

landreev commented Mar 24, 2020

landreev commented Mar 24, 2020

qqmyers commented Mar 24, 2020

Integration Tests Failing on Develop #6731

Integration Tests Failing on Develop #6731

Comments

djbrooke commented Mar 9, 2020

donsizemore commented Mar 9, 2020

djbrooke commented Mar 10, 2020

donsizemore commented Mar 10, 2020

landreev commented Mar 19, 2020

landreev commented Mar 19, 2020

landreev commented Mar 24, 2020

kcondon commented Mar 24, 2020

djbrooke commented Mar 24, 2020

qqmyers commented Mar 24, 2020

landreev commented Mar 24, 2020

qqmyers commented Mar 24, 2020

landreev commented Mar 24, 2020

landreev commented Mar 24, 2020

qqmyers commented Mar 24, 2020