Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SBML TestSuite semantic/01592 takes 15-25 minutes to run #39

Open
matthiaskoenig opened this issue May 23, 2020 · 10 comments
Open

SBML TestSuite semantic/01592 takes 15-25 minutes to run #39

matthiaskoenig opened this issue May 23, 2020 · 10 comments

Comments

@matthiaskoenig
Copy link
Collaborator

matthiaskoenig commented May 23, 2020

The test takes much too long to run. This creates a ton of problems. E.g. very long travis runtimes and also very long runtimes for the local tests. Tests should normally run < 100ms, upper bound is something of 1-5 seconds for very large and complicated tests. But minutes is too long.

This is most likely due to the delay in the model.

matthiaskoenig added a commit to hemilpanchiwala/SBSCL that referenced this issue May 23, 2020
@hemilpanchiwala
Copy link
Member

@matthiaskoenig, the test case 1287 from SBML Test Suite, also has a similar problem.

@draeger
Copy link
Member

draeger commented Aug 5, 2020

What are the reasons for the long run time? Are particular SBML elements / attributes used?

@hemilpanchiwala
Copy link
Member

The description of model 01592 test case from the SBML Test Suite:

category: Test
synopsis: Competing events with the same priority, jointly causing a parameter to monotonically increase, checking to make sure the two events are not exactly evenly distributed. NOTE: STOCHASTIC TEST. Your software may fail periodically; it is only supposed to succeed in the majority of cases.

This model contains two events with the same trigger, the same effective priority, both set 'persistent=false', and both of which disable the trigger of the other. This means that every .01 seconds, one fires and the other does not, at random, and increases the parameters Q or R, respectively. A third parameter, S, is assigned the value of Q+R, meaning that it doesn't matter which one fires; S will increase monotonically. A final parameter, 'error' checks to make sure neither Q nor R are chosen more frequently than the other--if the difference gets higher than 0.2, it triggers. Note: The 'errorLow' and 'errorHigh' parameters are a stochastic test, and may not always remain at '0' for all runs. If your software fails, try running it again with a new random number seed, and it may succeed. The values were chosen to be reasonable in the vast majority of cases, but still high enough to reveal problems in software that tends to pick both events either evenly or in an unbalanced manner. This test is a repeat and synthesis of tests 952 and 962, with the additional wrinkle that a delay is used in the assignments.

@draeger
Copy link
Member

draeger commented Aug 6, 2020

@hemilpanchiwala Could it also help here if we identify some successful seed value to speed up the test, i.e., does it explicitly use a random number generator?

@hemilpanchiwala
Copy link
Member

@draeger, do we use any seed in the simulation for SBML tests🤔?

@draeger
Copy link
Member

draeger commented Aug 6, 2020

I think not, but it was a question from my side - maybe we should discuss this. How do we randomly decide which event to pick if they are of equal priority? Are random numbers involved or are they in a randomized data structure (such as a Set), or do we deterministically always pick the first in a list?

@hemilpanchiwala
Copy link
Member

Yes, it just clicked to me that there is a method pickRandomEvent() in EquationSystem class. Maybe that can help.
Link: https://github.com/hemilpanchiwala/SBSCL/blob/fern/src/main/java/org/simulator/sbml/EquationSystem.java#L1726

@hemilpanchiwala
Copy link
Member

Also, this class RNG which is accessed in pickRandomEvent() has variable random_seed.
https://github.com/hemilpanchiwala/SBSCL/blob/fern/src/main/java/org/simulator/math/RNG.java

@draeger
Copy link
Member

draeger commented Aug 6, 2020

See, there is our random number.

@hemilpanchiwala
Copy link
Member

Yes, but it is accessed only once. I mean seed value is set only once in the simulation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants