Scala ScriptEngine implementation is substantially slower than others

[scabug]

I'm currently running a simple benchmark to decide which ScriptEngine I should incorporate in my project. This project requires for a massive amount of small, short-lived, yet changing script executions to be made concurrently (meaning that I can't just cache these scripts).

Therefore, the overhead of the method "AbstractScriptEngine.eval" and of the general ScriptEngine mechanism is crucial to me. I also care about the performance of "ScriptEngineManager.getEngineByName" but it's less critical, because in my context the engine itself can be cached much longer than the scripts themselves.

Based on my results and as of the scala compiler in version 2.12.0, it seems I'd have to pay, approximately, a 200 ms toll for evaluating a script that only prints a simple value to the standard output, and a 3500 to 4500 seconds toll for invoking the engine.

This basically means I can not use scala or at least this implementation of the compiler in my project.

I compared the performance of scala's implementation of JSR 223 to other engines (groovy, rhino, nashorn, jython, I also tried jruby but I'm running into a bug there). In every of the other three working scenarios, not only is the execution time of both methods much faster but it reduces dramatically when the process is repeated inside the same JVM, which leads me to believe that :

1- other JSR-223 implementations cache the compiler classes (and possibly more)
2- the scala implementation could probably be optimized in the same way, which means in turn that I could use scala in my project

So I went ahead and started profiling both calls (getEngineByName and eval). Here are the response time distributions in the sub-packages:

getEngineByName : !getEngineByName.JPG|thumbnail!

eval : !eval.JPG|thumbnail!

I might not be the only one facing this problem and I think this would improve scala's chances to be adopted by people with more of a Java background like myself.

I'm also copy-pasting the code the test program I used to compare the different engines (see below). For profiling I just iterated many times over the scala one.

Could someone please look at these findings and tell me if the suggested optimization (caching) can be implemented and if yes, would someone do it / by when?

Best Regards,

Dorian Cransac

import javax.script.Bindings;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
import javax.script.SimpleBindings;

public class TestEngine {

	public static void main(String...strings){

	for(String s : new String[] {"groovy", "scala", "rhino", "nashorn", "jython", "jruby"})
		//for(String s : new String[] {"scala"})
		{
			for(int i = 0; i <3; i++)
				speedTest(s);
		}		
	}
	
	public static void speedTest(String engineKey){
		long s1 = System.currentTimeMillis();
		ScriptEngine engine = new ScriptEngineManager().getEngineByName(engineKey);
		long end1 = System.currentTimeMillis();
		
		if(engine == null){
			System.out.println(engineKey + " engine not found.");
			return;
		}
		
		String script = null;
		if(engineKey.equals("jython"))
			script = "from java.lang import System as javasystem\njavasystem.out.println(\""+engineKey+"\")";
		else{
			script = "java.lang.System.out.println(\""+engineKey+"\")";
		}
		
		Bindings b = new SimpleBindings();
		//b.put("n", 10);
		

		
		try {
			long s2 = System.currentTimeMillis();
			//engine.eval("1 to n.asInstanceOf[Int] foreach java.lang.System.out.println", b);
			engine.eval(script);
			long end2 = System.currentTimeMillis();
			System.out.println("getEngineByName:" + (end1 - s1) + ", eval:" + (end2-s2));
		} catch (ScriptException e) {
			e.printStackTrace();
		}
	}

}

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-10089?orig=1
Reporter: Dorian Cransac (dcransac)
Affected Versions: 2.12.0
Attachments:

• eval.JPG (created on Dec 4, 2016 4:26:12 PM UTC, 145470 bytes)
• getEngineByName.JPG (created on Dec 4, 2016 4:26:12 PM UTC, 103041 bytes)

[scabug]

Dorian Cransac (dcransac) said (edited on Dec 4, 2016 4:50:16 PM UTC):
I forgot to paste the results of the benchmark themselves (3 iterations per engine type):

groovy
getEngineByName:123, eval:317
groovy
getEngineByName:1, eval:8
groovy
getEngineByName:2, eval:7
scala
getEngineByName:4505, eval:192
scala
getEngineByName:3633, eval:186
scala
getEngineByName:3934, eval:239
rhino
getEngineByName:150, eval:17
rhino
getEngineByName:6, eval:6
rhino
getEngineByName:3, eval:4
nashorn
getEngineByName:183, eval:120
nashorn
getEngineByName:6, eval:3
nashorn
getEngineByName:6, eval:5
console: Failed to install '': java.nio.charset.UnsupportedCharsetException: cp0.
jython
getEngineByName:2101, eval:7
jython
getEngineByName:1, eval:3
jython
getEngineByName:1, eval:2
Exception in thread "main" java.lang.NoSuchFieldError: O_TMPFILE
	at org.jruby.RubyFile.createFileClass(RubyFile.java:202)
	at org.jruby.Ruby.initCore(Ruby.java:1570)
	at org.jruby.Ruby.bootstrap(Ruby.java:1352)
	at org.jruby.Ruby.init(Ruby.java:1247)
	at org.jruby.Ruby.newInstance(Ruby.java:339)
	at org.jruby.embed.internal.AbstractLocalContextProvider.getGlobalRuntime(AbstractLocalContextProvider.java:82)
	at org.jruby.embed.internal.SingletonLocalContextProvider.getRuntime(SingletonLocalContextProvider.java:99)
	at org.jruby.embed.internal.EmbedRubyRuntimeAdapterImpl.runParser(EmbedRubyRuntimeAdapterImpl.java:167)
	at org.jruby.embed.internal.EmbedRubyRuntimeAdapterImpl.parse(EmbedRubyRuntimeAdapterImpl.java:94)
	at org.jruby.embed.ScriptingContainer.parse(ScriptingContainer.java:1239)
	at org.jruby.embed.jsr223.JRubyEngine.eval(JRubyEngine.java:89)
	at org.jruby.embed.jsr223.JRubyEngine.eval(JRubyEngine.java:142)
	at TestEngine.speedTest(TestEngine.java:44)
	at TestEngine.main(TestEngine.java:15)

[scabug]

Dorian Cransac (dcransac) said:
updated the code and benchmark results with groovy results, and translated a couple of words which were initially left in french.

[scabug]

@som-snytt said:
Currently, it will always compile your script. To avoid recompilation, use Compilable as here.

But modifying your example:

        script = "java.lang.System.out.println(n)";
        engine.put("n", -1);
        CompiledScript compiled = ((Compilable) engine).compile(script);
        Bindings b = engine.createBindings();
         
        for (int n = 0; n < 10; n++) {
            long s2 = System.currentTimeMillis();
            b.put("n", n);
            compiled.eval(b);
            long end2 = System.currentTimeMillis();
            System.out.println("getEngineByName:" + (end1 - s1) + ", eval:" + (end2-s2));
        }

results in

0
getEngineByName:536, eval:3
1
getEngineByName:536, eval:103
2
getEngineByName:536, eval:98

It looks like the first invocation is simple invocation, but subsequently it incurs a compilation. That could be improved, here.

[scabug]

Dorian Cransac (dcransac) said:
As stated in the description of the issue, unfortunately I can't use a CompiledScript because I can't cache nor pool them between calls to that section of the code. The scripts are different each time. Therefore, the performance of the compilation phase is critical to me.

I read about how scala supposedly has a tougher time compiling due to various specific features or constraints, but the thing I don't understand here is why is there no diminishing return for loading the Engine twice or more, and then why do I have to wait 200ms to compile a script with a *single *instruction?

Again : every other compiler takes a while to load the engine the first time (presumably because of class loading), but the 2nd, 3rd (etc) times are instantaneous. That's why I'm suspecting certain objects (classes in this case) could be cached in the JVM and probably aren't.

On a separate note : did you use the same piece of code as I did for the "getEngineByName" measurement? Strangely enough you're coming up with a value 5 times smaller than in my own executions (even though this value may be influenced by a number of factors such as CPU speed, classpath "size" and such, I can't see why this would be so much faster on your side).

Lastly :can you develop on why executions #2 and #3 are slower than execution #1 ?! I would have expected the opposite.

[scabug]

@som-snytt said:
As stated, currently it will always compile, so it doesn't seem to fit your use case. I don't know why you don't see getEngineByName get cheaper when warm. The warm cost includes initializing the compiler and running a compilation, which is done for interactive use.

[scabug]

Dorian Cransac (dcransac) said:
Ok fine. I guess I'll turn to another compiler / language, then.

[scabug]

@adriaanm said:
Scala is fundamentally a compiled language, with the scriptingengine/repl implemented on top of that. It has to do a full type check etc every time, whereas the other languages you're comparing against don't need to do that, so it sound like they are indeed a better fit for your application.

[scabug]

Dorian Cransac (dcransac) said:
I understand. And I appreciate the feedbacks to try and help guiding me.

Even though I read that type inference is costly by nature, I just wanted to make sure I wouldn't pass on this option just because of a more trivial performance issue and a corresponding potential optimization that we'd miss. That's why I went ahead and actually profiled that runtime phase.

Now if you guys tell me there's nothing that can be done about, I'll accept it :)

[scabug]

@adriaanm said:
It depends on your requirements, I guess. Once the compiler is running, the REPL works fast enough for interactive use (i.e., in the 100ms range). The same technique is used in a lot of big data worksheets, so it's certainly possible to use Scala interactively. If you're looking for 1ms response times, probably not.

[scabug]

@som-snytt said:
The part about reinvoking a CompiledScript is still an open issue. Once that is improved, it would be nice to be Invocable.

[scabug]

@adriaanm said:
Could you narrow the scope / description so it's in scope and reopen?

[scabug]

Dorian Cransac (dcransac) said:
@adriaan Is this intended for me? I'm not sure I fully understand the reinvocation problem. Maybe it's better if one of you creates a new issue referencing this one, as I won't be much help there.

[scabug]

@SethTisue said:
@som-snytt is there already a ticket on the CompiledScript issue?

[scabug]

@som-snytt said:
No, I think this would be that ticket if amended.