What happened to Pyjeon/RapydScript?

[v3ss0n]

So this is continuation of pyjeon's Rapydscript ? Why not a fork there ?

[v3ss0n]

just read the bottom of the change list, very well. and since Pyjeon's Rapydscript is practically dead , i think this should become main . So many new features compare to original.

[kovidgoyal]

You'd have to convince @atsepkov to relinquish maintainership of the original rapydscript for that to happen. When I first made this fork, he had no intention of doing that. So we decided I would continue working on -ng and he would merge in the changes from here as and when he found the time. Since then, he does not seem to have found much time -- but I have no idea whether he is willing to relinquish maintainership or not.

[atsepkov]

Kovid, that's not true. I've offered to give commit rights to original RapydScript multiple times. It was your decision to keep the forks separate, I still have the emails with that conversation. At this point I realize that the fundamentals of the language may differ too much to allow for a merge. With that said, you're still welcome to the write access for the original RS, if you desire it. What I don't want to do is delete it altogether in favor of this version, which doesn't even seem to have been forked properly.

[kovidgoyal]

I dont know what I would do with write access to the original repo, unless you are actually ready to merge in all the code from rs-ng, in particular the backwards incompatible changes. When we discussed this originally, you were not ready to do that. Has that changed?

Let me remind you that I only created rapydscript-ng after you said you were not willing to merge in all the changes I was making. And despite that decision, I have been scrupulously adding links to every single commit that fixes an open bug in the original RapydScript so that you can merge them in easily, despite the diverged codebases.

[atsepkov]

I understand, what I'm saying is I'm not simply being stubborn for the sake of being stubborn. I want to make sure other projects relying on original RS (including GlowScript) remain functional. There are many parts of your version that I've accepted back into original. There are, however, also parts that I prefer from the original version instead (such as the less-intrusive kwargs, performant true equality operator, and smaller baselib footprint). If we can address these issues in the merge, and figure out how to make RS still relevant in ES6 world, then I'm all for it.

[kovidgoyal]

The whole point of having a 0.x version number is that you have the freedom to break backwards compatibility. So IMO trying to preserve backwards compatibility should not be an overriding concern. If there is a good reason to break backwards compat, then it should be broken. For projects that are not willing to migrate their code, old releases of rapydscript remain available.

Now coming to the other points you mention:

1. less instrusive kwargs -- I dont understand this one at all. If anything the kwargs implementation in rs-ng is less intrusive, in that it does not require a magic decorator in order to support kwargs. Not to mention that its support for *args and **kwargs is far more robust and closer to that of python.

2. == operator. If by this you mean converting == into a === b || typeof a === 'object' && rich_compare(a, b) let me point out that:
   a) I already merged that into rs-ng with a link to your blog post in the commit, so you get the credit
   b) Your implementation is broken if b is an expression that has side effects for example, a == do_something() where do_something() has some side effect and returns a value. This is fixed in rs-ng

3. Smaller baselib footprint -- on this I'm afraid we disagree. To me, providing a consistent runtime environment that has the fundamental container types: string, list, set and dict, as well as important python global functions is more important that saving a few KB in baselib size. The overhead of trying to make all those parts of the baselib into optional dependencies is so high that it simply will not happen -- at least, I dont have the energy/motivation to make it happen.

[kovidgoyal]

Oh and as for relevance of RapydScript in an ES 6 world. My personal motivation for using RapydScript is that, as a python developer who also has to write lots of JavaScript, my productivity is much higher when that JavaScript can be written in a language close to python. That motivation is not affected by ES 6, at all.

[v3ss0n]

I hope my issue doesn't lead to a flame-war. I am sorry, I didn't knew any of communication issues between two maintainers. But to voice my opinions why Rapydscript is important and keep you guys working :

• Javascript sucks , ES6 can't help. Python is better than javascript in almost every way possible.
• Rapydscript and Rusthon's JS are only true serious Python transpilers for Javascript.
• These days , javascript become main language for client-side programming at web, we are left without choices, front-end LOC in javascript become 2x-3x more than serverside Python .

So guys , please continue working on Rapydscript and also RapydML . You will save a lot of Python Developers' Life. I hope There is no need for in-fighting .

[kovidgoyal]

Dont worry, this is not a flame war -- the question is simply whether @atsepkov and I can reach a compromise that allows us to merge the two codebases. If not, things will continue as they have been. I fully intend to keep working on rs-ng, as I use it extensively in my other projects.

[v3ss0n]

So it seems , this is an inevitable fork then.

as a user , here are my opinions

• Javascript frameworks are now huge , over megabytes , for pyjeon there is no problem of increasing a hundred kilobytes, if Its going to save hundreds of hours of developers and better experience and functionality for end-user. Even at developing Asian country like ours , average connections speeds are 3MB/s now . There is no point saving a few hundred kilobytes at all.

Oh and as for relevance of RapydScript in an ES 6 world.

• I can't agree more , javascript at larger scale is un-maintainable at best , we really need rapydscript , even with best practices in javascript/typescript with full object oriented , compare to python, just modular python functions are much much easier to maintain, RS is very relevant in JS6 world , especially huge projects .

[v3ss0n]

there is a good reason to break backwards compat, then it should be broken. For projects that are not

I have never seen a release of RapydScript so , i believe nobody would care if it breaks backward compatilbity , as old version user won't even update as they do not see any release at all.

[atsepkov]

@kovidgoyal the kwargs decorator has been replaced by a more subtle system almost a year ago. The kwargs decorator has been unnecessary since then, the new system automatically curries the individual call in kwargs when it sees ** unary prefix for the last argument. I.e.:

args = {'foo': 1, 'bar': 2}
test(**args)
-----
var args = {'foo': 1, 'bar': 2}
kwargs(test)(args)

There are also other subtle differences between the forks, such as magic function JS() being replaced with magic string and some others I may not be aware of. Some of these changes seem like a matter of individual preference and neither is compatible with Python. For these changes I'd want some sort of sanity check from the community first. And sure, the community may not be large/vocal enough, but I'd want incompatible changes to at least consider the use cases of other developers, I'm more than happy to provide my own, despite being less active in development now. @BruceSherwood has been providing feedback on RS through its evolution as well.

As for ES6, I agree that it will not clean up JS. However, it introduces features like list comprehensions, generators, and classes, many of which RS had to implement manually. Now may be a good time to simplify the language to make use of new ES6 syntax sugar when possible, and solutions like Babel.js may make backwards compatibility irrelevant.

[kovidgoyal]

The kwargs support I am talking about is more than just support for **kw, it means being able to do the following:

def a(a, b=1, c=2):
  return [a, b, c]

a(1, b=2) == [1, 2, 3]

Which you can do in rs-ng but not (at least the last time I checked) in RS without using the kwargs decorator.

Now to next point you raise:

JS() is a magic function. The problem with magic functions is that they break user code. Using a function for JS literals means that name JS cannot be used in user code and can even lead to subtle and hard to debug errors if the user uses that name without realizing the compiler is going to replace it with a javascript literal. For instance suppose the user uses an external javascript library that defines a function named JS() and then he calls it in RapydScript as JS('1'). that would become '1' in your fork instead of the result of calling the function JS() with the argument '1'.

Using a string literal for JS on the other hand is not magic. It has far less potential for user surprise, and does not restrict user code in any way. It is backwards incompatible, but for a good reason and is moreover a backwards incompatibility that is trivially caught by the linter I developed. In fact there are many more magic compile time function in original RS that I have converted to functions in the baselib, so that they are no longer magic, but instead simple global functions. For example, bool()

There is no backwards incompatible change I have made in rs-ng without a good reason. You are free to disagree with those reasons, but kindly do not dismiss them as personal preference.

You have not provided any reasons (as far as I can tell) for why you think the kwargs implementation in rs-ng is "more intrusive"? So can I assume you are OK with it? If not, I'd like to hear some reasons. If they are good reasons I am perfectly willing to change the implementation in rs-ng to address your concerns.

Similarly, you did not respond to 2) and 3) above, so can I assume you are OK with those as well? I am trying to nail down what exactly are your reasons for not wanting to merge, beyond the generic waiting for community feedback. If you do actually want community feedback perhaps you should ask for some -- make a new issue in the RS github project asking for feedback or a post in the RS google group. As it stands now, nothing will happen and the current situation will go on forever. Which is a pity for the RapydScript community.

Incidentally, at least @BruceSherwood seems to be happy with the changes in rs-ng, see #12

[atsepkov]

Both of those cases are handled as well (first case doesn't need kwargs):

def a(a, b=1, c=2):
    return [a, b, c]

a(1, b=2) == [1, 2, 3]
----
function a(a, b, c) {
    if (typeof b === "undefined") b = 1;
    if (typeof c === "undefined") c = 1;
}

kwargs(a)(a, {b: 2}) == [1,2,3]

The limitation in your version I didn't like was that function declaration required optional arguments to be considered a kwargs function, my version does not.

For #2, yes it sounds like your solution is good, haven't played with it

For #3, I'm not a fan, but don't have any solution to propose at the moment that's a good compromise.

For other magic functions like bool(), maybe I'm too anal about the performance overhead of a function call, I just don't want RS user to feel like they're taking a performance hit over JS user, I've often felt like that when using other Python-in-a-browser solutions, but JS today is much faster. Still, if I was to put that bool() in a loop, it would be quite a bit slower than !! which will be used by a JS dev.

[kovidgoyal]

That is good, if your kwargs implementation does support all the feautures of mine without needing the function declaration to have keyword arguments, then it is clearly superior, and I have no problem whatsoever with adopting it.

As for magic functions, IMO, people that require that last iota of performance from bool(), can always use literal javascript inline. I dont think that sort of optimizations should be baked into the language at compiler level. It's confusing why you cannot re-assign bool to your own custom function if you need to. In general I am not a fan of this kind of magic.

So to summarize, your remaining objections are:

1. kwargs -- I am happy to switch implementations, if yours is indeed a superset of mine

2. baselib size -- are you willing to live with the expanded baselib in rs-ng? As I dont see how any other type of compromise is possible there.

3. Community feedback -- all I have heard from the community about rs-ng is positive, but if you wish to seek more feedback before merging, you are welcome to do so. I just urge you to not put off the merge forever because of this reason.

4. Magic functions -- I think from your tone you are ok with making them global functions?

Anything else?

[kovidgoyal]

Playing around with your kwargs implementaion, the following fails:

def f3(a, b=2, **kwargs):
  pass
1:13: **kwargs in function definition is not implemented yet, work in progress

Is this something you intend to implement anytime soon? The ability to accept arbitrary keyword arguments is rather critical for me. I'm afraid I cannot switch implementations until it is implemented.

[kovidgoyal]

Another problem with your kwargs implementation, it parses the string representation of the function on every function call, when using keyword arguments. That seems like a lot of overhead. Not to mention that it will not work on JS engines where Function.prototype.toString() does not return the argument list. One such JS engine is duktape and this engine is actually used by at least one large scale consumer of RapydScript -- calibre.

[kovidgoyal]

I just committed code to remove the restriction on needing to define functions with keyword arguments. You can now do:

def f(a, b):
   return [a, b]
f(b=2, a=1) == [1, 2]

It has no per call overhead (apart from the unavoidable one of creating the kwargs object), works on all JS runtimes and does not rely on parsing the string representation of functions.

So I think we can scratch that of the list of reasons not to replace RS with rs-ng.

And while developing this solution, I also created a framework for doing arbitrary annotations to function objects, which will be used for optional typing in rs-ng, a la, python 3 PEP3107. See #16

[BruceSherwood]

I've been too busy to study this thread in the detail it merits, but I will say that I'm definitely interested in using rapydscript-ng in the GlowScript project, and I recently experimented with this, though ultimately I was blocked by the need to update the paperscript/acorn/streamline libraries as well, to handle the ES6 that is generated by ng. I even tried using Babel to convert the ES6 back to ES5 but ran into problems. I'm definitely in favor of moving to ES6. I'm specifically interested in having calling conventions as similar as possible to standard Python and have been sometimes confused about what works and what doesn't. I'm also particularly interested in having full implementation of dict() and its methods, which I believe ng now provides. Recently though I've been mainly engaged with Ruth Chabay in making the Jupyter version of VPython mature, and compatible with the GlowScript VPython API.

[atsepkov]

Kovid, could you also explain how the dicts and arrays work now in ng, I've glanced at the documentation and having syntax like a = v'[1,2]' for native array scares me. Likewise, the use of exclamation and extra punctuation characters in dict declaration was something that concerned me as well. As someone who worked with perl in the past, I'm very leery of using punctuation in new ways as that adds a lot of confusion to someone unfamiliar with it. I'll admit that I haven't had the time to play with it yet to understand the syntax changes in detail, but why couldn't we simply have the user to type out dict() for a Python dict? Rarely have I been bitten by verbosity, but quite a bit by "syntax sugar".

[BruceSherwood]

Good point, Alex, one that I had forgotten. Kovid made it clear that he wanted to favor a JavaScript interpretation of dictionaries (object literals), hence the unusual use of single quote. I do a lot of preprocessing of a user's VPython program, so in my experiment with ng I simply switched the default, so that the single quote meant JavaScript and its absence meant Python. I should have mentioned this to Kovid at the time of my experiment. It is certainly true that having RapydScript be a thin wrapper around JavaScript has many advantages, but I agree with you, Alex, that here is a case where the default should be Python behavior.

[kovidgoyal]

@BruceSherwood rapydscript-ng can generate either ES 5 or ES 6 code with a compiler flag. The only place where it insists on ES 6 is for the embedded web compiler.

@atsepkov Lists work just like in python. [1, 2] gives you an array object with all the same methods as a python list (except where the python names clash with existing array methods, in which case they are prefixed by py. Similarly, you can do list(any iterable) to generate a list. The reason your see v'[1,2]' in my code is an optimization, since constructing python like list objects is currently a little slow compared to plain JS arrays. The need for this optimization will go away in the future when the compiler can output only ES 6.

As for dicts, the normal python like dict syntax generates Javascript dicts. If you want python like dicts, then you can either use dict({'1': 'a'}) or {!'1':'2'}. The wrinkle comes in item access, where for python like dicts you have to do d[!'1'] instead of d['1']. This is unavoidable, unless you go down the road of virtualizing the [] operator, which is both a big performance hit and makes your dicts incompatible with external JS code.

[BruceSherwood]

The only usage of interest to me is the embedded web compiler. At glowscript.org a registered user writes a Python (VPython) program and the file is stored in the cloud. But when the user clicks "Run this program" the file is compiled and run in the client browser. I do understand the issue of wanting to facilitate compatibility with external JS code, but my usage situation is that it is far more important to be compatible with Python syntax.

[kovidgoyal]

Well if you are OK with having your users download a 6MB compiler I suppose I can add an option for outputting an ES 5 capable embedded compiler as well. Although, I really think that you should make the effort to move to ES 6 -- you will have to do it sooner or later, the benefits are too large to be ignored.

As for python vs. JS dicts, the situation in rs-ng is the same as it is in the original RS, except for in rs-ng you can actually create python compatible dicts if you want to, and rs-ng supports both dict comprehensions and follows python syntax for dict literals rather than JS syntax (neither of which is true in RS, at least the last time checked).

Not to mention that in rs-ng you have both lists and sets and all the python string functions (albeit accessible via the str object not from strings themselves, for reasons of JS compatibility). So as far as your use case goes, rs-ng is strict win over RS.

[kovidgoyal]

Oh and @atsepkov let me remind you that you wanted to use punctuation for operator overloading yourself, in the original RapydScript: atsepkov/RapydScript#43

[BruceSherwood]

I do want to switch to ES6, and it looks to me like I should switch to rs-ng. As I said, I was temporarily stopped by the necessity to update several other libraries to their ES6 versions, which I was not in a position to do at the moment due to other commitments. I should mention that I wrote JS code that handles string.format(...) that you might wish to use. It starts around line 330 in this file: https://github.com/BruceSherwood/glowscript/blob/master/lib/glow/api_misc.js

[kovidgoyal]

@BruceSherwood well now string.format() is part of the language, can be used by everyone out of the box and has its own test suite, which you are welcome to contribute to, if there are any corner cases I have bungled in my implementation.

https://github.com/kovidgoyal/rapydscript-ng/blob/master/src/baselib-str.pyj#L74
https://github.com/kovidgoyal/rapydscript-ng/blob/master/test/str.pyj#L32

Another major benefit (at least for me) in rs-ng string handling is support for python string literal syntax. All of the following work:

"\U0001F431" == '🐱'
"\N{nbsp}" == '\u00a0'
r'\n' == '\\n'

[BruceSherwood]

Looks like you went far beyond what I did, since I didn't consider unicode issues. I'll be happy to delete my own code!

[BruceSherwood]

Perhaps I should make a clear statement of what I need, as my usage situation is quite different from what were perhaps the original goals of RS, which I perceived to be making it possible for web programmers to use Python. I am the developer of GlowScript VPython (glowscript.org), and Ruth Chabay and I are currently the most active developers of IPython/Jupyter VPython ( vpython.org). We are working hard to make it possible to run as nearly as possible the same Python code, at least in fairly simple cases, in both environments, pure browser (GlowScript), especially for novice programmers, and Jupyter notebook, especially for professionals including computational scientists and advanced students who need full access to the large Python ecology of modules. One of the several reasons for wanting to be able to use the same API in both environments is to facilitate students who in their course of study may move from GlowScript to Jupyter. It's important to know that the use of GlowScript VPython, based on RS, is rising extremely fast, with over 8000 registered users at glowscript.org: http://www.glowscript.org/#/user/Bruce_Sherwood/folder/My_Programs/program/Users The meteoric rise seen in this graph began when Salvatore di Dio introduced me to Alex's work, which made it possible for people to write GlowScript programs using VPython rather than JavaScript. Most of the fast rise is almost certainly due to use by physical science and engineering students who are taking the college freshman intro physics course, using the textbook "Matter & Interactions" authored by Ruth and me ( matterandinteractions.org). Institutions using our textbook, and VPython, include Georgia Tech, Purdue, U Texas Austin, Cal State Long Beach, Carleton, St. Olaf, Wellesley, Haverford, etc. So GlowScript's use of RS is a big deal and rapidly getting bigger. As an example of what this means in the context of one of the topics in the present conversation, I'm sympathetic to Alex's view on strange punctuation for Python dictionaries and also sympathetic to Kovid's view on the same matter (the issue of external JS libraries). I would hope that RS can serve more than one need. In this particular case, it would be great if there were switches settable not only in node builds but also as options in the browser-embedded compiler. For example, I would set a switch to favor Python dictionaries but a web programmer might set the switch to favor JavaScript object literals. Finally a confession: Due not only to other pressures but also a weak grasp on the issues, I don't quite understand what is the actual present status of function argument handling in rs and rs-ng. A few months ago I did try to upgrade my copy of the RS compiler but ran into problems I was unable to solve (which I did report at the time), so I'm still using a version of RS from November 2014.

[kovidgoyal]

At some point in the future, I plan on introducing the ability to do

from __python__ import dict_literals, overload_getitem

which will work like from__future__ import in python and tell the compiler that all dict literals in that module should become python dicts and all item access should be overloaded via __getitem__
Eventually one could use this approach to enable per module operator overloading for all the operators in a way that does not degrade performance for people that do not need operator overloading. However, this is a somewhat longer term project and as it is of no use to me personally, it would help if people that were interested in it would contribute some code.

As for function calling, with my recent changes rs-ng should be identical to python in every respect but one (namely raising of TypeErrors when a function is called with incorrect number of arguments).

The original rapydscript's function calling compatibility with python is a strict subset of rs-ng.

Incidentally, @atsepkov while working on the recent changes, I realized that function calling was broken for many cases in both rs and rs-ng. It is now fixed in rs-ng. For example, in rs:

asd().x(b=1)
------
kwargs(asd().x).call(asd(), {b: 1})

asd() is being called twice in the generated JavaScript instead of once. This entire class of problems (when the expression in AST_BaseCall contains a call of its own, is now fixed in rs-ng.

[BruceSherwood]

from python is certainly worth doing. I should comment that for my own
parochial needs it would also be sufficient for the in-browser compiler to
accept a list of options, which for all I know is easier for you to
implement.

[kovidgoyal]

The hard (well laborious) part is not getting the options to the compiler, its getting the compiler to output the relevant code corresponding to those options :)

[kovidgoyal]

I implemented

from __python__ import dict_literals, overload_getitem

See https://github.com/kovidgoyal/rapydscript-ng#containers-listssetsdicts for details. There is no more special ! for dicts. Instead you use scoped flags.

@BruceSherwood you can just feed the above statement to the RS compiler once and then all future code that you compile with it will use python dicts/getitem. You can try it in the rapydscript web repl.

[BruceSherwood]

Thanks very good, Kovid. Thanks.

[kovidgoyal]

I'm closing this bug report since I think I have comprehensively addressed all concerns raised here. It is up to @atsepkov whether he wants to relinquish maintainership of rapydscript or keep things as they are. In either case, I will continue to work on rs-ng. If he decides not to relinquish maintainership he is welcome to take as much code as he likes from rs-ng. If he decides to relinquish maintainership, then we can discuss how to proceed. Let me close by saying thanks to @atsepkov for all his great work on the original RapydScript, which made my work on rs-ng possible.

[webmaven]

Not to actually reopen this issue, but given that the two codebases are diverging, perhaps a name-change of rs-ng to something entirely different is in order (-ng often implies forward compatibility with a future version of the original project).