conditionally parsing code via the %if feature pragma

[StefanKarpinski]

When one changes the language, it forces packages to choose between using a new feature and only working on new versions of Julia or not using the feature but supporting new and old versions. For non-syntactic changes, you can often work around this with fairly simple conditional definitions. For syntax changes (which are blessedly rare), this doesn't work. To allow code to gracefully handle new syntax while still parsing in older versions of Julia, we're going to introduce, ironically, a new syntax feature: pragmas. The first pragma will be the feature pragma, used as follows:

%if feature conditional_modules
# write code that uses the conditional_modules feature
%else
# fallback code that doesn't use the conditional_module feature
%end

A Julia parser will have a list of feature names. If the parser recognizes a feature name, it parses the file as if the source were the code between the %if and %else or if there is no %else until the %end. If the parser doesn't recognize the feature name, it parses the file as if the source were the code between the %else and the %end or as if the source were empty if there is no %else.

In general, %if pragmas nest:

%if A
  %if B
    # A and B
  %else
    # A but not B
  %end
%else
  # neither A nor B
%end

There should probably also be a %ifelse to handle the classic ambiguity here. Other pragmas, should we introduce them are single-line.

[vtjnash]

I'm not sure this is really necessary or a good idea. It reminds me too much of the bad parts of C. We can already do conditional include. Perhaps we should just have a basic feature test check function (ala python's future class), rather than depending on VERSION.

[tkelman]

Or feature test macro...

@supports_foo begin
...
end

no, yeah, nevermind, still needs to parse if it's a macro. ignore me.

[JeffBezanson]

The problem is that this is the only way I can see to skip invalid syntax.
If you use conditional include, you might have to copy code into both files.
On Jun 28, 2014 5:17 PM, "Tony Kelman" [email protected] wrote:

Or feature test macro...

@supports_foo begin
...
end

—
Reply to this email directly or view it on GitHub
#7449 (comment).

[StefanKarpinski]

@tkelman – macros don't help if there is new syntax that was previously a parser error.

@vtjnash, how do you propose to support both new and old versions across syntax changes without forking your code entirely? I know this is ugly but maintaining compatibility is an ugly business.

[ivarne]

We can also document this feature to be removed before Julia 1.0, if we don't like the look of it.

[mlubin]

We should be making it easy for package authors to maintain compatibility with the latest stable release, so I'd say this is a reasonable approach even if it's a bit ugly.

[vtjnash]

@StefanKarpinski and Jeff just use more include files!

Honestly, I just don't think this syntax will help improve compatibility, but I do think it will significantly impact readability

[StefanKarpinski]

This is only something to be used in dire need. It's not like this is meant to be used in any kind of normal programs so I don't see how it can affect readability. You use it when you absolutely have to.

[vtjnash]

I guess I don't see how that is much of an argument. Saying that this will be a rarely used feature doesn't make me feel that that this is a must have language feature. (And if the % character is actually up for grabs, I'm sure we could find better used for it).

I would rather not have this hastily implemented for 0.3, and just not have conditional modules for a while longer (or add them in some later RC, since they are a non-breaking change)

[ViralBShah]

I have been thinking about this since yesterday, and I would rather not do this feature hastily this late into the release cycle. I realize the importance, but in balance, I think we are ok breaking some eggs here.

[StefanKarpinski]

I'm not sure how much you guys have tried to maintain installations of packages and code bases using them, especially ones that are used by people on multiple different Julia versions, but this is a crucial feature. We are not all Julia developers who get to live with a bleeding edge Julia version. I'm still maintaining a setup with a 0.3-prerelease from two months ago. I had to patch several packages to make it work. Conditional modules don't break backwards compatibility but they completely wreck forwards compatibility, which is exactly the point of this feature.

[vtjnash]

As author and maintainer of Gtk, Glob, Polynomial, WinRPM (among others) -- all of which work on 0.2 and 0.3 AFAIK -- I don't see a strong benefit to this. It makes more sense if we intend to move to rolling release. I think we should just instead try to have more frequent release milestones.

[lindahua]

I have no strong opinion of this feature.

However, I am not sure this should be 0.3 stuff. There has been a long waiting of the 0.3 release, and it seems that there remains a lot of blocking issues. We should be cautious of adding more to that list.

Things that do not require a breaking decision and that are likely to run into long debates may be deferred to 0.4 cycle.

[staticfloat]

I think we do need good, solid ways of attacking this problem as it's going to continue to be a problem, but I'm with Jameson on this one, I don't really like this, as it feels too much like the #ifdef soup you get in C code when using autoconf, debug flags, etc...

Since we're specifically talking about syntax changes such that older parsers would get confused, I think using conditional includes is a better way to control what gets passed to the parser; no new syntax needed, identical functionality is preserved, and the only downside is that authors will need to write (N + 1) files instead of just one, where N is the number of mutually exclusive julia syntax subsets. As Stefan states, this is hopefully going to be a rare occurrence, so this shouldn't happen too much.

[tkelman]

Having to split things out into separate little files for every different piece of syntax also feels a little too reminiscent of old-style-Matlab-OOP. #ifdefs are ugly and I don't think anyone likes them, but they do serve a valid purpose. Another alternative to conditionally including snippet files without requiring new syntax would be conditionally eval(parse( on a here-string, but that sounds even worse.

[ViralBShah]

I fully support the idea. I am just not comfortable trying it this late in the release process, as doing it hastily just means that we will not get it right.

[tknopp]

While I do not really have an opinion on the proposed idea yet I would like to second @vtjnash opinion that it is a good idea to have more frequent releases. I can absolutely understand that in the open source world releases are something that are "evolving" as it was hard 9 month ago to predict all the great changes that made it into master until now. Nevertheless, it might be a good idea to say that we make e.g. a release every 6 month and the features that are not in after 5 month (feature freeze) have to wait for the next release cylce.

[ViralBShah]

Originally we talked about for 3 month releases which is clearly not practical. 6 months should be OK. Let's move this discussion to a different place though.

[tknopp]

Yes sorry this is of course a little off topic.

[quinnj]

@ViralBShah, I think the idea is this needs to be in 0.3 so that packages developed in the 0.4 cycle will be able to have backwards compatibility with a single package.

I'd be willing to try this out. I say we do it now and try it out during 0.4, with maybe an asterisk attached that says this is a very experimental feature that may not survive if it proves too much a hassle. This should be pretty developer focused anyway.

[StefanKarpinski]

@ViralBShah, I think the idea is this needs to be in 0.3 so that packages developed in the 0.4 cycle will be able to have backwards compatibility with a single package.

This. The key aspect of this feature is that not much needs to be decided – the above description is a complete design. If we don't include this in 0.3 then there will be no way aside from conditional includes to support both 0.3 and 0.4 with the same code base. Maybe that's ok, but maybe not. Regarding conditional includes, can all top-level constructs be done in a conditional include? I'm not certain that they can. If they can't then it's actually not a sufficient solution to this problem.

Another significant point is that #ifdef is so common in C because it's the only conditional compilation feature in C. We have lots of conditional compilation features and once the language stabilizes, this will become almost entirely unnecessary. But while we're developing the language, this is needed.

[vtjnash]

I would like to see a practical example of how this feature would help compatibility. The whole point of adding new syntax is enabling things that couldn't be done before, not giving simple alternatives. For example, conditional modules allows structuring code with dependencies in a completely different (and this less coupled) architecture. For another example, immutable types allowed completely different optimizations which were not previously achievable without significant effort via bitstypes. The common trait though is that a %if block does nothing to minimize code duplication or maintenance for these situations.

(All constructs can be done in a conditional include, or base wouldn't be able to compile -- the parser doesn't care if a file was conditionally or directly included, although in the future, code caching might)

[StefanKarpinski]

Simple example based on @tknopp's example:

module A
    export myf
    myf() = 1
%if feature conditional_modules
    module AB when B
%else
    module AB
%end
         importall ..A
         using B
         myf(::B) = 2
         # lots of other code to support using A and B together
    end
end

Writing module AB when B is a syntax error in an older Julia version. To work around this using conditional includes, you have to have two very slightly different copies of the entire module AB block – one with the when B and the other without it – everything else is identical.

If this facility isn't added to 0.3 then any usage of new syntax in 0.4 will force the packages using the new syntax to be 0.4-only. If we actually mean it when we say that we're going to maintain support for 0.3 better than we did with 0.2 then we need this feature in 0.3.

[tknopp]

I have a little mixed feelings about this and can understand both arguments.
One real issue is that package authors use master features in released package versions. I.e. when conditional modules hit master in the 0.4 development phase, IMHO no package should be released with that feature until 0.4 julia is release. It can be in the package master version but not in released versions for 0.3 julia. And to make this more practical so that we don't have to wait too long that feature lands in packages, shorter release cycles would be handy.

[StefanKarpinski]

If packages don't use features during the release cycle, how can we know if the features work?

[JeffBezanson]

Yes, it seems like it would be very useful to have alpha releases, to identify the point where major breaking changes happened.

[vtjnash]

not quite how these things are traditionally done.

Libuv has been doing this via 0.1.1-unstable, 0.1.2-unstable, 0.1.3-stable, 0.2.1-unstable etc.

The Linux kernel has been doing this with 3.3 (unstable), 3.4 (stable), 3.5 (unstable), 3.6 (stable), etc.

So it's not without precedent to tag more than just the final release series

[pao]

The Linux kernel has been doing this with 3.3 (unstable), 3.4 (stable), 3.5 (unstable), 3.6 (stable), etc.

The kernel hasn't been developed this way since the 2.6 series (which continues to this day) started. Right now on kernel.org, there's longterm stable support for 3.2, 3.4, 3.10, 3.12, and short-term stable support for both 3.14 and 3.15. Whether a kernel is maintained as a -stable tree or not has more to do with how many major vendors used it and invest in that maintenance. Note the skip from 3.4 to 3.10, for instance.

[IainNZ]

To add to the cacophony: I too have the concerns about adding language features, but less strongly than @vtjnash. I think they'd be OK because I don't think they'd be used too much, and where they are used it'd be clear why.
Having said that, I like the idea of more frequent snapshotting even more, especially because it formalizes something people are effectively already doing in practice. Something 3 months after 0.3 release with big stuff merged in, like a 0.4.0-alpha, which I recall @JeffBezanson referring to as his preference, sounds really nice.

[StefanKarpinski]

Snapshotting more frequently makes the original problem we're trying to address much worse, not better because it means there are more versions of Julia that people might be using at any one time and those people will want to be able to use packages that are compatible with their given Julia version.

[lobingera]

To to confuse a little bit more:
a) #ifdef is actually not a C language feature, it's a feature of the process how compilation is/should be done. To have something like this in julia, a mandatory/optional preprocessor should be defined.
b) %if as a language feature/parser feature is asking for trouble. Think about a) OR
c) switch on/off options for the runtime (this looks scary, but is providing the cleaner language).

For this release/snapshot thing two things would be inevitable: Regression/system testing with good coverage and package maintainers that take the job seriously (i have only good experiences with the second one in the julia community, for the first?).

[StefanKarpinski]

This is too contentious to go into 0.3 so I'm pushing it back to 0.4 and marking as speculative.

[tkelman]

Is this sufficiently covered by conditionally using include_string ? Not the prettiest, but already works. The amusing thing is that if we were to add some new syntax here, existing packages would need to use @static (if the new syntax previously parsed but had a different meaning) or include_string (if it caused a parser error) to use it without breaking on older versions of Julia, which has been the gradual upgrade path for most package maintainers now.

[StefanKarpinski]

We've survived this long without this, so I'm closing.