Notes on the theme "Programming With Data"

[gafter]

Programming languages are opinionated

Programming languages are opinionated, whether we like it or not.
Any given programming language makes some things easy, and other things not as easy.
If someone is programming and has to choose between the two, the language effectively encourages them to do things the easy way.
Sometimes the opinions of a language are intentional, but sometimes the opinions are incidental and do not align with the language author's own opinions.
When that happens, that is a language design smell.
When we design and modify a language, we should aim to have its opinions align with our own.

C# grew up opinionated

C# came preloaded with a set of opinions inherited from its predecessors.

• It is object-oriented: operations on a data type should be placed on the data type on which they operate. If an operation varies in behavior from one "kind" of instance to another, it should be virtual (internal dispatch). The exceptions to this, inherited from non-object-oriented C, are the integral and enum types, for which switch permits external dispatch.
• C# is imperative: algorithms are expressed as a sequence of steps described by statements.
• C# is mutation-based: objects are mutable and the purpose of computation is to mutate data.
• Concurrency is expressed by protecting mutable data with monitors (lock) to provide mutual exclusion during mutation.

Opinions have changed over time

As we've learned more about good software engineering practices, we've reconsidered these opinions and modified the language and its libraries accordingly.
We added extension methods to permit a non-virtual operation to be declared outside the type on which it operates while being used by the client as if it were a member.
We added a number of enhancements to make more declarative (and less imperative) code possible, such as expression-bodied methods, Linq, the "elvis" ?. operator, and local functions.
We added libraries that support fork-join concurrency and asynchronous computation pipelines.
We added auto-properties to make it easier to declare APIs with immutable members.
In all of these, we moved the language's opinion not by making the existing things harder, but by adding yet easier ways of doing things that align with our opinions of what is "right".

The opinionated purpose of Programming With Data features

This cluster of features aims to move C#'s opinion:

• From internal dispatch to external dispatch
• From mutation based to patterns that work well with immutable data.
• From imperative (statement oriented) to declarative (e.g. expression oriented)

One shortcoming of C#'s opinion is that internal dispatch is frequently inappropriate for programming-in-the-large (i.e. for data types that span component boundaries).
This is particularly true when the data type cannot anticipate all of the operations that clients will want to perform on it, or when the data type is described by a set of related types.
Extension methods do not help much because they are not virtual; they give the appearance of internal dispatch without actually providing support for it.
Two programming patterns that can be used to address this are the use of visitors, and switching on kind fields.
These are widely used throughout Roslyn.
Both permit constant-time dispatch based on the logical "kind" of the data type.
But these programming patterns are clumsy and error-prone.
C# 6 adds typeswitch (pattern-matching in the switch statement) as a more direct way of expressing external dispatch without these issues.
Typeswitch moves the language's opinion toward external dispatch, but currently it has a performance penalty that is enough to discourage many uses (which we are trying to address with runtime support).

The purpose of records and pattern-matching are to reorient C#'s opinions toward what are now understood to be better programming practices:

• APIs for exchange types (types that are interfaces between program components) are better operated upon using external dispatch than internal dispatch for operations that are not core to the meaning of the data types.
• Expression-oriented code can be more declarative and more clear than statement-oriented code; we seek to make it easier to express algorithms using expression-oriented approaches.
• Many design issues are simpler when exchange types have an immutable API surface. We don't seek to change C#'s opinion that variables should by default be immutable (though we are open to that), as we do not propose using fields for these API elements.

Example of a language design pitfall

It might seem that adding support for a hypothetical readonly modifier to parameters and locals would move the language's opinion in alignment with these directions.
That is not so, for two reasons.
First, readonly int x is not shorter than int x, so the opinion has hardly moved.
val instead of var is slightly better, but it moves an opinion from encouraging mutability of locals toward being neutral about the mutability of locals.
But it adds the opinion that types for locals should be inferred, which is not intended.
It also doesn't make immutable preferred because it isn't shorter.
Second, the benefits of immutability occur at API boundaries, not within a method body.
So the hypothetical change, even if it moved the language's opinion, would not do so in accordance with the reason for wanting to move the opinion.
(I'm not saying the readonly modifier would be a bad idea, just that it doesn't fit with this theme.)

For similar reasons, expression blocks are not particularly useful to move the language's opinion in a direction consistent with these preferences.
(Which is not to say it is a bad idea.)

A design principle for Programming With Data

There is a useful design criterion for this cluster of features that can make them more intuitive for programmers:
there should be a syntactic alignment between the principal operations on an exchange type.
The principal operations are declaration, creation, decomposiion, and dispatch.
We can draw a matrix of categories of types to see how we're doing so far:

Type	Declaration	Creation	Decomposition	Dispatch (pattern)
int	int x	3	int x	int x or 3
class with mutable fields	class Point { public int X, Y; }	new Point { X = 3, Y = 4 }	p.X	Point { X: int x, Y: int y } or Point { X: 3, Y: 4 }
anonymous type	-	new { X = 3, Y = 4 }	p.X	{ X: int x, Y: int y } or { X: 3, Y: 4 }
record class	class Point(int X, int Y); (proposed)	Point(3, 4) (proposed)	(int x, int y) = p	Point(int x, int y) or Point(3, 4)
tuple	(int x, int y)	(3, 4)	(int x, int y) = p	(int x, int y) or (3, 4)
List	List<int>	new List<int> { 3, 4 }	?	?
Dictionary	Dictionary<K,V>	new Dictionary<K,V> { { K, V } }	?	?

This illustrates our attempt to align the syntax between the different forms in which a type is used.
That alignment helps build the programmer's intuition and simplifies understanding of the features.
Hopefully we can keep the syntactic forms simple and aligned to continue that tradition.

[HaloFour]

Very interesting. From what I gather the interest here is in filling in the remaining squares of that matrix, namely with syntax to decompose and dispatch list and dictionary types, or more specifically for List<T> and Dictionary<TKey, TValue>? Given collection and indexer initializers, couldn't that be abstracted over anything list-like or dictionary-like?

[gafter]

These are my notes from what I presented yesterday morning at the LDM meeting, so I typed it up for use in the notes.

The original motivation for working this out ~6 years ago was to set possible agenda items for filling in this matrix.

Because the LDM membership has almost completely turned over since the first records proposal was written, some people don't recall the motivation for its syntactic choices and are considering alternatives that (in my opinion) could undermine its motivations. I tried to make the motivations explicit so that we could explicitly discuss their merits.

[TonyValenti]

@gafter -
I really love your write up and it made me think of the below for List/Dictionary:

var C = new List<int>{1,2,3,4,5,6,7,8,9};
if(C is [1, 2, var third, _, var fourth, ..., var eighth, _] && fourth >= 0){
  //At most, only one ... may appear
}

var D = new Dictionary<string, string>(){
 ["a"] = "alpha",
 ["b"] = "beta",
 ["z"] = "zeta",
};
if(D is [["a"] == "alpha", ["b"] is { } beta, !C["missing"]]){
 //C[key] is "Contains".
}
```

[MgSam]

I disagree with the repeated implication in the writeup that "shorter is better/easier". Perl is the quintessential counterexample to that.

[HaloFour]

@MgSam

I disagree with the repeated implication in the writeup that "shorter is better/easier". Perl is the quintessential counterexample to that.

I think the argument is that shorter naturally creates a pit of effort for users of the feature. Given two ways to do something a developer might tend for the version that requires less typing. There's certainly a legit argument that shorter can mean harder to read, but I don't think that these notes are arguing about that. In this case, if you have int x and readonly int x there's little incentive for a developer to use the latter unless they really want to make the local readonly, so the language isn't expressing an opinion that readonly is better.

Might be more accurate to say that developers tend to use the features that have the "nicer" syntax.

[MgSam]

@HaloFour
But "nicer" is totally subjective. And I think what is "nice" can get really distorted in the language-design bubble.

Take pattern matching. It's an alternative to if and switch syntax that has been around for 50 years. People who visit this repo and love the cutting edge and functional languages might find it easier to write and read; but does your average developer? Early in my career I worked at a firm where the use of lambda expressions was banned because the syntax was "too hard to understand". And though I disagreed with the policy, I can certainly understand the motivations behind it.

I hate to keep bringing it up, but I'll also point to the in keyword that the team hastily added. Sure, it's shorter. But no one outside of a few people in Redmond and this forum have any clue what it actually does. In their quest for shortness, the C# team enshrined a pit of failure into the language for all time.

[Clockwork-Muse]

Many design issues are simpler when exchange types have an immutable API surface. We don't seek to change C#'s opinion that variables should by default be immutable (though we are open to that), as we do not propose using fields for these API elements.

and

Second, the benefits of immutability occur at API boundaries, not within a method body.

.... I could have sworn F# allowed something along these lines:

let test =
    // Local variables inside a function are mutable by default.
    let x = 1
    x = x + 1
    x

// Would print out 2
printfn "%d" (test)

... but the interpreters I'm trying have the x = x + 1 be a boolean expression (and the result silently discarded), so I could be thinking of some other language.

[CyrusNajmabadi]

Early in my career I worked at a firm where the use of lambda expressions was banned because the syntax was "too hard to understand".

And it would have been pretty terrible to have used that anecdote to prevent what has been such a staggeringly useful and beneficial feature for the language.

[CyrusNajmabadi]

But "nicer" is totally subjective. And I think what is "nice" can get really distorted in the language-design bubble.

That's not really the argument here. The argument is: if you're going to have multiple syntaxes, with varying levels of size and complexity, the short and simple syntax should represent the most valuable and desirable cases.

That's the idea of pit of success here. Users should not have to add more and more syntax to get hte desired behavior. The desired behavior shoudl be the default and should be given by the simplest syntax. And opting out of that should require extra effort on their part.

[theunrepentantgeek]

Early in my career I worked at a firm where the use of lambda expressions was banned because the syntax was "too hard to understand".

When generics were introduced in C# 2, members of my team were against their use because the syntax was too hard to understand.

Concerns about nested generic types were widespread enough that FxCop introduced CA1006: Do not nest generic types in member signatures

Today, however, no-one blinks twice at the syntax of generics.

All syntax is learned. All new syntax is unfamiliar and looks hard.

(To be fair, I did blink (but only once) when I recently saw Lazy<IDictionary<string, Option<Person>>>.)

[SamPruden]

It might seem that adding support for a hypothetical readonly modifier to parameters and locals would move the language's opinion in alignment with these directions.
That is not so, for two reasons.
First, readonly int x is not shorter than int x, so the opinion has hardly moved.
val instead of var is slightly better, but it moves an opinion from encouraging mutability of locals toward being neutral about the mutability of locals.

There's also the option of introducing opinions in the compiler/IDE. For example, Roslyn could suggest that anything that can be readonly, should be. Obviously the language spec is independent of the compiler, but they're closely linked. I would say that a compiler suggestion like that would take val from neutral to preferred because it alters the path of least resistance - or at least, the path of least squiggles - for the vast majority of users.

Is this considered an insufficient way to introduce opinions? Is putting things in Roslyn considered cheating, because other compilers exist?

I obviously understand that it's better to embed the preference directly into the language spec, but I'm curious how you weight compiler features as an effective tool here.