Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SLURP: a Single Liberal Unified Registry of Haskell Packages #4

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

simonmar
Copy link
Member

@simonmar simonmar commented Jan 22, 2018

NOTE: the proposal is currently "parked" pending discussions of alternatives, see this comment.

Rendered proposal

Hackage has been extraordinarily successful as a single repository through which to share Haskell packages. It has supported the emergence of variety of tools to locate Haskell packages, build them and install them (cabal-install, Stack, Nix, ...). But in recent years there has been increasing friction over,

  • Hackage’s policies, especially concerning version bounds;
  • Hackage's guarantees, especially around durability of package content and metadata;
  • Hackage's features, especially the visual presentation and package documentation.

If we do not resolve this friction, it seems likely that the Haskell library ecosystem will soon “fork”, with two separate repositories, one optimised for Cabal and one for Stack. This would be extremely counter-productive for Haskell users.

Thus motivated, over the last few months we have talked a lot to colleagues, including ones in the Hackage and Stack communities. We have emerged with SLURP, a proposal that could go a long way towards supporting the upsides of a diverse ecosystem, without the sad downsides of forking into mutually-exclusive sub-communities.

Here is the SLURP proposal. We invite the Haskell community to debate it.

SLURP is meant to enable both Hackage and Stackage (and perhaps more services in the future) to in the future make choices autonomously without hurting other package services. But it will only work if the implementors of both Hackage and Stackage are willing to participate. We respect their autonomy in this matter, but we urge them to give this proposal serious consideration in the best interests of the community and Haskell's success. We have carefully designed SLURP to be as minimal and non-invasive as possible, so that it can be adopted without much trouble. Of course, we are open to debate about the specific details.

We do have an offer from someone willing to implement SLURP.

We also strongly urge members of the community to express clear views about the importance --- or otherwise --- of adopting something like SLURP. You are, after all, the community that GHC, Hackage, Stackage, Cabal, etc are designed to serve, so your views about what best meets your needs are critically important.

Mathieu Boespflug (@mboes)
Manuel Chakravarty (@mchakravarty)
Simon Marlow (@simonmar)
Simon Peyton Jones (@simonpj)
Alan Zimmerman (@alanz)

@hvr
Copy link
Member

hvr commented Jan 22, 2018

To be honest, I'm rather sceptical about this proposal being effective at achieving the stated goals. As such, I'm not convinced enough of its merits to support this.

@silky
Copy link

silky commented Jan 22, 2018

perhaps we could somehow trial this thing by running it in reverse: SLURP is created and exists soley as a mirror of, say, hackage and stackage. it somehow watches them and inputs new packages immediately; then we can see if anyone actually adopts it and finds it useful?

@phadej
Copy link

phadej commented Jan 22, 2018

To correct a terrible mistake:

In Hackage, for example, the presence of revisions may mean it will not.

That's not true. The tarballs aren't ever mutated on Hackage. To apply a revision client needs to consider Hackage's index (01-index.tar).

EDIT Check https://github.com/haskell-infra/hackage-trustees/blob/master/revisions-information.md for more information about revisions.


I don't see technical reason not to make SLURP tarballs immutable. What's the additional metadata and how it's fetched is up to particular repository (Hackage revisions, Stackage snapshots, ...)


I have to reread the proposal to get proper understanding of it to comment more.

@Profpatsch
Copy link

Profpatsch commented Jan 22, 2018

You can think of SLURP as an authoritative URL-shortening service for packages.

No, please.
This implies moving from a single point of failure with high pressure for availability (hackage) to multiple points of failure with availability discovery by chance.

If you say “federalized and decentralized”, this is exactly how to not build a federalized/decentralized system.

Semi-related opinion: Uniformity of data is actually a good thing.

@tonymorris
Copy link

-1

It is unfortunate what is happening. This will not solve it. It will make it worse.

@nc6
Copy link

nc6 commented Jan 22, 2018

After some thought, I feel as if this proposal at once goes too far and not far enough.

The key element to this proposal is to retain a single, authoritative mapping of package names (+versions) to sources. But this is actually a pretty strong guarantee which is already somewhat violated. For example, nix does patch some (few) Haskell packages in order to be compatible with its scheme. By keeping the single authoritative source, we don't work around issues such as that arising with cassava recently, or other issues where some changes in a package itself might cause issues with a particular distribution.

At the same time, allowing hackage to become just "one-of-many" would make it difficult to satisfy some of the goals set out here. If some critical package (either current or future) decides to move off of Hackage as their authoritative source, say, then the part of the ecosystem depending on that package becomes unavailable to cabal unless it moves over to using slurp directly, which would be a much bigger change, or unless Hackage ends up hosting a version of said package anyway, which would violate the slurp guarantees.

On the other hand, the more I think about it, the more a "soft fork" seems like not so much of a bad thing. If stackage moved to a model where, by default, packages come from Hackage but could be overridden on a case-by-case basis, then it's not clear that too much harm or fragmentation would result. Obviously the nightmare case is that there's a divergence in some core package, but this case seems pretty unlikely, since it benefits nobody. On the other hand, say, if two parsers happen to differ in such a way as to cause differing results on a package, temporarily forking that package until the parsers can be updated seems to be no bad thing. Again, this mirrors much of what already happens in nix or in any linux distribution, where packages are taken from upstream by default, but can be overridden if needed.

Obviously, one could still have the case where critical packages moved to being hosted elsewhere and thus became unavailable to cabal, but this would seem less likely since Hackage would still be considered the "central" package source, if no longer the completely authoritative one.

@clintonmead
Copy link

You make the following assertion:

If we do not resolve this friction, it seems likely that the Haskell library ecosystem will soon “fork”, with two separate repositories, one optimised for Cabal and one for Stack.

Could you detail why you think this is the case? As far as I understand, Stackage is just a subset of hackage. Stackage requires maintainers to get their package working with a particular snapshot, but hackage on the other hand just leaves it down to maintainers to try their best and if packages don't work together you can either try --allow-newer or submit a patch.

I think the current arrangement works well. Stackage for more mature packages you're prepared to put the extra work into keeping up to date with the ecosystem and Hackage for more experimental packages. You can always depend on a Hackage package in a Stackage package by just explicitly mentioning the Hackage package version.

Have I missed something about how this works? I don't see Hackage and Stackage as competitors, but completely complementary parts for slightly different levels of package maturity.

@mikeplus64
Copy link

I might be missing the point here. But...

One seriously great thing about both Stackage and Hackage is the documentation both provide. Packages basically all get documentation in the same familiar format, link into eachother happily via types, module links, functions etc., we even get lovely links to source files, and even lovelier links between source files! This is something I don't want to give up.

How will this work under SLURP, where I assume docs are neither guaranteed (if the canonical home for a package is just e.g. some guy's github repo) nor guaranteed to have a similar structure? It seems inevitable, if SLURP is the canonical home for packages, that it (or a hypothetical SLURPage) will be the canonical "giver of links" to the homepages of packages, and that it may as well generate API docs while it's at it.

@tomjaguarpaw
Copy link
Member

FYI, Mathieu Boespflug's handle is @mboes.

@haskell haskell deleted a comment from boes Jan 22, 2018
@gwils
Copy link

gwils commented Jan 22, 2018

This seems like an unnecessary complication in the ecosystem. I agree with the above comment that this multiplies our single point of failure.

Furthermore I don't see how SLURP will achieve its goals. It seems to me that implementing this proposal will cause the fork it aims to prevent.

@nomeata
Copy link

nomeata commented Jan 22, 2018

Besides the indiviual review points, I have an overall question:

I am not sure how it solves the problem. Both hackage and stackage will offer functionality beyond what is defined by SLURP, which is required by the corresponding tools to work (Stackage: the package sets; Hackage: the version bounds). But this means that stack resp. cabal-install will not actually be able to use SLURP, they will only be able to use the packages provided by their corresponding hosting-service. But if I want to provide package foo for users of both tools, I now have to upload it to both services. But SLURP allows the name foo to only point to one of them. So I will have to upload foo-stackage and foo-hackage? Clearly that is not the desired state of affairs … where did I make a mistake?

particular package tarball for (say) ``pkg-3.5.3`` always yields the
same result each time. (In Hackage, for example, the presence of
revisions may mean it will not.) Immutability, or lack thereof, is an
example of hosting-service specific policy that does not concern SLURP.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have serious doubts about this aspect.

Downstream packagers like Debian would have serious trouble if downloading pkg-3.5.3.tar.gz would yield different results later on. In general, it would cause huge amounts of confusion.

Also note that (AFAIK) hackage does not actually ship mutable tarballs. It ships updated .cabal files out-of-band (in the 00-index.tar), and cabal-install upon unpacking replaces the .cabal file.

Is this paragraph motivated by a real use case where hackage/stackage/whatever plan to actually ship tarballs that vary over time? Or is it in to cater for the Hackage revision feature, written with maybe a slightly wrong idea about how it actually works?

Could this paragraph be replaced by something like “We impose that getting a particular package tarball always yields the same results. Hosting services that plan to modify the package in some way should deliver this modification out-of-band in a way that is specifically understood by the corresponding client.”?

available from Hackage.

6.4 No need for extra user accounts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Copy link

@nomeata nomeata Jan 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about the work that went into making the Haskell package update and delivery mechanism not embarrassingly insecure? Both Stackage (with signed, S3-hosted indexes) and hackage (with hackage security) have given these things some serious thought. Are we going to lose that again, sacrificed on the altar of the least common denominator?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

registry.
- The *pkgname* must not be unlawful, threatening, abusive, libelous,
defamatory, obscene, offensive, indecent, pornographic, profane, or
otherwise objectionable.
Copy link

@nomeata nomeata Jan 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need be codified? Will it prohibit

acme-cutegirl: library and program: Maybe gives you a cute girl
acme-missiles: library: Cause serious international side effects.

which is currently on hackage?

I recommend to simply avoid codifying that, state that “trusteeds have the power to remove entries that abuse the system” and leave it to human common sense.

genuine Haskell packages. This means that the tarballs must include
package metadata and a build file that conforms to Cabal-the-spec.
This metadata must (1) identify the packages as *pkgname* and (2)
include valid license and copyright information.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we require the URL to be a https: url? (I.e. not http, not gopher, not ftp, not git)

@tfausak
Copy link

tfausak commented Jan 22, 2018

👍 I am strongly in favor of this proposal. I think it will allow all parties (Stackage, Hackage, Nix, and others) to continue improving independently without stepping on each other's toes or compromising the uniqueness of Haskell package names.

@HeinrichApfelmus
Copy link

I like this proposal: I think that a common namespace for Haskell packages is a thing that we definitely want in a shared ecosystem. This is a minimal proposal to get it.

A.

However, I do think that @nomeata has a good point: It may happen that a particular name is specific to one hosting-service and not useable by others — if the common package metadata is not enough for some hosting-service to provide their service. I don’t think that this problem can be avoided if we want to allow experimentation in metadata.

It is still better than a fork, because in this proposal, the package name stays unique, at least. (In a fork, we could have to very different packages with the same name.)

But I concur that a way to offer multiple variants of a single name for different hosting providers would be nice.

A2.

As I understand, the proposal does not actually allow experimentation in metadata. In this case, I think that we need to make the metadata specification a part of the proposal. The specification linked in the proposal is not good enough, because it is out of date. De facto, Cabal-the-spec is implicit in Cabal-the-library. The main problem with this is that it leaves open the question of “Who is allowed to change the common Spec in the future?”. Right now, the answer is “The people with push privileges to Cabal-the-library”. If the proposal wants to specify a common package metadata, then it also needs to address the question of ownership of the metadata format, thereby actually defining what common means.

B.

On point 6.4: “Name squatting”: I do not think that allowing PUT without authentication is a good way to solve this. Fortunately, this not about any core problem that the proposal wants to address, but it is about automated protection from SPAM. There are various tools for that, like user authentication, or CAPTCHAs. A very simple solution would be to allow PUT only from trusted hosting-services like Stackage and Hackage.

@tomjaguarpaw
Copy link
Member

There seems to be a contradiction in these two statements:

bytestring-0.10.8.2 is bytestring-0.10.8.2 no matter where you downloaded it from today (which might be a different place from where you'll download it tomorrow).

we do not impose that getting a particular package tarball for (say) pkg-3.5.3 always yields the same result each time

@mboes
Copy link

mboes commented Jan 22, 2018

@nomeata thanks for the review. Will respond here to the top-level comment.

I am not sure how it solves the problem. Both hackage and stackage will offer functionality beyond what is defined by SLURP, which is required by the corresponding tools to work (Stackage: the package sets; Hackage: the version bounds).

Hackage is currently a package store, plus a bunch of other things (documentation server, downloads counter, ratings tracker, etc). Mind you, there are plenty of other package stores too, but using them exclusively is currently problematic. For example, GitHub is a package store: it's possible to list versions and to download tarballs of releases associated with repositories. All other public instances of hackage-server are also package stores. And I have heard of several plans for more in the future.

Stackage is not exactly a package store: one way to see it is as a mirror of a controlled subset of Hackage. Packages aren't (currently) uploaded directly to Stackage.

Where does SLURP fit in? Well, if people are going to introduce their own public package stores (as the Simon's are saying will likely happen), then we need to make using these other package stores not problematic. I won't comment on these future package stores, so let's go back to the GitHub-as-package-store example: what's the problem with only hosting package tarballs directly on GitHub? It's that if tools start pulling from this package store directly, in addition to Hackage, then there is no point of coordination for choosing who owns what package name. That is to say, user A could upload package foobar to GitHub, even have it be included in future Stackage snapshots say, while user B might concurrently upload foobar to haskell.hackage.org. Without SLURP, user A and user B had no way to coordinate about package names and we end up with both user A and user B vying for the foobar name.

You might say, "yes but today whoever uploaded first to Hackage is whoever owns the name foobar". That's true. But if tomorrow a significant part of the community no longer does that (and this is not a theoretical concern), then we have a problem on our hands and introducing something like SLURP makes that scenario less problematic.

How would services like Stackage deal with multiple package stores the future? Just like it does now with Hackage: users send a pull request saying which new package to include in Stackage and Stackage mirrors (an immutable copy of) that, just like it does now). Hackage could do the same as well, by including in the 00-index.tar.gz file the .cabal metadata for all packages registered in SLURP, irrespective of their location. Or cabal-install could be updated. In sum, no need for uploading foo-hackage here and foo-stackage there.

SLURP focuses on one problem and one problem only: coordinating package name ownership and making sure the owner of package name is advertising available versions in a standard way.

@tomjaguarpaw
Copy link
Member

tomjaguarpaw commented Jan 22, 2018

SLURP focuses on one problem and one problem only: coordinating package name ownership and making sure the owner of package name is advertising available versions in a standard way.

Aha! Then I think you should make this point upfront in big bold letters (metaphorically) because currently the proposal seems to be a lot more wooly than that.

@tomjaguarpaw
Copy link
Member

If we do nothing it seems likely that the Stack community will create a separate Haskell package repository.

It would be nice if this was supported with some justification. At the moment SLURP looks like a solution in anticipation of a problem which may never occur.

@nomeata
Copy link

nomeata commented Jan 22, 2018

SLURP focuses on one problem and one problem only: coordinating package name ownership and making sure the owner of package name is advertising available versions in a standard way.

Is a possible name collision really the biggest problem that we are facing in light of a possible ecosystem split? (and are we not rather making such splits more likely?)

Here is a hypothesis: Every tool that will start pulling from more sources than just hackage (or just stackage) will just use full URLs to disambiguate the packages. Or come up with some other solution to distinguish https://hackage.haskell.org/package/foobar from https://github.com/alternative/foobar. Both stack.yaml files and caba.project files already support that – with the intention of supporting development versions, the infrastructure seems to be there already.

I am not sure if we are solving the right problem, and if there is a problem in the first place.

@mboes
Copy link

mboes commented Jan 22, 2018

@HeinrichApfelmus

The specification linked in the proposal is not good enough, because it is out of date. De facto, Cabal-the-spec is implicit in Cabal-the-library. The main problem with this is that it leaves open the question of “Who is allowed to change the common Spec in the future?”. Right now, the answer is “The people with push privileges to Cabal-the-library”.

Yup, and this is an important problem. Thanks for bringing it up. This proposal, however, takes things one step at a time. For now, Cabal-the-spec = Cabal-the-library as you say, for lack of anything better. If some folks could get together to propose a new working group or somesuch to keep Cabal-the-spec up-to-date, then that would be fantastic. It's just that changing what Cabal-the-spec means is out of scope for this particular proposal. In this proposal we just assume the status quo. If the meaning of Cabal-the-spec somehow changes in the future, then SLURP trustees will enforce that instead of the current specification-by-implementation.

@mpickering
Copy link

If we do nothing it seems likely that the Stack community will create a separate Haskell package repository.

Why would the stack community want to create a separate Haskell package repository?

I think this proposal is lacking motivation whilst the elephant in the room is not addressed.

@gbaz
Copy link
Collaborator

gbaz commented Jan 22, 2018

No tool knows how to use SLURP, nor will immediately know how to in the future. This means that tools will need to change to use SLURP regardless. If tools need to change, there is an easier way to change that involves no creation of any new upstream system, management of its hosting and backup, provisioning of security systems at all:

Simply allow namespacing prefixes to refer to different package registries, and allow cabal files (and stack.yaml files, etc) to declare which prefixes associate to which registries.

Mock syntax:

repo-aliases: foo-repo=https://foo.com/repo, bar-repo=https://bar.com/repo

build-depends: foo-repo+some-package, bar-repo+some-package

This proposal seems to introduce many confusing problems and solve no important ones.

@nc6
Copy link

nc6 commented Jan 22, 2018

So, on reflection there's an aspect of this that I missed. I still have the above concerns, but this might overrule them.

Thinking about name authority, there are two separate issues:

  • Deliberate overriding of the package under a given name to deal with distribution issues. This should be allowed and supported.
  • Accidental name clash following a hard fork (e.g. multiple actual repositories as storage locations, rather than persistent overlays).

I was largely thinking about the former situation, which it reads a bit like this proposal aims to prevent. But perhaps it's the latter situation. So if there is likely to be a hard fork, and if we want to stick with un-namespaced packages (at least for the time being) I can see the merit in this proposal.

@gbaz
Copy link
Collaborator

gbaz commented Jan 22, 2018

On a personal note, the process of arriving to this state has been frustrating. I put in enormous amounts of time on a prior iteration of this proposal that was utterly different than this one, and we arrived at a situation where there was no objection (among the 25-ish people cc-d on two different threads) to something that proposed to avoid a fork by improving and modifying the candidate mechanism to serve an "uncurated" purpose (through a new layer) and could lever existing tech at every step rather than introducing a new component.

The last message in that discussion, roughly 20 days ago, seemed positive on the direction that had gone.

Now, without, as best as I can tell, most of any of the people in those discussions having been followed up with, we have an entirely different proposal, sharing the same acronym, whose purpose is not to avoid a fork, but to facilitate one. In this proposal is no discussion or mention of the fact that a prior proposal, that seemed just fine and more straightforward had ever been considered, nor what people consider may have been wrong with it.

This is not how to coordinate a discussion, and it leaves a very bad taste in my mouth.

@chreekat
Copy link

Can a specific example of needing to put something on Stackage that cannot go on Hackage be added? That is what I understand the motivation of this proposal to be, if I have read between the lines correctly. It would strengthen the proposal to be more explicit.

@HeinrichApfelmus
Copy link

The main problem with this is that it leaves open the question of “Who is allowed to change the common Spec in the future?”. Right now, the answer is “The people with push privileges to Cabal-the-library”.

This proposal, however, takes things one step at a time. For now, Cabal-the-spec = Cabal-the-library as you say, for lack of anything better. [...] If the meaning of Cabal-the-spec somehow changes in the future, then SLURP trustees will enforce that instead of the current specification-by-implementation.

Sure, I can see that using Cabal-the-spec = Cabal-the-library for the time being is an acceptable solution. But it still needs to be recorded in the proposal itself! At a minimum, the sentence

"If the meaning of Cabal-the-spec somehow changes in the future, then SLURP trustees will enforce that instead of the current specification-by-implementation."

needs to be included in the proposal. My point is that it's not so much a matter of "What is the spec?" but "Who decides what the spec is?". The latter question has to be answered as part of the proposal text, otherwise the spec is effectively undefined / subject to control outside of this proposal.

(To facilitate discussion about this, it may make sense to remove the name "Cabal" from one of the concepts. For instance, the format could be called "Common Haskell package metadata format" instead of Chabal-the-spec, or the tool for installing packages from Hackage could be called hackage install.)

@mboes
Copy link

mboes commented Jan 22, 2018

@nomeata

Or come up with some other solution to distinguish https://hackage.haskell.org/package/foobar from https://github.com/alternative/foobar. Both stack.yaml files and caba.project files already support that – with the intention of supporting development versions, the infrastructure seems to be there already.

Yes, at the cost of not being able to combine both in the same project (that can become increasingly difficult when you start having 200+ transitive dependencies). Further, requiring ancillary data not part of the commonly agreed core to disambiguate. cabal-install uses cabal.project files, and Stack uses stack.yaml files, but neither interoperates with the other. Further still, if a package changes location (say from Hackage to GitHub or vice versa), then all this metadata, whether it's in the de-facto-standard .cabal files or in the non-standard cabal.project file (or stack.yaml), will need to be updated everywhere. That's why a key principle stated in the proposal is independence from provenance:

Package names should not depend on their provenance: bytestring-0.10.8.2 is bytestring-0.10.8.2 no matter where you downloaded it from today (which might be a different place from where you'll download it tomorrow).

That's why there's a difference between nicely supporting temporary overrides for in-development packages, and nicely supporting packages hosted in multiple places (not just Hackage) quasi-permanently.

Disambiguating names might seem like a small service to offer. It can nonetheless be of crucial importance. SLURP is a proposed means to do so in a way that requires little to no changes to existing tooling (and small changes to existing platforms). And the implementation effort is commensurately very small (including while preserving the security properties you seek - topic for another comment).

@23Skidoo
Copy link
Member

23Skidoo commented Jan 22, 2018

If the main reason for SLURP is that Stackage people want to fork Hackage, it's a bit strange that they are not among the proposal's authors.

@gbaz
Copy link
Collaborator

gbaz commented Jan 24, 2018

Let me add one further thought on openness. I realized earlier today that we have put in place all sorts of proposal and governance mechanisms for hackage changes, cabal changes, ghc changes, etc. People rely on these things, and so want to A) have a voice and B) be forewarned about what is in the pipeline.

That people could be so concerned about potential changes to stackage policies indicates to me that stack and stackage should also consider having some sort of open governance in this regard. E.g. if stackage is considering implementing a patch overlay, that also should have a thread in a public forum, with the opportunity for input. (Could the ecosystem-proposals repo serve this purpose?) Perhaps there would not be much input, but it would certainly cut down on speculation, and the additional considerations raised might be invaluable.

@gbaz
Copy link
Collaborator

gbaz commented Jan 24, 2018

@ElvishJerricco that won't work exactly, because overlays give you a bigger thing than you started with (it is a biased union, basically). Anyway, this isn't the thread for that -- there will be a place with room for discussion soon enough :-)

@mchakravarty
Copy link

@tonymorris wrote,

I don't think pursuing this angle is particularly constructive. But I thought you should know that there are several of us simply waiting, idle, hesitantly considering a comment or two (mostly not commenting at all), for this tactic to be recognised. Rightly or wrongly, it is occurring. We've seen it before.

Do not succumb to threats.

Ok, fair enough.

@fumieval
Copy link

  • Hackage’s policies, especially concerning version bounds;
  • Hackage's guarantees, especially around durability of package content and metadata;
  • Hackage's features, especially the visual presentation and package documentation.

I'm not convinced of the motivation at all. I must say I don't see any "friction" increasing over any of these.
I'd like to see real evidence and thoughts about them, if they exist.

If we do not resolve this friction, it seems likely that the Haskell library ecosystem will soon “fork”

Why the hell would anyone fork? It looks to me like the choice of words is causing a lot more friction.

@23Skidoo
Copy link
Member

Nice to see the discussion finally taking a constructive turn. Agreed that openness and transparency is the best way to move forward.

@simonpj
Copy link

simonpj commented Jan 24, 2018

Friends

Some of you may remember my Sept 2016 email about holding respectful conversations in the Haskell community. Once again, I urge (beg, actually) everyone to speak with respect about all our colleagues. It does great harm to attribute motives or unstated intentions to others, to demonise them, or to call them names. I have huge respect and admiration both for those who created Stack, and for those who created Hackage, and it is painful to see either group vilified. It is (I believe -- I accept that you may not) unjust to them and, regardless, it is corrosive of our common life.

The initiative for this proposal lies with the authors of the proposal and us alone. It's been clear for some time (at least two years) that that there is serious friction between the Stack and Hackage communities. As I said earlier in this thread, I believe that this friction has arisen because of differing technical perspectives, priorities, and working practices. SLURP is an attempt to reduce that friction by allowing the two ecosystems, and others, to cooperate (to some degree) and innovate without conflicting with each other. That doesn't solve the problem, but it might reduce friction substantially.

However, on this thread at least, there seems to be little support for the idea. And without such support, it's hard to see the Hackage folk, or the Stack folk, using SLURP even it existed. Fair enough. That’s what a public debate is for: to test, refine, and sometimes discard ideas.

Moreover, Michael's very helpful post (thank you Michael) and its follow-ups from Gershom and others, herald the beginning of a more constructive conversation. Let's allow time for that conversation to develop, hopefully into concrete actions.

Meanwhile, let’s "park" SLURP. We can always dust it off later, if that seems useful as things evolve.

With genuine thanks to those who have contributed.

Simon

@maxnordlund
Copy link

@simonpj would you mind adding a TL;DR edit to the description so people who stumble upon this thread (like me) finds your post easily?

Thanks for the kind words and the level headed conclusion.

@simonpj
Copy link

simonpj commented Jan 24, 2018

would you mind adding a TL;DR edit to the description

Good idea. (Revealing Github ignorance.) By "description" do you mean the rendered proposal? Or something else?

@nomeata
Copy link

nomeata commented Jan 24, 2018

Ideally both the rendered proposal, and the first comment in this particular thread. Probably @simonmar is the one who has the right to edit both.

@simonmar
Copy link
Member Author

@simonpj @nomeata done.

@maxnordlund
Copy link

Thank you

@mchakravarty
Copy link

Dear all, thank you very much for enduring with us this long and somewhat heated discussion. I want to especially thank @snoyberg for dropping his initial reservations about entering into the discussion. A direct public and constructive conversation between Stackage and Hackage folks is clearly much better than our own one-step removed attempts to broker a peaceful, but separate existence.

Much has been made out of the exact meaning of the word "fork". For those who read the actual proposal, I hope, it was clear that this was always about (1) where to get authoritative package names and (2) where packages are stored. The idea behind SLURP is to keep one central (unforked) authority for authoritative package names —a package name server if you will— but allow multiple (forked) package storage servers. (This is in contrast to the earlier proposal that @gbaz referenced, where package storage was also kept in one place.) It is the absolute minimum that we believe ought to be centralised. Our (the author's) motivation behind the proposal was that we really want to avoid a fractured (forked) package name space, where one package name (e..g, "lens") means two completely different things depending on which package server you go looking for it.

@gbaz and @snoyberg, may I ask what the next steps are? Will Hackage be able to accomodate the three issues that are raised in #4 (comment)? Too much of this discussion has happened behind closed doors and through intermediaries, as Michael pointed out. I like to politely request that the next steps be discussed in the open. What is the right forum for this?

Moreover, referring to Michael's blog post https://www.snoyman.com/blog/2018/01/slurp, specifically the text under the heading "Avoid uploading to Hackage", Stackage pulling packages directly from Git (side-stepping Hackage) exactly raises the naming issue (aka fork) that SLURP aims to address. Hence, unless Hackage and Stackage folks can resolve the situation such that independent sourcing from GutHub is not necessary, I like to ask everybody to come back to discussing SLURP. (I very much hope that this will not be necessary!)

Thanks!

@gbaz
Copy link
Collaborator

gbaz commented Jan 25, 2018

@mchakravarty I think this thread should remain parked as SPJ proposed and as Simon Marlow implemented. There's plenty else to be worked on in the meantime, in a hopefully less fraught setting.

I listed the two "action items" I have coming from this thread above.

To repeat:

I will try to draft a straightforward "uncurated" proposal with phasing such that the first portion can be implemented without tying it to collection or candidate projects. Hopefully (no promises) by end of week.

I think some guidelines on professional conduct expectations of trustees would be good as well. I'll ask if someone wants to draft them.

I also intend to write a further proposal on generalized namespacing without a central server, but this has less immediate priority.

Further, I would also urge the maintainers of stackage to consider an "action item" of their own as I suggested above:

...stackage should also consider having some sort of open governance in this regard. E.g. if stackage is considering implementing a patch overlay, that also should have a thread in a public forum, with the opportunity for input. (Could the ecosystem-proposals repo serve this purpose?) Perhaps there would not be much input, but it would certainly cut down on speculation, and the additional considerations raised might be invaluable.

In particular there should be a designated group of stackage trustees or the like who are listed and accountable for stackage policy, and there should be some open discussion forum in which prospective changes with ecosystem-wide impact, such as patch overlays or pulling-directly from git (roughly, things with impact to the stackage maintainer's agreement: https://github.com/fpco/stackage/blob/master/MAINTAINERS.md) are to be discussed broadly with those in the Haskell ecosystem prior to changes being made.

@maxnordlund
Copy link

One use for pulling directly from git that I didn't see mentioned is to distribute private packages. Hackage is purely for open source, and it's kind of an hassle to run your own server just to enable code sharing. But allowing cabal/stack to use any sort of git/darcs repository would make it trivial, or at least easier, to share code inside a company.

When I did that using bower (for our frontend), you needed to specify the entire URL to the repo (git+ssh://github.com/<foo>/<bar>.git). Which neatly sidesteps the dual lenses problem mentioned by @mchakravarty.

Anyway, that's just my two cents. I hope this leads to something good in the end.

@simonpj
Copy link

simonpj commented Jan 25, 2018

I suggest that we park the proposal, but I think it would be helpful to have some way for us all to continue to talk to each other about next steps, each of which will doubtless lead to a more detailed per-item discussion. That's why I haven't closed this thread yet. I can see a logic in closing it (since it's ostensibly about SLURP), but if so I wonder if we might want to open a new "next-steps" thread?

On the specifics, Gershom has helpfully offered to write two proposals about (a) an uncurated layer to Hackage and (b) generalised namespacing. Gershom: would you be happy to share them on this GItHub repo in due course? It provides a good way to share the conversation with a broader audience.

Doubtless much can and will take place in private discussion. But I'm strongly in favour of sharing the thinking more broadly.

@angerman
Copy link

angerman commented Jan 25, 2018

Doubtless much can and will take place in private discussion. But I'm strongly in favour of sharing the thinking more broadly.

This has been one of the main takeaways from this thread for me. We should try to minimize backchannelling and have some central public forum where these kinds of issues can be debated. Is this (the ecosystem proposals) supposed to be such a public forum?

@simonpj
Copy link

simonpj commented Jan 25, 2018

Is this (the ecosystem proposals) supposed to be such a public forum?

I think it is, yes. That's why Alan set it up in the first place. But, like you, I think the main thing is to have the conversation; I'm not fussy about where. But this is as good as any and better than most.

@mboes
Copy link

mboes commented Jan 25, 2018

@maxnordlund that's a nice feature to have. To a large extent, that's already possible today with cabal.project and stack.yaml files, and even with plain Git submodules. Those three mechanisms are all well suited, I think, when it comes to temporary (or private) packages. It's when packages are intended to be permanent and public, that having everything under a single namespace (be it flat or hierarchical) becomes useful to avoid name conflicts. This ground was covered previously in this thread (see early comments by @nomeata).

@simonpj since this PR is putting forth a specific proposal, which has been rejected, on second thought I suggest closing the PR and opening one or more issue tickets in this project's issue tracker. The closing words by @snoyberg, @simonpj, @mchakravarty and @gbaz bring this one to a nice conclusion. @snoyberg, since you raised a few specific issues in your blog post, I would suggest submitting those. I support @gbaz's proposal about Stackage trustees and policies. Sounds like a good ticket for the fpco/stackage repository.

@gbaz
Copy link
Collaborator

gbaz commented Jan 25, 2018

On the specifics, Gershom has helpfully offered to write two proposals about (a) an uncurated layer to Hackage and (b) generalised namespacing. Gershom: would you be happy to share them on this GItHub repo in due course? It provides a good way to share the conversation with a broader audience.

Yes, the uncurated proposal is intended for this repo. It never occurred to me that it would go anywhere else. The namespacing proposal is intended for the ghc-proposals repo, since it is at base about changes to ghc and ghc tooling directly.

@snoyberg
Copy link
Collaborator

The namespacing proposal is intended for the ghc-proposals repo, since it is at base about changes to ghc and ghc tooling directly.

I'm glad you mentioned that here. The other namespacing proposal I've seen batted around (very unofficially, no one's done any kind of write up AFAIK) is allowing Github-style namespacing of a package, and letting the build tool override which package is selected. As an example: a cabal file would still simply say build-depends: bytestring, but a cabal.project or stack.yaml file would be able to say either extra-deps: [bytestring-1.2.3] for the official bytestring of today, or snoyberg/bytestring-1.2.3 to get my special version that includes some patch (such as an experimental feature). It may be worth discussing relative merits outside of the GHC tracker first.

@gbaz
Copy link
Collaborator

gbaz commented Jan 25, 2018

We can accomplish that today, with overlays, already. I can't motivate why the proposal I want to make should go into the repo outside of essentially writing the proposal and putting it in the repo, at which point it will become clear. At that point, if people don't think it makes sense, we can discuss it there.

@gbaz
Copy link
Collaborator

gbaz commented Jan 27, 2018

Here is the uncurated hackage layer proposal: #6

@ghost
Copy link

ghost commented Jan 29, 2018

This reminds to much the way PureScript does it and it's a nightmare !!!

Please don't do it !!!

@gbaz
Copy link
Collaborator

gbaz commented Mar 27, 2018

I wrote upthread

This thread has tended to convince me that the assumption of a single global package namespace is not a good one. In particular -- we're haskellers, when do we make assumptions about a single global anything? I have an idea kicking around about how we might fix this at the root, in a way that is pretty painless to end-users. It would be a fair amount of engineering overall, so wouldn't land anytime soon. But I will try to turn this into something discussable soon, and that will also maybe improve things in the long-term, in terms of a more distributed and repo-overlay ridden world we're moving towards.

The result of this is the Provenance-Qualified Package Imports proposal, which people may be interested in: ghc-proposals/ghc-proposals#115

@gbaz
Copy link
Collaborator

gbaz commented May 9, 2018

As a followup, there was a request in this discussion for strengthened hackage trustee guidelines. That is now merged into the policies: haskell-infra/hackage-trustees#154 (https://github.com/haskell-infra/hackage-trustees/pull/154/files)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.