Webstack overhaul

[quinnj]

Hey anyone who happens to still read things around here :)

I'm opening this issue to discuss some of my recent efforts in a new (still unregistered) package HTTP.jl.

The julia webstack has long been the neglected child of the julia ecosystem, hobbling along with just enough attention to still be considered useable packages, but not much else. To be fair, the webstack originally started out as a HackerSchool project that thankfully provided basic functionality, but which unfortunately has atrophied as the original authors have moved on to bigger and better things. We found an unlikely hero in @malmaud when he stepped up and fixed some glaring usability issues with Requests.jl, as well as creating the MbedTLS.jl package to fill the glaring TLS need, but otherwise, the webstack as a whole has received very little attention over the last 3-4 years.

I have some personal and professional interests around building web services in Julia, but which require a solid stack. As I researched and dug into our webstack code, I originally was just aiming to make some improvements to HttpServer, but the more I looked around, the more I realized a complete overhaul was needed. In particular, most of the webstack code used outdated idioms from 3-4 years ago and was failing to utilize some of the great new languages features that have been released recently. There were also glaring issues of rampant code duplication due to the micro-separation of functionality between HttpCommon, HttpParser, URIParser, Requests, HttpServer; as well as a broad lack of testing in most areas.

The approach I've taken to the webstack overhaul has involved:

• Research: in particular, reading through the actual HTTP 1.1 spec to derive necessary functionality and generate tests, as well as other language http implementations. I became particularly interested in the implementation in go, due to the friendly licensing, modern-ness (i.e. implemented in recent years), and general web focus of the language
• Webstack consolidation: most other languages/libraries don't separate webstack functionality as granular as it is in julia; the most separation you generally see is between client vs. server. While the intent is right, there just aren't that many uses of having HttpCommon or HttpParser exist as stand-alone packages. In particular, this has led to a lot of code duplication in the webstack, as well as quite a few cases of "code squatting", i.e. code living in a repo that actually extends/is relevant for a different package entirely, not to mention the problem users have in figuring out where to request functionality or submit issues. In general, I think it's made the webstack less maintainable and harder to develop on/with having everything separated out. My first efforts in overhauling were to combine the existing webstack packages into a single git repo, making sure to preserve the git histories of each.
• Design from the top down: I tried to take what most high-level client/server implementations look like in other libraries/languages and come up with a Julia-version that makes the most sense. Starting at the highest-levels and working down helps drive the right abstractions and fuels testing.
• Overall webstack quality: one of my main goals in this endeavor is to make the webstack code "Base worthy"; meaning it's well-tested, well-documented, and as many warts as possible are excised, so that, as a package, it could be ready to be included in Base. Obviously there are a lot of other design decisions for including something like this in Base (or rather, the Julia standard library), and that's not ultimately my goal, just that it get to and remain in a state that it could.

State of progress:

• I know, I know, I probably should have posted this earlier, but my initial research/code exploration kind of snow-balled into the overhaul and I ended up having the time/motivation to do it, so I just started.
• The overhaul is about 2/3 of the way to a v0.1, IMO. The client-side has been totally revamped and I'm working on wrapping up a few remaining "robustness" features and filling in tests and docs. Server-side has been through a cycle or two of redesign, but there's still some core design left to do around utilizing Julia's built-in parallelism and how that affects the core server definition and event loop.
• In terms of dependencies, I'm aiming for one, julia or shared library otherwise. HTTP.jl will depend on MbedTLS.jl, but won't be hard-coded to it. MbedTLS provides an implementation of the TransportLayerSecurity.jl interface, that other TLS packages could do as well, and swapping out MbedTLS with another TLS package would be a matter of changing a single line. Currently, we're relying on the http-parser library from joyent/node, but the actual library code isn't more than 1K LOC and I have plans to do a pure reimplementation in Julia (shouldn't take more than a week or two); this will leave us with the single dependency of a TLS package.
• All of that said, I'd say the package as a whole is 2-3 months from being registered and tagged for v0.1.

I'd like to welcome anyone and everyone to provide feedback to the discussion here about what they think and ideas for moving the webstack forward. Also feel free to take HTTP.jl for a spin, it should be fairly functional already. I'd of course welcome all feedback and help designing, implementing, and testing the package from as many people as possible. Happy to open up a JuliaWeb gitter if we want a more fast-paced place to discuss if there's interest, otherwise, we can use github issues and the Julia discourse forum to discuss.

[MikeInnes]

Not too much to add on the technical side from me – this looks like a great plan and I'm excited to see this stuff really moving forward!

Looking at Go for inspiration is a great idea. I suggest also taking a look or two at Clojure's designs, interfaces and lessons learned – obviously the language and approach is virtually opposite from Go's and it may be useful to have another perspective when finding a middle ground for Julia.

I don't think you've mentioned the higher-level components like more framework-y packages, middleware, routing etc. Obviously you may not be as interested in those parts if you're not building a big website, and instead just want to build the foundation. Arguably that stuff is orthogonal anyway; eventually it'd be great to have an established, sensible default framework as a package on top of HTTP.jl and people can play with other approaches in their own packages.

[quinnj]

Great thoughts Mike. Could you point me to some files/docs on Clojure web design? I'd certainly be interested in reading.

Yeah, as far as higher-level components, I definitely want to avoid putting too much in HTTP.jl and really make it more of a foundation for the webstack. I have a prototype package that does endpoint mapping based on HTTP, but I'm not sure if it'll be something to promote over Mux.jl or anything. I plan on focusing my efforts on HTTP.jl before pushing more on it.

[IainNZ]

This all sounds great!

…

On 19 Dec 2016 4:05 pm, "Jacob Quinn" ***@***.***> wrote: Great thoughts Mike. Could you point me to some files/docs on Clojure web design? I'd certainly be interested in reading. Yeah, as far as higher-level components, I definitely want to avoid putting too much in HTTP.jl and really make it more of a foundation for the webstack. I have a prototype package that does endpoint mapping based on HTTP, but I'm not sure if it'll be something to promote over Mux.jl or anything. I plan on focusing my efforts on HTTP.jl before pushing more on it. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAqRm7c1ie5Dr5LRXG9ciuSGLIafkCTJks5rJqtagaJpZM4LQ0iP> .

[TheOnlySilverClaw]

I like what you're doing. One of my reasons to learn a new language (more than just the basics) is Java's lack of a basic HttpServer. Even the smallest ones around were several dozen megabytes large and required a bunch of documentation to get started.
If Julia manages to get a decent basic HTTP-package, I'm definitely sticking around.

[shashi]

Good plan! I would like to point out @sorpaas's https://github.com/sorpaas/HTTP2.jl which was built during GSoC. It would be SO NICE to have a reliable HTTP2 implementation.

[MikeInnes]

Things may have changed, but AFAIK the state of the art is http-kit and clj-http for the raw HTTP client. From what I can tell they're using channels now so it won't be completely far from the Go interfaces. It'd be worth taking a look at ring which starts to get a little higher-level and then there's compojure which does routing and stuff on top of that. Compojure went through a couple of interations and I think they had some interesting design discussions – unfortunately I'm missing any links but it sounds like that side will be less relevant to you anyway.

[luminarious]

Julia as a language looks lovely and a working webstack with maybe even something Sinatra-like would go a long way toward exciting the Java/PHP crowds, not to mention the Ruby and Python fans looking for a more performant alternative.

[StefanKarpinski]

@luminarious: Please see http://julialang.org/community/standards/ regarding sexualization of the term "Julia".

[luminarious]

@StefanKarpinski Amended, the metaphor got away from me.