-
Notifications
You must be signed in to change notification settings - Fork 887
Pyramid 2 brainstorm
Some ideas for changes to Pyramid. Compatible ones could go into Pyramid 1.x; larger ones would have to go into Pyramid 2.x.
Note: None of these ideas are set in stone. These are just some preliminary musings for discussion.
The only parts of Paste we use are "paster create", "paster serve", and some utility objects that have been mostly shadowed in pyramid. (and paste.fileapp.) Paste has a lot of cruft going back to 2005 and Cheetah, and many of its features have been spun off into separate packages which are now more up-to-date (WebOb, WebError, PasteOb). We're the de facto maintainers of Paste as Ian Bicking withdraws from package maintenance. The question is whether to continue patching Paste or to extract what we're using into new packages. (Paste will continue to be maintained for existing frameworks.)
"paster create" can be extracted to a new utility. Revamp the application-template code. Add validators for interactive template variables. Rename "application template" to "application skeleton" because the word template can be mistaken for a template file. Keep Tempita for templating? It does add an unnecessary dependency, but on the other hand it keeps compatibility with non-Pyramid frameworks that use it. Jeff Hammel has also put some work into a "paster create" alternative -- what is it?
"paster serve" can be extracted to a new utility. Would YAML be a better config format than INI? It can handle type conversions and nested structures, which INI can't. Logging is compatible with it, at least in Python 2.7 and 3.2 (dictConfig). Pyramid requires pipelines for error middleware, and YAML can express that directly as a nested structure. (On the other hand, it's useful for the application config to be top-level).
Should these utilities be under one command-line script (paster)? We could call it "glue" -- "Glue is the new Paste." It would have entry points for "create" and "serve", as well as those provided by the user or plugins. Alternatively, we could make top-level packages Create and Server with their own command. (Glue and Create are unregistered in PyPI; Serve is registered by James Gardner, a Pylons developer who may be willing to give up the name.) Another possibility for a command name is "karnak" (the ancient Egyptian city of temples). "skeleton" could also be a command name for a create command.
loadwsgi, loadserver, loadapp, and BetterConfigParser need better names. loadapp and loadserver need to be more documented and less recursive. BetterConfigParser needs to handle "%(here)s" automatically and do everything else loadapp does, so that people don't have to reinvent this. On the other hand, a YAML format would obviate it, and I think it has a better substitution syntax? Configuration should be more automatic: not just with "paster serve" but with all commands that take a config file.
PasteDeploy and PasteScript are closely related and are rarely used apart; they're both needed for "create" and "serve". Meld them together.
Work has been put into replacing PasteDeploy via Wiseguy. This code is not precious but represents one possible strategy to be able to ditch PasteDeploy.
Marrow is a namespaced refactoring of Paste, WebOb, and other utilities:
- marrow.blueprint (paster create) agnostic to underlying template lingua via alacarte, advanced interactive and command-line interrogation including secure password entry.
- marrow.script (paste script) using Python's powerful introspection features.
- marrow.wsgi.objects (WebOb) request/response/exceptions
- marrow.util (a lot of stuff including 2.6->3.1 forward compatibility)
- marrow.tags (12K page generations per second pure Python templating)
- marrow.server.http (C10K capable fully HTTP/1.1 compliant server)
- marrow.config (paste deploy) YAML-based configuration
- marrow.mailer (high-performance message delivery) a newer version of TurboMail with additional delivery transports (such as maildir)
MO: My main concerns are, marrow.script is not argparse-compatible, and marrow.server.http is asynchronous. I don't think we want to make such a large leap from multithreaded to asynchronous in the default server.
CM: I'm uncomfortable with the direction of these libs personally. They seem to be more researchy than practical in lots of cases.
GothAlice on argparse:
Avoiding argparse is a feature, not a bug. If you want argparse, what benefit does a scripting interface API give you? Effectively none. marrow.script is designed to allow you to write new callables (or use existing ones!) and have the command line interface Just Work. Even more importantly, marrow.script does not provide a single central script to run; your scripts aren't magically part of another package. Optparse/argparse/getopt are terribly non-Pythonic, even though argparse had the opportunity to fix things!
GothAlice on marrow.server.http:
The underlying communication stack (marrow.io/server) is fundamentally async. That particular fact has zero (and I mean it, zero) impact on writing WSGI applications that run under the HTTP/1.1 implementation. The underlying stack also has full support for a worker thread pool (using Futures) and multi-processing. Both of these details are handled for you by the web server; if you enable a thread pool, communication is still handled using async, but the WSGI application is executed in a thread. This allows you to tune a deployment for your requirements. Development or io-bound? Run a base server. CPU-bound? Run a threaded server. Need to scale? Add multi-processing to either of those.
GothAlice on build quality:
Proper releases of marrow packages require 100% unit test coverage, documentation, and multiple working examples. This takes the packages somewhat out of the realm of 'experimental' or plain 'research'. I've also successfully integrated blueprints, scripting, and marrow.tags into production, commercial code and managed to get WebCore and Pyramid both running under the HTTP server (with a tiny adapter decorator for WSGI1).
CM: "YAML config format" is not IMO a goal of any particular project, although it might be a side effect of replacing PasteDeploy (Wiseguy uses YAML currently). If we don't replace PasteDeploy, I don't think there's any benefit to changing PasteDeploy itself to use YAML.
There is some interest in replacing the INI config format with YAML. YAML can express ints, bools, lists, dicts, and ordered dicts (YAML: !!odict, Python: collections.odict). INI requires external routines to convert these. YAML's nested dicts are more flexible than INI's rigid two-level structure.
The main advantage of INI is that sysadmins who don't know Python can maintain it without having to learn Python's quoting rules. The two-level structure is also more useful than a Python module of scalar assignments (one-level). But INI is not a multi-language standard although people assume it is. It has no universal spec saying which characters are allowed in section names and keys. Python's ConfigParser interface is horrible, although it got better in Python 3.2.
This YAML config format is from Marco, a precursor to Pyramid. Marko's development was halted but we could revive parts of it for a new tool.
main:
component: pipeline
config:
apps: [ first, auth, core ]
first:
component: gzip-middleware
auth:
component: auth-middleware
core:
component: cascade
config:
apps: [ static, dynamic, app ]
static:
component: static
config: { directory: foo }
dynamic:
component: pyramid
config: { templates: foo }
app:
component: pyramid
config:
templates: bar
"New 'component' types can be created and plugged in via entry points. Each component type has a schema, so you can tell when some concrete configuration that tries to use it is wonky and you can theoretically generate forms or simple Q&A scripts from a preselected set of schemas that would spit out a configuration file. It'd need a good bit of work to give it feature parity with PasteDeploy/PasteScript though."
Make Chameleon, Mako, and PasteDeploy optional. Pyramid itself doesn't need formal dependencies that are only used by some application templates. These can be provided by pyramid_template_chameleon and pyramid_template_mako packages. PasteDeploy is not required when instantiating a Pyramid app directly. Akhet is becoming the "Pylons-like" Pyramid framework, which means that Pyramid itself can pare down some batteries if it wishes.
Delete code deprecated in Pyramid 1.0: pyramid.configuration.
Suggestions to simplify Pyramid by removing major features like Traversal or ZCA (Interfaces) will not be considered. These features are fundamental to Pyramid's goals. The complexity of the manual will be dealt with by higher-level frameworks such as Akhet, Khufu, and (TurboGears successor: Orion?), which can have simpler manuals. The complexity of the code, you'll just have to get used to.
Revamp the auth library. Chrism's idea. Perhaps put it under a new name and deprecate the existing one. We don't want to do this under GSOC, and maybe never.
Mailing list threads: