[1.0] Umbrella issue

[KyleAMathews]

Hey folks!

The remaining work (in my head at least) is starting to settle down into something coherent so I wanted to document my thinking here to get feedback on the vision for 1.0 and give those wanting to help out chunk sized pieces of work to pull off to work on so I can stop being the bottleneck for us shipping 1.0.

Goals for 1.0

To reiterate what I wrote last fall #419

Make building high-performance websites fun by:

• Enabling high performance by default. Anything that prevents Gatsby from generating the fastest possible website is a bug.
• Generating sites that have a rich webapp-like feel.
• Source all the data. Enable a plugin-driven data layer which lets Gatsby source and transform data from any number of sources.

TODOs

The following is a loose collection of TODOs for shipping 1.0. Some of them have issues already. Others not. If you want to work on one, please start an issue (or send a PR) claiming it. For more complex issues, please open an issue to discuss things before starting to write code. This is also not a complete list of course so feel free to add comments for other work that needs to done and I'll update the list. Also please file bugs you find as normal issues with the issue title prefixed with [1.0].

Check out the contributing guide for how to setup your development environment https://www.gatsbyjs.org/docs/how-to-contribute/

• Marketing
  • Design/implement front page
  • metatags on website
  • Favicon for site
• Documentation
  • Rewrite the Gatsby README.md for 1.0
  • Tutorial
  • APIs (automated from code comments hopefully). (TODO here still are docs for args to node APIs)
• Example websites
  • gatsbygram
  • using-drupal
  • Hacker News clone
  • Something with PDF processing
  • ecommerce store
  • i18n
  • animation
  • Anything else? Want example websites that are a) surprising (wait? a "static site generator" can do that?) b) use lots of Gatsby's official plugins (example sites are our integration tests) and c) simple so easy for people new to Gatsby to read through the code and understand it.
• Testing
  • Transformer plugins (e.g. JSON, YAML, Markdown)
  • Source plugins (e.g. Filesystem)
  • Caching
  • GraphQL query runner
• Flow types
  • everything (not a flow user yet so huge if someone wants to jump in on this...)
• Plugins
  • Add plugins various styling options e.g. sass, less, stylus, styled components, glamorous, anything else that's easy to get going and popular
  • PDF processing
  • Source plugins for popular data sources e.g. Wordpress, Drupal, hosted CMSs, Trello, Google Spreadsheets, etc.
  • Add support for word/paragraph counts to markdown Get word/sentence/paragraph count? remarkjs/remark#251
• webpack
  • Allow people to import all files types i.e. copy create-react-app's pattern here.
  • Support GATSBY-* environment variables
• babel
  • Use https://github.com/babel/babel-preset-env instead of a stage [1.0] make babel-preset-env the default babel setup instead of stage presets #975
  • Refactor to store babelrc state in Redux plus add action creators for plugins to add to the config.
• GraphQL data layer
  • Port JS frontmatter from 0.x (i.e. being able to reference export const data = {}) — should just be a transformer plugin
  • Add pagination "field" to all connections
  • Add aliases to children nodes in the graphql schema e.g. childrenMarkdownRemark.
• DX improvements
  • Add validation for options for each plugin.
  • Change filename of default layout component to layouts/index.js.
  • Change the default port that graphiql and gatsby serve-build run on so they don't interfere with the development server.
  • if hot update fails to apply, reload the page automatically.
  • Change .intermediate-representation to .cache and store data to be persisted between builds there.
  • split off the gatsby cli into its own package
• Performance
  • Tweak JS bundling strategy somewhat. Do commons bundle, bundle for each layout, bundle for routes, then a bundle for each page component and page data.
• Chores
  • Review package dependencies and remove ones we're not using anymore and update
    packages to their latest version

Post 1.0

• Copy default eslint checking from create-react-app

• Copy error modal from create-react-app

• Make starters searchable from CLI Browsable/searchable list of starters #56

• Upgrade to webpack 2

• Use Redux for managing the webpack config and for allowing plugins to modify it.

• Thinking in Gatsby

• Lots and lots of examples. Make them all searchable/browsesable inline in the website

• Add support for searching/browsing community plugins from website

• Themes

  • Get minimal theme system in place with rough plans for something more sophisticated. Basic idea is there'll be a gatsby-theme-core that all other themes will be based on. The core theme will include the basic Gatsby for the vanilla Gatsby experience. Then on top of that (themes are composable) add a number of other official themes e.g. gatsby-theme-sass, gatsby-theme-glamor, gatsby-theme-preact, gatsby-theme-markdown, gatsby-theme-blog, etc. Every theme will have an associated starter which will install the theme w/ a bit of sample content.

  • Setup build performance testing. Perhaps start with a script which auto-generates a site with 100, 1000, 10,000 pages and measures how long it takes to build them from scratch and with cached data.

[fabien0102]

Nice TODO list! For my part, I'm working on a gatsby-starter with full typescript support and some dev tool (jest, enzyme, xo, tslint, plop …) 😉

I've got a lot of work to do this starter functional (I want html.js and all pages/components in typescript) but it can be cool!

If someone wants to help (dev, tests, code reviews, suggestions…), you are welcome 😃

[KyleAMathews]

@fabien0102 cool! Yeah, if anyone else is working on their own 1.0 related ideas, please leave a comment!

[fabien0102]

@KyleAMathews For webpack config wrapper, why not use redux? (It will be easier to debug and logic with the rest of the stack)

[KyleAMathews]

@fabien0102 hmmmm good point. Yeah, it wouldn't be too much work to write our own custom wrapper and having it work exactly the same as the rest of the stack would as you say make it much easier to debug plus easier to use for people writing plugins as it'd work the same way as everything else. So yeah, just create custom action creators for modifying the webpack config in different ways. Sweet! I'll update the list above. Also, if you want to take this refactor on that'd be great!

[KyleAMathews]

@fabien0102 this is what I was planning on doing for babelrc as well so yeah, it does make tons of sense.

[fabien0102]

@KyleAMathews I can take the webpack config on redux refactor ;) (I just want to finish my typescript starter before 😉 )

BTW, if you can fix all tests before, it can be very cool :) (I prefer works in TDD for this kind of feature)

[craigmulligan]

Hey @KyleAMathews, super excited for v1.

I'm looking into how to create a source plugin.

It's based off a current metalsmith plugin we have written. It basically combines handlebars and markdown so you can create dynamic markdown documents using partials and generate bunch of pages from a few dynamic markdown files.

1. If I'm correct in understanding, I could watch these files and regenerate pages on each change and then call gatsbys createPages and it'd automatically update the graphql data?

EDIT: I just saw v1.0.0-alpha13 release which answers question 1 🎊

2. Is there a way to chain the source plugins, so I could use the fs plugin and then apply my transformations after?

3. Are there some hidden docs on source plugins or best to look at the fs plugin?

[KyleAMathews]

@craig-mulligan looks like an interesting idea!

So what you want to do I think is just write a transformer plugin. The filesystem source plugin handles already the watching of files. You just want to watch for new File nodes that it creates and then transform those based on your partials/handlebars into all the resulting Markdown nodes.

So files on disk are converted to File nodes which you convert to N markdown nodes.

Check out the various transformer plugins, you'll be doing something very similar, implementing onNodeCreate.

Perhaps move this to another issue too as you get into it as I need to start a docs page anyways for writing transformer plugins and helping you through the process would be a good time to do it.

[craigmulligan]

@KyleAMathews, sounds good! 👍

[gutenye]

Add plugins various styling options e.g. sass, less, stylus, styled components, glamorous, anything else that's easy to get going and popular

I just created a plugin for styled-components: https://github.com/gutenye/gatsby-plugin-styled-components.

In case someone needs it :)

[KyleAMathews]

@gutenye hey! great work on the plugin! But would you mind moving the plugin into this repo? I'd really like core plugins like this to be in the main repository to ensure that they're always up to date and really high quality. E.g. I'm making a lot of API changes right now and I don't want to have to bug you or others to update to the latest alpha when I make a breaking change. If everything is here, it's really easy to make a global copy/replace. Plus with our examples sites, we can ensure that all the plugins are working against every PR. I'm going to claim the NPM packages for all the other styling options plus other core plugins tomorrow so would appreciate it if you add this to the packages directory in the meantime. Would love your help building out other styling plugins here!

[gutenye]

@KyleAMathews Sure, PR is on the way :)

[KyleAMathews]

@gutenye sweet! Added you as a core-maintainer so you can merge PRs around this package and others you want to work on!

[gutenye]

Thanks.

…

On Tue, May 30, 2017 at 1:41 PM, Kyle Mathews ***@***.***> wrote: @gutenye <https://github.com/gutenye> sweet! Added you as a core-maintainer so you can merge PRs around this package and others you want to work on! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#796 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXCyJbG5T23HSzlbtOMZNmrLu9OJfMbks5r-6v1gaJpZM4M8Giz> .

-- JavaScript, Go, Linux and Open Source - g <http://GutenYe.com>uten.me - github.com/g <http://Github.com/GutenYe>utenye

[chiedo]

I noticed that everytime I create a build, it creates a new version of resources.

After two builds, I am left with this:

layout-component---index-eabb3eb87e4aebbe4e26.js.map
layout-component---index-eabb3eb87e4aebbe4e26.js
layout-component---index-2a79ddf4997c6690fecc.js.map
layout-component---index-2a79ddf4997c6690fecc.js

Is this intentional or are there plans to clear out data in the build process? If not, would you advise against doing a rm public/* before running the build command?

Thanks for all your hard work on this by the way!

[chiedo]

Also, for the gatsby-remark-responsive-image plugin, why not have the images stay where they were in the src/pages folder. Currently all the images are being saved to the public root. Maybe I'm being particular, but is there a reason that's the solution instead of keeping images where they were to begin with? For large blogs you could end up with thousands of images in the public root.

I can implement if you're in favor.

[KyleAMathews]

Is this intentional or are there plans to clear out data in the build process?

Keeping around old builds is actually a good thing as then people on older versions of a site (Gatsby sites are JavaScript apps) can still access bundles for their version of the site.

My plan is to delete old HTML files as we definitely want those gone so that deleted pages stop working.

[KyleAMathews]

Currently all the images are being saved to the public root.

They have to go here as that's where all the site's assets have to go... why would you want derivative images in your source folders? Gatsby is a build tool so it builds from source to built versions of the source. Images are just another variety of source material that Gatsby can build.

[chiedo]

Thanks @KyleAMathews I get the mentality on the old js builds but after 1000 builds, are you saying you'd expect there to be 1000 versions of the site with no cleanup strategy?

In regards to the images, I don't think I was clear. What I mean is this:

• If my source image is src/pages/blog/hello-world/hi.png wouldn't it be better to have the public image in public/blog/hello-world/... instead of public/...

[KyleAMathews]

are you saying you'd expect there to be 1000 versions of the site with no cleanup strategy?

Most built files wouldn't change but sure, I don't see a problem with this. Depending on the site, I can definitely see garbage collecting stale files after a while but in general, HD space is cheap so why not keep them around forever? It's trivial to clean them out yourself if for some reason that's a requirement for your site.

wouldn't it be better to have the public image

All built files are content addressable meaning their path is derived from their content. Which makes them safe to cache forever which is really valuable. So no, preserving their source path is not a priority.

[chiedo]

Cool @KyleAMathews that all makes sense.

Thanks!

[chiedo]

FWIW:

My concern was with github @KyleAMathews. I like to build locally and commit the build to git for deployments to 'dumb' servers that can just serve html and after 1000 builds I worried that things could get larger than github's limits? BUt maybe the size wouldn't be of any consequence and worst case, I can just clear the git history.

[KyleAMathews]

after 1000 builds I worried that things could get larger than github's limits

We can definitely revisit the question but most Gatsby site's output is tiny e.g. < 1 mb so size constraints from keeping old builds around won't generally be a problem.

[jbolda]

If this is the working assumption, how does that affect systems such as netlify that I am assuming builds and serves the new stuff, and takes down the old stuff? I'm thinking partially in terms of the service worker exploding if it expects old files that don't exist.

[chiedo]

I'm wondering if a better name for gatsby-remark-responsive-image would be gatsby-remark-build-images

My thinking is that the current name is a little misleading and the intent isn't clear. If I don't want images to be responsive, I may ignore that package all together not realizing that it's necessary to get my images from my markdown files so that they work in my build.

If we did change the name, then the default could be to just copy the default version of the image without creating all the responsive versions and we could have a responsive option that when set to true would do what it does now.

Just a thought. If it isn't helpful, feel free to ignore. :)

[KyleAMathews]

@chiedo could you please create new issues for these ideas/questions? Love the discussion/ideas just this issue isn't meant as a catch-all for 1.0 issues :-) thanks!

[chiedo]

@KyleAMathews yes sir! I misread what umbrella meant haha. I'll move this

[wiesson]

Any updates regarding the i18n topic? I tried to create a simple example by my own but I'm wondering if anybody would like to contribute.

What I did so far (it is a really simple example):

• Stored language texts in JSON documents (like indexEN, indexDE, indexFR for index page)
• Set language related meta data with helmet

File Structure
language files for index page

src/data/indexDe.json
src/data/indexEn.json
src/data/indexFr.json

Base Layout with Header (from the example)

src/layouts/index.js

Pages to load languages

src/pages/index.js // default is english
src/pages/de/index.js
src/pages/fr/index.js

Index Page Template

src/templates/IndexPage.js

What's missing
I need to set the language globally if the user enters /de or /fr and then prefix all links with that language. Any (simple) ideas? I would like to avoid to pull redux in to keep it simple.

Link (work in progress) to the repository: https://github.com/wiesson/gatsby-i18n-example

// edit

Just discovered gatsby-plugin-i18n. I will take a look at the plugin :)

[fabien0102]

@wiesson If it can help, I got an i18n system on my own website -> https://github.com/fabien0102/fabien0102.com

Just a lang props on index, and a filter in graphql queries 😉
https://github.com/fabien0102/fabien0102.com/blob/master/src/pages/en.jsx