Querying `allMarkdownRemark` with no markdown nodes fails #2212

LegNeato · 2017-09-23T20:17:07Z

If you have no markdown files in your page directory and have gatsby-transformer-remark enabled, querying for allMarkdownRemark throws an error. I would expect the query to be empty instead, as the support is there but there are no nodes.

The text was updated successfully, but these errors were encountered:

gre · 2017-09-24T12:35:31Z

a bit on that line, I'm having hard time figuring out where is allMarkdownRemark handled in the code. at no place there is a "allMarkdownRemark" endpoint so it feels magic naming inferring? (but where?)

LegNeato · 2017-09-24T12:39:05Z

It looks like the magic is for every node type registered (in this case MarkdownRemark) a field for all[Typename] (in this case allMarkdownRemark) is added to RootQueryType:

gatsby/packages/gatsby/src/schema/index.js

Line 13 in 524ff17

const typesGQL = await buildNodeTypes()

This only happens if there are nodes, hence this issue.

nihgwu · 2017-09-24T13:34:10Z

the MarkdownRemark node type will only be created when there is a markdown node, or there will be no MarkdownRemark node type at all, so you can't query allMarkdownRemark

LegNeato · 2017-09-24T13:49:39Z

Right, changing that behavior is what this issue is about 😄 . It was immensely confusing to install the transformer, have it in the config, yet have my queries marked as invalid. I spent a while debugging until I realized it was due to not having an existing .md file yet so there were no nodes. This also means if one were to transition off sources/transformers, a perfectly valid config can suddenly start throwing errors (like, for example if you deleted your last .md file).

While I don't think this is a huge issue, I would expect sources/transformers to register their GraphQL type and the "all" queries on source/transformer activation regardless of existing nodes.

This expectation is from the way GraphQL generally works...if you query a field that doesn't have data, you get null back rather than an error. There is a dividing line between the schema and the data in the schema, and that separation is broken when it comes to Gatsby and its plugins here.

djm · 2017-09-24T14:18:33Z

This one caught me too; I agree it's not the largest of issues but I think it would be a move that improved the developer experience.

The core issue for me was that my first thought when it occurred was not "oh, I don't have any markdown files in that folder yet", but rather "why hasn't the plugin registered the type? There must be an issue with how I installed and configured the plugin...". And that led me on a bit of a wild goose chase.

KyleAMathews · 2017-09-25T17:27:01Z

So this is interesting. I'm a bit surprised this is causing confusion. Let me try to explain my mental model for these parts of Gatsby.

At a high level this is how I see Gatsby


   +-----------+   +------------+   +----------------+
   |           |   |            |   |                |
   |  Source   +---> Transform  +--->  Infer GraphQL |
   |  Data     |   | Data       |   |  Schema        |
   |           |   |            |   |                |
   +-----------+   +------------+   +---------^------+
                                              |
                                              |
                  +-------------------+       |
                  |   GraphQL Queries+--------+
                  |                   |
                  |                   |
                  | React Web App     |
                  +-------------------+
                           |
                           v
                    Optimized Production
                    Build
                           +
                           |
                           v
                      Static Files Output

(I hope you enjoy my 5 minute ascii drawing :-D)

Source & Transformer plugins don't (generally) have anything to do with the GraphQL schema directly. They just "source" data from remote sources or "transform" data that source plugins pull in. So they don't create a schema, they create data (nodes in Gatsby-speak) which after they've finished their work, another Gatsby system comes along and infers a GraphQL schema from the nodes.

So if there's no markdown files, there's no File nodes with a mediaType of text/markdown which means gatsby-transformer-remark doesn't create an nodes of type MarkdownRemark which means there's no GraphQL types allMarkdownRemark or MarkdownRemark.

Transformer plugins don't need to know the type of nodes they're creating as often the type is only known at runtime e.g. "gatsby-source-contentful" creates nodes whose types are taken from the types of content created through the Contentful UI.

I get this is confusing as there's multiple data transformations going on and many layers of types going on as well.

I suppose a solution could be if a) a transformer plugin only creates "static" types (e.g. type names we know at code-writing time e.g. MarkdownRemark) then b) it could register with Gatsby this type name with any fields in knows for sure would be in the eventual type (generally not much).

Perhaps we just need to do a better job in docs explaining the setup so it's less surprising what's going on?

Other ideas?

fk · 2017-09-25T17:30:18Z

(To cut down drawing time: http://asciiflow.com/ ;) Edit: Took another look and seems you used it?! :D Nevermind, sorry for the noise ppl!)

djm · 2017-09-25T17:52:16Z

@KyleAMathews What you said makes full sense! The confusion on my part (I can't speak for LegNeato) came from making the assumption that the transformers have any say whatsoever in the graphql schema generation.

I think you're right, a documentation solution is probably enough – just telling users that transformers return just data (nodes) in the same sense source plugins do.

Edit: Perhaps if the confusion continues another solution could be that the transformers were made to report/log on the runs when they found nothing to transform; this may also help with people not realising they can remove certain transformers as they don't use them (for build perf). I can't think of a reason why you'd leave a transformer in if it wasn't transforming anything.

KyleAMathews · 2017-09-25T17:58:02Z

transformers were made to report/log on the runs when they found nothing to transform

Ooo! That's a great solution. But for helping people build good mental models of Gatsby and avoiding confusion about why things aren't showing up in GraphiQL but also, yeah having lots of unused transformers would slow things down somewhat (though not a ton most likely as most plugins just do a simple if check on nodes which is very cheap).

djm · 2017-09-25T18:10:45Z

I'm not by an editor at the mo, but I imagine it could probably happen in one place too? In whatever bit of code expects data from all the transformers.

KyleAMathews · 2017-09-25T18:34:22Z

Yeah that would work. We track which plugin creates which nodes. We could add a check for plugins which implement "sourceNodes" or "onCreateNode" and check that each of these plugins created at least one node and warn if not. Wanna make the PR?

djm · 2017-09-25T18:37:56Z

Yeah I'll take it, need to learn my away around those bits anyway!

KyleAMathews · 2017-09-25T18:54:54Z

Sweet! Add the validation code here:

gatsby/packages/gatsby/src/utils/source-nodes.js

Line 13 in e739a59

djm · 2017-09-26T12:24:56Z

@KyleAMathews

Had a crack at this last night and got pretty far but I need some guidance..

Example internal output from a test project:

# Plugins that implement sourceNodes or onCreateNode.
[ 'internal-data-bridge',
  'gatsby-source-filesystem',
  'gatsby-plugin-typography',
  'gatsby-transformer-remark',
  'gatsby-transformer-sharp',
  'default-site-plugin' ]

# state.nodes are owned by:
[ 'internal-data-bridge',
  'gatsby-source-filesystem',
  'gatsby-transformer-remark' ]

# Leaving the "should warn" crowd. These use the APIs but don't own nodes.

[ 'gatsby-plugin-typography',
  'gatsby-transformer-sharp',
  'default-site-plugin' ]

Out of that final list, I believe the only one we actually want to politely warn about is gatsby-transformer-sharp, as my project had no images to find and therefore it wasn't actually used.

However, the other two pose questions:

gatsby-plugin-typography implements sourceNodes but doesn't source anything.
default-site-plugin - sounds like it can be ignored?

What shall we do about these two? and do you know of any others that might be edge cases?

Thanks!

KyleAMathews · 2017-09-26T14:40:37Z

Sounds like the other plugins need a PR to remove the unused API implementation? Also realized that plugins can use onCreateNode to create nose fields.

djm · 2017-09-26T14:44:02Z

@KyleAMathews You realised that yesterday so the code checks for sourceNodes and onCreateNode already :)

I'm happy making the PR to change gatsby-plugin-typography, there's a comment in there which says:

// TODO add a new API hook e.g onPreBootstrap

Do you know the status of that?

djm · 2017-09-26T15:04:38Z

I think it'd have to be a new one, I don't see a similar API in api-node-docs, but the api-specification does mention it.

KyleAMathews · 2017-09-26T19:26:01Z

Yeah, it'd be new. Lemme add it and move gatsby-plugin-typography over.

KyleAMathews · 2017-09-26T19:51:08Z

@djm #2241

Alxandr · 2017-09-27T07:25:51Z

You might want to allow this warning to be turned off (by the plugin). If you want to be general about it, how about something like

exports.disabledWarnings = [
  'core/no-nodes-created'
];

djm · 2017-09-27T07:54:08Z

Yeah, that's something that can be added in time when it's definitely needed. For now, ensuring plugins don't use the incorrect APIs because there's nothing else available to them was the better option.

djm · 2017-09-27T12:50:36Z

Update on this: near completion - just need to update the tests and add a few new cases. I don't have any time tonight so I would expect the PR more likely on Friday/over the weekend. Thanks Kyle!

sebastienfi · 2017-10-02T15:47:47Z

@djm This is great, it will help a lot

djm · 2017-10-05T08:31:06Z

As ever, real life got in the way! This is coming along though:

Next step, update the tests then PR.

The warning serves two purposes: a) to notify those that are using the plugin that it hasn't created any nodes and can likely be removed, and b) to reduce the confusion caused by a GraphQL query for a specific node erroring as no nodes have been created yet. See gatsbyjs#2212 for more. As part of this work, two key changes have been made to the redux store: a) Each plugin now has a `nodeAPIs` key whose value retains an array of strings denoting the names of which Gatsby APIs the plugin implements. b) At the root of the store, a new key exists named `apiToPlugins` has been created which stores a simple key -> val map from the name of a Gatsby API, to an array containing the string names of all plugins which implement that API. a) is helpful when needing to know what APIs a given plugin implements and b) is helpful when code requires a quick look up of which plugins implement a specific API - we pay the cost once at creation time rather than iterating through the plugin objects whenever we need to know.

* Add warning when node-creating plugins create no nodes The warning serves two purposes: a) to notify those that are using the plugin that it hasn't created any nodes and can likely be removed, and b) to reduce the confusion caused by a GraphQL query for a specific node erroring as no nodes have been created yet. See #2212 for more. As part of this work, two key changes have been made to the redux store: a) Each plugin now has a `nodeAPIs` key whose value retains an array of strings denoting the names of which Gatsby APIs the plugin implements. b) At the root of the store, a new key exists named `apiToPlugins` has been created which stores a simple key -> val map from the name of a Gatsby API, to an array containing the string names of all plugins which implement that API. a) is helpful when needing to know what APIs a given plugin implements and b) is helpful when code requires a quick look up of which plugins implement a specific API - we pay the cost once at creation time rather than iterating through the plugin objects whenever we need to know. * Trigger builds

koluch · 2018-05-02T15:09:55Z

Hi,

Is there is any workaround for this issue? I want to make my site work fine when there are no Markdown files, but build fails with Unknown field 'allMarkdownRemark' on type 'RootQueryType'. How can I deal with it?

nileshgulia1 · 2018-05-03T11:46:51Z

@koluch I am also experiencing this issue, did you resolved your problem?

koluch · 2018-05-03T13:02:13Z

@nileshgulia1 I ended up with putting empty file in every folder I use for markdown files and then filtering them out in all the places I need them. It looks terrible, but it also fixed an issue with empty folders, because Git doesn't stores empty folders, and gatsby-source-filesystem is unable to handle non-existed folders properly :(

You can look at it here, this is the project I just started working on: https://github.com/koluch/he-he-2

nileshgulia1 · 2018-05-03T15:41:04Z

@koluch Thanks a lot, I will look into it :)

KyleAMathews · 2018-06-27T02:14:21Z

I believe this was fixed so closing

JGallardo · 2019-03-03T01:39:14Z

for anyone doing a google search and ends up here, your answer may be at
https://www.gatsbyjs.org/packages/gatsby-transformer-remark/

tfaieta · 2019-03-19T19:38:15Z

Hi all, really loving Gatsby, but I'm running into the same issue when I'm trying to build my site. Running gatsby develop works fine but I get this error when I run gatsby build:

 Error: Cannot query field "allMarkdownRemark" on type "Query".
 GraphQL request (3:9)
 2:       {
 3:         allMarkdownRemark(
            ^
 4:           limit: 1000,

The problems started after I added gatsby-mdx and reconfigured my gatsby-config and gatsby-node files. I currently have no .MD files and only have one .MDX file.

Here's a link to the repo if you all can help me out I'd appreciate it:
https://github.com/tfaieta/tfaieta.com

jonniebigodes · 2019-03-19T20:55:38Z

@tfaieta i've cloned your repo, installed the dependencies and issued gatsb build && gatsby serve. And i'm presented with this:

Doing a bit of digging it looks like you have installed gatsby-plugin-feed and activated it in the gatsby-config.js.
Disabling it will allow a sucessful build as you can see here:

If you need that plugin, a little more of a configuration might be needed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Querying `allMarkdownRemark` with no markdown nodes fails #2212

Querying `allMarkdownRemark` with no markdown nodes fails #2212

LegNeato commented Sep 23, 2017

gre commented Sep 24, 2017 •

edited

Loading

LegNeato commented Sep 24, 2017

nihgwu commented Sep 24, 2017

LegNeato commented Sep 24, 2017 •

edited

Loading

djm commented Sep 24, 2017 •

edited

Loading

KyleAMathews commented Sep 25, 2017

fk commented Sep 25, 2017 •

edited

Loading

djm commented Sep 25, 2017

KyleAMathews commented Sep 25, 2017

djm commented Sep 25, 2017

KyleAMathews commented Sep 25, 2017

djm commented Sep 25, 2017

KyleAMathews commented Sep 25, 2017

djm commented Sep 26, 2017 •

edited

Loading

KyleAMathews commented Sep 26, 2017

djm commented Sep 26, 2017 •

edited

Loading

djm commented Sep 26, 2017

KyleAMathews commented Sep 26, 2017

KyleAMathews commented Sep 26, 2017

Alxandr commented Sep 27, 2017 •

edited

Loading

djm commented Sep 27, 2017

djm commented Sep 27, 2017

sebastienfi commented Oct 2, 2017

djm commented Oct 5, 2017 •

edited

Loading

koluch commented May 2, 2018

nileshgulia1 commented May 3, 2018

koluch commented May 3, 2018

nileshgulia1 commented May 3, 2018

KyleAMathews commented Jun 27, 2018

JGallardo commented Mar 3, 2019

tfaieta commented Mar 19, 2019

jonniebigodes commented Mar 19, 2019 •

edited

Loading

tfaieta commented Mar 19, 2019

Querying allMarkdownRemark with no markdown nodes fails #2212

Querying allMarkdownRemark with no markdown nodes fails #2212

Comments

LegNeato commented Sep 23, 2017

gre commented Sep 24, 2017 • edited Loading

LegNeato commented Sep 24, 2017

nihgwu commented Sep 24, 2017

LegNeato commented Sep 24, 2017 • edited Loading

djm commented Sep 24, 2017 • edited Loading

KyleAMathews commented Sep 25, 2017

fk commented Sep 25, 2017 • edited Loading

djm commented Sep 25, 2017

KyleAMathews commented Sep 25, 2017

djm commented Sep 25, 2017

KyleAMathews commented Sep 25, 2017

djm commented Sep 25, 2017

KyleAMathews commented Sep 25, 2017

djm commented Sep 26, 2017 • edited Loading

KyleAMathews commented Sep 26, 2017

djm commented Sep 26, 2017 • edited Loading

djm commented Sep 26, 2017

KyleAMathews commented Sep 26, 2017

KyleAMathews commented Sep 26, 2017

Alxandr commented Sep 27, 2017 • edited Loading

djm commented Sep 27, 2017

djm commented Sep 27, 2017

sebastienfi commented Oct 2, 2017

djm commented Oct 5, 2017 • edited Loading

koluch commented May 2, 2018

nileshgulia1 commented May 3, 2018

koluch commented May 3, 2018

nileshgulia1 commented May 3, 2018

KyleAMathews commented Jun 27, 2018

JGallardo commented Mar 3, 2019

tfaieta commented Mar 19, 2019

jonniebigodes commented Mar 19, 2019 • edited Loading

tfaieta commented Mar 19, 2019

Querying `allMarkdownRemark` with no markdown nodes fails #2212

Querying `allMarkdownRemark` with no markdown nodes fails #2212

gre commented Sep 24, 2017 •

edited

Loading

LegNeato commented Sep 24, 2017 •

edited

Loading

djm commented Sep 24, 2017 •

edited

Loading

fk commented Sep 25, 2017 •

edited

Loading

djm commented Sep 26, 2017 •

edited

Loading

djm commented Sep 26, 2017 •

edited

Loading

Alxandr commented Sep 27, 2017 •

edited

Loading

djm commented Oct 5, 2017 •

edited

Loading

jonniebigodes commented Mar 19, 2019 •

edited

Loading