WIP Gatsby Integration

[markhuot]

CraftQL exposes the Craft content model through GraphQL. To do this we transform every content type/section/entry type in to a unique Object Type within GraphQL. To re-implement this in Gatsby would require re-defining the entire CraftQL schema in to a gatsby-node.js plugin, essentially duplicating the logic to turn Craft in to GraphQL.

Personally, I was hoping that Gatsby could query CraftQL directly, but that doesn't seem to be the case.

One example: exposes an event entry type, and all the custom fields on an event. You can then find any reverse relationships to the event such as the city the event is in, the hotels related to the event, etc… CraftQL exposes this, like so:

{
  entriesConnection(type:event){
    edges{
      node{
        id
        title
        ...on Event{
          city
          hotels{
            name
          }
        }
      }
      relatedEntries{
        edges{
          node{
            title
            url
            type{
              name
            }
          }
        }
      }
    }
  }
}

Short of re-implementing the current PHP logic of CraftQL into a Javascript-backed gatsby-node.js plugin I'm not sure how to expose this level of fidelity through to a Gatsby-backed React component?

Fair warning: I could be way over thinking this or approaching this from the wrong point of view. If so, sorry!

[markhuot]

I think this is related to gatsbyjs/gatsby#2165, which asks about a way to mirror an existing GraphQL schema and not have to recreate the entire thing in Gatsby.

[zauni]

I think you have the same problem as GraphCMS had. They also have a GraphQL API and they built a source plugin which doesn't look complicated: https://github.com/GraphCMS/gatsby-source-graphcms/tree/master/src
It looks like they just query the GraphQL API and based on the result they build nodes. The user of the source plugin has to build a query which can fetch all the relevant content: https://github.com/GraphCMS/gatsby-source-graphcms/blob/master/README.md#usage

So I think you don't have to duplicate the PHP logic but instead just use your API, build gatsby nodes and gatsby will handle the rest. But I don't know if it is really so simple :)

[markhuot]

Thanks for the thoughts @zauni, I'm giving this a try now, but I'm wondering… if I call createNode I get a GraphQL Object Type of CraftQLConnection. This works for simple use cases but I'd like to also create additional types. For example, CraftQLCraftSection and CraftQLCustomEntryType, etc… Because everything in Craft can have infinite custom fields, CraftQL creates new types for each of them to define the shape of the resource, correctly. I'm not seeing any way in the Gatsby API to create additional types other than the default type provided by the gatsby-node.js plugin.

[zauni]

In the gatsby docs for the "Node Interface" it says Gatsby automatically infers the structure of your site’s nodes and creates a GraphQL schema which you can then query from your site’s components. (https://www.gatsbyjs.org/docs/node-interface/#graphql)
This let me think that maybe you can't build the GraphQL Schema yourself. You can just build nodes like gatsby want to have them. Maybe it's not too hard to build the nodes, but you have quite sophisticated query capabilities in your API and I don't know if the generated GraphQL Schema from gatsby has the same capabilities. This would be interesting.

How is it going? Did you get something to work? Hopefully you get something working, this would be perfect to use CraftCMS as a backend for static sites.

[markhuot]

I'm still working on this and think the biggest issue is that Gatsby assumes your source API stinks and abstracts it completely away. E.g., If we had a JSON API out of Craft that only returned a list of "elements" then the Gatsby API would be a huge step up. However, we actually have a fantastic API out of Craft, already. We have full text searching, per-custom-field searching, image sizing, date formatting, etc…

For example, the left is what Gatsby provides and the right is CraftQL,

We can beef up the Gatsby side by exposing all the text fields out to native Gatsby fields, that would gain us a bit, but it'd never be comparable to the native Craft API. I'm still investigating what can be done here.

[KyleAMathews]

Hey! Sorry for jumping in here a bit late. A bit of exciting news, with gatsby v2 we're going to launch support for schema stitching! Which will mean you can reuse the native graphql API directly.

Gatsby assumes your source API stinks

Lol :-) I wouldn't put it quite that bluntly but yeah, the assumption was that you were pulling data from a rest API w/o many options so we'd easily do better than that. Craft seems like it has an awesome API so we wouldn't want to get in the way of that. Also gatsby was meant to fulfill use cases where you just dump in raw data e.g. markdown or csv or yaml files on disk so we had to provide something on top.

[KyleAMathews]

Oh one other thing. You can set node types to anything you like so you're definitely not limited there.

[markhuot]

Oooo! Schema stitching sounds perfect. Is this on a branch yet for testing?

Also gatsby was meant to fulfill use cases where you just dump in raw data e.g. markdown or csv or yaml files on disk so we had to provide something on top.

Yup, that’s makes sense.

You can set node types to anything you like so you're definitely not limited there.

Hrm, I don’t think I’m following. I see how to make new object types/nodes. But I’m not sure I see how a field on a Node can return anything other than a string. This question may be moot with schema stitching though.

[markhuot]

Just pulling in some related references. Schema stitching support in a future Gatsby release should make a CraftQL implementation very feasible. Until then I'm going to put this on hold because we have some Craft-based-PHP logic that can't be smartly ported over to a gatsby-node.js implementation.

• https://twitter.com/kylemathews/status/942842474271088640
• WIP: Add async SSR data fetching gatsbyjs/gatsby#2423
• https://dev-blog.apollodata.com/graphql-schema-stitching-8af23354ac37

[magicspon]

Hello,

Have you made any more progress with this? Super keen to have a play with gatsby integrated with craftcms.

cheers

[markhuot]

No progress yet @magicspon. There's been a number of conversations around schema-stitching with Gatsby and GraphQL such as: gatsbyjs/gatsby#4708.

[gusnips]

https://github.com/gusnips/gatsby-source-craftcms
I just added a plugin to gatsby to fetch data from craftQL
feel free to open an issue if you find any problems using
it's working on gatsby v1 and v2

[markhuot]

This is exciting @gusnips! I gave this a try and it seemed to work as I would expect. If I'm following correctly this takes all the data out of Craft at build time and makes it available through the usual Gatsby allContentType GraphQL fields. This definitely works today and is further than I've had a chance to push this integration so thanks!

Note for those reading through the thread: I'm still tracking schema stitching as another approach to integrating CraftQL in to Gatsby.

[gusnips]

If I'm following correctly this takes all the data out of Craft at build time and makes it available through the usual Gatsby allContentType GraphQL fields

Exactly, it was the only way I could get it working

Thanks!! Glad I could help

[rjgux]

This will be of interest to those looking at integrating with Gatsby gatsbyjs/rfcs#6

[magicspon]

I think this can be closed now.... 'gatsby-source-graphql' works like a charm.

{
	resolve: 'gatsby-source-graphql',
	options: {
		fieldName: 'craft',
		typeName: 'Craft',
		url: 'https://yoursite.com/api',
		headers: {
			Authorization:
				'bearer wibblewibble'
		}
	}
},

then in gatsby:

{
  craft {
    entries(section: [blog]) {
      __typename
      ... on Craft_Blog {
        title
        subtitle
        body {
          content
        }
        tags {
          id
          title
        }
      }
    }
  }
}

"lovely stuff"

[markhuot]

@magicspon, not sure if you're interested but in Gatsby 2 there is native schema stitching. That means you can query CraftQL directly from your React components now. No more need for a middle step of writing out the queries to a static source that you can then query. I'm planning to write up a demo of this shortly to better illustrate the amazing possibilities.

Notes for future reference:

• Proposal: Add plug-n-play support for arbitrary GraphQL sources gatsbyjs/gatsby#4708
• Add native graphql source RFC gatsbyjs/rfcs#6
• Add graphql-source-plugin gatsbyjs/gatsby#6475

[magicspon]

hey @markhuot , thanks for the heads up.

I've been using gatsby-source-graphql with craft/craftql and it's been pretty seamless, how does this differ?

ta

[zauni]

I think @magicspon already meant the native schema stitching with the official gatsby-source-graphql plugin. The old gatsby-source-graphql plugin from @wyze used graphql files for the queries.
It's a bit confusing that they have the same name...

[magicspon]

Yup.. I've been using the official one.

[wyze]

It's a bit confusing that they have the same name...

I had the original package which I then donated to Gatsby. Sorry for the confusion!

[pstinnett]

Is anyone having issues with gatsby-source-graphql / CraftQL in say... the past couple days? My set up was working great but now I'm getting errors and I'm unable to query.

From the Gatsby GraphiQL I'm trying:

{
  craft {
      entries(section: [pages]) {
       	__typename
        ... on Craft_Pages {
          id
          title
          uri
        }
      }
  }
}

which is returning

{
  "errors": [
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        0
      ]
    },
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        1
      ]
    },
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        2
      ]
    },
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        3
      ]
    },
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        4
      ]
    },
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        5
      ]
    },
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        6
      ]
    },
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        7
      ]
    },
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        8
      ]
    },
    {
      "message": "__typename did not match an object type: Craft_Pages",
      "locations": [
        {
          "line": 3,
          "column": 7
        }
      ],
      "path": [
        "craft",
        "entries",
        9
      ]
    }
  ],
  "data": {
    "craft": {
      "entries": [
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null
      ]
    }
  }
}

I'm using basically the same setup as @magicspon above, but the section is named pages instead of blog.

Things were working well until the past few days where I've hit this issue. Having trouble debugging it. Any help would be greatly appreciated!

[narration-sd]

@pstinnett @markhuot Yes -- I have a demo app for something interesting coming for Gatsby which uses a query much like that.

I've just tried it on Craft 3.1.19 as updated today, and can't build. I broke things down so I could run Gatsby Graphiql only, and the query then fails in a quite related way to yours.

The same query in CraftQL/general GraphQL form, without the Gatsby prefixes, still runs fine, on Craftiql where that ordinary form is expected.

When the Gatsby query fails, it asks me if I didn't mean any in a list of 'basic' Craft objects. This list doesn't include the sections and so forth of items I've added to make the Craft data model for the application.

I thus strongly suspect the problem is coming in Schema capture, so that Gatsby GraphQL isn't able to learn about data types added to base Craft.

This should be a pretty good hint to @markhuot Mark, especially if he's done something with Schema queries or Schema formats in answering them.

We've in fact already spotted trouble in this area, as for weeks at least Gatsby hasn't been able to find any of Mark's directives which go beyond the very basic. His @date was the example, as reported in #231

In #231 I posted a schema discovery query which finds his @date directive when run from Craft's query tool, but not when used from Gatsby's.

I ran this just now on the failing Gatsby iql. Results are too long to post here, but it sees my sections as CraftQL_SectionsEnum and CraftQL_EntryTypesEnum. and that's all.

I don't see composite definitions of the Entries objects for these sections, and I find only Craft-standard fields like title, uri, etc.. -- there is no item for an image field I have, for example.

This looks very much like the missing directive in the earlier case, and I think the tools are there for Mark to see what's not coming through to Gatsby.

I just ran the same schema discovery query on the same Craft using its local Craftiql, and got back lots of objects related to my sections and fields, just as you would expect, so there's some definite proof these just aren't getting through, very possibly by a format failing to match, would seem one likely cause.

As well, just heard from a friend who's having the same kinds of problems.

Wishing us eager fortune,
Clive

[narration-sd]

@markhuot, Mark, for your morning, I attach a report of things I've tried and possible portion of conclusion. Matt Stein, @mattstein has also been easter egging this as we chatted.

Steps taken, often with some degree of short binary search on versions, and reckoning from the fact that CraftQL was working ok here, apart from the mentioned & seemingly could be related @date issue, up to a day or two ago maximum:

• backed off Craft to 3.1.17 several days ago, no help
• backed off npm versions to several days ago, no help
• inspected CraftQL 2.x branch on repo, no changes in 12 days, no help
• began to suspect libraries of CraftQL, ran back webonyx/graphql-php, which should then be the culprit, as far as 0.11.0 as left side of binary search, vs. Mark's 0.11.6, no help. I do note that
  this lib is actually up to 0.12s, but again no help as Mark's not using latest
• It can be something webonyx/graphql-php uses, which perhaps escaped back compatibility, of course....
• As a wrapper on effort for this, I hooked up and ran a simple Gatsby Graphiql query against the hallowed Star Wars swapi gql server, as Gatsby uses for test examples. This has the same spec and unusual query forms as Craft-Gatsby ones do, and it discovered schema for it and ran the query fine.
• I also checked the web for any recent problems with the gatsby-source-graphql plugin that runs all this, and there's nothing
• conclusion: really seems CraftQL is not performing, but very likely to be a component of its components at some depth. It's not impossible that bringing CraftQL up to the latest webonyx/graphql-php version would make the repairs...

Mark, could mention that here this is holding up a Beta with interested customer looking over a participant's shoulder, of something I think you'll be interested in as it becomes more public. Best, Clive

[pstinnett]

@narration-sd Thanks for confirming all of that! @markhuot Should I break this off into a separate issue? Happy to provide any assistance that I can to help get this figured out.

[narration-sd]

@pstinnett You're welcome, and I have more....a long morning of digging holes got part of an answer.

• I know for a fact now that this is not due to something @markhuot changed, nor Craft itself. Reason: almost all my testing happened to be done with Gatsby config I forgot that pointed CraftQL queries to a remote staging server. Nothing's been changed there in the past two weeks. So this bust-up happens only in the presence of Gatsby project.json or gatsby-cli packages.
• I'd like to let Mark fully off the hook, but we can't do that until we see what actually has changed, thus whether it's something bad now about Gatsby's schema-collecting query, or CraftQL's ability to properly respond to it.
• CraftQL's graphiql continues to work fine, while Gatsby dev's fails in the same way as projects do, just with more ability to see that it's the Craft sections' information other than name that's all missing. Basis Craft entities are there. Nothing but name enums are, for actual Sections, Categories, etc. you've put into the Craft operating schema yourself. That's where the error happens, and also is one more way to see it's on Gatsby's side, since Mark hasn't changed his basis for his graphiql.
• I already last night tried backing off the Gatsby project's packages.json to a week ago, thanks to JetBrains local history. This didn't help, but in fact I think Gatsby likely has all their own packages spec'd with up-arrow versions, so that they would take the latest apollo, etc. even for older own versions. -

Thus on last point, anyone knowing a way to force an npm update or install to a specific date would be a hero here....

I'm thinking about ways to get the difference in good and failing schema queries out of CraftQL, not an easy one either. A break now, maybe will give a thought. Because we really need this back to working.

The p.s. to this is that underlying apollo code in the schemas area is machine written, and a nightmare. Thus to just effectively back version out, the thing....

[pstinnett]

@narration-sd Excellent work! I agree I now do not think this is a CraftQL issue. I spent most of the day away from this issue, but just searching now I've found this in the Gatsby repo: gatsbyjs/gatsby#12717

Looks like a similar (the same?) problem...

[pstinnett]

@narration-sd 1 more update for tonight. I do believe that issue linked above is what we're seeing. I set my gatsby version to 2.1.39 and ensured I was running gatsby develop using the project version (not global gatsby-cli) as npm run develop and my project is compiling again. Hopefully Gatsby team can fix this one up soon.

@markhuot Sorry for the spam! Thanks for all your hard work on this plugin.

[narration-sd]

@pstinnett no spam problem; this the good kind -- and great find.

Thank you kindly for passing it on.

While those Deutschland fellows eat up their evening, or next morning, I'm going to try your back-off to see if it gets us on the air right away here.

[narration-sd]

Ok, think I have this very nicely in place for us, thanks to you @pstinnett and an add from @mattstein.

• need to be sure gatsby(-cli) remains held on 2.1.39 until this fix this, across npm update commands
• need now a polyfill to stop today's non-understandable increasingly circular chatter on builds about core-js

This looks to do both, plus a freebie:

  "devDependencies": {
    "gatsby": "2.1.39",
    "@babel/polyfill": "^7.4.0",
    "prettier": "^1.16.4"
  },

@markhuot, I think you are officially off the hook :)

[narration-sd]

...the Gatsby (cli) team fix via 2.2.7 does appear to clear yesterday's difficulties - I tested on 2.2.8, and am leaving our repo package.json devDependencies line in, modified to ^2.2.8, to catch older usage, and to prepare for any further oncoming trains like this :)