Deferred Static Generation

[pragmaticpat]

As the title suggests, Deferred Static Generation is conceptually very similar to SSG. The only difference with Deferred Static Generation is developers can choose to defer building certain pages until the first time a user requests it. Deferred Static Generation gives developers greater control over their site’s build times.

As this is a new feature available in Gatsby 4, we're eager to hear about your experiences and use cases where you apply DSG in your Gatsby projects. Please share your questions and findings below!

Read all about Deferred Static Generation here

Thank you for building with Gatsby! 💜

[tlgimenes]

DSG is amazing! I think it will greatly decrease my building times. However, I'm not using createPage api directly, but rather Gatsby file system route api. I was wondering if there is a way of using DSG with File System route API or if I should go with SSR instead.

Thanks!

[hashimwarren]

I have the same question 👍

[moonmeister]

I believe the docs say this is not possible now but will be soon. Hopefully with GA.

[wardpeet]

It will be ready for GA 👍

[tlgimenes]

Ok, so just to keep track. It was postponed after GA: #33446

[tlgimenes]

Nice, it was released on https://www.gatsbyjs.com/docs/reference/release-notes/v4.1/

[ziimakc]

Is there a way to regenerate page after some time or using api call again?

[moonmeister]

Update: I'm assuming by cache you mean the Gatsby data layer cache.

no actually I meant the service that serves the website. So in Gatsby Cloud, that's Fastly. When Gatsby says a page is invalidated, you need to invalidate the page in your hosting service.

But isn't it Gatsby best practice to not cache html and page-data.json? So why would you need to invalidate those? Or is that not the case for DSG?

Better question, if I hit a DSG route that has no CDN cache, will it render everytime or is Gatsby OSS caching the generated page untill the data layer says otherwise?

[KyleAMathews]

there's a difference between the CDN cache and the browser cache. You don't want the CDN to be hitting the origin on every request.

[moonmeister]

there's a difference between the CDN cache and the browser cache. You don't want the CDN to be hitting the origin on every request.

Is this dock still accurate? https://www.gatsbyjs.com/docs/caching/

If not, what would the recommendation be?

UPDATE: I just re-read up on etags/must-revalidate and I get how despite the max-age=0 stuff still gets cached. Just not intuitive. Though it still doesn't explain why I'd need to tell the CDN to invalidate.

[KyleAMathews]

Yeah, that doc is accurate.

CDNs have edge servers (what they run) and origin servers (what you run). The edge servers have their own caches. So e.g. Gatsby Cloud has origin servers running on Google Cloud and we use Fastly as our edge servers. Fastly caches files for a site on their edge servers and serves all requests from there. They only go back to Gatsby Cloud when there's a new file or we tell Fastly that a file has changed.

So there's three caches — the browser cache (which is what the doc describes), the Fastly edge server cache (which Gatsby Cloud controls), and the Gatsby cache that's maintained as part of builds. They all feed forward to the next one.

This is a decent explanation about CDN architecture https://www.cdnetworks.com/knowledge-center/what-is-origin-server/

Hopefully that was as clear as mud :-D

Anyways — I'm guessing these questions are in regard to gatsby-plugin-fastify? If you're self-hosting with that, then I'm assuming you're fine as fastify's static serving would just notice that files in the public folder had changed and start serving those. For DSG, it'd need a way to write out the results of query running/html generation so that on the next build, it can incorporate that into the process. That's what I mean by there'll need to be some sort of API created for this.

[moonmeister]

Thanks Kyle. Much appreciated.

[laurenskling]

While giving Gatsby 4 a spin using gatsby-source-graphql, I found that DSG is not working :(

I've set my codebase https://github.com/laurenskling/using-gatsby-source-graphql to defer rendering the film pages, and now these pages get an errors prop.

Example on Gatsby Cloud: https://build-58701a40-9907-423a-9f26-9f80e4b3beea.gtsb.io/ZmlsbXM6Mg== Saying: Cannot query field "swapi" on type "Query". So the entire external Graphql isn't found.

Building and serving locally I do no get this specific error, but another one. I'm getting in my errors prop:

{
  message: "nodeFetch is not a function"
  path: (2) ['swapi', 'film']
}

[devtim123]

i also have the same problem rendering pages in DSG mode in combination with "gatsby-source-graphql" plugin. Here are my complete Error-Log from my local environment build:

Rendering html for "/new-news" failed. nodeFetch is not a function

  58 |       width
  59 |       height
  60 |     }
  61 |   }
  62 |   fieldPillarImageCaption
  63 |   fieldPillarImageCopyright
  64 | }
  65 |
  66 | query NodePillarNews($nid: String!) {
  67 |   drupal {
> 68 |     nodeById(id: $nid) {
     |     ^
  69 |       ...PillarNewsFragment
  70 |     }
  71 |   }
  72 | }
  73 |

  TypeError: nodeFetch is not a function

  - index.js:280050 exports.fetchWrapper
    /.cache/query-engine/index.js:280050:26

  - index.js:277211
   /.cache/query-engine/index.js:277211:13

  - index.js:266596 new Subscription
    /.cache/query-engine/index.js:266596:34

  - index.js:266678 Observable.subscribe
    /.cache/query-engine/index.js:266678:14

  - index.js:277099
    /.cache/query-engine/index.js:277099:20

  - new Promise

  - index.js:277098 toPromise
    /.cache/query-engine/index.js:277098:12

  - index.js:262679
    /.cache/query-engine/index.js:262679:80

  - index.js:255900
    /.cache/query-engine/index.js:255900:87

  - index.js:256008 new ValueOrPromise
    /.cache/query-engine/index.js:256008:21

  - index.js:255900 delegateRequest
    /.cache/query-engine/index.js:255900:16

  - index.js:255834 delegateToSchema
    /.cache/query-engine/index.js:255834:12

  - index.js:252395
    /.cache/query-engine/index.js:252395:120

  - index.js:252391
    /.cache/query-engine/index.js:252391:16

  - index.js:137622 wrappedTracingResolver
    /.cache/query-engine/index.js:137622:20

  - index.js:137622 wrappedTracingResolver
    /.cache/query-engine/index.js:137622:20

The generated cached file under "/.cache/query-engine/index.js" where the error occurse have following in line 280050:

const nodeFetch = __webpack_require__(1737); // this is passed to the Apollo Link
// https://www.apollographql.com/docs/link/links/http/#fetch-polyfill


exports.fetchWrapper = async (uri, options) => {
  const response = await nodeFetch(uri, options); //<-------- here the error occurs

  if (response.status >= 400) {
    throw new Error(`Source GraphQL API: HTTP error ${response.status} ${response.statusText}`);
  }

  return response;
};

I was trying to install node-fetch, but that did not fix the problem.

Any hint/help would be appreciating

[devtim123]

Maybe we should also mention that this error not happens on build time, it happens after request a page in the browser which is in DSG mode and served with "gatsby serve"

[JigneshWorld]

Yes I am getting same error on production build not during development

Any possible workaround?

[devtim123]

Testing this with Gatsby Version 4.1 the error still occurs with the gatsby-source-graphql. Are there any workarounds for using DSG with graphql-sources?

[vladar]

Probably caused by the issue in node-fetch when bundled with webpack: node-fetch/node-fetch#450

The fix described in node-fetch/node-fetch#450 (comment) should probably help. I encourage you to submit a PR! I guess it should just change a single line:

gatsby/packages/gatsby-source-graphql/src/fetch.js

Line 1 in 83ca1df

const nodeFetch = require(`node-fetch`)

to:

const nodeFetch = require(`node-fetch`).default

But please double check if it works for you in normal develop/build workflow!

[tlgimenes]

Hello Gatsby,

I was trying to use DSG in my project but I couldn't get the performance boost I was hoping and I decided to share my problem with you guys to get some feedback and solutions.
My project consists in an ecommerce with 11 million skus. I'm currently running on Gatsby v3 and I'm using a hybrid rendering approach where I SSG a small portion of these products during the build phase and I use client-side rendering for the rest. My current pages folder looks something like:

./src/pages/
├── index.tsx
├── [slug].tsx
└── {Product.slug}.tsx

I was giving Gatsby v4 a try and one of my hopes with DSG was to remove this hybrid rendering approach and move all my rendering to the backend for SEO purposes. With this I would be able to remove this [slug].tsx file and have all my product related code in one single page file.
When I started doing this, I've figured out that I would need to fetch and source all these 11 million products during the sourceNodes step, which is a HUGE performance hit, since fetching and sourcing all of this data can take quite a while and can take a considerable amount of RAM memory.

I then tried to take the opposite approach to the problem. I started wondering if I could remove the file {Product.slug}.tsx instead and use SSR in [slug].tsx to render these pages as the users visit these pages. I was taking a look to the SSR docs and apparently, the difference between SSR and DSG is the caching mechanism it uses. Apparently, SSR uses http caching, whereas DSG caches the generated pages per "deploy". The problem with http caching is that it does not ensure user/bot experience in my site since I always need to add a max-age to my pages. Also, another problem with SSR is that it does not accept graphql queries being made inside the getServerData function, which is bad since I would like to use the Gatsby schema during SSR

To me, the ideal solution for this problem would be a mix between SSR and DSG. I would like to write a file [slug].tsx where I can opt into the DSG caching strategy (per deploy instead of max-age) and being able to query the Gatsby graphql schema inside getServerData.

Thats it, let me know if you guys have faced a similar problem or if I'm overthinking this problem. Any ideas or suggestions are welcome, thanks!

[moonmeister]

Thanks for sharing. I think this is a really interesting use case where that would require being able to live query data through Gatsby's data layer that wasn't already fetched at build time. I'd love to see support for this as well.

For now qureyingthe data layer in DSG only has the data that was consumed via sourceNode at build time. That said, in your getServerData function you can't query the Gatsby data layer, but currently that wouldn't be helpful. What you can do is query your API directly. But, as you said, this isn't ideal cause your left using a time based cache. Using a Stale-while-revalidate cache here could be beneficial, or you could connect your data source directly your caching layer to purge cache when data changes.

Hopefully those ideas help for now. Hopefully we get a stronger long term solution for massive datasets like yours that don't require these work arounds.

[tlgimenes]

Thanks for your reply @moonmeister.

For now qureyingthe data layer in DSG only has the data that was consumed via sourceNode at build time. That said, in your getServerData function you can't query the Gatsby data layer, but currently that wouldn't be helpful

Oh, I thought that something like gatsby-source-graphql plugin or addThirdPartySchema API would work on this case. Maybe this can be a way for the future.

What you can do is query your API directly. But, as you said, this isn't ideal cause your left using a time based cache. Using a Stale-while-revalidate cache here could be beneficial, or you could connect your data source directly your caching layer to purge cache when data changes.

I thought about purging the cache directly, however I would like to be able to use this on 3rd party cloud providers, like gatsby cloud or netlify. That's why I was hoping to have a way to tell these providers that my page is deploy based not time based on a framework level.

Anyways, thanks for your reply! Look forward to see the advances in large datasets

[wardpeet]

My project consists in an ecommerce with 11 million skus. I'm currently running on Gatsby v3 and I'm using a hybrid rendering approach where I SSG a small portion of these products during the build phase and I use client-side rendering for the rest. My current pages folder looks something like:

Feel free to email me at [email protected]. 11m SKUs sounds daunting but I'm pretty sure it's going to be okay, so I'm happy to have a chat or look at your repo to see if we can make it work.

[3200pro]

Hi Everyone,

I'm integrating DSG into pages using a field at Sanity.io that shows the page should use DSG. I'm also using the file API routing solution to generate the pages.

Sending the selection of the field from Sanity through GraphQL is fine and Gatsby is reading it, however getting the GraphQL query noticed in the DSG Config is really tripping me up. How do you either:

• Add the GraphQL to the config function
• Read the Data in the config function

Here's a working .tsx file prior to adding the reference to pageType. How would you suggest I add it in. Every process I try says 'Data undefined'

{sanitySiteRoute.slug__current}.tsx

import React from 'react'
import { graphql } from 'gatsby'
import Layout from "../components/layout"
import Seo from "../components/seo"
  
const Page = ({ data }) => {
  const { title } = data.page
return (
  <Layout>
    <Seo title="Post" />
    <h1>{title}</h1>
    
  </Layout>
)
}

export default Page 

export const query = graphql`
query($id: String) {
  page: sanitySiteRoute(id: { eq: $id }) {
  title
    slug {
      current
    }
    pageType
  }
}
`

export async function config() {
  
  const dsgPosts = true // I'd like to use a ternary here. something like: pageType === 'DSG' ? true : false
 
// Alternatively adding the graphql here would work however whenever I attempt it I get a variety of errors that are difficult to troubleshoot because they only show on a gatsby build vs. gatsby deploy

  return ({ params }) => {
    return {
      defer: dsgPosts
    }
  }
}

[moonmeister]

One of the things we were asked when discussing this feature was, "Will this help Gatsby sites scale"? I answered yes. But I also miss understood a key aspect to the feature. When DSG was finally released it was released and we were told it maintained "atomic builds". This means the only performance savings are the "query running & page generation & upload time" (from kyle on the WPGQL slack). Because DSG doesn't fetch data from the data source at render time it must still be synced at build time leading scenarios like this:

My answer is now no/sometimes. Not sure the best way to solve this, but it sure would be nice if DSG could benefit sites with large datasets.

• Full context - https://wp-graphql.slack.com/archives/CJUBND25R/p1636997995096100

[moonmeister]

Follow up thought. If getServerData Supported using the Gatsby data layer to fetch current data this wouldn't be an issue and it kind makes sense to be in SSR and not DSG. No matter what it's called, the only expectation I'd have is to be able to bust the cache on data changes. This would make DSG much closer to ISR but allow cache busting on data changes not time.

[wardpeet]

If it takes the same amount of time it means his sourcing step, createPages step is very slow; With v4 and Parallell query running, queries should run faster. I'm happy to give more information when a build log is posted.

[franciscocobas]

Hello @wardpeet thanks for responding. Here it is the log:

15:08:29 PM:Cloning into '/usr/src/app/www'...
15:08:30 PM:Found package-lock.json with lockfileversion >= 2, using node 16
15:08:30 PM:Installing node version 16
15:08:30 PM:Downloading and installing node v16.13.0...
15:08:33 PM:Now using node v16.13.0 (npm v8.1.0)
15:09:15 PM:npm WARN deprecated @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
            npm WARN deprecated @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
15:09:15 PM:npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
15:09:16 PM:npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
15:09:16 PM:npm WARN deprecated [email protected]: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
15:09:17 PM:npm WARN deprecated [email protected]: babel-eslint is now @babel/eslint-parser. This package will no longer receive updates.
15:09:17 PM:npm WARN deprecated @hapi/[email protected]: Moved to 'npm install @sideway/address'
15:09:18 PM:npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
15:09:18 PM:npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
15:09:18 PM:npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
            npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
15:09:18 PM:npm
15:09:18 PM: WARN deprecated @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
15:09:20 PM:npm WARN deprecated @hapi/[email protected]: Switch to 'npm install joi'
15:09:21 PM:npm WARN deprecated
15:09:21 PM: [email protected]: This SVGO version is no longer supported. Upgrade to v2.x.x.
15:09:40 PM: added 1982 packages in 36s
15:09:40 PM: 282 packages are looking for funding
            run `npm fund` for details
15:09:58 PM:From https://github.com/my-gtihub-url
15:09:58 PM: * branch            upgrade-to-v4 -> FETCH_HEAD
15:09:58 PM:HEAD is now at 2ee5031 Upgrade gatsby to v4
15:09:58 PM:Your branch is up to date with 'origin/upgrade-to-v4'.
15:09:58 PM:Already on 'upgrade-to-v4'
15:10:00 PM:Node version v16.13.0
15:10:01 PM:Gatsby version Gatsby CLI version: 4.1.3
15:10:01 PM:Gatsby version: 4.1.4
15:10:01 PM:  Note: this is the Gatsby version for the site at: /usr/src/app/www
15:10:01 PM:BUILDS
15:10:05 PM:success open and validate gatsby-configs, load plugins - 1.777s
15:10:06 PM:success onPreInit - 0.004s
15:10:07 PM:success delete html and css files from previous builds - 0.742s
15:10:07 PM:info One or more of your plugins have changed since the last time you ran Gatsby. As
15:10:07 PM:a precaution, we're deleting your site's cache to ensure there's no stale data.
15:10:08 PM:success initialize cache - 0.861s
15:10:08 PM:success copy gatsby files - 0.047s
15:10:08 PM:success Compiling Gatsby Functions - 0.197s
15:10:08 PM:success onPreBootstrap - 0.207s
15:10:10 PM:success [gatsby-source-wordpress] ensuring plugin requirements are met - 1.778s
15:10:12 PM:success [gatsby-source-wordpress] ingest WPGraphQL schema - 2.173s
15:10:12 PM:success createSchemaCustomization - 3.978s
15:10:13 PM:success [gatsby-source-wordpress] fetch root fields - 0.866s
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/05/jaime-saavedra-20180809ac_016_1200w.jpg is taking a long time time (longer than 15 seconds). This file is 136.57 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Lado-F.jpg is taking a long time time (longer than 15 seconds). This file is 7.36 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2020/06/linng-cardozo.jpg is taking a long time time (longer than 15 seconds). This file is 25.05 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2020/06/labat.jpg is taking a long time time (longer than 15 seconds). This file is 33.11 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2020/09/20200908-eduardo-botinelli.jpg is taking a long time time (longer than 15 seconds). This file is 40.61 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2019/02/20180814-victor-hugo-morales_0.jpg is taking a long time time (longer than 15 seconds). This file is 34.54 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/04/Alvaro-Padron.jpg is taking a long time time (longer than 15 seconds). This file is 63.59 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/11/18986.jpg is taking a long time time (longer than 15 seconds). This file is 26.4 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/11/5eaa0537337c7.jpeg is taking a long time time (longer than 15 seconds). This file is 246.44 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/11/393.jpeg is taking a long time time (longer than 15 seconds). This file is 82.83 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2020/08/200811-CANZANI.jpg is taking a long time time (longer than 15 seconds). This file is 56.62 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/11/WhatsApp-Image-2021-11-03-at-16.23.43-1.jpeg is taking a long time time (longer than 15 seconds). This file is 185.59 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/11/Captura-de-Pantalla-2021-11-03-a-las-4.54.19-p.-m..png is taking a long time time (longer than 15 seconds). This file is 1.2 MB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/11/WhatsApp-Image-2021-11-03-at-16.23.43.jpeg is taking a long time time (longer than 15 seconds). This file is 185.59 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/11/Captura-de-Pantalla-2021-11-02-a-las-6.46.02-p.-m..png is taking a long time time (longer than 15 seconds). This file is 1.21 MB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/11/151700108_191339109453561_4861249124747591525_n.jpeg is taking a long time time (longer than 15 seconds). This file is 45.14 KB
15:21:05 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/10/solosalmediodia.jpeg is taking a long time time (longer than 15 seconds). This file is 93.39 KB
15:30:10 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Tabare.jpg is taking a long time time (longer than 15 seconds). This file is 786.08 KB
15:30:17 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/VIVIR-PARA-CONTARLO-2.png is taking a long time time (longer than 15 seconds). This file is 649.98 KB
15:30:49 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/M.ferreira-10-2.jpg is taking a long time time (longer than 15 seconds). This file is 818.26 KB
15:30:49 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/20210826-Mujica-Pereira.jpg is taking a long time time (longer than 15 seconds). This file is 514.11 KB
15:30:49 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/M.ferreira-7-2.jpg is taking a long time time (longer than 15 seconds). This file is 626.86 KB
15:30:49 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/M.ferreira-6.jpg is taking a long time time (longer than 15 seconds). This file is 663.28 KB
15:30:49 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/20210826-Banco-Hipotecario.jpg is taking a long time time (longer than 15 seconds). This file is 282.96 KB
15:30:49 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/PDAHenryBorgesTokio.jpeg is taking a long time time (longer than 15 seconds). This file is 191.32 KB
15:30:49 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/20210821-Cardoso-German-1.jpg is taking a long time time (longer than 15 seconds). This file is 309.76 KB
15:30:56 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Wilmar-4.jpg is taking a long time time (longer than 15 seconds). This file is 361.4 KB
15:30:56 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/05/Isabella.jpg is taking a long time time (longer than 15 seconds). This file is 166.77 KB
15:30:56 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Rafael-Porzecanski.jpg is taking a long time time (longer than 15 seconds). This file is 471.2 KB
15:30:56 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/04/Romanelli-Lustemberg.jpg is taking a long time time (longer than 15 seconds). This file is 65.67 KB
15:30:56 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Radio-Calvelo.jpg is taking a long time time (longer than 15 seconds). This file is 332.7 KB
15:30:56 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/CODICEN.jpg is taking a long time time (longer than 15 seconds). This file is 561.07 KB
15:30:56 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2020/12/20201218_Moratorio.jpg is taking a long time time (longer than 15 seconds). This file is 28.76 KB
15:30:56 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/20210809NC_030.jpg is taking a long time time (longer than 15 seconds). This file is 69.93 KB
15:30:56 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/20210207PV_Luc.jpg is taking a long time time (longer than 15 seconds). This file is 129.21 KB
15:31:10 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/LUBO-ADUSTO-2-2.png is taking a long time time (longer than 15 seconds). This file is 751.62 KB
15:31:18 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Pescadere.jpg is taking a long time time (longer than 15 seconds). This file is 157.49 KB
15:31:35 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/20200303JC_0482.jpg is taking a long time time (longer than 15 seconds). This file is 132.04 KB
15:31:35 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/natural-laura-falero.jpg is taking a long time time (longer than 15 seconds). This file is 346.48 KB
15:31:43 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2020/09/20200923_Padrón.jpg is taking a long time time (longer than 15 seconds). This file is 63.66 KB
15:32:07 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/LUBO-ADUSTO-8.png is taking a long time time (longer than 15 seconds). This file is 753.89 KB
15:32:07 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/mirate.png is taking a long time time (longer than 15 seconds). This file is 398.63 KB
15:32:07 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/espa.png is taking a long time time (longer than 15 seconds). This file is 348.48 KB
15:32:07 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/20210318JC_0462.jpg is taking a long time time (longer than 15 seconds). This file is 104.01 KB
15:32:13 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Mauro-Alvares.jpg is taking a long time time (longer than 15 seconds). This file is 175.02 KB
15:32:13 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/20190423JC_0073.jpg is taking a long time time (longer than 15 seconds). This file is 84.44 KB
15:32:13 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Bonomi-Calvelo.jpg is taking a long time time (longer than 15 seconds). This file is 409.24 KB
15:32:13 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/RAN20110526-003.jpg is taking a long time time (longer than 15 seconds). This file is 906.18 KB
15:32:13 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/01/15012021madurorendiciondecuentas.jpg is taking a long time time (longer than 15 seconds). This file is 40.77 KB
15:32:13 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/06/Daniel-Larrosa-Montevideo-Portal.jpg is taking a long time time (longer than 15 seconds). This file is 152.68 KB
15:32:13 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/20210820-spanish.jpg is taking a long time time (longer than 15 seconds). This file is 238.11 KB
15:32:42 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Tribunal-de-Cuentas-informe-Ale.jpg is taking a long time time (longer than 15 seconds). This file is 351.59 KB
15:32:42 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/06/20210614-Ejercito-Nacional.jpg is taking a long time time (longer than 15 seconds). This file is 535.38 KB
15:32:42 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/PDAJulianaCastro.jpg is taking a long time time (longer than 15 seconds). This file is 113.88 KB
15:32:42 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Manson.jpeg is taking a long time time (longer than 15 seconds). This file is 172.88 KB
15:32:42 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/07/PDAGorroTokio1.jpg is taking a long time time (longer than 15 seconds). This file is 115.08 KB
15:32:50 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/Ollas-Federico-Gutierrez.jpg is taking a long time time (longer than 15 seconds). This file is 416.82 KB
15:32:50 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/LUBO-ADUSTO-7.png is taking a long time time (longer than 15 seconds). This file is 675.67 KB
15:32:50 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/dudas-razonables-3.png is taking a long time time (longer than 15 seconds). This file is 537.4 KB
15:35:00 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/08/PDATokio_Vela1_DSC04571-2.jpg is taking a long time time (longer than 15 seconds). This file is 549.77 KB
15:36:20 PM:[gatsby-source-wordpress] Fetching https://my-wordpress-url-site.com/wp-content/uploads/2021/07/IDEA-DE-MIERCOLES.png is taking a long time time (longer than 15 seconds). This file is 933.11 KB
16:07:15 PM:success [gatsby-source-wordpress] creating nodes - 2792.851s
16:07:15 PM:success [gatsby-source-wordpress] fetching nodes - 3422.344s - 22424 total
16:07:15 PM:success Downloading remote files - 2785.297s - 6623/6623 2.38/s
16:07:16 PM:success Checking for changed pages - 0.000s
16:07:16 PM:success source and transform nodes - 3423.570s
16:07:17 PM:info Writing GraphQL type definitions to /usr/src/app/www/.cache/schema.gql
16:07:17 PM:success building schema - 1.444s
16:07:22 PM:success createPages - 4.206s
16:07:22 PM:success createPagesStatefully - 0.136s
16:07:22 PM:info Total nodes: 35709, SitePage nodes: 7041 (use --verbose for breakdown)
16:07:22 PM:success Checking for changed pages - 0.001s
16:07:22 PM:success Cleaning up stale page-data - 0.134s
16:07:22 PM:success onPreExtractQueries - 0.001s
16:07:25 PM:success [gatsby-source-wordpress] ingest WPGraphQL schema - 3.124s
16:07:29 PM:success extract queries from components - 7.201s
16:07:29 PM:success onPostBootstrap - 0.001s
16:07:29 PM:success write out redirect data - 0.004s
16:07:29 PM:info bootstrap finished - 3448.397s
16:07:29 PM:success write out requires - 0.026s
16:07:57 PM:warning ./node_modules/gatsby-source-wordpress/dist/gatsby-browser.js
16:07:57 PM:warning ./node_modules/gatsby-source-wordpress/dist/gatsby-browser.js
16:07:57 PM:Attempted import error: 'createRoot' is not exported from 'react-dom' (imported as 'ReactDOM').
16:07:57 PM:Attempted import error: 'createRoot' is not exported from 'react-dom' (imported as 'ReactDOM').
16:07:57 PM:success Building production JavaScript and CSS bundles - 27.542s
16:09:02 PM: [webpack.cache.PackFileCacheStrategy] Serializing big strings (1233kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)
16:09:02 PM: [webpack.cache.PackFileCacheStrategy] Serializing big strings (1233kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)
16:09:02 PM: [webpack.cache.PackFileCacheStrategy] Serializing big strings (1233kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)
16:09:10 PM:success Building Rendering Engines - 73.606s
16:09:22 PM:success Building HTML renderer - 11.293s
16:09:22 PM:success Execute page configs - 0.092s
16:09:25 PM:success [gatsby-source-wordpress] ingest WPGraphQL schema - 0.809s
16:09:25 PM:success Validating Rendering Engines - 3.644s
16:09:25 PM:success Caching Webpack compilations - 0.001s
16:09:47 PM:success run queries in workers - 21.353s - 644/644 30.16/s
16:10:52 PM:success Merge worker state - 0.033s
16:10:54 PM:success Rewriting compilation hashes - 0.001s
16:10:54 PM:success Writing page-data.json files to public directory - 0.385s - 641/641 1663.73/s
16:10:56 PM:success Building static HTML for pages - 3.495s - 641/641 183.41/s
16:10:56 PM:warning [gatsby-source-wordpress] There were 2790 files with file sizes that are above the maxFileSizeBytes config option and consequently were not fetched. Visit https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-wordpress/docs/debugging-and-troubleshooting.md#media-file-download-skipped for more info.
16:10:56 PM:warning [gatsby-source-wordpress] There were 3225 files with mime types that were included in the excludeByMimeTypes config option and consequently were not fetched. Visit https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-wordpress/docs/debugging-and-troubleshooting.md#media-file-download-skipped for more info.
16:11:00 PM:success onPostBuild - 3.583s
16:11:00 PM:Pages
16:11:00 PM:│ └   /agendacultural
16:11:00 PM:│ └   ...16 more pages available
16:11:00 PM:│ └   ...599 more pages available
16:11:00 PM:├ src/templates/events.tsx
16:11:00 PM:├ src/templates/post.tsx
16:11:00 PM:┌ src/templates/event.tsx
16:11:00 PM:│ ├   /agendacultural/juan-carlos-baglietto-y-lito-vitale-30-anos
16:11:00 PM:│ ├   /clientes-declarados-por-kirma-niegan-vinculos-con-la-firma-estonia-contratada-por-german-cardoso
16:11:00 PM:├ src/templates/post-no-image.tsx
16:11:00 PM:│ └ D ...6399 more pages available
16:11:00 PM:│ ├ D /la-victoria-electoral-socialdemocrata-en-alemania-es-un-cambio-relativo-porque-alli-no-hay-nada-radical-en-la-izquierda-con-chances-de-triunfar
16:11:00 PM:│ └   /a-dos-orillas/
16:11:00 PM:│ └   /basquetbol/
16:11:00 PM:│ └   /404.html
16:11:00 PM:├ src/pages/a-dos-orillas.tsx
16:11:00 PM:├ src/pages/alquimia.tsx
16:11:00 PM:├ src/pages/404.tsx
16:11:00 PM:│ └   /alquimia/
16:11:00 PM:├ src/pages/basquetbol.tsx
16:11:00 PM:│ ├   /404/
16:11:00 PM:│ └   /carnaval/
16:11:00 PM:│ └   /el-menu-del-dia/
16:11:00 PM:├ src/pages/carnaval.tsx
16:11:00 PM:├ src/pages/info24.tsx
16:11:00 PM:│ └   /
16:11:00 PM:│ └   /hablando-al-sur-jose-mujica/
16:11:00 PM:├ src/pages/index.tsx
16:11:00 PM:├ src/pages/hablando-al-sur-jose-mujica.tsx
16:11:00 PM:│ └   /columnistas/
16:11:00 PM:├ src/pages/el-menu-del-dia.tsx
16:11:00 PM:├ src/pages/columnistas.tsx
16:11:00 PM:│ └   /info24/
16:11:00 PM:├ src/pages/mejor-hablar.tsx
16:11:00 PM:├ src/pages/mis-notas.tsx
16:11:00 PM:├ src/pages/maldonado.tsx
16:11:00 PM:│ └   /informe-granjero/
16:11:00 PM:│ └   /mejor-hablar/
16:11:00 PM:├ src/pages/informe-granjero.tsx
16:11:00 PM:│ └   /maldonado/
16:11:00 PM:│ └   /internacionales/
16:11:00 PM:│ └   /mis-notas/
16:11:00 PM:├ src/pages/internacionales.tsx
16:11:00 PM:├ src/pages/otras-maneras.tsx
16:11:00 PM:│ └   /otras-maneras/
16:11:00 PM:├ src/pages/por-decir-algo.tsx
16:11:00 PM:│ └   /todo-por-la-misma-plata/
16:11:00 PM:├ src/pages/todo-por-la-misma-plata.tsx
16:11:00 PM:├ src/pages/tapalo-con-radio.tsx
16:11:00 PM:├ src/pages/ultimo-cuarto.tsx
16:11:00 PM:│ └   /ultimo-cuarto/
16:11:00 PM:│ └   /noticias/
16:11:00 PM:├ src/pages/noticias.tsx
16:11:00 PM:│ └   /por-decir-algo/
16:11:00 PM:│ └   /tapalo-con-radio/
16:11:00 PM:└ src/pages/universos-paralelos.tsx
16:11:00 PM:  └   /universos-paralelos/
16:11:00 PM:  ╭────────────────────────────────────────────────────────────────────╮
16:11:00 PM:  │     (SSG) Generated at build time                                  │
16:11:00 PM:  │   D (DSG) Deferred static generation - page generated at runtime   │
16:11:00 PM:  │   λ (Function) Gatsby function                                     │
16:11:00 PM:  ╰────────────────────────────────────────────────────────────────────╯
16:11:00 PM:  │                                                                    │
16:11:00 PM:  │                                                                    │
16:11:00 PM:  │   ∞ (SSR) Server-side renders at runtime (uses getServerData)      │
16:11:00 PM:info Done building in 3659.293220605 sec
16:11:23 PM:Uploading build results
16:11:52 PM:Done uploading build results

[moonmeister]

If it takes the same amount of time it means his sourcing step, createPages step is very slow; With v4 and Parallell query running, queries should run faster. I'm happy to give more information when a build log is posted.

My understanding is the slow step for him is when the data layer ingests data, not when data is requested from the DL. It's the latter that parallel query running improves, correct?

[wardpeet]

it's the latter yes. @franciscocobas your slowness happens because of slow file downloads. We're working on something and should have an experiment version in January so reduce the cost of downloading files.

[tsdexter]

I would like to use DSG to speed up build times however I'm confused about exactly how it works. When the page is being built with DSG does the build server go back to my WordPress GraphQL endpoint and fetch the data or does it just get it from the cached Gatsby data layer?

In other words, if I use DSG for every page, is gatsby-source-wordpress still going to call my graphql endpoint for all of the site data during a full build? This step alone is often timing out my builds even on an enterprise wp engine server. I would ideally like to have something set up where even a full, clean build does not need to source every piece of data from wordpress...is this possible?

[moonmeister]

Yes, DSG would fetch all data from WP. SSR would be the only way to fully deffer fetching of data.

[flogy]

Is it possible to somehow invalidate the cache after a periodic time interval? For example a page that displays the current day's weather could be rendered using DSG instead of SSR but should be regenerated on every request after midnight. Did somebody do something like this before or do you just use SSR or call the weather Api from the client for such use cases?

[iamakulov]

Hi 👋

This is less of a question and more of a feature request, but it would be great to make getServerData() compatible with DSR.

Our use-case:

• We have a page rendered with SSR

• In getServerData() for that page, we query several API endpoints, collect the data, and use it to render the page

• We tried switching the page to DSG to reduce the server response time (= build the page only on the first request instead of on every one)

• However, adding

  export async function config() {
    return ({ params }) => {
      return {
        defer: true,
      }
    }
  }

  at that page did nothing as getServerData() took precedence over defer: true

• We can’t remove getServerData() as we need to fetch the page data from third-party SDKs. Moving that data to a GraphQL layer would require a lot of work

We ended up using Cache-Control: s-maxage=86400, stale-while-revalidate to optimize the server response time, but the fact that getServerData() didn’t work with defer: true was pretty unintuitive.