Libsass/Too many files error on big sites when running server on MacOS

[FuadEfendi]

Error: error building site: TOCSS: failed to transform

What version of Hugo are you using (hugo version)?

0.128.1
0.128.2

Does this issue reproduce with the latest release?

Yes

Please see discussion at https://discourse.gohugo.io/t/hugo-embedded-web-server-limitations-error-tocss/50543 and https://github.com/orgs/hbstack/discussions/68

This is test site with randomly generated 40,000 files:
https://github.com/tokenizer-inc-canada/large-site-test

hugo server command fails with Error: error building site: TOCSS: failed to transform ... error message, and hugo command fails with different error message.

Another user, @razonyang tried on Windows, and it doesn't fail.

I tried on MacBook M1 Pro with 64G, and on MacBook Air M1, and it fails on both, tried with 0.127.0, 0.128.1, 0.128.2.

If I randomly delete any 10,000 files, build doesn't fail.

With configured "segments" feature, when I try to build small subset, build still fails, and fails in faster time.

[bep]

I have briefly looked at your site, and generating 40K content files with lots of random taxonomy values is blowing up the number of files, and is not what the taxonomies were designed for:

If you can create a more realistic breaking example, I can have a look at it, but I'm not going to fill my disk/CPU debugging this crazy stuff.

Please reopen if you have something more real.

[FuadEfendi]

I have few public websites (with average 35k -50k pages) having this issue on Mac (and not having with Netlify build). Finally I suspected that it cannot be malformed file, tried to reproduce, and reproduced.

[FuadEfendi]

I started encountering this error a few weeks ago and did a lot of research and spent many hours on it:

Error: error building site: TOCSS: failed to transform

Now, we documented it which is a good thing ;)

No need to waste time on it, I agree, it may unpredictably disappear in some new versions.

However, unfortunately, building by 20x smaller segments also fails with same error message... in Linux, I'd assume that ulimit nofile should be increased; not sure about Mac; could be internal bug in Mac.

[bep]

OK, I can reproduce this.

The odd thing is that even

hugo mod vendor
hugo server

Fails with ...

File to import not found or unreadable: /Users/bep/dump/large-site-test/_vendor/github.com/twbs/bootstrap/scss/_functions.scss.

This doesn't make sense, but it's certainly easier to reason about when having the deps vendored.

[FuadEfendi]

I can imagine weird path being generated by broken platform-specific tools, such as ../abc/../abc/../abc/.. of very large size, and we have file not found error; some internal tools are platform specific, in case of SCSS for example, SAAS compiler; maybe need to run npm audit fix etc.; something is platform-specific.

[ShaunaGordon]

I think I'm running into this on my M1, as well. I'm working on the docs for O3DE, which works on my Linux machine, and builds with hugo on my M1, but hugo server fails with the TOCSS error and the fix we've documented in other contexts doesn't apply. It only has about 2200 pages, so it's not a size issue, I don't think.

[FuadEfendi]

In my specific case, I was trying to "isolate" issue and found even exact number of files / threshold after which it happens (but I lost this number)

Can you check please your total number of files (including pagination and aliases)?

[FuadEfendi]

I tried O3DE too, number of files above 40,000; "threshold" which I found was around 37,000

                   |  EN    
-------------------+--------
  Pages            |  2215  
  Paginator pages  |     5  
  Non-page files   |    17  
  Static files     | 44236  
  Processed images |     0  
  Aliases          |     2  
  Cleaned          |     0  

Total in 16016 ms

I tried hugo server too, similar error. My only guess is that some NPM module has bug in it, and it is platform-specific module. For example, SCSS compiler. Or, maybe bug in Hugo "HTTP Server" implementation; really hard to track and who knows it may suddenly disappear in future release.

[FuadEfendi]

I deleted folders from "a" to "h" in o3de.org/static/docs/api/gems and hugo server starts successfully. Otherwise weird nonrelated error message TOCSS: failed to transform "blah.sass" (text/x-sass): "/Users/fefendi/java/web/o3de.org/assets/sass/custom.sass:28:1": File to import not found or unreadable: bootstrap/scss/functions.

(and I tried to set explicitly ulimit -n 1000000 in iTerm2)

[ShaunaGordon]

It's definitely something different within Hugo, but that's apparently platform specific. I pulled Hugo version 0.105.0 the other day (what the O3DE docs are currently built on; yeah, I know it's ancient, I currently have red tape in my way), and it works fine and builds nice and fast.

I'm going to toy around with Hugo versions, so I can at least narrow down what version triggers it, though I'm wondering if it's something to do with the lib/dart Sass handling changes that've happened over the past year or so.

[bep]

OK, sorry for the late follow up, I have some time tomorrow and will have another look at this mystery error.

[ShaunaGordon]

Okay, so I think I found the issue, at least enough to point you in the right direction to fix it.

I ran through old versions and found that, at least for my setup, 0.115.3 is the latest version that hugo server works without error for the O3DE docs. 0.115.4 is where I start running into the toCSS error. I've been able to successfully build with just hugo every time I've tried, even on 0.131.0.

Strangely, if I replicate what Fuad did, and delete A-H from the static/docs/api/gems folder in my repo, hugo server runs fine and doesn't give me the error, which I find particularly weird, because those are explicitly static files, so I wouldn't expect Hugo to do anything more than simply copy them over.

To try to further isolate things, I tossed the api folder into my blog repo, which doesn't use Bootstrap, but a fork of the Dream/Dream Plus theme, which imports its big dependencies from CDNs, and uses vanilla CSS on the local side. That pushed its file count well above 40k, and I'm able to serve it just fine, as demonstrated here:

I also toyed around with adding a toCSS function (copied from the docs) into a partial on my blog, and even with 40k files, it also serves fine. This says to me that the issue is deeper, but not OS-level deep.

Something I noticed was that the docs are explicitly setting "transpiler" "dartsass" in its options for toCSS, so I updated my O3DE template to do the same and lo and behold (after fixing the site's Sass file of stupid syntax errors), it serves! (And as a sanity check, I removed the transpiler option, but kept the syntax fixes, and that failed with the toCSS error).

Okay, so that must mean it's a libsass thing, right? Well...not quite. Every version of Hugo I'm using is using the same version of libsass, so if it was just a libsass thing, then the behavior wouldn't change across Hugo versions.

That means it's likely an interaction between Hugo and libsass.

In looking at what did change between 115.3 and 115.4, I found this change, which seems at least from a cursory look through the release notes, to be the most likely culprit, since it's the only one that appears to be dealing with file pathing. However, I'm not at all familiar with the details of the inner workings of Hugo, so it's entirely possible that it was either an unrelated change, or one that didn't make it into the release notes. Regardless, I hope this helps pinpoint the root cause.

[bep]

OK, I finally found some time to get to the bottom of this. After digging into the C++ code in Libsass I found that it fails to open with the error code meaning Too many open files.

This is related to Hugo's file watcher on MacOS.

Libsass is deprecated so I'm not sure I'm going to spend time patching it to improve the error handling, I'll think about this.

There are some OS settings related to getting rid of the "Too many open files", but my recommended workaround is to do:

hugo server --poll 5s

Or something for these biiiig sites.

[FuadEfendi]

I tried to run in separate terminal, with ulimit -n 1000000 and it didn't help; maybe just need to set default profile setting for MacOS

On Mac, default is 256; on Linux, 1024 (?); and it seems default setting on Linux works (I didn't test on Linux)

But then, we can reproduce the same issue on Linux, by setting ulimit -n 256 before executing hugo server

[ShaunaGordon]

@bep Did you see any indication for why newer versions of Hugo fail with the error, but older ones don't, even when they all report the same version of libsass?

Also, why are static files being opened (or at least "opened" as far as libsass and the OS are concerned) during this processing?

[jmooring]

Given that this problem seems to be related to LibSass, I can't think of a good reason to spend any time on it. LibSass was EOL'd 4 years ago.

[bep]

@ShaunaGordon This issue is not introduced by a code change in Hugo or Hugo's library, I double checked that. My best guess is that MacOS has changed its fread implementation. But ... I'm closing this; there's not much we can do about this with our current limited budget.

[FuadEfendi]

ulimit -n definitely plays role here; default on Mac is 256, but I set it to 2,000,000

Now, with "smaller" site I don't have any issue:

Start building sites … 
hugo v0.133.0+extended darwin/arm64 BuildDate=2024-08-17T19:57:41Z VendorInfo=brew


                   |  EN    
-------------------+--------
  Pages            | 61418  
  Paginator pages  |   927  
  Non-page files   |     0  
  Static files     |     3  
  Processed images |     9  
  Aliases          | 23578  
  Cleaned          |     0  

Built in 130317 ms
Environment: "development"
Serving pages from disk

Very fast, great improvements!

[FuadEfendi]

The only issue with SASS was that it doesn't correctly report underlying error: it must be too many open files instead of File to import not found or unreadable

(@razonyang - to confirm, I use github.com/hbstack/hb v0.12.4 module in this test)

[bep]

@FuadEfendi I reopened this; I guess this issue is annoying enough and very hard to debug for end users to warrant a one-line patch in libsass, so I have added this in bep/golibsass#12 which I will eventually pull into Hugo.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.