Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

state.buffer[0].length Cannot read properties of null (reading 'length') #2368

Closed
gal-checksum opened this issue Nov 23, 2023 · 19 comments · Fixed by #2422
Closed

state.buffer[0].length Cannot read properties of null (reading 'length') #2368

gal-checksum opened this issue Nov 23, 2023 · 19 comments · Fixed by #2422
Assignees
Labels
api: storage Issues related to the googleapis/nodejs-storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@gal-checksum
Copy link

Environment details

  • OS: Mac 13.5 (22G74)
  • Node.js version: 21.2.0
  • npm version: 10.2.3
  • @google-cloud/storage version: 7.6.0

Getting the following error when trying to download a file:

source-map-support.js:496 node_modules/stream-shift/index.js:16
    return state.buffer[0].length
  source-map-support.js:499 TypeError: Cannot read properties of null (reading 'length')
    at getStateLength (node_modules/stream-shift/index.js:16:28)
    at shift (node_modules/stream-shift/index.js:6:99)
    at Duplexify._forward (node_modules/duplexify/index.js:170:35)
    at PassThrough.onreadable (node_modules/duplexify/index.js:136:10)
    at PassThrough.emit (node:events:519:28)
    at emitReadable_ (node:internal/streams/readable:832:12)
    at processTicksAndRejections (node:internal/process/task_queues:81:21)

This only happens when I download the file from a certain project. If I copy the exact same config and code but initiate it within a script, it works

@gal-checksum gal-checksum added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Nov 23, 2023
@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/nodejs-storage API. label Nov 23, 2023
@unifractal
Copy link

I am seeing the same problem. But for me it seems to be an issue with most versions, not just 7.6.0. I have tested all the way back to 6.12 and it is broken for me

@gal-checksum
Copy link
Author

Has to do with this:
mafintosh/stream-shift#6

Downgrade your node.js version to below 21.2

@ddelgrosso1
Copy link
Contributor

@unifractal can you confirm you were also using node 21.2+?

@ddelgrosso1 ddelgrosso1 self-assigned this Nov 27, 2023
@PatricioPoncini
Copy link

Hi, I have the same problem. I cannot upload files larger than approximately 70KB. Does it work if I downgrade my Node.js version, or is it a library issue?

@unifractal
Copy link

unifractal commented Nov 27, 2023

Yes, i can confirm. I was also on node 21.2+ and downgrading to e.g. Node version LTS 18 fixed my problems.

Definitely a problem caused when upgrading node. It just sucks when it happens automatically using e.g. Google Cloud run 😂

@Idnan
Copy link

Idnan commented Nov 27, 2023

I was also having the same issue with NodeJS 21.2+ and downgrading to NodeJS 21.1+ fixed the issue.

@PatricioPoncini
Copy link

PatricioPoncini commented Nov 27, 2023

It doesn't work for me, even after downgrading Node.js to 21.1. I'm using Bun. Could it be an issue with Bun and this library? I posted a question; it's my first time using this library with Bun. My problem is when I want to download a file from my bucket

#2370

@ddelgrosso1
Copy link
Contributor

For those having problems on Node 21.2 it appears the solution is to downgrade to get around the problem. Looking at the provided stack trace it appears to mainly be coming from Node internal libs / Duplexify. It might be worth opening an issue for the Duplexify folks to see if they can resolve it on their side. We do intend to remove Duplexify from this library in the future (#2041) However, that will not happen until the next major revision to the library which is likely sometime next year.

@jontybrook
Copy link

Ditto this. Downgrading to node v20.10 resolved fixed it.

@drichardson
Copy link

FYI I ran into this with current LTS of node (20.11.0). I worked around it by removing the {destination: my_output} option from the download function and writing it to a file myself.

@mpalmerlee
Copy link

I experienced the same issue with Node 20 although my error message was slightly different:

TypeError: Cannot read properties of null (reading 'length')
    at getStateLength (/prod/api/node_modules/.pnpm/[email protected]/node_modules/stream-shift/index.js:16:28)
    at shift (/prod/api/node_modules/.pnpm/[email protected]/node_modules/stream-shift/index.js:6:99)
    at Duplexify._forward (/prod/api/node_modules/.pnpm/[email protected]/node_modules/duplexify/index.js:170:35)
    at PassThrough.onreadable (/prod/api/node_modules/.pnpm/[email protected]/node_modules/duplexify/index.js:136:10)
    at PassThrough.emit (node:events:518:28)
    at emitReadable_ (node:internal/streams/readable:832:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)

To work around I changed my code from:
await this.storage.bucket(config.gcp.storageBucket).file(gcsPath).download(options);

To:
await this.transferManager.downloadFileInChunks(gcsPath, options);

Like in this example downloadFileInChunksWithTransferManager.js

@russell-liu-splyt
Copy link

stream-shift 1.0.2 works fine with node 20.11.0

@mksony
Copy link

mksony commented Jan 25, 2024

stream-shift 1.0.2 works fine with node 20.11.0

Can confirm that 💯 we had to add an override to our package.json to enforce stream-shift 1.0.2.

@MilosLukicBM
Copy link

MilosLukicBM commented Jan 25, 2024

Can also confirm that setting stream-shift to 1.0.2 for all dependencies (used the overrides method) works with node 20.11.0 🎉

@MattBidewell
Copy link

MattBidewell commented Jan 29, 2024

Those who might end up here whilst running a Google Cloud Function, theres a chance the underlying image of your cloud function has been updated and this bug has manifested itself!

For the time being, you can add the following to your package.json to get around the issue the storage API is updated:

JSON

  "overrides": {
    "stream-shift": "1.0.2"
  }

Worth noting that stream-shift is a dependency of duplexify which appears unmaintained...

@majikandy
Copy link

majikandy commented Jan 29, 2024

Can also confirm this works in the package json to override to 1.0.2 using the overrides block above.

Using the functions-framework to run the function locally doesn't cause the stream-shift TypeError: Cannot read properties of null (reading 'length'), it only seems to happen when it runs in the cloud so we only became aware on a rebuild/deploy.

@danielduhh
Copy link

Thanks for the investigation!
Please upgrade to stream-shift 1.0.2 (a dependency of duplexify in this repo). This will fix the error. We have plans to move away from duplexify altogether -- you can track that here: #2041

@npomfret
Copy link

Had the problem with node v20.11, downgraded to v20.10 and the problem went away. Thank you.

@undesicimo
Copy link

downgrading to 20.10 seems to work for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/nodejs-storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.