[feat]: upgrade to mongodb driver 6 #499

DominusKelvin · 2023-09-30T09:35:33Z

This PR is now ready for merging

This is gonna be minor release of sails-mongo i.e v2.1.0

I am adding this so the broader Sails community using MongoDB can pitch in

This PR when landed will close issue #7275 on the sails repo

sailsbot · 2023-09-30T09:35:35Z

Thanks for submitting this pull request, @DominusKelvin! We'll look at it ASAP.

In the mean time, here are some ways you can help speed things along:

discuss this pull request with other contributors and get their feedback. (Reactions and comments can help us make better decisions, anticipate compatibility problems, and prevent bugs.)
ask another JavaScript developer to review the files changed in this pull request. (Peer reviews definitely don't guarantee perfection, but they help catch mistakes and enourage collaborative thinking. Code reviews are so useful that some open source projects require a minimum number of reviews before even considering a merge!)
if appropriate, ask your business to sponsor your pull request. (Open source is our passion, and our core maintainers volunteer many of their nights and weekends working on Sails. But you only get so many nights and weekends in life, and stuff gets done a lot faster when you can work on it during normal daylight hours.)
make sure you've answered the "why?" (Before we can review and merge a pull request, we feel it is important to fully understand the use case: the human reason these changes are important for you, your team, or your organization.)

Please remember: never post in a public forum if you believe you've found a genuine security vulnerability. Instead, disclose it responsibly.

For help with questions about Sails, click here.

lennyAiko · 2023-09-30T14:01:00Z

Hi @DominusKelvin

Happy to work on this with you. Do you have any current task to delegate? Or if you have series of steps to follow so I can hop on one of them.

I ask this in order to avoid future merge conflicts

luislobo · 2023-10-01T04:28:44Z

@DominusKelvin I've been spending a lot of time working on first cleaning up sails-mongo code to more current one, removing travis and appveyor and using github actions.

We have been talking with @Josebaseba (who has been a long time maintainer/contributor, like me) to sails-mongo.

We are all wasting time working on the same things in parallel.

Please check: #498

@mikermcneil We really need to get together and have a real action plan here.

luislobo · 2023-10-01T04:29:27Z

@DominusKelvin The goals is: once that PR is approved, it would be very simple to just migrate to 6 and have good code coverage in mongodb versions for both Linux and Windows as CI covers all combinations.

DominusKelvin · 2023-10-01T08:43:37Z

Hey @luislobo great work on the PR you are working on

I wish we've seen it earlier but I guess it sort of fall through the crack for us.

I notice a couple of things like that you've changed like updating var to let let/const. Really cool

Looking at something like create-record I think it's not quite trivial
to migrate as insertOne no longer returns the record that was created.

To move forward I think you, myself and, any other MongoDB user can see how to move forward on this as there are quite a lot of breaking changes we've got to consider

DominusKelvin · 2023-10-01T09:20:58Z

Also @luislobo and @Josebaseba if you want, we can have a Discord call in the Sailscasts community to talk about this and moving forward.

From the core team meeting last Thursday, @mikermcneil and myself have discussed prioritizing moving sails-mongo to the latest version of the mongodb driver so I think we can all move forward with this PR as there some contextual changes needed as well.

Let me know what you think.

Here is a link to the community

DominusKelvin · 2023-10-01T09:28:47Z

So one blocker I noticed(so far) is .insertOne no longer return the document inserted because it's complying to MongoDB's CRUD Result spec

So the options we have is to do a .findOne(this is what Mongoose ODM is doing) here

…a find

DominusKelvin · 2023-10-01T11:20:21Z

So I have moved create-record and create-each-record to call findOne and find respectively to respect the contract usage when a user pass in fetch: true as a meta option or call .fetch() in userland Waterline query

DominusKelvin · 2023-10-01T11:21:42Z

I will have to move .count() from the current implementation since cursor.count has been deprecated in versions of MongoDB driver later than 3.x. It will also fix this warning ⚠️ 👇🏾

DominusKelvin · 2023-10-01T14:02:21Z

Currently dealing with this error

luislobo · 2023-10-01T14:29:53Z

I can join in about an hour or so. I was already on SailsCast discord, just not active daily.

luislobo · 2023-10-01T14:35:12Z

Hey @luislobo great work on the PR you are working on

Thanks, I'd love to do pair review, merge and then work on the other things. This PR changes code everywhere since it changes var usage, so it can become a merge nightmare if we continue coding under the current master.

I wish we've seen it earlier but I guess it sort of fall through the crack for us.

I notice a couple of things like that you've changed like updating var to let let/const. Really cool

Looking at something like create-record I think it's not quite trivial to migrate as insertOne no longer returns the record that was created.

It is my understanding that current Waterline version works also like that, unless you chain it with fetch.

To move forward I think you, myself and, any other MongoDB user can see how to move forward on this as there are quite a lot of breaking changes we've got to consider

I'm down to it.

luislobo · 2023-10-01T14:36:43Z

So one blocker I noticed(so far) is .insertOne no longer return the document inserted because it's complying to MongoDB's CRUD Result spec

So the options we have is to do a .findOne(this is what Mongoose ODM is doing) here

Should only be done if "fetch" is called in a chain

https://sailsjs.com/documentation/reference/waterline-orm/models/create#?result

luislobo · 2023-10-01T14:38:01Z

BTW, to note: my PR also switches to GitHub Actions, removing AppVeyor (which is failing here) and Travis. And in the process, I've added test for MongoDB 4.4 (which is still LTS until Feb next year), 5.x, 6.x and 7.x

DominusKelvin · 2023-10-01T14:55:43Z

Got you @luislobo i think for now we can prioritize moving the codebase to support v6 of the adapter and also change let/const for the files we touch plus the infra things you mentioned can be added as well.

If you want we can pair right now to try fix some things I've noticed

Pair on Discord?

DominusKelvin · 2023-10-01T14:58:45Z

I can join in about an hour or so. I was already on SailsCast discord, just not active daily.

What's your handle so I ping you and we hop on a call together

luislobo · 2023-10-01T16:29:52Z

I'm back. I think I'm cromo over there

mikermcneil · 2023-11-22T18:47:54Z

lib/private/machines/create-each-record.js

@@ -105,15 +106,19 @@ module.exports = {
      //  ╠═╝╠╦╝║ ║║  ║╣ ╚═╗╚═╗  │││├─┤ │ │└┐┌┘├┤   ├┬┘├┤ │  │ │├┬┘ │││ └─┐ │
      //  ╩  ╩╚═╚═╝╚═╝╚═╝╚═╝╚═╝  ┘└┘┴ ┴ ┴ ┴ └┘ └─┘  ┴└─└─┘└─┘└─┘┴└──┴┘└─└─┘─┘
      // Process record(s) (mutate in-place) to wash away adapter-specific eccentricities.


^ This comment makes more sense down below above the try/catch when we process records.

And instead, before that, we need a new comment explaining what we're doing in terms of fetching these records.

Minor thing, but I misunderstood when first reading so others are likely to as well (this'll prevent future confusion)

mikermcneil · 2023-11-22T18:49:53Z

lib/private/machines/create-manager.js

-    // Use unified topology. MongoDB node maintainers recommends this to be enabled
-    // https://github.com/mongodb/node-mongodb-native/releases/tag/v3.2.1
-    // Use new url parser to remove warnings
-    _clientConfig = Object.assign({


I wonder if this could be related to the ? error @eashaw mentioned (no real reason why I'm suggesting that, other than at a glance, I'm noticing useNewUrlParser, so maybe related?)

mikermcneil · 2023-11-22T18:51:04Z

lib/private/machines/create-record.js

-      try {
-        processNativeRecord(phRecord, WLModel, s3q.meta);
-      } catch (e) { return exits.error(e); }
+      (function(iifeDone) { mongoCollection.findOne({ _id: nativeResult.insertedId }).then(function(phRecord) { iifeDone(undefined, phRecord);}).catch(function(err) { iifeDone(err);});})(function(err, phRecord) {


same as https://github.com/balderdashy/sails-mongo/pull/499/files#r1402546571

On the useNewUrlParser @mikermcneil that's no longer a supported config option in the newer version of the MongoDB driver. @eashaw can you confirm if that's the case for you?

mikermcneil · 2023-11-22T18:53:09Z

lib/private/machines/create-each-record.js

@@ -97,6 +97,7 @@ module.exports = {
        return exits.success();
      }//-•

+      var insertedIds = Object.values(nativeResult.insertedIds);



I think we should paste in (and slightly tweak) the assertion from here: https://github.com/balderdashy/sails-mongo/pull/499/files#diff-962fecaf7496f0dc0f4c49842e1e52f7035018902cd4530b86833883185c0166L106

that way we'll know if something changes in Mongo in the future and keep things just as easy to follow for migrating to future mongo driver releases

You are right @mikermcneil. I just added the assertions and tweaked it to match what we are checking.

package.json

mikermcneil

Everything looks good to me other than comments above.

@eashaw is looking at a failing test (and a second look at the create-manager function re: compatibility), but then assuming he doesn't run into anything else, I think this is ready to merge!

Thanks @DominusKelvin!

DominusKelvin · 2023-11-23T09:58:55Z

Everything looks good to me other than comments above.

@eashaw is looking at a failing test (and a second look at the create-manager function re: compatibility), but then assuming he doesn't run into anything else, I think this is ready to merge!

Thanks @DominusKelvin!

Thanks @mikermcneil. I think @eashaw has made a pass with the suggestions you made. @eashaw does the test still fails for you after you've made the changes? If not then I think we are definitely at the end of getting this merged.

Thanks @eashaw for the changes made.

DominusKelvin · 2023-11-23T10:17:56Z

@DominusKelvin So much easier to review. Thank you!

I am glad it is :)

…4 localhost address

DominusKelvin · 2023-12-05T07:57:34Z

Great job working on the CI passing @eashaw 🥳

luislobo · 2023-12-05T18:00:48Z

@eashaw @DominusKelvin

Shouldn't the tests MongoDB servers be updated?

The MongoDB server versions being tested on are EOL.

3.6.18: EOL April 2021
4.0.18: EOL April 2022
4.2.7: EOL April 2023

https://www.mongodb.com/support-policy/lifecycles

I suggest using:

MongoDB 4.4: EOL Feb 2024
MongoDB 5.0: EOL Oct 2024
MongoDB 6.0: EOL Jul 2025
MongoDB 7.0: EOL Aug 2026

luislobo · 2023-12-05T18:06:09Z

For the Travis version, the CI will have to be updated to a focal based Ubuntu, and the install script also be updated.

Links to download files are here:

https://www.mongodb.com/download-center/community/releases

I can help with this update if you want to go on this direction for this PR.

DominusKelvin · 2023-12-07T07:19:39Z

Hey @luislobo great suggestions. I think we can do the CI upgrades on a follow-up PR after this one.

DominusKelvin · 2023-12-07T07:24:22Z

Hey all, just to state the obvious 😁, this PR has been approved for merge and a new version will be cut out soon. Thank you all for your patience on this one :)

DominusKelvin · 2023-12-07T20:23:08Z

Hey there Sailors, I am pleased to announced that this PR has been merged and a minor release of sails-mongo - v2.1.0 has been made. Thank you all for your patience and do enjoy your latest sails-mongo adapter with the latest MongoDB driver.

You can read the release blog post for more information on upgrading to this new version of sails-mongo 👇🏾

https://blog.sailscasts.com/sails-mongo-v2-1-0

DominusKelvin added 4 commits September 29, 2023 12:10

feat: upgrade docker-compose.yml from deprecated syntax

bb3d120

chore: update mongodb driver to 6.1.0

b70cb73

feat: remove deprecated client option

fa906e5

feat: update connect() to promise-based syntax

5d1ef99

DominusKelvin mentioned this pull request Sep 30, 2023

Incompatibility with MongoDB 6.0.2 balderdashy/sails#7275

Closed

DominusKelvin added 7 commits October 1, 2023 10:43

chore: upgrade mongodb to 4.x and eslint to 8.x

6e918a5

chore: update ecmaVersion to 14

ec3dad7

fix: use findOne to get back the newly created document

c579e58

fix: use id not _id

15fe1c4

chore: update var declarations to const/let

6318039

chore: change comment for .insertOne end

0ec5228

feat: move to new syntax for insertMany and return records created vi…

2a39aad

…a find

feat: replace deprecated .count with .countDocuments

31e6a9a

mikermcneil requested changes Nov 22, 2023

View reviewed changes

mikermcneil reviewed Nov 22, 2023

View reviewed changes

package.json Show resolved Hide resolved

mikermcneil reviewed Nov 22, 2023

View reviewed changes

update comments

f190ffb

DominusKelvin requested a review from mikermcneil November 23, 2023 09:56

feat: add assertions for results returned from MongoDB

817c742

eashaw added 5 commits November 28, 2023 19:43

update node versions in CI config files

c151a31

update CI tests

4cc13ca

run travis test on ubuntu 16, comment out incompatible node versions

a7c8f6f

Update tests (localhost » 127.0.0.1)

49948b5

Revert test changes, update env variables in appveyor test to use ipv…

ecf1f42

…4 localhost address

mikermcneil approved these changes Dec 6, 2023

View reviewed changes

eashaw merged commit 7e8012e into balderdashy:master Dec 7, 2023
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat]: upgrade to mongodb driver 6 #499

[feat]: upgrade to mongodb driver 6 #499

DominusKelvin commented Sep 30, 2023 •

edited

Loading

sailsbot commented Sep 30, 2023

lennyAiko commented Sep 30, 2023

luislobo commented Oct 1, 2023

luislobo commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

luislobo commented Oct 1, 2023

luislobo commented Oct 1, 2023

luislobo commented Oct 1, 2023

luislobo commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

luislobo commented Oct 1, 2023

mikermcneil Nov 22, 2023

mikermcneil Nov 22, 2023

mikermcneil Nov 22, 2023

DominusKelvin Nov 23, 2023

mikermcneil Nov 22, 2023

DominusKelvin Nov 23, 2023

mikermcneil left a comment

DominusKelvin commented Nov 23, 2023

DominusKelvin commented Nov 23, 2023

DominusKelvin commented Dec 5, 2023

luislobo commented Dec 5, 2023

luislobo commented Dec 5, 2023

DominusKelvin commented Dec 7, 2023 •

edited

Loading

DominusKelvin commented Dec 7, 2023

DominusKelvin commented Dec 7, 2023

[feat]: upgrade to mongodb driver 6 #499

[feat]: upgrade to mongodb driver 6 #499

Conversation

DominusKelvin commented Sep 30, 2023 • edited Loading

sailsbot commented Sep 30, 2023

lennyAiko commented Sep 30, 2023

luislobo commented Oct 1, 2023

luislobo commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

luislobo commented Oct 1, 2023

luislobo commented Oct 1, 2023

luislobo commented Oct 1, 2023

luislobo commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

DominusKelvin commented Oct 1, 2023

luislobo commented Oct 1, 2023

mikermcneil Nov 22, 2023

Choose a reason for hiding this comment

mikermcneil Nov 22, 2023

Choose a reason for hiding this comment

mikermcneil Nov 22, 2023

Choose a reason for hiding this comment

DominusKelvin Nov 23, 2023

Choose a reason for hiding this comment

mikermcneil Nov 22, 2023

Choose a reason for hiding this comment

DominusKelvin Nov 23, 2023

Choose a reason for hiding this comment

mikermcneil left a comment

Choose a reason for hiding this comment

DominusKelvin commented Nov 23, 2023

DominusKelvin commented Nov 23, 2023

DominusKelvin commented Dec 5, 2023

luislobo commented Dec 5, 2023

luislobo commented Dec 5, 2023

DominusKelvin commented Dec 7, 2023 • edited Loading

DominusKelvin commented Dec 7, 2023

DominusKelvin commented Dec 7, 2023

DominusKelvin commented Sep 30, 2023 •

edited

Loading

DominusKelvin commented Dec 7, 2023 •

edited

Loading