RD-1291 Introduced TS to backend #1575

kubama · 2021-09-01T08:21:48Z

Description

Although this PR is massive it's not as bad as it looks. All changes made to production and test code is just migration to TS modules, so the set of applied changes is pretty consistent across these files. Additionally there are some required changes around modules mocking in jest tests - I can elaborate about these on our weekly, as there are some gotchas.
Core changes are done in just a few files - package.json, tsconfig.json and .eslintrc.json.
Unfortunately, a few files that were properly renamed appear as deleted/added after committing 🙁.

Screenshots / Videos

N/A

Checklist

My code follows the UI Code Style.
I verified that all tests and checks have passed.
I verified if that change requires a documentation update.
I added proper labels to this PR.

Tests

https://jenkins.cloudify.co/view/UI/job/Stage-UI-System-Test/1185/

Documentation

N/A

…fy-stage into RD-1291-backend-ts

backend/routes/Filters.ts

stepek · 2021-09-01T10:36:09Z

@kubama can you split #d447df2 to smaller commits?

kubama · 2021-09-01T11:29:14Z

@kubama can you split #d447df2 to smaller commits?

I could, but I'm not sure it would be of much help. As I stated in the description, the core changes are limited to just a few json files. All the changes to the actual source files are just conversion to TS modules. There are no changes to a single file that could be split across different commits.

qooban · 2021-09-01T12:29:52Z

Oh, quite big change. It'll take some time to review.

Unfortunately, a few files that were properly renamed appear as deleted/added after committing slightly_frowning_face.

Having one commit containing both - rename and migration-related changes - won't help. AFAIR from what Grzegorz found is that GitHub works better if you do the rename in one commit and the changes inside the file in the other commit.

https://jenkins.cloudify.co/view/UI/job/Stage-UI-System-Test/1177/

I see there's a problem with starting the backend:

14:15:02  > [email protected] wait-on-server /opt/cloudify-stage/backend
14:15:02  > wait-on tcp:localhost:8088 --timeout 5000 && echo 'Stage backend is up!'
14:15:02  
14:15:07  Error: Timeout
14:15:07      at Timeout._onTimeout (/opt/cloudify-stage/backend/node_modules/wait-on/lib/wait-on.js:113:10)
14:15:07      at listOnTimeout (internal/timers.js:554:17)
14:15:07      at processTimers (internal/timers.js:497:7)
14:15:07  npm ERR! code ELIFECYCLE
14:15:07  npm ERR! errno 1
14:15:07  npm ERR! [email protected] wait-on-server: `wait-on tcp:localhost:8088 --timeout 5000 && echo 'Stage backend is up!'`
14:15:07  npm ERR! Exit status 1
14:15:07  npm ERR! 
14:15:07  npm ERR! Failed at the [email protected] wait-on-server script.
14:15:07  npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Do we need any update in terms of installed packages on the server to execute ts-node? Maybe it should be installed globally? (npm i -g ts-node)

Other questions before I jump into the review (possibly tomorrow):

Did you manage to change all the files to .ts?
How many of them are already fully converted?
Could you provide some guidelines on how to do similar migration in Composer? (it'd be very much appreciated)

kubama · 2021-09-01T14:24:40Z

Having one commit containing both - rename and migration-related changes - won't help. AFAIR from what Grzegorz found is that GitHub works better if you do the rename in one commit and the changes inside the file in the other commit.

Yes, I didn't expect that as I did just minor updates to the renamed files. In fact git respected renames for most of the files.

I see there's a problem with starting the backend:

14:15:02  > [email protected] wait-on-server /opt/cloudify-stage/backend
14:15:02  > wait-on tcp:localhost:8088 --timeout 5000 && echo 'Stage backend is up!'
14:15:02  
14:15:07  Error: Timeout
14:15:07      at Timeout._onTimeout (/opt/cloudify-stage/backend/node_modules/wait-on/lib/wait-on.js:113:10)
14:15:07      at listOnTimeout (internal/timers.js:554:17)
14:15:07      at processTimers (internal/timers.js:497:7)
14:15:07  npm ERR! code ELIFECYCLE
14:15:07  npm ERR! errno 1
14:15:07  npm ERR! [email protected] wait-on-server: `wait-on tcp:localhost:8088 --timeout 5000 && echo 'Stage backend is up!'`
14:15:07  npm ERR! Exit status 1
14:15:07  npm ERR! 
14:15:07  npm ERR! Failed at the [email protected] wait-on-server script.
14:15:07  npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Do we need any update in terms of installed packages on the server to execute ts-node? Maybe it should be installed globally? (npm i -g ts-node)

Nothing extra needs to be done.

The issue here seems to be just the timeout - the backend now starts slightly longer, which is not unexpected. I increased the timeout and submitted new job (link in description updated).

Did you manage to change all the files to .ts?

No, db migrations and sequelize models (also used by migrations) are not changed. These are not directly imported, and the logic that handles them is part of cloudify-ui-common and used by composer as well - if we want these to be converted as well it will be best to first migrate composer backend to TS and then handle it as part of a separate ticket.

How many of them are already fully converted?

6 production source files and 3 test files are fully migrated.

Could you provide some guidelines on how to do similar migration in Composer? (it'd be very much appreciated)

Here are the steps in short:

Install ts-node and typescript as production dependencies
Rename all source files (except migrations and sequelize models) and test files to .ts
Replace all node occurrences with ts-node and all direct references to js files with their corresponding ts files
Look for all wildcards referencing js files and change them so they referenced both js and ts files
Convert all renamed files to valid TS modules (import/export)
Test files that used require inside functions will need to have module mocking updated:

replace all jest.doMock calls with jest.mock
move all jest.mock calls to top level
make sure all required jest.mock calls are done directly in the test file (not in an imported file)

Align eslint configuration

kubama · 2021-09-01T16:06:37Z

There is one system test failure - looks like an issue with widget backend. I'll investigate.

…fy-stage into RD-1291-backend-ts

qooban

I did partial review (78/124 files) so far. Will continue later, probably tomorrow (due to the planned meetings). Added some comments.

Very hard to review files which are detected as deleted/added. It's not the time right now, but I'd suggest for the future to consider splitting such huge PR into smaller pieces. I suspect it must be possible to have it done within few PRs, one building on top of the other.

backend/.eslintrc.json

backend/babel.config.js

backend/package.json

backend/serverSettings.ts

qooban · 2021-09-02T09:14:18Z

backend/test/.eslintrc.json

-    "settings": {
-        "import/resolver": {
-            "jest": {
-                "jestConfigFile": "jest.config.js"
-            }
-        }
-    },


What was the reason behind removing it?

Did you have problems with resolving .ts files? (if so, see my comment: https://github.com/cloudify-cosmo/cloudify-blueprint-composer/pull/921/files#r700138908)

Yes, I couldn't make it working.
In the end there are two rules that check the same thing: node/no-missing-import and import/no-unresolved. There is no point in having both, so I decided to disable the second one (thus removing the setting).

qooban · 2021-09-02T09:18:40Z

backend/test/routes/Config.test.ts

+import appConfig from '../../../conf/config.json';
+import userConfig from '../../../conf/userConfig.json';


If we bring back jest import resolver, we can avoid using such paths.

Not really - the import resolver impacts eslint only. Here the issue was tsc not resolving the modules as they were originally specified

backend/tsconfig.json

devServer.js

…fy-stage into RD-1291-backend-ts

kubama · 2021-09-02T11:14:04Z

It's not the time right now, but I'd suggest for the future to consider splitting such huge PR into smaller pieces. I suspect it must be possible to have it done within few PRs, one building on top of the other.

Thinking of it now I think it could be split into first introducing TS support without renaming any of the source files, and then gradually renaming the files and applying required changes (TS modules). Unfortunately I wasn't upfront aware of the scale of the changes that were required. I hope it's a lesson learned for composer backend work.

stepek

any new comments from me.

qooban

Second part of the review. 104/124 files reviewed, so a bit closer to finish.

backend/package.json

backend/samlSetup.ts

backend/handler/services/LoggerService.ts

backend/handler/services/ManagerService.ts

backend/test/migration.test.ts

backend/test/handlers/services/LoggerService.test.ts

backend/test/handlers/ArchiveHelper.test.ts

qooban

OK, I'm done with the review. I have no more additional comments. I didn't manage to review all the files with the same attention to detail, but I believe that we have some time until the next release to detect any potential problems.

backend/config.ts

…fy-stage into RD-1291-backend-ts

Co-authored-by: Jakub Niezgoda <[email protected]>

…fy-stage into RD-1291-backend-ts

qooban

Added two small comments.

package.json

devServer.ts

Co-authored-by: Jakub Niezgoda <[email protected]>

…fy-stage into RD-1291-backend-ts

qooban

I appreciate hard work on that migration 👍🏻

Could you summarize leftovers (migrations? DB models? any other?) and possibly add tickets for them? (labelled with UI stage typescript)

kubama · 2021-09-07T09:52:15Z

@qooban Created RD-3119 and RD-3120

Introduced TS to backend

d447df2

kubama added the internal Pull request with internal change (e.g. refactoring, CI changes, documentation update) label Sep 1, 2021

kubama and others added 6 commits September 1, 2021 10:22

Merge branch 'master' into RD-1291-backend-ts

00dea5f

Removed webpack config dependency on backend/consts.ts

32264d8

Merge branch 'RD-1291-backend-ts' of github.com:cloudify-cosmo/cloudi…

7f17a66

…fy-stage into RD-1291-backend-ts

Fixed eslint error

955d4f5

Corrected route implementation

724e8a0

Updated reference commit for coverage check

8205b8e

kubama requested review from qooban and stepek September 1, 2021 09:45

stepek reviewed Sep 1, 2021

View reviewed changes

backend/routes/Filters.ts Show resolved Hide resolved

Jakub Madej added 2 commits September 1, 2021 13:46

Minor update to ensure non-empty change set for coverage check

c8db33a

Added lodash typings

1d64531

Jakub Madej added 2 commits September 1, 2021 15:48

Fixed devServer

b81a37f

Extended backend startup timeout

08351fd

Jakub Madej and others added 5 commits September 2, 2021 09:46

Fixed logger widget backend helper

244cf05

Merge branch 'master' into RD-1291-backend-ts

ed1713f

Added test to satisfy coverage check

9267de2

Merge branch 'RD-1291-backend-ts' of github.com:cloudify-cosmo/cloudi…

3b564ee

…fy-stage into RD-1291-backend-ts

Merge branch 'master' into RD-1291-backend-ts

c6116e4

qooban reviewed Sep 2, 2021

View reviewed changes

Jakub Madej added 3 commits September 2, 2021 11:51

Corrected accidental change

df0e842

Merge branch 'RD-1291-backend-ts' of github.com:cloudify-cosmo/cloudi…

78987c1

…fy-stage into RD-1291-backend-ts

Added -T flag to ts-node invocations

02e338d

kubama requested a review from stepek September 2, 2021 11:15

stepek previously approved these changes Sep 3, 2021

View reviewed changes

qooban reviewed Sep 3, 2021

View reviewed changes

backend/config.ts Show resolved Hide resolved

Jakub Madej added 2 commits September 3, 2021 16:04

Migrated devServer to TS

64d8ffc

Changed devServer extension

ecb665d

kubama dismissed stepek’s stale review via ecb665d September 3, 2021 14:05

kubama and others added 9 commits September 3, 2021 16:06

Merge branch 'master' into RD-1291-backend-ts

2c8f690

Fixed devTrace npm script

8194f60

Merge branch 'RD-1291-backend-ts' of github.com:cloudify-cosmo/cloudi…

e7377cb

…fy-stage into RD-1291-backend-ts

Fixed typo in test name

bbb0b1f

Co-authored-by: Jakub Niezgoda <[email protected]>

Merge branch 'master' into RD-1291-backend-ts

cdffc1f

Fixed SAML config validation, added test

44ae37c

Merge branch 'RD-1291-backend-ts' of github.com:cloudify-cosmo/cloudi…

05a08d4

…fy-stage into RD-1291-backend-ts

Aligned project references configuration

6f50a30

Merge branch 'master' into RD-1291-backend-ts

8a4158f

kubama requested a review from qooban September 6, 2021 11:22

qooban reviewed Sep 6, 2021

View reviewed changes

package.json Outdated Show resolved Hide resolved

devServer.ts Outdated Show resolved Hide resolved

kubama and others added 4 commits September 6, 2021 14:49

Added separate check-types scripts for frontend and backend

5b5a2c8

Co-authored-by: Jakub Niezgoda <[email protected]>

Reused more consts

6b67ed4

Merge branch 'RD-1291-backend-ts' of github.com:cloudify-cosmo/cloudi…

aa1dacd

…fy-stage into RD-1291-backend-ts

Merge branch 'master' into RD-1291-backend-ts

6b84796

kubama requested a review from qooban September 7, 2021 07:54

qooban approved these changes Sep 7, 2021

View reviewed changes

kubama merged commit 1b7bf59 into master Sep 7, 2021

kubama deleted the RD-1291-backend-ts branch September 7, 2021 09:37

kubama mentioned this pull request Sep 7, 2021

RD-1291 Updated coverage reference commit #1587

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RD-1291 Introduced TS to backend #1575

RD-1291 Introduced TS to backend #1575

kubama commented Sep 1, 2021 •

edited

Loading

stepek commented Sep 1, 2021

kubama commented Sep 1, 2021

qooban commented Sep 1, 2021

kubama commented Sep 1, 2021

kubama commented Sep 1, 2021

qooban left a comment

qooban Sep 2, 2021 •

edited

Loading

kubama Sep 2, 2021

qooban Sep 2, 2021

kubama Sep 2, 2021

kubama commented Sep 2, 2021

stepek left a comment

qooban left a comment

qooban left a comment

qooban left a comment

qooban left a comment

kubama commented Sep 7, 2021

		import appConfig from '../../../conf/config.json';
		import userConfig from '../../../conf/userConfig.json';

RD-1291 Introduced TS to backend #1575

RD-1291 Introduced TS to backend #1575

Conversation

kubama commented Sep 1, 2021 • edited Loading

Description

Screenshots / Videos

Checklist

Tests

Documentation

stepek commented Sep 1, 2021

kubama commented Sep 1, 2021

qooban commented Sep 1, 2021

kubama commented Sep 1, 2021

kubama commented Sep 1, 2021

qooban left a comment

Choose a reason for hiding this comment

qooban Sep 2, 2021 • edited Loading

Choose a reason for hiding this comment

kubama Sep 2, 2021

Choose a reason for hiding this comment

qooban Sep 2, 2021

Choose a reason for hiding this comment

kubama Sep 2, 2021

Choose a reason for hiding this comment

kubama commented Sep 2, 2021

stepek left a comment

Choose a reason for hiding this comment

qooban left a comment

Choose a reason for hiding this comment

qooban left a comment

Choose a reason for hiding this comment

qooban left a comment

Choose a reason for hiding this comment

qooban left a comment

Choose a reason for hiding this comment

kubama commented Sep 7, 2021

kubama commented Sep 1, 2021 •

edited

Loading

qooban Sep 2, 2021 •

edited

Loading