feat: SITES-23133 [Importer] Report on the progress of a import job via a new jobs endpoint #508

bhellema · 2024-09-23T16:11:11Z

When querying the job progress via the endpoint eg. /tools/import/jobs/e551798c-eb17-49bc-beb3-4e3737734db4/progress

The result over time will look something like this:

{
	"pending": 11,
	"redirect": 0,
	"running": 0,
	"completed": 1,
	"failed": 0
}
// next request
{
	"pending": 9,
	"redirect": 0,
	"running": 0,
	"completed": 3,
	"failed": 0
}
// next request
{
	"pending": 0,
	"redirect": 0,
	"running": 0,
	"completed": 12,
	"failed": 0
}

When getting the job status and the processor will also add the redirectCount to the result:

{
	"id": "7e646a90-0f4b-4355-8d63-20681adfc360",
	...,
	"successCount": 12,
	"failedCount": 0,
	"redirectCount": 0
}

…a a new jobs endpoint

…a a new jobs endpoint * add write all domains scope

…a a new jobs endpoint

# Conflicts: # docs/index.html

…a a new jobs endpoint

github-actions · 2024-09-23T17:42:16Z

This PR will trigger a patch release when merged.

src/controllers/import.js

…a a new jobs endpoint

src/controllers/import.js

blefebvre · 2024-09-23T17:49:22Z

src/controllers/import.js

+   * - failed: URLs that have failed to import.
+   * - pending: URLs that are still being processed.
+   * - redirected: URLs that have been redirected.
+   * - running - URLs that are currently being imported.


Will running ever be non-zero? Since we do not have access to the data layer from the content-scraper, we do not update the DB when we begin processing a URL.

So this is interesting. We could do two things I guess with this.

remove running as it will never be non zero.

implement a new feature that will allow the content-scraper to message back / use a lamda to update the status of the import url job.

Is there value in constantly updating this field?

Currently, it will always be 0 or 1 since we have configured only a single Lambda invocation for the import queues, so effectively sequential processing

The cost of additional writes to the DB each time a new URL begins processing will add up for larger jobs

I don't think there is value at this point updating the field, at this point. I think at this point I should just eliminate the concept of running from the api-service. However if we start running jobs in parallel, then I think it might be interesting to revisit this.

blefebvre · 2024-09-23T17:52:12Z

src/support/import-supervisor.js

-    let updatedOptions = { ...options };
-
-    if (importScript) {
-      updatedOptions = { ...updatedOptions, hasCustomImportJs: true };
-    }
-
-    if (customHeaders) {
-      updatedOptions = { ...updatedOptions, hasCustomHeaders: true };
-    }
-


How will we expose these values for consumption both by API clients, as well as for use by our dashboards? cc @swetabar

We will have to provide new columns in the job for these values. We can log an new issue for this.

SITES-25634

src/support/import-supervisor.js

…a a new jobs endpoint

swetabar · 2024-09-24T15:14:22Z

src/controllers/import.js

+    READ: 'imports.read', // allows users to read the import jobs created with their API key
+    WRITE: 'imports.write', // allows users to create new import jobs
+    READ_ALL: 'imports.read_all', // allows users to view all import jobs
+    WRITE_ALL_DOMAINS: 'imports.write_all_domains', // allows users to import across any domain


Should we include this new scope in spacecat-shared as well? I'm planning to add it with my PR for https://jira.corp.adobe.com/browse/SITES-22756

# [1.66.0](v1.65.12...v1.66.0) (2024-09-24) ### Features * SITES-23133 [Importer] Report on the progress of a import job via a new jobs endpoint ([#508](#508)) ([78ef726](78ef726))

solaris007 · 2024-09-24T15:47:10Z

🎉 This issue has been resolved in version 1.66.0 🎉

The release is available on:

v1.66.0
GitHub release

Your semantic-release bot 📦🚀

Ben Helleman added 8 commits September 17, 2024 15:50

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

436ed0a

…a a new jobs endpoint

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

9ab4079

…a a new jobs endpoint

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

58e5016

…a a new jobs endpoint * add write all domains scope

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

5239cd8

…a a new jobs endpoint

Merge branch 'main' into bhellema/SITES-23133

801d23a

# Conflicts: # docs/index.html

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

c4306f5

…a a new jobs endpoint

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

13dfec4

…a a new jobs endpoint

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

10bafcd

…a a new jobs endpoint

bhellema self-assigned this Sep 23, 2024

bhellema added the enhancement New feature or request label Sep 23, 2024

bhellema requested review from blefebvre and swetabar September 23, 2024 16:12

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

7448239

…a a new jobs endpoint

blefebvre reviewed Sep 23, 2024

View reviewed changes

src/controllers/import.js Outdated Show resolved Hide resolved

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

3227ed7

…a a new jobs endpoint

bhellema requested a review from blefebvre September 23, 2024 17:46

blefebvre approved these changes Sep 23, 2024

View reviewed changes

Ben Helleman and others added 3 commits September 23, 2024 14:40

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

50b3586

…a a new jobs endpoint

fix: SITES-23133 [Importer] Report on the progress of a import job vi…

3f64cf3

…a a new jobs endpoint

Update README.md

d73387c

swetabar reviewed Sep 24, 2024

View reviewed changes

swetabar approved these changes Sep 24, 2024

View reviewed changes

bhellema merged commit 78ef726 into main Sep 24, 2024
4 checks passed

bhellema deleted the bhellema/SITES-23133 branch September 24, 2024 15:45

solaris007 added the released label Sep 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: SITES-23133 [Importer] Report on the progress of a import job via a new jobs endpoint #508

feat: SITES-23133 [Importer] Report on the progress of a import job via a new jobs endpoint #508

bhellema commented Sep 23, 2024 •

edited

Loading

github-actions bot commented Sep 23, 2024

blefebvre Sep 23, 2024

bhellema Sep 23, 2024 •

edited

Loading

blefebvre Sep 23, 2024

bhellema Sep 23, 2024 •

edited

Loading

blefebvre Sep 23, 2024

bhellema Sep 23, 2024

bhellema Sep 23, 2024

swetabar Sep 24, 2024

solaris007 commented Sep 24, 2024

feat: SITES-23133 [Importer] Report on the progress of a import job via a new jobs endpoint #508

feat: SITES-23133 [Importer] Report on the progress of a import job via a new jobs endpoint #508

Conversation

bhellema commented Sep 23, 2024 • edited Loading

github-actions bot commented Sep 23, 2024

blefebvre Sep 23, 2024

Choose a reason for hiding this comment

bhellema Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

blefebvre Sep 23, 2024

Choose a reason for hiding this comment

bhellema Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

blefebvre Sep 23, 2024

Choose a reason for hiding this comment

bhellema Sep 23, 2024

Choose a reason for hiding this comment

bhellema Sep 23, 2024

Choose a reason for hiding this comment

swetabar Sep 24, 2024

Choose a reason for hiding this comment

solaris007 commented Sep 24, 2024

bhellema commented Sep 23, 2024 •

edited

Loading

bhellema Sep 23, 2024 •

edited

Loading

bhellema Sep 23, 2024 •

edited

Loading