Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

reporting of credit usage in batch jobs #272

Closed
jdries opened this issue Nov 24, 2022 · 4 comments · Fixed by #317
Closed

reporting of credit usage in batch jobs #272

jdries opened this issue Nov 24, 2022 · 4 comments · Fixed by #317
Assignees
@bossie
Milestone
sap05-usability

Comments

@jdries
Copy link
Contributor

jdries commented Nov 24, 2022 •
edited
Loading

we now log resource usage in marketplace, can we retrieve the resulting credit usage from the marketplace and record it in batch job metadata?
If we use ETL api, the API will return amount of credits when we log resources, so that could be the solution.
@bossie bossie mentioned this issue Nov 28, 2022
Closed
@bossie bossie mentioned this issue Jan 19, 2023
Closed
@bossie
Copy link
Collaborator

bossie commented Jan 19, 2023

ETL API description including /addedvalue and /resources endpoint: https://etl.terrascope.be/docs/

bossie added a commit that referenced this issue Jan 19, 2023
@bossie
get (dev) access token from Keycloak #272
b13a79c
@bossie
Copy link
Collaborator

bossie commented Jan 20, 2023 •
edited
Loading

  • POST YARN metrics to /resources
  • for each unique process ID, if allowed, POST to /addedvalue
  • if Sentinel Hub costs, POST "shpu" to /resources
  • include credits in batch job metadata

bossie added a commit that referenced this issue Jan 20, 2023
@bossie
contact (dev) ETL API #272
a7827f8
bossie added a commit that referenced this issue Jan 20, 2023
@bossie
support logging resources and added value #272
8263d97
bossie added a commit that referenced this issue Jan 20, 2023
@bossie
ETL API nods approvingly #272
e284a72
bossie added a commit that referenced this issue Jan 20, 2023
@bossie
only send SHub PUs if necessary #272
c88700a
bossie added a commit that referenced this issue Jan 20, 2023
@bossie
incorporate ETL API in JobTracker #272
190c45e
@bossie bossie linked a pull request Jan 23, 2023 that will close this issue
272 reporting of credit usage in batch jobs #317
Merged
bossie added a commit that referenced this issue Jan 23, 2023
@bossie
add memory and time resources #272
1cfc68a
@bossie
Copy link
Collaborator

bossie commented Jan 23, 2023

The effect of these requests can be witnessed in Kibana.

bossie added a commit that referenced this issue Jan 23, 2023
@bossie
improve API #272
a41945b
bossie added a commit that referenced this issue Jan 23, 2023
@bossie
add missing file #272
98f710a
bossie added a commit that referenced this issue Jan 23, 2023
@bossie
improve logging #272
a13569d
bossie added a commit that referenced this issue Jan 23, 2023
@bossie
include credits in batch job metadata #272
3c79c82
soxofaan added a commit that referenced this issue Jan 23, 2023
@soxofaan
Fix openeo.keytab location in job_tracker-entrypoint
5421da3 
related to #236 #272 #317
bossie added a commit that referenced this issue Jan 23, 2023
@bossie
drop explicit "Content-Type" #272
fd4007f
bossie added a commit that referenced this issue Jan 23, 2023
@bossie
drop redundant url variable #272
b8ad3e9
@bossie
Copy link
Collaborator

bossie commented Jan 23, 2023

Implementing retries with requests: Open-EO/openeo-python-client#365 (comment)

soxofaan added a commit that referenced this issue Jan 23, 2023
@soxofaan
Vault.login_kerberos: add principal and keytab arguments
06b593a 
related to #236 #272 #317
@bossie bossie mentioned this issue Jan 24, 2023
Merged
bossie added a commit that referenced this issue Jan 24, 2023
@bossie
introduce connection pooling and retry #272
c68179c
bossie added a commit that referenced this issue Jan 24, 2023
@bossie
add remaining properties #272
bad5620
bossie added a commit that referenced this issue Jan 24, 2023
@bossie
remove method to test API credentials #272
7b5f347
bossie added a commit that referenced this issue Jan 25, 2023
@bossie
reuse OIDC auth implementation (dev) #272
8171bae
bossie added a commit that referenced this issue Jan 25, 2023
@bossie
foolproof the transition to (prod) https://etl.terrascope.be #272
0847cd6
bossie added a commit that referenced this issue Jan 25, 2023
@bossie
remove test code #272
143d4af
@bossie bossie mentioned this issue Jan 25, 2023
Closed
bossie added a commit that referenced this issue Jan 25, 2023
@bossie
specify currency for batch job costs #272
9d6d39c
bossie added a commit that referenced this issue Jan 25, 2023
@bossie
fixed argument types in calls to EtlApi #272
fd70224 
Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/job_tracker.py", line 202, in update_statuses
    duration_ms=finish_time - start_time,
TypeError: unsupported operand type(s) for -: 'str' and 'str'
bossie added a commit that referenced this issue Jan 26, 2023
@bossie
fix omitting SHub PUs #272
9ec4bff
bossie added a commit that referenced this issue Jan 26, 2023
@bossie
fix jobName, state and status values #272
f30e0bc 
POST https://etl-dev.terrascope.be/resources {'jobId': 'j-220b5d02390e412386c93508badb7074', 'jobName': None, 'executionId': 'application_1674538064532_9722', 'userId': 'jenkins', 'sourceId': 'TerraScope/MEP', 'orchestrator': 'openeo', 'jobStart': 1674682122903.0, 'jobFinish': 1674682639861.0, 'state': 'SUCCEEDED', 'status': 'finished', 'metrics': {'cpu': {'value': 2480, 'unit': 'cpu-seconds'}, 'memory': {'value': 4020798, 'unit': 'mb-seconds'}, 'time': {'value': 516958.0, 'unit': 'milliseconds'}, 'processing': {'value': 0.0, 'unit': 'shpu'}}} returned 400: {"status":400,"message":"Bad Request Exception","timestamp":"2023-01-25T21:38:55.734Z","path":"/resources"}
bossie added a commit that referenced this issue Jan 26, 2023
@bossie
fix job tracker updates wrt/ area DeepKeyError #272
d566036 
Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/job_tracker.py", line 207, in update_statuses
    area = deep_get(result_metadata, 'area', 'value', None)
  File "/opt/venv/lib64/python3.8/site-packages/openeo/util.py", line 393, in deep_get
    raise DeepKeyError(key, keys)
openeo.util.DeepKeyError: None (from deep key ('area', 'value', None))
soxofaan added a commit that referenced this issue Jan 27, 2023
@soxofaan
Introduce ConfigParam item for vault address
86c1a84 
related to #236 #272 #317
@bossie bossie mentioned this issue Jan 27, 2023
Closed
@jdries jdries added this to the sap05-usability milestone Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bossie bossie

Labels
None yet
Projects
None yet
Milestone
sap05-usability
Development

Successfully merging a pull request may close this issue.

272 reporting of credit usage in batch jobs Open-EO/openeo-geopyspark-driver
2 participants
@bossie @jdries