Another Pack from IBM (MONITOR_INGEST) #163

Anshika-Gautam · 2021-02-16T10:09:14Z

This is another pack from IBM similar to the first one monitor_mqtt

nmaludy · 2021-02-18T13:13:24Z

.circleci/config.yml

@@ -30,7 +30,7 @@ jobs:
      - run:
          name: Download dependencies
          command: |
-            git clone -b ${CI_BRANCH:-master} [email protected]:StackStorm-Exchange/ci.git ~/ci
+            git clone -b ${CI_BRANCH:-master} [email protected]:Anshika-Gautam/ci.git ~/ci


Need to fix this

Our pack is using mam-sdk which is dependent on iotfunctions package for its functioning. The "[email protected]:StackStorm-Exchange/ci.git ~/ci" is installing the pip version 9.0.3 which is not able to get the iotfunctions package from the specified git repository. It is resulting into the below mentioned error on circle:

`Using /home/circleci/virtualenv/lib/python3.6/site-packages
Finished processing dependencies for st2common==3.4.dev0

[[ -f /home/circleci/repo/requirements.txt ]]

echo 'Installing pack requirements from /home/circleci/repo/requirements.txt'
Installing pack requirements from /home/circleci/repo/requirements.txt

/home/circleci/virtualenv/bin/pip install -r /home/circleci/repo/requirements.txt
Collecting git+https://github.com/ibm-watson-iot/maximo-asset-monitor-sdk.git (from -r /home/circleci/repo/requirements.txt (line 2))
Cloning https://github.com/ibm-watson-iot/maximo-asset-monitor-sdk.git to /tmp/pip-o1f94hft-build
Warning: Permanently added the RSA host key for IP address '140.82.113.3' to the list of known hosts.
Collecting pandas-schema (from -r /home/circleci/repo/requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/9c/03/6d87ce8719dc57e44688096c05fb0efa61a08c6838816c9d991b1ece5b24/pandas_schema-0.3.5-py3-none-any.whl
Collecting jsonschema>=3.2.0 (from mam-sdk==0.0.0->-r /home/circleci/repo/requirements.txt (line 2))
Cache entry deserialization failed, entry ignored
Downloading https://files.pythonhosted.org/packages/c5/8f/51e89ce52a085483359217bc72cdbf6e75ee595d5b1d4b5ade40c7e018b8/jsonschema-3.2.0-py2.py3-none-any.whl (56kB)
100% |████████████████████████████████| 61kB 6.6MB/s eta 0:00:01
Collecting iotfunctions@ git+https://github.com/ibm-watson-iot/functions.git@production#egg=iotfunctions (from mam-sdk==0.0.0->-r /home/circleci/repo/requirements.txt (line 2))
Could not find a version that satisfies the requirement iotfunctions@ git+https://github.com/ibm-watson-iot/functions.git@production#egg=iotfunctions (from mam-sdk==0.0.0->-r /home/circleci/repo/requirements.txt (line 2)) (from versions: )
No matching distribution found for iotfunctions@ git+https://github.com/ibm-watson-iot/functions.git@production#egg=iotfunctions (from mam-sdk==0.0.0->-r /home/circleci/repo/requirements.txt (line 2))
You are using pip version 9.0.3, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Exited with code exit status 1
CircleCI received exit code 1`

We had removed the pip restriction in the package "[email protected]:Anshika-Gautam/ci.git ~/ci" and after that we are able to download all the dependencies for our pack. But the run test step is failing for the pack now as shown in snapshot below:

monitor_ingest/README.md

monitor_ingest/config.schema.yaml

nmaludy · 2021-02-18T13:20:09Z

monitor_ingest/config.schema.yaml

+  type: "string"
+  required: true
+json_schema_path:
+  description: "json Schema is must to validate CSV in case of action_type : DataClean"


What is this a JSON schema for?

The json schema defines particular format in which the data for ingestion should be supplied for our pack's data_ingest action

Does _path refer to a file on disk?

And does this mean that the file will need to exist on all st2actionrunner nodes?

@blag Yes _path refers to a file on disk. What are you trying point with st2actionrunner nodes being using the path? Can you please explain?
Note - The _path is mandatory to execute setup_entity action.

nmaludy · 2021-02-18T13:22:55Z

monitor_ingest/actions/data_clean_csv_action.py

+
+    def run(self):
+        # define validation elements
+        print('1. Starting data Clean Action ..')


Should replace prints with self.logger.debug()

updated the changes for print

nmaludy · 2021-02-18T13:26:12Z

monitor_ingest/actions/data_clean_csv_action.py

+            if errors:
+                errors_index_rows = [e.row for e in errors]
+                print('5. Cleaning input CSV data ..')
+                data_clean = data.drop(index=errors_index_rows)


I don't think it's great practice to hard code paths like this in your actions. You're going to run into problems with there is >1 instance of your action running at the same time

I had removed the hard code paths. Thanks for your suggestion.

nmaludy · 2021-02-18T13:26:48Z

monitor_ingest/actions/data_clean_csv_action.py

@@ -0,0 +1,102 @@
+import json


From a naming perspective, a better name would be data_clean_csv you can leave out the _action

nmaludy · 2021-02-18T13:27:06Z

monitor_ingest/actions/data_ingest_csv_action.py

@@ -0,0 +1,37 @@
+import yaml


same with the name here, maybe just data_ingest_csv

nmaludy · 2021-02-18T13:28:14Z

monitor_ingest/actions/setup_entity.py

+        dimension_data_path = None
+        function_data_path = None
+
+        if self._action_type == "SetupEntityAction":


curious why these are not parameters to the action and instead hard coded in the config?

our config file is actually working as a handler to perform different actions on the entity. We had updated the code to follow a modular approach now. Instead of if loops we tried using functions to call a particular action type.

…-incubator

abharast · 2021-03-02T09:36:09Z

Hi @nmaludy

could you please suggest us with the issue related to pip version as it made us blocked in this submission
#163 (comment)

-Abhay

blag · 2021-03-02T10:00:40Z

monitor_ingest/config.schema.yaml

+entity_name:
+  description: "Entity Name is must in case of action_type : LoadCsv"
+  type: "string"
+data_file_path:


Where does the file identified here come from?

@blag It is the same file which is residing on disk. This variable here is pointing to the path of the file.

blag · 2021-03-02T10:02:55Z

monitor_ingest/actions/setup_entity.py

+            raise ValueError('Missing action type key in config file')
+
+    def run(self):
+        operations_completed = {}


I do not like using a mutable dictionary as a global variable. That's not intuitive and difficult to debug. Since all of the setup_* functions are mutually exclusive, simply have them return the result you would like to return to StackStorm.

@blag removed mutable dictionary

blag · 2021-03-02T10:04:45Z

monitor_ingest/actions/setup_entity.py

+
+        """----------STATUS----------"""
+        self.logger.info('RESULT :')
+        for name, status in operations_completed.items():


All of the logic in this loop is overcomplicated. Why not just return a tuple of (success, result_text) to StackStorm? Seems much cleaner to me.

@blag simplified the logic to return boolean values

Anshika-Gautam · 2021-03-04T10:28:52Z

@nmaludy @blag we have a blocker regarding pip version for downloading dependencies. As mentioned in thread #163 (comment).

cognifloyd · 2021-03-07T05:30:42Z

StackStorm-Exchange/ci#102 was merged today which updates the pinned version of pip to 20.0.2

Please push another commit to restart the tests.

The following error was encountered while running circleci Traceback (most recent call last): File "/home/circleci/ci/.circle/validate.py", line 21, in <module> from st2common.models.api.pack import PackAPI File "/tmp/st2/st2common/st2common/models/api/pack.py", line 25, in <module> from st2common.util import schema as util_schema File "/tmp/st2/st2common/st2common/util/schema/__init__.py", line 112, in <module> "allOf": _validators.allOf_draft4, AttributeError: module 'jsonschema._validators' has no attribute 'allOf_draft4' Unable to retrieve pack name. In order to avoid this error the jsonschema is restricted to version 3.0.0

cognifloyd · 2021-03-08T18:18:49Z

Eww. That's a nasty dependency conflict.

ERROR: st2common 3.5.dev0 has requirement jsonschema==2.6.0, but you'll have jsonschema 3.0.0 which is incompatible.
ERROR: orquesta 1.3.0 has requirement jsonschema!=2.5.0,<3.0.0,>=2.0.0, but you'll have jsonschema 3.0.0 which is incompatible.
ERROR: mam-sdk 0.0.0 has requirement jsonschema>=3.2.0, but you'll have jsonschema 3.0.0 which is incompatible.

jsonschema._validators.allOf_draft4 was moved and dropped in the 3.0 series. So, st2common and orquesta will need to be updated to support jsonschema 3+ before packs can use newer versions of jsonschema.

Anshika-Gautam · 2021-03-09T06:23:50Z

@cognifloyd are you working on updating Stackstorm packs to use the latest versions of the packages like jsonschema and others. As you can see the conflict is because of the different versions these dependencies(st2,mam-sdk) are using.

cognifloyd · 2021-03-09T06:56:57Z

I have a variety of other things I'm working on contributing. Recently I helped to get pip pinned to the same version in several of the StackStorm repos. That's how I came across this new pack. :)

Would you be able to look into what it will take to update jsonschema in https://github.com/StackStorm/orquesta and https://github.com/StackStorm/st2 ?

CLAassistant · 2022-05-11T12:37:26Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

cognifloyd · 2022-05-23T16:09:36Z

I'm closing and reopening this to trigger the latest CI.

Anshika-Gautam added 19 commits February 16, 2021 13:45

Create README.md

57c61a8

Add files via upload

998ee90

Create setup_entity.py

26fbd72

Add files via upload

09e3192

Create clean_data_ingest_chain.yaml

be4bdcb

Create csvSchema.json

4dfb10d

Add files via upload

0831a94

Create errors.csv

41ec5c6

Add files via upload

dccd200

Create test_action_setup_entity.py

594ed17

Update README.md

e3e2049

Update pack.yaml

ad833a3

Update config.yml

5cfb18b

Update config.yml

b55228c

Update config.yml

cb52ee7

Adding iotfunctions to file

624d8e1

Update config.yml

83b8e8c

Update README.md

45b9385

Update requirements.txt

37cad34

nmaludy reviewed Feb 18, 2021

View reviewed changes

nmaludy suggested changes Feb 18, 2021

View reviewed changes

Anshika-Gautam and others added 5 commits February 19, 2021 18:37

Changes suggested 18 Feb

b1c9ee2

Changes suggested 18 Feb

c3c4ec9

Delete .idea directory

26f4413

reverting changes in .circleci/config.yml

1fd245a

Merge branch 'master' of https://github.com/maximo-developer/exchange…

d03fafa

…-incubator

Anshika-Gautam requested a review from nmaludy February 26, 2021 09:38

blag suggested changes Mar 2, 2021

View reviewed changes

removed mutable dictionary

c28478c

Anshika-Gautam requested a review from blag March 4, 2021 10:29

Anshika-Gautam added 2 commits March 8, 2021 11:25

rerun the pack to test pip changes in pipeline

4091de5

cognifloyd closed this May 23, 2022

cognifloyd reopened this May 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Another Pack from IBM (MONITOR_INGEST) #163

Another Pack from IBM (MONITOR_INGEST) #163

Anshika-Gautam commented Feb 16, 2021

nmaludy Feb 18, 2021

Anshika-Gautam Feb 19, 2021

nmaludy Feb 18, 2021

Anshika-Gautam Feb 19, 2021

blag Mar 2, 2021

Anshika-Gautam Mar 4, 2021 •

edited

Loading

nmaludy Feb 18, 2021

Anshika-Gautam Feb 19, 2021

nmaludy Feb 18, 2021

Anshika-Gautam Feb 19, 2021

nmaludy Feb 18, 2021

Anshika-Gautam Feb 19, 2021

nmaludy Feb 18, 2021

Anshika-Gautam Feb 19, 2021

nmaludy Feb 18, 2021

Anshika-Gautam Feb 19, 2021

abharast commented Mar 2, 2021

blag Mar 2, 2021

Anshika-Gautam Mar 4, 2021

blag Mar 2, 2021

Anshika-Gautam Mar 4, 2021 •

edited

Loading

blag Mar 2, 2021

Anshika-Gautam Mar 4, 2021 •

edited

Loading

Anshika-Gautam commented Mar 4, 2021

cognifloyd commented Mar 7, 2021

cognifloyd commented Mar 8, 2021

Anshika-Gautam commented Mar 9, 2021

cognifloyd commented Mar 9, 2021

CLAassistant commented May 11, 2022

cognifloyd commented May 23, 2022

Another Pack from IBM (MONITOR_INGEST) #163

Are you sure you want to change the base?

Another Pack from IBM (MONITOR_INGEST) #163

Conversation

Anshika-Gautam commented Feb 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anshika-Gautam Mar 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abharast commented Mar 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anshika-Gautam Mar 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anshika-Gautam Mar 4, 2021 • edited Loading

Choose a reason for hiding this comment

Anshika-Gautam commented Mar 4, 2021

cognifloyd commented Mar 7, 2021

cognifloyd commented Mar 8, 2021

Anshika-Gautam commented Mar 9, 2021

cognifloyd commented Mar 9, 2021

CLAassistant commented May 11, 2022

cognifloyd commented May 23, 2022

Anshika-Gautam Mar 4, 2021 •

edited

Loading

Anshika-Gautam Mar 4, 2021 •

edited

Loading

Anshika-Gautam Mar 4, 2021 •

edited

Loading