Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sensor raises Error when using sensor_service.list_values #5327

Open
Moradf90 opened this issue Aug 12, 2021 · 7 comments · May be fixed by #6134
Open

Sensor raises Error when using sensor_service.list_values #5327

Moradf90 opened this issue Aug 12, 2021 · 7 comments · May be fixed by #6134
Labels
bug regression

Comments

@Moradf90
Copy link

SUMMARY

when the sensor tries to read from the datastore (st2KV) with self.sensor_service.list_values(prefix'something'), it raises this Error:

ERROR [-] Sensor "MySensor" run method raised an exception: no such option in group api: max_page_size.
WARNING [-] Sensor "MySensor" run method raised an exception: no such option in group api: max_page_size.
KeyError: ('api', 'max_page_size')
    limit = limit or cfg.CONF.api.max_page_size
oslo_config.cfg.NoSuchOptError: no such option in group api: max_page_size
ERROR [-] ('api', 'max_page_size')
ERROR [-] limit = limit or cfg.CONF.api.max_page_size
ERROR [-] no such option in group api: max_page_size

STACKSTORM VERSION

v3.5.0

OS, environment, install method

using docker version

Steps to reproduce the problem

  1. create a new pack with sensor class :
from st2reactor.sensor.base import PollingSensor

class MySensor(PollingSensor):
    def __init__(self, sensor_service, config=None):
        super(SyslogAlertsSenderSensor, self).__init__(
            sensor_service=sensor_service,
            config=config,
            poll_interval=60
        )
        ...

    def poll(self):
        payloads = self.sensor_service.list_values(local=False, prefix='something')
        ...
  1. install the pack
  2. enable the sensor and look at the logs of st2sencorcontainer.

Expected Results

 DEBUG [-] Sensor my_pack.MySensor started.

Actual Results

ERROR [-] Sensor "MySensor" run method raised an exception: no such option in group api: max_page_size.

Workaround

add this code to sensor file:

from oslo_config import cfg

try:
    api_group = cfg.OptGroup('api')
    cfg.CONF.register_group(api_group)
    cfg.CONF.register_opts([cfg.IntOpt('max_page_size', default=50)], group=api_group)
except cfg.DuplicateOptError:
    pass

Thanks!
@ytjohn
Copy link
Contributor

ytjohn commented Oct 11, 2021

Introduced in #5176

@arm4b
Copy link
Member

arm4b commented Oct 11, 2021

@ytjohn sounds like a regression then since the st2 v3.5.0?

@ytjohn
Copy link
Contributor

ytjohn commented Oct 11, 2021

@armab Yes. We just ran across this Friday.

This line specifically:
https://github.com/StackStorm/st2/pull/5176/files#diff-e80db3ed79737fd467db4f0046b9a1668a1a2a433fa22bb611da25ffd1642bd7R92

While st2api config.py reads in api.max_page_size, this is not read in by the st2common or st2reactor. The sensor_wrapper in st2reactor appears to pass in a different config context, which doesn't include the same api options that st2common.services.datastore is expecting.

@rush-skills
Copy link
Member

This is still an issue (st2 3.8)

Not sure why the config is not read (even when explicitly added to st2.conf).
In our case, it was action_service failing

self.action_service.list_values(local=False, prefix=key_prefix)

fails with

st2.actions.python.CheckOldCeilings: DEBUG    Creating new Client object.
st2.actions.python.CheckOldCeilings: DEBUG    Retrieving all the values from the datastore
Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2262, in _get
    return self.__cache[key]
KeyError: ('api', 'max_page_size')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 395, in <module>
    obj.run()
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 214, in run
    output = action.run(**self._parameters)
  File \"/opt/stackstorm/packs/htcondor/actions/check_old_ceilings.py\", line 73, in run
    current_ceilings = self.get_ceilings_from_kv(key_prefix)
  File \"/opt/stackstorm/packs/htcondor/actions/check_old_ceilings.py\", line 43, in get_ceilings_from_kv
    kvp = self.action_service.list_values(local=False, prefix=key_prefix) # use global namespace for keys (local is action specific namespace)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 136, in list_values
    return self.datastore_service.list_values(local=local, prefix=prefix)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/services/datastore.py\", line 92, in list_values
    limit = limit or cfg.CONF.api.max_page_size
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2547, in __getattr__
    return self._conf._get(name, self._group)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2264, in _get
    value = self._do_get(name, group, namespace)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2282, in _do_get
    info = self._get_opt_info(name, group)
  File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2415, in _get_opt_info
    raise NoSuchOptError(opt_name, group)
oslo_config.cfg.NoSuchOptError: no such option in group api: max_page_size

Workaround

 kvp = self.action_service.datastore_service.list_values(local=False, prefix=key_prefix, limit=100)

@glafkosch
Copy link

glafkosch commented Apr 15, 2023 •
edited
Loading

I have upgraded to latest Stackstorm 3.8 version and I am getting also this issue. Any fix?

st2.actions.python.WorkflowCleanupAction: DEBUG Creating new Client object.\n\"auth.api_url\" configuration option is not configured\nst2.actions.python.WorkflowCleanupAction: DEBUG Retrieving all the values from the datastore\nTraceback (most recent call last):\n File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2262, in _get\n return self.__cache[key]\nKeyError: ('api', 'max_page_size')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 395, in <module>\n obj.run()\n File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 214, in run\n output = action.run(**self._parameters)\n File \"/opt/stackstorm/packs/tf/actions/workflow_cleanup.py\", line 14, in run\n items = self.action_service.list_values(local=False, prefix=key_prefix)\n File \"/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py\", line 136, in list_values\n return self.datastore_service.list_values(local=local, prefix=prefix)\n File \"/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/services/datastore.py\", line 92, in list_values\n limit = limit or cfg.CONF.api.max_page_size\n File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2547, in __getattr__\n return self._conf._get(name, self._group)\n File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2264, in _get\n value = self._do_get(name, group, namespace)\n File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2282, in _do_get\n info = self._get_opt_info(name, group)\n File \"/opt/stackstorm/st2/lib/python3.8/site-packages/oslo_config/cfg.py\", line 2415, in _get_opt_info\n raise NoSuchOptError(opt_name, group)\noslo_config.cfg.NoSuchOptError: no such option in group api: max_page_size\n"

My workflow_clean.py

from st2common.runners.base_action import Action

class WorkflowCleanupAction(Action):
    def run(self, scan_id):
        key_prefix = f"tf.StoreWorkerResultAction:{scan_id}"
        items = self.action_service.list_values(local=False, prefix=key_prefix)
        for item in items:
            self.action_service.delete_value(item.name, local=False)

        return (True, None)

Temporary Fix

        #items = self.action_service.list_values(local=False, prefix=key_prefix)
        items = self.action_service.datastore_service.list_values(local=False, prefix=key_prefix, limit=100)

@maxfactor1
Copy link
Contributor

Also experiencing this issue. However, I am also getting an issue where if I try to specify a 'limit' inside my sensor, I get this message:

list_values() got an unexpected keyword argument 'limit'

I verified in datastore.py that 'limit' should be a valid argument, but the wrapper for list_values doesn't include the option to pass 'limit' through.

sensor_wrapper.py snippet:

    def list_values(self, local=True, prefix=None):
        return self.datastore_service.list_values(local=local, prefix=prefix)

datastore.py snippet:

    def list_values(self, local=True, prefix=None, limit=None, offset=0):
        """
        Retrieve all the datastores items.

        :param local: List values from a namespace local to this pack/class. Defaults to True.
        :type: local: ``bool``

        :param prefix: Optional key name prefix / startswith filter.
        :type prefix: ``str``

        :param limit: Number of keys to get. Defaults to the configuration set at 'api.max_page_size'.
        :type limit: ``integer``

        :param offset: Number of keys to offset. Defaults to 0.
        :type offset: ``integer``

        :rtype: ``list`` of :class:`KeyValuePair`
        """
        client = self.get_api_client()
        self._logger.debug("Retrieving all the values from the datastore")
        limit = limit or cfg.CONF.api.max_page_size
        key_prefix = self._get_full_key_prefix(local=local, prefix=prefix)
        kvps = client.keys.get_all(prefix=key_prefix, limit=limit, offset=offset)
        return kvps

My workaround was to manually edit the sensor_wrapper.py to include all of the arguments in the upstream method in datastore.py like the following:

139
140     ##################################
141     # Methods for datastore management
142     ##################################
143
144     def list_values(self, local=True, prefix=None, limit=None, offset=0):
145         return self.datastore_service.list_values(local=local, prefix=prefix, limit=limit, offset=offset)
146
147     def get_value(self, name, local=True, scope=SYSTEM_SCOPE, decrypt=False):
148         return self.datastore_service.get_value(
149             name=name, local=local, scope=scope, decrypt=decrypt
150         )
151
...

@maxfactor1
Copy link
Contributor

I'll be submitting a PR to fix this, but in the meantime for anyone who is experiencing this issue, I think the 'correct' fix is to include the max_page_size option in st2common/st2common/config.py, as it seems like it may have been missed when this config option was added.

# Common API options
api_opts = [
    cfg.StrOpt("host", default="127.0.0.1", help="StackStorm API server host"),
    cfg.IntOpt("port", default=9101, help="StackStorm API server port"),
    #Add the below line as a workaround:
    cfg.IntOpt("max_page_size", default=100, help="Number of keys to get for kv list_values operations"),

@maxfactor1 maxfactor1 linked a pull request Feb 13, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug regression
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add missing api.max_page_size to config as IntOpt maxfactor1/st2
6 participants
@ytjohn @arm4b @rush-skills @maxfactor1 @Moradf90 @glafkosch