Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support custom SASL mechanisms #146

Closed
wants to merge 5 commits into from

Conversation

wbarnha
Copy link
Owner

@wbarnha wbarnha commented Mar 8, 2024

There is some interest in supporting various SASL mechanisms not
currently included in the library:

Adding these mechanisms in the core library may be undesirable due to:

  • Increased maintenance burden.
  • Unavailable testing environments.
  • Vendor specificity.

This PR provides a quick prototype for a pluggable SASL system (and potential alternate route to supporting dpkp#2255).

I don't want to be presumptuous about the support for custom SASL extensions on the behalf of the maintainers of kafka-python.
I hope that this quick example can serve as a venue for discussion about the viability of the idea!


Example

To define a custom SASL mechanism a module must implement two methods:

def validate_config(conn):
    # Check configuration values, available libraries, etc.
    assert conn.config['vendor_specific_setting'] is not None, (
        'vendor_specific_setting required when sasl_mechanism=MY_SASL'
    )

def try_authenticate(conn, future):
    # Do authentication routine and return resolved Future with failed
    # or succeeded state.

And then the custom mechanism should be registered before initializing
a KafkaAdminClient, KafkaConsumer, or KafkaProducer:

import kafka.sasl
from kafka import KafkaProducer

import my_sasl

kafka.sasl.register_mechanism('MY_SASL', my_sasl)

producer = KafkaProducer(sasl_mechanism='MY_SASL')

Notes

ABCs

This prototype does not implement an ABC for custom SASL mechanisms.
Using an ABC would reduce a few of the explicit assertions involved with
registering a mechanism and is a viable option. Due to differing feature
sets between py2/py3 this option was not explored, but shouldn't be
difficult.

Private Methods

This prototype relies on some methods that are currently marked as
private in BrokerConnection.

  • ._can_send_recv
  • ._lock
  • ._recv_bytes_blocking
  • ._send_bytes_blocking

A pluggable system would require stable interfaces for these actions.

Alternative Approach

If the module-scoped dict modification in register_mechanism feels too
clunky maybe the addtional mechanisms can be specified via an argument
when initializing one of the Kafka* classes?


This change is Reviewable

mattoberle and others added 2 commits August 20, 2021 13:47
There is some interest in supporting various SASL mechanisms not
currently included in the library:

* dpkp#2110 (DMS)
* dpkp#2204 (SSPI)
* dpkp#2232 (AWS_MSK_IAM)

Adding these mechanisms in the core library may be undesirable due to:

* Increased maintenance burden.
* Unavailable testing environments.
* Vendor specificity.

This commit provides a quick prototype for a pluggable SASL system.

---

**Example**

To define a custom SASL mechanism a module must implement two methods:

```py

def validate_config(conn):
    # Check configuration values, available libraries, etc.
    assert conn.config['vendor_specific_setting'] is not None, (
        'vendor_specific_setting required when sasl_mechanism=MY_SASL'
    )

def try_authenticate(conn, future):
    # Do authentication routine and return resolved Future with failed
    # or succeeded state.
```

And then the custom mechanism should be registered before initializing
a KafkaAdminClient, KafkaConsumer, or KafkaProducer:

```py

import kafka.sasl
from kafka import KafkaProducer

import my_sasl

kafka.sasl.register_mechanism('MY_SASL', my_sasl)

producer = KafkaProducer(sasl_mechanism='MY_SASL')
```

---

**Notes**

**ABCs**

This prototype does not implement an ABC for custom SASL mechanisms.
Using an ABC would reduce a few of the explicit assertions involved with
registering a mechanism and is a viable option. Due to differing feature
sets between py2/py3 this option was not explored, but shouldn't be
difficult.

**Private Methods**

This prototype relies on some methods that are currently marked as
**private** in `BrokerConnection`.

* `._can_send_recv`
* `._lock`
* `._recv_bytes_blocking`
* `._send_bytes_blocking`

A pluggable system would require stable interfaces for these actions.

**Alternative Approach**

If the module-scoped dict modification in `register_mechanism` feels too
clunky maybe the addtional mechanisms can be specified via an argument
when initializing one of the `Kafka*` classes?
Comment on lines +268 to +271
assert self.config['sasl_mechanism'] in sasl.MECHANISMS, (
'sasl_mechanism must be one of {}'.format(', '.join(sasl.MECHANISMS.keys()))
)
sasl.MECHANISMS[self.config['sasl_mechanism']].validate_config(self)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is so much cleaner. Thank you for writing this.

@wbarnha
Copy link
Owner Author

wbarnha commented Mar 17, 2024

Actually, let's pull in the tests and make life easier to manage this PR.

@wbarnha
Copy link
Owner Author

wbarnha commented Mar 17, 2024

On further though, it would actually be better to pull in changes from #147 into this PR.

@wbarnha
Copy link
Owner Author

wbarnha commented Mar 18, 2024

Resuming in #170.

@wbarnha wbarnha closed this Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants