Paramiko 2.9.x breaking SSH operator for some OpenSSH configurations

[talnicolas]

Apache Airflow version

2.2.4 (latest released)

What happened

After upgrading from 2.2.3 to 2.2.4 the Paramiko library upgraded from 2.8.1 to 2.9.2. After that change one of our SSH connections started to fail with the following error:

File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1334, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1460, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1516, in _execute_task
result = execute_callable(context=context)
File "/usr/local/lib/python3.9/site-packages/airflow/providers/ssh/operators/ssh.py", line 175, in execute
raise AirflowException(f"SSH operator error: {str(e)}")
airflow.exceptions.AirflowException: SSH operator error: encountered RSA key, expected OPENSSH key

After looking at Paramiko changelog we saw that since 2.9.0 they have made non-backward compatible changes to the way they manage algorithms. In our case we are trying to connect to a server that has the following version of openssh and openssl : OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013.

What you expected to happen

Our existing SSH connections to keep working regardless of the version of OpenSSH used.

How to reproduce

Connect through the 2.2.4 SSHOperator to a server that do not support either RSA2 or the server-sig-algs protocol extension.

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow-providers-ssh==1!2.4.0

Deployment

Astronomer

Deployment details

No response

Anything else

In their changelog Paramiko indicates that to workaround that issue it is possible to specify disabled_algorithms={'keys': ['rsa-sha2-256', 'rsa-sha2-512']} in either SSHClient <paramiko.client.SSHClient.init> or Transport <paramiko.transport.Transport.init>.

I'm wondering what could be the impact to add this to the SSHClient?

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[ashb]

What version of the SSH provider do you have installed?

[talnicolas]

Does this answer your question?

astro@***********:/usr/local/airflow$ pip show apache-airflow-providers-ssh
Name: apache-airflow-providers-ssh
Version: 1!2.4.0
Summary: Provider package apache-airflow-providers-ssh for Apache Airflow
Home-page: https://airflow.apache.org/
Author: Apache Software Foundation
Author-email: [email protected]
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: paramiko, astronomer-certified, pysftp, sshtunnel
Required-by:

[potiuk]

You are free to downgrade to earlier paramiko version. Current limitation of SSH Provider is paramiko>=2.6.0 - the fact that paramiko >=2.9.0 does not accept your key is likely for a good reason (security) and if you choose to continue using version that is older (and might contain some security vulnerabilities) rather than upgrading your key - you can still do it.

Follow the examples here:

https://airflow.apache.org/docs/docker-stack/build.html#example-of-upgrading-airflow-provider-packages

You can build your own image using this Dockerfile (feel free to do it)

FROM apache/airflow:2.2.4
RUN pip install --no-cache-dir paramiko<2.9.0

Expecting that newer version of Airflow is using an older and potentially vulnerable version of a dependency to allow you to use potentially vulnerable key format is a bit selfish.

[talnicolas]

Thank you for your insightful comment @potiuk.

Just a comment if I may, insulting people when they are just coming for advice from « experts » won’t help building a safe and inclusive community.

[potiuk]

Just a comment if I may, insulting people when they are just coming for advice from « experts » won’t help building a safe and inclusive community.

Unfortunately, I have no control whether you feel insulted or not, that was certainly not my intention, My intention was to raise your awareness that your expectations were a bit selfish as they did not include wider thinking about the community but focused on your own needs. This is what I think is a bit selfish. I just expressed how I perceived it.

But I certainly had no intention to insult you - if you feel insulted - sorrry for that - but again, I have completely no control on your feelings when I express mine. I suggest you to look at it at a wider context.

[talnicolas]

Indeed you do not have that control, just like I don’t have any control on your ability to understand people’s request.

I'm wondering what could be the impact to add this to the SSHClient?

If that doesn’t feel to you like someone who came here specifically to understand that wider context you are talking about, well I’m sorry too.

That conversation is sterile now, I wish you a good continuation.

[potiuk]

I'm wondering what could be the impact to add this to the [SSHClient]

By all means - please propose a PR to expose this. You are most welcome.

Indeed I missed that part.

I was focusing more on the expectations: "Our existing SSH connections to keep working regardless of the version of OpenSSH used". But If you marked something as a bug, where this is a rather a feature request to expose this feature to the operators - this bug will be closed as invalid. This is not a bug and your expectation is more of a wish, than expectation and is not a bug by any means. Maybe that's why the part where you proposed a feature (in otherwise a bug report) was simply, well, missed.

No hard feelings - really :). And again - if you felt "insulted" - then that was a total misunderstanding. I simply closed the bug, believing that you "complain on a bug that you expected to work" rather than "proposed a feature to make it work better". The latter is absolutely welcome - actually you don't even need to make an issue for it - just create a PR and proceed as usual. Airflow has almost 2000 contributors - mostly volunteers so you are free to propose this (and I will be happy to review and approve it if it follows all the criteria from CONTRIBUTING.rst - and this is a great way to become one.

Again, apologies if you felt insulted, that was definitely not intention.

[potiuk]

This is really what I consider as "non selfish" - Identify a problem that small part of others might have, and propose a PR so that not only my problem is fixed but also others. For an open-source software that you get for free this is absolutely the best way to proceed.

[swapniel99]

Hi,

Just putting this here in case someone else faces this issue.

https://askubuntu.com/questions/1409105/ubuntu-22-04-ssh-the-rsa-key-isnt-working-since-upgrading-from-20-04

In case the server is on 22.0.4 then RSA keys are considered obsolete.

For me applying this solved the issue.