Sample from docs error: Reason: Unsupported Media Type

[Atoms]

Expected Behavior

object added and on.create handler is run

Actual Behavior

kopf run worker.py --verbose --debug --standalone                                                                                1 ↵
[2019-07-04 12:10:47,468] asyncio              [DEBUG   ] Using selector: EpollSelector
[2019-07-04 12:10:47,502] kopf.clients.auth    [DEBUG   ] configured via kubeconfig file
[2019-07-04 12:10:47,511] kubernetes.client.re [DEBUG   ] response body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.99.100:8443"}]}

[2019-07-04 12:10:47,524] kubernetes.client.re [DEBUG   ] response body: {"apiVersion":"zalando.org/v1","items":[],"kind":"EphemeralVolumeClaimList","metadata":{"continue":"","resourceVersion":"89870","selfLink":"/apis/zalando.org/v1/ephemeralvolumeclaims"}}

[2019-07-04 12:10:56,869] asyncio              [INFO    ] poll took 9342.085 ms: 1 events
[2019-07-04 12:10:56,973] kopf.reactor.handlin [DEBUG   ] [wimd-streams/my-claim] First appearance: {'apiVersion': 'zalando.org/v1', 'kind': 'EphemeralVolumeClaim', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"zalando.org/v1","kind":"EphemeralVolumeClaim","metadata":{"annotations":{},"name":"my-claim","namespace":"wimd-streams"},"spec":{"size":"10G"}}\n'}, 'creationTimestamp': '2019-07-04T10:10:56Z', 'generation': 1, 'name': 'my-claim', 'namespace': 'wimd-streams', 'resourceVersion': '89883', 'selfLink': '/apis/zalando.org/v1/namespaces/wimd-streams/ephemeralvolumeclaims/my-claim', 'uid': 'd95088a8-02b2-47a4-bc62-a7b8946d6206'}, 'spec': {'size': '10G'}}
[2019-07-04 12:10:56,973] kopf.reactor.handlin [DEBUG   ] [wimd-streams/my-claim] Adding the finalizer, thus preventing the actual deletion.
[2019-07-04 12:10:56,973] kopf.reactor.handlin [DEBUG   ] [wimd-streams/my-claim] Patching with: {'metadata': {'finalizers': ['kopf.zalando.org/KopfFinalizerMarker']}}
[2019-07-04 12:10:56,973] root                 [DEBUG   ] Setting up syn executor
[2019-07-04 12:10:56,984] kubernetes.client.re [DEBUG   ] response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json","reason":"UnsupportedMediaType","code":415}

[2019-07-04 12:10:56,984] asyncio              [DEBUG   ] poll took 9.618 ms: 1 events
[2019-07-04 12:10:56,985] kopf.reactor.queuein [ERROR   ] functools.partial(<function custom_object_handler at 0x7f646c0c0048>, lifecycle=<function asap at 0x7f646a8dc1e0>, registry=<kopf.reactor.registries.GlobalRegistry object at 0x7f646c8fd240>, resource=Resource(group='zalando.org', version='v1', plural='ephemeralvolumeclaims'), event_queue=<Queue at 0x7f646a7becc0 maxsize=0 _getters[1]>, freeze=<asyncio.locks.Event object at 0x7f646a79d390 [unset]>) failed with an exception. Ignoring the event.
Traceback (most recent call last):
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 145, in worker
    await handler(event=event)
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kopf/reactor/handling.py", line 128, in custom_object_handler
    await patching.patch_obj(resource=resource, patch=patch, body=body)
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kopf/clients/patching.py", line 41, in patch_obj
    await loop.run_in_executor(config.WorkersConfig.get_syn_executor(), functools.partial(patch_func, **request_kwargs))
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kubernetes/client/apis/custom_objects_api.py", line 1951, in patch_namespaced_custom_object
    (data) = self.patch_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, body, **kwargs)
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kubernetes/client/apis/custom_objects_api.py", line 2057, in patch_namespaced_custom_object_with_http_info
    collection_formats=collection_formats)
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 393, in request
    body=body)
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kubernetes/client/rest.py", line 286, in PATCH
    body=body)
  File "/home/asterns/projects/wimd-operator/env/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (415)
Reason: Unsupported Media Type
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Thu, 04 Jul 2019 10:10:56 GMT', 'Content-Length': '263'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json","reason":"UnsupportedMediaType","code":415}

Steps to Reproduce the Problem

1. launch minikube
2. add crd from docs
3. launch sample operator from docs
4. apply object from docs

Specifications

• Platform: minikube
• Kubernetes version: (use kubectl version)

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"archive", BuildDate:"2019-06-20T22:55:20Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

• Python version: (use python --version)
  Python 3.7.3
• Python packages installed: (use pip freeze --all)

aiohttp==3.5.4
aiojobs==0.2.2
astroid==2.2.5
async-timeout==3.0.1
attrs==19.1.0
cachetools==3.1.1
certifi==2019.6.16
chardet==3.0.4
Click==7.0
google-auth==1.6.3
idna==2.8
iso8601==0.1.12
isort==4.3.21
kopf==0.17
kubernetes==10.0.0
lazy-object-proxy==1.4.1
mccabe==0.6.1
multidict==4.5.2
oauthlib==3.0.1
pip==19.1.1
pyasn1==0.4.5
pyasn1-modules==0.2.5
pylint==2.3.1
python-dateutil==2.8.0
PyYAML==5.1.1
requests==2.22.0
requests-oauthlib==1.2.0
rsa==4.0
setuptools==40.8.0
six==1.12.0
typed-ast==1.4.0
urllib3==1.25.3
websocket-client==0.56.0
wrapt==1.11

[nolar]

@Atoms Thanks for reporting. I could reproduce it locally — will take a look later.

As a quick workaround, please install the Kubernetes client version 9:

pip install 'kubernetes<10.0.0'

---

The error happens with kubernetes==10.0.0 only, which was released 03.07.2019 @ ~23:00 (this past night): https://github.com/kubernetes-client/python/releases/tag/v10.0.0

It seems, the error comes from the custom resource patching call. Kopf does not specify any media-/content-type, it is the client library's internals. So, it looks like an error in the client library itself — I will investigate further.

Perhaps, switching to pykube-ng (#15) can solve the issue, as there we specify the exact Content-Type of the patch.

[Atoms]

yes, i already tried to downgrade and it works, you know more internals of kopf and how it uses kubernetes client, can you open issue within kubernetes-client, and i think as workaround is there this can be closed as it's not kopf issue.

P.S. just started with kopf

[nolar]

Created kubernetes-client/python#866 with a reproducible example (unrelated to Kopf).

[nolar]

Meanwhile, I've released kopf==0.17.post1 hotfix with kubernetes<10.0.0 restriction — just to have it runnable by default, i.e. when using pip install kopf.

[nolar]

Found the cause (and added it to kubernetes-client/python#866, with links):

In 10.0.0, they have changed the content-type from application/merge-patch+json (a dict with fields to override) to application/json-patch+json (a list of operations per field, see wikipedia).

The data sent by Kopf was a dict as per application/merge-patch+json, and it worked until 10.0.0.

At the moment, there is no way how we can control the content-type used by Kubernetes client — the content-type selection is hard-coded, and does not depend on the content data itself.

In pykube-ng branch of Kopf, the content type is specified by us explicitly. So, this problem will not appear there.

[nolar]

So far, the issue was fixed with kubernetes>=10.0.1 some time ago.