Authentication error with app-specific passwords

[dsfaller]

Trying to authenticate with app-specific password gives an error.

I am trying to reproduce an error that happens with Home Assistant's iCloud Plugin (https://www.home-assistant.io/integrations/icloud/#app-specific-passwords) when using app-specific password (home-assistant/core#53926). I used the code example from toothrobber to see if pyiCloud alone works or not. Instead of successful authentication the same error Missing apple_id fieldis raised.

Environment

• pyiCloud release with the issue (pip show pyicloud): 0.10.2
• Last working pyiCloud release (if known): unknown
• Service causing this issue: authentication (base.py)
• Python version (python -V): 3.9.7
• Operating environment (project deps/Docker/Windows/etc.): Docker as well as MacOS

Traceback/Error logs

Setup Time Zone
Py iCloud Services
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pyicloud/base.py", line 344, in _authenticate_with_token
    req = self.session.post(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pyicloud/base.py", line 157, in request
    self._raise_error(code, reason)
  File "/usr/local/lib/python3.9/site-packages/pyicloud/base.py", line 186, in _raise_error
    raise api_error
pyicloud.exceptions.PyiCloudAPIResponseException: Missing apple_id field

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 10, in <module>
    api = PyiCloudService("***", "***")
  File "/usr/local/lib/python3.9/site-packages/pyicloud/base.py", line 268, in __init__
    self.authenticate()
  File "/usr/local/lib/python3.9/site-packages/pyicloud/base.py", line 328, in authenticate
    self._authenticate_with_token()
  File "/usr/local/lib/python3.9/site-packages/pyicloud/base.py", line 350, in _authenticate_with_token
    raise PyiCloudFailedLoginException(msg, error)
pyicloud.exceptions.PyiCloudFailedLoginException: ('Invalid authentication token.', PyiCloudAPIResponseException('Missing apple_id field'))

Additional information

[rusjoan]

Same thing. Sad that this issue remains without any reply for almost 2 weeks

[jm66]

Apparently apple_id and password are now required here:

pyicloud/pyicloud/base.py

Lines 336 to 341 in bab549a

	data = {
	"accountCountryCode": self.session_data.get("account_country"),
	"dsWebAuthToken": self.session_data.get("session_token"),
	"extended_login": True,
	"trustToken": self.session_data.get("trust_token", ""),
	}

But still get an exception:

PyiCloudAPIResponseException('Authentication required for Account. (421)'))

[dsfaller]

I did more digging with pyicloud (being really a noob on this...). It seems pyicloud does not use the official Apple APIs but instead fakes to be a web browser logging in (see comments e.g. in #152 ). Apparently, Apple changed the behaviour of the web interface which breaks pyicloud.

The problem of the mentioned passage in @jm66 's comment is probably not that the apple_id / password is missing, but that the tokens (dsWebAuthToken and trustToken) cannot be found in the initial request that submits the password (

pyicloud/pyicloud/base.py

Lines 318 to 322 in bab549a

	req = self.session.post(
	"%s/signin" % self.AUTH_ENDPOINT,
	params={"isRememberMeEnabled": "true"},
	data=json.dumps(data),
	headers=headers,

).

When trying to login directly with curl commands / web browser to iCloud with app-specific passwords this is not possible - so it will probably need a rewrite of pyicloud to use the official APIs to make this work again :(

Update
Just tested with a regular 2FA password, and I then correctly get the responses for trustToken, account_country, session_id and session_token during the initial auth step above.
My conclusion is that app-specific passwords will not work without a rewrite of pyicloud... happy to be convinced otherwise.

[gcobb321]

Think about it this way. App specific passwords are for iOS apps that are meant to be logged into by a user through the apps own interface, and probably written in Swift that communicates via iOS on the device. Pyicloud is a Python program running on a computer that accesses the Apple iCloud account through a web interface. The purpose, mechanism and interface are completely different. I doubt Pyicloud will ever be changed to support this.

[dsfaller]

I agree with your conclusion... but as I understood the available documentation from Apple, app-specific passwords are exactly for those apps (not UI tools!) that cannot handle the modern Apple-specific APIs for 2-factor-auth. E.g. Thunderbird for Mail and Calendar (-> https://discussions.apple.com/thread/252366911).
But as you rightfully state, Pyicloud access the iCloud web interface - and from a security perspective it makes a lot of sense, that app-specific passwords do not work in the web UI to avoid a leaked password getting full access to your Apple account.
So, yes: at the end of the day I also think it's unlikely that Pyicloud will ever be changed to support this.

[brapifra]

Interestingly enough, it works for me in some cases:

• Account created via web (not linked to any iphone, ipad...) + regular password ✔️
• Account created via web (not linked to any iphone, ipad...) + app-specific password ✔️
• Account from a friend (linked to his iphone) + app-specific password ❌

So I wonder, what is the difference between my friend's account and the one I created via web? Is it just because mine is not linked to any apple device?

[brettinternet]

In the Home Assistant implementation, using email/password without an app-specific password appears to cause Apple to email login notices daily. https://www.reddit.com/r/homeassistant/comments/nek69p/icloud_integration_keeps_causing_apple_emails_to/

[apumapho]

Update Just tested with a regular 2FA password, and I then correctly get the responses for trustToken, account_country, session_id and session_token during the initial auth step above. My conclusion is that app-specific passwords will not work without a rewrite of pyicloud... happy to be convinced otherwise.

@dsfaller I'm having similar issue and for some reason iCloud thinks I'm submitting the deprecated app-specific password but I want to trigger the 2FA flow. When you say you "tested with a regular 2FA password" what did you do differently?

[dsfaller]

@apumapho Actually, I did not change anything other than using the 'main' password (not the app-specific password) when logging in. This alone triggered the 2FA flow for me. But haven't tested in the last months again... try deleting the tmp files of pyicloud, maybe some cookies prevent the 2FA flow.

[apumapho]

Thanks, @dsfaller. Getting inconsistent results depending on where (source IP, server vs lambda, etc.) I'm connecting from. Will investigate a bit more and report back in case helpful for others.

[zefoo]

Also seeing this issue. No 2FA set up, but I do have my account paired with a device (which I haven't turned on in months).

[polskikrol]

Indeed, the lack of this implementation does break a few things. Would be useful if both the http hook and API methods were supported alongside, for full MFA login or app password based login. Not sure if permission and feature mappings are on parity with both access methods.