MT.1033 is giving conflicting results between securing registration and legacy authentication

[dszp]

I'm getting the following results (four times, one for each account I've currently excluded from securing registration policies for now) when I add in the CAWhatIf tag to my run:

Failed icon MT.1033: User should be blocked from using legacy authentication ([[email protected]])
Failed
Overview
Checks if the Conditional Access Policies for blocking legacy authentication is active and used.
See Block legacy authentication - Microsoft Learn
Test Results
No conditional access policy securing security info registration.
Category: Regular users
Source: /home/runner/work/maester-tests/maester-tests/tests/Maester/Entra/Test-ConditionalAccessWhatIf.Tests.ps1

Here's a redacted screenshot of the same from the downloaded action results:

Note that it's saying the user isn't blocked from legacy authentication (they are, without exclusion), but that's what it says, and then it says that it's really because a CA policy isn't securing security info registration (it actually is limiting these specific users in a separate policy to requiring registration from one allowed country only but they are never in the office so they aren't fully locked down in the same way, and the rest of the users have a more specific policy that allows only specific IPs). But the message conflicts and seems to be a bug saying legacy auth but really is security registration messaging?

[merill]

Thanks I have updated the error message to refer to the legacy auth check. This is what the code is checking.

For your specific issue, these users are not being blocked from legacy authentication.

MT.1033 feels like a duplicate of MT.1010 (Test-MtCaBlockLegacyExchangeActiveSyncAuthentication).

@f-bader I know 1033 uses the what-if api, however do we need that given we have 1010 which does a more accurate check for a block legacy auth policy?

We might need to deprecate 1033.

[dszp]

Thanks, good to know that it's relating to legacy auth! I have been adjusting some security registration policies which is one reason it's confusing (one enforced and one report-only for now). However, that doesn't make sense for these four users triggering on this; I have a Disable Legacy Authentication CA that hasn't changed recently and has only a single exclusion, a break glass group with a single break-glass account in it (none of the four users causing MT.1033 to fail are in the group). I'm not sure how legacy auth would be disabled for any of these users based on the CA policy:

(Include is set to All Users.)

I verified group membership, and see nowhere that these four accounts could be excluded from these rules. MT.1010 that you mention, passes in the same run, linking to the policy that I posted a screenshot of above. So I feel like something is still off here somewhere, I'm just not sure where...

[mh-dh]

i think my issue belongs maybe to this one, "MS.AAD.1.1: Legacy authentication SHALL be blocked." failed even so we have a policy which blocks legacy for all but breakglass, but "MT.1010: At least one Conditional Access policy is configured to block legacy authentication for Exchange ActiveSync." passes.

[f-bader]

@merill Yes, this is a duplicate use case but I would still keep it as MT.1010 does not check who is excluded from the conditional access policy. This way you can have MT.1010 as success but MT.1033 fails for users that were manually excluded.

What's your take on this?

[dszp]

Edit to add: I'm running the tests using -Tag All,CAWhatIf so the WhatIf tests are included in my notes below. Maybe some of this is related to #152?

I'm seeing a ton of failures that are either actually incorrect, or don't provide enough information to troubleshoot. I started with the original legacy auth ones from this report but since most of them seem to be similar or inter-related, I'm listing them all here together (I can move this to a new post if you'd prefer):

I have the following failures (skipping failures that are true, these are just failures I believe to be incorrectly failing):

MT.1024: Entra Recommendation - Enable policy to block legacy authentication.

Saying "You have 1 of 32 users that don’t have legacy authentication blocked." This is likely my single break glass account that is in a single break glass group, but that's intentional--that group is excluded from every policy.

I'm also seeing that my tenant fails:

MT.1024: Entra Recommendation - Require multifactor authentication for administrative roles.

Saying "You have 1 of 6 users with administrative roles that aren’t registered and protected with MFA." This is incorrect, I have no users, certainly no admin users, that are not registered with and protected by MFA (the break glass account is registered with it but excluded from requiring it). Another false report.

Then I have this failure:

MT.1024: Entra Recommendation - Ensure all users can complete multifactor authentication.

Saying "You have 1 of 32 users that aren’t registered with MFA." I cannot find any users that are not registered with MFA no matter how I search, so this appears to be wrong, but I can't prove it since there's no list here to work from. How do I get this list?

I have this failure:

MT.1026: No hybrid user with permanent role assignment on Control Plane.

This does list two users, one that's my admin that authenticates to the CIPP application only (a global admin) and the global admin account I use separate from my daily account. But this reports each of these users as "These directory role assignments for HybridUser exist: [user] with Global Administrator on scope directory (tenant-wide)" But my tenant has never had hybrid on-premises AD join in its history, though it goes back to probably just after the BPOS days, it's purely cloud, so saying I have a hybrid user with any role is incorrect.

I have five users who with repeated full errors showing with different usernames in each:

MT.1033: User should be blocked from using legacy authentication ([email protected])

This is not correct; I even deleted and re-created the Block Legacy Authentication CA policy from the template wizard (though it matched the very old one I already had). It has no exceptions other than the break glass group, which isn't listed in this list, but the others have no exceptions to blocking legacy auth and therefore this check must be wrong, or CA is not behaving correctly since no user account should be able to bypass the block legacy auth templated rule.

I also have the following successful checks for two accounts:

MT.1034: Emergency access users should not be blocked ([email protected])

But the two accounts shown are NOT my emergency access users, they are my two admins used for two separate CIPP installations, but I have no idea how they are listed in this rule with category of "Emergency access users".

I'm not sure what do to resolve these, it's not urgent but I while I'm OK with PowerShell, I'm not great with Graph API or Pester. I've read the docs and links provided when possible for some of these tests. I update my maester-tests about once a week; these errors are from the version I sync'd 5 days ago but the tests ran yesterday evening. I can update the tests again if there have been substantial changes since then, I just haven't had a chance to do the update again this week.

[f-bader]

Hi @dszp thank you for taking the time and providing all the details.

For the results of MT.1024: Entra Recommendation I'm sorry to inform you that we can't do much about those. We use the results from Entra recommendations using Graph.

To remove FP see here

The other failures I will have to look into the tests a bit more closely. Would it be possible that I request some data from your environment and Graph? If so I would contact you directly

[dszp]

@f-bader thanks! I'm fine trying to resolve some of the recommendation issues directly, in fact I did before but they seem to have changed since, hah! I noticed that in some of the reports they link to the recommendations directly which is nice!

I'm happy to provide additional information as you alluded to. Again, not in a hurry, just trying to understand, figure out where I'm misunderstanding something or determine if something is a bug instead, to get the tool and the directions better--I'd love for this to be useful for my team and clients beyond internal but I need to have more confidence in the results and my own understanding, whichever it is that lacks :-)

I'm available on Discord as DavidSzp and I'm in the Entra server and #maester channel, but happy to connect however and whenever is the most convenient for you.

[dszp]

Just to update here, today I ran the latest Maester test updates, which re-ran the tests when I pushed the changes to GitHub, and in the action run log I noticed the following, which may be related to some of the issues above (though, I recall seeing a couple of weeks ago that I needed to add additional API permissions to the app and I added these, so I'm not sure why the forbidden errors):

WARNING: The version '5.5.0' of module 'Pester' is currently in use. Retry the operation after closing the applications.

[edit to remove Maester ASCII text logo]

Test-MtConditionalAccessWhatIf: /home/runner/.local/share/powershell/Modules/Maester/0.0.130/public/Test-MtCaWIFBlockLegacyAuthentication.ps1:23
Line |
  23 |  … iesResult = Test-MtConditionalAccessWhatIf -UserId $UserId -IncludeAp …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Response status code does not indicate success: Forbidden (Forbidden).
Test-MtConditionalAccessWhatIf: /home/runner/.local/share/powershell/Modules/Maester/0.0.130/public/Test-MtCaWIFBlockLegacyAuthentication.ps1:23
Line |
  23 |  … iesResult = Test-MtConditionalAccessWhatIf -UserId $UserId -IncludeAp …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Response status code does not indicate success: Forbidden (Forbidden).
Test-MtConditionalAccessWhatIf: /home/runner/.local/share/powershell/Modules/Maester/0.0.130/public/Test-MtCaWIFBlockLegacyAuthentication.ps1:23
Line |
  23 |  … iesResult = Test-MtConditionalAccessWhatIf -UserId $UserId -IncludeAp …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Response status code does not indicate success: Forbidden (Forbidden).
Test-MtConditionalAccessWhatIf: /home/runner/.local/share/powershell/Modules/Maester/0.0.130/public/Test-MtCaWIFBlockLegacyAuthentication.ps1:23
Line |
  23 |  … iesResult = Test-MtConditionalAccessWhatIf -UserId $UserId -IncludeAp …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Response status code does not indicate success: Forbidden (Forbidden).
Test-MtConditionalAccessWhatIf: /home/runner/.local/share/powershell/Modules/Maester/0.0.130/public/Test-MtCaWIFBlockLegacyAuthentication.ps1:23
Line |
  23 |  … iesResult = Test-MtConditionalAccessWhatIf -UserId $UserId -IncludeAp …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Response status code does not indicate success: Forbidden (Forbidden).
Test-MtConditionalAccessWhatIf: /home/runner/work/maester-tests/maester-tests/tests/Maester/Entra/Test-ConditionalAccessWhatIf.Tests.ps1:29
Line |
  29 |              Test-MtConditionalAccessWhatIf -UserId $id -IncludeApplic …
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Response status code does not indicate success: Forbidden (Forbidden).
Test-MtConditionalAccessWhatIf: /home/runner/work/maester-tests/maester-tests/tests/Maester/Entra/Test-ConditionalAccessWhatIf.Tests.ps1:29
Line |
  29 |              Test-MtConditionalAccessWhatIf -UserId $id -IncludeApplic …
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Response status code does not indicate success: Forbidden (Forbidden).
WARNING: [[email protected]] has been defined as Break Glass and removed from TooManyGlobalAdminsAssignedToTenantAlert
WARNING: [[email protected]] has been defined as Break Glass and removed from TooManyGlobalAdminsAssignedToTenantAlert
🔥 Maester test report generated at test-results/test-results.html

Just in case that helps with anything

[f-bader]

MT.1033: User should be blocked from using legacy authentication ([email protected])
@Cloud-Architekt I checked with @dszp and in this env the onPremisesImmutableId was set but the onPremisesSyncEnabled was empty. Should we switch and create a separate test to validate that for cloud users the onPremisesImmutableId is null?

[f-bader]

@Cloud-Architekt Ping on this issue, should we switch?