Revealing Full Details of Authentication Methods and Why This Might Be a Privacy Issue

Soon after releasing V1.2 of the Tenant Passwords and MFA Report (to add details about per-user MFA states), I was asked if it was possible to add more information for authentication methods, like the phone number used for SMS responses. My response was that I had covered the topic of reporting the details of authentication methods in a previous article and it was simply a matter of using the code from that article, updating it slightly to deal with the device-based passkeys recently introduced for Entra ID.

Not everyone likes cracking open a PowerShell script to insert code that they didn’t write. I don’t like messing with other peoples’ code either and will usually write my own version when necessary. In any case, I found some time and upgraded the script to include the expanded details, available in V1.3 of the script in GitHub.

Reporting Authentication Methods

Figure 1 shows the information about authentication methods registered for a user account in V1.2 of the report. The information given use the names from the MethodsRegistered property returned by the Get-MgBetaReportAuthenticationMethodUserRegistrationDetail cmdlet from the Microsoft Graph PowerShell SDK.

The problem is that the names aren’t very user-friendly. If you’re used to working with authentication methods, you probably recognize the values and understand what they mean. If not, this information might be useless.

More detail about the methods is available by running the Get-MgUserAuthenticationMethod cmdlet. Even so, some manipulation is necessary to generate human-friendly output. I’d done most of the work before, so it was easy to generate more information for each method. For instance, in Figure 2 you can see the mobile phone number used for SMS challenges and the version of the Authenticator app used for push notifications.

Because the script captures details in a PowerShell list, it’s also possible to query the list to find information like who uses a YubiKey FIDO2 key with a command like:

$Report | Where-Object {$_.'Authentication Methods' -like "*Yubikey*"}

The Privacy Issue

All was going well when I realized that the information generated about authentication methods might include some PII data, like the mobile phone number used for SMS responses. In most instances, I don’t think this will be a problem because details like mobile phone numbers are often included in the properties of Entra ID user accounts. The email addresses used to recover passwords via the Self-Service Password Reset (SSPR) feature are often personal accounts, so they might be more of an issue.

However, the regulations covering access to PII differs from country to country and it’s a good idea to cover all bases. The script now has a PrivacyFlag parameter. It’s a switch parameter, so the value is false by default. If set to true by including the parameter when running the script or by setting the flag explicitly, the script generates the names of the authentication methods without any details.

$PrivacyFlag = $true

On to The Next Version

I am sure that many other good ideas about how to add value to a report like this exist within the community. If you do, suggest the change through the Office 365 for IT Pros GitHub repository (for this script or any of our other scripts). Many people create a fork of our repository and work on updates that way. Whatever’s easier for you…

Learn more about how Microsoft 365 applications and Entra ID work on an ongoing basis by subscribing to the Office 365 for IT Pros eBook. Our monthly updates keep subscribers informed about what’s important across the Office 365 ecosystem.

Aiming for Workload-Agnostic Retention

Exchange and SharePoint on-premises servers both included retention processing. Exchange’s Messaging Records Management (MRM 2.0) persists today in the form of mailbox retention policies. Microsoft 365 retention policies, first introduced in 2017, replaced SharePoint’s equivalent (here’s an example of SharePoint retention in use) as part of a strategy to implement workload-agnostic data governance. In other words, deploy Microsoft 365 retention processing that worked for all workloads.

The initial implementation of retention policies (and classification labels, as Microsoft then called retention labels) supported Exchange Online (including public folders), SharePoint Online, OneDrive for Business, and Skype for Business Online.

Compliance Records and Microsoft 365 Retention Processing

Teams was the first Office 365 application to store compliance records in Exchange Online mailboxes. Compliance records are mail items created to capture details of chats and channel conversations, including messages sent by people who don’t have Exchange Online mailboxes like on-premises users, guest accounts, and external (federated) users (the substrate stores these messages in special cloud-only mailboxes). Yammer adopted the same mechanism in 2020, but only for networks configured in Microsoft 365 native mode. Planner announced that they would follow likewise but have not yet delivered this capability.

If you check the number of compliance items in your mailbox, you might find that you have more than you imagine. Most of my chatting happens in other tenants, but even so, I have quite a few compliance records for Teams chats (and fewer for Yammer conversations). Exchange Online doesn’t charge the storage consumed by these messages against user mailbox quotas.

$Folders = Get-ExoMailboxFolderStatistics -Identity Tony.Redmond@office365itpros.com -IncludeOldestAndNewestItems -FolderScope NonIPM
$Folders | Where-Object {$_.Name -eq "TeamsMessagesData" -or $_.Name -eq "Yammer"}| Format-Table Name, ItemsInFolder, FolderSize, NewestItemReceivedDate -AutoSize

Name              ItemsInFolder FolderSize                     NewestItemReceivedDate
----              ------------- ----------                     ----------------------
Yammer                      330 4.144 MB (4,344,857 bytes)     23/08/2021 08:23:50
TeamsMessagesData         14815 1.547 GB (1,660,565,924 bytes) 23/08/2021 17:22:38

Like any other Exchange Online content, Microsoft Search includes the compliance records in its indexes, which makes the records available for eDiscovery. Communications compliance policies also process compliance records to detect policy violations such as threatening or offensive behavior.

In addition to their role in eDiscovery, compliance records are the basis for retention processing for their originating workloads. Instead of having to create separate jobs to apply retention policies against the workload repositories, a single processing job runs against the compliance records and synchronizes policy actions such as removing items with the workloads.

Originally, the Exchange Managed Folder Assistant (MFA) processed the Teams compliance records. This was a pragmatic step because MFA could process the compliance records along with other mailbox content and avoided the need to create a special retention assistant for Teams. When MFA processed Teams content, it synchronized the results back to Teams, which applied the deletions to its Cosmos DB-based message store. Teams clients then picked up the changes from the message store and removed items in local caches.

The implementation worked well for Teams channel conversations and chat messages, but a better solution was needed to deal with the growth of workloads supporting retention policies and the need to process special cases like conversations in Teams private channels. To solve the problem, Microsoft implemented a specific background assistant to handle retention processing for all Microsoft 365 workloads except Exchange Online (Exchange is different because it supports both the older mailbox retention policies in addition to Microsoft 365 retention policies).

The Retention Assistant and the Microsoft 365 Substrate

Today, the retention assistant evaluates retention policies against OneDrive for Business, SharePoint Online, Yammer, and Teams using the “digital twins” of workload items stored in the Microsoft 365 substrate. Digital twins are copies of content stored in the Microsoft 365 substrate (in Exchange Online) to make it easier for shared services like Search and artificial intelligence services to process items. It’s much easier for a service to process items in a single place than having to deal with multiple repositories, each requiring a different interface (API).

In some cases, the digital twins are not complete copies, but they’re sufficient to allow shared services to operate efficiently and effectively. If you want to hear more about how the substrate works, consider signing up for TEC 2021 to hear Microsoft CTO for Modern Workplace Transformation Jeffrey Snover discuss the topic.

Substrate items coming within the scope of retention policies are subject to retention periods and actions. As the assistant processes items, it checks if the item’s retention period has lapsed, and if this is the case, invokes the retention action. This could involve additional processing, like putting the item into a manual disposition cycle, or it could be a simple delete action. The assistant interacts with the underlying workloads to ensure compliance with retention policy actions. For instance, if the assistant determines that the retention period for a document stored in a SharePoint Online document library has expired, it instructs SharePoint Online to move the document into the site recycle bin.

The Black Box of Microsoft 365 Retention Processing

Although it’s possible to track the processing done by MFA for mailbox items, Microsoft has not yet made an equivalent capability available for the retention assistant. This means that administrators who want to validate the effectiveness of retention processing need to make manual checks of content in repositories like SharePoint Online sites to ensure that items which should retention policies should remove are no longer present. As evident in this useful flowchart, understanding how retention policies process items can be a complex business, especially when locations like sites or teams come within the scope of multiple policies.

Making sense of complex retention policies is why the retention assistant exists. I’m sure it does its job very well. It would just be nice to be able to validate and understand exactly what actions the assistant takes for different locations. Is that too much to ask?

Learn more about how Office 365 really works on an ongoing basis by subscribing to the Office 365 for IT Pros eBook. Our monthly updates keep subscribers informed about what’s important across the Office 365 ecosystem.

No More Views Over Rio

Office 365 notification MC204985 published on February 27 brings the news that Microsoft is updating the default background image displayed when users sign into Azure Active Directory. The change is being made to reduce the bandwidth consumed by the view of Rio de Janeiro (Figure 1).

According to Microsoft roadmap item 61054, the new image is “1% the size of the previous default image which will reduce bandwidth requirements and improve perceived page load times.” The change is intended to benefit users in bandwidth-constrained locations. Deployment starts in early April and full deployment should be achieved in early May.

Important to Tell Users

Normally the change in a background image isn’t worth commenting upon, but given the number of phishing attempts that trick unwary users into entering their credentials into a false site, it’s important that people are informed about the change and expect to see the new slimmed-down background image, which is startlingly different in its plainness (Figure 2).

Do It Yourself Sign-In Backgrounds

A sign-in to Azure Active Directory has two phases: collect the username and then collect the password. The screens displayed differ depending on what you connect to:

• If you connect using a generic URL for an Office 365 service, like outlook.office.com, the Azure Active Directory sign-in page first captures the username and then captures the password. This is the page that is changing.
• If you connect using a domain variable, like outlook.office.com/office365itpros.com, Azure Active Directory checks if branding is specified for the tenant and applies it when displaying pages to collect the username and then the password.

Tenants that add customized elements for the sign-in page, usually some branding elements like logos or corporate colors, don’t need to recustomize their sign-in page after Microsoft updates the Azure Active Directory sign-in page.

Applying custom branding is easy to do. I created the effect shown in Figure in less than ten minutes. The trick is to select the graphics you need in advance and make sure that they are the right dimensions (1920 x 1080 pixels for the background, less than 300 KB).

FIDO2 Keys

The availability of FIDO2 keys has authentication easier for me recently. I have keys for both USB and USB-C from eWBM (Goldengate G310 and G320) and Yubico. Both work really well as a second source for multi-factor authentication against Azure Active Directory. Instead of receiving a code via SMS when prompted to authenticate, I plug a key into my PC to make Azure Active Directory happy…

Understanding Azure Active Directory Authentication

All of which brings me to the topic of authentication. Understanding how the authentication flow works from the time that someone sees the sign-in screen through MFA challenges and so on to reach an application is important knowledge. As we move from the era of basic authentication (simple) to modern authentication (different), it’s a good idea to refresh what we know about this important topic.

A series of videos featuring Stuart Kwan, Principal Program Manager in the Microsoft Identity Division explain how Azure Active Directory authentication works. These videos are available:

Authentication fundamentals: The basics | Azure Active Directory

Authentication fundamentals: Web applications | Azure Active Directory

Authentication fundamentals: Web single sign-on | Azure Active Directory

Authentication fundamentals: Federation | Azure Active Directory

The topic of modern authentication is dealt with in:

Authentication fundamentals: Native client applications- Part 1 | Azure Active Directory

and

Authentication fundamentals: Native client applications- Part 2 | Azure Active Directory

Good stuff and worth viewing.

The Office 365 for IT Pros eBook covers Azure Active Directory in its own chapter. Lots of good information written by a master of directories.

Multi-Factor Authentication Should Be Enabled for Privileged Accounts

Update: This article describes a script using Graph APIs to generate a report showing the MFA status for accounts and highlights administrative accounts that aren’t MFA-enabled. Given the deprecation of the MSOL module, you should switch to the Graph version.

If, like me, you were impressed by the case laid out in the July 10 2019 blog entitled Your Pa$$word doesn’t matter by Alex Weinert (Microsoft), you might wonder how to take his advice to “turn on MFA” for accounts. The process can take some time and user education because you can’t really enable MFA for “average users” if you don’t prepare them to deal with the resulting challenges, roll out the Microsoft Authenticator app, and so on. And then there’s the ongoing need to find unprotected Azure AD accounts to coach their owners about the wonders of MFA.

Reporting Accounts with Administrative Roles

One immediate step you can take is to clamp down on accounts holding one or more Azure Active Directory administrative roles that are not MFA-enabled. Microsoft has an Azure Active Directory usage and insights report about authentication methods to inform tenants about the accounts that are/are not enabled for MFA and self-service password reset (Figure 1), but it doesn’t highlight accounts holding administrative roles.

We discussed how to create a report of Azure AD accounts and their MFA status in a previous post and we can build on the techniques explored there to construct a PowerShell script to report accounts holding an administrative role that need to be protected with MFA. You can grab a copy of the script from GitHub.

What the Script Does

The script is imperfect and could do with improvement in terms of optimization and error handling, but it works. Here’s what it does.

Azure Active Directory defines directory roles which can be assigned to accounts to allow those accounts to perform specific tasks. In this case, we’re interested in some of the more highly-permissioned roles like Exchange Admin, so we use the Get-AzureADDirectoryRole cmdlet to grab the GUIDs identifying these roles and put them in variables. We then call the Get-AzureADDirectoryRoleMember cmdlet to populate another set of variables with details of the accounts that hold each role.

Write-Host "Finding Azure Active Directory administrative roles..."
$UserAccountAdmin = Get-AzureADDirectoryRole | Where-Object {$_.DisplayName -eq ‘User Account Administrator’} | Select ObjectId
$TenantAdmin = Get-AzureADDirectoryRole | Where-Object {$_.DisplayName -eq ‘Global Administrator’} | Select ObjectId
$TeamsAdmin = Get-AzureADDirectoryRole | Where-Object {$_.DisplayName -eq ‘Teams Service Administrator’} | Select ObjectId
$ExchangeAdmin = Get-AzureADDirectoryRole | Where-Object {$_.DisplayName -eq ‘Exchange Service Administrator’} | Select ObjectId
$SharePointAdmin = Get-AzureADDirectoryRole | Where-Object {$_.DisplayName -eq ‘Sharepoint Service Administrator’} | Select ObjectId

# Find out the set of accounts that hold these admin roles in the tenant
$UserAccountAdmins = Get-AzureADDirectoryRoleMember -ObjectId $UserAccountAdmin.ObjectID | Select ObjectId, UserPrincipalName
$TenantAdmins = Get-AzureADDirectoryRoleMember -ObjectId $TenantAdmin.ObjectID | Select ObjectId, UserPrincipalName
$TeamsAdmins = Get-AzureADDirectoryRoleMember -ObjectId $TeamsAdmin.ObjectID | Select ObjectId, UserPrincipalName
$ExchangeAdmins = Get-AzureADDirectoryRoleMember -ObjectId $ExchangeAdmin.ObjectID | Select ObjectId, UserPrincipalName
$SharePointAdmins = Get-AzureADDirectoryRoleMember -ObjectId $SharePointAdmin.ObjectID | Select ObjectId, UserPrincipalName

The script then calls the Get-MsolUser cmdlet to create a collection of Azure Active Directory licensed accounts (yes, there’s an odd mix of the Azure AD V1 and V2 cmdlets in the script; that’s because I can’t work out how to get MFA information using the V2 cmdlets). Using the MFA report code described here, each account is checked to see if it is MFA-enabled. We then create an array of accounts which are not MFA-enabled. These accounts are checked to see if they hold one of the administrative roles we’re interested in. If an account holds one or more of those roles, we capture its details.

# Extract users whose accounts don't have MFA
$MFAUsers = $MFAReport | ? {$_.MFAUsed -ne "Enforced"}
If (!($MFAUsers)) { Write-Host "No privileged accounts found without MFA protection" ; break}

Write-Host "Checking MFA status for accounts holding admin roles..."
$i = 0
$Report = [System.Collections.Generic.List[Object]]::new() # Create output file 
# Check Admin Roles if MFA not enabled
ForEach ($User in $MFAUsers) {
  $Roles = $Null
  If ($UserAccountAdmins.ObjectId -Contains $User.ObjectId) {
         Write-Host $User.DisplayName "Account holds the User Account Admin role" -ForegroundColor Red 
         $Roles = "Account Admin" }
  If ($TenantAdmins.ObjectId -Contains $User.ObjectId) {
         Write-Host $User.DisplayName "Account holds the Tenant Admin role" -ForegroundColor Red 
         If ($Roles -eq $Null) { $Roles = "Tenant Admin" } Else { $Roles = $Roles + "; Tenant Admin" } }
  If ($TeamsAdmins.ObjectId -Contains $User.ObjectId) {
         Write-Host $User.DisplayName "Account holds the Teams Admin role" -ForegroundColor Red 
         If ($Roles -eq $Null) { $Roles = "Teams Admin" } Else { $Roles = $Roles + "; Teams Admin" } }
  If ($ExchangeAdmins.ObjectId -Contains $User.ObjectId) {
         Write-Host $User.DisplayName "Account holds the Exchange Admin role" -ForegroundColor Red
         If ($Roles -eq $Null) { $Roles = "Exchange Admin" } Else { $Roles = $Roles + "; Exchange Admin" } }
  If ($SharePointAdmins.ObjectId -Contains $User.ObjectId) {
         Write-Host $User.DisplayName "Account holds the SharePoint Admin role" -ForegroundColor Red 
         If ($Roles -eq $Null) { $Roles = "SharePoint Admin" } Else { $Roles = $Roles + "; SharePoint Admin" } }      
 If ($Roles -ne $Null) {Write-Host "User" $User.DisplayName "is assigned the following roles:" $Roles -ForeGroundColor Yellow;  $i++ 
    $ReportLine = [PSCustomObject]@{
       User      = $User.DisplayName
       UPN       = $User.UserPrincipalName
       Roles     = $Roles
       MFA       = $User.MFAUsed }   
   $Report.Add($ReportLine) } #End if
}

Reporting Unprotected Azure AD Administrative Accounts

As the code runs, it generates information about accounts which are not MFA-enabled but hold administrative roles (Figure 2). Apart from anything else, this is a good way to see what accounts hold administrative roles and ask whether they need to hold those roles.

Finally, a CSV file is generated with details of accounts holding Azure AD administrative roles which are not MFA-enabled and exported to a CSV file. Figure 3 shows details of what the file contains as viewed through the Out-GridView cmdlet. It’s easy to pick out the accounts whose security needs to be improved.

As always, we’re happy to hear about other approaches to the problem. Please post your ideas as a comment to this post.

Need more solutions to common Office 365 Admin problems? The Office 365 for IT Pros eBook is packed full of ideas…

Knowing Which Accounts Are MFA-Enabled and Those That Don’t

When an issue like the November 19 2018 Azure outage occurs, the first thing that an administrator needs to know is what users are affected. In this case, the problem was that users whose accounts were enabled for multi-factor authentication (MFA) couldn’t complete the sign-in process and therefore weren’t able to access Exchange, SharePoint, and other Office 365 apps.

You can, of course go to the Microsoft 365 Admin Center and manage the MFA status of accounts there (Go to Active Users, click the More drop-down, and select Multifactor Authentication setup). However, the GUI is awful (Figure 1). It’s not part of Office 365 (it comes from Azure), hasn’t been updated in years, is slow, and is painful to navigate when you have a large tenant with many accounts.

Because the GUI is so terrible and slow, I usually turn to PowerShell to manage MFA for Office 365 accounts. The cmdlets you need are in the MSOnline module.

PowerShell Report

To start, let’s find out what accounts are MFA-enabled. Here’s a script to find and report the MFA status of all user accounts. The output is a CSV file. In a situation where an MFA outage is happening and you can’t access your normal administrative account, you’ll need to sign into Office 365 with a breakglass account.

CLS
Write-Host "Finding Azure Active Directory Accounts..."
$Users = Get-MsolUser -All | ? {$_.IsLicensed -eq $True}
$Report = [System.Collections.Generic.List[Object]]::new() # Create output file
Write-Host "Processing" $Users.Count "accounts..." 
ForEach ($User in $Users) {
   $MFAMethods = $User.StrongAuthenticationMethods.MethodType
   $MFAEnforced = $User.StrongAuthenticationRequirements.State
   $DefaultMFAMethod = ($User.StrongAuthenticationMethods | ? {$_.IsDefault -eq "True"}).MethodType
   If (($MFAEnforced -eq "Enforced") -or ($MFAEnforced -eq "Enabled")) {
      Switch ($DefaultMFAMethod) {
        "OneWaySMS"             { $MethodUsed = "One-way SMS" }
        "TwoWayVoiceMobile"     { $MethodUsed = "Phone call verification" }
        "PhoneAppOTP"           { $MethodUsed = "Hardware token or authenticator app" }
        "PhoneAppNotification"  { $MethodUsed = "Authenticator app" }
      } #End Switch
    }
    Else {
          $MFAEnforced= "Not Enabled"
          $MethodUsed = "MFA Not Used" }
  
   $ReportLine = [PSCustomObject] @{
           User        = $User.UserPrincipalName
           Name        = $User.DisplayName
           MFAUsed     = $MFAEnforced
           MFAMethod   = $MethodUsed }
                 
    $Report.Add($ReportLine) 
} # End For

Write-Host "Report is in c:\temp\MFAUsers.CSV"
$Report | Select Name, MFAUsed, MFAMethod | Out-GridView
$Report | Export-CSV -NoTypeInformation c:\temp\MFAUsers.CSV

Figure 2 shows the contents of the report as viewed through the Out-GridView cmdlet.

Update: The script described in this post uses the Microsoft Services Online module (MSOL). Microsoft will deprecate the module at the end of 2022. You should upgrade scripts to use Graph API queries or cmdlets from the Microsoft Graph PowerShell SDK. See this post for information about how to extract and report details of authentication methods used by Azure AD accounts.

Azure Active Directory Sign-in Log

Because the refresh token of MFA-enabled accounts don’t necessarily expire during an outage, a more nuanced approach is to check the Azure Active Directory sign-in logs for events with error codes 50076, 50074, or 50058, all of which flag problems signing in with an MFA-enabled account. You can review sign-ins interactively or download events to a CSV file. You can then use Excel to look through the failed sign-ins or import the CSV file into Power BI to analyze it there. In either case, you’ll end up with a set of accounts who are experiencing problems signing into Office 365.

Disabling MFA for Accounts

When a cloud outage happens, there’s no guarantee when normal service will resume. As everyone discovered on November 18, the process to figure out the root cause of a problem, come up with a fix, test it, and then deploy into production across a massive multi-tenant infrastructure can take many hours. While this process was in going on, the only workaround administrators had was to disable MFA for user accounts. This is easily done for a specific user:

$MFAOptions = @()
Set-MsolUser -UserPrincipalName Kim.Akers@Office365itpros.com -StrongAuthenticationRequirements $MFAOptions

To disable a batch of users, simply form a collection and call the Set-MsolUser cmdlet for each account. In this case, we disable MFA for all members of the Marketing department. If you extract failed sign-in data from Azure Active Directory, you can create a CSV file holding the set of accounts to process and use it as the input.

$MFAOptions = @()
$Accounts = Get-MsolUser -Department Marketing | ? {$_.StrongAuthenticationMethods -ne $Null}
ForEach ($A in $Accounts) {
Set-MsolUser -UserPrincipalName $A.UserPrincipalName -StrongAuthenticationRequirements $MFAOptions}

Enabling MFA for Accounts

Later on when the outage is over, we need to re-enable MFA for the same set of accounts (we’ll assume that you have saved details of the accounts and populated those names in the $Accounts variable). The PowerShell snippet shown below sets up MFA based on one-way SMS (text) messages containing 6-digit codes.

$MFA = New-Object –TypeName Microsoft.Online.Administration.StrongAuthenticationRequirement
$MFA.RelyingParty = "*"
$MFA.State = "Enabled"
$MFAOptions = @($MFA)
$MFAMethod = New-Object –TypeName Microsoft.Online.Administration.StrongAuthenticationMethod
$MFAMethod.IsDefault = $True
$MFAMethod.MethodType = "OneWaySMS"
$MFAMethods = @($MFAMethod)

ForEach ($A in $Accounts) {
    Set-MsolUser –UserPrincipalName $A.UserPrincipalName –StrongAuthenticationRequirements $MFAOptions –StrongAuthenticationMethods $MFAMethods }

These examples aren’t a complete solution. Instead, they’re intended to give administrators a start to figure out how to manage MFA settings with PowerShell.

For more details about Office 365 MFA, see the online documentation.

We cover multi-factor authentication for Office 365 in Chapter 3 of the Office 365 for IT Pros eBook.

Updated November 22 with Microsoft preliminary root cause analysis and then on November 26 with the final version.

Two Outages in Ten Weeks

The second major outage for Azure multi-factor authentication (MFA) in two months brought some Office 365 to a halt on Monday, November 19. That is, until administrators understood what was happening and and disabled MFA for accounts to allow users to sign in.

The first outage occurred on September 4 when lightening struck Microsoft’s San Antonio datacenter. Post-postmortems published after the event (here’s the VSTS version) revealed how the impact of the outage rippled across multiple Microsoft cloud services, including MFA.

The Problem

The original problem statement was:

“Customers in Europe, Asia-Pacific and the Americas regions may experience difficulties signing into Azure resources, such as Azure Active Directory, when Multi-Factor Authentication is required by policy.”

The official word on the Azure incident history page says:

“A recent update was deployed to improve connections to caching services for the MFA service, this introduced a race condition which prevented users from being able to sign-in, or carry-out self-service password resets, when using MFA services…

Engineers initially rolled back the deployment which eliminated the connection between the Azure MFA service and the backend caching service. Engineers subsequently cycled impacted servers which allowed authentication requests to succeed.“

On November 26, Microsoft updated the root cause analysis to say:

“There were three independent root causes discovered. In addition, gaps in telemetry and monitoring for the MFA services delayed the identification and understanding of these root causes which caused an extended mitigation time. 

The first two root causes were identified as issues on the MFA frontend server, both introduced in a roll-out of a code update that began in some datacenters (DCs) on Tuesday, 13 November 2018 and completed in all DCs by Friday, 16 November 2018. The issues were later determined to be activated once a certain traffic threshold was exceeded which occurred for the first time early Monday (UTC) in the Azure West Europe (EU) DCs. Morning peak traffic characteristics in the West EU DCs were the first to cross the threshold that triggered the bug. The third root cause was not introduced in this rollout and was found as part of the investigation into this event.

1. The first root cause manifested as latency issue in the MFA frontend’s communication to its cache services. This issue began under high load once a certain traffic threshold was reached. Once the MFA services experienced this first issue, they became more likely to trigger second root cause.
2. The second root cause is a race condition in processing responses from the MFA backend server that led to recycles of the MFA frontend server processes which can trigger additional latency and the third root cause (below) on the MFA backend.
3. The third identified root cause, was previously undetected issue in the backend MFA server that was triggered by the second root cause. This issue causes accumulation of processes on the MFA backend leading to resource exhaustion on the backend at which point it was unable to process any further requests from the MFA frontend while otherwise appearing healthy in our monitoring.”

The worrying part of the story is that a code update proved to be unreliable when introduced into production, which is not good for Microsoft’s cloud quality and testing regimes.

Early-morning Incident

The incident started at 04:39 UTC and stopped users completing the MFA secondary challenge and sign-into services like Office 365. For instance, the text message containing the code to prove that the account owner has the device registered for the account never arrives, meaning that the challenge shown below can never be completed.

According to Microsoft, the problem started to ease at around 14:45 UTC after a hotfix was deployed. It takes a long time to deploy code fixes across a massive infrastructure and many tenants were affected by the problem for several hours afterwards. I first managed to authenticate with MFA at 18:23 UTC. Others were not so lucky and the lack of connectivity persisted for several hours afterwards. The incident slowly wound down and, at the time of writing, the situation is being monitored by Microsoft but everything is working.

Overall, Monday wasn’t a great day for users or administrators alike. MFA-enabled accounts couldn’t access Office 365 applications if their refresh token expired and they needed to go through the MFA sign-in process to reauthenticate. Administrators, whose accounts are more likely to be protected by MFA, hit the same issue and lost access to Office 365 and Azure portals.

Communication Blues

During the incident, Microsoft communicated with customers via the Office 365 Admin Center and the Azure status page, but didn’t always give the same story in both places. For instance, around 14:00, on the service health page of the Office 365 portal, we learned:

While we continue to develop the code update, we’re exploring additional workstreams to find a path to mitigation.

While at the same time, the Azure portal told watchers that:

Engineers have explored mitigating a back-end service via deploying a code hotfix, and this is currently being validated in a staging environment to verify before potential roll-out to production. Engineers are also continuing to explore additional workstreams to expedite mitigation. 

Obviously, the text posted on the Azure portal gave more complete information. One wonders why the people responsible for updating the portals couldn’t have used the same story?

On the one hand it’s reasonable that Azure should have its own communications because its services are used by more than Office 365. On the other, Microsoft runs both services and it is strange to have Office 365 give less information than is publicly available elsewhere.

Microsoft Says to Use MFA

Microsoft recommends that Office 365 tenants use MFA. The priority is to protect accounts with privileged access, like tenant administrators, followed by high-profile accounts, like those used by executives. However, for maximum protection against hacker attacks, all Office 365 user accounts should use MFA.

Microsoft reinforces the message by giving tenants that use MFA a big boost in their Secure Score (if that means anything to you). Generally speaking, I agree with Microsoft and think that all accounts should be protected. Until, that is, something bad happened and users can’t sign into Office 365 or any other Microsoft cloud service because of an MFA failure. It’s worth underlining that the problem only surfaces for new connections or when a user’s access token expires and needs to be renewed. While the access token is still valid, users can continue to connect even with a broken MFA service.

Disable MFA to Keep Working

The question then becomes what should a tenant in case of an extended an MFA outage when users need to get into Office 365 or other services and can’t because they cannot complete the MFA process. The obvious answer is to disable MFA for affected user accounts while the outage continues and then re-enable the accounts for MFA immediately the outage is over and normal service resumes. Of course, this assumes that you can still sign into an administrator account to reset MFA for users. But keeping an admin account that isn’t secured with MFA is a bad idea, isn’t it?

Breakglass Accounts

Not if it’s a “breakglass” account. In other words, a privileged account that can be used in case of emergency and other administrator accounts are unavailable for some reason. See this article for a discussion on the topic as well as some advice from Microsoft on how to manage emergency administrative accounts for Azure Active Directory.

The lesson of the outage are clear. If they use MFA (as they should), Office 365 tenants need to be prepared to deal with outages. Knowing what accounts are protected with MFA is a start, being able to disable MFA if needed is a good idea (and revert once the problem eases), and having a breakglass account is also sensible.

The process to enable Office 365 accounts for multi-factor authentication is covered in Chapter 3 of the Office 365 for IT Pros eBook. We’re not so hot on disabling MFA…