October 15 Date for First Round of Azure MFA Requirement Enforcement

Microsoft’s decision to enforce multifactor authentication (MFA) for access to Azure properties certainly caused a great deal of kerfuffle. The original May 14, 2024 announcement didn’t help itself by proclaiming that “Microsoft will require MFA for all Azure users,” an assertion that led people to believe that any sign-in to Azure AD (Entra ID) would require MFA. At the time, I thought the idea is sound but the communication was woeful.

Roll forward to June 27, 2024, and a follow-up post to the Core Infrastructure and Security blog in the Microsoft Technical Community attempted to clear the confusion. Given the number of questions since, the signs are that people are still unsure what’s happening. Let me try to unravel the situation.

First, not everyone is affected by the change. MFA is only enforced for access to some Azure cloud properties. According to message center notification MC862873, Microsoft will require MFA for access to the Entra admin center, Azure portal, and Intune admin center (including services like Windows 365 Cloud PC) on or after October 15, 2024. Although I am not certain, Windows 365 Cloud PC seems like the only place where non-administrators might be affected by the MFA requirement.

Phase 2 of the implementation in early 2025 will see the requirement spread to the Azure CLI, Azure PowerShell, and infrastructure as code (IaC) tools. The good news is that only administrator accounts typically use these sites. Normal users are unaffected by the change. And anyway, administrator accounts should be protected by MFA. If they are not, the tenant has other problems.

Second, Microsoft 365 tenants that use Security Defaults already mandate MFA for connections to administrator portals.

Third, tenants that don’t use Security Defaults and have conditional access policies in place to control connections instead probably have a policy to require MFA for connections to administrative sites. Microsoft even publishes a policy template to make it easy for tenants to enable this control.

Workload identities, such as the managed identities used with Azure Automation, are unaffected. However, if Azure Automation runbooks include user identities (username and password credentials) for authentication, they might not work after Microsoft deploys the requirement for MFA access to Azure. For instance, if a runbook uses the Azure PowerShell module, it must use a managed identity or service principal (app) to connect.

Break glass accounts are affected too. If ever needed, these accounts are likely going to access Azure administrative sites and use Azure administrative tools, so the new guidance is to modify the previous practice of using a long and complicated password and add the protection for the accounts with a strong MFA authentication method like FIDO2 or certificates.

Postponing the Azure MFA Requirement

Microsoft says that they will allow a grace period to tenants who need some extra time. Organizations that use non-Microsoft MFA solutions might be in this category (support for external MFA providers is in preview). If in doubt, use the link in the message center notification post to request a postponement (Figure 1).

More information is available in the Microsoft planning document covering the new requirement.

Microsoft’s Script to Reveal Access to Azure Administrative Tools

In their post, Microsoft says that the Export-MsIdAzureMfaReport cmdlet from the MSIdentityTools PowerShell module as a way to uncover accounts likely to be affected by the change. The cmdlet “exports the list of users that have signed into the Azure portal, Azure CLI, or Azure PowerShell over the last 30 days by querying the sign-in logs,” so it’s a useful way to get an insight into who might be affected by the MFA requirement. I ran the cmdlet for my tenant and found that I need to act for an account used for utility background jobs (Figure 2).

The data used by the cmdlet is available to administrators to create their own version. For instance, the script covered in this article explains how to combine data from several sources to create a picture of MFA usage within a tenant.

No Need to Panic – the Azure MFA Requirement is a Good Thing

The bottom line is that there’s no need to panic unless you have a bunch of background jobs that use user credentials for authentication or forget to update your break glass accounts. Normal users will make unperturbed progress through the change. This is predominantly a good update to force administrators who haven’t yet understood the absolute need to protect their accounts with MFA to change their behavior. After all, why leave an open door ready for the bad guys to kick in?

Support the work of the Office 365 for IT Pros team by subscribing to the Office 365 for IT Pros eBook. Your support pays for the time we need to track, analyze, and document the changing world of Microsoft 365 and Office 365.

HVE and ECS Compete for Different Customers of Microsoft Cloud Email Services

I need to apologize to some of the subscribers to the Office 365 for IT Pros eBook. Over the last few weeks, I’ve been using you as the targets for emails sent using Exchange Online High-Volume Email (HVE) and the Azure Email Communication Service (ECS).

Both solutions focus on sending large quantities of email. HVE is more internal-focused but can handle external messages. HVE is part of Exchange Online and intended to help customers move off on-premises servers to handle traffic generated by multi-functional devices and applications. ECS is a standalone offering that can handle large volumes of external email such as newsletters, subject to thresholds set by Microsoft. According to Microsoft, ECS is very popular and handles large amounts of messages daily.

HVE is in preview and is free to use today. When it’s generally available, HVE will likely cost for some traffic. ECS is already a pay-as-you-go service that must be funded by an Azure subscription.

Seeking Test Email Targets for Microsoft Cloud Email Services

When setting out to test the effectiveness of emailing solutions, you need large numbers of target recipients. Little is to be learned by sending a couple of messages to a few internal recipients. To run a better trial, I decided to use HVE and ECS to send reminder messages to subscribers of the 2024 edition of the Office 365 for IT Pros eBook to ask if they wanted to take advantage of an offer to extend their subscription. Sending email to ask people to buy something or take out a subscription seemed like a pretty good scenario to test the useability of HVE and ECS.

Comparing HVE and ECS

Overall, HVE is easier to use. Less setup is required, and the PowerShell used to generate and submit messages is based on the old (deprecated) Send-MailMessage cmdlet. No shortage of articles can be found on the internet to tell you how to use Send-MailMessage. Because of the need to provide an email service for apps and devices, HVE uses a restricted form of basic authentication with the SMTP AUTH protocol. Support for modern authentication is coming, but using basic authentication for internal messages will make the switchover to HVE much easier.

HVE reporting (Figure 1) is basic. More comprehensive reporting is built into ECS. In both cases, feedback from sent messages is minimal, so figuring out what happened to messages is tough. ECS can tell you the number of messages it failed to send but HVE is silent on this point. However, HVE is in preview and Microsoft says that they will deliver better reporting when the solution is generally available.

The ECS setup is more complicated if you’re unaccustomed to dealing with Azure resources and billing. ECS uses an Entra ID app for authentication and to prove that an app (like a PowerShell script) has the right to submit messages to the service. Creating and submitting messages to ECS is similar to using Graph-based cmdlets like Send-MgUserMail. Some differences exist because a different API is used, but the basics of building a hash table of message parts and converting it to JSON before sending won’t be unfamiliar.

Throttling and thresholds were the biggest issue I encountered with both ECS and HVE. It took a little while to find where limits applied in practice and to investigate ways around them. Microsoft has a documented process for applying for higher limits for ECS but my ability to navigate the process failed and I never managed to achieve a higher threshold. Microsoft is careful with HVE while it is in preview and some limitations (like the 2,000 external recipients per tenant daily) are hardcoded and won’t change until the software reaches general availability.

Testing of both Microsoft Cloud Email Services Proves Valuable

As always, the opportunity to conduct realistic tests over a sustained period proved invaluable in gaining an understanding about how HVE and ECS work. In my case, sending thousands of reminder messages to Office 365 for IT Pros subscribers certainly taught me a lot. You can read more about my experiences in articles covering HVE and ECS in-depth. Other articles about HVE and ECS are available on the internet, but most are content to send just a few test messages and then declare success. That’s no way to exercise a high volume email system.

If you’re interested in one of these services, my advice is to spin up both and test using a sample of messages that your organization needs to send. Exchange Online tenants will, I think, select HVE, but I can see why ECS has its attractions especially if the focus is on sending large quantities of email to external recipients. Beauty is in the eye of the mail sender.

Learn about using Exchange Online and the rest of Office 365 by subscribing to the Office 365 for IT Pros eBook. Use our experience to understand what’s important and how best to protect your tenant.

Azure MFA Required for Connections from July 2024

Updated

Microsoft’s May 14 announcement that they will require multifactor authentication (MFA) for access to Azure services certainly kicked up a heap of questions. The sad fact is that Microsoft has a good message to communicate around increasing the security of connections to the Azure portal (and assumedly for Azure PowerShell sessions) but failed miserably to communicate that message.

After reading the announcement, my take is that Microsoft will deploy the requirement for MFA for connections to Azure services from July 2024 onward. Microsoft says that they will communicate with tenant administrators with details about what they plan to do and when they will do it, and that the deployment will be “gradual and methodical to minimize impact on your use cases.”

The Reasons to Use Multifactor Authentication

Excellent reasons exist to use MFA to protect connections. Anyone who uses basic authentication (username and password) for administrator accounts (or any user account) is playing with fire because their account is a prime target for compromise. Microsoft cites two different numbers (99.2% and 99.9%) for the ability of MFA to block attacks like password sprays (I’ve seen both figures cited elsewhere), but this slip of the pen doesn’t matter.

What does matter is that MFA offers better protection for account compromise, especially if you use strong authentication methods like the Microsoft Authenticator app, including the recently-added support for passkeys.

Another important point is that the Entra ID community is not doing a great job of deploying and using MFA. According to Microsoft VP for Identity Security Alex Weinert, MFA protected 38% of Entra ID accounts in February 2024. Perhaps the recent announcement of support for external authentication methods will help drive the percentage higher because organizations can leverage investments in MFA solutions that don’t come from Microsoft.

Communication Issues Around Azure MFA

Good as MFA undoubtedly is, Microsoft just didn’t get their point across.

First, Microsoft didn’t clarify which users will need to use MFA. Including the phrase “for all Azure users” in the announcement title made a major contribution to the confusion. My understanding is that MFA will be required to connect to the Azure portal, so that limits the set of affected users to people who sign into the Azure portal to work with subscriptions, resource groups, automation accounts, billing, and so on. In short, not your average Microsoft 365 user (who probably don’t know or want to know about the Azure portal).

Update: Microsoft posted a comment to the article saying that MFA applies to, “All users signing into Azure portal, CLI, PowerShell, or Terraform to administer Azure resources are within the scope of this enforcement.”

Second, Microsoft didn’t say how they will enforce MFA. The text points to the MFA setup wizard in the Microsoft 365 admin center (Figure 1), which focuses heavily on enforcing MFA through conditional access policies.

Conditional access policies work very well, but they require Entra ID P1 licenses. This is probably not an issue in enterprise tenants where Entra ID premium licenses cover many different features, but it could be a problem for small businesses. It’s the same issue around imposing extra cost that occurs in Microsoft’s campaign to move Office 365 per-user MFA to conditional access policies.

Perhaps Microsoft plans to use a mechanism like the way Security Defaults requires accounts with administrator roles to use MFA with the Authenticator app. In other words, no conditional access policies and no need for premium licenses. Of course, if organizations want to use conditional access policies to enforce MFA for inbound connections they can do so and fulfil the requirements of Azure. Microsoft says that no-opt is available except through an exception process that isn’t yet defined.

A long time ago when I started to write magazine articles, an editor told me not to assume that the reader understood the topic I wrote about and to answer questions in the text that I assumed people already knew the answers to. That good advice has stood the test of time. I often feel that Microsoft communicates in a way where they assume the target readership understands the full context of the topic being discussed. It would be nice if they wrote text that is a lot more specific and complete.

Rushing to Embrace Security

Microsoft’s Security Future initiative is a worthy venture, but it seems like Microsoft engineering groups are rushing to implement blocks to meet their schedule rather than understanding that announcing what could be a major change in mid-May for implementation in July (initially for the Azure portal) is not appreciated by customers. It’s not as if tenant administrators only need to concentrate on securing Azure better. Every engineering group in the Microsoft 365 ecosystem is tightening security and the cumulative workload created for tenant administrators is something that I don’t think individual program managers contemplate.

The net is that no one can argue against better security connections to Azure services if implemented in a measured and well-communicated manner. It seems like Microsoft’s May 14 announcement was a tad rushed and that’s a real pity.

Stay updated with developments across the Microsoft 365 ecosystem by subscribing to the Office 365 for IT Pros eBook. We do the research to make sure that our readers understand the technology.

Azure Transitions Administrative Roles to Azure RBAC Roles

On March 11, Microsoft sent a note to Azure administrators titled “Transition to role-based access control (RBAC) in Azure by 31 August 2024.” The note pointed to an article explaining that Microsoft will retire Azure’s classic deployment model on the same date in favor of Azure Resource Model.

Microsoft 365 Consumes a Surprising Amount of Azure

Like many Microsoft 365 administrators, I labor under the illusion that I don’t use Azure all that much. Then I realized that I use:

• Azure Automation accounts and managed identities to run PowerShell scripts (runbooks).
• Azure Key Vault to hold secure credentials used with PowerShell scripts.
• Connectors to bring unified audit log events into Microsoft Sentinel.
• Microsoft 365 Backup for SharePoint Online, Exchange Online, and OneDrive for Business paid for on a PAYG basis through an Azure subscription. Other SharePoint Premium features use the same PAYG payment model.
• Microsoft Translator for tasks like translating local language values for sensitivity label display names.
• External identities monthly active user (MAU) licensing to pay for guest user access to premium Entra ID features.

And of course, Teams consumes a mass of Azure services and Copilot for Microsoft 365 executes queries against Large Language Modules running in Azure.

In short, despite thinking that my Azure administration needs were minimal, I use Azure for different reasons and the number of those reasons has grown steadily over the last five years. I suspect that this is the same for other tenants. I therefore needed to pay attention to Microsoft’s email telling me to transition classic administrative roles (Co-administrator and Service administrator are the two roles called out) before August 31.

Finding Classic Administrative Roles

I followed the documented steps to access the Azure Portal, select my subscription, and use the IAM option to find classic administrators (Figure 1).

It came as no surprise that my account was listed as the service administrator. When you sign up for an Azure subscription, the account used automatically assumes the service administrator role. The account also is the account administrator. The difference between the two roles is that the service administrator can cancel subscriptions. If needed, the account administrator can grant themselves the service administrator role and become all-powerful. As few accounts as possible should possess such powerful roles.

Azure wants to get away from all-powerful roles and use roles that more accurately reflect the granular nature of the work individuals need to do. This is the principle of RBAC. Within Azure RBAC, the Owner role (limited to the scope of a subscription) is the equivalent of a co-administrator. However, the changeover to RBAC creates an opportunity to assess why user accounts have the co-administrator role and what tasks they perform that require them to possess such a role. It’s possible that one of the Azure RBAC roles is better suited to the work that an individual does. If so, they should be assigned that role.

Switching to Azure RBAC Roles

Being a small and relatively simple tenant, reviewing the current roles and reassigning new roles was straightforward. Instead of the classic service administrator role, my account now has the Owner RBAC role. Instead of the co-administrator role, I assigned the Contributor RBAC role.

I assigned the Azure RBAC roles two weeks ago. Everything has worked perfectly since, and no one has complained that they’re missing the ability to do something. No doubt I am lucky in that respect!

On a serious note, the experience demonstrated that Microsoft 365 has a huge and growing dependency on Azure that tenant administrators should pay attention to. Although it’s not compulsory to be a master of Azure, it does pay to have a workable acquaintance with how Azure works and how to manage the parts that you use.

Support the work of the Office 365 for IT Pros team by subscribing to the Office 365 for IT Pros eBook. Your support pays for the time we need to track, analyze, and document the changing world of Microsoft 365 and Office 365.

Azure Automation for IT Pros

I’ve spent a lot of time working with Azure Automation over the last few years. It’s an extremely useful facility for tenant administrators who want to run PowerShell scripts using a more modern mechanism than offered by Windows Scheduler. This is especially true so in large tenants where processing hundreds or thousands of objects is common, which is why I started to use Run As accounts with Azure Automation.

Converting scripts to run on Azure Automation isn’t too difficult, once you understand the headless nature of the beast and that PowerShell runs on virtual machines spun up for the purpose. The biggest issue often faced when moving scripts from running interactively to being an Azure Automation runbook is how to create output from scripts, but it’s possible to send email, post to Teams channels, and create files in SharePoint document libraries.

Microsoft seems to communicate with developers and administrators (aka IT Pros) in different ways. For instance, the news about the retirement of Azure Automation Run As accounts on September 30, 2023, didn’t appear in any notification in the Microsoft 365 admin center. In fact, apart from the notices posted in Azure Automation documentation (like that shown in Figure 1), I can’t find a formal announcement from Microsoft.

Informing the Technical Community About the Run As Retirement

The possibility exists that I might not be looking hard enough. Normally, I am reasonably proficient with search (Google), but the first hit I find is a 27 September 2022 Microsoft Answers post saying “On 30 September 2023, we’ll retire the Azure Automation Run As account that you use for Runbook authentication.” I can find an earlier “plan for change” note for July 2022 in the What’s new in Azure Automation page. Apart from that, Microsoft seems to have updated the documentation on 18 October 2022 (here’s the FAQ).

I suppose that it’s reasonable to expect people to learn about developments from documentation. In this instance, I think Microsoft dropped the ball and didn’t do a great job of telling people what’s going to happen when Run As accounts retire.

Managed Identities Are a Better Solution

The logic for retiring Run As accounts is undeniable. A better and more secure solution (managed identities) exists. Run As accounts authenticate using a self-signed certificate that needs to be renewed yearly. Microsoft has removed the ability to renew these certificates from the Azure portal, meaning that Run As accounts are counting down to a time when they won’t be able to authenticate. Microsoft has a script to renew certificates for Run As accounts and the script will run after September 30, 2023. However, Run As accounts will then be unsupported, which isn’t a great situation for production components.

The nice thing about managed identities from an Office 365 perspective is that the important PowerShell modules used for automation support managed identities. Some do so very smoothly (like the latest Exchange Online management module, where even the latest RBAC for applications feature supports managed identities) and some do it with a little extra work. For example, V1.0 of the Microsoft Graph PowerShell SDK needs to get an access token from the Azure Automation account that owns a managed identity while V2.0 will be able to sign in using a managed identity. Here’s an example of a simple runbook that:

• Connects to the Azure Automation account using a managed identity.
• Gets an access token from Azure AD.
• Uses the access token to connect to the Graph with Connect-MgGraph.
• Retrieves the service domain (like office365itpros.onmicrosoft.com) using the Get-MgOrganization cmdlet.
• Uses the service domain and a managed identity to connect to Exchange Online.
• Lists details of user mailboxes.

# Connect to Microsoft Graph with Azure Automation
Connect-AzAccount -Identity
$AccessToken = Get-AzAccessToken -ResourceUrl "https://graph.microsoft.com"
Connect-MgGraph -AccessToken $AccessToken.Token
# Get Tenant service domain using Get-MgOrganization
$TenantName = (Get-MgOrganization).VerifiedDomains | Where-Object {$_.IsInitial -eq $True} | Select-Object -ExpandProperty Name
# Connect to Exchange Online
Connect-ExchangeOnline -ManagedIdentity -Organization $TenantName 
Get-ExoMailbox -RecipientTypeDetails UserMailbox | Format-Table DisplayName, UserPrincipalName

When V2.0 of the Microsoft Graph PowerShell SDK is available, you’ll be able to replace the first three lines of code with a simple Connect-MgGraph -Identity.

Another example of using a managed identity with Exchange Online is to monitor events gathered in the audit log to detect and report events that might indicate potential tenant compromise. Running the script on an Azure Automation schedule makes sure that audit events are checked without human intervention.

Time to Move Forward

Apart from the poor communication, I don’t have any problem with Microsoft’s decision to retire Run As accounts. They worked as a mechanism to connect resources to Azure Automation. We’re just moving on to adopt a new approach. Microsoft documents the migration steps to move from a Run As account to use managed identities. It’s a manual process, but not onerous.

Assign the Teams Administrator Permission to Apps and Automation Accounts

Last week, I explained how to assign the Exchange.ManageAsApp permission to an Azure automation account. The permission allows an automation runbook to sign into Exchange Online using a system-assigned managed identity and run cmdlets in the Exchange Online management module. Essentially, the permission gives the automation account the right to act like an Exchange administrator who signs in interactively to run Exchange Online cmdlets. The upshot is that any script written for the Exchange Online module becomes a candidate to be run by Azure automation.

The same situation applies to the Teams PowerShell module. When someone runs the Connect-MicrosoftTeams cmdlet to connect to the Teams management endpoint in an interactive session, they can use all the rights and permissions held by their Entra ID account. If those rights and permissions include a role like Global administrator or Teams administrator, they can perform management operations for Teams at the tenant level.

Giving an App the Right to Act as an Administrator

To use the cmdlets in the Teams PowerShell module in an Azure automation runbook, the automation account must hold the permission to act as an administrator. This requirement applies to all methods of authentication used when apps use the module in background jobs, including when a runbook uses a managed identity.

A runbook connects to Teams with a managed identity like this:

Connect-MicrosoftTeams -Identity

When authenticating the connection, Entra ID evaluates the roles and permissions held by the automation account that owns the runbook. It’s equivalent to a user when they connect to Teams in an interactive PowerShell session. If the account holds a Teams administrator role, it can run cmdlets to update the Teams tenant configuration, alter policies, and update accounts.

Assigning the Teams Administrator Permission (Role)

In the previous article, we assigned the role to manage Exchange Online (which holds the Exchange.ManageAsApp permission) to the automation account. The same principle applies to Teams. Instead of assigning the role to allow the app to manage Exchange Online, we assign the role to allow it to manage Teams. As explained in the other article, the Entra admin center doesn’t support role assignment to automation accounts, so this must be done using PowerShell.

This code:

• Connects to the Microsoft Graph.
• Populates a variable with details of the service principal for the Skype and Teams Tenant Admin API app. This is an enterprise app installed into tenants to allow components like the Teams admin center to manage Teams. Its app id is always 48ac35b8-9aa8-4d74-927d-1f4a14a0b239.
• Find the Application_access role in the set held by the Skype and Teams Tenant Admin API app.
• Populate a variable with details of the automation account to use. In this example, the account is called TeamsAutomationAccount.
• Populate the parameters to make the role assignment.
• Run the New-MgServicePrincipalAppRoleAssignment cmdlet to assign the Application_access role to the service principal of the automation account.

Here’s the code:

Connect-MgGraph -Scopes AppRoleAssignment.ReadWrite.All
# Fetch details of the Teams management app
$TeamsApp = Get-MgServicePrincipal -Filter "AppId eq '48ac35b8-9aa8-4d74-927d-1f4a14a0b239'" 
$AppPermission = $TeamsApp.AppRoles | Where-Object {$_.DisplayName -eq "Application_access"} # Create the payload for the assignment
$ManagedIdentityApp = Get-MgServicePrincipal -Filter "displayName eq 'TeamsAutomationAccount'"
$AppRoleAssignment = @{
"PrincipalId" = $ManagedIdentityApp.Id
"ResourceId" = $TeamsApp.Id
"AppRoleId" = $AppPermission.Id }
# Assign the role to the service principal for the managed identity.
New-MgServicePrincipalAppRoleAssignment -ServicePrincipalId $ManagedIdentityApp.Id -BodyParameter $AppRoleAssignment

After the code runs, check the permissions for the automation account and you should find that the Teams administrative role is present (Figure 1).

Remember that the access granted only allows the automation account to run administrative operations. Other operations might need consent for a Teams Graph permission.

Working with Teams Policies

The current version (4.7) of the Teams PowerShell module doesn’t support using a managed identity to run the older cmdlets from the Skype for Business Connector (like Get-CsTeamsMessagingPolicy). This is because Microsoft needs to update these cmdlets to support managed identities.

If you need to run the older cmdlets in an automation runbook, you can sign into Teams with an account that holds the Teams administrator role. For instance, you could store the credentials for an account in Azure Key Vault and use those credentials with Connect-MicrosoftTeams.

Azure Automation and Teams

Using an automation account with scheduled runbooks is an excellent way to make sure that administrative jobs get run. Being able to use common Microsoft 365 PowerShell modules like Teams expands the scope of possible work. A managed identity relieves the need to maintain credentials and keeps things more secure. All in all, it’s a nice combination.

Learn more about how the Office 365 applications really work on an ongoing basis by subscribing to the Office 365 for IT Pros eBook. Our monthly updates keep subscribers informed about what’s important across the Office 365 ecosystem.

Debugging, Cost, and Tracking for Azure Automation PowerShell

As you might have noticed, I’ve spent some time recently investigating how to exploit Azure Automation with Microsoft 365 PowerShell modules, exploring topics such as using Microsoft Graph PowerShell SDK with Run As accounts to Managed Identities with Exchange Online.

I cheerfully admit that I am still an Azure Automation PowerShell novice, but I’ve enjoyed probing the ins and outs of topics like Managed identities and understanding how things fit together to run jobs on the Azure infrastructure. Working with Azure Automation is certainly different to putting together PowerShell scripts for interactive use, but it’s a technique that I believe Microsoft 365 tenant administrators should master because it’s a great (and secure) way to run resource-intensive scripts on a scheduled basis. Or indeed, scripts that need to run periodically to adjust settings, like the one to update new teams to block team members from creating regular channels. In any case, here are three lessons I’ve learned over the last year or so.

Debugging Azure Automation PowerShell is Different

Debugging code in Azure runbooks is one of the more interesting challenges. When a runbook (PowerShell script) executes, it essentially runs on a headless server. Your code either runs or it doesn’t, and there’s no way to step through commands in a script as you can interactively. If code doesn’t run as expected, it becomes a matter of breaking code up into segments to identify where the problem occurs and inserting old-time output statements to gain clues about what might be going wrong.

Microsoft documentation recommends using the output stream to log progress and information about processing. That’s just a sophisticated way of saying that you insert Write-Output commands in the code to output details of variables. I did this with PRINT statements when I was a COBOL programmer in the 1980s and I’m still doing it today. Technology changes, but some debugging techniques remain in situ. It works both for the output of scheduled jobs and when you execute a runbook in the test pane.

Figure 1 shows the output of a run of the Azure Automation version of the script used to generate the Microsoft 365 Groups Expiration Report. The original article explaining the principle behind the script is available here.

Cost of Azure Automation PowerShell is Minimal

You need a paid Azure subscription to use Azure Automation. The subscription pays for costs such as the small amount of storage used to store runbooks and the compute costs for running the code. Originally, I was concerned that I would rack up considerable costs, especially when running scheduled jobs. However, the actual costs are very modest and most months I end up with a total of a couple of dollars.

The bottom line is that running PowerShell scripts with Azure Automation is unlikely to break the bank unless you execute some highly resource-intensive scripts on a very frequent basis. But in saying that, it’s always a good idea to keep an eye on costs until you’re happy that you understand the cost profile. Suffice to say that I am no longer concerned about expenditure on Azure Automation, at least not for the work that I do (famous last words…).

Tracking Access for Azure Automation PowerShell

Tenant administrators tend to like to know what’s happening. To gain an insight into the usage of Azure Automation accounts, we can analyze the Azure AD sign-in logs for the service principals of these accounts. For example, this code uses the Microsoft Graph PowerShell SDK to connect and fetch audit records for service principal events. It then groups the records to show the count of events for each service principal:

Connect-MgGraph -Scopes "AuditLog.Read.All","Directory.Read.All"
Select-MgProfile Beta
[array]$AuditRecords = Get-MgAuditLogSignIn -Filter "(signInEventTypes/any(t:t eq 'servicePrincipal'))" -Top 2000 -Sort "createdDateTime DESC"
$AuditRecords | Group-Object ServicePrincipalName | Sort-Object count -Descending | Format-Table Name, Count

Name                                                              Count
----                                                              -----
ExoAutomationAccount_Y6LgjDYIfPnxmFzrqdbaClsnTD/gN4BNnVMywiju5hk=   119
Graph Microsoft 365 Groups Membership Report                         84
Clean up Exo Mailboxes                                                6
PS-Graph                                                              6
ServiceCommunications                                                 5
PopulateOrgContacts                                                   2

Each access to a resource (like signing into the Graph) generates an audit record.

$AuditRecords | ? {$_.ServicePrincipalName -eq "Clean up Exo Mailboxes"} | Format-Table ServicePrincipalName, CreatedDateTime, ResourceDisplayName, ipaddress

ServicePrincipalName   CreatedDateTime     ResourceDisplayName IPAddress
--------------------   ---------------     ------------------- ---------
Clean up Exo Mailboxes 12/08/2022 05:33:12 Microsoft Graph     51.171.212.123
Clean up Exo Mailboxes 07/08/2022 15:32:34 Microsoft Graph     51.171.212.123
Clean up Exo Mailboxes 03/08/2022 16:24:01 Microsoft Graph     78.17.104.68
Clean up Exo Mailboxes 02/08/2022 23:23:50 Microsoft Graph     78.17.104.68
Clean up Exo Mailboxes 02/08/2022 17:08:26 Microsoft Graph     51.171.212.123
Clean up Exo Mailboxes 01/08/2022 20:41:35 Microsoft Graph     51.171.212.123

In passing, the large number of sign-ins for the service principal for the app used by the Graph-based Microsoft 365 Groups Membership Report arise from a mistake I made when publishing the report script in GitHub. Unhappily, I included the tenant identifier and app identifier for my tenant (now fixed), and people continue to try to run the script using those values. In the last week, I’ve seen sign in failures from many locations, as I discovered by looking at sign in logs with a failure status.

ForEach ($R in $AuditRecords) {
 $Status = $R.Status.ErrorCode
 $City = $R.Location.City
 If ($Status -ne 0) {
 Write-Host ("{0} Status {1} City {2}" -f $R.CreatedDateTime, $Status, $City, $R.IPAddress) }
}
26/08/2022 20:20:49 Status 7000215 City Sanger
26/08/2022 18:16:44 Status 7000215 City Edmonton
25/08/2022 19:34:10 Status 7000215 City Montreal
25/08/2022 19:33:47 Status 7000215 City New Berlin
25/08/2022 19:24:09 Status 7000215 City Montreal
25/08/2022 11:16:23 Status 7000215 City Leigh
25/08/2022 11:11:15 Status 7000215 City Amsterdam
25/08/2022 10:42:51 Status 7000215 City London
25/08/2022 07:54:12 Status 7000215 City Frederiksvaerk
24/08/2022 19:44:03 Status 7000215 City Montreal
23/08/2022 20:57:05 Status 7000215 City Wellington
23/08/2022 13:43:48 Status 7000215 City Kobenhavn

The script contains a note to tell people to change the values to match a registered app in their tenant, but it seems to be ignored. The moral of the story is to always be sure to obscure authorization details in scripts available publicly.

Azure AD sign-in records for managed identities are fetched with:

[array]$AuditRecords = Get-MgAuditLogSignIn -Filter "(signInEventTypes/any(t:t eq 'servicePrincipal'))" -Top 2000 -Sort "createdDateTime DESC"

These records tell us what resources (services) the scripts run by the managed identity use. For instance, these records are generated by calls made to connect to Teams (as an administrative app), run a Graph request, fetch secrets from Azure Key Vault, and fetch an access token using the Azure management API.

$AuditRecords | Format-Table CreatedDateTime, ResourceDisplayName

CreatedDateTime     ResourceDisplayName
---------------     -------------------
30/08/2022 15:35:02 Skype and Teams Tenant Admin API
30/08/2022 15:35:01 Microsoft Graph
30/08/2022 15:35:00 Azure Key Vault
30/08/2022 15:35:00 Azure Key Vault
30/08/2022 15:35:00 Azure Key Vault
30/08/2022 15:34:49 Windows Azure Service Management API

More to Discover About Azure Automation PowerShell

A great deal of information is available for administrators to fetch and analyze to help understand the usage of Azure Automation in their tenant. I’m not done with my investigations yet. I expect to find other interesting aspects of Azure Automation to investigate. And when I do, I shall share my findings.

Insight like this doesn’t come easily. You’ve got to know the technology and understand how to look behind the scenes. Benefit from the knowledge and experience of the Office 365 for IT Pros team by subscribing to the best eBook covering Office 365 and the wider Microsoft 365 ecosystem.

Storing Script Secrets Safely in Azure Key Vault

In my article about posting messages to Teams channels using Azure Automation runbooks, I discuss how to use the incoming webhook connector and the Submit-PnpTeamsChannelMessage cmdlet from the PnP PowerShell module. The incoming webhook connector posts a snippet about a report while the PnP cmdlet can post a much bigger body, in this case, a HTML-formatted report created by an Azure Automation runbook.

In a related article, I explained that I had used account credentials stored in the Azure Automation account to authenticate and be able to store the report in a SharePoint Online document library. The runbook used the same credentials to post the message to the Teams channel. I explained that I used this approach because I couldn’t find an easy-to-use method (in PowerShell) to impersonate the identity of a team member to allow me to post information to the site and channel belonging to the team.

Setting up an Azure Key Vault

Several people suggested that I should store the account credentials somewhere else, which brought me to Azure Key Vault, defined as “a cloud service for securely storing and accessing secrets.” Secrets can be anything you want to store securely, like account names, passwords, or website URLs. The point is that once a secret is in a vault, it can only be accessed by an account or app allowed access to that vault.

Before you can create your own vault, you need an Azure subscription. Key Vault. The cost is minimal to store and retrieve a small number of secrets, so that shouldn’t be an issue, especially for organizations with existing Azure subscriptions.

To start, go to the Key Vaults section of the Azure portal and select the option to create a new vault. As you can see in Figure 1, this is where you need to specify the Azure subscription to use along with the resource group (I used an existing resource group, but you can choose to create a new one), and the hosting Azure region (usually the one closest to your location). It takes Azure a few minutes to create the new vault and then it’s ready to store secrets.

Storing and Managing Secrets

The runbook uses several variables.

• Account credentials to log into SharePoint Online with PnP.
• The site URL used to connect to PnP.
• The group identifier of the team hosting the channel where the script posts messages.
• The channel identifier of the target channel.
• The inbound webhook identifier to post messages.

Azure Key Vault can store all these items as secrets. In Figure 2, I’m storing the SharePoint site URL.

There’s nothing complicated about the secret storage process, and it won’t take long to populate the full set of secrets required by the runbook (Figure 3).

Interestingly, Azure detected that I stored a webhook in a secret and posted a message using the webhook, which turned up in Teams (Figure 4). Although I can’t confirm this, it looks like Microsoft uses TruffleHog to scan items stored in Azure Key Vault.

After storing our secrets, the next thing to do is to allow access to the accounts and apps that use the secrets. Azure Key Vault uses access policies for this purpose. I used vault access policies rather than those based on Azure Role-based access control.

Each vault access policy determines what an account or app can do with the secrets. In my case, apps only need to read the secrets (a GET operation). In Figure 5, I’m creating an access policy based on the Secret management template. The access policy allows seven secret permissions because I haven’t trimmed the set down to just Get. The important thing in Figure 5 is that I’ve chosen to allow access to the service principal used for the Run As account. Essentially, this means that when the Run As account authenticates, it can gain access to the secrets in the vault.

Accessing Secrets from the Runbook Script

As noted above, the original runbook script uses a credential stored in the Azure Automation account. The updated script (available from GitHub) replaces that credential and the other variables with lookups against Azure Key Vault. The code used to connect to Azure and fetch the secrets is as follows (the code to connect to Azure is dependent on some other variables established earlier in the script).

$AzConnection = Connect-AzAccount -Tenant $Connection.TenantId -ApplicationId `
   $Connection.ApplicationId -CertificateThumbPrint $Connection.CertificateThumbprint `
  -ServicePrincipal
# Get username and password from Key Vault
$UserName = Get-AzKeyVaultSecret -VaultName "Office365ITPros" -Name "ExoAccountName" -AsPlainText
$UserPassword = Get-AzKeyVaultSecret -VaultName "Office365ITPros" -name "ExoAccountPassword" -AsPlainText
# Create credentials object from the username and password
[securestring]$SecurePassword = ConvertTo-SecureString $UserPassword -AsPlainText -Force
[pscredential]$UserCredentials = New-Object System.Management.Automation.PSCredential ($UserName, $SecurePassword)
# Get Site URL to use with PnP connection
$SiteURL = Get-AzKeyVaultSecret -VaultName "Office365ITPros" -name "SPOSiteURL" -AsPlainText
# Target channel identifier for incoming webhook connector
$TargetChannel = Get-AzKeyVaultSecret -VaultName "Office365ITPros" -name "IncomingWebhookId" -AsPlainText
# Target team and channel in that team to which we post a message containing the report
$TargetTeamId = Get-AzKeyVaultSecret -VaultName "Office365ITPros" -name "TargetTeamId" -AsPlainText
$TargetTeamChannel = Get-AzKeyVaultSecret -VaultName "Office365ITPros" -name "TargetChannelID" -AsPlainText

Getting Secrets with Normal PowerShell

The code shown above works when executed in a runbook. With a tweak to the connection to Azure, the same code can fetch secrets from Azure Key Vault when run interactively or in “normal” scripts. As with the runbook, the prerequisite is that a vault access policy must allow the account used to access the vault. With the policy in place, you can run the Connect-AzAccount cmdlet to sign in and execute the same code to retrieve the secrets (Figure 6).

Alternatively, you can use the SecretManagement PowerShell module and map your key vault. For example, these commands configure the Key Vault for access through cmdlets from the SecretManagement module and then retrieve one of the secrets stored in the vault:

$AzureSubscription = "35429342-a1a5-4427-9e2d-551840f2ad2a"
Register-SecretVault -Module Az.KeyVault -Name MyKV -VaultParameters @{ AZKVaultName = "Office365ITPros"; SubscriptionId = $AzureSubscription}
$AccountName = Get-Secret -Name ExoAccountName -Vault MyKv -AsPlainText

The decision to use one method or another is entirely up to you. Both do the job.

The Obvious Issue

We’re still using a username and password as account credentials. Although it’s true that we have the secrets safely stashed in Azure Key Vault, it’s still single-factor authentication. You can mitigate this somewhat by using an account with an obscure name and a very complex password, but it’s still a potential weakness. Time to do some more research.

A Safe Place for Secrets

My use of Azure Key Vault is simple but effective. I was pleased with how easy it was to store secrets and access secrets. Apart from a secure place to keep sensitive data, it also means that it’s easy to update variables used in a runbook script. I could make the change directly in the runbook script and the code would be a little faster to run, but I do like the idea of being able to secure important information. Give Azure Key Vault a try and see if it works for you!

Learn how to exploit the data available to Microsoft 365 tenant administrators through the Office 365 for IT Pros eBook. We love figuring out how things work.

Microsoft 365 Tenants Need to Link their Entra ID to Azure Subscription

Tenant administrators are all too aware of the growth of guest user accounts in tenant directories over recent years. The success of Teams and the use of guest accounts in sharing SharePoint Online and OneDrive for Business documents are the biggest factors in driving the growth in guest accounts. As we’ll discuss, some premium features of Microsoft 365 Groups require consideration of Entra ID guest user licensing.

Apart from cluttering up the directory, guest accounts don’t do any harm. You can try to identify and remove obsolete accounts using a variety of methods such as checking the Entra ID sign-in logs to discover the last sign in to the account or using the Office 365 audit log and message tracking logs to figure out if guest accounts are active.

However, one thing you should keep an eye on is the requirement to license guest accounts if you use premium Entra ID features like conditional access policies or dynamic Microsoft 365 groups. In the past, the rule was that guest accounts needed premium licenses at a 1:5 ratio to Entra ID premium licenses. In other words, each premium license covers five guest accounts. Guest accounts don’t need licenses for “normal” activity such as accessing a team or opening a shared document. Entra ID access reviews can help control the need for licenses by forcing group owners to validate continued membership of guests in their groups.

External Identities Licensing Change

In September 2020, Microsoft announced a change in licensing for external identities (Azure B2B and B2C collaboration). Instead of requiring customers to buy premium licenses to cover guest accounts, the new monthly active users (MAU) billing model allows up to 50,000 free MAU for premium activities monthly. Licenses are still needed for tenant accounts that use premium features.

The definition on Microsoft’s billing model for Entra ID external identities page explains that MAU is “the count of unique users with authentication activity within a calendar month.” In other words, the MAU threshold covers all authentication activity by 50,000 external identities (like guest accounts) in a month. Any individual identity within that set can authenticate as many times as they like. If a tenant exceeds the 50,000 MAU threshold, Microsoft bills for authentications by subsequent external identities. Pricing varies according to market and whether an authenticated external identity uses Entra ID P1 or P2 features (see MAU pricing). As an example, in the U.S., an engra ID P1 MAU costs $0.00325.

To date, Microsoft hasn’t done much to enforce the changeover to MAU pricing, and it’s very possible that Microsoft’s change in licensing strategy passed tenant administrators by without registering. It certainly made no impact on me. However, the signs are that some new features might require tenants to use MAU billing, which requires customers to link their Entra ID tenant to an Azure subscription. If you’ve already done this, you don’t need to do anything else as Microsoft bills you based on the MAU model. If you haven’t, you’ll need to link your tenant to an existing or new subscription.

Switching Entra ID Guest User Licensing to MAU Billing

On the surface, the process to switch to MAU billing seems straightforward:

• Create a new Azure subscription or identify an existing subscription to use for MAU billing.
• Go to the External Directories blade in the Entra admin center and select the Linked subscriptions option. Figure 1 shows the result of successfully linking Entra ID to a Azure subscription.
• Select your directory (most tenants have just one).
• Click Link subscription to select the Azure subscription and resource group (within the subscription) to use for MAU billing. Click Apply to link the directory to the subscription.

Registering the Entra ID Resource Provider

In my case, linking proceeded smoothly until Azure rejected my chosen subscription with the error:

The subscription is not registered to use namespace ‘Microsoft.AzureActiveDirectory’. See https://aka.ms/rps-not-found for how to register subscriptions.

The referenced page contains a lot of information about fixing various problems but nothing I could see relating to Entra ID. Some research (aka web searches) revealed that Microsoft.AzureActiveDirectory is the name of the resource provider for Entra ID. As you might imagine, not every resource provider is registered for every Azure subscription, so the solution is to register Entra ID for the subscription.

You can do this in two ways. First, go to the Subscriptions section of the Azure portal and select the subscription you want to use. Now select resource providers and look for Microsoft.AzureActiveDirectory in the set of providers. Select and register the provider. Figure 2 shows that the provider is registered, which is what you want to see.

Those wanting to live on the edge can register the provider using the Azure Cloud Shell. Start a session by clicking the Cloud Shell icon in the menu bar (it’s the icon which looks vaguely like PowerShell). This opens a small pane in the Azure portal into which you can type commands (you have a choice of Bash-like or PowerShell-like environments).

Accessing Cloud Shell from the Azure portal logs into your account automatically. All you need to do is run two commands to select the subscription you want to update and then register the Microsoft.AzureActiveDirectory provider with the subscription:

Az account set –-subscription "Visual Studio Enterprise Subscription"
Az provider register –-namespace Microsoft.AzureActiveDirectory

If you access the Cloud Shell directly (https://shell.azure.com/), you’ll need to sign in first with:

Az login

In either case, after registering the provider, you can link the subscription to Entra ID and use the MAU billing model.

It seems strange that Microsoft hasn’t optimized the Entra admin center to make sure that a selected subscription has access to Entra ID and if not, offer the administrator to register Entra ID with the subscription. There should be no need to force administrators to solve the problem when software can do it automatically.

Extra SMS Charges

Although Microsoft allows for 50,000 free MAU monthly, the MAU pricing page says:

A flat fee of $0.03 is billed for each SMS/Phone-based multi-factor authentication attempt.

Note the wording. The charge applies whether the attempt to send an SMS code is successful or not and covers the telephony charge involved in sending the SMS. The charge does not apply when external identities use the Microsoft Authenticator app for MFA verification, which is another good reason to encourage guest accounts to use the app.

Entra ID Guest User Licensing Works for Microsoft and Tenants

I’m sure Microsoft likes the new MAU pricing model for external identities because it gives them more control and visibility over the volume of guest account activity with premium Entra ID features. The old 1:5 licensing model was unenforceable and probably ignored in many tenants. On the upside, because MAU pricing is linked to Azure subscriptions, tenants gain more insight into the activity level for guest accounts too. I’ll be keeping an eye on costs as time goes by.

So much change, all the time. It’s a challenge to stay abreast of all the updates Microsoft makes across Office 365. Subscribe to the Office 365 for IT Pros eBook to receive monthly insights into what’s happening.

Office 365 Dependent on Azure and Affected by Its Downtime

Data compiled by Gartner and Krystallize Technologies puts Azure behind Amazon Web Services (AWS) and Google in terms of uptime (availability) in North America in 2018. Although the availability of Azure is very high (99.9792%), Office 365 tenants have been affected by several recent outages that underscore the dependency Office 365 has on Azure.

Multiple Recent Azure Issues

The worst thing a cloud service can have is multiple incidents highlighted and discussed in the technology press. The January 24-25 failure of Azure Active Directory, the MFA outage in November, and the unfortunate lightening strike incident in September were followed by a DNS issue last week. Throw in the Teams outage in February caused by a problem with Azure KeyVault (the problem was with a Teams component), and there’s enough problems associated with Azure services to tarnish Azure’s image.

But let’s go back to that uptime number again. 99.9762% is extraordinarily high for a massive cloud infrastructure. No breakdown is given about which Azure components contributed to downtime, so it’s hard to know if there’s any fundamental problem with Azure or it’s just had a run of bad luck recently. I think it’s more like the latter.

Some of the problems have been down to bad luck (lightening). Others are due to a lack of resilience in the infrastructure (having a single MFA service for multiple regions), operation issues (DNS), or programming (Teams use of Azure KeyVault). In other words, there’s no single thread of weakness that causes Azure to fail. Each issue is being addressed appropriately by making services more robust and resilient, distribution across more datacenters, improvements in operational processes, or code updates.

Office 365 and Azure

When people think about Office 365 and Azure, much of the attention focuses on Azure Active Directory. Each Office 365 tenant comes with a basic Azure Active Directory subscription (to allow Office 365 to store information about accounts and settings). But the relationship between Office 365 and Azure is much wider and deeper. Some applications, like Teams, consume dozens of Azure micro-services to avoid the need to recreate wheels. Instead of building monolithic applications like Exchange and SharePoint (based largely because of their on-premises heritage), new applications seek to reuse the Office 365 software parts bin. The result is a mass of dependencies.

The availability of Azure should concern Office 365 tenants. Even with the recent glitches, Azure is still doing pretty well. And as GeekWire says “even if Microsoft lags AWS and Google in reliability scoring, unless your company is blessed with world-class operations talent, Microsoft is likely still better at operating data centers than most companies managing their own servers.”