Why Some People Can’t Use Shared Files with Copilot for Microsoft 365

After reading the article about the new sensitivity label advanced setting to block access for Microsoft content services to confidential Office documents, a reader asked why some users can use some documents shared using company-wide links with Copilot for Microsoft 365 while others cannot. The situation seemed a little strange because it happened for documents shared with everyone in the organization. The problem couldn’t be due to a sensitivity label because the capability only just rolled out and is limited to the Office applications.

The answer is in Microsoft’s documentation for secure file sharing, which says: “Creating a People in your organization link will not make the associated file or folder appear in search results, be accessible via Copilot, or grant access to everyone within the organization. Simply creating this link does not provide organizational-wide access to the content. For individuals to access the file or folder, they must possess the link and it needs to be activated through redemption.”

In other words, sharing a file with everyone in your organization is only the first step in the process of making information available to Copilot for Microsoft 365. A company sharing link that arrives in your inbox or is shared through a Teams chat is dormant until you redeem it by using the link. At that time, SharePoint Online checks that your account belongs to the organization to conform your access to the file. If confirmed, the file joins the set of “shared with you” information, which makes it available to Copilot for Microsoft 365.

Testing Company-wide Sharing Links with Copilot

A simple test proves the point. Create a file that contains some information that’s unlikely to exist elsewhere within the company. In my case, I created a Word document about a fictional digital SLR camera called the Bunsen BX7. Now share the file with a company-wide link (Figure 1).

After signing into another account, open Copilot for Microsoft 365 chat and attempt to find some information about the topic in the file. Copilot should return nothing because a Bing search of the internet and a Microsoft search of company resources available to the account turn up no mention of the topic. But if you now go and use the link to open the file, Copilot can find the information and use it in its responses.

Figure 2 shows a Copilot for Microsoft 365 chat session. The first prompt about the Bunsen BX7 turns up nothing and Copilot responds with some generic text about digital cameras. The second prompt is after redemption of the company-wide sharing link. Copilot is able to find the document and use the information in its response. You can see that the shared document is listed as a source for the response.

The Desirability of Company-wide Links

The mystery of why some people can use shared documents with Copilot for Microsoft 365 is solved, but thoughts now turn to whether organizations should restrict the use of company-wide links for sensitive documents. The value of these links is that they allow anyone in the organization to access content. The downside is that it’s too easy to create and use company-wide links, which then creates the temptation for people to use these links to share confidential files wider than the organization wants the information to be known.

To guide users away from company-wide links to create sharing links for specific people instead, you can modify the SharePoint tenant configuration to make direct links the default option. Even better you can update individual site settings to disable company-wide links (anyone links are also disabled). For example, the first command sets direct links as the tenant default; the second disables company-wide links for a specific site.

Set-SPOTenant -DefaultSharingLinkType Direct

$Site = "https://office365itpros.sharepoint.com/sites/BlogsAndProjects"
Set-SPOSite -Identity $Site -DisableCompanyWideSharingLinks Disabled

If your organization uses sensitivity labels, you could also consider applying a label that restricts access to a small group of users. That way, even if someone sends a document outside the organization as an email attachment, external recipients won’t be able to open it.

The Challenge of Managing Information in an AI World

The advent of AI assistants creates new information governance challenges for Microsoft 365 tenants. Slowly but surely mechanisms are being developed to help organizations cope and manage the potential for information leakage and misuse. Some Microsoft solutions are no more than sticking plasters to allow customers to progress their Copilot deployments, but overall, the situation seems to be improving. Let’s hope that the trend continues and the current AI hype lives up to its promise.

Helping People Build Better KQL Queries, One Search at a Time

In May 2021, Microsoft introduced a new user interface for content searches (including core eDiscovery) in the Microsoft 365 compliance center. At the time, I was critical about the change because the new interface is slower and (still) buggier than the old. Nevertheless, change is an ongoing influence in cloud services, and you’ve got to imagine that the developers introduce nee features or upgrades to existing capabilities for good reason. At least, that’s the theory.

Which brings me to a change which happened some time ago that I completely overlooked. Message center notification MC288050 (October 4) announced the arrival of a Keyword Query Language (KQL) editor to help compliance administrators build queries for content searches, including the searches used in core eDiscovery and advanced eDiscovery. Microsoft 365 roadmap item 88582 reports that the KQL editor is still in preview as of November 2021.

Condition Card Builder

Put simply, the value of the KQL editor is that it stops people making mistakes when they compose queries for content searches. Up to now, building a query involved handcrafting the set of keywords and conditions necessary to find information. The GUI offered some help, such as date pickers to set a date range for a query, but not much. The roll-your-own query facility is now referred to as the condition card builder and it’s still available for those who like composing KQL queries. Figure 1 shows a query using two keywords (search terms) and a condition.

The KQL Editor

You can switch between the condition card builder and KQL editor as often as you like. In this instance, switching to the KQL editor reveals the query created using the entered keywords and conditions. In effect, we have a starting point to refine the query (Figure 2).

Let’s say that we’re only interested in Word documents which contain the keywords. Type AND to add a new clause and then at least the first two characters of a property to search against. The KQL editor then proposes the matching conditions and operators (like :, for equals) which can be used. Microsoft refers to these hints as “property restrictions.” The hint used in Figure 3 is “file” because we want to search for a specific file type. The KQL editor suggests autocompletion of matching properties, and we can select Filetype: and then docx to complete the condition.

If the KQL editor notices a syntax error or some missing element, it flags the issue and tells you what the problem is (Figure 4). The experience is somewhat like that of editing code using an ISE and makes it easy to ensure that a query is valid and will run when submitted for processing.

Another interesting facility is the autocompletion for user principal names when searching based on email recipients (Figure 5). This works for the From, To, Recipients, and Participants properties. The names come from the tenant’s Azure AD and include guest accounts.

You can also copy and paste queries from other searches to use as the basis for a new search. This isn’t a huge advantage for simple searches, but it saves time when you deal with complex, multi-condition search queries. After pasting a query into the KQL editor, the editor checks the query to make sure that it doesn’t contain errors.

A Step Forward

The KQL editor is a nice addition to content searches. Even with the error checking done by the KQL editor, it’s still possible to create queries that just don’t work. For instance, I was able to compose this query in the KQL editor, which accepted the syntax without any problem:

“Azure B2B Connect*” AND “Teams” AND (filetype:”doc OR filetype:pdf”) AND LastModifiedTime=”this year” AND (from:James.Abrahams@office365itpros.com OR from:Ben.James@Office365itpros.com)

The filetype condition has quotation marks in the wrong place and the query mixes email and document conditions together (the Author property is more appropriate to search for documents).

In any case, you can’t expect to disengage your brain entirely when composing search queries. The KQL editor helps, but humans need to create and check the queries and validate that the result results are those expected.

So much change, all the time. It’s a challenge to stay abreast of all the updates Microsoft makes across Office 365. Subscribe to the Office 365 for IT Pros eBook to receive monthly insights into what happens, why it happens, and what new features and capabilities mean for your tenant.

Even For Tenants Who Don’t Use Records Management

Records management is a Microsoft solution to help enterprises manage items (email, documents, and lists) marked as formal company records. Office 365 E5 or Microsoft 365 compliance licenses are required to implement the Records management solution, but Microsoft uses the Settings section of Records management to host the GUI for controls over how SharePoint Online and OneDrive for Business work for retention processing. Today, three important controls for retention labels are available (Figure 1).

The settings apply to all sites in a tenant (no site-specific controls ae available). These are:

• Deleting content labeled for retention: Until recently, SharePoint Online blocked users from deleting labeled items while OneDrive for Business allowed them to do so. To achieve consistency across the two applications, Microsoft changed SharePoint Online to behave like OneDrive for Business, meaning that users can delete labeled items and SharePoint Online will store the items in the site’s preservation hold library until their retention period expires. Some organizations prefer the previous behavior because they believe that users should not remove labeled items. If this option is set, users see an error if they attempt to remove a labeled item. To proceed, a site administrator or user with permission must remove the label to allow deletion to happen or replace the label with one that does not have a delete action. This control is not linked to Records management.
• Configure record versioning: By default, record versioning is on, meaning that users can unlock items assigned a record label and edit their content. If off, items assigned a record label remain locked and updates are not possible after their creation. In effect, record labels then act like regulatory record labels.
• Allow editing of record properties: Apart from its content, an item has metadata like its title and other attributes. By default, users can edit items assigned a record label to update metadata. If this control is off, users cannot update item metadata after creation.

More Flexible Records Management

The last two options are needed for a new Records management capability where organizations can create retention labels for records which, when applied, leave items unlocked (MC306685, Microsoft 365 roadmap item 88509).

Up to now, when you apply a record label to a document, SharePoint Online or OneDrive for Business immediately lock the file to stop it being edited. The metadata (for instance, a document title) is changeable when a file is locked, but to update its contents, the user must first unlock it. This isn’t difficult (how hard it is to move a slider from locked to unlocked?), but it’s more convenient to keep a document in an unlocked state until its content is final, at which point it can be locked and behave as in the past.

Two options are available for organizations to decide if they want to allow users to edit locked records. Both are on by default, meaning that users can lock/unlock files and update metadata. However, if an organization wants record labels to behave much like regulatory record labels, they can toggle one or both settings to off. In this state, users cannot update the content or metadata of documents assigned retention labels marked as records. It’s not something to do on a whim, but it will make compliance administrators happy because they gain some extra flexibility in records management.

Insight like this doesn’t come easily. You’ve got to know the technology and understand how to look behind the scenes. Benefit from the knowledge and experience of the Office 365 for IT Pros team by subscribing to the best eBook covering Office 365 and the wider Microsoft 365 ecosystem.

Restrict Processing to Specific File Types Using Auto-Label Retention Policies

A reader asks if it’s possible to limit the processing of a Microsoft 365 retention policy to certain file types. In this case, they wish to remove any Word documents, Excel spreadsheets, and PDF files from the SharePoint Online sites used by Teams while leaving other items stored in SharePoint like the OneNote notebook and .mht files used by the Teams wiki.

Before answering the question, it’s important to understand that Teams retention policies only process messages from channel conversations (including private channels) and chat. If you want a retention policy to process the SharePoint content owned by a team, you need the policy to process Microsoft 365 Groups. This is because Teams uses Microsoft 365 Groups for its identity and membership service.

The challenge then is how to amend a retention policy for Microsoft 365 Groups to only process certain file types. However, Microsoft 365 retention policies operate on a “whole container” basis. In other words, they apply the same retention settings to all items found in the target locations. In this case, Microsoft 365 Groups. Although you can refine a policy to process selected groups, the GUI in the Microsoft 365 compliance center doesn’t allow you to modify a policy to perform selective processing by file type.

Assigning Content Queries with PowerShell

PowerShell cmdlets are available to manage Microsoft 365 standard and auto-label retention policies. To access these cmdlets, connect to the Exchange Online management endpoint and then run the Connect-IPPSSession cmdlet to connect to the compliance endpoint. You can then run the Get-RetentionCompliancePolicy cmdlet to view the set of retention policies defined in the tenant and the Get-RetentionComplianceRule to view the rule defined for each policy. Each retention policy divides into the general policy settings and the rule dictating the retention action performed by the policy.

If you look at retention policy rules, you’ll see a parameter called ContentMatchQuery. Checking Microsoft’s documentation for Set-RetentionComplianceRule, we discover that this parameter contains a content query in Keyword Query Language (KQL) to filter the content scanned by the policy. A KQL-based content query is a commonplace object within Microsoft 365 as it’s used for SharePoint search, eDiscovery content searches, and automatic labeling policies.

It is possible to update a standard retention policy to add a content query. However, because no GUI exists for this purpose, you’ve got to add the content query with PowerShell. For example, the content query added to the target retention policy finds any Word document, Excel worksheet, or PDF file:

Set-RetentionComplianceRule -Identity "Retention for Specific File Types" -ContentMatchQuery "filetype:doc* filetype:xls* filetype:pdf"

You can test a content query by inputting it to SharePoint Search. If the search returns the expected files, you know the filter is good and will work with a retention policy.

The Problem with Standard Retention Policies

Or can it? Well, the filter works, and the retention policy will process only the specified files, but Microsoft doesn’t support this scenario. At least, they don’t include this scenario in their testing as they introduce new functionality (like adaptive scopes), which means that something might break inadvertently. That’s not a good situation for a compliance solution.

The ContentMatchQuery parameter exists because a KQL content filter is a way to select items for auto-label retention policies. Examples of the use of KQL queries in auto-label retention policies are to apply retention labels to Teams meeting recordings or to files with specific sensitivity labels.

In the earliest days of Office 365 retention policies, as they were then, you could attach a content query to a standard retention policy, but this has not been a supported scenario since Microsoft removed the GUI to enter KQL queries for retention policies. However, Microsoft left the ability to define a content query for a standard retention policy through PowerShell intact, and that’s what has created the inconsistency which exists today. To address the issue, Microsoft should update the documentation for Set-RetentionComplianceRule to say that the ContentMatchQuery parameter is unsupported with standard retention policies.

Use Auto-Label Policies

The approved solution is an auto-label retention policy. These policies accept KQL content queries like the example cited above. When SharePoint Online applies a retention policy with a content query, it finds matching files and assigns the retention label defined in the policy to the items. The retention period and action defined in the retention label controls what happens to the files. For example, the label might define a retention period of one year after which SharePoint Online should remove the labeled files.

One advantage of using an auto-label retention policy is that users see the retention label applied to matching files and can change the label if necessary. This doesn’t happen with standard retention policies because all processing happens in the background and users won’t necessarily be aware of the deletion of files.

The downside of auto-label retention policies is that any account which comes within the scope of the policy (for instance, all members of the Microsoft 365 groups processed by a policy) require Office 365 E5 or Microsoft 365 compliance licenses. This is not a problem if your organization already has these licenses, but there’s quite a step up from Office 365 E3 (needed for standard retention policies) to automatic processing, even before Microsoft’s March 1 price increase.

The bottom line is that you can live on the edge and use content queries with standard retention policies until something breaks (or Microsoft disables the capabilities) or use auto-label retention policies for selective processing based on file types. Going with a supported approach is always a better choice, even if it might cost some extra license fees.

Insight like this doesn’t come easily. You’ve got to know the technology and understand how to look behind the scenes. Benefit from the knowledge and experience of the Office 365 for IT Pros team by subscribing to the best eBook covering Office 365 and the wider Microsoft 365 ecosystem.

Making Sure Confidential Documents are Retained

By their very nature, sensitivity labels are intended to mark documents and files as containing important information. With this thought in mind, it makes sense to apply retention labels to files based on the sensitivity of the information they contain. Given that they know the content, you can ask users to assign appropriate retention labels to files, but humans are imperfect and often forget, which is where auto-label retention policies come in.

Auto-label retention policies run in the background to check Exchange Online messages, and files in SharePoint Online sites and OneDrive for Business sites. Auto-label retention labels also support Microsoft 365 Groups, meaning that they apply to the messages in group mailboxes and the files in the SharePoint Online team sites belonging to groups (including Teams). The basic principles of auto-label retention policies are:

• Identify the objects to label through a content query. The query could be the presence of a sensitive information type known to Microsoft 365, like a credit card number. Microsoft 365 includes over 250 different sensitive information types, and organizations can create their own types to handle business requirements. Organizations can also create trainable classifiers based on business documents and use classifiers with auto-label policies. Finally, you can use a search constructed with the Keyword Query Language (KQL), which is what we’ll use.
• Define a retention label for the policy to apply when it finds content matching its conditions. You can choose any retention label defined in the organization.

Auto-label retention policies are an advanced compliance feature, meaning that any account which comes within the scope of a policy must have an appropriate license (like Office 365 E5 or Microsoft 365 compliance).

Working Through an Example

In this example, we’ll create an auto-label retention policy to assign a retention label to documents and messages protected by the Highly Confidential sensitivity label. To do this, you:

• Connect to the compliance endpoint with PowerShell by connecting to Exchange Online and then running the Connect-IPPSSession cmdlet.
• Find the unique identifier (GUID) for the selected sensitivity label by running the Get-Label cmdlet. The ImmutableId property contains the GUID.

Get-Label | ? {$_.DisplayName -eq "Highly Confidential"} | Select-Object -ExpandProperty ImmutableId

Guid
----
9ec4cb17-1374-4016-a356-25a7de5e411d

• Use SharePoint search to test the KQL query for the auto-label policy. The search term is in the form InformationProtectionLabelId:9ec4cb17-1374-4016-a356-25a7de5e411d wherethe managed SharePoint property used to hold sensitivity labels (InformationProtectionLabelId) is combined with the GUID identifying the sensitivity label you want to search for. Run the search and open one of the documents returned by the search to check that it has the correct sensitivity label. If no documents are found, it might indicate that the GUID is incorrect or that your account has access to no documents assigned this sensitivity label.
• If the search term finds the correct documents, go to the Information governance section of the Microsoft 365 compliance center to create an auto-label retention policy. The condition of the policy uses the same search term as the content query to find the target documents. The policy action applies a suitable retention label to keep the documents for the desired period. Figure 1 shows the KQL query inserted in the settings of an auto-label retention policy.

• Configure the policy with target locations. Remember to use Microsoft 365 Groups to cover SharePoint sites owned by groups and teams. Publish the policy when everything is complete.
• After ten days or so, check that documents with the sensitivity label have the correct retention label, remembering that if a user assigns a retention label to a document, an auto-label policy won’t replace it.

The ten days mentioned above is an estimate rather than a guarantee. It can take SharePoint Online anything from seven days to two weeks for a new auto-label retention policy to become operational and start to apply retention labels.

Retention and Sensitivity

If you have the necessary licenses, auto-label retention policies are a great way to make sure that important information is kept for as long as required or that other information is removed once no longer required. Another example is to apply retention labels to Teams meeting recordings (a more flexible option than the default Teams-only retention for meeting recordings).

Microsoft’s original labeling plan features labels that had both retention and sensitivity capabilities. That plan fell by the wayside, perhaps because such labels might have been very complex to implement and manage. We now must implement retention labels and sensitivity labels separately. Auto-label retention policies are one way to bring the two together in some small way.

The Office 365 for IT Pros eBook includes chapters with in-depth coverage of both retention labels and sensitivity labels. If you’re planning a deployment which includes these components, you can benefit from our insight.

Second Example of Trend Emerges with Offer to Use Communications Compliance

I don’t like the trend now emerging in Microsoft 365 Data Loss Prevention (DLP) where Microsoft uses DLP policies as a conduit to sell other Microsoft 365 solutions. A case can probably be made to extend a DLP policy to cover Teams, but the October 21 announcement in MC293000 that Microsoft will “surface” recommendations to use Communications Compliance within the DLP workflow (Figure 1) is a step too far.

You might think it is awfully helpful for Microsoft to make suggestions about making better use of DLP. This feeling would be justified if the recommendations improved DLP. However, as I point out when discussing the need to move away from Exchange DLP policies, extending a policy to Teams is not something which can often be done automatically. Likewise, gaining “the ability to apply DLP policy insights to your insider risk practice to better identify user behavior and intent” by configuring a communications compliance policy is not something that should happen as the result of a prompt at the end of updating a DLP policy.

Monitoring Communications

Communications compliance policies monitor interactions between people to detect problems like using offensive or threatening language. The communications covered are email, Teams, Yammer, and Skype for Business conversations (soon to disappear). For Teams and Yammer (only networks configured in Microsoft 365 mode), monitoring happens against the compliance records captured in Exchange Online. Matching occurs using one or more of the trainable classifiers available within a tenant (Figure 2), including those configured by the tenant.

Since the launch of communications compliance in 2019, Microsoft has done a good job of building out the set of available classifiers and expanding language coverage. The image classifiers are language independent. Classifiers won’t catch everything, but they improve over time and the idea is to detect gratuitous and persistent offenders rather than picking up every conceivable issue.

Policy matches result in referrals to human reviewers to check the content and context of the problem messages. The reviewers can decide if a policy violation is present and if so, how best to deal with the offender. All of which is grounded in an organizations HR policies and procedures, and probably heavily influenced at a local level.

There’s lots to like about communications compliance and it’s a good solution for Microsoft 365 to offer. However, this is not a solution that every organization needs or is comfortable with. Communications compliance has a hint of big brother is watching you about it that makes many people uncomfortable. Its implementation requires careful planning to ensure that the organization is prepared and that everyone involved in policy creation and operation from HR to reviewers to managers understand their roles and how to deal with offenses. This is not a project to start on a whim.

Inappropriate Connection

All of which makes me think that it is inappropriate for Microsoft to link DLP with communications compliance. There’s too big a jump between monitoring for inadvertent disclosure of sensitive corporate information outside the organization (the normal DLP scenario) to checking internal communications to detect violations in tone and language. I don’t see the natural connection between policies largely under the control of IT (DLP) and those where HR has huge influence and oversight.

One thing that links both suggestions Microsoft surface within DLP is that they need Office 365 E5 or Microsoft 365 E5 Compliance licenses. Office 365 E3 covers DLP for Exchange and SharePoint, but you need E5 for Teams (a differentiation that’s always seemed strange and inexplicable). Communications compliance is a premium E5 feature. I hope that Microsoft isn’t simply using DLP to push higher-price features to customers. That’s a tactic which might seem reasonable inside Microsoft, but it’s just tacky out in the real world.

PS. Microsoft will run a webinar about moving Exchange DLP policies to Microsoft 365 DLP policies on November 9. Register here.

Keep up to date with developments in compliance and other areas of Microsoft 365 by subscribing to the Office 365 for IT Pros eBook. Our monthly updates make sure that our subscribers understand the most important changes happening across Office 365.

Aiming for Workload-Agnostic Retention

Exchange and SharePoint on-premises servers both included retention processing. Exchange’s Messaging Records Management (MRM 2.0) persists today in the form of mailbox retention policies. Microsoft 365 retention policies, first introduced in 2017, replaced SharePoint’s equivalent (here’s an example of SharePoint retention in use) as part of a strategy to implement workload-agnostic data governance. In other words, deploy Microsoft 365 retention processing that worked for all workloads.

The initial implementation of retention policies (and classification labels, as Microsoft then called retention labels) supported Exchange Online (including public folders), SharePoint Online, OneDrive for Business, and Skype for Business Online.

Compliance Records and Microsoft 365 Retention Processing

Teams was the first Office 365 application to store compliance records in Exchange Online mailboxes. Compliance records are mail items created to capture details of chats and channel conversations, including messages sent by people who don’t have Exchange Online mailboxes like on-premises users, guest accounts, and external (federated) users (the substrate stores these messages in special cloud-only mailboxes). Yammer adopted the same mechanism in 2020, but only for networks configured in Microsoft 365 native mode. Planner announced that they would follow likewise but have not yet delivered this capability.

If you check the number of compliance items in your mailbox, you might find that you have more than you imagine. Most of my chatting happens in other tenants, but even so, I have quite a few compliance records for Teams chats (and fewer for Yammer conversations). Exchange Online doesn’t charge the storage consumed by these messages against user mailbox quotas.

$Folders = Get-ExoMailboxFolderStatistics -Identity Tony.Redmond@office365itpros.com -IncludeOldestAndNewestItems -FolderScope NonIPM
$Folders | Where-Object {$_.Name -eq "TeamsMessagesData" -or $_.Name -eq "Yammer"}| Format-Table Name, ItemsInFolder, FolderSize, NewestItemReceivedDate -AutoSize

Name              ItemsInFolder FolderSize                     NewestItemReceivedDate
----              ------------- ----------                     ----------------------
Yammer                      330 4.144 MB (4,344,857 bytes)     23/08/2021 08:23:50
TeamsMessagesData         14815 1.547 GB (1,660,565,924 bytes) 23/08/2021 17:22:38

Like any other Exchange Online content, Microsoft Search includes the compliance records in its indexes, which makes the records available for eDiscovery. Communications compliance policies also process compliance records to detect policy violations such as threatening or offensive behavior.

In addition to their role in eDiscovery, compliance records are the basis for retention processing for their originating workloads. Instead of having to create separate jobs to apply retention policies against the workload repositories, a single processing job runs against the compliance records and synchronizes policy actions such as removing items with the workloads.

Originally, the Exchange Managed Folder Assistant (MFA) processed the Teams compliance records. This was a pragmatic step because MFA could process the compliance records along with other mailbox content and avoided the need to create a special retention assistant for Teams. When MFA processed Teams content, it synchronized the results back to Teams, which applied the deletions to its Cosmos DB-based message store. Teams clients then picked up the changes from the message store and removed items in local caches.

The implementation worked well for Teams channel conversations and chat messages, but a better solution was needed to deal with the growth of workloads supporting retention policies and the need to process special cases like conversations in Teams private channels. To solve the problem, Microsoft implemented a specific background assistant to handle retention processing for all Microsoft 365 workloads except Exchange Online (Exchange is different because it supports both the older mailbox retention policies in addition to Microsoft 365 retention policies).

The Retention Assistant and the Microsoft 365 Substrate

Today, the retention assistant evaluates retention policies against OneDrive for Business, SharePoint Online, Yammer, and Teams using the “digital twins” of workload items stored in the Microsoft 365 substrate. Digital twins are copies of content stored in the Microsoft 365 substrate (in Exchange Online) to make it easier for shared services like Search and artificial intelligence services to process items. It’s much easier for a service to process items in a single place than having to deal with multiple repositories, each requiring a different interface (API).

In some cases, the digital twins are not complete copies, but they’re sufficient to allow shared services to operate efficiently and effectively. If you want to hear more about how the substrate works, consider signing up for TEC 2021 to hear Microsoft CTO for Modern Workplace Transformation Jeffrey Snover discuss the topic.

Substrate items coming within the scope of retention policies are subject to retention periods and actions. As the assistant processes items, it checks if the item’s retention period has lapsed, and if this is the case, invokes the retention action. This could involve additional processing, like putting the item into a manual disposition cycle, or it could be a simple delete action. The assistant interacts with the underlying workloads to ensure compliance with retention policy actions. For instance, if the assistant determines that the retention period for a document stored in a SharePoint Online document library has expired, it instructs SharePoint Online to move the document into the site recycle bin.

The Black Box of Microsoft 365 Retention Processing

Although it’s possible to track the processing done by MFA for mailbox items, Microsoft has not yet made an equivalent capability available for the retention assistant. This means that administrators who want to validate the effectiveness of retention processing need to make manual checks of content in repositories like SharePoint Online sites to ensure that items which should retention policies should remove are no longer present. As evident in this useful flowchart, understanding how retention policies process items can be a complex business, especially when locations like sites or teams come within the scope of multiple policies.

Making sense of complex retention policies is why the retention assistant exists. I’m sure it does its job very well. It would just be nice to be able to validate and understand exactly what actions the assistant takes for different locations. Is that too much to ask?

Learn more about how Office 365 really works on an ongoing basis by subscribing to the Office 365 for IT Pros eBook. Our monthly updates keep subscribers informed about what’s important across the Office 365 ecosystem.

Sensitive Information Types for Use with DLP

Data Loss Prevention (DLP) isn’t the most exciting topic, but it’s an important way to protect sensitive information stored in Exchange Online, SharePoint Online, OneDrive for Business, and Teams. Office 365 E3 licenses are needed to use DLP policies. The exception is Teams, which for some bizarre reason requires Office 365 E5.

Matching Sensitive Data

The foundation for DLP is the ability to find sensitive information within items. Microsoft 365 does this by scanning items for matches against definitions of sensitive information types as items are added to workloads, mostly when new or changed information is indexed.

The definition for a sensitive information type is a pattern identified by a regular expression or function. For instance, credit card numbers are matched if a fourteen or sixteen digit number is found which complies with Luhn’s algorithm (also used to check other sensitive information types like Canadian Social Security numbers). Additional confidence during the matching process is gained by the presence of other evidence close to the matched term. For instance, the word “Visa” or “MasterCard” close to a number which passes Luhn’s test increases the percentage chance that the number is a credit card.

Organizations can create their own sensitive information types to match information specific to their business, like customer numbers or project identifiers. These definitions join the set of common sensitive information types defined by Microsoft for use in DLP policies.

New Sensitive Information Types

Office 365 notification MC230755 published on 18 December brings the news that Microsoft has added 49 new sensitive information types to its set, which now includes 201 definitions. The new sensitive information types are now rolling out.

The definitions of sensitive information types created by Microsoft are described online, including the pattern and keywords used in the matching process. You can also get a quick count of the current set by running the Get-DlpSensitiveInformationType cmdlet. In this case, we see the 201 standard Microsoft definitions and 3 added by the organization:

$Dlp = Get-DlpSensitiveInformationType
$Dlp | Group Publisher | Format-Table Name, Count

Name                              Count
----                              -----
Microsoft Corporation               201
Office 365 for IT Pros                3

Microsoft says that the new definitions “unbundle” European Union definitions for driver’s license, passport, and social security numbers. In other words, instead of using generic definitions for these types, country-specific definitions are available for individual European Union countries like Latvia, Hungary, and Luxembourg (Figure 1).

If you’ve been using the Euro definitions in DLP policies, Microsoft recommends that you consider upgrading to country-specific sensitive information types if available to increase the accuracy of matching.

We cover DLP in Chapter 22 of the Office 365 for IT Pros eBook. Subscribe now to stay up to date with changes across Office 365. Our monthly updates will surprise and delight you!

Windows 10 and Edge Deliver Signals for DLP Evaluation

Announced as Generally Available on November 10, Endpoint DLP is a Microsoft 365 offering which uses signals generated by actions performed on Windows 10 workstations to evaluate against DLP policies. Supported actions include copying files to removable media like a USB or to a network share, printing files, uploading to a cloud app, or copying data to the clipboard.

Microsoft leverages its control of Windows and Edge by avoiding the need to deploy additional agents to monitor activity on a workstation. The necessary code to detect actions and submit them for DLP evaluation is incorporated into Windows 10 (version 1809 or later) and recent versions of the Edge browser.

Edge is the preferred browser because it understands how to respect endpoint DLP policies, and you can block other browsers from accessing files protected by policies. For instance, you could block Chrome or Firefox from opening a Word document if a specific retention label is present.

Not an Office 365 Feature

Before you can use Endpoint DLP, you need Microsoft 365 E5 licenses or either the Microsoft 365 E5 information protection and governance or compliance add-ons. This is understandable given that Windows 10 is bundled in the Microsoft 365 suite. Being able to gather information from Windows is a big part of the Endpoint DLP value proposition and it’s important that users have access to builds of Windows which include the DLP code. Having a Microsoft 365 license makes it more likely that users will be current, and not run something like an old Windows 7 or Windows 8 device.

Workstations used by licensed accounts can be onboarded (enabled) through the Microsoft 365 compliance center to start the flow of signals for DLP evaluation, unless they are already enrolled for Windows Defender, in which case Endpoint DLP works without any further configuration.

Looking for Violations

Once a workstation is enabled, actions taken by the user are monitored for potential violations against policy using the same kind of conditions as used to monitor Office 365 activity. For example, attempts to upload documents containing credit card numbers can be detected and stopped. Supported file formats include Office documents, PDF, text, and source code.

Endpoint DLP settings for the organization can be adjusted in the Microsoft 365 compliance center (Figure 1) to reduce the amount of noise in signals by excluding certain folders like the recycle bin, temp folder, or folders used for non-work files. It’s also possible to allow uploads to specific cloud services without generating a violation. Policy thresholds can be set to generate alerts when a large number of similar events happen. For instance, a policy could alert administrators if someone prints more than twenty documents assigned the Confidential sensitivity label.

Checking Devices

When Endpoint DLP is available in a tenant, DLP policies can be created for a target location called Devices, just like choosing SharePoint or Exchange as policy locations. The normal approach is to separate device policies from those used with Office 365 workloads, but you can combine them. Device policies have separate settings for restrictions to enforce when conditions are met (Figure 2).

Signals to SIEM

Apart from being used by DLP, the signals generated by devices can be gathered and analyzed in a SIEM. An example using Azure Sentinel is described in this article.

Good for Some Organizations

Some organizations will like Endpoint DLP very much. Others will not be interested because of the cost of Microsoft 365 licenses, presence of non-Windows devices, or because they’ve invested in other solutions. In either case, this is an area that’s worth keeping an eye on because the signs are that Microsoft is taking advantage of its Information Protection, Office, and Windows assets to create a compelling unified DLP story.

For more independent information about Endpoint DLP, read this article by MVP Anders Onevinn.

For more information about DLP for Office 365 workloads (Exchange, SharePoint, OneDrive, and Teams), read chapter 22 of the Office 365 for IT Pros eBook.

Error in Retention Processing Removes Personal Chats

Reading Monday’s report in the Register about the problem KPMG suffered when erasing Teams personal chat data for 145,000 users, you might ask the question “how did this happen”? The answer is that an error was made in a change applied to an Office 365 retention policy applied to Teams personal chat. Instead of removing a user from the policy, the update applied the policy to the entire KPMG deployment.

Teams compliance records for personal chats are captured in user mailboxes. These records are used by content searches, eDiscovery cases, and other data governance features like communications compliance policies. Office 365 retention policies to control Teams data use the Exchange Managed Folder Assistant (MFA) to process the compliance records in mailboxes according to policy settings. Exchange Online synchronizes the deletions made by MFA to remove compliance records from the mailboxes to Teams, which then removes the items from its data store in Azure Cosmos DB. The cycle completes when the deletions synchronize from Teams to clients. The overall process used to take far longer than it used to.

KPMG Error

Errors happen in life and in IT. I don’t have direct knowledge of what happened in this case, but it looks very like an administrator updated the retention policy for Teams personal chat and applied it to everyone instead of excluding a user from the policy. It’s easy to do if you’re not paying attention (Figure 1).

Only messages in personal chats were affected. Files shared in chats remained unaffected in user OneDrive for Business accounts. Different settings in retention policies for Teams apply to channel conversations, so these messages were unaffected too.

Restrictive Policies for Chats

Many organizations apply restrictive retention policies to Teams personal chat, which is one of the reasons why Microsoft enabled a 1-day retention period for this data. The logic is that personal chat is much like the discussions in WhatsApp and all business discussion should be conducted through Teams channels. That’s a reasonable approach but one that founders on the simple fact that Teams supports group chats for up to 250 users. You can do a lot of business with 249 others, especially if you follow the advice to move debates into chats before presenting an agreed position in a team channel.

Avoiding Errors in Retention Processing

I’m sure KPMG has good change control policies in place to make sure that the right change is made at the right time. You could question making a change to retention policies manually in such a large organization, but on the face of it the proposed change seemed straightforward and easy.

Other large tenants develop and deploy PowerShell scripts to automate management operations and test and debug those processes in test tenants (as the Register report notes, “to automate service execution and remove human intervention in policy management”). However, it can be argued that the nature of retention policies is that they don’t change all that often, making the investment to automate this operation less attractive than others.

The basic fix for something like this is to make sure that anyone who makes a change understands the technology and knows what will happen if they update a retention policy. Asking someone qualified to check that the right change is being made before it is committed is also possible. Such is the benefit of hindsight.

Proponents of backup services for Office 365 will say that they could rescue the situation by being able to restore the deleted compliance records back to user mailboxes. This will certainly solve the eDiscovery gap. Regretfully, no API exists to allow restore back so that the missing data appears as chats, leaving users no better off than they are now. There’s no silver restore bullet available in this scenario.

Retention policies can be complicated and they work differently across Exchange Online, SharePoint Online, OneDrive for Business, and Teams. Learn how to master retention processing by reading Office 365 for IT Pros.

Email Exfiltration Controls for Office 365 connectors

Updated: 18 June 2021

In May, I wrote two articles about how Office 365 tenants can restrict users autoforwarding email from their Exchange Online mailboxes. The first article covered OWA, the second more general restrictions. In the second article, I pointed out that Power Automate (aka Flow) cheerfully ignores any restrictions imposed by Exchange Online, thus giving those who want to transfer email outside the organization a handy way to accomplish their goal.

That was then and this is now. Microsoft has just introduced some additional capabilities to help tenants control “email exfiltration” through Office 365 connectors. The immediate use case is to stop Power Automate flows sending, forwarding, or replying to email. Exfiltration is an interesting word to choose, and one that will be unfamiliar even to native English speakers. One definition I found that seems to fit is that data exfiltration is any unauthorized movement of data. In this instance, we want to keep email inside Exchange Online so that it’s exposed to compliance and data governance tools, so the unauthorized movement of data is of messages to an external email address.

Exfiltration Headers

There’s nothing complicated in the new controls. Some well-understood and reliable mechanisms are deployed to detect and stop outbound email generated by Power Automate addressed to external recipients. What’s changed recently is that Power Automate now adds an SMTP x-header to messages to identify its traffic. For example, I created a flow to fire when a new item is added to a SharePoint list. The message sent has the following headers:

x-ms-mail-application: to identify that the message comes from Power Automate. For example, my flow generated the following header. The underlined identifier is important because it can be used to allow or block messages from specific flows.

Microsoft Power Automate; User-Agent: azure-logic-apps/1.0 (workflow d356b212a66640dab94fd13546ca88d8; version 08586039113867675952) microsoft-flow/1.0

x-ms-mail-operation-type: to identify whether the message is a send, forward, or reply. In this instance, SharePoint Online creates a new message, so the action noted is Send. The value can also be Forward. Either will work.

To find this information, I sent the message to an Outlook.com address and examined it with the Message Header Analyzer after it was delivered (Figure 1).

Implementing an Email Exfiltration Block in a Transport Rule

Anyone who has ever created an Exchange transport (mail flow) rule knows that all outbound mail passes through the transport service, which examines and applies the conditions set in rules. In this instance, the rule is very simple. Figure 2 shows all that’s needed for a complete block of all email sent to external recipients via Power Automate flows.

The rule is: If the recipient is external, check if the x-ms-mail-application header is present and contains the words “Power Automate.” If it does, block the message and send the user a reject notification.

The rule conditions and action are: If the recipient is external, check if the x-ms-mail-application header is present and contains the words “Power Automate.” If it does, block the message and send the user a reject notification.

You can compose some nice text to explain the problem to the user which Exchange Online will insert into the reject message (Figure 3).

Microsoft’s article explains how to add conditional processing and exceptions. You might want to allow some flows to run because they are needed to send email to invoke an external process, or you might want to allow flows from specific senders or addressed to specific recipient addresses because you’re happy that the email is necessary and doesn’t compromise the organization’s data governance policy.

Good Flow Controls

The email exfiltration control is simple and effective. It’s just strange that it’s taken Microsoft four years since the introduction of Flow in April 2016 to figure out that controls are needed over email generated by Power Automate. In their defense, the data governance landscape was very different in April 2016 and Office 365 did not have the same kind of compliance feature set that’s available now.

Communication Compliances Policies Depend on Compliance Records Captured by Microsoft 365 Substrate

In a change so good that it deserved two identical Office 365 notifications (MC218305 and MC218304). Microsoft revealed on July 11 that communications compliance policies now support hybrid deployments by monitoring Teams chat and channel messages sent by users with on-premises Exchange mailboxes.

Communication compliance policies replaced supervision policies in April 2020 as part of Microsoft’s Insider Risk solution. Background processes monitor user communications in email, Teams, and Skype for Business Online to detect potential violations of regulatory, legal, or business policies. To check Teams messages, the processes use the compliance records captured in user and group mailboxes.

Communications compliance policies required Office 365 E5 or Microsoft 365 E5 compliance licenses.

Cloud-Only Mailboxes or Shards

In hybrid deployments, the Microsoft 365 substrate creates special cloud-only mailboxes for users with on-premises mailboxes. These mailboxes, or “shards,” cannot be accessed by users or any administrative tools available to tenants. The substrate creates compliance records in the cloud-only mailboxes to capture details of messages sent by hybrid users.

Content from the mailboxes used for hybrid users are indexed and are discoverable by Office 365 content searches, but only if you ask Microsoft to enable “app content” searches. When this is done, an extra option appears in content search settings (Figure 1).

It’s also possible to create a content search which includes hybrid mailboxes by setting the AllowNotFoundExchangeLocationsEnabled parameter to $True for the New-ComplianceSearch cmdlet.

Why Few Extra Violations Might Be Detected

The change made to communications compliance policies extend their reach to process the messages held in the cloud-only mailboxes. As reported by the Exchange engineering group at Ignite 2019, hybrid deployments often move most of their mailboxes to Exchange Online to take advantage of the functionality available there. The mailboxes which remain on-premises are there for a specific reason which stops them being moved to Exchange Online.

The update might not detect many more violations because it’s likely that a copy of offending messages are already available in a tenant user’s mailbox or group mailbox. However, it increases the coverage by communications compliance policies by scanning conversations from the scenario when all the participants in personal or group chats are hybrid users.

Understanding the detail behind how compliance policies work inside Office 365 is key to constructing a solid data governance framework for a tenant. Learn more from the in-depth coverage in the Office 365 for IT Pros eBook.

Capturing Crucial Office 365 Audit Data Requires E5 Licenses

In January 2019, Microsoft announced that they were adding an event called MailItemsAccessed to the set of audited operations captured in the Office 365 audit log. Microsoft claimed that the new event would “capture details of when a message in a mailbox is opened by the mailbox owner, delegate (someone with read access to the mailbox) or using administrative access” leading to audit information delivering “comprehensive forensic coverage of mailbox accesses.”

Time moved on and in March 2019, Microsoft said that they had halted the deployment of MailItemsAccessed to Office 365 tenants. Software has a habit of hitting delays and it was speculated that the overhead involved in gathering a massive number of message access events would place a strain on Exchange Online.

All went quiet for a while, which prompted me to ask Microsoft in June what was happening. They provided an odd statement that faintly indicated that the MailItemsAccessed event might appear in Q3 (July to September).

Crucial Security or Compliance Audit Events

Q3 came and went without a trace of any message access being captured in the Office 365 audit log. But last month Microsoft released documentation for Advanced Audit in Microsoft 365 (now Purview Audit Premium) which makes it clear that MailItemsAccessed is now regarded as the first example of a “crucial” security or compliance-related audit event included in their advanced audit offering. Previously, Microsoft called these events “high-value.” In either case, Microsoft defines the event as “one that can help you investigate possible breaches or other forensic-related investigations.”

Update October 19: Microsoft has released three additional crucial events to handle email sends and searches of mailboxes and sites.

In a nutshell, if you want to see information about who accessed an item in a mailbox, you need to buy some Office 365 E5, Microsoft 365 E5 or Microsoft 365 E3 with Compliance licenses.

Some MailItemsAccessed records can be found in the Office 365 audit log for my tenant audit and viewed using the Search-UnifiedAuditLog cmdlet or the Audit log search (Figure 1). But all the records that have turned up so far (in about a month) are for “sync” activities for various folders like the Inbox. Sync records aren’t very exciting because all they record is the synchronization of a complete folder using a client like Outlook desktop. The really interesting data lie in bind records, which record access to individual messages.

It’s also interesting to learn that Exchange Online applies throttling for MailItemsAccessed events. If a mailbox generates more than 1,000 bind events in a 24-hour period, Exchange Online stops recording MailItemsAccessed events for bind operations for another 24 hours before resuming capture of these events. Microsoft says that less than 1% of mailboxes are subject to throttling.

You can download an example of how to extract and report MailItemsAccessed audit events from GitHub.

Audit Log Retention Policies

Apart from capturing crucial audit events, the advanced audit feature also allows tenants to configure audit log retention policies. These policies work much like mailbox retention policies. You define a retention policy for selected audit events with a set retention period and Office 365 removes those items after that period. A tenant supports up to 50 audit log retention policies.

This example runs the New-UnifiedAuditLogRetentionPolicy cmdlet to create an audit retention policy to remove any SearchQueryPerformed event executed by the background app@sharepoint process after three months instead of the twelve-month retention of audit events if the tenant has E5 licenses.

New-UnifiedAuditLogRetentionPolicy -Name "90-day Retention SearchQueryPerformed by app@sharepoint" -Description "Remove SearchQueryPerformed events from the app@sharepoint process after 90 days" -RecordTypes SharePoint -Operations SearchQueryPerformed -UserIds "app@sharepoint" -RetentionDuration ThreeMonths -Priority 8

You can only manage audit log retention policies with PowerShell using cmdlets accessible by connecting to the Compliance Center endpoint.

Purging the Office 365 Audit Log

You can choose to apply retention for any of the events captured in the Office 365 audit log and keep them for three, six, nine, or twelve months. That is, you can keep audit events for longer than 90 days for accounts with E5 licenses. Office 365 restricts E3 accounts to a 90-day retention period, which is also the period for which you can search audit events in the Compliance Center. Searches earlier than this point must be done with the Search-UnifiedAuditLog PowerShell cmdlet.

It’s a good idea for tenants who either want precise control over how long audit data is retained or want to clean up events that don’t add much value in terms of investigations. SharePoint is a notoriously “chatty” application when it comes to the capture of audit events, so I can see why tenants  might decide to keep important events like FileUploaded or FileAccessed for as long as possible while removing some of the chatter after 90 days.

Communication Woes

I don’t have any issue with Microsoft classifying the MailItemsAccessed event as crucial and demanding a premium for its capture into the audit log. Only some tenants will be interested in these events and they might well have E5 licenses already. I can also see the sense of not imposing a huge overhead on Office 365 to capture these events for E3 tenants. It’s just a pity that the communication around the introduction of MailItemsAccessed and its evolution to become a crucial audit event has been so fractured and incoherent. Microsoft can do better.

We track developments in Office 365 auditing, including the kind of events you can extract from the audit log, in a chapter in the Office 365 for IT Pros eBook. Knowing what goes on in a tenant is important and the audit log holds the answers to many mysteries.

An Unclear Announcement About Legal Holds for Teams

The wording of Microsoft’s February 2 announcement (MC202846) that legal hold is now supported for Teams private channels might have confused some. The announcement starts with “we have begun rolling out legal hold for Microsoft Teams,” which isn’t accurate. It has been possible to put the group mailboxes used by Teams on legal hold via PowerShell or by including group mailboxes in holds owned by eDiscovery cases for quite a while. For example, to set a group mailbox on litigation (everything is retained hold), you can run the command:

Set-Mailbox -Identity MyTeam -LitigationHoldEnabled $True -GroupMailbox

The real meaning of MC202846 is that holds are now supported to control the compliance records created for conversations in private channels. As noted in this article, private channels don’t have a group mailbox, so the same capture mechanism for compliance records used for regular channels doesn’t work.

Holding Teams Private Channel Conversations

When messages are posted to regular channels, the Microsoft 365 substrate captures copies of the messages and stores them in the Team Chat folder of the group mailbox belonging to the team. The lack of a group mailbox for private channels means that the substrate stores compliance records for Teams private channels in the mailboxes of all the members of the private channel, which is the same approach taken to capture records of 1:1 and group chats. Therefore, compliance records for a team are divided as follows:

• Messages posted to Teams regular channels. Stored in the Team Chat folder of the group mailbox belonging to the team.
• Messages posted to Teams private channels. Stored in the Team Chat folder of the mailboxes belonging to all private channel members.

Team Chat is a sub-folder of the Conversation History folder. “Team Chat” is the English language name. If you want to be sure that you’re accessing the right folder with PowerShell, check the folder type. For example, I often use a command like this to discover when the last compliance record was written to a mailbox:

Get-ExoMailboxFolderStatistics -Identity O365ITPros -FolderScope ConversationHistory -IncludeOldestAndNewestItems | ?{$_.FolderType -eq "TeamChat"} | Format-Table Name, ItemsInFolder, NewestItemReceivedDate   
                                       
Name      ItemsInFolder NewestItemReceivedDate
----      ------------- ----------------------
Team Chat          2469 4 Feb 2020 16:03:05

Teams Compliance Records Are Copies

Despite the efforts of some backup vendors, aided and abetted by a lack of understanding about Teams compliance records, it is untrue that messages stored in Exchange mailboxes are real Teams message data that are a good backup source. The Teams message store is in Azure CosmosDB, and the mailbox items are incomplete copies created as Outlook mail items. The upside is that because the compliance records exist in Exchange mailboxes, they are indexed and therefore discoverable by Office 365 content searches, available for retention processing, and suitable targets for holds.

Distinguishing Teams Private Channel Messages

The problem with storing copies of private channel messages alongside copies of personal data is how data governance processing can distinguish the items. After all, you probably don’t want the retention policy set for personal chats to apply to private channel messages. To solve the problem, compliance records for private channels are marked with a different source, allowing components like the Managed Folder Assistant to ignore private channel data when processing retention policies.

Code in the Managed Folder Assistant also handles ELC (Electronic Lifecycle) processing, a fancy name for checking if items must be retained because they come within the scope of a hold. ELC checks items before they are removed from a mailbox and keeps a copy if required by a retention policy or hold. Microsoft has updated the hold logic to allow processing of private channel items, which then means that private channel items now support holds.

Clients can’t get at the Team Chat folder to view or remove items (as seen in Figure 1, you can use MFCMAPI to look at the items), so all compliance records for private channels created since their introduction are still in group mailboxes. In effect, a hold existed for these items. After the update rolls out, holds placed on the mailboxes of members of a private channel will include the messages posted to that channel.

Holding Private Channel Messages

Because all members of a private channel store copies, it’s enough to put the mailbox of a single member of a private channel on hold to impose the hold on the messages posted to that private channel. The obvious flaw in this strategy is that if the chosen member leaves the organization and their mailbox is deleted, the hold lapses. The workaround is to include the mailboxes of two, or three members in the hold, but what happens if all the chosen members leave?

It would be better if the addition of a group mailbox to a hold created implicit holds on all private channel content stored in member mailboxes, but that’s not the way things work. At least, not for now.

Compliance is such an interesting topic! Seriously, when you need to understand Office 365 data governance, consider leveraging the wealth of experience in the Office 365 for IT Pros eBook.

Old Security and Compliance Center Split in Two

Office 365 Notification MC202599 posted on January 30 tells tenants that the Microsoft 365 compliance center and Microsoft 365 security center portals are being rolled out in February 2020 with worldwide completion by early March. These portals were originally announced in April 2018 and have been significantly upgraded since (see this post for a discussion of some shortcomings that existed in the preview versions about a year ago). Tenants with Microsoft 365 subscriptions already have access.

The new portals will replace the Office 365 Security and Compliance Center (SCC) introduced in 2016. Microsoft is dividing the functionality found in the SCC across two portals to better reflect the work done in each. It’s a reasonable thing to do considering:

• The number of new features added in the security and compliance areas since 2016 (like sensitivity labels) and the expansion of functionality to handle extra workloads. The SCC was becoming a catch-call for anything remotely connected to security, compliance, or data governance.
• Although administrators might do everything in small tenants, in larger enterprises a division of work often exists and those who handle compliance issues tend not to be the same people who deal with tenant security.
• Many enterprises have upgraded their subscriptions from Office 365 to Microsoft 365. The new portals deliver a common interface for security and compliance work across all areas of Microsoft 365. At least, that’s the vision.

Not Quite Ready for a Total Switchover

The SCC will remain available at https://protection.office.com/homepage for some time to come because not all of the functionality available in it has been transferred to the new portals. It takes time to untangle everything and move code to the new locations, which is why the Microsoft 365 compliance center has a link to the SCC. At this point, the compliance center seems more complete and useful than the security center.

I don’t really have strong feelings about the change. To me, it’s more important that features work all the time, something that could never be said of the SCC in the past. While acknowledging the difficulty of slip-streaming functionality into a portal at a hectic rate, the sad lack of attention to detail was distressing at times. Recently, the SCC seems to have settled down, perhaps because the developers left it alone while they concentrated on the new portals.

Let’s hope that the quality of the new portals is better than the SCC and that Microsoft focuses effort into making sure that all the basic functionality works robustly instead of new and glitzy features like the compliance score. I consider it strange that 75% of a possible maximum score is gained by Microsoft managing controls as a cloud provider (Figure 2).

It’s also annoying that many of the rating used to increase the score could be automatically calculated and are not. For example, the improvement actions include advice such as “implement spam filter” (isn’t that what Exchange Online Protection is doing?) and “implement ATP safe links” (ditto) and “black legacy authentication (has Microsoft looked at the settings active in the tenant?). Oh well, things will improve over time. Won’t they?

The advent of the Microsoft 365 Security and Compliance portals brings joy to the hearts of book authors. We have to refresh all our content to make sure that we refer to the right option in the right portal when we describe functionality. Expect the switchover to happen in the Office 365 for IT Pros eBook over the next few monthly updates.

Classic AIP Client and Label Management in Azure Portal to Cease

On January 6, Microsoft announced the deprecation of the classic Azure Information Protection (AIP) client and management of labels through the Azure portal (Figure 1). Following the discovery of some”technical issues” (and probably some customer feedback), Microsoft withdrew the announcement, which said that support for the client and management of its labels through the Azure portal would cease on March 31, 2021.

Microsoft hasn’t communicated what the issues blocking the deprecation are or when they will be resolved. A discussion in the Azure Information Protection Yammer community (available to join by anyone) said that Microsoft is “still committed to move all AIP customers as soon as possible to the new Unified Labeling client...” (Figure 2).

Unity of Focus

The focus of the Microsoft Information Protection team is on the unified labeling version of the AIP client, so-called because its labels are shared across multiple platforms. Microsoft’s public position expressed in April 2019 is that “going forward new features will be included in the Azure Information Protection unified labeling client whereas we’re not planning to add new features to the Azure Information Protection client.” They make a strong case that the unified client should be used for deployments if at all possible. Microsoft has published a list of features from the classic client which it does not intend to deliver in the unified client. If any of the missing features are critical to your deployment, you should discuss the situation with your Microsoft representative.

Given that the now-retracted announcement about the deprecation of the classic AIP client was made, we can anticipate that the deprecation will happen at some point in the near future. With this in mind, using the unified AIP client is the best route forward.

Office 365 Sensitivity Labels

In the Office 365 world, unified labels are known as sensitivity labels. Unified labels are unavailable in GCC. Office 365 tenants manage sensitivity labels through the Security and Compliance Center (or the new Microsoft 365 Security Center). The labels are published to users through label policies and can then be used to classify Office files according to their sensitivity. Recent versions of the Office click to run applications include native support for labels for Windows, Mac, and mobile, meaning that you don’t need to deploy the AIP client to use labels to protect confidential information, including the rights-management based encryption of email, Office documents, and PDF files.

Native support is in preview for Office online apps and the SharePoint Online/OneDrive for Business browser interfaces. Another preview allows sensitivity labels to apply classifications and control some aspects of Office 365 Groups, Teams, and SharePoint Online team sites. Both previews are expected to progress to general availability throughout Office 365 in a month or so.

Licensing

Office 365 E3 and E5 (and equivalent) tenants do not need additional licenses to use sensitivity labels. Tenants only need to deploy the AIP client if they want to apply labels to files stored outside Office 365 or to use some of the features not included in sensitivity labels, such as the AIP scanner. This article compares the current labeling capabilities of the classic, unified, and native clients. Microsoft says that some of the features missing from the unified client will appear during 2020.

Impact on Office 365 Tenants

In most cases, tenants who use the AIP client have migrated from the classic client to the unified client over the last year, so the deprecation should not have much impact unless you use the perpetual version of the Office desktop applications (including Office 2019), as this software does not include native support for sensitivity labels. In all cases, if you use the AIP client, you need Azure Information Protection P1 or P2 licenses, depending on the level of desired functionality.

Office 365 for IT Pros keeps an eye on what’s happening with Microsoft Information Protection and Office 365 sensitivity labels in Chapter 24. It’s a compelling read, if you’re into protection.

Legacy eDiscovery Tools Due for Removal by Mid-2020

Update: On March 27, Microsoft announced that they are postponing the scheduled retirement of the legacy eDiscovery tools for three months. The new retirement date is July 1, 2020.

Over the holiday period, Microsoft issued a note on December 30, 2019 about their retirement of Legacy eDiscovery tools. The original version of the note dealt with the retirement of the Exchange eDiscovery tools and version 1 of Office 365 Advanced eDiscovery. Microsoft subsequently refreshed their note on January 8, 2020 to add more information about the retirement of:

• Exchange Online in-place holds and eDiscovery.
• Office 365 Advanced eDiscovery V1.
• The Search-Mailbox cmdlet.

Although Microsoft had flagged its deprecation since 2018 the inclusion of the Search-Mailbox cmdlet in the revised document came as a surprise. Quite why Microsoft decided to issue a stripped down version on December 30 and a much more comprehensive version nine days later is not understandable. All it did was cause confusion.

Exchange Online in-Place Holds and eDiscovery

Unless you’re in the habit of running Exchange Online searches through PowerShell, you might have missed the news that Microsoft has been warning about the deprecation of the *-MailboxSearch cmdlets (the foundation of in-place hold and searches) for some time. These cmdlets first appeared in Exchange 2010 when the email server gained the ability to set in-place holds on mailbox content uncovered by eDiscovery searches. If you run searches through PowerShell (Figure 1), you see the warning that new searches cannot be created from April 1, 2020 (now July 1, 2020) and the cmdlets will disappear on July 1, 2020 (now October 1, 2020).

The Exchange Online Admin Center (EAC) gives much the same information. (Figure 2)

Office 365 Advanced eDiscovery

Office 365 Advanced eDiscovery (Figure 3) came from the Equivio acquisition in 2015 to become part of Office 365 E5 (also available as an add-on).

Version 1 of dvanced eDiscovery is replaced by a new version which is a more developed and easier-to-use edition of the original technology designed to serve the same function: make it possible for investigators to find relevant and interesting content in very large eDiscovery sets (think millions of items). V2 is still part of Office 365 E5.

Dealing with massive eDiscovery cases is a specialist business and it’s unlikely that large numbers of Office 365 tenants are affected by the deprecation, a feeling underlined by the fact that V2 of Advanced eDiscovery has been live inside Office 365 for several months now.

Moving from Exchange Online In-Place Holds

Microsoft’s original announcement posted on December 30 said:

“The In-Place eDiscovery and Holds tool in the Exchange admin center is also being retired. This tool is used for searching, holding, and exporting mailbox content in Exchange Online. Similar functionality is available in the Microsoft 365 compliance center.“

The similar functionality referred to in the statement comprises of Office 365 content searches and eDiscovery cases. Office 365 searches are faster, scale to deal with much more data, and include more than Exchange mailbox data, so there’s really no good reason to continue using the Exchange Online variant. Unless of course you have to because the organization has live eDiscovery cases running.

Microsoft’s document points to detailed steps for tenants to use PowerShell to recreate in-place holds and replace them with holds in Office 365 eDiscovery cases. The process works (eventually – you might need to tweak the PowerShell code), but tenants are advised to consult their legal advisors to ensure that the steps taken to establish new holds, test that the holds retain the right information, and release the old holds are documented in such a way that they can survive legal challenge.

No Export to Discovery Mailboxes

One piece of functionality that isn’t available with Office 365 eDiscovery cases is the ability to export search results to an Exchange discovery mailbox where the items can be reviewed. Microsoft suggests that you should use Advanced eDiscovery Review sets instead. This is fine, until you find out that Advanced eDiscovery requires Office 365 E5 or Microsoft 365 licenses, a substantial cost bump over E3. With that fact in mind, their other suggestion to export the results from an Office 365 content search to a PST and import the PST into a discovery mailbox is more practical, even if it uses a PST (a thing always to be avoided) and requires a lot more manual interaction.

Final Deprecation of Search-Mailbox

After warning that its deprecation was coming since 2018, Microsoft has given a date for removal of the very useful Search-Mailbox cmdlet. I covered this topic elsewhere last August and concluded that it would be unwise for Microsoft to remove Search-Mailbox because the replacement capabilities offered by Office 365 content searches don’t cover all the cmdlet’s functionality, not least in the ability of Search-Mailbox to remove more than 10 items from a mailbox at a time.

Microsoft originally said that they would remove Search-Mailbox from Exchange Online on April 1, 2020, subsequently revised to July 1, 2020. The cmdlet is still available in December 2020 (the last time I checked), complete with a notice that it is unsupported.

Although I understand Microsoft’s desire to remove what they view as an old cmdlet that can only handle a single workload and replace it with new cmdlets that work across Office 365, it is a pity that they have chosen to pursue this deprecation without upgrading Office 365 content searches to deliver the same features.

eDiscovery is covered in Chapter 20 of the Office 365 for IT Pros eBook. We stopped covering workload-specific eDiscovery technology several editions ago. Not because the technology isn’t interesting: we just had better material to discuss.

Office Depends on Unified Labeling Client

Office 365 Message Center Notification MC193997 (24 October) brings the news that OWA will soon be able to automatically apply Office 365 sensitivity labels to outbound email (Office 365 roadmap item 56649). Automatic application is based on the detection of sensitive data types in content. This feature requires accounts to have Office 365 E5 licenses and is being deployed to Targeted Release tenants with roll-out due to complete by the end of 2019.

Outlook and the other Office desktop applications can apply sensitivity labels automatically if the Azure Information Protection client is installed on a PC. Microsoft is now upgrading Office applications to support the same functionality and OWA is the first application to receive the capability. Microsoft announced the preview for support for sensitivity labels in the Office Online apps and SharePoint Online at the Microsoft Ignite 2019 conference.

Auto-Labeling on Send

OWA applies automatic labeling when messages are sent and uses the content of the message body to decide if the test for the sensitive data is satisfied. The other Office applications perform automatic labeling when files are saved.

The sensitive data types used to detect content for auto-labeling are the same as used elsewhere in Office 365 (for instance, by data loss prevention policies) and the same concepts of defining the number of occurrences of the data type that must exist in a file or message together with the confidence level that the data type is what it seems to be are used to define when a match exists.

Defining Labels for Automatic Protection

Figure 1 shows the properties of a sensitivity label with auto-labeling enabled. In this case, the label will be applied to any document or file where a client detects the existence of a single credit card number. Typically, you combine this feature with content marking to apply a header or footer to warn users that sensitive data is present and encryption (if desired) to protect the information. The message displayed to the user is free-form text to communicate to users when their items are automatically labelled. In this case a suitable message might be “Automatic protection applied because this message contains a credit card number.”

Priority Dictates Which Label to Apply

It is possible that several sensitivity labels invoke automatic labeling and match against content in an item. When this happens, Office 365 applies the label with the highest priority as ordered in the sensitivity label policy published to the user. The first label in a policy has the lowest priority while the last has the highest priority (most sensitive).

Need to know more about Office 365 Sensitivity Labels? Look no further than Chapter 24 of the Office 365 for IT Pros eBook. It’s packed full of information on this topic.

Successful Experts Live (Norway) Event

The blog has been relatively quiet this week because I’ve been traveling to Germany and Norway to speak at the European Collaboration Summit (Wiesbaden) and Experts Live Norway (Oslo). I discussed if Teams will replace Email at ECS (you can grab a copy of the presentation here) before making a rapid departure for the airport.

Experts Live events are community run at either country or European level. In this case, Ståle Hansen, who writes Chapter 16 about Teams Meetings and Voice in the Office 365 for IT Pros eBook, asked me to come to Oslo to speak at the Norwegian event.

Experts Live Norway was very successful and a good example of how professional and interesting a national-level community event can be. The agenda included a good mixture of sessions covering topics like Azure, programming with the Graph API, Teams, and Office 365. Speakers were a mixture of Microsoft employees, MVPs, and subject matter experts. Most sessions were in English.

All About Office 365 Data Governance (in very little time)

I spoke about Office 365 Data Governance (you can download a PDF of the presentation below) and naturally didn’t manage to fit everything I wanted to say into the allotted time. Such is life. I’ve just got to try and stop telling stories and asking questions…

If, like me, you haven’t been to Oslo in a while, the city is very modern and vibrant. The number of electric cars is very noticeable with lots of Teslas and BMW i3s and an occasional Jaguar i-Pace. All in all, an enjoyable event.

Need help understanding Office 365 data governance and compliance technologies? Look no further than the Office 365 for IT Pros eBook, where we devote several chapters to unraveling these mysteries.

Sometimes You Want to Get Rid of Inactive Mailboxes

Updated 17 April 2023

Sometimes Microsoft doesn’t communicate changes made to PowerShell cmdlets that introduce interesting new functionality. There’s so much change in the service that they could be forgiven for an occasional slip-up, unless of course you need to use the specific feature that is undocumented.

New Parameters for Set-Mailbox

Which brings me to the well-known Set-Mailbox cmdlet, which boasts two parameters called ExcludeFromOrgHolds and ExcludeFromAllOrgHolds, a fact highlighted by MVP Vasil Michev in his ongoing crusade to discover what’s hidden in the corners of Office 365.

These parameters allow administrators to exclude some or all org-wide retention holds from inactive mailboxes. Remember that an inactive mailbox is one belonging to an Azure AD account that has been deleted but is kept because a hold exists on the mailbox. The hold can be any form of hold supported by Exchange Online, including litigation holds and those set by Office 365 retention policies. Retention holds come in two flavors, org-wide and non-org-wide (in other words, holds that apply to all mailboxes and those that apply to selected mailboxes).

Excluding an org-wide hold means that when Exchange evaluates whether to keep an inactive mailbox, it ignores that hold. If all org-wide holds are ignored, the inactive mailbox will only be kept if a specific non org-wide hold exists.

Controlling Org-Wide Holds on Inactive Mailboxes

Why do these parameters exist? Well, Microsoft introduced inactive mailboxes several years ago as a way for organizations to keep mailboxes around for compliance purposes without having to pay for Office 365 licenses. The most common use case is when mailboxes are kept for ex-employees. The idea is that a tenant will apply a hold to keep the mailboxes inactive for the desired period and then release the hold when the mailboxes are no longer needed.

Org-wide holds apply to both active and inactive mailboxes. Over time, it’s possible that a tenant will add new org-wide holds. The effect is that the set of inactive mailboxes is likely to grow because any mailbox that is deleted will become inactive because one or more org-wide holds exist.

Keeping inactive mailboxes is good if intended. It’s not so good if you don’t want or need those mailboxes. One of the principles of data governance in Office 365 is that tenants should be able to decide what data to keep and what to remove, and keeping inactive mailboxes longer than they should be goes against that principle. I imagine that Microsoft introduced these cmdlets to give tenants the ability to decide what org-wide holds should apply to inactive mailboxes.

Discovering Org-Wide Holds

Org-wide holds are registered in the Exchange Online organization configuration. To see the set, run the PowerShell command:

# Retrieve org-wide holds for the Exchange Online 
Get-OrganizationConfig | Select-Object -ExpandProperty InPlaceHolds

mbx15382841af9f497c83f9efe73e51888d:1
mbx9696959111f74ecda8a40aef97edd2c2:1
mbx703105e3b8804a1093bb5cb777638ea8:1
grp703105e3b8804a1093bb5cb777638ea8:1
mbxc1e2d6f1785d4bf8a7746a26e58e5f66:1
grpc1e2d6f1785d4bf8a7746a26e58e5f66:1
mbxf6a1654abdba4712a43c354e28a4d56c:2
grpf6a1654abdba4712a43c354e28a4d56c:2

The holds we’re interested in start with mbx. Those starting with grp apply to Office 365 Groups. The values following are GUIDs that point to the retention policies defining the holds. If you’re interested in understanding how to resolve the GUID to find the retention policy, see the Compliance chapter in the Office 365 for IT Pros eBook.

Excluding Org-Wide Holds from Inactive Mailboxes

To exclude specific org-wide holds from a mailbox, run the Set-Mailbox cmdlet and pass the GUIDs for the holds you want to exclude in a comma-separated list for the ExcludeFromOrgHolds parameter. Use the same format for the GUIDs as reported by Get-OrganizationConfig. When you run the command, Exchange updates the InPlaceHolds property for the mailbox to note the excluded holds.

# Exclude specific org-wide holds from a mailbox
Set-Mailbox -Identity Kim.Akers -ExcludeFromOrgHolds "mbx9696959111f74ecda8a40aef97edd2c2:1", "mbx19200b9af08442529be070dae2fd54d3:1" 

Microsoft recommends that you use the distinguished name or ExchangeGUID property to identify the mailbox. This is to be absolutely sure that a unique value is passed because if you exclude the holds for the wrong inactive mailboxes, you run the risk that Exchange will remove these mailboxes permanently when it evaluates the holds that exist on them.

To remove all org-wide holds from a mailbox, run Set-Mailbox and pass the ExcludeFromAllOrgHolds parameter. Because you’re now removing all org-wide holds, it’s even more important to be certain that you’re processing the right mailboxes.

#Exclude all org-wide holds from the target mailbox 
Set-Mailbox -Identity $Mbx.DistinguishedName -ExcludeFromAllOrgHold

The Effect of Exclusion

I wrote a script to exclude all org-wide holds from the inactive mailboxes in my tenant. Here’s the relevant code to retrieve org-wide holds from the Exchange Online configuration and exclude inactive mailboxes from the mailbox holds. Figure 1 shows the script running.

[array]$InPlaceHolds = Get-OrganizationConfig | Select-Object -ExpandProperty InPlaceHolds
$InPlaceHoldsMbx = $InPlaceHolds | Where-Object {$_ -like "*mbx*"}

[array]$InactiveMbx = Get-ExoMailbox -InactiveMailboxOnly -ResultSize Unlimited | Select-Object -ExpandProperty Alias 

ForEach ($Mbx in $InactiveMbx) {
   Write-Host ("Excluding inactive mailbox {0} from org-wide holds" -f $Mbx)
   $Status = Set-Mailbox -Identity $Mbx -ExcludeFromOrgHolds $InPlaceHoldsMbx }

Immediately Set-Mailbox processes a mailbox, Exchange evaluated the holds to decide whether to remove the mailbox. After the script finished, the number of inactive mailboxes reduced from 39 to 17. This proves that you need to be ultra-careful when you exclude any org-wide hold from an inactive mailbox.

For more information about managing Exchange Online mailboxes, read Chapter 6 in the Office 365 for IT Pros eBook to discover even more valuable tips and techniques.

Checking Communications for All Users in an Office 365 Tenant

Office 365 supervision policies allow tenants to monitor the communications of users in email and Teams. The usual situation is that a company needs to ensure that certain groups of users don’t infringe regulations in their communications inside or outside the company. Supervision policies allow this by capturing a set percentage of messages matching predefined criteria for reviewers to examine.

Typically, you use distribution lists to define the set of users whose communications are reviewed. The background assistants that examine email and Teams messages expand the distribution list membership to know what individual users to monitor. If a new member joins the distribution list, they are added to the supervision group. If they leave the list, they are removed.

You can’t use dynamic distribution groups for supervision policies because the overhead of continually evaluating the group membership would be too high. The question therefore arose of how to maintain a distribution list when you want a supervision policy to check the email sent by every user in a tenant.

PowerShell Server-Side Filtering

My solution is to build a relatively simple PowerShell script to scan for mailboxes that are not in a distribution list and add those mailboxes to the list. The list is then used by a supervision policy to monitor whatever traffic is needed, perhaps to make sure that no one in the tenant calls any other user something offensive in email (defining the list of offensive terms is an interesting exercise).

In any case, the basis of the script is that you use one of the fifteen custom attributes available for Exchange Online mailboxes to store an indicator to show if the mailbox has been added the distribution list. The reason for choosing a custom attribute is that the Get-Mailbox cmdlet supports server-side filtering against these attributes, so retrieving a set of mailboxes that aren’t on the list is faster than if you use a property that needs client-side filtering. Server-side filtering means that Exchange does the work before returning a set of objects while client-side filtering means that you fetch all objects from Exchange and then filter them on the client. As 15,000 mailboxes were involved in this case, server-side filtering is a big win.

Creating and Populating the Distribution List for the Supervision Policy

With the decision made about the technique to use, the code is simple. The first thing to do is to create a distribution list (later you might like to hide this DL from the Exchange address lists so that no one tries to use it for other purposes):

New-DistributionGroup -Name "Supervisory Review Mailboxes" -Alias SRReview -ManagedBy TRedmond -MemberDepartRestriction 'Closed' -MemberJoinRestriction 'Closed' 

Now we can create a set of mailboxes that are not marked and then add those mailboxes to the DL. We also update the attribute for each mailbox to indicate that the mailbox is now in the DL. Note that we are careful to have Get-Mailbox only find user mailboxes because only people generate communications for supervision policies to monitor. There’s no sense in processing room mailboxes, shared mailboxes, and the like.

$Mbx = (Get-Mailbox -RecipientTypeDetails UserMailbox -ResultSize Unlimited -Filter {CustomAttribute7 -eq $Null})
$i = 0
ForEach ($M in $Mbx) {
   Write-Host "Adding" $M.DisplayName
   Add-DistributionGroupMember -Identity SRReview -Member $M.DistinguishedName -ErrorAction SilentlyContinue
   Set-Mailbox -Identity $M.Alias -CustomAttribute7 SRAdded 
   $i++}
Write-Host $i "mailboxes added to Supervisory Review distribution list"

The first time the script runs, it will take some time to complete because it finds and processes all user mailboxes. Thereafter, if you run the script weekly to pick up new mailboxes, it will be much faster because the call to Get-Mailbox will find only the new mailboxes.

For more information about supervision policies, read Chapter 21 of the Office 365 for IT Pros ebook. We have many other PowerShell examples throughout the book too.

Reason for Backup #101: Stop Accidental File Deletion

All companies seek reasons for customers to buy their products. Vendors who sell backup for Office 365 advance reasons to convince customers that they need to extract and copy data from Exchange Online, SharePoint Online, OneDrive for Business and other data to their cloud service. Often, one cited reason is that Office 365 doesn’t protect against accidental deletion of important information by users.

For SharePoint Online, the story usually goes something like this: a user deletes a document and doesn’t realize that they need the document until after SharePoint processes it through the first and second-level recycle bins and is permanently removed. In other words, our user takes more than 93 days to realize their mistake. Apparently, this loss of awareness happens to many otherwise rational human beings working today.

The unfortunate SharePoint administrator must then tell the user that their document is irretrievable. Backup vendors say that they can help because the lost document can be fetched from their repository. Cue happiness for the user and relief for the wise administrator who had the foresight to invest in the backup solution.

Backups Have Value

I’m not against backup solutions for Office 365. Backups have their place in highly regulated industries or when auditors insist that Office 365 tenants should keep a copy of their data outside the primary service. Some backup vendors, like Spanning, understand the challenges that exist to serve Office 365, including data sovereignty, protocols, and access to some data. Other companies don’t seem quite so aware of the technology inside Office 365 apps.

But it’s important to challenge statements like “Generally speaking, in most online services, the only backup you have for your organization’s data is via the recycle bin, which is automatically purged after a fixed period of time. After that, your data is gone forever.”

Office 365 Data Governance Changed the Game

This statement was true for SharePoint Online until the introduction of the Office 365 data governance framework in 2017. The SharePoint recycle bin still exists and uses two-phased progress towards final deletion. What’s different is that you can stop users removing important files from SharePoint Online libraries by using Office 365 retention policies and labels. Retention policies and labels are included in the Office 365 E3 plan.

First, you can assign retention labels with defined retention periods to documents. Users cannot delete a document that has a retention label until its retention period elapses. For example, if a document has a retention label for two years, it cannot be removed until it is two years old – or two years since the last modification (depending on the label settings).

Retention labels can be assigned manually or through policy. Default labels can be defined for a library so that SharePoint assigns the same label to every document unless a label was previously assigned. Or you can assign a retention label to a folder and have all files in the folder inherit that label.

Office 365 Retention Policies Work Too

And if you don’t want to use retention labels, you can deploy an Office 365 retention policy to ensure that no document is removed from SharePoint Online until it reaches a certain age. For instance, you might say that documents can’t be removed until they are five years old. In this case, users can remove documents from libraries that go through the recycle bin process, but when the normal 93-day limit is reached, SharePoint continues to keep the documents (in the site’s preservation hold library) until their retention period expires.

You can even apply a preservation lock to Office 365 retention policies to stop administrators reducing the retention period assigned in a policy. This is not something to do without thought, but it does help Office 365 satisfy the requirements of regulations governing the retention of electronic records in several industries.

Retrieving Held Files

Files retained by SharePoint in a site’s preservation hold library can’t be restored from the recycle bin, but they can be found and recovered by a content search. Or they can be copied from the preservation hold library. These are both administrator operations, but they are baked into Office 365 and easier (and much cheaper) than using a separate backup service.

SharePoint Can Hold Deleted Files For Much Longer Than 93 Days

The myth that SharePoint Online only keeps deleted files for 93 days is easy to understand if you view the situation through the lens of the functionality built into SharePoint on-premises servers, which don’t support Office 365 retention capabilities. But it completely misses the point that Office 365 has a data governance framework to help companies keep important information for as long as needed.

Tenant administrators should understand the technology that they use, and backup vendors have the right to make a strong case for their technology. However, it would be nice if discussions about the need for backup were based on business needs and technology facts rather than myths and half-truths.

Need help to understand Office 365 retention labels and policies? Look no further than Chapter 19 of the Office 365 for IT Pros eBook where this topic is discussed in great detail.

More Progress towards Enabling Sensitivity Labels

Along with announcing its intention to include licenses for Information Protection in Office 365 E3 and E5 plans, Microsoft made further progress to encourage widespread use of Office 365 sensitivity labels by upgrading policies to include some auto-label capabilities and shipping an update for the “unified labeling” preview for the Azure Information Protection (AIP) client.

The biggest barrier for adoption for sensitivity labels today is lack of support in Office apps (desktop, mobile, and online) for the labels. To bridge the gap until General Availability (expected later this year), Microsoft released a different version of the Azure Information Protection client. The “unified labeling” version reads label and policy information from Office 365 (sensitivity labels and policies are found in the Security and Compliance Center) instead of Azure. The unified labeling client has just been updated and can be downloaded here.

Some Work Still to do for Sensitivity Labels

The preview of the unified labeling client (V2.0.747.0 ) only works for Windows workstations. When installed, the unified labeling client adds a Sensitivity button to the Office desktop apps. By comparison, the regular version of the AIP client adds a Protect button. Both buttons serve the same function. They display a list of all the labels available to the user (from all applicable policies) to allow them to select which label to apply to a message or file.

Long term, the Office apps will have native (in-built) support sensitivity labels and you won’t need to deploy any other software to apply labels and have them mark and protect (encrypt) content. The idea is that you should be able to apply labels to Exchange Online messages (with OWA and Outlook) and files stored in SharePoint Online and OneDrive for Business.

I also expect Microsoft to overhaul the the limited (and old) support for rights management in SharePoint Online to make it easier for site owners to apply default labels. Some work also needs to be done to update the SharePoint Online and OneDrive for Business web apps to allow users to apply sensitivity labels, probably in much the same way as they can apply retention labels today.

Once sensitivity labels are fully deployed inside Exchange Online and SharePoint Online, it is reasonable to anticipate that Microsoft to enable support for sensitivity labels to other Office 365 apps.

Because Office 365 sensitivity labels and Azure Information Protection labels share common underpinnings, sensitivity labels can also be applied to files outside Office 365, in which case they act like AIP labels.

Auto-Label Settings for Sensitivity Labels

Office 365 administrators are used to the concept of using auto-label policies to assign retention labels to content discovered by background processes to match conditions set in a policy. Sensitivity labels have their own take on auto-labeling. Briefly:

• Auto-label conditions are set for a label instead of by policy.
• Matching is only possible against Office 365 sensitive data types. Auto-label policies for retention labels can also match against keywords.
• Applications that support sensitivity labels action the label settings when they detect matches. For instance, if you create a Word document and include a credit card number, the match is detected when the document is saved and the (AIP) client executes the auto-label action. In the example below, the action is to apply the label.

This form of auto-labeling has been supported by AIP labels for a couple of years, so its appearance inside Office 365 is evidence of the work going on to create functional equivalence between AIP and sensitivity labels.

Note that auto-label is a premium feature that requires Azure Information Protection P2 licenses. In the world of Office 365, it’s likely that access to this functionality will be controlled by the new Information Protection for Office 365 -Premium licenses in Office 365 E5 or the Advanced Protection and Compliance SKU.

Need more information about Sensitivity Labels and Encryption through rights management in Office 365? Head over to the Office 365 for IT Pros eBook and read Chapter 24!

Preparing for Office 365 Sensitivity Labels

Microsoft’s 15 February announcement (MC173614) that they are updating the Office 365 E3, E5, and Advanced Protection and Compliance SKUs to include new Information Protection service plans might have surprised some. After all, Office 365 E3 and E5 tenants are already automatically enabled for rights management and can use the feature to protect email and documents.

What’s happening is that Microsoft is clearing the decks to prepare for the general availability of Office 365 sensitivity labels and the predictable rise in interest about protecting Office 365 content, especially that stored in Exchange Online, SharePoint Online, and OneDrive for Business. It’s also likely that Microsoft will extend the reach of sensitivity labels to other Office 365 apps, including Teams.

Azure Information Protection Licenses

Today, a lacuna exists in licensing terms. Azure Information Protection (AIP) is the technology built on top of rights management. AIP labels can apply protection (encryption) or just mark content (for instance, with a footer). AIP labels can be used to protect content stored inside Office 365, but no integration exists between these labels and Office 365 apps because the predominant use of AIP labels is to mark content stored outside Office 365.

To use AIP labels to protect content, you need an AIP license. The license comes in two forms – standard and premium. The premium license covers automatic labeling, where applications like Word and Excel can apply labels based on content detected in files. Sensitivity labels support automatic labeling (enabled in the latest preview of the AIP client), and I anticipate that this will be a premium feature.

Clarifying Office 365 Licensing

Up to now, it has been assumed that because Office 365 E3 and E5 tenants are automatically enabled for rights management, their existing licenses cover protection applied by sensitivity labels. The new service plans clarify the matter. Although Microsoft’s announcement isn’t clear on the point, it seems logical that Office 365 E3 will include Information Protection for Office 365 – Standard in its list of service plans and Office 365 E5 will include Information Protection for Office 365 – Premium. This approach clarifies the licensing issue and allows for premium features like automatic labeling to be restricted to the higher Office 365 E5 SKU.

Because Information Protection is a separate service plan within a SKU (like Yammer or To-Do), you will be able to selectively enable or disable it for users. For instance, you might not want some people to apply sensitivity labels until they receive training and understand how protection works.

You don’t have to do anything to prepare for the change. The new service plans will turn up in March and once they appear in your tenant, you can enable or disable Information Protection for accounts through the Office 365 Admin Center or PowerShell.

For more information about Information Protection, read Chapter 24 of the Office 365 for IT Pros eBook. There’s lots of stuff there about encryption, rights management, templates, and AIP.

Sensitivity Labels are a Game Changer

Today’s Petri.com post discusses the use of Microsoft 365 sensitivity labels through an updated set of Office desktop applications coming soon. A previous post reviewed the migration from Azure Information Protection (AIP) labels. Of course, you can create and deploy sensitivity labels to protect Exchange and SharePoint content without going anywhere near AIP. In the long term, AIP labels are only needed if you want to protect content that isn’t stored inside Office 365.

The important point is that AIP labels and sensitivity labels share a common foundation in the Azure Information Protection service and the set of rights management templates published through that service. Both update the same file metadata and both use the same permissions.

Rights management has been around for a long time. I think the technology got a bad rap because it was deemed complex and unwieldy.  Sensitivity labels change the dynamics because they are easy to create and publish, and easy for users to apply to Office documents stored inside SharePoint and to email sent by Exchange Online. For these reasons, sensitivity labels will make protection through rights management and encryption a daily part of Office 365 life.

Rights and Permissions

Protection means that a user cannot access content unless they have the rights to do so. Furthermore, once a user accesses content, the permissions assigned to them (the rights) dictate what they can do (print, edit, forward, reply, etc.). Protecting documents and email gives authors confidence that they control that content. For instance, adding a new recipient to a reply to protected message is useless from the perspective of that recipient because they don’t get the right to open the content because they’re not in the set assigned to the original message. All in all, protecting Office 365 content is a good thing.

The Downside of Protection

Even good technology can have its downside and protection is no different. Once you protect a document, you lose some functionality. The biggest issue is that Office 365 cannot search the content because it can’t decrypt the content to index it. This means that content searches and eDiscovery must rely on document metadata for its indexes. If users populate the metadata with terms that search can use to find documents, it might not be so much of a problem. But users are humans and humans often don’t do such a good job with metadata.

Of course, if a content search finds some protected content, you then face the further difficulty of what to do with it. Investigators might want to review the content to check whether it’s needed for eDiscovery purposes, but the content is encrypted. The solution is to use super-user privilege to decrypt the content. A technical solution exists, but dealing with encrypted files can be painful.

ISVs and Protection

In addition to the issues thrown up inside Office 365, any ISV who deals with Office 365 content needs to understand if the advent of sensitivity labels and increased use of rights management within Office 365 impacts their product. If a product depends on gaining access to content, it’s going to run into a brick wall when it tries to access protected content.

No Argument Against Protection

You can’t really argue against the goodness of securing access to confidential information. Sensitivity labels give users control over their information, and they should know what’s confidential and needs to be protected. Some user education is needed to ensure that everyone knows how best to use the range of visual markings and protection available through sensitivity labels, but overall, this is a very good feature that’s arriving into Office 365.

To read more about sensitivity labels, rights management, and encryption, go to Chapter 24 of the Office 365 for IT Pros eBook.

The Need for a Nice Clean Mailbox

Note: Search-Mailbox is due for deprecation on July 1, 2020. See this post for more information.

Another question, this time from the Facebook Office 365 group:

“How can i delete a whole bunch of emails in a shared mailbox using the online mail browser? (instantly)“

It’s quite common to find that a clean-up is needed for shared mailboxes. It might be possible to do the job manually by selecting and removing the messages, or using OWA’s Cleanup mailbox option, but both options can take a long time to run and move items into the Deleted Items folder, where you might want to remove the items permanently.

Cleaning Options

The OWA options are user-driven and can be applied to any mailbox to which a user has access. The other available options are administrative actions. Let’s assume that you want to cleanup a mailbox with an Inbox folder of 100,000 items. This is well under the Exchange Online folder limit of 1 million items, but it’s still a bunch of data to process. Here are three obvious actions that can be taken:

1. Use the Search-Mailbox cmdlet to remove items from the mailbox. The upside is that Search-Mailbox can remove the items permanently; the downside is that Search-Mailbox only returns 10,000 items at a time, so it can only remove 10,000 items. Ten searches are needed to remove our 100,000 items.
2. Apply an Exchange mailbox retention policy to the Inbox folder to remove all items after they are “x” days old (let’s say 7). The upside is that the Managed Folder Assistant does its processing in the background and can remove items permanently. The downside is that the Managed Folder Assistant might not process the mailbox for another seven days (the workcycle used in Exchange Online), and you will have to wait for its completion to see the results of its work.
3. Create a new mailbox and switch it in to replace the old mailbox. The upside is that you get a clean mailbox immediately, which is what you want. The downside is that you might need to recover items from the old mailbox before it is discarded. As was pointed out to me, make sure that the new mailbox has the LegacyExchangeDN of the old mailbox as a proxy X.500 address so that messages sent to the mailbox using addresses stored in user autoaddress caches don’t NDR.

Other available methods include creating some Exchange Web Services (EWS) code to delete the items or run an Office 365 content search to find the items and then remove them. There are two things to remember about using a content search to remove items. First, the actions supported by content searches only include soft deletes. Second, a content search can only remove 10 items at a time.

The Best Choice

If you need a clean mailbox as quickly as possible, the new mailbox approach might be best. If you want to keep the existing mailbox only but need it cleaned up as quickly as possible, Search-Mailbox is probably best used (assuming that your account has the necessary permissions).

But overall, if you want to impose control on a mailbox that tends to swell in terms of messages, use retention policies to keep a cap on what’s stored. Because it can apply policies to specific folders, an Exchange mailbox retention policy is more precise than an Office 365 retention policy.

——————————————-

For more information about mailbox management, read Chapter 6 of the Office 365 for IT Pros eBook. Retention policies and the Managed Folder Assistant are covered in Chapter 19.

ESPC18 in Denmark

Next Wednesday, I shall be speaking at the European SharePoint Conference (ESPC18) in Copenhagen, Denmark. My chosen topic is “Office 365 Data Governance,” which is somewhat challenging because the underlying technology changes all the time. The recent introduction of sensitivity labels into Office 365 (available to E3 and E5 plans) is a good example because the new labels make protection (encryption) through rights management more accessible than ever before to Office 365 tenants.

Other examples are the ongoing efforts by the Teams development group to support different aspects of data governance. This year, they’ve added support for Office 365 retention policies and expanded their ability to capture compliance records for hybrid and guest users, including external people pulled into a 1:1 chat. Teams will soon support Office 365 data loss prevention policies too.

Microsoft has also reinforced the importance of the Office 365 audit log by increasing the retention period for records to 365 days for E5 users. On the other hand, Microsoft is still struggling with the problem of truncated records for Azure Active Directory events reported in September. All in all, data governance is an interesting area to review.

Expanding Views

One point I will be making is that Microsoft is not investing in workload-specific functionality for data governance within Office 365. Any new features that come along apply to all workloads that support the data governance framework (Yammer is still a notable outlier).

What this means is that anyone dealing with data governance topics like retention, protection, compliance, eDiscovery, and auditing needs to change their own mindset away from the products that they might know well to understand how things are done in a pan-Office 365 way.

Two base workloads exist inside Office 365 – Exchange and SharePoint. An Office 365 administrator absolutely needs to understand at least one of these workloads inside-out, but they also need to know how the other workload functions and how they two interoperate and support other applications like Teams and Planner. It’s a new world.

Presentation Slides

The slides I used for the session are posted online: Exploring Compliance With Office 365 – ESPC18

—————————————————-

If you can’t get to Copenhagen for ESPC18, you can still read about Office 365 data governance in Chapters 19 (retention), 21 (reporting and auditing), 22 (data loss prevention), and 24 (sensitivity labels and rights management) in the Office 365 for IT Pros eBook.

Reporting Files with Labels

Let’s assume that your users have applied Azure Information Protection or Office 365 sensitivity labels to a bunch of documents. How can you create a report of files to know which files are labelled and protected?

PowerShell to the Rescue

As it turns out, you can use PowerShell to examine the Azure Information Protection properties of files and extract the necessary information and use that to create our report. As always, an example helps to illustrate the point.

This PowerShell script looks for any Excel and Word documents in a folder (which could be a folder holding files copied by the OneDrive sync client from a SharePoint Online or OneDrive for Business document library). Each file is checked for the presence of an Azure Information Protection (AIP) or Office 365 sensitivity label (the same metadata is used). You need to be a tenant or AADRM administrator to be able to run the code.

$Report = @()
$Files = (Get-ChildItem -Path "c:\temp\" -Include *.docx, *.xlsx -Recurse)
ForEach ($F in $Files) {
$FileName = "C:\Temp\" + $F.Name
$TemplateName = $Null
$Status = (Get-AipFileStatus -Path $FileName)
 If ($Status.IsLabeled -ne $False) {
 If ($Status.RmsTemplateId -ne $Null) {
    $TemplateId = [GUID]($Status.RMSTemplateId)
    $TemplateName = (Get-RMSTemplate -Identity $TemplateId.Guid ErrorAction SilentlyContinue ).Name }
    $ReportLine = [PSCustomObject]@{
         File        = $F.Name
         IsLabeled   = $Status.IsLabeled
         LabelId     = $Status.MainLabelId
         Label       = $Status.MainLabelName
         Date        = $Status.LabelDate
         RMSGuid     = $Status.RMSTemplateId
         RMSTemplate = $TemplateName
         Owner       = $Status.RMSOwner }
 $Report += $ReportLine
}}
$Report | Export-CSV -NoTypeInformation c:\Temp\LabeledFiles.csv

Outputting Details

If a file has a label, we extract details of the label and the underlying rights management template. One interesting thing that I discovered when writing the script is that the Get-RMSTemplate cmdlet fails when passed the GUID of a template used by an Office 365 sensitivity label. The GUIDs are correct, but for some reason the cmdlet fails. The output for an individual file that has a label with protection is:

File        : ABPs and Teams.docx
IsLabeled   : True
LabelId     : 81955691-b8e8-4a81-b7b4-ab32b130bff5
Label       : Secret
Date        : 13 Nov 2018 12:29:42
RMSGuid     : c7fc2174-097c-4123-9cad-15f1a32cb145
RMSTemplate : Secret
Owner       : Tony.Redmond@office365itpros.com

Script Output

The output for the script is a CSV file that can be opened and analyzed with Excel or Power BI.

This script is included in our coverage of protecting Office 365 content in Chapter 24 of the Office 365 for IT Pros ebook. There’s another 44 pages about protection to read there…

Limiting Data

Following on the Dutch report slamming Microsoft for potential GDPR violations in how it deals with personal data extracted from Office and Office 365, my thoughts turned to how you can stop some of the data flows.

Lots of Workloads, Lots of Audit Data

No control is possible over the internal telemetry Office 365 apps send back to Microsoft. All we can think about is the switches and controls available in Office 365. One problem that’s immediately apparent is the sheer number of workloads. If you look at the events ingested in the Office 365 audit log, we get:

• Exchange Online.
• SharePoint Online.
• OneDrive for Business.
• Teams.
• Planner.
• Yammer.
• Azure Active Directory.
• Power BI.
• Office 365 eDiscovery.
• Stream.
• Flow.
• Kiazala.
• Dynamics 365.

The wide spectrum of activities encompassed in the list partially explains the 20 to 30 different engineering groups who are interested in the Office events mentioned in the Dutch DPIA report.

Pause the Audit Log

You can stop information flowing into the Office 365 audit log by running the command:

Set-AdminAuditLogConfig -UnifiedAuditLogIngestionEnabled $False

This pauses ingestion into the audit log. However, my belief is that the pre-ingestion events stay in the generating workload. For example, if Teams generates an audit event when a new channel is added to a team, that event is stored somewhere in the Teams Azure service before the Office 365 audit log ingests it. If ingestion is stopped, the Teams events are still in Teams.

Exchange Mailbox Auditing

There’s no documentation about how to stop audit events accumulating in many of the Office 365 workloads. You can disable mailbox auditing in Exchange Online by running the command:

Set-OrganizationConfig AuditDisabled $True

However, this only controls mailbox auditing and has no effect on administrative auditing.

SharePoint Online

The documentation for SharePoint Online auditing describes how SharePoint generates data for the Office 365 audit log, but is unclear what might happen if ingestion to that log is paused. It might be that SharePoint reverts to legacy audit data collection, but there’s no certainly on the matter.

A setting in the SharePoint Online Admin Center allows control over the Office Graph (now the Microsoft Graph) by stopping SharePoint capturing information about how people interact with documents. If you turn this off, Office 365 features that depend on the Graph (like Delve) are a lot less effective.

Azure

Azure Active Directory is critical to Office 365 and every tenant uses a free instance as its directory service. Azure Active Directory retention policies state that data is held for between 7 and 90 days depending on its type and your licenses.

As for the rest of Azure, a lot of information is logged and there doesn’t seem to be much control at a tenant level over what’s stored and how long storage lasts.

Work to Do

This note only scratches the surface of the work that would need to be done by an Office 365 tenant to understand exactly what data flows due to auditing activities to Microsoft and what that data might hold in terms of personal data under GDPR. And then to decide what to do if they wanted to limit some of those flows.

I’m not advocating that any Office 365 tenant should disable audit logging for any workload. Too much value is to be gained from analyzing the content of the Office 365 audit log to understand what happens in the tenant, how users behave and misbehave, and the course of events that might need to be documented (here’s an example).

Your Data. You Own It

One thing’s for sure. Microsoft has some work to do to deliver the commitment made in the Office 365 Trust Center where they say:

With Office 365, it’s your data. You own it. You control it.

What we need is true control for customers over the information gathered from across Office 365 about user activities and stored in Microsoft databases. It will be interesting to see how Microsoft seeks to assuage the issues raised in the Dutch DPIA over the coming months.

For more information about the Office 365 audit log, read Chapter 21 of the Office 365 for IT Pros eBook.

Brightening Label Names

Retention labels are a security and compliance feature. And like most things associated with security and compliance, the names given to labels are usually pretty boring. Names like Keep for Audit or Confidential or even Top Secret hardly stir the blood. The same is true for sensitivity labels, used to assign permissions to files and messages and to protect their content with encryption. For the remainder of this article, I’ll use label to refer to both retention and sensitivity labels.

To be fair, those charged with managing compliance for an Office 365 tenant might not want to excite users. But then again, they might want to add a touch of emphasis to a label, and you can do that with some special characters.

Getting Graphic

All characters you see have a code that tells the computer what to display on screen. Pressing keyboard keys inserts the codes for the most common characters into documents, programs, or anything else you might want to input into on a computer. But beyond the range of “normal” characters, we find special characters. To make the special characters appear on-screen, you must enter key combinations such as pressing the ALT key and then 26 on a numeric keypad to generate the right-arrow character → or ALT and 9781 to generate a snowman character  (here’s a good article to read on the topic).

Apart from the special characters, there’s the code to generate Unicode symbols, such as ALT+128274 needed for a lock  as well as a set of emojis (not that I ever recommend the use of an emoji in an Office 365 label).

Adding a Graphic to a Label Name

To add a special character to an Office 365 label, first create it in a Word document by inputting the necessary key combination. Then go to the Office 365 Security and Compliance Center and create a new label. Input the text part of the name and then cut and paste the symbol from the Word document to end up with something like Figure 1.

Complete the rest of the settings for the new label and save the label. Finally, add the label to a label policy and publish it to the Office 365 workloads.

Using the Graphic Label

After a short period, the label is available to SharePoint Online and OneDrive for Business (it takes longer to publish to Exchange because the Managed Folder Assistant must process mailboxes to make new labels available). As you can see in Figure 2, the label appears along with all the other published labels and can be applied to a document in the same way as any other label.

Exchange Retention Tags

Graphic symbols also work in Exchange retention tags. In Figure 3, we see two labels with graphics appear in OWA. The first (Locked Down) is created as an Office 365 label and published to Exchange, where it shows up as a personal tag. The second (Keep 10,000 days) is an Exchange personal retention tag.

PowerShell

The same technique works with other types of labels such as Azure Information Protection labels. As it should. The only issue I have run into is that the PowerShell console doesn’t like graphic symbols and treats them as non-printing characters. But you can cut and paste values containing graphic characters and use them with PowerShell. For example, to get details of the retention tag shown in Figure 3:

Get-RetentionPolicyTag -Identity "Keep 10,000 days "

Name Type Description
---- ---- -----------
Keep 10,000 days  Personal Managed Content Settings

What Will Microsoft Support Do?

I can’t see that any great harm is caused by using graphic symbols in labels. After all, the symbols are just character codes that computers can process and Office 365 is designed to be multilingual and cope with different character sets (like the way Teams deals with Hebrew and Arabic).

I haven’t tested the willingness of Microsoft Support to accept that symbols can be valid components of label names. It’s possible that some deep and dark flaw is lurking out there. And remember, once you give a name to an Office 365 label, you can’t change it because the label might have been applied to content. That small but very important point is likely the one that will stop people being more colorful with their labels.

To learn more about Office 365 labels, read Chapter 19 of the Office 365 for IT Pros eBook. Chapter 19 also explains how the Exchange Online Managed Folder Assistant works.

Content Searches Find Teams Compliance Items

When someone leaves your company, you might need to preserve their Office 365 data. The steps needed to preserve user information stored in Email, OneDrive, and SharePoint are straightforward, but what about the messages the employee sent using Teams? As it turns out, an Office 365 content search or an Office 365 Data Subject Request (a specialized version of a content search designed to meet a GDPR requirement) is a good way to retrieve information about Teams messages. To complement the items found by a content search, you can gain more insight into the user’s activities in Teams from the Office 365 audit log. For more information on this topic, see the Petri.com article.

And when you’ve read the Petri article, pick up a copy of the Office 365 for IT Pros eBook and get even more technical information about content searches (including DSRs) in Chapter 20 – and maybe even the stuff we’ve written about Teams in Chapter 13.

Using PowerShell To Prove Policies Remove Compliance Records

Some folks raised the point in discussions in the Microsoft Technical Community that there’s no way to know whether a Teams retention policy is working. As you probably know, a retention policy is an automated background process designed to remove items from Teams after a retention period elapses.

This Petri.com article explains that Office 365 generates no reports to help administrators know if their carefully-crafted retention policy is effective, so if you want to know whether items are being removed, you have to check the group mailbox for a team under the control of the policy to see if the Exchange Online Managed Folder Assistant has processed the mailbox and removed some items. Running the Get-MailboxFolderStatistics cmdlet against the folder in the group mailbox where Teams compliance records are stored is key to knowing that items are being removed.

Things get even more complicated because a further synchronization process must occur to replicate any deletions made by the Managed Folder Assistant back to the Azure-based Teams data services, followed by replication down to Teams clients to make the deleted items disappear from user view. The process works, but it can take up to a full week, depending on the load on the Office 365 infrastructure.

For more information about Teams, see Chapter 13 of Office 365 for IT Pros. And then be sure to read up about Office 365 retention policies in Chapter 19.