Counting Fetched Objects is a Hard Computer Problem

All software has its own foibles, something clearly evident in the Microsoft Graph PowerShell SDK. You become accustomed to the little oddities and workaround known issues and all is well. But when the underlying foundation of software causes problems, it can cause a different level of confusion.

Take the question posed by MVP Aleksandar Nikolić on Twitter about how many accounts a Get-MgUser command will return (Figure 1).

The answer is 4 even though the command explicitly requests the return of the top 3 matching objects. Why does this happen? It’s all about Graph pagination and the way it’s implemented in Microsoft Graph PowerShell SDK cmdlets.

Pagination and Page Size

To understand what occurs, run the command with the debug switch to see the actual Graph requests posted by Get-MgUser. The first request is against the Users endpoint and requests the top 2 matching objects.

https://graph.microsoft.com/v1.0/users?$top=2

Note that the Graph request only includes a $top query parameter. This sets the page size of the query and matches the PageSize parameter in the Get-MgUser command. The Top parameter used with Get-MgUser has no significance for the Graph query because it’s purely used to tell PowerShell how many objects to show when the command completes. The use of Top in different contexts is confusing, but few people look behind the scenes to see how the cake is made.

The Graph request respects the page size and fetches 2 objects. However, we asked for 3 objects so some more work is needed to fetch the outstanding item. Microsoft’s Graph documentation says “When more than one query request is required to retrieve all the results, Microsoft Graph returns an @odata.nextLink property in the response that contains a URL to the next page of results.” The skiptoken or nextlink lets the command know that further data remains to be fetched, so it continues to fetch the next page with a request that includes the skiptoken:

https://graph.microsoft.com/v1.0/users?$top=2&$skiptoken=RFNwdAIAAQAAAFI6NWUyZWI1YWIub2ZmaWNlMzY1aXRwcm9zLmNvbV9lbWVhLnRlYW1zLm1zI0VYVCNAUmVkbW9uZEFzc29jaWF0ZXMub25taWNyb3NvZnQuY29tKVVzZXJfYmNmYTZlY2ItY2M1Yi00MzU3LTkwOWMtMWUwZTQ1MDg2NGUzuQAAAAAAAAAAAAA

The follow up request fetches the remaining page containing 2 more objects and completes processing. The person running the command sees 4 objects from the two pages. In effect, the Get-MgUser cmdlet ignores the instruction passes in the Top parameter to only show 3 objects.

The same processing happens with different combinations of page size and objects requested, and it’s the same for other cmdlets too. For instance, the command:

Get-MgGroup-Top 7 -PageSize 3

Returns 9 group objects because 3 page fetches are necessary. It seems odd, and it’s odder still that running Get-MgGroup -Top 7 without specifying a page size will return exactly what we asked for (7 objects), while using a larger page size returns all the objects that can be packed into the page:

Get-MgUser -Top 3 -PageSize 8 | Format-Table Id, DisplayName

Id                                   DisplayName
--                                   -----------
44c0ca9c-d18c-4466-9492-c60c3eb78423 12 Knocksinna (Gmail)
bcfa6ecb-cc5b-4357-909c-1e0e450864e3 Email Channel for Exchange GOMs
da1288d5-e63c-4118-af62-3280823e04e1 GOM Email List for Teams
de67dc4a-4a51-4d86-9ee5-a3400d2c12ff Guest member - Project Condor
3e5a8c92-b9b6-4a45-a174-84ee97e5693f Arthur Smith
63699f2f-a46a-4e99-a068-47a773f9af11 Annie Jopes
7a611306-17d0-4ea0-922e-2924616d54d8 Andy David 
d8afc094-9c9b-4f32-86ee-fadd63b112b2 Aaron Jakes

Frustrating Paging and Display

The typical page size for a Graph request is 100 objects (but this can differ across resources), so it’s unusual to use the Top parameter to request a limited set of objects that’s larger than the default page size. Usually, I bump the page size up to 999 (the maximum) to reduce the number of requests made to fetch large quantities of user or group objects. Using a large page size can significantly affect the performance of queries retrieving large numbers of objects.

The conclusion is that changing the default page size for a Microsoft Graph PowerShell SDK cmdlet overrides the Top parameter. This kind of thing is commonly known as a bug and it’s very frustrating. The Graph requests work perfectly but then something gets in the way of restricting the output to the required number of objects.

Selecting Properties to Use

The same kind of problem arises when Microsoft changes the way Graph requests respond. For instance, this week I was asked why a script I included in an article about reporting Entra ID Managers and their Direct Reports didn’t work. The article dates from April 2023, so neither the text nor the script code is ancient.

Sometime in the intervening period, Microsoft made a change that affected the set of default properties returned by the Get-MgUser cmdlet (probably in the transition to V2.0 of the Microsoft Graph PowerShell SDK). The result meant that some of the properties returned when the script was written are not returned today. The fix is simple: use the Property parameter to specify the properties you expect to use in the script:

[array]$Users = Get-MgUser -Filter "assignedLicenses/`$count ne 0 and userType eq 'Member'" -ConsistencyLevel eventual -CountVariable Records -All -PageSize 999 -Property Id, displayName, userprincipalName, department, city, country

I believe Microsoft made the change to reduce the strain on Graph resources. It’s annoying to be forced to update scripts because of external factors, especially when cmdlets appear to run smoothly and generate unexpected output.

More Handcrafting Required for the Microsoft Graph PowerShell SDK

The issues discussed here make me think that Microsoft should dedicate more engineering resources to perfecting the Microsoft Graph PowerShell SDK instead of creating a new Entra PowerShell module that duplicates Microsoft Graph PowerShell SDK cmdlets. The statement’s been made that the Entra cmdlets are better because they’re “handcrafted,” which I understand means that humans write the code for cmdlets. T

It’s nice that the Entra module gets such attention, but it would be nicer if the Graph PowerShell SDK received more human handcrafting and love to make it more predictable and understandable. Even Entra ID would benefit from that work.

Stay updated with developments across the Microsoft 365 ecosystem by subscribing to the Office 365 for IT Pros eBook. We do the research to make sure that our readers understand the technology.

Report More than 200 Loop Workspaces Requires Fetching Pages of Data

In November 2023, I wrote about a PowerShell script I developed to report the storage consumed by Loop workspaces. The script worked. That is, it worked until a tenant had more than 200 workspaces at which point the script ceased to report details for any more workspaces. This point was recently made to me by a reader after they discovered that the script didn’t produce the desired results in their tenant. Obviously, I didn’t do enough testing to encounter the limit.

Investigation (reading the documentation for the Get-SPOContainer cmdlet) revealed that the cmdlet implements a primitive form of pagination. The default mode is to fetch the first 200 workspaces, but if you know that more workspaces exist, you can add the Paged parameter to the cmdlet.

Odd Pagination to Fetch More Loop Workspaces

APIs implement pagination when they want to limit the amount of data that an app can fetch in one operation. The Graph APIs use pagination for this reason (some of the cmdlets in the Microsoft Graph PowerShell SDK can perform automatic pagination). The idea is that an app fetches the first page, checks to see if a token (pointer) to the next page is present, and if so, the app uses the token to fetch that page. The process continues until the app has fetched all available pages.

In the case of the Get-SPOContainer cmdlet, if more workspace data are available, the 201^st record in the set fetched from SharePoint is a pointer to the next page of (up to) 200 workspaces. Oddly, the information is in the form of a string followed by the actual token. Here’s an example:

Retrieve remaining containers with token: UGFnZWQ9VFJVRSZwX0NyZWF0aW9uRGF0ZVRpbWU9MjAyNDAzMzAlMjAwMCUzYTU5JTNhMjUmcF9JRD0yMDA=

To fetch the next page, run the Get-SPOContainer cmdlet and specify both the Paged and PagingToken parameters. The value passed in the PagingToken parameter is the token extracted from the record referred to above. The code must also remove the record from the set that will eventually be used for reporting purposes because it doesn’t contain any information about a workspace. For example:

$Token = $null
If ($LoopWorkspaces[200]) {
    # Extract the token for the next page of workspace information
    $Token = $LoopWorkSpaces[200].split(":")[1].Trim()
    # Remove the last item in the array because it's the one that contains the token
    $LoopWorkspaces = $LoopWorkspaces[0..199]
}

Looping to Fetch All Pages

A While loop can then fetch successive pages until all workspaces are retrieved. The curious thing is that at the end of the data, Loop outputs a record with the text. “End of containers view.” It’s just odd:

While ($Token) {
    # Loop while we can get a token for the next page of workspaces
    [array]$NextSetofWorkSpaces = Get-SPOContainer -OwningApplicationID a187e399-0c36-4b98-8f04-1edc167a0996 `
      -PagingToken $Token -Paged
    If ($NextSetofWorkSpaces[200]) {
        $Token = $NextSetofWorkSpaces[200].split(":")[1].Trim()
        $NextSetofWorkspaces = $NextSetofWorkspaces[0..199]
    } Else {
        $Token = $Null
        If (($NextSetofWorkSpaces[$NextSetofWorkspaces.count -1]) -eq "End of containers view.") {  
            # Remove the last item in the array because it contains the message "End of containers view."
            $NextSetofWorkspaces = $NextSetofWorkspaces[0..($NextSetofWorkspaces.count -2)]
        }             
    }
    $LoopWorkspaces += $NextSetofWorkspaces
}

Eventually, you have an array of all the Loop workspaces and can report it as in the previous script (Figure 1).

The script with the updated code can be downloaded from GitHub.

Another Example of SharePoint PowerShell Strangeness

I have no idea why the Loop developers thought it was a good idea to implement their unique style of PowerShell pagination in the Get-SPOContainer cmdlet. What they should have done is implement the All cmdlet as done elsewhere, like the Get-SPOSite cmdlet. Supporting easy retrieval of all workspaces together with server-side filtering capability would be more than sufficient for most scenarios and would result in simpler code to develop and maintain.

Last month, I wrote wondering if Microsoft cared about SharePoint PowerShell. This is yet another example of strangeness in SharePoint PowerShell that reinforces my feeling that no one in Microsoft does care.

Support the work of the Office 365 for IT Pros team by subscribing to the Office 365 for IT Pros eBook. Your support pays for the time we need to track, analyze, and document the changing world of Microsoft 365 and Office 365.

Creating a List of Teams

I’ve written a couple of articles about using Microsoft Graph queries with PowerShell to access data that you can’t normally get to with cmdlets. For instance, this example explains how to report the somewhat bizarre email addresses assigned to Teams channels. When a channel is email-enabled, people can post to the channel by sending email to the assigned address, which is a good way to introduce information into Teams. Apart from being posted as new topics in the channel, messages are captured in the channel’s folder in the SharePoint document library belonging to the team.

Limited Data Returned by Design

But as a comment to the article notes, when you use the Invoke-WebRequest cmdlet to send a Graph command to fetch information about the set of Teams in a tenant, the Graph responds with 100 teams. This is what the Graph intends to do because it doesn’t want the response to be too large. And the response is good enough to prove the principle of working with the Graph through PowerShell. However, once you get to production, you probably need to deal with more than 100 teams and need to be able to fetch all the teams in the tenant and process them.

Processing Four Thousand Teams

A few days ago, Mike Tilson posted a note in the Office 365 Facebook group asking if it is “possible to run a report (hopefully via PowerShell) to report on what Microsoft Teams apps (third party apps like Polly) are installed across your environment?” As it happens, the esteemed technical editor for the Office 365 for IT Pros eBook, Vasil Michev, had written a script to report on apps and tabs. I took his script and made (in my mind) some improvements, and I gave Mike a copy of the script.

Mike’s response was that the script worked great in a small environment but had 2,000+ teams (among 4,000-odd groups) to report on in production. The code needed to be upgraded to process larger numbers.

NextLink is the Key

The solution is a thing called a nextlink, available when Microsoft Graph queries have more data to provide to clients than is returned to the original request (server-side paging). A page is the set of data returned for a Graph call and several pages might need to be retrieved to fetch the full set of objects you want to process The documentation says “When a result set spans multiple pages, Microsoft Graph returns an @odata.nextLink property in the response that contains a URL to the next page of results.” In effect, the nextlink tells the Graph the next set of data to return if an application wishes to request it following its first call. To be sure that you get all data, you need to fetch it page by page until the nextlink is null. This is called pagination.

If you use the Graph Explorer to play with Microsoft Graph queries, you’ll see that a nextlink turns up in calls like “all groups in my organization.” You know it’s a nextlink because it includes the term “skiptoken” as in https://graph.microsoft.com/v1.0/groups?$skiptoken=X%274453707402 (a real nextlink is much longer). Microsoft Graph queries can generate nextlinks after fetching less than 100 items. For instance, the default number of folders retrieved from a mailbox is 10.

Solving the Problem

The uprated code:

• Makes the original call to fetch teams and stores the object identifier and display name for each team in a hashtable. A hashtable is suitable when you only need to store two properties. If you need to store more, create and populate a generic list instead.
• Checks if a nextlink is returned.
• If a nextlink is found, make another call to the Graph to fetch another set (page) of data and store those items in the hashtable. This call uses the nextlink to tell the Graph where to start retrieving data.
• Continue until the nextlink is null, meaning that no more data remains to be fetched.

Here’s the code, based on an original solution created by Mike Tilson.

$Uri = "https://graph.microsoft.com/V1.0/groups?`$filter=resourceProvisioningOptions/Any(x:x eq 'Team')"
[array]$Teams = Invoke-WebRequest -Method GET -Uri $Uri -ContentType "application/json" -Headers $Headers | ConvertFrom-Json
If ($Teams.Value.Count -eq 0) { Write-Host "No Teams found - exiting!"; break }
$Teams.Value.ForEach( {
   $TeamsHash.Add($_.Id, $_.DisplayName) } )
$NextLink = $Teams.'@Odata.NextLink'
While ($NextLink -ne $Null) {
   $Teams = Invoke-WebRequest -Method GET -Uri $NextLink -ContentType $ctype -Headers $headers | ConvertFrom-Json
   $Teams.Value.ForEach( {
      $TeamsHash.Add($_.Id, $_.DisplayName) } )
   $NextLink = $Teams.'@odata.NextLink' }

Get-Team Works Too

Another pragmatic solution to the problem of how to fetch all teams in a tenant is to use the Get-Team cmdlet. Using Get-Team is much slower than the code listed above (minutes instead of seconds), but the cmdlet handles the paging for you.

After fetching the set of teams, we can begin to process each team to discover what apps and tabs are installed in it. The Graph calls in the script to fetch channels, tabs, and apps don’t use paging. A team can have up to 200 channels, so I guess the call to fetch channels might need to change to handle such a well-endowed (and possibly confusing) team.

Still Work to Do

Although we can now fetch all the teams in a tenant, things aren’t perfect yet. I found a problem processing archived teams, where the attempt to retrieve app information fails. I can’t see how to identify an archived team from the information returned, so more research is needed. (Update: check the isArchived property in team settings to find archived teams).

Another issue is that the auth token expires after an hour and stops the script. A refresh token is needed at this point. (Update: the latest version of the Teams and Groups Report script illustrates how to renew an access token).

If you’d like to improve the code and make it even better, you can get the script from GitHub. In the meantime, enjoy using Microsoft Graph queries to process data with PowerShell.

Need help understand how to use PowerShell to manage Office 365 Groups and Teams? The Office 365 for IT Pros eBook contains a ton of examples to help you get going.