How to build your own (meaningful) Microsoft 365 Copilot usage reporting system

This is the final* in my 3-part series about how to build your own meaningful Microsoft 365 Copilot (referred to as “Copilot” for the rest of this article) usage reporting.

Unfortunately, what I keep hearing from customers is that they’ve not found the existing data provided to be overly useful. Given that Copilot has been available for over a year, this isn’t great as it makes it challenging for those organisations to drive further adoption.

But if you want to understand more about that, read my original piece that compares the four different methods available to get Copilot usage.

Then, read the next piece about how and why you might need to hijack Viva Insights reporting data to provide identifiable Microsoft 365 Copilot usage metrics – as the native reporting is too high level for most organisations to find actually useful.

Finally, we come to this piece – where I’m making my solution available to you, so that you can make something meaningful for your organisation. And if you’re really game, you can mash a couple of them together. 🙂

The Challenge

I don’t need to dwell too much here, but to quickly recap (in case you haven’t bothered to read the two previous articles):

• Data provided via the admin centre and by extension from the Microsoft 365 usage reports via Microsoft Graph show last used data for Copilot in some of the Microsoft 365 apps/services.
• Data provided via Viva Insights (and by extension Copilot usage report) is anonymised and grouped by managers, and only those who have 10 direct reports or more.
• Data provided by the Microsoft 365 Audit Log is a rich vein of information, shows far too much information for this purpose, and cannot be scoped or filtered to just Copilot.

So, we are left with our fourth and (current) solution to this problem…

The Solution

Buried inside the Microsoft Graph API are some endpoints that show Copilot usage on a user level.

(NOTE: These endpoints are currently in beta, so use at your own peril as it may change at any point.)

The resource type we’re looking at here is called “aiInteraction” and shows us Copilot interactions across Microsoft 365. The apps/services I’ve found it to show are:

• Copilot Chat (formerly known as Business Chat)
• Excel
• Forms
• Loop
• Planner
• OneNote
• Outlook
• PowerPoint
• SharePoint
• Stream
• Teams
• Whiteboard
• Word

Disappointingly, Viva Engage still doesn’t show up.

While I have been expecting to see that show up, I wouldn’t however expect to see any data for Power BI, Power Automate, Power Apps, or any of the other Copilot versions out there as they are not “Microsoft 365 Copilot”.

Also disappointing was that while my original method using the Audit Log method allowed me see a more granular level of Teams usage (i.e. whether it was in a chat, channel, or meeting) – unfortunately this method does not show any of those.

Skipping some of the tech talk, here’s the kind of outputs you can expect to see with my solution.

Original Design

The original design was built at the start of this year, but unfortunately I only had an organisation with 88 licensed users to test it on, and they had many Copilot licenses for long – so there wasn’t a great deal of data to display.

Updated Design

Thankfully I was able to work with fellow MVP Lindsay Shelton who was able to deploy it into her organisation that had 950 licensed Copilot users, and as such we had a lot more data to work with.

However, her organisation has a retention policy that Teams chats after 30 days which we discovered also deletes Copilot interactions as well. There is a M365 roadmap item to separate these, however we still had some great data to work with despite a reduced history.

Additionally, Lindsay was able to make some improvements to the report, which I am very grateful for!

The team in Lindsay’s organisation found this data to be MUCH more useful than what was provided natively, and it also highlighted some inconsistencies with the native reporting data as well.

Deploying the Solution

There are several components involved here:

• Entra ID app registration to interact with Microsoft Graph
• Power Automate workflows to capture Copilot interactions for licensed users
• Dataverse tables to store the data
• Power BI report to display the data

I’ve made the solution files available from my GitHub repository for you to deploy in your own organisation, but there’s a few things the technical folks will need to know in order to make it work.

Like many Hollywood movies these days where you go along thinking you’re seeing the final in the series, it turns out it’s part 1.

* This is one of those cases, as I needed to separate out the solution overview from the technical bits behind the scenes. So this is part 1, and….

 

 

 

 

Oh wait… here it is… the thrilling conclusion to the series!

 

 

 

 

---

Also published on Medium.