Activity Logging (aka Read Audit) - why is it important for a Dynamics developer

Yesterday’s just a memory, tomorrow is never what it’s supposed to be. — Bob Dylan

As GDPR came to life earlier this year, anyone working with customer data should have asked: how do we handle complex audit scenarios? Like:

• Who has viewed sensitive data ?
• Has anyone exported a given data set from the system?

These questions should have been security concerns long before, along with the ones like “Who published something in production?”, “When was the last bulk delete triggered?”. But GDPR is a good trigger.

The big news is that since June 2018, auditing from the Dynamics SDK layer can be viewed and accessed in a single spot, out of the box, without customisation! Sounds too good? The magic words are “Activity Logging”!

https://protection.office.com

Activity Logging has been around since 2016 for Office 365. It is a neat feature, which gathers audit records from the Microsoft stack into a single location.

The official documentation can be found HERE, but these are the most important details:

• audit needs to be turned on (this will change) for an Office 365 organisation, and it can take up to 24 hours to fully start the service
• alternatively, it can be reached from the general Office 365 admin portal (Reports / Security&Compliance) — though it is not yet functioning
• wide range of events is logged from the MS stack: SharePoint, Exchange Online, PowerBI, Azure AD, MS Teams, Dynamics, MS Stream, …
• audit records are generated asynchronously: either 30 minutes or 24 hours after the event (see details on link)
• there are services, for which you need to additionally enable audit (eg PowerBI)
• audit logs can be exported into CSV file(s)
• max 5.000 event logs are displayed at once, and max 50.000 records can be exported (corresponding to applied filters)
• some record types contain additional details about the activity, but the detail information is truncated at 3.060 characters
• audit logs are kept for 90 days

Dynamics in spotlight

First of all, activity logging for Dynamics has its own documentation HERE, and the most important information is that it only works with PRODUCTION instances!

In a nutshell, you need to enable it in System Settings within Dynamics, and then in protection.office.com to get the basic service. To also be able to log record access, it needs to be configured on a one-by-one basis on the entities.

What do we get, aka the schema

The official documentation is not in line with the real audit records, so please find my completed and corrected schema below. To translate the enumerations, THIS PAGE gives a good start.

Random thoughts

Noise

As it stands, the audit generates a lot of “noise”. The most outstanding for me is the "UserLoggedIn” event, which takes most of my log table, even on a trial instance with just a single user.

Normal day-to-day activity will add up quickly on a middle sized tenant, so get comfortable using the filters!

Microsoft promises to revisit the design and filter out logs that are not deemed useful. On the other hand, you could use a capability with insights (such as Office 365 Advanced Security Management) to detect suspicious activities automatically.

Dynamics experience

As for Dynamics, the end user experience is imperfect: to reach the logs you have to navigate out of your instance. Also reversed, some of the logs are not directly linked to the related records — I know, IDs, but remember the end users! It can end up being quite tedious — you need to check, what information is held in the records and which were retrieved by a random RetrieveMultiple request.

I have additional concerns. While logging record access, only the filter is audited and we don’t really know, which exact fields were exposed to the user. The same is true for exporting. Similarly, the audit only tells us, when a user imports a solution, but not the name of it, and not even it’s ID.

Retention

The 90 days retention is short if audit is used to identify data breaches. Based on the 2018 annual IBM report, an average of 196 days are spent by companies before breach detection (LINK). And that is the average.

Although, to comply with GDPR, there is no set minimum period defined, but “ the processor (…) shall make the record [of processing activities] available (…) on request”. Therefore, it is strongly advised to periodically move these log records to a storage tool or service.

Conclusion

I am amazed and excited already that the data is there without any customisation! We have the power to read it, analyse it, report on it, or use it — and not just to comply with GDPR and data security policies.

As the “big data” buzzword is still around, and the era of insight services with AI has come, we could do a lot more, eg discover and validate various user adoption patterns.

So, what are your thoughts, how do you use or plan to use Activity Logging? Do you have any special concerns? What additional features would you be happy to see there?