Reducing your Office 365 attack surface — Part 2: Monitoring your environment

A little more than 3 months ago, I published the first part of this series on how to harden your Office 365 environment. I’ve received a lot of very positive feedback and quite some people reached out while waiting for Part 2. I’m happy to present this second part.

Before diving into the details, I want to note a few developments:

• Office 365 has been renamed to Microsoft 365 — I decided to keep the title anyway for consistency. However, I’ll use the new name going forward in this post.
• Don’t expect miracles. The end-user side of Microsoft 365 is very polished, but the admin side is buggy from time to time and is far from the “end-user experience”.
• As mentioned in the previous post, Microsoft is constantly updating and improving — be aware that some UI elements might have changed. Also, I expect that the provided functionality for monitoring will improve rapidly. If you’re reading this article more than 2 months after the initial publish date (May 2020), keep in mind that there is a significant chance that not everything will be applicable.

Since the whole M365 suite is still in heavy development, there are a number of different solutions available for security monitoring. Each of them covers different parts and each of them has its own pros and cons. The following table is a comparison of the pros, cons and the current pricing.

Next to the “built-in” Microsoft solutions, there is quite a market of 3rd party vendors that offer such solutions. So the above table is not a complete comparison, but just the Microsoft top 3.

Unified Audit Logs

First topic to discuss are the monitoring features which are included in all (basic) Microsoft 365 subscriptions. This is a short list. The only source of log data which is useful is the so-called “unified audit log”. Luckily, we can already do some useful stuff with it.

All the recommendation in this blog post are purely my personal recommendations, use at your own risk :-) Each recommendation is aimed at getting insight in unusual behavior, which might be an attack. I’ve incorporated all my experience with abusing M365 environments in red teams to make the recommendation as tangible as possible. Also, there is no one-size-fits-all, you really need to consider if the recommendation here fits the way your organization uses M365.

First step is to enable this log, if you haven’t done so already. I highly recommend everybody to enable the audit logs anyway. Even if you’re not going to set up any alerting . This may be a very useful source of evidence in case of a breach (except the Audit/Sign-in Logs in Azure, but these aren’t detailed enough for investigating a breach). You can read here how to enable unified audit logging.

Since audit logging is meant for audit purposes and not for monitoring purposes, we have to access some compromises here. As described on this page, some event may take up to 24 hours to appear in the audit log. In practice, it seems to be quite a bit better than that. Still, delays of around 2–3 hours are fairly common. But then again, it’s an audit log. And it’s free.

Creating monitoring rules is something which is very organization specific and requires a lot of tweaking. However, I’ve a couple of rules which might be more universally applicable. Since this is not a point-and-click tutorial, I’m going to assume that you can find your way in setting these up, based on the guidelines below.

Alerts on using admin accounts. I find it very useful to know when someone uses their Microsoft 365 admin account to login. Every time this happens at FalconForce, I check with the relevant admins (out-of-band) if they’ve indeed used their admin account. And the same happens vice versa. We’ve agreed to check-up on each other. This rule is only applicable for SMEs though, if your company is so big that admins are logging in every day for maintenance — this is not going to work for you. To set up this rule, go to “Audit log search” under the “Search” menu option in the “Security & Compliance” dashboard. Make a new “alert policy” and set the settings as indicated in the next screenshot.

Alert settings for admin logins.

There are two catches here. First, you have to select each account you want to alert on individually. So every time you add a new admin account, you have to remember to modify this rule. There is no support for wildcards. Second, consider sending the alerts to a mailbox which is not part of your M365 environment. So if an attacker has admin access to your environment, he won’t be able to manipulate the alerts before they come in.

Malware on Sharepoint/OneDrive. Another rule which works more or less universally is the detection of known malware. I haven’t investigated how good this detection is and you don’t need to rely on it for malware detection, but if it detects malware — you need to know and you need to act. I won’t include a screenshot for this one, but you want to alert on the activity “Detected malware in file”. Leave the “users” field empty to make the rule applicable to all users.

From here on, the rules get more noisy and might or might not work for your environment. Depending on your environment, you can consider and test the effectiveness of alerting on the following activities.

• Created anonymous link — A link that provides everyone access to the resource. The noise one this one depends on if your organization allows and uses anonymous links.
• Site administration activities — Depending on how many site admins you have and how often you perform “site admin actions” you can consider to enable alerting some or all of the activities. Especially the following sub-activities for site administration are worth considering: “Added geo location admin”, “Changed device access policy”, “Changed network access policy”, “Created send to policy”, “Changed sharing policy”.
• User administration activities — Again depending on the size of your organization and how often you onboard new users. Especially the following sub-activities: “Added user” (especially for small organizations), “Changed/Reset user password” (especially if you use passwordless logins).
• Directory administration activities — If these activities are performed at all, should usually be done by admin. Consider alerting, preferably on non-admin users.
• Published app (Teams) — If new apps are published in Teams, it might be malicious. Someone publishing a new custom Teams app is quite rare. Sidenote, I haven’t tested this, but I presume this applies only to “company specific” apps and not for public apps.

Note that this list is only limited to the functionality of M365 which we use. For a lot of applications such as PowerBI, Shifts, Stream, PowerApps, etc. I can’t make any proper recommendation as we’re not using it.

Finally, a lot of the activities would be useful to alert on if you would be able to fine-tune more parameters. However, because UAL doesn’t provide this feature, it’s not useful to alert on it.

Inside the “Security & Compliance” dashboard, there is one other place where you can create alerts. Go to “Alerts” => “Alert policies”. This option allows you to create some of the alerts as described before, but with just a little bit more fine-grained alerting rules. Moreover, it supports some “higher level” rules, which you cannot define in UAL. So I recommend to create a few more alerts here. Click on “New alert policy” and try the following alert. Again, this is not a one-fits-all but should inspire you on how to get started.

• Detected malware in an email message. Both inbound & outbound.
• Malicious email detected.
• Created mail forward/redirect rule.

You can set the other settings such as severity and category.

Microsoft Cloud App Security (MCAS)

My view on MCAS is a bit mixed. It is definitely a big step up from the Unified Audit Logs. The rule-engine is more detailed and allows for creating more sophisticated rules, with zero code. It also has some intelligence built-in and can detect things like “activity from infrequent country”, “impossible travel”, “mass downloads” and way more. But at the same time the delays seems to be even longer her compared to Unified Audit Logging, some events seem not recorded and still the type of queries you can build is sometimes limited. Then again, just € 3 per user for all these features is fairly cheap.

Since this blog is not meant as review or recommendation of the products themselves, I just leave it at that. Just get a free 1 month trial and see if it works for you. If you can’t easily get a trial for whatever reason, I’ve compiled a list of all the connectors & rules which are available in MCAS. You can download it here.

I’m skipping the part on how you’re supposed to setup MCAS, there’s already enough documentation on that. Assuming you’ve installed everything, these are the rules I highly recommend to enable/create:

• Malicious OAuth app consent
• Leaked credentials
• Malware detected
• Suspicious inbox manipulation rule
• Suspicious inbox forwarding
• Activity from anonymous IP addresses
• Activity from suspicious IP addresses
• Impossible travel
• Unusual impersonated activity (by user)
• Unusual administrative activity (by user)
• Ransomware activity
• Misleading publisher names for OAuth
• Activities from suspicious user agents*

*The default list of “suspicious user agents” is a fairly limited blacklist. This can be improved and turned into a whitelist if using Sentinel.

For the following rules, I suggest a bit more careful thought. Depending on your organization type, they might or might not be effective:

• Risky sign-in — only with Azure AD integration and Azure AD P2 license.
• Unusual file deletion — some attacks try to delete as many files as possible from your SharePoint/OneDrive and ask you for ransom. This rule can help you to detect such deletion and get your files back (within 30 days).
• Activity from infrequent country — only useful if (business) travel abroad by employees is very limited.
• Logon from outdated browser/OS — only useful if your employees manage their own machines used to log into M365.
• Externally shared source code — if source code is part of your IP.
• External user added in Teams — only if collaborating with externals in Teams is rare/disabled.
• Mass deletion of Teams — you can set after how many Teams deletions this rule should trigger
• Stale externally shared files — only useful for the really small organizations, where the M365 admin can check with file-owner

MCAS can monitor even more cloud applications. You can connect it to other cloud applications, but also ingest logs of a lot (on-prem) appliances and build similar rules. I have no experience with the latter and hence cannot comment on it’s usefulness.

On top of all the default rules, here are some custom rules I recommend to create:

• Administrative activity from a unknown IP address — admin activity either from non-corporate IP or non-whitelisted IP. This can also be the external IP of the M365 admin or some other trusted IPs. There is a catch, you need to add some more filters to prevent false positives triggered by internal Microsoft activities on your environment. There is not a lot of documentation on the things Microsoft is doing in the backend for your M365 environment, but quite some events are logged — we don’t want those events to trigger alerts. You can ignore these activities by excluding IP addresses which are tagged as Azure or O365. Excluding Azure is a vulnerability as potentially someone with an Azure VM might be tagged as Azure — but it’s worth the trade-off against all the false-positives that would otherwise be triggered.

The settings for “Administrative activities from untrusted IPs”

• Activity outside of home country — If your employees don’t travel regularly, you can use this rule to alert on any activity from abroad. Again, there are some catches for filtering out false-positives. You need to exclude the “TIMailData-Inline” activity. You need to exclude the IP addresses tagged as in the second screenshot. And finally, you need to exclude all events where IP address is empty. See screenshots below.

• Changes to Exchange transport rules — Transport rules in Exchange are sometimes abused to forward emails without setting mail rules. This query allows alerting for that. Should be very accurate as transport rules are very rarely touched, once set up.

• External user activity — this rules is only useful if you’ve completely disabled external user activity on your M365. Once set up, it alerts on any external activity in your environment.

Modified audit log configuration — if the (unified) audit log configuration or its retention period is modified, you might want to be notified. This activity is rare and suspicious enough for your attention.

Changes in security policy/rules — If you have Office ATP, you can create some advanced security policies. Any changes to these policy should be verified. The following events can be monitored.

Azure Sentinel

The most advanced solution for monitoring in the Microsoft Suite is Azure Sentinel. Discussing all the possibilities of Sentinel is way beyond this research, I’ll only cover what’s relevant for monitoring M365. If you haven’t heard of Sentinel before; it’s Microsoft’s “cloud native” SIEM, including some awesome automation. As an added bonus, Sentinel is being picked up by the IT security community and a lot of content is being publicly shared for Sentinel which you can re-use.

However, the product is still fairly young and it doesn’t yet feel completely “ready”. During this research, I got some unexpected behavior. So when you plan to implement Sentinel, make sure to calculate in some extra slack time for unexpected weirdness in Azure/Sentinel.

For example, it took around 72 hours to get the Sign-In logs into Log Analytics / Sentinel instead of the 15 minutes or so, as documented. Similarly, when building some “Logic Apps” used for automated responses to incidents, the Logic App at some point broke after a save, by something which looked like a paring bug in the Logic Apps engine. Again, I’m not saying to stay away because of these bugs, but just be prepared.

To have proper coverage for your monitoring solution, you need login data. To get this data into Sentinel the easy way, you need to have an Azure AD P2 license. There are different bundles that contain AAD P2, but if you don’t have it yet, the separate license is €7,90 per user per month.

If this is too much for you, there is a cheaper solution. You can build something to get the login events from the Unified Audit Log to Sentinel with custom apps and connectors. However, in my opinion, this is a far more inferior solution. You get the logs with a significantly bigger delay (up to 24h according to the documentation), need to do custom log parsing, need to invest time to build and deploy the solution, need to invest in keeping the solution running, etc. Just get the P2 license, unless cost is really a big issue for you and time and speed are not.

I’m not going to get into the details on how to connect Sentinel to your M365 environment. However, I do recommend to connect at least the following data connectors:

• Office 365 — For SharePoint and Exchange log data
• Microsoft Cloud App Security — For integrating MCAS incidents in your Sentinel flow. Only applicable if you have an MCAS license
• Azure Active Directory — For Sign-In logs and AuditLogs of Azure AD
• Azure Active Directory Identity Protection — For “risky events” in Azure AD

For each data connector, make sure to browse the “recommended workbooks” and “relevant analytic templates” (see next screenshot). These contain great pre-made content to get insights in your environment.

Again, I will keep the same structure as above. I’ll highlight the existing queries that I recommend to use and propose some custom queries.

If you open up “Analytics” and go to “Rule Templates”, you’ll find a lot of Microsoft proposed rules (See next screenshot). I highly recommend to consider each of them, as every environment has different needs and requirements.

Having said that, I think the following M365 queries are relevant for almost any kind of organization:

• Create incidents based on Azure Active Directory Identity Protection
• Create incidents based on MCAS alerts (if applicable)
• Office policy tampering
• Exchange AuditLog disabled
• MFA disabled for a user (ALWAYS MFA!!)

For certain types of organizations (small, always working from the office, no remote working, etc.), the following additional list is also useful:

• SharePointFileOperation via devices with previously unseen user agent
• Anomalous sign-in location by user account and authentication application
• SharePointFileOperation via previously unseen IPs
• Multiple users email forwarded to same destination
• Attempts to sign-in to disabled accounts
• Rare and potentially high-risk Office operations

Again I want to re-iterate. This is not a one-size-fits-all solution. Consider which of those rules works in your organization.

The great thing about these queries is that you can easily tweak the parameters to fit your needs. If you want an alert on previously unseen IPs, but it gives you too much noise due to mobile connections, you can filter based on ISP. Or you can exclude unseen IPs from your home country. You have all the flexibility in these queries. As a matter of fact, your security team should be constantly working on tweaking the parameters to find a suitable balance. Watch out for over-fitting though.

On top of these built-in rules/workbooks/notebooks, the Sentinel GitHub repository contains good queries and hunts for Office and Azure AD. Spend some time to look through all of them and see if they can help in your environment. The following is a list of my favorites:

• 3 types of mail-forwarding 1 2 3
• Legacy authentication Attempt (you should disable Legacy authentication if possible)
• New admin activities

And on top of these, GitHub contains a lot of Kusto Query Language (KQL) content (of varying quality).

The final addition I have to all of these monitoring madness is a query and a building block that we at FalconForce have been working on. I believe that a serious IoC is a login attempt which is aborted at the MFA request. Depending on how you’ve setup MFA, the end-user might or might not be alerted. If users use the Microsoft Authenticator Push notifications, SMS or Voice MFA, they’ll get that. If they just type over the 6 digit TOTP, there is no alert on a missed MFA.

To detect that, I’ve written a KQL query which detects sessions in which login attempts are aborted/not finished when MFA is requested. The query can be found here. Of course, this is noisy as this will happen with normal usage as well. That’s why the follow up of such an alert should be an automatic message to the user to either confirm or dispute the aborted MFA. Confirmation obviously requires MFA. If confirmed, the incident is automatically closed in Sentinel. If disputed or no response, the ticket gets escalated to high risk.

We’ve developed a proof-of-concept for Teams, but it’s still too early to release. We’ll release this building block with Teams (and perhaps Slack) support and MFA confirmation as soon as it’s ready.

What else?

Obviously, this was an information overload. It’s meant as a reference guide on the various options for security monitoring for Microsoft 365, but then again it’s far from complete. There are so many awesome things to build in this area that the possibilities are endless.

Feel free to reach out if you have any questions or comments. Happy to have a discussion and brainstorm new ideas.