A custom SharePoint newsletter service

In a customer project, we had the task to implement a custom newsletter service for SharePoint. You may wonder: Why are you not using the news digest feature?
Well, the answer to that was that it didn’t match the requirements of the customer.

Requirements

• The application should be able to send emails that contain the content of a news post (except images).
• The application should transform relative links into absolute links.
• The application should remove unnecessary HTML fragments like webpart elements, no value attributes, custom SharePoint attributes, class names, empty <div> elements, empty <span> elements.
• The application should support a basic template mechanism to define the layout for the mail.
• The application should run on a single server that is not exposed to the outside.
• The application should be run regularly with the Windows TaskScheduler to fetch news.
• The application should only process news once.
• The application should contain all dependencies without the need to install additional software.

Technologies

For the implementation of our Azure AD application, we decided to use .NET Core (C#). It is like SharePoint a fundamental part of the Microsoft ecosystem and therefore, there is a large and active developer community that provides many good libraries (like PnPCore) around it.

Our application uses the following 3rd party packages:

• AngleSharp (for working with the HTML DOM)
• PnP.Core (for accessing the SharePoint/Microsoft Graph API)
• Serilog (for logging)

Getting busy

The first step was to figure out a way to retrieve the newsletter list. With PnP.Core this was pretty straightforward (simplified code):

With this small piece of code, we were able to get the correct SharePoint list instance which contained all of our newsletter items. But as always problems arose. We questioned ourselves:

What should we do when the list grows bigger over time? Fetching all the items again and again? Even if we already processed them?

Sounds like a really bad plan. Also because SharePoint (Online) will use throttling mechanisms when you are sending too many requests.
In the end, we decided to use change queries because SharePoint keeps track of changes made to sites, lists, list items, and more. To get an idea of how that works you can head over here.
You will get a change token when sending a request to the SharePoint REST API. When you store that token you can use it for the next request to get only the changed items. We stored it in a simple text file because it does not contain any sensitive information.
After that our application was able to receive only changed items. Great! The next requirement was to only process a newsletter once. For that purpose, we saved the unique list item id (after processing it) in a text file and called it a day!

Processing a single list item

To be able to send a newsletter mail we first needed to process the received list item and request some more information. We needed the following data:

• UniqueId
• Title
• Author
• Recipients (custom field)
• AsTestMail (custom field)
• PublishDate
• Topic (custom field, taxonomy)
• Content (CanvasContent1)
• Url
• AttachmentUrl (custom field)

The properties Author, UniqueId, Title, AsTestMail, PublishDate, Url, and AttachmentUrl were already in the right format and did not require any custom processing. The CanvasContent1 property will get its own section because it was more complex to deal with it.

Recipients

The initial value we received was a IEnumerable<> of a type implementing the interface IFieldUserValue. Because the IFieldUserValue type did not had all informations we needed we retrieved the IGraphUser representation for it:

Topic

The initial value we received here was a (unique) TermId. Because we needed the string representation/real value we were forced to look it up from the term store:

Transforming ‘CanvasContent1’

The CanvasContent1 property contains the HTML body of the list item. Unfortunately, it’s pretty cluttered with elements that we don’t want or can’t display in our mail body. So the next step was to unclutter it. Here we used AngleSharp because it made working with the DOM a breeze. You should check it out!

Notice: Some of the snippets below could be optimized even more (for example by the usage of LINQ instead of a for-each loop), but they are currently running exactly like that “battle-proven” in productive.

Unprocessed CanvasContent1 with Text + YouTube Webpart

Loading CanvasContent1

To work with AngleSharp we need to parse the plain string into an IHtmlDocument:

Removing webparts

Webpart elements are pretty easy to identify. They are always represented by a html <div> and have the custom html attribute named data-sp-webpart. You can find and replace them like that:

Removing images

This requirement is even easier to implement. We just need to find all html <img> elements and remove them:

Removing attributes without value

For this we are getting all elements of the document and searching for attributes with empty value. With a little bit of our beloved LINQ this is also pretty easy to do:

Removing custom SharePoint attributes

All custom SharePoint attribute names start with data-sp. So we used that as an identifier.

Removing class attributes

Due to the fact that we have access to all document elements we are simply searching for the html class attribute.

Removing empty div and span elements

Because of the pre processing above we can simply replace <span> and <div> elements in the document body.

Transforming relative links to absolute links

The last requirement for the newsletter body. Finally! And it’s a pretty common task. Just search all <a> tags, check their value, and if it’s a relative URL replace it.

Bringing it all together + final result

Now we should have a result which we can send without problems via mail. Let’s bring all parts together!

And the final result is:

The processed CanvasContent1

The template “engine”

After processing the CanvasContent1 we are now ready to craft our mail. To be flexible we decided to implement a simple template “engine”. The template is loaded from a local HTML file and contains different placeholders. These placeholders are then replaced with the actual values. The available placeholders are:

• %RECIPIENTS%
• %PUBLISH_DATE%
• %TOPIC%
• %TITLE%
• %CONTENT%
• %AUTHOR%
• %ATTACHMENT_URL%
• %NEWSLETTER_URL%

I think the placeholder names are speaking for themselves. The template we are currently using in productive is this:

Final step — Sending the mail

Unfortunately PnP.Core does not offer a (direct) way to send an email. But it does allows us to instantiate a GraphServiceClient which can do the job! We only need to use the PnP authentication provider like this:

Afterwards we must create an instance of the Message class like this:

Now we are authenticated, have our Message instance, and are ready to send! As the last step just retrieve the user who should send the mail and call the SendMail() method:

And this is the final result from the view of the recipient:

Final email

I hope you liked my blog post about our adventure to build a custom SharePoint newsletter and could gather some insights from it 😊.