Integration of Iterable into Sololearn: Elevating User Experience
In this article, we explore how we integrated Iterable into Sololearn’s systems. First, we will overview our service structure and how we communicate. Then, we will dive into the solution we carefully selected to meet our current business needs. To show you how our solution can grow and be tracked, we will also share some practical code examples and lessons we learned along the way.
Introduction:
Sololearn, the world’s largest code learning platform and community, selected Iterable, a robust marketing automation and customer engagement platform, to enhance its marketing strategies and user experience. Iterable was adopted to help Sololearn manage campaigns, personalize messages, and deliver relevant content to users through various channels. Sololearn’s integration strategy was designed to be comprehensive and efficient, aiming to avoid extra costs or data disruption. This move is part of Sololearn’s commitment to continuous improvement and to delivering engaging, high quality content.
Section 1: Context and proposed solution
1.1 Our Current Service Setup
Our backend services setup is currently a mix of a small, tightly-packed core and a large network of smaller, connected services, or microservices. These microservices talk to each other through a series of HTTP requests and domain events. This setup primarily involves several services associated with specific bounded contexts, which include User Information, Authentication, Subscriptions, Hearts, Leagues, and Onboarding. To get a better idea of this setup, take a look at the diagram below.
1.2 The Solution We Selected
When searching for an integration solution that met Sololearn’s changing needs, we decided to go with an orchestrator pattern. This pattern is key to our strategy, helping us cut down on unnecessary database calls while also improving data across our systems. This optimization reduces the number of calls from N to only one, which not only makes things more efficient but also speeds up the addition of new domain event handlers. This approach, backed up by code examples we will provide, ensures our solution can be reused and quickly adapted to new requirements.
We use a strict “fail-first” validation approach, thoroughly checking event data for consistency before it is sent on. This careful checking process ensures that only data that meets our exact needs goes through the orchestrator to the subsequent services, protecting data integrity and system coherence.
The integration of new third-party services is made easy and seamless in our organization, thanks to an efficient subscription model and the use of a well-understood pattern. Any developer within our team can effortlessly incorporate a new service and subscribe to the pertinent event queues, while also having the ability to filter by the event types that serve their interest.
Our existing infrastructure, which includes Service Bus and containerization, aligns perfectly with our integration goals. Its scalability and adaptability match our vision for future growth and changing needs. This compatibility makes the integration process more streamlined and strengthens our ability to scale.
The diagram below gives a detailed view of how these new services integrate smoothly with our existing system.
For Iterable’s specific needs, as mentioned earlier, we must handle a special situation where user data is updated before events get to Iterable. Importantly, most events require both, updates to user data and the sending of custom events.
Section 2: Services Implementation
When it comes to integration, the Orchestrator Service is key to making sure events and data flow smoothly across Sololearn’s complex system. Below, you will see the important parts and code snippets that make this service work.
2.1 Orchestrator Service implementation code samples
The first step in implementing the Orchestrator Service is to register your domain event handler. This is accomplished using the following code:
services.AddSingleton<IMessageHandler<UserAuthenticationEvent>, UserAuthenticationEventHandler>();Next, define the validation logic for the new Orchestrator Event object, which involves setting rules for various parameters such as the user’s email, signup date, and account status, among others.
public class UserAuthenticationValidator: OrchestratorEventValidator
{
public UserAuthenticationValidator() : base()
{
RuleFor(i => i.Payload.User.Email).NotEmpty().EmailAddress();
RuleFor(i => i.Payload.User.SignupAt).NotEmpty().LessThan(DateTime.Now);
RuleFor(i => i.Payload.User.IsAccountActive).NotEmpty();
RuleFor(i => i.Payload.Device!.Id).NotNull().NotEqual(0);
RuleFor(i => i.Payload.Device!.IpAddress).NotNull().SetValidator(i => new IpAddressValidator());
RuleFor(i => i.Payload.Subscription).NotNull();
RuleFor(i => i.Payload.Subscription!.MonetizationStatus).Equal(MonetizationStatusType.Free);
RuleFor(i => i.Payload.Language).NotNull();
RuleFor(i => i.Payload.Language!.Locale).IsInEnum().NotNull();
RuleFor(i => i.Payload.Login).NotNull();
RuleFor(i => i.Payload.Login!.Provider).IsInEnum().NotNull();
}
}After defining the validation logic, declare the new Orchestrator Event object, which involves data transformation and enrichment.
public class UserAuthenticationEventHandler : OrchestratorHanlder<UserAuthenticationEvent>
{
public override AbstractValidator<OrchestratorEvent> GetValidator(UserAuthenticationEvent @event)
{
return new UserAuthenticationEventValidator();
}
public override Task<ProccessedMessage> ProcessAsync(UserAuthenticationEvent @event)
{
var deviceClient = await _userInfoService.GetDeviceClientsAsync(@event.DeviceId, @event.ClientId);
var orchestratorEvent = new OrchestratorEvent()
{
Type = Consts.OrchestratorEventType.SignUpEvent,
Payload = new OrchestratorEvent.OrchestratorEventPayload()
{
Device = new Device()
{
Id = @event.DeviceId,
Platform = DeviceUtils.GetPlatform(@event.ClientId),
IpAddress = deviceClient!.IpAddress
},
Language = new Language()
{
Locale = (LocaleType)deviceClient!.LocaleId,
},
User = new User()
{
Id = @event.UserId.ToString(),
Email = @event.Email,
FullName = @event.Name,
IsAccountActive = @event.IsActivated,
SignupAt = @event.CreationDate
},
Subscription = new Subscription()
{
MonetizationStatus = MonetizationStatusType.Free
},
Login = new Login()
{
Provider = GetProvider(@event.Method)
}
}
};
var processedMessage = new ProccessedMessage();
processedMessage.Events.Add(orchestratorEvent);
processedMessage.Properties.Add(new KeyValuePair<string, object>("EventType", orchestratorEvent.Type));
return processedMessage;
}
}Finally, define the base class that contains the pipeline logic.
public abstract class OrchestratorHandler<Event>
: IMessageHandler<Event>, IOrchestratorValidator<Event>
where Event : IntegrationMessage
{
protected IMessageBusPublisher<OrchestratorEvent> _eventBusPublisher;
protected ILogger<OrchestratorHandler<Event>> _logger;
protected OrchestratorHandler(IMessageBusPublisher<OrchestratorEvent> eventBusPublisher, ILogger<OrchestratorHandler<Event>> logger)
{
_eventBusPublisher = eventBusPublisher;
_logger = logger;
}
/// <summary>
/// The entry point for all events
/// </summary>
/// <param name="message"></param>
/// <returns></returns>
public async Task Handle(Event message)
{
var serviceBus = new { MessageId = message.Id , message.CreationDate, Type = message.GetType().Name };
using var _ = new LogContextEnricher()
.AddProperty("MT_ServiceBus", serviceBus, true)
.Push();
var proccessedMessage = await ProcessAsync(message);
try
{
proccessedMessage.Events.ForEach(e =>
{
GetValidator(message).ValidateAndThrow(e);
});
}
catch (Exception ex)
{
_logger.LogError(ex, "Validation failed");
throw new ValidationException(ex.Message);
}
await PublishAsync(proccessedMessage);
}
/// <summary>
/// Executes event processing
/// </summary>
/// <param name="event">Input event from service bus</param>
/// <returns>
/// List of processed events with their property messages
/// </returns>
public abstract Task<ProccessedMessage> ProcessAsync(Event @event);
/// <summary>
/// Validates the new event
/// </summary>
/// <param name="event"></param>
/// <returns></returns>
public abstract AbstractValidator<OrchestratorEvent> GetValidator(Event @event);
/// <summary>
/// Send the processed message to the event bus
/// </summary>
/// <param name="proccessedMessage"></param>
/// <returns></returns>
protected virtual async Task PublishAsync(ProccessedMessage proccessedMessage)
{
if (!proccessedMessage.Events.Any())
return;
foreach (var currentMessage in proccessedMessage.Events)
{
await _eventBusPublisher.PublishAsync(currentMessage, proccessedMessage.Properties);
using var _ = new LogContextEnricher()
.AddProperty("MT_OrchestratorEvent", currentMessage, true)
.Push();
_logger.LogInformation("Published event");
}
}
}2.2 Iterable Service implementation code samples
For the Iterable Service, start by registering the Orchestrator Event handler for the new event type.
services.AddScoped<IOrchestratorEventHandler<OrchestratorEvent>, SignUpOrchestratorEventHandler>()Next, define the handler that publishes events and user attributes to their specific message queue.
public class SignUpOrchestratorEventHandler : BaseOrchestratorEventHandler
{
private readonly IMessageBusPublisher<UserAttributesMessage> _userAttributesServiceBusPublisher;
private readonly IScheduledMessageBusPublisher<EventsMessage> _eventsScheduledMessageBusPublisher;
private readonly IMapper _mapper;
public SignUpOrchestratorEventHandler(
IMessageBusPublisher<UserAttributesMessage> userAttributesServiceBusPublisher,
IScheduledMessageBusPublisher<EventsMessage> eventsscheduledMessageBusPublisher,
IMapper mapper,
ILogger<SignUpOrchestratorEventHandler> logger) : base(logger)
{
_userAttributesServiceBusPublisher = userAttributesServiceBusPublisher;
_eventsScheduledMessageBusPublisher = eventsscheduledMessageBusPublisher;
_mapper = mapper;
}
protected override async Task<bool> PublishUserAttributesAsync(OrchestratorEvent @event)
{
var userAttributesMessage = _mapper.Map<UserAttributesMessage>(@event);
userAttributesMessage.SignupAt = @event.Payload.User.SignupAt!.Value.ToIterableString();
userAttributesMessage.Email = @event.Payload.User.Email!.ReplaceTempEmail();
userAttributesMessage.UserFirstName = UserUtils.GetFirstName(@event.Payload.User.FullName);
userAttributesMessage.UserFullName = @event.Payload.User.FullName;
userAttributesMessage.SuperControlGroup = ExperimentUtils.IsSuperControlGroup(@event.Payload.User.Id);
userAttributesMessage.ActivatedAccount = @event.Payload.User.IsAccountActive!.Value;
userAttributesMessage.Locale = LanguageUtils.GetLanguageCode(@event.Payload.Language);
userAttributesMessage.Ip = @event.Payload.Device!.IpAddress!;
userAttributesMessage.MonetizationStatus = SubscriptionUtils.GetMonetizationStatus(MonetizationStatusType.Free);
userAttributesMessage.LastPlatformName = DeviceUtils.GetPlatformName(@event.Payload.Device!.Platform!.Value);
await _userAttributesServiceBusPublisher.PublishAsync(userAttributesMessage);
return true;
}
protected override async Task<bool> PublishEventAsync(OrchestratorEvent @event)
{
var eventMessage = _mapper.Map<EventsMessage>(@event);
eventMessage.Payload = new EventsMessage.EventsMessagePayload
{
SignUpComplete = _mapper.Map<SignUpCompleteEventPayload>(@event)
};
await _eventsScheduledMessageBusPublisher
.PublishScheduledAsync(eventMessage, Enumerable.Empty<KeyValuePair<string, object>>(),
DateTimeOffset.Now.AddSeconds(Consts.ServiceBus.EventsDelay));
return true;
}
}Then define the handler for the user attributes message queue that sends a request to Iterable.
public class UserAttributesMessageHandler : IMessageHandler<UserAttributesMessage>
{
private readonly ILogger<UserAttributesMessageHandler> _logger;
private readonly IServiceProvider _serviceProvider;
public UserAttributesMessageHandler(ILogger<UserAttributesMessageHandler> logger, IServiceProvider serviceProvider)
{
_logger = logger;
_serviceProvider = serviceProvider;
}
public async Task Handle(UserAttributesMessage message)
{
var serviceBus = new { MessageId = message.Id, message.CreationDate, Type = message.GetType().ShortDisplayName() };
var user = new { Id = message.UserId.ToString() };
using var _ = new LogContextEnricher()
.AddProperty("MT_ServiceBus", serviceBus, true)
.AddProperty("MT_User", user, true)
.Push();
_logger.LogInformation("Consuming message from user-attributes-ext-tracking-iterable-queue queue");
using var scope = _serviceProvider.CreateScope();
var iterableClient = scope.ServiceProvider.GetRequiredService<IIterableClient>();
await iterableClient.SendUserAttributesAsync(message);
}
}Finally, define the handler for the events message queue that sends the request to Iterable.
public class EventsMessageHandler : IMessageHandler<EventsMessage>
{
private readonly ILogger<EventsMessageHandler> _logger;
private readonly IServiceProvider _serviceProvider;
public EventsMessageHandler(ILogger<EventsMessageHandler> logger, IServiceProvider serviceProvider)
{
_logger = logger;
_serviceProvider = serviceProvider;
}
public async Task Handle(EventsMessage message)
{
var serviceBus = new { MessageId = message.Id, message.CreationDate, message.Type };
var user = new { Id = message.UserId.ToString() };
using var _ = new LogContextEnricher()
.AddProperty("MT_ServiceBus", serviceBus, true)
.AddProperty("MT_User", user, true)
.Push();
_logger.LogInformation("Consuming message from events-ext-tracking-iterable-queue");
using var scope = _serviceProvider.CreateScope();
var iterableClient = scope.ServiceProvider.GetRequiredService<IIterableClient>();
await iterableClient.SendEventAsync(message);
}
}Section 3: Real-World Scenario
Let’s consider a real-world scenario outlined in the workflow at the end of this section, and break it down into pieces to understand the steps involved in updating user data in our CRM and in sending notifications.
User registration
Upon downloading the application, the user initiates the sign-up process. Our internal authentication service verifies the data input by the user. If the data is validated, the user is registered in the database and a user authentication event is subsequently published.
The orchestrator service, which is subscribed to the user authentication topic, processes this event. It enriches the event into a new object that is then published in a separate topic designated for orchestrator events.
The Iterable service is responsible for handling this event type. It splits the event into two distinct messages: one for creating the user profile in Iterable, and the other for sending a custom event to Iterable. This custom event activates a specific campaign, resulting in the user receiving a personalized welcome email.
Course enroll
When a user enrolls in a course, several backend processes are triggered, culminating in a course enrollment event.
Similar to the previous case, the orchestrator service processes this event, transforms it, and publishes a new message. However, in this instance, a custom event is not sent to Iterable. Instead, the user’s attributes are updated to enable the sending of scheduled campaigns through specific segmentation to users who are enrolled in a course but have been inactive for some time.
Course completion
Upon completion of a course by a user, backend processes are initiated, leading to a ‘certificate generated’ event. As in the initial case, we update the user’s attributes and generate an event to send the newly generated certificate to the user. The user will then receive their certificate in their email inbox.
Section 4: Advice from Our Experience
- Like any other initiative, start with a Minimum Viable Product (MVP); kick off with the most crucial communications like welcome emails and password reset emails; and gradually add events in a natural way to ensure user data remains consistent. In our present application, despite handling over 30 events, we have successfully reduced the attributes we transmit to less than half of their original number.
- Thanks to the above point, we can understand much more accurately how far we are from reaching the quotas of these services. Our initial implementation used bulk calls because we thought we were exceeding the quota, but in reality, we didn’t even approach half of it.
- Logs are super important for this type of application. Thanks to our logs, we have been able to deeply understand how Iterable works — specifically when problems arise. We use logs to identify the root issues underlying the problems and to answer questions from our customer success team.
