Audit Log Approach with Spring AOP and FluentBit
This is the journey of audit logs …
As Pudo Team, our need is logging changes, which are triggered by our REST API’s, over the database entities and logging these with the desired fields(who made ? when ? what was the change ?).
But first, let’s look out what are the benefits of audit log.
- Increased Security : improves security by logging suspicious events.
- Proving Compliance : proving compliance with common regulations like HIPAA and PCI DSS etc.
- Detailed Inside : gives a detailed inside over the system and business.
- Risk Management : plays a key role in risk management strategy with the strength of data
Audit log fields can be changed according to your needs. Our log structure is shown below:
{
"request": "",
"before": {},
"after": {},
"difference": {},
"modifiedBy": {},
"date": ""
}Before filling out the json fields, you need to know we have some assumptions;
- all entities should have the “modifiedBy” field
- on the API side, custom annotated methods should check if the entity already exists in the database (if you need to get “before” data this is crucial).
Let’s clarify some points and fill out the fields, thanks to Spring AOP, we can get input and output of the methods.
The first check pointcut is “does entity exist ?” To do this check, we call findById(). It means that the output of the findById() is our “before” data.
The second pointcut is custom annotated method calls. We wrote a custom @Loggable annotation that decides which method’s input and output should be tracked. Returning value of the annotated method is our “after” data.
After this point, we can extract the “difference” and the “date” is already on the hand. How do we know who made this change? We assume that all entities should have the “modifiedBy” field as default so if you want to make changes over the APIs, you need to tell who are you. The request should also include a “modifiedBy” field.
Definition of @Loggable annotation:
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Loggable {}Definition of AuditLog:
public class AuditLog {
private Object request;
private Object before;
private Object after;
private Object difference;
private String modifiedBy;
private Long date;
}
public class FluentbitLog {
private AuditLog auditLog;
}Definition of audit log service:
@Aspect
@Component
@Slf4j
public class AuditLogService {
public static final String MODIFIED_BY = "modifiedBy";
private static final ThreadLocal<AuditLog> auditLogThreadLocal = new ThreadLocal<>();
private final Gson gson = new Gson();
@Pointcut("@annotation(com.trendyol.pudoauditlog.auditlog.annotations.Loggable)")
public void loggables() {
}
@Pointcut("execution(* org.springframework.data.repository.CrudRepository.findById(..))")
public void findById() {
}
@AfterReturning(value = "loggables()", returning = "dto")
public void loggableMethodCall(JoinPoint jp, Object dto) {
try {
final AuditLog auditLog = auditLogThreadLocal.get();
auditLog.setRequest(gson.toJson(jp.getArgs()));
auditLog.setAfter(gson.toJson(dto));
auditLog.setDate(Instant.now().toEpochMilli());
auditLog.setDifference(getDifference(auditLog.getBefore(), auditLog.getAfter()));
auditLog.setModifiedBy(getModifiedBy());
log.info(gson.toJson(FluentbitLog.builder().auditLog(auditLog).build()));
} catch (Exception exception) {
log.warn("[AuditLog] error occurred while loggableMethodCall logging ! {}", exception.getMessage());
}
}
@AfterReturning(value = "findById()", returning = "entity")
public void findByIdCall(JoinPoint jp, Optional<?> entity) {
try {
AuditLog auditLog = AuditLog.builder().build();
entity.ifPresent(o -> auditLog.setBefore(gson.toJson(o)));
auditLogThreadLocal.set(auditLog);
} catch (Exception exception) {
log.warn("[AuditLog] error occurred while findByIdCall logging ! {}", exception.getMessage());
}
}
private String getModifiedBy() throws JsonSyntaxException {
JsonObject afterObjectJson = gson.fromJson(auditLogThreadLocal.get().getAfter(), JsonObject.class);
return String.valueOf(afterObjectJson.get(MODIFIED_BY));
}
private String getDifference(String before, String after) throws JsonSyntaxException {
return StringUtils.hasText(before) ? extractDifferences(before, after) : null;
}
private String extractDifferences(String before, String after) {
Type mapType = new TypeToken<Map<String, Object>>() {
}.getType();
Map<String, Object> firstMap = gson.fromJson(before, mapType);
Map<String, Object> secondMap = gson.fromJson(after, mapType);
final MapDifference<String, Object> difference = Maps.difference(firstMap, secondMap);
return gson.toJson(Map.of("diff", difference.entriesDiffering(), "added", difference.entriesOnlyOnRight()));
}
}Definitions of annotation and service are extracted as a library. To use this feature, you just need to add dependency to your project and put the custom @Loggable annotation on the desired methods.
We still have some questions, How we collect and store audit logs ?
Fluentbit comes into play. It is strong log processing and distribution tool that can transfer logs to the Kafka.
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[FILTER]
name grep
match kube.*
regex log .*auditLog.*
[OUTPUT]
Name kafka
Match kube.*
# insert your kafka broker url
Brokers borkerips
Topics topicname
Timestamp_Key @timestamp
Retry_Limit false
# hides errors "Receive failed: Disconnected" when kafka kills idle connections
rdkafka.log.connection.close false
# producer buffer is not included in http://fluentbit.io/documentation/0.12/configuration/memory_usage.html#estimating
rdkafka.queue.buffering.max.kbytes 10240
# for logs you'll probably want this ot be 0 or 1, not more
rdkafka.request.required.acks 1Above Fluentbit config express that you can filter just desired prefixed(“auditLog”) logs and then distribute them to topics on the Kafka.
After we put our logs into Kafka, we can save the log wherever we want. We prefer to use the Timescale database which is specified for time series data.
Conclusion
In this article, I tried to explain importance of the audit logging and our approach with the Spring AOP and the Fluentbit.
Thank you for reading and feel free to contact me if you have any questions.
