Elasticsearch, a powerful search and analytics engine, is often the go-to choice for developers when they need to handle large volumes of data with real-time search capabilities. With the introduction of .NET 7’s Minimal APIs, building web services for Elasticsearch has become even more streamlined. In this article, we’ll explore how to create a generic CRUD service for Elasticsearch and expose it using .NET 7’s Minimal APIs.
Setting the Stage
Before diving in, let’s understand the components we’ll be working with:
- Generic Elasticsearch Service: A reusable service that provides CRUD operations for any entity type.
- Minimal APIs: A new feature in .NET 7 that allows developers to build web APIs with minimal boilerplate.
Note: Before diving deeper into the intricacies of our Elasticsearch service, it’s crucial to emphasize a foundational expectation: the service presumes that the Elasticsearch client is correctly set up and functioning optimally. An intriguing aspect of our setup is our choice of tooling. Even though we’re interfacing with an Elasticsearch 8.x instance, we’ve chosen the .NET NEST client version 7.x. This decision stems from a specific feature that’s indispensable to our operations: the ‘Query DSL operators for combining queries’. These operators allow for a more intuitive and streamlined way to craft complex Boolean Queries. As of our last update, this feature isn’t fully embraced in the latest .NET client 8.x. It’s a reminder of the dynamic nature of technology, where occasionally, older tools might offer specific functionalities that newer iterations haven’t fully integrated yet. See release notes for more details.
The Generic Elasticsearch Service
1. Generic IElasticSearchService<T> Interface:
This generic interface is designed to offer CRUD operations for any entity type T in Elasticsearch. By being generic, it ensures code reusability across different entity types.
public interface IElasticSearchService<T> where T : class
{
IElasticSearchService<T> Index(string indexName);
Task<BulkResponse> AddOrUpdateBulk(IEnumerable<T> documents);
Task<T> AddOrUpdate(T document);
Task<BulkResponse> AddBulk(IList<T> documents);
Task<GetResponse<T>> Get(string key);
Task<ISearchResponse<T>?> Query(SearchDescriptor<T> sd);
Task<bool> Remove(string key);
Task<DeleteByQueryResponse> BulkRemove(IDeleteByQueryRequest<T> queryReq);
}Here’s a brief overview of the methods defined in the interface:
Index(string indexName): Sets the index name for the operations.AddOrUpdateBulk(IEnumerable<T> documents): Adds or updates multiple documents in bulk.AddOrUpdate(T document): Adds or updates a single document.AddBulk(IList<T> documents): Adds multiple documents in bulk.Get(string key): Retrieves a document by its key.Query(SearchDescriptor<T> sd): Executes a search query against Elasticsearch.Remove(string key): Deletes a document by its key.BulkRemove(IDeleteByQueryRequest<T> queryReq): Deletes multiple documents based on a query.
2. Implementation of IElasticSearchService:
The ElasticSearchService<T> class implements the IElasticSearchService<T> interface. Here's how it provides CRUD operations for the Article entity:
public class ElasticSearchService<T> : IElasticSearchService<T> where T : class
{
private string IndexName { get; set; }
private readonly IElasticClient _client;
public ElasticSearchService(IElasticClient client)
{
_client = client;
IndexName = typeof(T).Name.ToLower() + "s";
}
public IElasticSearchService<T> Index(string indexName)
{
IndexName = indexName;
return this;
}
public async Task<BulkResponse> AddOrUpdateBulk(IEnumerable<T> documents)
{
var indexResponse = await _client.BulkAsync(b => b
.Index(IndexName)
.UpdateMany(documents, (ud, d) => ud.Doc(d).DocAsUpsert())
);
return indexResponse;
}
public async Task<T> AddOrUpdate(T document)
{
var indexResponse =
await _client.IndexAsync(document, idx => idx.Index(IndexName));
if (!indexResponse.IsValid)
{
throw new Exception(indexResponse.DebugInformation);
}
return document;
}
public async Task<BulkResponse> AddBulk(IList<T> documents)
{
var resp = await _client.BulkAsync(b => b
.Index(IndexName)
.IndexMany(documents)
);
return resp;
}
public async Task<GetResponse<T>> Get(string key)
{
return await _client.GetAsync<T>(key, g => g.Index(IndexName));
}
public async Task<List<T>?> GetAll()
{
var searchResponse = await _client.SearchAsync<T>(s => s.Index(IndexName).Query(q => q.MatchAll()));
return searchResponse.IsValid ? searchResponse.Documents.ToList() : default;
}
public async Task<ISearchResponse<T>?> Query(SearchDescriptor<T> sd)
{
var searchResponse = await _client.SearchAsync<T>(sd);
return searchResponse;
}
public async Task<bool> Remove(string key)
{
var response = await _client.DeleteAsync<T>(key, d => d.Index(IndexName));
return response.IsValid;
}
public async Task<DeleteByQueryResponse> BulkRemove(IDeleteByQueryRequest<T> queryReq)
{
var response = await _client.DeleteByQueryAsync(queryReq);
return response;
}
}Constructor:
The constructor initializes the IElasticClient instance and sets the default index name based on the type T. For the Article entity, the default index name would be "articles".
Indexing:
Index(string indexName): This method simply sets the index name for subsequent operations. It's useful if you want to override the default index name.
Adding/Updating:
AddOrUpdateBulk(IEnumerable<T> documents): This method uses theBulkAsyncmethod of theIElasticClientto add or update multiple documents. It uses theUpdateManymethod to perform upsert operations (update if exists, insert if not).AddOrUpdate(T document): This method uses theIndexAsyncmethod of theIElasticClientto add or update a single document. If the document already exists in the index, it will be updated; otherwise, it will be added.AddBulk(IList<T> documents): This method uses theBulkAsyncmethod of theIElasticClientto add multiple documents in bulk using theIndexManymethod.
Reading:
Get(string key): This method uses theGetAsyncmethod of theIElasticClientto retrieve a document by its key (ID).Query(SearchDescriptor<T> sd): This method uses theSearchAsyncmethod of theIElasticClientto execute a search query against Elasticsearch. TheSearchDescriptor<T>parameter allows for building complex queries.
Deleting:
Remove(string key): This method uses theDeleteAsyncmethod of theIElasticClientto delete a document by its key (ID).BulkRemove(IDeleteByQueryRequest<T> queryReq): This method uses theDeleteByQueryAsyncmethod of theIElasticClientto delete multiple documents based on a query.
Exposing the Service with Minimal APIs
.NET 7’s Minimal APIs provide a concise way to define web endpoints. Instead of traditional controllers, you can define routes directly in the Program.cs or a separate configuration method.
Here’s how our Article CRUD operations can be exposed using Minimal APIs:
using Ardalis.Result;
using Ardalis.Result.AspNetCore;
using MediatR;
using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using Taxbox.Application.Features.Articles;
// ... other using statements ...
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
// Register MediatR services
app.Services.AddMediatR(typeof(GetArticleByIdRequest).Assembly); // Assuming all requests are in the same assembly
// ElasticSearchService
services.AddScoped(typeof(IElasticSearchService<>), typeof(ElasticSearchService<>));
// ... other services ...
var mediator = app.Services.GetRequiredService<IMediator>();
app.MapGet("/api/article/{id}", async (string id) =>
{
return await mediator.Send(new GetArticleByIdRequest(id));
});
app.MapGet("/api/article/list", async (GetAllArticlesRequest request) =>
{
return await mediator.Send(request);
});
app.MapPost("/api/article", async (CreateArticleRequest request) =>
{
return await mediator.Send(request);
});
app.MapPut("/api/article/{id}", async (string id, UpdateArticleRequest request) =>
{
return await mediator.Send(request with { Id = id });
});
app.MapDelete("/api/article/{id}", async (string id) =>
{
return await mediator.Send(new DeleteArticleRequest(Id: id));
});
app.MapPost("/api/article/bulkAddOrUpdate", async (BulkAddOrUpdateArticlesRequest request) =>
{
return await mediator.Send(request);
});
app.MapPost("/api/article/bulkRemove", async (BulkRemoveArticlesRequest request) =>
{
return await mediator.Send(request);
});
// ... other endpoints ...
app.Run();A few things to note:
- Dependency Injection: In Minimal APIs, you can directly inject services into your endpoint delegate. In the example above, we’re injecting the
IMediatorservice. - Routing: The routing is more concise. You define the HTTP verb and the route directly.
- No Controllers: There’s no need for controller classes, which reduces the boilerplate code.
- Authorization: If you need to apply authorization, you can still do so using middleware or attributes.
Remember, this is a basic transformation. Depending on your application’s complexity, you might need to make additional adjustments, especially if you’re using features like model validation, exception handling, or custom middleware.
Article Entity
public class Article
{
public string Id { get; set; } = NewId.NextGuid().ToString();
public string Title { get; set; } = null!;
public Metadata? Metadata { get; set; }
public string? HtmlContent { get; set; }
public string? Content { get; set; }
public IList<string>? AuthorIds { get; set; } = new List<string>();
public IList<Author>? Authors { get; set; } = new List<Author>();
public DateTime? CreatedAt { get; set; }
public DateTime? UpdatedAt { get; set; }
public DateTime? PublishedAt { get; set; }
public IList<string>? Tags { get; set; }
public bool IsPublic { get; set; }
public bool IsPublished { get; set; }
public bool IsDraft { get; set; }
public string? CoverImage { get; set; }
public string? ThumbnailImage { get; set; }
public IList<ArticleAttachment>? Attachments { get; set; }
public string? Category { get; set; }
public string? Slug { get; set; }
}
public class ArticleAttachment
{
public string File { get; set; } = null!;
public string Type { get; set; } = null!;
}
public class Metadata
{
public string? Category { get; set; }
public string? Language { get; set; }
public int? Views { get; set; }
public override string ToString()
{
return $"{Category} {Language} {Views}";
}
}Let’s look at the GetAllArticlesRequest :
public record GetAllArticlesRequest : GetAllArticlesRequestBase, IRequest<PaginatedList<GetAllArticlesResponse>>
{}GetAllArticlesRequestBase :
public record GetAllArticlesRequestBase
{
public virtual bool? IsPublic { get; set; }
public string? Title { get; set; }
public Metadata? Metadata { get; set; }
public string? Content { get; set; }
public IList<string>? AuthorIds { get; set; }
public DateTime? CreatedAt { get; set; }
public DateTime? UpdatedAt { get; set; }
public DateTime? PublishedAt { get; set; }
public IList<string>? Tags { get; set; }
public bool? IsPublished { get; set; }
public bool? IsDraft { get; set; }
public string? SourceFields { get; set; }
public string? FreeTextSearch { get; set; }
public string? SortBy { get; set; }
public string? SortOrder { get; set; }
public int CurrentPage { get; set; } = 1;
public int PageSize { get; set; } = 10;
public string? Category { get; set; }
public string? Slug { get; set; }
public override string ToString()
{
return
$"{Title} {Metadata} {Content} {AuthorIds} {CreatedAt} {UpdatedAt} {Tags} {IsPublished} {SourceFields} {FreeTextSearch} {IsDraft} {PublishedAt}";
}GetAllArticlesValidator :
public class GetAllArticlesValidator : AbstractValidator<GetAllArticlesRequestBase>
{
public GetAllArticlesValidator()
{
RuleLevelCascadeMode = ClassLevelCascadeMode;
var allowedFields = new List<string>
{
"Title",
"CreatedAt",
"UpdatedAt",
"PublishedAt",
"IsPublished",
"IsDraft",
"IsPublic",
"Slug",
"Category"
};
RuleFor(x => x.SortBy)
.Must(x => x == null || allowedFields.Contains(x))
.WithMessage($"SortBy must be one of the following: {string.Join(", ", allowedFields)}");
}
}GetAllArticlesResponse :
public record GetAllArticlesResponse
{
public string Id { get; init; } = null!;
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string Title { get; set; } = null!;
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public Metadata? Metadata { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string Content { get; set; } = null!;
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string HtmlContent { get; set; } = null!;
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
public IList<string> AuthorIds { get; set; } = new List<string>();
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public DateTime? CreatedAt { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public DateTime? UpdatedAt { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public DateTime? PublishedAt { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public IList<string>? Tags { get; set; }
public bool IsPublic { get; set; }
public bool IsPublished { get; set; }
public bool IsDraft { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? CoverImage { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? ThumbnailImage { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public IList<ArticleAttachment>? Attachments { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Category { get; set; }
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public string? Slug { get; set; }
}GetAllArticlesHanlder :
public class GetAllArticlesHandler : IRequestHandler<GetAllArticlesRequest, PaginatedList<GetAllArticlesResponse>>
{
private readonly IElasticSearchService<Article> _esService;
public GetAllArticlesHandler(IElasticSearchService<Article> esService)
{
_esService = esService;
}
public async Task<PaginatedList<GetAllArticlesResponse>> Handle(GetAllArticlesRequest request,
CancellationToken cancellationToken)
{
QueryContainer qd;
if (string.IsNullOrEmpty(request.ToString()) || string.IsNullOrWhiteSpace(request.ToString()))
{
qd = new QueryContainerDescriptor<Article>().MatchAll();
}
else
{
qd = BuildQueryDescriptor(request);
}
if (request.CurrentPage <= 0)
{
request.CurrentPage = 1;
}
if (request.PageSize <= 0)
{
request.PageSize = 10;
}
var availableFields = typeof(Article).GetProperties().Select(p => p.Name);
IEnumerable<string> enumerable = availableFields as string[] ?? availableFields.ToArray();
var fields = request.SourceFields?.Split(',').ToArray() ?? Array.Empty<string>();
var validFields = fields.Intersect(enumerable, StringComparer.OrdinalIgnoreCase);
var sd = new SearchDescriptor<Article>()
.Index("articles")
.Query(_ => qd)
.From((request.CurrentPage - 1) * request.PageSize)
.Size(request.PageSize)
.Source(s => s.Includes(i => i.Fields(validFields.ToArray())))
;
if (!string.IsNullOrEmpty(request.SortBy) && !string.IsNullOrWhiteSpace(request.SortBy))
{
if (!enumerable.Contains(request.SortBy, StringComparer.OrdinalIgnoreCase))
{
request.SortBy = GetAllArticlesRequestConstants.DefaultSortBy;
}
if (request.SortOrder != GetAllArticlesRequestConstants.Ascending &&
request.SortOrder != GetAllArticlesRequestConstants.Descending)
{
request.SortOrder = GetAllArticlesRequestConstants.Descending;
}
var sort = new SortDescriptor<Article>().Field(request.SortBy,
request.SortOrder == "asc" ? SortOrder.Ascending : SortOrder.Descending);
sd.Sort(_ => sort);
}
var resp = await _esService.Query(sd);
var list = new List<GetAllArticlesResponse>();
if (resp?.Hits != null)
{
foreach (var hit in resp.Hits)
{
if (hit.Source == null) continue;
hit.Source.Id = hit.Id;
list.Add(hit.Source.Adapt<GetAllArticlesResponse>());
}
return new PaginatedList<GetAllArticlesResponse>(list,
(int)resp.Total, request.CurrentPage, request.PageSize);
}
return new PaginatedList<GetAllArticlesResponse>();
}
private QueryContainer BuildQueryDescriptor(GetAllArticlesRequest request)
{
var should = new QueryContainer();
var must = new QueryContainer();
if (!string.IsNullOrEmpty(request.FreeTextSearch))
{
should = should || new MatchPhraseQuery()
{
Field = Infer.Field<Article>(f => f.Title), Query = request.FreeTextSearch, Boost = 3,
};
should = should || new MatchQuery
{
Field = Infer.Field<Article>(f => f.Title),
Query = request.FreeTextSearch,
Boost = 2,
Fuzziness = Fuzziness.Auto,
Operator = Operator.Or
};
should = should || new WildcardQuery
{
Field = Infer.Field<Article>(f => f.Title),
Value = $"*{request.FreeTextSearch}*",
Boost = 1,
CaseInsensitive = true
};
should = should || new MatchPhraseQuery()
{
Field = Infer.Field<Article>(f => f.Content), Query = request.FreeTextSearch, Boost = 1,
};
should = should || new MatchQuery
{
Field = Infer.Field<Article>(f => f.Content),
Query = request.FreeTextSearch,
Boost = 1,
Fuzziness = Fuzziness.Auto,
Operator = Operator.And
};
should = should && new BoolQuery { MinimumShouldMatch = 1 };
}
if (!string.IsNullOrEmpty(request.Title))
{
must = must && new MatchQuery
{
Field = Infer.Field<Article>(f => f.Title), Query = request.Title, Boost = 2
};
}
if (!string.IsNullOrEmpty(request.Content))
{
must = must && new MatchQuery
{
Field = Infer.Field<Article>(f => f.Content), Query = request.Content, Boost = 1
};
}
if (request.AuthorIds is { Count: > 0 })
{
must = must && new TermsQuery { Field = Infer.Field<Article>(f => f.AuthorIds), Terms = request.AuthorIds };
}
if (request.Tags is { Count: > 0 })
{
must = must && new TermsQuery { Field = Infer.Field<Article>(f => f.Tags), Terms = request.Tags };
}
if (request.CreatedAt != null)
{
must = must && new DateRangeQuery
{
Field = Infer.Field<Article>(f => f.CreatedAt),
GreaterThanOrEqualTo = request.CreatedAt,
LessThanOrEqualTo = request.CreatedAt
};
}
if (request.UpdatedAt != null)
{
must = must && new DateRangeQuery
{
Field = Infer.Field<Article>(f => f.UpdatedAt),
GreaterThanOrEqualTo = request.UpdatedAt,
LessThanOrEqualTo = request.UpdatedAt
};
}
if (request.IsPublished != null)
{
must = must && new TermQuery
{
Field = Infer.Field<Article>(f => f.IsPublished), Value = (bool)request.IsPublished
};
}
if (request.IsPublic != null)
{
must = must && new TermQuery
{
Field = Infer.Field<Article>(f => f.IsPublic), Value = (bool)request.IsPublic
};
}
if (request.Category != null)
{
must = must && new MatchQuery { Field = Infer.Field<Article>(f => f.Category), Query = request.Category };
}
if (request.Slug != null)
{
must = must && new TermQuery { Field = Infer.Field<Article>(f => f.Category), Value = request.Category };
}
return should && must;
}
}Understanding the GetAllArticlesHandler Class:
The GetAllArticlesHandler is a mediator between the API request and the Elasticsearch service. It processes the request, constructs a tailored query, fetches the data, and returns it in a structured, paginated format. It's a testament to the power of clean architecture, where business logic is neatly encapsulated, making the system maintainable and scalable.
Let's break it down:
Purpose: This class is designed to handle requests to fetch articles. It implements the IRequestHandler interface, indicating it's a handler for a specific request type, in this case, GetAllArticlesRequest.
Dependencies:
IElasticSearchService<Article> _esService: The handler relies on a generic Elasticsearch service tailored forArticleentities. This service provides the necessary CRUD operations.
Constructor: The handler’s constructor injects the Elasticsearch service, ensuring it has the necessary tools to fetch data.
Handle Method:
- This is the core method where the action happens. It takes in a
GetAllArticlesRequestand returns a paginated list of articles. - It first constructs a query based on the request parameters. For instance, if no specific search terms are provided, it defaults to fetching all articles.
- It then ensures valid pagination parameters are set.
- The method also allows for field selection, ensuring only relevant data is fetched.
- Sorting capabilities are also provided, allowing articles to be ordered based on specific fields.
- Finally, it queries the Elasticsearch service and constructs the response.
BuildQueryDescriptor Method:
- This private helper method constructs a complex query based on the request. It allows for free-text search, filtering by title, content, authors, tags, dates, and more.
- It uses a combination of
should(optional criteria) andmust(mandatory criteria) clauses to build a comprehensive search query. - I’m using Operator overloading feature from the NEST .NET Client to combine the queries together with an easy-to-understand syntax.
Similar to the GetAllArticlesHanlder handling, we can implement the following classes:
- GetArticleByIdHandler
- CreateArticleHandler
- UpdateArticleHandler
- DeleteHandler
- BulkRemoveHandler
- BulkAddOrUpdateHandler
These classes can have Request, Response, and Validator classes.
Benefits of This Approach:
- Reusability: The generic Elasticsearch service can be used for any entity type, reducing code duplication.
- Simplicity: Minimal APIs reduce boilerplate, making the codebase cleaner and more readable.
- Flexibility: The generic service can easily be extended or modified without affecting the API layer.
Conclusion
Combining the power of Elasticsearch with the simplicity of .NET 7’s Minimal APIs offers a compelling way to build efficient and maintainable web services. Whether you’re building a content management system, an e-commerce platform, or any application that requires robust search capabilities, this approach provides a solid foundation for your development needs.
