Building a Generic File Export Utility with Joda Beans and Spring Reactor
Why generic file export library?
In most of the applications there is a business requirement to export/download or dump files from a data set either queried from database or any other source. Mostly developers implement this feature by writing separate Java bean to File column mapping for each use case and format of file or somewhat home grown generic library using java reflection etc. Both of these approaches either lead to a lot of boilerplate code or performance issues with Java reflection.
The idea is to have the ability to access the properties of a Java bean by name without using reflection and iterating over the properties of file to write them in file columns one by one. Apart from ability to treat Java beans as a Collection and hence iterate over bean properties, Joda Beans offers a lot more which is used to develop a generic file export library, which can be used by any Java based application. Another challenge is to have a mechanism to provide the metadata such as file header column names and extendable/customizable file writing strategies as exporting to any required format CSV, Excel, PDF etc. It can perform better if the data collection part and writing this data to file happen in separate threads (Producer/Consumer).
Adding joda-file-export to Your Maven Project
The library uses Lombok and needs at least java 8 to compile and run. So you need to setup Lombok in your favorite IDE
Add below maven depenencies into your project’s pom.xml
Add following build plugins
Git source contain the library code into below two packages
- com.xebia.util.export Complete joda-file-export library code, copy this code into your source directory
- com.zcompany.example All example code elaborating the use of library, play with the examples to understand the usage
Code is bundled as Spring boot application, So simply run the application and browse through swagger at
Hit following URL to test file download
Downloading a CSV file with name sample in default context, writing blank if a property’s value is found to be null. Using global
jodaConverter), to convert properties to string while writing to file, which ever property class's type converter is registered with
Executing the file export
Defining POJO as Joda Bean
The export candidate Java bean to be used as data container while writing data to file must be written in an opinionated manner making it as a Joda bean as follows.
- POJO must be annotated with
@BeanDefinitionand extend either
- All properties whose data is to be written in file must be annotated with
@PropertyDefinition. If any property is to be ignored while file writing then do not annotate the property with
@PropertyDefinition, hence you can ignore any properties if required.
- All primitive properties or non Joda bean properties must be annotated with
- A property which is itself a Joda bean must not be annotated with
@Exportannotation, but just with
- The column name in the file where the POJO property would be exported must be given as
columnNameattribute of the
- You may have some reusable DTOs being using in multiple export candidate Bean. So depending upon use case, you may want to have the given column name in composed class or may want to override them with different column names in another export candidate Bean. The same can be done by using
@ExportOverrideannotation. If single property column name is to be overridden then use
@ExportOverrideor if multiple properties of the composed bean needs to be overridden then use the companion
- The nested composed classes export metadata can also be overridden by providing the complete property path as follows. Ex. Overriding nested beans property metadata
fieldName = "currency.source"
- There could be some use cases where a class is being used in multiple scenarios but you may need different file structures exported. In such cases you can give a context name to given property as follows. While exporting the file you can optionally give a context name as an attribute of
context. If you need to exclude some properties in exported file in a specific context or scenario but include the same in another context or scenario then you can mark the property with a unique context name.
- While export only the properties with no context given (default context) and specified context will be included in exported file whose context names matches the given Context in
ExportContext. The properties with no given contexts i.e. in default context would always be exported but the properties given with one or multiple context names would only be exported if the export is executed in a given context.
Exporting the file with above data bean in a context is as follows, So the following code would export agent property along with all other with default or no context properties but ignore bank property in the exported file
File writing strategies
You may either want to download a file in a web application or just dump the same at a given location. These are just different strategies of file writing. By default there are two strategies bundled out of box with library i.e.
DownloadExcelFileStrategy.java to download file either in CSV or Excel format. But you may need your own custom file writing strategy in following cases
- Need to export file in a different format such as PDF or RTF
- Need to customize excel sheet column styles
- Need to externalize file header column names in a properties file
- Rather than downloading the file, you may need to dump a file at a given location.
- By default the order of columns in file would be as per the order of properties defined in bean, in case you need to change the default order.
- Or any other reason as per your need
So you can define any new strategy of file writing simply by implementing
FileWriterStrategy.java interface. One such custom strategy is given in the examples,
ExternalizedHeaderLabelsDumpCSVStrategy.java as follows.
And then use above strategy in your file export as follows.
The order of columns in the file can be customized as per your needs by providing a custom File writer strategy and reordering the columns headers and row data in following methods of the strategy implementation.
The properties of bean are written in file in String format only. So it is a very basic requirement to convert the bean properties to a string in a particular format. For example you may need to format date time to a specific format while writing to file. So you can define your custom type converters as follows. All such type converters then needs to be registered in Joda
StringConvert as follows. The library then automatically converts the data in a bean property depending upon the Class of property. For example by defining following type converter and registering it to global
StringConvert joda converter, each property of
BigDecimal type would be precised to 8 decimal points in exported file.
Similar to above type converter you may have others also. After defining all such type converters register them in global Joda converter
StringConvert as follows.
ExportContext expects an instance of
StringConvert while exporting the file. It is recommended to have a singleton instance of
StringConvert, but in case you need different converters for same class such as in one case you want to write Boolean value as YES/NO but in other case ENABLED/DISABLED, then you may need to create multiple instances of
StringConvert and use respective instance as per the conversion required. You can find following type converters in source code.
Going the Reactive way
Normally the data to export is fetched from some database. If data set is small then you can just fetch a collection and pass the collection to export API. Or if the data set is large you may need to fetch the data page by page or in batches and sequentially push the data into a
Flux using any of programmatically generating
Flux strategy. There are some databases like Postgres or MongoDB, which have native reactive supporting JDBC drivers and you can simply get a
Flux from their Spring Data repository.
Or you can programmatically push your data into
Flux and pass the same to Export API as follows.
If there are multiple data sources such as some Queue, API and DB etc., then you can you Spring Reactor’s thread safe
FluxSink as follows.
EmitterProcessor is thread safe so you can push data into
FluxSink simultaneously,the same would be available in the
Flux for consumption. File writing happens in single thread to retain the order of rows. As with Spring Reactor's
Flux, nothing happens until the
Flux is subscribed. Hence the export is only started once you call the
export() method on
There could be IO related exceptions or Joda mis-configuration of beans. In all error scenarios, all exceptions are wrapped into a single unchecked exception with different error codes and description message. In case any exception occurs the same is logged. You may get the error code and description from logs and take the required measures but you can not catch any of the exceptions as they occur in separate thread while processing the
Ideally the exported file would have a fixed structure. In case the data is in different structure, you need to normalize the data as per the model to be exported. As of now no collection is supported in POJO bean except Map, that too with restrictions. Even its highly unlikely to have a collection in the file export candidate bean, because in that case you may not have a fixed number of columns in the file, which is almost always required in file export use cases. For example if data bean is supposed to have a list then you may not be able to have a fixed number of columns in the file as list content may vary for different records.
A Map collection is supported in file export candidate bean with following restrictions.
- It is recommended be used only with a fixed number of entries otherwise the number of columns in file would be too many
- Both Key and Value of the map must be custom class object.
- The Key class must implement
Distinguishable.javainterface, overriding two methods.
descriminator()method must return a unique String for each Key in the Map.
label()method must return a String to be prefixed in all columns of corresponding Key's value columns.
- The value must be a Joda bean, refer to below class for reference.
The export candidate bean may also compose other beans till any depth. But if while file export any of the composed joda bean is found as
null, then the export would fail. So you need to make sure none of the joda bean objects in the data set is
null. The primitives and non Joda beans properties can obviously be
null. So as given in the examples
Cost.java is a joda bean composed in multiple export candidate classes, the value of
Cost should never be null. If you do not have any Cost value then simply initialize it with
null sell and buy values.
Find source code on Github repository joda-file-export