Building a Generic File Export Utility with Joda Beans and Spring Reactor

Rajveer Singh
Dec 16, 2019 · 8 min read

Why generic file export library?

In most of the applications there is a business requirement to export/download or dump files from a data set either queried from database or any other source. Mostly developers implement this feature by writing separate Java bean to File column mapping for each use case and format of file or somewhat home grown generic library using java reflection etc. Both of these approaches either lead to a lot of boilerplate code or performance issues with Java reflection.

The idea is to have the ability to access the properties of a Java bean by name without using reflection and iterating over the properties of file to write them in file columns one by one. Apart from ability to treat Java beans as a Collection and hence iterate over bean properties, Joda Beans offers a lot more which is used to develop a generic file export library, which can be used by any Java based application. Another challenge is to have a mechanism to provide the metadata such as file header column names and extendable/customizable file writing strategies as exporting to any required format CSV, Excel, PDF etc. It can perform better if the data collection part and writing this data to file happen in separate threads (Producer/Consumer).

Getting started

Building joda-file-export

Adding joda-file-export to Your Maven Project

The library uses Lombok and needs at least java 8 to compile and run. So you need to setup Lombok in your favorite IDE

Add below maven depenencies into your project’s pom.xml

Add following build plugins

Git source contain the library code into below two packages

  • com.xebia.util.export Complete joda-file-export library code, copy this code into your source directory
  • com.zcompany.example All example code elaborating the use of library, play with the examples to understand the usage

Code is bundled as Spring boot application, So simply run the application and browse through swagger at

http://localhost:8080/swagger-ui.html

Hit following URL to test file download

http://localhost:8080/api/jodaExport/collectionDownload?fileName=inter_bank_rates&downloadFileType=CSV

http://localhost:8080/api/jodaExport/reactiveDownload?fileName=rx_sample&downloadFileType=CSV

http://localhost:8080/api/jodaExport/withContextDownload?contextName=bank_specific&fileName=var_rates&downloadFileType=EXCEL

http://localhost:8080/api/jodaExport/withContextDownload?contextName=agent_specific&fileName=var_rates&downloadFileType=CSV

Using joda-file-export

Downloading a CSV file with name sample in default context, writing blank if a property’s value is found to be null. Using global StringConvert ( jodaConverter), to convert properties to string while writing to file, which ever property class's type converter is registered with jodaConverter.

Executing the file export

Defining POJO as Joda Bean

The export candidate Java bean to be used as data container while writing data to file must be written in an opinionated manner making it as a Joda bean as follows.

  • POJO must be annotated with @BeanDefinition and extend either Bean or ImmutableBean class.
  • All properties whose data is to be written in file must be annotated with @PropertyDefinition. If any property is to be ignored while file writing then do not annotate the property with @PropertyDefinition, hence you can ignore any properties if required.
  • All primitive properties or non Joda bean properties must be annotated with @Export annotation.
  • A property which is itself a Joda bean must not be annotated with @Export annotation, but just with @PropertyDefinition
  • The column name in the file where the POJO property would be exported must be given as columnName attribute of the @Export annotation.
  • You may have some reusable DTOs being using in multiple export candidate Bean. So depending upon use case, you may want to have the given column name in composed class or may want to override them with different column names in another export candidate Bean. The same can be done by using @ExportOverride annotation. If single property column name is to be overridden then use @ExportOverride or if multiple properties of the composed bean needs to be overridden then use the companion @ExportOverrides annotation.
  • The nested composed classes export metadata can also be overridden by providing the complete property path as follows. Ex. Overriding nested beans property metadata fieldName = "currency.source"
  • There could be some use cases where a class is being used in multiple scenarios but you may need different file structures exported. In such cases you can give a context name to given property as follows. While exporting the file you can optionally give a context name as an attribute of @Export annotation in context. If you need to exclude some properties in exported file in a specific context or scenario but include the same in another context or scenario then you can mark the property with a unique context name.
  • While export only the properties with no context given (default context) and specified context will be included in exported file whose context names matches the given Context in ExportContext. The properties with no given contexts i.e. in default context would always be exported but the properties given with one or multiple context names would only be exported if the export is executed in a given context.

Exporting the file with above data bean in a context is as follows, So the following code would export agent property along with all other with default or no context properties but ignore bank property in the exported file

File writing strategies

You may either want to download a file in a web application or just dump the same at a given location. These are just different strategies of file writing. By default there are two strategies bundled out of box with library i.e. DownloadCSVFileStrategy.java and DownloadExcelFileStrategy.java to download file either in CSV or Excel format. But you may need your own custom file writing strategy in following cases

  • Need to export file in a different format such as PDF or RTF
  • Need to customize excel sheet column styles
  • Need to externalize file header column names in a properties file
  • Rather than downloading the file, you may need to dump a file at a given location.
  • By default the order of columns in file would be as per the order of properties defined in bean, in case you need to change the default order.
  • Or any other reason as per your need

So you can define any new strategy of file writing simply by implementing FileWriterStrategy.java interface. One such custom strategy is given in the examples, ExternalizedHeaderLabelsDumpCSVStrategy.java as follows.

And then use above strategy in your file export as follows.

The order of columns in the file can be customized as per your needs by providing a custom File writer strategy and reordering the columns headers and row data in following methods of the strategy implementation.

Type converters

The properties of bean are written in file in String format only. So it is a very basic requirement to convert the bean properties to a string in a particular format. For example you may need to format date time to a specific format while writing to file. So you can define your custom type converters as follows. All such type converters then needs to be registered in Joda StringConvert as follows. The library then automatically converts the data in a bean property depending upon the Class of property. For example by defining following type converter and registering it to global StringConvert joda converter, each property of BigDecimal type would be precised to 8 decimal points in exported file.

Similar to above type converter you may have others also. After defining all such type converters register them in global Joda converter StringConvert as follows. ExportContext expects an instance of StringConvert while exporting the file. It is recommended to have a singleton instance of StringConvert, but in case you need different converters for same class such as in one case you want to write Boolean value as YES/NO but in other case ENABLED/DISABLED, then you may need to create multiple instances of StringConvert and use respective instance as per the conversion required. You can find following type converters in source code.

Going the Reactive way

Normally the data to export is fetched from some database. If data set is small then you can just fetch a collection and pass the collection to export API. Or if the data set is large you may need to fetch the data page by page or in batches and sequentially push the data into a Flux using any of programmatically generating Flux strategy. There are some databases like Postgres or MongoDB, which have native reactive supporting JDBC drivers and you can simply get a Flux from their Spring Data repository.

Or you can programmatically push your data into Flux and pass the same to Export API as follows.

If there are multiple data sources such as some Queue, API and DB etc., then you can you Spring Reactor’s thread safe EmitterProcessor and FluxSink as follows.

The EmitterProcessor is thread safe so you can push data into FluxSink simultaneously,the same would be available in the Flux for consumption. File writing happens in single thread to retain the order of rows. As with Spring Reactor's Flux, nothing happens until the Flux is subscribed. Hence the export is only started once you call the export() method on ExportContex.

Error handling

There could be IO related exceptions or Joda mis-configuration of beans. In all error scenarios, all exceptions are wrapped into a single unchecked exception with different error codes and description message. In case any exception occurs the same is logged. You may get the error code and description from logs and take the required measures but you can not catch any of the exceptions as they occur in separate thread while processing the Flux.

Collection support

Ideally the exported file would have a fixed structure. In case the data is in different structure, you need to normalize the data as per the model to be exported. As of now no collection is supported in POJO bean except Map, that too with restrictions. Even its highly unlikely to have a collection in the file export candidate bean, because in that case you may not have a fixed number of columns in the file, which is almost always required in file export use cases. For example if data bean is supposed to have a list then you may not be able to have a fixed number of columns in the file as list content may vary for different records.

A Map collection is supported in file export candidate bean with following restrictions.

  • It is recommended be used only with a fixed number of entries otherwise the number of columns in file would be too many
  • Both Key and Value of the map must be custom class object.
  • The Key class must implement Distinguishable.java interface, overriding two methods.
  • The descriminator() method must return a unique String for each Key in the Map.
  • The label() method must return a String to be prefixed in all columns of corresponding Key's value columns.
  • The value must be a Joda bean, refer to below class for reference.

Known issues

The export candidate bean may also compose other beans till any depth. But if while file export any of the composed joda bean is found as null, then the export would fail. So you need to make sure none of the joda bean objects in the data set is null. The primitives and non Joda beans properties can obviously be null. So as given in the examples Cost.java is a joda bean composed in multiple export candidate classes, the value of Cost should never be null. If you do not have any Cost value then simply initialize it with null sell and buy values.

Resources

Find source code on Github repository joda-file-export

Know more about Joda Beans and Spring Reactor

Xebia Engineering Blog

The Xebia Engineering Blog

Rajveer Singh

Written by

Xebia Engineering Blog

The Xebia Engineering Blog

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade