Appresso: Building a World-Class Data Integration Platform
Developers of popular enterprise application integration (EAI) middleware reveal the secrets behind their product’s success and longevity.
The Japanese enterprise middleware market is a notoriously hard one to crack. To get the inside scoop on what it takes to build a successful product, we interviewed three key development team members at Tokyo-headquartered Appresso Corporation about their journey to build middleware optimized for today’s era of cloud, big data, and IoT.
Their product, DataSpider Servista, is known for enabling the integration of a comprehensive range of data sources without the need for coding. Its many adapters can connect a wide variety of systems, applications, databases, and other data sources hosted either on-premise or in the cloud. It can be used to automate a broad range of processes, thereby significantly improving an organization’s operational efficiency.
Used by over 3,000 companies, DataSpider Servista has been ranked №1 for Customer Satisfaction in the Japanese market for the past four consecutive years by Nikkei BP Consulting’s annual report on enterprise application integration (EAI) software.
We interviewed the following members of the Appresso’s Development Division (first name, last name):
- Katsuhiko Azuma, Deputy Director
- Takumi Toki, Development Manager
- Jung-su Kim, Quality Assurance Manager
This transcript has been translated from Japanese and edited for clarity.
First, to give our readers some context, please tell us about your company.
Azuma:
Sure. It’s a bit complicated, but here goes:
- We work for a company called Appresso in Tokyo that develops and supports various enterprise software products. We’re a venture business that started in 2000 and grew to become well-known in the Japanese enterprise market for tackling challenging IT infrastructure issues.
- In 2013, we were acquired by Saison Information Systems, one of Japan’s major systems integration service providers. They also have a successful enterprise software business and bought us because our product portfolio complements theirs. We’ve since made our products fully compatible. Their HULFT business division now handles sales of our products.
- Saison Information Systems is owned by Credit Saison, is a financial services company that is a household name in Japan thanks to their consumer credit card business.
- They are affiliated with the Mizuho Financial Group, the second largest financial services group in Japan.
I guess you could say that this kind of structure is relatively common in the Japanese corporate world.
Give us an overview of your product, DataSpider Servista.
Azuma:
Well, in a nutshell, it is software that lets you connect various systems and have them share data. Please allow me to explain.
This kind of software is increasingly necessary. As enterprises grow over the years, they acquire businesses, build new businesses in-house, and cultivate working relationships with partners. Of course, in an ideal world, each party would use the same technologies for each system (i.e. platforms, file formats, data encoding standards, databases, and software, etc.) so that they could talk to each other seamlessly. However, such standardization is not universal.
In an ideal world, we would migrate everything to a single system or uniform set of standards, but this is seldom feasible in reality. A far easier approach is to use middleware to integrate them. This is the role our product, DataSpider Servista (DSS) performs. The first version was released in 2001. Since then, we’ve developed over eighty adapters compatible with major data destinations. For example, IBM Notes, kintone, Salesforce, and SAP applications can all work together the way you need them to.
Major cloud services like Microsoft Azure and Amazon Web Services (AWS) interact smoothly with DSS, too. Plus, we’re at the forefront of the latest services and applications on offer, so that we can continue to provide the most needed connecting adapters.
Mostly, DSS automates the process of coding in the Java programming language. The UI lets you easily set up data connections and automated processes by simply dragging-and-dropping icons. It can save many hours, dramatically reduce the need for coding skills, and automatically produce high-quality code. Then, later, if something needs to be changed or adjusted, it’s relatively quick and easy. It’s much more flexible, and you don’t need to bring in a coding wizard each time. It also means that if ownership of the project needs to change, the hand-over is much smoother because you don’t need to know the code inside-out.
DSS remains flexible when working with various databases, protocols, and file formats. It’s just as easy to cancel or reroute a connection with an adapter as it is to set one up in the first place.
Some customers must be concerned that it might modify existing systems.
Azuma:
Yes, that’s certainly an issue that smart CIOs are mindful of when selecting a middleware product.
While we take a modern approach to software development, we are very conservative in our approach because we’re dealing with mission critical installation scenarios. We’re handling data, which is the lifeblood of any modern enterprise. From the very beginning, we’ve built DSS from the ground up to ensure it plays by the rules and doesn’t modify existing systems. For example, when connecting with Salesforce, it follows all of Salesforce’s standard protocols.
How has the product evolved over the years?
Azuma:
Our software is well over fifteen years old. Usually, when software changes over such an extended period, the source code becomes messy. However, we pride ourselves on having clean and optimal code. It’s one reason for the high performance and reliability of our software. It also enables us to remain nimble and continuously optimize the product to keep up with the times.
Our mission for DSS has always been for it to serve as a platform that improved an enterprise’s operational efficiency through facilitating data integration at any scale.
When implementing an integration using DSS, there are three overall phases: Design, Development, and Testing. We are always seeking feedback from our customers and sales partners on how to improve and add value to these phases.
What enhancements to the Design phase will the latest version bring?
Azuma:
In this most recent update for DSS, the design phase now allows users to calculate how much time they will save by implementing an integration. It will also output a detailed quotation giving specifications of the integration and an estimate of hours required to implement the overall project. This ability to generate estimates for large development projects before starting them is invaluable.
Toki:
There’s an interesting backstory for how we came up with this functionality. For years, our customers had been manually counting the icons to try and calculate the expected development hours. Then one customer came up with a hack — they would select all the icons, click “Delete,” and then it would say, “Are you sure you want to delete __ icons?” That way, they could get the exact number immediately. This gave us the push we needed to develop the project estimation feature. But, we didn’t stop at only giving them a count, we created much more detailed and practical output.
What enhancements have you added to the development phase?
Azuma:
Overall, our goals with the development phase are to empower our users to be more efficient and produce optimal code for their needs. In this release, we’ve added three major features: Script Version Comparison, Global Custom Logic , and Output Mapper Processing Data Log.
How does the Script Version Comparison feature help?
Azuma:
This allows comparison of two versions of a script to see the differences easily. We use color codes such as green for additions, orange for updates, and gray for deletions to make it easier to understand visually. This improves test efficiency by narrowing down the range to be tested, and identifying the cause when problems occur.
The larger a project is, the more rigorous the management needs to be. For example, if you move an icon slightly and save it, an updated version of the script will be generated. The developer will say, “Since I just changed the position of the icon, the content of the process has not changed, so there’s no need to test it.” But, without that evidence, the project leader has no choice but to say, “Well, since the version number has changed, you’ll need to test it just to make sure.”
If they’d had this new revision comparison feature, it would have been easy for the project leader to see that only the x and y axes of the icon had changed, and so additional testing would be unnecessary. This is one way it makes things more efficient.
How does the Global Custom Logic feature help?
Azuma:
This allows users to define a set of logic once and then use it throughout their project. Then, to make changes, you just update one instance, and all other instances are automatically updated. We had this before, but have greatly enhanced it with the latest update.
Toki:
It’s not always obvious what we should focus our development resources on. For example, our former Local Custom Logic was well received and saved our customers a lot of time. However, we knew there must be scope for making things more efficient. From this, we came up with the scheme for our Global Local Custom Logic functionality for the latest version.
How does the Output Mapper Processing Data Log feature help?
Azuma:
It basically makes it easier to find bugs and fix them quickly. Users can identify the cause when an error occurs in the Mapper and keep track of whether the configuration you set up is behaving as expected.
How about the testing phase?
Azuma:
The DSS product’s evolution over the years has reflected the input we have received directly from developers involved in large integration projects. As a company, we seek to alleviate our customers’ pain. The key is knowing what questions to ask and interpreting their issues into action points that we can undertake to produce solutions. By taking a hard look at customers’ problems, we could come up with solutions and implement them as features.
For example, our customers told us, “Thanks to DSS, our development time has been halved, but we’re still spending many hours on testing.” This helped us reorganize our development priorities for this latest update.
While testing is, of course, critical to the success of any integration project, it can be laborious and inefficient. Our aim is to save our users time while helping them find and resolve issues before deployment.
In the latest version, we introduced a testing framework. It’s not a single function, but rather is a generic term for multiple functions and guidelines that support the testing of scripts developed with DSS. You can automate testing scripts by building tests according to guidelines.
This saves the user a lot of time by sparing them from having to develop their own test methods. We’ve designed it in such a way that the content of the test and its results are reported in an easy-to-understand manner.
You can easily repeat the same test as many times as you need when adjustments are made, or you get change requests. The test results are automatically output as an Excel and can be audited.
It also has a function to check the consistency of data, which helps remove human error.
Toki:
When we talk to Java developers and want to explain how it works, we tell them it’s like JUnit. i.e. It’s a framework that collectively provides various functions supporting the creation of automatic tests and reporting functions.
Overall, our testing features automate processes that would otherwise take many hours to perform manually. This means that if you make changes, you can rerun the same test immediately to ensure quality.
Kim:
We’ve brought our years of experience with software quality assurance (QA) into these tools.
To what degree do your customers’ requests dictate the direction of your product development?
Azuma:
Our customers share a lot of details of their projects. Many of them have been with us for years, and we’ve developed strong relationships built on trust. We’re very grateful for the level of collaboration they enable.
However, we don’t just automatically implement features randomly based on one-off interactions. We take the time to really understand where they are coming from and see things from multiple perspectives. We also must take a holistic view and carefully prioritize the development of new features and making improvements to existing ones. It’s tough to please everybody all the time, but overall, this approach has been working well.
Toki:
During the process of digging into the customer’s issues, we sometimes notice things that our customer had overlooked. We’re careful to ask them, “What do you want to achieve?” Rather than, “What kind of features do you want us to build?”
Kim:
By listening carefully to our customers, we can get a fresh perspective on the many facets of our product and work to improve it with each iteration.
How does your team approach the development process?
Azuma:
For the past few years, we have used the Scrum approach. It’s an iterative and incremental agile software development framework for managing product development. For this latest DSS version update, we created a team of nine members.
We run one Sprint per week, releasing the functions developed in small units internally for review. For this in-house review, anyone can participate regardless of their department. We ask for input from our colleagues in our sales, pre-sales, and support teams. We repeat this development loop, immediately reflecting the feedback we got to improve the product.
Toki:
The advantage of this approach, as compared to the more traditional waterfall model, is that we can set development periods, called “sprints,” and release updates more frequently. This allows us to get feedback from our customers, which we then incorporate into further changes and enhancements. We’re much more efficient thanks to this method.
Of course, we also involve our sales and support departments closely since they are communicating with customers regularly and receiving a lot of constructive feedback through their respective channels.
Kim:
In past years, we would wait until the entire development process was complete and then enter into an internal review process. WE would inevitably come up with various issues, but it has harder to make changes. With the Scrum approach, quality management department has become easier, and our development costs have been reduced. The test phase is smoother and we can make fixes immediately.
Were the one-week sprints tough to pull off?
Azuma:
They were certainly challenging to begin. We had to devise a way to assemble a schedule with a volume of development tasks that could be completed in time for our weekly internal review. It was like assembling a puzzle each week.
We would write tasks that we expected would take one hour or less on sticky notes and arrange them on a wall. When a task was completed, it would be moved. To save our developers from having to run back and forth to the wall, we had designated team members acting as runners.
We used this analog method for managing small tasks since it gave us good visibility. However, for longer term and more complex tasks, we had to use a project management tool.
Toki:
Each week-long sprint consists of five business days. Within these, four are used for development and testing. The remaining day is devoted to reviewing and planning what actions we can take to increase productivity during the next sprint. We prepare a rigorous work plan for each day.
In recent years, an increasing number of top development teams are adopting one-week cycles because the shorter feedback loop makes them more effective.
If you sprint in a short cycle, it becomes a bottleneck when people’s desks are too far away from the wall. We have our team members gather nearby to make an environment where we can talk with each other as soon as something comes up.
Although it is focused and challenging work, there was also a sense of accomplishment each time we complete a task and get to move one of the sticky notes. Completed tasks are then summarized at the end of the day in a Google spreadsheet, shared, and confirmed during the morning meetings. We also discuss these management spreadsheets and data from various sources to get more perspectives on the status of the project. We’ve tried different things to make the process more enjoyable such as making the wall more like a baseball game scoreboard and so on.
Kim:
We introduced standing desks in our office, which helped productivity. It also made it easier to gather for impromptu team meetings.
What are your plans?
Azuma:
We will continue to seek feedback from our customers on how we can make our product even easier to use. This is the driving force behind our development efforts.
Toki:
Yes. We want to stay focused on developing features that create value for our customers. For new customers, we’d like them to try our product and give us their feedback.
Kim:
Within my specialty of quality control, I’d like to further enhance our product’s ability to run tests for large projects and make it more productive for users.
