Revelation Software’s OpenInsight 10 — What have you been doing for 8 years?

People usually like being right…but not me, not this time :-(

Something like 8–9 years ago, Mike came to Sydney running a conference about this amazing new OpenInsight version 10 update in the works and coming “soon”. He was quick to demonstrate all its bells and whistles modern interfaces and external database connectivity, explaining that moving to its new 64-bit architecture would improve speed and reliability, even going to re-support the Linux Linear Hash driver that was seemingly scrapped back in version 3.0.

All the buzzword of the period were flying around the room, everyone was excited and it all sounded amazing, finally OpenInsight was coming out of the dark ages, lets do this!

Today I am Beta testing OpenInsight 10, yes, some 8 years later it has finally made it what might be described as a Beta, however my verdict is still out on that. To tell you the truth I am quite disheartened and disappointed when I was hoping on a whim and a prayer that it was all going to be worth the wait, at the same time entering into it knowing full well that the likelihood of being a disappointment was quite high.

First attempts at migrating our application across by the methods explained in the document as well as by a standard Application Backup and Restore and then trying to hash it out when it didn’t work have been poor failures thus far hitting roadblocks at every turn.

Fundamental core level cracks seem to be quite apparent in what seems to be an attempt to piece back together a major restructure of the entire platform after tacking in new features left right and centre in a hope to improve its visual aesthetic (and hopefully more than just that).

Are you ranting?

Yes, I most certainly am, I am a frustrated developer and business owner that has been waiting for this promise of features and tools to come through for 8 years that is now feeling trapped by an outdated development environment that I can’t develop or obtain any improvement with that we have our 20+ year old application written in that would require hundreds of thousands of dollars to re-write and probably not a lot of assistance from the the platform developers to improve so it feels like i’m just sort of pissing money into the wind waiting for a miracle.

tl;dr: First impressions last, and mine has unfortunately been anything but good so far, these opinions expressed are my own, and others may have their own opinion. For some additional context, my frustrations extend way past the agonising wait for OpenInsight 10's “fix it all” features.
So having said all that, apologies in advance if it comes across as rude, these are my current experiences and I am more than happy to change my opinions if you can show me solutions or prove me otherwise but I am going to write this knowing that there are others out there in the same boat that are not or can not saying anything in the hope that maybe something good will come of it for the all the end-users out there.
To this day…even through reaching out for assistance, updates, advice, even offering freely and fully my services to assist them with development expecting nothing in return other than being able to use these in my own application and letting them provide that to their paying customers…everything has seemingly fallen on deaf ears or not interested.

The rumour mill is full of talk about internal control issues on the project itself, or that “they are trying to perfect it before it is released, thats why its taking so long”, even bets on if it would ever even eventuate or if it was just a pipe-dream…of which only those in the know would know if any are fact, but Revelation is a very closed-shop when it comes to any discussion of the software/future developments. In fact, even when I have tried to contact them, they blindly refer my call to my local reseller.

What would be awesome for OpenInsight?

If version 9 is now superseded by the impending release of version 10, why not create a semi-open source version “9C” or similar community edition of it, make it an applicant based group of developers that can work on and submit features and put the participants in the program under a mutual NDA so that everyone is covered, at the end of the day, participants get the benefit of making the bits they need to work better, and OpenInsight not only gets input, participation, and free coding and ideas from participants for features they actually want and need the product advances with features that it may not before have had or work as well as they do now…would almost be a ”WinWin Solution” as you might like to call it ;)

It might not work, or it might advanced rapidly in a short space of time and include things that you could implement into version 10 that you never thought or had the resources to do before…some of the developers out there may just surprise you…you have a willing community, why not use it?

The Beta

OpenInsight 10 boasts a totally new IDE environment stemming its inspiration from the likes of Microsoft’s Visual Studio product, scrapping the traditional development IDE which everyone has grown to know since OpenInsight made its move to Windows from the DOS days of Advanced Revelation (ARev).

OpenInsight 10 — Development IDE
OpenInsight 9.4 — Development IDE

Was this necessary? To be honest, I don’t think so. It is quite inconsistent and certainly not refined, others may beg to differ, but certainly not what I would expect for over 8 years of development to receive a Beta that looks and feels like it does today.

All the bells and whistles that were shown off back at that conference have pretty much remained that for the period, since then things have changed, in a BIG way in the years since, technologies have advanced and shifted and from what I can tell, this new release puts them up to a standard that is current for that period of 8 years ago, maybe even 4–5 years ago, so now they need to basically work again to get up to current by fixing all the things that remain the same issues that we were having in versions 9, 8 and even 7.

This development platform boasts our business’s primary investment, and income, with hundreds of thousands, maybe even millions of man-hours and lines of code going into developing our application atop this platform over more than 20 years. While we always knew OpenInsight was behind the technology trends, we are paying developers that try to support and develop the system further, not just for us, but for itself. It is sad to say it but it is completely evident now that the platform is the biggest bottleneck in our business causing us a huge loss in potential, interleaving with current technologies and as a result income.

I have managed to hack my way to get modern technologies to work with OpenInsight over the years, managing real-time native mobile apps and currently GraphQL data services, but they still lack the abilities of any sort of scaleable speed and reliability compared to other solutions when it comes to selecting and reading/writing records.

From what I have experienced so far with this new Beta release, my worst fears are apparent that it is now potentially going to cost us hundreds of thousands of dollars in labour to basically start at the beginning of the application performing testing and tweaking, fixing and rewriting the system to suit this new version, and it is probably then still going to have the same or similar issues as before OR its years off completion and we’re stuck with the issues we currently have in version 9.

Now I know full well that I am one of the younger developers that use OpenInsight, at 33, I would say that the average age of developers on this platform could potentially be double that based on past user conferences…

My current development stacks outside of OpenInsight tend to rely on Linux, web and cloud based technologies, modern languages as well as large scale databases; most of which being open-source and having arisen at scale in the last few years. Look at the likes of Pokemon Go’s GoChat running on AWS’s DynamoDB scaling to 500,000 users on $100 worth of hardware in 5 days.

The power of using specialised components written by specialists companies and combining them together to create an environment that is fast, secure and reliable means that you are not relying on the one “jack-of-all-trades” environment to fix a bug in their database system because they are busy trying to create a patch for a new version of a front end that is being released and only 3 clients are having the issue anyway so they can wait…its an ecosystem that interleaves and advances everyone while everyone is doing what they are good at.

Don’t try and do everything well, thats just not feasible.
Do one thing well…then another…then another…and so on…

The issues and concerns most developers I believe are deeper seeded under the hood than the look and feel of the development environment or its fancy (not so fancy) web server. I think you would find that developer and even end-users would be totally fine with the older-fashioned look if you could improve the speed 10 fold, provide some external data access (internal and external) at its core and throw in front of it an abstracted data layer that some external Node, Go, PHP or even Python web server could feed off to serve the web requests because it is tried and tested, they are good at it and why do you need to re-create it when it works great?

Revelation’s core product in my opinion is its MultiValue database system, namely its interesting spin on MultiValue fields combined with calculated columns and relational indexes and its native manipulation of this data inside its programming language.

Well nowadays there are a lot of database systems that title themselves as “multi-value” and do just that, so what is OpenInsight edge in the current market of multi-value? Is that why they are trying to vamp up the UI front end and promote it as a multi-value Microsoft Access?

The competitive edge in my mind is probably the calculated columns, and the SQL-like select statements of the indexes and its Basic language syntax. It works okay, but it isn’t ageing well, and its slow in comparison. Connectivity, agile development, web and distributed services reign supreme in todays ecosystem, and this is where it falls behind; its client-server architecture and aged core programming generating speed issues, it’s Windows only limitations and data throughput to run across a network limiting it running in anything past a Terminal Services solution to a remote site.

Now while we all use the system and structure our data in different ways, discussions with multiple users combined with my own experiences from using this system for over 20 years lead me to speed issues with the database, indexes and more underlying functions that iterate there way up to increase these issues stemming from its flat file database access system, fixed size reads on the database from the ARev days, no server-side processing or selecting, and that little black-box the “Universal Driver” that somehow manages all the network traffic for the system.

I have spent weeks on end travelling down rabbit holes in an attempt to improve the speed and issues as far as I could either work out or the system would let me as a developer end-user of the platform, and my result was always the same, no matter how much I could improve the speed of components like indexing and selects, the inherent processes that occurred when selects were being run on the system or records read in meant that my speed improvements were still then followed by the slow code meaning that I would end up with either a slightly improved result, no change or slower process that before.

What Would I Like To See?

Instead of the path they have taken to produce this new version that is at the moment very unstable and potentially no more advantageous other than looks that its predecessor (which if this is the only development environment you use then it is probably leaps and bounds ahead of what you had). Why not work WITH the community of developers, new and old and work out where the biggest bottlenecks lie and improve those first, then worry about making it pretty.

The reactions and vibe I have gotten from trying to bring up these issues and find ways to solve or improve these for everyone on their Paid for developer forum for the application was very much from the OpenInsight side, “you don’t know what your talking about”, “it works fine, your doing it wrong”, “nobody else is having this issue”, “we are working on it”, “the new version will fix that” or deathly silence, and then when issues are brought up, multiple developers pop out of the woodwork to back the issues they are having or want to help fix it.

C’mon guys, you are making people pay to use your developer community and then not actually listening to their cries for help while they are drowning at the bottom of the well.

OpenInsight 9 would probably still be perfect and work fine if there was a few improvements made or access to lower level functions granted to developers to implement additional and eternal databases to the system, some sort of even basic implementation of multi-threading and more options when it comes to indexing.

Fix access speed and method for the database system

Have the database system run server-side, not solely client side, let it do the processing, get rid of what I can only assume is the fixed block read size of the database so that records can be read and written faster, or; Let us write our own base level components to implement or access data via our own systems, core implement (lower than the current examples) Redis, Mongo, Couch, DynamoDB, SQLite, MySQL, PostgreSQL, ANYTHING ELSE even!

Let us hook onto the Universal driver and get some sort of change feed that could be feed into other services to assist like ElasticSearch, Kinesis, or some sort of cache at that level, anything would improve this area.

Could I Improve It?

As an example to this I implemented an ANSI C wrapper for SQLite and combined that with some functions into OpenInsight to move the data access to SQLite by creating a MFS that interrupted the normal chain of events to read and write the data. It is about 100 lines of code and compiles into one single DLL file including SQLite and is 175kb.

The Results?

Yes, I could improve the speed and access. The solution improved read and write, selects could be performed on the data in milliseconds instead of minutes, very impressed.

The Issues?

The MFS’s required to perform the reads and writes. It still fired off a pile of other functions on the chain that slowed down the process, it’s unfortunately not at a low enough level in the environment to avoid the slowness that OpenInsights database introduces by its database system, index processing etc.

Indexes

Oh my god, talk about re-inventing the wheel it would seem. Is is a proud thing or just have to do it their way? The multitude of solutions out there that specialists have developed in the land of indexing and we’re stuck with this, just use an existing solution and stop trying to manually index the database or at least make it an option for people that want to do things quickly and efficiently.

Could I Improve It?

Again I took that implementation above of the ANSI wrapper for SQLite, created some more programs that imitated the process that the standard indexer would perform in terms of writing the Key to the record and the indexed field values to the SQLite database (this included indexes on multi-value fields and uses the JSON field features of SQLite).

The Result?

No more waiting or having to update or rebuild indexes, an MFS would be attached to the table that monitored the indexed fields for the indexing system and updated the fields accordingly in the SQLite database on the fly. Calling a custom FList program with a standard SQL select statement returned the keys almost instantly.

The Issues?

It has to use an MFS, once again issues arise with the dreaded MFS, its not a low enough level of access or control to implement this system, or at least not stable enough for production yet.

The other issue is that you can only select on indexed fields, as that is all that was being stored against they key in this example, if the entire record was being stored in SQLite, any and all fields would be selectable with a standard SQL string and fields could be indexed as required.

Could I Improve It Another Way?

The second attempt lead me to a server-based approach using Sprezzatura’s Engine Server component. A lot of slowness comes from the fact that when you run a report or process that relies heavily on large datasets, the indexing, sorting and then reading of the records takes a long time, and then when you run this process across a network, the process is exaggerated ten-fold as you are now introducing the additional bottleneck of the network, not to mention choking everything in the process.

I developed a simple program I called RList_Server, passing it a file name and select query would yield the results from the server instead of passing all the information to the client. It would instead use a seperate engine server on the server that would process the result, returning the list of keys to the client machine.

The Results?

Speed improvements on the client manchines on a network, they now perform near server access speeds, selects are read in then converted into an active select by the client. There is still a few caveats to this process to consider however.

The Issues?

As I was saying there is still a few caveats, take multiple selects (reducing statements) for instance, traditionally you would select by an indexed field that was going to yield the largest reduction in record size, then you would query on that result set to reduce the list of results further. In this implementation, the server does not have that initially reduced set of keys, therefore returning a larger dataset each time that would need to be compared and reduced by the client.

Possible solutions could be made work with this process by possibly sending through a complete list of select statements to be run sequentially, however sometimes there is a requirement to get a record count after certain selects or read those records then reduce further to consider.

In Summary

I want OpenInsight to be good, and it can be, and parts of it already are, I also want to help, and so does the community…from what I can tell, Revelation don’t seem to want any help and they give the impression that they think they know what is best for the community without (from my knowledge) ever reaching out to actually see what the community wants or needs in a new version…I think they have taken on way too much with this new version and it has had a negative effect as a result.

Let’s at least get this conversation started, lets get some understanding, accountability and lets actually get somewhere with this program…please!?