This is a little write-up about the work done in June, and how it incorporates into the general roadmap for the project. Included here are also some ideas to be implemented in the near future.
1. Work with tests and CI (continuous integration) testing
The main thing with this step was looking into the tests that were used to test the NECContext of the old completion, and make them run for our completion (via CompletionContext). The reason this was the priority rather than adding more tests (and making sure all of them worked) for the CompletionEngine that we added last summer, was because the main part we are using for actual in-menu completion right now is CompletionContext. CompletionEngine was a stand-alone attempt at a completion tool to get more understanding of how completion worked, and at the moment is not so useful (more about how we do plan to make it useful in #3 of this post).
So, how was the rerunning of the tests done? First, we just made a subclass which would run the old completion tests for the new completion controller and context. Then, some of the tests were failing for the new one so we started fixing the ones where the implemented fix would help for both the old and the new completions. Later we faced a problem: fixing some of the tests (redoing them so the checks were for something which intuitively you should have in the completion) made them fail when run for the old completion. So then the idea was to copy the whole infrastructure into our own repository and fix tests there, and later decide what works best (ideally, the goal of the project is to get rid of the old completion, so the original version of the tests is to be deleted entirely).
additional info: Another thing is that while documentation has been largely up-to-date since the beginning, there was still a point in improving the documentation while working on the tests, which we did.
As for CI (we used Travis), it was successfully set up for the Github repository of the project and all the tests are being run every time a new commit is pushed. Right now, it keeps failing because we are still fixing the tests, but once the main chunk of the work is done, CI will indeed prove very useful in making sure that further progress that is made throughout the course of the project doesn’t break anything else.
2. Single model instead of Typed, Untyped and Empty models
The model class in the old completion, NECModel, is essentially one of the main points of the current code completion infrastructure in Pharo. It is basically a storage point for entries to be completed and the place where most behaviour is determined (where we are during completion, how the typing is done, how the list of entries is updated and so on). NECModel has three subclasses NECEmptyModel, NECTypedModel and NECUntypedModel. Our idea is that in the new version of the completion this can be simplified by having the one main model (CompletionModel) instead of the three, and having specific typed/untyped model behaviour determined on specific (lower) levels.
The easiest way to start with just the one model, is to make it the default for the completion (at first it will break) and keep adding methods until it doesn’t break anymore. We managed to cut down the number of methods in half and generally simplify the architecture to have the least possible workable version (playground completion is successful in most of the cases; the only problem is non-generalised labeling of the entries’ types, as in the screenshot below):
The next challenge, and a much more difficult one, remains making it work not just for playground but also for the editor (which is a separate and much more complicated case). However, for this we already have some tooling necessary, and we only have to figure out a way to make it work (which is not necessarily the easiest thing to do but should be manageable). See next paragraph for more information about this.
3. Work to do in the nearest future
The key in making the new completion work both for the playground and the editor, is to go back to the work we already did with CompletionEngine and MatchedNodeProducer last summer. As I already mentioned, it was a stand-alone attempt at code completion, for which we devised its own Spec and a set of tests, and it worked well. However, at the point we did that, we had no way of actually making it work as a native Pharo tool — for that we needed to go back and see how it was done in the old completion.
Having looked at how it was done, an idea emerged: reuse most of the functionality of the old completion, but keep eliminating unnecessary code and exchanging overly complicated code with ours. In short, merge the two in the most efficient way possible.
At the moment we have neared the limit of only eliminating things we don’t need and so now, once again, we need to find a way to attach parts of new completion to the old completion carcass. This means using CompletionEngine to see how we determined the playground/editor difference, the cursor position, method name, the list of temporary variables, etc, and MatchedNodeProducer, which does the typing that is the only typing method used in CompletionEngine (therefore, something that successfully works, as we have tested it extensively in the past).
Once that is done, we will also be able to implement another part of the plan towards simplifying the implementation — getting rid of the entries (as in, the different subclasses of NECEntry which are responsible for typing in the old completion) and only using MatchedNodeProducer.
additional info: Another idea we have for simplifying the existing completion is to get rid of OCompletion (NOC), which at the moment is an entirely separate implementation of code completion that hardly anyone uses (it’s not a default in Pharo so not everyone is even aware it exists — I sure hadn’t known about it before I started reading up on autocompletion in general and its implementations in Pharo in particular sometime in early 2018).
For the fans of OCompletion: don’t worry, with the completion architecture we are working on, the technique used in OCompletion can be implemented as simply one of the sorting options in the new completion.
Hope this glimpse into the work progress and the thought process involved in making decisions concerning how to proceed has been of interest. If you have any questions or suggestions, don’t hesitate to let me know either here, on the mailing list, on Discord or on Twitter.
P.S. this is the Github link if anyone wants to dive deep into the classes I keep mentioning in the blog post.