Building a Mongo Browser in Pharo

by T.

This article explains how the author used GT Tools to provide a simple Mongo database browser within Pharo. It also explains how Mongo and frameworks like Voyage play together. We assume the reader to be a little bit familar with the Pharo lively programming environment already.


Pharo 4 and GT-Tools

The latest Pharo 4.0 version includes a nice new toolset called GT-Tools. The abbreviation “GT” means Glamorous Toolkit and represent a set of tools that are based on the Glamorous framework.

The basic idea behind the GT Toolkit is not only to provide new tools for but to include also a toolset that is moldable. This way tools can easily get configured to be used as part of the development environment (IDE) but also for other tasks for instance as tools within an own application. Currently there are two primary tools that are integrated:

  • GT Inspector
    An adoptable inspector browsing objects in different representations
  • Spotter
    A moldable interface to search through objects and navigate through relationships

Mongo

I guess I do not have to explain Mongo DB. It is a nice NoSQL database that allows to store documents in JSON/BSON format. Mongo’s document centric model (in contrast to the relation database model of an RDBMS) fits very well with the object representation in OO languages like Smalltalk.

To get started I first installed Mongo on my machine. I used Windows operating system here and after the installation finished it was easy to start the database daemon from the command line:

 >C:\MongoDB\bin\mongod —dbpath “C:\Databases\Mongo”

This provides a running Mongo DB instance storing databases in the given folder.

Nice — but I would like to stay within the Pharo environment even to start Mongo. Nothing easier than that: in Pharo one can click into the world menu and select Tools -> Configuration Browser and load the “OSWindows” project. This project (also written by the author) allows to get better control on the underlying operating system as part of the Operation system (OS) series for Pharo.

This way one can start Mongo simply by constructing the commandline using Smalltalk:

|executable dbPath cmd|
executable := ‘C:\MongoDB\bin\mongod.exe’ surroundedBy: $” asString.
dbPath := ‘C:\Databases\Mongo' surroundedBy: $” asString.
cmd := executable,’ —dbpath ‘, dbPath.
WinProcess createProcess: cmd

Evaluating this expression runs the Mongo DB executable as well and the mongo daemon is now also listening on localhost on the default Mongo port 27017.

Side note: if I were on Linux (Ubuntu or others) one could have loaded “OSUbuntu” or “OSUnix” from the config browser and use UnixProcess class to start Mongo.


MongoTalk — have Smalltalk with Mongo

MongoTalk is a project to communicate to a running Mongo instance from within the Smalltalk environment. The original Mongo driver for Squeak/Pharo was started in 2010 by Kent Beck. At least at that time the initial MongoTalk project for Smalltalk appeared on the SqueakSource code hosting platform. Later the project was moved to SmalltalkHub as part of a MongoTalk team.

One can load MongoTalk from the config browser as well with a few simple clicks:

As we have a Mongo DB alread running we can easily connect to the running instance on the standard port from within Pharo now:

Mongo default open

If Mongo is running on another port or server one can use:

Mongo host: 'localhost' port: 12345

We are ready to ask the Mongo object for existing databases:

Mongo default open databases

This expression will return an array of database objects. At a minimum it returns two databases by default: ‘admin’ and ‘local’.

It is also easy to add and inspect an own database object:

Mongo default open databaseNamed: 'MyDB'

As this database ‘MyDB’ is new it is still empty and without data. Lets move on to get some data in.


Voyage — Persisting some objects

While MongoTalk is the basic driver that allows you to connect to a running Mongo instance there is a nice project built on top called VoyageMongo. Voyage allows for easy persistance in Pharo by saving any regular Smalltalk object easily into a database. While Voyage itself is a database independent common layer the more specific VoyageMongo project is a project to use Voyage with Mongo DB as a backend. Behind the scenes VoyageMongo is using MongoTalk as one can easily guess.

So lets open the config browser again and loaded VoyageMongo project into the Pharo image. With the help of this framework it is easy to persist some domain objects into our new database ‘MyDB’.

The initial step to be done is to tell Voyage about the Mongo database by evaluating the following expression in a workspace:

|repo|
repo := VOMongoRepository
host: VOMongoRepository defaultHost
database: 'MyDB'.
VORepository setRepository: repo.

To save some objects we use a simple domain model here. To implement it

  1. Open a new system browser from the world menu
  2. Create a new package ‘MyKillerApp-Core’
  3. Add a class Person with two attributes firstName and sureName

We need some accessor methods (getters/setters) for the two instance variables. So no need to code these methods manually — we can directly use the Nautilus System Browser in Pharo because this tool provides a nice feature to autogenerate these methods:

Additionally we implement the following two methods on the class side of our new class Person:

 isVoyageRoot
^ true

This is a method we provide to VoyageMongo. Returning true basically means that instances of this class will be root objects in the persistence graph of Voyage and for Mongo persistency this means they will be stored in an own document collection in the database.

voyageCollectionName
^ 'Persons'

With this method we provide the name of the collection to be used by VoyageMongo. Note that in a real application scenario these two methods would be extensions to our domain class in an own persistency package, but for now let’s keep them in the MyKillerApp-Core package.

With this saving an object in the database is easy. Lets create and persist our first person, Homer Simpson:

(Person new)
firstName: 'Homer';
sureName: 'Simpson';
save

and his wife Marge Simpson as well

(Person new)
firstName: 'Marge';
sureName: 'Simpson';
save

additionally his boss Mr. Burns:

(Person new)
firstName: 'C. Montgomery';
sureName: 'Burns';
save

After evulating these expressions the objects are stored in the database. How can we see and check if this really happened?

One can easily install and use external tools like the nice Robomongo (a user interface for Mongo databases) and connect to a running Mongo instance. With such an external tool we can check that our Pharo persistence framework has created three documents for the three persons:

As they are now persisted we can use Voyage to easily query the objects back from the database in the Pharo image. As an example lets search for the members of the Simpsons family:

Person selectMany: [:each | each sureName = 'Simpson' ]

By evaluating this code in a workspace we get two result objects back: Homer and Marge. Internally the framework performs a query on the database and converts the data back into objects.


Checking with Mongo

The previous section showed how easy persistence can be in Pharo. But this article is not about Voyage persistency. We just used it to get a collection “Persons” with three documents (one for each movie character) filled into our “MyDB” Mongo database.

As we now have our database ready lets dig deeper into MongoTalk and the GT Tools. First we would like to ask Mongo which databases are now available by printing again the result of the following statement:

Mongo default open databases

which returns an Array (local MyDB admin) for the three available databases. We can see that our “MyDB” database is included.

Lets inspect the database using the following expression:

(Mongo default open databaseNamed: 'MyDB') inspect

A new inspector appears. Unfortunately this default inspector shows only some basic infos like the name of the database but not the included Mongo collections:

To retrieve the collections from the database we explicitly have to ask the “MyDB” database for the number of included collections and inspect that again in our Pharo environment:

|db|
db:= Mongo default open databaseNamed: 'MyDB'.
db collections inspect

We have only one Mongo collection so far (“Persons”) - so we can directly access it from the returned array with the #first method. This way we can again dig deeper into the structures:

|db|
db:= Mongo default open databaseNamed: 'MyDB'.
db collections first

So this last expresssion gives us direct access to the Persons collection.

But again this is not very helpful. The three documents within our Persons collection are not visible in the standard inspector as they were not loaded from the database. Which means we need again another expression to return the three includes documents by querying from the database:

|db|
db:= Mongo default open databaseNamed: ‘MyDB’.
db collections first select: Dictionary new

Sad. Even when we use the inspector it is cumbersome to browse through the objects by evaluating new expression one after the other — is’nt it? How can we solve this?


Better inspection — GT-Tools to the rescue

Is’nt there a better way to navigate and inspect these objects? Especially as we now know that we can navigate already by using Smalltalk expresssions:

  • from Mongo to the database
  • from a database to included collections
  • and from a collection to its persisted documents

Lets get deeper into GT-Tools. If you havent already done I would recommend the reader to checkout some web infos/documentation on GT-Tools first, there are various source like the official webpage or blog posts on extending the inspector itself.

Basically all GT-Tools (like Spotter or the GT Inspector) can be extended and customized easily. For our need here we have to know that we can extend the inspector representation of an object by providing a method annotated with specific pragmas.

With this knowledge the first step done is to create a new package “Mongo-Pharo-Tools” and implement the following instance side method in the Mongo class:

gtInspectorDatabasesIn: composite
<gtInspectorPresentationOrder: 40>
<gtInspectorTag: #basic>

composite list
title: 'Databases';
display: [ self databases ];
when: [ self isOpen ]

As we see this method includes some GT-Tools pragmas defining that we want to have an own inspector representation.

Lets see the effect by inspecting our Mongo instance again:

Mongo default open inspect

and magically we have a new tab called “Databases” listing all the databases:

By using the <gtInspectorPresentationOrder: x> pragma the inspector will find your custom inspector extension, the provided number x (here 40) is some kind of weight to specify the tab order compared to existing tabs.

The <gtInspectTag: aSymbol> is for grouping the extensions and will be explained later.

As you see the method has one argument that is provided from the outside: a composite view that we can ask to provide a list view. We have to provide the title and contents of the list —in this case our collection of databases.

The additional #when: in the method makes sure that this new tab is only displayed when the mongo connection is open. That means the tab will not be there when you just inspect Mongo without opening it first:

Mongo default inspect

But lets move on and inspect the Mongo DB instance again with our new “Database” tab:

Mongo default open inspect

Now when we click on one of the items in the database list the inspector shows a new pane to the right inspecting the selected Mongo database object:

Cool — with this we have our first navigation step and can already navigate from a Mongo object to an included database like “MyDB”.

That’s not enough. For the new MongoDatabase instance that is shown on the right inspector pane we now want to have a new additional tab again showing the collections of the database. Therefore we need to add another instance side method extending our inspector toolset. This time we provide a method on the MongoDatabase class:

gtInspectorCollectionsIn: composite
<gtInspectorPresentationOrder: 40>
<gtInspectorTag: #basic>

composite list
title: 'Collections';
display: [ self collections ];
when: [ self collections notEmpty ]

This tab should only be shown when we have collections in the database, therefore we check if the collection is empty or not.

Inspecting

Mongo default open

again we can now navigate:

  • from Mongo to the database
  • from a database to included collections

To implement our third and last navigation step (from a collection to its persisted documents) we again need to add another instance side method, this time it has to be on class MongoCollection:

gtInspectorDictionariesIn: composite
<gtInspectorPresentationOrder: 40>
<gtInspectorTag: #basic>

composite list
title: 'Documents';
display: [ self select: Dictionary new ];
when: [ (self select: Dictionary new) notEmpty ]

Again we get a new tab and we can now see the three documents stored in the Persons collection.

But the default representation of the three document items is not what we expect as we can not distinguish the items in the list. Robomongo is still better and displays the document id for each document — can we have that in our new Pharo tool as well? Yes!

One hast know that in Mongo databases each document has a unique object id, typically represented by the value of the “_id” key in the JSON representation.

So the only thing we have to do is to use the #format: method to provide a better string representation of the document objects when displayed as a list items:

gtInspectorDictionariesIn: composite
<gtInspectorPresentationOrder: 40>
<gtInspectorTag: #basic>

composite list
title: 'Documents';
display: [ self select: Dictionary new ];
format: [:each | (each at: #_id) asString ];
when: [ (self select: Dictionary new) notEmpty ]

Much better! Did you see the small bullets at the bottom of the window? You can use them to navigate back and forward.

So with three simple extension methods we are now able to navigate through our virtual object graph with our customized inspect. Amazing!


Give some stylish icons

Are we satisfied already? Not yet. While we can change the displayed list item text with the #format: message it would be nice to display some icons similar to the Robomongo tool.

There are nice icon collections available on the net — we would like to use icons from Eclipse as a similar icon set is already integrated into Pharo. One can access the internal Pharo icon collection by inspecting:

Smalltalk ui icons

and as we see: in Pharo 4 this inspector is also customized to show you the icons in a list in an own tab:

Exactly what we need for our own lists. From the title we see that the object displayed here is an instance of EclipseUIThemeIcons. Open it in the Smalltalk Browser — we see that it inherits from ThemeIcons class.

Yay — in this class we find the inspector extension method that provides the above icon list:

gtInspectorIconsIn: composite
<gtInspectorPresentationOrder: 40>
composite list
title: 'Icons';
display: [
self iconSelectors asSortedCollection
collect: [ :each | each -> (self perform: each) ] ];
icon: [ :each | each value ];
format: [ :each | each key ]

From this method we can easily guess that we can use an #icon: with a block as argument to provide the icon for each list element. The argument to the block is the object that should be represented by the list item — so you can even provide different icons depending on the object / object state.

But how to provide our own custom icons in the environment?

First we have to download a small *.gif or *.png file to our hard disk and then use the Pharo file browser (“Tools” -> “File Browser” in world menu) to open the file in the Pharo environment.

This will open the small picture as a morph on the Pharo desktop. You can select the Halo of the morph by using a pressing a special key while clicking the morph (on my Windows this is SHIFT + middle mouse button).

From the morphs Halo we can select “inspect morph” on the gray halo button which opens an inspector on the image morph object. Here we can use the evaluation pane in the inspector to ask the image morph for its form (using #form) object and the form itself for its string representation (by using #storeString).

One has to know that in Smalltalk a store string is a string that represents a Smalltalk expression that can be used to recreate an object. So in this case we get an expression that allows us to (re)create a color form of the iconic picture.

We can clip the resulting expression into the clipboard. This screenshot should help understanding how one can get such a string expression:

If this is too cumbersome you can use your powerful Pharo environment to solve this with a little workspace script:

Clipboard clipboardText: 
(Form fromFileNamed: ‘c:\temp\datasheet.gif’) storeString

With a Smalltalk expression in the clipboard we can now provide an own extension method on class ThemeIcons with the icon we want to use:

mongoDatabaseIcon

^ icons
at: #'mongoDatabaseIcon'
ifAbsentPut: [ (Form
extent: (16@16)
depth: 32
fromArray: #( 0 0 0 0 425091964 1029269120
... ) offset: (0@0)) ].

Note that we shortened the method here and as you see from the code the icons will be cached in a dictionary because it would costly to recreate them anytime the theme is asked for an icon. This also saves memory and too many form instances floating around. Also note that while you can put a generic icon to class ThemeIcons it is possible to provide also icons in its subclasses like WateryThemeIcons, EclipseUIThemeIcons, …

So depending on the UI theme selected in the Pharo settings (“System” -> “Settings” -> “Appearance” -> “User interface theme”) a different icon can be used.

As we now have a method providing the icon we can check the availability of the icon form:

Smalltalk ui icons mongoDocumentIcon

and use the new icon by modifying our inspector extension method in class MongoCollection:

gtInspectorDictionariesIn: composite
<gtInspectorPresentationOrder: 40>
<gtInspectorTag: #basic>

composite list
title: 'Documents';
display: [ self select: Dictionary new ];
format: [:each | (each at: #_id) asString ];
icon: [ Smalltalk ui icons mongoDocumentIcon ];
when: [ (self select: Dictionary new) notEmpty ]

After providing more icons and adopting the other inspector presentation methods in class Mongo and MongoDatabase our inspector looks much nicer:


Less is more — tags and filters

Nice — but still not satisfying. We cann add new presentations but can we remove other ones like the “Raw” and the “Meta” tab. They are not helpful and look disturbing if one just wants to use our customized inspector as a Mongo database browser.

Again GT-Tools come to the rescue: first we define an additional tag “mongo” for each of our three presentation methods in class Mongo, MongoDatabase and MongoCollection:

gtInspectorDictionariesIn: composite
<gtInspectorPresentationOrder: 40>
<gtInspectorTag: #basic>
<gtInspectorTag: #mongo>

composite list
title: 'Documents';
display: [ self select: Dictionary new ];
format: [:each | (each at: #_id) asString ];
icon: [ Smalltalk ui icons mongoDocumentIcon ];
when: [ (self select: Dictionary new) notEmpty ]

Next step is that we want to instantiate and open an own GTInspector tool taking this tag as a filter. This can be done with the help of the class GTInspectorTagFilter.

Here is a script that uses this:

 |inspector|
inspector := GTInspector new.
inspector title: 'Mongo Database browser'.
inspector presentationFilter:
(GTInspectorTagFilter new
defaultTag: #mongo;
addAndSelectTag: #mongo).
inspector openOn: Mongo default open.

Wow — only a few lines of code and we are now ready with an own Mongo Browser that is available within our Pharo image. No need to switch to an external tool.


Finalize the MongoBrowser

Using a script anytime we want to use this new Mongo tool would be very cumbersome again. We would have to remember how it was done and how this custom inspector will be constructed.

As creating new classes is free in Pharo as in “free as in beer” we just add another class to the system:

Object subclass: #MongoBrowser
instanceVariableNames: ‘’
classVariableNames: ‘’
category: ‘Mongo-Pharo-Tools-UI'

and open a class side method #open:

open
<script>
  |inspector|
inspector := GTInspector new.
inspector title: 'Mongo Database browser'.
inspector presentationFilter:
(GTInspectorTagFilter new
defaultTag: #mongo;
addAndSelectTag: #mongo).
inspector openOn: Mongo default open.

You have noticed the <script> pragma. A while ago I introduced this as a new feature in standard Pharo 4 to be able to mark unary class methods that can act as a callable script. The effect is that you can not only call this method with a regular Smalltalk expression:

MongoBrowser open

but you can also click on an icon in Nautilus to run this method:

There is another variant of this <script> pragma (that additionally works on instance side) and that accepts a string with a Smalltalk expression.

We can use in an #openOn: method where one can pass any other Mongo instance as a parameter:

openOn: aMongo
<script: 'self openOn: Mongo default open'>
    |inspector|
inspector := GTInspector new.
inspector title: ‘Mongo Database browser’.
inspector presentationFilter:
(GTInspectorTagFilter new
defaultTag: #mongo;
addAndSelectTag: #mongo).
inspector openOn: aMongo.

So one can use the MongoBrower also on other Mongo instances beside the local one and even browse databases remotely or on different ports:

MongoBrowser openOn: (Mongo host: '192.168.1.22' port: 9092) open

The other nice effect is that this reduces our initial #open method to less code because we can call #openOn: in it with the default Mongo.
After applying this refactoring it looks much cleaner:

open
<script>
   self openOn: Mongo default open

One final touch

We now have MongoBrowser as a new tool — but would’nt it be cool to make it available beside all the other nice IDE tools in the Pharo “Tools” menu?

This is easily done with another icon and a class side method on our new shiny class MongoBrowser using the pragma <worldMenu>:

menuCommandOn: aBuilder 
<worldMenu>
(aBuilder item: #'Mongo Browser')
parent: #Tools;
action: [ self open ];
icon: Smalltalk ui icons mongoIcon

Tips and Tricks

This story would be endless as one could always change the environment always one step more. In Pharo you can change nearly anything: like most Smalltalk systems the Smalltalk image is a dynamic object system that can be explored and changed all the way down. You can change and adopt the whole IDE and even the Smalltalk language— but take care such power requires knowledge and discipline.

If you want to use the power of this system and the now included GT Tools in Pharo 4 for own needs I recommend to browse through the system itself.

Check out provided example methods like in GTInspectorMethodListFilter or GTInspectorTagFilter. Learn about Glamour possibilities with included example browsers:

GLMExamplesBrowser new openOn: GLMBasicExamples

Check out the code in SUnit test cases — it also helps to understand how GT classes should work together.

Especially use the new included Pharo spotter search tool to enter and search users of the pragmas mentioned in this article. You can learn a lot from the system itself:

Covering other new tools like the Playground or the Spotter was not part of this journey — but they work similar and it should be easy to find out with the mentioned explorer facilities.


Summary

Pharo 4 will be another major step for the Pharo open source community. Beside many bugfixes, cleanups, new features like Slots, MetaLinks and many others it also includes better toolsets. Especially the GT-Tools extensions allow you to easy adopt existing tools to own specific needs — not only for development tools but also for end user applications.