Tcl — Domain Specific Language Made Easy

A recurring topic in building software tools is coming up with some configuration system. Almost every piece of real software would ship with some sort of configuration system, be it Windows Registry or nginx config files.

A very interesting pattern we always see is: software developers love to create their own configuration languages. I am not saying this is bad, but maybe we could step back and rethink before we start out to do so: do we really need to invent something? Is our piece of software that unique that no existing system could describe it?

A few years ago when I first completed my game engine prototype in C++, I started to integrate scripting to enable writing game stage and define custom events with ease. After some internet research, I eventually ended picked Tcl.

Why Tcl? Because it is extremely easy to define your own DSL (Domain Specific Language) piggybacked upon it. The interface to C is extremely simple. However there is Lua falling into the same category but I just felt Tcl could be more expressive. In the end, the scripting looks more like plain English. Here is an excerpt from one of the game stage script,

Beacon create beacon
beacon set position {200 210 110}
beacon set text "Enemy scouts"

(Re) Introducing Tcl

From Wikipedia,

Tcl (pronounced “tickle” or tee cee ell /ˈtiː siː ɛl/) is a high-level, general-purpose, interpreted, dynamic programming language. It was designed with the goal of being very simple but powerful

It seems like nothing informative. With its full expanded name, we get more information: Tool Command Language. Yes, this is a very special programming language of its own like, a language based upon commands. A command is like a function, something callable.

Let’s explore more.

In Tcl, every operation is a command. The basic structure of the command invocation is:

command_name arg1 arg2 arg3 

To use the return value of a command, use brackets,

set a [sum_all 1 2]

The above means evaluate command sum_all with arguments 1 and 2 then set the return value to variable a. As illustrated, in Tcl we use set command (every operation is a command) to assign a value to a variable.

To define a new command, use proc command,

proc proc_name args body

A quick example,

proc sum_all args {
    return [expr [join $args +]]
}

Dissecting the code above: we define a new command named sum_all, which takes variadic args. The command will sum all the numbers passed in.

Command expr will do the math evaluation of its arguments and command join will return a string from the first list argument, joined by the second argument.

You might have noticed that we don’t have any quotes around the + sign, its because in Tcl every value will be treated as a String so we could ignore quotes if there is no ambiguity. Well, there is no free lunch, this string representation for every value used to have some performance impact in earlier versions of Tcl because of type conversions but in later versions, Tcl implemented dual ported structure to store values: a value is a structure with a string representation and a native type value side by side.

No surprise, flow control structures are commands too. Let’s look at if command

if {$a == 1} {
    puts 1
} else {
    puts 0
} 

Rewriting it into one line,

if {$a == 1} {puts 1} else {puts 0}

We can see that it is merely another command call with command name if, with 4 arguments supplied!

There is a special built-in command named unknown. It will be invoked whenever an unknown command is called. A common usage would be (from Tcl Manual),

# Save the original one so we can chain to it
rename unknown _original_unknown

# Provide our own implementation
proc unknown args {
    puts stderr "WARNING: unknown command: $args"
    uplevel 1 [list _original_unknown {*}$args]
}

This makes building dynamic commands possible.

You can almost use any allowed symbols as proc names, thus the language opens a whole new horizon of defining your DSL.

Example 1: A Useless TclML

Just to demonstrate how fun it is to work with Tcl, let’s create a useless DSL, which resembles XML but does nothing except for outputting Tag names in order.

What we would like to achieve is something like this:

<tml>
  <node_a>
    <node_b>
    </node_b>
    <node_c>
    </node_c>
  </node_a>
</tml>

We would like to make the above a valid Tcl script and when fed into our Tcl interpreter to evaluate, it gives the following result,

Begin tag: tml
Begin tag: node_a
Begin tag: node_b
End tag: node_b
Begin tag: node_c
End tag: node_c
End tag: node_a
End tag: tml

Here is how to achieve it (as demo only, skipped things like namespace ).

set tagstack {}
proc unknown args {
  global tagstack
  set cmdname [lindex $args 0]
  set s [string index $cmdname 0]
  set s1 [string index $cmdname 1]
  set e [string index $cmdname end]
  if {$s == "<" && $e == ">"} {
     if {$s1 != "/"} {
         # Begin of tag
         set tagname [string range $cmdname 1 end-1]
         lappend tagstack $tagname
         puts "Begin tag: $tagname"
     } else {
         # End of tag
         set tagname [string range $cmdname 2 end-1]
         if {[lindex $tagstack end] == $tagname} {
             set tagstack [lreplace $tagstack end end]
             puts "End tag: $tagname"
         } else {
           error "no matching closing tags"
         }
    }
  } else {
  }
}

This is basically playing around with unknown . When an unregistered command is called, unknown command will be invoked. We check if the command starts with < and end with > to determine if it is a tag, then depending on the second character we can check if it s a open tag or a closing tag.

Example 2: Building lambda function support

Lambda functions are supported by most of modern languages, let’s add one to Tcl. Again having global variables is not great, we could have used upvar or namespace to make it cleaner, but it is only a simple demo.

set c 0
proc λ args {
   global c
   incr c
   set pv [lsearch $args ->]
   uplevel 1 eval [list proc lambda$c [list [lrange $args 0 [expr $pv-1]]] [list [lindex $args end]]]
   return lambda$c
}

Then let’s try it out,

[λ x y -> { puts [expr $x + $y] }] 10 12

If we do not want special symbols like λ, we could use some combination like /| (a slash and a pipe symbol)— we could almost use anything as command names!

We can assign the lambda function to a variable,

set x+y [λ x y -> { puts [expr $x + $y] }]
${x+y} 5 1

Yes, our variable name is x+y.

Example 3: Adding Immutability support

Okay let’s go even further, adding more functional styles. We want to have a new statement say let that creates immutable variables. So whenever a variable is create with let, we could not change it again with let.

set varsdict [dict create]
proc let {name eqsign value} {
    global varsdict
    if { $eqsign != "=" } {
        error "usage: let var = val"
    }
if [dict exists $varsdict $name] {
        error "Variable $name already set ([dict get $varsdict $name]). Can't mutate."
    } else {
        dict set varsdict $name $value
        uplevel 1 eval "set $name $value"
    }
}
let a = 10
puts "value: $a"
# The line below will error out. 
let a = 11

The output will look like something:

value: 10
Variable a already set (10). Can't mutate.
    while executing
"error "Variable $name already set ([dict get $varsdict $name]). Can't mutate.""
....

But apparently you could still use built-in set command to change the variables. It only works if you stick to using let command.

Summary

This is not a crash course or tutorial on Tcl instead only a quick introduction to this ~30 year old language (first appeared in 1990). Tcl is still widely used in telecomm industry for network device configuration and test automation. If you are interested, please checkout the official website www.tcl.tk and play with the official tutorial https://www.tcl.tk/man/tcl8.5/tutorial/tcltutorial.html.