Vim Script Parser written in Go

6 min readSep 20, 2016

Introduce go-vimlparser

haya14busa/go-vimlparser

go-vimlparser - :zap: Vim Script Parser written in Go

github.com

I developed go-vimlparser, which is the Vim script parser written in Go. It’s the fork of https://github.com/ynkdir/vim-vimlparser, which is the Vim script parser written in Vim script (wow!).

ynkdir/vim-vimlparser provides Vim script, Python, and JavaScript vimlparser. Vim script implementation is the original one. Python and JavaScript implementations are generated by (ad-hoc) VimL to Python/JavaScript translator.

Python and JavaScript implementation may be fast enough, but… I like golang these days and vimlparser implementation written in Go should be faster, easy to use by type support(completion, doc, etc…) and easy to provide binary tools to users for CI or editor support. So, I started to develop go-vimlparser (I listed some reason but actually it’s just for fun).

VimL to Golang (ad-hoc) translator

I didn’t have a plan to write Vim script parser from scratch, so first I needed to develop VimL to Golang translator and translate vimlparser Vim script implementation to golang one.

I could refer the Viml to Python translator to develop VimL to Golang translator and it really helped me. However, Go is typed language and isn’t flexible compared to Python nor JavaScript. So I needed to add type annotations, to write some function by hand, to run ad-hoc rewrite in translation, and to modify source Vim script implementations (as small as possible and changes are not specific for golang).

The work was little troublesome, but by incrementally implementing translations for each node and writing and running tests, I managed to develop gocompiler.vim! This is the generated vimlparser.go. Please compare it with the original Vim script implementation.

Provide Go-ish interface

The generated vimlparser.go doesn’t export types and functions, and I want more Go-ish rich interface in the first place, so I wrote some helper files to export internal interface.

I wrote go-vimlparser/ast package to provide rich node type information and use go/export.go to convert internal node type to public ast.Node type. It becomes easier to manipulate Vim AST by providing rich Go-ish interface.

Inspect node recursively

For example, you can easily traverse node by ast.Inspect. It should be useful to create lint tools, static analyzer or something.

Output:

src.vim:2:5:	s:c
src.vim:2:11:	1.0
src.vim:3:5:	X
src.vim:3:9:	F
src.vim:3:11:	3.14
src.vim:3:17:	2
src.vim:3:21:	s:c

The interface is heavily inspired by https://golang.org/pkg/go/ast/. Let me say thank you to The Go Authors for awesome works!.

Profiling and Performance Improvement

Once I finished to implement initial golang implementation, I added benchmark tests and run profiling.

In addition to the blog article Profiling Go Programs, Brad Fitzpatrick’s talk in yapc asia 2015, Profiling & Optimizing in Go is really helpful to run profiling and improve performance. I highly recommend to see the note and watch the video if you haven’t see it.

$ go test -run=”^$” -bench=”.” -cpuprofile=prof.cpu -memprofile=prof.mem
$ go tool pprof go-vimlparser.test prof.cpu
$ go tool pprof --alloc_space go-vimlparser.test prof.mem

pprof tool help shows lots of commands, but I mainly used top, list [func] and web.

From a profiling result, it seems there were lot’s of unneeded allocation and it affects the performance, so I mainly reduced allocations and the parsing speed is improved by about 2x! ref: example commit and pull-request.

The Fastest Vim Script Parser!

Thanks to the Go language speed itself and little performance optimization, go-vimlparser becomes the fastest Vim script parser!

Benchmarks

Benchmark is parsing vimlparser.vim (5195 lines) and outputting S-expression like format of parsed result. (e.g. let x = 1 -> parse -> compile -> (let = x 1))

$ pwd
/home/haya14busa/src/github.com/ynkdir/vim-vimlparser

$ git rev-parse HEAD
2fff43c58968a18bc01bc8304df68bde01af04d9

$ wc -l < autoload/vimlparser.vim
5195

$ time vim -u NONE -N --cmd "let &rtp .= ',' . getcwd()" --cmd "silent call vimlparser#test('autoload/vimlparser.vim')" -c ":q"
vim -u NONE -N --cmd "let &rtp .= ',' . getcwd()" --cmd  -c ":q"  48.88s user 0.05s system 99% cpu 48.942 total

$ python3 -V
Python 3.5.0

$ time python3 py/vimlparser.py autoload/vimlparser.vim > /dev/null
python3 py/vimlparser.py autoload/vimlparser.vim > /dev/null  4.17s user 0.04s system 99% cpu 4.236 total

$ pypy3 -V
Python 3.2.5 (b2091e973da69152b3f928bfaabd5d2347e6df46, Mar 04 2016, 07:08:30)
[PyPy 2.4.0 with GCC 5.3.0]

$ time pypy3 py/vimlparser.py autoload/vimlparser.vim > /dev/null
pypy3 py/vimlparser.py autoload/vimlparser.vim > /dev/null  2.63s user 0.06s system 99% cpu 2.694 total

$ node --version
v4.2.3

$ time node js/vimlparser.js autoload/vimlparser.vim > /dev/null
node js/vimlparser.js autoload/vimlparser.vim > /dev/null  0.77s user 0.04s system 125% cpu 0.644 total

$ go get github.com/haya14busa/go-vimlparser/cmd/vimlparser
$ time vimlparser autoload/vimlparser.vim > /dev/null
vimlparser autoload/vimlparser.vim > /dev/null  0.25s user 0.03s system 114% cpu 0.244 total

Vim script: 48.88s
Python3: 4.17s
pypy3: 2.63s
node: 0.77s
Go: 0.25s

Go implementation is the fastest!

The speed is important for I or someone to develop or use Vim script linter, code formatter, static analyzer, etc…

(Vim script is a lot slower than I expected and I’m surprized with node performance compared to Python…)

CLI tool

go get github.com/haya14busa/go-vimlparser/cmd/vimlparser

go-vimlparser provides a command which parses Vim script from stdin or files and outputs S-expression like format.

$ echo 'let x = 1' | vimlparser
(let = x 1)
$ vimlparser autoload/vimlparser.vim | head -n 5
; vim:set ts=8 sts=2 sw=2 tw=0 et:
;
; VimL parser - Vim Script Parser
;
; License: This file is placed in the public domain.

As a Lint Tool

You can use it to detect syntax error. It doesn’t check much things compared to existing lint tools for Vim script, but it runs fast even if you pass lots of files to it.

$ vimlparser **/*.vim > /dev/null
test/test_err_funcarg_duplicate.vim:1:20: vimlparser: E853: Duplicate argument name: b
test/test_err_funcarg_firstline.vim:1:14: vimlparser: E125: Illegal argument: firstline
test/test_err_funcarg_lastline.vim:1:14: vimlparser: E125: Illegal argument: lastline
test/test_err_funcarg.vim:1:44: vimlparser: E125: Illegal argument: a:bar
test/test_err_funcname.vim:11:10: vimlparser: E128: Function name must start with a capital or contain a colon: foo
test/test_err_toomanyarg.vim:1:9: vimlparser: E740: Too many arguments for function
test/test_err_varname.vim:1:5: vimlparser: E461: Illegal variable name: foo:bar
test/test_issue16_err_line_continuation_lnum2.vim:3:9: vimlparser: E488: Trailing characters: z
test/test_issue16_err_line_continuation_lnum.vim:2:9: vimlparser: E488: Trailing characters: z
test/test_neo_tnoremap.vim:1:1: vimlparser: E492: Not an editor command: tnoremap <Esc> <C-\><C-N>
test/test_noneo_tnoremap.vim:1:1: vimlparser: E492: Not an editor command: tnoremap <Esc> <C-\><C-N>
test/test_xxx_colonsharp.vim:2:6: vimlparser: unexpected token: :
test/test_xxx_err_funcarg_space_comma.vim:19:14: vimlparser: E475: Invalid argument: White space is not allowed before comma

For Vim

" sample
command! LintVimLParser :silent cexpr system('vimlparser ' . expand('%') . ' > /dev/null')
augroup lint-vimlparser
  autocmd!
  autocmd BufWritePost *.vim LintVimLParser
augroup END

I found and fixed some syntax errors in Vim runtime files and plugins I use. Example:

go-vimlparser is fast and it takes 12 seconds to run command for all vim plugins files (250621 lines) in my environment.

$ wc -l ~/.vim/.dein/repos/**/*.vim
250621 total
$ vimlparser ~/.vim/.dein/repos/**/*.vim > /dev/null  12.42s user 0.20s system 111% cpu 11.326 total

Contribution to original vim-vimlparser

Through the development of go-vimlparser, I found some bugs and support new syntax in Vim 8.0, Lambda feature. By implementing vimlparser in Go, I could find some typos, measure code coverage, can implement new feature and debug it easily (I can write new code in go implementation first with helps by golang tools and port the changes to original Vim script implementation).

I enjoyed go-vimlparser development!

I like both Vim and Go. It was really fun to see Vim script parser implementation written in Vim script(!) through the development and know how awesome @ynkdir’s work is. It was fun to read and port go/ast related interface and I can learn Go development practice and good Go-ish interface. It was fun to translate Vim script to Go and develop the fast Vim script parser.

I’m really happy if you enjoy the go-vimlparser possibility, develop interesting tools using go-vimlparser and make Vim script development ecosystem more healthy.

Next…

I’ll try to develop vimfmt, Vim script version of gofmt, by using go-vimlparser. I think it will be somewhat difficult and challenging task, and it might be useless tools if the interface go wrong. Anyway, I’ll continue to maintain go-vimlparser and try to develop some tools.

Thank you for reading this.

Happy Vimming and enjoy writing Go!