Vim Script Parser written in Go
I developed go-vimlparser, which is the Vim script parser written in Go. It’s the fork of https://github.com/ynkdir/vim-vimlparser, which is the Vim script parser written in Vim script (wow!).
VimL to Golang (ad-hoc) translator
The work was little troublesome, but by incrementally implementing translations for each node and writing and running tests, I managed to develop gocompiler.vim! This is the generated vimlparser.go. Please compare it with the original Vim script implementation.
Provide Go-ish interface
The generated vimlparser.go doesn’t export types and functions, and I want more Go-ish rich interface in the first place, so I wrote some helper files to export internal interface.
I wrote go-vimlparser/ast package to provide rich node type information and use go/export.go to convert internal node type to public ast.Node type. It becomes easier to manipulate Vim AST by providing rich Go-ish interface.
Inspect node recursively
For example, you can easily traverse node by ast.Inspect. It should be useful to create lint tools, static analyzer or something.
The interface is heavily inspired by https://golang.org/pkg/go/ast/. Let me say thank you to The Go Authors for awesome works!.
Profiling and Performance Improvement
Once I finished to implement initial golang implementation, I added benchmark tests and run profiling.
In addition to the blog article Profiling Go Programs, Brad Fitzpatrick’s talk in yapc asia 2015, Profiling & Optimizing in Go is really helpful to run profiling and improve performance. I highly recommend to see the note and watch the video if you haven’t see it.
$ go test -run=”^$” -bench=”.” -cpuprofile=prof.cpu -memprofile=prof.mem
$ go tool pprof go-vimlparser.test prof.cpu
$ go tool pprof --alloc_space go-vimlparser.test prof.mem
pprof tool help shows lots of commands, but I mainly used top, list [func] and web.
From a profiling result, it seems there were lot’s of unneeded allocation and it affects the performance, so I mainly reduced allocations and the parsing speed is improved by about 2x! ref: example commit and pull-request.
The Fastest Vim Script Parser!
Thanks to the Go language speed itself and little performance optimization, go-vimlparser becomes the fastest Vim script parser!
Benchmark is parsing vimlparser.vim (5195 lines) and outputting S-expression like format of parsed result. (e.g. let x = 1 -> parse -> compile -> (let = x 1))
$ git rev-parse HEAD
$ wc -l < autoload/vimlparser.vim
$ time vim -u NONE -N --cmd "let &rtp .= ',' . getcwd()" --cmd "silent call vimlparser#test('autoload/vimlparser.vim')" -c ":q"
vim -u NONE -N --cmd "let &rtp .= ',' . getcwd()" --cmd -c ":q" 48.88s user 0.05s system 99% cpu 48.942 total
$ python3 -V
$ time python3 py/vimlparser.py autoload/vimlparser.vim > /dev/null
python3 py/vimlparser.py autoload/vimlparser.vim > /dev/null 4.17s user 0.04s system 99% cpu 4.236 total
$ pypy3 -V
Python 3.2.5 (b2091e973da69152b3f928bfaabd5d2347e6df46, Mar 04 2016, 07:08:30)
[PyPy 2.4.0 with GCC 5.3.0]
$ time pypy3 py/vimlparser.py autoload/vimlparser.vim > /dev/null
pypy3 py/vimlparser.py autoload/vimlparser.vim > /dev/null 2.63s user 0.06s system 99% cpu 2.694 total
$ node --version
$ time node js/vimlparser.js autoload/vimlparser.vim > /dev/null
node js/vimlparser.js autoload/vimlparser.vim > /dev/null 0.77s user 0.04s system 125% cpu 0.644 total
$ go get github.com/haya14busa/go-vimlparser/cmd/vimlparser
$ time vimlparser autoload/vimlparser.vim > /dev/null
vimlparser autoload/vimlparser.vim > /dev/null 0.25s user 0.03s system 114% cpu 0.244 total
- Vim script: 48.88s
- Python3: 4.17s
- pypy3: 2.63s
- node: 0.77s
- Go: 0.25s
Go implementation is the fastest!
The speed is important for I or someone to develop or use Vim script linter, code formatter, static analyzer, etc…
(Vim script is a lot slower than I expected and I’m surprized with node performance compared to Python…)
go get github.com/haya14busa/go-vimlparser/cmd/vimlparser
go-vimlparser provides a command which parses Vim script from stdin or files and outputs S-expression like format.
$ echo 'let x = 1' | vimlparser
(let = x 1)
$ vimlparser autoload/vimlparser.vim | head -n 5
; vim:set ts=8 sts=2 sw=2 tw=0 et:
; VimL parser - Vim Script Parser
; License: This file is placed in the public domain.
As a Lint Tool
You can use it to detect syntax error. It doesn’t check much things compared to existing lint tools for Vim script, but it runs fast even if you pass lots of files to it.
$ vimlparser **/*.vim > /dev/null
test/test_err_funcarg_duplicate.vim:1:20: vimlparser: E853: Duplicate argument name: b
test/test_err_funcarg_firstline.vim:1:14: vimlparser: E125: Illegal argument: firstline
test/test_err_funcarg_lastline.vim:1:14: vimlparser: E125: Illegal argument: lastline
test/test_err_funcarg.vim:1:44: vimlparser: E125: Illegal argument: a:bar
test/test_err_funcname.vim:11:10: vimlparser: E128: Function name must start with a capital or contain a colon: foo
test/test_err_toomanyarg.vim:1:9: vimlparser: E740: Too many arguments for function
test/test_err_varname.vim:1:5: vimlparser: E461: Illegal variable name: foo:bar
test/test_issue16_err_line_continuation_lnum2.vim:3:9: vimlparser: E488: Trailing characters: z
test/test_issue16_err_line_continuation_lnum.vim:2:9: vimlparser: E488: Trailing characters: z
test/test_neo_tnoremap.vim:1:1: vimlparser: E492: Not an editor command: tnoremap <Esc> <C-\><C-N>
test/test_noneo_tnoremap.vim:1:1: vimlparser: E492: Not an editor command: tnoremap <Esc> <C-\><C-N>
test/test_xxx_colonsharp.vim:2:6: vimlparser: unexpected token: :
test/test_xxx_err_funcarg_space_comma.vim:19:14: vimlparser: E475: Invalid argument: White space is not allowed before comma
command! LintVimLParser :silent cexpr system('vimlparser ' . expand('%') . ' > /dev/null')
autocmd BufWritePost *.vim LintVimLParser
I found and fixed some syntax errors in Vim runtime files and plugins I use. Example:
go-vimlparser is fast and it takes 12 seconds to run command for all vim plugins files (250621 lines) in my environment.
$ wc -l ~/.vim/.dein/repos/**/*.vim
$ vimlparser ~/.vim/.dein/repos/**/*.vim > /dev/null 12.42s user 0.20s system 111% cpu 11.326 total
Contribution to original vim-vimlparser
Through the development of go-vimlparser, I found some bugs and support new syntax in Vim 8.0, Lambda feature. By implementing vimlparser in Go, I could find some typos, measure code coverage, can implement new feature and debug it easily (I can write new code in go implementation first with helps by golang tools and port the changes to original Vim script implementation).
I enjoyed go-vimlparser development!
I like both Vim and Go. It was really fun to see Vim script parser implementation written in Vim script(!) through the development and know how awesome @ynkdir’s work is. It was fun to read and port go/ast related interface and I can learn Go development practice and good Go-ish interface. It was fun to translate Vim script to Go and develop the fast Vim script parser.
I’m really happy if you enjoy the go-vimlparser possibility, develop interesting tools using go-vimlparser and make Vim script development ecosystem more healthy.
I’ll try to develop vimfmt, Vim script version of gofmt, by using go-vimlparser. I think it will be somewhat difficult and challenging task, and it might be useless tools if the interface go wrong. Anyway, I’ll continue to maintain go-vimlparser and try to develop some tools.
Thank you for reading this.
Happy Vimming and enjoy writing Go!