Vim Script Parser written in Go

haya14busa
Sep 20, 2016 · 6 min read

Introduce go-vimlparser

VimL to Golang (ad-hoc) translator

I didn’t have a plan to write Vim script parser from scratch, so first I needed to develop VimL to Golang translator and translate vimlparser Vim script implementation to golang one.

Provide Go-ish interface

The generated vimlparser.go doesn’t export types and functions, and I want more Go-ish rich interface in the first place, so I wrote some helper files to export internal interface.

Inspect node recursively

For example, you can easily traverse node by ast.Inspect. It should be useful to create lint tools, static analyzer or something.

src.vim:2:5:	s:c
src.vim:2:11:	1.0
src.vim:3:5:	X
src.vim:3:9:	F
src.vim:3:11:	3.14
src.vim:3:17:	2
src.vim:3:21:	s:c

Profiling and Performance Improvement

Once I finished to implement initial golang implementation, I added benchmark tests and run profiling.

$ go test -run=”^$” -bench=”.” -cpuprofile=prof.cpu -memprofile=prof.mem
$ go tool pprof go-vimlparser.test prof.cpu
$ go tool pprof --alloc_space go-vimlparser.test prof.mem

The Fastest Vim Script Parser!

Thanks to the Go language speed itself and little performance optimization, go-vimlparser becomes the fastest Vim script parser!

Benchmarks

Benchmark is parsing vimlparser.vim (5195 lines) and outputting S-expression like format of parsed result. (e.g. let x = 1 -> parse -> compile -> (let = x 1))

$ pwd
/home/haya14busa/src/github.com/ynkdir/vim-vimlparser

$ git rev-parse HEAD
2fff43c58968a18bc01bc8304df68bde01af04d9

$ wc -l < autoload/vimlparser.vim
5195

$ time vim -u NONE -N --cmd "let &rtp .= ',' . getcwd()" --cmd "silent call vimlparser#test('autoload/vimlparser.vim')" -c ":q"
vim -u NONE -N --cmd "let &rtp .= ',' . getcwd()" --cmd  -c ":q"  48.88s user 0.05s system 99% cpu 48.942 total

$ python3 -V
Python 3.5.0

$ time python3 py/vimlparser.py autoload/vimlparser.vim > /dev/null
python3 py/vimlparser.py autoload/vimlparser.vim > /dev/null  4.17s user 0.04s system 99% cpu 4.236 total

$ pypy3 -V
Python 3.2.5 (b2091e973da69152b3f928bfaabd5d2347e6df46, Mar 04 2016, 07:08:30)
[PyPy 2.4.0 with GCC 5.3.0]

$ time pypy3 py/vimlparser.py autoload/vimlparser.vim > /dev/null
pypy3 py/vimlparser.py autoload/vimlparser.vim > /dev/null  2.63s user 0.06s system 99% cpu 2.694 total

$ node --version
v4.2.3

$ time node js/vimlparser.js autoload/vimlparser.vim > /dev/null
node js/vimlparser.js autoload/vimlparser.vim > /dev/null  0.77s user 0.04s system 125% cpu 0.644 total

$ go get github.com/haya14busa/go-vimlparser/cmd/vimlparser
$ time vimlparser autoload/vimlparser.vim > /dev/null
vimlparser autoload/vimlparser.vim > /dev/null  0.25s user 0.03s system 114% cpu 0.244 total
  • Python3: 4.17s
  • pypy3: 2.63s
  • node: 0.77s
  • Go: 0.25s

CLI tool

go get github.com/haya14busa/go-vimlparser/cmd/vimlparser
$ echo 'let x = 1' | vimlparser
(let = x 1)
$ vimlparser autoload/vimlparser.vim | head -n 5
; vim:set ts=8 sts=2 sw=2 tw=0 et:
;
; VimL parser - Vim Script Parser
;
; License: This file is placed in the public domain.

As a Lint Tool

You can use it to detect syntax error. It doesn’t check much things compared to existing lint tools for Vim script, but it runs fast even if you pass lots of files to it.

$ vimlparser **/*.vim > /dev/null
test/test_err_funcarg_duplicate.vim:1:20: vimlparser: E853: Duplicate argument name: b
test/test_err_funcarg_firstline.vim:1:14: vimlparser: E125: Illegal argument: firstline
test/test_err_funcarg_lastline.vim:1:14: vimlparser: E125: Illegal argument: lastline
test/test_err_funcarg.vim:1:44: vimlparser: E125: Illegal argument: a:bar
test/test_err_funcname.vim:11:10: vimlparser: E128: Function name must start with a capital or contain a colon: foo
test/test_err_toomanyarg.vim:1:9: vimlparser: E740: Too many arguments for function
test/test_err_varname.vim:1:5: vimlparser: E461: Illegal variable name: foo:bar
test/test_issue16_err_line_continuation_lnum2.vim:3:9: vimlparser: E488: Trailing characters: z
test/test_issue16_err_line_continuation_lnum.vim:2:9: vimlparser: E488: Trailing characters: z
test/test_neo_tnoremap.vim:1:1: vimlparser: E492: Not an editor command: tnoremap <Esc> <C-\><C-N>
test/test_noneo_tnoremap.vim:1:1: vimlparser: E492: Not an editor command: tnoremap <Esc> <C-\><C-N>
test/test_xxx_colonsharp.vim:2:6: vimlparser: unexpected token: :
test/test_xxx_err_funcarg_space_comma.vim:19:14: vimlparser: E475: Invalid argument: White space is not allowed before comma
" sample
command! LintVimLParser :silent cexpr system('vimlparser ' . expand('%') . ' > /dev/null')
augroup lint-vimlparser
  autocmd!
  autocmd BufWritePost *.vim LintVimLParser
augroup END
$ wc -l ~/.vim/.dein/repos/**/*.vim
250621 total
$ vimlparser ~/.vim/.dein/repos/**/*.vim > /dev/null  12.42s user 0.20s system 111% cpu 11.326 total

Contribution to original vim-vimlparser

Through the development of go-vimlparser, I found some bugs and support new syntax in Vim 8.0, Lambda feature. By implementing vimlparser in Go, I could find some typos, measure code coverage, can implement new feature and debug it easily (I can write new code in go implementation first with helps by golang tools and port the changes to original Vim script implementation).

I enjoyed go-vimlparser development!

I like both Vim and Go. It was really fun to see Vim script parser implementation written in Vim script(!) through the development and know how awesome @ynkdir’s work is. It was fun to read and port go/ast related interface and I can learn Go development practice and good Go-ish interface. It was fun to translate Vim script to Go and develop the fast Vim script parser.

Next…

I’ll try to develop vimfmt, Vim script version of gofmt, by using go-vimlparser. I think it will be somewhat difficult and challenging task, and it might be useless tools if the interface go wrong. Anyway, I’ll continue to maintain go-vimlparser and try to develop some tools.

haya14busa

Written by

V!mm!shment Th!s World! https://github.com/haya14busa