Native node.js apps are totally on fire.
Today I had a harsh problem. I was trying to fix the update scripts of my lil’ engine lychee.js so that all updates of third-party runtimes can be downloaded and integrated more easily by end-users and that our lychee.js runtime repository contains all binaries and literally zero installation overhead for end users.
We don’t like doing Terminal stuff and that’s what we think is the bad part about node.js. Third-party end users need graphical user interfaces to get better efficiency — or do you create PSD files in the Terminal with running libmagick tools and ffmpeg? I guess no.
That being said, I initially tried only to fix the nw.js update script. After realizing that nw.js is now over 126MB big (former 22MB size) I realized that we can’t go the nwjs route. Electron is even bigger, I ain’t get started with that. Taking a glimpse at the screenshot, this is getting ridiculous.
I mean, for lychee.js we only need a TCP network API, a Filesystem API, a Canvas / WebGL API and the possibility to play sounds. That’s it. Literally everything else is total bloat that decreases end-users experience.
Going back the route to node.js I have a little toy project that tries to get into the state of bundling apps with node.js and libSDL. It’s harder than you might think on first sight, but I ripped partially the awesome code of node-sdl2 where the maintainer @zetsingithub is sadly not replying. I owe him much respect because he basically taught me how to use the `ffi` node module (their documentation is also completely burning down the incompatibility of cross-platform river).
Anyways, the node-sdl-runtime tries to ship a bundled ready-to-use prebuilt node.exe with all third-party modules being already installed. Children, normal humans and whatever third-party user should NOT need to install npm on their system or even try to get a proper bash running. That’s developer shit that nobody cares about in the real world.
Okay, now to the hard part: node_modules dependencies and shrinkwrap.
If you haven’t heard of npm shrinkwrap, it’s basically the idea to fix dependencies of third-party recursive dependencies of libraries if the maintainer of said library is too lazy to update their package.json.
Currently, all things native are totally broken. Anything related to char arrays is using the iconv library or, if being a linked data type in the ffi world, uses ref-wchar. Dependency chain for the modules looks like this, all relying NOT on stable semantic versions, all pinning the version to a ~specific version:
- node-sdl > ref-wchar@1.0.1 > iconv@2.1.10 > nan@too_old_version
- node-sdl > iconv@2.2.0 > nan@2.2.1
Okay, to explain what nan is: It’s basically a native library that is required pretty much everywhere as it includes the headers for v8-specific methods. That means if nan is out of date, NOTHING with node-gyp and native node.js modules will work. Only with nan@2.2.1+ the fixes for node v6.0.0+ arrived.
Let’s get this straight with semantic versioning. Major.Minor.Bugfix. That’s how the version is structured. So WHAT is the point if the whole npm ecosystem is relying on specific pinned versions and not, say on nan@2 as a version? Yes, you are right, somebody has to file over 6 pull requests just to fix a single dependency that is used in multiple libraries. Either that; or he can try getting npm-shrinkwrap.json to run, which is a bad joke as a data format. Impossible to maintain by any human, overly complex recursive structure and it looks like this for a single dependency, ALL in there being necessary (yes, you manually have to google up all current versions and have a notepad ready to note down all version differences): https://github.com/Artificial-Engineering/node-sdl-runtime/blob/master/npm-shrinkwrap.json
{
"dependencies": {
"ref-wchar": {
"version": "1.0.1",
"from": "ref-wchar@1.0.1",
"dependencies": {
"iconv": {
"version": "~2.2.0",
"from": "iconv@~2.1.10",
"dependencies": {
"nan": {
"version": "~2.3.5",
"from": "nan@~2.2.1"
}
}
}
}
}
}
}Granted, npm install automatically respects an existing npm-shrinkwrap.json file. This is good. But the hints printed out are totally useless, could’ve just echoed “Error code 0x1231321” like in the Windows days and would make no difference in debugging efficiency.
Don’t get me wrong. I’m trying to fix things. I’m not trying to rage only without constructive suggestions. The pain I had as a developer needs to be automated, so this is what I needed to do to fix shit that was on fire:
- Create a pull request for iconv to update nan which was denied because the maintainer better wants to fix this in future when it breaks again and it’s too late. I’m okay with that, but from the product perspective he should not be okay with that.
- Create a pull request for ref-wchar to update iconv dependency.
- Create an npm-shrinkwrap.json file was the hardest part (stackoverflow solutions…), but I managed to fix this locally.
So what’s the rage here? Shit is too complex to fix. That’s the quintessence and from the product perspective, NPM is a total fail here because they could not care less about their user group(developers).
How could that have been solved in the NPM world?
- Create a command like npm install --flatten-all that flattens all dependencies in the node_module folder and allows to IGNORE all recursive dependencies and sticks to enforced semantic versions
- Respect semantic versions and ignore sticky versions in dependencies. Why? Because that’s the point of semantic versioning. If NOBODY is able to automatically get security fixes of ANY library — what’s the point? Literally, semantic versioning is useless if the package manager does not enforce to use it. NPM tries to literally reinvent the shared libraries concept. They seemingly never looked up /usr/lib/*.so.oldversion symlinks on UNIX systems.
- Create an npm-shrinkwrap configuration file that respects flattened structures. That means have a text file (like in python?) with module ^sticky_major_version, separated by line breaks.
