How I found the 1740 Meteor apps (thx Fast Render)
Last week, I released some statistics of 1740 Meteor apps I’ve scraped.
People asked us how we built this list so here is the explanation.
We focused on 2 main websites that are known to contains lot of Meteor apps; madewith and spotting.
MadeWith
madewith was developed by MDG as a place where everybody can post apps made with Meteor. It displays 14 apps on the main page and there’s a “Load More” button to load more apps. It means it has a pagination system to get the app list.
After looking inside the code, We easily found there’s a mini mongodb collection called Apps on the client containing app information.
Second step, find the subscription name, in the code we see:
newAppsHandle=Meteor.subscribeWithPagination(“newApps”,14)
So I opened the browser console and change 14 to 1000 and now, we have all the data:
A small extract to get the urls from the collection with a simple javascript command and here we go:
We have 131 apps, not so bad.
Spotting
Spotting is a really cool chrome extension that warn you when you are browsing a website made with Meteor. It’s fun because there’s a small competition with other people and it’s easy to use (no need to manually post links).
As you can see on the homepage, there’s no “load more” button so we could imagine the subscription is limited by the server.
Since it’s opensource, the source code is available so it’s easy to check and confirm the intuition:
So how can we get the whole data? Fast Render!
Fast Render is an amazing package created by Arunoda that make Meteor website load and render faster.
Usually, when you load a Meteor app, the browser loads a small .html, then the unique .js and .css file generated by Meteor. Then, it creates a websocket, get the data through this websocket and finally renders the data into the DOM.
With Fast Render, the data are directly added inside the .html so the browser can render them immediately, before the websocket is created.
The problem here in Spotting is that the programmer forgot to add the limit in the RouterController so Fast Render adds all the data inside the .html sent to the client.
With all the data in the .html, it’s quite easy to parse and get all the spotted Meteor apps.
I contacted the owner of the website and the good news is that it’s really easy to fix, he just has to add the ‘limit’ option in the find().
Conclusion
As you can see, there’s nothing complex or hard to get the list.
If you use Fast Render, it’s always a good idea to check regularly which data are in the .html to be sure you don’t send things you should not.
Until 2 days ago, there was also a potential issue with Fast Render if you are using shared domains like *.meteor.com. If it’s your case, be sure to update to the latest Fast Render version to fix this issue.
FastRender is really an amazing package that make a amazing job, just be sure to use it correctly.
We’ll regularly post articles about security, nice packages, cool websites made with Meteor so don’t forget to subscribe to the Meteor Secret newsletter.
