SECrawl beta: audit for search, with search

SECrawl_-_Enterprise_Website_Crawler__built_on_search

Does anyone need yet another website crawler?

There’s no shortage of crawlers out there, and for most people, my web-based SEO crawler, Screaming Frog or URL Profiler is more than enough.

There is a gap though — no two websites are alike, yet we look at the same metrics and cookie-cutter recommendations again and again. Or vast tables of complex text data that Excel was never built to analyse or manipulate.

Introducing SECrawl

SECrawl is a search engine, and an enterprise-level website analysis tool that offers:

Flexibility

Because SECrawl is a search engine, the variety of reports and queries you can perform on the data are only limited by the data itself.

There are over 30 advanced, Google-style search operators which you can combine to provide deep insight into your website’s content and meta data.

The problem with using spreadsheets to analyze website data is that they’re terrible at dealing with arrays and nested data structures. So for the same reason NoSQL databases have become popular for building websites, we need similar flexibility in analysing big data that doesn’t conform to tabular structures.

Speed

SECrawl is built to be fast; both in terms of how quickly it crawls a website, and the speed at which you can start playing with the data.

Another advantage of using a search engine in the back-end is that a lot of hugely complex queries and reports can be completed in less than a second.

Scalability

The technology behind SECrawl is robust, and all has been proven to scale massively.

It has been tested so far with crawling millions of URLs with no significant slow-down in crawl speed or analysis.

Portability

All search result based reports (which means most of them) are directly exportable into an XLSX file.

So if you’re very comfortable in Excel, you can narrow down the big data you want to analyse using SECrawl, then PivotTable to your heart’s content once you’ve got a manageable number of pages to analyse.

Accessibility

Obviously using a web-based solution means you can use the crawler from any web browser.

Content audits

Not only does SECrawl index the content every page it crawls, it also picks up the author and publish time, which allows you to do nifty things such as view your most prolific authors, or see Google N-Gram style charts of your language usage over time.

language-trends
Language Trends — analyse content published in a Google N-grams style report

By combining advanced search operators, you can also quickly narrow down a dataset of millions of pages to a more focused view.

What next?

SECrawl is still very much in beta, but you can email me for an invite code to try it out for free.