Mining Common Crawl with PHP

The Idea

The challenge

First steps

$filename = ‘warc’;
$handle = fopen($filename, “r”);
if ($handle) {
while (($line = fgets($handle)) !== false) {
$line_num++;
if (substr($line,0,16)==’WARC-Target-URI:’) {
$targetUrl = trim(substr($line,17));
}

$posTheme = stripos($line, ‘wp-content/themes/’);
if ($posTheme !== false) {
$posTheme += 18;
$posThemeEnd = strpos($line, ‘/’, $posTheme);
if ($posThemeEnd !== false) {
$t = substr($line, $posTheme, $posThemeEnd — $posTheme);
$lineToWrite = $targetUrl. ‘ ‘ . $t.”\n”;
if ($lineToWrite != $lastLineWritten) {
file_put_contents(‘out.txt’, $lineToWrite, FILE_APPEND | LOCK_EX);
$lastLineWritten = $lineToWrite;
}
}
}

// progress
if ($line_num % 1000000 === 0) {
echo $line_num.”\n”;
}
}
fclose($handle);
} else {
echo “error opening the file \n”;
}

Scaling Up

Scaling up Part 2

Getting the results from S3

The Results

pub 651165 
h4 415293
genesis 69626
twentyeleven 67447
twentytwelve 53274
twentyten 52642
divi 45945
avada 44101
twentyfourteen 34825
twentyfifteen 28499
enfold 24288
twentythirteen 22825
responsive 19185
canvas 19157
default 18907
x 12937
atahualpa 11723
salient 11477
graphene 11445
u-design 10877
twentysixteen 10393
dt-the7 9687
avada-child-theme 9520
premium 9412
headway 9307
suffusion 9264
cherryframework 9239
customizr 9153
sahifa 8996
vantage 8842
hueman 8132
jupiter 7866
bridge 7453
thesis_18 6928
thematic 6840
prophoto5 6726
betheme 6443
divi-child 6396
pinboard 6249
chameleon 6089
weaver-ii 6074
ipinpro 5938
newspaper 5685
mantra 5555
pagelines 5422
sparkling 4954
builder 4880
thesis 4703
karma 4631
enfold-child 4559

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Paulius Rimavičius

Paulius Rimavičius

More from Medium

Tutorial: Docker PHP Development setup for the generation of TikTok attention span developers

Example Dockerfile

How To Install PHP On Windows OS — 6 Steps To Follow

How To Install PHP On Windows OS

How to connect PHP application to MSSQL, setup on 2 different servers.

URLs manipulation in PHP (the easy way)

Change your URLs with a simple fluent PHP class