May 20: Removing duplicates from a JavaScript array of non-primitives (without lodash!)

Ah, the humble array! It’s the Swiss Army knife of JavaScript data structures: nothing fancy, but it does what it says on the can. Arrays are a convenient way to group variables and values together, and understanding them is crucial to working with web APIs.

But unfortunately, like everything else in JavaScript, arrays have limits, quirks and caveats galore — and using them effectively requires smart thinking and MacGyver-esque workarounds. In particular, filtering non-primitive duplicates is an array problem so common it’s a near-ubiquitous interview subject. If I’m handling a mailing list for a client as an array, for example, I likely don’t want duplicate recipients in my array of addresses. And maybe my client is Al Pacino’s character from The Devil’s Advocate and insists I work without an external library like lodash.

Since I’ve encountered this issue in my career and in personal projects, let’s explore a few approaches!

Background: Primitive Love

In JavaScript, a primitive is one of the four Ts, representing text, a total, a truth or typelessness:

• Text: a character, 'c', or a 'string';
• Total: a Number() like 3 or 3.14 or 0b11;
• Truth: a Boolean() (true or false)
• Typeless: undefined, nullor NaN

The relevant defining and distinguishing characteristic of the four Ts is strict comparability: two primitives are equal if they have strictly the same value:

1 === 1;
   > true
'primitive' === 'primitive';
   > true
false === false;
   > true

Since Array methods like includes() and indexOf() rely on strict comparability, this property of primitives makes filtering duplicates of them from an array relatively painless:

function filterPrimitiveDuplicates( array ) {
   return array.filter( ( element, index ) => {
      return !array.slice( 0, index ).includes( element );
   } );
}
filterPrimitiveDuplicates( [ 5, 4, 5, 2, 5 ] )
   > [ 5, 4, 2 ]
filterPrimitiveDuplicates( "there is no there there".split( " " ) )
   > [ 'there', 'is', 'no' ]

There’s another relatively new addition to the JavaScript toolkit that relies on strict equality: sets. A JavaScript Set() is a non-indexed collection of strictly unique values, just like a mathematical set. Set()s are fast and well-supported by most browsers, and we can take advantage of their uniqueness constraint in a function to filterPrimitiveDuplicates() just like the one above:

function filterPrimitiveDuplicates( array ) {
   return [ ...new Set( array ) ];
}

To strip primitive duplicates, all we have to do is turn our array into a Set() , then spread it out back into an array! And it may be hard to believe, but casting back and forth between an array and a Set() is slightly (about 25%) faster than looping through an array‘s elements and comparing them with includes()!

Back to the Primitive

Unfortunately for our Advil budget, non-primitives like arrays and objects are not comparable. Two arrays, even if they have the exact same values in the exact same order, are not equal — strictly or even weakly — if they have different addresses in memory:

[ 5 ] === [ 5 ];
   > false           // ugh
[ 1, 2 ] == [ 1, 2 ];
   > false           // yuck

This applies to Set(), includes() and indexOf() just as it does with twequals and threequals:

const arrayOfNonPrimitives = [ [ 1, 2 ], [ 3, 4 ], [ 5, 6 ], [ 1, 2 ] ];
new Set( arrayOfNonPrimitives ) ;
   > Set(4) { [ 1, 2 ], [ 3, 4 ], [ 5, 6 ], [ 1, 2 ] } // fail
arrayOfNonPrimitives.includes( [ 1, 2 ] );
   > false                                             // awful
arrayOfNonPrimitives.indexOf( [ 1, 2 ] );
   > -1                                                // my eyes

So how do we filter duplicates from an array of non-primitives if its elements aren’t strictly comparable? How about we cast them to a primitive datatype that is strictly comparable — like strings? I can already tell this is gonna get messy…

Casting arrays to strings

JavaScript’s toString() or join() methods will do nicely for this task; when called without arguments on an array, either yields a string of each element separated by commas — and these strings, praise the gods, are comparable:

[ 1, 2, 3 ].join();
   > '1,2,3'
[ 1, 2, 3 ].toString();
   > '1,2,3'
[ 1, 2, 3 ].toString() === [ 1, 2, 3 ].join();
   > true

Let’s filter like before, but add some casting:

function filterArrayDuplicates( array ) {
   return array.filter( ( element, index ) => {
      return !array.slice( 0, index ).join().includes( element.join() );
   } );
}
filterArrayDuplicates( [ [ 1, 2 ], [ 3, 4 ], [ 1, 2 ] ] );
   > [ [ 1, 2 ], [ 3, 4 ] ]           // hallelujah

The same idea applied to our approach with Set()s contains graphic content and may not be appropriate for younger readers:

function filterArrayDuplicates( array ) {
   return [ ...new Set( array.map( element => element.join() ) ) ].map( element => element.split( "," ).map( element => parseInt( element ) ) );
}

Let’s tease this out into multiple lines a little before I run screaming off into the woods to become a primitive:

const castToString = element => element.join();
const castToInteger = element => parseInt( element );
const splitByCommasThenCastToInteger = element => element.split( "," ).map( castToInteger );
function filterArrayDuplicates( array ) {
   const castEachElementToString = array.map( castToString );
   const castToSetAndBackAgain = [ ...new Set( castEachElementToString ) ];
   return castToSetAndBackAgain.map( splitByCommasThenCastToInteger );
}

This approach becomes clearer when I pre-define all those nasty (but important!) callbacks. First, I castEachElementToString with a map() function; then, I castToSetAndBackAgain; and finally, I use another map() function to splitByCommasThenCastToInteger. Whew!

Casting objects to strings

My will to live slowly grows weaker and weaker as I realize that toString() and join() will not work on objects — because there is no Object.join() method, and the default conversion of an Object toString() is ridiculous and unhelpful:

{ one: 1, two: 2, three: 3 }.toString();
   > '[object Object]'           // what is this sorcery
{ one: 1, two: 2, three: 3 }.join();
   > Uncaught: TypeError         // how could a merciful God do this

It’s all fun and games until [object Object] calls

We’ll need to get creative and find another way to compare Objects to each other to determine correctly if they contain the same values. One way is JSON.stringify(), which was almost certainly not intended for a use like this, but nevertheless yields a passable primitive version of an Object:

JSON.stringify( { one: 1, two: 2, three: 3 } );
   > '{"one":1,"two":2,"three":3}'     // I guess that's better (?!)

The only problem with JSON.stringify() is that it will yield different results if key/value pairs were inserted in a different order, even if those key/value pairs are all exactly the same:

thisObject = { one: 1, two: 2, three: 3 };
thatObject = { three: 3, two: 2, one: 1 };
JSON.stringify( thisObject );
   > '{"one":1,"two":2,"three":3}'
JSON.stringify( thatObject );
   > '{"three":3,"two":2,"one":1}'
JSON.stringify( thisObject ) === JSON.stringify( thatObject );
   > false                       // this is the life I've chosen

To work around this hysterical and ludicrous problem, I’ll use two Object prototype methods: Object.keys() and Object.values(). We’ll split objects into arrays of their keys() and values(), then sort() and cast each toString(). When doing that, two objects with identical key/value pairs will always yield the same results, even if those pairs are out of order:

function compareObjects( thisObject, thatObject ) {
   return Object.keys( thisObject ).sort().toString() === Object.keys( thatObject ).sort().toString() && Object.values( thisObject ).sort().toString() === Object.values( thatObject ).sort().toString();
}
compareObjects( thisObject, thatObject )
   > true                       // it's over, Mr. Frodo

Remember above I explained that Set() relies on comparability of its elements to enforce its uniqueness. Since there’s no straightforward way to cast an object to a primitive effectively while guaranteeing its comparability, we can’t use Set() like above. But, we can use filter() by combining our compareObjects() method with findIndex(), which returns the index of the first element in the array that satisfies the testing callback:

function compareObjects( thisObject, thatObject ) {
   ...
}
function filterObjectDuplicates( array ) {
   return array.filter( ( element, index ) => {
      return index === array.findIndex( elementToCompare => compareObjects( element, elementToCompare ) );
   } );
}
filterObjectDuplicates( [ { one: 1 }, { two: 2 }, { three: 3 }, { one: 1 } ] )
   > [ { one: 1 }, { two: 2 }, { three: 3 } ] // sweet victory

Conclusion: Primitive feeling, primitive way

It’s important to notice that in all the examples I used in the above circus of horrors, the non-primitive array elements themselves contained exclusively primitives. There was no nesting of non-primitives within non-primitives, like…

[
  {
    name: 'Josh',
    languages: [ 'English', 'Spanish' ],
    diet: { vegetarian: false, glutenFree: false }
  },
  {
    name: 'Worf',
    languages: [ 'English', 'Klingon' ],
    diet: { vegetarian: false, glutenFree: true }
  },
]

Nor were there any irregular arrays with more than one data type:

[ 56, null, true, { name: "Josh"}, 9.5, "Spanish" ]

The more irregular and nested your array is, the more tweaking and head-scratching will be needed to remove duplicates. Only you can decide how best to adjust and handle edge-cases and irregularities. Just remember the core idea: cast elements to primitives and make sure those primitives are strictly, uniquely comparable!