7 Ways to Remove Array Duplicates in Javascript

How to remove duplicate element?

In real development work, we often encounter the handling of a set of data de-duplication. In JavaScript, there are several ways to filter out duplicates from an array, and I’ll share with you the different ways of de-duplication in this article.

Here is the array we want to filter out duplicates:

Original array

1. Set

new Set() is my favorite method for everyday development because it’s the easiest to use of all the de-duplication methods.Set is a new type introduced by ES6, and the difference between it and Array is that the data in the Set type can’t have duplicate values. Of course, there are some methods of Array that Set can’t call.

Using new Set() to remove duplicates

First use new Set() to convert the original array to Set type data, and then convert the Set type data to a new array which has been filtered out duplicates. When we talk about Set to Array, we can use Array.from() or we can use structural way […new Set(arr)].

filtered array result

Set de-duplication also works for NaN and undefined de-duplication, because both NaN and undefined can be stored in a Set, and NaNs are treated as the same value as each other (although in js: NaN !== NaN)

2. double for + splice

The array elements are compared one by one in a two-level for loop, and duplicates are removed by the splice method.

Using for-loop and splice() to remove duplicates

filtered array result

This method is not able to filter out NaN because NaN !== NaN when compare.

3. indexOf / includes

Create a new empty array, traverse the array that needs to be de-duplicate, push the array elements into the new array, and judge whether the new array already contains the current element before pushing, if not, then push it. This method also cannot filter out NaN. To judge whether an array already contains the current element, use the array method indexOf or includes.

indexOf

The indexOf() method of Array instances returns the first index at which a given element can be found in the array, or -1 if it is not present.

Using indexOf() to remove duplicates

filtered array result

includes

The logic of includes is similar to indexOf, we can use it to judge whether an array contanins an element.

The includes() method of Array instances determines whether an array includes a certain value among its entries, returning true or false as appropriate.

Using includes() to remove duplicates

filtered array result

Because includes can correctly find variables of type NaN, it can de-duplicate data of type NaN.

difference between includes() and indexOf()

We see from the example, includes(NaN) returns true, while indexOf(NaN) returns -1.

4. filter

We can use filter() + indexOf().

The filter() method of Array instances creates a shallow copy of a portion of a given array, filtered down to just the elements from the given array that pass the test implemented by the provided function.

The feature of indexOf is to return the index of the first position contained in the target to be found, we can filter out only the first item of each individual data, the remaining duplicates will be filtered out.

Using filter() and indexOf() to remove duplicates

filtered array result

Here the output does not contain NaN, because indexOf() can’t judge NaN, that is, arr.indexOf(item) === index returns false.

5. Map / Object

Map

The Mapobject is a data structure provided by JavaScript that is structured as a key-value pair and remembers the original insertion order of the keys. Any value (object or raw value) can be used as a key or a value.

Using Map object to remove duplicates

An array element is stored as a key of a map, and then the has() and set() methods are combined to determine whether the key is duplicated or not.

filtered array result

NaN can also be de-duplicated after using Map(), because Map makes a judgment that NaN is equal to NaN, leaving all other values to be judged as equal or not based on the result of the === operator.

Object

Use Object type to filter out is is similar to use Map, this way mainly using the feature that property name of the object can not be repeated.

Using Object object to remove duplicates

filtered array result

6. sort

Use sort() to sort, then traverse and compare neighboring elements based on the sorted result.

Using sort() to remove duplicates

If the current item is not equal to the previous item, push into the new array.

filtered array result

This method changes the original position of the array and cannot de-duplicate NaN data.

7. reduce

The reduce() executes a callback function for each element of the array in turn, taking four arguments: the initial value initialValue (or the return value of the previous callback function), the current element value, the current index, and the array from which reduce was called.

Using reduce() and includes() to remove duplicates

When initializing, we define a new array, and each time we loop, we determine whether the values of the old array have already been stored in the new array, and if not, we add them to the new array.

filtered array result

Thanks for reading!