How to Quickly Get A Unique Array of Objects in JavaScript

Photo by Clay Banks on Unsplash

Part of the struggle of being a developer is a lack of uniform system standards. I was recently building a REST API to interface with a legacy SOAP API and ran into the issue of poorly constructed endpoints, no filtering, and duplicate records. JavaScript is a very flexible language but does not have a built-in unique array method to help with troublesome APIs. Instead, I had to be creative to quickly get a unique collection of record objects using JavaScript.

Working with Unique Values in JavaScript Arrays

Unlike other scripting languages, JavaScript has no unique iterative method for dealing with collections. Instead, you are forced to use one or more forEach, map, filter, or reduce methods.

With a flat array of values, this task is not difficult. There are multiple ways to approach getting only unique values from an array. The first leverages the array methods forEach and includes:

Figure 1. Iterative approach to checking unique values fo primitive type array.

This doesn’t have the best speed because we have O(N) time every time we call includes for each iteration. We have an additive factorial or triangular number time complexity as we get something like:

O(1) + O(2) + O(3) + … O(N)

To make this efficient, we create a hash to track our values, reducing each loop iteration time complexity to O(1):

Figure 2. Hash lookup method for unique values in a collection.

This is still somewhat complex. Let’s make it really simple. In math, we have a term called a set. A set is a collection of distinct objects grouped together without any specific order.

If we are worried about order, there are methods for that. Now, we are looking for distinct objects. We must understand that these objects refer to primitive types, not complex types like hash objects.

To get our unique collection of objects we have a single line:

Figure 3. Getting unique values from primitive collection using sets.

It’s that simple… for primitive objects. Unfortunately, we are dealing with database records that are not primitive. We will have to approach this from a different direction.

How to Remove Duplicate Objects with JavaScript

We’ve looked at how to create a unique collection for primitive types, but we really care about getting a collection of unique objects. This is more difficult as objects in JavaScript have something called object references.

Two objects in JavaScript with all the same keys and values could be equivalent, but they might not be. It depends on whether or not they share an object reference. Object references are like fingerprints. They are unique to records.

Each JavaScript object is assigned an object reference on instantiation, but each object’s reference is not necessarily unique. If you create two separate objects with the same keys and values, they will have different object references. If you create an object and then create a new object by assigning the first object’s value to the second, they will have the same object reference.

Why is this important for unique objects in a collection? It is important to understand that this is the reason our simple primitive filtering methods in the previous section do not work.

In an ideal world, the solution for getting a collection of unique objects is not challenging. We can leverage a combination of methods mentioned above and the built-in JSON object:

Figure 4. Using Set and JSON object to get unique values from object collection.

With this approach, we turn all the objects into strings, which are primitive values. We can then use the Set to get distinct objects before we pass the unique values to parse to return usable JavaScript objects.

This simple solution makes some assumptions about the shape of our data. One of the main assumptions is that there are no timestamps. Usually, we included createdAt and updatedAt fields on records to help track their lifecycle.

If this is the case, we may have duplicate records with differing timestamps, but all the same attribute values. This would lead us to include duplicate records. We can solve that by removing the timestamps from the objects or creating a composite key.

A composite key combines the fewest fields necessary to ensure a unique value. For our scenario, we can combine an order number, customer ID, and total cost for a composite key to ensure our filtered records are unique. This allows us to leave records intact.

The solution would look like this:

Figure 5. Getting unique values from object collection using composite keys.

As you can see, there are not always straightforward ways to get unique collections of potentially duplicated records. Review your data and check it against your business cases. I hope this helped.