When working with large amounts of data the use of indexes will greatly improve the time it takes for your queries to run by storing part of a collection’s data in a form that is easy to traverse. To add some indexes to your collections you could run some functions directly via the Mongo Shell or Spring Data can be used to handle it for you. As the title suggests that it was we will be looking into in this post.
Lets start with some background information about why we should use indexes. As mentioned in the introduction indexes allows us to query vast amounts of data in a more efficient way which reduces the time taken to retrieve the results. This might seem negligible with smaller sets of data but as the size of documents and collections increase this time difference between having indexes or not is definitely recognisable.
Now lets get onto what this post is about, applying indexes to documents using Spring Data. This is done through the use of the the various index annotations that Spring Data provides for use with MongoDB, which include:
@Indexed
specifies a field that will be indexed by MongoDB.@CompoundIndex
specifies a class that will use compound indexes.@TextIndexed
specifies a field that will part of the text index.@GeoSpacialIndexed
specifies a field that will be indexed using MongoDB’s geospacial indexing feature.
This post will focus on the @Indexed
and @CompoundIndex
annotations.
A very important thing to mention before we go any further is that to use the index annotations within a document class the @Document
annotation needs to be applied. Without this annotation the documents will be created and used correctly but no indexes will be created.
In this post I used Spring Boot to run and test the code although none of this code will be shown here, the required setup and foundation information needed for this post can be found in Getting started with Spring Data and MongoDB.
@Indexed
This annotation is how we mark a single field as being indexed which is the equivalent to the following MongoDB command.
db.COLLECTION_NAME.createIndex({FIELD_NAME: 1})
Where COLLECTION_NAME
is obviously the name of the collection, which when using Spring Data will be the name of the class that is being used or the name specified in the @Document
annotation that has been applied to the class. FIELD_NAME
is the name of field that the @Indexed
annotation has been applied to.
It also comes with various properties that allow us to control how the index is applied.
background
when set to true the index will be applied in the background allowing read and write operations to occur while the index is being built. The equivalent MongoDB command isdb.COLLECTION_NAME.createIndex({FIELD_NAME:1}, {background: BOOLEAN})
direction
specifies the sort order of the index which is ascending by default. The equivalent MongoDB command isdb.COLLECTION_NAME.createIndex({FIELD_NAME:SORT_ORDER})
where
SORT_ORDER
is 1 forIndexDirection.ASCENDING
and -1 forIndexDirection.DESCENDING
.dropDups
when set to true applies an unique index to the first occurrence of a key and removes all subsequent duplicated documents from the collection, although this command was deprecated in MongoDB 3.0.expireAfterSeconds
specify the number of seconds that documents in the collection are retained for. When this property is used the index can be referred to as a TTL (Time-To-Live) index. This property can only be used on fields that represent a date. The equivalent MongoDB command isdb.COLLECTION_NAME.createIndex({FIELD_NAME:1}, {expireAfterSeconds: TIME})
name
provide a name for the index, otherwise the name will be automatically generated to the name of the field. The equivalent MongoDB command isdb.COLLECTION_NAME.createIndex({FIELD_NAME:1}, {name: INDEX_NAME})
sparse
when true the index only references documents that contain the indexed field. The equivalent MongoDB command isdb.COLLECTION_NAME.createIndex({FIELD_NAME:1}, {sparse: BOOLEAN})
unique
when set to true reject all documents that contain a duplicate value for the indexed field. The equivalent MongoDB command isdb.COLLECTION_NAME.createIndex({FIELD_NAME:1}, {unique: BOOLEAN})
useGeneratedName
when set to true it will ignore the given index name from thename
property if provided and use the MongoDB generated name instead, which will look likefieldName_1
.
There are some remaining options that are available via MongoDB directly but not through the Spring Data annotations which will need to be applied manually to the collection via the shell if you wish to use them. These include: v
the index version number and weights
which specifies the significance of an indexed field relative to other indexes.
An import piece of information to note, changing the properties in the annotation when the index has already been created for the collection will cause an exception to occur when Spring Data tries to create the new index. Therefore you might need to drop the original index or question if you are really meant to be changing it in the first place.
Now that we know the properties that are available lets see them in action. Below there are two code snippets that make up an example that use @Indexed
on a few fields and makes use of some of the properties.
In the Person document each index has been given a name, the first_name_index
is sorted in descending order and the expire_after_seconds_index
will cause documents to be removed after being in the collection for 10 seconds (yes I know you probably wouldn’t actually put this on a date of birth field!).
In the Address document the index address_line_one_index
has been created. This is an embedded document that has been used inside the Person
document and will cause its index to be created slightly differently than the earlier piece of code. The index will be placed onto the field address.addressLineOne
where address
is a field in the Person
document and addressLineOne
is a field in the embedded Address document.
To test the use of the indexes I created “some” test data… 100,000 records to be exact so I could make the time difference between querying with and without indexes more significant. To check how long it took for the query to execute I ran a find query with the explain method added on the end.
db.person.find({"firstName":"firstName_2500"}).explain("executionStats")
I don’t personally know anyone called “firstName_2500” but it made creating the data much simpler. Anyway, the generated data contained documents that were duplicated 4 times, for example 4 documents called “firstName_2500” were created. By running the above query we know that we are looking for 4 documents in a collection of 100,000… that’s a pretty small percentage of the total documents that we actually want.
When ran without an index on firstName
{ ... more stats ... "executionStats" : { "executionSuccess" : true, "nReturned" : 4, "executionTimeMillis" : 111, "totalKeysExamined" : 0, "totalDocsExamined" : 100000, ... more stats ... }
From looking at the statistics we can see that all 100,000 documents were examined for the query results even though only 4 were returned. This caused it to take 111 milliseconds, which doesn’t seem like much but as the collection size keeps increasing this time will only become greater. For consistency I ran this query multiple times with execution times varying from 60 to 200 milliseconds.
When ran with an index on firstName
{ ... more stats ... "executionStats" : { "executionSuccess" : true, "nReturned" : 4, "executionTimeMillis" : 0, "totalKeysExamined" : 4, "totalDocsExamined" : 4, ... more stats ... }
These execution results look much better. Only 4 documents were examined compared to 100,000 which were read without the index and this leads to the execution time being much faster, in this example it was actually so small it couldn’t display the actual time. I also ran this query multiple times and each time it returned an execution time of 0 milliseconds.
I also ran the query a bit later, some time after the 10 second time to live which was marked by the expireAfterSeconds
property and no results were returned. So either I did something wrong and deleted the documents myself or the TTL index worked correctly.
That’s probably enough time spent on the @Indexed
annotation which provides a good basis moving forward, therefore some information will be skipped over while explaining the @CompoundIndex
annotation.
@CompoundIndex
This annotation is placed onto a class that represents a document. The equivalent MongoDB command is
db.COLLECTION_NAME.createIndex({FIELD_NAME_1: 1, FIELD_NAME_2: 1})
The format follows the same as the command for creating a singular index but instead takes in multiple fields, in this example I have only used two fields but more could be added.
The annotation shares all of the properties that the @Indexed
annotation had available although direction has been deprecated as the sort order is specified in a different property as explained below.
As demonstrated by this example the fields that are added to the compound index are specified by the def
property inside the annotation. If you compare this to the equivalent MongoDB command to add the index manually you can see that it is virtually the same. As mentioned earlier the direction property has been deprecated from this annotation since the sort orders are specified within the def
property.
The order that the fields are specified within the def
property are important and represent the order that the index will sort the fields. In the example above documents are ordered in ascending order of firstName
values and then salary
values in descending order. Another important piece of information is that when manually applying a sort the fields in the sort method must appear in the same order as the index and can only sort on the original sort order or it’s inverse, if these conditions are violated the sort order on the index will not be efficiently used or not used at all.
Therefore the index in the example could be sorted by the following Mongo queries
db.person.find({$and: [{"firstName":"firstName_2500"}, {"salary":{$gt:0}}]}).sort({"firstName":1, "salary":-1})
and
db.person.find({$and: [{"firstName":"firstName_2500"}, {"salary":{$gt:0}}]}).sort({"firstName":-1, "salary":1})
But not by
db.person.find({$and: [{"firstName":"firstName_2500"}, {"salary":{$gt:0}}]}).sort({"firstName":1, "salary":1})
or
db.person.find({$and: [{"firstName":"firstName_2500"}, {"salary":{$gt:0}}]}).sort({"salary":-1, "firstName":1})
More information about sorting compound indexes can be found in the MongoDB docs.
If you wanted to add multiple compound indexes to your document class you will quickly realise that we need to go about it in a different way (can’t have multiple annotations of the same type applied to the class). Thankfully there is a way around this with the aptly named @CompoundIndexes
annotation which simply contains a collection of @CompoundIndex
annotations which work as explained earlier.
There is not much to say about the @CompoundIndexes
annotation so the example below shows it being added to a document class which will create the indexes when inserted.
I think it’s about time to wrap this post up. In this post we looked at the @Indexed
and @CompoundIndex
annotations that can be applied to a class that is marked with @Document
which when inserted will create the indexes that have been specified within the class. We have also looked briefly at what indexes actually do and how they can decrease query times by a significant amount.
Hi,
I have a structure as follows:
SuperClass
Class1, Class2, Class3
I set @Indexed(name = “originFilename_index”) into a field of SuperClass, but I want it’s created for Class1 and Class2, but not for Class3.
It is possible, in this scenario, to avoid index creation for Class3 ?
Thanks in advance
LikeLike
Oh, I think it’s very simple…I can create the necessary indexes on top of each class instead of on individual fields
Is it right ?
LikeLike
Hi Alessio,
I don’t think there is a solution for what you are trying to do. I’m pretty sure the
Indexed
annotation can only be used on columns (properties of the class) so you can’t add it to the class itself.If you want the columns on two out of three classes then just add the columns to those classes directly. It might be slightly more duplication but it’s the only way I see this working.
It might also be worth reviewing your document structures, as it could be possible to restructure them to better suit what you are trying to achieve.
Thanks
Dan
LikeLike
It is possible to declare indexes at the top of classes. For example:
@CompoundIndexes({
@CompoundIndex(name = “My_Index”, unique = true, def = “{‘field1’ : 1, ‘field2’}” )
})
LikeLike
Oh sorry, haven’t looked at this for a while.
In that case then maybe you can define the shared fields in the super class but add the indexes on the children themselves. It might be possible to mark indexes on the super class and add extras to the children via annotating the child class.
Honestly I am not 100% sure on this and would need to play around with it myself to figure it out.
If you do come up with a good solution, let me know!
Thanks
Dan
LikeLike
Hi, please help me) I can not create the index ttl, I follow the documentation, my application starts without errors and warnings, after I execute db.plan.getIndexes (); and I do not see my indexes
LikeLike
Hi Vlad,
It has been so long since I wrote this that I am not sure I can be of much help.
Thanks Dan
LikeLike
Hi Dan
Why is that index is created each time a Mongo Connection is established using @Indexed on a class property? Doesn’t this cause an overload on the Database?
LikeLike
As far as I am aware, the index should only be created once. If it already exists the index won’t be created again.
LikeLike
Hi
I want to define a variable for text with the length of 300 characters Which one I have to choose
please let me know
LikeLike
From what I can find, there is no specific limit on a single field except that a single document cannot be above 16MB.
https://docs.mongodb.com/manual/reference/limits/
https://stackoverflow.com/questions/21524287/mongoid-data-type-for-text
LikeLike