GroupDocs.Search for .NET 26.10 is now available. This is a major release: the index format has been redesigned, so indexes take less space on disk, need half as much RAM and open 2 to 4 times faster. Search, deletion, optimization and image search are accelerated several to hundreds of times, searches no longer wait for indexing, and index data are protected with checksums and atomic updates.

Existing indexes created by versions 22.7–26.8 are converted to the new format without re-indexing. The release contains 22 changes: 4 new features, 15 improvements and 3 fixes.

Performance highlights

Most gains grow with the size of the index and the number of its segments.

What Gain in 26.10
RAM required by a loaded index 2 times less
Opening an index 2 to 4 times faster
Index size on disk Grows in proportion to the indexed data; the larger the index, the greater the saving
Search for a word in large indexes Up to 2 times faster; rare words 10 to 40 times faster
Fuzzy search in large indexes 2.5 to 5 times faster
Wildcard search in large indexes 2.5 to 3 times faster
Wildcard, regular expression, numeric range and fuzzy search by chunks 7 to 150 times faster
Date range search 2 to 27 times faster
Deleting documents Hundreds of times faster, 4 times less memory, no additional disk space
Index optimization 4 times faster, 4 times less memory
Peak RAM during indexing 2 times less
Retrieving the stored text of a document Up to 15 times faster
Image search data 2.9 times less memory, loaded 6.6 times faster, saved 100 times faster; reverse image search up to 1.8 times faster
Searching while the index is being modified No longer blocked
Index reliability Checksums and atomic changes; interrupted operations are completed or rolled back
Search threads All processor cores by default; can be changed at any time

What’s new in this release

Key Summary Category
SEARCHNET-3665 Implement the ability to change the number of search threads at any time Feature
SEARCHNET-3666 Implement an option to replace unpaired surrogate characters Feature
SEARCHNET-3667 Implement an option to skip terms consisting only of blended characters Feature
SEARCHNET-3668 Implement index version update to the new index format without reindexing the documents Feature
SEARCHNET-3669 Implement optimization of disk space consumption of an index Improvement
SEARCHNET-3670 Implement optimization of RAM consumption of a loaded index Improvement
SEARCHNET-3671 Implement optimization of index loading speed Improvement
SEARCHNET-3672 Implement optimization of search speed in large indexes Improvement
SEARCHNET-3673 Implement optimization of fuzzy search Improvement
SEARCHNET-3674 Implement optimization of wildcard search Improvement
SEARCHNET-3675 Implement optimization of deleting and updating documents Improvement
SEARCHNET-3676 Implement optimization of the index optimization operation Improvement
SEARCHNET-3677 Allow searching while the index is being modified Improvement
SEARCHNET-3678 Improve index reliability Improvement
SEARCHNET-3679 Implement optimization of memory usage during indexing Improvement
SEARCHNET-3680 Implement optimization of retrieving document text Improvement
SEARCHNET-3681 Implement optimization of image search Improvement
SEARCHNET-3682 Implement handling letter case identically on all platforms Improvement
SEARCHNET-3688 Implement optimization of search by chunks Improvement
SEARCHNET-3683 Fix the default number of search threads Fix
SEARCHNET-3684 Prevent inconsistent indexing after changing term dictionaries Fix
SEARCHNET-3685 Prevent merging indexes that process text differently Fix

Major features

Index storage and reliability

  • The index size now grows in proportion to the indexed data, so large indexes with many segments save the most space.
  • A loaded index needs half as much RAM, and opening an index is 2 to 4 times faster.
  • Index data are protected with checksums, and all changes are applied atomically. If an operation is interrupted by an error, a power failure or process termination, the index stays in its previous consistent state.

Search

  • Only the segments that contain the query terms are read, which speeds up word, fuzzy, wildcard and chunked search.
  • Search uses all processor cores by default, and the number of threads can be changed at any time via IndexSettings.SearchThreads.
  • Searches and index-modifying operations (indexing, update, deletion, merging, optimization) no longer block each other. Each search works with a consistent snapshot of the index.

Indexing and maintenance

  • Deleting documents no longer rewrites the index: documents are excluded from search immediately and physically removed during optimization.
  • Index optimization is 4 times faster and uses 4 times less memory; very large indexes are optimized into several segments of limited size.
  • Indexing consumes about half as much peak RAM.

Text processing

  • Letter case is handled identically on .NET Framework and .NET, so search results do not depend on the platform.
  • Unpaired surrogate characters are replaced with U+FFFD instead of being removed, and terms that consist only of blended characters are no longer indexed.

Public API changes

Two properties have been added to the GroupDocs.Search.IndexSettings class:

  • Boolean ReplaceUnpairedSurrogates – replaces unpaired UTF-16 surrogate characters with the Unicode replacement character (U+FFFD). Enabled by default.
  • Boolean SkipBlendedOnlyTerms – excludes from indexing the terms that consist only of blended characters. Enabled by default.

Both settings are fixed when an index is created.

Code example

The following example updates an index created by a previous version to the new index format without re-indexing the documents.

string sourceIndexFolder = @"c:\MyOldIndex\"; // Specify path to the index of a previous version
string targetIndexFolder = @"c:\MyNewIndex\"; // Specify path to the folder for the updated index

IndexUpdater updater = new IndexUpdater(); // Creating an index updater

if (updater.CanUpdateVersion(sourceIndexFolder)) // Checking whether the index version can be updated
{
    VersionUpdateResult result = updater.UpdateVersion(sourceIndexFolder, targetIndexFolder); // Updating the index version; the source index is not changed
}

Upgrade notes

Take the following changes into account when upgrading from a previous version:

  • The index format has changed. Indexes of previous versions must be converted with the IndexUpdater class before they can be opened. Source documents are not required, the converted index is saved to a different folder, and the source index is not modified.
  • Term dictionaries are fixed once an index contains documents. The alphabet, tokenization dictionary, character replacements, stop words, and the UseStopWords and UseCharacterReplacements settings must be configured before the first document is indexed. A later change throws InvalidOperationException.
  • Incompatible indexes are not merged. Index.Merge throws InvalidOperationException if the merged indexes split text into terms differently.
  • Search is multithreaded by default. With the default IndexSettings.SearchThreads, search uses as many threads as there are processor cores. To limit the number of threads, set the property explicitly.
  • Two new index settings are enabled by default. To keep the previous behavior, set ReplaceUnpairedSurrogates and SkipBlendedOnlyTerms to false before creating an index.
  • Deleted documents are physically removed during optimization. Call Index.Optimize to reclaim the space.
  • The number of chunks has changed. Search by chunks no longer returns a chunk for every segment. Iterate over the chunks while SearchResult.NextChunkSearchToken is not null instead of relying on the number of segments.

How to get the update

NuGet

  • NuGet – Install the latest package:
# Package Manager
NuGet\Install-Package GroupDocs.Search -Version 26.10.0

# .NET CLI
dotnet add package GroupDocs.Search --version 26.10.0

# Package Manager
NuGet\Install-Package GroupDocs.Search.NETFramework -Version 26.10.0

# .NET CLI
dotnet add package GroupDocs.Search.NETFramework --version 26.10.0

Direct download

Resources