GroupDocs.Search for .NET 26.11 is now available. This release makes repeated and simultaneous searches much faster, cuts the memory allocated by indexing up to 4.5 times and secures the search network with node authentication. Indexing in a search network is 5 to 20 times faster, and the GroupDocs.Search NuGet package now includes builds for .NET Framework 4.8, .NET 6.0, .NET 8.0 and .NET 10.0.

Indexes created by version 26.10 open without conversion. The release contains 18 changes: 5 new features, 12 improvements and 1 fix.

Performance highlights

The full release notes describe the test setup of each figure.

What Gain in 26.11
Repeated regular expression search Up to 300 times faster
Repeated fuzzy search 4 to 30 times faster
Simultaneous searches of an index Median time of 8 simultaneous searches from 5.6 ms to 0.18 ms; 4 to 18% more searches per second
Fuzzy search with one mistake 1.8 times faster, 4.5 times faster with 8 search threads
Date range search Up to 10 times faster, up to 59 times faster by chunks; long ranges return complete results
Memory allocated by indexing Up to 4.5 times less
Indexing with stored document text Up to 2.2 times faster
Retrieving stored document text 3.5 times faster, 11 times less memory
Merging indexes 3.5 times faster, almost 50 times less memory
Indexing in a search network 5 to 20 times faster, far fewer index segments in the shards
Registering 20,000 documents on the master node From 27.1 s to 0.12 s, 0.8 MB written to disk instead of 7.6 GB
Disk space of an index Can be reduced at the cost of loading speed

What’s new in this release

Key Summary Category
SEARCHNET-3729 Implement an option to reduce the disk space occupied by an index at the cost of loading speed Feature
SEARCHNET-3730 Implement authentication of search network nodes Feature
SEARCHNET-3731 Implement OCR support for document extraction in a separate process Feature
SEARCHNET-3732 Implement search by chunks in an index repository Feature
SEARCHNET-3733 Implement highlighting of words produced by a custom word splitter Feature
SEARCHNET-3734 Optimize search by chunks Improvement
SEARCHNET-3735 Implement a shared thread pool for concurrent searches of an index Improvement
SEARCHNET-3736 Implement reuse of matched words in repeated searches Improvement
SEARCHNET-3737 Implement the IDisposable interface in the Cancellation class Improvement
SEARCHNET-3738 Extend the default alphabet with additional scripts Improvement
SEARCHNET-3739 Accelerate date range search and ensure complete results for long date ranges Improvement
SEARCHNET-3740 Reduce memory consumption of indexing and of operations with stored document texts Improvement
SEARCHNET-3741 Accelerate indexing in a search network with a large number of documents Improvement
SEARCHNET-3742 Implement batch indexing and parallel processing of shard requests in a search network Improvement
SEARCHNET-3743 Implement the dispose pattern in the Index, IndexRepository, Cancellation, ExtractionHost and SearchNetworkNode classes Improvement
SEARCHNET-3693 Implement TFM-Split Packaging Strategy for GroupDocs.Search NuGet Package Improvement
SEARCHNET-3747 Implement NuGet packages for .NET Framework 4.8, .NET 6.0, .NET 8.0, .NET 10.0 Improvement
SEARCHNET-3746 Fixed indefinite waits of a search network for lost or unresponsive nodes Fix

Major features

Search

  • Repeated wildcard, regular expression, numeric range and fuzzy searches reuse the words found by earlier searches with the same query until the index changes. The kept words take up to 8 MB of memory per index.
  • Simultaneous searches of an index share its search threads (IndexSettings.SearchThreads), so short searches no longer wait for long ones.
  • Date range search reads each day, month and year from the index once instead of searching every date of the range separately. Long ranges are no longer cut off at the occurrence limit, and a range of all possible dates, which failed before, is now searched in seconds.
  • Search by chunks gets all of these gains, and an index repository can now be searched by chunks as well: IndexRepository.SearchNext continues the search index by index.
  • Highlighting and Index.GetDocumentText can divide the text with the custom word splitter used for indexing, so the words that the splitter produces, for example words of a text without spaces, are highlighted too.
  • The default alphabet of new indexes includes Hebrew, Devanagari, Georgian, extended Latin, Greek and Hangul letters and combining diacritical marks. Words such as “Việt” are no longer split into parts, and Korean and Hebrew words are indexed and found whole.

Indexing and index storage

  • The new IndexSettings.UseFastLoading setting lets an index on disk drop the additional data that makes its loading several times faster: the index takes less disk space but loads longer. The setting can be changed at any time, also for an index that already contains documents.
  • Indexing allocates up to 4.5 times less memory per document and, with stored document text, runs up to 2.2 times faster. Retrieving stored text is 3.5 times faster with 11 times less memory, and merging indexes is 3.5 times faster with almost 50 times less memory.
  • The text of images in documents extracted in a separate process can now be recognized: ExtractionHost.OcrConnector sets the OCR connector of the extraction process. If recognition is enabled but the extraction process has no OCR connector, the index reports it through the ErrorOccurred event.

Search network

  • Nodes authenticate each other with a shared secret key (TcpSettings.NetworkKey), and every message is signed: a computer without the key can neither give commands to a node nor answer for one. The key is never sent over the network. A slave node can also restrict the addresses from which it accepts the master node (TcpSettings.AllowedMasterAddresses).
  • Indexing is 5 to 20 times faster and creates far fewer index segments: documents that arrive while a shard is busy are indexed in batches of up to 1,000 documents or 16 MB, and the shards of a node process their requests in parallel.
  • Registering indexed documents on the master node no longer slows down as their number grows, and the list of the documents of a shard is kept safely when a write fails or the process crashes.
  • The master node detects lost and unresponsive nodes: a slave node reports every 5 seconds, and calls no longer wait forever for a node that is gone. TcpSettings.ResponseTimeout can also limit how long the master node waits for a response.

API and packaging

  • Index, IndexRepository, Cancellation, ExtractionHost and SearchNetworkNode implement the standard .NET dispose pattern, so derived classes can release their own resources. Disposing a Cancellation releases the timer of a cancellation scheduled by CancelAfter.
  • The GroupDocs.Search NuGet package now includes builds for .NET Framework 4.8, .NET 6.0, .NET 8.0 and .NET 10.0 in addition to .NET Framework 4.6.2 and .NET Standard 2.1, and selects the build for the target framework of a project automatically.

Public API changes

The following members have been added:

  • GroupDocs.Search.IndexSettings:
    • Boolean UseFastLoading – keeps the additional data that makes an index on disk load several times faster. Enabled by default; set it to false to save disk space.
  • GroupDocs.Search.Scaling.Configuring.TcpSettings:
    • String NetworkKey – the secret key of the search network, at least 16 characters long and the same on all nodes.
    • String[] AllowedMasterAddresses – IPv4 addresses or host names from which a slave node accepts the connection of the master node.
    • Int32 ResponseTimeout – the time in milliseconds that the master node waits for the response of a node to a request. No limit by default.
  • GroupDocs.Search.ExtractionHost:
    • IOcrConnector OcrConnector – the OCR connector of the separate extraction process.
  • GroupDocs.Search.IndexRepository:
    • SearchResult SearchNext(ChunkSearchToken) and SearchResult SearchNext(ChunkSearchToken, Cancellation) – continue a search by chunks in an index repository.
  • GroupDocs.Search.Options.TextOptions:
    • IWordSplitter WordSplitter – the word splitter used by highlighting and Index.GetDocumentText; inherited by HighlightOptions.
  • GroupDocs.Search.Common.Cancellation now implements IDisposable:
    • void Dispose() – releases the timer of a scheduled cancellation; the scheduled cancellation is not requested.
  • Index, IndexRepository, Cancellation, ExtractionHost and SearchNetworkNode:
    • protected virtual void Dispose(Boolean disposing) – the dispose pattern method for derived classes.

Behavior changes that may require changes in your code are listed in the upgrade notes below.

Code examples

Search an index repository by chunks

The following example searches two indexes of a repository chunk by chunk, one index after another.

string indexFolder1 = @"c:\MyIndex1\"; // Specify the path to the first index folder
string indexFolder2 = @"c:\MyIndex2\"; // Specify the path to the second index folder
string documentsFolder1 = @"c:\MyDocuments1\"; // Specify the path to a folder containing documents to index
string documentsFolder2 = @"c:\MyDocuments2\"; // Specify the path to another folder containing documents to index

using (IndexRepository repository = new IndexRepository()) // Creating an index repository
{
    Index index1 = repository.Create(indexFolder1); // Creating the first index in the repository
    index1.Add(documentsFolder1); // Indexing documents from the first folder
    Index index2 = repository.Create(indexFolder2); // Creating the second index in the repository
    index2.Add(documentsFolder2); // Indexing documents from the second folder

    SearchOptions options = new SearchOptions();
    options.IsChunkSearch = true; // Enabling search by chunks

    SearchResult result = repository.Search("Einstein", options); // Getting the first chunk of the first index
    Console.WriteLine("Documents: " + result.DocumentCount);
    while (result.NextChunkSearchToken != null)
    {
        result = repository.SearchNext(result.NextChunkSearchToken); // Getting the next chunk, index by index
        Console.WriteLine("Documents: " + result.DocumentCount);
    }
}

Protect a search network with a shared key

Every node of a search network gets the same secret key. A slave node accepts the master node only from its address, and the master node limits the wait for responses of the nodes.

string basePath = @"c:\SearchNetwork\"; // Specify the path to the folder of the search network
string networkKey = Environment.GetEnvironmentVariable("SEARCH_NETWORK_KEY"); // The same key of at least 16 characters on every node

// On the server of slave node 1: the connection of the master node is accepted from 10.0.0.5 only
TcpSettings slaveSettings = new TcpSettings(49101, 10000, 10000, 10000, 1000);
slaveSettings.NetworkKey = networkKey;
slaveSettings.AllowedMasterAddresses = new string[] { "10.0.0.5" };
SearchNetworkNode node1 = SearchNetworkNode.CreateSlaveNode(1, basePath + "Node1", slaveSettings);
node1.Start();

// On the server of the master node (10.0.0.5): the responses of the nodes are awaited no longer than 10 minutes
Configuration configuration = Configuration.Load(basePath + "Configuration.xml"); // Loading the search network configuration
TcpSettings masterSettings = new TcpSettings(49100, 10000, 10000, 10000, 1000);
masterSettings.NetworkKey = networkKey;
masterSettings.ResponseTimeout = 600000;
SearchNetworkNode node0 = SearchNetworkNode.CreateMasterNode(
    0, basePath + "Node0", masterSettings, new FileLogger(basePath + "Log.txt", 4.0), configuration);
node0.ConfigureAllNodes();
node0.Start();

Reduce the disk space of an index

The following example creates an index that takes less disk space, and then makes it load fast again.

string indexFolder = @"c:\MyIndex\"; // Specify the path to the index folder
string documentsFolder = @"c:\MyDocuments\"; // Specify the path to a folder containing documents to index

IndexSettings settings = new IndexSettings();
settings.UseFastLoading = false; // The index takes less disk space but loads longer
using (Index index = new Index(indexFolder, settings, true)) // Creating an index with the settings
{
    index.Add(documentsFolder); // Indexing documents from the specified folder
}

using (Index index = new Index(indexFolder)) // Opening the existing index
{
    index.IndexSettings.UseFastLoading = true; // The data is created the next time the index is loaded
}

Upgrade notes

Take the following changes into account when upgrading from a previous version:

  • Indexes changed by version 26.11 cannot be opened by earlier versions. Version 26.11 opens indexes of version 26.10 without conversion, but once it creates or changes an index (by indexing, deleting documents, optimization or a change of settings), earlier versions can no longer open it. Upgrade all applications that share an index folder at the same time, including the nodes of a search network.
  • .NET Framework projects use the GroupDocs.Search package. Replace GroupDocs.Search.NETFramework with GroupDocs.Search, which now includes the .NET Framework 4.6.2 and 4.8 builds and selects the right build automatically. There is no version 26.11 of GroupDocs.Search.NETFramework.
  • All nodes of a search network are upgraded together. Nodes of versions 26.10 and 26.11 do not connect. To protect the network, set the same TcpSettings.NetworkKey on every node: nodes with different keys, or a node with a key and a node without one, do not connect, and both write the reason to their logs. A node without a key works as before and writes a warning to its log.
  • Messages of a search network are signed but not encrypted. Keep the nodes in a trusted network or connect them through an encrypted tunnel.
  • Lost nodes are detected. A node that sends nothing for 15 seconds, for example a process paused in a debugger, is treated as lost: calls that wait for it throw SearchException, and its documents are sent to other nodes or reported by the ErrorOccurred event. By default the master node waits for responses without a limit; set TcpSettings.ResponseTimeout to limit the wait.
  • IndexSettings.SearchThreads is shared by all searches of an index. One search uses as many threads as before, but K simultaneous searches now run on max(K, SearchThreads) threads instead of K × SearchThreads.
  • Only new indexes get the extended alphabet. Indexes of earlier versions, including those updated by IndexUpdater, split the words of these scripts as before, and Index.Merge refuses to merge a new index with an older one. To merge them, export the Alphabet and TokenizationDictionary of the older index with ExportDictionary and import them into the new index with Clear and ImportDictionary before adding documents.
  • Occurrence limits count the parts of dates once. SearchOptions.MaxTotalOccurrenceCount and MaxOccurrenceCountPerTerm count each occurrence of a day, month or year once per search, or once per chunk, instead of once for every date of the range. A date range that reached a limit before may now find all its dates.
  • Search by chunks in an index repository goes index by index. IndexRepository.Search with IsChunkSearch returns the first chunk of the first index. Continue with IndexRepository.SearchNext while SearchResult.NextChunkSearchToken is not null, or search without chunks to get all documents in one result.
  • OCR in a separate extraction process uses ExtractionHost.OcrConnector. Set the connector in the program of the extraction process and build that program with the same GroupDocs.Search version as the application. The OCR connector of the indexing options is not used there.
  • Network indexing progress is reported per shard response. NodeEventHub.IndexingProgressChanged occurs once for every response of a shard, which may report several documents, so ProcessedDocuments can grow by more than one.

How to get the update

NuGet

  • NuGet – Install the latest package. It works in .NET and .NET Framework projects and selects the build for the target framework automatically: .NET Framework 4.6.2 or 4.8, .NET Standard 2.1, .NET 6.0, .NET 8.0 or .NET 10.0.
# Package Manager
NuGet\Install-Package GroupDocs.Search -Version 26.11.0

# .NET CLI
dotnet add package GroupDocs.Search --version 26.11.0

Direct download

Resources