lifti Wiki Rss Feed

Updated Wiki: Home

MikeGoatly — Mon, 27 May 2013 09:14:06 GMT

LIFTI provides a simple API that allows you to build a searchable full text index. This can be useful for document retrieval or simply allowing .NET objects to be searched for by associated text.

There are 3 samples (4 if you count the timing comparison application) included in the solution – these should give you a pretty good idea how to get started. They’re also included as binaries in the download.

This project started as a single blog entry and has grown into something that may actually be genuinely useful. The basic principle behind LIFTI is that it allows you to index items (or their keys) against text associated to them - once the text has been indexed it is stored in a compact and searchable form, meaning that the overall text for each item need not be persisted.

The documentation is now up to date with the 0.5 release – I’ll probably look to extend the documentation further in the near future..

LIFTI is currently compiled against version 3.5 of the .NET Framework – I’m probably going to be moving it to version 4.0 in the next release – if you want to influence this in one way or the other make your thoughts known!

Installing LIFTI

Manual Installation

LIFTI is available from the downloads page – download and extract the binaries yourself and reference them in your project.

Nuget

LIFTI is also available as a nuget package. Nuget is a great way to add and manage your project dependencies.

Links

My blog (http://www.goatly.net) is where a lot of articles about LIFTI have, and will continue, to appear.

Updated Wiki: Constructing a full text indexes

MikeGoatly — Mon, 27 May 2013 09:11:29 GMT

Constructing an in-memory full text index

The code to create a new full text index class looks like this:

var index = new FullTextIndex<MyType>(); 
var updatableIndex = new UpdatableFullTextIndex<MyType>();

From this you can see that all forms of the full text index class are generic types defining the type of item that the index will contain.

There are two approaches that you can take to indexing items: indexing keys and indexing .NET objects.

Indexing keys

The first approach is to index a unique key for your items. This means that the index itself will only store a reference or id to the indexed item. This might be an integer, a string, a Guid or other primitive type.
The constructor for this approach looks like this (Note that the FullTextIndex generic type is int):

var index = new FullTextIndex<int>()

The main benefit to this approach is that once an item has been indexed the associated text doesn’t need to be retained in memory.

Indexing .NET objects

If your items are always loaded into memory and they contain all the text that you want to index, you can index them directly:

var index = new FullTextIndex<Customer>();

Here the generic type of FullTextIndex is Customer.

Obviously the drawback to this approach is that all the instances are retained in memory, along with all their associated properties, including the text.

Constructing a persisted full text index

Full text indexes can be created by passing the path to the persisted file into the constructor:

var indexFilePath = Path.Combine(
    Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData),
    Path.Combine("MyApplication", "MyTextIndex.dat");

var index = new PersistedFullTextIndex<int>(indexFilePath);

As with the FullTextIndex and UpdatableFullTextIndex, the persisted full index is a generic type. It is strongly recommended that you use a key-based approach to the index, by restricting the type that you store in the index to a primitive type, e.g. int, string, etc.

If you do need to store some other type in the index, such as a Guid or custom class, then this blog post should get you started.

Updated Wiki: Home

MikeGoatly — Mon, 20 May 2013 10:20:23 GMT

LIFTI provides a simple API that allows you to build a searchable full text index. This can be useful for document retrieval or simply allowing .NET objects to be searched for by associated text.

Installing LIFTI

Manual Installation

LIFTI is available from the downloads page – download and extract the binaries yourself and reference them in your project.

Nuget

LIFTI is also available as a nuget package. Nuget is a great way to add and manage your project dependencies.

Updated Wiki: Documentation

MikeGoatly — Thu, 20 Sep 2012 21:22:13 GMT

Getting started

The Quick Start is a really good place to start – it will show you briefly how to use the index classes and describes them at a high level.

The downloadable source for each release contains several samples for you to investigate. You can also download the API documentation generated from the source code.

Basic operations

Constructing full text indexes
Find out how to construct the various full text index classes, FullTextIndex, UpdatableFullTextIndex and PersistedFullTextIndex.

Word splitters
Understand the various ways that word splitters can be used to process text being indexed and searched.

Searching the full text index
Learn the LIFTI query syntax used to parse queries with the LiftiQueryParser.

Advanced documentation

Locking behaviour

Persistence File Format

Updated Wiki: Persistence file format

mikegoatly — Thu, 20 Sep 2012 21:20:37 GMT

Updated Wiki: Home

MikeGoatly — Wed, 06 Jul 2011 13:06:57 GMT

LIFTI provides a simple API that allows you to build a searchable full text index. This can be useful for document retrieval or simply allowing .NET objects to be searched for by associated text.

There are 3 samples (4 if you count the timing comparison application) included in the solution – these should give you a pretty good idea how to get started. They’re also included as binaries in the download.

This project started as a single blog entry and has grown into something that may actually be genuinely useful. The basic principle behind LIFTI is that it allows you to index items (or their keys) against text associated to them - once the text has been indexed it is stored in a compact and searchable form, meaning that the overall text for each item need not be persisted.

The documentation is now up to data with the 0.5 release – I’ll probably look to extend the documentation further in the near future..

Installing LIFTI

Manual Installation

LIFTI is available from the downloads page – download and extract the binaries yourself and reference them in your project.

Nuget

LIFTI is also available as a nuget package. Nuget is a great way to add and manage your project dependencies.

Updated Wiki: Home

MikeGoatly — Mon, 27 Jun 2011 18:44:36 GMT

Project Description
LIFTI provides a simple API that allows you to build a searchable full text index. This can be useful for document retrieval or simply allowing .NET objects to be searched for by associated text.

There are 3 samples (4 if you count the timing comparison application) included in the solution – these should give you a pretty good idea how to get started. They’re also included as binaries in the download.

This project started as a single blog entry and has grown into something that may actually be genuinely useful. The basic principle behind LIFTI is that it allows you to index items (or their keys) against text associated to them - once the text has been indexed it is stored in a compact and searchable form, meaning that the overall text for each item need not be persisted.

The documentation is now up to data with the 0.5 release – I’ll probably look to extend the documentation further in the near future..

Nuget

LIFTI is available as a nuget package. Nuget is a great way to add and manage your project dependencies.

Updated Wiki: Documentation

MikeGoatly — Mon, 13 Jun 2011 16:58:42 GMT

Getting started

The Quick Start is a really good place to start – it will show you briefly how to use the index classes and describes them at a high level.

The downloadable source for each release contains several samples for you to investigate. You can also download the API documentation generated from the source code.

Basic operations

Constructing full text indexes
Find out how to construct the various full text index classes, FullTextIndex, UpdatableFullTextIndex and PersistedFullTextIndex.

Word splitters
Understand the various ways that word splitters can be used to process text being indexed and searched.

Searching the full text index
Learn the LIFTI query syntax used to parse queries with the LiftiQueryParser.

Advanced documentation

Locking behaviour

Updated Wiki: Home

MikeGoatly — Fri, 10 Jun 2011 15:04:13 GMT

There are 3 samples (4 if you count the timing comparison application) included in the solution – these should give you a pretty good idea how to get started. They’re also included as binaries in the download.

This project started as a single blog entry and has grown into something that may actually be genuinely useful. The basic principle behind LIFTI is that it allows you to index items (or their keys) against text associated to them - once the text has been indexed it is stored in a compact and searchable form, meaning that the overall text for each item need not be persisted.

The documentation is now up to data with the 0.5 release – I’ll probably look to extend the documentation further in the near future..

Updated Wiki: Home

MikeGoatly — Thu, 09 Jun 2011 21:28:24 GMT

There are 3 samples (4 if you count the timing comparison application) included in the solution – these should give you a pretty good idea how to get started. They’re also included as binaries in the download.

This project started as a single blog entry and has grown into something that may actually be genuinely useful. The basic principle behind LIFTI is that it allows you to index items (or their keys) against text associated to them - once the text has been indexed it is stored in a compact and searchable form, meaning that the overall text for each item need not be persisted.

The documentation is now up to data with the 0.5 release – I’ll probably look to extend the documentation further in the near future..

Updated Wiki: Documentation

MikeGoatly — Thu, 09 Jun 2011 21:26:36 GMT

Getting started

The Quick Start is a really good place to start – it will show you briefly how to use the index classes and describes them at a high level.

The downloadable source for each release contains several samples for you to investigate. You can also download the API documentation generated from the source code.

Basic operations

Constructing full text indexes
Find out how to construct the various full text index classes, FullTextIndex, UpdatableFullTextIndex and PersistedFullTextIndex.

Word splitters
Understand the various ways that word splitters can be used to process text being indexed and searched.

Searching the full text index
Learn the LIFTI query syntax used to parse queries with the LiftiQueryParser.

Advanced documentation

Locking behaviour

Updated Wiki: Constructing a full text indexes

MikeGoatly — Thu, 09 Jun 2011 21:08:21 GMT

Constructing an in-memory full text index

The code to create a new full text index class looks like this:

var index = new FullTextIndex<MyType>(); 
var updatableIndex = new UpdatableFullTextIndex<MyType>();

From this you can see that all forms of the full text index class are generic types defining the type of item that the index will contain.

There are two approaches that you can take to indexing items: indexing keys and indexing .NET objects.

Indexing keys

var index = new FullTextIndex<int>()

The main benefit to this approach is that once an item has been indexed the associated text doesn’t need to be retained in memory.

Indexing .NET objects

If your items are always loaded into memory and they contain all the text that you want to index, you can index them directly:

var index = new FullTextIndex<Customer>();

Here the generic type of FullTextIndex is Customer.

Obviously the drawback to this approach is that all the instances are retained in memory, along with all their associated properties, including the text.

Constructing a persisted full text index

Full text indexes can be created by passing the path to the persisted file into the constructor:

var indexFilePath = Path.Combine(
    Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData),
    Path.Combine("MyApplication", "MyTextIndex.dat");

var index = new PersistedFullTextIndex<int>(indexFilePath);

Updated Wiki: Constructing an in-memory full text index

MikeGoatly — Thu, 09 Jun 2011 18:15:02 GMT

Constructing an in-memory full text index

The code to create a new full text index class looks like this:

var index = new FullTextIndex<MyType>(); 
var updatableIndex = new UpdatableFullTextIndex<MyType>();

From this you can see that all forms of the full text index class are generic types defining the type of item that the index will contain.

There are two approaches that you can take to indexing items: indexing keys and indexing .NET objects.

Indexing keys

var index = new FullTextIndex<int>()

The main benefit to this approach is that once an item has been indexed the associated text doesn’t need to be retained in memory.

Indexing .NET objects

If your items are always loaded into memory and they contain all the text that you want to index, you can index them directly:

var index = new FullTextIndex<Customer>();

Here the generic type of FullTextIndex is Customer.

Obviously the drawback to this approach is that all the instances are retained in memory, along with all their associated properties, including the text.

Updated Wiki: Temporary Post Used For Theme Detection (64b6ffae-5ded-44c1-b553-8dcf597ee12b - 3bfe001a-32de-4114-a6b4-4005b770f6d7)

MikeGoatly — Thu, 09 Jun 2011 18:03:52 GMT

This is a temporary post that was not deleted. Please delete this manually. (aa2e0171-ac71-4684-a238-08240730f1f4 - 3bfe001a-32de-4114-a6b4-4005b770f6d7)

Updated Wiki: Using the LIFTI query language

MikeGoatly — Thu, 09 Jun 2011 15:58:06 GMT

The LIFTI Query Language

When you have constructed a FullTextIndexer instance and set its QueryParser property to a LiftiQueryParser, you will be able to search the full text index using the LIFTI query language. This will allow you to write queries using the following operators:

Query	Meaning
West	West must appear in the text.
doc*	Words that starts with doc must appear in the text.
west & wing	The words west and wing must appear in the text
west wing	The words west and wing must appear in the text - the default operator is & if none is specified between search words
west \| wing	The words west or wing must appear in the text
west ~ wing	west and wing must appear near to each other (within 5 words) in the text
west ~> wing	west must be closely followed by wing (within 5 words) in the text
west >> wing	west must precede wing anywhere in the text
"the west wing"	The words the west wing must appear in sequence in the indexed text.
"notr* dam*"	A word starting with notr must be immediately followed by a word starting with dam.

If you specify two search words in a query without a joining operator, then they will automatically be combined with an and (&) operator, so the queries west & wing is equivalent to west wing. (Just to clarify, searching for "west wing" is different - this indicates that the words should appear in sequence - the previous searches only enforce that the words appear somewhere in the text.)

Longer queries

Query operators can be combined, so you can write a query such as the & west & wing to search for items that have all 3 words indexed against them.
Naturally you can mix and match operators, so "west wing" & "oval office" would search for items with the phrases "west wing" and "oval office" indexed against them.
When combining operators you need to be aware of operator precedence. Operators will be processed in the following order:

Positional operators (~, ~> and >>)
And (&)
Or (|)

This means that west & wing | oval office would search for items containing both west and wing, or items containing oval and office.

You can specify your own precedence by using brackets
This means that you could write west & (wing | oval) & office to search for items containing west and office and either wing or oval.

Updated Wiki: Word splitters

MikeGoatly — Thu, 09 Jun 2011 15:54:19 GMT

Word splitters

In LIFTI a word splitter is a class implementing IWordSplitter - these classes are capable of breaking a piece of text into individual words that should be contained in the index. There are currently three word splitters defined in the LIFTI assembly:

WordSplitter (Currently the default)
StemmingWordSplitter
XmlWordSplitter

Word splitters are used in two parts of the LIFTI code:

During the indexing process - to split the text associated to an item into constituent words.
During the searching process - search words are also passed through a splitter, ensuring that adjustments made to a word when it was indexed are also applied to these criteria.

You can set the word splitter on an index using the WordSplitter property:

index.WordSplitter = new StemmingWordSplitter();

If you need different behaviour to be applied to the indexing and searching process, then you can additionally set the SearchWordSplitter property. If SearchWordSplitter is left null it defaults to the value held against the WordSplitter property, meaning that you won’t normally need to set it. The XmlWordSplitter section of this page shows an example of when you might use the SearchWordSplitter property.

WordSplitter

This is the default word splitter that a LIFTI full text index is constructed with. It simply breaks a string apart on white spaces. Characters that are not letters or digits are treated as whitespace, apart from apostrophes which are just ignored. This means that the following statement:

Simon's latest phrase is "Simon’s space-monkeys rule"

Would be broken into the following words:

Simons
latest
phrase
is
space
monkeys
rule

Each word is returned only once as an instance of SplitWord, which also provides an array of integers indicating the positions in the document at which the words were encountered. For the word “Simons” above, that array would have contained 0 and 4.

StemmingWordSplitter

The stemming word splitter breaks words apart using the same rules as the standard word splitter. Each individual word is subsequently processed using an implementation of a Porter stemmer which strips off word suffixes. For example, connection, connects and connecting all become connect.

Although this incurs a small cost, it also makes the index more accessible when the text being indexed is free-form English, as the user is more likely to get relevant results. Additionally it has the effect of making the index smaller, as the number and length of indexed words will be reduced.

Currently only English words are officially supported, as the stemming algorithms differ between languages.

XmlWordSplitter

The XML word splitter is designed to only return words that appear in text content contained within nodes in an XML document.

Consider the following XML:

<document type="phrase">
    The quick brown <emphasis>fox</emphasis> jumped
    over the lazy <emphasis>dog</emphasis>.
</document>

When run through the XML word splitter, the following words (and associated positions) will be returned:

Word	Word index
THE	0, 6
QUICK	1
BROWN	2
FOX	3
JUMPED	4
OVER	5
LAZY	7
DOG	8

Notice that the word “the” is reported at positions 0 and 6 – the word index is relative to first word in the document, regardless of whether there are XML elements that interrupt the flow of the text.

One thing to consider when using the XmlWordSplitter is that you probably won’t want

When you construct an XmlWordSplitter instance, you need to specify the type of word splitting behaviour that should be applied to the chunks of text within the XML document itself.

This example shows how you would configure an index to use an XmlWordSplitter that stems the words that it returns using a StemmingWordSplitter:

var wordSplitter = new StemmingWordSplitter();
index.WordSplitter = new XmlWordSplitter(wordSplitter);
index.SearchWordSplitter = wordSplitter;

Here you can see usage of the SearchWordSplitter property of the index to differentiate the indexing and searching word splitting behaviours. If you really wanted to search using XML text as well, then you wouldn’t need to specify SearchWordSplitter.

Updated Wiki: Quick Start

MikeGoatly — Thu, 09 Jun 2011 09:55:20 GMT

Quick start

Using LIFTI to index .NET classes in-memory

The LIFTI API is really straightforward to use, a simple example would be:

// People is a class defined with properties Name and Remarks
IEnumerable<Person> people = LoadPeople(); 

// Create the index
var index = new FullTextIndex<Person>(); 

// Index the people against the text in their remarks data
index.Index(people, p => p.Remarks);

// Search for all the people with "great" and "boss" in their remarks (in any order)
foreach (Person person in index.Search("great boss"))
{
    Console.WriteLine(person.Name); 
}

By default the full text index will not handle word derivatives, e.g. searching for CALLING will not return items indexed against the word CALL or CALLED. LIFTI provides a stemming word splitter which enables this kind of searching - to enable it just change the full text index constructor to:

var index = new FullTextIndex<Person>()
{
    WordSplitter = new StemmingWordSplitter()
};

You can also use a richer querying syntax to search the index if you want, just change the index so it is using an instance of the LiftiQueryParser class:

var index = new FullTextIndex<Person>()
{
    WordSplitter = new StemmingWordSplitter(),
    QueryParser = new LiftiQueryParser()
};

Updatable indexes

The FullTextIndex class doesn’t allow items to be modified once they are index – it is essentially write-once-per-item. If you need to be able to index the same item again, or remove an item from the index, then you can use the UpdatableFullTextIndex class instead.

Using LIFTI to create a persisted full text index

The FullTextIndex and UpdatableFullTextIndex classes require fully re-populating every time your application starts. If you want your index to be persisted across application restarts, you can use the PersistedFullTextIndex class instead. In this case you will typically want to store the id of the related item in the index - following on from the previous examples, this might be an PersonId of type Int32:

// Create the index - this will automatically load any data that was previously indexed
var index = new PersistedFullTextIndex<int>(@"c:\MyIndexData\PeopleIndex.dat")
{
    WordSplitter = new StemmingWordSplitter()
}; 

// Index new or updated people
index.Index(people, p => p.PersonId, p => p.Remarks);

// Search for all the people with "great" and "boss" in their remarks (in any order)
foreach (Person person in index.Search("great boss"))
{
    Console.WriteLine(person.Name); 
}

Indexes aren’t restricted to storing integers as keys – you can quite easily store strings or other value types. This final example indexes the paths to documents against the text that they contain:

private PersistedFullTextIndex<string> index;

void Main()
{
    this.index = new PersistedFullTextIndex<string>(@"c:\MyIndexData\DocumentIndex.dat")
    {
        WordSplitter = new StemmingWordSplitter(),
        QueryParser = new LiftiQueryParser()
    };

    this.index.Index(Directory.GetFiles(@"c:\MyDocumentStore"), f => File.ReadAllText(f));

    Console.WriteLine("Searching for documents containing the words 'design' and 'document'");
    foreach (var fileName in this.index.Search("design & document"))
    {
        Console.WriteLine("Document file name:", fileName);
    }
}

Updated Wiki: Home

MikeGoatly — Wed, 08 Jun 2011 22:17:47 GMT

There are 3 samples (4 if you count the timing comparison application) included in the solution – these should give you a pretty good idea how to get started. They’re also included as binaries in the download.

This project started as a single blog entry and has grown into something that may actually be genuinely useful. The basic principle behind LIFTI is that it allows you to index items (or their keys) against text associated to them - once the text has been indexed it is stored in a compact and searchable form, meaning that the overall text for each item need not be persisted.

~~The documentation is slowly coming together - I'd be interested in getting feedback regarding any holes that are missing.~~
The documentation is now out of date since the 0.5 release – I'll be updating it over the next couple of days.

Updated Wiki: Home

MikeGoatly — Wed, 08 Jun 2011 21:59:50 GMT

Version 0.5 of the library introduces the PersistedFullTextIndex which automatically persists the index to a backing file store as it is being used. This opens up the possibility of using it in an application to index text associated to data stored in a database that doesn’t provide its own full text indexing services, such as SQL Server Compact.

This project started as a single blog entry and has grown into something that may actually be genuinely useful.

The basic principle behind LIFTI is that it allows you to index items (or their keys) against text associated to them - once the text has been indexed it is stored in a compact and searchable form, meaning that the overall text for each item need not be persisted.

~~The documentation is slowly coming together - I'd be interested in getting feedback regarding any holes that are missing.~~
Important: The documentation is now out of date since the 0.5 release - I'll update it very shortly.

Updated Wiki: Using the FullTextIndexer class

MikeGoatly — Tue, 07 Jun 2011 21:31:39 GMT

The LIFTI API is really straightforward to use, a simple example would be:

// People is an imaginary class with properties Name and Remarks
// People will be indexed on the text in their remarks data
IEnumerable<Person> people = LoadPeople(); 

FullTextIndexer<Person> indexer = new FullTextIndexer<Person>(p => p.Remarks); 
indexer.Index(people); // Index the loaded people

// Search for all the people with "great" and "boss" in their remarks (in any order)
foreach (Person person in indexer.Search("great boss"))
{
    Console.WriteLine(person.Name); 
}

By default the full text indexer will not handle word derivatives, e.g. searching for CALLING will not return items indexed against the word CALL or CALLED. In v0.2 LIFTI provides a stemming word splitter which enables this kind of searching - to enable it just change the full text indexer constructor to:

FullTextIndexer<Person> indexer = new FullTextIndexer<Person>(p => p.Remarks)
{
    WordSplitter = new StemmingWordSplitter()
};

lifti Wiki Rss Feed

Updated Wiki: Home

Installing LIFTI

Manual Installation

Nuget

Links

Tutorials

Recent articles

Older links

Updated Wiki: Constructing a full text indexes

Constructing an in-memory full text index

Indexing keys

Indexing .NET objects

Constructing a persisted full text index

Updated Wiki: Home

Installing LIFTI

Manual Installation

Nuget

Links

Tutorials

Recent articles

Older links

Updated Wiki: Documentation

Getting started

Basic operations

Advanced documentation

Updated Wiki: Persistence file format

Updated Wiki: Home

Installing LIFTI

Manual Installation

Nuget

Links

Tutorials

Recent articles

Older links

Updated Wiki: Home

Nuget

Links

Older links

Updated Wiki: Documentation

Getting started

Basic operations

Advanced documentation

Updated Wiki: Home

Links

Older links

Updated Wiki: Home

Links

Older links

Updated Wiki: Documentation

Getting started

Basic operations

Advanced documentation

Updated Wiki: Constructing a full text indexes

Constructing an in-memory full text index

Indexing keys

Indexing .NET objects

Constructing a persisted full text index

Updated Wiki: Constructing an in-memory full text index

Constructing an in-memory full text index

Indexing keys

Indexing .NET objects

Updated Wiki: Temporary Post Used For Theme Detection (64b6ffae-5ded-44c1-b553-8dcf597ee12b - 3bfe001a-32de-4114-a6b4-4005b770f6d7)

Updated Wiki: Using the LIFTI query language

The LIFTI Query Language

Longer queries

Updated Wiki: Word splitters

Word splitters

WordSplitter

StemmingWordSplitter

XmlWordSplitter

Updated Wiki: Quick Start

Quick start

Using LIFTI to index .NET classes in-memory

Updatable indexes

Using LIFTI to create a persisted full text index

Updated Wiki: Home

Links

Older links

Updated Wiki: Home