Introduction
TeamWox groupware initially offers effective and convenient search over modules from standard delivery. But how to handle data from custom modules? The answer is obvious - you need to enhance your module functionality with searching feature using TeamWox API.
The Search module enables data searching in all TeamWox modules. When you search for data, user access permissions are taken into account. In this article we will consider working with the Search module's API. Using the Hello World module as an example, you will learn how to organize data indexing and searching for custom module.
How the 'Search' Module Works
'Search' Module API
Example of Implementing Search for a Custom Module
1. Including 'Search' Module API into a Project
2. Implementing the ISearchClient Interface
3. Extending Module's Manager
4. Indexing
4a. Indexing a Single Record
4b. Full Re-indexing of a Module
5. Display Search Query Results
6. Demonstration
How the 'Search' Module Works
Searching information in TeamWox groupware is performed using the Search module, that comes in standard delivery. This module equally works with all other TeamWox modules - first it indexes and processes their data and then users are able to search information on a separate page by entering search queries.
According to the TeamWox ideology, to extend module's functionality you must implement an interface from TeamWox API. As for search, in custom module we must implement the ISearchClient interface provided by search module API.
Consider the principle of the 'Search' module on the following scheme:
Next, for simplicity we will imply the search module as Search and custom module or module from standard delivery as Module.
There are two interaction modes of Search and Module.
1. Indexing Data Real-Time. Module applies to Search to inform of changed data.
2. Full Re-indexing. Search applies to a certain Module in case of full re-indexing that is carried out either by schedule or by request of TeamWox administrator. In this case Module provides all available information about its data to Search.
Module's developer decides how to store data (in DBMS or in file storage) and what data to provide Search for indexing and display.
Display Search Results
Search stores only identifiers of Module's records (+ other service information) in a special storage (see also the How to Speed Up TeamWox - Store Components on Different Drives article). To display search query results, Search applies to Module to fill information about found record. Record id is passed as an argument.
Search' Module API
Search' Module API (this module ID is TWX_SEARCH) is included into TeamWox SDK. You can find it in the "<TeamWox SDK_install_dir>\SDK\API\Search.h" file. Consider API interfaces:
- ISearchManager - the main interface used to invoke Search functionality.
- ISearchIndexer - interface of indexing data.
- ISearchClient - Module must implement this interface to interact with Search.
- ISearchResult - interface to display search results. This object is passed into module to fill data. "Set" methods are used to fill object with data and "Get" methods - to display data on page with search results.
- ISearchRequest - interface of requesting data with custom access rights.
You can find the complete description of the 'Search' module in TeamWox SDK documentation.
Example of Implementing Search for a Custom Module
Consider the implementation of custom module search on example of the Hello World module. As a starting point you can download and unzip the archive for the previous publication Setting Up Custom Modules Environment - Part 2 from TeamWox SDK articles series. The final variant with all modifications described is available in attachment to this article.
Currently, for the Hello World module we have implemented the following features: adding records and storing them in DBMS and file storage, and also creating reports for these data. The next step of extending functionality - ability to index module's data for their searching, and also maintaining search index in up-to-date state when modifying or deleting data.
Our implementation will be based on the Board module source code, that is available in TeamWox SDK.
1. Including 'Search' Module API into a Project
1. Open the "HelloWorld.sln" project, then in Visual Studio open the "stdafx.h" file.
2. Include the "Search.h" file. In our example module project is located in the "<TeamWox SDK_install_dir>\Modules\HelloWorld\" folder, so we will use the relative path up two levels. Correct it if you have unpacked the project into a different folder.
//--- #include "..\..\SDK\API\Core.h" #include "..\..\SDK\API\Search.h" //---
3. Add the "<TeamWox SDK_install_dir>\SDK\API\Search.h" file to the project by placing it to the "\Header Files\API\" folder.
2. Implementing the ISearchClient Interface
1. In the project add new class for the search client by placing it in managers category (according to accepted hierarchy).
2. Declare the CHelloWorldSearch class that implements methods of the ISearchClient interface. We will implement the ISearchClient::Release method right here in class declaration.
//+------------------------------------------------------------------+ //| TeamWox | //| Copyright 2006-2010, MetaQuotes Software Corp. | //| https://www.metaquotes.net | //+------------------------------------------------------------------+ #pragma once //+------------------------------------------------------------------+ //| Class for the search client | //+------------------------------------------------------------------+ class CHelloWorldSearch : public ISearchClient { private: IServer *m_server; // TeamWox server's interface class CHelloWorldManager &m_manager; // our internal interface public: CHelloWorldSearch(IServer *server,class CHelloWorldManager &manager); ~CHelloWorldSearch(); //--- Interface methods void Release() { delete this; } TWRESULT Reindex(ISearchManager *search_mngr); TWRESULT FillResult(const Context *context, ISearchResult *result); TWRESULT ModifyRequest(const Context *context, ISearchRequest *request); private: void operator=(CHelloWorldSearch&) {} }; //+------------------------------------------------------------------+
3. Specify the ISearchClient interface (and the CHelloWorldSearch class that implements it) in the list Hello World module's implemented interfaces.
//+------------------------------------------------------------------+ //| Get module's interfaces | //+------------------------------------------------------------------+ TWRESULT CHelloWorldModule::GetInterface(const wchar_t *name, void **iface) { //--- Check if(name==NULL || iface==NULL) return(RES_E_INVALID_ARGS); //--- if(StringCompareExactly(L"IToolbar",name)) { *iface=&m_toolbar; return(RES_S_OK); } if(StringCompareExactly(L"IModuleMainframeTopBar",name)) { *iface=static_cast<IModuleMainframeTopBar*>(this); return(RES_S_OK); } if(StringCompareExactly(L"IModuleTips", name)) { *iface=static_cast<IModuleTips*>(this); return(RES_S_OK); } if(StringCompareExactly(L"IWidgets",name)) { *iface=&m_widgets; return(RES_S_OK); } if(StringCompareExactly(L"ISearchClient",name)) { *iface=new (std::nothrow) CHelloWorldSearch(m_server,m_manager); if(*iface==NULL) ReturnError(RES_E_OUT_OF_MEMORY); return(RES_S_OK); } //--- return(RES_E_NOT_FOUND); }
4. Re-indexing and displaying search results in fact will be implemented in manager's module, while in the ISearchClient::Reindex and ISearchClient::FillResult methods we will simply pass control into the appropriate methods of module's manager.
- ISearchClient::Reindex
//+------------------------------------------------------------------+ //| Full Re-indexing of Module | //+------------------------------------------------------------------+ TWRESULT CHelloWorldSearch::Reindex(ISearchManager *search_manager) { //--- Start re-indexing in manager return(m_manager.SearchReindex(search_manager)); }
- ISearchClient::FillResult
//+------------------------------------------------------------------+ //| Fill search result | //+------------------------------------------------------------------+ TWRESULT CHelloWorldSearch::FillResult(const Context *context, ISearchResult *result) { //--- return(m_manager.SearchFillResult(context,result)); }
5. In our example there is no need to implement request with custom access rights.
//+------------------------------------------------------------------+ //| Modify requests | //+------------------------------------------------------------------+ TWRESULT CHelloWorldSearch::ModifyRequest(const Context* /*context*/, ISearchRequest* /*request*/) { return(RES_E_NOT_IMPLEMENTED); }
3. Extending Module's Manager
In module's manager we have to implement previously mentioned methods of re-indexing records (SearchReindex) and displaying search results (SearchFillResult). For this we have to get reference to the ISearchManager interface.
On updating record (the CHelloWorldManager::Update method) we will invoke procedure of its indexing (the SearchReindexRecord method), and on deleting record (the CHelloWorldManager::InfoDelete method) - we will delete record information and its comments from the search index (the SearchRemoveRecord method).
1. In class of module's manager declare pointer to the search manager interface.
//+------------------------------------------------------------------+ //| Module's manager | //+------------------------------------------------------------------+ class CHelloWorldManager : public IReportsProvider { private: static HelloWorldRecord m_info_records[]; // List of public information static HelloWorldRecordAdv m_advanced_records[]; // List of information with limited access //--- SET THE VALUE ONLY WHEN INITIALIZING IServer *m_server; // Reference to server IFilesManager *m_files_manager; // files manager ISearchManager *m_search_manager; // 'Search' module's manager IComments *m_comments_manager; // Comments manager
2. As the Search is a separate NON SYSTEM module, we must get its interface (and interfaces of all other NON SYSTEM modules) on the second step of initialization.
- Description
//+------------------------------------------------------------------+ //| Module's manager | //+------------------------------------------------------------------+ class CHelloWorldManager : public IReportsProvider { public: CHelloWorldManager(); ~CHelloWorldManager(); //--- TWRESULT Initialize(IServer *server, int prev_build); TWRESULT PostInitialize();
- Implementation
//+------------------------------------------------------------------+ //| Second step of manager initialization | //+------------------------------------------------------------------+ TWRESULT CHelloWorldManager::PostInitialize() { //--- Checks if(m_server==NULL) ReturnError(RES_E_INVALID_ARGS); //--- Get reference to interface of the search. In case of failure just continue to work (don't inform of it) m_server->GetInterface(TWX_SEARCH,L"ISearchManager",(void**)&m_search_manager); //--- return(RES_S_OK); }
3. In module's manager declare the following methods:
- SearchReindexRecord - Indexes records when it is changed
- SearchRemoveRecord - Deletes record from search index
- SearchFillResult - Generates page with search results
- SearchReindex - Performs full re-indexing of module's data
- ReadDescription - Utility method of reading record's contents from file storage
- SearchRecordAdd - Indexes contents read from file storage
//+------------------------------------------------------------------+ //| Module's manager | //+------------------------------------------------------------------+ class CHelloWorldManager : public IReportsProvider { public: //--- Search TWRESULT SearchReindex(ISearchManager *search_manager); TWRESULT SearchFillResult(const Context *context,ISearchResult *result); private: //--- TWRESULT ReadDescription(HelloWorldRecord* record, wchar_t** description, size_t* description_len, size_t* description_alocated); //--- TWRESULT SearchRecordAdd(ISearchManager* search_manager, ISearchIndexer* indexer, HelloWorldRecord* record, const wchar_t* description, size_t description_len); //--- TWRESULT SearchReindexRecord(const Context* context,HelloWorldRecord* record); TWRESULT SearchRemoveRecord(const Context* context,INT64 record_id);
4. Add the call of re-indexing procedure to the CHelloWorldManager::Update method. So that on any change of record its information indexing is guaranteed.
//+------------------------------------------------------------------+ //| Add and save new record in HELLOWORLD table | //+------------------------------------------------------------------+ TWRESULT CHelloWorldManager::Update(const Context *context,HelloWorldRecord *record,AttachmentsModify *attachments) { ............................. //--- Re-index the message SearchReindexRecord(context,record); //--- return(RES_S_OK); }
5. Add the call of procedure of deleting record information and its comments from search index to the CHelloWorldManager::InfoDelete method.
//+------------------------------------------------------------------+ //| Delete record from HELLOWORLD table | //+------------------------------------------------------------------+ TWRESULT CHelloWorldManager::InfoDelete(const Context *context,INT64 id) { ............................. //--- Remove record from search index SearchRemoveRecord(context,id); //--- return(RES_S_OK); }
4. Indexing
In TeamWox groupware indexing is performed with a certain delay that depends on frequency of adding new content and its amount. This approach optimizes the load on server. First data are accumulated, and then (when certain amount is accumulated or after some time) they are processed in batch mode.
In the HelloWorldManager.h file create the EnHelloWorldSearchId enumeration containing two IDs. These IDs will determine categories of indexed records: first - the contents of record, second - contents of comments.
enum EnHelloWorldSearchId { HELLOWORLD_BODY_SEARCH_ID =0, HELLOWORLD_COMMENT_SEARCH_ID=1 };
4a. Indexing a Single Record
1. First create pointer to the ISearchIndexer interface. Every time when re-indexing starts, new indexer object is created. It accumulates all data related to a record.
//--- Get indexer's object CSmartInterface<ISearchIndexer> indexer; if(RES_FAILED(res=m_search_manager->GetSearchIndexer(&indexer)
2. Record's contents are read and then after processing they are added to the indexer.
//--- Read description ReadDescription(record,&description,&description_len,&description_allocated); //--- Add/Update record in search index SearchRecordAdd(m_search_manager,indexer,record,description,description_len);
3. Comments is the standard TeamWox feature. Developers don't need to create procedures of indexing their contents. All the necessary tools are provided by the IComments interface. Particularly, indexing comments is performed using the IComments::SearchReindexRecord method.
//--- Indexing comments m_comments_manager->SearchReindexRecord(context->sql,m_search_manager,indexer,0,NULL,0, HELLOWORLD_COMMENT_TYPE_COMMENTS,0,record->id, HELLOWORLD_COMMENT_SEARCH_ID);
4. Key steps in implementing the SearchRecordAdd method.
- First we will put record's data into the indexer's object. In our case it is plain text without HTML formatting tags.
In our example we will give more weight to titles (2.0) and less weight to description (1.0). So that record with keywords in its title will be more relevant than record with the same keywords only in description.
//--- indexer->AddText(SEARCH_PLAIN_TEXT,record->name,2.0f); //--- if(description!=NULL && description_len>0) indexer->AddText(SEARCH_PLAIN_TEXT,description,description_len,1.0f);
- After that we have to juxtapose these data with some unique id using the ISearchManager::Insert method. In our example it will consist of two parts: the HELLOWORLD_BODY_SEARCH_ID constant (from the EnHelloWorldSearchId enumeration we have created) and record's id in DBMS.
//--- Add indexed data
search_manager->Insert(HELLOWORLD_MODULE_ID,SEARCH_GENERIC,HELLOWORLD_BODY_SEARCH_ID,record->id,indexer);In general, it is enough to specify only one of the record1 or record2 arguments in the ISearchManager::Insert method. Ability to specify second id makes implementation of search more flexible.
5. When record's contents are deleted, the search index is purged from record contents and its comments' contents.
//+------------------------------------------------------------------+ //| Delete record from search index | //+------------------------------------------------------------------+ TWRESULT CHelloWorldManager::SearchRemoveRecord(const Context *context,INT64 record_id) { //--- Checks if(context==NULL || m_search_manager==NULL || m_comments_manager==NULL) ReturnError(RES_E_INVALID_ARGS); //--- m_search_manager->Remove(HELLOWORLD_MODULE_ID,SEARCH_GENERIC,HELLOWORLD_BODY_SEARCH_ID,record_id); //--- m_comments_manager->SearchRemove(context,m_search_manager,HELLOWORLD_COMMENT_TYPE_COMMENTS,record_id,HELLOWORLD_COMMENT_SEARCH_ID); //--- return(RES_S_OK); }
4b. Full Re-indexing of a Module
This procedure is similar to indexing a single record. The Search module clears the search index and sequentially indexes all records from selected modules. The difference is in the range of use - in some cases it is more efficiently to perform full re-indexing rather than to index each record:
- Emergency stop of server. In this case full re-indexing is launched automatically.
- Emergency stop of server while performing full re-indexing. In this case once started procedure must be completed.
- Errors in a module.
- Massive deleting of records.
Full re-indexing is performed in the service time according to schedule (once a week). For this reason search indices are not included into backup.
Consider implementing the SearchReindex method of full re-indexing.
1. In one query select all records.
//--- CSmartSql sql(m_server); if(sql==NULL) ReturnError(RES_E_FAIL); //--- Text of SQL request to get row from HELLOWORLD table by specified ID. char query_select_string[]="SELECT id,name,category,description_id,attachments_id FROM helloworld"; //--- "Bind" data to parameters of request SqlParam params_query_select_string[] ={ SQL_INT64, &record.id, sizeof(record.id), SQL_WTEXT, record.name, sizeof(record.name), SQL_LONG, &record.category, sizeof(record.category), SQL_INT64, &record.description_id,sizeof(record.description_id), SQL_TEXT, record.attachments_id,sizeof(record.attachments_id) }; //--- Send request if(!sql.Query(query_select_string, NULL, 0, params_query_select_string, _countof(params_query_select_string))) ReturnErrorExt(RES_E_SQL_ERROR,NULL,"helloworld record query failed");
2. Perform indexing of all records one by one. Just like for a single record, we perform indexing of all records and their comments.
//--- Get response while(sql->QueryFetch()) { description_len=0; //--- Read description ReadDescription(&record,&description,&description_len,&description_allocated); //--- Add/Update record in search index SearchRecordAdd(m_search_manager,indexer,&record,description,description_len); //--- Indexing comments m_comments_manager->SearchReindexRecord(sql.Interface(), search_manager, indexer,0,NULL,0, HELLOWORLD_COMMENT_TYPE_COMMENTS, 0,record.id, HELLOWORLD_COMMENT_SEARCH_ID);
3. It is recommended to display debug messages about current state of re-indexing process (for example, after every 500 successfully indexed records). At the end display the total number of indexed records.
//--- if((++count)%500==0) ExtLogger(NULL, LOG_STATUS_INFO) << count << " records re-indexed"; //--- Clean-up ZeroMemory(&record,sizeof(record)); } //--- ExtLogger(NULL, LOG_STATUS_INFO) << count << " records re-indexed"; //--- if(description!=NULL) { delete [] description; /*description=NULL;*/ } //--- Release request sql.QueryFree(); //--- return(RES_S_OK);
5. Display Search Query Results
Search query returns only IDs of found records. Module's developer decides what part of record's contents to display using these IDs.
Consider implementing the ISearchClient::FillResult method (in the CHelloWorldManager::SearchFillResult method).
1. Get IDs of found records.
//--- Check parameters const SearchResult *data=result->GetResult(); if(data==NULL || data->module_id!=HELLOWORLD_MODULE_ID) ReturnErrorExt(RES_E_INVALID_ARGS,context,"invalid parameters for search result");
2. We have to determine, what type of records the found id is referring to - either to record itself, or to its comments.
//--- Determine where to get information from switch(data->record1) { case HELLOWORLD_COMMENT_SEARCH_ID: m_comments_manager->SearchFillResult(context,result,data->record2,comment_url,_countof(url),&record_id); break; case HELLOWORLD_BODY_SEARCH_ID: default: record_id=data->record2; }
In case of comments all the necessary tools are provided by the IComments interface. Particularly, displaying search results in comments is performed using the IComments::SearchFillResult method.
3. Get record from DBMS by found id.
//--- if(record_id==0) return(RES_E_NOT_FOUND); //--- if(RES_FAILED(res=Get(context,record_id,&record))) { ExtLogger(context,LOG_STATUS_ERROR) << "failed to get record info #" << record_id << " for search [" << res << "]"; return(res); }
4. Now we have to fill page with search results. In general, such a page consists of record title (as hyperlink), brief description (includes part of found record's contents), information about modification date, list of assigned workers and so forth. In our example we will limit ourselves with title and description. Modification date is displayed in search result automatically.
- Title
//--- Set the title
result->SetTitle(record.name);- Link for title
//--- Generate URL if(comment_url[0]!=0) StringCchPrintf(url,_countof(url),L"/helloworld/number_two/view/%I64d?%s",record_id,comment_url); else StringCchPrintf(url,_countof(url),L"/helloworld/number_two/view/%I64d",record_id); //--- result->SetUrl(url);
- Description
//--- Generate description if(data->record1==HELLOWORLD_BODY_SEARCH_ID && record.description_id>0) { if(RES_SUCCEEDED(m_files_manager->FileInfoGet(NULL,record.description_id,&file_info)) && file_info.size>0 && (file_info.size%2)==0) { wchar_t description_stat[512]={0}; //--- if(file_info.size>sizeof(description_stat)) { UINT64 len =file_info.size; wchar_t *description=new wchar_t[size_t(len/sizeof(wchar_t))]; //--- if(description==NULL) ReturnErrorExt(RES_E_OUT_OF_MEMORY,context,"failed not enough memory for description content"); //--- if(RES_SUCCEEDED(m_files_manager->FileRead(NULL,record.description_id,description,&len,0))) result->AddText(SEARCH_PLAIN_TEXT,description,int(len/sizeof(wchar_t))); //--- delete []description; } else { UINT64 len=file_info.size; if(RES_SUCCEEDED(m_files_manager->FileRead(NULL,record.description_id,description_stat,&len,0))) result->AddText(SEARCH_PLAIN_TEXT,description_stat,int(len/sizeof(wchar_t))); } } }
6. Demonstration
Let's demonstrate implemented features.
1. On the Page 2 add new record in WYSIWYG editor.
2. Wait a while of perform full re-indexing of the Hello World module.
3. Search previously added data.
4. Search results are displayed on a separate page.
Conclusion
As you see, it is not difficult to implement the search functionality in your module. In the second part we will talk about filtering search results.
- How to Add a Ready-made Module to TeamWox
- How to Add Page into TeamWox Module
- Building User Interface
- Interaction with DBMS
- Creating Custom Reports
- TeamWox File Storage - Part 1
- TeamWox File Storage - Part 2
- Setting Up Custom Modules Environment - Part 1
- Setting Up Custom Modules Environment - Part 2
2011.09.07