TeamWox SDK: Search and Filtering - Part 2

 

Introduction

From the first part of the TeamWox SDK: Search and Filtering article you've learned how to integrate the general mechanism of finding information (provided by the Search system module) into TeamWox custom modules. In this article we will talk about how to use this technology to solve a typical problem for many modules - filtering a large number of records.

  Filtering
  Example Of Implementing Filtering In A Custom Module
    1. Manager Of Programming Languages List
    2. Parameters Of Filtering
    3. Get Data For Filtering
  HTTP API
  User Interface
  Demonstration

 

Filtering

Filtering is a selection of data subset with retaining the way this data is displayed in a module. The need for filtration becomes vital when you navigate a long list of records. As you work with large amounts of data, search and filtering (as a subset of search) using SQL queries become ineffective or even impossible. To implement a full-text search, different DBMS extensions are used.

TeamWox groupware was created according to principles of increased fault-tolerance and maximum performance. These principles became a priority in the "Search" module implementation.

The "Search" module quickly displays search results for large amounts of indexed information. The search is universal and allows you to search standard fields by keywords and display representation in a standard form. For custom modules with non-standard and specific fields a separate implementation is needed. We have considered an example of it in the previous article. If the search mechanism is already implemented in a module, adding filtering is pretty easy.

 

Example Of Implementing Filtering In A Custom Module

The HelloWorld module (included in TeamWox SDK) implements an example of filtering data on the List of Records page. You can download and install the updated TeamWox SDK from this page. The Hello World module source codes are attached to this article.

Let's describe the key points of implementation.

 

1. Manager Of Programming Languages List

The changes here are associated with filtering criteria, which can not be foreseen in the standard search. The principle of full reindexing of records has not changed. But after indexing main text of records, we will run fast indexing of records to be filtered. Indexing is fast because only single field is indexed, not the entire contents of a record. 

1.1. The code of full-text reindexing is virtually unchanged. The control criterion has been added - common type of search:

//+------------------------------------------------------------------+
//| Re-index all records                                             |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldManager::SearchReindex(ISearchManager *search_manager,int search_type)
  {
..............................
      //--- Read description
      if(search_type==SEARCH_GENERIC)
         ReadDescription(&record,&description,&description_len,&description_allocated);
      //--- Add/Update record in search index
      SearchRecordAdd(m_search_manager,indexer,search_type,&record,description,description_len);
      //--- Indexing comments
      if(search_type==SEARCH_GENERIC)
         m_comments_manager->SearchReindexRecord(sql.Interface(),search_manager,indexer,0,NULL,0,
                                                 HELLOWORLD_COMMENT_TYPE_COMMENTS,0,record.id,HELLOWORLD_COMMENT_SEARCH_ID);
..............................
  }

1.2. Declare two constants of user-defined types of search. Enumeration of additional search parameters should exceed the SEARCH_USER constant.

enum EnHelloWorldSearchId
  {
   HELLOWORLD_BODY_SEARCH_ID   =0,
   HELLOWORLD_COMMENT_SEARCH_ID=1,
//---
   HELLOWORLD_SEARCH_FILTER         =SEARCH_USER,
   HELLOWORLD_SEARCH_FILTER_CATEGORY=SEARCH_USER+1
  };

1.3. The method of indexing a single record has undergone minimal changes. As for the full reindexing, the second step is the indexing of data to be filtered.

//+------------------------------------------------------------------+
//| Indexing single record                                           |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldManager::SearchReindexRecord(const Context *context,HelloWorldRecord *record)
  {
.....................................
//--- Add/Update record in search index
   SearchRecordAdd(m_search_manager,indexer,SEARCH_GENERIC,record,description,description_len);
   m_comments_manager->SearchReindexRecord(context->sql,m_search_manager,indexer,0,NULL,0,
                                           HELLOWORLD_COMMENT_TYPE_COMMENTS,0,record->id,HELLOWORLD_COMMENT_SEARCH_ID);
//--- reindex for filter
   SearchRecordAdd(m_search_manager,indexer,HELLOWORLD_SEARCH_FILTER,record,description,description_len);
.....................................
  }

1.4. For user-defined type of search we will only index the field of programming language category.

TWRESULT CHelloWorldManager::SearchRecordAdd(ISearchManager *search_manager,ISearchIndexer *indexer,int search_type,
                                             HelloWorldRecord *record,const wchar_t *description,size_t description_len)
...............................
//---
   if(search_type==SEARCH_GENERIC)
     {
      if(description!=NULL && description_len>0)
         indexer->AddText(SEARCH_PLAIN_TEXT,description,description_len,1.0f);
     }
   else
     {
      indexer->AddUserItem(HELLOWORLD_SEARCH_FILTER_CATEGORY,record->category, 1.0);
     }
...............................

1.5. Don't forget to clean the search index from the filtering data when deleting records.

//+------------------------------------------------------------------+
//| Delete record from search                                        |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldManager::SearchRemoveRecord(const Context *context,INT64 record_id)
  {
//--- Checks
   if(m_search_manager==NULL)                    return(RES_S_OK);
   if(context==NULL || m_comments_manager==NULL) ReturnError(RES_E_INVALID_ARGS);
//---
   m_search_manager->Remove(HELLOWORLD_MODULE_ID,SEARCH_GENERIC,HELLOWORLD_BODY_SEARCH_ID,record_id);
   m_comments_manager->SearchRemove(context,m_search_manager,HELLOWORLD_COMMENT_TYPE_COMMENTS,record_id,HELLOWORLD_COMMENT_SEARCH_ID);
//---
   m_search_manager->Remove(HELLOWORLD_MODULE_ID,HELLOWORLD_SEARCH_FILTER,HELLOWORLD_BODY_SEARCH_ID,record_id);
//---
   return(RES_S_OK);
  }

 

2. Parameters Of Filtering

2.1. Create a list of the filtering parameters. As this list can be further expanded, we will write it as a structure. In this way we won't need to alter the definition of methods and won't lose binary compatibility of API calls from other modules. In our example, filtering can be performed either by category of programming language, or by keyword in a record name.

//+------------------------------------------------------------------+
//| Parameters for filtering                                         |
//+------------------------------------------------------------------+
struct HelloWorldFilter
  {
   const wchar_t    *keyword;
   int               category;
  };

2.2. In the class of programming languages ​​list manager (CHelloWorldManager) declare the InfoFilter method. It will get indexed data for displaying on the page.

//----
TWRESULT          InfoFilter(const Context *context,HelloWorldFilter *filter,
                             HelloWorldRecord *records,size_t start,int *count,int *total);

 

3. Get Data For Filtering

By its purpose, this stage is similar to getting all the data for displaying (the CHelloWorldManager::InfoGet method) - the list of records is filled by request. However, the implementation is different. The query first goes not to DBMS, but to the Search module, which returns the IDs of records that meet search criterion.

Let's take a closer look at the CHelloWorldManager::InfoFilter method implementation.

3.1. Preparation/Binding request parameters

The search is performed very quickly as indexed data contain minimum of information. In addition, the speed is increased by limiting the number of requested records (in our case 40).

Set the search parameters. At the least they should include the IDs of modules whose data you want to filter, the number of modules, the type of search (in our case - custom one that we've defined above in 1.2.) and the search keyword.

//---
   SearchFilter  search_filter={0};
   SearchResul   results[40]  ={0};
...............................
//--- fill in the parameters of search/filtering
   int module_id              =HELLOWORLD_MODULE_ID;
   search_filter.modules      =&module_id;
   search_filter.modules_count=1;
   search_filter.type_id      =HELLOWORLD_SEARCH_FILTER;
   search_filter.keyword      =(filter->keyword!=NULL ? filter->keyword : L"");

In our example, we will also specify a custom search criterion - programming language category (see step 1.2.).

//---
   SearchFilter::UserItem  extra[1]     ={0};
   size_t                  extra_index  =0;
...............................
//---
   if(filter->category>0 && extra_index<_countof(extra))
     {
      extra[extra_index].type_id=HELLOWORLD_SEARCH_FILTER_CATEGORY;
      extra[extra_index].id     =filter->category;
      extra_index++;
     }
//---
   if(extra_index>0)
     {
      search_filter.extra      =extra;
      search_filter.extra_count=extra_index;
     }

3.2. Request. The Search module uses the ISearchManager::Search method to request IDs of required records.

//---
   int      cnt      =0;
   TWRESULT res      =RES_S_OK;
...............................
//---
   cnt=_countof(results);
...............................
   if((res=m_search_manager->Search(context, SEARCH_ORDER_BY_RELEVANCE, &search_filter, results, 
                                    (start>0 ? start-1 : start), &cnt, total))==RES_E_NOT_FOUND)
     {
      *count=0;
      return(RES_S_OK);
     }
   else
     {
      if(RES_FAILED(res)) ReturnError(res);
     }

3.3. Process results of request. Using previously obtained list of record IDs, get these records from DBMS.

//---
   size_t   max_count=*count,index=0;
...............................
   for(size_t i=0;i<cnt && i<max_count; i++)
     {
      //--- check the type of result
      if(results[i].record1!=HELLOWORLD_BODY_SEARCH_ID) continue;
      //--- get info about record
      if(RES_FAILED(res=Get(context,results[i].record2,&records[index])))
        {
         ExtLogger(context,LOG_STATUS_ERROR) << "failed to get record #" << results[i].record2 << " [" << res << "]";
        }
      else
        {
         index++;
        }
     }
//---
   *count=index;

In general, the principle of indexing has not changed: we have divided it into two successive steps, differing only by the scope of data being indexed.

 

HTTP API

Now let's consider how data indexed for filtering is requested from the view page. Recall that in our example, data viewed on the PageNumberTwo page is formed by the WriteJson method.

The main difference between search and filtering is that the common search results are displayed on a separate page of the Search module, and filtered data are displayed on the module's page. Accordingly, to filter displayed data we need to add the appropriate controls in User Interface (see next) that will allow us to specify parameters for the filtering.

1. In our example, an HTTP request will either use a keyword (a string is requested), or one of the possible values ​​of the Category field (a number is requested). The server side changes are minimal - you need to get filtering parameters from request and call the CHelloWorldManager::InfoFiler method that obtains filtered data.

//+------------------------------------------------------------------+
//| Display list of records in JSON format                           |
//+------------------------------------------------------------------+
TWRESULT CPageNumberTwo::WriteRecords(const Context *context)
  {
.......................................
//---
   if(context->request->Exist(IRequest::GET,L"k") || context->request->Exist(IRequest::GET,L"category"))
     {
      HelloWorldFilter filter={0};
      //---
      if(context->request->Exist(IRequest::GET,L"k"))
         filter.keyword=context->request->GetString(IRequest::GET,L"k");
      //---
      if(context->request->GetInt32(IRequest::GET,L"category")>0)
         filter.category=context->request->GetInt32(IRequest::GET,L"category");
      //---
      if(RES_FAILED(res=m_manager->InfoFilter(context,&filter,m_info,m_start,&m_info_count,&m_info_total)))
         ReturnErrorExt(res,context,"failed get info records");
     }
.............................................

2. Otherwise, we will request all the data.

.............................................
   else
     {
      //---
      if(RES_FAILED(res=m_manager->InfoCount(context,&m_info_total)))
         ReturnErrorExt(res,context,"failed get count of records");
      //---
      m_start=max(1,context->request->GetInt32(IRequest::GET,L"from")+1);
      int max_page   =max(0,int((m_info_total+_countof(m_info))/_countof(m_info))-1);
      //---
      if(m_start>max_page*_countof(m_info)+1)
         m_start=max_page*_countof(m_info)+1;
      //---
      if(RES_FAILED(res=m_manager->InfoGet(context,m_info,m_start,&m_info_count)))
         ReturnErrorExt(res,context,"failed get info records");
     }
................................................

In fact, when sending an HTTP request that includes filtering criteria, the Search module is inquired in the background and provides a list of indexed records that meet specific criterion (see implementation of the CHelloWorldManager::InfoFilter method).

The output format in both cases remains the same.

 

User Interface

Consider the user interface layout that uses the functionality implemented above.

1. Add a filter button next to the search field in the header of the List of Records page. This will allow filtering by keyword - entering the name of programming language category in the search field.

In the page template you can do this by calling the .Filter(url) method of the PageHeader control. The argument passed to the Filter method is URL used to request data for filtering. The PageHeader control contains all the necessary functionality to include entered keyword into the filtering parameters.

//+----------------------------------------------+
//| Controls                                     |
//+----------------------------------------------+
var header = TeamWox.Control("PageHeader","#41633C")
                .Command("<lngj:MENU_HELLOWORLD_LIST>","/helloworld/index","<lngj:MENU_HELLOWORLD_LIST>")
                .Command("<lngj:MENU_HELLOWORLD_NEW>","/helloworld/number_two/edit/","<lngj:MENU_HELLOWORLD_NEW_DESCR>")
                .Help("helloworld/number2")
                .Search(65536)
                .Filter("/helloworld/number_two/");

2. The second way to specify filtering parameters will be a dropdown list. From this list you can select a category of programming language from given values: all, compiled and interpreted.

To do this, in the page template mark up space for a drop-down list and its caption. For example, you can do it using a table layout.

//---
var filter_category;
//--- Select category
TeamWox.Control("Layout",{
    type: "table",
    items: [[
        //--- Dropdown list caption
        TeamWox.Control("Label",'<lngj:HELLOWORLD_INFO_LIST_CAPTION />'),
        //--- Dropdown list
        filter_category=TeamWox.Control("Input","combobox","category",0,{options:[[0,"All"],[1,"Compiled"],[2,"Interpreted"]]})
                           .Style({width:"180px"}),
        //--- Padding the remaining space
        TeamWox.Control("Label",'').Style({width:"100%"})
    ]]
}).Style({padding:"0px 0px 16px 10px"});
filter_category.Append("onchange",LoadData);

3. In all cases (displaying all the records, filtering by category, filtering by keyword) data for the table are inquired the same - if user specifies a filter criterion, it is included into request parameter. This key point of implementation is highlighted in the source code.

//---
function LoadData(startFrom,perPage) {
    var params={};
    //---
    if(startFrom==undefined) startFrom=pages.Item();
    //---
    if(startFrom>0) params.from=startFrom;
    //---
    var keyword=header.SearchControl().Input().Value();
    if(keyword!="") params.k=keyword;
    //---
    var category=filter_category.Value();
    if(category>0) params.category=category;
    //---
    TeamWox.Ajax.get('/helloworld/number_two/',params,
    {
        onready: function(text) {
            Update(TeamWox.Ajax.json(text));
        },
        onerror: function(status) {
            alert(status);
        }
    });
}

For more information about implementation details of other utility methods working with data, see the helloworld\templates\number2.tpl page template.

 

Demonstration

To demonstrate the implemented functionality on the "List of Records" page, it must be filled with data. We have added several programming languages ​​of different categories.

1. First, let's demonstrate the filtering by keyword:

2. Now let's filter records by category from the drop down list.

First compiled languages:

then interpreted:

3. Both types of implemented filtering can work together, allowing you to narrow down the criteria to required level.

 

Conclusion

For this and subsequent articles, the Hello World training module has been redesigned and adapted to the current features of TeamWox server. Now all the pages contain explanations of the demonstrated features. All the texts are taken out of the page templates and put in the helloworld.lng language file. These texts are available for translation into any language supported in TeamWox groupware.

In our next articles we are going to highlight importing and exporting data in custom modules.

---

2012.01.31