TeamWox SDK: File Storage - Part 1

Introduction

In TeamWox groupware two types of storages are used to store and manipulate data: Firebird DBMS and file storage. We have considered how TeamWox interacts with its DMBS in the TeamWox SDK: Interaction with DBMS article. In this article we will talk about file storage. You'll know what file storage is, how it is implemented and where it is used in TeamWox.

For convenience, this article will be split into two parts. In the first part in addition to theory we will consider a simple example of how to save record in the file storage, then read and delete it. In the second part we'll describe the tools used to work with TeamWox file storage: WYSIWYG editor, attachments and comments on example of the Helloworld training module.

TeamWox File Storage

In TeamWox data are stored in 2 ways:

The reason of this division is that in terms of performance it's inefficient to store large amounts of data in relational databases. Moreover, if modules' data would have been stored in the database, it will be impossible to manage caching strategy (simply there would be no free memory for indices).

So far as system performance and fault tolerance were of paramount importance during TeamWox development, all of these features have been taken into account. Use of TeamWox file storage gives the following benefits:

• Performance. Since BLOB data type is not used in TeamWox DBMS, the size of database remains minimal. DBMS cache is used to store indices.

• Data caching. The most frequently used data up to 30-80 MB (depending on free memory size and system architecture) are cached in memory, that in turn improves system performance.

• Data compression. Applies to all files except for some MIME types of files that are initially compressed (video, mp3, archives, etc.).

Application of TeamWox file storage:

• Texts of e-mails, tasks, posts on board, documents, comments, etc. in the Tasks, Board, Documents, Organizations, Team, Contacts and Products modules.

• Files (images, video, audio, diagrams, etc.) downloaded via built-in WYSIWYG editor, or attached by user in tasks, comments, etc.

File storage data are located in the <TeamWox_install_dir>\data folder. Data for each module are stored in appropriate \data\<module name> subfolder. When users begin to fill module with data, in the \data\<module name>\ folder the new 1\ folder is created. In this folder files with names like 1, 2, 3, etc. and dat extension are created. Module's data are written in these very files. When this folder will count 100 of such files, the server creates folder 2\, then (when it is filled) folder 3\, etc.

To work with file storage (with records stored in it, to be more precisely) module must implement the IFilesManager interface, that we will now consider.

 

Interface for Working with Data

TeamWox API provides the IFilesManager interface to work with data in file storage. Its basic methods are methods of saving data (FileStore), reading data (FileRead) and deleting data (FileDelete). They are similar to the C.R.U.D. methods, that we have implemented to interact with DBMS.

In the FileStore method, stored data can be assigned with certain flags that define module logic. If the flag is not set (value 0), then data are stored in the file storage at once. In this case the MIME type of file is detected, and then the file is either compressed (text, images, etc.) or stored without compression (archives, videos, music, etc.).

The FILE_FLAG_TEMP flag marks stored data as temporary. When you save data, several actions are performed and some of them may complete unsuccessfully. If transaction for whatever reason had failed, files remain marked as temporary (or saved with temporary flag). In this case, when service time of TeamWox server will come, all temporary data will be removed and file storage won't be clogged with corrupted data.

The FILE_FLAG_COMPRESSED flag marks data as compressed, i.e. compression is not applied when saving these data. The FILE_FLAG_UNKNOWN_MIME flag allows developer to manually specify MIME type for the file.

If data uploaded to file storage are marked as temporary, the FilesCommit method can confirm upload by removing flag of temporary record and setting identifiers of record, to which these files are linked.

The FileInfoGet allows to get information about file, when it is read from file storage.

Here are descriptions of some methods of working with files, that will be used in this article.

 

Save record in file storage from memory

virtual TWRESULT  FileStore(const Context *context,const void *src,FileInfo *info,INT64 *file_id,int flags)

Parameter	Type	Description
*context	Context	Context of request processing.
*src	void	Pointer to memory space, that is written in file storage. The size is specified in the FileInfo *info.
*info	FileInfo	Information about record.
*file_id	INT64	Record ID in file storage. If file_id=0, then new record is created, otherwise - existing record is overwritten.
flags	int	Flags of the EnFilesFlags enumeration.

 

Delete record from file storage

virtual TWRESULT  FileDelete(const Context *context,INT64 file_id)

Parameter	Type	Description
*context	Context	Context of request processing.
*file_id	INT64	Record ID in file storage.

 

Read record from file storage into memory

virtual TWRESULT  FileRead(const Context *context,const INT64 file_id,void *dst,UINT64 *dst_len,const UINT64 offset)

Parameter	Type	Description
*context	Context	Context of request processing.
*file_id	INT64	Record ID in file storage.
*dst	void	Memory space where record will be read to.
*dst_len	UINT64	Size of data to be written (in bytes).
offset	UINT64	Offset in file storage record, from where record is being read.

 

Get record information

virtual TWRESULT  FileInfoGet(const Context *context,const INT64 file_id,FileInfo *info)

Parameter	Type	Description
*context	Context	Context of request processing.
*file_id	INT64	Record ID in file storage.
*info	FileInfo	Information about record.

 

Confirm saving of record with temporary flag

virtual TWRESULT  FilesCommit(const Context *context,const INT64 *file_ids,const int count,const INT64 type_id,const INT64 record_id) 

Parameter	Type	Description
*context	Context	Context of request processing.
*file_ids	INT64	Array of record IDs.
count	int	Number of record IDs.
type_id	INT64	Type of record. Specified by developer and depends on module logic. Taken from the FileInfo structure.
record_id	INT64	Record ID, that you need to set to the files, specified in the file_ids array. Taken from the FileInfo structure.

 

Example of Adding Text Record in File Storage

In this article as an example we will consider saving simple text in file storage. Other variants of working with files will be discussed in the second part of this article. We will continue to work with the Hello World training module. Source codes with all the changes are attached to this article.

 

1. Getting the IFilesManager Interface

To begin with, in manager class we need to get the pointer to the class object, that implements the IFilesManager interface.

1.1. Declaration

• CHelloWorldManager

private:
   IFilesManager    *m_files_manager;                  // files manager

1.2. Implementation

• Constructor - CHelloWorldManager::CHelloWorldManager()

//+------------------------------------------------------------------+
//| Constructor                                                      |
//+------------------------------------------------------------------+
CHelloWorldManager::CHelloWorldManager() : m_server(NULL), m_files_manager(NULL), m_next_id(0)
  {
//---
//---
  }

• Module initialization - CHelloWorldManager::Initialize(IServer *server, int prev_build)

//---
   if(RES_FAILED(res=m_server->GetInterface(L"IFilesManager",(void**)&m_files_manager)) || m_files_manager==NULL)
      ReturnErrorExt(res,NULL,"failed to get IFilesManager interface");

1.3. As the IFilesManager interface is got only once during module initialization and then it won't change, synchronization is not required. And for convenience, we will declare method, that allows you to quickly get this interface while processing requests for pages.

• CHelloWorldManager

public:
   IFilesManager*    FilesManagerGet() { return(m_files_manager); };

2. Expanding Data Structure

To store records in file storage, we need to expand data structure by adding the record ID into it.

2.1. Add new field to the HELLOWORLD table. This will be the record ID.

• HelloWorld.h

//+------------------------------------------------------------------+
//| Record structure                                                 |
//+------------------------------------------------------------------+
struct HelloWorldRecord
  {
   INT64             id;
   wchar_t           name[256];
   int               party;
   INT64             description_id;
  }; 

• CHelloWorldManager::DBTableCheck - Check existing/add new fields to the table.

//---
   if(RES_FAILED(res=sql->CheckTable("HELLOWORLD",
       "ID              BIGINT        DEFAULT 0   NOT NULL,"
       "NAME            VARCHAR(256)  DEFAULT ''  NOT NULL,"
       "PARTY           INTEGER       DEFAULT 0   NOT NULL,"
       "DESCRIPTION_ID  BIGINT DEFAULT 0 NOT NULL",
       "PRIMARY KEY (ID)",
       "DESCENDING INDEX IDX_HELLOWORLD_ID_DESC (ID)",
                                             NULL,
                                             NULL,
                                              0)))
      ReturnError(res);

2.2. Accordingly update SQL queries in C.R.U.D. methods implementations.

• CHelloWorldManager::InfoGet - Select fields from table with sorting.

//--- Text of SQL request to select records from HELLOWORLD table and sort them by ID. 
   char query_select[]="SELECT id,name,party,description_id FROM helloworld ORDER BY id ROWS ? TO ?";
//--- "Bind" data to parameters of request
   SqlParam params_query_select[] ={
      SQL_INT64, &rec_info_get.id,            sizeof(rec_info_get.id),
      SQL_WTEXT,  rec_info_get.name,          sizeof(rec_info_get.name),
      SQL_LONG,  &rec_info_get.party,         sizeof(rec_info_get.party),
      SQL_INT64, &rec_info_get.description_id,sizeof(rec_info_get.description_id)
      };

• CHelloWorldManager::Get - Получение строки из таблицы.

//--- Text of SQL request to get row from HELLOWORLD table by specified ID. 
   char query_select_string[]="SELECT id,name,party,description_id FROM helloworld WHERE id=?";

//--- "Bind" data to parameters of request
   SqlParam params_query_select_string[] ={
      SQL_INT64, &record->id,            sizeof(record->id),
      SQL_WTEXT,  record->name,          sizeof(record->name),
      SQL_LONG,  &record->party,         sizeof(record->party),
      SQL_INT64, &record->description_id,sizeof(record->description_id)
      };

• CHelloWorldManager::Update - Save/update existing record.

//--- Text of SQL request to add new record in HELLOWORLD table
   char query_insert[] ="INSERT INTO helloworld(party,name,description_id,id) VALUES(?,?,?,?)";
//--- Text of SQL request to modify existing record in HELLOWORLD table
   char query_update[] ="UPDATE helloworld SET party=?,name=?,description_id=? WHERE id=?";
//--- "Bind" data to parameters of request
   SqlParam params_query[] ={
      SQL_LONG,  &record->party,         sizeof(record->party),
      SQL_WTEXT,  record->name,          sizeof(record->name),
      SQL_INT64, &record->description_id,sizeof(record->description_id),
      SQL_INT64, &record->id,            sizeof(record->id)
      };

 

3. Working with Files in File Storage

In manager we will implement methods of saving and deleting data from storage, as data in our example are linked to records in the database.

3.1. Save records - CHelloWorldManager::Update. The FilesCommit method removes the temporary flag from the uploaded data, and after that they are stored permanently. We are saving data by specified ID and linking them with record ID in the table.

   TWRESULT res=RES_S_OK;
...........................
//---
   if(record->description_id!=0 && RES_FAILED(res=m_files_manager->FilesCommit(context,&record->description_id,1,0,record->id)))
      ExtLogger(context,LOG_STATUS_ERROR) << "failed to commit description [" << res << "]";

3.2. Delete data - CHelloWorldManager::InfoDelete. Data are removed from storage by record ID.

   TWRESULT res=RES_S_OK;
   HelloWorldRecord record={0};
//--- проверки
   if(context==NULL || m_server==NULL || m_files_manager==NULL || id<=0) ReturnError(RES_E_INVALID_ARGS);
   if(context->sql==NULL)                                                ReturnError(RES_E_INVALID_CONTEXT);
................................
//--- Delete record contents from file storage
   if(RES_FAILED(res=m_files_manager->FileDelete(context,record.description_id)))
      ExtLogger(context,LOG_STATUS_ERROR) << "failed to delete record #" << id << " description [" << res << "]";

 

4. HTTP API

In HTTP API we will implement methods of uploading data into file storage and reading them from file storage. Since in the Hello World module data are edited on the PageEdit page, we will implement methods of uploading and reading data only in this page class. Optionally you can implement reading data on the PageNumberTwo page by yourself.

4.1. CPageEdit - To work with file storage from the PageEdit page, in this page class declare the pointer to the file manager. To the input text declare appropriate character array.

private:
   IFilesManager    *m_files_manager;
   wchar_t          *m_description;

4.2. In the PageEdit class constructor, initialize pointer to the file manager and array for the text string.

• Constructor

//+------------------------------------------------------------------+
//| Constructor                                                      |
//+------------------------------------------------------------------+
CPageEdit::CPageEdit() : m_server(NULL), m_files_manager(NULL), m_description(NULL)
  {
//---
   ZeroMemory(&m_record,sizeof(m_record));
//---
  }

• Destructor. Free allocated memory at the end of page processing.

//+------------------------------------------------------------------+
//| Destructor                                                       |
//+------------------------------------------------------------------+
CPageEdit::~CPageEdit()
  {
   if(m_description!=NULL)
     {
      delete [] m_description;
      m_description=NULL;
     }
//---
   m_server       =NULL;
   m_files_manager=NULL;
  }

4.3. While processing PageEdit request let's get the file manager.

• CPageEdit::Process

//---
   m_server       =server;
   m_files_manager=manager->FilesManagerGet();

4.4. Declare and implement method of saving record in file storage.

• Declaration. Strings, passed into this method, will be saved in the file storage record.

private:
   TWRESULT          StoreDescription(const Context *context,const wchar_t *description);

• Implementation. Before you save text, specify file type, file size (in bytes) and indicate MIME type for correct processing of file contents by web browser. In the FileStore method the last argument is the FILE_FLAG_TEMP flag, that marks data uploaded into file storage as temporary.

//+------------------------------------------------------------------+
//| Save description                                                 |
//+------------------------------------------------------------------+
TWRESULT CPageEdit::StoreDescription(const Context *context,const wchar_t *description)
  {
   TWRESULT res=RES_S_OK;
//--- checks
   if(m_files_manager==NULL || context==NULL || description==NULL) ReturnError(RES_E_INVALID_ARGS);
   if(context->user==NULL)                                         ReturnError(RES_E_INVALID_CONTEXT);
//---
   FileInfo description_info={0};
//--- Fill out the structure with information about the record
   description_info.type=TW_FILE_TYPE_HTML;
   description_info.size=(wcslen(description)+1)*sizeof(wchar_t);
   StringCchCopy(description_info.mime_type,_countof(description_info.mime_type),L"text/html");
//--- If there is no record yet, fill out other fields
   if(m_record.description_id==0)
     {
      description_info.type_id  =0;
      description_info.record_id=m_record.id;
     }
//--- Write data to the file storage
   if(RES_FAILED(res=m_files_manager->FileStore(context,(void*)description, &description_info, &m_record.description_id, FILE_FLAG_TEMP)))
     {
      ExtLogger(context,LOG_STATUS_ERROR) << "CPageEdit::StoreDescription: failed to store description";
      //---
      return(res);
     }
//---
   return(RES_S_OK);
  }

4.5. Declare and implement method of reading/checking data from file storage.

• Description

private:
   void              CheckDescription();

• Implementation. First, we will read information about the file, then allocate memory for it, and after that we will read file contents into this memory using specified buffer size. In our example text files will be small, and in this case you don't have to specify the offset to read them.

  If module logic will imply reading of large files, then records data must be read in blocks by specifying the offset in the FileRead method.

//+------------------------------------------------------------------+
//| Check description                                                |
//+------------------------------------------------------------------+
void CPageEdit::CheckDescription()
  {
   TWRESULT res=RES_S_OK;
//--- checks
   if(m_files_manager==NULL) return;
//--- Have already read
   if(m_record.description_id==0 || m_description!=NULL) return;
//---
   FileInfo blob_info={0};
//--- 1. Get the record information in file storage
   if(m_files_manager->FileInfoGet(NULL,m_record.description_id,&blob_info)==RES_S_OK)
     {
      //--- 2. Allocate memory for the record contents
      UINT64 sz    =UINT64((blob_info.size+sizeof(wchar_t))/sizeof(wchar_t));
      m_description=new (std::nothrow) wchar_t[sz];
      //---
      if(m_description==NULL)
        {
         ExtLogger(NULL, LOG_STATUS_ERROR) << "CPageEdit::CheckDescription: failed to allocate memory";
         return;
        }
      //---
      ZeroMemory(m_description,size_t(blob_info.size));
      //--- 3. Read the record contents into the memory
      sz *= sizeof(wchar_t);
      if(RES_FAILED(res=m_files_manager->FileRead(NULL,m_record.description_id,m_description,&sz,0)))
         ExtLogger(NULL, LOG_STATUS_ERROR) << "CPageEdit::CheckDescription: failed to load file record [" << res << "]";
     }
  }

4.6. In the CPageEdit::OnUpdate method that saves records in database, add the call of method, that saves data in file storage, that we've implemented earlier. We will pass text via the description token.

//---
   if(RES_FAILED(res=StoreDescription(context,context->request->GetString(IRequest::POST,L"description"))))
      ExtLogger(context,LOG_STATUS_ERROR) << "failed to store description";

4.7. Implement the token with name description in the CPageEdit::Tag method.

//---
   if(TagCompare(L"description",tag))
     {
      CheckDescription();
      //---
      if(m_description!=NULL && m_description[0]!=NULL)
        {
         context->response->WriteSafe(m_description,IResponse::REPLACE_JAVASCRIPT);
        }
      //---
      return(false);
     }

5. User Interface

In user interface of editing HELLOWORLD table records (the edit.tpl template) let's add the field for entering text that will be stored in file storage. The input text will be contained in the description token, that we've implemented earlier in the PageEdit class.

5.1. In the Form control add the new element to the array of parameters for the items key. Here we will use the Input.Textarea control that adds the input area for multiline text.

      items   : [
                  [
                     TeamWox.Control('Label','<lngj:HELLOWORLD_NAME>','name'),
                     TeamWox.Control('Input','text','name','<tw:name />')
                  ],
                  [
                     TeamWox.Control('Label','<lngj:HELLOWORLD_PARTY>','party'), 
                     TeamWox.Control('Input','combobox','party','<tw:party />', {
                        options : [
                           [0,'<lngj:HELLOWORLD_REPORT_REPUBLICAN />'], 
                           [1,'<lngj:HELLOWORLD_REPORT_DEMOCRATIC />'],
                           [2,'<lngj:HELLOWORLD_REPORT_DEMOREP />'],
                           [3,'<lngj:HELLOWORLD_REPORT_FEDERALIST />'],
                           [4,'<lngj:HELLOWORLD_REPORT_WHIG />'],
                           [5,'<lngj:HELLOWORLD_REPORT_NATUNION />'],
                           [6,'<lngj:HELLOWORLD_REPORT_NOPARTY />']
                     ]})
                  ],
                  [
                     TeamWox.Control('Label','<lngj:HELLOWORLD_DESCRIPTION>','description'),
                     TeamWox.Control('Input','textarea','description','<tw:description />').Style({height: "120px"})
                  ]
                ]

5.2. Add translations for the text label, that is attached to the new input field.

[eng]
;---
HELLOWORLD_NAME                ="Name"
HELLOWORLD_PARTY               ="Party"
HELLOWORLD_DESCRIPTION         ="Description"

[rus]
;---
HELLOWORLD_NAME                ="Имя"
HELLOWORLD_PARTY               ="Партия"
HELLOWORLD_DESCRIPTION         ="Описание"

5.3. Compile the module, update the template on the server and run TeamWox. On the edit page type some text and save it.

Data are successfully saved in the file storage. You can ensure that, as there are no error messages. Also you can see that size and date of the <TeamWox server>\data\helloworld\1\1.dat file have changed.

For convenience of development process, you can also supply the implementation of the IFilesManager interface methods with debugging messages using the CSmartLogger class.

 

Conclusion

We have considered what file storage is and how interact with it via modules. On the Hello World module example you have learned how to save plain text as a record in file storage.

In the second part, we'll consider the real-life examples of using file storage in TeamWox modules. You will learn how to add comments to records, attach files to messages and insert different files in TeamWox WYSIWYG editor.

---

2011.02.02