MyDMS

MyDMS 1.7.2 is now available and has been uploaded onto SourceForge. Numerous user interface improvements have been made, including a new menu layout, new chooser dialogs and CSS work-arounds to accommodate problems encountered with IE6.

There are also 2 new translations, Czech and Slovak, and a minor alteration to the LDAP authentication code to allow for ldaps:// connections.

MyDMS 1.7.0 is now available and has been uploaded onto SourceForge. This release is primarily intended to address a fundamental limit on the scalability of the storage subsystem and is essential for all systems that intend to store tens of thousands of documents on MyDMS.

Also included are various small changes to the stylesheets to improve the user interface, as well as a Drupal theme that matches MyDMS.

Last, but not least, a Hungarian language translation has been included and there are one or two little bug fixes, mostly to do with properly parsing escaped characters.

There is a hard limit on the number of files that MyDMS can manage, due to that way in which the original underlying file system structure for storing documents was designed.

Currently, MyDMS creates a new directory as a container for every single file that it will store. Each directory never contains more than one file, and directories are never re-used. However, some file systems, such as UFS (and its derivatives) have a hard-coded limit on the number of sub-directories a directory can contain (in fact it is a limit on the number of hard links, but sub-directories are registered within directory inodes as hard links). For UFS, this is a limit of 32767 directories, including the special cases, “.” and “..”.

In an ideal world, it would be better to tear down the existing mechanism and start again from scratch. However, this inevitably breaks backwards compatibility, and so it is necessary to devise an alternative strategy for managing the directory structure.

A solution proposal has been developed that has a nested structure and uses the database to help control the allocation of directory names.The database has 2 new tables, described as follows:

CREATE TABLE `tblDirPath` (
`dirID` int(11) NOT NULL auto_increment,
`dirPath` varchar(255) NOT NULL,
PRIMARY KEY (`dirPath`,`dirID`)
) ;

CREATE TABLE `tblPathList` (
`id` int(11) NOT NULL auto_increment,
`parentPath` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ;

dirID represents a leaf directory within the content file structure and parentPath is the path to its parent subdirectory, relative to settings->_contentDir. dirPath is equivalent to parentPath. When a request for a new storage location is made, MyDMS reads the last directory entry recorded the database table. If the value of dirID is equal to the directory link limit, then parentPath is updated such that it represents a new parent directory. For example:

limit = 32765;

parentPath = “1/2/3″;

In this case, if dirID = 32765, then parentPath is updated to “1/2/4″ and dirID is reset back to 0.

If parentPath = “32765/32765″ and dirID = 32765, then parentPath would change to “0/0/0″, and dirID would be set to 0.

In addition, the size of the “dir” field in tblDocumentContent has been increased to 255 to allow for longer directory paths:

ALTER TABLE `tblDocumentContent` CHANGE `dir` `dir` VARCHAR( 255 ) NOT NULL ;

A new makeDir() function has also been introduced to allow MyDMS to create complete directory paths, equivalent to “mkdir -p”. This functionality is available in PHP5 , but is not provided in PHP 4.

The new mechanism is currently undergoing testing on one of the large MyDMS deployed within my company. This instance has tens of thousands of documents and they unfortunately hit the file system limit 2 days ago. We introduced a work-around by moving the content directory to a backup location and creating soft links in the original directory that points them at the backups.

Small MyDMS instances (fewer than discrete 32765 files) are not affected by the limitation on hard links within a directory, so the use of the new file structure is optional. A flag has been set in the Settings file that allows users to choose which storage method they prefer. Users with large DMS deployments should seriously consider this new system. Backwards compatibility is completely preserved and there is no loss of existing files or data and no need to create a data migration strategy. Existing files are kept in situ.

This change, along with some other minor alterations, will be packaged up and released as 1.7.0 in due course.

The first beta release of MyDMS 1.6.0 is now available and has been uploaded onto SourceForge.
This package has been extensively tested on two production systems over the last 4 weeks and many bug fixes and enhancements have been made. All of the  languages files have been populated with the new phrases, but are incomplete — where there are gaps, the english language phrase has been used as a default. Can I ask that users of the other language packs review the content and submit patches to me via SF.net.

Once 1.6.0 has completed beta testing and is released, I will be re-organising the CVS repository to make way for MyDMS 2.0. This will be a major departure from MyDMS 1.x as I intend to re-work the entire database structure so that I can take better advantage of the newest stable release of MySQL 5. The user interface and user experience will be essentially unchanged (except where an improvement can be made, of course) but the system will be made more robust through proper use of transactions, simplification of the table structure, and so on. Performance and scalability will also increase beyond even the improvements in 1.6.0!
Lots of work. This means that the databases for MyDMS 1 and MyDMS 2 will be incompatible. As a result, there will be a huge focus on migration tools. Also, development on MyDMS 1 will not stop — features will continue to be back-ported into the MyDMS 1 branch and new releases made. I’m not about to isolate users!

Work on MyDMS 2 will commence in January 2007.

Has it been quiet here lately? Yes, I think perhaps it has. Just shows you how busy I’ve been .

A new CVS snapshot (20060920) has been packaged and made available. There is a new feature that allows the user to view historical information from previous document revisions. This information includes the list of reviewers/approvers and their comments. Also bundled into this release are some bug fixes and new styles. That’s right: MyDMS now is now available in the traditional blue and a slightly more unorthodox green.

After some extensive internal testing, it looks like most of the bugs have been fixed. What’s missing are updates to the languages files (currently only the English language file is up to date). The test strings have been expanded upon in order to be able to capture all of the output text in a way that can be localised. This means that the lang.inc files are now much bigger, and I could use some help in getting them up-to-date.

I’ve created a new snapshot, based on feedback and some user acceptance testing carried out over the last 2 weeks. Found some bugs, fixed some things, etc. I’ve also put together the basic installation / upgrade procedure for MyDMS 1.6.x, which is one of the last prerequisites for 1.6.0 beta. Based on my current work-load, I think you can expect the first real beta in the next week or so.

I guess I’d better write some documentation for the release as well.

CVS snapshot is available, as always, from SF.net.

Hello all.

I’m looking for people to help out with support requests coming from Windows users. I don’t have access to any Windows servers and do not have any expertise with services such as IIS, so I am unable to help much when users report problems on this platform.
Are there any volunteers out there who could devote a little time to assisting these poor, encumbered souls? It’s not a lot of work, but there is this one fellow on the SF.net forums who can’t get the file conversion stuff to work on his machine.

Any takers?

If you have left a comment recently and have had no response, I apologise. The Wordpress moderation features are pretty strict and so comments have been queued in the moderator queue rather than being posted. I hadn’t noticed that there were comments waiting to be approved, so it’s all my fault. Since the comments for a post are switched off after a period of time, I shall be responding to everyone individually.

In my defence, I have been working hard to get this software ready for release. However, this is a poor excuse, and I am sorry.

And there was much rejoicing.

Internet Explorer has some unusual attitudes toward margins or containers or, who knows… There’s something not right somewhere. It also treats forms differently from other block level containers (well, divs anyway), so I’ve been on a merry dance to fix the container rendering problem.

The fix has been committed to CVS. New packages will be released at the weekend or on Monday — I have a couple of major deadlines to meet this week, so please bear with me.

A minor update to the development snapshot. LDAP support has been added in as an authentication option. I’ve also taken this opportunity to separate the CVS snapshots from the stable release on the SourceForge repository.