Custom Options : Message Deduping

Message Deduping
Transend Migrator has the ability to prevent duplicate message items from migrating. This is called deduping. This feature is helpful in the event you want to restart a migration. It is also helpful when you restore many email archives from a particular person, and you want to output only the unique messages.
 
When deduping is turned on and when you perform an email migration, Transend Migrator performs a calculation called an MD5 hash on each message item to see if it is unique. It writes that the results of the MD5 hash into memory. At the end of your migration, the MD5 information is taken from memory and written to file.
 
The calculation is performed on the following metadata items to see if they are unique or different.
 
 
These default metadata items can be changed, and you can also add additional requirements such as:
 
 
Enable Deduping
1.
2.
3.
4.
5.
6.
 
When no parameter is entered, Transend Migrator will first write the MD5 hash value of each message into memory, and then write that information into a file called md5.dat, and this file will be created inside the same directory as the master log file (tmlog.html).
 
Suggested Deduping Options
Sometimes, a migration will fail and end abnormally. When this happens, the MD5 information in memory does not get written to disk. Therefore, when this happens you will have some duplicate email items if you decide to migrate all message folders over again. Therefore, you can force Transend Migrator to write the MD5 information to disk immediately instead of waiting until the migration is complete.
 

1.
Turn on deduping by enabling the Custom Option Setting called, “Migrate Unique Messages.
2.
 
 
 
3.
 
 
4.
 
Deduping in Batch Mode
In batch mode, it is common practice to create an md5 hash file for each account migrated. In batch mode, variables are used so substitute a variable, such as $var1, for the name of the dedup file. Often, a variable is a user name so consider using the user name as the variable. It is easier to keep track of the file if it is related to the user name, and you will only have to use one variable.
 
Note
You cannot share the dedup file with another process. Therefore, create a unique dedup file for each account migrated.