Thursday, July 14, 2011

Compacting :- What is it and Why must I do it.


This document predates the official Thunderbird support documentation at https://support.mozilla.org/en-US/kb/compacting-folders

What is it?

Thunderbird stores all of your local messages in a file on the local computer.  One file for each folder that appear in Thunderbird.  When you delete or move a message it is marked as deleted in the file that contains it and ignored by Thunderbird.  This is great for performance, if  the program stopped to actually write a new file without the message in question every time you moved or delete a message it would be so slow as to be unusable.  But like all good things there is a price to pay.


Over time the file wastes a significant amount of space, becomes bloated and unwieldy. The only real solution for this is to create a new file without all of the left over junk.  This is called compacting in Thunderbird.  Those who have used Outlook will be familiar with a prompt asking to compress.  This is exactly the same thing with a different name.

Why Must I do it?

There are some general housekeeping reasons such as to save space.  But the main reason is to maintain Thunderbirds speed and to make the file small enough for Thunderbird to actually use.  Files over 4Gb in Windows and 2Gb ion Linux and Mac systems are unreadable and present either error messages, or can appear as mail simply downloading and never getting to your inbox.  So compacting is an essential part of the process of keeping Thunderbird not only working, but working effectively.

Edit:  I have had some people point out that the capacity of the NTFS file system is in exabytes and that this is rubbish. Nothing could be further from the Truth.  Thunderbird has this limitation and ignoring it will increase your chance of data loss at some future point in time.

Edit 2: Bug 462665 has finally been fixed for release in Thunderbird 12.  This brings to an end the 4Gb limit for local files.  It does not remove the need to compact, but it does remove on of the major  side effects of not compacting, that of having the file exceed to 4Gb limit.

Making it less frequent

Thunderbird compacts when a file contains deleted messages that total a preset size.  The default in Thunderbird 5 is 100Mb, but those who go through the upgrade process don't don't necessarily get the benefit of this change, all they see is the regular request to compact because their threshold is still set to the pre V5 level of 100kb.

On the tools Menu go to Options > advanced > Network and disk space and change the 'compact when it will save over' entry to 100Mb or more.  If you get lots of mails with large attachments you may have to double or triple the 100Mb.

If Thunderbird appears bent on compacting every time you delete something.  There is the possibility that the compact process is not completing, either through interruption (new mail arriving or your anti virus scanner scanning the file)

On the file menu, select offline and then work offline.
Allow any download or sync to occur.
On the View menu, select folders and then ALL
From the file menu select Compact Folders
Go make coffee, this might take a while. Particularly if you have lots of mail
In the bottom left hand corner of your Thunderbird screen eventually a Done Compacting message will appear.

Now you can change your folder view back to whatever it was originally and go back online.

In some circumstances Thunderbird can hang when the compact prompt appears.  This hang apparently affects version 3.x.x - 6.0.1.  A fix was implemented in version 7 

I lost my mail when Compacting

Please complete my survey of those that hard lost data after compact. and see my post on what this is all about

This does appear to happen,  the first obvious point is that Thunderbird can not read the index for the file while compacting is in progress, so the mail in the folder will simply disappear while the compact is in progress. Following the completion of the process the mail will reappear.  At other times after the compact there will be no messages show in the folder due to some sort of interruption the the process causing an empty index to be created.  The cause of this is not really understood, but right clicking the folder and selecting properties and then repair will force a re-index and the mail will reappear as if by magic.

More Fatal apparent data loss can occur when compacting.  If the  underlying file is corrupt then the compact will write a file that can at best be described as truncated.  This is not compact loosing your mail, it is compact showing you the corruption and demonstrating the extent of that corruption.  One leading cause of this sort of corruption is anti virus programs that think they understand Thunderbird's filing system quarantining mails and leaving the file in a corrupt state.

Other symptoms of corruption in the mail store are;
  • Previously deleted messages reappearing
  • Messages with the date of 1969
  • Message bodies containing bits of other messages
  • Messages in the message list not matching what is displayed when clicked
  • Thunderbird apparently going off to sleep before messages in a folder are displayed

48 comments:

  1. Hi Matt,
    very interesting post indeed.
    Just a question: is it possible to delete ONLY the attached files in the sent messages folder ?
    I want to keep trace of the sent messages, but normally the attached files are also stored somewhere else in the PC and they fill so much room uselessly in the Thunderbird folder.
    Thank you
    Carlo (from Italy)
    cgpicchi@libero.it

    ReplyDelete
  2. Dear Matt, Can you possibly help me? I have multiple problems suddenly with TB 6.0.1. Running on Win 7. First, whenever I try compact certain boxes such Inbox and Send, it crashes TB. Second, it looks like all MSF files were lost (I have about 300 folders) and it is taking days to rebuild them. Even the folder with one message in it, that is already compacted, takes several hours to build the MSF.. Third, after the MSF file is rebuilt and it all works, the next day the MSF file seems to have been lost or has ceased to function, even without any new message being put in them. So it spends hours remaking them. Fourth, even when a folder has the functioning MSF file (I can see from the number of files in lower rt hand corner of the screen), but it still doesn't show those messages, or not until 3-4 hours of processing. Fourth, even when it does show the messages, it often has to rebuild the summary each time that I open the folder, even though no message has been added or deleted. Fifth, the address book reports that the abook or mab file is locked. It was all fine till 3 days ago. I don't know who else to ask! Thanks so much. I am Robbie,in NY.

    ReplyDelete
  3. Great info, thanks! Why isn't this info in Thunderbird Help. When you search Help for "Compact" it says 0 results!

    ReplyDelete
  4. This is not on the official site, because of something of a dispute I had with them over where it should go. The result was they linked to it.

    Beyond that the search function on the official site appears to be having some problems finding things.

    ReplyDelete
  5. What a great idea compacting is, however my TB just did it and I have no record of any previous e-mail which means I can't find a business e-mail I need urgently. Is there any way to get it back

    ReplyDelete
  6. I'm unable to rename or delete any folder in Thunderbird. I tried compacting folders after the rename or delete, and the folder came back (it was empty of any messages). Now I am afraid to create needed folders as they will persist forever. The one that keeps returning is for a deceased correspondent. I want to remember him, but not this way.

    ReplyDelete
  7. From reading all this, I'm afraid to even think about compacting! I don't want to lose my e-mails and I just want to be able to empty my trash file of all the e-mails I've deleted. Evidently, that's not possible--Now it seems it's not possible to do anything as easily as push a button. I had Outlook before and had no problems like this! Evidently I need technical support to just empty my trash! Yikes!

    ReplyDelete
  8. Folks if you want support on an individual bases please post here http://getsatisfaction.com/mozilla_messaging I spend more than enough time there without trying to do it on my blog as well.

    @ Anonymous Compacting does not loose mails. I thought I made that clear in the text. What it does is show you what was already gone because of corruption.

    @ Yvonne Most likely not. This sort of mishap is one of the reasons IMAP mail was invented, so there is still a copy in the cloud. Please note that most corruption is a direct result of activity by anti virus scanners. They 'clean your mail of the virus they found and completely corrupt your mail store in the process.

    ReplyDelete
  9. This was helpful, and I can guess now that my big problem is because I’ve never bothered with compacting. I thought my version of ThB updated itself, but now I'm hearing there is a version 8 and I have had something like v.3. I like ThB, but not when it loses my Inbox, which it did a month ago and then did again yesterday. Something is terribly wrong. The article gave me a clue - two clues, compacting and my anti-virus program, Kaspersky, may also be in the problem. I thought I had a ferocious virus, especially from the sound Kaspersky made. In any case my list of messages in the inbox disappeared. The list is there, but each one opens to a blank page. There is nothing there. Now I have made copies of the profile and put it on an external drive. I downloaded the latest version of ThB, installed it over the older version in the application folder. I restarted--many times—and I now I can’t get past a message that says:
    “Close Thunderbird! A copy of Thunderbird is already open. Only one copy of Thunderbird can be open at a time.’
    What is it finding that makes it think the application is running? There is not another copy of the application open. I threw out the cache for Thunderbird. I’ve saved the old profile on an external drive. Now, taking courage in both hands, I’ll try throwing away the Profile in the Library, to see if it starts a new one. That worked, so at least I have a mail program again. Now, how can I bring in my folders for the mail stored in subject folders, and how to retrieve the inbox which was not lost for the other account? Can I just put those folders into the new profile?
    I really thank you for your article. I could not have managed without it. Thank you very much.
    nandita

    ReplyDelete
  10. Nandita, you may well find that your anti virus program has indeed found a virus and quarantined your whole inbox

    ReplyDelete
  11. mjmanka@chemicalsoft.comDecember 4, 2011 at 1:45 AM

    I've not had my email messages disappear after compacting, but it did now. In the folder where I think the messages should be, the inbox and inbox.mfs files are missing whereas in my other mail boxes they exist. Is there a backup of the two missing files? If I proceed to close Thunderbird, these files will be recreated by Thunderbird with no messages. This is what happened in the past. Thunderbird needs to backup all files being compacted prior to compacting in case they are deleted. I can't believe this isn't done as a matter of common sense and then provide a recovery mechanism for the user. I would also suggest that compacting of files need not be automatic but let the user choose it or perform manual compacting upon notification from Thunderbird.

    ReplyDelete
  12. Please can me help someone?

    I compacted my Inbox in my TB 8.0 while running a anti-vir program. After compacting the inbox was empty, all my Mails are lost...panic!

    ReplyDelete
  13. Hi - Using version 8 I lost all of my in-bax when compacting - fortunately I use a Western Digital back-up drive that utomatically backs up changed files and I was able to recover all of my in-box from this.

    My question is - I have a file called nstmp appear in my inbox - that seems to have backed up my inbox e-mails from the compacting process - but cannot find anything about this in help.

    Does anyone know anything about this?

    ReplyDelete
  14. Good day. For everybody. Use the last version of TD (8.0) you will find a delete item that was not available before. You will find all your deleted messages, you can also transfer your previous mail files to the Local Folder to have the deleted email reappear.

    ReplyDelete
  15. I just hit "compact" by mistake and lost all my inbox e-mails. I am using Thunderbird 9.0.1. Where do I find the "delete item that was not available before"? Thank you.

    ReplyDelete
    Replies
    1. Ask your support questions in the Thunderbird support forum. http://getsatisfaction.com/mozilla_messaging

      One thing I can assure you is that the Compact process did not loose any mail. What vanished has been missing for some time. The compact forces a bew index and in the process uncovers corruption.

      The principle cause of inbox corruption is anti virus programs that think they know how to quarantine things that they don't like and don't.

      Delete
    2. What do you mean by "the Compact process did not loose any mail. What vanished has been missing for some time." I had been able to access and read all of the files in my Inbox until I selected Compact, at which point my Inbox became empty; all of the files had disappeared.

      Delete
  16. I have an issue where Thunderbird seems to think that there is no space on the device (USB), when in fact there is plenty, although Windows properties also says it is full?? I have archived a load of messages to hard disk, tried reindexing and with compacting, still getting the message that there is not enough space to do so. Any ideas?

    Stephen

    ReplyDelete
  17. I am getting multiple copies of the same message as many as 8 I delete and then from the trash box delete there also. The next time I log in the same thing happens

    ReplyDelete
  18. Wow. I read this because I did not know what ThunderBird compression was. Now I know, it might be time to pay 100 bucks for Outlook....

    ReplyDelete
    Replies
    1. To what point? Outlook has an identical function. Almost all mail programs do. Outlook has it's own problems with data corruption as well. 32 Million hits for corrupt PST when I looked this morning.

      You are not the only one who appears to think Outlook is a silver bullet, but you and they are wrong. All mail programs have problems and the grass is always greener on the other side of the fence.

      Delete
    2. Have been using outlook for last 7 years. Did not need to compress as i archive message folders in separate .pst files. these i load when i require. Only issue was search was restricted to default pst folder but New docfecher indexes .pst files too. No issue there :)

      Delete
  19. After my Tbird automatically updated on Feb 15, I rec'd email fr Tbird saying 70% of max reached in inbox. An hour or so later, I began getting emails from Tbird, each one telling me the sender and subject of an email which had been bounced because my inbox was at capacity. I've compacted folders; disposed of thousands of emails in my inbox, then re-compacted as you recommended; moved my entire inbox cache of emails to a "local" folder, and re-compacted folders. All to no avail. Still I am getting an individual notice in my inbox telling me it is at capacity and a specific email has been bounced.

    My Tbird software was automatically updated again yesterday, and I hoped it was a fix for this problem, but to not effect. Still getting the emails telling me of "bounced" emails. Anyone else having this particular experience? And, Matt, thanks a lot for your article and comments above.

    ReplyDelete
    Replies
    1. A number of senders have told me they sent emails that were returned undeliverable due to full mailbox on my end. I've deleted 1,000+ old messages from inbox and sentbox, moved many older messages from inbox to yearly files, compacted everything, and still get High Memory Useage message and reports of bounced emails similar to Wayne. How can I get more usable space ... especially after deleting and then emptying trash??

      Delete
  20. why do we have to have the compact part of TB? In the past month whenever I compact I loose the messages that I had in my inbox and they never come back. Can you turn off the compact button? Aren't they supposed to go inot a folder? When I look in the archives, there is only 1 folder with 2 messages on it.

    ReplyDelete
  21. I lost my whole Inbox just now, so this is broken.

    ReplyDelete
  22. I had a number of very important messages that I was keeping in my Inbox and now they are gone and no attempt at Repair has succeeded.

    ReplyDelete
  23. All of my inbox messages disappeared. I get a message stating my inbox is full and I should delete and compact. However it shows no messages to delete. Zero. I tried compacting but there are none to compact. I sent myself a test message and I just get the inbox full message.

    ReplyDelete
  24. My inbox messages disappeared also. Repair did not work. Now what? How can I get them back??

    ReplyDelete
  25. All the messages in my inbox dissappeared repair does not work, how can I get them back??????

    ReplyDelete
  26. I sent this comment to Mozilla Support Forum today....

    There appears to be a regular problem with TB deleting either Inbox or sub-folder mail during compacting. it has happened to me several times in last five years and I am more careful now about how I catalog mail and moving messages out of the Inbox into subfolders. I have auto-compact at 20MB and also take manual backups to an EHDD using MozBackup every month or so.

    Today however I lost over 1000 messages in a subfolder during the compact process - they were too recent to be covered by any backup; although the folder exists in the user profile all the .msf and blank filenames are gone. Attempts at Repair yield nothing.

    Worse still I have looked back over one month using ShadowExplorer to see if I can retrieve them from earlier Vista shadows and these too are empty of all content - even though other subfolders have clearly changed their content over this same period. The email files were there this morning so it appears TB is deleting the shadow files as well.

    To be honest I have had enough and will be looking at other mail clients now - I know this problem exists for other people because the web is full of similar experiences. Can someone from Mozilla possibly explain to me how this problem has remained unresolved for so long and what if any remedy I can now take to find my mail ? I am using TB 13.0.1 running on Vista.

    ReplyDelete
  27. All email in my inbox lost on compacting ... what's the fix to get back my mail???

    ReplyDelete
  28. Yeah, I have to agree with Jimmy. This is F****** ridiculous. Never had this kind of problem with any other email client. This is 2012 right? I have compacted and compacted every which way. automatically, manually, changed limit for auto. defragged my hard drive. no matter what, my hard drive stays fairly heavily fragmented. and just guess what the culprits are? hmmmmm........I wonder. Freaking thunderbird folders/files. Makes me nuts and very annoyed. I really try to support mozilla as much as possible. But this is just too much. I can deal with firefox bloat to a degree (despite how quickly it bloats and slows down with just 6 add ons) but this is insane and infuriating. Goodbye Thunderbird. Give me a shout when you grow up and join the adult programs and are ready for work. I dont have the time for this crap.

    ReplyDelete
    Replies
    1. Compacting is not defraging. Do you have Thunderbird open while you defragment your drive? You can't defrag open files, that is in the windows primer on deragmentation.

      Delete
  29. Great I HAVE TRANSFERED to THunderbird and when ask to compact folders on numerous occasions have agreed and now have lost all my e mails? why? where are they, why cant I find something in basic English to get the messages in the folders back. If I loose these messages filed away I am in s..t street!!!!!

    ReplyDelete
  30. How can we increase the cache beyond 1024Mb? Or increase the "saving" of space beyond 2048Mb? I don't want to compact my emails - I want to make use of my massive disc space - we mostly have Tb's of disc space not Mb's now!!
    Can we please have a Thunderbird revision to do this?
    Thanks guys!!

    ReplyDelete
  31. WHERE IS MY MAIL! I WILL PAY IF I HAVE TO, TO KEEP MY MAIL!!! THERE IS ROOM ON MY COMPUTER FOR ***MY*** MAIL! MY BOYFRIEND DIED AND I ***DO NOT*** WANT TO EVER LOSE THAT MAIL!!! PLEASE DO NOT FORCE THESE KIND OF LOSSES ON PEOPLE! GIVE US A MAIL PROGRAM THAT DOES NOT DESTROY OUR MAIL, WHICH CAN ALSO INCLUDE OLD BUT NEEDED PASSWORDS! STOP STEALING AND DESTROYING IMPORTANT THINGS TO US!
    I DO NOT ****EVER*** WANT TO COMPACT MY MAIL! STEALING MAIL FROM US IS NOT RIGHT AND IS NOT A SOLUTION!!!

    ReplyDelete
  32. I am traumatized by this loss of my mail. It is so NOT RIGHT!

    ReplyDelete
  33. What kind of heartless person designs such a thing?

    ReplyDelete
  34. There has GOT to BE a better way to do this and not lose our mail. I am scared spitless because I HAVE lost important and precious mail I can never get back, hard work, passwords and codes, and special mail from people who have died that were dear to me and I wanted their mail. I am not compacting because I have had those "blank emails" or mail with no attachments of pictures anymore. I don't want this lack of technical time consuming understanding to snare me. Surely there is a better way to store this mail that WE WANT on our own computers and not let strangers manage it. We buy the space! Please help us. Revolutionize email and give us our mail just as it was!

    ReplyDelete
  35. Also, I want it on MY computer, not the cloud, in case I can't pay that one month and then LOSE all my property, work and email!

    ReplyDelete
  36. Hi, im reading a lot about compact folder right now and on multiple sites people have reported loss of mails. Ive never used it before and consequently some of my local folders are huge.
    Is this loss of mails still a thing in 60.8.0 or was it fixed?

    ReplyDelete
  37. Furthermore will it ONLY delete mail from MBOX storage format that have been marked as deleted and nothing else?
    Or is the compact folder process doing more?

    ReplyDelete
  38. There HAS TO BE another way! and a better one. Use your energy to find it and not destroy our mail. I too lost a lot of important mail. Sorry, hard to trust after that!

    ReplyDelete
  39. Someone out there is going to get very rich when they make emails files that we can manage and not lose much better. Emails are so needed for business and person use - they need to interact, and I don't mean syncing! Today we have to copy and paste them, but a program that can help us with them is a dream we need to see come true. Yes, I've lost files before and there are things I cannot get back from those who have passed on, or special codes that I need that I thought was safe in email! We lose programs because we lost the link or password (since the computer kept it. Important conversations, business reminders and suggestions, important pictures, too much that you just don't want to see vaporized. Without question there can be a better email system that is a whole new easier working set up. This is good for now, but it is way too fragile. Someone will get smart and brave. It would be wonderful to have a folder at the end of each year that could convert to a storage file of letters/emails and then use keywords to find them. Then we could transfer them from computer to computer as needed since computers play out every 3 years or so. A worthy back up system would be awesome!

    ReplyDelete
  40. This thing should be brought UP to date or DELETED

    ReplyDelete
    Replies
    1. What is "this thing". If you are talking about my 2011 post about compacting, perhaps you should use the official Thunderbird support location where compacting is discussed and this prototype of the support document was placed a decade ago and where it has been updated. See https://support.mozilla.org/en-US/kb/compacting-folders

      Delete
  41. Hi, I understand what compacting does but I'm not clear on what impact this will have on deleted items. If I delete an email, say from my inbox, does it get copied to deleted items? If so, compacting the inbox makes sense, otherwise I'll now have two copies. However, if deleted items is merely a filter of emails marked as deleted from other folders, then compacting the inbox isn't going to be congruent with expectations. Does anyone happen to know which of these scenarios (if either) is an accurate description of what actually happens?

    ReplyDelete