Quicken File Ballooned up to almost 2GB - Help!

DavidVC

Hello,

Been a Quicken user forever. I've got a single file with all my data and it's been a normal size. Maybe a tad big but nothing out of control - till now. I just noticed that it was taking way too long to close the file and I peeked at the size - 1.9GB! This is all of a sudden, there is no way it was that big a few days ago. I'm on the Mac subscription so the program updates nearly every time I use the App. I apply the updates and move on. I do have Time Machine so I can roll back to an earlier file but I don't want to lose the data I have in there. Something happened and I'm not sure how to fix it.

I did try making a backup and then restoring from the backup. No change. The backup is also 1.9GB as is the newly created file. Very odd.

Anyone have any ideas how I can get it back to normal?

Thank you

jacobs

That certainly does sound strange and worrisome, especially since there is no file utility to compress or rebuild a Quicken data file.

I'd start with a little investigation. Right-click on your Quicken data file, and select Show Package Contents from the pop-up menu. A Quicken data file is actually a special type of Mac file called a package, which is like a wrapper around a bunch of files and folder to make them appear like one file for the user. When you Show Package Contents, it shows what's inside the wrapper. there should be about 10 folders and one plain file called "data". That file is your actual Quicken database file. What we're interested in is the size of that file, and perhaps of what's in the other folders. My "data" file is 72 MB with about 30 years of data in it. Yours might be larger or smaller depending what sort of data you have, but it certainly shouldn't be up in the GB range. How big is it?

Is the "data" file isn't huge, your next task is to find out what is taking up all the space inside this Quicken data file package. Open each folder and look for anything that's really large. Report back what you find.

DavidVC

Hey thanks for your reply. I had been down that road some time ago and found a slew of duplicate files inside the container. However, this new 2020 format is not the same. When I Control+Click the file, the are only three things. A folder called "locks", a file called "data", and the monster, a 1.9GB file called "catalog." My "data" file is tiny, 25KB (not a typo)...it's this "catalog" file that's taking up all the space. What is that do you suppose?

jacobs

Aha! That tells me your Quicken data file is not being stored locally on your Mac; it's being stored on Dropbox mostly likely, or possibly iCloud. That's a no-no, and the source of your problem; you want to get your Quicken data file back on your local Mac. 

DavidVC

Indeed it is on Dropbox...that's a feature I have liked as it lets me use the file from home or work, and on my laptop or desktop. Honestly I've been using it that way for years, matter of fact, Quicken (used to, or maybe still does) give you extra Dropbox space with your membership.

Anyway, even when I copy the file local to my Mac and out of Dropbox, the file is still the very same 1.9GB in size. How can I get it back to normal?

Bonus question if you care to elaborate, why is storing it on Dropbox a bad thing to do?

Thanks for your help - I greatly appreciate it.

jacobs

Storing a backup file on Dropbox is fine; it puts a "wrapper" around the file that Dropbox doesn't penetrate. 

But using your live data file on Dropbox invites Dropbox to move portions of an active database between the cloud and your local desktop and control locking and unlocking parts of the data as it does so. Dropbox mostly handles this okay, but sometimes things go wrong. The most common problem seems to be that some of the permissions for the files and folders within your Quicken data file package get messed up in moving between different computers, which can result in you getting locked out of your data file entirely.

I have no idea why your file is bloated. Perhaps someone else who works with Dropbox more can offer an explanation. You can see what happens if you do File > Save a Backup and then Restore from Backup, but I don't think this will accomplish anything. Otherwise, since you said that this happened in just the past couple days, you might be best off going to a backup file from just before that which is smaller, and then catching up your most recent work. (Do you have a backup file in a Time Machine backup? If not, that's something else I can't recommend strongly enough going forward.)

The preferred workflow for moving between two Mac is to Save a Backup at the end of your Quicken session, and move that backup file to Dropbox. The next time you want to use Quicken, copy the latest backup to one of your Macs and open that. As long as you take the extra few seconds to always copy a backup file to Dropbox and to always pull that file down to start from, then you're assured of having the latest file no matter which computer you're using.  

Chris_QPW

Note I'm a Windows user, but I think I can provide a bit more information on what might be going on.

First on databases in general backing them up tends to require cooperation between the software managing the database and the software backing it up because database contain lots of "links" between different parts.  Like updating a transaction might means changing several tables, and the Quicken software would expect those updates to be consistent.  If the backup happens while the database is doing this update parts might get update, and other parts not updated.  Then strange things can happen.

For instance one known problem in Quicken Windows (that luckily seldom happens) is that the date in an automatic reminder gets corrupted somehow, and when the user opens the data file Quicken thinks it needs to catch up and enter thousands of transactions.  Needless to say Quicken become unresponsive and the data file grows immensely.

You might be able to find out what part of the database grew really large by opening up that "file" which is really a "package" and see what is so big, but in truth that will most likely not lead to any kind of solution since these aren't going to be user editable files.  You will just find a huge file or more.

Also database almost never "shrink".  To optimize speed when they are "deleting" they tend to just mark records "deleted" instead of actually deleting them.  This saves time in the actual writing out of the file, and later when you ask to add something else it can just reuse those records instead of having to expand the files to get more space.

So it is very unlikely that you can reduce the size other than going back to older copy, because first off whatever "expanded" the database hasn't been reversed, and second just because of the way the database are optimized.

It isn't that Dropbox is taking up the space and moving it to a local one moves a copy that doesn't include "Dropbox overhead" will be smaller.  The "expansion" is in the the Quicken database files themselves.

jacobs

@Chris_QPW  I'll respectfully disagree regarding Quicken Mac. We've definitely seen cases like this before with people who have their active files on Dropbox. As discussed higher in this thread, storing the Quicken data file on Dropbox actually changes what's in the 'package' file. Dropbox creates a "catalog" file which isn't something Quicken Mac creates. What's inside that catalog file, and why it sometimes blows up in size with duplicates of parts of or duplicate copies of the SQL database file, is not clear. But it's not something Quicken Mac is doing; it's definitely Dropbox.

Chris_QPW

@jacobs I'm sorry my mistake for thinking that it worked the same as on Windows.

DavidVC

Hello guys, thanks again for the continued assistance here. So I understand a bit more about the whole Dropbox thing but here is what doesn't make sense. I've copied the file local to my computer. I've opened it and saved a backup. Then I restored from that backup - which in theory should rebuild the file. Yet that rebuilt file has the same structure with the whole "catalog" file which is NOT a package and I can't open deeper to purge any conflicted copies or whatever Dropbox does to mess it all up. To me, that doesn't make any sense whatsoever. I would think the rebuilt file would be the normal quicken format.

I do have backup of course but I can see now that this has been a gradual process. Only in the last few weeks has it gone to the monster file it is now. But reverting to a copy a month or two old would be a problem because of all the lost transactions. There must be a way to save the good parts of my file and purge the bad...

Let me know if you guys have any other ideas and thanks!

jacobs

@DavidVC  Saving a backup and restoring from the backup unfortunately does not "rebuild" the file. That's why I wrote above that it was worth a try but I didn't think it would help. The backup file is very similar to a making any file into a .zip file with File > Compress in the Finder; Quicken does some compression and puts a wrapper around the file.

In the old Quicken 2007, there was the Save A Copy command, which actually re-wrote a file, record by record, into a new file, so it was useful at times for solving file corruption issues. Which was necessary because the old database was prone to various corruption issues. There's no such utility with modern Quicken Mac, as modern SQL databases don't have the types of issues older database did. 

Again, the problem is whatever Dropbox does to modify the internal contents of the Quicken data file package, which transforms a number of the Quicken folders and files into its "catalog" file. I don't know how Quicken works with the file once it's in this state; there might be some invisible aliases, but I don't use Dropbox and don't know what more to advise. The only sure thing I know is reverting to an older backup -- and I understand that you'd lose a bunch of work you'd prefer not to.

One other possible suggestion: you could try exporting the file to Quicken Windows format, and then starting a new file and importing the Quicken Windows data. This technique is sometimes used as a last resort if there are unexplained connectivity issues with a file. but be aware that this will preserve just your data; you'll have to re-create reports and budgets, set up your register columns again, and other set-up steps -- which is why I call it a last resort. In your case, you might try it -- the export and import will just tie up the computer for a bit -- to see if that cleanup would be less painful than going back to an older backup file. 

DavidVC

I called Quicken Support, which was not the best, but the guy was willing to try some things. He ended up suggesting just what you said, I exported as a Quicken for Windows and imported into a new file. This worked surprisingly well except that for some reason, when I linked my accounts, it basically added double transactions for the last 3 months.

I have not used the file yet so if there is some magic to NOT have that happen, I'd welcome that! I wish I could tell it not to import anything older than today because the file was current when I created it.

Anyway, the file size is now TINY as I suspected it would be. I also looked at my backups, the file went from 300MB to 600MB then last week went from 600MB to 1.9GB! Crazypants.

The newly imported file is 132MB...

jacobs

DavidVC said:

for some reason, when I linked my accounts, it basically added double transactions for the last 3 months.

Those transactions probably don't retain the financial institution ID number through the export and import that Quicken uses to prevent duplicate transactions. So this is a one-time problem after a change like this; once you remove the duplicates, you shouldn't see more. 

lhossus

Sorry I am late to this party ...

One thing I would try if I had this problem is to go inside the Quicken data file package and simply remove any extra files that are present there. The fact that the data file works with extra files present suggests that this is possible.

Do this by comparing the data file package content to a known good data file package content. If nothing else, you might just create a new empty Quicken data file.

I have no experience with Dropbox, so cannot say more. But maybe you will have to move things around to be in the proper folders, given that Dropbox seems to be playing some tricks with the structure.

Another thing you could try: Take a copy of your last "good" data file (from your backups) and replace the "data" file there with the one from your most recent working data file (the one that is huge). Or maybe just create a new Quicken data file and replace its "data" file with the one from your most recent data file.

Just ideas. No guarantees. But if you could get one of these ideas to work, it likely will benefit others in the future.

In case in might be helpful, here is the Package Content of my data file:

DavidVC

@lhossus , apparently Dropbox changes the file so that structure no longer exists and none of my backups have it (I've been using it on DB for years...) Anyway my rebuild (which I will document) is almost complete and I'll post up details about how to do it. My only remaining issue remains that about 3 month of transactions were duplicated and I can't find any automated way to remove them. It's taking me forever to do...total bummer.