Can not backup 4GB file - also QDF keeps growing in size (QMac)
I have reached a citical point in my daily usage of Quicken de Luxe on MacOS.
Version 7.2.1 (Build 702.49920.100)
MacOS Ventura 13.5
Basically, I can not backup, quit or upgrade Quicken anymore because of bugs in
the program. The program will only store part of the data.
The problem started when the file became larger than 4 GB and it smells like
a "32-bit" problem.
The portfolio parameters are:
securities : 2225
categories : 562
transactions : 102,760
accounts:
updated : 33
non-updates : 27
File Size:
- saved/backup : 1.7 GB
- running : 4.4 GB (control +click/show package: data, all other folders nearly empty)
Memory : 1.98 GB
Following is a workaround but might not work in the future and is therefore not
a long-term solution. While the program is running , make a copy of the work file which
is in my case 4.4 GB. Only after making that copy you can quit the program or
hit the upgrade button (which effectively makes you quit the program).
This copy is your backup. So if you want to save periodically backup's you have
to copy again the running file.
Start up the program using these copies. The actual Quicken backups are
much smaller because I assume that at opening new structures are populated.
I have several times tried to discuss this with support but the responses
are that either my file is too large and I should start a new file or I have
to make a backup first before continuing the discussion.
As a side note, it is hard to understand where all that disk space goes:
My estimate would be:
transactions : 100,000 (transactions) x 10 (fields) x 8 Bytes = 8MB
historical daily data : 2,225 (securities) x 365 (days) x 20 (fields: OHLC Vol ..) x 8 Bytes = 130 MB
Is there anybody out there that has this experience too ?
Comments
-
just following along -
how did you get such large numbers of Securities, Categories, & Transactionsguessing Options Trading ?
securities : 2225
categories : 562
transactions : 102,760
0 -
Do you have any attachments in your data file?
My data file has about 70,000 transactions — so about 70% of yours. And my data file is only about 92 MB, or about 2% the size of yours!
I have about 325 categories, but I can't envision of the number of securities has any impact on file size.
Now, I only have about 100 securities, so far less than you. But as you say, even the storage of security prices for 2200 securities shouldn't take up too much space.
So there's some mystery here about why your file size is so massive.
I'd suggest trying one thing just as an experiment: export your data (File > Export > Quicken Transfer File (QXF)"), then create new file (File > New), and select to start from a Quicken Windows file, and point to the exported file. (If that doesn't work, do File > New > Start from Scratch, and then do File > Import > Quicken Windows File (QDX, QXF).
This new file will not have any of your custom reports, and you will have to re-set your register display settings — so you may not want to continue with it if it doesn't solve a problem — but it should have all your transactions, securities, categories, payees, etc. I'd be curious to know the size of the new file.
Quicken Mac Subscription • Quicken user since 19931 -
@jacobs - would the QXF transfer file also contain the attachments, if any existed ?
0 -
No, documents aren't retained in a transfer file.
Quicken Mac Subscription • Quicken user since 19930 -
@Jacobs , I read previous answers from you advocating the QXF path and my back was enough to the wall to take the jump.
I managed today to bring my Quicken file back up, luckily I have no attachments. Resurrecting the Quicken file from a QXF file went fairly smooth (A few transactions had to be edited, not completely trivial to find them between the 100,000 transactions).
So here some statistics:
File sizes:
- qxf file: 130.7 MB (pretty close to my simple estimate of 138 MB)
- saved/backup : 224.8 MB
- running : 541.5 MB (control +click/show package: data, all other folders nearly empty)
securities : 2006 (so ~ 200 disappeared)
categories : 562
transactions : 102,090 (so ~ 600 disappeared)
@Ps56k2 Using "tax-alpha" strategies nicely work you through the 5000 US stocks
The question remains what caused this bloated file and does the file management of Quicken have a "32bit" issue. The last question is for me at the moment academic. The first one might have to do with several Quicken software updates where "data restructuring" was performed.
0 -
a) I can't answer, or even fathom a guess, why your data file was so bloated. I don't know if you save periodic data files over time — I save a manual backup every 3-4 months and leave those backups around for years — but old backups or looking back through Time Machine, if you use it, might enable you to zero in on when your Quicken file ballooned in size. I think it's a good guess that something may have gone wrong during one of the version updates which updated the database structure, but pinning thant down now might be impossible, or useless. We haven't seen any other reports like yours, so it's not likely a widespread bug.
b) What do you mean by repeatedly suggesting there is a "32-bit issue"? As you're likely aware, Apple ceased allowing 32-bit applications several years ago, starting with macOS Catalina. And The Quicken Mac application is compiled to run in native mode on M-series Mac hardware. So I don't think there's any issue with the application not being up to modern Mac standards. Are you instead referring to the database structure? Quicken Mac uses SQLite, the same industrial-powered database used throughout the operation of macOS. There is no limit on the number of records, if that's what you're suggesting.
Quicken Mac Subscription • Quicken user since 19930 -
@jacobs indeed I am referring to 32-bit integer usage. The fact that the underlying MacOS libraries have moved on to 64-bit is no guarantee that all internal Quicken code did too. From my observations across the years I see that not enough backtesting is done by the development team. Since you are running also a large system, you remember for sure the download times for transactions after some software change around October last year …
0 -
Looking at the lots for each stock all information concerning "minimum gain" sale is lost in the QXF transition. :-(
So from a tax perspective completely useless.
0 -
Looking at the lots for each stock all information concerning "minimum gain" sale is lost in the QXF transition.
Can you explain how this is, or what you see? When you sell shares of a security and click to Select Lots, the pop-up Specify Lots window simply lists all the purchases of shares in the history of the security in that account. The Minimum Gain selection should just be doing the math between the purchase price and selling price of each lot to select the ones with the smallest gain. What information is missing after your import into a new file?
Since you are running also a large system, you remember for sure the download times for transactions after some software change around October last year
I mostly enter my transactions manually. But in the accounts I download in my testing file, I did not notice a major change in time last October.
The fact that the underlying MacOS libraries have moved on to 64-bit is no guarantee that all internal Quicken code did too.
I believe that making Quicken Mac fully 64-bit compatible for memory management was built into this modern-generation Quicken Mac since it debuted.
Quicken Mac Subscription • Quicken user since 19930 -
@jacobs As of 8/8/2023 I look for instance at the MSFT lots in the newly rebuild file from the QFX file and compare it with the original (bloated file) . The rebuild was done according to the setting of the account , so FIFO or LIFO. My original file was obtained by choosing for each transaction "minimum gain" and therefore contains still lots from many years ago.
(minimum gain is also unfortunately poorly implemented, not taking short/long term into account)
I will now have to go through each sell trade in each account starting at the oldest date and choose "minimum gain".
I do each day a download of the transactions. There are on average 50/day and I rather not enter them by hand.
Sure their intention was to make the code 64-bit compatible, but neither you or I saw the code.
0 -
Ahhh, now I understand what you were saying. For Sell transactions in the past, Quicken imported them using the setting for the account (FIFO or LIFO). I assume that if you were to enter a new Sell transaction today, you would be able to select Minimum Gain in the dialog box.
(Much of my investments are in mutual funds which, in the real world, use Average Cost Basis — and Quicken Mac doesn't support Average cost Basis. As a result, I have to treat Quicken's reports on gains as estimates rather than actual values; I use the year-end 1099s and such from my brokerages for actual tax filing purposes. For that reason, I don't pay much attention to Quicken's sales by lots feature.)
Quicken Mac Subscription • Quicken user since 19930 -
The main reason I do accounting (in Quicken) is to get the quarterly tax estimates right.
0 -
Yup, I get it. I was just pointing out that there are multiple issues with Quicken Mac producing accurate tax info.
Back to the QXF export/import I originally suggested: I noted the new file might not be viable for using going forward due to loss of various settings, but would at least by a test to determine if your original file bloat would go away. Unfortunately, there's no way to know why your original file has this exceptionally large size, nor a way to "fix" it. So I think you're stuck with either using the original file (which seems problematic and unlikely to ever get better) or using the new file but having to re-create all your settings, reports, etc. — and now you've identified that security sales by specific lot are part of what needs to get re-created. Sorry! 😉
Quicken Mac Subscription • Quicken user since 19930 -
Oh, apologies if it seems that I suggest regretting the QXF path ! It is/was the only option going forward ! I have 2 months to edit the sell transactions before the next quarter ends 💪
0 -
Good luck powering through those edits. Hopefully you don't have too massive a number of Sell transactions to edit!
Quicken Mac Subscription • Quicken user since 19930 -
@jacobs, after many edits I have my portfolio restored starting from a "clean" qxf file.
Unfortunately, the file is growing again. The reader can see above that I started with 541.5 MB on Aug 7
(saved/backup with compression : 224.8 MB) and on August 19 it is 759.5 MB (saved/backup with compression : 313 MB). No attachments are stored, this is just the data file in the package.
It turns at that every "accounts update", accessing JPMorgan and UBS, increases the file by 20-25MB. So if I do it twice in a row it is 40-50MB. Of course the second download has no new transactions. I expect to hit in about 150 days to hit the 4GB again …
0 -
That's so odd. I haven't seen any other users report this type of activity (although most probably don't check their file size very often 😉).
One question I don't think we touched on previously: where is your data file located. Specifically: is it on your local hard drive, as opposed to a cloud storage service such as iCloud or Dropbox? (If you store your data file in a folder which is on your Desktop or in your Documents folder, you might have those folders saving to iCloud in your System Settings.) Storing the data file on a cloud storage service is known to produce a number of undesirable results, some of which may relate to file size. It's a long shot, but I thought I'd ask.
Quicken Mac Subscription • Quicken user since 19930 -
@edmondoffermann I have a similar problem but not as drastic as yours. I first reported a year ago that my file seems to grow about 100k whenever I enter a few simple transactions - maybe two or three payments and about the same number of credit card entries plus maybe ten downloaded investment transactions. My working file then was 254mb and the compressed was 88mb. Now the working file is 316mb and 104mb compressed. I have about 94,000 transactions going back to 1989. I tried exporting the file to QXF and creating a new file from it and the result was a file size even bigger. Quicken is stored locally and only backups go to the cloud. Mobile sync is off.
0 -
@jacobs, yes I have the files stored in a folder in the Documents folder which is stored by default on iCloud . I will remove it from there to a place that is not iCloud stored and do my tests again. It is important to point out that although iCloud storage is used I do not "Optimize Mac Storage". So all my files are aways physically on my hard drive and accessed from there.
@magoodm, yes I am aware of your postings and very helpful back and forth with @jacobs
0 -
@jacobs, created a folder in my top directory, copied the file to it (584 MB) and did twice a "Accounts update". Now I have a 604 MB file (and of course no new transactions downloaded).
0 -
@jacobs @magoodm , I am touching on two bugs in Quicken: 1) Quicken files > 4 GB can not be backup-ed and therefore the program can not be quitted or upgraded etc. 2) transaction downloads increase the file size regardless of the number of downloaded transactions (and therefore reaching in a finite amunt of time the 4 GB).
Reading the exchanges between @jacobs and @magoodm and observing the changing file sizes during backing up and/or quitting it started downing on me what was happening in the Quicken code. I am still convinced it is a 32-bit bug (aka casting a long into an int). It actually can be circumvented by not selecting the option to encrypt the file in Settings/Security. It is a misnomer and this option causes the file to be compressed besides encryption. So the bug is in invoking the compression code.
0 -
I'm at a loss to explain these large jumps in file size. I just looked again at my testing file, which has 30 years of transactions and is only 86.1 MB in size. I launched it, did an Account Update, and it remains 86.1 MB. In this file I have only one account connected for downloading; it downloaded a few new transactions.
I cannot understand what would cause a file to increase by 20 MB after doing an Update All Accounts twice; that's equivalent to one-quarter of my 30 year-old file. Perhaps someone else here might have ideas about what could be going on.
The only further troubleshooting step I can think of is to disconnect all but one account from downloading, and do an account update. If the file size jumps up, disonnect that one and try another one. Try them one at a time to see if you can isolate that perhaps one financial institution's download are causing the big leap in file size. But it might be a pain in the butt to do this, and I don't know if it will ultimately be helpful.
I feel like you'd need to get a Quicken programmer to look at what is changing in the file after each account update, but unfortunately I don't think front-line Quicken Support will understand or substantively engage to get you bumped up to someone who could really dig into this to see what's going on. I think the more you can pin it down to account updates, and ideally account updates from a specific financial institution, the better the chance you could get a Support representative to watch it with you on your screen, see what you're seeing, and escalate the issue.
Quicken Mac Subscription • Quicken user since 19930 -
@jacobs , First of all I appreciate all your comments and all the time you put into this forum, you rock !! That being said, this whole structure of support by Quicken is another story. I think that I know which financial institution is causing the problem and I kicked up enough dust there that a meeting is organized with their team. I pointed exactly where the 4GB problem is, let's see if a Quicken coder will contact me ;-)
0 -
I think the bigger problem than backing up a giant file is that your file shouldn't be anywhere near 4GB in size.
For what it's worth, in creating a backup, I believe Quicken is just invoking a macOS core utility ("tar") to compress the file. It's not Quicken code, so if there's a problem there, it's likely a macOS issue. I do not encrypt my files, though, so I have no experience with how that impacts file storage or backups.
Quicken Mac Subscription • Quicken user since 19930 -
tar is a shell / system command that would have to be called outside the program. More likely that lib zip is used and that it is called with the wrong argument in Quicken, but we are only guessing here. Fact is that there is no problem without compression.
0 -
hi all .. should we ask the Admins to split this baffling case into … 4GB back up - vs - QDF file keeps growing ?
Or just leave it ?
Also should we edit the topic to reflect the File Growing problem … vs 4GB0 -
@Ps56k2 , just for the record, I started after getting this 4GB problem with a "clean" QXF file. But since then I am working with the native Quicken file that keeps growing with each "download transactions".
0
