New Quicken File

Pilotbum

I've been a Quicken user for 28 years. That said my present file is ALL 28 years of financial data. I have had some Quicken Tech support reps tell me it doesn't matter how big the file is and others tell me I should trim it down every couple years. One tech support rep told me they recommend starting a new file every several years regardless as there are some aspects of my old data file that cannot be upgraded. Some of which I've seen first hand throughout the years. 

Anyway, my file is finally showing some data errors that are not fixable through a Validation or Super Validation so 2021 I'm going to toss in the towel and start a new file. As I'm trying to plan the best way to do this I'm curious if I should start a brand new file from scratch or if exporting/importing things such as Memorized Payees, Securities, Categories & Business list would be possible/advisable given the age of my original file (1992).

GeoffG

If you are willing to put in the time and effort and it sounds as if your are, I would strongly suggest starting from new and rebuild from there - no imports. If you want a clean file without skeletons, that would be the cleanest. I've rebuilt test files from scratch using all account online updates and was able to relatively easily create a file that was up and running fairly quickly, albeit without much history. You will always have the existing file for history reference as needed. My $0.02

Pilotbum

Thanks, Geoff. I was pretty sure that would be the advice I'd be given and thought it would probably be the best course of action; just wanted to get other opinions. Things seem to be functioning fine with the present file, just a couple errors that I can't get to go away and errors seem to be all too frequent. I'll certainly keep the old file around for reference. 

splasher

Have you done a File->File Operations->Copy to see if it resolves your issues.

My data file goes back to 1996 and I will just about do anything to keep it in a single file.

If you do start over, you have to make sure that you keep opening the old data file to keep it up with the software changes otherwise it will become useless.

Rocket J Squirrel

I would try importing some things via QIF. It should be fine to export/import the category list, for example, and save yourself the trouble of recreating all your favorite categories. And really, if you're starting fresh with File > New, importing other lists via QIF should be benign. It's just a text format which probably won't carry corruption along with it. I don't think I'd trust it with actual transactions, but payees & securities might be OK. I'd try importing such lists one at a time and see how it goes.

Pilotbum

splasher said:

Have you done a File->File Operations->Copy to see if it resolves your issues.

My data file goes back to 1996 and I will just about do anything to keep it in a single file.

If you do start over, you have to make sure that you keep opening the old data file to keep it up with the software changes otherwise it will become useless.

I have not tried that but wouldn't really expect any change if it's just a copy? Wouldn't it just be copying the corruption? 

I'm the same way, I've tried to keep all my data in the same file for as long as I could but am now finally starting to think it's time. It's not uncommon to get a long list of errors with a validation (and I run the validate frequently). Recently I've had 3 corrupt scheduled transactions that it says it fixed but every validation it says the same thing, "corrected 3 scheduled transactions..." so I don't think they're actually getting repaired. I tried deleting all my scheduled transactions and setting them back up and that didn't help so I'm assuming it means scheduled transactions that have been entered in the registers. 

Pilotbum

Rocket J Squirrel said:

I would try importing some things via QIF. It should be fine to export/import the category list, for example, and save yourself the trouble of recreating all your favorite categories. And really, if you're starting fresh with File > New, importing other lists via QIF should be benign. It's just a text format which probably won't carry corruption along with it. I don't think I'd trust it with actual transactions, but payees & securities might be OK. I'd try importing such lists one at a time and see how it goes.

I always thought all of my data was not more than text & numbers in database tables but one time when I was talking to tech support on the phone I was told there's more to the data files and there have been some aspects that have not been able to be upgraded to later versions and that's why closed/hidden accounts, for example, continue to show up in my category/account list in the transaction entry fields. 

Chris_QPW

Pilotbum said:

I always thought all of my data was not more than text & numbers in database tables but one time when I was talking to tech support on the phone I was told there's more to the data files and there have been some aspects that have not been able to be upgraded to later versions and that's why closed/hidden accounts, for example, continue to show up in my category/account list in the transaction entry fields. 

I have done a lot of data conversions in my career, in fact it is one of the main things I did and there is some truth that this statement, but I do also think it is being used as an excuse for the lack of a feature or a bug.

Your closed/hidden accounts continue to show in the category/account list because of bugs/lack of proper design.  Not because it isn't possible.

If you think about it the very fact that the account name shows up in the category field is "strange". It was clearly done to save space on the GUI and to force the fact that you can't categorize a transfer. And as such you immediately have the problem is that entry a category or is it an account?

The [Account] show up in the category field, so it seems like it should be treated as a category, not an account. But note that you can't hide it!

So "hiding an account" really doesn't directly apply, does it?
Well people think it does, and clearly the developers are confused about it too.

In a old data fie "hide account" will not hide [Account] from the list, but it will in a new data file.  So it is just on "old data file/conversion problem", right?

Not so fast!
There is a setting "show hidden categories" that popup category list for the field.
If you select it in a New data file the hidden "account names" will not show up!

This is just a series of bad design/bugs it isn't a "conversion problem".

Real conversion problems come up when a feature changes or is removed and there isn't really any conversion path.  As in the old data file doesn't have the information to fill in what the new one needs.   An example of this kind of problem would be in the old database you had an "asset account" and it wasn't really marked as a "house".  Now you want to attach the Zillow option only to "house accounts", it isn't really possible to do the conversion 100%.

splasher

Pilotbum said:

splasher said:

Have you done a File->File Operations->Copy to see if it resolves your issues.

My data file goes back to 1996 and I will just about do anything to keep it in a single file.

If you do start over, you have to make sure that you keep opening the old data file to keep it up with the software changes otherwise it will become useless.

I have not tried that but wouldn't really expect any change if it's just a copy? Wouldn't it just be copying the corruption? 

I'm the same way, I've tried to keep all my data in the same file for as long as I could but am now finally starting to think it's time. It's not uncommon to get a long list of errors with a validation (and I run the validate frequently). Recently I've had 3 corrupt scheduled transactions that it says it fixed but every validation it says the same thing, "corrected 3 scheduled transactions..." so I don't think they're actually getting repaired. I tried deleting all my scheduled transactions and setting them back up and that didn't help so I'm assuming it means scheduled transactions that have been entered in the registers. 

This Copy operation is at the transaction level and not just a DOS copy of the file, so in some cases it will resolve some issues.

Pilotbum

@Chris_QPW You obviously know way more about this than I do but you more or less have supported an opinion that I have had for many years. I've also always suspected that much of Quicken is based on old DB technology that has not been upgraded. That is however based on VERY LIMITED knowledge of the subject and a hunch more than anything. 

@splasher Thanks for the info. It didn't help. Did the copy and 2 more validates and still got the same error, "3 scheduled transactions corrected".. every time I validate. 

One thing I still may try is to go back to a backup on the 11th where I noted on the backup file I validated it without errors. I've tried to keep all my data in one file over the years and there's another part of me that just thinks it may be time to start over...

Another problem that I've had recently the Quicken tech support has blamed on an "old corrupt data file" is I track my income in gross earnings and itemize the paychecks using the paycheck feature. My paychecks have some line items that may not be present one month and present another. So sometimes on a scheduled income (paycheck feature) I will add or delete a line item for a particular entry. Too much of this however and the arithmetic in the different sections of the paycheck don't add up correctly, and the total will be off. In other cases I'll delete one line item in say the post-tax section of the paycheck and it'll also delete line items a different section. Quicken had me start a new data file thinking it was corrupt data and I got it to do the same thing in the new data file... so apparently NOT the data file. I was wondering if for some reason or another too many line items on their paycheck feature caused the problem. Thought about just doing my income in split transactions like I did before the paycheck feature existed since I really don't think the paycheck feature is more than a graphically enhanced split transaction anyway. 

Would love your guys' feedback... 

splasher

You are correct, the paycheck wizard is just a split transaction with a fancy front end to "make it easier" for those that don't understand how to do it the "old way".
I do know that if your paycheck varies from paycheck to paycheck, you want to change the transaction AFTER you enter it versus changing the reminder from the Tools menu.

Pilotbum

@splasher Just to clarify, I don't change the reminder but rather when entering the reminder I'll change either the values in various line items or add/remove line items when entering from the reminder. Are you saying I should enter the reminder as saved in a scheduled transaction and then go back and edit it after it's been entered into the register? 

Chris_QPW

Well your problem with the paycheck certainly sounds like a bug to me.  I personally don't do what you do so I can't verify it one way or another.  What I do is leave in all the possible lines, but set the non used ones to zero when entering them.  One reason I do that is because I set the non used ones to an average for predicting the income/expense for the budget for the month/year.  The other reason is just easier than adding and deleting lines all the time.

I would add that there also have been all kinds of reports of problem with the paycheck entries for people that use the cloud sync to Quicken Web/Mobile.  And now that the same cloud sync is being used for Express Web Connect accounts, if this is in such an account that might be the problem.  The paycheck reminders I use for my wife's paycheck are in an account setup with Direct Connect.

There is one thing that I think the paycheck does that is "special" over a straight split transaction.  For contributions to an account like a 401K it uses a hidden tax account which of course you can't do in a split transaction.  It can be simulated, but it is "different" in the way the entries will look.  But more to the point for your problem, is that there are hidden entries, and I can certainly see a bug where changing them up a lot where most people don't change them, might lead to causing such a bug to appear.

And at least for the regular reminders they have programmed it that it does different things where if you change it while entering the transaction in comparison to entering it, and then coming back and changing the values.

splasher

Yes, save it first, then change it.  

Pilotbum

@Chris_QPW that's what I do now--a list of all the possibilities and leave many at a zero amount. That seems to have solved the problem. 

Re: Paycheck feature... 

1. I don't use the Cloud Sync. I've tried it and LOATHE that they store my banking credentials in the cloud. That should be my option, IMO. I would rather the cloud just sync mobile devices and let my desktop do the downloading from banks. I also have 2 financial institutions that lock me out after a second device (cloud sync) tries to download transactions requiring a phone call to reset. Every. Time. These banks only allow one thing to do the downloads for security. That said, I've tried it and quickly stopped. That was long ago. Would that have a long-term affect on the file? 
2. The paycheck account is also Direct Connect. 
3. I travel a lot for work. Since Mobile/Web isn't a workable solution for me I sync my Quicken data file between my desktop at home and my laptop I travel with. Could this be causing me the problems? I have not been able to ever notice a pattern associated with errors and syncing between devices. (I actually sync via my NAS)

As far as the scheduled transactions go are you saying all scheduled transactions will have similar issues if you change the values during entry rather than entering the saved transaction and then coming back and then changing the values? For example, my utility bill for Gas & Electric estimates based on time of year. I itemize the gas and electric charges and change the values (which are obviously never the same) as I enter the transaction. Would I be better served to do the same as @splasher suggested with paychecks and enter the transaction as the reminder is programed and then going back to edit the values later? 

Years ago I used the Quicken Mobile for my palm pilot and it worked pretty well. That's the kind of functionality I would like, just sync my transactions between my devices and let me store my credentials on my device ONLY. 

Chris_QPW

#1 I doubt that using Quicken Sync for a short time way in the past is really going to affect your data file now.  But also keep this in mind:
https://community.quicken.com/discussion/7882641/qcs-express-web-connect-is-cloud-sync

#3 I take it that you copy your data file to the NAS and then copy it to the other machine from it.  They don't recommend actually opening the data file on the NAS because a NAS isn't as reliable as a disk, and Quicken has nothing in it to correct any errors.  But copying to and NAS and back is just fine.

On the scheduled transactions.  What I was referring to isn't exactly like the problem with the paycheck reminder.  In the past if you changed the transaction while entering it in from a reminder, the reminder itself wouldn't change.  Then not too long ago they changed the behavior that it would change the reminder for future transactions.

That is the main reason for the people that didn't want that reminder for the future to change, to first enter it in the register and then change it there where it is already a regular transaction and as such will not affect the future reminders.  So a similar thought is that maybe that kind of behavior was somehow introducing a bug in the paycheck reminder.

And yeah, Quicken Mobile worked quite different than Quicken Cloud Sync.
Quicken Cloud Sync is much more powerful in what it can do like budget and other data, but it also means that a bug can potentially cause much more damage to the data file.

Pilotbum

So what you're saying in point #1 is my express web connect accounts are the same as cloud sync? Does that mean my banking credentials for those institutions are being stored in the Quicken Servers as well? 

I do not open/edit/use my Quicken data file from the NAS. The file is stored locally on my machine and syncs to the NAS as a backup and to sync to my laptop. The data file is only accessed locally on the computer actually running Quicken. 

Do you use the Cloud Sync? I liked the mobile apps for the most part and the web app and the functionality it provided but did not like my creds in the cloud and I didn't like that it would download and match transactions without my requesting it to do so. Like I said, I'd like to do that from my desktop app only. I am old fashioned in that I accept downloaded transactions manually. 

Chris_QPW

Pilotbum said:

So what you're saying in point #1 is my express web connect accounts are the same as cloud sync? Does that mean my banking credentials for those institutions are being stored in the Quicken Servers as well? 

Yes they are. Note that your credentials were always being stored on the Intuit servers for Express Web Connect.  Now with them changing to QCS for syncing they are on both Quicken servers and Intuit servers.

Old system:
Quicken -> Intuit servers -> financial institution (credentials stored on Intuit servers so that they can log in each night and download transactions)

New system:
Quicken -> QCS/Quicken servers -> Intuit servers -> financial institution

The way data is transferred between Quicken and QCS is "cloud sync".
What is visible to Quicken Web/Mobile is different when Express Web Connect is only used, but the syncing between Quicken and QCS is the same.

Pilotbum said:

Do you use the Cloud Sync? I liked the mobile apps for the most part and the web app and the functionality it provided but did not like my creds in the cloud and I didn't like that it would download and match transactions without my requesting it to do so. Like I said, I'd like to do that from my desktop app only. I am old fashioned in that I accept downloaded transactions manually. 

.No I don't, except for in a separate data file for testing purposes.

Chris_QPW

You might also want to check out this thread:
https://community.quicken.com/discussion/7882395/can-quicken-inc-please-update-the-article-on-how-quicken-connects-to-financial-institutions

Pilotbum

I just spent a few hours restoring a backup file that had no errors. Then after restoring doing another validation that resulted in no errors. Then I added all the missing transactions (20 days worth) and ran another validation that produced a **** ton of errors every time I validated. I don't understand. I'm starting to wonder if my desktop application is doing something to corrupt the file. 

Greg_the_Geek

What is the full path to the folder that contains your Quicken data file on the hard drive? Do you have Controlled Folder Access enabled in Windows Defender? Are you using OneDrive, Dropbox or Google Backup?

Pilotbum

The path is C:\Users\Randy\Quicken
Controlled Folder Access is disabled, always has been.
I am using none of those services for backup. I do use Synology Drive to backup to my Synology NAS MOST of the time, however, when I restored the clean data file without errors and added the 20 days of transactions the restored file was in a folder on the Window Desktop where there is no syncing/backing up taking place. Additionally worth noting, I use the NAS to store my backups. I only use the Drive client to sync the files when I'm preparing to travel and want to move a current copy of my data file to my laptop. Then I sync back when I return home. It is not constantly syncing. AND I have been home (i.e., not syncing the Quicken folder) since August. 

Greg_the_Geek

Have you checked your C: drive for errors?

Pilotbum

Yup. No errors. 

Chris_QPW

My guess would be there is something in that data file, and you are making the right choice to go with a new one.

Pilotbum

That's what I've been afraid of... Just hope this one can get me to the end of the year... 

Greg_the_Geek

Do you have another computer you can try? Have you tried adding transactions one at a time and then validating? Have you tried copying your data file from within Quicken and then adding transactions to the copied file?

Pilotbum

I do have another computer I can try. That's next. 

I've tried copying the file, but not copying the restored file... I'll give that a go. 

GeoffG

Pilotbum said:

splasher said:

Have you done a File->File Operations->Copy to see if it resolves your issues.

My data file goes back to 1996 and I will just about do anything to keep it in a single file.

If you do start over, you have to make sure that you keep opening the old data file to keep it up with the software changes otherwise it will become useless.

I have not tried that but wouldn't really expect any change if it's just a copy? Wouldn't it just be copying the corruption? 

I've not seen if you responded to splasher's initial comment. No this is not just a copy. It is a significant tool for rebuilding a damaged table structure. If you have not performed this, I would strongly encourage you to do so. It may not fix the issue, and if it does not, you will have to decide to stay with a damaged file, or act on your initial question.

Chris_QPW

GeoffG said:

Pilotbum said:

splasher said:

Have you done a File->File Operations->Copy to see if it resolves your issues.

My data file goes back to 1996 and I will just about do anything to keep it in a single file.

If you do start over, you have to make sure that you keep opening the old data file to keep it up with the software changes otherwise it will become useless.

I have not tried that but wouldn't really expect any change if it's just a copy? Wouldn't it just be copying the corruption? 

I've not seen if you responded to splasher's initial comment. No this is not just a copy. It is a significant tool for rebuilding a damaged table structure. If you have not performed this, I would strongly encourage you to do so. It may not fix the issue, and if it does not, you will have to decide to stay with a damaged file, or act on your initial question.

When you do a File -> File Operations -> Copy this isn't equivalent to doing a copy in Windows Explorer.  It copies the database record by record.  Not only does it not copy "deleted" records, the very fact that it has to read those records means that it is going through all the linked data and such and as such it can leave some kinds of errors behind.  This is why for bad problems both the copy and validate are suggested.

@Pilotbum I going to throw out one more thing, a full QIF to QIF recreating of the data file.  There is about zero chance that the "corruption" will be transferred in this method because of the completely different formats of the data.  But the main reason it isn't done is because of the difficulty and all the limitations (both in what can be converted and what is left behind).  For instance open security options will not transfer correctly the QIF format has no way to record the lots.  And the reports, reminders, budgets, ... are left behind.

But what I have found to be the hardest part is the transfers.  There is an option on the QIF import "special handling for transfers", but in my opinion they mess that up!.

Quicken does a terrible job of matching transfers during a QIF import.  With that option off both sides of the transfer are left in and Quicken has to do the matching correctly (and doesn't).  If you use that option what it does is remove the transfers from the "from account" during the import and with only one side of the transfer imported this puts in the transfer and it doesn't have to be matched on the other side.
The problem with this is they should have done it with the "to side account" not the "from account".  Think of a paycheck it has split lines going to various accounts, say a 401K account.  It is these split lines that will be removed.  It is the transfer in the 401K account that will be kept. So all transfers in a split will be broken up into multiple transactions.

So what I did when I tried recreate my data file because of the Online Bills function corrupting it beyond use was to instead go through the QIF file and change the transfers to standard categories.  I lose the ability to jump between the two accounts on that transfer transaction, but accounting wise this is just "double book accounting" where you have offsetting categories opposed to linked transfers.

It worked pretty well and here is the documentation on all of it and the pitfalls that I and others point out.  BTW that corruption problem was so widespread that Quicken Inc wanted to really work on it and as such I gave them my un-sanitized data file to work on to find and fix the problem. It took a days of their developers digging into and trying various technics to fix the data.  Even the CEO got involve in testing with my data file. In the end I did get my data file back in an uncorrupted state.

http://www.quicknperlwiz.com/changetransfers.html

It is a real shame that the QXF import only supports non investment accounts or that would be my suggested way to create a new data file.  On Quicken Mac where it is available that is exactly how they handle such problems.

Pilotbum

I wish I could get developers and the CEO involved with my problem! I have a hard enough time getting tech support that doesn't speak broken english. 

I'm about to toss in the towel on fixing this data file and just stick with the original plan of babying this corrupt file through to the end of the year and starting over. I've tried every combination and suggestion y'all have offered and errors are still showing up. I've restored, copied, rebuilt and still continue to have errors. 

I've made one attempt to the qif approach and see what you're saying about the transfers. I then tried a second using your transfer renaming tool and the accounts all seemed to be accurate, just without the transfers (as you said). I can see that is a permanent change since the two transactions in two different accounts are no longer connected with no way to reconnect them. 

IF I do get a new file with a qif import to work out for me, is it truly a new file or does the old data still cause problems in a new file the way quicken support has told me?