Malfunctions, especially in trying to reconcile

nutward

When trying to reconcile my bank account I discovered:
1. Quicken had un-reconciled all transaction back to 8/14/19 (my transactions had been reconciled up through 11/7/19)
2. Quicken did not include any deposits in the deposit column (only charges); in fact no deposits were included in either column of the reconciliation page.

Another less critical matter: I download transactions yesterday. I closed the program, leaving about 15-20 transactions which had not been reviewed. Today I booted Quicken and found that all of my blue review dots had been removed.

I called Quicken support a while ago today (4/8/2020) and was told to revert to an earlier backup file until I found one that worked correctly and then use that one. Obviously, that will entail manually recreating all of my entries from that point forward. That will involve a tremendous amount of work, if it's possible at all.

Thanks for any help anyone can give me.

Snowman

Issue #1 is that you should create a separate backup after EACH time you use Quicken and you should use the option that adds the current date to the file name.  I backup everyday and keep one years worth of backups.  These backups are backed up to another drive and finally to another drive that is kept offsite.  That has rescued my bacon from the exact situation that you are now in.

Issue #2 you need to validate your data file on a regular basis.  This will tell you that you file is not corrupted or if it is what corrective steps to take.  Keeping all of the backups that I do is not enough.  At least once a week you need to validate the data file.  If you do not do this you may just be backing up corrupted data to a backup file that itself will be corrupted.

Going back to a working file will cause to you have to recreate many entries but it is not the end of the world.  By correcting the 2 issues above you will hopefully never have to face this situation again.

Snowman

Issue #1 is that you should create a separate backup after EACH time you use Quicken and you should use the option that adds the current date to the file name.  I backup everyday and keep one years worth of backups.  These backups are backed up to another drive and finally to another drive that is kept offsite.  That has rescued my bacon from the exact situation that you are now in.

Issue #2 you need to validate your data file on a regular basis.  This will tell you that you file is not corrupted or if it is what corrective steps to take.  Keeping all of the backups that I do is not enough.  At least once a week you need to validate the data file.  If you do not do this you may just be backing up corrupted data to a backup file that itself will be corrupted.

Going back to a working file will cause to you have to recreate many entries but it is not the end of the world.  By correcting the 2 issues above you will hopefully never have to face this situation again.

nutward

Thanks for the help, Snowman. I'm paranoid about backups, so I do the things you do, plus some others. I don't lack for backups. The problem is I don't know exactly when things went wrong, so I don't know how far back to go. My issue became apparent when I tried to reconcile my bank account, which I hadn't done in several months. I suppose the lesson is: reconcile more frequently.

Your second suggestion is the one I really needed. I do not validate on a regular basis. During the many years I've used Quicken, I haven't validated more than 3 or 4 times. In fact, I had to check help on how to do it. So I'll get to it.....

Again thanks for the help -- and for the promptness as well.

Boatnmaniac

@nutward You might want to consider letting Quicken reconcile your accounts to your online balance after each update.  If there are any reconciliation issues, they will be identified immediately so you can address them right away.

As an added bonus from my perspective:  Once I got used to this and found that this auto reconciliation process is really reliable, I know longer felt a need to reconcile to a paper statement and hardly even bother ever looking at paper statements anymore.  That's what tools are supposed to do...make your life easier, not harder.

Chris_QPW

Just my general notes on this thread.

On validate. Quicken Inc doesn't suggest doing it frequently, and I don't either.  Validate can be "destructive", this is why it automatically makes a backup of your data file each time you run it.  Its tries to make the data "consistent", depending on what it finds wrong that can mean removing data, which may or may not be visible to the user.  Validate has always been suggested as "if you have problems" kind of thing.

On reconciling I think this come under the same heading as the old "How often should I save my document?"  Answer "That depends on how much you want to recreate if something goes wrong."

Like @Boatnmaniac I currently use and like reconciling to online balances, but it isn't 100% without problems.  I like to reconcile right after I download, but some financial institutions don't always do this right.  For instance sometimes sending new transactions without updating the online balance.  For some I have to wait until the next day to do the reconcile.

On automatic reconcile, I have this turned off, but this probably doesn't apply to people in this thread.  I also use automatic transaction entry.  The two modes do not work well together.  In automatic transaction entry mode you look each "New" or "New Match" transaction in the register looking for the correct category and such and then mark it Reviewed.  This is basically the same as Accept in non automatic entry mode.

The problem with using automatic reconcile with this is that as soon as those transactions are downloaded into the register it kicks off and even though it fine for the amounts, it doesn't care anything about the categories.  So you can reconcile just fine, but have messed up categories.  And reconciling clear the "New"/"New Match" status so you can't check them after the fact.

BTW a person recently ran into a rather strange problem with automatic reconcile.  They had it on, and then let their subscription expire.  Well Quicken hides the Online Services tab in that case.  And the only place you can turn off automatic reconcile/online reconcile is on that tab.

nutward

Thanks Boatnmaniac. That sounds like a great idea, but I don't know how to do it. I've used Quicken for a long time, but I still don't know much more about it than the basics. As with many old duffers, pencil and paper are more my bailiwick than bits and bytes. Can you point me to some instructions on reconciling accounts to an online balance? Would it be in Q's help file? I would be grateful for some enlightenment on this.

And again, thank you.

Boatnmaniac

@nutward - You're welcome.  Yes, Quicken Help will walk you through it.  Pull up Help and do a search for "auto reconcile" and then click on the link that says "How do I turn on Auto Reconcile?".

As mentioned by @Chris_QPW , the main issue with auto reconcile I run into (and only with one checking account and one credit card out of more than a dozen such accounts in 4 different Quicken files) is that sometimes the FI will download a cleared transaction but the online balance has not been updated.  If it happens, you can cancel the auto reconciliation while it's in process and then run OSU again the next day and everything reconciles well then.

I also don't like auto entry of downloaded transactions.  As @Chris_QPW said, this doesn't give you the opportunity to enter/modify categories before they get accepted into the register.  It also might not correctly match with any bills/income or manual transactions entered into the register.  So, I manual accept all downloaded transactions to make sure they are correctly categorized and matched.  Then when that is done I click on "Done" which kicks off the auto reconcile process.

Boatnmaniac

Chris_QPW said:

BTW a person recently ran into a rather strange problem with automatic reconcile.  They had it on, and then let their subscription expire.  Well Quicken hides the Online Services tab in that case.  And the only place you can turn off automatic reconcile/online reconcile is on that tab.

Morale to the story, then, is to not let your subscription expire, huh?  :-:smile:

nutward

Thanks to Snowman, Boatnmaniac, and Chris_QPW. With all the info you have given me, surely I can solve my problem. If not, I should probably return to pencil and paper.

So it's comes down to this:
To renew or not to renew, that is the question:
Whether 'tis nobler in the mind to suffer
The slings and arrows of Quicken reconcile,
Or to take Arms against a Sea of incorrect entries?
(with apologies to Hamlet)

Chris_QPW

I thought I would add some on getting this setup since frankly it is a bit confusing even if you look at the help (or at least what is needed to back out of these choices).

You start at the Reconcile dialog (Ctrl+R):

The option I highlighted turns this dialog off.  As in you select it, and then select OK, and then Ctrl+R it will go directly to the reconcile dialog, skipping this dialog. Needless to say once you have done that, you can't use this dialog to reverse your choice.

This also where you select the Auto reconcile, well that implies reconciling to the online balance (it can't work any other way).  And that in turn implies/forces the Always use Online Balance.... to be on, including "Don't ask again".

So how do you get this dialog back?

Settings/gear Icon -> Edit Account Details -> Online Services tab

Snowman

@ Chris_QPW.  "On validate. Quicken Inc doesn't suggest doing it frequently, and I don't either.  Validate can be "destructive", this is why it automatically makes a backup of your data file each time you run it. "

i respectfully have to disagree with the above statement.  So what if there is "harm" if a backup is made?  The greater "harm" is when you are going along and a "problem" does not show for many months after it first happens.  This happened to me a number of years ago and I found out I had 5 months worth of corrupt backups.  My main checking account had been corrupted.  I had to go back 5 months, restore from that backup, download and manually input all of the transactions for that account including transfers between accounts.  It took over 65 hours to do. 

That is when I started to validate at LEAST once a week if not more.  So if you still say "do not do regular file validations" then how do you avoid the above situation?  Some "problems" are easy to see, others like the one I had do not.

Chris_QPW

First off let me say that either action is mostly an "opinion".  So I'm certainly not saying that one way or the other is "100% correct".  And the reason it is just an opinion is because there is no telling if the file Validate produces is always equal or better than the input data file.

Next the only reason I mentioned the backup is to show that the Quicken developers felt that the operation is risky.  If the operation was like entering a transaction, there would be no backup.  Validate and especially super Validate have the potential of rendering your data file useless.  In fact there have been times where Validate has corrupted everyone's data file that used it because of a bug in a given patch release.  Whenever you are going through a database and checking it for consistency and trying to repair inconsistencies and other kinds of problems you risk damaging it even more.  And it can be damaged in a "hidden" way.  Validate isn't magic, it is just more code.  Data corruption is of course something that is talked about a lot, and sometimes used by the support people to "give up" and tell people to start over.  But in my estimate 90% or more of the "corruption" is created by the Quicken software, not the user or the user's hardware.

Luckily most of the time when a user has "data corruption" the data file is still usable, especially after running Validate.  It is quite rare that a data file becomes totally unusable. in these cases Validate can't fix the problem, and there isn't any indication that Validate would have fixed the problem "in the past".  I suppose it is possible that a number of problems might accumulate in a data file and then get to a point where it can't be repaired, but I suspect that if that happens at all, it is extremely rare.  More likely the data file is damaged in a way that Validate can't fix no matter when it is applied.

If I was going to Validate on a regular basis I would not use the resulting file unless I saw it found problems and if it did I would be very sure to look over my data file as much as possible to be sure it is still OK to use.

Chris_QPW

Actually let me summarize the above.

The decision to Validate on a regular basis or not depends on what part of Quicken code you want to be paranoid about. 

GeoffG

Chris_QPW said:

Validate and especially super Validate have the potential of rendering your data file useless.  In fact there have been times where Validate has corrupted everyone's data file that used it because of a bug in a given patch release. 

Isn't it better to regularly perform a validate and discover a data corruption in a recently validated file, than run validate only as prescribed and find the last six months or more of backups are hopelessly corrupt?

After twice experiencing bug induced corruption I use File Copy and validate weekly.

Chris_QPW

GeoffG said:

Chris_QPW said:

Validate and especially super Validate have the potential of rendering your data file useless.  In fact there have been times where Validate has corrupted everyone's data file that used it because of a bug in a given patch release. 

Isn't it better to regularly perform a validate and discover a data corruption in a recently validated file, than run validate only as prescribed and find the last six months or more of backups are hopelessly corrupt?

After twice experiencing bug induced corruption I use File Copy and validate weekly.

Let me summarize my opinion, I don't trust Validate any more, and maybe less than any other code in Quicken.  So if I was going to use it on a regular basis I would using it to try to detect that data corruption has crept in and most likely if it found a problem I would just as likely go to a backup before the problem than use the resulting validated file.

One thing you see all the time suggested on here is described this way:

"I have xxxx problem"

Try Validate

OK that fixed the problem.

End of conversation.

To me the conversion should be like this:

"I have xxxx problem"

Try restoring from a backup before the problem.

OK that fixed it.  OR No I still have the problem.

OK as a last resort try Validate.

OK that fixed it.  OR no that didn't fix it, sorry then you have to somehow rebuild your data file from scratch (maybe QIF export/import)

GeoffG

Chris_QPW said:

Let me summarize my opinion, I don't trust Validate any more, and maybe less than any other code in Quicken.  So if I was going to use it on a regular basis I would using it to try to detect that data corruption has crept in and most likely if it found a problem I would just as likely go to a backup before the problem than use the resulting validated file.

Fair enough. That is why Snowman and I perform regular validations, not to fix, but to alert of an issue that requires a restore.

nutward

Wow. You guys are PHDs and I'm just a kindergartner. And welcome, GeoffG to the conversation. So let me see if my basic take-away captures the essence of what has been said: 1) if corrupt data is obvious to me restore a backup from a time when I know my data was correct, 2) use validate as a last resort, or 3) use validate regularly, say, once a week, not to fix problems, but to find them, 4) restore from a good back.

GeoffG

1) Yes - the challenge is that corrupt data is not always obvious and can remain dormant

2) Depends on who you ask

3) Depends on who you ask

4) Yes - which requires a validate to confirm

Chris_QPW

I think I might state is these a bit different even though overall it is correct.

1) Prefer restoring from a backup to fix problems if at all possible over using Validate to fix them.

2) Using validate to check for problems on a regular basis, but most people are actually going to be "using" the validated file if not told otherwise.  As in if you run validate, the file you end up with it the validated one, not your original file. You have to take extra steps to get back to your original file.  This seems like the safest procedure to me.

But let me say that in fact I believe all these problem are extremely rare and as such it would be very hard to judge if "do nothing",  "validate as a check only", or "validate and use" would have any real difference "percentage wise".

What I think more determines that "rate of corruption" has to do with the features the user tend to use.  For instance I rate the use of "cloud sync" as "above average risk".

Also the one time I almost lost use of my data was when I was using the Bill Presentment.  In fact I had rebuilt my whole data file because it had corrupted my data file beyond use (there was a bug introduced in it).  The only reason I have that data file now is because  I submitted it unchanged to Quicken Inc to be used by developers and the whoever needed to including the CEO as a way to find the root cause of that problem.  They did multiple operations and over several days to recover my data.

Sherlock

Since you're all providing opinions regarding reconciliation and validation, I thought I'd share ours:

Reconciliation: We reconcile regularly to statements obtained from the financial institution as the online balance is not reliable.

Validation:  We very rarely validate to minimize the risk of a bad repair.

nutward

Hello Sherlock, thank you for your comments. They bring us back to where I started this discussion. I have always reconciled to a statement sent out by my bank (either paper or digital). However, my most recent attempt to do so, went horribly awry, as explained in my opening question. I'm happy to continue the practice; I'm accustomed to it, and since I'm retired, time is rarely an issue. However, for the moment, I've got to do something about the data corruption that surfaced at my last attempt.

From all that's been said, it seems I should start by restoring a previous backup when things worked correctly, and then go from there. So, I'm going to give it a shot. After I get things back to where they should be, I'll re-read all of the helpful info you guys have given me and decide whether I want to try using my online balance, auto reconcile, etc.

Thank each of you for your input.

Snowman

Yes I agree I do not use any of the cloud "features" nor any of the "bill presentment features" because I feel they are very high risk.  I should have mentioned that.

The only problem with restoring a backup first is how do you know that the backup is not corrupted?  Make a backup first, then validate which is what I do once a week.

From a database standpoint ( I have written many relational databases in my time) the data is only as good as the structure and integrity of the database.  I also wrote them that before backing up it checked the integrity of the database first, then when it was OK, create the backup.

This is what Quickbooks does.  I know that Quickbooks and Quicken are completely different animals but the logic flow is the same.  Check the integrity of the data BEFORE doing the backup.