Quicken Web Scraping for downloads [Edited]

solmssen

I'm really unhappy with Quicken. I'm an IT guy and I spend more time managing IT for Quicken that I do for some of my clients. This web scraping business has to end. You need to talk to financial institutions for data download in a reliable and consistent way, or the whole thing is a bust, it doesn't make my life easier at all. Get an API and OAUTH2 set up, or get out of the data download business, the way you're doing it now is just a promise you're not keeping.

Sherlock

To avoid the scraping, only use the Direct Connect or Web Connect connection methods.  If you haven't already, you may want to review: https://www.quicken.com/support/how-quicken-connects-your-bank

Chris_QPW

(Note through this I talk about Intuit, because in truth they are still the provider for this service, Quicken Inc is too small for this, and pays them).

This is a problem that Quicken Inc can solve.  In fact it is a problem no one can solve.
As long as there isn't any enforced standard on the financial institutions nothing will ever change.

In reality "web scraping" is probably a very small percentage of how the data is downloaded, but that doesn't really matter.

The OFX standard was created for this very reason.  It was pushed by Intuit, Microsoft, CheckFree, and "the financial institutions". At its highest only about 4,500 financial institutions adopted it out of more than 35,000 financial institutions in the US alone.  That number is now down to about 2,500.

And that is why "Express Web Connect" was created.  If understand that "Web Connect" is downloading a QFX file and importing it, and that the QFX file actually has the "response" to what would be the OFX request for the transactions, then the "Express" becomes clear.  As in log in for the user and download/import that QFX file.

But even this runs into a problem, not all the financial institutions will even generate a QFX file.
So Intuit expanded that to working with the financial institution to use an "agreed method" to download the data and convert it into this format for Quicken.  Again the financial institutions didn't just say let's all use this API, no they all have their own opinion on what they will provide.  And note it would be even more of crazy, but what happens in a lot of cases is the smaller financial institutions farm such work out to "providers" and as such Intuit can get the agreed upon method for more than one financial institution at a time.

But even more than the getting/converting of the data the biggest problem is just logging in.  The logging is for the most part "as the user" on the "financial institution's website".  These website login methods of course aren't designed for non interactive access.  And you can see the result as the security methods have change with multiple factor authentication and such gets pushed into this domain.

Can OAUTH2 help?  Certainly, but the first thing that is needed is for the financial institutions to agree to that.  And there is the "unsolvable" part of all of this.

And I might add that all these players think they know best, when in fact it certainly doesn't look like it.

Capitol One had some of the worse login problems and such, and to their credit they decided they wanted to work with Intuit for a better solution.  So they worked together and came up with Express Web Connect +.

So all their problems are solved right?
Nope one of the number one complaints on her has to do with Capitol One.

So much for them throwing together a better solution.

Note that Quicken Inc talk about the "promise" of this connection method, and how they hoped other financial institutions would adopt it.  So far I don't think any others have, and from what I can see that is a good thing.

But the story doesn't stop there.  Investment transactions use to be the domain of Direct Connect/OFX/QFX only.  And now Charles Schwab will be the first to use some kind of new "connection method".  My hope is that all this is a different way to connect (maybe even OAUTH2) and the underline protocol of OFX which has been proven to be a good protocol will still be used.  But in typical Quicken Inc/Intuit style they have refused to state any details on what this new connection method is.

But in reality I don't expect it to solve any problems with connecting for Quicken.  I think there is a possibility that it might be a more secure connection method than the security model that OFX is using, but I don't expect it so solve connection problems for two simple reasons.  One is that it will not be adopted by most of the financial institutions.  I will note that I still see OFX records with 1.0 version, even though the current version is 2.3. That shows you how eager the financial institutions are to keep up with such things.

And the second reason is that given the record of what happened with the Express Web Connect + that Capitol One brought to the table (which is no better, or even worse than before) I don't trust them to implement a good solution.  OAUTH2 was around when they created Express Web Connect + and they ignored it for some reason.

solmssen

I understand the difference, but I don't want to be in the download business either - Web Connect means I need to download a file every time I want to update, and I have to do that for my Apple Card. Citibank has me using Express Web Connect for checking and savings and Direct Connect for investments, Amex it's not even clear what it's doing. It's all a giant mess, it's much too hard to keep running, and I just wish it all worked better. The point of aggregating all this info is to make my life easier, and it's not.

Chris_QPW

solmssen said:

The point of aggregating all this info is to make my life easier, and it's not.

And unless the financial institutions get some kind standard this is never going to change.
A few years ago the EU mandated that their financial institutions support one or both of two such APIs (one being OFX), the US and Canada are "free countries" so we have to put up with chaos.

Quicken isn't alone in this.  I have tried using aggregation with Mint, my financial institutions, and such.  And there isn't one that is trouble free.  They all suffer from the same lack of a standard.

And it isn't going be puny Quicken Inc that is going to change this.

TTSguy

 I use web connect and have eliminated all the old problems I had with Express Web Connect.

Chris_QPW

I use Direct Connect on all my accounts and I have had very little problems over the years with it.
It doesn't solve all the problems, but it is far an way better than Express Web Connect.

And hope that I'm clear in what I was saying.  I completely agree with @solmssen that a standardized API/protocol is much better than the "ad hock" agreements that is Express Web Connect/aggregator type approach.  After all it already exists in the OFX standard (Direct Connect) and is already proven itself.

But the fundamental problem is that no one can dictate to the financial institutions to force them to use it or anything like it.

And given that one has to choose for themselves the way they are going to deal with the problem.  Asking Quicken Inc to fix the problem isn't going to get there.

If there was such a solution it would have been done a long time ago.  It isn't like Quicken Inc (or Intuit before them) loves all the complaints about the system not working.

And with trying to fix it with Capital One/Express Web Connect + and they seem to be working on another connection method (which I believe is more about security than it is about reliability) they clearly care about this problem.  They don't need to be told that the problem exists.  They deal with it every day.

But I see no way that they can provide a solution that will work for everyone using the 35,000+ financial institutions, because of the simple fact that even if they had the perfect way of doing it, they don't have "pull" to make the financial institutions use it.

solmssen

@Chris_QPW your comments are enlightening and frustrating at the same time - helping me to recognize the magnitude of the issue and making me mad that a standard hasn't taken root. It's unfortunate, and it's holding back progress.