One Step Update
Thought — how to have "One Step Update" window show more info or detail when performing an update? Right now I have ~25 updates (some accounts use same update etc). When running an update, some take longer than others to complete and there is really no granularity on the stage or status of the update. Some connections seem to hang up but then you don't really get an indication of which is the problem child unitl it returns an error (if it does return an error).
Just curious if there was items running in the back ground that could be converted to some type of progress bar or other indication of connections or progress that would give more insight? Or am I the only one?
Comments
-
I think the answer might be "maybe", but it isn't really all that useful of information.
For Direct Connect you can look at the OFX log. For Express Web Connect and Express Web Connect + you look at the cloud sync log file.
So, comparing what it has in those logs let's compare that to the status dialog shortly after I have started it:
Citi Cards is Direct Connect, and only takes a few seconds to complete. What one will find in this log is a "SEND" message requesting the transactions, and then the "RECV" message with the results. Not for whatever reason they can only do one login at a time, and as such if you have more than one Direct Connect login you can expect them to one at a time complete. But if for whatever reason it is taking lots of time all Quicken knows it that it is waiting until either it gets a result or times out.
If I look at Cloud Sync log, it is similar. There are more "setup" messages being sent, but it gets to a point where it is just waiting. And here you have to understand that it does these all at the same time. The "Waiting" in this case isn't like if you had multiple Direct Connect accounts where each would have to wait its turn to send/receive messages. It will be sending to all at once and will spend most of the time waiting for a response.
So, yes for the "maybe" they might be more dynamic in putting up the "Complete" messages for the ones that have completed first, but the reality is that they can't exit until they all finish one way or another. And as soon as the last one finishes, they can put up the final results error messages/codes and all.
The bottom line is that Quicken is basically kept in the dark and has no idea about "progress" other than "sent", "waiting", "done".
Quicken is talking to the Quicken Connection Services (Sync to Cloud), it isn't even talking to Intuit, let alone the financial institution.
Signature:
This is my website: http://www.quicknperlwiz.com/0 -
I do think that Quicken could make significant improvements in the feedback and responsiveness of One Step Update. If we only consider EWC/EWC+ connections (since that it the method all connections are being migrated to), my inspection of the Cloud Sync Log seems to show the following:
- The sync process occurs serially, one Financial Institution (FI) at a time — they are not done in parallel. The Quicken desktop app sends a request to sync a single institution and then polls the Quicken servers until that is sync task is completed. It then moves to the next FI and repeats until all are done. This begs the question of why they don't submit the sync requests in parallel as it could greatly reduce overall time for OSU (My assumption is that they're trying to throttle the load on their servers, although I would argue that approach is misguided).
- Given that the desktop app polls the servers fairly frequently (every 3 seconds), the servers could potentially provide more complete feedback as to progress than simply "isProcessing: true". This is definitely more complex from an implementation point of view on their servers, but progress info could be made to propagate to the client. My guess is that investing in those changes isn't worthwhile to them if, in a typical use case, they expect the sync of an individual FI to complete quickly enough that interim status info would be fleeting. Also, if they use a third-party aggregator for any of these integrations, that aggregator may not provide feedback that could be passed on.
- My experience is that the OSU status screen doesn't show a "complete" indicator for any EWC/EWC+ FI until all FIs are completed. Again, since each FI is processed individually, the client does know which FIs are completed and which are not. (This would be true even if they processed the FIs in parallel.) Why the UI isn't updated to show this info is odd as it seems like that would be a fairly easy fix.
One of my biggest questions/comments for Quicken is that they share what third party/parties are involved in the aggregation process beyond the FIs. For example, many services use Plaid or Yodlee. If they are using a third party, it would be good to know that so that, when OSU issues occur, we users could understand that perhaps this issue is outside of Quicken's control (even if we expect Quicken to press for a timely resolution).
0 - The sync process occurs serially, one Financial Institution (FI) at a time — they are not done in parallel. The Quicken desktop app sends a request to sync a single institution and then polls the Quicken servers until that is sync task is completed. It then moves to the next FI and repeats until all are done. This begs the question of why they don't submit the sync requests in parallel as it could greatly reduce overall time for OSU (My assumption is that they're trying to throttle the load on their servers, although I would argue that approach is misguided).
-
@tivolo I think you need to look at those logs a little closer. What you were seeing is basically statements that say I need the information for this financial institution. I need information for this one. I need this one this one this one and on and then there is basically a go to it. After that Quicken is just asking every few seconds. Are you done? Once it comes back, you’ll see it basically immediately finishes it. While it’s waiting and asking every few seconds, are you done, it’s getting no responses from the server. When the server is finished, you get this one done. And then the whole process is ended. At that point is we are Quicken starts actually processing the return data.
edit: note that the request at the beginning, take just a few seconds. But the waiting is minutes before any response from the server.Signature:
This is my website: http://www.quicknperlwiz.com/0 -
@Chris_QPW My understanding of the process from the logs is the same as yours: the first request from the Quicken app to the server for each FI kicks off a server-side "job" which will perform the aggregation process. Because that processing is happening asynchronously (with respect to the Quicken app), the app polls the server for the status of that job every 3 seconds until it completes. Currently, when the server responds to a polling request, it appears that the only two statuses it sends to the application are a generic "AGGREGATION_IN_PROGRESS" (i.e. "not done yet") and "OK" (i.e. "completed"). When Quicken sees the "OK" response, it knows to stop polling as that aggregation for that FI is complete.
My suggestion is that the polling response could return finer-grained status information to the application, e.g. "QUEUED", "AUTHENTICATING", "REQUESTING TRANSACTIONS", "WAITING FOR FI RESPONSE", … Of course, since the client is polling (as opposed to receiving "push" notifications from the server), it will only see snapshots of the server-side process at 3 second intervals. That means it may not see all (or any) of these statuses.
For example, if the entire FI processing occurs in less than 3 seconds, the app will receive the "OK" message in its first polling attempt. So, on the UI, it would go straight from "Updating…" to "Complete" after 3 seconds. However, if the aggregation takes longer than 3 seconds, Quicken will poll multiple times and could report back a better indicator of progress than it does currently.
This is particularly important in cases where things are really slow, as has been the case in these past few days. For me, aggregation is currently taking 2+ minutes for each FI. Having more detailed process info might be useful and, at a minimum, could at least show me that progress is being made (albeit slowly). For example, if I saw the job was stuck in the "QUEUED" stage for a long time but then completed quickly once started, I would deduce that Quicken's servers are likely overloaded. On the other hand, if it's hung in "WAITING FOR FI RESPONSE", then the problem is likely on the FI's side.
0 -
I certainly can't be sure, but I don't think even the Quicken servers have the information you seek.
In other words, I highly doubt that Quicken has ever hung up the first "level" of communications. It is either so fast that it isn't worth reporting on the status dialog, or you will get "Can't connect to Quicken Server" pretty much immediately.
Back when Express Web Connect didn't go through the Quicken Connection Services (QCS)/Quicken Cloud dataset the "process" was much more segmented and you could see they different statuses appear. One of the first thing I noticed when they switched to using QCS is different people reported seeing different statuses flash by. As in they had the old statuses in there, but they basically became meaningless.
If you have ever done a Sync to Mobile/Web you will notice that except for the very first time, it is actually quite a fast operation.
So, that would mean that it probably isn't where that two minute is coming from:
The rest of this is the connection between QCS and Intuit, and Intuit and the financial institution.
We have no insight into that because the log only contains information between Quicken and QCS.
Could they possibly have more information between QCS and Intuit that they could pass a long? Maybe, but I'm guessing they aren't about to change that protocol just to make the statuses nicer.
Signature:
This is my website: http://www.quicknperlwiz.com/0 -
You certainly could be right that Quicken simply doesn't have the info needed to provide more detailed status — it very much depends on the capabilities of whatever system/provider is doing the aggregation. That's the part that's completely opaque (to me, at least).
You seem to indicate that Intuit is still in the mix, essentially acting as the aggregation service provider for Quicken. I wasn't aware of this and wasn't sure if this is speculation on your part or if you've seen Quicken confirm this. When Quicken forked off from Intuit, I wasn't sure how the IP around aggregation was split. Certainly it makes sense that Intuit still would need these services as well, but I'm not sure if/how the partnership between the two companies works.
0 -
The part that Intuit is Quicken Inc's aggregator isn't speculation, that they have confirmed many times. And yes, Intuit still needs those services for QuickBooks, and of course it was the easiest way to spin off Quicken. Quicken Inc doesn't have the personal to be their own aggregator, and changing to a new one would be a really big job. Not something you do when you have just been sold.
Personally, I don't think Intuit is a very good aggregator. What's more I know that the Quicken group always had problems with the "connection group" responding to their requests. One employee expressed the opinion that they would be better once Quicken Inc was a customer instead of just another group in Intuit. But I doubt that. And I haven't seen anything that suggests that they have got more responsive.
And yeah, most of what I "know" is either from doing tests/looking at logs and such because Quicken Inc follows Intuit in the opinion that they shouldn't tell their customers how their systems work. And of course, speculation based on what I see and what I might have done in the same situation.
I did do quite a bit of testing though on this and I do know that it is in "parallel" for Express Web Connect and Express Web Connect + for the simple reason that if I did them individually (I have mine and my wife's accounts and banking/credit cards/investment accounts with Chase plus two Express Web Connect Macy's and Syncronoy) they will take much longer than if I do them together. It still baffles me that it takes so long, and so predictable. When Express Web Connect went directly to the Intuit server, how fast it would return really depended on the financial institutions. Some being very fast, and some being very slow. But with this new system, it seems like everyone is getting this same 2 to 3 minute "performance".
It is entirely possible that the Quicken Server has more information but is just not setup to provide it to the "client".
The way I see this has evolved is that they had Sync to Mobile/Web to the Quicken Connection Services/Quicken Cloud dataset (which also is the "database" for the Mobile/Web apps). That alone needed to have that server kick off a request to the Intuit servers periodically to keep the transaction data up to date even if the user was away from their Desktop computer. And you can force and update too in the Web app (didn't see the same in the Mobile app, had to restart).
So, that it isn't really surprising when they wanted to get in the middle of this to track what is up and down, that they would just basically drop the Express Web Connect interface that went directly to Intuit in favor of Sync to Mobile/Web with a request to it to get new transactions. Which is basically what the update on the Web app would be doing.
BTW I think it is very telling that Quicken (the program) is polling every few seconds. Polling is the most primitive way to do this. It implies that there isn't a good "call back" system for the Quicken Connection Services.
Signature:
This is my website: http://www.quicknperlwiz.com/0 -
The funny thing is that Quicken used to do this. You would see each financial institution go from waiting to sending to receiving to complete. Now they all say waiting and then they are all magically done.
0 -
That is because the whole process changed.
One thing I also don't have to speculate on is the flow for Express Web Connect (and now Express Web Connect +) isn't like it was about two years ago (maybe more, my sense of time doesn't work in "years" 🤣.)
Before the flow was:
Quicken (the program) ←> Intuit ←> financial institution
Now it is:
Quicken (the program) ←> Quicken Connection Services {sync} ←> Intuit ←> financial institution
So, the actual communications for fetching the transactions from the financial institution is one more server away from Quicken (the program).
Before Express Web Connect was logged in the OFX logs, and now it is in the Cloud Sync Log.
Signature:
This is my website: http://www.quicknperlwiz.com/0
