"Synced" status on Android not always correct (especially after item delete)

[Jayden_Navarro]

Steps to reproduce

Delete an item in the Android app. Status goes from “Saved” to “Synced”, then close the app immediately.

Expected result

Item is deleted in Dynalist backend, so viewing on another client (after refreshing/resyncing) shows the item removed.

Actual result

Occasionally (not always) the item is not deleted on the backend, and it appears on other clients even after full Dynalist site refresh. I have to reopen the Dynalist app on my phone (despite it having said it was synced before closing it before) and wait a few seconds then the delete is pushed through.

Environment

Android Pie.

Additional comments

It seems that the “Synced” status is not properly tied to “ALL pending updates (i.e. inserts/deletes/updates/etc.) are flushed/acked”. When I reopen the app it knows that there is an outstanding delete that needs to be synced (as a few seconds after reopening the app it does get pushed up), however the status does not reflect this state and prematurely reports “Synced”.

I use the “Synced” status as an indicator it’s safe to close the app and all of my updates have been fully synced (hence why I made this: Custom CSS to show sync status on top bar in mobile app), so the fact that this is not always the case severely affects the mobile experience as I am forced to just leave the app open for 10 seconds after an update just to be fully sure it has synced.

[samdark]

I think that’s the same issue as Edits don’t get synced if I close the app too quickly (bug)

[Jayden_Navarro]

This is a slightly different issue. The issue linked above is if you immediately close Dynalist on mobile, in which case you will only be in “Saved” state not yet “Synced”.

This issue is much more rare, and in my opinion is a bigger “bug” from a data integrity point of view as it says it’s “Synced” and yet it isn’t! It seems to be specifically related to deleting items as opposed to all types of edits.

[samdark]

Right. I think I’ve experienced both. This one happened only once or twice.

[Jayden_Navarro]

Just wanted to give this a bump as I still hit this issue frequently.

Another thing that does not seem to be taken into consideration when marking as “Synced” is collapsing items. I’ve had many times where I collapse an item and it says “Synced” on mobile but it isn’t actually persisted (e.g. when I open a new Dynalist tab in Chrome it isn’t collapsed).

It seems like for actual item edits (writes) the persistence mechanism is robust, however for deletes and “collapses” (and likely other view changes?) it does not appear to be using the same robust persistence mechanism (or at least not being tied into the “Synced” status). It would be great if all of these changes were considered as “writes” from a persistence perspective so that when we see “Synced” we are guaranteed that all changes we have made have been pushed to the servers.

[Shida]

Not entirely sure about deletes but I believe collapse isn’t taken into account by the sync state on purpose - most people don’t consider it part of important data, and it’s ok if it’s lost or unsynced in most cases.

As to why deletes aren’t showing unsynced, I’m not sure. In your experience does it only happen with deletes and not at all with create/edit/move actions? AFAIK delete is considered a “move” so it should go through the same code path as a moving action.

[Jayden_Navarro]

I’m fairly certain I have not seen this happen with creates/edits. I don’t perform moves frequently enough on mobile to be able to confidently say I have or have not seen it happen with move.

[Jayden_Navarro]

Out of curiosity, does the “Synced” status mean that the changes have been acked as received by the server, or just that they’ve been placed on the output queue/buffer and the OS has marked them as “sent”? I’m not familiar with web or mobile development, so not sure what kind of mechanisms are involved.

I’ve just seen what looks like a race condition (five minutes ago) where I added a new item, it displayed “Synced” then I immediately closed the app and the changes weren’t fully pushed. When I looked on the web it had the first few words of the item, but it was cut off and didn’t have the whole thing. Then when I reopened the app it still said “Synced” but a couple of seconds later it updated on the web to the whole item.

[Shida]

It should only say Synced when the app receives acks from the server that it received your changes… so if it says synced on your app and you don’t see some of your changes on the server that sounds like a bug to me.

If you make changes after a sync happens the app is supposed to revert back to unsynced/saved state.

[Shida]

Oh my, I think I spot what went wrong… The mobile “synced” display actually doesn’t use the same code path as the desktop “synced” status display which I was thinking of (because I wrote the code for the desktop one). Seems like there’s a few delays in the mobile version that makes it inaccurate in some cases. Will fix this asap and it’ll be released in the next version update along with the rest of the changes for this month!

[Jayden_Navarro]

Fantastic, thank you Shida!

[Jayden_Navarro]

Hi Shida. Even after the latest update, I am still experiencing syncing consistency issues on mobile.

Here are steps that can fairly reliably repro (1 in 3):

1. Use this custom CSS to display the sync status in the top bar, required to ensure you can quickly determine what the sync status is: Custom CSS to show sync status on top bar in mobile app
2. Create a new item, and slowly add characters to the item (roughly 1 character per second).
3. Wait until it shows “Synced”, then stop typing characters. Make sure that it continues to say “Synced” for at least a few seconds (showing that if it was going to move to “Save now” that would have happened already).
4. Open the list in a new tab on a computer. Verify that the last character(s) are missing from the item (if they are not missing, repeat from step 2 until there are missing characters).
5. Wait as much time as you need to feel confident that mobile should have synced up these last characters by now (10 seconds?). As it showed “Synced” it should NOT require more time to sync up, but this part is just to prove that the update is lost entirely, it’s not simply delayed and the “Synced” status displayed prematurely.
6. Refresh the web page and observe that the last characters are still missing.
7. Open the dot-menu in the top right of the mobile app and click “Refresh”.
8. Sync the web page, and verify that the characters are now properly added.

Here is a screen capture of the repro: https://streamable.com/22dec

It shows that the last character “j” is not properly synced.

Note that I showed it wasn’t synced on a mobile browser to visually illustrate the inconsistency in the screen recording, but you can more easily check for inconsistency using the desktop web app.

This repro shows that it is possible for updates to be lost, and the mobile app to improperly report as “Synced” even when it has content that is not pushed to the server. The content is not actually lost, as refreshing the app causes it to finally push up the change, but it still breaks the consistency model that “Synced” actually means all changes are present on the server.

This also is more serious than simply reporting “Synced” too early, as the behavior observed here indicates that new item content updates are missed entirely. As you can see in the video, it moves from “Saved” to “Synced” right as the new character is being added, and then never moves back to “Save now”, which is likely a race condition that leads to the lost update.

I believe this may be causing the “collapsed view” bug as well. Even though user data is a separate code path from actual content changes, it seems like a generic issue with the mobile syncing mechanism.

[Jayden_Navarro]

I was browsing the bugs page, and saw this comment you made in another post, which is really enlightening:

@Shida

Internally, dynalist uses a “diff” system to sync changes to the server. We store a local copy of your current Dynalist file, and a local copy of the “last synced” Dynalist file. When it’s time to send changes to the server, a “diff” is created by comparing these files and summarizing the difference.

When we compute the diff, an additional step of optimization is done to avoid scanning the whole file: we mark each item with a version number, and when they’re edited locally, the version number is bumped to indicate that it’s been changed. When computing a diff, any item with version numbers below the last synced number is ignored.

Given these details, I was curious whether the item version number was not incremented, or if it just didn’t realize we needed to perform a sync-up.

So I reproed the issue again, but rather than refreshing to force it to sync I modified a different item. This caused both items to sync-up immediately, so it seems that the verison number is correctly incremented, but the app just doesn’t realize there are new changes that need to be synced up.

This is different than the behavior I observed for collapsed items, which did not sync-up when a different item was modified. Therefore they are likely isolated issues (unless I’m missing something).

[Shida]

I’ve tried to repro this on my dev setup and it never happens. So I decided to do a code inspection instead according to your description.

What I found was a race condition that could happen when you precisely make a modification during the time a sync was happening, and nothing again (which explains the lack of repro on my dev since internet was instant).

Specifically, after a sync happens, the document’s dirty flag is cleared, even though it may have been set after the sync has started. I’m writing a fix for it now. It doesn’t seem urgent enough to require a deployment right away, so I’ll queue it up for deployment along the next release.

[Jayden_Navarro]

Great, thank you!

Now I’ll try to find a reliable repro for the collapsed state issue (as that seems completely unrelated to this)