Collapsing item on mobile not getting synced to web

🐛Bugs
#11

As it still was inconsistent after uncollapsing/recollapsing repeatedly on the mobile side, I don’t think this could be the issue.

Is this server side pushed to each syncing client, or on the mobile client it sends out a list of items where the collapsed states have changed which then are replayed by the server to each syncing client?

If it’s server side, how does it determine which items it should send to the web client, does it internally track the synced node version ID for each connected session to identify which nodes differ and which updates to send? If so, if the client was offline for a while could this client session be cleared from the server side, leading to inconsistency?

If these updates are pushed from the mobile side, is it possible if I closed the app too quickly they weren’t sent out? Are these updates the only way that the server detects collapse state? If so then this argument doesn’t hold water as new clients showed the proper collapse state.

#12

On the server side, we store the latest collapsed/expanded state and a version number that it was changed in. Each client holds its own “last synced version” which is presented to the server on sync. The server selects all collapse/expand states from the document that has version number strictly above the one presented and sends the list back to the client, as well as the current version number.

#13

This “last synced version” is stored per item/node right? And this per-node version is sent to the server for all nodes in the document on every sync? Or are the version numbers of only a subset of nodes sent?

Also, is this version number specifically tied to collapse state changes, or is it a more general version number for the node?

#14

No, only one “last synced version” is stored on the client per document. It’s only used for syncing collapsed changes, and it is different than the versioning/syncing structures used for syncing document content.

The last synced version for collapse state is sent to the server, and the server returns all nodes that have collapsed version numbers higher than the client’s last version, which is the subset of all nodes that had their collapsed state changed since last time the client had synced with the server.

#15

@Shida I still experience this frequently and I’m really curious what’s causing it. Based on the protocol described I can think of a few possible scenarios:

  1. There’s some race condition, like you described above, where the collapse state is applied before the item’s children are synced down. In this case, is there a console error message when a collapse is applied to an empty item, or is it silently ignored? You noted that this shouldn’t happen, but maybe there’s some small edge case here related to Chrome/Windows backgrounding a tab (I’ve only seen this bug when resuming my PC after being asleep for a while).

  2. The client side last synced version is somehow bumped without getting the collapse state changes.

  3. There’s some client side bug that has to do with it being the last item in the list combined with some race condition causing its collapse state to be out of sync.

The first two scenarios don’t line up with the behavior I observed where collapsing/recollapsing the “problem item” doesn’t sync to the broken client, as those should be properly reflected on update even if we missed the initial update.

I’m going to leave the network tab open for a few days and hopefully this repros, then I’ll be able to see if there are collapse_changes in the update bundle_binary which should determine whether it’s a server side or client side issue.

1 Like
#16

@shida I finally have a repeatable repro! It’s a fun little edge case that I backed into from thinking about the protocol (thanks for going into the details there, it wouldn’t have been possible without that!).

Have Dynalist open in two tabs, A and B. Open dev console in tab A.

  1. Create item “Foo” in A with child item “Bar” and collapse “Foo”. Make sure it syncs to B.
  2. In the “network” tab of dev console on A change from “Online” to “Offline”.
  3. In A, zoom into the “Foo” item and delete “Bar”. The item is now no longer collapsed in A since it has no children.
  4. In B, zoom into the “Foo” item and add a new item “Baz”. It should remain collapsed and now have two children.
  5. Change A back to “Online” and sync changes (e.g. Ctrl-s).

Tab A should now be in an inconsistent state where it’s uncollapsed with child “Baz”, as opposed to B and and other clients which will have “Baz” and be collapsed. No matter how many times you sync or perform new changes tab A will remain in this inconsistent state.

The root cause is that an offline tab can move into an uncollapsed state while other clients (and thus the server) never become uncollapsed. This means there are no collapse_changes to sync down to the uncollapsed client since from the perspective of the server the item never became uncollapsed (all other clients always had at least one child).

When I’ve experienced this problem I don’t think it was exactly like this repro, but it likely has the same root cause, so if the fix is fundamental enough it should hopefully resolve it (as opposed to just patching this one edge case). I still need to figure out what the exact case is when I’ve hit this in the past…

One robust solution is to always provide syncing clients the collapse state of new items’ parents, which should hopefully resolve this issue.

EDIT: Thinking about this a bit more, it’s likely that when I’ve experienced this I’ve quickly deleted the last item in a collapsed child and then put my computer to sleep before the collapsed change is synced, but after the item deletion synced (was able to repro locally that that can occur). I think even switching tabs is sometimes sufficient to prevent the collapsed state syncing from occurring.

#17

Wow nice debugging. I think that explains the problem well and is probably what’s happening here. I’ll see how we can get this fixed!

1 Like
#18

@shida I just had this happen again and I’m positive that the collapse event (on mobile) happened after the diff version stored by the web client, so it’s a different issue than the one reproed above (that earlier one only applies if from the server side it doesn’t think there were any collapse events).

I have two theories:

  1. The client collapse event is somehow not bumping the collapse version number. Maybe from the client’s perspective the item collapse state hasn’t changed since the last sync (the item deletion and addition happened from other clients and the API, so before syncing down it’s “collapsed”), so when I collapse the item it doesn’t increment the collapse document version even though it does sync the collapse state back to the server (verified by seeing the item collapsed in a new client). Is this possible? When it decides to increment the version number does it diff to the last sync state and only increment if they differ, or is it a simple “did any collapse events happen”? If it’s the former, this could be the issue. More generally, is incrementing the version done client side or server side? I think this only makes sense if it’s done client side.

  2. There’s a race on stale clients update. Is something like the following possible?
    a. Client sees update that removes item, making it uncollapsed.
    b. Client thinks it’s now up to date since it processed an update and pulls down collapse diffs which includes the collapse. Since item has no children in the client it’s a no-op. The document version number on the client is updated, so this collapse diff is essentially lost.
    c. Client sees update that adds item. Since the collapse state was already applied (and ignored) the item is now improperly uncollapsed.

Before the client pulls down the collapse diffs, does it ensure it’s fully synced to the latest version and not just processed partial updates? If so, then this second theory doesn’t make sense.

#19

I was able to repro theory #2 (update race)!

Here’s what I see in the network tab (some contents removed for brevity/privacy):

(1) bundle_binary

"user_version":1931
"added":[]
"removed":["T8FQfQlQbCi06qUBDmIkzRLd"]
{"id":"T8FQfQlQbCi06qUBDmIkzRLd","parent_old":"y8JtkASdlDsmcq-4Odz0n-UJ","index_old":0,"index_new":0}

(2) bundle_binary

"user_version":1933
"collapse_changes":[{"node_id":"y8JtkASdlDsmcq-4Odz0n-UJ","collapsed":true}]

(3) bundle_binary

"user_version":1933
"added":["gFtYTJrJ2Zx7USbnzLcCYbvM"]
"removed":[]
{"id":"gFtYTJrJ2Zx7USbnzLcCYbvM","parent_old":"","parent_new":"y8JtkASdlDsmcq-4Odz0n-UJ","index_old":0,"index_new":1}

So it seems that we aren’t guaranteed to be fully synced before we fetch the collapse changes.

From the server’s perspective the collapse event must occur on or after the new item is added, so this just appears to be a client syncing issue (eagerly fetching collapse changes before all changes synced down).

I’m curious if the same fix can be used for both of these bugs. If whenever an item is added it includes the collapse state of its parent in the meta that may work (unless add/remove updates can be processed out of order). A fix that only ensures we are fully synced before grabbing collapse changes will not fix the earlier bug.

#20

I’ve actually now reproed theory #1 as well (client doesn’t properly bump version number). Although I think this is essentially the same problem as the very first repo.

Repro steps:

  1. Open Dynalist in a tab (A).
  2. Open Dynalist in a new tab (B). Create an item with a child and collapse it. Ensure this is fully synced to tab A (including being collapsed).
  3. In tab B, verify the current document version number by inspecting user_version for the bundle_binary that is pulled down after hitting Ctrl-s in the network tab.
  4. In tab B, zoom into the collapsed item and delete the child, then immediately create a new child item and zoom out and collapse the item.
  5. Don’t Ctrl-s, just wait for the page to naturally sync.
  6. Open Dynalist in a new tab ( C ) and ensure that the item is collapsed, meaning that the server thinks the item is collapsed.
  7. In tab A, wait for it to naturally sync. If the item is uncollapsed, the issue reproed. Otherwise repeat from the top until this occurs.
  8. Inspect the user_version in tab A (or any tab for that matter). It is likely the same as in step 3, meaning the version wasn’t bumped even after the item was collapsed.

Based on this, I assume the version bump is done on the server side, and since all it saw was that the item was collapsed before and after the client synced it didn’t think there was a difference.

1 Like
#21

Yeah the version handing is all server sided. Will think of a better solution in the coming days!

1 Like
#22

Will be fixed in the upcoming release on web, with mobile release coming a bit later!

1 Like
#23

I’m wondering what the rule is here. Is it that there is ONE collapsed state for a node, irrespective of who looks at it?

#24

It’s stored on the server side as a per-user/per-document blob, which contains all the node’s collapsed state.

#25

Per user but not per device? Okay. Just wondering.

#26

It should be synced across devices, at least on web. Desktop might work a bit different.

#27

@Shida Have any changes gone out related to this already? I noticed something odd today on mobile I’ve never seen before (even if mobile code hasn’t changed, maybe affected by server side logic change). I had an item with no children, and I’m sure that my mobile client was up to date. I added two child items using Quick Dynalist (API), and when I reopened the android app the item was collapsed. This shouldn’t happen since the item was childless (therefore uncollapsed), and then new child items were added. Could recent changes have caused this unexpected behavior?

#28

It definitely looks like this change has started rolling out as the first repro I gave is fixed (thanks for that!). Although I’m seeing some other weird behavior now, likely due to these changes:

On web:

  1. Create item with one child and collapse.
  2. Zoom into item and delete the child.
  3. Zoom out (item will not have any children, so is uncollapsed).
  4. Zoom into the item again and add a child.
  5. Zoom out, and observe that the item is collapsed again.

NOTE: If on step 4 you add the child while zoomed out it will properly be uncollapsed. It’s only an issue when you add the child while zoomed in to the parent item.

#29

I also have a new repro (slightly different than the original one, but in a similar vein) that shows that this isn’t fixed.

On the web:

  1. Create item with one child and collapse (Foo -> Bar).
  2. Go offline (dev tools -> network).
  3. Zoom in and delete the child item.
  4. Zoom out and add a new child item “Baz”. The item will be uncollapsed.
  5. In a new tab, zoom into the item and add a second item “Qux”.
  6. Reconnect the first client and sync up.
  7. Verify the reconnected client is uncollapsed, while all other clients are still collapsed (even though all clients do show the proper child items “Baz” and “Qux”).

I’m guessing that in this recent update there’s some tentatively_collapsed state that’s kept for nodes with deleted children that haven’t synced up yet, is that correct? But when you add a new child item zoomed out the client “knows” that it is no longer collapsed, leading us back to the original bug. This type of logic likely also explains the other issue I reported one post above.

In this case I actually think it makes more sense for the server-side state to be “uncollapsed” if we’re following a “last-write-wins” model. This is made trickier by the fact that we don’t want to uncollapse items if they only became empty (no new children added), like in the original repro.

Here’s an idea for what might fix this: For each node that experiences a collapse state change keep track of the following (clearing out this state after a sync occurs):

  1. The last synced collapse state for the node.
  2. The current collapse state for the node (e.g. if you explicitly uncollapsed the item OR deleted all children, it is uncollapsed).
  3. Whether the collapse state change is tentative or definite. An explicit collapse/uncollapse is a definite change, whereas an item becoming uncollapsed due to all children being deleted is tentative. Adding the first child to an item is also a definite change (since the item is now visibly uncollapsed).

If a change is tentative, after a sync it should apply the following state changes (in this order of precedence):

  1. Any new collapse state changes synced down for that node.
  2. The last synced collapse state for the node.

If a change is definite, it should override any collapse state changes it sees and sync up its collapse state to the server (last-write-wins).

(this also relies on the server side explicitly setting nodes without any children as “uncollapsed”, including bumping the user version number when this occurs. I think this was the old behavior, but after the update this seems to have changed.)

It looks like the current solution is doing something like 1 and 2, but maybe it’s missing the 3rd part?

If we follow this new model, then the above repro should result in the server-side incrementing the user version number, and forcing all other clients to be uncollapsed. For the original repro, the reconnecting client’s change would only be tentative, so once it resyncs it would re-apply the original collapse state it had for the item (collapsed), which would make it consistent with the other clients.

#30

I followed your steps and got a repro. Upon further investigation this is caused by a bug in the sync algorithm where anytime you upload changes, but there are remote changes to be downloaded, the server rejects the upload, but the collapse changes were discarded accidentally and not uploaded on retry.

A fix has been prepared and has been deployed to web.

3 Likes