Strange drift Sony FX6 Zoom F4

I used a Sony FX6 and a Zoom F4 recorder. Both had a Tentacle attached permanently. FPS was set to 29,97 on all devices. Sequence in premiere is also at 29,97.  Syncing TC in post gave serious trouble and I ended up syncing with audio instead of TC. Looking at my sequence I now see that whenever there is a new audio or video clip, audio is behind by 1 frame, but this difference becomes larger quite quickly and after ten minutes audio is like over 20 frames behind (with both clips running normally at 100% and no cuts). Does anybody have any idea, how a clip that starts out in sync (well, nearly) can be out of sync (according to TC) by the end of it?