Fujifilm X-T4 with Canon / Sigma lenses AF problem in stills

Hey everyone,

I have been searching for this forum on this topic but I did not found an exact answer for my problem.
I have just bought a Fuji X-T4 camera and a Viltrox ef-fx1 mount adapter to be able to use my current lenses. They are Canon, Sigma and Samyang glasses mostly with EF / EF-S mounts. As far as I am aware, the Viltrox adapter should make the communication between the Fuji body and the 3rd party lenses in terms of AF and exif info.
I have tried a lot of different settings, but the AF problem comes down to this:
I have set the camera to AF-S mode with a single point AF with back button focusing. I guess this is the single and most simple way to check AF on lenses for stills. So if I press and hold the back (AF/ON) button, the focus is set sometimes perfectly, sometimes like half-way. It means that I need to release the button, and press it several more times to get a perfectly sharp focus. I realised that it works well when the focus needs to be set from a distant subject to a closer one or the other way around. So if the focus is very far from the intended spot, it works well. When I change the focus area between objects with little distance difference (e.g. one is 5m away, the other is 6m away), the camera changes the focus just a little bit which is not enough, so the selected area remains blurry. I need to press the AF-ON button again, and it does a better precision, and if I press it several times, it finally finds the right AF. It never happens in the case when I select objecst that are far from each others in depth.
I tried it with my Sigma 17-50mm f2.8, a Canon 50mm STM, a 55-250mm STM, a Tokina 11mm f2.8 lens. They all seem to be suffer from the same phenomenon.
If I set the PRE-AF on, so the camera tries to constantly set AF, it is sometimes a bit better, sometimes not. And I cannot check the focus with the zoom-in check screen if this is set to ON.
With a lens that is capable of manual override (such as the Canon 50mm STM), I can set the focus mode to Manual on the body, but leave the AF switch to on on the lens. In this case I can see the focus peaking and I am able to use the AF-ON at the same time. If I use this, the accuracy is a bit better of the AF, but many times I need to override it woth rotating the focus ring.
So, what I do not understand, that it seems that the Fuji camera seems to be able to decide the perfect AF in some cases (when bigger AF shift needed), and finds the focus ok in cases when it is half-set (the image is still blurry but the optimal setting is close). Why is it? It is like there is a threshold of deciding if the focus is already set and this threshold is quite bad. When the focus area is blurry, it sets it well, but if it is a little blurry, it leaves this as it is.
I tried also the Focus / Release priority settings for AF-S and AF-C. Nothing really changed. I tried the tracking speed and sensitivity as well for AF-C custom menu, but if I understand correctly, this is for AF-C mode only.
The firmware are updated for the body and for the adapter too.
Any ideas?