The problem is not the use of Google Drive but the use of Google Apps in general. The Google services access your data in the background and send it to Google. You can see this very well if you install a firewall and an adblocker. For example, I use "AF-Wall+" and "Blokada". I get dizzy when I look at the protocols of these apps.
So the goal is to run a smartphone without Google Apps. This is possible without any problems with FOSS apps. There are free and open software alternatives to any Google App (https://switching.social/). I already sync Enpass through my nextcloud via webdav (DAVX5). Since Enpass cannot be installed without the Playstore I can't install my purchased version on my smartphone without gapps. Unfortunately a reason to look around for an alternative.