You can get the camera height with camera.positionCartographic.height, see: https://cesium.com/docs/cesiumjs-ref-doc/Camera.html?classFilter=camera#positionCartographic
You can get that value, and then put that in your flyTo options. Instead of using an invisible entity, you can just pass in a lat/lon (use viewer.camera.flyTo). This is what I did to get the “fly to a location but keep the camera height” in my app here: 🌍 Waking up and Sleeping 🌎 - Live Twitter Map.
If your user is going to be really close to the ground, you may need to sample terrain height at the lat/lon you’re flying to, since a height of 500 meters relative to the ellipsoid in one place on Earth may be underground in another place.