You can actually see how screen space error is computed from the geometric error here:
It’s essentially scaling it based on how much screen space the tileset is estimated to be taking up. So a tile may have a geometric error of 10, but it’s so far from the camera that you can’t notice this error, then it would have a screen space error of 0. So the idea is just projecting the geometric error onto the screen. With that said, it is largely an approximation (both the geometric error computed during tiling, and the projected screen space error computed at runtime). Getting better approximations for these I think is not an easy problem, but you may find some writing on this out there since it is not specific or unique to 3D Tiles.