I’m curious why you chose to implement point clouds in this way, since massive point clouds are already well supported in CesiumJS with 3D Tiles.
I haven’t looked at your implementation, but have you seen this article on using the Primitive API (https://cesium.com/docs/tutorials/geometry-and-appearances/) ? What you could do is detect when the camera moves, and destroy/recreate the geometry as needed. Using the Primitive API will give you more control over how to batch this geometry, or create it asynchronously if needed etc.
I didn’t realize your goal was to display 3D buildings though. I think this approach is definitely going to be harder and slow for large data. This is why we handle the tiling and clamping to ground as an offline process, so it can be fast at runtime even with hundreds of millions of buildings (see Announcing Cesium OSM Buildings). Do you have a way of exporting your data as CityGML or KML/COLLADA?