You will need to create an Entity on which you may specify PointGraphics properties if you want Cesium to handle time for you. Was there something in particular that lead you to use PointPrimitive?
It sounds like you will be creating your own file format to send to the client which would avoid most of the parsing overhead associated with CZML. That might give you enough of a performance boost to manage each time slice in an EntityCollection and let Cesium handle time.
Alternatively, you could manage a list of PointPrimitiveCollections and then subscribe to the Scene’s preUpdate Event in which you would hide the points in the “active” PointPrimitiveCollection and show the points in the time appropriate PointPrimitiveCollection.
There isn’t currently a built-in way to do time-dynamic 3D Tiles, but one way is to create a 3D Tileset for each time slice, and show/hide these as the timeline changes.