I do not know how the linked project actually implements the visualization. But from the appearance and behavior, and from quickly skimming over the code, I think that the buildings do not use textures at all. For example, the “windows” that you are talking about seem to be procedurally generated [e.g. from this part - some guesses here, but it looks like this).
There are options for styling the buildings, e.g. starting with
and looking at the example sandcastle at
But creating real textures for the OSM buildings would require generating actual texture coordinates based on the shape, extrusion height and many other things. There are some related threads, for example, for Cesium For Unreal, at Easiest Way to make a "Night" time scene? - #2 by agallegos , listing why this is not really easy to accomplish.
It might be possible to achieve something that causes the buildings to “not look so empty” (i.e. also generating something like procedural windows), by using a CustomShader - Cesium Documentation , but there’s no off-the-shelf solution for that.