Increasing load speed of CesiumURLTemplateRasterOverlay with local data

I’m trying to use CesiumUrlTemplateRasterOverlay to load local data using a localhost url: http://localhost:8080/tiles/16/{z}/{x}/{y}.png
The 16 in this example is an hour, and I have data for every hour from 12-17.
I can change the data shown by changing the hour in the URL in UE editor, which refreshes the data and collects the data for a new specified hour. e.g. http://localhost:8080/tiles/12/{z}/{x}/{y}.png.

It is working as intended, but I would like the data load to be more instant, such that the user could change the hour with a slider and see a more instant change.

Settings:
Projection: Geographic
Specify Tiling Scheme: false
Minimum level: 12
Maximum level: 16
I have generated the corresponding folder structure for level 12-16.
I have tried tile size 256 and 512, but they seem to load equally fast.
It does not look like Maximum Simultaneous Tile Loads changes much either.

From what I can tell, the whole terrain is reloaded. This might be a contributor, since I’m using Google Photorealistic 3D Tiles.

I also get a lot of errors such as:

Received response code 404 for image http://localhost:8080/tiles/12/4/24/14.png.

Which I find strange, since the 4 in this case would mean its trying to load Z level 4, even if I defined minimum to be 12.

Any ideas for load time improvements?

Regards, Gustav