Initial nDisplay integration into Cesium sample project crashes on startup (remote nDisplay hosts) 5.2

Hello, we have several successful nDisplay projects with properly working nDisplay configurations. (5 machines, 5 displays).

Upon adding a Cesium tileset to a working nDisplay project - in this case the Photorealistic google tiles, the nDisplay portion of the project no longer works. The project runs in the editor and when run as an isolated single build, but the nDisplay remote displays crash upon creation (or shortly thereafter).

I am looking for some guidance on integration of these two elements (Rendering Cesium tiles + remote machine nDisplay configurations).

  1. nDisplay loads an independent version of the same project and syncs frames with a Primary node, so each Unreal project is its own instance of the same Cesium-based render project.
  2. We have 5 machines running 5 4k displays simultaneously

Questions:

  1. Any simple fix that I am overlooking to allow a headless build to be run remotely?
  2. Has anyone built an nDisplay example with Cesium that may have some insight or issues they came across and resolved?
  3. Do I need to do anything specific on each remote machine (logged into Cesium ION with the same account) to allow for the simultaneous instances to run and not be in conflict?

Any help would be greatly appreciated! Happy to share logs, images, etc. to further it along.

Thanks!

1 Like

Some more color on the nDisplay integration:

  1. We have six machines in total:

    • A head machine that runs the nDisplay switchboard and launches the other 5 remote Unreal projects
    • 5 machines that are each displaying a viewport of the same scene within the same project (N, E, S, W and down)
    • Tested #1: Each machine is referencing the same networked folder, logged in with the same Cesium Ion account and same Cesium Ion token
    • Tested #2: Each machine is referencing a local version of the same project (not shared), logged in with the same Cesium Ion account and same Cesium Ion token
    • Tested #3: Each machine is referencing a local version of the same project (not shared), logged in with the same Cesium Ion account and using different Cesium Ion tokens for the same project
    • Tested with all available tilesets and Unreal 5.2 & 5.3
    • All sample projects run smoothly as independent projects on each machine, no tile rendering issues
  2. We have ne through all available logs on the nDisplay and Unreal side, but could use some more debugging from the Cesium side.

    • Is there an issue pulling multiple tiles from multiple machines simultaneously with the same app and token?
    • Is there some tile rendering issue that is triggered on nDisplay start?

Any help would be greatly appreciated @agallegos @Kevin_Ring ?

Hi,

we have pretty much exactly the same setup, i.e. one primary node and five nodes in our nDisplay cluster.
Our Unreal projects are on the same network share and all 6 nodes are accessing this one project.

Unfortunately we have not run into the problems you encountered.
Do all other nDisplay projects work well for you?

What I would check is:

  1. Make sure that on every machine running Unreal all plugins that are used in the project(s) are installed and it’s exactly the same versions. Not just cesium but also others
  2. Try with locally hosted Cesium tiles. The easiest would probably be to create or download Cesium tilesets, put them in a folder and use a simple python server. Then you can add the Cesium tiles from url with http://X.X.X.X:8000/tileset.json or similar
    This way you could check if the problem is with Cesium Ion (i.e. the account or tokens) or with something else.
    Be sure to use an IP address of the main node that all cluster nodes can access

I’m using this python script to do so:

import http.server
import socketserver
import os

CWD = os.path.join(os.path.dirname(os.path.abspath(__file__)), os.pardir,"CesiumData")

PORT = 8000
ADDRESS = "X.X.X.X" # an IP address of your primary node that all cluster nodes can access

class Handler(http.server.SimpleHTTPRequestHandler):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, directory=CWD, **kwargs)

# handler = http.server.SimpleHTTPRequestHandler

with socketserver.TCPServer((ADDRESS, PORT), Handler) as httpd:
    print(f"Server started at {ADDRESS}:{PORT}")
    httpd.serve_forever()

Hello! Really appreciate the reply - added some more to our testing process and good to know that someone else is running from the same project location.

All other nDisplay projects work well in our environment, so this is specific to the Cesium project.

In fact, we can add and remove the Cesium Tileset actor (any of the provided tilesets) and when removed, the project runs just fine in nDisplay - add it back in and it crashes. Other Cesium actors work (sun, pawn, etc.) Very strange.

  1. For sanity check, we duplicated everything again to make sure all versions of plugins and Unreal matched specifically.
  2. Loaded a blank tileset scene without a specific API call and it still crashed
  3. Removed the blank tileset and it worked again
  4. We are working on the local tileset through local URL option now and will report back for future knowledge. Thanks for the code snippet.

Again, thanks for the input!

Is there an issue pulling multiple tiles from multiple machines simultaneously with the same app and token?

Generally, no. However, the Google Photorealistic 3D Tiles have a fairly tight rate limit for the initial request. If all 6 machines simultaneously tried to make their initial request to the Google tile server, some of them might fail.

Is there some tile rendering issue that is triggered on nDisplay start?

Not to my knowledge.

Since you’re crashing, the next step is to get the call stack of the crash. We’re unlikely to be able to help you without that. That means building a Debug (or at least Development) configuration.

Thanks for the feedback - much appreciated. We have been digging through the output and logs, but have yet to find a clear culprit for the crash.

We will reach back out with more specific debugging output.