FEEDBACK REQUESTED: Native AI Copilot for Sandcastle

General
1

Hello Cesium Community,

The purpose of this post is to open a technical discussion regarding the next evolution for Sandcastle: integrating a native, agentic AI Copilot.

As AI-assisted development tools continue to evolve, we’re curious about your experiences using them with CesiumJS and Sandcastle. What has worked well? What challenges have you encountered?

The Vision: An Integrated AI Agent

We’re exploring whether a native AI agent integrated directly into Sandcastle could enhance the development experience. Our hypothesis is that tight integration with the Sandcastle environment could reduce iteration time and errors by enabling new capabilities, such as:

  • :building_construction: Rapid Scaffolding: Build applications with idiomatic, up-to-date CesiumJS code.
  • :lady_beetle: Intelligent Debugging: Read console output to suggest and apply fixes automatically.
  • :globe_with_meridians: Smart Asset Integration: Connect with Cesium ion to find and load appropriate assets for a given prompt.

Before diving into implementation, we want to ensure we’re building something that will have maximum impact for the community. Your ideas and feedback are essential.

:speech_balloon: We’d Love Your Feedback

Here are a few areas where your input would be especially helpful:

  1. Your Experience:
    Have you used any AI code generation tools (like GitHub Copilot, ChatGPT, etc.) while working with Sandcastle or CesiumJS? What worked well, and what didn’t?

  2. Ideal Use Cases:
    If you had an integrated Cesium AI agent, what is the first complex task you would ask it to build?

  3. Workflow Bottlenecks:
    What are the most repetitive or frustrating parts of your current Cesium development workflow that an agent could automate?

  4. Beyond Code Generation:
    What other tasks would you want an AI to handle? For example:

    • Managing camera views
    • Generating flight paths
    • Optimizing 3D Tiles styling
    • Something else?

  5. Vision & Concerns:
    What does the ideal AI-assisted development experience look like to you? What technical challenges or pitfalls should we avoid?

This is an open exploration, and your feedback will help shape this from a concept into a valuable tool for the entire community. Whether you’re excited, skeptical, or somewhere in between – we’d love to hear from you.

Looking forward to the discussion!

4 Likes
2

This is a great initiative. I am currently trying to integrate a CHAT GPT chat bot interface into my VR game so the user / gamer can speak to the AI BOT , ask questions about the area on the earth they are looking at etc… or ask to go to a particular spot on earth having descibed it. Currently I am using speech to tect to get a prompt for Geo Coding to return a longitude and latitude but more richness could be added through a chat gpt interface I am developing. This would require separate API licensing for GPT alongside that for Cesium ION so having everything integrated into Cesium with just one licence agreement for ION tile streaming and AI integration would be great if it offered the power of AI response offered through chat GPT :wink: Thanks

2 Likes
3

Personally, I would prefer an “AI Agent friendly” interface to Cesium docs & examples that I can use with whatever AI coding set-up I am currently using (instead of “another” plugin).

The key capability would be the semantic / hybrid search through the docs and examples (but made to be consumed by the LLM, not humans).

A step further, perhaps a custom Cesium skill:
https://github.com/huggingface/skills (<- the open source version of skills), where custom prompts, bash scripts that LLMs can call, and MCP server setups can be bundled and easily added to Claude Code, Copilot, etc.

If this stays well maintained by Cesium, this could be a massive impact on dev productivity - especially because it can be smoothly integrated into existing development workflows.

1 Like
4

Hi Justin,

I apologize for not responding sooner.

Your Experience

I’ve used chatGPT extensively while developing CesiumJS Sandcastle apps. Coding CesiumJS has worked very well. With each new LLM model, coding works better. Back in o3, there were some of the data structures within CesiumJS that the LLM was not aware of. Other than that, I’ve generally been able to get the code I wanted with only a little massaging. Generating UI code with ChatGPT for Sandcastle apps has also worked really well. For me, it’s the easiest way to add controls to an app.

Ideal Use Cases

I actually have a list of uses cases I already implemented with GPT.

  • Map the F2 layer of the * ionosphere, (Hamie Carter and I * presented this at last year’s Cesium Developer Conference
  • Map in map view so the user can have two different perspectives on the same geospatial data.

Workflow Bottlenecks

In every case where automation was required, I asked the LLM to create code that automated the workflow rather than using the LLM as an automation tool.

The biggest conceptual bottlenecks involved learning portions of the CesiumJS libraries that I inuited must exist (camera contro for example) but didn’t know specifically how to use or control. Another bottleneck is in understanding the limitations and advantages of the different ellipsoid models for the Earth model.

Beyond Code Generation

In my experience, it’s been much easier to have the AI generate code for the items in the list rather than trying to do those actions on its own. I think an MCP could also be setup to handle those operations, but I’d have to think about the ROI of having the AI have that capability in and of itself, rather than just generating code to manage those things specifically for each individual app.

I think it would be valuable to have the LLM inspect data sources and summarize their structure. From there though, I’d still have the LLM generate code to routinely use the data source with Cesium JS.

Vision and Concerns

I’ve linked a video of how I envision LLMs being incorporated into Cesium Sandcastle. There are three aspects I think a tool for workflows like the hurricane and satellite demos, (the satellite demo sprung out of working on my answer to your comment), should focus on:

  1. Data sources
  2. Visual feedback
  3. Coding

The screen layout highlights these aspects. The left pane is split into three working panels, an LLM chat panel for describing maps, an independent LLM chat panel for finding and talking about data sources, and a tool panel for tracking LLM chats and code progression. The panel on the right side of the screen is bigger and is for both viewing the maps produced and for viewing code as well as code revisions.

Data is what makes the map interesting in these flows, so it’s important that it be easily visible and open for users to chat about.

As I work on these maps, I keep a record of
the code generated along with my changes in a github gist. I do this so that I can try newly generated code from the LLM confidently because I know that I can revert to my last working version if the LLM produces something I don’t like, (or if it just flat out breaks the code.)

I also like to keep an easy to search and reuse record of the chat itself. LLMs have an issue in that their context eventually fills to overflowing. At that point, they tend to make successive coding mistakes. By having my chats in a searchable database, it’s easier to reconstruct my work—leaving out any false leads or no longer interesting explorations to conserve context space—if I need to start over in a new chat session.

There are tools available that facilitate this sort of chat recording and reconstruction. My preferred one at the moment is a Python package called llm that works in conjunctions with Datasette.

I believe all of this is realizable with current technology and would enable people to build useful mapping prototypes quickly. All of the prompts, maps, and code snippets in the video are from my recent work on amateur radio satellite maps.

I’d enjoy discussing this further with you if you’d like!

Best Regards,
Hamilton Carter

2 Likes
5

@yazzap That sounds like a killer feature for a VR game. Out of curiosity, what speech-to-text are you using right now? Is it one of the open source models or a cloud-hosted service? I know there are some pretty good options out there these days.

Regarding your unified ecosystem vision, that is really helpful to hear, especially as we navigate architectural requirements and see what is possible for Cesium ion integration. At first, these efforts are going to be primarily open source, but deeper ion integration is also definitely something we are looking at down the road. I think there is a lot of potential there.

6

@Joscha_During Again, couldn’t agree more with your take on this. Following up on our discussion over on LinkedIn, I want to summarize a couple of the key takeaways/topics here for the benefit of the wider community.

1. Skills vs. Sub-agents We discussed whether a Cesium implementation should prioritize the Skill format or try to define the Sub-agents (orchestration) themselves.

  • The Consensus: The high-leverage move is to prioritize robust Skills (tools + system prompts + documentation) rather than enforcing a specific orchestration flow.

  • The Reasoning: Sub-agents are essentially just “Agents + Skills + Identity.” If Cesium provides the modular building blocks (e.g., a terrain-expertise.md system prompt or a specific MCP server), developers can easily assemble their own “Terrain Expert” sub-agent on the fly using their preferred LLM.

  • The Balance: While this approach empowers power users to “meta-engineer” their own complex workflows, providing these Skills as a cohesive package or plugin serves as a strong, opinionated recommendation that helps the majority of users get started immediately.

2. Portability & The Claude Code Ecosystem We also touched on the emerging skills standard and claude code plugs, while at the same time considering concerns regarding coding assistant lock-in.

  • Internal Validation: It was actually quite validating to hear you bring up the Claude Code ecosystem as a potential starting point. Internally, a significant portion of our initial prototyping has been centered around workflows compatible with Claude Code, given its position at the epicenter of these emerging standards.

  • The “Lock-in” Reality: I have found that because Skills are effectively an open standard, they are incredibly portable. The specifications we write for the Claude Code CLI context are almost immediately usable by other agents (Codex CLI, Gemini, etc.) with minimal friction. Time spent developing nice “plugins” for Claude Code is also likely time well spent for other ecosystems’ equivalent of plugins (now and in the future).

7

Hi I was testing with some of the cloud hosted models but want to avoid another charge for that service if possible via another provider so was looking at stand alone open source stuff integrated but not having so much luck….. Have dropped this for now as want to get an MVP on the market without the STT / TTS services but this will come in the future after I have game on the market etc…but would be good if this was supported out of the box within the Cesium framework I am using within unreal engine for quest 3 :wink:

1 Like
8

Hi Hamilton,

I watched the video you linked and I have to say, the thoughtfulness you put into that design is genuinely impressive. The way you structured the layout to explicitly separate the data sourcing tools from the coding chat shows you have really lived through these friction points. It is rare to get such high-fidelity feedback, so thank you for putting that together.

Your sentiment on having the LLM “create code to automate a workflow” rather than using the LLM itself as the automation tool is spot on. That is a very popular sentiment across the industry right now, and it forms the basis for a lot of the “Code Mode” approaches we are seeing in newer tools. It is especially effective in cases where direct tool calling (MCP) doesn’t scale well…code acts as a much more flexible abstraction layer.

I think a lot of what you are describing, specifically that clean three-pane layout with data context, chat, and live preview, naturally lends itself to the workflows emerging in modern AI-enabled IDEs and IDEs paired with CLI-based coding agents.

While overhauling the Sandcastle UI to match that exact layout would be a massive undertaking, I think we can actually achieve the spirit of your workflow even more effectively through a custom VS Code Extension. In fact, we recently started exploring this exact path: Cesium Visual Studio Code Extension - We Need Your Input!

To tie this back to the other discussions in this thread (specifically with @Joscha_During), I think your vision is fully compatible with the concept of Agentic Skills. Instead of building a rigid UI inside the browser, we could provide the “Cesium Skills” (the data inspection and code generation logic you mentioned). You could then plug those skills into a VS Code workflow that offers that “Direct Rendering/Preview” capability right alongside your code. This would give you the freedom to manage your chat history and “undo” states (via Git) exactly as you described, without being constrained by a browser session.

I think an extension that offers specialized rendering of Cesium scenes right inside the IDE, powered by these portable skills, would arguably make for an even more powerful experience than a browser-only tool.

Would love to hear more of your thoughts on whether an official VS Code extension would satisfy that “clean workflow” requirement for you.

9

@Joscha_During We started exploring RAG based MCP Servers to address AI Hallucination in Cesium Development. As a potential solution we’re experimenting with Context7 ( Context7 - Up-to-date documentation for LLMs and AI code editors ) MCP Server: Fight AI Hallucination in Cesium Development — Your Feedback Is Needed! What do you think about this potential solution? Does it cover “AI Agent friendly” interface workflows?

1 Like
10

Hi Justin,

Apologies for not seeing this till today. (I gotta work out my notification settings :slight_smile: .)

I think the VS Code integration could work. From the point of view of non-LLM comfortable, and non-IDE comfortable users, will the OG sandcastle still be available? I can easily hop over to VS Code, but when I started working with CesiumJS and CZML it would have been an extra step to get started and might have slowed me down a bit on adoption.

Best Regards,

Hamilton Carter

415-799-1629

11

Hi Margarita,

I love this! Thanks! It won’t be my go-to for specific applications, but it’ll make a very nice framework to

  1. Learn MCP with CesiumJS (I posted a video to LinkedIn today about your camera view branch:
    Cool new MCP from Cesium! I've been tracking along with Cesium's AI integrations and wandered into a brand new MCP implementation to allow LLMs to control the CesiumJS viewer (it started… | Hamilton Carter
  2. Facillitate voice control of map and data visualization via the LLM. I think this will be really cool for automating visualization operations with a given data set.
  3. Contribute with other data visualization efficiency apps in the future.

Best Regards,

Hamilton Carter

415-799-1629

1 Like