Talking to a Figure: Two Routes from Natural Language to JSXGraph – JSXGraph AIR Compass and JSXGraph AIR for Visual Studio Code

Carsten Miller

Center for Mobile Learning with Digital Technology, University of Bayreuth, Germany

carsten.miller@uni-bayreuth.de

Abstract

Dynamic mathematics software offers two approaches, each with its own advantages and disadvantages. Direct manipulation requires no syntax, but provides an instance rather than a description. Code provides precision and the ability to be reused, but requires knowledge of syntax, a barrier for learners and teachers preparing material. Large language models can remove this barrier by generating construction code from a written description. In practice, however, when a point is dragged, the displayed figure can diverge from the code, so subsequent instructions act on a representation that no longer matches what is on screen.

In this presentation, I will introduce two tools from the Artificial Intelligence Renderer (AIR) project that address this issue from different perspectives: one is a standalone environment and the other is an extension to a professional editor.

JSXGraph AIR Compass is a browser-based environment built around JSXGraph. A cloud assistant, a local assistant and a board window all write into a single source window, which renders the live preview. Every draggable element is instrumented, so when a point, glider, slider, line, circle or polygon is moved, its new position is written straight back into the source code. For supported elements, the source stays in sync with the displayed state, and the model receives the last state that the user left on the screen. Language, code and figure all describe one state, meaning that follow-up instructions such as “colour this circle red” no longer reset the figure to its initial state.

Global properties of the drawing area, such as axes, grid, viewport and background, are held in a board layer of their own. These properties are stated in natural language and kept editable so that they persist across constructions. The same prompt reaches either an external provider or a model running on the user’s own machine; on the local path no input leaves the room, which removes the transfer question that otherwise dominates classroom use. Generated code is matched against a blocklist before it runs. The interface is available in German and English. JSXGraph AIR for Visual Studio Code takes a different approach: it uses the editor for authoring and turns a browser tab into a live preview. A small local bridge server monitors two files and transmits their content via WebSocket whenever they are updated. Beyond AI, what makes this useful is that the browser preview always reflects the current file, so handwritten code, pasted snippets and constructions generated by GitHub Copilot are treated identically. This means the tool is just as usable with the AI turned off. Header content is admitted through a structured allowlist (external stylesheets, scripts over HTTPS, meta elements and inline styles, each of which can be switched on or off individually), so libraries such as MathJax can be loaded deliberately. Finished work can be exported as a single, self-contained HTML file or copied to the clipboard.

The presentation will demonstrate both tools live and set them against our sketchometry project. Gesture and language are two attempts at achieving the same goal — reaching a construction without writing syntax — and they both have their strengths and weaknesses.

Both are working prototypes. Neither has been evaluated in a classroom setting, and no claims are made about learning outcomes. Write-back is text-based and covers the above-named element types, not arbitrary constructions. The file bridge is one-way, so dragging updates the in-page source rather than the watched file. Local inference also requires hardware that schools may not have.