Using DSL Over HTML for AI Agent UI Understanding
Technical insight on why Domain-Specific Languages are more effective than HTML for describing UIs to AI agents, with prompt engineering tips.
Why DSL Beats HTML for AI Agent UI Understanding: A Developer's Deep Dive
Originally inspired by a practical reverse-engineering experiment shared by @cholf5
If you've ever tried to feed a UI screenshot into an AI agent and asked it to reason about or replicate that interface, you've probably defaulted to one of two approaches: describe it in plain English or dump the HTML source. It turns out both of these are suboptimal. A growing number of AI engineers are discovering that Domain-Specific Languages (DSLs) offer a dramatically better way to describe UIs to AI agents — and the reasoning behind this might change the way you build your automation pipelines.
Let's break down why.
The Experiment: Reverse-Engineering a UI with ChatGPT
The insight here comes from a straightforward but illuminating experiment. The developer took a screenshot of a UI, handed it to ChatGPT, and asked a simple but powerful question:
"What format is best for describing this UI so that an AI agent can work with it effectively?"
The expected answer? HTML. After all, HTML is the native language of the web, it's structured, parseable, and universally understood.
But ChatGPT's answer was surprising: HTML is not the best choice for agent-driven UI tasks. Instead, it recommended using a DSL — a Domain-Specific Language purpose-built for pure, clean UI description.
This isn't just a stylistic preference. There's solid technical reasoning behind it, and once you hear it, it's hard to argue against.
Why HTML Falls Short for AI Agents
HTML is an extraordinary tool for browsers. It tells rendering engines exactly how to display content, handle events, and structure a document. But when you're feeding UI descriptions to an AI agent rather than a browser, all that richness becomes noise.
Here's what goes wrong with HTML in an agentic context:
- Tag pollution: HTML is littered with structural tags (
<div>,<span>,<section>) that carry zero semantic meaning for an AI trying to understand what a UI element does, not how it's rendered. - Style interference: Inline styles, class names like
btn-primary-lg-rounded, and deeply nested DOM structures force the model to wade through presentation logic before it can extract intent. - Code generation bias: When you describe a UI in HTML and ask an agent to generate code from it, the model tends to reproduce the HTML structure — even if the target platform is React, SwiftUI, Flutter, or a Python GUI framework. The HTML scaffolding bleeds into the output.
- Verbosity: A simple login form in HTML can be 50–80 lines. The same form described in a clean DSL can be 8–12 lines. Token efficiency matters, especially in agentic loops.
The core issue is that HTML conflates presentation with structure, and AI agents primarily care about semantics — the what and why, not the how.
The DSL Advantage: Pure Description, Better Results
A well-designed UI DSL strips everything down to its semantic essence. Instead of telling the agent what tags to render, you tell it what elements exist, what they do, and how they relate to each other.
Here's a practical comparison. Imagine a simple login screen:
HTML approach:
<div class="container mt-5">
<div class="card shadow-sm p-4">
<h2 class="card-title text-center">Sign In</h2>
<form id="login-form">
<div class="form-group mb-3">
<label for="email">Email Address</label>
<input type="email" class="form-control" id="email" placeholder="[email protected]" />
</div>
<div class="form-group mb-3">
<label for="password">Password</label>
<input type="password" class="form-control" id="password" />
</div>
<button type="submit" class="btn btn-primary w-100">Login</button>
</form>
</div>
</div>
DSL approach:
Screen: LoginScreen
Title: "Sign In"
Form: LoginForm
Field: email
type: email
label: "Email Address"
placeholder: "[email protected]"
required: true
Field: password
type: password
label: "Password"
required: true
Action: submit
label: "Login"
style: primary
width: full
The DSL version is cleaner, shorter, and laser-focused on intent. An AI agent reading this DSL knows exactly what the screen contains, what actions are available, and what data needs to be captured — without any rendering baggage.
When you then ask the agent to generate code from this description, it's free to produce idiomatic output in any target language or framework, because it isn't anchored to HTML conventions.
Crafting the Right Prompt for UI-to-DSL Extraction
One of the most actionable takeaways from this experiment is the prompt pattern used to get ChatGPT to generate a DSL from a UI screenshot. Here's a refined version you can use directly in your workflows:
You are a UI analysis expert. I will give you a screenshot of a UI.
Your task is to describe the UI using a clean, structured DSL
(Domain-Specific Language) — not HTML, not CSS, not code.
The DSL should:
- Identify each screen or view by name
- List all UI components (inputs, buttons, labels, containers, etc.)
- Capture the type, label, placeholder, state, and any relevant behavior
- Describe layout relationships (e.g., stacked, side-by-side, modal overlay)
- Note interactive actions and their triggers
- Avoid any rendering-specific details (colors, fonts, pixel sizes)
Output only the DSL. Be concise and structured.
This prompt works exceptionally well with GPT-4o and Claude's vision models. The resulting DSL can then be fed into a second agent pass for code generation, test case creation, or UI automation scripting.
Practical Use Cases
- UI reverse engineering: Capture legacy interfaces and convert them to modern frameworks without manually reading DOM trees.
- Cross-platform generation: Describe a web UI once in DSL, then generate iOS, Android, and web implementations in separate passes.
- Agent-driven QA: Use DSL descriptions as the basis for generating test plans and automation scripts (e.g., Playwright, Appium).
- Design-to-code pipelines: Feed Figma exports or screenshots into a DSL extraction step before code generation to improve accuracy and reduce hallucinations.
Conclusion: Rethink Your UI Representation Layer
The lesson here is subtle but significant: the format in which you describe a problem to an AI agent shapes the quality of its output. HTML was designed for browsers, not for reasoning engines. When you remove the rendering layer and speak to the agent in pure semantic terms — what a UI is rather than how it looks — you unlock better comprehension, cleaner code generation, and more reliable automation.
DSL-based UI description is still an emerging practice, and there's no single standard yet. But that's also an opportunity: you can design a DSL that fits your specific domain, whether that's enterprise dashboards, mobile apps, or embedded device interfaces.
Start small. Take a screenshot of any UI you're working with. Run it through a vision-capable model with the prompt above. See what DSL comes out — and then try generating code from that DSL in your target framework. The results might just change how you think about the entire UI automation stack.
Want to explore more AI agent patterns and prompt engineering techniques? Stay tuned to ClawList.io for weekly deep dives into the tools and workflows powering the next generation of AI automation.