Extend Assist with home management help, starting with documentation

[nielsrowinbik]

Problem statement

Home Assistant's Assist feature is built for home control: turning lights on, adjusting the thermostat, running a script. But managing a home, configuring automations, diagnosing problems, understanding what a setting does, is a different kind of task, and Assist offers no help with it today. Home maintainers who need guidance are on their own, searching docs, browsing forums, or turning to external AI tools that know nothing about their setup.

This is the first step in a broader effort to bring AI-powered management help into Home Assistant. For the full vision, see the AI in Home Assistant vision document (TODO: Add link). This opportunity establishes the split between "control" and "management" as a product concept, and validates it with a first concrete capability: documentation access.

Community signals

• Multiple custom integrations exist that add AI-assisted home management to Home Assistant, indicating strong demand for this kind of capability. These projects take two different approaches: external tooling via an MCP server that accesses HA through an API (most notably ha-mcp, with 2600 stars on GitHub), and internal tooling built into HA itself (such as the community-built ha_claw). This opportunity assumes an internal tooling pattern; the choice between these two approaches is an open architectural question listed in Risks.
• This demand is confirmed by the numerous posts on Reddit, like this one, and articles on tech news outlets, like this one on The Verge, all explaining how powerful being able to use AI with Home Assistant can be.

While none of these ask for what we've proposed in this opportunity specifically, they clearly signal the enormous amount of potential of our overall vision that this opportunity contributes to.

Scope & Boundaries

In scope

• Scaffolding a set of management APIs next to the existing control APIs (called Assist).
• Extending the Assist window's Assistant picker with clarity on what each Assistant is able to do.
• HA documentation retrieval as the first management capability: the assistant can look up and explain documentation in response to maintainer questions

Not in scope

• Context awareness (knowing what the user is looking at when they open Assist) — that is a separate opportunity (Make Assist aware of what the maintainer is working on #157)
• Any authoring or editing actions such as creating automations or modifying dashboards
• Expanding or modifying the existing home control capabilities of Assist

Foreseen solution

Controlling or managing, that distinction will be made at the Assistant level by giving the Assistant access to a set of APIs.

<TODO: Add image of Assistant settings>

In this opportunity we'll scaffold a set of management APIs, just like we have the Assist APIs for home control. The first actual tool that will be available as part of that new set will be the ability to read Home Assistant documentation.

We would like to see documentation retrieval be delivered as a hosted service by Nabu Casa, offered for free as part of Home Assistant Cloud. Rather than shipping the docs and a retrieval engine to every install, the HA instance sends the maintainer's question to a Nabu Casa RAG service, which searches a vector store of the documentation and returns the relevant passages for the assistant to answer from.

The vector store is versioned against HA releases. The instance reports its running version, and the service queries the matching store, so a maintainer always gets answers for the version they actually have installed, not the latest release. This also lets documentation corrections reach users between HA releases, without waiting for an update.

Keeping retrieval server-side means no docs corpus, vector index, or embedding model ships to the device, as none of that work runs on low-power hardware like a Raspberry Pi that Home Assistant is commonly run on. The tradeoff is that the capability is cloud-connected: instances without HA Cloud or without internet access will not have it. A fully local alternative is out of scope for this opportunity, but not ruled out as future work.

Risks & open questions

• Nabu Casa delivery dependency: This approach assumes Nabu Casa builds and operates the hosted RAG service, including the versioned indexing pipeline and per-release re-embedding. Early signals are positive, but this still needs further shaping with NC to confirm scope, ownership, and the operational commitment before the opportunity can be executed.
• Implementation details to settle during shaping: embedding model choice (including multilingual support, given HA's international user base), chunking strategy, and how the instance communicates its version to the service.

Appetite

To be set.

Execution issues

No response

Decision log

Date	Decision	Outcome

[jlpouffier]

AS discussed already @nielsrowinbik
I think the main decision boils down to where this documentation too, but also all future tools will live.

I see two routes

• External: An MCP server that accesses Home Assistant via some API, like this https://github.com/homeassistant-ai/ha-mcp
• Internal: Just like our intents are. Just like this https://github.com/ha-china/ha_claw

As you can see the community is already building projects going these two routes. We just need to drive a decision now so that we can iterate on the tools creation later on.

This opportunity is just the first one, so a decision is needed

[frenck]

Putting my view here so it lands publicly. I have raised this in our internal direction conversations and it belongs on the issue too.

The direction is right. A management-oriented capability for maintainers, admin-only, documentation-aware to start. That fits where the broader AI work has been heading, and I agree with the framing of splitting "control" from "management" as a product concept.

The surface is wrong IMHO. Folding management mode into Assist couples a new product (AI-assisted home management) to the identity of an existing one (voice control). Users will expect voice-control behavior from the management mode, and management-mode behavior from voice control. We end up holding two products inside one surface for the long run, and the coupling will be hard to undo later. This is a strategic call, not a UX-testing call. The "toggle discoverability" risk in the body treats it as the latter.

There is also a fit problem the body does not call out. The Assist dialog as it stands today is too narrow for the kind of interaction this opportunity is asking it to host. Looking something up in documentation, getting an explanation, or asking how to approach a task is multi-turn and reads-heavy. The current dialog was sized for "turn on the kitchen light", not for "explain what this option does and what the tradeoffs are". The scope explicitly excludes changes to the surface or layout, which means we are dropping a wider use case into a UI that was sized for a narrower one.

The "AI in Home Assistant vision document (TODO: Add link)" reference is still a TODO in the body. That document is what this opportunity is supposed to hang off of. Without the link, the betting table cannot see the framework this is a piece of.

On the core itself. Even the documentation capability looks safer than it is, because it inherits two architectural calls this opportunity is making by implication.

The first is the surface call described above (lives in Assist, or in a new sidekick).

The second is the tooling-layer call that @jlpouffier just flagged: do AI tools live external to HA via an MCP server (the ha-mcp pattern that this opportunity cites as a demand signal), or internal to HA as native plumbing (the intent-style pattern, with ha_claw as a community example). These are different patterns with different consequences for portability, latency, security, and the integration surface.

The opportunity picks the internal-Assist surface and (implicitly) an internal-tooling pattern. Both are valid choices. Neither is named as a choice in the opportunity. The ha-mcp citation under Community signals actually points at the opposite tooling pattern from the one this opportunity is built on, which makes the signal-architecture mismatch worth calling out.

What I would suggest:

• Figure out our broader direction first: the chat-based sidekick shape this opportunity assumes, versus the sprinkled-across-the-UI shape that Make AI features in Home Assistant easy to discover and set up and disable #53 assumes. Both opportunities sit downstream of that decision.
• Resolve the surface call (Assist vs sidekick with its own identity) and the tooling call (external MCP vs internal intents) explicitly, before this opportunity is approved. They are different decisions that this opportunity bundles into one default.
• Decouple the documentation capability from these assumptions. Specify it as a capability that can attach to whichever direction, surface, and tooling layer we land on.
• Cross-link Make AI features in Home Assistant easy to discover and set up and disable #53 as the sibling on the direction axis. Cross-link @jlpouffier's comment above as the sibling on the tooling axis.
• Fill in the link to the vision document, or strip the reference if it is not ready yet.
• Acknowledge all three architectural questions in Risks. Right now the surface question is implicit in the toggle-discoverability bullet, and the direction and tooling questions are not surfaced at all.

Open to being wrong on any of these if there is a path that works that I am not seeing.

../Frenck

                       

Blogging my personal ramblings at frenck.dev

[Hedda]

Towards the broader direction, could the new Open Floor Protocol (OFP) that the Open Voice Interoperability Initiative is now promoting potentially align and be a strategic fit for extending Assist in the long-run?

• https://voiceinteroperability.ai/

As I understand it is specifically designed so different assistants can discover each other, hand off tasks, and collaborate. The initiative basically describes OFP as enabling assistants to locate other assistants with specific skills and delegate requests between systems.

• https://github.com/orgs/home-assistant/discussions/3832

OFP is made for interoperability and delegation between assistants/services, which could become relevant if and when extending Assist will evolve further toward broader workflows with multi-agent coordination. It may be worth considering whether future Assist architecture and/or HA Cloud's own offering could benefit from compatibility with OFP concepts rather than creating entirely custom assistant-to-assistant orchestration.

At least it would be one possible way forward to introduce a standard path and avoid custom assistant-to-assistant communications.

Using their standard would also connect the Open Home Foundation the with other broader open assistant interoperability efforts.

[nielsrowinbik]

Replying to both @jlpouffier and @frenck at once because the points overlap.

On the vision document

Fair, the TODO link is a gap. I've drafted a document in which I make a decision and build the case for it, but I haven't managed to finish it yet. I intended to do it much sooner after the roadmapping session, but life got in the way. I will pick this back up and finish it by the first half of next week, update the initiative above (and any others on the same topic), and ping you.

Having this document available will also clear up the confusion caused between this proposal and #53. These appear to be making different assumptions on how to surface AI features. They don't, but that's an issue primarily with the wording of #53. I will address this in a reply to your comment there @frenck.

On the tooling-layer call

@jlpouffier is right that this needs to be resolved before the opportunity is approved, and @frenck is right that the ha-mcp citation in Community signals actually points at the opposite pattern from the one the proposal assumes. I shouldn't have left that implicit. I've added an explicit item to risks and open questions, and we should resolve it separately before this moves forward.

On using Assist as the UX surface

@frenck writes that "users will expect voice-control behavior from the management mode, and management-mode behavior from voice control." That assumes Assist is fundamentally a voice product. It may have been marketed as such, but it doesn't have to remain like that. Assist, within the context of the Home Assistant UI, is a text input with optional voice. That's the same shape as ChatGPT, Claude, and every other AI chat interface.

Even if Assist is currently architecturally a voice product, it makes more sense to brand the proposed extension as the next evolution of Assist rather than introducing an entirely new concept next to it: splitting control and management into two separate surfaces would mean two chat interfaces, both accepting text and voice, with no meaningful distinction in how the user reaches them. That's the worse UX outcome in my opinion. I believe evolving what Assist means is the right call here.

On the dialog being "too narrow": the scope explicitly excludes layout changes for this opportunity, but not permanently. Starting with documentation retrieval in the existing dialog is a deliberate choice to keep the scope of this opportunity limited, not an architectural commitment to keeping it small forever. In fact, an opportunity already exists to address this problem: #158. I'll make this link explicit in the opportunity description.

[marcinbauer-ohf]

To the discussion about reusing the assist dialog, I’d like to add that our primary argument for adopting this approach is based on the industry’s UX pattern for interacting with chat or text. In this pattern, there’s a single place for all types of text input from the user, regardless of the output. This design minimizes cognitive load on users even before they decide whether they want to control their home or create something. By asking users to click to access a chat interface located elsewhere (but visually similar) with a different name, just to change their desired action, we introduce unnecessary friction.

The common touchpoint for users interacting with text is a text field. Looking at the history of development of LLM products like ChatGPT, Claude, and others, we can observe that they initially started with a single text field capable of capturing user input and generating a limited text output. Over time, their output capabilities have expanded, and most LLM products now share the same input mechanism. However they offer different modes of operation (as seen with Claude) or allow users to invoke various tools to pinpoint the desired output. I believe we are at the same point with the "Assist" dialog, and we can certainly re-use it and expand it. That said there are design decisions that can make the separation of the "modes" more apparent.

[jlpouffier]

Hey @nielsrowinbik : We decided to move this opportunity back to shaping because we feel the decision of the tools location is needed.
I know you are already coordinating this (With @balloob )
Please keep shaping and ping us once you have a clear proposal!