Chroma: e2e testing library for Polkadot wallet integrations

[preschian]

Project Abstract

Chroma is an end-to-end (E2E) testing library specifically designed for Polkadot wallet interactions. It enables developers to write automated tests that interact with real wallet extensions, providing a comprehensive testing solution for decentralized applications (dApps) in the Polkadot ecosystem.

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• I have read and followed the program guidelines.
• I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
• The software delivered for this grant will be released under an open-source license specified in the application.

[keeganquigley]

Hi @preschian sorry for the delay here. This looks interesting and certainly fills a gap when it comes to wallet integration testing. Can you confirm the following:

• Would it be limited to Chromium browsers?
• Does it have to be run in non-headless mode?
• While the need is real, the target audience may be relatively niche. How would you plan to get the other ecosystem wallet providers to adopt your project?

[ditavia-br]

@preschian Thank you for the application. Do you know this project: https://github.com/Synthetixio/synpress
? I’ve used it before with Metamask, and it works nicely. It can be integrated with Playwright and claims to be flexible enough to support other wallets. Could you evaluate whether it would be possible, or preferable, to integrate with this project instead of creating a new one? Also, what limitations did you find in the other project (https://github.com/ChainSafe/cypress-polkadot-wallet
)?

[preschian]

Hi @keeganquigley and @ditavia-br thank you very much for the response

Would it be limited to Chromium browsers?

Yes, based on Playwright documentation it's limited to Chromium browsers only. But I think this is more than enough, especially since Chromium is still dominantly used by users at the moment.

Does it have to be run in non-headless mode?

It can run in headless mode. More precisely, I haven't tested heavily in headless mode yet. I'll try to reword this grant application document later.

While the need is real, the target audience may be relatively niche. How would you plan to get the other ecosystem wallet providers to adopt your project?

Do you mean wallet providers have to use this library? Actually the main target is dapp developers. Dapp developers use this library to test their dapps. Wallet providers don't need to adopt this library, they're just optional.

But conversely, this library will expand to support other wallet providers in the Polkadot ecosystem. As long as the wallet is open source.

Do you know this project: Synthetixio/synpress

Yes, there are quite a lot of options in other ecosystems. Actually I've been researching for quite a while, and have tried several other libraries. And now trying that library again I'm getting an error:

• cli is still alpha version
• manifest extension error appears

manifest error	unable to run

Could you evaluate whether it would be possible, or preferable, to integrate with this project instead of creating a new one? Also, what limitations did you find in the other project (ChainSafe/cypress-polkadot-wallet)?

My answer might be a bit biased. But since I haven't found a suitable library, I tried to create a new one.

For example with ChainSafe/cypress-polkadot-wallet. Dapps in other ecosystems already use embedded wallets extensively. Because that library modifies the window implementation, that library cannot be used. Some other limitations are in the "Similar Projects in Polkadot Ecosystem" section.

wallet not detected with cypress-polkadot-wallet:

[preschian]

I just tested regarding headless mode, the results look ok. But the results might be different with other wallet extensions or certain conditions that haven't been covered yet.

GitHub actions: https://github.com/preschian/create-dot-app/actions/runs/18152695184
Pull request: https://github.com/preschian/create-dot-app/pull/161/files

[ditavia-br]

@preschian thanks for the reply. I'm not willing to support the development of a completely new tool, since that often leads to yet another open-source project with little usage or maintenance. I like the project and think it’s needed, but it would be better if it were implemented as a plugin or extension to an already well-known and widely used tool. This would help avoid duplicating code that already exists elsewhere and could be reused. There are tools in the EVM ecosystem that work well for end-to-end testing, so it would be great to have them working with Polkadot wallets as well.

[preschian]

Hi @ditavia-br, thanks for the feedback. Even though my proposal might not be approved, I'll probably still maintain it and spread it to our ecosystem. At least I'll use it in projects I'm building like Chaotic or include it in create-dot-app, so the templates are more complete with e2e testing.

As mentioned in previous posts, my problem with other ecosystems is I can't find one that works on new projects. The difference also is that this library plans not to include wallets that are not open-source. Because of reasons related to maintaining that wallet version. Extracting from chrome extensions also isn't necessarily reliable forever.

Besides that, a few days ago I found this https://github.com/TenKeyLabs/dappwright/issues/491, the team from Velocity labs seems to want to use e2e as well. But there are obstacles. I'm planning to try to reach out to them to see if my library has the same problem or not.

[preschian]

hi @ditavia-br 👋🏻

here are some early impressions from several developers about chroma that might be worth considering

• https://forum.polkadot.network/t/kitdot-build-web2-like-apps-on-polkadot/15303/16?u=0xpresc
• https://forum.polkadot.network/t/kitdot-build-web2-like-apps-on-polkadot/15303/18?u=0xpresc

to make it clearer why I decided to create this library:
I haven’t found any existing library from other ecosystems that fits well for Polkadot ecosystem. For example, synpress throws errors on the latest Playwright version, you can see the screenshot of the “manifest error” in my earlier post.

I also noticed that Velocity Labs tried to use an existing library, dappwright, but it still fails in CI/CD.

As far as I know, most existing libraries try to support non-open-source extensions. During my research, I found that extracting Chrome extensions is quite hacky and I’m not a fan of that approach, since it can’t be guaranteed to work forever. On top of that, controlling the wallet version in that flow is tricky, exactly like the issue mentioned here: https://github.com/synpress-io/synpress/pull/1291#issuecomment-3223747304.

So, if we were to integrate Polkadot wallets into an existing ecosystem library, the effort wouldn’t just be adding the wallet, it would also require fixing their core functions. Moreover, if builders in Polkadot use a non-open-source wallet from those libraries and encounter issues, it would result in a poor developer experience.

Meanwhile, in my early version, it’s already possible to connect both Polkadot and Ethereum addresses and run successfully in CI/CD: https://github.com/avalix-labs/chroma-examples/blob/cce24378982ac970699b15cb3e3489d958779939/tests/dapp/hydration.spec.ts#L35-L47

Actually, I don’t mind expanding or integrating into an existing library. If there’s already a library that’s good enough. For instance, AssetHub (Kusama and Paseo) now supports Solidity smart contracts, but it’s not yet available in Thirdweb. In that situation, I’d prefer finding a way to make Thirdweb support AssetHub, because its main functionality works well, and other EVM-compatible parachains like Moonbeam already have support.

[ditavia-br]

@preschian Thanks for the comment. E2E testing in Web3 applications is not a new challenge. My main concern here is the long-term maintenance of the tool. Building a new one would not leverage the community benefits that come from adding a feature to an already widely used project.

Furthermore, the issues you mentioned with Synpress would also occur if you maintained this new tool over time. Each time a dependency releases a new version, whether it’s Playwright or a wallet, you would need to adjust the software to stay compatible. For a tool that’s already in use, you get help from the community with these updates. For instance, there’s already an open PR to support a more recent version of Playwright.

So, while I agree that the problem you’re addressing is important, developing an entirely new tool has a higher chance of becoming an abandoned project in the future than simply adding new features to an existing one. For this reason, I’ll stand by my position and not support building a completely new E2E testing tool. Don't need to be Synpress, can be any tool that is already widely adopted for this problem and can be adjusted to Polkadot ecosystem.

[preschian]

hi @ditavia-br thank you so much for your response.

For instance, there’s already an open PR to support a more recent version of Playwright.

Actually, during my research, this is one of the reasons why I decided to create a new one. If you look, that PR is from one of the core contributors. But after 7 months, there’s still no update. So how much longer do we have to wait? It’s uncertain when it will be fixed. In fact, it should already be tested with Playwright v1.55, because in that version Chrome extension v2 is no longer supported. playwright release notes v1.55

For this reason, I’ll stand by my position and not support building a completely new E2E testing tool. Don't need to be Synpress, can be any tool that is already widely adopted for this problem and can be adjusted to Polkadot ecosystem.

So, what should I do next? I’ve already tried looking for existing tools, but none are compatible with the latest version of Playwright. Even Velocity Labs has tried using tools other than Synpress and still failed. From what I know, most existing tools try to support non-open-source wallets, which causes issues whenever Chromium upgrades its extension manifest version. They use hacky methods like unzipping Chrome extensions, instead of building the extension directly from open-source code.

If there’s already an existing tool that’s good and only supports open-source wallets, please let me know. I’ll try integrating Polkadot wallets there.

My question is: if builders in the Polkadot ecosystem want to use E2E testing for their dApps, what should they use?
If we have to wait for fixes like in Synpress, who knows how long that’ll take? Meanwhile, Chroma is already ready to use.

[ditavia-br]

@preschian Thanks for your answer. Let’s look at it this way: if a well-established project in another ecosystem is already facing maintenance issues, what makes you confident that your project won’t face the same challenges? What is your long-term plan to ensure the sustainability of your project? How do you intend to make it maintainable in the long run and avoid the same problems they encountered? What will you do differently to make your project more sustainable?

[preschian]

Hi @ditavia-br, thanks again for your thoughtful follow-up!

That’s a fair question. Honestly, I also share the same concern. Many tools start strong but fade out when maintenance becomes heavy. So here’s how I plan to make Chroma more sustainable in the long run:

• Focus on compatibility, not complexity. I’m keeping the architecture lightweight and Playwright-first. That means fewer custom wrappers, so every Playwright update only needs minimal patching instead of a rewrite.

• Open contribution model. I’m quietly exploring internship opportunities to bring in Web2 developers from my region and introduce them to Polkadot development. It’s a small step to build a contributor base while helping new engineers learn and get involved in maintaining tools like create-dot-app and Chroma.

• CI/CD testing for each wallet. Once the base APIs are stable, I’ll automate regression tests for major wallet integrations. That helps catch breaking changes faster when wallets or Playwright update.

• Dogfooding across projects. I’ll actively use Chroma in my own projects (like create-dot-app and Chaotic), so it naturally stays up to date. If something breaks, I’ll notice it right away.

The idea is to keep Chroma small, useful, and community-friendly rather than a big monolith that depends on one maintainer.

[ditavia-br]

@preschian I was wondering if it would make sense to include a small maintenance milestone in this proposal, or if that would fall outside your current scope.

Also, I believe that one of the main factors in keeping an open-source project healthy is ensuring it finds real users. If the community finds value in it, sustainability tends to follow naturally. I think the project is valuable, my only suggestion would be to make sure there’s a clear commitment to maintaining the project for some time and to actively engage potential users. Long-term success will depend not only on delivering the code but also on fostering adoption.

If the project is completed successfully, you could consider approaching wallet teams to explore ways of supporting or promoting the work.

[preschian]

hi @ditavia-br 👋🏻 thanks again for the suggestion!

I’ve added a clear Maintenance Commitment section in the updated proposal. For me, maintaining the project after completion is already a given. I’ve been maintaining create-dot-app, another open-source tool I built for Polkadot developers, and I plan to do the same for Chroma for at least 6 to 12 months after delivery.

I fully agree that sustainability depends on real users and adoption, so I’ll keep engaging with dApp and wallet teams to ensure Chroma stays relevant and useful over time.

[semuelle]

Thanks for the clarification. Happy to support.

[ditavia-br]

Happy to support and looking forward to seeing long-term adoption of the software.

[keeganquigley]

Thanks @preschian for the changes, happy to approve also.

[preschian]

Thank you everyone for the approvals! Excited to move forward and keep refining Chroma together 💪

[tien]

Not trying to be a party crasher here, but for DApp E2E testing, won't you just need to inject a signer/headless-extension?

[preschian]

Hi @tien, no worries, I really appreciate the feedback 👌

Actually, injecting a signer is exactly what I try to avoid for this use case.
When doing E2E testing for a dApp, I want the flow to be as close as possible to how a real user interacts, for example:

user clicks "Connect Wallet" → approves wallet provider → selects account → performs on-chain transaction → sees result

But with signer injection, the flow becomes more artificial, like:

inject signer → perform on-chain transaction → result

Each dApp also implements wallet connection differently. For example, some store user data in localStorage or cookies after the “Connect Wallet” step.
Not to mention cross-chain cases and embedded wallets.

So, I prefer leaving the signer logic to the existing wallet provider instead of patching or re-implementing it.
You can see an example here:
https://github.com/avalix-labs/chroma-examples/blob/main/tests/dapp/turtle-cool.spec.ts

[tien]

I believe "Headless-wallet" is the other thing that you are looking for here. It would be this exact same flow you mentioned:

user clicks "Connect Wallet" → approves wallet provider → selects account → performs on-chain transaction → sees result

The signer logic is exactly the same as all wallets uses the same core keyring implementation, but even then, that doesn't matter at all since it's just the InjectedExtension interface from the DApps’ POV.

[preschian]

But we’d still need to create an adapter layer for that “headless-wallet” and maintain one for each supported chain. From a development and maintenance perspective, it’s much easier to rely on existing wallets like Talisman, which already supports Polkadot, Ethereum, and Solana. For now, the main focus is to make things work reliably and stay stable, especially for cross-chain and embedded wallet cases.

That said, if this requirement becomes relevant in the future, I’m very open to collaborating on it 🦾

[tien]

But we’d still need to create an adapter layer for that “headless-wallet” and maintain one for each supported chain.

If we are talking about Polkadot chains, then only one implementation is needed for any number of chains? Which, a basic headless version, should be relatively short and easy to implement, I believe. We are talking about one to two days of work or up to one week max for a well-encapsulated version.

@semuelle @keeganquigley @ditavia-br pardon for asking, but shouldn’t you guys be the ones to make these most basic of evaluations?

[preschian]

it’s much easier to rely on existing wallets like Talisman, which already supports Polkadot, Ethereum, and Solana

hi @tien, perhaps you missed the part I mentioned earlier. I wasn’t referring only to Polkadot, even though that’s the main focus. If you check my very first commit in this PR, you’ll see this already included:

- **Multi-chain Support**: Native support for both Polkadot and Ethereum chains through Talisman integration  
- **Real Extension Testing**: Works with actual wallet extensions rather than modified implementations

What you call a "headless-wallet" is actually quite similar to what some Ethereum projects call a "Mock Wallet". In my opinion, that falls under modified or mocked implementation details. After experimenting with that approach, I found it simpler and more reliable to just use existing wallets.

The main reason, as I mentioned before, is multi-chain and embedded wallet support. This way, if more dApps like Hyperbridge, Hydration, or Turtle appear in the Polkadot ecosystem with multi-chain connections, they can benefit from the same setup.

On top of that, there’s also https://polana.network/. As far as I remember, when I first submitted this proposal, Talisman didn’t yet support Solana. So later, if Solana support is added, I won’t need to dive into its spec or implementation details. I can just update Talisman and adjust the UI flow for Solana.

[tien]

In my opinion, that falls under modified or mocked implementation details

It is not modified/mocked (in the sense that you think) when you are still signing transactions with a real key and via a real "wallet interface”.

I found it simpler and more reliable to just use existing wallets

This is simply not possible. You can make other points, but this one is just not. Like this is the entire implementation of a headless Polkadot wallet version.

[laurogripa]

I've used the Cypress plugin in the past and it worked like a charm. I get that it might be a bit outdated for the newest versions of Cypress, and it also could use a few more features, but I wouldn't discard it. It can even be used as a base for a Playwright alternative.

[preschian]

hi @tien @laurogripa 👋🏻 thank you both for taking the time to respond to this proposal

This is simply not possible

I’m not sure what exactly you meant by “not possible”. Have you checked this repository?
https://github.com/avalix-labs/chroma-examples

It already runs successfully both in CI/CD and locally, and supports multi-chain connections.

Regarding @chainsafe/cypress-polkadot-wallet, this has actually been discussed earlier by one of the curators. You can find my full explanation in this proposal, where I also listed the key points:

The closest comparable project is @chainsafe/cypress-polkadot-wallet, which provides Cypress-based testing for Polkadot wallets. Initially, my research aimed to follow their implementation approach with a similar implementation but adapted for Playwright, as it seemed like a promising foundation for Polkadot wallet testing. However, through direct testing experience with this package, several critical limitations were discovered that ultimately led me to develop Chroma with a different approach:

Maybe you can check the points I mentioned in this proposal about why I decided not to follow the same approach from ChainSafe. I’m very open to any feedback or suggestions.

[tien]

When I said "This is simply not possible", I meant that it's not possible for "Headless wallet" to be "less simple and less reliable" by any definitions.

[tien]

Now to address your other points

Framework Choice: Uses Playwright instead of Cypress, offering better extension support and performance

Like I said before, this is the entire meat of that library, it does not matter at all whether that is being ran by Cypress or Playwright.

Implementation Approach: Does not modify wallet internals like injectedWeb3, maintaining compatibility with real wallet behavior. Additionally, with the ongoing discussion to deprecate signPayload in favor of the new transaction interface (https://forum.polkadot.network/t/toward-an-interoperable-transaction-interface-introducing-createtransaction/15060), Chroma's approach of not modifying injectedWeb3 implementations ensures better future compatibility

You can similarly do the same with whatever supersede signPayload & it would be like another 200 LoCs. Also "real wallet" behaviour is whatever implement InjectedExtension interface & real wallet does inject/modify the global injectedWeb3

Multi-chain Support: Native support for both Polkadot and Ethereum chains through Talisman integration

Other chains also have "headless" wallet for E2E testing, for example: https://github.com/cawabunga/headless-web3-provider

If there isn't for any particular chain, I can guarantee you that implementing one is almost certain to be simpler than to juggle with real wallet internals. And that it would be of great values.

Real Extension Testing: Works with actual wallet extensions rather than modified implementations

As I've mentioned earlier, headless wallet IS real wallet.

Embedded Wallet Compatibility: In my proof of concept with Talisman wallet, I demonstrate support for embedded wallet solutions such as those from Thirdweb or Privy.io, providing comprehensive testing coverage for dApps using embedded wallet integrations

You are not testing DApps, but wallets at this point. DApps have zero exposure to wallets’ internals other than through a fixed interface.

[tien]

I take no pleasure in having to point these out btw

1. I don't want things to appear unwelcoming for builder like yourself
2. I hate even the notion of "punching down"
3. When there's $$$ on the table, me being involved does nothing but to put a target on my head
4. Most importantly, this is simply not my job, cc @semuelle @keeganquigley @ditavia-br

But you have to understand that a treasury funded bounty need to meet certain standards.

[preschian]

You can similarly do the same with whatever supersede signPayload & it would be like another 200 LoCs. Also "real wallet" behaviour is whatever implement InjectedExtension interface & real wallet does inject/modify the global injectedWeb3

I think we’ve found the main point where we agree to disagree on this part.

Maybe for you, a "headless" implementation can still be reliable and simpler. But for now, I personally find that using existing wallets is the simpler approach, especially when it comes to multi-chain cases.
It also means I don’t need to check or handle chain specs outside of the Polkadot ecosystem.

For this case, I prefer maintaining the ui-flow first rather than focusing on the chain-spec.

You are not testing DApps, but wallets at this point

If we strictly think that way, then libraries like Synpress or Dappwright probably wouldn’t have any users at all.
In reality, even Velocity Labs’ Turtle dApp, as I recently found out, is trying out Dappwright for their own use case.
If we follow that line of thought, they should also be looking for a headless alternative to replace @chainsafe/cypress-polkadot-wallet. https://github.com/velocitylabs-org/turtle/pull/499/files#diff-012b982f97a0d326aeec58dd3b789484b05a944d609afe1b8ed5e299fc485f61

[tien]

I think we’ve found the main point where we agree to disagree on this part.
Maybe for you, a "headless" implementation can still be reliable and simpler. But for now, I personally find that using existing wallets is the simpler approach, especially when it comes to multi-chain cases.
It also means I don’t need to check or handle chain specs outside of the Polkadot ecosystem.
For this case, I prefer maintaining the ui-flow first rather than focusing on the chain-spec.

How is handling internals un-specced UI state less flaky than dealing with fixed spec interfaces?

If we strictly think that way, then libraries like Synpress or Dappwright probably wouldn’t have any users at all.

Honestly they shouldn't, if your dapp only deal with specced behaviour & can leverage programmatically controllable wallets for E2E testing, then please do. Tagging @noahjoeris cuz Velocity was mentioned 🙏

[preschian]

How is handling internals un-specced UI state less flaky than dealing with fixed spec interfaces?

Because I lock the wallet version being used instead of always pulling the latest extension release. For example, the current Polkadot-JS version used is v0.61.7, specifically the master-chrome-build.zip package.

Honestly, I think most of the questions you’re raising were already brought up by one of the curators earlier. But from what I see, our discussion here seems to be more about how to implement it rather than how to solve the actual problem.

Personally, I prefer to focus on solving the actual problem first by taking the simplest path. The main issue, in my view, is that there is currently no E2E testing library that supports multi-chain setups for the Polkadot ecosystem. That’s the part I’m focusing on right now.

In the end, I believe most library users don’t really care how the implementation works internally. What matters to them is simply that they can use the E2E testing library to test their dApps effectively.

[tien]

I prefer to focus on solving the actual problem first

This is why I hate having to join the discussion like this, I don't want to discourage this.

In the end, I believe most library users don’t really care how the implementation works internally.

For a $30k, we absolutely should care. We should care regardless, if we don’t care as the "top of the funnel", how will that filter down to end users?

there is currently no E2E testing library that supports multi-chain setups for the Polkadot ecosystem

And wallet integration is just one line item in that equation, which is already easily “solvable," just maybe not matching some preference, hence not a priority. There are definitely more to testing than just wallets (e.g. if you bundle in wallets with other stuffs like Chopstick for example), you are always welcome to explore more.

I just wish that this process was given more guidance from the start, so it didn't build up to be adversarial like this.

[laurogripa]

Hey, @preschian, thanks for your reply!

Maybe you can check the points I mentioned in this proposal about why I decided not to follow the same approach from ChainSafe. I’m very open to any feedback or suggestions.

I read the whole thread and your proposal, which is why I said the Cypress plugin could use some updating and some new features; I'm just not sure about creating a whole new opinionated project. I share some of the same concerns mentioned in the thread, namely usage and maintainability, but also fragmenting collaboration.

However, I also don't want to appear unwelcoming, our devx is already hard as it is. I hope I'm not discouraging you.

[keeganquigley]

Hi everyone, thanks for the high-quality discussion here. We welcome community members to participate. I want to briefly summarize the points raised to ensure we are all on the same page. Correct me if I'm wrong, but:

It seems we have two valid but distinct philosophies on testing being debated:

The "Headless" Approach (highlighted by @tien): Focusing on reliable, fast, and maintainable testing of dApp logic using simulated wallets. The argument is that this covers the vast majority of use cases without the "brittleness" of testing UI elements. Since it might be tricky to maintain with UIs constantly changing, tests might break down the line, etc.

The "fully E2E" Approach (proposed by @preschian): Focusing on integration testing with the actual, "injected" wallet extension. The argument is that there is currently a gap in tooling for developers who need to verify the full user experience, especially given the maintenance issues with existing tools like Synpress.

If I understand correctly, @tien is arguing that the cost of testing this small gap isn't worth the instability it could create.

While @tien makes excellent points regarding the efficiency and stability of headless testing, we believe there is value in funding @preschian’s approach to solve the specific problem of real-world extension interaction. Lowering the barrier to entry for developers who need "true" UI verification is a net positive for the ecosystem.

Since this application has been approved, we are going to proceed with Chroma. However, @preschian, please take @tien's feedback seriously regarding the maintenance challenges of UI-based testing. The success of this project will largely depend on its ability to remain stable even when wallet UIs update.

Thanks for bringing this to our attention and I hope this helps to shed light on the situation from the bounty side.