Skip to content

Commit 55876fc

Browse files
authored
Merge pull request #97 from morpho-org/feature/integ-3967-refactor-datasources
Refactor: Repo
2 parents 8d4a2ff + f1daa34 commit 55876fc

128 files changed

Lines changed: 11586 additions & 4666 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/agents/liquidation-engineer.md

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,20 +25,28 @@ You are a domain expert on the Morpho Blue liquidation bot and the Morpho Blue p
2525

2626
### Liquidation bot architecture
2727
- Read and understand the CLAUDE.md at the project root for the full architecture overview
28+
- Six packages: `apps/config`, `apps/client`, `apps/data-providers`, `apps/hyperindex`, `apps/liquidity-venues`, `apps/pricers`
2829
- Executor contract patterns: how `LiquidationEncoder` builds batched calldata via `executooor-viem`
2930
- Multi-chain execution: how `script.ts` launches one bot per chain config
3031

3132
### Venue integration patterns
32-
- `LiquidityVenue` interface: `supportsRoute` and `convert`
33+
- `LiquidityVenue` interface in `apps/liquidity-venues/src/liquidityVenue.ts`: `supportsRoute` and `convert`
3334
- How venues are ordered and tried sequentially in `apps/config/src/config.ts`
34-
- Factory pattern: config exports string names, client owns implementations
35+
- Factory pattern in `apps/liquidity-venues/src/factory.ts`: config exports string names, implementation package owns classes
3536
- Common patterns: ERC4626 unwrapping, DEX swaps, wrapper unwrapping, Pendle PT redemption
3637

3738
### Pricer integration patterns
38-
- `Pricer` interface: `price(client, asset)` returns USD price
39-
- Factory pattern matching venues
39+
- `Pricer` interface in `apps/pricers/src/pricer.ts`: `price(client, asset)` returns USD price
40+
- Factory pattern in `apps/pricers/src/factory.ts` matching venues
4041
- Caching strategies (see DefiLlama pricer)
4142

43+
### Data provider integration patterns
44+
- `DataProvider` interface in `apps/data-providers/src/dataProvider.ts`: `init`, `fetchMarkets`, `fetchLiquidatablePositions`
45+
- Factory pattern in `apps/data-providers/src/factory.ts`: creates shared instances across chains
46+
- HyperIndex provider: self-hosted Envio indexer with GraphQL API
47+
- MorphoApi provider: queries the Morpho API directly
48+
- Config in `apps/config/src/dataProviders/hyperindex.ts`: chain deployment blocks and factory addresses
49+
4250
### EVM/DeFi best practices
4351
- Token decimal handling: always use `parseUnits`/`formatUnits`, never manual exponentiation
4452
- BigInt precision: `WAD = 10^18`, `wMulDown` from `utils/maths.ts`, rounding direction

.claude/agents/reviewer.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -47,11 +47,12 @@ Read each changed file and check for violations. Focus areas:
4747
- Chain ID assumptions are documented
4848
- New chain additions include all required config fields
4949

50-
**Venue/pricer patterns (P1)**:
51-
- New venues implement the `LiquidityVenue` interface (`supportsRoute` + `convert`)
52-
- New pricers implement the `Pricer` interface (`price`)
53-
- Registered in the factory switch statement
54-
- Type name added to the union type in config
50+
**Venue/pricer/data-provider patterns (P1)**:
51+
- New venues implement the `LiquidityVenue` interface (`supportsRoute` + `convert`) in `apps/liquidity-venues/src/`
52+
- New pricers implement the `Pricer` interface (`price`) in `apps/pricers/src/`
53+
- New data providers implement the `DataProvider` interface (`init`, `fetchMarkets`, `fetchLiquidatablePositions`) in `apps/data-providers/src/`
54+
- Registered in the respective factory switch statement (`apps/liquidity-venues/src/factory.ts`, `apps/pricers/src/factory.ts`, `apps/data-providers/src/factory.ts`)
55+
- Type name added to the union type in config (`apps/config/src/types.ts`)
5556
- Config constants exported from config package
5657
- Tests added
5758

@@ -67,8 +68,8 @@ Read each changed file and check for violations. Focus areas:
6768
- Individual venue/pricer failures don't crash the bot
6869

6970
**Test coverage (P2)**:
70-
- New venues have liquidity venue tests
71-
- New pricers have pricer tests
71+
- New venues have tests in `apps/liquidity-venues/test/vitest/`
72+
- New pricers have tests in `apps/pricers/test/vitest/`
7273
- Tests follow existing patterns (use `encoderTest` for venues, `test` fixture for pricers)
7374

7475
### 4. Output format

.claude/commands/add-chain.md

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,10 @@ Ask the user for:
1717
4. **Vault whitelist** — list of vault addresses, or `"morpho-api"` for API-based discovery
1818
5. **Which liquidity venues to enable** (ordered list from existing `LiquidityVenueName` values)
1919
6. **Which pricers to enable** (ordered list from existing `PricerName` values, optional)
20-
7. **Flashbots toggle** (`useFlashbots`)
21-
8. **Block interval** (optional)
22-
9. **Liquidation buffer bps** (optional, default 50)
20+
7. **Which data provider to use** (`"morphoApi"` or `"hyperIndex"`)
21+
8. **Flashbots toggle** (`useFlashbots`)
22+
9. **Block interval** (optional)
23+
10. **Liquidation buffer bps** (optional, default 50)
2324

2425
## Steps
2526

@@ -73,6 +74,7 @@ Add an entry to `chainConfigs` in `apps/config/src/config.ts`:
7374
chain: <chainImport>,
7475
wNative: "<wNativeAddress>",
7576
options: {
77+
dataProvider: "<dataProviderName>",
7678
vaultWhitelist: <whitelist>,
7779
additionalMarketsWhitelist: [],
7880
liquidityVenues: [<ordered venue list>],
@@ -97,7 +99,22 @@ Check and update per-chain config files that have `Record<number, ...>` mappings
9799

98100
For each enabled venue/pricer, add the chain ID entry with the correct addresses. Ask the user for any chain-specific addresses needed.
99101

100-
### 4. Reminder
102+
### 4. Add HyperIndex config (if using hyperIndex data provider)
103+
104+
Add the chain to `apps/config/src/dataProviders/hyperindex.ts`:
105+
106+
```typescript
107+
[<chainId>]: {
108+
morphoStartBlock: <block>,
109+
metaMorphoFactoryStartBlock: <block>,
110+
adaptiveCurveIrmStartBlock: <block>,
111+
preLiquidationFactoryStartBlock: <block>,
112+
},
113+
```
114+
115+
Ask the user for the deployment block numbers.
116+
117+
### 5. Reminder
101118

102119
After scaffolding, remind the user:
103120
- Set up environment variables:
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Add Data Provider
2+
3+
Interactive workflow for scaffolding a new data provider. Follow CLAUDE.md "How to Add a New Data Provider" exactly.
4+
5+
## Input
6+
7+
Ask the user for:
8+
1. **Provider name** (camelCase, e.g. `theGraph`) — this becomes the `DataProviderName` union member and directory name
9+
2. **Whether the provider needs config constants** (deployment blocks, subgraph URLs, etc.)
10+
3. **Whether the provider needs an `init()` step** (e.g. spinning up infrastructure, waiting for backfill)
11+
12+
## Steps
13+
14+
### 1. Config package (`apps/config`)
15+
16+
**a) Add to union type** in `apps/config/src/types.ts`:
17+
- Add `"<name>"` to the `DataProviderName` union type (maintain alphabetical order within the union)
18+
19+
**b) If config constants are needed**, create `apps/config/src/dataProviders/<name>.ts`:
20+
- Follow the pattern in `apps/config/src/dataProviders/hyperindex.ts` for per-chain config records
21+
- Use `Record<number, ...>` keyed by chain ID
22+
- Export from `apps/config/src/dataProviders/index.ts`
23+
24+
### 2. Data Providers package (`apps/data-providers`)
25+
26+
**a) Create provider implementation** at `apps/data-providers/src/<name>/index.ts`:
27+
28+
```typescript
29+
import type { AccrualPosition } from "@morpho-org/blue-sdk";
30+
import type { Account, Address, Chain, Client, Hex, Transport } from "viem";
31+
32+
import type { DataProvider } from "../dataProvider";
33+
34+
export class <ClassName> implements DataProvider {
35+
async init(): Promise<void> {
36+
// TODO: implement initialization (or remove if not needed)
37+
}
38+
39+
async fetchMarkets(
40+
client: Client<Transport, Chain, Account>,
41+
vaults: Address[],
42+
): Promise<Hex[]> {
43+
// TODO: implement market fetching
44+
throw new Error("Not implemented");
45+
}
46+
47+
async fetchLiquidatablePositions(
48+
client: Client<Transport, Chain, Account>,
49+
marketIds: Hex[],
50+
): Promise<AccrualPosition[]> {
51+
// TODO: implement position fetching
52+
throw new Error("Not implemented");
53+
}
54+
}
55+
```
56+
57+
Replace `<ClassName>` with an appropriate PascalCase class name (e.g. `TheGraphDataProvider`).
58+
59+
**b) Register in factory** at `apps/data-providers/src/factory.ts`:
60+
- Add import for the new class
61+
- Add `case "<name>":` to the switch returning `new <ClassName>()`
62+
63+
**c) Add re-export** in `apps/data-providers/src/index.ts`:
64+
- Add `export * from "./<name>";`
65+
66+
### 3. Reminder
67+
68+
After scaffolding, remind the user:
69+
- Set `options.dataProvider` to `"<name>"` in relevant chain configs in `apps/config/src/config.ts`
70+
- Data providers are multi-chain: a single instance is shared across all chains using that provider
71+
- The factory calls `init()` once before the provider is used — use it for any async setup
72+
- Run `/test` to validate the scaffold compiles and tests pass

.claude/commands/add-pricer.md

Lines changed: 10 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,9 @@ Ask the user for:
2222
- Include all chains from `apps/config/src/config.ts`
2323
- Export from `apps/config/src/pricers/index.ts`
2424

25-
### 2. Client package (`apps/client`)
25+
### 2. Pricers package (`apps/pricers`)
2626

27-
**a) Create pricer implementation** at `apps/client/src/pricers/<name>/index.ts`:
27+
**a) Create pricer implementation** at `apps/pricers/src/<name>/index.ts`:
2828

2929
```typescript
3030
import type { Account, Address, Chain, Client, Transport } from "viem";
@@ -44,31 +44,26 @@ export class <ClassName> implements Pricer {
4444

4545
Replace `<ClassName>` with an appropriate PascalCase class name (e.g. `PythPricer`).
4646

47-
**b) Register in factory** at `apps/client/src/pricers/factory.ts`:
47+
**b) Register in factory** at `apps/pricers/src/factory.ts`:
4848
- Add import for the new class
4949
- Add `case "<name>":` to the switch returning `new <ClassName>()`
5050

51-
**c) Add re-export** in `apps/client/src/pricers/index.ts`:
51+
**c) Add re-export** in `apps/pricers/src/index.ts`:
5252
- Add `export * from "./<name>";`
5353

5454
### 3. Test scaffold
5555

56-
Create `apps/client/test/vitest/pricers/<name>.test.ts`:
56+
Create `apps/pricers/test/vitest/<name>.test.ts`:
5757

5858
```typescript
59-
import { describe, expect, beforeEach } from "vitest";
59+
import { describe, expect } from "vitest";
6060

61-
import { <ClassName> } from "../../../src/pricers";
62-
import { test } from "../../setup.js";
61+
import { <ClassName> } from "../../src";
62+
import { test } from "../setup.js";
6363

6464
describe("<name> pricer", () => {
65-
let pricer: <ClassName>;
66-
67-
beforeEach(() => {
68-
pricer = new <ClassName>();
69-
});
70-
71-
test("should test price", async ({ client }) => {
65+
test("should return a price", async ({ client }) => {
66+
const pricer = new <ClassName>();
7267
// TODO: test with known assets
7368
// Example: expect(await pricer.price(client, USDC)).toBeCloseTo(1, 3);
7469
expect(true).toBe(true);

.claude/commands/add-venue.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -22,15 +22,15 @@ Ask the user for:
2222
- Include all chains from `apps/config/src/config.ts`
2323
- Export from `apps/config/src/liquidityVenues/index.ts`
2424

25-
### 2. Client package (`apps/client`)
25+
### 2. Liquidity Venues package (`apps/liquidity-venues`)
2626

27-
**a) Create venue implementation** at `apps/client/src/liquidityVenues/<name>/index.ts`:
27+
**a) Create venue implementation** at `apps/liquidity-venues/src/<name>/index.ts`:
2828

2929
```typescript
3030
import type { ExecutorEncoder } from "executooor-viem";
3131
import type { Address } from "viem";
3232

33-
import type { ToConvert } from "../../utils/types";
33+
import type { ToConvert } from "../types";
3434
import type { LiquidityVenue } from "../liquidityVenue";
3535

3636
export class <ClassName> implements LiquidityVenue {
@@ -39,7 +39,7 @@ export class <ClassName> implements LiquidityVenue {
3939
throw new Error("Not implemented");
4040
}
4141

42-
async convert(encoder: ExecutorEncoder, toConvert: ToConvert): Promise<ToConvert> {
42+
async convert(executor: ExecutorEncoder, toConvert: ToConvert): Promise<ToConvert> {
4343
// TODO: implement conversion logic
4444
throw new Error("Not implemented");
4545
}
@@ -48,21 +48,21 @@ export class <ClassName> implements LiquidityVenue {
4848

4949
Replace `<ClassName>` with an appropriate PascalCase class name (e.g. `CurveV2Venue`).
5050

51-
**b) Register in factory** at `apps/client/src/liquidityVenues/factory.ts`:
51+
**b) Register in factory** at `apps/liquidity-venues/src/factory.ts`:
5252
- Add import for the new class
5353
- Add `case "<name>":` to the switch returning `new <ClassName>()`
5454

55-
**c) Add re-export** in `apps/client/src/liquidityVenues/index.ts`:
55+
**c) Add re-export** in `apps/liquidity-venues/src/index.ts`:
5656
- Add `export * from "./<name>";`
5757

5858
### 3. Test scaffold
5959

60-
Create `apps/client/test/vitest/liquidityVenues/<name>.test.ts`:
60+
Create `apps/liquidity-venues/test/vitest/<name>.test.ts`:
6161

6262
```typescript
6363
import { describe, expect } from "vitest";
64-
import { encoderTest } from "../../setup.js";
65-
import { <ClassName> } from "../../../src/liquidityVenues/index.js";
64+
import { encoderTest } from "../setup.js";
65+
import { <ClassName> } from "../../src/index.js";
6666

6767
describe("<name> liquidity venue", () => {
6868
const liquidityVenue = new <ClassName>();

.claude/skills/review/SKILL.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -51,11 +51,12 @@ For each file, check for violations of CLAUDE.md standards. Focus areas:
5151
- Chain ID assumptions are documented
5252
- New chain additions include all required config fields
5353

54-
**Liquidity Venue / Pricer Patterns**:
55-
- New venues implement the `LiquidityVenue` interface
56-
- New pricers implement the `Pricer` interface
57-
- Registered in the factory switch statement
58-
- Type name added to the union type in config
54+
**Liquidity Venue / Pricer / Data Provider Patterns**:
55+
- New venues implement the `LiquidityVenue` interface in `apps/liquidity-venues/src/`
56+
- New pricers implement the `Pricer` interface in `apps/pricers/src/`
57+
- New data providers implement the `DataProvider` interface in `apps/data-providers/src/`
58+
- Registered in the respective factory switch statement
59+
- Type name added to the union type in config (`apps/config/src/types.ts`)
5960
- Config constants exported from config package
6061
- Tests added
6162

.github/workflows/test.yml

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ jobs:
5454
env:
5555
RPC_URL_1: ${{ secrets.RPC_URL_1 }}
5656

57-
bot:
57+
client:
5858
runs-on: ubuntu-latest
5959
steps:
6060
- name: Checkout code
@@ -73,6 +73,32 @@ jobs:
7373
with:
7474
version: stable
7575
- name: Run liquidation execution tests
76-
run: pnpm test:bot
76+
run: pnpm test:client
77+
env:
78+
RPC_URL_1: ${{ secrets.RPC_URL_1 }}
79+
80+
hyperindex:
81+
runs-on: ubuntu-latest
82+
timeout-minutes: 30
83+
steps:
84+
- name: Checkout code
85+
uses: actions/checkout@v4
86+
- name: Set up pnpm
87+
uses: pnpm/action-setup@v4
88+
- name: Set up Node.js
89+
uses: actions/setup-node@v4
90+
with:
91+
node-version-file: .nvmrc
92+
cache: pnpm
93+
- name: Install dependencies
94+
run: pnpm i --frozen-lockfile
95+
- name: Install Foundry
96+
uses: foundry-rs/foundry-toolchain@v1
97+
with:
98+
version: stable
99+
- name: Build config package
100+
run: pnpm build:config
101+
- name: Run hyperindex tests
102+
run: pnpm test:hyperindex
77103
env:
78104
RPC_URL_1: ${{ secrets.RPC_URL_1 }}

0 commit comments

Comments
 (0)