blocktx

Author: carbolymer

cardano-rpc/docs/node-kernel-access/plan-fetchblock-tx-phases.md

@@ -0,0 +1,165 @@
+# FetchBlock tx deserialisation phases
+
+Populates `Block.body.tx` in the FetchBlock response.
+Each phase compiles independently and can be reviewed as a separate commit.
+
+## Background
+
+FetchBlock currently returns `nativeBytes` (raw CBOR), `header` (slot, hash, height), and `timestamp`.
+The `body.tx` field is empty.
+The proto `Tx` message has 14 fields - implementing them all at once is too large to review.
+This plan breaks the work into incremental phases, each adding a subset of fields.
+
+The parsed block is already available via `Consensus.GetBlock` in the same `getBlockComponent` call.
+`fromConsensusBlock` + `getBlockTxs` gives `[Tx era]` - no new consensus APIs needed.
+
+## Existing building blocks in Type.hs
+
+- `txOutToUtxoRpcTxOutput` - TxOutput (address, coin, assets, datum, script)
+- `scriptDataToUtxoRpcPlutusData` - PlutusData
+- `simpleScriptToUtxoRpcNativeScript` - NativeScript
+- `referenceScriptToUtxoRpcScript` - Script (reference scripts)
+- `utxoRpcBigIntToInteger` - BigInt (inverse)
+- `scriptWitnessIndexToRedeemerPurpose` - redeemer purpose mapping
+- `mkProtoRedeemer` - Redeemer construction
+
+Missing: TxInput, Withdrawal, Certificate, WitnessSet, AuxData, Collateral, Multiasset (for mint), GovernanceActionProposal, and the top-level `Tx era -> Proto Tx` composition.
+
+## Phase 1: Scaffold - extract txs from block, populate hash + fee
+
+Minimal end-to-end wiring with the simplest fields.
+
+- Change `fetchBlock` to also return the parsed `BlockInMode` (via `fromConsensusBlock` on `GetBlock`)
+- In `fetchBlockMethod`, call `getBlockTxs` to get `[Tx era]`
+- For each tx, populate:
+  - `hash` - `getTxId` serialised to raw bytes
+  - `fee` - from `txFee` on the tx body
+  - `successful` - `True` (all blocks in ChainDB have validated txs; phase 1 does not distinguish collateral-return scripts)
+- Set `Block.body.tx` on the response
+- Add an E2E test asserting tx count, hash, and fee for a submitted tx
+
+Build: `cabal build cardano-rpc`
+
+## Phase 2: Inputs, outputs, reference inputs
+
+The core spending data. Reuses existing `txOutToUtxoRpcTxOutput`.
+
+- Add `txInputToProto :: TxIn -> Proto TxInput` (tx_hash + output_index, without as_output or redeemer - those come later)
+- Map `txIns` -> `Tx.inputs`
+- Map `txOuts` -> `Tx.outputs` (reuse `txOutToUtxoRpcTxOutput`)
+- Map `txInsReference` -> `Tx.reference_inputs`
+- Extend E2E test to assert input/output fields
+
+Build: `cabal build cardano-rpc`
+
+## Phase 3: Validity, mint, withdrawals
+
+Simple scalar/list fields.
+
+- Map `txValidityLowerBound` + `txValidityUpperBound` -> `Tx.validity` (TxValidity: start, ttl)
+- Map `txMintValue` -> `Tx.mint` (repeated Multiasset: policy_id + assets)
+- Map `txWithdrawals` -> `Tx.withdrawals` (repeated Withdrawal: reward_account + coin, no redeemer yet)
+- Extend E2E test with a minting tx
+
+Build: `cabal build cardano-rpc`
+
+## Phase 4: Collateral
+
+- Map `txInsCollateral` -> `Collateral.collateral`
+- Map `txReturnCollateral` -> `Collateral.collateral_return`
+- Map `txTotalCollateral` -> `Collateral.total_collateral`
+- Set `Tx.collateral`
+- `Tx.successful` - refine: check `IsValid` flag on Alonzo+ txs
+
+Build: `cabal build cardano-rpc`
+
+## Phase 5: Witnesses
+
+- Map vkey witnesses -> `WitnessSet.vkeywitness`
+- Map scripts (Plutus + native) -> `WitnessSet.script`
+- Map plutus datums -> `WitnessSet.plutus_datums` (reuse `scriptDataToUtxoRpcPlutusData`)
+- Map redeemers -> `WitnessSet.redeemers` (reuse `mkProtoRedeemer`)
+- Map bootstrap witnesses -> `WitnessSet.bootstrapWitnesses`
+- Wire redeemers to `TxInput.redeemer` and `Withdrawal.redeemer`
+
+Build: `cabal build cardano-rpc`
+
+## Phase 6: Certificates
+
+The largest proto message (19 oneof variants).
+
+- Map each `TxCert era` variant to the corresponding `Certificate` oneof
+- Pre-Conway certs: stake registration/deregistration, delegation, pool registration/retirement, genesis delegation, MIR
+- Conway+ certs: reg, unreg, vote deleg, stake vote deleg, committee hot/cold, DRep reg/unreg/update
+- Wire certificate redeemers
+
+Build: `cabal build cardano-rpc`
+
+## Phase 7: Auxiliary data + governance proposals
+
+- Map tx metadata -> `AuxData.metadata`
+- Map auxiliary scripts -> `AuxData.scripts`
+- Map governance action proposals -> `Tx.proposals` (Conway+)
+- This completes all 14 Tx fields
+
+Build: `cabal build cardano-rpc`
+
+## Phase dependency graph
+
+```
+Phase 1 (scaffold: hash, fee, successful)
+  |
+  v
+Phase 2 (inputs, outputs, reference inputs)
+  |
+  v
+Phase 3 (validity, mint, withdrawals)
+  |
+  v
+Phase 4 (collateral, successful refinement)
+  |
+  v
+Phase 5 (witnesses + redeemer wiring)
+  |
+  v
+Phase 6 (certificates)
+  |
+  v
+Phase 7 (auxiliary data, governance proposals)
+```
+
+## Design decisions
+
+- **Incremental field population.**
+  Each phase adds fields to the same `Tx` proto message.
+  Unset proto fields default to empty/zero, so partial responses are valid.
+
+- **No new consensus APIs.**
+  `fromConsensusBlock` + `getBlockTxs` already exist in cardano-api.
+  The work is entirely in the cardano-rpc mapping layer.
+
+- **Reuse existing conversions.**
+  `txOutToUtxoRpcTxOutput`, `scriptDataToUtxoRpcPlutusData`, `mkProtoRedeemer` etc. are already battle-tested in the Query and Submit handlers.
+
+- **Witnesses and redeemers split from inputs/withdrawals.**
+  Phase 2-3 populate inputs and withdrawals without redeemers.
+  Phase 5 adds the witness set and wires redeemers back to their inputs/withdrawals.
+  This avoids a large cross-cutting change.
+
+- **Certificates are a separate phase.**
+  The Certificate oneof has 19 variants spanning pre-Conway and Conway eras.
+  Isolating it keeps reviews focused.
+
+- **Dijkstra safety.**
+  `conwayEraOnwardsConstraints` crashes at runtime for Dijkstra.
+  Phases 6-7 (certificates, governance proposals) must use
+  `caseShelleyToBabbageOrConwayOrDijkstra` and pattern match on concrete
+  `ConwayEraOnwards` constructors in the right arm.
+  The cardano-api lenses `proposalProceduresTxBodyL` and `votingProceduresTxBodyL`
+  use `conwayEraOnwardsConstraints` internally - avoid them, go through the ledger
+  tx body directly with explicit era constraints.
+
+- **Work through the ledger tx directly.**
+  `ShelleyTxBody` is deprecated.
+  Access the ledger `Tx` via the `ShelleyTx` constructor and use ledger lenses
+  (`bodyTxL`, `witsTxL`, `auxDataTxL`, `isValidTxL`) rather than cardano-api wrappers.

cardano-rpc/src/Cardano/Rpc/Proto/Api/UtxoRpc/Sync.hs

@@ -5,15 +5,20 @@ module Cardano.Rpc.Proto.Api.UtxoRpc.Sync
   ( module Proto.Utxorpc.V1beta.Sync.Sync
   , module Proto.Utxorpc.V1beta.Sync.Sync_Fields
   , module Proto.Utxorpc.V1beta.Cardano.Cardano
-  , header
+  , module Proto.Utxorpc.V1beta.Cardano.Cardano_Fields
   )
 where
 
 import Network.GRPC.Common
 import Network.GRPC.Common.Protobuf
 
 import Proto.Utxorpc.V1beta.Cardano.Cardano
-import Proto.Utxorpc.V1beta.Cardano.Cardano_Fields (header)
+import Proto.Utxorpc.V1beta.Cardano.Cardano_Fields hiding
+  ( hash
+  , height
+  , slot
+  , timestamp
+  )
 import Proto.Utxorpc.V1beta.Sync.Sync
 import Proto.Utxorpc.V1beta.Sync.Sync_Fields
 

cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Predicate.hs

@@ -4,6 +4,7 @@
 
 module Cardano.Rpc.Server.Internal.UtxoRpc.Predicate
   ( matchesUtxoPredicate
+  , exactAddressPredicate
   , extractAddressesFromPredicate
   , matchesAddressPattern
   , matchesAssetPattern
@@ -24,8 +25,10 @@ import Cardano.Rpc.Proto.Api.UtxoRpc.Query qualified as UtxoRpc
 import RIO hiding (toList)
 
 import Data.ByteString qualified as BS
+import Data.ProtoLens (defMessage)
 import Data.Set qualified as Set
 import GHC.IsList
+import Network.GRPC.Spec (Proto (..))
 
 -- | Check if a UTxO entry matches a 'UtxoPredicate'.
 -- All present fields are combined with AND logic.
@@ -89,6 +92,20 @@ matchesAddressPattern pat addr =
         _ -> BS.null $ pat ^. UtxoRpc.delegationPart
     _ -> BS.null $ pat ^. UtxoRpc.delegationPart
 
+-- | A 'UtxoPredicate' matching UTxOs at the exact address.
+exactAddressPredicate
+  :: IsCardanoEra era
+  => AddressInEra era
+  -> Proto UtxoRpc.UtxoPredicate
+exactAddressPredicate address =
+  Proto $
+    defMessage
+      & UtxoRpc.match
+        .~ ( defMessage
+               & UtxoRpc.cardano
+                 .~ (defMessage & UtxoRpc.address .~ (defMessage & UtxoRpc.exactAddress .~ serialiseToRawBytes address))
+           )
+
 -- | Serialise a 'PaymentCredential' to raw bytes (the key or script hash).
 serialisePaymentCredential :: PaymentCredential -> ByteString
 serialisePaymentCredential (PaymentCredentialByKey h) = serialiseToRawBytes h

cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Sync.hs

@@ -15,14 +15,15 @@ import Cardano.Rpc.Proto.Api.UtxoRpc.Sync qualified as U5c
 import Cardano.Rpc.Server.Internal.Error
 import Cardano.Rpc.Server.Internal.Monad
 import Cardano.Rpc.Server.Internal.Tracing ()
+import Cardano.Rpc.Server.Internal.UtxoRpc.Type (txToUtxoRpcTx)
 import Cardano.Rpc.Server.NodeKernelAccess
 
 import RIO
 
 import Data.ByteString qualified as BS
 import Data.ProtoLens (defMessage)
 import Data.Time.Clock.POSIX (utcTimeToPOSIXSeconds)
-import Network.GRPC.Spec (GrpcError (GrpcInternal, GrpcInvalidArgument, GrpcNotFound), Proto)
+import Network.GRPC.Spec (GrpcError (GrpcInternal, GrpcInvalidArgument, GrpcNotFound), Proto (..))
 
 -- | Handle the @FetchBlock@ SyncService RPC method.
 -- Fetches a block from ChainDB by slot and header hash.
@@ -51,13 +52,15 @@ fetchBlockMethod request = do
   headerHash <-
     deserialiseFromRawBytes (proxyToAsType (Proxy @(Hash BlockHeader))) hashBytes
       & either (const throwInvalidHash) pure
-  (rawBytes, BlockNo height) <-
+  (rawBytes, BlockInMode era block) <-
     fetchBlock nodeKernelAccess slot headerHash >>= maybe throwNotFound pure
   eraHistory <- readEraHistory
   timestampMs <-
     slotToUTCTime systemStart eraHistory slot
       & either (const throwPastHorizon) (pure . round . (* 1000) . utcTimeToPOSIXSeconds)
-  let blockHeader =
+  let BlockHeader _ _ (BlockNo height) = getBlockHeader block
+      txs = forEraInEon era [] $ \sbe -> Proto . txToUtxoRpcTx sbe <$> getBlockTxs block
+      blockHeader =
         defMessage
           & U5c.slot .~ unSlotNo slot
           & U5c.hash .~ hashBytes
@@ -68,7 +71,6 @@ fetchBlockMethod request = do
         .~ ( defMessage
                & U5c.nativeBytes .~ rawBytes
                & U5c.cardano . U5c.header .~ blockHeader
+               & U5c.cardano . U5c.body . U5c.tx .~ txs
                & U5c.cardano . U5c.timestamp .~ timestampMs
            )
-
--- TODO: cardano.body.tx - needs full block deserialisation + UTxO RPC tx mapping

cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type.hs

@@ -15,6 +15,8 @@ module Cardano.Rpc.Server.Internal.UtxoRpc.Type
   , txInTxOutToAnyUtxoData
   , anyUtxoDataUtxoRpcToUtxo
   , txOutToUtxoRpcTxOutput
+  , txToUtxoRpcTx
+  , txoRefUtxoRpcToTxIn
   , utxoRpcTxOutputToTxOut
   , protocolParamsToUtxoRpcPParams
   , simpleScriptToUtxoRpcNativeScript
@@ -661,6 +663,23 @@ utxoRpcTxOutputToTxOut txOutput = do
       datum
       referenceScript
 
+-- | Convert a transaction from a fetched block to the UTxO RPC 'UtxoRpc.Tx' message.
+-- Populates hash, fee and successful.
+-- TODO: inputs, outputs, certificates, withdrawals, mint, reference inputs,
+-- witnesses, collateral, validity, auxiliary data and proposals are not mapped yet.
+txToUtxoRpcTx
+  :: ShelleyBasedEra era
+  -> Tx era
+  -> UtxoRpc.Tx
+txToUtxoRpcTx sbe (ShelleyTx _ ledgerTx) =
+  shelleyBasedEraConstraints sbe $
+    defMessage
+      & U5c.hash .~ serialiseToRawBytes (fromShelleyTxId (L.txIdTx ledgerTx))
+      & U5c.fee .~ getProto (inject (ledgerTx ^. L.bodyTxL . L.feeTxBodyL))
+      -- all transactions in a block stored in ChainDB have been validated;
+      -- TODO: reflect the IsValid flag for collateral-consuming transactions
+      & U5c.successful .~ True
+
 utxoRpcBigIntToInteger
   :: forall m
    . HasCallStack

cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess.hs

@@ -78,7 +78,8 @@ grabNodeKernelAccess =
     Just nodeKernelAccess ->
       pure nodeKernelAccess
 
--- | Fetch a raw block and its block number from ChainDB by slot and header hash.
+-- | Fetch a raw block and its parsed era-contextualised form from ChainDB
+-- by slot and header hash.
 fetchBlock
   :: MonadIO m
   => NodeKernelAccess
@@ -87,9 +88,9 @@ fetchBlock
   -- ^ Block slot number
   -> Hash BlockHeader
   -- ^ Block header hash
-  -> m (Maybe (ByteString, BlockNo))
-  -- ^ Raw CBOR bytes and block number, or 'Nothing' if not found
+  -> m (Maybe (ByteString, BlockInMode))
+  -- ^ Raw CBOR bytes and the block in era context, or 'Nothing' if not found
 fetchBlock NodeKernelAccess{chainDb} slot (HeaderHash shortHash) = do
   let point = Consensus.RealPoint slot (Consensus.OneEraHash shortHash)
-      component = (,) <$> fmap BSL.toStrict Consensus.GetRawBlock <*> fmap Consensus.blockNo Consensus.GetBlock
+      component = (,) <$> fmap BSL.toStrict Consensus.GetRawBlock <*> fmap fromConsensusBlock Consensus.GetBlock
   liftIO $ Consensus.getBlockComponent chainDb component point