# CosmWasm Connect SDK
Querying Connect prices using CosmWasm and Connect SDK
**Building with Connect? Join our [Discord](https://discord.gg/amAgf9Z39w)**!
The Connect SDK provides bindings for the `x/oracle` and `x/marketmap` modules.
## Adding Connect SDK To Your Contract
The version of the Connect SDK for your contract depends on the Connect protocol version of the chain.
| Connect Protocol Version | Connect SDK Version |
| ------------------------ | ------------------- |
| v1.x | v0.1.0 |
| v2.x | v0.2.0 |
How to find the Connect Protocol version
The protocol version of Connect can be found on the chain by either:
1. Checking the chain's `go.mod` file.
2. Checking the required version in the [quickstart](../validators/quickstart#run-connect-sidecar)
Add the following line to the `dependencies` section of your `Cargo.toml`:
`connect-sdk = { git = "https://github.com/skip-mev/connect-sdk", tag = "CONNECT SDK VERSION HERE", package = "connect-sdk" }`
## Safely Accessing Price Data
The following checks should be made before utilizing a price from Connect in a contract:
1. currency-pair exists within the `x/oracle` and `x/marketmap` modules.
2. currency-pair is `enabled` within the `x/marketmap`.
3. price `block_height` is not older than a few blocks.
4. price nonce is not 0.
### Code Example
This code example is for `v0.1.0` of the Connect SDK.
```rust contract.rs
use connect_sdk::bindings::query::ConnectQuery;
use connect_sdk::bindings::querier::ConnectQuerier;
use connect_sdk::bindings::marketmap::types::CurrencyPair;
use cosmwasm_std::{Deps, Env, StdResult, Int128, StdError};
fn do_something_with_price(
deps: Deps,
env: Env,
currency_pair: CurrencyPair
) -> StdResult {
let connect_querier = ConnectQuerier::new(&deps.querier);
let base = currency_pair.base.to_uppercase();
let quote = currency_pair.quote.to_uppercase();
// Check if the market exists and is enabled
let market = connect_querier.get_marketmap_market(base.clone(), quote.clone())?;
if !market.market.ticker.enabled {
return Err(StdError::generic_err("market is not enabled"));
}
// Check price validity
let price_response = connect_querier.get_oracle_price(base, quote)?;
if price_response.nonce == 0 {
return Err(StdError::generic_err("price has never been updated"));
}
let max_price_age: u64 = 3; // adjust based on appetite for price freshness
let price_age = env.block.height - price_response.price.block_height.unwrap();
if price_age > max_price_age {
return Err(StdError::generic_err("price is too old"));
}
// We can now do something with the price
let valid_price = price_response.price.price;
// Placeholder for actual price processing
Ok(valid_price)
}
```
## Full Example Contract
Example Contract here: [example](https://github.com/skip-mev/connect-sdk/tree/trunk/contracts/x-oracle-passthrough).
# Getting Prices
Getting prices from the Connect oracle
**Building with Connect? Join our [Discord](https://discord.gg/amAgf9Z39w)**!
## Summary
Connect prices are stored within the [x/oracle](https://github.com/skip-mev/connect/v2/tree/main/x/oracle) module.
On very specific chains - right now only dYdX - they live in a different storage module (in dYdX's case, `x/prices`).
These prices are updated on a per-block basis when there is a sufficient delta from the last block's price. They can be accessed natively by CosmWasm smart contracts, other modules, or those with access to chain state.
Connect's market configuration is stored in the [x/marketmap](https://github.com/skip-mev/connect/v2/tree/main/x/marketmap). This module, unlike `x/oracle`, does **not** store price data. Instead, it stores which currency pairs are supported and how they are configured.
### Getting Supported Assets
Every chain will have a different set of supported assets. You can find out which assets are supported on your chain by either running:
1. (REST): `curl http://localhost:1317/connect/marketmap/v2/marketmap`
2. (application cli): `appd q marketmap marketmap`
3. (gRPC): `grpcurl -plaintext localhost:9090 connect.marketmap.v2.Query/MarketMap`
This will return a JSON list of supported assets with associated metadata.
### Accessing Connect Prices over node APIs and RPC
To access **all** Connect prices (as of the last committed block):
1. (REST): `curl http://localhost:1317/connect/oracle/v2/get_prices`
2. (gRPC): `grpcurl -plaintext localhost:9090 connect.oracle.v2.Query/GetPrices`
To get a **specific** currency pair:
1. (Get all currency pairs request) `appd q oracle currency-pairs`
2. (Get price request) `appd q oracle price [base] [quote]`
### Price Metadata within Connect
When calling `getPrices` via the above methods, you are returned an array of `GetPriceResponse`, each of which contains the following metadata about individual prices:
1. `QuotePrice`
2. nonce
3. decimals
4. ID
`GetPriceResponse` looks like this:
```protobuf query.proto
// GetPriceResponse is the response from the GetPrice grpc method exposed from
// the x/oracle query service.
message GetPriceResponse {
// QuotePrice represents the quote-price for the CurrencyPair given in
// GetPriceRequest (possibly nil if no update has been made)
QuotePrice price = 1 [ (gogoproto.nullable) = true ];
// nonce represents the nonce for the CurrencyPair if it exists in state
uint64 nonce = 2;
// decimals represents the number of decimals that the quote-price is
// represented in. It is used to scale the QuotePrice to its proper value.
uint64 decimals = 3;
// ID represents the identifier for the CurrencyPair.
uint64 id = 4;
}
```
Inside `QuotePrice`, you can fetch for the currency-pair:
1. price
2. timestamp of last update
3. blockheight of last update
`QuotePrice` looks like this:
```protobuf query.proto
// QuotePrice is the representation of the aggregated prices for a CurrencyPair,
// where price represents the price of Base in terms of Quote
message QuotePrice {
string price = 1 [
(cosmos_proto.scalar) = "cosmos.Int",
(gogoproto.customtype) = "cosmossdk.io/math.Int",
(gogoproto.nullable) = false
];
// BlockTimestamp tracks the block height associated with this price update.
// We include block timestamp alongside the price to ensure that smart
// contracts and applications are not utilizing stale oracle prices
google.protobuf.Timestamp block_timestamp = 2
[ (gogoproto.nullable) = false, (gogoproto.stdtime) = true ];
// BlockHeight is height of block mentioned above
uint64 block_height = 3;
}
```
# Integration
Integrating Connect with Blockchain Applications
Connect is business-licensed software under the Business Source License (BSL). The source code is viewable; however, please reach out to us if you are interested in integrating. We are limiting the number of partnerships we engage with in 2024. We apologize in advance if we reach capacity and are unable to accommodate new integrations.
This document will guide you through integrating Connect in your application.
## Requirements
* Go 1.23+
* Cosmos SDK v0.50+
## Integrating Connect
Integrating Connect into your Cosmos SDK application requires a few simple steps.
Add the oracle configuration to your application.
Import and add the `x/marketmap` and `x/oracle` Modules to your app.
Set up the Oracle client the application will use to get prices from the Connect oracle.
Set the PreBlock ABCI method, which is responsible for aggregating prices and writing them to the application's state.
Configure vote extensions with compression and storage strategies.
## Application Configuration
The application node's configuration must be extended so the oracle configuration can be read into the node through `app.toml`.
The application should contain a custom configuration struct with a `"github.com/cosmos/cosmos-sdk/server/config"` embedded.
Note: application function and type names may vary. The names in the following steps are arbitrary for example purposes only.
```go config.go
// CustomAppConfig defines the configuration for the app.
type CustomAppConfig struct {
serverconfig.Config
// ... other configurations
Oracle oracleconfig.AppConfig `mapstructure:"oracle" json:"oracle"`
}
```
Next, append the Oracle's default config template to the custom application template.
```go config.go
func CustomConfigTemplate() string {
return serverconfig.DefaultConfigTemplate + oracleconfig.DefaultConfigTemplate
}
```
Finally, add a default configuration.
```go config.go
func DefaultConfig() (string, CustomAppConfig) {
serverConfig := serverconfig.DefaultConfig()
// edit serverConfig as needed
oracleCfg := oracleconfig.AppConfig{
Enabled: true,
OracleAddress: "localhost:8080",
ClientTimeout: time.Second * 2,
MetricsEnabled: true,
}
customConfig := CustomAppConfig{
Config: *serverConfig,
Oracle: oracleCfg,
}
return CustomConfigTemplate(), customConfig
}
```
The template and default configuration should be passed into `server.InterceptConfigsPreRunHandler` in the application's root command.
Example:
```go root.go
package cmd
import (
// ...
"github.com/cosmos/cosmos-sdk/server"
)
func NewRootCmd() *cobra.Command {
// ....
customAppTemplate, customAppConfig := DefaultConfig() // call function from previous step
return server.InterceptConfigsPreRunHandler(cmd, customAppTemplate, customAppConfig, cometConfig)
}
```
## Keepers
Add [x/marketmap](https://github.com/skip-mev/connect/v2/blob/main/x/marketmap/README.md) and [x/oracle](https://github.com/skip-mev/connect/v2/tree/main/x/oracle) keepers to the application.
```go app.go
package app
import (
// ... other imports
marketmapkeeper "github.com/skip-mev/connect/v2/x/marketmap/keeper"
oraclekeeper "github.com/skip-mev/connect/v2/x/oracle/keeper"
)
type App struct {
// ... other fields
OracleKeeper *oraclekeeper.Keeper
MarketMapKeeper *marketmapkeeper.Keeper
}
```
Then, add them to the dependency injection system.
```go app.go
err := depinject.Inject(
// ... other arguments
&app.MarketMapKeeper,
&app.OracleKeeper,
)
```
Finally, once the app is built with the `appBuilder`, finish the initialization of the `MarketMapKeeper` by setting the hooks.
```go app.go
app.App = appBuilder.Build(db, traceStore, baseAppOptions...)
app.MarketMapKeeper.SetHooks(app.OracleKeeper.Hooks())
```
## Oracle Client
Create a method to construct and return the oracle client and metrics.
```go oracle.go
package app
import (
"context"
"github.com/cosmos/cosmos-sdk/server/types"
oracleconfig "github.com/skip-mev/connect/v2/oracle/config"
oracleclient "github.com/skip-mev/connect/v2/service/clients/oracle"
servicemetrics "github.com/skip-mev/connect/v2/service/metrics"
)
// initializeOracle initializes the oracle client and metrics.
func (app *App) initializeOracle(appOpts types.AppOptions) (oracleclient.OracleClient, servicemetrics.Metrics, error) {
// Read general config from app-opts, and construct oracle service.
cfg, err := oracleconfig.ReadConfigFromAppOpts(appOpts)
if err != nil {
return nil, nil, err
}
// If app level instrumentation is enabled, then wrap the oracle service with a metrics client
// to get metrics on the oracle service (for ABCI++). This will allow the instrumentation to track
// latency in VerifyVoteExtension requests and more.
oracleMetrics, err := servicemetrics.NewMetricsFromConfig(cfg, app.ChainID())
if err != nil {
return nil, nil, err
}
// Create the oracle service.
oracleClient, err := oracleclient.NewPriceDaemonClientFromConfig(
cfg,
app.Logger().With("client", "oracle"),
oracleMetrics,
)
if err != nil {
return nil, nil, err
}
// Connect to the oracle service (default timeout of 5 seconds).
go func() {
app.Logger().Info("attempting to start oracle client...", "address", cfg.OracleAddress)
if err := oracleClient.Start(context.Background()); err != nil {
app.Logger().Error("failed to start oracle client", "err", err)
panic(err)
}
}()
return oracleClient, oracleMetrics, nil
}
```
## ABCI and Vote Extensions
Configure the ABCI methods and vote extensions.
Define a method to contain the logic where these will be configured.
```go oracle.go
func (app *App) initializeABCIExtensions(oracleClient oracleclient.OracleClient, oracleMetrics servicemetrics.Metrics) {}
```
Within this method, do the following:
* **Setup Proposal Handler:** This handler will be used in `PrepareProposal` and `ProcessProposal` to fill proposals with the oracle data.
* **Set PreBlocker:** The application's `PreBlocker` will be configured to write price data to state before transactions are executed.
* **Set Vote Extensions:** Set the vote extension handlers on the application that will handle adding price data to the node's consensus votes.
Start with setting up the proposal handler.
```go oracle.go
package app
import (
oracleclient "github.com/skip-mev/connect/v2/service/clients/oracle"
servicemetrics "github.com/skip-mev/connect/v2/service/metrics"
"github.com/cosmos/cosmos-sdk/baseapp"
"github.com/skip-mev/connect/v2/abci/proposals"
compression "github.com/skip-mev/connect/v2/abci/strategies/codec"
"github.com/skip-mev/connect/v2/abci/strategies/currencypair"
"github.com/skip-mev/connect/v2/abci/ve"
)
func (app *App) initializeABCIExtensions(oracleClient oracleclient.OracleClient, oracleMetrics servicemetrics.Metrics) {
// Create the proposal handler that will be used to fill proposals with
// transactions and oracle data.
proposalHandler := proposals.NewProposalHandler(
app.Logger(),
baseapp.NoOpPrepareProposal(),
baseapp.NoOpProcessProposal(),
ve.NewDefaultValidateVoteExtensionsFn(app.StakingKeeper),
compression.NewCompressionVoteExtensionCodec(
compression.NewDefaultVoteExtensionCodec(),
compression.NewZLibCompressor(),
),
compression.NewCompressionExtendedCommitCodec(
compression.NewDefaultExtendedCommitCodec(),
compression.NewZStdCompressor(),
),
currencypair.NewDeltaCurrencyPairStrategy(app.OracleKeeper),
oracleMetrics,
)
app.SetPrepareProposal(proposalHandler.PrepareProposalHandler())
app.SetProcessProposal(proposalHandler.ProcessProposalHandler())
}
```
Next, set up the `PreBlocker`. This involves:
* **Aggregate Function:** Setting the aggregator function that combines all reported prices into one final price per currency pair.
* **Currency Pair Strategy:** Setting the currency pair strategy. For this example, we will use the `DeltaCurrencyPairStrategy` which encodes/decodes the price as the difference between the current price and the previous price. While other strategies are available, we recommend this one for most applications.
* **Data Compression Codecs:** Setting the compression strategy for vote extensions and extended commits.
```go oracle.go
package app
import (
oracleclient "github.com/skip-mev/connect/v2/service/clients/oracle"
servicemetrics "github.com/skip-mev/connect/v2/service/metrics"
oraclepreblock "github.com/skip-mev/connect/v2/abci/preblock/oracle"
compression "github.com/skip-mev/connect/v2/abci/strategies/codec"
"github.com/skip-mev/connect/v2/abci/strategies/currencypair"
"github.com/skip-mev/connect/v2/pkg/math/voteweighted"
)
func (app *App) initializeABCIExtensions(oracleClient oracleclient.OracleClient, oracleMetrics servicemetrics.Metrics) {
// ...
// Create the aggregation function that will be used to aggregate oracle data
// from each validator.
aggregatorFn := voteweighted.MedianFromContext(
app.Logger(),
app.StakingKeeper,
voteweighted.DefaultPowerThreshold,
)
veCodec := compression.NewCompressionVoteExtensionCodec(
compression.NewDefaultVoteExtensionCodec(),
compression.NewZLibCompressor(),
)
ecCodec := compression.NewCompressionExtendedCommitCodec(
compression.NewDefaultExtendedCommitCodec(),
compression.NewZStdCompressor(),
)
// Create the pre-finalize block hook that will be used to apply oracle data
// to the state before any transactions are executed (in finalize block).
oraclePreBlockHandler := oraclepreblock.NewOraclePreBlockHandler(
app.Logger(),
aggregatorFn,
app.OracleKeeper,
oracleMetrics,
currencypair.NewDeltaCurrencyPairStrategy(app.OracleKeeper), // IMPORTANT: always construct new currency pair strategy objects when functions require them as arguments.
veCodec,
ecCodec,
)
app.SetPreBlocker(oraclePreBlockHandler.WrappedPreBlocker(app.ModuleManager))
}
```
Next, configure the vote extensions using the vote extension codec, extended commit codec, and aggregator function from the previous step.
```go oracle.go
package app
import (
"time"
oracleclient "github.com/skip-mev/connect/v2/service/clients/oracle"
servicemetrics "github.com/skip-mev/connect/v2/service/metrics"
"github.com/skip-mev/connect/v2/abci/ve"
"github.com/skip-mev/connect/v2/abci/strategies/currencypair"
"github.com/skip-mev/connect/v2/abci/strategies/aggregator"
)
func (app *App) initializeABCIExtensions(oracleClient oracleclient.OracleClient, oracleMetrics servicemetrics.Metrics) {
// ... snip ...
// Create the vote extensions handler that will be used to extend and verify
// vote extensions (i.e. oracle data).
voteExtensionsHandler := ve.NewVoteExtensionHandler(
app.Logger(),
oracleClient,
time.Second, // timeout
currencypair.NewDeltaCurrencyPairStrategy(app.OracleKeeper), // IMPORTANT: always construct new currency pair strategy objects when functions require them as arguments.
veCodec,
aggregator.NewOraclePriceApplier(
aggregator.NewDefaultVoteAggregator(
app.Logger(),
aggregatorFn,
// we need a separate price strategy here, so that we can optimistically apply the latest prices
// and extend our vote based on these prices
currencypair.NewDeltaCurrencyPairStrategy(app.OracleKeeper), // IMPORTANT: always construct new currency pair strategy objects when functions require them as arguments.
),
app.OracleKeeper,
veCodec,
ecCodec,
app.Logger(),
),
oracleMetrics,
)
app.SetExtendVoteHandler(voteExtensionsHandler.ExtendVoteHandler())
app.SetVerifyVoteExtensionHandler(voteExtensionsHandler.VerifyVoteExtensionHandler())
}
```
Finally, call these methods back in `app.go`, directly after setting the `x/marketmap` hooks.
```go app.go
app.MarketMapKeeper.SetHooks(app.OracleKeeper.Hooks())
// oracle initialization
client, metrics, err := app.initializeOracle(appOpts)
if err != nil {
return nil, fmt.Errorf("failed to initialize oracle client and metrics: %w", err)
}
app.initializeABCIExtensions(client, metrics)
```
## Initializing Modules
In order for the application to use Connect properly, the following is required:
* Set the consensus parameters to enable vote extensions
* Initialize `x/marketmap` with initial markets
```go oracle.go
package app
import (
"slices"
tmtypes "github.com/cometbft/cometbft/proto/tendermint/types"
sdk "github.com/cosmos/cosmos-sdk/types"
consensustypes "github.com/cosmos/cosmos-sdk/x/consensus/types"
"github.com/skip-mev/connect/v2/cmd/constants/marketmaps"
)
func (app *App) setupMarkets(ctx sdk.Context) error {
// add core markets
coreMarkets := marketmaps.CoreMarketMap
markets := coreMarkets.Markets
// sort keys so we can deterministically iterate over map items.
keys := make([]string, 0, len(markets))
for name := range markets {
keys = append(keys, name)
}
slices.Sort(keys)
for _, marketName := range keys {
// create market
market := markets[marketName]
err := app.MarketMapKeeper.CreateMarket(ctx, market)
if err != nil {
return err
}
// invoke hooks. this syncs the market to x/oracle.
err = app.MarketMapKeeper.Hooks().AfterMarketCreated(ctx, market)
if err != nil {
return err
}
}
return nil
}
```
For new chains, or to test the integration, the method above can be called in `InitChainer`. Connect will begin posting prices to the chain once the `VoteExtensionsEnabledHeight` is reached.
```go app.go
package app
func NewApp(
logger log.Logger,
db dbm.DB,
traceStore io.Writer,
loadLatest bool,
appOpts servertypes.AppOptions,
baseAppOptions ...func(*baseapp.BaseApp),
) *App {
// ...
// initialize the chain with markets in state.
app.SetInitChainer(func(ctx sdk.Context, req *types.RequestInitChain) (*types.ResponseInitChain, error) {
consensusParams, err := app.ConsensusParamsKeeper.Params(ctx, nil)
if err != nil {
return nil, err
}
consensusParams.Params.Abci = &types.ABCIParams{
VoteExtensionsEnableHeight: 5, // must be greater than 1
}
_, err = app.ConsensusParamsKeeper.UpdateParams(ctx, &consensustypes.MsgUpdateParams{
Authority: app.ConsensusParamsKeeper.GetAuthority(),
Block: consensusParams.Params.Block,
Evidence: consensusParams.Params.Evidence,
Validator: consensusParams.Params.Validator,
Abci: consensusParams.Params.Abci,
})
if err != nil {
return nil, err
}
// initialize module state
app.OracleKeeper.InitGenesis(ctx, *oracletypes.DefaultGenesisState())
app.MarketMapKeeper.InitGenesis(ctx, *marketmaptypes.DefaultGenesisState())
// initialize markets
err := app.setupMarkets(ctx)
if err != nil {
return nil, err
}
return app.App.InitChainer(ctx, req)
})
// ...
}
```
For live running chains, use an upgrade handler. Note: Connect will not post prices to the chain until the upgrade is executed.
```go app.go
package app
func NewApp(
logger log.Logger,
db dbm.DB,
traceStore io.Writer,
loadLatest bool,
appOpts servertypes.AppOptions,
baseAppOptions ...func(*baseapp.BaseApp),
) *App {
// ...
connectUpgradeName := "connect-upgrade" // placeholder value, use a real upgrade name.
app.UpgradeKeeper.SetUpgradeHandler(connectUpgradeName, func(ctx context.Context, plan upgradetypes.Plan, fromVM module.VersionMap) (module.VersionMap, error) {
migrations, err := app.ModuleManager.RunMigrations(ctx, app.Configurator(), fromVM)
if err != nil {
return nil, err
}
consensusParams, err := app.ConsensusParamsKeeper.Params(ctx, nil)
if err != nil {
return nil, err
}
consensusParams.Params.Abci = &tmtypes.ABCIParams{
VoteExtensionsEnableHeight: ctx.BlockHeight() + int64(10), // enables VE's at current_height + 10.
}
_, err = app.ConsensusParamsKeeper.UpdateParams(ctx, &consensustypes.MsgUpdateParams{
Authority: app.ConsensusParamsKeeper.GetAuthority(),
Block: consensusParams.Params.Block,
Evidence: consensusParams.Params.Evidence,
Validator: consensusParams.Params.Validator,
Abci: consensusParams.Params.Abci,
})
if err != nil {
return nil, err
}
// add the markets to the chain state.
err := app.setupMarkets(ctx)
if err != nil {
return migrations, err
}
return migrations, nil
})
upgradeInfo, err := app.UpgradeKeeper.ReadUpgradeInfoFromDisk()
if err != nil {
panic(fmt.Errorf("failed to read upgrade info from disk: %w", err))
}
// add the x/marketmap and x/oracle stores.
if upgradeInfo.Name == connectUpgradeName {
app.SetStoreLoader(
upgradetypes.UpgradeStoreLoader(
upgradeInfo.Height,
&storetypes.StoreUpgrades{
Added: []string{marketmaptypes.ModuleName, oracletypes.ModuleName},
Renamed: nil,
Deleted: nil,
},
),
)
}
// ...
}
```
## Running the Node
Once the chain is properly configured, head over to the [Quickstart](../validators/quickstart) guide to learn how to start the node with a Connect sidecar.
## Need Help?
Need help with your integration? Feel free to reach out to us on [Discord](https://discord.gg/amAgf9Z39w).
# Marketmap
Reference documentation for the Marketmap module.
## Overview
The `Marketmap` module is responsible for managing and storing a configuration that informs the Connect oracle of which markets to fetch data for, and which providers to use to fetch them.
## Concepts
### Authority, Admin, And MarketAuthority
The `Marketmap` module contains three levels of access to interact with the module.
#### Authority
The `Authority` is the only account allowed to change the module's [Params](#params). By default, this account is set to the governance address. However, you may edit the configuration of the module to be any address.
#### Admin
The `Admin` can only *remove* an address from the market authority list via `RemoveMarketAuthorities`.
#### MarketAuthority
A `MarketAuthority` is assigned by the module `Authority`. There can be any number of market authorities. The market authorities are able to create and update markets in the `Marketmap`. Specifically, only a `MarketAuthority` may send the following transactions:
* CreateMarkets
* UpdateMarkets
* UpsertMarkets
### Market
A market consists of a `Ticker` (i.e. BTC/USD) and a list of `ProviderConfig`s. A `Ticker` contains data about a specific currency pair. A `ProviderConfig` contains data that informs the Oracle of how to query for the currency pair in the `Ticker`.
```go
// Market encapsulates a Ticker and its provider-specific configuration.
type Market struct {
// Ticker represents a price feed for a given asset pair i.e. BTC/USD. The
// price feed is scaled to a number of decimal places and has a minimum number
// of providers required to consider the ticker valid.
Ticker Ticker `protobuf:"bytes,1,opt,name=ticker,proto3" json:"ticker"`
// ProviderConfigs is the list of provider-specific configs for this Market.
ProviderConfigs []ProviderConfig `protobuf:"bytes,2,rep,name=provider_configs,json=providerConfigs,proto3" json:"provider_configs"`
}
```
### ProviderConfig
The `Name` field refers to one of the providers listed in the [Providers](/developers/providers) document.
```go
type ProviderConfig struct {
// Name corresponds to the name of the provider for which the configuration is
// being set.
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
// OffChainTicker is the off-chain representation of the ticker i.e. BTC/USD.
// The off-chain ticker is unique to a given provider and is used to fetch the
// price of the ticker from the provider.
OffChainTicker string `protobuf:"bytes,2,opt,name=off_chain_ticker,json=offChainTicker,proto3" json:"off_chain_ticker,omitempty"`
// NormalizeByPair is the currency pair for this ticker to be normalized by.
// For example, if the desired Ticker is BTC/USD, this market could be reached
// using: OffChainTicker = BTC/USDT NormalizeByPair = USDT/USD This field is
// optional and nullable.
NormalizeByPair *types.CurrencyPair `protobuf:"bytes,3,opt,name=normalize_by_pair,json=normalizeByPair,proto3" json:"normalize_by_pair,omitempty"`
// Invert is a boolean indicating if the BASE and QUOTE of the market should
// be inverted. i.e. BASE -> QUOTE, QUOTE -> BASE
Invert bool `protobuf:"varint,4,opt,name=invert,proto3" json:"invert,omitempty"`
// MetadataJSON is a string of JSON that encodes any extra configuration
// for the given provider config.
Metadata_JSON string `protobuf:"bytes,15,opt,name=metadata_JSON,json=metadataJSON,proto3" json:"metadata_JSON,omitempty"`
}
```
### Ticker
```go
// Ticker represents a price feed for a given asset pair i.e. BTC/USD. The price
// feed is scaled to a number of decimal places and has a minimum number of
// providers required to consider the ticker valid.
type Ticker struct {
// CurrencyPair is the currency pair for this ticker.
CurrencyPair types.CurrencyPair `protobuf:"bytes,1,opt,name=currency_pair,json=currencyPair,proto3" json:"currency_pair"`
// Decimals is the number of decimal places for the ticker. The number of
// decimal places is used to convert the price to a human-readable format.
Decimals uint64 `protobuf:"varint,2,opt,name=decimals,proto3" json:"decimals,omitempty"`
// MinProviderCount is the minimum number of providers required to consider
// the ticker valid.
MinProviderCount uint64 `protobuf:"varint,3,opt,name=min_provider_count,json=minProviderCount,proto3" json:"min_provider_count,omitempty"`
// Enabled is the flag that denotes if the Ticker is enabled for price
// fetching by an oracle.
Enabled bool `protobuf:"varint,14,opt,name=enabled,proto3" json:"enabled,omitempty"`
// MetadataJSON is a string of JSON that encodes any extra configuration
// for the given ticker.
Metadata_JSON string `protobuf:"bytes,15,opt,name=metadata_JSON,json=metadataJSON,proto3" json:"metadata_JSON,omitempty"`
}
```
### Params
`Params` define the authenticated addresses that can mutate the state of the `Marketmap`.
```go
// Params defines the parameters for the x/marketmap module.
type Params struct {
// MarketAuthorities is the list of authority accounts that are able to
// control updating the marketmap.
MarketAuthorities []string `protobuf:"bytes,1,rep,name=market_authorities,json=marketAuthorities,proto3" json:"market_authorities,omitempty"`
// Admin is an address that can remove addresses from the MarketAuthorities
// list. Only governance can add to the MarketAuthorities or change the Admin.
Admin string `protobuf:"bytes,2,opt,name=admin,proto3" json:"admin,omitempty"`
}
```
## Messages
The following messages can be included in [transactions](https://docs.cosmos.network/main/learn/advanced/transactions) to mutate the state of the `Marketmap`.
### MsgCreateMarkets
`MsgCreateMarket` creates a new `Market`.
```go
// MsgCreateMarkets defines a message carrying a payload for creating markets in
// the x/marketmap module.
type MsgCreateMarkets struct {
// Authority is the signer of this transaction. This authority must be
// authorized by the module to execute the message.
Authority string `protobuf:"bytes,1,opt,name=authority,proto3" json:"authority,omitempty"`
// CreateMarkets is the list of all markets to be created for the given
// transaction.
CreateMarkets []Market `protobuf:"bytes,2,rep,name=create_markets,json=createMarkets,proto3" json:"create_markets"`
}
```
### MsgUpdateMarkets
`MsgUpdateMarkets` updates an existing `Market`.
```go
// MsgUpdateMarkets defines a message carrying a payload for updating the
// x/marketmap module.
type MsgUpdateMarkets struct {
// Authority is the signer of this transaction. This authority must be
// authorized by the module to execute the message.
Authority string `protobuf:"bytes,1,opt,name=authority,proto3" json:"authority,omitempty"`
// UpdateMarkets is the list of all markets to be updated for the given
// transaction.
UpdateMarkets []Market `protobuf:"bytes,2,rep,name=update_markets,json=updateMarkets,proto3" json:"update_markets"`
}
```
### MsgUpsertMarkets
`MsgUpsertMarkets` will update a `Market` if one already exists. If a `Market` does not exist, it will create one instead.
```go
// MsgUpsertMarkets defines a message carrying a payload for performing market upserts (update or
// create if does not exist) in the x/marketmap module.
type MsgUpsertMarkets struct {
// Authority is the signer of this transaction. This authority must be
// authorized by the module to execute the message.
Authority string `protobuf:"bytes,1,opt,name=authority,proto3" json:"authority,omitempty"`
// CreateMarkets is the list of all markets to be created for the given
// transaction.
Markets []Market `protobuf:"bytes,2,rep,name=markets,proto3" json:"markets"`
}
```
### MsgParams
`MsgParams` updates the `Marketmap` parameters.
```go
// MsgParams defines the Msg/Params request type. It contains the
// new parameters for the x/marketmap module.
type MsgParams struct {
// Params defines the new parameters for the x/marketmap module.
Params Params `protobuf:"bytes,1,opt,name=params,proto3" json:"params"`
// Authority defines the authority that is updating the x/marketmap module
// parameters.
Authority string `protobuf:"bytes,2,opt,name=authority,proto3" json:"authority,omitempty"`
}
```
### MsgRemoveMarketAuthorities
`MsgRemoveMarketAuthorities` removes a market authority from the `Params`.
```go
// MsgRemoveMarketAuthorities defines the Msg/RemoveMarketAuthoritiesResponse
// request type. It contains the new addresses to remove from the list of
// authorities
type MsgRemoveMarketAuthorities struct {
// RemoveAddresses is the list of addresses to remove.
RemoveAddresses []string `protobuf:"bytes,1,rep,name=remove_addresses,json=removeAddresses,proto3" json:"remove_addresses,omitempty"`
// Admin defines the authority that is the x/marketmap
// Admin account. This account is set in the module parameters.
Admin string `protobuf:"bytes,2,opt,name=admin,proto3" json:"admin,omitempty"`
}
```
## Queries
The following [queries](https://tutorials.cosmos.network/academy/2-cosmos-concepts/9-queries.html) are available to retrieve data about the state of the `Marketmap`.
### MarketMap
The `Marketmap` query returns the full `Marketmap` in state.
**Request:**
```go
// MarketMapRequest is the query request for the MarketMap query.
// It takes no arguments.
type MarketMapRequest struct {}
```
**Response:**
```go
// MarketMapResponse is the query response for the MarketMap query.
type MarketMapResponse struct {
// MarketMap defines the global set of market configurations for all providers
// and markets.
MarketMap MarketMap `protobuf:"bytes,1,opt,name=market_map,json=marketMap,proto3" json:"market_map"`
// LastUpdated is the last block height that the market map was updated.
// This field can be used as an optimization for clients checking if there
// is a new update to the map.
LastUpdated uint64 `protobuf:"varint,2,opt,name=last_updated,json=lastUpdated,proto3" json:"last_updated,omitempty"`
// ChainId is the chain identifier for the market map.
ChainId string `protobuf:"bytes,3,opt,name=chain_id,json=chainId,proto3" json:"chain_id,omitempty"`
}
```
### Market
Market returns a specific `Market` from the `Marketmap`.
**Request:**
```go
// MarketRequest is the query request for the Market query.
// It takes the currency pair of the market as an argument.
type MarketRequest struct {
// CurrencyPair is the currency pair associated with the market being
// requested.
CurrencyPair types.CurrencyPair `protobuf:"bytes,1,opt,name=currency_pair,json=currencyPair,proto3" json:"currency_pair"`
}
```
**Response:**
```go
// MarketResponse is the query response for the Market query.
type MarketResponse struct {
// Market is the configuration of a single market to be price-fetched for.
Market Market `protobuf:"bytes,1,opt,name=market,proto3" json:"market"`
}
```
### LastUpdated
LastUpdated returns the height at which the `Marketmap` was last updated.
**Request:**
```go
// LastUpdatedRequest is the request type for the Query/LastUpdated RPC
// method.
type LastUpdatedRequest struct {}
```
**Response:**
```go
// LastUpdatedResponse is the response type for the Query/LastUpdated RPC
// method.
type LastUpdatedResponse struct {
LastUpdated uint64 `protobuf:"varint,1,opt,name=last_updated,json=lastUpdated,proto3" json:"last_updated,omitempty"`
}
```
### Params
Params returns the `Marketmap`'s `Params`.
**Request:**
```go
// ParamsRequest is the request type for the Query/Params RPC method.
type ParamsRequest struct {}
```
**Response:**
```go
// ParamsResponse is the response type for the Query/Params RPC method.
type ParamsResponse struct {
Params Params `protobuf:"bytes,1,opt,name=params,proto3" json:"params"`
}
```
## Proto Definitions
Proto definitions for all types, queries, and messages can be found [here](https://github.com/skip-mev/connect/v2/tree/main/proto/connect/marketmap).
# Providers
Connect pulls data from `Providers`. `Providers` utilize public REST APIs, RPCs, and websockets from either centralized or decentralized exchanges. Below is a list of all available `Providers` in Connect.
# Centralized Exchange Providers
### REST API
* binance\_api
* bitstamp\_api
* coinbase\_api
* kraken\_api
* polymarket\_api
### Websocket
* binance\_ws
* bitfinex\_ws
* bitstamp\_ws
* bybit\_ws
* coinbase\_ws
* crypto\_dot\_com\_ws
* gate\_ws
* huobi\_ws
* kraken\_ws
* kucoin\_ws
* mexc\_ws
* okx\_ws
# Decentralized Exchange Providers
### REST API
* uniswapv3\_api-ethereum
* uniswapv3\_api-base
* raydium\_api
* osmosis\_api
# Introduction
## What is Connect?
**Connect** is an oracle that securely delivers offchain data to onchain applications using a chain's native validator set and security. This documentation will guide you through using Connect; whether you're a **validator** or **developer**.
Learn how to setup and run Connect with your infrastructure.
Learn how to use the Connect API to access data.
### Features
#### Security
Connect leverages the chain’s security, giving the fastest updates possible, and removing the requirements for any 3rd party systems.
#### Performance
Connect can support over 2000 currency pairs and price feeds, allowing the launch of thousands of permissionless on-chain markets. Ultimately, the speed of Connect is determined by the speed of your application.
#### Support
Connect comes with a 1-day SLAs for adding new feeds, and 24/7 on-call support and maintenance by the Skip team.
#### UX
By leveraging new advancements in consensus like vote extensions & ABCI++, Connect guarantees a millisecond-fresh oracle update every block, allowing applications to build without sacrificing UX for safety.
## Ready to Connect?
If you're interested in using Connect, please check out our [website](https://skip.build) and reach out to us on [Discord](https://discord.gg/PeBGE9jrbu) to get started.
# Architecture
Learn how Connect works with ABCI++
The Connect sidecar is an out-of-process service that efficiently fetches prices from various data sources and runs aggregation logic
to combine them into a single price for each feed. The application uses GRPC requests to the sidecar to fetch the latest updates to update on-chain prices from over 20+ providers.
## On-chain aggregation
Connect uses [ABCI++](https://docs.cometbft.com/v0.38/spec/abci/) to separate Oracle aggregation into secure and efficient steps.
### Extend Vote / Verify Vote
The `ExtendVote` and `VerifyVote` methods of ABCI++ are where a price is first queried.
* The validators fetch prices from the sidecar from a series of `providers` (e.g. Binance / Coinbase) for each currency pair.
* For each pair, the median is taken between all `providers`.
* Each validator then submits their final prices to the chain via the ABCI++ `ExtendVote` method.
* `VerifyVote` is used to ensure that the submitted data is valid--i.e. it can be unmarshalled by another validator.
### Prepare Proposal
During `PrepareProposal`, the vote extensions from the previous block are aggregated by the block proposer into their block proposal, after various checks are run on them.
* Connect ensures that the set of vote extensions comprise the required minimum stake required for a price update (default of 2/3).
* It also ensures that the vote extensions are valid and can be understood by the application.
* Finally, it encodes the vote extensions and injects them into the top of the block proposal as a pseudo-transaction ignored by the chain.
For more information on vote extensions in general, refer to [the Cosmos SDK docs](https://docs.cosmos.network/main/build/abci/vote-extensions).
### Process Proposal
`ProcessProposal` is identical to PrepareProposal, except it is run on every validator to validate the block proposal.
* If the validator comes to the conclusion that the injected votes are valid and comprise the required minimum stake, it will accept the proposal.
* If not, the validator will reject the proposal.
### Finalize Block
The end of a price's journey is in the `Preblock` step.
* Here, the injected transaction data is unmarshalled back into vote extensions, and the application takes a stake-weighted median of the prices reported by every validator.
* The resulting canonical price for each pair is stored in the `x/oracle` module and can be queried by any application or RPC.
### Full Flow
* This full set of steps repeats on every block resulting in a continuous stream of guaranteed prices. The oracle is enshrined in the application by the
validator set, and so is fundamentally equivalent to chain liveness (i.e. the oracle can't go down without the chain going down)
# Security
Connect Security Properties
## Basics
**Connect is as secure as the chain it is deployed on** - This is the meaning of a "restaked" oracle - Connect leverages all available
security that the chain can provide by using the existing validator set and tying updates to consensus via vote extensions.
**Prices require 2/3 participation from validators to be posted on chain** - We require a minimum of 2/3 of stake weight to contribute
to a price aggregation for that price to actually update chain state. This is the same stake weight required to commit blocks.
**Connect avoids any external dependencies** - Connect injects price data directly into the chain it's deployed on. This means there's no
additional chain, token, relayer, or other outside actor dependency. The same operators that run the chain run the oracle.
**Connect price updates are atomic with liveness** - Since Connect data is signed atomically with `PreCommit` votes via Vote Extensions, chains can
configure their application (or parts of their application) to depend on per-block price updates. This makes almost all forms of oracle attacks impossible,
and allows applications to avoid having to build their own UX roadblocks to ensure safety.
## Impact of Vote Extensions
Connect updates when over 2/3 of the validator set (by stake weight) suggest a series of prices, and then aggregates across
them using a stake-weighted median. This price data is committed to the chain by validators through [vote extensions](https://docs.cosmos.network/main/build/abci/vote-extensions).
Vote extensions are arbitrary metadata that is signed atomically with a validator's `PreCommit` vote. In Connect's case, that metadata is
locally observed prices for a series of `CurrencyPairs` (e.g. `ETH/USDC`).
Given blocks cannot progress without 2/3 of stake weight submitting votes, blocks also cannot progress without 2/3 of vote extensions.
Additionally, the `x/oracle` module (where prices are stored on chain) requires at least 2/3 of voting power to have contributed to individual
price updates. This means that every price update has the same participation as block validity itself.
## Manipulation Thresholds
In the standard configuration of Connect, it requires `1/3` of stake to be manipulated to post an incorrect oracle price. This is because:
* `2/3` of stake is required to post any update (as described above)
* half of that stake must be manipulated to post incorrect prices in the *same direction* to move the stake-weighted median final price posted on-chain
* That is, (`2/3 / 2 = 1/3`) of stake must be manipulated to post a malicious price.
However, Connect can support additional constraints to increase this level to `2/3` of stake. To do this, the final on-chain prices are re-checked by all validators on the network in the `ProcessProposal` step to enforce some minimum deviation from their local prices are not exceeded. `2/3` of stake weight is required to vote "yes" on this check, raising the overall security back to that of the chain itself.
There are tradeoffs to increasing the security to this level. Validators may reject proposals more often if there is high volatility in price updates between them, which could result in longer periods of oracle downtime in periods of crypto market instability.
## Importance of Operational Excellence
Security assumptions, no matter how good, are no substitute for the oracle operators (the chain's validator set) keeping high uptime, reliable, and accurate price updates. Connect makes this easy for validators, providing an ample amount of operational support and tooling for alerting, upgrading, and maintaining a Connect instance.
# Application
Reference for the available metrics in the Application
## Exposed Metrics
By default, Cosmos SDK application expose metrics on port `26660`. These metrics are in the [Prometheus](https://prometheus.io/docs/introduction/overview/) format and can be scraped by any monitoring system that supports Prometheus format.
## Oracle Client Connection Metrics
* **oracle\_response\_latency:** Histogram that measures the time in nanoseconds between request/response of from the Application to Oracle.
* **oracle\_responses:** Counter that measures the number of oracle responses.
## ABCI Metrics
The following metrics are measured in the following ABCI methods: ExtendVote, PrepareProposal, ProcessProposal, VerifyVoteExtension, FinalizeBlock.
* **ABCI\_method\_latency:** Histogram that measures the amount of seconds ABCI method calls take.
* **ABCI\_method\_status:** Counter that measures the number of ABCI requests.
## VE/Price Metrics
* **message\_size:** Histogram that tracks the size of vote-extensions and extended commits that Connect transmits.
* **oracle\_prices:** Gauge that tracks prices written to application state.
## Network Metrics
* **oracle\_reports\_per\_validator:** Gauge that tracks the prices each validator has reported for any block per ticker.
* **oracle\_report\_status\_per\_validator:** Counter that tracks the number of reports per validator and their vote status (absent, missing\_price, with\_price).
# Oracle
Reference for the available metrics in the Connect oracle
## Dashboard Terminology
* **Market:** The pair of assets that are traded against each other. For example, the BTC/USD market is the market
where Bitcoin is traded against the US Dollar.
* **Price Feed:** A price feed is indexed by a price provider and a market. For example, the Coinbase API provides a price feed for the BTC/USD market.
* **Price Provider:** Service that provides price data for a given market. For example, the Coinbase API is a price provider for the Coinbase markets.
* **Market Map Provider:** Service that supplies the markets that the side-car needs to fetch data for.
## Exposed Metrics
Connect also exposes metrics on the `/metrics` endpoint on port `8002` by default. These metrics are in the Prometheus format and can be scraped by any monitoring system that supports Prometheus format.
### Health Metrics
The following health metrics are emitted in Connect.
* **side\_car\_health\_check\_system\_updates\_total:** Counter that increments every time the sidecar updates its internal state. This is a good indicator of the sidecar's overall health.
* **side\_car\_health\_check\_ticker\_updates\_total:** Counter that increments every time the side-car updates the price of a given market. This is a good indicator of the overall health of a given market.
* **side\_car\_health\_check\_provider\_updates\_total:** Counter that increments every time the side-car utilizes a given providers market data. This is a good indicator of the health of a given provider. Note that providers may not be responsible for every market. However, the sidecar correctly tracks the number of expected updates for each provider. This metric can be quite noisy, so consider omitting it from dashboards if that becomes an issue in your Grafana instance.
### Price Metrics
Connect exposes various metrics related to market prices. These metrics are useful for monitoring the health of the price feeds and the aggregation process. The rate of updates for these metrics is likely going to be much greater than the health metrics. This is expected as we poll the providers much more frequently than we update all prices. As such, the metrics below are not expected to be updated at the same rate as the health metrics. However, they are greatly useful to quickly identify issues with the price feeds.
* **side\_car\_provider\_price:** The last recorded price for a given price feed.
* **side\_car\_provider\_last\_updated\_id:** The last UNIX timestamp for a given price feed.
### Aggregated Price Metrics
* **side\_car\_aggregated\_price:** The aggregated price for a given market. This price is the result of a median aggregation of all available price feeds for a given market. This is the price clients will see when querying the side-car.
### HTTP Metrics
Connect exposes various metrics related to HTTP requests made by the sidecar - including the number of requests, the response time, and the status code. These metrics can be used to monitor the health of the sidecar's HTTP endpoints.
* **side\_car\_api\_http\_status\_code:** The status codes of the HTTP response made by the side-car.
* **side\_car\_api\_response\_latency\_bucket:** The response latency of the HTTP requests made by the side-car.
### WebSocket Metrics
Connect exposes various metrics related to WebSocket connections made by the sidecar. These metrics can be used to monitor the health of the sidecar's WebSocket connections.
* **side\_car\_web\_socket\_connection\_status:** This includes various metrics related to the WebSocket connections made by the side-car.
* **side\_car\_web\_socket\_data\_handler\_status:** This includes various metrics related to whether WebSocket messages are being correctly handled by the side-car.
* **side\_car\_web\_socket\_response\_time\_bucket:** This includes the response time of the WebSocket messages received by the side-car.
# Overview
Overview of metrics in Connect
The Connect sidecar is equipped to emit Prometheus metrics to provide observability into the health of the oracle sidecar.
### Metrics Emitted by Connect
Connect will emit the following metrics from the oracle sidecar:
* Liveness counter for every update cycle
* Counter for the number of times a market has been updated
* Price updates per ticker per provider
* Aggregate price updates per ticker
* Count for number of providers used in a price aggregation
* Count for number of times a provider included a price used in aggregation
* Connect binary build information
### Metrics Emitted by Application
* Oracle client response latency
* Number of Oracle responses
* ABCI method latency
* Size of messages
* Aggregate ticker prices written to app state
* Prices reported by each validator
* Status of reports from each validator
### Need More Metrics?
If you find yourself wanting more observability than what is provided, please open an issue on our [GitHub repo](https://github.com/skip-mev/connect/).
# Setup
How to set up Connect metrics
This document will guide you on how to utilize Connect's metrics.
## Prerequisites
* You have a [Prometheus](https://prometheus.io/) instance
* You have a [Grafana](https://grafana.com/) instance
* You've configured a data source in Grafana for your Prometheus instance
* You have set the metrics [configuration](/validators/configuration#env-vars) in the Connect sidecar
If you're not sure how to set up Prometheus and Grafana, see this [guide](https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-prometheus/).
## Import Dashboards
Import the following dashboard JSON files to view the available metrics in Grafana. If you do not know how to import a Grafana dashboard, see this [guide](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/).
### Sidecar
[Sidecar Dashboard JSON file](https://github.com/skip-mev/connect/v2/blob/main/grafana/provisioning/dashboards/side-car-dashboard.json)
### Application
[Application Dashboard JSON file](https://github.com/skip-mev/connect/v2/blob/main/grafana/provisioning/dashboards/chain-dashboard.json)
# Advanced Setups
Using Connect with Advanced Validator Infrastructure
## Using Remote Signers
Remote signers can be used with Connect, however only certain versions are compatible.
### Horcrux
**Required Version:** v3.3.0+ [https://github.com/strangelove-ventures/horcrux/releases](https://github.com/strangelove-ventures/horcrux/releases)
#### With `Horcrux-Proxy`
**Required Version:** v1.0.0+ [https://github.com/strangelove-ventures/horcrux-proxy/releases](https://github.com/strangelove-ventures/horcrux-proxy/releases)
### TMKMS
**Required Version:** v0.13.1+ [https://github.com/iqlusioninc/tmkms/tags](https://github.com/iqlusioninc/tmkms/tags)
## Using Distributed Validators
Connect can be used within a distributed validator setup. To do so, simply apply the same `app.toml` changes to *each* validator node.
Head over to [configuration](configuration#application) to see the necessary application-side configurations.
# Configuration
Reference document for configurable values in Connect and Applications.
This document serves as a reference for ways to further configure the Connect sidecar, as well as in depth explanations of the application node's configuration.
## Connect Sidecar
Connect can be configured by both flags and environment variables.
### Env Vars
The following values are read via environment variables:
| Key | Default | Description |
| ------------------------------------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CONNECT_CONFIG_HOST` | `"0.0.0.0"` | The address Connect will serve requests from. WARNING: changing this value requires updating the `oracle_address` in the `app.toml` configuration. |
| `CONNECT_CONFIG_PORT` | `"8080"` | The port Connect will serve requests from. WARNING: changing this value requires updating the `oracle_address` in the `app.toml` configuration. |
| `CONNECT_CONFIG_METRICS_ENABLED` | `"true"` | Enables prometheus metrics. |
| `CONNECT_CONFIG_METRICS_PROMETHEUSSERVERADDRESS` | `"0.0.0.0:8002"` | The address of your prometheus server instance. |
### Flags
| Flag | Default Value | Description |
| -------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--market-map-endpoint` | `""` | The listen-to endpoint for market-map. This is typically the blockchain node's gRPC endpoint. |
| `--oracle-config` | `""` | Overrides part of the Oracle configuration. This does not override the *entire* configuration, only the part of the configuration specified in the json file passed in. |
| `--run-pprof` | `false` | Run pprof server. |
| `--pprof-port` | `"6060"` | Port for the pprof server to listen on. |
| `--log-std-out-level` | `"info"` | Log level (debug, info, warn, error, dpanic, panic, fatal). |
| `--log-file-level` | `"info"` | Log level for the file logger (debug, info, warn, error, dpanic, panic, fatal). |
| `--log-file` | `"sidecar.log"` | Write logs to a file. |
| `--log-max-size` | `100` | Maximum size in megabytes before log is rotated. |
| `--log-max-backups` | `1` | Maximum number of old log files to retain. |
| `--log-max-age` | `3` | Maximum number of days to retain an old log file. |
| `--log-file-disable-compression` | `false` | Compress rotated log files. |
| `--log-disable-file-rotation` | `false` | Disable writing logs to a file. |
| `--metrics-enabled` | `true` | Enables the Oracle client metrics. |
| `--metrics-prometheus-address` | `"0.0.0.0:8002"` | Sets the Prometheus server address for the Oracle client metrics. |
| `--host` | `"0.0.0.0"` | The address the Oracle will serve from. |
| `--port` | `"8080"` | The port the Oracle will serve from. |
| `--update-interval` | `250000000` | The interval at which the oracle will fetch prices from providers. |
| `--max-price-age` | `120000000000` | Maximum age of a price that the oracle will consider valid. |
## Application Node
The blockchain application is configured under the `[oracle]` heading in your application's `app.toml` file.
```toml app.toml
# ... other sections
[oracle]
# Enabled indicates whether the oracle is enabled.
enabled = "true"
# Oracle Address is the URL of the out of process oracle sidecar. This is used to
# connect to the oracle sidecar when the application boots up.
oracle_address = "CONNECT_ADDRESS_HERE:CONNECT_PORT_HERE" # default Connect port is 8080.
# Client Timeout is the time that the application is willing to wait for responses from
# the oracle before timing out.
client_timeout = "250ms"
# MetricsEnabled determines whether oracle metrics are enabled. Specifically,
# this enables instrumentation of the oracle client and the interaction between
# the oracle and the app.
metrics_enabled = "true"
# PriceTTL is the maximum age of the latest price response before it is considered stale.
# The recommended max age is 10 seconds (10s). If this is greater than 1 minute (1m), the app
# will not start.
price_ttl = "10s"
# Interval is the time between each price update request. The recommended interval
# is the block time of the chain. Otherwise, 1.5 seconds (1500ms) is a good default. If this
# is greater than 1 minute (1m), the app will not start.
interval = "1500ms"
```
# FAQs
Frequently asked questions about Connect.
Yes, you can run it anywhere - but please defer to any chain-specific recommendations if there are any!
No, IPv6 is currently not supported for sidecar-node communication.
We are currently working on supporting this feature. It will be available in a future release.
No. Prices are stored in `x/oracle` module, and only stores the most recently posted price. However, you can use blockchain indexers or inspect past blocks to see the prices committed on previous heights.
Prices within Connect are committed on a one-block delay, since validators use the vote extensions from block n-1 to securely submit their price data for block n.
Most of the time, prices will update every single block. Price updates happen when over 2/3 of validators are correctly running the Connect sidecar.
Prices will not update on any given block if:
* The market is disabled within x/marketmap
* Less than 2/3s of validators (by stake weight) contributed to a price update. This can happen if not enough validators run Connect, or there is a massive, widespread outage across providers.
Upgrading the Connect binary can be done out of band of the chain’s binary. If you have a load balancer, CNAME, etc., in front of your sidecar you can simply start up the new version and switch out which version traffic is being directed to during live chain validation.
If you are running the Connect sidecar in a container you can shut down the container, pull the updated container image and relaunch your container to update.
If you are running the binary via systemd or other management tool, you will need to stop the process and re-launch using the newly released binary.
The node will still be able to participate in consensus without the sidecar, and will begin attaching prices to blocks once Connect is available. In the worst case, an upgrade in any of these manners will cause you to miss including vote extensions for a single block which should have no negative effects on you or the network.
No.
No, the Connect binary is very lightweight. On a 36 GB Macbook Pro M3, a Connect instance fetching 125 markets took up only 50MB of memory, and 6% of the CPU.
If you're a validator and need help getting your infrastructure setup, head over to our [Discord](https://discord.com/invite/hFeHVAE26P) and let us know what chain you're validating for in the `#waiting-room` channel.
If you don't see your question here, please reach out to us on [Discord](https://discord.com/invite/hFeHVAE26P).
# Quickstart
Connect Validator Quickstart
The following instructions assumes you are running a single validator full node with a remote instance of Connect. If you are not running a full node, you do not need to run Connect.
This document will guide you through setting up and running Connect, as well as configuring your node to receive data from Connect.
## Installation
### Using Curl
```bash
curl -sSL https://raw.githubusercontent.com/skip-mev/connect/main/scripts/install.sh | sudo bash
```
### From GitHub Releases
Head over to our [GitHub Releases](https://github.com/skip-mev/connect/releases) page and download the binary that matches your machine's architecture.
### From Source
To install from source, you'll need [Git](https://git-scm.com/) and [Go](https://go.dev/).
Enter the commands below to install the binary.
```shell
git clone git@github.com:skip-mev/connect.git
cd connect
git checkout $(git describe --tags $(git rev-list --tags --max-count=1))
make install
```
## Verify Installation
Let's check Connect is properly installed on your machine.
You should see the version printed in your terminal after running the following command:
```shell
connect version
```
## Run Connect Sidecar
This tab provides general instructions for starting the Connect sidecar.
If you are running one of the chains listed in the tabs above, please refer to those instructions.
To run Connect, which starts the service on the default port of `8080`, enter the following command:
```shell
connect --market-map-endpoint=":"
```
**The required version for Connect with dYdX v7 is `v1.0.13`.**
First, please ensure you've received your **API keys** for the relevant decentralized provider nodes. If you have not received API keys, please reach out to the provider team in the relevant communication channels.
Next, place your API keys under their corresponding URLs in the following file and save it to your system. **Keep the file path handy** as we will pass it into a flag when running Connect.
We will also supply an edited configuration for the `dydx_migration_api` which facilitates graceful migration from dydx's `x/prices` to `x/marketmap`.
For the `dydx_migration_api` provider, **make sure to fill in the URL for the REST endpoint and gRPC endpoint of your node** (in that order). The migration API will *not* work unless the REST API endpoint is the **first** endpoint in the endpoints list.
```json oracle.json
{
"providers": {
"dydx_migration_api": {
"api": {
"endpoints": [
{
"url": "http://"
},
{
"url": ":"
}
]
}
},
"raydium_api": {
"api": {
"endpoints": [
{
"url": "https://solana.polkachu.com",
"authentication": {
"apiKeyHeader":"x-api-key",
"apiKey":"API KEY"
}
},
{
"url": "https://connect-solana.kingnodes.com",
"authentication": {
"apiKeyHeader":"x-api-key",
"apiKey":"API KEY"
}
},
{
"url": "https://solana.lavenderfive.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://solana-rpc.rhino-apis.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://dydx.helius-rpc.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
}
]
}
},
"uniswapv3_api-ethereum": {
"api": {
"endpoints": [
{
"url": "https://ethereum.lavenderfive.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://ethereum.polkachu.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://connect-eth.kingnodes.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://ethereum-rpc.rhino-apis.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
}
]
}
},
"uniswapv3_api-base": {
"api": {
"endpoints": [
{
"url": "https://base-rpc.rhino-apis.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://connect-base.kingnodes.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://base.lavenderfive.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://base.polkachu.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
}
]
}
}
}
}
```
With the `oracle.json` file path, enter the following command to run Connect.
```shell
connect \
--marketmap-provider dydx_migration_api \
--oracle-config path/to/oracle.json
```
**The required version for Connect with dYdX v6 is `v1.0.13`.**
First, please ensure you've received your **API keys** for the relevant decentralized provider nodes. If you have not received API keys, please reach out to the provider team in the relevant communication channels.
Next, place your API keys under their corresponding URLs in the following file and save it to your system. **Keep the file path handy** as we will pass it into a flag when running Connect.
We will also supply an edited configuration for the `dydx_migration_api` which facilitates graceful migration from dydx's `x/prices` to `x/marketmap`.
For the `dydx_migration_api` provider, **make sure to fill in the URL for the REST endpoint and gRPC endpoint of your node** (in that order). The migration API will *not* work unless the REST API endpoint is the **first** endpoint in the endpoints list.
```json oracle.json
{
"providers": {
"dydx_migration_api": {
"api": {
"endpoints": [
{
"url": "http://"
},
{
"url": ":"
}
]
}
},
"raydium_api": {
"api": {
"endpoints": [
{
"url": "https://solana.polkachu.com",
"authentication": {
"apiKeyHeader":"x-api-key",
"apiKey":"API KEY"
}
},
{
"url": "https://connect-solana.kingnodes.com",
"authentication": {
"apiKeyHeader":"x-api-key",
"apiKey":"API KEY"
}
},
{
"url": "https://solana.lavenderfive.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://solana-rpc.rhino-apis.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://dydx.helius-rpc.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
}
]
}
},
"uniswapv3_api-ethereum": {
"api": {
"endpoints": [
{
"url": "https://ethereum.lavenderfive.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://ethereum.polkachu.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://connect-eth.kingnodes.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://ethereum-rpc.rhino-apis.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
}
]
}
},
"uniswapv3_api-base": {
"api": {
"endpoints": [
{
"url": "https://base-rpc.rhino-apis.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://connect-base.kingnodes.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://base.lavenderfive.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://base.polkachu.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
}
]
}
}
}
}
```
With the `oracle.json` file path, enter the following command to run Connect.
```shell
connect \
--marketmap-provider dydx_migration_api \
--oracle-config path/to/oracle.json
```
**The required version for Connect with dYdX v5 is `v1.0.5`.**
First, please ensure you've received your API keys for Raydium markets. If you have not received API keys, please reach out to the team in the relevant channels.
Next, place your API keys under their corresponding URLs in the following file and save it to your system. Keep the file path handy as we will pass it into a flag when running Connect.
```json oracle.json
{
"providers": {
"raydium_api": {
"api": {
"endpoints": [
{
"url": "https://solana.polkachu.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://connect-solana.kingnodes.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://solana.lavenderfive.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://solana-rpc.rhino-apis.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
},
{
"url": "https://dydx.helius-rpc.com",
"authentication": {
"apiKeyHeader": "x-api-key",
"apiKey": "API KEY"
}
}
]
}
}
}
}
```
With your dYdX node's REST API endpoint and the `oracle.json` file path, enter the following command to run Connect.
```shell
connect \
--marketmap-provider dydx_api \
--market-map-endpoint https:// \
--oracle-config path/to/oracle.json
```
**The required Connect version for the Neutron chain is `v1.0.12`.**
To run Connect, which starts the service on the default port of `8080`, enter the following command:
```shell
connect --market-map-endpoint=":"
```
**The required version for Connect with Stargaze is `v1.0.12`.**
You need to configure a custom API endpoint for use with the Osmosis provider, `https://rest.osmosis-1.interchain-apis.com`.
Set the following `oracle.json` configuration file. **Keep the file path handy** as we will pass it into a flag when running Connect.
```json oracle.json
{
"providers": {
"osmosis_api": {
"api": {
"endpoints": [
{"url": "https://rest.osmosis-1.interchain-apis.com"}
]
}
}
}
}
```
With the `oracle.json` file path, enter the following command to run Connect.
```shell
connect \
--market-map-endpoint=":" \
--oracle-config path/to/oracle.json
```
**The required Connect version for the Warden chain is `v1.0.12`.**
To run Connect, which starts the service on the default port of `8080`, enter the following command:
```shell
connect --market-map-endpoint=":"
```
**The required Connect version for the Initia chain is `v1.0.12`.**
To run Connect, which starts the service on the default port of `8080`, enter the following command:
```shell
connect --market-map-endpoint=":"
```
### Verify Connect
To verify Connect is working, run the following command:
```shell
curl 'http://localhost:8080/connect/oracle/v2/prices' | jq .
```
The output of the command should look similar to this:
```json
{
"prices": {
"ATOM/USD": "920650000",
"BITCOIN/USD": "3980283250000",
"DYDX/USD": "273682500",
"ETHEREUM/BITCOIN": "5842000",
"ETHEREUM/USD": "232550500000",
"POLKADOT/USD": "638800000",
"SOLANA/USD": "8430350000"
},
"timestamp": "2024-01-23T01:15:09.776890Z"
}
```
## Run Application Node
In order for the application to get prices from Connect, we need to add the following lines under the `[oracle]` heading in the `app.toml`.
Remember to change the `oracle_address` value to the address of your Connect sidecar.
```toml app.toml
# ... other sections
[oracle]
enabled = "true" # if you are not running a full node, set this to "false"
oracle_address = ":8080"
client_timeout = "250ms"
metrics_enabled = "true"
interval = "1500ms"
price_ttl = "10s"
```
Once your `app.toml` is configured, you may start your node as normal.
## Additional Steps
Using a remote signer? Have a distributed validator setup? Check out the [advanced setups](advanced-setups) to learn how to properly configure your validator infrastructure.