-
Notifications
You must be signed in to change notification settings - Fork 359
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add derive.assets.all #4509
Add derive.assets.all #4509
Changes from all commits
a275f0c
9b5202d
a6fb968
cb3f11b
60c296b
bcb2336
c5a2edd
4c5ec40
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
// Copyright 2017-2022 @polkadot/api-derive authors & contributors | ||
// SPDX-License-Identifier: Apache-2.0 | ||
|
||
import type { EventRecord, Hash } from '@polkadot/types/interfaces'; | ||
import type { FrameSystemEventRecord } from '@polkadot/types/lookup'; | ||
import type { Vec } from '@polkadot/types-codec'; | ||
import type { DeriveApi } from '../types'; | ||
import type { DeriveAsset, FetchedAssetsEntries, FetchedAssetsMetadataEntries } from './types'; | ||
|
||
import { combineLatest, concat, EMPTY, map, Observable, of, switchMap, take } from 'rxjs'; | ||
|
||
const ASSET_EVENTS: string[] = [ | ||
'Created', | ||
'Destroyed', | ||
'MetadataCleared', | ||
'MetadataSet', | ||
'OwnerChanged', | ||
'TeamChanged', | ||
'Issued' | ||
]; | ||
|
||
export function extractAssetEventsHash (events: Vec<FrameSystemEventRecord>): Observable<Hash> { | ||
const filtered = events.find(({ event: { method, section } }) => section === 'assets' && ASSET_EVENTS.includes(method)); | ||
|
||
return filtered ? of(events.createdAtHash as Hash) : EMPTY; | ||
} | ||
|
||
function concatAssetData ([maybeAssets, maybeMetadatas]: [maybeAssets: FetchedAssetsEntries, maybeMetadatas: FetchedAssetsMetadataEntries]): DeriveAsset[] { | ||
const result: DeriveAsset[] = []; | ||
|
||
maybeAssets.forEach(([id, asset], index) => { | ||
if (asset.isSome) { | ||
const { accounts, admin, approvals, freezer, isFrozen, isSufficient, issuer, minBalance, owner, sufficients, supply } = asset.unwrap(); | ||
const { decimals, deposit, name, symbol } = maybeMetadatas[index][1]; | ||
|
||
result.push({ | ||
accounts: accounts.toBn(), | ||
admin, | ||
approvals: approvals.toBn(), | ||
decimals: decimals.toNumber(), | ||
deposit: deposit.toBn(), | ||
freezer, | ||
id: id.args[0], | ||
isFrozen: isFrozen.toHuman(), | ||
isSufficient: isSufficient.toHuman(), | ||
issuer, | ||
minBalance: minBalance.toBn(), | ||
name: name.toUtf8(), | ||
owner, | ||
sufficients: sufficients.toBn(), | ||
supply: supply.toBn(), | ||
symbol: symbol.toUtf8() | ||
} as DeriveAsset); | ||
} | ||
}); | ||
|
||
return result; | ||
} | ||
|
||
/** | ||
* @name all | ||
* @returns An array containing all assets with metadata | ||
*/ | ||
|
||
export function all (api: DeriveApi): Observable<DeriveAsset[]> { | ||
const initialBlockHash = api.rpc.chain.subscribeNewHeads() | ||
.pipe(take(1)) | ||
.pipe((val) => val.pipe(map(({ hash }) => hash))); | ||
|
||
return concat( | ||
initialBlockHash, | ||
api.query.system.events() | ||
jacogr marked this conversation as resolved.
Show resolved
Hide resolved
|
||
.pipe(switchMap((events: Vec<EventRecord>) => extractAssetEventsHash(events))) | ||
) | ||
|
||
.pipe( | ||
switchMap((blockHash: Hash) => | ||
combineLatest([ | ||
api.query.assets.asset.entriesAt(blockHash), | ||
api.query.assets.metadata.entriesAt(blockHash) | ||
]) | ||
) | ||
Comment on lines
+77
to
+82
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a very, very heavy operation. Basically you refresh everything no matter what has changed on which id. The events actually have the ids of those changed, so should pin-point to those and only update those. (crowdloans is an example of this - it tries to not get everything each time something has changed, rather it only refreshes based on the events received. As the state grows, this approach would just become slower and slower...) For the initial load, 100% - we need all, on subsequent changes, this is the hammer approach and exceptionally ineffective and adding a lot of load. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Understood. Though still not sure how to solve it. Reason we use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So if we can get a minimal reproducable sample with API code, can actually log that on Substrate. (There could obviously be an issue with the apps UI implementation itself) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Take a look at this issue, which is also parachain-related - /~https://github.com/paritytech/substrate/issues/10743 Would love a sample on a single connection where - (a) event is available I honestly cannot replicate this (relating to assets) on a stand-alone node. |
||
) | ||
.pipe(map(concatAssetData)); | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,226 @@ | ||
// Copyright 2017-2022 @polkadot/api-derive authors & contributors | ||
// SPDX-License-Identifier: Apache-2.0 | ||
|
||
import type { GenericEventData, Vec } from '@polkadot/types'; | ||
import type { AssetId, EventRecord, Hash, Header } from '@polkadot/types/interfaces'; | ||
import type { PalletAssetsAssetMetadata } from '@polkadot/types/lookup'; | ||
import type { DeriveApi } from '../types'; | ||
|
||
import { concatMap, delay, from, Observable, ObservableInput, of, Subscription } from 'rxjs'; | ||
|
||
import { ApiPromise } from '@polkadot/api'; | ||
import { StorageKey } from '@polkadot/types'; | ||
import { BN } from '@polkadot/util'; | ||
|
||
import { createApiWithAugmentations } from '../test/helpers'; | ||
import { all } from './all'; | ||
import { DeriveAsset } from './types'; | ||
|
||
export const ALICE = '5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY'; | ||
export const BOB = '5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty'; | ||
|
||
const FIRST_ASSET_ID = 15; | ||
const SECOND_ASSET_ID = 24; | ||
const THIRD_ASSET_ID = 100; | ||
|
||
const FIRST_BLOCK_HASH = '0x6443a0b46e0412e626363028115a9f2cf963eeed526b8b33e5316f08b50d0dc3'; | ||
const SECOND_BLOCK_HASH = '0x05bdcc454f60a08d427d05e7f19f240fdc391f570ab76fcb96ecca0b5823d3bf'; | ||
|
||
describe('assets derive', () => { | ||
let api: ApiPromise; | ||
let mockedApi: DeriveApi; | ||
let assets: DeriveAsset[] = []; | ||
let sub: Subscription; | ||
|
||
const createAssetStorageKey = (id: number): StorageKey<[AssetId]> => { | ||
const assetId = api.createType('AssetId', id); | ||
const key = new StorageKey<[AssetId]>(api.registry, api.query.assets.asset.key(assetId)); | ||
|
||
key.setMeta(api.query.assets.asset.creator.meta); | ||
|
||
return key; | ||
}; | ||
|
||
beforeAll(() => { | ||
api = createApiWithAugmentations(); | ||
mockedApi = { | ||
query: { | ||
assets: { | ||
asset: { | ||
entriesAt: (hash: Hash) => valueAt(hash, { | ||
[FIRST_BLOCK_HASH]: [ | ||
[createAssetStorageKey(FIRST_ASSET_ID), api.createType('Option<AssetDetails>', { | ||
isSome: () => true, | ||
isSufficient: undefined, | ||
owner: api.createType('AccountId', BOB), | ||
unwrap: () => ({}) | ||
})], | ||
[createAssetStorageKey(SECOND_ASSET_ID), api.createType('Option<AssetDetails>', { | ||
isSome: () => true, | ||
isSufficient: true, | ||
owner: api.createType('AccountId', ALICE), | ||
unwrap: () => ({}) | ||
})], | ||
[createAssetStorageKey(THIRD_ASSET_ID), api.createType('Option<AssetDetails>', { | ||
isSome: () => true, | ||
owner: api.createType('AccountId', BOB), | ||
unwrap: () => ({}) | ||
})] | ||
], | ||
[SECOND_BLOCK_HASH]: [ | ||
[createAssetStorageKey(FIRST_ASSET_ID), api.createType('Option<AssetDetails>', { | ||
isSome: () => true, | ||
isSufficient: undefined, | ||
owner: api.createType('AccountId', BOB), | ||
unwrap: () => ({}) | ||
})], | ||
[createAssetStorageKey(THIRD_ASSET_ID), api.createType('Option<AssetDetails>', { | ||
isSome: () => true, | ||
owner: api.createType('AccountId', BOB), | ||
unwrap: () => ({}) | ||
})] | ||
] | ||
}) | ||
}, | ||
metadata: { | ||
entriesAt: (hash: Hash) => valueAt(hash, { | ||
[FIRST_BLOCK_HASH]: [ | ||
[createAssetStorageKey(FIRST_ASSET_ID), api.createType('AssetMetadata', { | ||
decimals: 8, | ||
name: 'TestToken', | ||
symbol: 'TT' | ||
}) as PalletAssetsAssetMetadata], | ||
[createAssetStorageKey(SECOND_ASSET_ID), api.createType('AssetMetadata', { | ||
decimals: 10, | ||
name: 'TestTokenExtra', | ||
symbol: 'TTx' | ||
}) as PalletAssetsAssetMetadata], | ||
[createAssetStorageKey(THIRD_ASSET_ID), api.createType('AssetMetadata', { | ||
decimals: 12, | ||
name: 'Kusama😻', | ||
symbol: 'KSM🤪' | ||
}) as PalletAssetsAssetMetadata] | ||
], | ||
[SECOND_BLOCK_HASH]: [ | ||
[createAssetStorageKey(FIRST_ASSET_ID), api.createType('AssetMetadata', { | ||
decimals: 8, | ||
name: 'TestToken', | ||
symbol: 'TT' | ||
}) as PalletAssetsAssetMetadata], | ||
[createAssetStorageKey(THIRD_ASSET_ID), api.createType('AssetMetadata', { | ||
decimals: 12, | ||
name: 'Kusama😻', | ||
symbol: 'KSM🤪' | ||
}) as PalletAssetsAssetMetadata] | ||
] | ||
}) | ||
} | ||
}, | ||
system: { | ||
events: () => from<ObservableInput<Vec<EventRecord>>>([ | ||
[] as unknown as Vec<EventRecord>, | ||
Object.assign([{ | ||
event: { | ||
data: [{ module: { error: 9, index: 34 } }, { | ||
class: 'Normal', | ||
paysFee: 'Yes', | ||
weight: 397453000 | ||
}] as unknown as GenericEventData, | ||
index: api.createType('EventId', '0x0001'), | ||
method: 'Destroyed', | ||
section: 'assets' | ||
}, | ||
phase: { ApplyExtrinsic: 1 }, | ||
topics: [] as unknown as Vec<Hash> | ||
} as unknown as EventRecord] as unknown as Vec<EventRecord>, { createdAtHash: api.createType('Header', { number: new BN(2) }).hash }) | ||
]).pipe(concatMap((val) => of(val).pipe(delay(100)))) | ||
} | ||
}, | ||
rpc: { | ||
chain: { | ||
subscribeNewHeads: () => from<ObservableInput<Header>>([ | ||
api.createType('Header', { number: new BN(1) }) | ||
]) | ||
} | ||
} | ||
} as unknown as DeriveApi; | ||
}); | ||
|
||
beforeEach(() => { | ||
jest.useFakeTimers(); | ||
}); | ||
|
||
afterEach(() => { | ||
jest.useRealTimers(); | ||
assets = []; | ||
sub.unsubscribe(); | ||
}); | ||
|
||
it('gets current assets list', () => { | ||
sub = all(mockedApi).subscribe((value) => { assets = value; }); | ||
|
||
expect(assets).toHaveLength(3); | ||
expect(assets[0].id.toNumber()).toBe(FIRST_ASSET_ID); | ||
expect(assets[1].id.toNumber()).toBe(SECOND_ASSET_ID); | ||
expect(assets[2].id.toNumber()).toBe(THIRD_ASSET_ID); | ||
|
||
expect(assets[0].name).toBe('TestToken'); | ||
expect(assets[1].name).toBe('TestTokenExtra'); | ||
expect(assets[2].name).toBe('Kusama😻'); | ||
}); | ||
|
||
it('updates assets list after an asset event', () => { | ||
sub = all(mockedApi).subscribe((value) => { assets = value; }); | ||
|
||
expect(assets).toHaveLength(3); | ||
|
||
jest.advanceTimersByTime(200); | ||
|
||
expect(assets).toHaveLength(2); | ||
expect(assets[0].id.toNumber()).toBe(FIRST_ASSET_ID); | ||
expect(assets[1].id.toNumber()).toBe(THIRD_ASSET_ID); | ||
|
||
expect(assets[0].name).toBe('TestToken'); | ||
expect(assets[1].name).toBe('Kusama😻'); | ||
}); | ||
|
||
it('does not update assets list for a non asset event', () => { | ||
const mockedApiWithNonAssetEvent: DeriveApi = { | ||
...mockedApi, | ||
query: { | ||
...mockedApi.query, | ||
system: { | ||
events: () => from<ObservableInput<Vec<EventRecord>>>([ | ||
[] as unknown as Vec<EventRecord>, | ||
Object.assign([{ | ||
event: { | ||
data: [{ module: { index: 34 } }, { | ||
class: 'Normal', | ||
paysFee: 'Yes', | ||
weight: 397453000 | ||
}] as unknown as GenericEventData, | ||
index: api.createType('EventId', '0x0001'), | ||
method: 'IdentitySet', | ||
section: 'identity' | ||
}, | ||
phase: { ApplyExtrinsic: 1 }, | ||
topics: [] as unknown as Vec<Hash> | ||
} as unknown as EventRecord] as unknown as Vec<EventRecord>, { createdAtHash: api.createType('Header', { number: new BN(2) }).hash }) | ||
]).pipe(concatMap((val) => of(val).pipe(delay(100)))) | ||
} | ||
} | ||
} as unknown as DeriveApi; | ||
|
||
sub = all(mockedApiWithNonAssetEvent).subscribe((value) => { assets = value; }); | ||
|
||
expect(assets).toHaveLength(3); | ||
|
||
jest.advanceTimersByTime(200); | ||
|
||
expect(assets).toHaveLength(3); | ||
}); | ||
}); | ||
|
||
function valueAt <T> (hash: Hash, value: Record<string, T>): Observable<T> { | ||
return of(value[hash.toHex()]); | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
// Copyright 2017-2022 @polkadot/api-derive authors & contributors | ||
// SPDX-License-Identifier: Apache-2.0 | ||
|
||
export * from './all'; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
// Copyright 2017-2022 @polkadot/api-derive authors & contributors | ||
// SPDX-License-Identifier: Apache-2.0 | ||
|
||
import type { Option, StorageKey } from '@polkadot/types'; | ||
import type { AccountId, AssetId } from '@polkadot/types/interfaces'; | ||
import type { PalletAssetsAssetDetails, PalletAssetsAssetMetadata } from '@polkadot/types/lookup'; | ||
|
||
import { BN } from '@polkadot/util'; | ||
|
||
export type FetchedAssetsEntries = [StorageKey<[AssetId]>, Option<PalletAssetsAssetDetails>][]; | ||
export type FetchedAssetsMetadataEntries = [StorageKey<[AssetId]>, PalletAssetsAssetMetadata][]; | ||
|
||
export interface DeriveAsset { | ||
readonly owner: AccountId; | ||
readonly issuer: AccountId; | ||
readonly admin: AccountId; | ||
readonly freezer: AccountId; | ||
readonly supply: BN; | ||
readonly deposit: BN; | ||
readonly minBalance: BN; | ||
readonly isSufficient: boolean; | ||
readonly accounts: BN; | ||
readonly sufficients: BN; | ||
readonly approvals: BN; | ||
readonly isFrozen: boolean; | ||
readonly name: string; | ||
readonly symbol: string; | ||
readonly decimals: number; | ||
readonly id: AssetId; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rpc.chain.getHeader
returns the latest, no sub.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean changing to:
const initBlockHash = api.rpc.chain.getHeader().pipe(val => val.pipe(map(({ hash }) => hash)))
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed.
api.rpc.chain.getHeader().pipe(map(({ hash }) => hash))