Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add derive.assets.all #4509

Closed
wants to merge 8 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions packages/api-derive/src/assets/all.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
// Copyright 2017-2022 @polkadot/api-derive authors & contributors
// SPDX-License-Identifier: Apache-2.0

import type { EventRecord, Hash } from '@polkadot/types/interfaces';
import type { FrameSystemEventRecord } from '@polkadot/types/lookup';
import type { Vec } from '@polkadot/types-codec';
import type { DeriveApi } from '../types';
import type { DeriveAsset, FetchedAssetsEntries, FetchedAssetsMetadataEntries } from './types';

import { combineLatest, concat, EMPTY, map, Observable, of, switchMap, take } from 'rxjs';

const ASSET_EVENTS: string[] = [
'Created',
'Destroyed',
'MetadataCleared',
'MetadataSet',
'OwnerChanged',
'TeamChanged',
'Issued'
];

export function extractAssetEventsHash (events: Vec<FrameSystemEventRecord>): Observable<Hash> {
const filtered = events.find(({ event: { method, section } }) => section === 'assets' && ASSET_EVENTS.includes(method));

return filtered ? of(events.createdAtHash as Hash) : EMPTY;
}

function concatAssetData ([maybeAssets, maybeMetadatas]: [maybeAssets: FetchedAssetsEntries, maybeMetadatas: FetchedAssetsMetadataEntries]): DeriveAsset[] {
const result: DeriveAsset[] = [];

maybeAssets.forEach(([id, asset], index) => {
if (asset.isSome) {
const { accounts, admin, approvals, freezer, isFrozen, isSufficient, issuer, minBalance, owner, sufficients, supply } = asset.unwrap();
const { decimals, deposit, name, symbol } = maybeMetadatas[index][1];

result.push({
accounts: accounts.toBn(),
admin,
approvals: approvals.toBn(),
decimals: decimals.toNumber(),
deposit: deposit.toBn(),
freezer,
id: id.args[0],
isFrozen: isFrozen.toHuman(),
isSufficient: isSufficient.toHuman(),
issuer,
minBalance: minBalance.toBn(),
name: name.toUtf8(),
owner,
sufficients: sufficients.toBn(),
supply: supply.toBn(),
symbol: symbol.toUtf8()
} as DeriveAsset);
}
});

return result;
}

/**
* @name all
* @returns An array containing all assets with metadata
*/

export function all (api: DeriveApi): Observable<DeriveAsset[]> {
const initialBlockHash = api.rpc.chain.subscribeNewHeads()
.pipe(take(1))
.pipe((val) => val.pipe(map(({ hash }) => hash)));
Comment on lines +66 to +68
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rpc.chain.getHeader returns the latest, no sub.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean changing to:
const initBlockHash = api.rpc.chain.getHeader().pipe(val => val.pipe(map(({ hash }) => hash)))?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. api.rpc.chain.getHeader().pipe(map(({ hash }) => hash))


return concat(
initialBlockHash,
api.query.system.events()
jacogr marked this conversation as resolved.
Show resolved Hide resolved
.pipe(switchMap((events: Vec<EventRecord>) => extractAssetEventsHash(events)))
)

.pipe(
switchMap((blockHash: Hash) =>
combineLatest([
api.query.assets.asset.entriesAt(blockHash),
api.query.assets.metadata.entriesAt(blockHash)
])
)
Comment on lines +77 to +82
Copy link
Member

@jacogr jacogr Feb 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very, very heavy operation. Basically you refresh everything no matter what has changed on which id. The events actually have the ids of those changed, so should pin-point to those and only update those.

(crowdloans is an example of this - it tries to not get everything each time something has changed, rather it only refreshes based on the events received. As the state grows, this approach would just become slower and slower...)

For the initial load, 100% - we need all, on subsequent changes, this is the hammer approach and exceptionally ineffective and adding a lot of load.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. Though still not sure how to solve it. Reason we use entriesAt is initially that multi did not get the updates from latest (non-finalized) block (polkadot-js/apps#6874 (comment)). As there is no entriesAt(blockHash, keys), I understand we need to load all, and then update state based on data from events (asset and metadata changes)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if we can get a minimal reproducable sample with API code, can actually log that on Substrate. (There could obviously be an issue with the apps UI implementation itself)

Copy link
Member

@jacogr jacogr Feb 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at this issue, which is also parachain-related - /~https://github.com/paritytech/substrate/issues/10743

Would love a sample on a single connection where -

(a) event is available
(b) query for that item is not updated

I honestly cannot replicate this (relating to assets) on a stand-alone node.

)
.pipe(map(concatAssetData));
}
226 changes: 226 additions & 0 deletions packages/api-derive/src/assets/assets.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
// Copyright 2017-2022 @polkadot/api-derive authors & contributors
// SPDX-License-Identifier: Apache-2.0

import type { GenericEventData, Vec } from '@polkadot/types';
import type { AssetId, EventRecord, Hash, Header } from '@polkadot/types/interfaces';
import type { PalletAssetsAssetMetadata } from '@polkadot/types/lookup';
import type { DeriveApi } from '../types';

import { concatMap, delay, from, Observable, ObservableInput, of, Subscription } from 'rxjs';

import { ApiPromise } from '@polkadot/api';
import { StorageKey } from '@polkadot/types';
import { BN } from '@polkadot/util';

import { createApiWithAugmentations } from '../test/helpers';
import { all } from './all';
import { DeriveAsset } from './types';

export const ALICE = '5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY';
export const BOB = '5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty';

const FIRST_ASSET_ID = 15;
const SECOND_ASSET_ID = 24;
const THIRD_ASSET_ID = 100;

const FIRST_BLOCK_HASH = '0x6443a0b46e0412e626363028115a9f2cf963eeed526b8b33e5316f08b50d0dc3';
const SECOND_BLOCK_HASH = '0x05bdcc454f60a08d427d05e7f19f240fdc391f570ab76fcb96ecca0b5823d3bf';

describe('assets derive', () => {
let api: ApiPromise;
let mockedApi: DeriveApi;
let assets: DeriveAsset[] = [];
let sub: Subscription;

const createAssetStorageKey = (id: number): StorageKey<[AssetId]> => {
const assetId = api.createType('AssetId', id);
const key = new StorageKey<[AssetId]>(api.registry, api.query.assets.asset.key(assetId));

key.setMeta(api.query.assets.asset.creator.meta);

return key;
};

beforeAll(() => {
api = createApiWithAugmentations();
mockedApi = {
query: {
assets: {
asset: {
entriesAt: (hash: Hash) => valueAt(hash, {
[FIRST_BLOCK_HASH]: [
[createAssetStorageKey(FIRST_ASSET_ID), api.createType('Option<AssetDetails>', {
isSome: () => true,
isSufficient: undefined,
owner: api.createType('AccountId', BOB),
unwrap: () => ({})
})],
[createAssetStorageKey(SECOND_ASSET_ID), api.createType('Option<AssetDetails>', {
isSome: () => true,
isSufficient: true,
owner: api.createType('AccountId', ALICE),
unwrap: () => ({})
})],
[createAssetStorageKey(THIRD_ASSET_ID), api.createType('Option<AssetDetails>', {
isSome: () => true,
owner: api.createType('AccountId', BOB),
unwrap: () => ({})
})]
],
[SECOND_BLOCK_HASH]: [
[createAssetStorageKey(FIRST_ASSET_ID), api.createType('Option<AssetDetails>', {
isSome: () => true,
isSufficient: undefined,
owner: api.createType('AccountId', BOB),
unwrap: () => ({})
})],
[createAssetStorageKey(THIRD_ASSET_ID), api.createType('Option<AssetDetails>', {
isSome: () => true,
owner: api.createType('AccountId', BOB),
unwrap: () => ({})
})]
]
})
},
metadata: {
entriesAt: (hash: Hash) => valueAt(hash, {
[FIRST_BLOCK_HASH]: [
[createAssetStorageKey(FIRST_ASSET_ID), api.createType('AssetMetadata', {
decimals: 8,
name: 'TestToken',
symbol: 'TT'
}) as PalletAssetsAssetMetadata],
[createAssetStorageKey(SECOND_ASSET_ID), api.createType('AssetMetadata', {
decimals: 10,
name: 'TestTokenExtra',
symbol: 'TTx'
}) as PalletAssetsAssetMetadata],
[createAssetStorageKey(THIRD_ASSET_ID), api.createType('AssetMetadata', {
decimals: 12,
name: 'Kusama😻',
symbol: 'KSM🤪'
}) as PalletAssetsAssetMetadata]
],
[SECOND_BLOCK_HASH]: [
[createAssetStorageKey(FIRST_ASSET_ID), api.createType('AssetMetadata', {
decimals: 8,
name: 'TestToken',
symbol: 'TT'
}) as PalletAssetsAssetMetadata],
[createAssetStorageKey(THIRD_ASSET_ID), api.createType('AssetMetadata', {
decimals: 12,
name: 'Kusama😻',
symbol: 'KSM🤪'
}) as PalletAssetsAssetMetadata]
]
})
}
},
system: {
events: () => from<ObservableInput<Vec<EventRecord>>>([
[] as unknown as Vec<EventRecord>,
Object.assign([{
event: {
data: [{ module: { error: 9, index: 34 } }, {
class: 'Normal',
paysFee: 'Yes',
weight: 397453000
}] as unknown as GenericEventData,
index: api.createType('EventId', '0x0001'),
method: 'Destroyed',
section: 'assets'
},
phase: { ApplyExtrinsic: 1 },
topics: [] as unknown as Vec<Hash>
} as unknown as EventRecord] as unknown as Vec<EventRecord>, { createdAtHash: api.createType('Header', { number: new BN(2) }).hash })
]).pipe(concatMap((val) => of(val).pipe(delay(100))))
}
},
rpc: {
chain: {
subscribeNewHeads: () => from<ObservableInput<Header>>([
api.createType('Header', { number: new BN(1) })
])
}
}
} as unknown as DeriveApi;
});

beforeEach(() => {
jest.useFakeTimers();
});

afterEach(() => {
jest.useRealTimers();
assets = [];
sub.unsubscribe();
});

it('gets current assets list', () => {
sub = all(mockedApi).subscribe((value) => { assets = value; });

expect(assets).toHaveLength(3);
expect(assets[0].id.toNumber()).toBe(FIRST_ASSET_ID);
expect(assets[1].id.toNumber()).toBe(SECOND_ASSET_ID);
expect(assets[2].id.toNumber()).toBe(THIRD_ASSET_ID);

expect(assets[0].name).toBe('TestToken');
expect(assets[1].name).toBe('TestTokenExtra');
expect(assets[2].name).toBe('Kusama😻');
});

it('updates assets list after an asset event', () => {
sub = all(mockedApi).subscribe((value) => { assets = value; });

expect(assets).toHaveLength(3);

jest.advanceTimersByTime(200);

expect(assets).toHaveLength(2);
expect(assets[0].id.toNumber()).toBe(FIRST_ASSET_ID);
expect(assets[1].id.toNumber()).toBe(THIRD_ASSET_ID);

expect(assets[0].name).toBe('TestToken');
expect(assets[1].name).toBe('Kusama😻');
});

it('does not update assets list for a non asset event', () => {
const mockedApiWithNonAssetEvent: DeriveApi = {
...mockedApi,
query: {
...mockedApi.query,
system: {
events: () => from<ObservableInput<Vec<EventRecord>>>([
[] as unknown as Vec<EventRecord>,
Object.assign([{
event: {
data: [{ module: { index: 34 } }, {
class: 'Normal',
paysFee: 'Yes',
weight: 397453000
}] as unknown as GenericEventData,
index: api.createType('EventId', '0x0001'),
method: 'IdentitySet',
section: 'identity'
},
phase: { ApplyExtrinsic: 1 },
topics: [] as unknown as Vec<Hash>
} as unknown as EventRecord] as unknown as Vec<EventRecord>, { createdAtHash: api.createType('Header', { number: new BN(2) }).hash })
]).pipe(concatMap((val) => of(val).pipe(delay(100))))
}
}
} as unknown as DeriveApi;

sub = all(mockedApiWithNonAssetEvent).subscribe((value) => { assets = value; });

expect(assets).toHaveLength(3);

jest.advanceTimersByTime(200);

expect(assets).toHaveLength(3);
});
});

function valueAt <T> (hash: Hash, value: Record<string, T>): Observable<T> {
return of(value[hash.toHex()]);
}
4 changes: 4 additions & 0 deletions packages/api-derive/src/assets/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
// Copyright 2017-2022 @polkadot/api-derive authors & contributors
// SPDX-License-Identifier: Apache-2.0

export * from './all';
30 changes: 30 additions & 0 deletions packages/api-derive/src/assets/types.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
// Copyright 2017-2022 @polkadot/api-derive authors & contributors
// SPDX-License-Identifier: Apache-2.0

import type { Option, StorageKey } from '@polkadot/types';
import type { AccountId, AssetId } from '@polkadot/types/interfaces';
import type { PalletAssetsAssetDetails, PalletAssetsAssetMetadata } from '@polkadot/types/lookup';

import { BN } from '@polkadot/util';

export type FetchedAssetsEntries = [StorageKey<[AssetId]>, Option<PalletAssetsAssetDetails>][];
export type FetchedAssetsMetadataEntries = [StorageKey<[AssetId]>, PalletAssetsAssetMetadata][];

export interface DeriveAsset {
readonly owner: AccountId;
readonly issuer: AccountId;
readonly admin: AccountId;
readonly freezer: AccountId;
readonly supply: BN;
readonly deposit: BN;
readonly minBalance: BN;
readonly isSufficient: boolean;
readonly accounts: BN;
readonly sufficients: BN;
readonly approvals: BN;
readonly isFrozen: boolean;
readonly name: string;
readonly symbol: string;
readonly decimals: number;
readonly id: AssetId;
}