Substrate Monthly Substrate Technical Newsletter — June Issue

This monthly newsletter is authored by Substrate Evangelist @AmadeusGB.

Hi, to all developers interested in Substrate technology:

Substrate Monthly — Ecosystem Technical Newsletter, produced by OneBlock+, summarizes technical progress of the Substrate framework and updates activities in the Polkadot ecosystem each month to get the community updated. Subjects include critical technical submissions, modifications and releases, design discussions, references, technical community updates and so on.

The following is the June issue. Enjoy~

Important Submissions and Releases

1. Fix Core Sharing and Utilize Scheduling Lookahead This update fixes core sharing issues and improves the use of scheduling lookahead, enhancing the efficiency of parachain scheduling and validation. Key points include:

• Core Sharing Fix: Resolves issues with events not being processed correctly when multiple parachains share the same core, ensuring parachains take turns using the core in sequence. Fixes core allocation logic, avoiding resource contention and deadlocks.
• Scheduling Lookahead Improvement: Utilizes scheduling_lookahead to allow validators and collators to pre-build and process blocks. Adjustments in scheduling algorithms optimize block generation lookahead, improving processing speed and resource utilization.
• Test Coverage: Introduces comprehensive unit and integration tests to validate the correctness and stability of core sharing and scheduling lookahead functions. These tests cover various edge and exception cases to ensure system reliability under different loads.

These technical improvements significantly enhance the efficiency of parachain scheduling and validation, improving overall system performance and stability.

2. Block Import and Validation Refactor This update refactors block import and validation logic, with key improvements including:

• Stateless Validation: Implements stateless block validation before import, reducing dependency on Mutex.
• Concurrent Checking: Allows concurrent checking of multiple competing blocks if the parent block exists, optimizing import logic.
• Independent Function Refactor: Extracts import_handler and verify_single_block_metered as independent functions for reuse.
• Improved Testing: Covers new logic and paths, ensuring functional stability.

This refactor improves the efficiency and maintainability of block import and validation.

3. Skip Tree Routing Calculation to Accelerate Finalization This update optimizes tree routing calculation logic to improve block synchronization and finalization efficiency. Key changes include:

• Skip Tree Routing Calculation: Skips tree routing calculation when leaves().len() == 1, significantly reducing computational overhead, applicable in most single-leaf cases.
• Long Distance Optimization: During long-distance finalization, checks the lowest common ancestor of all leaves, avoiding full-tree computation and using caching to speed up processing.
• Regression Testing and Verification: Adds test cases to ensure functionality correctness and stability under various conditions.

These improvements significantly enhance the performance of block synchronization and finalization processes.

4. Add IBP Boot Nodes for Kusama Network This update adds IBP boot nodes for the Kusama network’s People parachain. Key points include:

• New Boot Nodes: Adds several new boot nodes such as Dwellir, Gatotech, and Amforc, updating the boot node list to the latest configuration.
• Test Coverage: Conducts comprehensive testing with --reserved-only and --reserved-nodes options to ensure node stability and connectivity in the Kusama and Polkadot networks.
• Network Stability: Improves network node distribution, enhancing overall network stability and reliability.

These improvements significantly enhance the availability and performance of the Kusama network.

5. Glutton Module: Increase Parachain Block Length This update enhances the Glutton module to support increased parachain block lengths. Key points include:

• Block Length Configuration: Adds a block_length parameter to allow filling parachain blocks based on runtime configuration, optimizing stress testing.
• Random Data Filling: Uses junk data to fill blocks, increasing PoV (Proof of Value) size to simulate more realistic load testing scenarios.
• Remaining Weight Handling: Burns remaining weight in on_idle, adjusted by the configured storageparameter.

These improvements enhance the testing capability and flexibility of the Glutton module.

6. Approval-voting: Add Debug Information for Absent Validators This update adds debug logs in the approval-voting module to identify absent validators and parachains. Key points include:

• Debug Logs: Records information about absent validators and parachains, aiding in identifying and analyzing common absence issues.
• Index Querying: Uses validator_index to query chain information, precisely locating absent validators.
• Improved Monitoring: Enhances the ability to monitor validator behavior, facilitating debugging and maintenance.

These improvements increase the efficiency of validator monitoring and issue diagnosis.

7. Add Runtime::OmniNode Variant to polkadot-parachain This update introduces the Runtime::OmniNode variant in polkadot-parachain. Key changes include:

• Multi-chain Support: The Runtime::OmniNode variant supports running multiple parachains in the Polkadot network, enhancing flexibility.
• CLI Tool Improvement: Optimizes the command-line interface (CLI) tool, enhancing user experience.
• Code Refactoring: Refactors parts of the code, improving maintainability.

These changes enhance the functionality and user experience of polkadot-parachain.

8. RPC Server: Add Health/Readiness Endpoints This update adds health and readiness endpoints to the RPC server to better monitor service status. Key points include:

• Health Check Endpoint: /health provides basic health information about the node, including the number of connected nodes and sync status.
• Readiness Endpoint: /health/readiness checks if the node is ready to process requests, returning detailed internal state information.
• Improved Monitoring: These endpoints provide more detailed and timely node status information, aiding operational monitoring and debugging.

This feature enhances node monitoring capabilities, improving system maintainability and reliability.

Design Proposals and Discussions

1. Support for Unpaid Asset Transfers in XCM This issue explores solutions for supporting unpaid asset transfers in XCM. Key discussion points include:

• Existing Options: Currently, only custom local XCM and logic can be used, which is error-prone and lacks future scalability.
• Future Version Support: Introduce XCMv5 or add an “unpaid” variant to pallet_xcm::transfer_assets() for more reliable unpaid execution.
• Proposed Method: Introduce an auxiliary pallet to build and execute unpaid asset transfers on both local and remote sides via pallet-xcm, reducing complexity and error risk.

This discussion aims to provide a secure and reliable implementation for unpaid asset transfers, enhancing XCM’s flexibility and usability.

2. Adjust the Current Inflation Formula This design proposal outlines a blueprint to make the existing inflation formula adjustable. Key points include:

• Parameter Module Introduction: Proposes introducing a pallet-parameters module to allow modifying inflation parameters via StakingAdmin, with restrictions on adjustments beyond specific ranges requiring Root authority.
• Dynamic Parameter Adjustment: Allows dynamic adjustment of inflation rates based on network needs, optimizing the network economic model.
• Permission Control: Ensures permission management during inflation adjustments to limit potential abuse risks.

This design aims to enhance the flexibility and security of inflation parameter adjustments.

3. New #[runtime] Macro Should Use Actual Rust Types for Allocation This issue discusses that the #[runtime]macro should use actual Rust types instead of simplified forms for allocation. Key discussion points include:

• Current Issue: The existing macro assigns types to crate names rather than actual Pallet<T> objects, causing code to deviate from Rust standards.
• Proposed Change: Update the macro definition to point to Pallet<T> objects, making the code more accurate and Rust-compliant. For example, updating pub type System = frame_system to pub type System = frame_system::Pallet<Runtime>.
• Compatibility: Ensure the new macro definition does not break compatibility with existing user code while providing a better foundation for future development.

This discussion ensures the macro definition is more accurate, improving code readability and standardization.

4. XCM Token Matching Too Lenient This issue explores the overly lenient matching logic of the XCM token matcher within tuples. The main discussion points include:

• Existing Issue: The current matcher continues matching on all errors when executing within tuples, which can lead to storage pollution and hard-to-debug errors. The matcher should only continue on AssetNotFoundor Unimplemented errors, not all errors.
• Proposed Change: Modify the implementation of MatchesFungibles and MatchesNonFungibles to continue matching only on the specified errors and return an error otherwise. For example:

match Tuple::matches_fungibles(a) {
      Err(AssetNotFound | Unimplemented) => (),
      o => return o,
}

• Consistency Improvement: Discussion on the inconsistency in return types of Matches<Singular> and Matches<Plural>, suggesting a unified return type of Result.

These discussions and changes aim to improve the accuracy of XCM token matching and system stability.

5. XCM: Multi-hop Mixed Asset Transfers Failing Due to Lack of Transfer Fees This issue explores the reasons for and solutions to the failure of multi-hop mixed asset transfers. The main discussion points include:

• Existing Issue: During multi-hop mixed asset transfers (e.g., from a parachain to an asset hub and then to the relay chain), the lack of transfer fees at intermediate hops (such as the asset hub) causes InitiateTeleportto fail. The existing solution cannot handle complex transfers, especially involving a mix of ReserveAssetand Teleport.
• Proposed Solution: Charge transfer fees from the transferred assets (if available) in the executor, ensuring sufficient fees for each step of the transfer.
• Improvement Goal: Simplify the multi-hop asset transfer process and improve the success rate by directly charging fees from the transferred assets.

This discussion aims to enhance the multi-hop asset transfer mechanism, ensuring XCM’s flexibility and reliability.

6. Block-Based Collators This issue discusses shifting from slot-based collators to block-based collators to improve performance under high load. The main discussion points include:

• Existing Issue: The current slot-based collators need to wait for the relay chain block to verify timestamps, causing performance bottlenecks under high load.
• Proposed Solution: Use the relay chain block number as the time metric, storing the last generated relay chain block number s and calculating the legitimate block producer index k based on n and s when generating new blocks. This reduces wait time and speeds up block generation.
• Improvement Goal: Improve block production efficiency by reducing the wait time for relay chain blocks, supporting elastic scaling.

This discussion aims to optimize the block production process and enhance system performance and scalability under high load.

7. XCM Asset Transfer Program Builder This issue discusses the introduction of an XCM asset transfer program builder. The main discussion points include:

• Background and Current State: Cross-chain asset transfers are currently difficult to generalize, with each parachain using its own asset transfer pallet, complicating integration. The introduction of XCM programs offers high flexibility, but building and executing low-level XCM programs is complex and error-prone.
• Proposed Solution: Introduce a Rust builder pattern tool that simplifies XCM program construction through high-level helper tools and libraries. This tool will perform sanity checks, handle asset relocation across different chains, and generate reliable XCM programs.
• Example: A specific code example is provided, demonstrating how to use this tool for complex cross-chain asset transfers, such as transferring assets from the Hydration Network to Moonbeam.

This proposal aims to improve the usability and reliability of XCM asset transfers, simplifying developers’ work.

8. Treasury API for Cross-Chain Transfers This issue proposes an API for scheduling XCM asset transfers from chain X, extracting assets on chain Y, and depositing them on chain Z. The main discussion points include:

• Existing Issue: The current treasury expenditure calls only support transferring assets to chain Y, and the beneficiary must be on chain Y, lacking flexibility for complex cross-chain asset transfer needs.
• Proposed Solution: Introduce LocatableAssetId and LocatableBeneficiary types to allow the API to handle more complex cross-chain transfers. The new API will enable asset extraction and deposit across different chains, improving flexibility and compatibility.
• Implementation Example:

fn treasury_spend (
   asset_kind: LocatableAssetId {
      location: xcm::Location,
      asset_id: xcm::AssetId,
   },
   beneficiary: LocatableBeneficiary {
      location: xcm::Location,
      account: xcm::Location,
   }
);

This design enhances the flexibility of treasury calls, enabling them to support more complex cross-chain asset transfer scenarios.

Documentation and Resources

1. Custom RPC V2 Reference Documentation This update introduces new custom RPC reference documentation, detailing the implementation and usage of custom RPC interfaces. Key contents include:

• Background Introduction: An overview of the storage query and subscription capabilities provided by Polkadot-SDK, explaining the necessity and application scenarios of custom RPC.
• Creating Custom RPC: Detailed step-by-step guide covering how to define and implement new RPC methods, including necessary code examples and annotations.
• Node Integration: Introduction on how to integrate custom RPC into nodes to ensure they operate correctly within the network.
• Testing and Validation: Provides testing strategies and examples to ensure the correctness and stability of custom RPC functionalities.

This documentation provides developers with comprehensive guidance to help them quickly implement and integrate custom RPC functions.

2. Rewrite “Add Pallet to Runtime” Tutorial This update rewrites the “Add Pallet to Runtime” tutorial to replace the deprecated Nicks Pallet. Main changes include:

• New Pallet: Replacing the Nicks Pallet with the Lottery Pallet to ensure the tutorial content remains modern and relevant.
• Detailed Steps: Provides detailed steps from creation to integration and configuration of the new Pallet, covering code examples and explanations.
• Build and Test: Guides users to successfully build and test the new Pallet integration, ensuring all steps are smooth.

This documentation update improves the practicality and operability of the tutorial, helping developers better master Substrate development.

Technical Ecosystem and Community

1. Substrate Introductory Course, 15th Session: On the evening of May 31, the 15th session of the Substrate introductory course seminar was held, attracting nearly 700 developers. Mr. Jun Zhou shared the JAM chain and its technical advantages and introduced the upgraded course content, including basic theory, Polkadot ecosystem, Rust and Substrate technology learning, and mini hackathon practical exercises, helping novice developers quickly master Substrate development skills. The course offers multiple incentives and referral opportunities, attracting widespread attention.
2. W3F Announces Second Round of Decentralized Voices Delegates, OneBlock+ Granted 4.2 Million DOT Voting Rights: The Web3 Foundation announced the second round of delegates for the Polkadot and Kusama Decentralized Voices (DV) program. The OneBlock+ community became the only Asian Polkadot representative selected, receiving 4.2 million DOT voting rights. This program aims to incentivize community members to actively participate in on-chain governance by delegating voting rights, increasing network transparency and inclusivity. The list of delegates includes 10 Polkadot representatives and 10 Kusama representatives, emphasizing the importance of governance transparency and community participation.
3. USD 100 Million Fund to Boost Polkadot Ecosystem Development, JAM Implementers Awards Launched: Harbour Industrial Capital announced the launch of a USD 100 million fund to support the development of the Polkadot ecosystem. This fund will aid the JAM chain implementers awards, encouraging developers to create more implementations for the JAM chain, enhancing the ecosystem’s diversity and stability. Harbour Industrial Capital’s CEO Max Rebol detailed this plan in an interview, emphasizing its vision and mission for the Polkadot ecosystem.
4. Gavin and Vitalik on Stage Together! How Do These Decade-Long Friends View Web3 Now?: On May 31, Gavin Wood and Vitalik Buterin had a fireside chat at the EthPrague event, discussing topics such as the Web3 community, governance, incentive mechanisms, and competition among L1 chains. Gavin explained the technical advantages of the JAM chain, while Vitalik emphasized the growth of the Ethereum ecosystem. The two tech leaders explored the impact of conflicts on community building and shared their insights on on-chain and off-chain governance. This conversation provided valuable perspectives on the future development of Web3.
5. Polkadot Treasury Allocates 3 Million DOT to Promote DeFi Growth: The Polkadot community approved three proposals through the OpenGov system, allocating 3 million DOT to enhance the DeFi ecosystem. Hydration received 2 million DOT for liquidity mining incentives and non-custodial liquidity. StellaSwap received 1 million DOT to increase and deepen the market depth of strategic assets. These initiatives demonstrate the important role of community-driven decisions in advancing the Polkadot ecosystem.
6. Polkadot Co-Founder Rob Proposes “Plaza” System Chain: Polkadot co-founder Rob proposed evolving AssetHub into a system chain named “Plaza,” integrating asset issuance, smart contracts, staking, and bridging functions to unify user experience. The Plaza plan aims to enhance scalability and usability by centralizing resources and functions, simplifying operations for developers and users. The proposal also includes integrating smart contracts into Plaza to support more decentralized applications and services.

Cross-Chain Protocols

1. Allowing Transfers of Any Asset Between Two Asset Hubs: This update allows transfers of any asset, including Ethereum assets, between Rococo and Westend asset hubs. Key details include:

• Cross-Chain Asset Reserve Management: Allow Rococo asset hub to act as a reserve manager for assets on the Westend asset hub and vice versa. Ethereum contracts can also act as reserve managers.
• Dual Bridge Connection: Westend can access Ethereum assets through a dual bridge connection (W<>R<>Eth).
• Enhanced Testing: Added simulation tests for different assets (local, trusted, external/bridged) in AHR -> AHW and AHW -> AHR directions.

This functionality sets the stage for bridging between Polkadot and Kusama, enhancing cross-chain asset management flexibility and compatibility.

2. Runtime API for LocationToAccount Conversion: This update introduces a new Runtime API for converting Location to AccountId in XCM operations. Key details include:

• API Functions: Provide convert_location and convert_account methods for bidirectional conversion between Location and AccountId.
• Unified Module: Merge xcm-runtime-api and xcm-fee-payment-runtime-api into a single module to simplify the structure.
• Test Coverage: Added corresponding tests in all testnets to ensure functionality and stability.

This feature enhances the flexibility and usability of XCM operations, providing better support for cross-chain communication.

3. Snowbridge: Syncing Snowfork Repository: This update syncs several improvements from the Snowfork repository, including:

• Ethereum Client Sync: The Ethereum client sends the execution head as proof when receiving messages, optimizing performance by not verifying all execution heads.
• Transfer Handling: If the AH sovereign account balance is insufficient, message processing does not fail, but relayers will not receive rewards.
• Local Testing: Includes end-to-end tests, Cargo tests, and Clippy tests to ensure the stability of new features.

These enhancements improve the functionality and reliability of Snowbridge.

4. Introducing Slot-Based Collators: This update introduces a new slot-based collator to optimize block generation timing and performance. Key changes include:

• Slot-Based Generation: Collators wait for the next parachain slot before starting block production. Compared to proactive collators, this reduces the waiting time for relay chain blocks.
• Task Separation: Introduced separate block building and collating tasks in the cumulus/client/consensus/aura module, improving task management clarity.
• Testing and Verification: Added a test environment with a 2-second slot duration in cumulus-test-runtimeand tested the new collator's performance through ZombieNet.

This improvement significantly increases block generation efficiency, especially under high-load scenarios.

About OneBlock+

OneBlock+ is the first and the largest blockchain developer community in China. At OneBlock+, we provide full support for developers with their substrate studies and further set off their career paths. We host Polkadot Hackathons every season to attract top-notch developers to build and innovate for the prosperity of the ecosystem. As a greater China technology resource integrator, OneBlock+ also partners with developers, communities, business elites, and key media who have business insights and experiences in the blockchain industry to provide educational events, such as technical courses, webinars, AMAs, and offline events for the industry. Want to shape the crypto world together? Come and join us today!

Twitter / Telegram / Discord / YouTube