Amazon Introduces ResNeSt: Strong, Split-Attention Networks

Synced
Synced
Apr 24, 2020 · 4 min read
Image for post
Image for post

The ResNet (residual neural network) neural network debuted in 2015, and quickly proved itself — winning the prestigious CVPR 2016 Best Paper Award. ResNet also took first place on three tasks in the ImageNet competition and aced the detection and segmentation tasks in the COCO competition. Over the past four years, the ResNet paper has been cited over 40,000 times, and many variations of the network have appeared.

The latest ResNet improvement comes courtesy researchers from Amazon and UC Davis, who this week unveiled their Split-Attention Networks, ResNeSt. The new network inherits ResNet’s concise and universal features and shows significant performance improvement without a large increase in the number of parameters, surpassing previous models such as ResNeXt and SEnet.

“Although image classification models continue to evolve, most downstream applications such as object detection and semantic segmentation are still using ResNet variants as the backbone network because of its simple and modular structure.”

Hang Zhang, an Applied Scientist at Amazon Lab 126 and the paper’s first author, says classification networks are usually the core component of downstream applications. While many recent classification network designs do not retain ResNet’s basic modular design, ResNet is still widely used for research on existing mainstream applications. The ResNeSt variant is therefore designed to be directly applied to such mainstream models.

Image for post
Image for post

In the paper, researchers propose a modular Split-Attention block that can distribute attention to several feature-map groups. The Split-Attention block is a computational unit composed of the feature-map group and split attention operations. By stacking those Split-Attention blocks in the style of ResNet, researchers were able to produce this new variant. ResNeSt maintains the overall ResNet structure and can be used directly for downstream tasks, without adding additional computational effort.

Image for post
Image for post
Overview of ResNeSt, SE-Net and SK-Net.

In experiments, ResNeSt outperformed other networks with similar model complexity, while its image classification performance on ImageNet easily surpassed SKNet, SENet, ResNetXt, and ResNet.

ResNeSt-50 achieves a top-1 accuracy of 81.13 percent on ImageNet, which is 1 percent higher than the previous SOTA ResNet variant. This improvement is especially meaningful for downstream tasks such as object detection and semantic segmentation. Also, in the object detection task, if researchers replace the ResNet-50 backbone network with ResNeSt-50, the ResNeSt backbone network can improve the mAP (mean Average Precision) of the model on Faster-RCNN and CascadeRCNN by about 3 percent compared with the standard ResNet baselines.

Image for post
Image for post

The paper also introduces a number of training strategies which have great reference value for the current work of general AI practitioners. For more detailed information please check out the GitHub project page.

The paper ResNeSt: Split-Attention Networks is on arXiv.

Author: Herin Zhao | Editor: Michael Sarazen

Thinking of contributing to Synced Review? Synced’s new column Share My Research welcomes scholars to share their own research breakthroughs with global AI enthusiasts.

Image for post
Image for post

We know you don’t want to miss any story. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.

Image for post
Image for post

Need a comprehensive review of the past, present and future of modern AI research development? Trends of AI Technology Development Report is out!

2018 Fortune Global 500 Public Company AI Adaptivity Report is out!
Purchase a Kindle-formatted report on Amazon.
Apply for Insight Partner Program to get a complimentary full PDF report.

Image for post
Image for post
Synced

Written by

Synced

AI Technology & Industry Review — syncedreview.com | Newsletter: http://bit.ly/2IYL6Y2 | Share My Research http://bit.ly/2TrUPMI | Twitter: @Synced_Global

SyncedReview

We produce professional, authoritative, and thought-provoking content relating to artificial intelligence, machine intelligence, emerging technologies and industrial insights.

Synced

Written by

Synced

AI Technology & Industry Review — syncedreview.com | Newsletter: http://bit.ly/2IYL6Y2 | Share My Research http://bit.ly/2TrUPMI | Twitter: @Synced_Global

SyncedReview

We produce professional, authoritative, and thought-provoking content relating to artificial intelligence, machine intelligence, emerging technologies and industrial insights.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store