GUI Component Design Gallery Through Invisible Crowdsourcing
This post summarises our research paper titled “Gallery D.C.: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery” by Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. The paper will be presented at the ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW) on November 11th, 2019. A GUI component design gallery is proposed in http://mui-collection.herokuapp.com/.
Design sharing (Figure 1) is a routine activity through which design knowledge, creativity and aesthetics is exchanged among designers. Through design sharing, the designers can gain feedback from the community and boost their portfolio. Meanwhile, the repository of the shared GUI designs provides an opportunity to learn about GUI designs, gain design inspiration and understand design trends.
Invisible crowdsourcing is one of the emerging Web 2.0 based phenomena and enables a broader collaboration indirectly with thousands of designers who craft the design of the world-wide popular mobile apps. Existing design sharing platforms have three fundamental limitations in terms of design practicality, design granularity (11 types of GUI components for Android APP (Figure 2)) and design knowledge discovery to demonstrate the problem of the practical use of certain design ideas in real applications, supporting the sharing of the overall design of the GUI only and tag-based search of the GUI designs, respectively.
With millions of real-world applications — each comprising multiple UI designs of human creativity and aesthetics — the application market provides an opportunity to learn about practical GUI designs on a truly massive scale, even though the GUI of these applications are not created for the purpose of design sharing and knowledge discovery. However, collecting and inspecting real-application GUIs in an ad-hoc fashion cannot effectively exploit this gold mine of GUI design resources. We found that the application introduction screenshot (Figure 3) in market usually illustrates the most important features and the best-designed GUIs of an application which enables us to train a deep-learning object detection model (Figure 4) to automatically wirify (i.e., decompose) the component into a set of GUI components.
To enhance the model training, we develop a GUI-specific image-augmentation method to transform the application real world screenshots into similar style of application introduction screenshots. In addition to the component metadata, we also detect the primary color of GUI components in HSV color space [2], and collect app metadata such as app category and download times.
Based on the collected real world application and introduction GUI screenshots, the GUI components in these screenshots, and the application and GUI component metadata, we build a web application Gallery D.C. (http://mui-collection.herokuapp.com/) for GUI designers and developers to access the large-scale GUI and GUI component designs. The users can perform multi-faceted search by component types, size, color, application category and/or displayed text. The system can report the design demographics of the GUI designs in the entire gallery or in the design search results. The system also allows users to select and visually compare the GUI designs of different companies in a comparative shopping manner. These design search, summarization and comparison features enable advanced design knowledge discovery of real-world application GUI designs on a truly massive scale, beyond the content sharing of GUI design artworks. Watch the video for more information on Gallery D.C.
Conclusion
We present a deep-learning based approach for harvesting invisible crowdsourcing GUI design resources in the application market. Those invisible design resources are made explicit in our design gallery, they complement existing design sharing platforms by exposing designer to not only design creativity and aesthetics but also design practicality. Furthermore, our design gallery contains both whole GUI designs and GUI components which give designers more flexibility to access design knowledge at different granularities. Last but not least, our design gallery supports multi-faceted design search, design demographics summarization, and design comparison shopping. These design applications can be incorporated into existing design sharing platforms to extend them beyond content sharing and curation.
This blog is posted by Sidong Feng, with edits from Chunyang Chen. Sidong Feng is a research student at Australian National University.
Citation: Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, Jinshui Wang. 2019. Gallery D.C.: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 180 (November 2019), 22 pages.
Reference
[1] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems. 91–99.
[2] Bashir Muhammad and Syed Abd Rahman Abu-Bakar. 2015. A hybrid skin color detection using HSV and YCgCr color space for face detection. In Signal and Image Processing Applications (ICSIPA), 2015 IEEE International Conference on. IEEE, 95–98.
