TurkEyes logo TurkEyes

A web-based toolbox for crowdsourcing attention data

We present a toolbox of four different user interfaces for collecting crowdsourced attention data, without using eye tracking. These interfaces can be used to approximate eye movements, measure importance of graphic design elements, capture exploration on large-scale visualizations, measure attention over time, and more. The goal of this project is to make human attention data an accesible tool for researchers and creators.

Figure comparing all four interfaces.
Newman, A., McNamara, B., Fosco, C., Zhang, Y.B., Sukhum, P., Tancik, M., Kim, N.W., Bylinskii, Z. TurkEyes: A Web-Based Toolbox for Crowdsourcing Attention Data. In ACM CHI, 2020.

ZoomMaps icon ZoomMaps

Participants use the pinch-zoom gesture on a mobile phone to explore image content.

Publications

  • Bylinskii, Z., Newman, A., McNamara, B., Tancik, M. , Madan, S., Durand, F., Oliva, A. ZoomMaps: Using Zoom to Capture Areas of Interest on Images. Vision Sciences Society (VSS) Conference, 2019
  • Newman, A., McNamara, B. Oliva, A., Bylinskii, Z. Exploring Attention on Large-scale Visualizations using ZoomMaps, a Zoomable Crowdsourced Interface Vision Sciences Society (VSS) Conference, VISXVISION Satellite Event, 2019

CodeCharts icon CodeCharts

Participants specify where on an image they gazed using a grid of codes that appears after image presentation.

Publications

  • Rudoy, D., Goldman, D.B., Shechtman, E., & Zelnik-Manor, L. Crowdsourcing Gaze Data Collection. Collective Intelligence conference, 2012
  • Fosco, C.*, Newman, A.*, Sukhum, P., Zhang Y.B., Zhao, N., Oliva, A., Bylinskii, Z. How many glances? Modeling multi-duration saliency. SVRHM Workshop at NeurIPS, 2019.

ImportAnnots icon ImportAnnots

Participants paint over regions of a design they consider important using binary masks.

Publications

  • O'Donovan, P., Agarwala, A., & Hertzmann, A. Learning Layouts for Single-Page Graphic Designs. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2014
  • Bylinskii, Z., Kim, N.W., O'Donovan, P., Alsheikh, S., Madan, S., Pfister, H., Durand, F., Russell, B., & Hertzmann, A. Learning Visual Importance for Graphic Designs and Data Visualizations. ACM Symposium on User Interface Software and Technology (UIST), 2017

BubbleView icon BubbleView

Participants click to deblur/expose small bubble regions on an otherwise blurry image.

Publications

  • Kim, N.W.*, Bylinskii, Z.*, Borkin, M., Gajos, K.Z., Oliva, A., Durand F., & Pfister, H. BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention. ACM Transactions on Computer-Human Interaction (TOCHI), 2017
  • Kim, N.W., Bylinskii, Z., Borkin, M., Oliva, A., Gajos, K., & Pfister, H. A Crowdsourced Alternative to Eye-tracking for Visualization Understanding. Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI), 2015

Authors

Special Thanks To