TurkEyes logo TurkEyes

A web-based toolbox for crowdsourcing attention data

We present a toolbox of four different user interfaces for collecting crowdsourced attention data, without using eye tracking. These interfaces can be used to approximate eye movements, measure importance of graphic design elements, capture exploration on large-scale visualizations, measure attention over time, and more. The goal of this project is to make human attention data an accessible tool for researchers and creators.

Figure comparing all four interfaces.
Newman, A., McNamara, B., Fosco, C., Zhang, Y.B., Sukhum, P., Tancik, M., Kim, N.W., Bylinskii, Z. TurkEyes: A Web-Based Toolbox for Crowdsourcing Attention Data. In ACM CHI, 2020.
Paper icon
Paper
 GitHub logo
GitHub
 ArXiv icon
ArXiv
 Poster icon
Poster

Video presentation

This is the video presentation for our paper at CHI 2020. You can also watch on YouTube.

ZoomMaps icon ZoomMaps

Participants use the pinch-zoom gesture on a mobile phone to explore image content.

Publications

  • Bylinskii, Z., Newman, A., McNamara, B., Tancik, M. , Madan, S., Durand, F., Oliva, A. ZoomMaps: Using Zoom to Capture Areas of Interest on Images. Vision Sciences Society (VSS) Conference, 2019
    Poster
  • Newman, A., McNamara, B. Oliva, A., Bylinskii, Z. Exploring Attention on Large-scale Visualizations using ZoomMaps, a Zoomable Crowdsourced Interface Vision Sciences Society (VSS) Conference, VISXVISION Satellite Event, 2019
    Slides
GitHub logo
GitHub
 Demo icon
Demo

CodeCharts icon CodeCharts

Participants specify where on an image they gazed using a grid of codes that appears after image presentation.

Publications

  • Rudoy, D., Goldman, D.B., Shechtman, E., & Zelnik-Manor, L. Crowdsourcing Gaze Data Collection. Collective Intelligence conference, 2012
    ArXiv BibTeX
  • Fosco, C.*, Newman, A.*, Sukhum, P., Zhang Y.B., Zhao, N., Oliva, A., Bylinskii, Z. How many glances? Modeling multi-duration saliency. SVRHM Workshop at NeurIPS, 2019.
    Extended Abstract Poster
GitHub logo
GitHub
 Demo icon
Demo

ImportAnnots icon ImportAnnots

Participants paint over regions of a design they consider important using binary masks.

Publications

  • O'Donovan, P., Agarwala, A., & Hertzmann, A. Learning Layouts for Single-Page Graphic Designs. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2014
    Website BibTeX
  • Bylinskii, Z., Kim, N.W., O'Donovan, P., Alsheikh, S., Madan, S., Pfister, H., Durand, F., Russell, B., & Hertzmann, A. Learning Visual Importance for Graphic Designs and Data Visualizations. ACM Symposium on User Interface Software and Technology (UIST), 2017
    Website BibTeX
GitHub logo
GitHub
 Demo icon
Demo

BubbleView icon BubbleView

Participants click to deblur/expose small bubble regions on an otherwise blurry image.

Publications

  • Kim, N.W.*, Bylinskii, Z.*, Borkin, M., Gajos, K.Z., Oliva, A., Durand F., & Pfister, H. BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention. ACM Transactions on Computer-Human Interaction (TOCHI), 2017
    Website BibTeX
  • Kim, N.W., Bylinskii, Z., Borkin, M., Oliva, A., Gajos, K., & Pfister, H. A Crowdsourced Alternative to Eye-tracking for Visualization Understanding. Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI), 2015
    Paper BibTeX
GitHub logo
GitHub
 Demo icon
Demo

Authors

Special Thanks To