• Challenge: speedperception.meteorapp.com/
• Findings & Repo: github.com/pahammad/SpeedPerception
• Team:
  • Qingzhu (Clark) Gao: Data Scientist @ Instart Logic
  • Parvez Ahammad: Head of Data Science & Machine Learning @ Instart Logic
  • Prasenjit Dey: Software Engineer @ Instart Logic

Phase-1 crowd sourced 07/28/2016 - 09/30/2016.

Hypothesis 1: Visual metrics will perform better than non-visual/network metrics

Hypothesis 2: No single metric can explain human choices with 90%+ accuracy

Hypothesis 3: User will not wait until “Visual Complete” to make their choice (despite the explicit instruction to wait until video turns grey)

• 5,444 sessions, of which 51% were complete and valid
• 77,482 votes, of which 75% were valid

Perception of speed and UX strongly impacted by popups / overlays

Hypothesis 1: Visual metrics will perform better than non-visual/network metrics

• Does presence of visual jitter / interstitials interfere with metric performance?
  • Can metrics be improved?
• Will there be different trends for sites that are free of visual jitter like modals and overlays?
• Is it possible to automatically predict the presence of jitter to help choose a better set of metrics?

Hypothesis 1: Visual metrics will perform better than non-visual/network metrics

Hypothesis 2: No single metric can explain human choices with 90%+ accuracy

Hypothesis 2: No single metric can explain human choices with 90%+ accuracy

• There appears to be no one unicorn metric but, is there a combination synthetic metric (joint ML model) that will do a better job?
• People only looked two videos and made the call. Is there some additional information that we can extract from videos that will improve our models?

Hypothesis 3: User will not wait until “Visual Complete” to make their choice (despite the explicit instruction to wait until video turns grey)

• How do visual jitter & interstitials impact perceived performance?
  • Do they interfere with metric performance?
  • Can metrics be improved?
  • Are sites free of visual jitter like modals and overlays viewed as more performant?
  • Is it possible to automatically predict the presence of jitter to help choose a better set of metrics?
• Does a long DOM Content Loaded impact perceived performance?

User Experience > Developer Experience