Parvez Ahammad

Head of Data Science and Machine Learning
Instart Logic

Twitter: @perceptPA
Blog: www.parvez-ahammad.org

Play

• Time to Load
• Time until usable
• Jitter
• Responsiveness
• Smoothness

Download

# of resources: images, fonts, HTML, scripts, and CSS loaded

Parse

File size of above resources

Execute

Parsing & Painting

Perceived Performance

Users perception of the speed of the load and reaction time.

1. HTTP request with protocol, host, port, and path.
2. DNS lookup (find host IP)
3. Socket opened between browser and server hosting the initial request content
4. HTTP request sent
5. Server handles request
6. Server sends HTTP response
7. The browser receives and parses response
8. Initial DOM tree is built
9. Additional requests are made based on resources requested in initial response, including images, stylesheets, and scripts.
   
   Go back to step 2 or 4 for each resource request.
10. Stylesheets (blocking) are parsed. CSSOM built.
11. Scripts (blocking) are parsed and executed. DOM updated.
12. Content gets rendered

• HTTP request
• DNS lookup
• TCP Connect
• HTTP request sent
• Server Magic
• Server Sends response
• Browser receives/parses response
• Resources fetched from Cache
• Parse & Execute Scripts
• Render Site

• Each request: go back to DNS lookup or HTTP Request
• Stylesheets are blocking
• Scripts are blocking.

Load Times: 3,729ms v. 3,768ms

Visually Complete: 16s v. 8.7s

Play

TTFB

Time from initial navigation until first byte is received by browser

Start Render

Time from start of initial navigation to the first non-white content painted to browser.
Green Line in WPTest

Load Time (onLoad)

From navigation start to document complete

Load Time (Fully Loaded)

Metrics collected up until there was 2s of no network activity after Document Complete.

Speed Index

Proportional representation of how quickly user-visible content was rendered.

Metric on above-fold visual Quality of Experience

• Created by Patrick Meenan (Google)
• Used on WebPage Test

Aggregate function on quickness of above-the-fold visual completion:

• 
• 4,462
• 5,902

• Frame-by-frame VC progress is computed from pixel-histogram comparisons

  

• Pixel-wise similarity (mean histogram difference a.k.a. MHD) doesn’t capture visual perception!
  • Perception of Shape / Color / Object similarity

Aggregate function on quickness of above-the-fold visual completion:

• 
• 4,462
• 5,902

• Update: Frame-by-frame VC progress computation using SSIM

  

Frame-by-frame VC progress computation using SSIM

• SI and PSI: linearly correlated
• Visual jitter / layout thrashing? PSI > SI
  • PSI appears higher when visual jitter exists (Pop-up ads / large lay-out changes / etc.)
• SSIM based visual progress measurements match human perception more closely than MHD
• SSIM / MHD swap doesn’t affect websites without visual jitter

Speed Index

• Primarily focused on progress of above-fold loading
• Does not account for layout stability

Perceptual Speed Index

• A perceptually oriented metric to measure above-fold visual QoE
• Designed to account for visual jitter (layout stability)
• Complementary to SI

• PSI and SI open sourced on GitHub: github.com/WPO-Foundation/visualmetrics
• For more details and explanations: http://www.parvez-ahammad.org/blog/perceptual-speed-index-psi-for-measuring-above-fold-visual-performance-of-webpages or http://bit.ly/2ldyo9n
• PSI in action: GoogleChrome Lighthouse
• SI in action: WebPageTest.org

• Challenge: speedperception.meteorapp.com/
• Findings & Repo: github.com/pahammad/SpeedPerception
• Team:
  • Qingzhu (Clark) Gao: Data Scientist @ Instart Logic
  • Parvez Ahammad: Head of Data Science & Machine Learning @ Instart Logic
  • Prasenjit Dey: Software Engineer @ Instart Logic

Phase-1 crowd sourced 07/28/2016 - 09/30/2016.

Hypothesis 1: Visual metrics will perform better than non-visual/network metrics

Hypothesis 2: No single metric can explain human choices with 90%+ accuracy

Hypothesis 3: User will not wait until “Visual Complete” to make their choice (despite the explicit instruction to wait until video turns grey)

• 5,444 sessions, of which 51% were complete and valid
• 77,482 votes, of which 75% were valid

Perception of speed and UX strongly impacted by popups / overlays

Hypothesis 1: Visual metrics will perform better than non-visual/network metrics

• Does presence of visual jitter / interstitials interfere with metric performance?
  • Can metrics be improved?
• Will there be different trends for sites that are free of visual jitter like modals and overlays?
• Is it possible to automatically predict the presence of jitter to help choose a better set of metrics?

Hypothesis 1: Visual metrics will perform better than non-visual/network metrics

Hypothesis 2: No single metric can explain human choices with 90%+ accuracy

Hypothesis 2: No single metric can explain human choices with 90%+ accuracy

• There appears to be no one unicorn metric but, is there a combination synthetic metric (joint ML model) that will do a better job?
• People only looked two videos and made the call. Is there some additional information that we can extract from videos that will improve our models?

Hypothesis 3: User will not wait until “Visual Complete” to make their choice (despite the explicit instruction to wait until video turns grey)

• How do visual jitter & interstitials impact perceived performance?
  • Do they interfere with metric performance?
  • Can metrics be improved?
  • Are sites free of visual jitter like modals and overlays viewed as more performant?
  • Is it possible to automatically predict the presence of jitter to help choose a better set of metrics?
• Does a long DOM Content Loaded impact perceived performance?

User Experience > Developer Experience