lighthouse performance output is significantly different from the browser inbuilt one

[fasatrix]

Hi there,
I noticed that the outputted metrics are significantly different from the browser inbuilt one, why is that(am I doing anything wrong)?

test:

import { chromium } from 'playwright';
import type { Browser } from 'playwright';
import { playAudit } from 'playwright-lighthouse';
import { test as base } from '@playwright/test';

export const lighthouseTest = base.extend<
    {},
    { port: number; browser: Browser}
    >({
    port: [
        async ({}, use) => {
            await use(9222);
        },
        { scope: 'worker' },
    ],

    browser: [
        async ({ port }, use) => {
            const browser = await chromium.launch({
                args: [`--remote-debugging-port=${port}`],
                headless: false,
            });
            await use(browser);
        },

        { scope: 'worker' },
    ],

});

lighthouseTest.describe('Lighthouse', () => {
    lighthouseTest('should pass lighthouse tests', async ({ page ,port }) => {
        await page.goto('https://angular.io');
        await playAudit({
            page,
            port,
            opts: { screenEmulation: { disabled: true } },
            thresholds: {
                performance: 40,
                accessibility: 50,
                'best-practices': 50,
                seo: 50,
                pwa: 50,
            },
        });
    });
});

From the inbuilt lighthouse

What I have noticed is that the output from your lib is very closed to the mobile output. However I added opts: { screenEmulation: { disabled: true } } to make sure mobile device is not run

[ryanrosello-og]

hey @fasatrix , I was also experiencing wild numbers when I run my tests. I then stumbled upon this article which has a good explanation as to why the numbers may fluctuate and offers up some suggestions on how to mitigate.

https://github.com/GoogleChrome/lighthouse/blob/master/docs/variability.md

[fasatrix]

hey @fasatrix , I was also experiencing wild numbers when I run my tests. I then stumbled upon this article which has a good explanation as to why the numbers may fluctuate and offers up some suggestions on how to mitigate.

https://github.com/GoogleChrome/lighthouse/blob/master/docs/variability.md

Hey, @ryanrosello-og thanks for that. My tests were run from the same computer on the same network multiple times using the web browser version and this package. So the environment was consistently the same however results were consistently different

[badsyntax]

What I have noticed is that the output from your lib is very closed to the mobile output. However I added opts: { screenEmulation: { disabled: true } } to make sure mobile device is not run

Can you try with different configs?

eg:

import lighthouseDesktopConfig from 'lighthouse/lighthouse-core/config/lr-desktop-config';
// import lighthouseMobileConfig from 'lighthouse/lighthouse-core/config/lr-mobile-config';

await playAudit({
  // ...
  config: lighthouseDesktopConfig,
  // ...
})

[das-en]

I am having the same issue as @fasatrix . Any update on this?

[YonatanKra]

Hi,

I'm having the same thing for a11y as well.

Lighthouse in dev tools reports a11y errors while the a11y score when using playwright-lighthouse is 100.

Here's my config:

	const config = {
		extends: 'lighthouse:default',
		settings: {
			maxWaitForFcp: 15 * 1000,
			maxWaitForLoad: 35 * 1000,
			formFactor: 'desktop',
			screenEmulation: {
				mobile: false,
				width: 1350,
				height: 940,
				deviceScaleFactor: 1,
				disabled: false,
			},
			emulatedUserAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
			skipAudits: [
				'html-has-lang',
				'document-title'
			],
		},
	};

	await testWrapper?.screenshot({
		path: './snapshots/select-a11y.png',
	});

	await playAudit({
		page: page,
		thresholds: {
			accessibility: 100,
		},
		port: 9222,
		reports: {
			formats: {
				html: true,
			},
			name: 'select-a11y',
		},
		config,
	});

In the dev tools I see 13 audits while in the playwright-lighthouse output I see only 3.

Any idea why?

[Kolby-Udacity]

I'm seeing the same problem. Running the test in my browser will return a performance score of ~85, running it in playwright I get ~70, running it in CI I get around ~50.

[vincerubinetti]

I have the same issue.

I have a Vite React SPA, and I'm using this package for automated performance and accessibility tests. The performance scores when I use the package are consistently less than half of when I run Lighthouse directly in Chrome:

Lighthouse Package	Dev Tools Lighthouse

Both cases are running on a production build of my app, with npm run build && npm run preview (Vite's mechanism for building an optimized production build and serving it to be previewed locally). My Playwright is configured to use this same command for all tests.

This is not variability in between runs/devices/whatever. This is analyzing the same page multiple times on the same device. Something is amiss. But I'm not sure if the problem lies in this package, Lighthouse itself, or Vite.

I've tried using the config recommendations above, with no success.

Another thing to note: running playwright test with the --debug flag (opens a headed browser and lets you step through the execution of tests one line at a time) and running the Dev Tools Lighthouse in the opened browser there produces the same degraded performance result.