php-wasm/web : Explore composer usage

[mho22]

I discovered that the php.cli method is accessible with @php-wasm/universal through PHP, and I wanted to test it in the browser using loadWebRuntime.

However, it appears that the C function _wasm_add_cli_args cannot be found. This function comes from the WITH_CLI_SAPI option, which is only available in php-wasm/node.

I’m trying to understand what might be missing:

• The WITH_CLI_SAPI option in web-kitchen-sink
• Removing the access to php.climethod in the web environment

 
 

FYI : Here is the error I encountered when running php.cli:

php_8_2-UUV366BB.js?v=cb94a143:5407 Uncaught (in promise) TypeError: func is not a function
    at Object.ccall (php_8_2-UUV366BB.js?v=cb94a143:5407:15)
    at PHP.cli (chunk-O65H5HB5.js?v=cb94a143:1718:34)
    at cli.js:77:9

 
 

I’m not sure if this is a real issue. I look forward to any insights you can provide.

[mho22]

I successfully tested compiling php-wasm/web/kitchen-sink with WITH_CURL and WITH_CLI_SAPI, and it indeed makes php.cli() work properly. I suppose this is probably related to [#1039] and this is only a matter of time.

[mho22]

I take this "issue" to go further : In my quest to make the composer phar file work. I made the use of the php proxy server on 127.0.0.1:5263 port :

const code = `
    // Set up proxy server
    $_SERVER[ 'cgi_http_proxy' ] = 'http://127.0.0.1:5263';
    $_SERVER[ 'https_proxy' ] = 'http://127.0.0.1:5263';

    // Set Composer arguments
    $_SERVER[ 'argv' ] = [ "composer", "--working-dir=/test", "diagnose" ];
    $_SERVER[ 'argc' ] = 3;

    require( '/capsules/vendor/bin/composer.phar' );`;

const result = await php.cli( [ 'php', '-r', code ] );

Unfortunately : the proxy handles only GET, POST, HEAD and OPTIONS methods, no CONNECT method.

This is the diagnose stack trace :

Checking composer.json: OK
Checking platform settings: OK
Checking git settings: No git process found
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: FAIL
[Composer\Downloader\TransportException] curl error 56 while downloading https://repo.packagist.org/packages.json: Received HTTP code 405 from proxy after CONNECT
Checking HTTP proxy with http: OK http://127.0.0.1:5263
Checking HTTP proxy with https: FAIL
[Composer\Downloader\TransportException] curl error 56 while downloading https://repo.packagist.org/packages.json: Received HTTP code 405 from proxy after CONNECT
Checking github.com rate limit: FAIL
[Composer\Downloader\TransportException] curl error 56 while downloading https://api.github.com/rate_limit: Received HTTP code 502 from proxy after CONNECT
Checking disk free space: OK
Checking pubkeys: FAIL
Missing pubkey for tags verification
Missing pubkey for dev verification
Run composer self-update --update-keys to set them up
Checking Composer version: FAIL
[Composer\Downloader\TransportException] curl error 56 while downloading https://getcomposer.org/versions: Received HTTP code 405 from proxy after CONNECT
Checking Composer and its dependencies for vulnerabilities: WARNING
Failed performing audit: curl error 56 while downloading https://repo.packagist.org/packages.json: Received HTTP code 405 from proxy after CONNECT
Composer version: 2.7.7
PHP version: 8.2.10-dev
PHP binary path:
OpenSSL version: OpenSSL 1.1.0h  27 Mar 2018
curl version: 7.69.1 libz 1.2.5 ssl OpenSSL/1.1.0h
zip: extension present, unzip not available, 7-Zip not available

@adamziel I know this is not a priority but if you have any insights about how to handle CONNECT properly, this can become really interesting.

[mho22]

I decided to go further on making composer phar work. I modified the current proxy server to run without the use of php -S because of additional Headers added everytime a request was sent.

I additionnally upgraded openssl to version 1.1.1u to enable LTS1.3.

Here is a Gist with the proxy.php file I run with php cors-proxy/proxy.php :

The result is as follows when running composer diagnose -vvv :

const code = `

    // Set up proxy server
    $_SERVER[ 'cgi_http_proxy' ] = 'http://127.0.0.1:5263';
    $_SERVER[ 'https_proxy' ] = 'http://127.0.0.1:5263';
    $_SERVER[ 'SSL_CERT_FILE' ] = '/dist/ca-bundle.crt';

    // Set Composer arguments
    $_SERVER[ 'argv' ] = [ "composer", "--working-dir=/capsules", "diagnose", "-vvv" ];
    $_SERVER[ 'argc' ] = 3;

    require( '/capsules/vendor/bin/composer.phar' );`;

const result = await php.cli( [ 'php', '-r', code ] );

I didn't find today why there was a timeout. I know that this is due to a connection not made within the 10 seconds from line 195 of composer/src/Composer/Util/Http/CurlDownloader.php :

curl_setopt($curlHandle, CURLOPT_CONNECTTIMEOUT, 10);

And my proxy blocking during these 10 seconds on line 63:

$input = socket_read( proxy::$client, 4096 );

I decided to run the HTTPS request separately with php-wasm/node cURL :

 const composer = `

    $ch = curl_init( "https://getcomposer.org/versions" );

    curl_setopt( $ch, CURLOPT_PROXY, "http://127.0.0.1:5263" );
    curl_setopt( $ch, CURLOPT_CAINFO, "/dist/ca-bundle.crt" );
    curl_setopt( $ch, CURLOPT_VERBOSE, true );

    curl_exec( $ch );

    curl_close( $ch );
`;

const result = await php.cli( [ 'php', '-r', composer ] );

In the same spirit.

Result :

And it works...Composer's behavior seems strange. I feel like I'm missing something important.

@adamziel @brandonpayton @bgrgicak — Is this something relevant? Should I continue investigating this?

[mho22]

Nevermind, I finally got it working with Composer 1 using php-wasm/node

 
 

Still not working on browser. Proxy is not reachable.

[mho22]

I finally managed to make things work on browser. Unfortunately a dynamic CAPem is needed and a TODO has been added by @adamziel months ago in #1093. The next step would be to dynamically export a CAPem made with the correct commonName. First one being repo.packagist.org, then api.github.com then getcomposer.org.

 
 

For the screenshot, I manually modified these lines :

// @TODO: Create these certificates in FetchWebsocketConstructor based on the requested host
const { certificate, privateKey } = createCertificate({
-	commonName: 'downloads.wordpress.org', //wsUrl.searchParams.get('host') || ''
+	commonName: 'repo.packagist.org',
});

const CAPair = { certificate, privateKey };
// const CAPair = createCertificate({
// 	commonName: ''
// });

export const CAPem = forge.pki.certificateToPem(CAPair.certificate);

But if I want this to work with dynamic domains I will have to overwrite the certificate based on the

this.host = wsUrl.searchParams.get('host')!;

 
 

But for now, only the original certificate is sent to composer diagnose :

    php.writeFile( '/home/web_user/.composer/ca-bundle.crt', CAPem );

    const code = `

        $_SERVER[ 'SSL_CERT_FILE' ] = '/home/web_user/.composer/ca-bundle.crt';
        $_SERVER[ 'argv' ] = [ "composer", "--working-dir=/test", "diagnose" ];

        require '/test/vendor/bin/composer.phar';`;

    await php.cli( [ 'php', '-r', code ] );

 
 

I still tried composer install but a general error occurs :

  [Composer\Downloader\TransportException]
  The "https://repo.packagist.org/packages.json" file could not be downloaded: Failed to open stream: Operation timed out

This looks like composer install requests are not triggered by the websocket send method.

Something similar occurs when using Composer v2. But this is probably linked with the fact that v2 uses curl requests under the hood and these are not intercepted by the websocket.

[adamziel]

Oh, this is such a great work @mho22! Generatic a dynamic ca.pem sounds like the best thing we can do, thank you for exploring that! The CORS part will get easier once we host a generic CORS proxy (that will happen) and open it up to all URLs (that may or may not happen).

Actually, shipping a nice API for generating those certificates would unblock #1093 as that's the part we're missing. Forge JS can be quite slow and block the entire UI for a good few seconds. I was thinking about the following two ways of speeding it up:

• Delegate the work to a new Worker() to avoid blocking the UI. Requesting each new domain would still introduce a few seconds of a delay until the certificate is ready so that's not ideal.
• See how far can we get with the browser's native crypto API. It's asynchronous and seems way faster. I'm just not sure whether it can handle all the steps in the ca.pem generation process or only a few of them. If it's only a few, it could make sense to lean on that and polyfill the missing bits from Forge JS.

[mho22]

I would be absolutely delighted to help complete task #1093, but I am not yet well informed about how the different steps work. So, I will need to dig deeper. I am trying to make the commands composer diagnose and composer install work with all its complexity. And for now I am stuck with :

• Generating dynamic ca.pem. Of course.
• Not every requests are intercepted by the websocket send function. Especially composer install.

Is "trying to make composer diagnose or composer install work" an interesting use case ?

[adamziel]

Is "trying to make composer diagnose or composer install work" an interesting use case ?

I think so! I've heard contributors asking about using composer in the browser. This could be interesting for the larger PHP community as well and would get us a composer playground for free. Once we integrate XTerm.js with the Playground UI, we could expose composer as a command. This would be a major step towards having an in-browser WordPress / PHP IDE.

[adamziel]

Not every requests are intercepted by the websocket send function. Especially composer install.

Interesting! Is that with #1093 applied? It should capture all network calls, it's weird if it doesn't. I remember installing composer packages in the past using @php-wasm/cli and it uses the same WebSockets tunneling mechanism.

[adamziel]

I got this from Claude, it's probably incorrect but it's fast :D It may or may not be a good starting point for CA cert generation:

/**
 * Generate a CA.pem certificate pair dynamically in the browser with no dependencies
 * using just the Browser-native crypto API.
 */

async function generateCaPem() {
	const certInfo = {
		serialNumber: '1',
		validity: {
			notBefore: new Date(),
			notAfter: new Date(Date.now() + 1000 * 60 * 60 * 24 * 365),
		},
		subject: {
			commonName: 'Root CA',
		},
		issuer: {
			commonName: 'Root CA',
		},
		extensions: {
			basicConstraints: {
				critical: true,
				cA: true,
			},
			keyUsage: {
				digitalSignature: true,
				keyCertSign: true,
			},
		},
	};

	const crypto = window.crypto;
	const encoder = new TextEncoder();
	const decoder = new TextDecoder();

	const caKey = await crypto.subtle.generateKey(
		{
			name: 'RSASSA-PKCS1-v1_5',
			modulusLength: 2048,
			publicExponent: new Uint8Array([1, 0, 1]),
			hash: 'SHA-256',
		},
		true,
		['sign', 'verify']
	);

	// Create a simple ASN.1 structure for the certificate
	const tbs = encoder.encode(JSON.stringify({
		version: 3,
		serialNumber: certInfo.serialNumber,
		issuer: certInfo.issuer,
		subject: certInfo.subject,
		validity: {
			notBefore: certInfo.validity.notBefore.toISOString(),
			notAfter: certInfo.validity.notAfter.toISOString(),
		},
		extensions: certInfo.extensions,
	}));

	const signature = await crypto.subtle.sign(
		{
			name: 'RSASSA-PKCS1-v1_5',
		},
		caKey.privateKey,
		tbs
	);

	// Combine TBS and signature into a simple certificate structure
	const cert = encoder.encode(JSON.stringify({
		tbsCertificate: decoder.decode(tbs),
		signatureAlgorithm: 'sha256WithRSAEncryption',
		signatureValue: btoa(String.fromCharCode(...new Uint8Array(signature))),
	}));

	const caPem = `-----BEGIN CERTIFICATE-----\n${btoa(decoder.decode(cert))}\n-----END CERTIFICATE-----`;
	const caKeyPem = await exportKeyToPem(caKey.privateKey);

	return { caPem, caKeyPem };
}

async function exportKeyToPem(key) {
	const exported = await crypto.subtle.exportKey('pkcs8', key);
	const exportedAsBase64 = btoa(String.fromCharCode(...new Uint8Array(exported)));
	return `-----BEGIN PRIVATE KEY-----\n${exportedAsBase64}\n-----END PRIVATE KEY-----`;
}

generateCaPem().then(({ caPem, caKeyPem }) => {
	console.log({ caPem, caKeyPem });
});

[mho22]

I’ve applied all the code changes from #1093. I also successfully installed composer packages using @php-wasm/node back then, but for some unknown reason, I'm now experiencing timeouts instead of seeing send function calls. Interestingly, this issue doesn't occur when running composer diagnose, which works fine despite the lack of dynamic certificate handling.

I recall setting disable_functions=curl_exec,curl_multi_exec in php.cli. Could this be related?

Anyway, thanks for your insights and help. Back to work for me!

[mho22]

It was indeed related to disable_functions, specifically with proc_open and popen. Now, let's look into the certificates!

[sinanisler]

curl :(

[adamziel]

Solving #1093 would allow enabling curl in the browser!

[mho22]

I couldn't resist and tried @adamziel new #1926 from this morning, and I managed to retrieve the first data from composer ! Unfortunately it doesn't work exactly as expected...

With Composer 2 :

But as you can see : Downloading https://repo.packagist.org/packages.json resulted in the actual download of the packages.json file 🎉 , and I think this should probably be the first validation of the list above.

There is still something not working when trying to decode data from curl, a headers problem based on the curl error.

Edit > Next step : transform these FAIL into OK

[mho22]

Some weird things happen when running php composer diagnose -vvv. The result without dev logs is shown above. But here is some screenshots with them.

HTTP :

According to Composer, the HTTP request failed...But I got the correct content.

HTTPS :

Same here, HTTPS request failed... But I got the correct content...

Maybe the problem comes from my proxy...

P.S.: Sorry for the screenshots flood.

[mho22]

Here it is ! :

Still a Warning but, 🎉 !

The previous reason why it didn't work was due to the proxy being on HTTPS.

And after 12 minutes of processing and 1128 lines in the Chrome browser console :

I sucessfully ran composer on a personal project !

But 12 minutes is a bit too long isn't it 😄 ?

[mho22]

16 seconds for a unique composer package CowsayPHP for a basic test :

• Download a git repository
• Download Composer Phar file
• Install Composer dependencies with php-wasm based on composer.json file in repository
• Use a dependency and show result in console and DOM

[adamziel]

Nice @mho22! I wonder if that previous test would be faster now that we have curl 🤔

[mho22]

@adamziel Unfortunetaly, these processes are actually made with curl enabled. I echoed curl_version to illustrate it.

So the previous 12 minutes test was with curl enabled.

I maybe missed something, because I admit the two chronos are pretty slow. I need to try it again on the trunk branch.

[mho22]

@adamziel My bad, I retried with the trunk branch and for the two tests :

First one : it reduced from 12 minutes to 7 minutes
Second one : it reduced from 16 seconds to 1 second

Looks promising !

I will investigate to find if this time can be even more reduced.