Full Web Access with Proxy to bypass Iframe Blockers on Major Sites

[vtempest]

Brett,
THIS. IS. THE. BEST. EFFING. WEBSITE. DEMO. ON. THE. ENTIRE. INTERNET.

Do not stop improving this, it has huge potential as AI workspace of the future that you login to from mobile, laptop anywhere. I want to combine this with Claude AI computer use automation

Suggestion: add vscode in a browser from coder.com and also add the option for a linux server backend that user has to specify the ssh login, then it will run the needed setup for persistent data and full linux system emulation to install any app then pipe its xvfb virtual frames into canvas with NoVNC.

Full Web Access: major sites like nyt.com block iframing but I have developed solution just for this. Code: https://github.com/vtempest/ai-research-agent/tree/master/src/crawler Demo Setup this proxy and pipe it thru this virtual browser which strips anti-bot, also rewrite the href so all links are proxied.

Lets keep working on this together!

you can disable cors via a chrome extension using a simple script calling the Networking API.

/**
 * <b>Allow CORS - Chrome Extension API.</b> Add (Access-Control-Allow-Origin: *) rule to the response 
 * header of all requests to enable cross-domain requests to scrape data from other domains. 
 * CORS is voluntarily enforced by browser frontends, but not by curl or backend
 * requests. Allowing CORS on the front-end means no proxy server is needed and web apps 
 * have Browser App functionality and can turn any data into an API.
 * [There are several extension enabling CORS.](https://chromewebstore.google.com/detail/allow-cors-access-control/lhobafahddgcelffkeicbaginigeejlf?)
 * It is suggested to allow only on specific initiator domains to prevent errors 
 * commonly caused on secure sites like Claude.ai. Needs [Chrome API 
 * "permissions"](https://developer.chrome.com/docs/extensions/reference/api/declarativeNetRequest):[ "declarativeNetRequest"]
 * 
 *  * @param {string[]} initiatorDomains - The domains allowed to initiate CORS request
 * @param {string} receivingDomains - The domains allowed to receive the request
 * @returns {void}
 * @category Chrome Extension
 * @example chrome.runtime.onInstalled.addListener(() => {
    modifyChromeAPIAllowCORS(["qwksearch.com"]);
  });
 */
export function modifyChromeAPIAllowCORS(
  initiatorDomains = "*://*/*",
  receivingDomains = "*"
) {
  return chrome?.declarativeNetRequest?.updateDynamicRules({
    removeRuleIds: [1],
    addRules: [
      {
        id: 1,
        priority: 1,
        action: {
          type: "modifyHeaders",
          responseHeaders: [
            {
              header: "Access-Control-Allow-Origin",
              operation: "set",
              value: "*",
            },
            {
              header: "Access-Control-Allow-Methods",
              operation: "set",
              value: "GET, POST, OPTIONS, PUT, DELETE, PATCH",
            },
          ],
        },
        condition: {
          urlFilter: receivingDomains,
          initiatorDomains,
          resourceTypes: [
            "main_frame",
            "sub_frame",
            "stylesheet",
            "script",
            "image",
            "font",
            "object",
            "xmlhttprequest",
            "ping",
            "csp_report",
            "media",
            "websocket",
            "other",
          ],
        },
      },
    ],
  });
}

[DustinBrett]

Brett, THIS. IS. THE. BEST. EFFING. WEBSITE. DEMO. ON. THE. ENTIRE. INTERNET.

Do not stop improving this, it has huge potential as AI workspace of the future that you login to from mobile, laptop anywhere. I want to combine this with Claude AI computer use automation

Suggestion: add vscode in a browser from coder.com and also add the option for a linux server backend that user has to specify the ssh login, then it will run the needed setup for persistent data and full linux system emulation to install any app then pipe its xvfb virtual frames into canvas with NoVNC.

Full Web Access: major sites like nyt.com block iframing but I have developed solution just for this. Code: https://github.com/vtempest/ai-research-agent/tree/master/src/crawler Demo Setup this proxy and pipe it thru this virtual browser which strips anti-bot, also rewrite the href so all links are proxied.

Lets keep working on this together!

Thank you very much! Funny you mention AI as I have been adding a lot of AI features recently.

I do have parts of VS Code (Monaco Editor & XTerm.js) already running on my site, so the main missing thing is putting it together into an IDE. There are also a few other ways to run VS Code in the browser, but I like the DIY version personally. VNC is something I am considering, there are also Wayland compositors such as https://github.com/udevbe/greenfield.

Indeed CORS makes the iframe/Browser app harder, but I do not currently desire to hardcode any kind of proxy in. I am thinking of eventually making something to allow proxying or special access for that iframe.

Thanks for the suggestions and support!

[bitnom]

Why do all that madness when you can just go websocket>socket (proxy). The easiest way I know would be https://github.com/qwj/python-proxy on a server (Websockets supported) and a service worker in daedalOS. That way you don't have to worry about making some weird networking abstraction when you code an app, and no more cors troubles. For the public demo you could just whitelist hosts server-side.

I'm guessing there's something like python-proxy for node. I just don't know what it is. If there isn't though, just be like "BYOProxy" and suggest how to do it with python-proxy, which would be a single command via its CLI.

[bitnom]

That's cool. That reminds me of how webvm handled it using tailscale. I suppose that offloaded some of the responsibility. Cloudflare has a similar thing.

related #476

I'm kicking around the idea of using daedalOS for one of my projects. If I do, I'll be down this rabbit-hole of networking, I guess ahead of your putting much time to it. It's fine by me. I prefer to focus on backend as much as I can get away with it.

[vtempest]

Why go through all that madness of requiring users to setup a paid VPN tailscale when you just need one simple server[side proxy with limits or if you want it clientside, you can disable cors via a chrome extension using a simple script calling the Networking API. Another main issue is that a proxy is not enough for most sites, you need a full dom renderer like puppeteer for modern apps that load data dynamically

/**
 * <b>Allow CORS - Chrome Extension API.</b> Add (Access-Control-Allow-Origin: *) rule to the response 
 * header of all requests to enable cross-domain requests to scrape data from other domains. 
 * CORS is voluntarily enforced by browser frontends, but not by curl or backend
 * requests. Allowing CORS on the front-end means no proxy server is needed and web apps 
 * have Browser App functionality and can turn any data into an API.
 * [There are several extension enabling CORS.](https://chromewebstore.google.com/detail/allow-cors-access-control/lhobafahddgcelffkeicbaginigeejlf?)
 * It is suggested to allow only on specific initiator domains to prevent errors 
 * commonly caused on secure sites like Claude.ai. Needs [Chrome API 
 * "permissions"](https://developer.chrome.com/docs/extensions/reference/api/declarativeNetRequest):[ "declarativeNetRequest"]
 * 
 *  * @param {string[]} initiatorDomains - The domains allowed to initiate CORS request
 * @param {string} receivingDomains - The domains allowed to receive the request
 * @returns {void}
 * @category Chrome Extension
 * @example chrome.runtime.onInstalled.addListener(() => {
    modifyChromeAPIAllowCORS(["qwksearch.com"]);
  });
 */
export function modifyChromeAPIAllowCORS(
  initiatorDomains = "*://*/*",
  receivingDomains = "*"
) {
  return chrome?.declarativeNetRequest?.updateDynamicRules({
    removeRuleIds: [1],
    addRules: [
      {
        id: 1,
        priority: 1,
        action: {
          type: "modifyHeaders",
          responseHeaders: [
            {
              header: "Access-Control-Allow-Origin",
              operation: "set",
              value: "*",
            },
            {
              header: "Access-Control-Allow-Methods",
              operation: "set",
              value: "GET, POST, OPTIONS, PUT, DELETE, PATCH",
            },
          ],
        },
        condition: {
          urlFilter: receivingDomains,
          initiatorDomains,
          resourceTypes: [
            "main_frame",
            "sub_frame",
            "stylesheet",
            "script",
            "image",
            "font",
            "object",
            "xmlhttprequest",
            "ping",
            "csp_report",
            "media",
            "websocket",
            "other",
          ],
        },
      },
    ],
  });
}

[DustinBrett]

Indeed madness all around. I don't desire to get into a backend proxy with "limits" or making a Chrome extension. The browser is a proof of concept with some neat ideas but I am ok that it doesn't allow anyone to go anywhere through a system I provide.

[vtempest]

Indeed madness all around. I don't desire to get into a backend proxy with "limits" or making a Chrome extension. The browser is a proof of concept with some neat ideas but I am ok that it doesn't allow anyone to go anywhere through a system I provide.

That would not be a concern if you did the simple chrome CORS-unblocker which takes seconds to install and enables them to surf via their own IP and download torrents via own IP.
You might look into webvm and find a way to do puppeter style dom render in frontend wasm

https://github.com/leaningtech/webvm

[DustinBrett]

This Browser app is a small part of my site and I don't desire to ask or make users install an extension in order for an app to work a certain way. I don't want to promote people coming to my site so they can go to other sites through my site,.