Suggestions and improvements after attempting to integrate it into my project:

[aguerrieri82]

Hi, congratulations on the huge work. I've started to make my own XR game engine. Long story short, I've tried everything out there, but nothing fits the needs with a decent iteration time. Since I'm not going to render on windows/swapchains and I need deeper control of the underlying backend, I've found some issues (not everything is related to my project) that I'm trying to synthesize here. If this is not the right place, I will open an issue for every point made here.

Android Build

• 🚨 Android build fails on Windows since it tries to invoke a bash script to merge some .a files into a single archive. Please provide the Windows version of the script or update your documentation. I've spent hours on this, then fell back to WSL.

• Please provide the library files ready to download, just like for the Windows platform. Hardly anybody will use the Java/JNI interface, since it likely needs to be multi-platform, and people prefer using native libraries.

OpenGL (Windows)

• In order to share the context, we need both the device context (HDC) and GL Context (HGLRC). If you call GetDC on the same Windows handle, that will be a new one and incompatible. So .sharedContext must point to a struct like this:

    struct Context{
        HGLRC glContext;
        HDC hDC;
    };

Also, we must publicly expose both values from the PlatformWGL class if we want to delegate Filament to create the context and share it afterward.

• The render thread must release the context in order to pass the control to another thread (wglMakeCurrent(whdc, NULL)), but I cannot access to it. I've made two virtual methods on Platform.

    virtual bool onPreExecute() noexcept  { return true; }
    virtual void onPostExecute() noexcept  { }

that are called inside Engine::execute in order to handle this case. When onPreExecute returns false, the command queue is not processed. Processing the queue while the context is owned by another thread will cause hundreds of assertion failures.

Vulkan

• I need to pass my custom instance/device extensions to the driver. Please make such a list public (currently it's static and private: static ExtensionSet VulkanPlatform::getRequiredInstanceExtensions()).
• 🚨The method VulkanDriver::importTextureR is not implemented, and no warning/exception is given. I spent hours trying to understand what Texture::Builder::import does. The implementation was quite simple; please integrate it into the main branch.

 void VulkanDriver::importTextureR(Handle<HwTexture> th, intptr_t id,
        SamplerType target, uint8_t levels,
        TextureFormat format, uint8_t samples, uint32_t w, uint32_t h, uint32_t depth,
        TextureUsage usage) {

    auto vktexture = mResourceAllocator.construct<VulkanTexture>(th, mPlatform->getDevice(),
            mAllocator, mCommands.get(), (VkImage)id, backend::getVkFormat(format), samples, w, h, usage, mStagePool);

    mResourceManager.acquire(vktexture);
}

Memory managment on vertex/index buffers and textures

• Any call to Texture::setImage or IndexBuffer::setBuffer, etc., are not immediately committed, but I need to free the memory as soon as the GPU has loaded the data. How can I do that? I can call Engine::flushAndWait, but I'm not 100% sure that Filament won't need that buffer anymore. Can you please add a flag to such buffers to opt-in for "release the memory when you are done"?

Off-screen rendering

• The method beginFrame always requires a swapchain. If I'm rendering off-screen on a render target, I don't need one. I've been successful in avoiding those calls by simply invoking renderStandaloneView, but I don't know if it's a 'hack' or if it's okay.
• endFrame will cause a call to SwapBuffers on PlatformWGL, waiting for the v-sync. Please, make a public method for each platform to disable the v-sync (a VR device has its synchronization mechanism, and I need to render on the provided texture as soon as the device requests it).

Surface orientation

• I need to dynamically load some models into my scene, and 99% of them only have normals + UVs. Lighting requires tangents, and OrientationBuilderImpl::buildWithUvs was not implemented with a stride. Here is my implementation:

SurfaceOrientation* OrientationBuilderImpl::buildWithUvs() {

    const auto nstride = this->normalStride == 0 ? sizeof(float3) : this->normalStride;
    const auto ustride = this->uvStride == 0 ? sizeof(float2) : this->uvStride;
    const auto pstride = this->positionStride == 0 ? sizeof(float3) : this->positionStride;

    #define STRIDE_GET(v, i, s, t)  (*(t*)(((char*)v) + (i * s)))

    vector<float3> tan1(vertexCount);
    vector<float3> tan2(vertexCount);
    memset(tan1.data(), 0, sizeof(float3) * vertexCount);
    memset(tan2.data(), 0, sizeof(float3) * vertexCount);
    for (size_t a = 0; a < triangleCount; ++a) {
        uint3 tri = triangles16 ? uint3(triangles16[a]) : triangles32[a];
        assert_invariant(tri.x < vertexCount && tri.y < vertexCount && tri.z < vertexCount);
        const float3& v1 = STRIDE_GET(positions, tri.x, pstride, float3);
        const float3& v2 = STRIDE_GET(positions, tri.y, pstride, float3);
        const float3& v3 = STRIDE_GET(positions, tri.z, pstride, float3);
        const float2& w1 = STRIDE_GET(uvs, tri.x, ustride, float2);
        const float2& w2 = STRIDE_GET(uvs, tri.y, ustride, float2);
        const float2& w3 = STRIDE_GET(uvs, tri.z, ustride, float2);
        float x1 = v2.x - v1.x;
        float x2 = v3.x - v1.x;
        float y1 = v2.y - v1.y;
        float y2 = v3.y - v1.y;
        float z1 = v2.z - v1.z;
        float z2 = v3.z - v1.z;
        float s1 = w2.x - w1.x;
        float s2 = w3.x - w1.x;
        float t1 = w2.y - w1.y;
        float t2 = w3.y - w1.y;
        float d = s1 * t2 - s2 * t1;
        float3 sdir, tdir;
        // In general we can't guarantee smooth tangents when the UV's are non-smooth, but let's at
        // least avoid divide-by-zero and fall back to normals-only method.
        if (d == 0.0) {
            const float3& n1 = STRIDE_GET(normals, tri.x, nstride, float3);
            sdir = randomPerp(n1);
            tdir = cross(n1, sdir);
        } else {
            sdir = {t2 * x1 - t1 * x2, t2 * y1 - t1 * y2, t2 * z1 - t1 * z2};
            tdir = {s1 * x2 - s2 * x1, s1 * y2 - s2 * y1, s1 * z2 - s2 * z1};
            float r = 1.0f / d;
            sdir *= r;
            tdir *= r;
        }
        tan1[tri.x] += sdir;
        tan1[tri.y] += sdir;
        tan1[tri.z] += sdir;
        tan2[tri.x] += tdir;
        tan2[tri.y] += tdir;
        tan2[tri.z] += tdir;
    }

    vector<quatf> quats(vertexCount);
    for (size_t a = 0; a < vertexCount; a++) {
        const float3& n = STRIDE_GET(normals, a, nstride, float3);
        const float3& t1 = tan1[a];
        const float3& t2 = tan2[a];

        // Gram-Schmidt orthogonalize
        float3 t = normalize(t1 - n * dot(n, t1));

        // Calculate handedness
        float w = (dot(cross(n, t1), t2) < 0.0f) ? -1.0f : 1.0f;

        float3 b = w < 0 ? cross(t, n) : cross(n, t);
        quats[a] = mat3f::packTangentFrame({t, b, n});
    }
    return new SurfaceOrientation(new OrientationImpl( { std::move(quats) } ));
}

Poor performance

On the Meta Quest 3 device (Snapdragon XR2 Gen 2 - Adreno (TM) 740), I've experienced very poor performance with both OpenGL and Vulkan in stereo rendering (two views bound to two different frame buffers). By poor, I mean achieving only 40 fps while rendering a single cube with a color-only shader (no post-processing, no shadows, no MSAA, no IBL, just one sunlight). Maybe I'm doing something wrong or the V-Sync is kicking in. In my render loop, I'm only updating the world matrix and the camera view/proj; the scene remains unchanged once created.

With my self-made OpenGL ES rendering engine / shader system, for a not too complex scene, I was easily able to maintain 72fps, which is the minimum required in VR to avoid motion sickness.

Here my code

Minor

• 🚨In ShadowMap::SceneInfo::SceneInfo, the following assert is 99% false due to floating-point approximation. Add an epsilon or remove the assert (in debug, kicks in every frame).

assert_invariant(transpose(viewMatrix)[3] == float4(0, 0, 0, 1));

That's it for now, thanks :)