BLOB data loading very slowly

[benkoppe]

Hello,

I'm using this package in a Sveltekit project, and loading BLOB data into a ReadableStream like this:

const row = result[0];
const blob = row[blobColumn] as BlobFunction;

const stream = new ReadableStream({
    start(controller) {
        blob((err, name, event) => {
            if (err) {
                reject(err);
            } else {
                event.on('data', (chunk) => {
                    console.log('new data chunk');
                    controller.enqueue(chunk);
                });
                event.on('end', () => {
                    controller.close();
                    db.detach();
                });
                event.on('error', (error) => {
                    reject(error);
                });
            }
        });
    }
});

resolve(stream);

However, the BLOB data is loading very slowly, at about 1-10 kbps, and I can't figure out why. I know I didn't write this Stream implementation in the best way, but this isn't the issue - the slow data loading happens regardless. Is this some sort of limitation with this package? Thanks!

[mateusvieites]

I dont know about this package, but using iconv-lite:

async function adjustBlob(data): Promise<string> {
    return new Promise<string>((resolve, reject) => {
        const chunks: Buffer[] = [];

        const onData = (chunk: Buffer) => {
            chunks.push(chunk);
        };

        const onEnd = () => {
            const buffer = Buffer.concat(chunks);
            let final = buffer.toString("base64");

            if (!isBase64Image(final)) {
                const bufferConverted = Buffer.from(final, "base64");
                final = iconv.decode(bufferConverted, "win1252"); //My system uses 1252 só change to uft8 here
            }

            resolve(final);
        };

        data((err: any, name: string, e: any) => {
            if (err) {
                reject(err);
                return;
            }
            e.on("data", onData);
            e.on("end", onEnd);
        });
    });
}

[benkoppe]

Thank you! Unfortunately, the data is still coming in slowly, so I think it must be a problem with the package or my connection somehow. I'll probably adopt that implementation, though.

[mreis1]

@benkoppe did you manage to solve your problem?

In my case, writing and fetching is extremely slow when dealing with files around 1mb or more (images, pdfs, etc) and connecting with a remote database.

If the server is running on the same server, we notice a huge improvement (local connections are always much faster) but still i think there's some refactoring work to be done in the part the handles the blob upload for us.

[benkoppe]

@mreis1 I'm developing remotely but in production the database will be running locally, so thanks! Hopefully that'll solve this issue.