[FEATURE REQUEST] I2S Speaker Output

[rjsachse]

I'm not sure if this is a too far stretch for the great project, but I think having a speaker will would be great.
either play audio through upd or wav from sd card, then would be able to make this the perfect project for a door cam or surveillance cam with 2 way audio and telemetry

just a idea...

[s60sc]

could you give a more specific use case for the speaker.

[rjsachse]

Some Ideas

1. Doorbell camera. Push button and plays a selectable doorbell chime sound.
2. Doorbell camera with 2 way audio so you can talk to the person at the door
3. Intercom so one mjpeg2sd could talk to another in a different room
4. Pet camera so when your away you can see and talk to your pets
5. Application voice feedback like Wifi Connected, camera recording etc.
6. Surveillance camera with human voice deterrent pre recorded like "Recording Started"
7. Make car/truck/tank sounds like a horn or rev engine through remote control
8. Robot with vision, hearing and a voice in 1 perfect package

I know a few ideas could be classed as 1 but trying to give user cases. To be able to play sounds from sd card or stream to device, would be nice but both would be great. I know it could be difficult to implement. Think this would be a great next step into this project.
Using a i2s MAX98357A amp board. Would only need 1 extra pin if using with i2s mic too

Off Topic.
I am starting to get my head around your code and started to add a oled screen, hlk-ld2410 radar, also updated telemetry.cpp to use just BME280 or BMP280 or GY91 not conflicting, not sure if you wanted a pull request or leave telemetry.cpp as a user edit. Happy to help in anyway I can.

[s60sc]

I'll start with browser on mic enabled pc to send stream to esp32

telemetry.cpp intended as a user edit

[Ritashaiev]

what pins are you using for for i2s on esp32s3 cam wroom?

[rjsachse]

You can pretty much use any free pins for i2s

[Ritashaiev]

You can pretty much use any free pins for i2s

i tried a couple but some of them not quite ideal.
i modify them in the menu but there's what i got:
pin 14 may cause the led out of control
pin 3 or 4 may interrupt the camera
it worked immediately after pressing the reset button, but not working after a reboot
pin 46/47/21 works but it only records sound in the first 3 seconds
i will try other pins later or maybe reinstall the app

[rjsachse]

What is the pin map of your board. Sorry I don't have that board.

[Ritashaiev]

What is the pin map of your board. Sorry I don't have that board.

Freenove ESP32-S3-WROOM CAM Board (Compatible with Arduino IDE), Onboard Camera Wireless, Python C Code, Detailed Tutorial, Example Projects https://amzn.asia/d/eibWUoQ
i bought the esp32s3 from the link

i don't know if this pin map is correct, the board looks the same

[rjsachse]

Are you talking about microphone or speaker

[Ritashaiev]

Are you talking about microphone or speaker

sry, microphone. i didn't think about using i2s as a speaker. maybe i reply in a wrong issue

[rjsachse]

47,21,42,41,1,14 should be free pins according to pin map. Yes this a feature request for i2s speaker. Not issue about microphone, I have no issues with microphone that I have noticed

[Ritashaiev]

the tricky thing is when i used pin14 it kept led on. idk why

[rjsachse]

Wrong place for issue. Create a issue, if there is a issue with a selected board in appGlobals.h

[s60sc]

Implemented code in v9.7.1 to stream device microphone via browser to speaker on esp32. This works fine on my VoiceChanger app, but not on this app due to lag probably as too much else going on. The speaker code and notes is in audio.cpp

[rjsachse]

For the life of me, I could not find the setting. Need to change appSpecific.cpp config going to submenu 9, not 3. Just found the issue, but there is no time to fix.

[s60sc]

Sorry, I did that to hide the parameters so it wouldn't confuse anyone else, in your copy change 9 to 3

[rjsachse]

Glad I worked it out lol, sorry not going to be able to test till about Sunday

[rjsachse]

I had a quick go today. Sort of got it working through phone and pc, but the mic disconnects randomly and get this error message sometimes
[14:39:24.739 ERROR @ webServer.cpp:413] websocket receive failed with ESP_ERR_INVALID_STATE

[s60sc]

As before there is too much going on in the app to handle the input stream in a timely manner. May need a separate ESP32 dedicated to remote mic. I will be uploading the latest voicechanger app in a few days, could use that as a basis.

[rjsachse]

Looking forward to trying the voicechanger app. Is there anyway to use the same serial clock and LR clock for both the mic and speaker like in this app esp32-walkie-talkie

[s60sc]

Voicechanger v1.3 uploaded.

Clock pin can be shared if compatible, eg mic I2S SCK & amp I2S BCLK

[rjsachse]

Seems to work OK. I get websocket errors sometimes. Still haven't had much time to play around with it as life gets in the way. Very thankful for what you have done, please don't remove code.

[rjsachse]

Ok been having troubles getting this working again since arduino v3. Thought is was my amp board, waited for a new one, with the same result with the amp just playing a high pitch squeal. found a bug and created pull request. What was happening is that I have the i2s pins predefined in camera pins and found a issue in audio.cpp sd and ws pins was swapped. Haven't had a chance to see if it works yet.

Also seems in the latest version the amp option is removed from web server

[s60sc]

I removed the amp option as it added clutter for something that few people would use as its too much work for the S3 with this app. Better to use a separate esp as an intercom. But if you can get it to work efficiently I'll reinstate it.

[rjsachse]

sounds good, not necessarily want it as an intercom, just more want to play wav files from SD when recording starts, or motion detected like the buzzer but a voice saying recording. i do more PCB hardware than software and just want proof of concept to finalize my board first. made a test program to record a message and play it back, just looking at how to add it to this to see how it works

[rjsachse]

Ok, I just re-added the amp settings back in and it works really good on arduino esp core v3. Did not run into any issues. Seems the new i2s driver is alot cleaner and faster. Maybe worth putting it back in?

[s60sc]

Does your fork include what youve tested. If so I'll try it out when I have time in the next couple of days

[rjsachse]

I have a branch "audio" that has the amp output. Haven't uploaded the latest as im trying to play wav from sd card. Done a quick check today, seems better through IE than chrome. Seems there is a issue with the resample to 16hz too? The web audio api has a resample feature to lower the Hz's to 16000 now but only latest editions I think.

Am thinking to create a python server that handles the streams and audio of multiple
devices. Maybe all through websockets and have python on a Rpi or something to handle telegram and email etc to save resources.

Only issue is time. Loving this project but hard too find the time

Also do you know if there is a way to see what tasks are running? Not just the number of tasks running or what resources are left to use. (Sorry if off topic)

[rjsachse]

Ok done a bit of a deep dive in web audio api. Should be able to change sample rate without a script. Hopefully have a working example with gain and visualisation in the next few weeks (time pending). Web audio api can do a fair bit with processing. Any tips in what you have learned would be great

[s60sc]

I have been looking at web audio api to stream to a PC speaker, so that an esp can act as one end of an intercom and a PC mic and speaker can act as the other end of the intercom, which will be incorporated in the voicechanger app. I've not had time to do much so far. The common.js file contains the existing code for PC mic streaming to esp

[rjsachse]

ok have put the remote code back in app and works good on my end, (my branch amp) just needed to increase mic gain. i am trying proof of concept at the moment so I can finish hardware design.

haven't been able to get remote mic voice changer working yet. mic and amp work.

my java script is a bit rusty been many years since last time coding in java,
yes time is hard to find. appreciate everting you have done so far. thanks again.

[rjsachse]

had a little bit of time today and tested the amp. works great like said previously have to increase mic gain the make amp louder. change some code in java to remove the resample and now just grabs at 16000.

[rjsachse]

I have created a 2 way audio using websockets. The branch is here 2WayAudio works really good on my end haven't had much time to test though. Needing help with timing wondering is circularBuffer would help.

If you are able to give a test and any suggestions on how to improve the code would be great. Just remember to change board and upload the new data

Any feedback would be great. Would be great to see this implemented

[s60sc]

interesting, I'll make time in a couple of days

[rjsachse]

sorry i have changed some code to clean up, still have some issue's with timing. ran out of time to look into further

[s60sc]

I've finished some other work, so can start to look at your code

[rjsachse]

i just fixed the audio streaming jumpy when sending audio. the issue was it was sending data before it finished playing.

[s60sc]

The browser speaker tested ok, saves me a job as I much prefer C to javascript.
I'm now rewriting audio.cpp as I had trouble understanding my own code as it is too convoluted.

[rjsachse]

Yer I understand that., I tried to just copy the style you had already. . It is still experimental and not sure how to do the webpage design for mic etc buttons. Was playing with volume visualisation and volume control on browser side, but might confuse people with there is 2 mic volume, 1 for device and other for browser. Maybe just simple buttons is best

[s60sc]

I've uploaded v10.2 with reworked audio.cpp and javascript

[rjsachse]

haven't tested yet but noticed audio.cpp not updated still on ver10.0

[rjsachse]

found the issue there is 2 audio.cpp file one is Audio.cpp and the other is audio.cpp

[rjsachse]

seems to work great beside the amp volume clips @ gain 4 so only under 3 works for me
looking at the code it is more a linear volume control but we hear logarithmically

int8_t adjVol = ampVol * 2;
adjVol = adjVol > 5 ? adjVol - 5 : adjVol - 7;

so if ampVol = 0:
adjVol = 0 * 2 = 0
adjVol = 0 - 7 = -7 (0 less than 5 so reduce by 7 equal -7
then 1 = -5
2 = -3
3 = 1 (3*2=6 so 6 is more than 5 so reduce by 5 equal 1)
4 = 3
5 = 5
6 = 7
7 = 9
8 = 11
9 = 13
10 = 15 (seems really high)

void browserMicInput(uint8_t* wsMsg, size_t wsMsgLen) {
  // input from browser mic via websocket, send to esp amp
  if (micRem && !wsBufferLen) {
    wsBufferLen = wsMsgLen;
    memcpy(wsBuffer, wsMsg, wsMsgLen);
    int8_t adjVol = ampVol * 2; // use web page setting
    if (adjVol) {
      // increase or reduce volume, 6 is unity eg midpoint of web slider
      adjVol = adjVol > 5 ? adjVol - 5 : adjVol - 7; 
      // apply volume control to samples
      int16_t* wsPtr = (int16_t*) wsBuffer;
      for (int i = 0; i < wsBufferLen / sizeof(int16_t); i++) {   
        // apply volume control 
        wsPtr[i] = adjVol < 0 ? wsPtr[i] / abs(adjVol) : constrain((int32_t)wsPtr[i] * adjVol, SHRT_MIN, SHRT_MAX);
      }
    }
    I2Sstd.write(wsBuffer, wsBufferLen);
    wsBufferLen = 0;
  }
}

here is a logarithmic scale for both amplification and attenuation
float adjVol = pow(10, (ampVol - 5) / 10.0); // Logarithmic scaling
ampVol 0 = adjVol 0.316
1 = 0.398
2 = 0.501
3 = 0.631
4 = 0.794
5 = 1.0 //no amplification or attenuation midpoint no adjustment
6 = 1.259
7 = 1.585
8 = 2.00
9 = 2.512
10 = 3.162

void browserMicInput(uint8_t* wsMsg, size_t wsMsgLen) {
  // Input from browser mic via WebSocket, send to ESP amp
  if (micRem && !wsBufferLen) {
    wsBufferLen = wsMsgLen;
    memcpy(wsBuffer, wsMsg, wsMsgLen);
    float adjVol = pow(10, (ampVol - 5) / 10.0); // Logarithmic scaling
    if (adjVol != 1.0) { // Check if volume adjustment is needed
      int16_t* wsPtr = (int16_t*) wsBuffer;
      for (int i = 0; i < wsBufferLen / sizeof(int16_t); i++) {   
        wsPtr[i] = constrain((int32_t)wsPtr[i] * adjVol, SHRT_MIN, SHRT_MAX);
      }
    }
    I2Sstd.write(wsBuffer, wsBufferLen);
    wsBufferLen = 0;
  }
}

unless i don't understand this correctly

haven't tested yet

[s60sc]

at some point I will look at the Espressif Audio Development Framework