No, I'm not talking WiFi or Bluetooth. I just want to get rid of this clew of wires:

so I designed a couple of PCB's that should cover most of my use cases, and ordered them at JLCPCB:

Nice, huh? That's 3D rendering, they didn't arrive yet. We'll see how good they actually are in a couple of weeks.

So it all has been working nicely for the past couple of weeks, with one exception. Every few days I see unexpected reboots, and about once a month - permanent freeze in at least one of the boards. Luckily, ESP32 has reset reason register, and I have made it available in the UI:

The manual explains:

The interrupt watchdog makes sure the FreeRTOS task switching interrupt isn’t blocked for a long time. This is bad because no other tasks, including potentially important ones like the WiFi task and the idle task, can’t get any CPU runtime. A blocked task switching interrupt can happen because a program runs into an infinite loop with interrupts disabled or hangs in an interrupt.

Great! Now I need to know which task is responsible for this. We know ESP-IDF dumps some useful info to UART0 when the watchdog is triggered . How nice would it be to send this info over to my logger that passes it on to the rsyslog daemon over UDP! Regretfully, not in this case. If context switches are blocked, WiFi/networking isn't going to function properly, so it is unlikely we're going to succeed with UDP transmission. Additionally, upon reviewing ESP-IDF's panic handler, there isn't a way to redirect or capture an output without modifying the source. So I thought the easiest would be setting up another Raspberry nearby, connect it to ESP32's UART0 and capture the output continuously, which I did. Results were puzzling, to say the least. Every time I saw different offending task in the log, and sometimes there was nothing in the log at all! I started to suspect this may be a hardware issue. I needed a confirmation, so I set up 2 identical ESP32 modules, one on a dev socket, powered by USB, and the other one - "in the field" with all the manual soldering and a bunch of wires. Uploaded identical binaries, connected identical peripheral devices. In a week, my first module had 0 issues, still up and running, and the second one froze once and auto-rebooted 3 times. Definitely a hardware problem. Could it be a faulty ESP32 chip? No problem, replaced it and ran the test again - nothing changed. My first setup is still up towards the end of the second week, and the second one reboots and freezes fiercely.

Is now the time to RTFM? Yes! Espressif has lots of PDFs on ESP32, and the solution must be there. The only problem is - too much reading and me being too lazy. Let me rather find a Guru and ask him. Sinclair Macgregor was very kind to help me. He suggested that the issue is probably related to the power supply:

I think you are describing a "brown out" a uc type of reset error that can hang a system, I think you have power or vreg issues. From what I see, I believe your main issue for stability MAY be related to regulated voltage supply issues in dynamic switching conditions (you may not see this without use of oscilloscope to capture the events as they may happen quickly enough to leave the IC in an indeterminate state - ie not enough time OFF to actually reset - or not low enough to actually reset but enough to cause what is known as a brown reset where SOME functions of IC reset but not all. Other possible sources of issues look to be the isolation strategy.

It was a revelation. I suddenly remembered that I have another ESP32 powered by a battery, with awful wiring, but an uptime of 46 days!

Those freezing and rebooting setups were powered by these modules:

as I needed 5v for some of the sensors, and then I used voltage regulator (LD1117 or similar) for 3.3v to power the ESP32. And of course I didn't use any of the capacitors recommended by the datasheet. Real heros don't read datasheets, lol. Now the capacitors are in place, and the problem is solved. Don't be like me, Read The F**ng Manuals!

It started as a simple home automation project. I was looking for a way to do the most basic things like track temperature/humidity, switch the lights, open water valve in the greenhouse, turn on the fan when it gets too hot, etc. So I looked into some ready-made automation kits and... decided to have some fun building my own! The idea was to set up WiFi-capable devices all over the place (luckily, my WiFi coverage allows that) and wire them to the switches/relays/sensors etc. My first try was Raspberry PI + USB relay module, driven by a python script. Works well for one or two locations, but if you want a dozen - it is an overkill and gets pricey. The next thing I tried was STM32 ("Blue Pill") module wired to ESP8266. STM32 is great to read multiple sensors and control several devices, but doesn't have any networking features. ESP8266 is a WiFi chip, but has a very limited number of I/O ports. Together they make a great combo, but there are cons. For example, I had to maintain 2 different firmwares, implement a protocol for the chips to talk to each other, put up with reliability issues because all connections were manually soldered and looked like this:

Another thing is the amount of available RAM/flash, that is pretty restrictive on both chips.

Finally, I found this beautiful ESP32 module that had everything I could ever think of, and more - in a single tiny module, for a very competitive price (N.B.: no, I'm not paid to advertise ESP32, nor am I affiliated with Espressif in any way. I was just truly fascinated by the chip).

Now I could easily connect 5 different sensors, 2 relays, 2 H-bridges, several LEDs and a buzzer, and make it all accessible over WiFi! The last missing piece of the puzzle was the software. At first, Arduino seemed like a very natural choice. There's support for ESP32, you get access to all the modules that mostly work well, the community is great, learning is easy, and it is a de-facto standard for simple MCU projects. But... there's a flipside:

• Arduino is too unified. It attempts to cover all sorts of MCUs, with different goals, features, hardware capabilities and design principles. What if I need access to hardware-specific features of ESP32, that aren't available, let's say, in STM32? ESP32 has its own development platform (ESP-IDF), and if you really want to get the most of the hardware, you have to use ESP-IDF. The problem is, Arduino went its own way. It accesses some of the hardware directly, bypassing the ESP-IDF. It runs startup code to initialize Arduino-specific features, which was not desirable in my case. It uses pre-configured (and, possibly, patched) version of ESP-IDF, and if you want to re-configure it for your particular needs (which is a common practice for ESP-IDF application), you're in for another challenge. So what happens if I want to use Arduino and ESP-IDF at the same time, without dirty hacks and workarounds? Seems like mission impossible, but no. There's a wonderful PlatformIO, that's not only a better alternative to Arduino's native IDE, but also allows to use Arduino as a library, on top of ESP-IDF! Now this should be the Ultimate Solution, I thought... and faced another handful of issues.
• Arduino wasn't designed for multitasking. No, it is perfectly possible to do multitasking in Arduino, there are scheduling libraries that offer pseudo-multitasking, and if there's native MCU support (which is the case for double-core ESP32) - you can always call native functions directly. At the same time, most of the Arduino libraries don't expect to be run in a multitasking environment, they don't care about synchronization, locking and race conditions, so one has to either patch the source, or apply various hacks to achieve stable performance. One example would be the OneWire library. This is a well-designed and mature library that works perfectly on AVR, STM32 (and I would expect, on other single-core MCUs in a single-thread mode). OneWire protocol depends on proper timing, so the library disables interrupts when sending/receiving data packets, with the naive belief that interrupts are the only possible cause of inaccurate time measurements and prolonged delays. Wrong! ESP32 context switches must be disabled as well, or the code must be re-designed to allow for a context switch to occur safely at the very moment when the protocol expects to see state change on the 1WIRE bus. As a result, this library fails consistently on ESP32 with multiple running tasks. For me, only about 30% of the measurements were successful and the rest were failing. The workaround was suggested and implemented by Chuck Todd in his fork of OneWire library. It is pretty straightforward - disable context switches for the duration of 1WIRE reads/writes, and voila, it works! But what about the rest of the tasks? Now they can't run until 1WIRE has finished its business... Far, far from perfect, IMHO.
• As I mentioned above, Arduino sometimes accesses the hardware directly, by reading/writing memory-mapped registers and ports. ESP-IDF does the same. So for the most of common stuff like writing to the serial port, we have 2 similar blocks of code that need to be included in the resulting binary, unnecessarily bloating its size.
• Using Arduino as a library together with ESP-IDF requires some (heavy - i think) modifications of both cores. Given that this is a community effort, there's a (substantial) time lag to see new ESP-IDF releases ported to PlatformIO's way of using it together with Arduino. For me, this was the last drop and I made a tough decision in favor of ESP-IDF alone, dropping Arduino completely, and never regretted it.

I don't know about you, but if I read this far, I would be puzzled. Multitasking? Concerns about firmware size? Just to switch a relay or get a temperature reading? The OP must be going nuts! Actually, that could be the case. After a couple of successful ESP32-driven projects, I got greedy. I wanted to see more and more features, and, the most important, I was not constrained by Arduino any longer. Here's what I wanted to achieve:

• True multitasking, so I could poll for a sensor in a simple loop and not worry about blocking other tasks.
• Modular architecture - a simple #include <XXX> followed by a call to useXXX(...); should be enough to make use of an external device attached to ESP32.
• Extensible design - adding new devices (or porting from Arduino) should be easy.
• Unified communication protocol to speak to the outer world - whether to get a sensor reading, or to send a control command to switch a relay - via different communication channels (serial port, MQTT, HTTP, WebSockets, etc). Human-readable. JSON-based.
• Unified logic for loading, using and storing module-specific settings. Human-readable. JSON-based.
• Unified asynchronous event dispatching system to tie different modules together, while reducing dependencies between the modules to a minimum.
• Over-the-air firmware updates with few simple clicks or unattended - when new firmware becomes available on the server.
• Logging with configurable appenders, for example, to save error logs to the flash memory and retrieve it when needed, or to send the log messages instantly to rsyslog server over UDP - very useful feature for debugging new code without physical access to the chip and/or it's UART0.
• Get sensor readings into influxdb time series database via MQTT and Telegraf and visualize it with Chronograf, or any other time-series database/UI.
• Allow to configure multiple WiFi APs and select optimal/available one automatically.
• If no WiFi SSID/password was pre-configured in the firmware, or no connection could be made within configured time frame, launch AP with captive portal, allowing to connect from any smartphone and configure WiFi options.
• Comprehensive Web-based User Interface (yes, I'm that crazy)!
  • Single page ReactJS - based app.
  • WebSockets to read/write state and settings of the running firmware and control attached devices.
  • Modular design - follows backend modules.
  • Served directly from the ESP32's IP address over WiFi, with an option to be served from a different server (to save flash space, for instance)
  • Displays general system/runtime info, such as RAM/flash usage/fragmentation, uptime, WiFi connection details, per-task info,
  • Allows to control pin modes from UI (switch between digital/ADC/DAC/PWM/...), read/write pin state, voltage, PWM duty/freq., etc.
  • Allows to scan popular buses for connected devices (I2C and MODBUS - get connected devices' addresses, 1WIRE - get codes of connected devices).
• All of the above - in the reusable library, with only the needed code to be included into the binary for a particular project.
• Must fit in the 2MB (actually, a little less) to make OTA updates possible on a standard ESP32 dev module with 4MB SPI Flash.
• Supervisor app:
  • Controls multiple devices via WebSockets API.
  • Provides simple higher-level scripting API with something as simple as Javascript, to run custom scenarios (for example, if the rain is detected by device A, tell device B to close the window).
  • Notifications to Telegram channel or any other messenger.
  • No central server - I don't want to pay for a VPS, domain a static IP. Rather, I want to have it on a couple of Raspberries (behind the NAT, dynamic IP) and make it accessible from any part of the world. TOR hidden services is one way to make it work.

Looking back, it's been about 6 months of part-time engagement, and most of the above (except the supervisor app) was implemented to a (not perfect, but) usable extent. I have 11 setups, driving different devices and feeding sensor data into the Influx server running on a Raspberry. Most of the boards have had an uptime of > 30 days (the rest have hardware issues due to manual soldering, caught by WDTs causing auto-reboot every now and then).

So I thought maybe there are other crazy guys and girls like me out there, who enjoy this stuff as much, and decided to share the source/docs. It may help enthusiasts to build something alike, to learn the MCU or just to have fun! :-)