ESP32: Solving Issues with Arduino Development

Where I demonstrate identifying and solving specific issues with developing code for the ESP32 using the Arduino framework.

Use Both the IDE and the CLI

While my entries are primarily related to using arduino-cli, I will use the IDE for a couple of reasons:

1. To test something that might work or should work, however, it isn’t. For example, if I buy a new board, I’ll use the IDE and blink to simply check if the board works.
2. To upload data to the SPIFFS (of LittleFS ) filesystem. Its very easy (see below) to add SPIFFS file upload capability to the IDE. Its not quite as easy using the command line and/or I haven’t spent time attempting to understand it.
3. To find the USB port. Sometimes the easiest method of determing the USB port is to fire up the IDE, attach the board and see what port shows up.

Sublime Text Integration

Moved to here - Automation

SPIFFS -> LittleFS

Switched from SPIFFS to LittleFS , it appears that LittleFS is faster (2-3x) and more reliable. Haven’t confirmed either, however, no issues after switching. (For more information as to why LittleFS, see joltwallet on Github)

1. The latest 2.0.2 code has LittleFS capabilities, however, it is critical to spell correctly as shown, LittleFS. It is not LITTLEFS (all-caps). Replaced (already performed) all references from SPIFFS and SPIFFS.h to LittleFS and LittleFS.h.
2. A new tool is required: Arduino ESP32 filesystem uploader, download the latest, extract and replace esp32fs.jar in the Arduino/tools folder then restart the IDE.
3. Once IDE is restarted, I found I needed to erase all Flash to get things to work. It is in the dropdown shown when ESP32 Sketch Data Upload is selected, it is at the very bottom. Perform this step then select ESP32 Sketch… and use the LittleFS selection to upload the data folder.

LittleFS Requirements

In order to use LittleFS, you need two things:

1. Arduino IDE tools to upload data

• *In ~/.arduino15/packages/esp32/tools, confirm mklittlefs exists. This is the tool which will make a LittleFS filesystem on the ESP32.
• The .jar file in …/Arduino/tools/ESP32FS/tool/ must be replaced by the one on github, otherwise the three options will not appear in the Arduino dropdown Tools -> ESP32 Sketch Data Upload. As both .jar files have the same name, I added .spiffs to the end of the old one to remind me.

2. Arduino framework tools to use LittleFS in your code

• The Arduino core package as of v2.00 now contains LittleFS.h as indicated by this note. Make sure your IDE and arduino-cli are both version 2.0 or higher. The one I use is 2.0.2
• The actual header file required to use LittleFS is located here: …/.arduino15/packages/esp32/hardware/esp32/2.0.2/libraries/LittleFS/src. Make sure it appears there, if not, reload the ESP32 package for the Arduino.

Confirm both elements are correct.

LittleFS Utility added

Use serial monitor to gather information about installed files on the ESP32 via LittleFS. Sample output:

Initializing FS...done.
File system info:
        Total:    1507328 bytes
        Used:      315392 bytes
        Free:     1191936 bytes

        favicon.png               717
        gauge.min.js.map       229040
        gauge.min.js            44751
        index.html               2547
        lightbulb.svg             547
        script.js                3623
        style.css                1782
        switch-closed.svg         738
        switch-open.svg           858

Code is here

IDE SPIFFS Upload

Random Nerd Tutorials has a nice tutorial on this. A couple of comments:

• Once you have downloaded esptool_py and unzipped it to a folder, you will need to move it to the Arduino folder (the one with the libraries), except in this case, move it to the tools folder. (You might need to create it.)
• As RNT says, move the entire unzipped folder, so you get Arduino -> tools -> ESP32FS -> tool esp32fs.jar.
• Once you have done that, restart the IDE and you will see the under Tools -> ESP32 Sketch Data Upload
• And to use it, you must be in a viable sketch folder with a folder called data. Inside of the data folder are the files you wish to upload. Also make sure you are attached to your board. When all of this is true, click on Tools -> ESP32… and watch the console for messages. If successful you will see messages to that effect.

-v for Verbose

Verbose is your friend. If you aren’t able to understand a specific error message or why something isn’t being found, use the -v option to gain a better understanding. For example, I believed the libraries needed to be installed in ~/Library/Arduino15/libraries, however, once I installed libraries in that folder, I still had a “not found” message.

I executed a *ac lib list -v" command and saw the following:

INFO[0000] Adding libraries dir                          dir=/Users/lkoepsel/Documents/Arduino/libraries location=user
...
INFO[0000] Executing `arduino-cli lib list`
Name              Installed Available    Location              Description
AsyncTCP          1.1.1     -            LIBRARY_LOCATION_USER -
ESPAsyncWebServer 1.2.3     -            LIBRARY_LOCATION_USER -

INFO[0000] Done

Missing Libraries

The number one thing to check if you are having issues at the beginning with missing libraries, is to ensure you are running the latest core framework. Run “arduino-cli core list” to see the version you are running. As of March, 2023 the latest is 2.0.2 for ESP32. If yours is not atleast 2.0, you will have issues. Make sure your arduino-cli.yaml file has the correct board reference:

# if using a 3rd party board, add to yaml file
board_manager:
  additional_urls:
    - https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json

If you are having difficulties with header files not being found, more than likely a library hasn’t been installed or its been referenced incorrectly. The command to list the installed libraries is arduino-cli library list or abbreviated below:

ac lib list
Name              Installed Available    Location              Description     
AsyncTCP          1.1.1     -            LIBRARY_LOCATION_USER -               
ESPAsyncWebServer 1.2.3     -            LIBRARY_LOCATION_USER -   

If the library desired isn’t listed you will more than likely run into this problem.

# The process in general is: (during compilation)
/home/lkoepsel/Desktop/esp32-weather-station/esp32-weather-station.ino:29:10: fatal error: sps30.h: No such file or directory
[#include](bear://x-callback-url/open-tag?name=include)  <sps30.h>
                                                          ^~~~~~~~~
compilation terminated.

The first thing to try is to use the library search function to find it in the Arduino repository. I’ve removed much of the text associated with the command to simplify the explanation. Note the key text is “includes sps30.h”, which is the missing header file.

# search for the missing header file w/o “.h”
ac lib search sps30
…
Name: “sensirion-sps”
…
Provides includes: sps30.h

If found, it is fairly easy to install the library.

# install the library
arduino-cli lib install “sensirion-sps” 
…
Installed sensirion-sps@1.0.0

Library can’t be found using arduino-cli

If unable to find a built library xxyyzz, search Github or google esp32 xxyyzz

1. More than likely, it will be a Github repository with a release zip file or you will need to click on the green Code button then click on Download ZIP.
2. Go to Downloads and select xxyyzz.zip, unzip xxyyzz.zip to folder xxyyzz. You might need to rename it by removing -master from the end of the folder name.
3. Move folder xxyyzz to /home/Arduino/Libraries

Pyserial

If you have upload to board issues, confirm pyserial is installed and works:

python -m serial.tools.miniterm /dev/ttyUSB0
# if python can't find pyserial
pip3 install pyserial

USB Issues

Linux has permission issues with the USB port, if you have installed pyserial, however it can’t connect to the board, it might be a USB permissions problem.

# if above has a permissions issue, attempt with sudo
sudo python -m serial.tools.miniterm /dev/ttyUSB0

# if it works with sudo, run sudo dmesg to identify port, look for lines shown below
sudo dmesg
…
[16700.904179] usb 1-2: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00
[16700.904184] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
…

# get the idProduct and idVendor and fill in below
sudo nano /etc/udev/rules.d/50-myusb.rules
SUBSYSTEMS==“usb”, ATTRS{idVendor}==“10c4”, ATTRS{idProduct}==“ea60”, GROUP=“plugdev”, MODE=“0660”, TAG+=“uaccess”
# save and exit from nano, test again. If it doesn't work, 
# reboot, to ensure the file is read.

Arduino IDE 2.x No module named ‘serial’

While I normally use the homebrew installed python3, which installs it at /opt/homebrew/bin/python3, I found the Arduino IDE was using /Library/Developer/CommandLineTools/usr/bin/python3, which is the native version of python3. While I had pyserial installed for the homebrew version, I needed to do the same for the native version.

/Library/Developer/CommandLineTools/usr/bin/pip3 install pyserial

Did the trick.

Arduino IDE 2.x No Extensions

I wanted to use the IDE to upload the Data folder to the ESP32, however, the Arduino IDE can’t use it. The new IDE is based on Electron, while the extensions and the older Arduino 1.x IDE is based on Java.

Follow this thread for more information: Github:Arduino - Missing Support for external tools

Board Boot Timeout

If you attempt to upload and see the following:

esptool.py v3.1
Serial port /dev/cu.usbserial-0001
Connecting........_____....._____....._____....._____....._____....._____....._____

A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header
Error during Upload: Failed uploading: uploading error: exit status 2

Your board requires a Boot press in order to upload. This doesn’t happen to me on my Linux platform, however it does happen to me on my Mac. The solution (thanks to Random Nerd Tutorials)! In a nutshell, press Boot prior to uploading code or attach a 10 uF electrolytic capacitor between EN and GND. See the post for more details.

Identify the Port

Running the pyserial program can help identify the port number you need to use to upload code. For example:

--- Available ports:
---  1: /dev/cu.BLTH         'n/a'
---  2: /dev/cu.Bluetooth-Incoming-Port 'n/a'
---  3: /dev/cu.usbserial-01F4D348 'CP2104 USB to UART Bridge Controller - CP2104 USB to UART Bridge Controller'
--- Enter port index or full name:

We can see that the 3rd option cu.usb… would be the port we want to use. We copy the full name and use that with the -p option on the arduino-cli upload command. On the same system, each board might report with a different port number. For example, the Adafruit Feather ESP32 shows /dev/cu.usbserial-01F4D348 on my Mac, while the NODEMCU-32 board shows /dev/cu.usbserial-0001, while both boards show as /dev/ttyUSB0 on my Linux system.

Note: Don’t forget about tab completion. While the numbers at the end of usbserial seem cryptic, you can type “/dev/cu.usb” then hit tab and more than likely it will find and enter the correct port number.

Clean Out Old Files

I changed the name of a file and I kept getting error messages based on the old file’s contents. In order to force a complete recompilation of the code (or to clean the code) you need to use --clean in your compilation command. As in:

ac compile --clean -b esp32:esp32:featheresp32 WiFIAP

From arduino-cli documentation:

--clean Optional, clean up the build folder and do not use any cached build

Consider SVG for icons

Most of the ESP32 projects involve some sort of Webserver. And as appropriate its nice to add visual elements to the website. While icons are widely available in sizes, all are going to be substantially larger than 600-700 bytes. SVG icons are drawn on the site and require far less space. For example, a simple one showing a open switch consumes 651 bytes. Two places I’ve found are:

• https://icons.getbootstrap.com/
• https://www.svgrepo.com/collection/electric-circuits/

To add them, simple put the rather large text into the body of the site:

          <p class="card-title">
            <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentColor" class="bi bi-lightbulb" viewBox="0 0 16 16">
              <path d="M2 6a6 6 0 1 1 10.174 4.31c-.203.196-.359.4-.453.619l-.762 1.769A.5.5 0 0 1 10.5 13a.5.5 0 0 1 0 1 .5.5 0 0 1 0 1l-.224.447a1 1 0 0 1-.894.553H6.618a1 1 0 0 1-.894-.553L5.5 15a.5.5 0 0 1 0-1 .5.5 0 0 1 0-1 .5.5 0 0 1-.46-.302l-.761-1.77a1.964 1.964 0 0 0-.453-.618A5.984 5.984 0 0 1 2 6zm6-5a5 5 0 0 0-3.479 8.592c.263.254.514.564.676.941L5.83 12h4.342l.632-1.467c.162-.377.413-.687.676-.941A5 5 0 0 0 8 1z"/>
            </svg>
            </i> GPIO 2</p>
          <p>

More on SVG

While its easy AND it works to add the SVG code into the index.html file, it gets messy fast. A preferable method is to serve it as a file, however to do that you need to tell the webserver what to do with it. This isn’t difficult to do, however, it doesn’t seem to be well documented.

For every file, you wish to display, you need to add a server handler. For example, to serve the svg files you need to do the following:

  // Route to load lightbulb.svg file
  server.on("/lightbulb.svg", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send(SPIFFS, "/lightbulb.svg", "image/svg+xml");
  });

  // Route to load switch-closed.svg file
  server.on("/switch-closed.svg", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send(SPIFFS, "/switch-closed.svg", "image/svg+xml");
  });

  // Route to load switch-open.svg file
  server.on("/switch-open.svg", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send(SPIFFS, "/switch-open.svg", "image/svg+xml");
  });

And you need to add the SVG files to the data folder, here is an example of the lightbulb.svg file which I used from Bootstrap icons:

<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" fill="currentColor" viewBox="0 0 16 16">
  <g>
    <path d="M2 6a6 6 0 1 1 10.174 4.31c-.203.196-.359.4-.453.619l-.762 1.769A.5.5 0 0 1 10.5 13a.5.5 0 0 1 0 1 .5.5 0 0 1 0 1l-.224.447a1 1 0 0 1-.894.553H6.618a1 1 0 0 1-.894-.553L5.5 15a.5.5 0 0 1 0-1 .5.5 0 0 1 0-1 .5.5 0 0 1-.46-.302l-.761-1.77a1.964 1.964 0 0 0-.453-.618A5.984 5.984 0 0 1 2 6zm6-5a5 5 0 0 0-3.479 8.592c.263.254.514.564.676.941L5.83 12h4.342l.632-1.467c.162-.377.413-.687.676-.941A5 5 0 0 0 8 1z"/>
  </g>
</svg>