ESP32: Automation

9 minute read
,

Where I discuss different methods of automating the process of compiling/linking/loading code for an embedded microcontroller.

Update March 13, 2022

For automation to work well, it needs to work on multiple platforms along with other automation. For example, I use git as my versioning system, and GitHub for Dashboard, the repository for this entry. The problem I had with Option 2 ESP32 and Make below was that it is difficult to incorporate environmental variables. Environmental variables are values used locally to define specific aspects of your development environment. I use the environmental variables, AVR_PORT and AVR_MCU in my AVR_C code to set the microcontroller and serial port. On my Mac in my .zshrc file, I have

export AVR_PORT=/dev/cu.usbmodem4101 
export AVR_MCU=atmega328pb

and on my Linux system, I use

export AVR_PORT=/dev/ttyUSB0
export AVR_MCU=atmega328pb

And in my AVR_C makefiles, I am able to reference $(AVR_PORT) to get the name of the local serial port.

As I develop my programs across three different platforms, Linux, WSL, and macOS, this approach allows me to use the same git repository on all three platforms immediately and not make changes to makefiles etc.

However!

For reasons, I had not been able to solve, arduino-cli and the Makefile example aren’t able to use the same approach. And after several days of attempting multiple methods of solving and learning far more about makefile variable substitution than I wished, I realized there was an easy fix.

Create another makefile called env.make and and use an include to pull it in to the Makefile. This file will be ignored by git as the name of the file in .gitignore. Each folder with an executable .ino file requires a Makefile and a env.make. Be sure BOARD/PORT/BUILD are all capitalized! The one on my mac looks like this (this file resides in the same folder of Makefile and source files):

BOARD=esp32:esp32:featheresp32
PORT=/dev/cu.usbserial*
BUILD=build

The new Makefile looks like this:


include env.make

.PHONY: default lint all flash clean

default: all flash clean

lint:
	cpplint --extensions=ino --filter=-legal/copyright *.ino

all:
	arduino-cli compile --fqbn $(BOARD) --output-dir $(BUILD) ./

flash:
	arduino-cli upload --fqbn $(BOARD) --port $(PORT) --input-dir $(BUILD)

clean:
	rm -r build

I’ve test this approach on both macOS and Linux and works like a charm!

My recommendation is to use this approach. It provides all of the capabilities desired, automation, local customization, error notification, and multiple processes. The latest git repository has the right Makefile, all you need to do is to add your own env.make as shown above.

The discussion below regarding Option 2 ESP32 and Make still stands, however, I have slightly modified the approach. Use the two Makefiles (one in the GitHub repository and one you will need to create locally) described above.

Note: File Structure

Here is the simple file structure for blink:

ls -l blink/
Makefile
blink.ino
env.make

Makefile and blink.ino will be tracked by git, while env.make will not, which requires the user to add it themselves. (similar to arduino_secrets.h)

Introduction

The process of compiling, linking and uploading code to your embedded microcontroller happens frequently in developing an application. It could happen as much as 20-30 times a day. As such, you want it to be as fast as possible as well as as easy as possible, hence automation becomes key.

For example, assuming 30 iterations, a difference of 5 keystrokes and 30 seconds, quickly becomes, 150 keystokes and 15 minutes of lost effort and time. More importantly, automation decreases the chance for mistakes and losing your focus on developing your code.

This discussion will focus on Sublime Text (ST) and how to automate the process when using ST.

Overview of Sublime Test Build

The process of compilation, linking and uploading is called “build” or “building”. Sublime Text has a capability under Tools called Build. You access it via Tools -> Build or Ctrl/Cmd-B (Ctrl = control key for Windows/Linux and Cmd = Command key for macOS). However, it is rare that this capability will work without some setup. The following steps need to happen:

  1. Tools -> Build System needs to indicate the correct Build for the files being edited.
  2. If you are building for an embedded processor, more than likely it won’t be one of the existing Build Systems. Which means you would need to select the element at the bottom of the list New Build System… and create the appropriate build system.
  3. There is one more capability, which I want to call out Tools -> Build With or Shift-Ctrl/Cmd-B, which we will cover later. This capability allows for options with in the Build process.

Two Options

For our purposes, there are two options for creating a build system, one, using the sublime-build process and the other using Make. The former is a custom process within ST which is quite good as it provides for not only the build process, it also provides for identifing errors and connecting them back to the file where they occurred. The latter is an application which is quite powerful and extremely well-supported. We’ll use both approaches for this discussion.

1. ESP32 and sublime-build

One of the reasons the Arduino team came up with arduino-cli was to support automation and a great value of Sublime Text, is its ability to be automated. So, not unlike chocolate and peanut butter, its imperative we get these two elements to work together to be automated!

As mentioned above, Sublime Text has a capability called Build Integration. It is the Tools -> Build menu option. The first thing we need to do is to add a small piece of code to perform the automation.

{
    "cancel": {"kill": true},
    "shell": true,
    "file_patterns": ["*.ino", ".c", "*.cpp", "*.h"],
    "cmd": ["arduino-cli compile -b esp32:esp32:featheresp32", "$folder"],
    "file_regex": "^(..[^:\n]*):([0-9]+):?([0-9]+)?:? (.*)$",
    "variants": [
      {
          "name": "Upload",
          "cmd": ["arduino-cli upload -b esp32:esp32:featheresp32 -p /dev/cu.usbserial*", "$folder"]
      }
  ]
}

Steps to follow:

  1. Select and copy the text above.
  2. In ST, go to Tools -> Build System -> New Build System (at the bottom)
  3. This will open a new text window, untitled.sublime-build, and completely replace the contents in the file with the text above.

Note: You will need to change the following:

  1. USB port to the name of the USB port on your system, in Linux it might be /dev/ttyUSB0. There is one place to change the USB port.

  2. The FQBN (fully qualified board name) to the one you are using such as esp32:esp32:esp32doit-devkit-v1. There are two places to change the board name.

  3. Save this file with the name arduino-cli.sublime-build, it will save it in one of the following locations:

  • Windows: %APPDATA%\Sublime Text 3/User
  • macOS: ~/Library/Application Support/Sublime Text 3/User
  • Linux: ~/.config/sublime-text-3/User
  1. In ST open an Arduino sketch folder, in this case, I’ll use blink. I’m in the blink folder and have clicked on the blink.ino sketch.
  2. To ensure you are using the right build system, go to Tools -> Build System and select arduino-cli.
  3. Press Ctrl/Cmd-B and arduino-cli will begin to compile.
  4. If the compile was successful, press Shift-Ctrl/Cmd-B, then the down arrow and arduino-cli will upload the file to the ESP32.

Note: In the future, it might be easier to remember:

  • Shift-Ctrl-B <return> will compile.
  • Shift-Ctrl-B <down arrow> <return> will upload.

Pretty quickly, it becomes muscle memory and your complilation/upload steps become that much faster!

If you have problems

  1. If nothing happens, then the file has JSON formatting issues. Be sure the file looks identical to the one above with the proper USB and board naming. Many times, ST will show you the JSON formatting issue with a red box or line.

Note: JSON Formatting Comments:

  • Look for commas at the end of each line except for the last parameter
  • Be sure to have balanced quotes and all quotes must be double quotes.
  • Have a colon after the quote on each key-word.
  1. If you get a shell error, such as “arduino-cli…. can’t be found”, your path inside of ST isn’t correct, the fastest way to fix this is to use the full name of the path for arduino-cli. In my case in Linux, it was /home/lkoepsel/bin/arduino-cli. You can find this path using the command “which arduino-cli” at the command line.
  2. You must be in the folder of the sketch you wish to compile and upload, in this case its the same as the Arduino IDE as it looks for the name of an ino file with the same name as the folder.
  3. You don’t have to compile then upload, if you simply wish to upload, then use the key sequence to do that.
  4. This version has a file_regex line which will highlight the errors found when compiling code. Its preliminary, so I wouldn’t use it solely to find errors, however, it will put a red error, along with an error message by each line which has an error. Double-clicking on the line in the console, will take you to the line in the specific file which has an error.

2. ESP32 and Make (Use Update above, process remains the same)

This video - Arduino CLI and the art of command line is excellent as to how to use make along with arduino-cli to automate the build process. Using Make is an excellent idea as I also use it in my Arduino Uno automation. As I said before, Make is extremely powerful and complex, so it is easy to go wrong quickly. I recommend you use the Makefile recommended in the video (also below) as is. If you want to make changes, go slowly, check your work and refine over time.

My process

  1. I use this makefile (called Makefile) in each folder:
BOARD=esp32:esp32:featheresp32
PORT=/dev/cu.usbserial*
BUILD=build

.PHONY: default lint all flash clean

default: all flash clean

lint:
	cpplint --extensions=ino --filter=-legal/copyright *.ino

all:
	arduino-cli compile --fqbn $(BOARD) --output-dir $(BUILD) ./

flash:
	arduino-cli upload --fqbn $(BOARD) --port $(PORT) --input-dir $(BUILD)

clean:
	rm -r build
  1. I have added this file (called Make.sublime-build) and I save it to Preferences -> Browse Packages -> Packages -> Makefile -> Make.sublime-build. In other words, I save the file named Make.sublime-build in a folder called Makefile, as compared to the one above, where I saved arduino-cli.sublime-build in a folder called Users.
{
	"cmd": "make",
	"shell": true,
	"file_regex": "^(..[^:\n]*):([0-9]+):?([0-9]+)?:? (.*)$",
	"selector": "source.makefile",
	"keyfiles": ["Makefile", "makefile"],

	"variants":
	[
		{
			"name": "Flash",
			"shell_cmd": "make flash"
		},
		{
			"name": "Clean",
			"shell_cmd": "make clean"
		},
		{
			"name": "Lint cpp",
			"shell_cmd": "make lint"
		}
	]
}
  1. In this situation, you would select the Build System -> Make and if you were to select Build With… you would see a dropdown of:
Make
Make - flash
Make - clean 
Make - lint cpp

Make

This command will perform all of the steps, indicated by default in the Makefile, which means it will compile writing the hex file to a build folder in the current folder, it will upload the proper hex file in the build folder to the microcontroller than it will delete the build file.

Make - flash

Will use the local build folder and will upload the code to the microcontroller. This has less value then usual as the build folder is deleted each time you run the Make command above.

Make - clean

Deletes the build folder.

Make - lint cpp

Runs a program called lint for C++ which will help show you possible formatting and naming issues. While lint calls them errors, this isn’t the same thing as a compilation error, it is a linting error. Which means the code doesn’t follow specific standards as to coding. I use when I have a compile error, which I can’t solve and sometimes linting helps.

Comments powered by Talkyard.