IOT

Color Detection & Tracking with ESP32 CAM & OpenCV

Color detection is required for object recognition and is also employed in a variety of picture editing and sketching programs. This project uses the ESP32 CAM Module and OpenCV to detect and track colors. As a result, any specific colors will be detected during live video streaming. This method differs from other ESP32-CAM Color Detecting methods in that no color detection code is written for the microcontroller. Instead, we’ll utilize our laptops to write Python code. This strategy speeds up the procedure.

We’ll use a Client-Server connection to send the values from our Python code to the ESP32-CAM later. HSV, or Hue Saturation Value conversion, is the color detecting method utilized here.

The ESP32-CAM module, which is a tiny camera module with the ESP32-S chip, was utilized in this project. Apart from the OV2640 camera and many GPIOs for connecting peripherals, it also has a microSD card slot for storing photographs captured by the camera.

We’ll start by writing C code for the ESP32 CAM, then install Python and the necessary Python libraries. Later, we’ll look at how to use OpenCV with Python programming. This is an important tutorial because it will allow you to apply any type of image processing or machine learning on live video without having to write code in the Arduino IDE.

Hardware Required:

  • ESP32-CAM Board-AI-Thinker ESP32 Camera Module
  • FTDI Module-USB-to-TTL Converter Module
  • USB Cable-5V Mini-USB Data Cable
  • Jumper Wires-Female to Female Connectors

ESP32 CAM Module

The ESP32 Based Camera Module was developed by AI-Thinker.  The controller contains a Wi-Fi + Bluetooth/BLE chip and is powered by a 32-bit CPU. It has a 520 KB internal SRAM and an external 4M PSRAM. UART, SPI, I2C, PWM, ADC, and DAC are all supported by its GPIO Pins.

The module is compatible with the OV2640 Camera Module, which has a camera resolution of 1600 x 1200 pixels. A 24-pin gold plated connector links the camera to the ESP32 CAM Board. A 4GB SD Card can be used on the board. The photographs captured are saved on the SD Card.

ESP32-CAM Features :

  • The smallest 802.11b/g/n Wi-Fi BT SoC module.
  • Low power 32-bit CPU, can also serve the application processor.
  • Up to 160MHz clock speed, summary computing power up to 600 DMIPS.
  • Built-in 520 KB SRAM, external 4MPSRAM.
  • Supports UART/SPI/I2C/PWM/ADC/DAC.
  • Support OV2640 and OV7670 cameras, built-in flash lamp.
  • Support image WiFI upload.
  • Supports TF card.
  • Supports multiple sleep modes.
  • Embedded Lwip and FreeRTOS.
  • Supports STA/AP/STA+AP operation mode.
  • Support Smart Config/AirKiss technology.
  • Support for serial port local and remote firmware upgrades (FOTA).

ESP32-CAM FTDI Connection

  • There is no programmer chip on the PCB. So, any form of USB-to-TTL Module can be used to program this board. FTDI Modules based on the CP2102 or CP2104 chip, or any other chip, are widely accessible.
  • Connect the FTDI Module to the ESP32 CAM Module as shown below.
ESP32 CAM FTDI Module Connection
ESP32-CAMFTDI Programmer
GNDGND
5VVCC
U0RTX
U0TRX
GPIO0GND

Connect the ESP32’s 5V and GND pins to the FTDI Module’s 5V and GND. Connect the Rx to UOT and the Tx to UOR Pin in the same way. The most crucial thing is that you must connect the IO0 and GND pins. The device will now be in programming mode. You can remove it once the programming is completed.

Project PCB Gerber File & PCB Ordering Online

If you don’t want to put the circuit together on a breadboard and instead prefer a PCB. EasyEDA’s online Circuit Schematics & PCB Design tool was used to create the PCB Board for the ESP32 CAM Board. The PCB appears as seen below.

The Gerber File for the PCB is given below. You can simply download the Gerber File and order the PCB from https://www.nextpcb.com/

Download Gerber File: ESP32-CAM Multipurpose PCB

Now you can visit the NextPCB official website by clicking here: https://www.nextpcb.com/. So you will be directed to the NextPCB website.

  • You can now upload the Gerber File to the Website and place an order. The PCB quality is excellent. That is why the majority of people entrust NextPCB with their PCB and PCBA needs.
  • The components can be assembled on the PCB Board.

Installing ESP32CAM Library

Another streaming process will be used instead of the general ESP webserver example. As a result, another ESPCAM library is required. On the ESP32 microcontroller, the esp32cam library provides an object-oriented API for using the OV2640 camera. It’s an esp32-camera library wrapper.

Download the zip library as shown in the image from the following Github Link

After downloading, unzip the library and place it in the Arduino Library folder. To do so, follow the instructions below:

Open Arduino -> Sketch -> Include Library -> Add .ZIP Library… -> Navigate to downloaded zip file -> add

Source Code/Program for ESP32 CAM Module

Color Detection & Tracking with ESP32 CAM Module source code is available here. Copy and paste the code into the Arduino IDE.

#include <WebServer.h>
#include <WiFi.h>
#include <esp32cam.h>
const char* WIFI_SSID = “ssid”;
const char* WIFI_PASS = “password”;
WebServer server(80);
static auto loRes = esp32cam::Resolution::find(320, 240);
static auto midRes = esp32cam::Resolution::find(350, 530);
static auto hiRes = esp32cam::Resolution::find(800, 600);
void serveJpg()
{
auto frame = esp32cam::capture();
if (frame == nullptr) {
Serial.println(“CAPTURE FAIL”);
server.send(503, “”, “”);
return;
}
Serial.printf(“CAPTURE OK %dx%d %db\n”, frame->getWidth(), frame->getHeight(),
static_cast<int>(frame->size()));
server.setContentLength(frame->size());
server.send(200, “image/jpeg”);
WiFiClient client = server.client();
frame->writeTo(client);
}
Void handleJpgLo()
{
if (!esp32cam::Camera.changeResolution(loRes)) {
Serial.println(“SET-LO-RES FAIL”);
}
serveJpg();
}
Void handleJpgHi()
{
if (!esp32cam::Camera.changeResolution(hiRes)) {
Serial.println(“SET-HI-RES FAIL”);
}
serveJpg();
}
Void handleJpgMid()
{
if (!esp32cam::Camera.changeResolution(midRes)) {
Serial.println(“SET-MID-RES FAIL”);
}
serveJpg();
}
Void setup(){
Serial.begin(115200);
Serial.println();
{
using namespace esp32cam;
Config cfg;
cfg.setPins(pins::AiThinker);
cfg.setResolution(hiRes);
cfg.setBufferCount(2);
cfg.setJpeg(80);
bool ok = Camera.begin(cfg);
Serial.println(ok ? “CAMERA OK” : “CAMERA FAIL”);
}
WiFi.persistent(false);
WiFi.mode(WIFI_STA);
WiFi.begin(WIFI_SSID, WIFI_PASS);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
}
Serial.print(“http://”);
Serial.println(WiFi.localIP());
Serial.println(” /cam-lo.jpg”);
Serial.println(” /cam-hi.jpg”);
Serial.println(” /cam-mid.jpg”);
server.on(“/cam-lo.jpg”, handleJpgLo);
server.on(“/cam-hi.jpg”, handleJpgHi);
server.on(“/cam-mid.jpg”, handleJpgMid);
server.begin();
}
Void loop()
{
server.handleClient();
}

You must make a little adjustment to the code before uploading it. Change the SSID and password variables to match the WiFi network you’re using.

Compile the code and upload it to the ESP32 CAM Board. However, you must follow a few steps each time you post.

  • When you push the upload button, make sure the IO0 pin is shorted to the ground.
  • If you notice dots and dashes during uploading, immediately press the reset button.
  • Remove the I01 pin shorting with Ground and push the reset button one more after the code has been uploaded.
  • If the output is still not the Serial monitor, push the reset button once again.

Now you can see a similar output as in the image below.

Python Library Installation

  • In order for the live video stream to appear on our computer, we must develop a Python script that allows us to retrieve the video frames. The first step is to get Python installed. Go to python.org and download Python.
  • Once downloaded, install Python.

Install NumPy, and OpenCV,  libraries from the command prompt.

  • type: pip install numpy and press enter. After the installation is done.
  • type: pip install opencv-python and press enter.

Now open Idle code editor or any other python code editor.

Python Code + Color Detection using ESP32 CAM

Let’s take a closer look at the OpenCV technique for Color Detection and Tracking with the ESP32 CAM. Our major goal is to identify color, thus we’ll use the OpenCV library to perform HSV conversion.

The OpenCV library includes a conversion called HSV (hue saturation value), often known as HSB (hue, saturation, brightness). It’s widely used to recognize specific hues. However, there are some consistent parameters associated with detection that change from different objects and locations.

We convert the RGB image to an HSV image and then set a lower and upper bound for all three values (hue, saturation, and brightness), so anything that falls inside those bounds is what we want to see.

For further in-depth understanding, you can see the OpenCV documentation.

  • Let’s see the ESP32 CAM Python Color Detection code. Create a new file and then copy and paste the code below.
  • Let’s look at the Python Color Detection code for the ESP32 CAM. Make a new file and put the code below into it.


import cv2
import urllib.request
import numpy as np
def nothing(x):
pass
#change the IP address below according to the
#IP shown in the Serial monitor of Arduino code
url=’http://192.168.1.61/cam-lo.jpg’
cv2.namedWindow(“live transmission”, cv2.WINDOW_AUTOSIZE)
cv2.namedWindow(“Tracking”)
cv2.createTrackbar(“LH”, “Tracking”, 0, 255, nothing)
cv2.createTrackbar(“LS”, “Tracking”, 0, 255, nothing)
cv2.createTrackbar(“LV”, “Tracking”, 0, 255, nothing)
cv2.createTrackbar(“UH”, “Tracking”, 255, 255, nothing)
cv2.createTrackbar(“US”, “Tracking”, 255, 255, nothing)
cv2.createTrackbar(“UV”, “Tracking”, 255, 255, nothing)
while True:
img_resp=urllib.request.urlopen(url)
imgnp=np.array(bytearray(img_resp.read()),dtype=np.uint8)
frame=cv2.imdecode(imgnp,-1)

hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
l_h = cv2.getTrackbarPos(“LH”, “Tracking”)
l_s = cv2.getTrackbarPos(“LS”, “Tracking”)
l_v = cv2.getTrackbarPos(“LV”, “Tracking”)
u_h = cv2.getTrackbarPos(“UH”, “Tracking”)
u_s = cv2.getTrackbarPos(“US”, “Tracking”)
u_v = cv2.getTrackbarPos(“UV”, “Tracking”)
l_b = np.array([l_h, l_s, l_v])
u_b = np.array([u_h, u_s, u_v])

mask = cv2.inRange(hsv, l_b, u_b)
res = cv2.bitwise_and(frame, frame, mask=mask)
cv2.imshow(“live transmission”, frame)
cv2.imshow(“mask”, mask)
cv2.imshow(“res”, res)
key=cv2.waitKey(5)
if key==ord(‘q’):
break

cv2.destroyAllWindows()



  • Now before you run this code make sure you replace the IP address that you got in the Arduino Serial monitor with the one which we are having in the URL variable in the above code.
  • Now Run the code by pressing the F5 button. After running the code you can see three screens that appear in front of you.

We want to track the blue color in the above frame. As a result, we slide the tracking bars in the Tracking window to alter the LH, LS, LV, UH, US, and UV values. Adjust the values of these bars such that white color appears only in blue colored spots in the Mask window, as shown in the images below.

Although I am in the dark with the blue-colored item, as you can see above. It can still detect it.

You have successfully detected the blue color.

Similarly, you can identify any specific colors by altering the Tracking bars.

The next step is to draw a contour around the colored image and specify the detected color’s average center coordinates. Save the LH, LS, LV, UH, US, and UV values from the previous program in the Trackbar window.

Create a new file and paste the code below into it.

import cv2
import urllib.request
import numpy as np
def nothing(x):
pass
url=’http://192.168.1.61/cam-lo.jpg’
##”’cam.bmp / cam-lo.jpg /cam-hi.jpg / cam.mjpeg ”’
cv2.namedWindow(“live transmission”, cv2.WINDOW_AUTOSIZE)
l_h, l_s, l_v = 92, 57, 50
u_h, u_s, u_v = 142, 153, 178
while True:
img_resp=urllib.request.urlopen(url)
imgnp=np.array(bytearray(img_resp.read()),dtype=np.uint8)
frame=cv2.imdecode(imgnp,-1)
#_, frame = cap.read()

hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
l_b = np.array([l_h, l_s, l_v])
u_b = np.array([u_h, u_s, u_v])

mask = cv2.inRange(hsv, l_b, u_b)
cnts, _ = cv2.findContours(mask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
for c in cnts:
area=cv2.contourArea(c)
if area>2000:
cv2.drawContours(frame,[c],-1,(255,0,0),3)
M=cv2.moments(c)
cx=int(M[“m10”]/M[“m00”])
cy=int(M[“m01”]/M[“m00”])
cv2.circle(frame,(cx,cy),7,(255,255,255),-1)
cv2.putText(frame,”blue”,(cx-20, cy-20),cv2.FONT_HERSHEY_SIMPLEX, 1, (0,0,255),2)

res = cv2.bitwise_and(frame, frame, mask=mask)
cv2.imshow(“live transmission”, frame)
cv2.imshow(“mask”, mask)
cv2.imshow(“res”, res)
key=cv2.waitKey(5)
if key==ord(‘q’):
break

cv2.destroyAllWindows()

Now make sure you have changed the l_h, l_s, l_v, u_h, u_s & u_v variables with the values of LH, LS, LV, UH, US, and UV from the previous code. And run the program. Similar output must be visible.

You have successfully detected the blue color. Similarly, by adjusting HSV values and building as many contours as you want, you may identify any given hue or color.

Conclusion:

This is how you can use the ESP32 CAM and OpenCV to detect and track colors. . We will be back soon with more informative blogs soon.

Leave a Reply

Your email address will not be published.