This is a note of what I did and learned for making video recordings on Raspberry Pi CM4
First of all
These are my development environments
Hardware
- Raspberry Pi CM4 Model B
- Waveshare Raspberry Pi CM4 IO board (https://www.waveshare.com/cm4-io-base-b.htm)
- ReSpeaker 2-Mics Pi HAT (https://wiki.seeedstudio.com/ReSpeaker/)
- IMX219
Software
- Raspberry Pi OS Bullseys
- Picamera2 (https://datasheets.raspberrypi.com/camera/picamera2-manual.pdf)
- PyAudio (https://people.csail.mit.edu/hubert/pyaudio/)
Background
I wanted to make something using audio on Raspberry Pi. I did some work using a camera/video with Raspberry Pi for fun and job but I had never done anything with audio. So I decided to create a program to create a video with voice.
What I did
Firstly, I googled how to create an audio system in Raspberry Pi and found a GitHub repository provided by Microsoft. It is quite organized and easy to follow. So I decided to use it for my reference.
Set up microphon
Then I need to microphone to record sounds. I bought a ReSpeaker 2-Mics Pi HAT like the reference and followed the setup guidance.
I was using Bookworm first, not Bullseys, for the OS but it didn’t work. Specifically, after running the install.sh script, there is no result of aplay -l . I tested the install script in Raspberry Pi 3 + Bullseye, CM4 + IO board + Bullseys, and CM4 + IO board + Bookworm. Then only CM4 + IO board + Bookworm didn’t work. I think it doesn’t support the new OS currently (2024/02/24).
Also, I bought the USB audio adapter as well. When the speaker hat doesn’t work, I can tell where the problem comes from like my code or setup using the audio adapter. It’s quite cheap and much worth to invest to do an efficient investigation.
Set up camera
I set up an IMX219 camera and Picamera2. The following file is my configuration for it.
Also, I need to install Picamera2. Currently, it can be installed through apt. I executed the following commands in my Raspberry Pi. You can find the official way to install it in the docs.
# check update
sudo apt update -y
sudo apt upgrade -y
# picamera
sudo apt install -y python3-picamera2https://datasheets.raspberrypi.com/camera/picamera2-manual.pdf
Test recording
I created script to create video and audio with bash script before creating python code. Before doing that, make sure your connection with camera and mic is detected with following commands.
$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: vc4hdmi0 [vc4-hdmi-0], device 0: MAI PCM i2s-hifi-0 [MAI PCM i2s-hifi-0]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: vc4hdmi1 [vc4-hdmi-1], device 0: MAI PCM i2s-hifi-0 [MAI PCM i2s-hifi-0]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 2: seeed2micvoicec [seeed-2mic-voicecard], device 0: bcm2835-i2s-wm8960-hifi wm8960-hifi-0 [bcm2835-i2s-wm8960-hifi wm8960-hifi-0]
Subdevices: 1/1
Subdevice #0: subdevice #0
ats@voice:~ $ arecord -l
**** List of CAPTURE Hardware Devices ****
card 2: seeed2micvoicec [seeed-2mic-voicecard], device 0: bcm2835-i2s-wm8960-hifi wm8960-hifi-0 [bcm2835-i2s-wm8960-hifi wm8960-hifi-0]
Subdevices: 1/1
Subdev$ libcamera-hello --list-cameras
Available cameras
-----------------
0 : imx219 [3280x2464] (/base/soc/i2c0mux/i2c@1/imx219@10)
Modes: 'SRGGB10_CSI2P' : 640x480 [103.33 fps - (1000, 752)/1280x960 crop]
1640x1232 [41.85 fps - (0, 0)/3280x2464 crop]
1920x1080 [47.57 fps - (680, 692)/1920x1080 crop]
3280x2464 [21.19 fps - (0, 0)/3280x2464 crop]
'SRGGB8' : 640x480 [103.33 fps - (1000, 752)/1280x960 crop]
1640x1232 [41.85 fps - (0, 0)/3280x2464 crop]
1920x1080 [47.57 fps - (680, 692)/1920x1080 crop]
3280x2464 [21.19 fps - (0, 0)/3280x2464 crop]These are commands for recording audio and video.
arecord -d $duration audio.wavlibcamera-vid -t $duration -o video.h264Afterword, I converted the h264 format to MP4 and combined the audio and video using ffmpeg commands.
# Convert h264 to mp4
ffmpeg -i video.h264 -c copy video.mp4
# Combine audio and video
ffmpeg -i video.h264 -i audio.wav -c:v copy -c:a aac output.mp4Recoding with Python
After testing recordings with CLI, I started to write codes with Python. I used Picamera2 and PyAudio.
They are quite simple and there are a lot of samples to try. I put my code samples below.
from picamera2.encoders import H264Encoder, Quality
from picamera2 import Picamera2
import pyaudio
import numpy as np
import wave
from multiprocessing import Process
import time
RECORD_SECONDS = 120
VIDEO_FILENAME = 'video.h264'
AUDIO_FILENAME = 'audio.wav'
def record_video():
picam2 = Picamera2()
picam2.configure(picam2.create_video_configuration())
encoder = H264Encoder()
picam2.start_recording(encoder, VIDEO_FILENAME, quality=Quality.HIGH)
time.sleep(RECORD_SECONDS)
picam2.stop_recording()
def record_audio():
RESPEAKER_RATE = 16000
RESPEAKER_CHANNELS = 2
RESPEAKER_WIDTH = 2
# run getDeviceInfo.py to get index
RESPEAKER_INDEX = 1 # refer to input device id
CHUNK = 1024
WAVE_OUTPUT_FILENAME = AUDIO_FILENAME
p = pyaudio.PyAudio()
stream = p.open(
rate=RESPEAKER_RATE,
format=p.get_format_from_width(RESPEAKER_WIDTH),
channels=RESPEAKER_CHANNELS,
input=True,
input_device_index=RESPEAKER_INDEX,)
print("* recording")
frames = []
for i in range(0, int(RESPEAKER_RATE / CHUNK * RECORD_SECONDS)):
data = stream.read(CHUNK)
# extract channel 0 data from 2 channels, if you want to extract channel 1, please change to [1::2]
a = np.frombuffer(data,dtype=np.int16)[0::2]
frames.append(a.tobytes())
print("* done recording")
stream.stop_stream()
stream.close()
p.terminate()
wf = wave.open(WAVE_OUTPUT_FILENAME, 'wb')
wf.setnchannels(1)
wf.setsampwidth(p.get_sample_size(p.get_format_from_width(RESPEAKER_WIDTH)))
wf.setframerate(RESPEAKER_RATE)
wf.writeframes(b''.join(frames))
wf.close()
if __name__ == '__main__':
video_thread = Process(target=record_video)
audio_thread = Process(target=record_audio)
video_thread.start()
audio_thread.start()
video_thread.join()
audio_thread.join()One note here. I used threading instead of multiprocessing first. Then I got the following error sometimes.
[0:17:04.054080068] [2379] WARN V4L2 v4l2_videodevice.cpp:2007 /dev/video0[17:cap]: Dequeue timer of 1000000.00us has expired!
[0:17:04.054175105] [2379] ERROR RPI pipeline_base.cpp:1374 Camera frontend has timed out!
[0:17:04.054197512] [2379] ERROR RPI pipeline_base.cpp:1375 Please check that your camera sensor connector is attached securely.
[0:17:04.054219827] [2379] ERROR RPI pipeline_base.cpp:1376 Alternatively, try another cable and/or sensor.Then I researched the error quickly and fount a github issue.
Eventually, I’m not sure the reason but I was assuming recording audio and video at the same time could be consuming a lot CPU. So I created a script to check the CPU usage.
When I used threading and started recording, the CPU usage went to 97%. I guess this was the reason to stop recording video. When I used multprocessing and started recording, the CPU usage was around 60% and I came not to have the error.
replaced threading with mulitprocessing to use multiple cores. Afterword, I came not to have the error.
That’s it!