Driving assessment

In-Cab Surveillance Drives Innovation and Growth of ST’s Global Shutter Image Sensor

Article by: Stephen Las Marias

STMicroelectronics’ global shutter portfolio captures distortion-free images in cabin surveillance applications.

The Driver Monitoring Systems (DMS) Market is expected to reach $8.77 billion by 2032, growing at a compound annual growth rate (CAGR) of 7.5% over the forecast period 2022-2032 , according to Future Market Insights (IMF).

This growth is primarily driven by increasingly stringent government regulations regarding driver, passenger and pedestrian safety. In fact, according to the World Health Organization, road crashes kill around 1.3 million people every year, and between 20 and 50 million more people suffer non-fatal injuries, many with disabilities. due to their injury.

Human error is one of the critical factors contributing to this figure. Speeding, for example. An increase in average speed, according to the WHO report, is directly related to both the probability of an accident and the severity of the consequences of the accident. Another problem is driving under the influence of alcohol and any psychoactive substance or drug.

Distracted driving is also a factor and a growing concern in this regard. According to the WHO, drivers using mobile phones are about four times more likely to be involved in a crash than drivers not using mobile phones. It also slows down reaction times, such as braking reaction time as well as reaction to traffic lights.

According to the IMF, DMS is also essential for Level 3 and 4 autonomous vehicles, which require drivers to re-engage and be prepared to control the vehicle at vital points during the journey. IMF notes that the continued development of autonomous vehicles along with the regulations governing them are expected to create more lucrative opportunities for the DMS market.

In-Cab Surveillance Driving Innovations in Image Sensors

Besides driver monitoring, another safety feature in cars is cabin or occupancy monitoring.

Vincent Lin

“With driver monitoring, of course, we focus on the driver. The first point, which is the most important, we need to know the driver’s attention, distraction or drowsiness, because these three behaviors will affect driving,” says Vincent Lin, Technical Marketing Manager, Imaging Division, Asia-Pacific, at STMicroelectronics. “With cabin or occupancy monitoring, we focus more on passenger monitoring, especially child detection.”

This system, CMS/OMS (Cabin/Occupancy Monitoring System), is now included in the European New Car Assessment Program (Euro NCAP) ratings, which is a performance assessment program for automotive safety that helps consumers, families and businesses compare vehicles more easily and identify the safest choice for their needs.

A key element here is the onboard camera. For the DMS, which only needs to monitor the driver’s face, the lens doesn’t need to be very large, according to Lin. A field of view (FOV) of 50° or 60° is sufficient, with a resolution of 1 to 2.3Mp.

“An important point is that with driver monitoring you have to use NIR [near infrared] camera because you also have to monitor a driver while driving at night,” says Lin. “When you drive at night, in a very dark environment, you need to have a light source on your face. In this case, since you cannot use visible light, you must use NIR light.

OMS is different because the system must monitor the entire cabin. “Because we have to do object detection or recognition, we need a color image. And because the space is wider, of course, our lens has to be wider and the resolution has to be more high because we want to know all the details in this area,” says Lin.

According to Lin, the trend is to develop an all-in-one camera, which will integrate a camera for CMS/OMS and a camera for DMS. “In occupancy monitoring, we need an RGB color image sensor, while we need a NIR image sensor for driver monitoring,” Lin explains. “In this case, if you want to use a single camera to implement an all-in-one system, that camera should be RGB-NIR, with a single camera supporting both RGB color and mono NIR.”

STMicroelectronics, one of the world’s leading semiconductor suppliers, has a portfolio of high-speed image sensors using a “global shutter” to capture distortion-free images in applications such as DMS and CMS/OMS.

“For the first generation, we have four products: the 1.6 Mp VD5661A and 2.3 Mp VD5761A sensors for DMS applications; and for OMS, the VD6763A color RGB sensor and the VD1762A all-in-one sensor, which is RGB-NIR,” Lin explains.

Measuring 2.6 x 2.5 mm and 3.6 x 4.3 mm respectively, ST’s VD55G0 and VD56G3 are the smallest in the market in terms of resolution and exhibit low pixel-to-pixel crosstalk at all frame lengths. wave, especially in the near infrared, ensuring high contrast for superior image clarity.

ST’s first-generation global shutters feature a high MTF, which means very high contrast; very flexible data management; and high dynamic range (HDR). ST’s advanced pixel technology, including full deep trench isolation (DTI), enables extremely small 2.61 x 2.61 μm pixels that combine low stray light sensitivity (PLS), efficiency high quantum (QE) and low crosstalk in a single array layer—attributes that other pixel technologies cannot achieve simultaneously.

The ST approach enables a small pixel on a backside illumination array (BSI), which enables space-saving vertical stacking of the optical sensor and associated signal processing circuitry on the bottom array to achieve an extremely small sensor size and to incorporate more key features, including a fully self-contained optical flow block.

The lower die is fabricated in ST’s 40nm technology and integrates digital and analog circuitry. The high-density, low-power digital circuit incorporates hardware features, including an exposure algorithm with statistics collected from 336 zones, automatic defect correction, and automatic dark calibration. The fully self-contained low-power optical flow block can compute 2,000 motion vectors at 60 fps.

Besides DMS and OMS applications, these sensors are suitable for augmented and virtual reality (AR/VR), simultaneous localization and mapping (SLAM) and 3D scanning.

“Our second-generation product is called VB56G4A,” Lin explains. “It’s a 1.5Mp Mono, which is suitable for DMS. There are three key points for this product: firstly, its high sensitivity. doesn’t have to be very powerful. The second is the footprint. The sensor is very small, so you can build smaller camera modules which will be easier for the mechanical design. The third is the integrated processing: for example , we have smarter automatic exposure algorithms in the sensor that can help the CPU manage the exposure, so it doesn’t need to be managed by the host.”

ST’s VB56G4A global shutter sensor leverages the company’s internal investment in manufacturing advanced 3D stacked back-illuminated image sensors (BSI-3D). These are more sensitive, smaller, and more reliable than conventional front-illuminated (FSI) sensors typically used in first-generation DMSs. The sensor achieves high quantum efficiency (QE), reaching 24% at a near-infrared wavelength of 940 nm, with a linear dynamic range of up to 60 dB. This allows a simple, low-power, non-visible LED emitter to provide adequate illumination for the sensor. Operation outside the visible spectrum also ensures consistent response in day or night riding and in bright or overcast conditions.

The high QE of the sensor, combined with a pixel size of only 2.6 µm, optimizes total power consumption and camera size. Additionally, the built-in automatic exposure control provides ease of use and simplifies application software design by minimizing system interaction with the sensor.

And after

For its next-generation global shutters, Lin notes that ST will continue to improve pixel performance and focus on better power management to ensure ultra-low power consumption.

“We will also increase the number of embedded systems or processors in the sensor,” he says. “We also aim to reduce the pixel size to enable smaller camera dimensions with higher resolution. These points are our priority for our next generation of products.

While most cabin monitoring systems only have basic functions, future designs should move towards a smart cockpit concept. “Your cameras, DMS or OMS, will be linked to the safety devices of the car. Take for example the airbag. Normally you have a lot of airbags in a car these days, maybe 10 or even more”, explains Lin “If in every accident you trigger all the airbags, the cost will be very high and sometimes it’s not safe. When you have an OMS camera, you will know where the passenger is and therefore which airbag to trigger.” It’s kind of a new app.

Another future application is gesture recognition, which can help control the entertainment system or the air conditioning system. “We also see a new application very recently: payment or payment by facial identification,” says Lin. “In the future, some people may want to buy something or pay money in the entertainment system because it may have an app store. You can pay using your face ID through the DMS camera.

Etienne Las Marias is the publisher of EETimes Asia. He can be contacted at [email protected]