39. Digital camera interface pixel pipeline (DCMIPP)

39.1 DCMIPP introduction

The DCMIPP is the pixel pipeline section of a high-resolution camera subsystem: it gets pixels from a parallel or a CSI interface, and after some processing (such as decimation, cropping, downsize, color conversion, gamma correction, auto-exposure) dumps them to the memory.

DCMIPP supports multiple types of external sensors, among others:

• • dumb sensors (without internal ISP), which output raw Bayer pixels
• • smart sensors (with an internal ISP), which usually output RGB or YUV pixels
• • smart sensors with internal compression, which output a bit-stream (such as JPEG, or H264)
• • sensors with interlaced video, which output their odd and even fields sequentially

The DCMIPP input interface integrates a parallel interface (up to 16 bits at 120 MHz, with internal/external synchronization), and makes it possible to abut an external CSI-2 host (one to two data lanes, up to 2.5 Gbps/lane).

A first common part of the DCMIPP selects the input exclusively from the parallel or the CSI interfaces. Data go to dedicated pipeline(s) before they are sent to memory for further processing or display purposes.

Table 328. Available pipelines

Figure 396 shows the DCMIPP main functions, namely the integrated parallel interface, the abutted CSI-2 host, the ISP functions in Pipe1 and shareable with Pipe2, the differentiated applicative post-processing pipelines with their crop, downsize, and pixel formatting.

Figure 396. DCMIPP overview

In the ISP processing, and therefore before the post-processing pipes (Pipe1 and Pipe2), the component dynamic is reduced to 8 bits per component. If the software intends to benefit of the wider dynamic, the input data must be dumped via Pipe0, which keeps the input pixel format (10/12/14 bpp) when storing them in memory.

Table 329. Glossary

39.2 DCMIPP main features

• • Parallel input interface:
  • – Pixel rate: up to 16 bits in parallel, 100 Mpixel/s (typically 1080p30, maximum 2048x2048 on processing pipelines after decimation).
  • – Pixel format: RGB565, RGB888, YUV422, raw Bayer/Mono 8/10/12/14, and ByteStream (JPEG)
• • CSI-2 host input interface:
  • – Pixel rate: 32 bits in parallel at 200 Mpixel/s (typically 5 MPixel x30), supports MIPI® CSI-2 v1.3
  • – Pixel format: all MIPI CSI-2 v1.3, among others RGB565, RGB888, YUV422, raw Bayer (see Table 335 )
  • – Abutted CSI-2 host IP: 1-to-2 DL at up to 2.5 Gbps/lane
  • – Features: Interleaved packets, four virtual channels
• • Pipeline maximum resolution:
  • – Dimensioned for 5 Mpixel (typically 2560 x 2048 or 2688 x 1944) before decimation
  • – Maximum pipe resolution: 4094 x 4094
  • – Maximum demosaicing or downsize or YUV420 width: 2688 pixels
• • Pipe pixel rate: 200 Mpixel/s peak (150 Mpixel/s average + blanking)
• • Maximum frame per second (fps): typical examples are defined with 30 fps, it is possible to reach higher rates (for example 120 fps) by decreasing image resolution to guarantee the maximum pixel rate constraints.
• • Flow selection and frame control:
  • – CSI-2 interleaved packets supported (with some restrictions)
  • – Capture mode in Continuous vs. Snapshot mode
• • Byte-to-pixel conversion:
  • – Enables the above flexible input pixel formats to be extracted and fed to pipes
  • – Non-extracted pixel formats can still be dumped via Pipe0, to allow an ad-hoc software extraction
• • Statistic removal:
  • – To remove unused metadata (like statistics) from received pictures
• • Bad pixel removal (automatic detection and correction of bad pixels from sensor array)
• • Decimation of one pixel every 1/2/4/8:
  • – Decimation to fit the demosaicing (maximum 2688 pixel width)
  • – Raw Bayer format support, via a decimation of components (2 by 2)
• • Integrated image processing block, which makes possible to connect very low cost camera modules without any embedded ISP
  • – Black level calibration
  • – Statistics extraction
    • > To extract average R, G, B, L values (for auto exposure and white balance)
    • > To sort any R, G, B, L value into 12 dynamic bins (for contrast enhancement)
  • – Exposure control (amplifies the exposure up to 256x)
    • > To smoothly adapt to dark exposures

• – Raw Bayer to RGB conversion (demosaicing)
  • > Mid quality conversion for display and software analysis purposes (high quality conversion can be done in software upon trigger, for still-image only)
  • > Algorithm with a 4 x 3 heuristic analysis matrix
• – Color conversion to adapt to the sensor and tune the illumination
  • > Configurable coefficients, to map any linear tuning
• – Contrast enhancement
  • > Non-linear luminance tuning, linearized across eight luminance segments.
• • Decimation on Pipe0 (dump pipe)
  • – Pure sub-sampling, ratio 1 vs. 1, 2, 4 in X and ratio 1 vs. 1, 2 in Y (no X if CSI source)
• • Parallel pipelines for parallel applications:
  • – Pipe0 (for data dump) for a direct dump without processing
  • – Pipe1 (for main usage) with downsize, ColorConv, decimation, multi-planar
  • – Pipe2 (for ancillary usage) with downsize, decimation, coplanar
• • Downsize
  • – Decimation, optional pre-processing, sub-sampling ratio 1 vs. 1, 2, 4, 8 in X and Y
  • – Box-filtering, with any decimal ratio up to 8x8, on Pipe1 and Pipe2
• • Up to eight regions of interest (ROI) can be defined for displaying purpose (only supporting RGB data format)
  • – Up to 64 line colors can be selected by the software, per ROI
  • – Up to four line widths can be configurable (1, 2, 4, 8 pixels, common to all ROIs)
  • – Pipe0 does not support ROI
• • Gamma conversion for display:
  • – Non-linear transform, with seven interpolated segments, with static coefficient for \( G = 2.2 \)
• • Color conversion 2 for RGB to YUV conversion:
  • – to/from YUV: 12 coefficients (BT.601, 709, any), with optional range ext/clamp
• • Output pixel format:
  • – Pipe0: any data as-is, Y/Rb: 8/10/12/14 statistics, bit streams
  • – Pipe2: 888, 565, 422-1, Y8, ARGB and RGBA (coplanar only)
  • – Pipe1: adds 422-2, 420-2, 420-3 (multi-planar possible) to Pipe2 formats
• • AXI master:
  • – Output FIFOs to drain pixel/data to the memory (linked to the memory latency) from the pipe(s)
  • – The unique AXI-Master interface implements one client per pipe, with semi- or multi-planar pipes using, respectively, two or three clients
  • – Double buffer mode capability per pipe
  • – Address wrapping to write into a ping-pong buffer
• • Overrun detection:
  • – Notification in case of frame impacted by pixel congestion (no impact on the following frame if the congestion cause is solved)

39.3 DCMIPP functional description

39.3.1 DCMIPP block diagram

The block diagram of the DCMIPP is shown in Figure 397 .

Figure 397. DCMIPP block diagram

39.3.2 DCMIPP pads and internal signals

Table 330. DCMIPP input/output pads

Table 331. DCMIPP input/output pins

39.3.3 DCMIPP reset and clocks

Table 332. DCMIPP clocks

Clocks and pixel rate limitations

This paragraph lists the DCMIPP limitations and bottlenecks:

• • The target is to sustain an average pixel rate (PixRateAvg):
  • – A 5 Mpixel sensor at 30 fps requires 150 Mpixel/s (CSI-2 interface)
  • – Higher resolution sensors can be supported, but with lower FPS (for example 10 Mpixels at 15 fps)
• • Vertical and horizontal blanking:
  • – The full system (sensor, interface, pipe, AXI) embeds limited capacity FIFOs. The idle times that the sensor inserts during its blanking periods impact all the downstream elements, including interface, pipe, and AXI. A pixel rate (PixRatePeak) higher than average must be used. The idle time due to the blanking can reach 33% of the frame duration, sometimes more (check sensor specification).
  • – \( PixRatePeak = 1.33 * PixRateAvg = 200 \text{ Mpixel/s} \) in our example.
• • Parallel interface clock (dcmipp_pxclk):
  • – On the parallel interface, pixels are received sequentially, in one, two or three cycles. The maximum implemented clock (120 MHz) constrains the peak pixel rate, depending on the ratio of cycles per pixel.
  • – \( PixRatePeak \leq dcmipp\_pxclk / (1, 2 \text{ or } 3) = 120 / (1, 2 \text{ or } 3) \)
  • – Clock frequency:
    • > \( dcmipp\_pxclk \leq dcmipp\_ker\_ck \) (due to resync interface between them)
    • > \( dcmipp\_pxclk \leq dcmipp\_aclk \)
  • – Consequences: see Table 334 .

Table 334. Parallel interface maximum resolution (80 MHz)

• – Clock setting: \( dcmipp\_pxclk \geq PixRatePeak * Ratio(1,2,3) \)

• • Pipe clock dcmipp_ker_ck (when using the CSI-2 host interface)
  • – CSI-2 PHY provides data bit at the clk_bit rate, on DL parallel data lanes.
  • – Rationale: internally, the CSI-2 host interface packs the serial stream on 32-bit words, resynchronized on the pipe clock ( dcmipp_ker_ck ).
  • – Clock setting: \( dcmipp\_ker\_ck \geq clk\_bit * DL / 32 \text{ bpp} = 156.25 \text{ MHz} \) .
  • – Rationale 2: The pixel pipe processes 1 pixel/clk, so that dcmipp_ker_ck must be higher than the wanted PixRatePeak .
  • – Clock setting: \( dcmipp\_ker\_ck \geq PixRatePeak \) .
  • – For small footprint pixels (like Raw8), the DCMIPP is limited by its maximum dcmipp_ker_clk , and can be unable to handle the maximum CSI pixel rate ( \( clk\_bit * DL / 8 \text{ bpp} \) for Raw8).
• • Downsize:
  • – The line buffer of the vertical downsize operator is limited to a width of 2688 pixels after the horizontal downsize operation, limiting the resolution, when the downsizing is active, to 2688 x 1944 (1944 is the usual maximum height with a width of 2688), thus 5.1 Mpixel.
  • – If a camera resolution higher than 5 Mpixel is targeted, a basic decimation has to be performed to decrease the resolution to a maximum of 5 Mpixel before processing the downsizing operation.
• • AXI clock:
  • – The pixels must be reclocked from the processing to the AXI domain, resulting in
    • > \( dcmipp\_ker\_ck \leq dcmipp\_ackl \)
  • – The AXI bus must drain the running pipelines. The maximum footprint of pixel (or data) pipes is 32 bpp (Pipe1 and Pipe2) and 16 bpp (Pipe0). AXI is on 64 bits, with an efficiency of 8 / 9 for the typical 64-byte burst.
  • – \( dcmipp\_ackl \geq dcmipp\_ker\_ck * (32 \text{ bpp} + 32 \text{ bpp} + 16 \text{ bpp}) / (64 \text{ bit} / \text{ bus}) * 9 / 8 \text{ (efficiency)} = dcmipp\_ker\_ck * 1.4 \) .

39.3.4 DCMIPP maximum resolution

The limitations related to the maximum resolution supported by the DCMIPP are:

• • Maximum width and height across pipes: 4094 x 4094 pixels
  • – Registers and counters are limited to 12 bits
• • Maximum performance: 333 Mpixel/s on each pipe
  • – Larger pixel rate can be used, but with less pipe active or lower memory latency
  • – Maximum pixel rate on each pipe must remain below the pixel clock
• • Maximum width on demosaicing: 2688 components wide
  • – limitation related to the Line-buffer size
• • Maximum width after downsize: 2688 pixels
  • – limitation related to the Line-buffer size
• • Maximum width for conversion to YUV420: 2688 pixels
  • – limitation related to the Line-buffer size
• • Max line stride: 32767 bytes
  • – Registers and counters, to handle 4094 pixels x 32 bits x 2 (provision for interleaving)

39.3.5 DCMIPP minimum requirements for frame structure

DCMIPP imposes the following minimum requirements for the frame architecture:

Blanking phase

• • Dump pipe
  • – The minimal duration of the blanking between two frames must be an equivalent of 666 datawords of 32 bits
• • Pixel pipe
  • – Vertical blanking: minimum is 5 lines. If the line is less than 12 pixels, a minimum of 24 pixels equivalent for the vertical blanking parameter is requested
  • – Horizontal blanking: a minimum duration of 12 pixels is required between two lines

Data image area

• • Vertical image area: minimum one line (two lines if raw Bayer conversion is active)
• • Horizontal image area: minimum two pixels
• • The minimum total frame size is fixed to 36 pixels

39.3.6 Description of DCMIPP pixel format support

Table 335 lists the supported pixel formats in the input interfaces (parallel, CSI), as input to the pixel Pipe1 and Pipe2, and in the three output pipes.

Table 335. Supported pixel formats

Table 335. Supported pixel formats (continued)

1. Index is used to link the input pixel format to the DCMIPP_PRCR register. The index has a meaning only for camera connected with the parallel interface
2. Pixel formats on 16 bpp (like RGB565 or YUV422-1) can be received either on an 8-bit interface (using 2 cycles), or on a 16-bit input interface (using one cycle). Pixel formats on 24 bpp (like RGB888) can be received either on an 8-bit interface (using three cycles), or on a 12-bit input interface (using two cycles).
3. In Byte stream mode, the bytes are dumped consecutively with only a 32-bit padding (if needed) at the end of the frame (or JPEG stream). There is no 32-bit padding operation inside the stream until the VSYNC deassertion (end of frame, or end of JPEG stream).
4. If the data format is different from those in this table, user can choose "others" configuration into bit field FORMAT in the DCMIPP_PRCR register, using bit-field EDM to select how many data bits must be captured in one pixel clock. It is possible to capture two 8-bit data mapped on 16-bit parallel input data in a single cycle (for instance FORMAT = Others, EDM = 100, i.e. 16-bit capture on each pixel clock). Data are 32-bit padded at the end of a line to have a complete line aligned on 32-bit width.
5. On the parallel interface input, some sensor pixel formats are supported by the DCMIPP, by selecting a wider pixel format in DCMIPP, by mapping the sensor wires onto the MSB of the DCMIPP interface. On the input missing pins, it is recommended to replicate the sensor MSB pins onto the missing input LSB pins. This helps achieving full dynamic on the extended pixel format. The work-around can be applied on the following formats (sensor output on left, mapping on DCMIPP format on right):
   • - Raw6: Raw8 with 1-cycle input, processed as a 8 bpp
   • - Raw7: Raw8 with 1-cycle input, processed as a 8 bpp
   • - RGB444: RGB565 with 1-cycle input, processed as a 16 bpp
   • - RGB555: RGB565 with 1-cycle input, processed as a 16 bpp
   • - RGB666: RGB888 with 3-cycle input, processed as a 24 bpp
6. The same bit mapping is used for the input RGB888 and YUV444, the input xRGB and xYUV, and similarly for the outputs. The difference between RGB and YUV is the color reference, set in the color conversion that occurs after the input or before the output.

39.4 DCMIPP input and flow control

This section describes the functional elements of the DCMIPP. For each, it details the features, the software configuration, and provides a software configuration example (when needed).

39.4.1 DCMIPP common configuration

The DCMIPP common configuration register (DCMIPP_CMCR) enables the selection of the input interface—either parallel DCMI or serial CSI-2—from which to fetch the input pixels. It also allows swapping the R/B color components of the incoming pixels and selecting the pipe used by the frame counter.

Table 336. DCMIPP_CMCR bit function

39.4.2 Parallel input interface

The parallel interface is a flexible pixel interface that clocks in a bus of 8- to 16-bit in parallel, with an externally provided clock, and adds vertical and horizontal synchronization, provided either with specific pins (external sync) or flagged via specific data (as in CCIR601).

Interface (all inputs)

• • Clk: clock up to 120 MHz, internally named dcmipp_pxclk
• • Data: 16 IO pins, up to 120 Mbps/pin, 8- to 16-bit, pixels potentially serialized
• • HSync, VSync, but no enable (EN)

Features

• • Horizontal and vertical synchronization:
  • – External, via the HSync and VSync pins.
  • – Internal, via the CCIR601 flagging.

  Data provided during the horizontal or vertical blanking cannot be captured.

• • Input pixel format (see Section 39.3.6 )
  • – Binary stream like JPG
  • – Single cycle data:
    • > Raw Bayer and monochrome: 8/10/12/14 bits in parallel
    • > YUV422, RGB565:16 bpp on 16 bits in parallel
  • – Double-cycle data:
    • > YUV422, RGB565:16 bpp on 8 bits in parallel
    • > RGB888/YUV444: 24 bpp on 12 bits in parallel
  • – Triple-cycle data:
    • > RGB888/YUV444: 24 bpp on 8 bits in parallel
• • Flexible input:
  • – MSB-vs-LSB bits can be swapped to ease PCB placement
  • – Cycle0-vs-Cycle1 can be swapped (for 2-cycle long pixels).
  • – Red-vs-blue components can be swapped (for RGB, or U-vs-V for YUV), this last swap operates only on the pixel pipes and not the dump pipe.

Hardware synchronization mode

In this mode two synchronization signals (DCMIPP_HSYNC and DCMIPP_VSYNC) are used.

Depending on the camera module/mode, data can be transmitted during horizontal/vertical synchronization periods. DCMIPP_HSYNC/DCMIPP_VSYNC act as blanking signals, as all data received during DCMIPP_HSYNC/DCMIPP_VSYNC active periods are ignored.

To correctly transfer images in the RAM buffer, data transfer is synchronized with the DCMIPP_HSYNC/DCMIPP_VSYNC signals. When the hardware synchronization mode is selected, and capture is enabled (CPTREQ bit set in DCMIPP_PxFCTCR), data transfer is synchronized with the deassertion of the DCMIPP_VSYNC signal (next start of frame).

Transfer can then be continuous, with successive frames transferred by the IP-Plug to successive buffers or the same/circular buffer.

Embedded data synchronization mode

In this synchronization mode, the data flow is synchronized using embedded 32-bit codes, using the 0x00/0xFF values not used in data anymore. There are four types of codes, all with 0xFF00 00XY format. The embedded synchronization codes are supported only in 8-bit parallel data capture (in the DCMIPP_PRCR register, the EDM[2:0] bits must be programmed to “000”). For other data widths, this mode generates unpredictable results.

Note: Camera modules generate up to eight synchronization codes when in interleaved mode, while the DCMIPP reacts to one single code. As a consequence, an interleaved flow with an embedded synchronization has one every other frame discarded as not detected.

Mode 2

Four embedded codes signal the following events:

• • Frame start (FS)
• • Frame end (FE)
• • Line start (LS)
• • Line end (LE)

The XY values in the 0xFF00 00XY format of these codes are programmable (see Section 39.14.9 ).

A 0xFF value programmed as a frame end means that all the unused codes are interpreted as valid frame end codes.

In this mode, once the camera interface has been enabled, the frame capture starts after the first occurrence of the frame end (FE) code followed by a frame start (FS) code.

Mode 1

An alternative coding is the camera mode 1. This mode is ITU656 compatible.

The codes signal another set of events:

• • SAV (active line) - Line start
• • EAV (active line) - Line end
• • SAV (blanking) - Start of line during interframe blanking period
• • EAV (blanking) - End of line during interframe blanking period

This mode can be supported by programming the following codes:

• • \( FS \leq 0xFF \)
• • \( FE \leq 0xFF \)
• • \( LS \leq SAV \) (active)
• • \( LE \leq EAV \) (active)

An embedded unmask code is also implemented for frame/line start and frame/line end codes. Using it, it is possible to compare only the selected unmasked bits with the programmed code. User can therefore select a bit to compare in the embedded code and detect a frame/line start or frame/line end. This means that there can be different codes for the frame/line start and frame/line end with the unmasked bit position remaining the same.

Example:

• • \( FS = 0xA5 \)
• • Unmask code for \( FS = 0x10 \)

In this case the frame start code is embedded in bit 4 of the frame start code.

Note: FEC sequence must be sent before the transfer of the first frame, otherwise other codes are not decoded and the first frame can be lost after the DCMIPP has been enabled. After the first FE sequence, the following frame is captured based on FS sequence.

Error conditions

Error conditions can be detected when using the embedding synchronization data modes. Flags PRERRF in DCMIPP_CMSR2 and ERRF in DCMIPP_PSR are used for this function. An interruption can be triggered based on error detection if bit PRERRIE in

DCMIPP_CMIER is set for the global interrupt (or bit ERRRIE in DCMIPP_PRIER for a local interrupt line).

For instance, such kind of wrong sequences or values generate an error:

• – FF 00 00 XY sequence detected with XY value different from the one set for FS/LS/FE/LE
• – LS = 0xFF or LE = 0xFF
• – If an LS or FS sequence follows an LS sequence

Software configuration

Table 337. DCMIPP_PRCR bit function

The embedded synchronization codes are configured by means of DCMIPP_PRESR and DCMIPP_PRESUR registers (see Table 338 ).

Table 338. DCMIPP_PRESR and DCMIPP_PRESUR bit function

Table 338. DCMIPP_PRESER and DCMIPP_PRESUR bit function (continued)

39.4.3 Interface from CSI-2 host

The CSI-2 host and DPHY_RX (both external to the DCMIPP) provide the data from the CSI-2 camera sensor, forward them into the DCMIPP using the ISB byte header interface.

The ISB byte header features are:

• • CSI-2 clock interface aligned with the DCMIPP processing clock (333 MHz)
• • Data width: 32 bit (data and CSI headers are interleaved)
  • – Frame synchronization short packet
  • – Frame number, Data ID, and packet type are provided (see Section 39.7.9 ).
• • Line Sync short packets are not received.
• • Headers of data long packet:
  • – Packet length, Data ID, and packet type are provided (see Section 39.7.9 ).
• • Payloads of data long packet:
  • – Data as 32-bit words, as collected by the CSI-2 host.
• • Packet CRC is not received.

39.4.4 Input selection

Input selection block is ahead of any pipe to select the camera sensor module interface connected to the DCMIPP. Two interfaces are supported:

• • Parallel interface: up to 16-bit parallel interface with physical synchronization signals (DCMIPP_VSYNC/DCMIPP_HSYNC) or software embedded ones (as in CCIR601).
• • MIPI CSI-2 interface: Host ISB byte header internal connection from the CSI2 host controller placed in front of the DCMIPP.

Both interfaces are exclusive. The selected one is applied to all the input pipes.

The camera sensor interface is selected by means of bit INSEL of the DCMIPP_CMCR register.

39.4.5 Flow selection

The module, present in each pipe, handles the flow control around the CSI2 camera sensor interface. For each pipe, the software needs to configure which virtual channel (VC) and which Datatypes (DT) are processed by the pipe.

For Pipe0 and Pipe1, multiple Datatypes can be selected whereas only one virtual channel can be defined for a frame. Configuration can be changed frame by frame if the application requires it, offering some flexibility.

For Pipe2, when the pipe is independent with PIPEDIFF = 1, only one virtual channel and only one data type by frame can be selected.

The software has to reconfigure the flow selection module to handle interleaved frames from a sensor if it uses virtual channels as a mean to make the differentiation, accepting to lose frames since there can be only one virtual channel active for a given frame (it decreases the FPS).

Multiple cameras data flow can be received simultaneously through the CSI2 ISB header bus. For instance, for a system having two different camera data flows, it is possible to redirect one virtual channel on a pipe and the other one on the other pipe. It can be interesting to dump them on Pipe0, but only one frame over two since the flow selection block has to be reconfigured to switch from one VC to another one.

The flow selection module is used also to share some image processing features (demosaicing, color conversion, statistics, contrast enhancement, auto exposure, statistics removal and basic decimation) between Pipe1 and Pipe2, whatever the camera sensor interface solution (parallel interface or CSI2 interface): when bit PIPEDIFF in the DCMIPP_P1FSCR register is 0, or the input is on the parallel interface that provides unique flow, the Pipe2 samples pixels from the output of the ISP of Pipe1 (see Figure 396 ). PIPEDIFF must be reconfigured when Pipe1 and Pipe2 are disabled.

Software configuration

The following configurations must be set first, because static in the DCMIPP_PxFSCR registers:

Pipe0

• • DTMODE[1:0]: Mode for CSI flow selection into that pipe
  • – 0: only flow VC/DTIDA is forwarded in the pipe
  • – 1: flows VC/DTIDA and/or VC/DTIDB are forwarded in the pipe
  • – 2: all Datatypes of the selected virtual channel, except the VC/DTIDA or VC/DTIDB, are forwarded in the pipe
  • – 3: all Datatypes of the selected virtual channel are forwarded in the pipe
• • VC[1:0]: Virtual channel ID of the CSI flow to select for further processing on the pipe
• • DTIDA[5:0]: Data type ID A to select in the CSI flow for further processing in the pipe
• • DTIDB[5:0]: Data type ID B to select in the CSI flow for further processing in the pipe
• • PIPEN: activation of the Pipes, to be set after the above configurations.

Pipe1

• • DTMODE[1:0]: mode for CSI flow selection into that pipe
  • – 0: only flow VC/DTIDA is forwarded in the pipe
  • – 1: flows VC/DTIDA and VC/DTIDB are forwarded in the pipe
• • VC[1:0]: virtual channel ID of the CSI flow to select for further processing on the pipe
• • DTIDA[5:0]: data type ID A to select in the CSI flow for further processing in the pipe
• • DTIDB[5:0]: data type ID B to select in the CSI flow for further processing in the pipe
• • FDTFEN: when the DTIDX is detected depending on DTMODE, the software forces the data type format to the value set into FDTF[5:0] for the pixel pipe processing. It can be used, for instance, to address sensor 3D camera through specific bridge using CSI2

“user data type” to make distinction between left and right camera sensor (each camera has its own user data type value, but sharing the same pixel format).

• • FDTF[5:0]: data type value to be forced
• • PIPEDIFF: Pipe2 to take same input as Pipe1 or differentiated:
  • – 0: Pipe1 and Pipe2 take the flow selection configuration of Pipe1. Pipe1 shares the image processing functions with Pipe2.
  • – 1: Pipe1 and Pipe2 are considered as two independent pipes, without any shared configuration.
• • PIPEN: activation of the pipes, to be set after the above configurations.

Pipe2

• • VC[1:0]: Virtual channel ID of the CSI flow to select for further processing on the pipe.
• • DTIDA[5:0]: Data type ID A to select in the CSI flow for further processing in the pipe
• • FDTFEN: When the DTIDX is detected depending on DTMODE, the software forces the data type format to the value set into FDTF[5:0] for the pixel pipe processing. It can be used to address sensor 3D camera through specific bridge using CSI2 “user data type” to make distinction between left and right camera sensor for instance (each camera having its own user data type value, but sharing the same pixel format).
• • FDTF[5:0]: Data type value to be forced
• • PIPEN: activation of the pipes, to be set after the above configurations.

If PIPEDIFF = 0 in DCMIPP_P1FSCR, the configuration bits are directly those set in the DCMIPP_P1FSCR register.

The above bit fields are meaningful when PIPEDIFF = 1.

39.4.6 Frame counter

A frame counter is available for a tag purpose and counts all the frame received on the selected pipe.

The counter is 32-bit, read-only. It is active at least when bit PIPEN of the pipe to whom it is connected is enabled (PIPEN = 1).

It provides an (almost) unique frame number, with a loop time of 4.5 years (if 30 fps).

The counter can be incremented, depending on the PSFC in DCMIPP_CMCR, at the FrameStart event of the selected virtual channel, when a CSI-2 camera sensor is connected, or at the FrameStart event of the pipe when the camera sensor module is connected with parallel interface.

The frame counter is cleared by setting CFC in the DCMIPP_CMCR register.

Software configuration

The frame counter must be configured before it is used for the pipe. Software action(s) must be performed in the DCMIPP_CMCR register:

• – Select on which pipe the frame counter is connected by configuring bit PSFC[1:0]
• – Frame counter runs as soon as the selected pipe is active (PIPEN = 1)

Note: It is possible to have the Frame counter run on a pipe, while having the pipe not flowing any pixel: this is achieved with pipe enable active (PIPEN = 1) to have the frame counter run, but with CPTREQ = 0 to avoid any pixel flow.

39.4.7 Frame control

The module is replicated for each dump and pixel pipe. It handles the frame control and the capture of each pipe, grouping data into frames, and capturing them when requested.

Definitions

There are two notions of frame events

• • Event used inside the hardware (provided by the input interfaces):
  • – FrameStart: when a frame starts at the input of the pipes (at frame control)
    • > Parallel interface with VSync fall
    • > CSI-2 interface: SoF (Start-of-Frame) packet
  • – FrameEnd: when a frame finishes at the input of pipes (at frame control)
    • > Parallel interface: VSync rise (End-of-Frame)
    • > CSI-2 interface: EoF (End-of-Frame) packet
• • Events exposed to software via DCMIPP events and interrupts:
  • – VSYNC
    • > When a frame starts, observed at the input of the pipes (at frame control)
    • > It indicates that the current frame has started and has sampled its shadow registers, and that the software can start to configure the next frame
    • > It is recommended to use that interrupt to trigger the software that reconfigures the DCMIPP.
  • – FRAME END
    • > When a frame finishes, observed at the output of pipes (when the last pixel has been written on the AXI)
    • > It indicates that the current frame has finished writing all its data to memory, and that the application can immediately start using them
    • > It is recommended to use that interrupt to trigger the software that forwards the memory buffer to other applications

Features

• • Frame delineation: allows to dump data with the required frame granularity.
  • – Extracts, per pipe, the FrameStart and FrameEnd event, based on the input Start-of-Frame and End-of-Frame.
    • > The FrameStart immediately generates a VSYNC event.
    • > The FrameEnd ripples till the end of the pipe, at the pixel packer level, where it generates the FRAME event.
  • – Based on the algorithm detailed in Extraction algorithm for FrameStart and FrameEnd .
• • Capture mode: can capture a frame snapshot, or capture continuously.
• • Capture rate: can capture all frames, or one out of 2, 4, 8. It is only active when the Capture mode is in Continuous mode.
• • Frame status: stores infos about the last captured frame (PxLSTFRM, PxLSTLINE), re-sampled at the FrameEnd event.

Extraction algorithm for FrameStart and FrameEnd

The extraction of the FrameStart and FrameEnd is straightforward.

In CSI-2 mode, each pipe is handling a unique virtual channel ID by frame. In case of interleaved flow in input of the DCMIPP, the software can decide to:

• • Dedicate each virtual channel on a single pipe in case of multiple pipes, each of them having its own virtual channel number selected
• • Decide to decrease the FPS ratio by changing the pipe configuration swapping with the other virtual channel ID (mainly in case of single pipe)

The interlaced mode can be handled by the software in CSI-2 mode by changing the pipe configuration frame by frame.

The following usecases are supported and detailed in the figures below:

• • Default single VC: straightforward behavior, support in both pixel and dump pipes ( Figure 398 )
• • Overlapping VC (supported on a single pipe via the loss of one frame when switching)
  • – It can be supported by affecting each VC on a dedicated pipe ( Figure 400 )
• • Exclusive VC (supported in both pixel and dump pipes), see Figure 399 :

The use cases with two exclusive VCs can be handled either using one single pipe, by reconfiguring it from frame to frame, or by splitting the flow on more than one pipe, each of them handling its own VC:

• • each flow arrives exclusively in one pipe and is processed as an independent frame, with its Start-of-Frame and End-of-Frame
• • FrameStart and FrameEnd are generated per VC flow.

Figure 398. VC flow processed by one pipe

1. 1. If frame counter is attached to the pipe and if reset is not requested when changing VC.

Use cases

The Continuous mode is typically used to send to display or to software-analysis a continuous stream of frames.

The Snapshot mode is typically used to dump a high-resolution frame.

The Frame rate mode is typically used for a low-power continuous analysis, where only one frame every eight is dumped and analyzed.

The frame informations are typically used for interlaced video use cases, where the sensors transmit alternatively odd and even fields (top vs. bottom), and where software must be able to retrieve the type of field. Depending on the sensor, it can be retrieved from:

• • the VC-ID, if the sensor uses one VC for one field and another for the other
• • the LSB of the counted number of lines in that frame (indicates if the number is odd or even)
• • the LSB of the frame counter extracted from the CSI information (if counting this pipe), which can be odd or even.

Software configuration

The frame control supports two different capture modes:

• • Snapshot mode: the software captures single frames
• • Continuous mode: the software captures a continuous sequence.

The timing diagram of these two modes is shown in Figure 401 .

Figure 401. Snapshot (CPTMODE = 1) and Continuous (CPTMODE = 0) capture modes

Table 339. DCMIPP_PxFCTCR bit function

1. 1. Bit read from the DCMIPP_PxSR register.

Configuration example

The software operations when using the Snapshot mode are the following:

1. 1. Software sets CPTMODE = 1, to use the Snapshot mode.
2. 2. Software sets PIPEN = 1, to let the flow selection send data to PipeN.
3. 3. Software sets CPTREQ = 1, to request the capture of one frame.
4. 4. At the first following VSync, the HW samples CPTREQ at 1, with the following impact:
   1. a) The capture effectively starts: pixels flow into the pipe and are dumped in memory.
   2. b) CPTACT is set to 1, to mention that a capture is currently ongoing, and that it is best to not modify the configuration of the pipe operators, unless shadowed.
   3. c) CPTREQ is reset to 0, so that only a single frame is dumped.
5. 5. At the following capture complete interrupt
   1. a) CPTACT is reset to 0, to signal that the capture is over.
   2. b) The capture is complete, no more pixels are flowing, hence software can restart to update the configuration of non-shadowed registers without any issue, and use the captured pixels in memory. Depending upon the sensor blanking, this period can be short.
6. 6. At the next VSync, CPTREQ usually is sampled at 0, so that no more frames are captured
   • – As soon as CPTREQ is reset to 0 the software can set it again to 1, to request a capture at the following frame. It means that a continuous sequence of frame can be captured by setting again CPTREQ to 1, continuously frame after frame.

The software operations when using the Continuous mode are the following:

1. 1. Software sets CPTMODE = 0, to use the Continuous mode.
2. 2. Software sets PIPEN = 1, to let the flow selection send data to PipeN.
3. 3. Software sets CPTREQ = 1, to request the continuous capture of frames.
4. 4. At the first following VSync, CPTREQ is sampled at 1, with the following impact:
   • – As in Snapshot mode, the capture starts, and CPTACT is set at 1.
   • – Unlike the Snapshot mode, CPTREQ is not modified and remains at 1.
5. 5. At the following capture complete interrupt:
   • – As in Snapshot mode, CPTACT is reset to 0, capture stops, software can shortly reconfigure the pipes, and software can use the captured pixels.
6. 6. At the next VSync, and as long as CPTREQ remains at 1, the capture restarts, similarly as described above.
7. 7. To stop capturing later frames, software resets CPTREQ to 0.
   • – An ongoing capture continues until completion.
8. 8. At the next VSync, CPTREQ is sampled at 0, and the next capture does not restart.

39.4.8 Pipe deactivation

It is possible to abort a pipe to stop any frame acquisition and to potentially offer a mean to reprogram it completely (including non-shadowed registers), or to let it fully disabled for some time. The following operating mode has to be considered to correctly stop the pipe:

1. 1. Disable the CPTREQ bit of the DCMIPP_PxFCTCR register of the corresponding pipe
2. 2. Poll CPTACT = 0 status bit in the DCMIPP_CMSR1 register to check if the pipe is no longer active (idle state)
3. 3. Disable PIPEN in the DCMIPP_PxFCTCR register.

When disabling the pipe, the last frame is processed completely by respecting the above operating mode.

Note: To disable the parallel interface (PREN = 0 in the DCMIPP_PxFSCR), it is recommended to first disable the pipes considering the above operation mode, before switching off the parallel interface.

39.5 Pipe0 (dump pipe)

39.5.1 Overview

Pipe0 works as a dump pipe: it extracts data from the camera sensor module and dumps them (as-is) to the targeted memory, with some basic decimation and cropping 2D operations in between.

Figure 402. Pipe0 (dump) architecture overview

Pipe0 retrieves data from the camera sensor module connected to the DCMIPP through the 16-bit parallel interface or the CSI-2 interface, by means of the CSI-2 host controller peripheral. The selection between interfaces is managed by means of bit INSEL in the DCMIPP_CMCR register.

The flow selection chooses the virtual channels and data types to be processed by the pipe from the CSI2 host controller, see Section 39.4.5 for more details.

The frame controller handles mainly the camera acquisition mode (continuous or snapshot, frame rate), see Section 39.4.7 for more details.

When the input data from the camera is valid and supposed to be processed by Pipe0, 2D-cropping and decimation operations can be configured by the software. A dump counter combining with a limit amount of data to be set is offered to handle unknown length of data or to avoid the amount of data to be too wide within a frame.

Pipe0 can be reconfigured from time to time. Some events can be used to trigger the software reconfiguration routines (refer to Section 39.13.2: Interrupts ).

It is also possible to write into the shadow registers within the processing of a frame. The values are loaded into the physical register based on the start of frame event.

39.5.2 Decimation

The decimation allows to cheaply downsize a frame.

Based on parallel interface camera module:

• – Factor 1x, 2x, 4x at data/byte level depending on the configuration of bit field BSM[1:0] of the DCMIPP_P0PPCR register. The decimation is not working directly at pixel level, but it is based on either byte or 16-bit word granularity (for padded raw 10, for instance) to make equivalent decimation to one pixel out of two. Bit OEBS is used to start from the odd or even byte (or data) to capture first.
• – The decimation can also be based on entire line, by capturing all the lines, or one line out of two thanks to bit LSM. Here, again, choice is given to the software to reject odd or even lines within the frame by configuring bit OELS.

When the camera sensor module is designed on CSI-2 interface, the decimation can be based filter out potentially, one line out of two. Indeed, each packet is by protocol composed for an entire line.

Note: Care must be taken when the CSI-2 header is dumped, as it is 32-bit wide, and it increases by one unit the data counter, taking part of the decimation. It is advised not to use the CSI header option when decimation takes place.

Table 340. DCMIPP_P0PPCR bit function

39.5.3 Crop/statistics selection/suppression

The dump pipe (Pipe0) has the capability to handle 2-D crop processing. The vertical cropping is based on line, thanks to HSYNC event (from the physical IOs or the embedded code detection if selected). It can be extracted from the CSI-2 packet information (each packet is one line wide by protocol).

The horizontal area on which the crop is applied is based on data (32-bit wide), and not on pixels since this pipe is not directly considering the pixels. It handles the data flow with 32-bit granularity. The CROP functionality is not supported when the pipe is conveying JPEG format. The ENABLE bit must be kept cleared into DCMIPP_P0SCSZR to avoid any unpredictable behavior.

The area to be captured within the frame has to be specified configuring the registers DCMIPP_P0SCSTR and DCMIPP_P0SCSZR. The starting point on the two axis are set accordingly as well as the width in both directions. It is possible to take the data inside or outside this area by means of bit POSNEG in the DCMIPP_P0SCSZR register. Some sensors can send sensor configuration data and statistics (histogram) on the very first and last lines of a frame. By specifying active the area outside of this window selection (POSNEG = 1), only statistic data are extracted by the way, and the software can decide to apply some processing on the data to correct, such as contrast or exposure.

Features

• • 2D-Crop operation with granularity:
  • – Lines in the vertical axis
  • – 32-bit data in the horizontal axis
• • Possibility to capture data inside or outside the defined window (valid image data or statistic data)
• • Cropping with 0 value for HSIZE[11:0] or VSIZE[11:0] has no crop effect to the dimension for which the 0 value has been applied (vertical or horizontal axis).

Software configuration

Registers DCMIPP_P0SCSTR and DCMIPP_P0SCSZR have to be configured to set the crop feature into the dump pipe:

1. 1. Configure the cropped horizontal starting point with HSTART[11:0] and the width with HSIZE[11:0] (both with a 32-bit data granularity).
2. 2. Configure the cropped lines, starting at line VSTART[11:0] and with a height of VSIZE[11:0] (both with a line granularity)
3. 3. If any value HSIZE or VSIZE is set to 0, the hardware does not consider crop operation in the vertical or horizontal dimension for which the 0 value has been applied, so that all the pixels in that dimension are sampled.
4. 4. Select the inner or the outer part of the window for the data capture by configuring POSNEG bit.
5. 5. Enable the crop feature setting ENABLE bit. This bit must be cleared when JPEG is selected as the input format for Pipe0.

Note: Cropping out the picture size (with too large HSTART or VSTART) leads to a not guaranteed processed frame.

39.5.4 Header insertion

The module works on the dump pipe(s) and it optionally inserts an header before the data of a former CSI packet.

Features

• • Optionally inserts non-data information to help software interpreting the following data payload. The non-data info is Frame sync short packets and Header of data long packets (both non-data info into a 32-bit word), mapped as follows:
  • – Frame sync short packet: defined by Frame type = SOF or EOF
    • > bits 31:16: Frame number
    • > bits 15:8: Reserved (forced to 0)
    • > bits 7:4: Virtual channel ID in CSI-2_v1.3, bits 7:6 = 00
    • > bits 3:0: Frame type (SOF = 0x1 or EOF = 0x2)
  • – Header of data long packet (defined by below Frame type = DATA)
    • > bits 31:16: Amount of bytes in the payload
    • > bits 15:8: Data type ID in CSI-2_v1.3, bits 15:14 = 00
    • > bits 7:4: Virtual channel ID in CSI-2_v1.3, bits 7:6 = 00
    • > bits 3:0: Frame type (DATA = 0x0)

To keep consistency between the amount of bytes provided by the header versus those effectively dumped, it is recommended to not activate the header insertion simultaneously with the crop or decimation, as these functions modify the amount of dumped data.

When header insertion is active, the FrameEnd packet is dumped as a header, and is considered as the unique part of an additional line, inducing an event.

Software configuration

It is configured with the DCMIPP_P0PPCR register:

• • HEADEREN:
  • – 0: No header insertion
  • – 1: Insertion of the headers, aligned on 32 bits.

39.5.5 Dump counter

The dump counter is present on dump pipe. It is used to count the amount of data that are dumped in that frame, and to potentially limit the amount dumped if too large. It allows the software to know the size of dumped buffer, and to make sure that no dump is made out of a preallocated buffer.

It is specifically useful when dumping a content whose length is unknown prior to reception, like an encoded JPG stream. When dumping a pixel frame, the size is known thanks to the configured width and height of the frame.

The counter is counting 32-bit data for almost all the input formats, even if the value is expressed in number of bytes. The counter increment is 4-bytes granularity for all formats except the JPEG byte stream input mode, for which the counter granularity is 8-bit because the application does not know the amount of dumped data.

Features

• • Dump counter, counting the amount of bytes dumped. It counts the amount of dumped 32-bit words (the increment is 4), except when connected to a parallel interface camera and dumping a 8-bit data in byte stream mode (see FORMAT bit field in the DCMIPP_PRCR register), where it counts per byte (the increment is 1).
  • – The counter saturates at 0x03FF_FFFF.
  • – At FrameEnd, the counter value is recopied in a readable shadow register and the counter itself is reset to start the count of the next frame.
• • Dump limit, that clamps the dumped words below a maximum limit:
  • – If the counter reaches the limit, all following bytes are deleted until the next VSync.
  • – The counter continues to count over the limit, to indicate how many bytes have been received and must have been dumped if not deleted.
• • Interrupt limit is triggered when some bytes are deleted due to dump limit crossed.
• • CSI headers, when dumped (HEADEREN = 1), participate to the counted and limited data volumes.
• • Monochrome or raw Bayer padding (alignment of 10/12/14 bpp to 16 bpp) is also taken into account in data volumes.

Software configuration

Table 341. DCMIPP_P0DCCNTR and DCMIPP_P0DCLMTR bit function

Note: The dump counter sees the CSI-2 packet headers as normal 32-bit words. Raw Bayer and monochrome pixels on 10/12/14 bpc are padded onto 16 bits, and are thus slightly larger than their 10/12/14 bit size.

39.5.6 Double buffer mode

The dump pipe uses an AXI master interface to dump the data from the internal FIFO to the external memory. In the application, it is possible to handle frame data swapping memory area frame by frame. The double buffer mode fills up this function. There are two memory address registers set to initialize the base addresses of these memories areas. Each start of captured frame event swaps the memory base address to handle double buffering mode.

The double buffer mode can be used to allow post-processing on a buffer (frame buffer) while the other buffer is read to be displayed (display buffer).

Software configuration

Double buffering mode requires an activation, as well as addresses configuration, to define the two memory areas in which data are consecutively stored frame by frame at the output of the pipeline:

• • DCMIPP_PxPPM0AR1 and DCMIPP_PxPPM0AR2 must be filled with the memory addresses associated to the double buffer memory locations.
• • Bit DBM in the DCMIPP_PxPPCR register(s) is set to enable the double buffer mode.

39.6 Pipe1 (ISP part)

39.6.1 Overview

Pipe1 is a pixel pipeline in which some image processing (ISP) and post-processing functions are integrated, to relieve the software from very specific (time and memory consuming) operations. The ISP part implements functions like:

• • Statistics removal within the dataflow
• • Bad pixel detection to automatically remove noisy pixels
• • Basic decimation to reduce high resolution camera sensor to fit with the image processing maximum resolution tolerance
• • Auto exposure and statistics extraction based on pixel analysis
• • Demosaicing algorithm to convert raw Bayer input pixels to RGB ones
• • Color conversion and contrast enhancement functions

Pipe1 ISP block diagram is shown in Figure 403 .

Figure 403. Pipe1 ISP architecture view

    graph LR
    IN[From INPUT] --> PS[Pipe Sel]
    subgraph PS [Pipe Sel]
    FS[Flow select
(CSI-2, VC, DT)]
    end
    PS --> IP
    subgraph IP [ISP Pipe]
    SR[Stats removal
(first/last line)] --> BPR[Bad pixel removal]
    BPR --> D[Decimation
(ratio 1,2,4,8)]
    D --> BL[Black level
(subtracts per RGB)]
    BL --> E[Exposure
(multi and shift per RGB)]
    E --> DEM[Demosaicing
(Raw Bayer-to-RGB)]
    DEM --> CC[Color conv
(configuration coefficients)]
    CC --> C[Contrast
(LUT in luminance)]
    SE[Statistics extractor
(3 accumulators)]
    BL --> SE
    E --> SE
    DEM --> SE
    end
    IP --> PC
    subgraph PC [Pipe Ctrl]
    FC[Frame control
(capture, rate)]
    end
    PC --> OUT[To post-processing]
  

Pipe1 is the most featured pixel pipeline, even if the image processing functions can be shared with Pipe2 (refer to Section 39.8.2 ).

Pipe1 retrieves data from the camera sensor module connected to the DCMIPP with the 16-bit parallel interface or with the CSI-2 interface, thanks to the external CSI-2 host controller peripheral. The selection between interfaces is managed by means of bit INSEL in the DCMIPP_CMCR register.

The flow selection chooses the virtual channels and data types to be processed by the pipe from the CSI2 host controller. Refer to Section 39.4.5: Flow selection for more details.

The frame controller handles mainly the camera acquisition mode (continuous or snapshot, the frame rate). Refer to Section 39.4.7: Frame control .

Pipe1 can be reconfigured from time to time. Some events can be used to trigger the software reconfiguration routines (refer to Section 39.13.2: Interrupts ).

It is also possible to write into the shadow registers during the processing of a frame. The values are loaded into the physical register based on the following start of frame event.

39.6.2 Byte-to-pixel conversion

The module works on the pixel pipes: it gets 32-bit words from the input (parallel or CSI) interfaces, extracts the pixels depending on the provided pixel format, and maps them onto

the triple-component pixel pipeline, that is 14 bpc at this stage (and 8 bpc later downstream).

Functionality

• • Extracts various input pixel formats to a triple R, G, B lane
  • – All CSI-2 formats of versions CSI v1.3 and v2.0 are supported, except Raw 6/7 bpc, YUV422-10 bpc, YUV420 (see Table 335 )
• • YUV420-to-YUV444 conversion is not supported
  • – Pipe1 and Pipe2 cannot extract and process YUV420 pixels received from the CSI interface
• • YUV422-to-YUV444 conversion:
  • – Done by replication of the U and V components. It thus assumes that the location of the common chroma is defined at mid-distance between the two pixels.
• • Extension of short triple component (RGB/YUV) to the output triple 8 bpc:
  • – The extension is done by replicating the input MSB onto the missing output LSB. The replication onto the LSB allows to map any input dynamic on the full downstream dynamic.
  • – Examples of 5 and 6 bpc extended to the output 8 bpc:
    • > Input Di[4:0]: Output Do = Di[4:0] and Di[4:2]
    • > Input Di[5:0]: Output Do = Di[5:0] and Di[5:4]
• • Keeps as-is the single-components, usually wide:
  • – The monochrome and raw Bayer 10/12/14 bpc are fed into the pipes as-is (the monochrome assumes a Y, with U = V = 128).

Software configuration

The used pixel format depends upon the source of the pixels:

• • CSI-2 interface: the pixel format is embedded in the received CSI-2 packets, and defined in the data type field of the packet header.
  • – When FDTFEN = 0, the byte-to-pixel conversion uses the data type provided by the CSI-2 packet, and works automatically, without need for any configuration.
  • – When FDTFEN = 1, the data type is configured by software in the FDTF field.
• • Parallel interface: the pixel format is determined by the source of the pixels, i.e. the camera sensor. The byte-to-pixel extracts the pixels based on the pixel format provided by the software in the parallel interface configuration, in the field of the DCMIPP_PRCR register.

39.6.3 Statistics removal

The module is used to removing potential statistic data inserted by a sensor, prior to starting the demosaicing conversion that is multi-pixel and would be impacted by remaining statistics. It is typically used by sensors with a parallel interface, which does not have any way to distinguish between statistics and pixel data. CSI-2 sensors typically split these flows using different virtual channel IDs.

Features

• • Suppression of some of the first lines of the frame.
• • Suppression of the last lines of the frame, after a configured line N.

Software configuration

Table 342. DCMIPP_P1SRCR bit function

Configuration example

If the received frame contains four lines of statistics, followed by 100 lines of pixels and eight lines of ancillary data, the configuration to extract the 100 lines of pixels is the following:

• • CROPEN = 1, to have the removal active
• • FIRSTLINEDEL = 4, to remove the first four lines
• • LASTLINE = 100, to keep the next 100 lines

39.6.4 Bad pixel removal

This module aims at detecting and correcting artifacts generated by a bad pixel on the sensor array. It is based on a correlation extraction, and not on fixed locations.

Functionality

• • Works for raw Bayer flows only, as RGB flows are assumed as already processed.
• • Scans all the arriving components, and extracts the horizontal continuity correlation versus the two left and two right neighbors, on both the local component and on the adjacent raw Bayer component:
  • – if a component is seen as too light, it is declared as bad
  • – too dark components are not extracted, to avoid false positives
• • A component assumed bad is replaced by the horizontal average of its two neighbors (with the same component).

Software configuration

Table 343. DCMIPP_P1BPRCR and DCMIPP_P1BPRSR bit function

Configuration example

A too aggressive bad pixel detection might detect false bad pixels and erroneously correct them (it can occur for very thin light vertical lines that cannot be distinguished from bad pixels in the implemented horizontal filter). To avoid such artifacts, it is advised to setup a software loop-back that tunes the filter strength so that the detected bad pixels remain within a predefined valid range.

Start configuration (or without software feedback loop):

• • ENABLE = 1 (enabled)
• • STRENGTH = 0 (high tolerance)

Software feedback loop:

Assumes a maximum amount of 100 bad pixels on the array (any other value can be used).

• • BADCNT read: gets the amount of corrected bad pixels of the last frame
• • Low-pass filter on the BADCNT, and/or hysteresis
  • – BADCNT > 100: too aggressive, decrease STRENGTH
  • – BADCNT < 30: too tolerant, increase STRENGTH (assuming there are bad pixel on the array)

Note: The expected amount of bad pixels (100 in the above example) can be set to other values, or made dependent upon the equivalent gain of the sensor (analog gain, exposure duration). A high-gain picture has more noisy pixels and can accept a more aggressive filter to smooth the noise, while a low-gain picture has mostly correct pixels, only very uncorrelated pixels need to be corrected.

39.6.5 Input decimation

The decimation makes it possible to cheaply downsize a frame, by a factor 2, 4, or 8, in both horizontal and vertical directions. It supports a special mode adapted to a raw Bayer format in input.

The decimation is redundant with the downsize, that brings a higher quality (the decimation discards the data of the dropped pixels, while the downsize averages all pixels and does not lose information).

The decimation has two main usages:

• • Reduces the width of very high resolution sensors to one accepted by the raw Bayer-to-RGB conversion. Indeed, the raw Bayer-to-RGB embeds a line buffer that limits its processed images to a maximum width of 2688 pixels. The decimation makes possible, for instance, to plug a 12 Mpixel sensor (4000 x 3000), decimate its width by a factor 2x to 2000 pixels wide, and have it converted to RGB.
• • Complements the downsize (limited to ratio of 8 x 8), to have a combined downsizing ratio of 64 x 64. It is recommended to have the largest ratio implemented in the downsize, and the decimation complement to that. For instance, for a ratio 30x, is it recommended to have a ratio 4x in decimation, and a ratio \( 30 / 4 = 7.5 \) in the downsize.

Features

• • Forwards one item every 1, 2, 4, 8 pixels, independently, vertical and horizontal.
• • Supports raw Bayer formats, where RG-GB components must remain grouped. In that mode the above forwarded items are not pixels, but a double component (for example R and G or G and B for a RGGB format), in both vertical and horizontal directions. For

instance, with a decimation of 2x, the input sequence R0, G0, R1, G1, R2, G2, R3, G3 gets forwarded as R0, G0, R2, G2 and not R0, R1, R2. The same applies vertically.

Software configuration

Table 344. DCMIPP_P1DECR bit function

The raw Bayer mode in decimation is indirectly activated when the Datatype is defined as raw Bayer.

39.6.6 Black level calibration

This module suppresses a potential positive black level offset, which happens when the sensor sends non-null pixels when capturing a black picture.

Functionality

• • Subtracts an offset from the R, G, B components.
• • A different constant is used for R vs. G vs. B, including within raw Bayer pixels.

Software configuration

Table 345. DCMIPP_P1BLCCR bit function

The offset subtracted is aligned components MSB: for instance, if components are defined on 12 bits, the 8-bit offset subtracted for red is BLCR « 4.

Configuration example

It is possible to use the statistics extraction, to extract the long-term most-dark R, G, B components, and assign their darkness as black. Their RGB values become the black level offset for each RGB.

For a raw Bayer sensor with 12-bit components, if the measured black level offset for red is 64 (out of a maximum of 4095), configure BLCR as 16.

39.6.7 Exposure compensation and white-balance calibration

This module aims at compensating bad exposure (too short/dark or too long/light), and at compensating a bad color balance to recenter the colors to more white ones.

The exposure and white balance operations are purely linear, and do not distort colors if performed correctly.

Functionality

• • Multiplies R, G, B by a configured floating-point constant.
• • A different constant is used for R vs. G vs. B, including within raw Bayer pixels.

Software configuration

Table 346. DCMIPP_P1EXCR1 and DCMIPP_P1EXCR2 bit function

The idle setting (that sends out a similar value as input) is:

• • MULTR = MULTG = MULTB = 128
• • SHFR = SHFG = SHFB = 0

Similar idle couples are MULTx = 64 and SHFx = 1, or 32 and 2, or 16 and 3.

Configuration example for exposure alone

It is recommended to use the exposure control in conjunction with the contrast enhancement below in the pipe: the idea is to deliberately keep an exposure lower than ideal, to avoid burning the whites: the contrast enhancement, thanks to non-linear operator on luminance, lightens the dark pixels, and reduces the dynamic of the light pixels without burning them.

It is recommended to use the statistics to extract the average luminance across the whole frame (done best by extracting the R, G, B averages, and computing the luminance from the RGB).

The idea is to push the average output luminance to 128, or if working with the contrast enhancement too, to push it to a lower value, for instance 64.

When measuring an average luminance of 26 (target is 64), the exposure factor to apply is \( 64 / 26 = 2.5x = 1.25x * 2 \) , hence the following configuration must be set:

• • \( MULTR = MULTG = MULTB = 128 * 1.25 = 160 \)
• • \( SHFR = SHFG = SHFB = 1 \)

Configuration example for an additional white balance

An example of easy way to recenter the colors is using the Gray-world algorithm.

This algorithm assumes that a default scene has an inherent balance of its red, green, and blue pixels, so that the average values of the red, green, and blue components across the ideal picture are the same. If the measured averages are not the same, it means that there has been a distortion of the white balance, that must be corrected by amplifying or reducing the red, green, and blue strengths in the exposure module, thanks to the differentiated red vs. green vs. blue multipliers and exponents.

In practice this is the same approach used for the exposure compensation, but, instead of forcing the output luminance to a default target (64 in the example), user must force the red, green, and blue averages towards the same exposure target.

As an example:

• • Exposure target is defined as 64.
• • Measured red is 26: red factor is \( 64 / 26 = 2.5x \) , hence \( MULTR = 160 \) , \( SHFR = 1 \) .
• • Measured green is 32: green factor is \( 64 / 32 = 2.0x \) , hence \( MULTG = 128 \) , \( SHFG = 1 \) .
• • Measured blue is 10: blue factor is \( 64 / 10 = 6.4x \) , hence \( MULTB = 205 \) , \( SHFB = 2 \) .

39.6.8 Demosaicing

The module works on the pixel pipes, and implements a mid-quality conversion from raw Bayer 8 bpp to an RGB 24 bpp, with no implicit half-sizing: the output RGB pixel rate is the same as the input component rate.

The conversion targets real-time but medium quality purposes, like display or low-resolution software analysis. For higher quality conversions, it is recommended to convert the raw Bayer in software, with higher quality algorithms but limited to non-real-time, thus still-picture applications.

The algorithm takes the surrounding \( 3 \times 3 \) components in input, applies linear heuristics to determine the most likely missing components, and generates an output RGB888 pixel.

To provide the \( 3 \times 3 \) input matrix to the algorithm, the demosaicing module embeds a double line-buffer, \( 2 \times 2688 \) components wide. As such, it limits the maximum width of a converted demosaicing buffer to 2688 components/pixels wide. Wider camera sensor resolution can be connected and goes through the first decimation stage in front of the demosaicing operation.

Note: This demosaicing conversion algorithm can create some artifacts, like Zipper. Such artifacts are much reduced by a later downsizing or even suppressed for a \( 2 \times 2 \) downsizing.

Applications like display, video encode, or software analysis usually require such a downsize, so the artifacts become invisible.

The underlying heuristic filters try to extract and adapt to edges, horizontal and vertical lines, and lone pixels.

Functionality

• • Input pixel format: from 1 (raw Bayer) to three (RGB/YUV) components on 8 bits
• • Max input width of 2688 components
• • Demosaicing conversion:
  • – 2-line buffer
  • – provides a 3 x 3 matrix to the algorithm + one trailing component (2 left)
  • – Extraction and optimization for specific shapes:
    • > Edges (on left, right, top, bottom)
    • > Lines (horizontal and vertical)
    • > Peaking (lone pixels detection)
• • Output pixel format: three RGB components on 8 bits.

Software configuration

Table 347. DCMIPP_P1DMCR bit function

The four filters, EDGE, LINEH, LINEV, PEAK, improve the zoomed accuracy of the filtered picture (thus mostly luminance correctness, like no Zipper artifact), however at the expense of some chrominance blurring or distortion. Some scenes can require specific settings:

• • Scenes where chrominance distortion is very visible (like a mostly black and white scene viewed by an RGB sensor) can require lower strength of the filters.
• • On the opposite, scenes heavily loaded with geometric shapes (and thus source of artifacts) can require strengthening the filters.

Globally, 6/4/4/2 is a good compromise for the filter strength.

Configuration example

If the camera sensor is raw Bayer, with a grid with a red top-left pixel, the next configuration can be used:

• • ENABLE = 1 to have the demosaicing active
• • TYPE = 0 to have a top-red pixel, thus a RGGB grid
• • EDGE, LINEH, LINEV, PEAK = (4, 4, 4, 4) as typical setting (so an internal strength of 8 for each parameter).

39.6.9 Color conversion

This module implements a color conversion with flexible (fully programmable coefficients), which can be used to map the following features:

• • YUV-to-RGB: when the pixels of a YUV-sensor are used as RGB (display, GPU)
• • RGB-to-YUV: when the pixels of a RGB-sensor are used by software (for example video encode)
• • Tone change: to map special color references (sepia), emphasize some tone/colors
• • Monochrome: to compute a monochrome from RGB/YUV with a specific linear mix
• • Expansion: to handle a YUV (or RGB) expansion or compression
• • Gamma: to emulate a gamma correction via a linear scale

Features

• • Linear rotation of color space: 3 x 3 matrix + 3 x 1 column.
• • Clamping optional on either [16 to 235] for Y, R, G, B and [16 to 240] for U, V.

Software configuration

Table 348. DCMIPP_PxCCyy (yy = R0, R1, G0, G1, B0, B1) bit function

Configuration example

Table 349 provides examples of the most used conversions:

• • To / from RGB from/to YUV (as SDTV ITU-R BT.601 or HDTV BT.709).
• • With a full (0 to 255) or a reduced (16 to 235) dynamic for Y and RGB, 16 to 240 for CbCr.
• • Matrix fields are defined as XY bit fields in the color conversion coefficient registers, with X being the input R or G or B, and Y being the output R or G or B. For conversions to/from YUV, the R must be assumed as Cr (= V), G must be assumed as Y, and B must be assumed as Cb (= U).
• • Coefficients are defined signed, and multiplied by 256 when there are decimals (the matrix coefficients). Negative values are on 11 bits: \( N11(\text{val}) = \text{XOR}(\text{val}, 0x7FF) + 1 \) , on 10 bits for Added_Coeff: \( N10(\text{val}) = \text{XOR}(\text{val}, 0x3FF) + 1 \) .

Table 349. Color conversion: examples of coefficients (continued)

39.6.10 Contrast enhancement

This module ends the image processing pipe. It makes possible to enhance the contrast of the frame, by non-linear emphasis of the luminance of the pixels, and to provide more luminance quantification steps at highly used luminances (usually central grays), by reducing the amount of quantification steps less used by the picture (usually the few very dark and very light pixels).

Functionality

• • Inputs RGB from the pipe
• • Computes the luminance of the local pixel
• • Extracts the desired luminance increase/decrease factor, in function of the input luminance, non-linearly (using a look-up table)
• • Multiplies the input RGB with the amplification factor
• • Outputs the clamped RGB components on 8 bits

Software configuration

Table 350. DCMIPP_P1CTCR1,2,3 bit function

Configuration example

There are multiple approaches to handle contrasts.

It is possible to flatten the spectrum of luminance density (that is, to try have a similar amount of pixel per each segment of luminance), which is to fill each of the eight segments of luminance with 12.5% of the pixels of the frame.

An intermediate approach is to proceed with only the upper and lower half pixels: half of pixels must have a luminance below and another half above the mid-dynamic (128 for 8-bit components).

It is advised to use the statistics extraction module, sample the luminance after the exposure control (to avoid an impact on the exposure compensation), and sort it in the 12 available bins (refer to Section 39.6.11 ). From the extracted bins the user must linearly extrapolate the luminance of the median of the amount of pixels. The luminance of median pixel must be increased/decreased to be moved to the targeted output median luminance, which is recommended to be the median dynamic (128).

For instance, a 400 kpixels VGA picture can have 200 kpixels below a luminance of 64, and the other 200 kpixels above that value. The ideal luminance amplification factor can be set to \( 128 / 64 = 2x \) :

• • pixels darker than 64 are lightened by a factor 2x
• • pixels lighter than 64 are lightened a bit less, decreasingly
• • pixel of luminance of 255 get an idle, 1x, factor.

The above assertions can be used to compute the LUMx values to configure:

• • Ratio 2x from lum = 0 to 64:
  • – LUM0 = LUM1 = LUM2 = 32.
• • Decreasing ratio from 32 to 256, best exponentially decreasing:
  • – LUM3 = 27
  • – LUM4 = 23
  • – LUM5 = 20
  • – LUM6 = 18
  • – LUM7 = 17
  • – LUM8 = 16 (the maximum luminance, 256, must remain idle).

39.6.11 Statistics extraction

This module extracts several statistics across the frame, to provide inputs to the software image compensation algorithms, that configure feedbacks to the following implemented hardware modules: black level calibration, exposure compensation, white balance, and contrast enhancement.

The statistics are accumulated during a full frame, and sampled at the end of the frame, thus allows the software to have the full next frame to read the accumulated statistics, process them, define the adapted configuration for the hardware feedback modules, and to configure these modules.

Thanks the shadowing mechanism, the newly configuration parameters are active at the following frame, thus active at an average two frames after the statistics are accumulated. The statistics extraction block is considering all frames selected by means of the DCMIPP_P1FSCR configuration registers. Statistics block is running even when the frame is not supposed to be captured, respect to the DCMIPP_FCTCR configuration register.

The latency is not an issue, as the captured scene is expected to be almost static, and typical software compensation algorithms filter the statistic variations to avoid oscillations and visual artifacts (the low-pass filter typically has a one-second period).

Functionality

• • Three statistics accumulators available, allowing three statistics extractions per frame.
  • – Different types of statistics can be extracted frame over frame
• • 32-bit accumulators, allowing to accumulate 8-bit samples over 16 Mpixels.
• • Average mode, that extracts averages (accumulates component values)
  • – Capability to discard extreme values (if value is <16 or >240, same for 32 or 64).
• • Binning mode, which extracts presences (accumulates 256 per fit)
  • – 12 available bins (values: <4, <8, <16, <32, <64, <128, and reverse to 256).
• • Capability to define a sub-rectangle within the frame where statistics are extracted
  • – Allows to focus the statistics on a specific region of interest.
• • Raw Bayer supported, with the statistics using the specific locally available R, G, B components of the raw Bayer
• • Statistics can be performed on either the input data (before any pixel modification) or after demosaicing, to benefit of a full RGB and thus luminance presence per pixel.

Software configuration

Table 351. DCMIPP_P1STyCR (y = 1, 2, 3), DCMIPP_P1STySR (y = 1, 2, 3), DCMIPP_P1STSTR and DCMIPP_P1STSZR bit function

Table 351. DCMIPP_P1STyCR (y = 1, 2, 3), DCMIPP_P1STySR (y = 1, 2, 3), DCMIPP_P1STSTR and DCMIPP_P1STSZR bit function (continued)

Note: If HSIZE or VSIZE = 0 when the CROPEN bit is set, the block internally forces the “0” value at the maximum for the corresponding bit field (i.e. 0xFFE).

Table 352. Statistics extraction: collected data vs. modes

Configuration example for exposure control

For exposure control and white balance calibration, user would typically extract the average values of the luminance, and of the R, G, B. It can be done in a single frame capture campaign, with the following settings:

• • ENABLE = 1 (enabled)
• • MODE = 0 (average mode)
• • SRC = 0 for red, 1 for G, 2 for B (upper values, before processing)
• • BINS = 0 (all pixel values)
• • CROPEN = 1 (1 below: to select a 300 x 220 close to top-left corner: 10,10)
• • VSTART = 10
• • HSTART = 10
• • VSIZE= 220
• • HSIZE= 300

After one frame (after a VSync), the Accu0, Accu1, Accu2 counters contain, respectively, the average red, green, blue accumulated across the 300 x 220 rectangle.

The value to expect from the Accu0, Accu1, Accu2 can be computed according to the following hypotheses:

• • Amount of pixels: 300 x 220 (or full frame if statistics has CROPEN = 0)
• • Amount of components:
  • – Raw Bayer-red: 1/4, as one pixel every four in raw Bayer has a red component.
  • – Raw Bayer-green: 1/2, as one pixel every two in raw Bayer has a green component.
  • – Raw Bayer-blue: 1/4, as one pixel every four in raw Bayer has a blue component.
  • – RGB: 1, as all pixels contain R, G, B components.
• • Average value on 8 bits: 128 typically for a mid-gray picture (RGB: 128, 128, 128).
• • Re-sampling: 1/256 (re-sampling accumulator to ACCU at end of frame).

The expected values in the accumulators are:

• • Accu0 (red): \( 300 \times 220 / 4 \times 128 / 256 = 8250 \)
• • Accu1 (green): \( 300 \times 220 / 2 \times 128 / 256 = 16500 \)
• • Accu2 (blue): \( 300 \times 220 / 4 \times 128 / 256 = 8250 \)

Configuration example for contrast enhancement

For contrast enhancement, the amount of pixels with a luminance inside a bin is typically extracted. As statistics are extracted 3 by 3, and as 12 bins can be accumulated, data are extracted during four consecutive frames, with three different bins for each frame.

The typical configuration is the following:

• • ENABLE = 1 (enabled)
• • MODE = 1 (bin mode)
• • SRC = 3 (upper value, luminance selected)
• • BINS = 0 (then, at next frame 1, then 2, then 3)
• • CROPEN = 0 (no stat sub-rectangle)

After one frame (after a VSync), Accu0, Accu1, Accu2 counters contain the amount of pixels fitting bins <4, <8, <16, respectively.

The value to expect from the Accu0, Accu1, Accu2 can be computed by multiplying the following:

• • Amount of pixels: \( 640 \times 480 \) (assuming a full frame resolution of VGA).
• • Amount of components: 1 (all pixels contribute to the luminance, even in raw Bayer).
• • All pixels contribute to the statistics, both in raw Bayer and in RGB. In raw Bayer, a fake statistics is extracted, assuming a RGB-balanced pixel (Gray-world algorithm).
• • Average value for bins: 256 (as in Bin mode, 256 is added for each fitting presence).
• • Re-sampling: 1/256 (at end of frame)

The resulting value in accumulators are:

• • Accu0 (bin < 4): \( 640 \times 480 \times 1 \times 256 / 256 \times 1\% = 3072 \) (assuming 1% of pixels have lum < 4)
• • Accu1 (bin < 8): \( 640 \times 480 \times 1 \times 256 / 256 \times 3\% = 9000 \) (assuming 3% of pixels have lum < 8)
• • Accu2 (bin < 16): \( 640 \times 480 \times 1 \times 256 / 256 \times 8\% = 24000 \) (assuming 8% of pixels have lum < 16)

At the following frame, with BINS = 1 , the statistics for luminance <32, <64, <128 are extracted, providing figures like 72k (25%), 144k (50%), and 210k (75%).

Such values indicate a median luminance \( \sim 64 \) , as 50% of the pixels have a luminance \( < 64 \) .

39.7 Pipe1 (post-processing part)

39.7.1 Overview

The post-processing Pipe1 (whose diagram is shown in Figure 404 ) is a pixel pipeline in which post-processing functions are integrated to relieve the software from very specific operations (time and memory consuming), like:

• • Crop, to keep only the part of the picture required by the targeted application
• • Decimation, as a cheap preprocessing before the downsize
• • Downsize, to reduce the picture resolution (size reduction)
• • ROI signaling
• • Gamma correction before sending the data to the memory for further displaying operations
• • YUV conversion with possible down-sampling option, to optimize the memory size needed to store the corresponding image or to match a YUV420 video encoder.

Figure 404. Block diagram

graph LR
    subgraph Pipe1_Ctrl [Pipe1: Ctrl]
        FC[Frame control
capture, rate]
    end
    subgraph Pipe1_Post [Pipe1: Post-processing]
        C[Crop
sub-rectangle]
        D[Decimation
1, 2, 4, 8x]
        DS[Downsize
h, 8x]
        R[ROI
8 regions]
        G[Gamma
g = 2.2]
        RY[RGB-to-YUV
cscf = any]
        D420[444-to-420
chroma downsampling]
        PP[Pixel packer
to pixel format]
    end
    subgraph AXI [AXI]
        AM[AXI-MASTER]
    end
    FromISP[From ISP] --> FC
    FC --> C
    C --> D
    D --> DS
    DS --> R
    R --> G
    G --> RY
    RY --> D420
    D420 --> PP
    PP --> AM
    AM --> AXIBus[AXI bus 64 bits]
  

The applicative Pipe1 is the most featured pixel pipeline, even if the image processing functions can be shared with Pipe2 (refer to Section 39.8.2 ).

Pipe1 can be reconfigured from time to time. Some events can be used to trigger the software reconfiguration routines (refer to Section 39.13.2 ).

It is also possible to write into the shadow registers during the processing of a frame. The values are loaded into the physical register based on the following start of frame event.

39.7.2 Pixel 2D cropping

The pixel 2D cropping aims at extracting and forwarding a rectangle of pixels.

The cropping assigns an X and Y location to its input pixels, and selects a configured rectangle among the pixels, and forward the pixels of this rectangle farther in the pixel pipe. The pixels around the selected rectangle are discarded.

Functionality

• • Counts lines thanks to HSync (if from parallel interface) or packets (if from CSI-2)
• • Counts pixels inside a line
• • Positive cropping (forwards a selected rectangle of pixels in the pipe)
• • Cropping with 0 value for HSIZE[11:0] or VSIZE[11:0] forces the internal crop to get the maximum value on the parameter where the software sets to the 0 value

Software configuration

Table 353. DCMIPP_PxCWSTR and DCMIPP_PxCRSZR bit function

If the cropped window is larger than the received window, only the received pixels are forwarded (no pixels are inserted to match the window size).

If the software sets 0 in one or both horizontal/vertical size bit fields (HSIZE, VSIZE), the crop forces internally the value to be at the maximum on the corresponding bit field.

Configuration example

To select a small rectangle (size 100 x 50), whose upper pixel is located at position (30, 10), provide the following setting:

• • ENABLE = 1 to crop a sub-rectangle
• • HSTART, VSTART = (30, 10) to start cropping at x = 30, y = 10 of the received picture
• • HSIZE, VSIZE = (100, 50)

The most bottom-right selected pixels have a position (129, 59) from the received picture.

The frame forwarded is handled downstream in the pipe as a 100 x 50 frame.

39.7.3 Decimation pre-downsize

The decimation (prior to downsize) makes it possible to cheaply downsize a frame by a factor 2, 4, or 8x, in both horizontal and vertical directions.

The decimation is partially redundant with the downsize, that brings a higher quality (the decimation discards the data of the dropped pixels, while the downsize averages all pixels and does not lose information).

The decimation complements the downsize (limited to ratio of 8 x 8), to have a combined downsizing ratio of 64 x 64. It is recommended to have the largest ratio implemented in the higher quality downsize, and have needed the decimation complement to that. For instance, for a ratio 30x, is it recommended to have a ratio 4x in decimation, and a ratio \( 30 / 4 = 7.5 \) in the downsize.

Features

• • Forwards one item (typically pixel) every 1, 2, 4, 8 pixels, independently vertical and horizontal.

Software configuration

Table 354. DCMIPP_PxDCCR bit function

39.7.4 Downsize

This module implements a downsize based on box filtering, which allows ratios from 1x to 8x, independently, in both X and Y directions.

The box filter has properties close to a bilinear filter for small ratios (from 1x to 2x), but provides the advantage of avoiding some Moiré effect for higher ratios (from 2x to 8x) thanks to using all the samples.

Functionality

• • Features:
  • – Integer + decimal ratio, different vertically and horizontally.
  • – Ratio: 1.0x to 8x
  • – Limited to a maximum output buffer width of 2688 pixels due to the needed linebuffer.
• • Operations:
  • – Increments defined with 3.13 precision (<1 pixel error after maximum width of 4094)
  • – Internal computations for filter defined with 3.3 precision
  • – Accumulates skipped pixels to allow ratio up to 8x with saturation on 11 bits
  • – Weighting of shared pixels, to allow good quality at ratio close to 1
  • – Independent H filter, then V filter
  • – Init phase: left of left-most pixel is aligned on input pixel (see below Pi0, Po0)

Software configuration

Table 355. DCMIPP_PxDSCR, DCMIPP_PxDSRTIOR, DCMIPP_PxDSSZR

Figure 405 shows the principle of the computations:

• • \( Po0 = (1.00 \times Pi0 + 1.00 \times Pi1 + 0.25 \times Pi2) / 2.25 \)
• • \( Po1 = (0.75 \times Pi2 + 1.00 \times Pi3 + 0.50 \times Pi4) / 2.25 \)
• • \( Po2 = (0.50 \times Pi4 + 1.00 \times Pi5 + 0.75 \times Pi6) / 2.25 \)

Figure 405. Downsize with coverage filtering

Configuration example

• • ENABLE = 1 to activate the downsize
• • Downsize ratio 1x: HRATIO = VRATIO = 8192, HDIV = VDIV = 1023
• • Downsize ratio 2x: HRATIO = VRATIO = 16384, HDIV = VDIV = 512
• • Downsize ratio 3x: HRATIO = VRATIO = 24576, HDIV = VDIV = 341
• • Downsize ratio 8x: HRATIO = VRATIO = 65535, HDIV = VDIV = 128

39.7.5 Regions of interest (ROIs)

Pixel pipes can defined up to eight different ROIs to highlight areas on a picture. The purpose of this feature is to draw one rectangle with user size defined for each enabled ROI.

There is no hardware processing within the defined area applied on the image. As the aim is to draw rectangles on a display, the ROI is supposed to be used when the data flow is RGB at the output of the image processing blocks.

The line width is configurable commonly for each ROI thanks to bit-field ROILSZ[1:0] of the DCMIPP_PxCMRICR register. The color line is configurable per ROI to make easy visual distinction between each region of interest.

Each configuration is valid for a frame and it may be changed frame by frame by reprogramming the registers by software.

Each ROI can be enabled independently for displaying purpose.

Figure 406 shows a basic example of two ROIs displayed within a frame to isolate two flights (it is only visual information). The application can decide to crop the area to work on these two ROIs, by reprogramming the crop area for the next frames to make appropriate analysis linked to the application target.

Figure 406. Example with two ROIs

Software configuration

The first action is the configuration of DCMIPP_PxCMRICR, the common register to activate the ROI and to select the line width applied to all the active ROIs within a frame:

• • ROIEYEN (y = 1 to 8): each ROI can be enabled during the frame.
• • ROILSZ: select the line width in pixel (1, 2, 4 or 8 pixels wide).

The second step is the configuration of each ROI, independently, through registers DCMIPP_PxRIyCR1 and DCMIPP_PxRIyCR2. The color line, among others, is set during this step.

• • VSTART and HSTART bit fields allow to set the origin point of the rectangle for a given ROI. The origin is located on the top-left pixel, within the colored lines.
• • VSIZE and HSIZE bit fields are used to set up the vertical and horizontal size in pixel for the area to draw. The size of the rectangle is measured the colored lines.
• • CLR, CLG, CLB are used to select the color among 64. Each component has its two bits duplicated to 8-bit, to get an RGB888 at the output of the ROI block

39.7.6 Gamma conversion

This feature implements a gamma compression on each R, G, B component, using a static gamma exponent 2.2, to store gamma-compressed pictures, as defined in the sRGB standard.

Features

• • The gamma is implemented as a 7-segment linear curve.
• • It crosses the following way-points, in logarithm spacing:
  • – (0,0)
  • – (4,39)
  • – (8,53)
  • – (16,72)
  • – (32,99)
  • – (64,136)
  • – (128,186)
  • – (256,256)

Software configuration

ENABLE = 1 in the DCMIPP_PxGMCR registers activate the color conversion.

39.7.7 YUV conversion

This block is a replication of the one described in Section 39.6.9 , used to convert RGB in YUV color format (basically, since the conversion YUV to RGB is assumed to be performed by the color conversion block into the image processing main function). Bit fields are the same of the color conversion module, only the register name and the memory mapping are different. The register functions are the same, the register names are DCMIPP_P1YUVxxx instead of DCMIPP_P1CCxx.

39.7.8 Chroma down-sampling

The Chroma down-sampling module reduces the chroma resolution from its arriving YUV444 format to the potential YUV420 output, and sends them to a different channel.

Features

• • Downsizes in H first, if YUV420, with (0.5 + 0.5) filter
• • Downsizes in V then, if YUV420, with (0.5 + 0.5) filter
• • Memory: embeds one half-line buffer for each U and V

Software configuration

Configuration is indirectly taken from the pixel packing output format, located in the DCMIPP_PxPPCR register.

• • The Chroma down-sampling is active when the output pixel format is YUV422-1, YUV422-2, YUV420-2, YUV420-3.

Output

• • One to three channels: RGB/Y, UV (or U) and V.

39.7.9 Pixel packing

This module works on the dump pipe (Pipe0) and pixel pipes (Pipe1 .. N), setting the arriving pixels in memory words.

Features

• • DBM addresses: double buffer mode with their associated address locations if the application need to swap memory address for one frame to the next one.
• • Flexible pitch (line stride): width up to 4094 pixels, 32 bpp, potentially interleaved.
• • Pixel format: all usual formats, as indicated in Section 39.3.6 (Pipe1 allows semi-planar and full planar buffer, Pipe2 coplanar buffer only).
• • Swap R-vs-B: swap the red vs. the blue components of the output pixel.
• • Padding: MSB vs. LSB padding of 10/12/14 bit onto 16 bit. (such formats are supported only on the dump pipe).
• • Header enable: allows to dump (or not) the CSI packet headers, to allow software an easier extraction of data in case of Interleaved CSI packets in the captured buffer.
• • Multi-line event: event generated (for DMA and INT) at each multiple of sent lines.
• • Multi-line address wrapping: allows to write the full frame across a ping-pong buffer, thanks to wrapping back to the BaseAddress after a power-of-2 amount of lines.

Note: Some features (for example padding) are available only on the dump pipe, while others (like swap R-vs-B) are available only on the pixel pipes.

Table 356. DCMIPP_PxPPCR bit function

Table 356. DCMIPP_PxPPCR bit function (continued)

Double buffer mode

The dump pipe uses an AXI master interface to dump the data from the internal FIFO to the external memory. In the application, it is possible to handle frame data swapping memory area frame by frame. The double buffer mode fills up this function. There are two memory address registers set to initialize the base addresses of these memories areas. Each start of captured frame event swaps the memory base address to handle double buffering mode. The double buffer mode can be used to allow post-processing on a buffer (frame buffer), while the other buffer is read to be displayed (display buffer).

Software configuration

Double buffering mode requires an activation and addresses configuration to define the two memory areas in which data are consecutively stored frame by frame at the output of the pipeline:

• • DCMIPP_PxPPM0AR1 and DCMIPP_PxPPM0AR2 must be filled with the memory addresses associated to the double buffer memory locations.
• • Bit DBM in the DCMIPP_PxPPCR registers is set to enable the double buffer mode.

Streaming across a ping-pong buffer

The DCMIPP has the capability (pixel pipes only) to stream pixels to a slave IP (like VENC or any) without storing the full frame in the main memory, by writing to a short intermediate multi-line buffer. It can do so by configuring the LINEMULT, LMAWE and LMAWM fields.

As an example, the slave IP is a JPG encoder that expects to be provided lines of macroblocks (16x16 pixels). The DCMIPP writes into a double (ping-pong) buffer, with each 16 lines high.

The streaming flow is as follows:

1. 1. At a start of frame, the DCMIPP writes its pixels at an incremented address starting at the buffer BaseAddress (M1A/M2A).
2. 2. After having written 16 lines, it emits a hardware Line event (dcmpp1_pXline_evt). At the reception of this hardware trigger, the slaved IP (JPG encoder here) starts to read the first 16-line buffer at the BaseAddress, and to encode its pixels.
3. 3. The DCMIPP continues to write at the following addresses, thus after line 16.
4. 4. After having written another 16 lines, the DCMIPP emits another Line event, to that the slaved IP can read and process the following 16 lines (at BaseAddress + Line16).
5. 5. The DCMIPP wraps back to the BaseAddress, writes the following lines (32 to 47) starting at this BaseAddress, and then sends a trigger to the slave IP to process these new 16 lines.

The needed configuration is the following one:

• • LINEMULT = 0x4, to send an event after having written each 16 lines
• • LMAWM = 0x5, to wrap the address after having written each 32 lines
• • LMAWE = 1, to enable the address wrapping of the address.

39.7.10 Overrun detection

This logic handles flow-control hazards: the DCMIPP is provided a continuous flow of data (via parallel or CSI2 interfaces) without any wait/hold capability, while downstream of DCMIPP, the access to memory can be temporarily stuck.

In some cases the internal FIFOs of the DCMIPP get full and pixels may be deleted. The overrun detection logic handles such hazard at best.

Features

• • Software gets warned in case of an hazard:
  • – Interrupt OVR is triggered at the first deletion of a pixel, repeated at every frame, as long as the overrun hazard is present.
  • – It is recommended to skip / trash the impacted frame:
    • > Overflowing pixels are deleted, so that when the access to memory resumes, the remaining of the line is discarded (not written out). The normal processing resumes at the start of the next line. Potentially, several full lines can get discarded.
• • No Frame-to-Frame impact in case of an hazard:
  • – VSync resets what is needed in the DCMIPP, so that any hazard on a previous frame has no impact at all on the pixels dumped in next frames.

Software configuration

• • Interrupt DCMIPP_PxSR.OVRF is warned in case of hazard.

39.8 Pipe2 (post-processing)

39.8.1 Overview

Pipe2, compared to Pipe1, is a lighter pixel pipeline offering no embedded image processing blocks. This pipe can work in standalone mode, or share the image processing functions with Pipe1 (refer to Section 39.8.2 ).

Figure 407. Pipe2 architecture overview

graph LR
    subgraph Inputs
        ISP[From ISP] --> SEL[SEL]
        Input[From input] --> SEL
    end
    SEL --> Pipe2_Ctrl[Pipe2: Ctrl]
    subgraph Pipe2_Ctrl
        FC[Frame control capture, rate]
    end
    Pipe2_Ctrl --> Pipe2_Post[Pipe2: Post-processing]
    subgraph Pipe2_Post
        direction LR
        Crop[Crop sub-rectangle] --> Decimation[Decimation 1, 2, 4, 8x]
        Decimation --> Downsize[Downsize 1x, 8x]
        Downsize --> ROI[ROI 8 regions]
        ROI --> Gamma[Gamma g = 2.2]
        Gamma --> PixelPacker[Pixel packer to pixel format]
    end
    PixelPacker --> AXI_MASTER[AXI MASTER]
    AXI_MASTER --> AXI_Bus[AXI bus 64 bits]
  

The main blocks (controller and post-processing general functions) of Pipe2 are almost the same as those of Pipe1. The flow selection and frame controller allow to configure Pipe2 behavior when it is used in standalone, meaning without image processing heritage from Pipe1. Refer to Section 39.4.5 and Section 39.4.6 for more details about the use cases and the configuration of the pipe.

The set of post-processing blocks is reduced compared to Pipe1, as neither the YUV conversion nor the down-sampling YUV444 to YUV420 are integrated in Pipe2. This pipe can have its own image crop area, its dedicated downsizing, and take benefit of the gamma correction, like Pipe1. Refer to Section 39.7 and to its subsections for more details on how to use and configure by software these post-processing functions.

Software configuration

To use Pipe2 as a standalone pixel pipe, without any image processing functions shared with Pipe1, bit PIPEDIFF must be set in the DCMIPP_P1FSCR register. This bit has a meaning only when the flows can be differentiated, that is, when the camera connected and processed by the DCMIPP is a CSI2 camera interface.

In this non-shared mode, it is required that different flows are sent to Pipe1 and Pipe2, or a non-supported behavior can occur.

The flow to Pipe1 and the flow to Pipe2 may potentially be interleaved at packet level.

Pipe2 may be reconfigured time to time. Some events may be used to trigger the software reconfiguration routines (refer to Section 39.13.2: Interrupts ).

It is possible also to write into the shadow registers within the processing of a frame. The values are loaded into the physical register based on the start of frame event.

39.8.2 Pipe1 and Pipe2 sharing image processing functions

Following the application, it may be interesting to share the image processing functions belonging to Pipe1 with Pipe2 when the dataflow is shared for both. One pipe may be used for display operation, and the other one used for statistics or software analysis. In this case PIPEDIFF bit of DCMIPP_P1FSCR register must be cleared.

If the CSI-2 module is connected to the DCMIPP, the VC DT and DTMODE of Pipe2 are the same as those configured into DCMIPP_P1FSCR register, making non relevant the values for these fields into the DCMIPP_P2FSCR register. However, bits CPTREQ and CPTMODE remain active into the DCMIPP_P2FSCR register to control the flow inside Pipe2.

39.9 Application use cases

There are several application use cases that can fit the DCMIPP architecture, a few of them are described in this section.

39.9.1 Parallel interface camera sensor module

A parallel interface camera sensor can transfer images as well as statistics. It is easier to manage the software processing by splitting the statistics and the image data on different memory storages. Two pipes are used in such case, with Pipe0 extracting the statistics and Pipe1 dealing with the image data (whatever the data format supported by the DCMIPP). Both work in parallel on the same frame, frame by frame. Each output FIFO attached to the pipe conveys the data on the targeted memory for further processing.

The camera sensor module may use some lines of the frame to transfer statistics or sensor configuration data. These lines can be placed at the beginning and at the end of the frame (see Figure 408 ), even if there is no real standard.

Figure 408. Use case: Pipe0 (statistics), Pipe1 (pixels)

Software configuration

• • Pipe0: select the crop area to extract statistics only
  • – Configure HSTART[11:0] = 0 and VSTART[11:0] bits in the DCMIPP_P0SCSTR register from the line number corresponding to the first pixel data line (after first statistics line).
  • – Configure HSIZE [11:0] (horizontal screen resolution in number of data 32-bit equivalent to the horizontal resolution of the screen in pixels) and VSIZE [11:0] to the value corresponding to the offset line number from the first pixel data area. These bits are in the DCMIPP_P0SCSZR register.
  • – Set bit POSNEG of the DCMIPP_P0SCSZR register to reject data belonging to the rectangle defined, to keep the outside area of the rectangle, where the statistics are taken in this use case.
• • Pipe1: Select the area to extract the statistics into DCMIPP_P1SRCR register, by configuring bit FIRSTLINEDEL[2:0] and LASTLINE[11:0] (it avoids to start the image processing on statistics data). Only the pixel data are forwarded, if needed, to the image processing blocks (for example demosaicing, auto exposure, contrast enhancement) for further processing.

39.9.2 CSI2 camera sensor module

The CSI2 protocol allows, thanks to Datatypes, to make distinction between different data categories. The application may request to handle interlaced video as well as interleaved data flow like described below.

Single virtual channel involving two pipes (for example Pipe0 and Pipe1)

• • Use case: the CSI provides two data flows in parallel, typically RGB (or raw Bayer) with its sideband statistics.
• • Handling: the statistics are dumped by the dump pipe, while the pixels are sent to the pixel pipes.

Figure 409. Statistics and RGB data for single virtual channel data flow

Single virtual channel involving three pipes (for example Pipe0, Pipe1 and Pipe2)

Another use case ( Figure 410 ) employs different Datatypes in addition to the statistics within the CSI-2 flow based on a single virtual channel.

Figure 410. Three pipes within a single virtual channel in CSI2 mode

Interleaved packets (based on multiple virtual channels)

• • Use case: the CSI provides two pixel flows in parallel, typically a JPG-compressed flow and an RGB888 or raw Bayer flow.
• • Handling: the JPG is dumped by the dump pipe, while the pixels are sent to the pixel pipes.

Both Pipe1 and Pipe2 can process these pixels. The shared demosaicing mode can be used, hence both Pipe1 and Pipe2 benefit of the potential demosaicing conversion.

Figure 411. Handling interleaved packet

Figure 412. Handling interleaved packets across two pixel pipes

Interlaced video

• • Use case: the CSI provides the odd and even frames of an interlaced video onto two different virtual channels (usually with a same data type).
• • Handling: the flows of both the virtual channel of the odd field and the virtual channel of the even field are sent to the pixel pipes.
• • Single pixel pipe may be configured if the image processing function is required by the application to handle both interlaced frame (refer to Figure 413 ). In such case, a pipe reconfiguration is needed frame by frame to change the virtual channel number and the address memory for the output buffer if the software distinctly keeps the odd and even frame buffers in their respective memory locations.
• • Pipe1 and Pipe2 may be used if the application software does not need image processing functions (refer to Figure 414 ). Each pipe is configured with its own virtual channel and its own output buffer address in memory. The split of the odd and even frames is explicitly defined in a static way, avoiding any pipe reconfiguration between two frames.
• • Interlaced video can be managed with a single virtual channel identifier, using a different data type to identify odd and even frames. In such case, the CSI2 bridge uses user Datatypes to distinguish between them.
• • Limitation: the frames of the two virtual channels must be time-exclusive, by default the case for usual interlaced video sensors.

Figure 413. Interlaced video (based on single pixel pipe)

Figure 414. Interlaced video (based on two pixel pipes)

Software configuration

The following configuration must be set first into the static DCMIPP_PxFSCR and DCMIPP_CMCR registers:

• • INSEL: Selects the input flow (parallel vs. CSI) in the DCMIPP_CMCR register
• • DTMODE[1:0]: mode for CSI flow selection into that pipe:
  • – 0: only flow VC/DTIDA is forwarded in the corresponding pipe.
  • – 1: flows VC/DTIDA and/or VC/DTIDB are forwarded for Pipe0, and VC/DTIDA and VC/DTIDB for Pipe1.
  • – 2: all flows, except the VCDTIDA or VCDTIDB, are forwarded in the pipe (this last setting, 3, is available only for Pipe0).
  • – 3: all Datatypes from the selected virtual channel are forwarded in Pipe0. It is valid only for Pipe0 (the pixel formats that are not supported by Pixel pipes are suppressed).

This bit field is not valid for Pipe2.

• • DTIDA/DTIDB: Data type selection ID-A and ID-B for Pipe0 and Pipe1.
• • DTIDA: Data type selection for the pixel Pipe2 when PIPEDIFF = 1.
• • FDTFEN: Forces the value of the data type (value defined in FDTF[5:0]) to be processed by the pixel Pipe1 and Pipe2, based on the DTIDA/DTIDB detection in the flow selection block of the pipe. Not available for Pipe0.
• • FDTF[5:0]: Data type value overwritten the one received by the CSI-2 physical node when equal to DTIDA/DTIDB. Not available for Pipe0.
• • PIPEDIFF: Pipe2 to take same input as Pipe1 or differentiated:
  • – 0: Pipe2 takes the same VC/DTIDx as well as the same FDTFEN and FDTF[5:0] configured into the DCMIPP_P1FSCR register.
  • – 1: Pipe1 and Pipe2 take their own VC/DTID, FDTFEN and FDTF[5:0] values in the data flow.

After all the pipes have been configured, the processing can be activated:

• • PIPEN: Set to 1 to start the frame control logic of the pipe. When reset at 0, it immediately resets the frame control logic, which results in immediately stopping an ongoing dump (CPTREQ = 0, CPTACT = 0), see Figure 398 , Figure 399 , Figure 400 .
• • CPTREQ: Set to 1 to activate the capture at the next start-of-frame in the DCMIPP_PxFSCR register.

Note: To be cleanly sampled for the next frame, the VCIDA, VCIDB, DTIDA, and DTIDB fields must be updated after a VSync and after the first HSync.

Configuration example

The following list details five possible use cases, among others interlaced video and interleaved frames, and a 3D sensor (left + right eye):

• • Default, with a parallel dump of statistics and processing of an image on both Pipe1 and Pipe2:
  • – Image forwarded to the pixel Pipe1 and Pipe2, and processed, with the following settings:
    • > Pipe1 / DTMODE = 0, to extract a single flow.
    • > Pipe1 / VC / DTIDA = vv_dddddd, with VC and DTIDA matching the image flow
    • > Pipe1 / PIPEDIFF = 0, to send the flow to both Pipe1 and Pipe2
  • – Statistics are extracted and dumped on Pipe0 with the following config:
    • > Pipe0 / DTMODE = 0, to extract a single flow.
    • > Pipe0 / VC / DTIDA = vv_dddddd, with VC and DTIDA matching the statistics flow.
• • Interlaced video, with the consecutive but exclusive odd and even (requiring the use of the image processing functions):
  • – Interlaced video, if having differentiated VC and DT, is forwarded to the pixel Pipe1 and Pipe2, and processed, with the following settings:
    • > Pipe1 / DTMODE = 0, to extract the VC /DTIDA flow
    • > Pipe1 / VC/DTIDA = vv_dddddd, with VC and DTIDA matching the odd or the even field by reprogramming between two frames
    • > Pipe1 / PIPEDIFF = 0, to send the flow to both Pipe1 and Pipe2
    • > Pipe1 / Pipe2 / CPTMODE = 1, Snapshot mode
    • > Pipe1 / Pipe2 / CPTREQ = 1 alternatively for Pipe1 and Pipe2, frame by frame
  • – Any other (different) data can be potentially dumped with the following settings:
    • > Pipe0 / DTMODE = 2, to extract all but flows A and B
    • > Pipe0 / VCDTIDA = Pipe1 / VC / DTIDA corresponding to the odd field
    • > Pipe0 / VCDTIDB = Pipe1 / VC / DTIDB corresponding to the even field

The software needs to reprogram frame to frame the DTIDx fields of the DCMIPP_P1FSCR register to select the data to process into Pipe1 or Pipe2. The software sets CPTMODE bit to work in Snapshot mode for Pipe1 and Pipe2 in the DCMIPP_P1FCTCR and DCMIPP_P2FCTCR registers. Alternatively, bit CPTREQ is set frame by frame in one pipe then in the other one to allow the acquisition of data on the allocated pipe. The data type identifying the odd or even fields needs to be updated into the DCMIPP_P1FSCR register only, as Pipe2 shares the image processing block of Pipe1.

• • Interlaced video, with consecutive but exclusive odd and even (bypassing the image processing functions):
  • – Interlaced video, if having differentiated VC and/or DT, is forwarded to pixel Pipe1 and Pipe2, and processed, with the following settings:
    • > Pipe1 / DTMODE = 0, to extract the VC /DTIDA flow
    • > Pipe1 / VC / DTIDA = vv_dddddd, with VC and DTIDA matching (for instance) the odd field data type
    • > Pipe2 / DTMODE = 0, to extract the VC /DTIDA flow

• > Pipe2 / VC / DTIDA = vv_dddddd, with VC and DTIDA matching (for instance) the even field data type
• > Pipe1 / PIPEDIFF = 1, to consider Pipe1 and Pipe2 as totally independent
• > Pipe1 / Pipe2 / CPTMODE = 0, Continuous grab mode
• > Pipe1 / Pipe2 / CPTREQ = 1 to capture data corresponding to the VCDTIDA attached to the corresponding pipe
• – Any other different data can be potentially dumped with the following settings:
  • > Pipe0 / DTMODE = 2, to extract all but flows A and B
  • > Pipe0 / VCDTIDA = Pipe0 / VC / DTIDA corresponding to the odd field
  • > Pipe0 / VCDTIDB = Pipe0 / VC / DTIDB corresponding to the even field

In this case the interlaced video must be differentiated with the data type and not with the virtual channel (since there is a single VC by pipe), if the application wants to dump data other than the odd and even frames on Pipe0.

• • Interleaved flows, with a parallel dump of an RGB picture and a JPEG-encoded flow:
  • – The RGB picture is extracted and forwarded to pixel Pipe1 to be processed, with the following settings:
    • > Pipe1 / DTMODE = 0, to extract one flow, the RGB picture
    • > Pipe1 / VC / DTIDA = vv_dddddd, with VC and DTIDA matching the RGB picture data type
    • > Pipe1 / PIPEDIFF = 1, to send the RGB on Pipe1 and use Pipe2 for other purposes
  • – The JPEG-encoded stream is transmitted under the CSI user-defined format, is extracted and sent to the pipe to be dumped, thanks to the following:
    • > Pipe0 / DTMODE = 0, to extract one flow, the JPEG-encoded flow
    • > Pipe0 / VCDTIDA = vv_dddddd, with VC and DTIDA matching the JPEG stream data type
    • > Pipe0 / VCDTIDB = not used
  • – Another pixel picture, like YUV, can be dumped using Pipe2, by overwriting fields:
    • > Pipe2 / DTMODE = 0, to extract a second flow for the YUV picture
    • > Pipe2 / VCDTIDA = vv_dddddd, with VC and DTIDA matching the YUV picture data type
• • 3D sensor, with a parallel transmission of two RGB or YUV pictures: one with data type DT1 for the left eye, and one with data type DT2 for the right eye: the DCMIPP can process each eye on a different pipe (for example left eye on Pipe1 and right eye on Pipe2). Note that, as Pipe1 and Pipe2 are differentiated, the demosaicing is assigned to Pipe1 only and is not available for Pipe2. The left and right eye flows are similar (raw Bayer or RGB), hence they both need to be RGB and not raw Bayer:
  • – The left eye RGB picture is extracted and forwarded to the pixel Pipe1:
    • > Pipe1 / DTMODE = 0, to extract one flow, the RGB picture
    • > Pipe1 / VC / DTIDA = vv_dddddd, with VC and DTIDA matching the left eye data type (DT1)
    • > Pipe1 / PIPEDIFF = 1, to split and differentiate the flow between Pipe1 and Pipe2
  • – The right eye RGB picture is extracted and forwarded to the pixel Pipe2:
    • > Pipe2 / VC / DTIDA = vv_dddddd, with VC and DTIDA matching the right eye

1. data type (DT2)
2. • – Another flow can be dumped onto the dump pipe:
     • > Pipe0 / DTMODE = 0, to extract one flow
     • > Pipe0 / VCDTIDA = vv_ddddd, with VC and DTIDA matching the flow to dump for instance
   • • Sensors in 3D, with left and right image inside a unique global image:
     • – If the left and right images are differentiated with a left and right part of a single image, that double image can be extracted and duplicated to both the pixel pipes, and there, cropped differently at input of Pipe1 and Pipe2 respectively
   • • Sensors with multiple DT:
     • – The sensor is supported by sending all (or most) of the flows to the dump pipe, and dumped consecutively, inserting ahead of each dumped packet an header containing the VC, DT, and packet length. See Section 39.7.9 for more details on the header format. One or two pixel flows can still be extracted and processed by Pipe1 and Pipe2, and potentially suppressed from the extraction of the data dump, thanks to DCMIPP_PxFSCR.DTMODE.

39.9.3 Force data type format from CSI-2 data flow (pixel pipes only)

Application may require to use a user data type in the CSI-2 data flow without specifying the format used for the pixel transfer from a camera bridge to the DCMIPP, thanks to CSI-2 interface physical link.

However, the pixel pipes need to know the data format to process them in a proper way along the pipeline. It is possible to force a data type by software thanks to bits FDTF[5:0] and FDTFEN[5:0]. Only the pixel pipes (Pipe1 and Pipe2) can benefit of this capability. Indeed, Pipe0 manages the data and not the pixel itself.

If FDTFEN bit is set in the DCMIPP_PxFSCR register, the data type set into DTIDA/DTIDB (coupled with DTMODE) is used to detect the data type to be overwritten by FDTF[5:0] value. The value used to force the data type must follow CSI-2 data type format specification to identify the CSI-2 normalized data format.

Note: The forced data type value is unique. This means that, if two data types have been selected to be propagated into Pipe1 (DTMODE = 1), both DTIDA and DTIDB detected in input of the pipe are considered by Pipe1 with the forced value FDTF[5:0] if bit FDTFEN is set.

If the virtual channel and data type values are identical between Pipe1 and Pipe2, the force value FDTF[5:0] needs to be aligned within these two pipes if the respective FDTFEN bits are set, otherwise the behavior is not guaranteed.

If PIPEDIFF = 1, any data type format linked to a raw format can be forced on Pipe2 since it is not sharing the demosaicing block of Pipe1.

Typical application example: 3D sensors camera (top-bottom)

Some CSI-2 bridges concatenate data from left and right camera in a top-bottom approach. It means that within a frame, first half of the lines are the ones generated by the left camera, whereas the half last lines of a frame are attached to the right camera sensor. The bridge may typically make a distinction between lines coming from left camera and right camera by identifying the data type with a user data type value linked to a camera.

The user data type does not identify a pixel format, but rather a data linked to a camera. To help the pipeline process in a correct way the pixel, the software must inform the pipeline about the pixel format in input attached to the user Datatypes.

Let us imagine that the application has to separate pixels from the left camera from those of the right camera. Two pixel pipes can be configured for that, with, for instance, Pipe1 for left camera and Pipe2 dedicated to extract pixels from the right camera, as shown in Figure 415 .

Figure 415. 3D sensors (top-bottom) processed by two pixel pipes

DCMIPP_P1FSCR configuration

• – PIPEDIFF = 0, Pipe1 shares the demosaicing block with Pipe2 to transform, for instance, Raw10 in RGB888 for the output pixel format.
• – VC = Virtual channel number
• – DTMODE = 1, both Datatypes (0x32 and 0x33) selected to be propagated into Pipe1 through the demosaicing, shared with Pipe2.
• – DTIDA = 0x32, to let pixel data from the left camera sensor (identified by the user data type 0x32) to be processed by the demosaicing block, and further Pipe1 post processing stages (crop, downsize).
• – DTIDB = 0x33, to let pixel data from the right camera sensor (identified by the user data type 0x33) to be processed by the demosaicing block, and further Pipe2 post processing stages (crop, downsize).
• – FDTF[5:0] = 0x2B, to inform about the Raw10 data type normalized value in CSI-2 specification as input pixel format of the pipes.
• – FDTFEN = 1, to force the user Datatypes DTIDA and DTIDB to be decoded as FDTF[5:0] input pixel format value.
• – PIPEN = 1, to activate Pipe1.

Pipe1 crop registers (DCMIPP_P1CRSTR and DCMIPP_P1CRSZR) must be configured to crop only the first half lines of the frame corresponding to the pixels coming from the left camera sensor.

DCMIPP_P2FSCR configuration:

• – VC / DTIDA / DTIDB / FDTF[5:0] / FDTFEN = don't care since it is the same as the one defined in DCMIPP_P1FSCR (as PIPEDIFF = 0).
• – PIPEN = 1 to activate Pipe2.

Pipe2 crop registers (DCMIPP_P2CRSTR and DCMIPP_P2CRSZR registers) must be configured to crop only the second half lines of the frame corresponding to the pixels coming from the right camera sensor.

39.10 Pixel format description

This section describes the pixel formats used in the parallel interface and in the pixel pipes and dump pipe. The CSI-2 pixel format is described in the CSI-2 v1.3 Standard.

The support for the pixel formats, per interface (parallel, CSI-2, pixel output, dump output) are summarized in Table 357 , Table 335 .

Note: In the following tables the components are represented with their symbolic color: red and Cr (= V) in red, blue and Cb (= U) in blue, green in green, and Y, monochrome, and all raw Bayer components in gray.

39.10.1 Parallel interface formats

This paragraph describes the input pixel format as supported by the parallel interface of the DCMIPP and the possible swap combinations.

Note: The parallel interface does not input specifically RGB444/RGB555 (and RGB666). However, a sensor with these output can connect them onto the DCMIPP, by selecting RGB565 (and RGB888), and by either connecting the missing bits with the MSB of the sensor output or by strapping them.

Table 357. Parallel interface input pixel formats

Table 357. Parallel interface input pixel formats (continued)

Table 358. Correspondence between index and DCMIPP_PRCR register values

To adapt to non-standard sensors, the parallel interface is flexible and allows to swap bits, cycles, components, namely:

• • Bits: LSB-vs.-MSB bits are swapped, across the whole 16-bit IO pins.
• • Cycles: cycle 1/2 and cycle 2/2 are swapped. No impact for 1 or 3 cycles.
• • Components: R-vs-B components are swapped (or V-vs-U if YUV).

Table 359 shows these permutations, based on the RGB565 format. It lists the sampled bits of cycles 1 and 2, on pins 0 to 7, and for each of them returns the assignation of the RGB output components.

Table 359. Parallel interface input pixel formats

39.10.2 Pixel pipe formats

Table 360. Pixel pipe output pixel format ⁽¹⁾

Table 360. Pixel pipe output pixel format ⁽¹⁾ (continued)

1. Format identifies the value set in bitfield FORMAT[3:0] of DCMIPP_PxPPCR registers (x = 1 to 2).

39.10.3 Dump pipe formats

The dump pipe, thanks to its capability to dump, supports more formats than the pixel pipe, namely:

• • CSI-2 packet headers and data
• • CSI-2 raw Bayer pixels on 6/7bits (that can be dumped as-is but are not supported by the pixel pipe and hence cannot be processed as pixels).
• • Monochrome and raw Bayer pixels on 10/12/14 bits, padded on 16 bpp.

Table 361 details the support for these additional pixel formats. Other formats are assumed to be pixels with the same width. The other pixel formats used by the dump pipe are similar to the pixel formats used by the pixel pipe.

Table 361. Dump pipe OUTPUT pixel formats ⁽¹⁾

1. PN indicates Pixel N.

39.10.4 AXI IP-Plug

The AXI IP-Plug dumps the data of the pipes (via the AXI) to the memory. It is unique, shared by all the pipes.

Features

• • Pipe ID: AXI IP-Plug differentiates, via its ID, the source of the pipe. It allows to differentiate the QoS depending upon the clients.
• • FIFOs: sized to handle up to 2 µs latencies (1 µs average round-trip + 1 µs memory slowdown), with, by default, at reset:
  1. a) Pipe0: 5 Gbps, thus up to 1.25 Kbytes
  2. b) Pipe1-Y/RGB: 200 Mpixel/s peak x 8bpp / 32bpp, thus up to 1.6 Kbytes (if ARGB)
  3. c) Pipe1-U: 200 Mpixel/s peak x 8 bpp, thus up to 400 bytes
  4. d) Pipe1-V: 200 Mpixel/s peak x 4 bpp, thus up to 200 bytes
  5. e) Pipe2: 200 Mpixel/s peak x 32 bpp, thus up to 1.6 Kbytes
• • Burst length: 64 and 128 bytes for all the pipes.
• • Outstanding capability: 16 outstanding, to cope with 2 GB/s maximum bandwidth on 128 bytes bursts.
• • Bandwidth limiter (via an outstanding limitation per pipe): avoids a last-line dump that would uselessly overload the memory bandwidth.
• • Unique AXI-QOS: unique QOS for the DCMIPP (without differentiation between pipes that must all work correctly with their real-time constraint).
• • The total FIFO size is 5120 bytes.

Common software configuration

The AXI IP-Plug has a common configuration in the DCMIPP_IPGR1/2/3/8 registers.

The common registers are writable only when the IP-Plug is in Idle mode (see PSTART below to lock it, or below for an example).

• • MEMORYPAGE: for efficient handling of DRAM: bursts do not cross this module.
• • PSTART: request to freeze the plug, to reconfigure it safely.
• • IDLE: status to mention the plug is frozen, to reconfigure it safely.
• • IPPID[7:0], ARCHIID[4:0], REVID[4:0], DID[5:0]: version IDs of the IP-Plug.

Per-client software configuration

The AXI IP-Plug has a per-client configuration in the registers DCMIPP_IPCxR1/2/3.

The client registers are writable only when the IP-Plug is in Idle mode (see PSTART in common registers above to lock it, or below for an example).

The clients (1..n) are numbered following the sequence Pipe0..m. Inside each pipe, there are several potential clients when the planar feature is needed.

For instance, the DCMIPP with its three pipes (Pipe0, Pipe1, Pipe2) and with Pipe1 allowing a full planar capability (three buffers: Y, U, V), has the IP-Plug clients, numbered as follows:

• • Client1 = Pipe0
• • Client2 = Pipe1Y/RGB
• • Client3 = Pipe1U
• • Client4 = Pipe1V
• • Client5 = Pipe2

The configurable per-client features are the following:

• • OTR: outstanding transactions per client (Outstanding = OTR + 1)
• • TRAFFIC[2:0]: burst size: best select 128 bytes, thus TRAFFIC = 4
• • WLRU[3:0]: proportion of total bandwidth: \( WLRU = BW_i / \sum (\text{all BW}) \)
• • SVC: system virtual channel, keep SVC = 0
• • DPREGSTART: local base-address of start of FIFO for this client
• • DPREGEND: local base-address of end of FIFO (included word) for this client

Configuration computations

IP-Plug common configuration:

• • PSTART: set to 1, to put IP-Plug idle and configure its registers G1/3, R1/3
• • IDLE: read and wait till 1, to get it idle
• • MEMORYPAGE: 4, to optimize for memory pages of 256 bytes

IP-Plug per-client configuration:

• • OTR (client): outstanding requests per client, among a total of 16 requests.
  • – The provided value (OTR), is the outstanding amount minus 1.
• • DPREG size (client): RAM shared among client, among a total of 640 x 64-bit words
  • – DPREGSTART = 0 (or after another client location).
  • – DPREGEND = 127 for \( ((127 - 0) + 1) \times 8 \text{ B} = 1 \text{ KB} \) , as example if 1 KB is assigned to this client. The value to program is 127, as the word is included. DPREGEND is defined to include the last given word of memory, so that a FIFO of 128 words (starting at 0) has DPREGEND = 127.
• • WLRU (client): Weighted Least Recently Used arbitration ratio, represents the bandwidth ratio for each client. The maximum ratio WLRU value is 16, meaning that the most demanding client will have 16 times the bandwidth of the least demanding client. The value to configure in DCMIPP_IPCxR2 is the WLRU ratio minus 1.

Each client is provided a proportion of the total FIFO size and of the total outstanding available transactions, in function of the local peak bandwidth. Indeed, the DCMIPP implements buffers to smooth the bandwidth, but they are not large enough to buffer full lines. The proportion of the peak bandwidth of each pipe is estimated as:

• • \( \text{PeakBandwidthProportion (PBP)} = \text{OutputWidth (OW)} \times \text{PixelFormatSize (PFS)} \)
• • \( \text{OTR(client)} = \text{Total\_OTR} \times \text{PBP(client)} / \sum \text{All(PBP)} - 1 \)
• • \( \text{DPREG\_Size(client)} = \text{Total\_DPREG\_Size} \times \text{PBP(client)} / \sum \text{All(PBP)} \)

As an example, an xRGB 720p display output, a VGA 422 video output (for software), and no statistics, results in:

• • \( \text{PBP(Display)} = 1280 \times 4 = 5120 \)
• • \( \text{PBP(Video)} = 640 \times 2 = 1280 \)
• • \( \text{OTR(Display)} = 16 \times 5120 / (5120 + 1280) - 1 = 12 \)
• • \( \text{OTR(Video)} = 16 \times 1280 / (5120 + 1280) - 1 = 3 \)
• • \( \text{DPREG\_Size(Display)} = 640 \times 5120 / (5120 + 1280) = 512 \)
• • \( \text{DPREG\_Size(Video)} = 640 \times 1280 / (5120 + 1280) = 128 \)

Video is assigned to Pipe1 (= Client2) and display to Pipe2 (Client5):

• • IP-Plug common registers:
  • – DCMIPP_IPGR1.MEMORYPAGE = 2 (default, 256 bytes)
  • – DCMIPP_IPGR2.PSTART = 1 (locks IP-Plug for reconfiguration)
  • – DCMIPP_IPGR3.IDLE to wait until = 1 (to wait before continuing)
• • Video as Pipe1, statically mapped onto IP-Plug Client2:
  • – DCMIPP_IPC2R1.OTR = 3
  • – DCMIPP_IPC2R1.TRAFFIC = 4 (default)
  • – DCMIPP_IPC2R3.DPREGSTART = 512
  • – DCMIPP_IPC2R3.DPREGEND = 639
• • Display as Pipe2, statically mapped onto IP-Plug Client5:
  • – DCMIPP_IPC5R1.OTR = 11
  • – DCMIPP_IPC5R1.TRAFFIC = 4 (default)
  • – DCMIPP_IPC5R3.DPREGSTART = 0
  • – DCMIPP_IPC5R3.DPREGEND = 511
• • IP-Plug common registers:
  • – DCMIPP_IPGR2.PSTART = 0 (unlocks IP-Plug after configuration)

Configuration example 1: Single RGB dump

It is the simplest but most usual configuration, with the DCMIPP dumping to a single xRGB buffer. It has thus a single pipe active, the dump is coplanar (no side U or V planes), and to emphasize, it does not dump separate statistics (with Pipe0), nor does use the second pixel pipe (Pipe2).

All the AXI capabilities are focused on the single active pipe, which gets all the FIFO space and outstanding capability of the DCMIPP. With this setting, a camera dump of 200 Mpixel/s peak on 32 bpp (800 MB/s bandwidth) benefits of:

• • 5 Kbytes FIFO, allowing a memory peak slowdown 6 µs (a very high value).
• • 16 outstanding of 128 bytes, allowing thus an average memory latency of 2.5 µs (a high value).
• • IP-Plug common registers:
  • – DCMIPP_IPGR1.MEMORYPAGE = 2 (default, 256 bytes)
  • – DCMIPP_IPGR2.PSTART = 1 (locks IP-Plug for reconfig)
  • – DCMIPP_IPGR3.IDLE to wait until = 1 (to wait before continuing)
• • Single dump on Pipe1 (main pipe), statically mapped onto IP-Plug Client2:
  • – DCMIPP_IPC2R1.OTR = 15 (maximum assigned here)
  • – DCMIPP_IPC2R1.TRAFFIC = 4 (default, 128 bytes/burst)
  • – DCMIPP_IPC2R2.WLRU = 0 (not used as single client)
  • – DCMIPP_IPC2R2.SVCMAPPING = 0 (non-user)
  • – DCMIPP_IPC2R3.DPREGSTART = 0 (single FIFO, start at lowest)
  • – DCMIPP_IPC2R3.DPREGEND = 639 (single FIFO, end at highest)
• • IP-Plug common registers:
  • – DCMIPP_IPGR2.PSTART = 0 (unlocks IP-Plug after configuration)

Configuration example 2: All pipes active

This is the most flexible configuration of the DCMIPP: the five clients of the IP-Plug are configured to dump the maximum bandwidth they can receive).

Note that, as the bus capabilities are shared among the clients, that configuration is less resistant to memory latencies than the above configuration with only a single RGB dump.

The capabilities of the DCMIPP are shared among the clients with the following assumptions:

• • Client1 for Pipe0: dump of raw Bayer pixels, padded to 16 bpp, BW = 2 / 12
• • Client2 for Pipe1-Y: dump of ARGB pixels, on 32 bpp, BW = 4 / 12
• • Client3 for Pipe1-UV: dump of UV components, on 8 bpp, BW = 1 / 12
• • Client4 for Pipe1-V: dump of V component, on 4 bpp, BW = 1 / 12
• • Client5 for Pipe2: dump of ARGB pixels, on 32 bpp, BW = 4 / 12

The total bandwidth proportion, when all pipes are used, is 12. As some configurations (like the outstanding capability) are divided in other granularities, Pipe1-Y/RGB has some advantages, sometimes is used alone (as in the above simpler use case).

With these settings, a camera dump of 200 Mpixel/s peak on 32 bpp (800 MB/s bandwidth) benefits, for an average pipe like Pipe2, of:

• • 1.5 Kbytes FIFO, allowing a memory peak slowdown of ~2 µs
• • Four outstanding of 128 bytes, allowing thus an average memory latency of 0.65 µs, (acceptable but not so large): the DCMIPP must get priority to access the memory.

It results in the following register configuration:

• • IP-Plug common registers:
  • – DCMIPP_IPGR1.MEMORYPAGE = 2 (default, 256 bytes)
  • – DCMIPP_IPGR2.PSTART = 1 (locks IP-Plug for reconfiguration)
  • – DCMIPP_IPGR3.IDLE to wait until = 1 (to wait before continuing)
• • Client1 for Pipe0: (raw Bayer, 16 bpp) with 2 / 12 of the BW:
  • – DCMIPP_IPC1R1.OTR = 1 (2 outstanding, among 16)
  • – DCMIPP_IPC1R1.TRAFFIC = 4 (default, 128 bytes/burst)
  • – DCMIPP_IPC1R2.WLRU = 1 (2 parts of BW, among 12)
  • – DCMIPP_IPC1R2.SVCMAPPING = 0 (non-user)
  • – DCMIPP_IPC1R3.DPREGSTART = 0 (96 parts of FIFO, among 640)
  • – DCMIPP_IPC1R3.DPREGEND = 95 (96 parts of FIFO, among 640)
• • Client2 for Pipe1-Y/RGB: (ARGB, 32 bpp) with 4 / 12 of the BW:
  • – DCMIPP_IPC2R1.OTR = 5 (6 outstanding, among 16)
  • – DCMIPP_IPC2R1.TRAFFIC = 4 (default, 128 bytes/burst)
  • – DCMIPP_IPC2R2.WLRU = 3 (4 parts of BW, among 12)
  • – DCMIPP_IPC2R2.SVCMAPPING = 0 (non-user)
  • – DCMIPP_IPC2R3.DPREGSTART = 96 (224 parts of FIFO, among 640)
  • – DCMIPP_IPC2R3.DPREGEND = 319 (224 parts of FIFO, among 640)
• • Client3 for Pipe1-UV: (UV, 16 bpp) with 2 / 12 of the BW:
  • – DCMIPP_IPC3R1.OTR = 1 (2 outstanding, among 16)
  • – DCMIPP_IPC3R1.TRAFFIC = 4 (default, 128 bytes/burst)
  • – DCMIPP_IPC3R2.WLRU = 0 (1 part of BW, among 12)
  • – DCMIPP_IPC3R2.SVCMAPPING = 0 (non-user)
  • – DCMIPP_IPC3R3.DPREGSTART = 320 (64 parts of FIFO, among 640)
  • – DCMIPP_IPC3R3.DPREGEND = 383 (64 parts of FIFO, among 640)
• • Client4 for Pipe1-V: (V, 8 bpp) with 1 / 12 of the BW:
  • – DCMIPP_IPC4R1.OTR = 1 (2 outstanding, among 16)
  • – DCMIPP_IPC4R1.TRAFFIC = 4 (default, 128 bytes/burst)
  • – DCMIPP_IPC4R2.WLRU = 0 (1 part of BW, among 12)
  • – DCMIPP_IPC4R2.SVCMAPPING = 0 (non-user)
  • – DCMIPP_IPC4R3.DPREGSTART = 384 (32 parts of FIFO, among 640)
  • – DCMIPP_IPC4R3.DPREGEND = 415 (32 parts of FIFO, among 640)
• • Client5 for Pipe2: (ARGB, 32 bpp) with 4 / 12 of the BW:
  • – DCMIPP_IPC5R1.OTR = 3 (4 outstanding, among 16)
  • – DCMIPP_IPC5R1.TRAFFIC = 4 (default, 128 bytes/burst)
  • – DCMIPP_IPC5R2.WLRU = 3 (4 parts of BW, among 12)

• – DCMIPP_IPC5R2.SVCMAPPING = 0 (non-user)
• – DCMIPP_IPC5R3.DPREGSTART = 416 (224 parts of FIFO, among 640)
• – DCMIPP_IPC5R3.DPREGEND = 639 (224 parts of FIFO, among 640)
• • IP-Plug common registers:
  • – DCMIPP_IPGR2.PSTART = 0 (unlocks IP-Plug after configuration)

39.11 Shadow registers

A dump and/or pixel pipe must have the capability to be reconfigured without stopping it. Some registers can be written in the middle of a frame without impacting the actual frame acquisition. The values are stored into some shadow registers before being written in their corresponding physical registers, based on a trigger event like described in Table 362 .

Each physical register in the register map has its address increased by 0x200 respect to its own shadow register address.

Table 362. Shadow and physical registers

Table 362. Shadow and physical registers (continued)

1. Bit PIPEN of this register is not shadowed.

It is mandatory to refresh the shadow registers within a frame to prepare the context to be ready for the next start of frame, to avoid any unpredictable behavior during the acquisition. The software must then react to the following interrupt sources to launch the shadow registers accesses:

• • At VSYNC interrupt (FrameStart at start of pipe) an interrupt is generated, and the software can update the shadow registers loaded into the physical registers at the end of the current frame (so active for the next frame). The software has an entire frame period to complete the update.
• • At LINE interrupt line selection can be used also to trigger the software to update the physical registers with the new frame context to consider. The application code must ensure that the update takes place before the end of frame event (current frame). If not, the context can be partially considered from the next frame, dealing with inconsistent settings.

In any case, the software must ensure that the shadow registers are updated before the end of the current frame, to be active from the next frame. Updates started and not completed before the end of current frame result in inconsistent frame acquisition for the next frame (mixing old and new configurations).

It is recommended to change non-shadowed registers when the corresponding pipe is disabled and in an idle state (refer to Section 39.4.8: Pipe deactivation ).

39.12 DCMIPP low power modes

This section describes the behavior of the DCMIPP versus the modes listed in Table 363 .

Table 363. DCMIPP low power modes

39.13 DCMIPP interrupts

This section describes the DCMIPP interrupts, and more globally, the features that are related to real-time, namely:

• • Free-running: DCMIPP can run without CPU involvement
• • Shadow registers: relieve timing constraints from the software
• • Interrupts: source of interrupt
• • Event pins: to drive external IPs, like a DMA

39.13.1 Free-running DCMIPP

After a camera flow has been configured and established, it is able to run permanently, without any software involvement. The following features are available:

• • Permanent pipe activation, for Pipe0, 1, 2: when set, the respective pipe dumps frame data, for an unlimited amount of frames.
  • – DCMIPP_PxFCTCR.CPTMODE = 0
  • – DCMIPP_PxSR.CPTACT = 1
  • – DCMIPP_PxFCTCR.FRATE = xx (depending on desired frame subsampling rate)
• • Double buffered frames: the DCMIPP can be defined with two BaseAddress pointers for the output buffers:
  • – Software can set up the two BaseAddress pointers to let the hardware reacting to the VSYNC event to change from one buffer BaseAddress to the other one, and this alternatively.

39.13.2 Interrupts

The DCMIPP handles five interrupt pins, the fifth being the OR of the first four.

A register set is associated to each interrupt to provide the following functionality:

• • Unmasked events: reports an event that occurred and has not yet been cleared
• • Masks: to mask unwanted events
• • Masked events: not readable, any active unmasked bit triggers the interrupt pin
• • Clear register: write to clear any active event

The interrupt pins are active when high: they go high when triggered by an unmasked event, and remain high until all the masked events of that interrupt are cleared.

The interrupts lines and registers are:

1. 1. Parallel interface, with the next interrupt events handled:
   • – ERR: bad embedded synchronization detected.
     • > Unexpected behavior in the parallel interface, usually not triggered if no issues.
     • > It is due to an unexpected sensor behavior, or to a transmission error.
2. 2. Pipe0 (dump), with the next interrupt events handled:
   • – OVR: data overflow: the memory was too slow vs. received pixels.
     • > Unexpected behavior in Pipe0, usually not triggered if no issues.
     • > Same recommendation as for above OVR: skip the impacted frame.
   • – LIMIT: received volume is larger than the maximum allowed dump volume.
     • > Unexpected behavior in Pipe0, usually not triggered if no issues.
     • > The transmitted flow is too long. More space must be allocated in memory to store the transmission of a next frame.
   • – VSYNC: main interrupt, where most software can sit.
     • > Trigger: at VSync (mid blanking), permanent (even if no dump active).
     • > Typically to reconfigure slowly the pipes (at least shadow registers) for next frame
     • > Typically to trigger usage of the previously captured frame.
   • – FRAME: secondary interrupt, as backup for fast-software
     • > Trigger: after last data dump of this pipe, inactive if no capture has occurred.
     • > Typically to quickly reconfigure non-shadow registers (during only vertical blanking)
     • > Typically to trigger usage of the previously captured frame.
   • – LINE: interruption for stripe-based operators trigger: after every 1/2/4/8/16/32/64/128 dumped lines and last line. It is measured and extracted at end of pipe, close to output to memory.
     • > Typically to trigger fast/reactive software or hardware stripe-based operators.

Note: The LINE flag (LINEF bit) is set at the end of frame, to trigger out software event when the frame height is not a multiple of the selected number of lines for which the LINEF event is triggered. The last part of the frame can, in such case, have a lower number of lines than the selection. The event/interrupt is generated even if the number of lines is incomplete. The software must consider that this last event has a lower than expected number of lines when reaching the end of frame (FRAME event).

1. 3. Pipe1 (main pipe)
   • – Same events handled as Pipe0: OVR, VSYNC, FRAME, LINE, but no LIMIT as only pixels are captured after a potential crop.
2. 4. Pipe2 (ancillary pipe)
   • – Same events handled as Pipe1: OVR, VSYNC, FRAME, LINE.
3. 5. Common interrupt: groups the events of the above interrupts in a single register set and drives its own interrupt line.
   • – The fifth interrupt is designed for systems where a single driver handles the whole DCMIPP: a single access to this interrupt status allows to retrieve the status of the whole DCMIPP.
   • – The first four interrupts are designed for systems where a driver handles a pipe

and (potentially) another one drives the parallel interface. In that case, each driver handles its own individual interrupt register set, without need for semaphores to arbitrate conflicting accesses.

There is also a common interrupt that can be generated if enabled, when a transfer error is detected during an AXI transfer from the IP-Plug to memories. Error flag ATXERR in the DCMIPP_CMSR2 register is set and an interrupt can be generated to inform the software about this error status. There is no specific hardware action linked to this error, it is up to software to handle the transfer error situation.

On multiple-pipe architecture, the error informs that at least one AXI client has a transfer error, but does not identify from which client.

The DCMIPP interrupts are summarized in Table 364 . An event that generates an interrupt does it at two locations:

• – once in the common interrupt register set (when driven by a single CPU)
• – once in the local interrupt register set (when driven by multiple CPUs).

Table 364 provides the local (register and bit) and global (bit only, as all bits are in the same DCMIPP_CMSR2 register) locations.

Table 364. DCMIPP interrupts

Table 364. DCMIPP interrupts (continued)

39.13.3 Event pins

Event pins are exposed to let an ancillary HW IP, like a central DMA, to synchronize with them. The event pins are the unmasked synchronization events (VSYNC, FRAME, LINE, HSYNC) of the pipes.

An internal events generates a pulse (15 APB cycles long) on its pin.

Table 365. Event connection

39.14 DCMIPP registers

The registers are split into groups with a same prefix, as shown in Table 366 .

Table 366. DCMIPP registers organization

39.14.1 DCMIPP IP-Plug global register 1 (DCMIPP_IPGR1)

Address offset: 0x000

Reset value: 0x0000 0002

Bits 31:25 Reserved, must be kept at reset value.

Bit 24 QOS_MODE : Quality of service

Set of functions enabling to build and configure an architecture meeting bandwidth and latency requirements.

Bits 23:3 Reserved, must be kept at reset value.

Bits 2:0 MEMORYPAGE[2:0] : Memory page size, as power of 2 of 64-byte units:

0x0: 64 bytes
0x1: 128 bytes
0x2: 256 bytes
0x3: 512 bytes
0x4: 1K bytes
0x5: 2K bytes
0x6: 4K bytes
0x7: 8K bytes

39.14.2 DCMIPP IP-Plug global register 2 (DCMIPP_IPGR2)

Address offset: 0x004

Reset value: 0x0000 0000

Bits 31:1 Reserved, must be kept at reset value.

Bit 0 PSTART : Request to lock the IP-Plug, to allow reconfiguration.

0: No lock requested, IP-Plug runs on demand by background HW.

1: Lock requested: IP-Plug freezes shortly (see IDLE bit when lock is active).

PSTART must be reset to 0 after configuration is completed, to restart the IP-Plug.

39.14.3 DCMIPP IP-Plug global register 3 (DCMIPP_IPGR3)

Address offset: 0x008

Reset value: 0x0000 0001

Bits 31:1 Reserved, must be kept at reset value.

Bit 0 IDLE : Status of IP-Plug

0: IP-Plug is running (on demand by background HW)

1: IP-Plug is currently locked and can be reconfigured

IDLE is set after a request by setting PSTART at 1, and reset by resetting PSTART at 0.

39.14.4 DCMIPP IP-Plug identification register (DCMIPP_IPGR8)

Address offset: 0x01C

Reset value: 0xAA04 0314

Bits 31:24 IPPID[7:0] : IP identifier (0xAA)

Bits 23:21 Reserved, must be kept at reset value.

Bits 20:16 ARCHIID[4:0] : Architecture identifier (0x04)

Bits 15:13 Reserved, must be kept at reset value.

Bits 12:8 REVID[4:0] : Revision identifier (0x03)

Bits 7:6 Reserved, must be kept at reset value.

Bits 5:0 DID[5:0] : Division identifier (0x14)

39.14.5 DCMIPP IP-Plug Clientx register 1 (DCMIPP_IPCxR1)

Address offset: 0x020 + 0x10 * (x - 1), (x = 1 to 5)

Reset value: 0x0000 0004

Bits 31:12 Reserved, must be kept at reset value.

Bits 11:8 OTR[3:0] : Maximum outstanding transactions

0: Disabled. No outstanding transaction limitation (except via FIFO size)

1: Maximum two outstanding transactions ongoing.

...

15: Maximum 16 outstanding transactions ongoing.

Bits 7:3 Reserved, must be kept at reset value.

Bits 2:0 TRAFFIC[2:0] : Burst size as power of 2 of 8-byte units

0x0: 8 bytes

0x1: 16 bytes

0x2: 32 bytes

0x3: 64 bytes

0x4: 128 bytes

Other values: Reserved

39.14.6 DCMIPP IP-Plug Clientx register 2 (DCMIPP_IPCxR2)

Address offset: 0x024 + 0x10 * (x - 1), (x = 1 to 5)

Reset value: 0x0001 0000

Bits 31:20 Reserved, must be kept at reset value.

Bits 19:16 WLRU[3:0] : Ratio for WLRU[3:0] arbitration

A client gets a portion of the total bandwidth = Ratio(client) / Sum(all ratios)

0x0: Ratio part = 1

0x1: Ratio part = 2

...

0xF: Ratio part = 16

Bits 15:0 Reserved, must be kept at reset value.

39.14.7 DCMIPP IP-Plug Clientx register 3 (DCMIPP_IPCxR3)

Address offset: 0x028 + 0x10 * (x - 1), (x = 1 to 5)

Reset value: 0x007F 0000, 0x013F 0080, 0x018F 0140, 0x01BF 0190, 0x027F 01C0

Bits 31:26 Reserved, must be kept at reset value.

Bits 25:16 DPREGEND[9:0] : End word (AXI width = 64 bits) of the FIFO of Clientx.

The addressed word is included in the FIFO, so that next DPREGSTART is DPREGEND + 1.

Bits 15:10 Reserved, must be kept at reset value.

Bits 9:0 DPREGSTART[9:0] : Start word (AXI width = 64 bits) of the FIFO of Clientx.

39.14.8 DCMIPP parallel interface control register (DCMIPP_PRCR)

Address offset: 0x104

Reset value: 0x0000 0000

Bits 31:27 Reserved, must be kept at reset value.

Bit 26 SWAPBITS : Swap LSB vs. MSB within each received component

0: As received

1: Swapped MSB vs. LSB

Bit 25 SWAPCYCLES : Swap data (cycle 0 vs. cycle 1) for pixels received on two cycles

0: Default

1: Swap active: the data of cycle 1 is used before the data of cycle 0.

The swap must not be activated by software for pixels received in one or three cycles.

Bit 24 Reserved, must be kept at reset value.

Bits 23:16 FORMAT[7:0] :

0x1E: YUV422

0x22: RGB565

0x24: RGB888 (= YUV444)

0x2A: RAW8

0x2B: RAW10

0x2C: RAW12

0x2D: RAW14

0x4A: monochrome 8-bit

0x4B: monochrome 10-bit

0x4C: monochrome 12-bit

0x4D: monochrome 14-bit

0x5A: byte stream (JPEG, compressed video)

Other values: data are captured and output as-is only through the data/dump pipeline (for example JPEG or byte input format).

The monochrome Y input is inserted in the pipe as YUV pixels, with the U and V components set to neutral, to represent a gray color.

Bit 15 Reserved, must be kept at reset value.

Bit 14 ENABLE : Parallel interface enable

0: Parallel interface disabled to lower power consumption

1: Parallel interface enabled

The parallel interface configuration registers must be correctly programmed before enabling this bit.

Bit 13 Reserved, must be kept at reset value.

Bits 12:10 EDM[2:0] : Extended data mode

0x0: Interface captures 8-bit data on every pixel clock
0x1: Interface captures 10-bit data on every pixel clock
0x2: Interface captures 12-bit data on every pixel clock
0x3: Interface captures 14-bit data on every pixel clock
0x4: Interface captures 16-bit data on every pixel clock
Other values: Reserved

Bits 9:8 Reserved, must be kept at reset value.

Bit 7 VSPOL : Vertical synchronization polarity

This bit indicates the level on the VSYNC pin when the data are not valid on the parallel interface.

0: VSYNC active low
1: VSYNC active high

Bit 6 HSPOL : Horizontal synchronization polarity

This bit indicates the level on the HSYNC pin when the data are not valid on the parallel interface.

0: HSYNC active low
1: HSYNC active high

Bit 5 PCKPOL : Pixel clock polarity

This bit configures the capture edge of the pixel clock

0: Falling edge active
1: Rising edge active

Bit 4 ESS : Embedded synchronization select

0: Hardware synchronization data capture (frame/line start/stop) is synchronized with the HSYNC/VSYNC signals.

1: Embedded synchronization data capture is synchronized with synchronization codes embedded in the data flow.

Valid only for 8-bit parallel data. HSPOL/VSPOL are ignored when this bit is set.

Bits 3:0 Reserved, must be kept at reset value.

39.14.9 DCMIPP parallel interface embedded synchronization code register (DCMIPP_PRESCR)

Address offset: 0x108

Reset value: 0x0000 0000

Bits 31:24 FEC[7:0] : Frame end delimiter code

This byte specifies the code of the frame end delimiter. The code consists of four bytes in the form of 0xFF, 0x00, 0x00, FEC.

If FEC is programmed to 0xFF, all the unused codes (0xFF00 00XY) are interpreted as frame end delimiters.

Bits 23:16 LEC[7:0] : Line end delimiter code

This byte specifies the code of the line end delimiter. The code consists of four bytes in the form of 0xFF, 0x00, 0x00, LEC.

Bits 15:8 LSC[7:0] : Line start delimiter code

This byte specifies the code of the line start delimiter. The code consists of four bytes in the form of 0xFF, 0x00, 0x00, LSC.

Bits 7:0 FSC[7:0] : Frame start delimiter code

This byte specifies the code of the frame start delimiter. The code consists of four bytes in the form of 0xFF, 0x00, 0x00, FSC.

If FSC is programmed to 0xFF, no frame start delimiter is detected, but the first occurrence of LSC after an FEC code is interpreted as the start of frame delimiter.

39.14.10 DCMIPP parallel interface embedded synchronization unmask register (DCMIPP_PRESUR)

Address offset: 0x10C

Reset value: 0x0000 0000