How To Use Pyav Or Opencv To Decode A Live Stream Of Raw H.264 Data?

The data was received by socket ,with no more shell , they are pure I P B frames begin with NAL Header(something like 00 00 00 01). I am now using pyav to decode the frames ,but i

Solution 1:

After hours of finding an answer for this as well. I figure this out myself.

For single thread, you can do the following:

rawData = io.BytesIO()
container = av.open(rawData, format="h264", mode='r')
cur_pos = 0whileTrue:
    data = await websocket.recv()
    rawData.write(data)
    rawData.seek(cur_pos)
    for packet in container.demux():
        if packet.size == 0:
            continue
        cur_pos += packet.size
        for frame in packet.decode():
            self.frames.append(frame)

That is the basic idea. I have worked out a generic version that has receiving thread and decoding thread separated. The code will also skip frames if the CPU does not keep up with the decoding speed and will start decoding from the next key frame (so you will not have the teared green screen effect). Here is the full version of the code:

import asyncio
import av
import cv2
import io
from multiprocessing import Process, Queue, Event
import time
import websockets

defdisplay_frame(frame, start_time, pts_offset, frame_rate):
    if frame.pts isnotNone:
        play_time = (frame.pts - pts_offset) * frame.time_base.numerator / frame.time_base.denominator
        if start_time isnotNone:
            current_time = time.time() - start_time
            time_diff = play_time - current_time
            if time_diff > 1 / frame_rate:
                returnFalseif time_diff > 0:
                time.sleep(time_diff)
    img = frame.to_ndarray(format='bgr24')
    cv2.imshow('Video', img)
    returnTruedefget_pts(frame):
    return frame.pts

defrender(terminated, data_queue):
    rawData = io.BytesIO()
    cur_pos = 0
    frames_buffer = []
    start_time = None
    pts_offset = None
    got_key_frame = Falsewhilenot terminated.is_set():
        try:
            data = data_queue.get_nowait()
        except:
            time.sleep(0.01)
            continue
        rawData.write(data)
        rawData.seek(cur_pos)
        if cur_pos == 0:
            container = av.open(rawData, mode='r')
            original_codec_ctx = container.streams.video[0].codec_context
            codec = av.codec.CodecContext.create(original_codec_ctx.name, 'r')
        cur_pos += len(data)
        dts = Nonefor packet in container.demux():
            if packet.size == 0:
                continue
            dts = packet.dts
            if pts_offset isNone:
                pts_offset = packet.pts
            ifnot got_key_frame and packet.is_keyframe:
                got_key_frame = Trueif data_queue.qsize() > 8andnot packet.is_keyframe:
                got_key_frame = Falsecontinueifnot got_key_frame:
                continue
            frames = codec.decode(packet)
            if start_time isNone:
                start_time = time.time()
            frames_buffer += frames
            frames_buffer.sort(key=get_pts)
            for frame in frames_buffer:
                if display_frame(frame, start_time, pts_offset, codec.framerate):
                    frames_buffer.remove(frame)
                if cv2.waitKey(1) & 0xFF == ord('q'):
                    breakif dts isnotNone:
            container.seek(25000)
        rawData.seek(cur_pos)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    terminated.set()
    cv2.destroyAllWindows()

asyncdefreceive_encoded_video(websocket, path):
    data_queue = Queue()
    terminated = Event()
    p = Process(
        target=render,
        args=(terminated, data_queue)
    )
    p.start()
    whilenot terminated.is_set():
        try:
            data = await websocket.recv()
        except:
            break
        data_queue.put(data)
    terminated.set()

Solution 2:

Its normal getting 3~4 seconds delay because you are reading encoded data and decoding it takes time via on CPU.

1. If you have GPU hardware, you can use FFMPEG to decode H264 by GPU. Here is an example.
2. If you don't have a GPU, decoding H264 on CPU always will cause delays. You can use FFMPEG for effective decoding but this will also decrease total delay almost 10%