![\"Green](\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/a\/af\/Green_screen_live_streaming_production_at_Mediehuset_K%C3%B8benhavn.jpg\")
<\/p>\n<\/p>\n
If you want to create videos using FFmpeg<\/a> there is a basic pipeline setup to go with. We will first take a short overview over this pipeline and then focus on each individual section.<\/p>\n <\/p>\n I’m assuming you have already captured your video\/audio data. Since this step is highly platform dependent it will not be covered in this tutorial. But there are plenty of great tutorials on this from other people: using v4l2<\/a> and using pulseaudio<\/a><\/p>\n You can pack all of these components into one process\/thread which makes handling memory a little easier and reduces copying of large memory chunks. If you are planning on using OpenGL for pixel manipulation and hardware acceleration like Quick SYNC<\/a> for encoding, it might be a good idea to isolate these steps into their own threads so they won’t mess with the rest of your program. This however makes memory handling and communication between the sections much more complicated. Also keep in mind that some of the libav*-calls (eg. sws_scale()) might be blocking.<\/p>\n If you interact with the FFmpeg API you need to use three data types:<\/p>\n As promised we will now take a closer look on each step of our processing pipeline. There will be quite a lot of code but hopefully this will help make starting with libav* a little less painful.<\/p>\n The first step in the scaling component is to set up a Sws_context<\/a>. This only has to be done once in your program. Also set the input and target resolution as well as the respective pixel formats. As mentioned earlier we have to transform the pixel format from a packed to a planar format. If you want to change the resolution use the bicubic resampling algorithm, since it produces the best image quality with okish performance. With the last parameter you can tune your resampling algorithm.<\/p>\n From now on everything has to be done on a per-image-basis, so you have to wrap the next calls in some kind of loop.<\/p>\n First we allocate an output buffer for the scaler. Because the scaler has to copy all the data anyway we don’t have to trouble ourselves with reusing buffers from the capturing process.<\/p>\n With av_image_alloc()<\/a> we can allocate the actual memory in the AVFrame-container. The buffer size alignment doesn’t seem to influence anything, even the libsws-sourcecode doesn’t give any clue as to what value to use. It could be optimization for hardware instructions on the CPU but I couldn’t find prove for that.<\/p>\n The last step is to call sws_scale()<\/a>. The source_data and source_linesize parameters are both arrays with an entry for each plane of the source image (4 in total). Because we are provided with a packed pixel format from our webcam, only the first element of the source-arrays will be set.<\/p>\n After you have passed your scaled frame to the encoder you have to free the frame yourself using av_frame_free()<\/a>.<\/p>\n Encoding<\/span><\/p>\n Now to the fun part. With a mandatory call to avcodec_register_all()<\/a> we initialize the avformat library. Afterwards we can get a handle to the codec we want to use.<\/p>\n Next we have to configure the encoder. You might want to adjust these settings according to your needs.<\/p>\n Because most of your users will watch the stream either on a computer or mobile device with a 60Hz display, you should set the framerate only to a divider of 60 to avoid stuttering.<\/strong> When using h264 you can also set codec presets<\/a>. These influence the image quality as well as the encoding time needed, with lower speeds yielding better image quality but longer encoding times and vice versa. “ultrafast”, “superfast” and “veryfast” seem to be the only presets that can keep up with 1080p 60fps while livestreaming (and they have a nice ring to it). This was tested with an Intel Core i5-6200U<\/a>.<\/p>\n With the avcodec_send_frame()<\/a> call we can send our raw frames to the encoder. This of course is also done on a per-image basis, so once again you have to wrap it in a loop. Because the encoder copies the frame to an internal buffer we can then safely free all our frame buffers.<\/p>\n A word on timestamps: simply incrementing an integer isn’t the correct way to do it, but it seems to work since it meets the monotonic-requirement of the encoder.<\/p>\n The correct (but in my case untested) solution would be to use this formula below: You basically have to increment the pts for each interval on your timebase, even if you haven’t read a frame. Therefore you could substitute the skipped_frames-variable with the number of past timebase intervals since the previous_pts.<\/p>\n To read encoded packets from your encoder you simply call the avcodec_recieve_packet()<\/a> function. Because the encoder may combine information from several input frames into one output frame, the first few calls to avcodec_recieve_packet() will not return any packets but an EAGAIN-error.<\/p>\n When ending your stream you will have to do two things:<\/p>\n This steps are usually called “draining” or “flushing” the encoder.<\/p>\n To set up the muxer we first have to set our output format. While the av_guess_format()<\/a> call doesn’t seem to be the most pretty solution it works fairly well.<\/p>\n With avformat_new_stream()<\/a> we create both an audio and a video track. You can also create subtitle tracks or add multiple audio tracks for different languages in your video. Because at playback time the decoder has to know which codec has been used to encode the tracks we have to embed this information in our output format. The IDs for the codecs can be found here<\/a>. Because avcodec_parameters_from_context()<\/a> sets only codec-specific settings we have to set the timebase and framerate of our tracks manually.<\/p>\n The muxer has to know where to write the resulting bytestream. Therefore we must use an IO-context. You can get this by either using the avio_open2()<\/a>-function or by creating your own custom context. In any case the muxer will handle calling these functions, you don’t have to worry about that. Since I wanted to write the output to an unix domain socket I had to use a context with a custom write-callback. If you want to read more about custom io, here<\/a> is a tutorial.<\/p>\n First we have to set up a buffer for the bytestream which we then provide to avio_alloc_context()<\/a>. The third parameter sets the buffer to be writeable (0 if you want read-only). The fourth parameter can be used to pass custom data to the IO-functions. The last three parameters are the functions for reading input (not required here since we are not decoding anything), seeking (also only needed when building a player) and writing.<\/p>\n To add the IO-context to the muxer set the pb field<\/a> of the muxer context.<\/p>\n The custom writing function has the following signature. You can access the muxer’s bytestream via the buffer-parameter.<\/p>\n Before we can start to put the actual data into our output format we first have to write a header. Here you can also provide additional flags to the muxer. The av_dict_set()<\/a>-function consumes all the flags it can process from your dictionary. The “live” option tells the muxer to output frames with strictly ascending presentation timestamps and prevents the muxer from reordering frames. Also the muxer writes the entire header at the beginning of the video (wit a placeholder for the length of the video) instead of just a placeholder for the entire header.<\/p>\n With everything set up we can now send packets to our muxer. Once again this is done on a per-packet basis, a loop would do nicely here.<\/p>\n To add the packets to the correct track (audio or video) we need to add an identifying stream index to each packet. The index of the track is simply incremented for each call to avformat_new_stream().<\/p>\n Now for the timestamp-magic-part: Because some containers (e.g. Matroska) force a fixed timebase on their tracks (in this case 1\/1000) we need to scale the timestamps of each track (timebase 1\/60) to match the containers timebase. If we woulnd’t do this, decoders would play the video with a wrong framerate, which would at best look funny.The basic pipeline<\/h2>\n
video capturing --> scaling ----> encoding \\\n \\\n muxing --> ....\n \/\naudio capturing --> filtering --> encoding \/<\/pre>\n
\n
\nA list of the FFmpeg pixel formats can be found here<\/a>.<\/li>\nDatatypes<\/h2>\n
\n
\nYou can allocate an AVFrame by simply calling the provided constructor.
\nDepending on your capture technique you can reuse the structs and simply replace the pointers. In this case, you can reset your frame with av_frame_unref()<\/a> to its original state. A call to av_frame_unref() will free all your buffers. Keep in mind that this will also reset all your fields.<\/li>\n<\/ul>\nAVFrame* raw_frame = av_frame_alloc();\n\nwhile() {\n \/\/ DO STUFF\n av_frame_unref(raw_frame);\n}\n\nav_frame_free(&raw_frame);<\/pre>\n\n
\/\/ uint8* buffer contains the muxed bytestream\nvoid custom_io_write(void* opaque, uint8_t *buffer, int32_t buffer_size);<\/pre>\n
More details, more code<\/h2>\n
Resampling<\/h3>\n
int source_width = 1920, source_height = 1080;\nint target_width = 1920, target_height = 1080;\n\nstruct SwsContext* scaler = sws_getContext(\n source_width, source_height, AV_PIX_FMT_YUYV422,\n target_width, target_height, AV_PIX_FMT_YUV422P,\n SWS_BICUBIC, NULL, NULL, NULL\n);<\/pre>\n
AVFrame* scaled_frame = av_frame_alloc();\n\nscaled_frame->format = AV_PIX_FMT_YUV422P;\nscaled_frame->width = target_width; \nscaled_frame->height = target_height;\n\nav_image_alloc(\n scaled_frame->data, scaled_frame->linesize, \n scaled_frame->width, scaled_frame->height, \n scaled_frame->format, 16);<\/pre>\n
sws_scale( scaler,\n (const uint8_t * const*) source_data, source_linesize,\n 0, source_height,\n scaled_frame->data, scaled_frame->linesize);<\/pre>\n
av_frame_free(&scaled_frame);<\/pre>\n
avcodec_register_all();\nAVCodec* codec = avcodec_find_encoder(AV_CODEC_ID_H264);<\/pre>\n
\n If you are using 23.976, 24, 25 or 50 frames per second there might be something wrong with your setup.<\/strong>
\n Also only use progressive scanning!<\/strong><\/p>\nAVCodecContext* encoder = avcodec_alloc_context3(codec);\n\nencoder->bit_rate = 10 * 1000 * 10000;\nencoder->width = 1920;\nencoder->height = 1080;\nencoder->time_base = (AVRational) {1,60};\nencoder->gop_size = 30;\nencoder->max_b_frames = 1;\nencoder->pix_fmt = AV_PIX_FMT_YUV422P;\n\nav_opt_set(encoder->av_codec_context->priv_data, \"preset\", \"ultrafast\", 0);\n\navcodec_open2(encoder, codec, NULL);<\/pre>\nAVFrame* raw_frame = scaled_frame; \n\nraw_frame->pts = pts++;\navcodec_send_frame(encoder, raw_frame);\n\nav_freep(&raw_frame->data[0]);\nav_frame_free(&raw_frame);\n<\/pre>\n
int64_t previous_pts = 0; \n\nraw_frame->pts = previous_pts + 1 + skipped_frames;\nprevious_pts = raw_frame->pts;<\/pre>\n
AVPacket encoded_frame; \nint got_output = avcodec_receive_packet(encoder, &encoded_frame);\n\nif(got_output == 0) {\n \/\/ yeah :)\n}\n<\/pre>\n\n
Muxing<\/h3>\n
AVFormatContext* muxer = avformat_alloc_context();\n\nmuxer->oformat = av_guess_format(\"matroska\", \"test.mkv\", NULL);\n\nAVStream* video_track = avformat_new_stream(muxer, NULL);\nAVStream* audio_track = avformat_new_stream(muxer, NULL);\nmuxer->oformat->video_codec = AV_CODEC_ID_H264;\nmuxer->oformat->audio_codec = AV_CODEC_ID_OPUS;\n\navcodec_parameters_from_context(video_track->codecpar, encoder); \nvideo_track->codecpar->codec_type = AVMEDIA_TYPE_VIDEO;\n\nvideo_track->time_base = (AVRational) {1,60};\nvideo_track->avg_frame_rate = (AVRational) {60, 1};<\/pre>\nint avio_buffer_size = 4 * KB;\nvoid* avio_buffer = av_malloc(avio_buffer_size);\n\nAVIOContext* custom_io = avio_alloc_context (\n avio_buffer, avio_buffer_size,\n 1,\n (void*) 42,\n NULL, &custom_io_write, NULL);\n \nmuxer->pb = custom_io;\n<\/pre>\n
int custom_io_write(void* opaque, uint8_t *buffer, int32_t buffer_size);<\/pre>\n
AVDictionary *options = NULL;\nav_dict_set(&options, \"live\", \"1\", 0);\navformat_write_header(muxer, &options);<\/pre>\n
\nThis has to be done both for the presentation and decoding time stamps. Because the documentation is very vague on how to use this function: av_rescale_q()<\/a> first expects the timebase of your track (1\/60) and then the target timebase of your container (1\/1000).<\/p>\n