How to Use FFmpeg Filters to Jazz Up Your Audio and Video Files

Introduction

You may have come across the FFmpeg utility before. It is that open-source command-line utility that can perform simple multimedia format conversions such as these:

ffmpeg -i video.avi video.mp4

ffmpeg -i audio.wav audio.mp3

FFmpeg is smart enough to pre-empt your intentions. For example, if you want to extract the audio from a video file, you do not have to strain yourself with details about codecs and stuff. FFmpeg will automatically choose the best settings:

ffmpeg -i video.mp4 audio.mp3

A lot more FFmpeg functionality is in the form of filters. The syntax for FFmpeg filters is obscure and the documentation is a bit cryptic. So, most people avoid them. Not anymore 'cos I'll learn ya.

Background

For several years, I did not bother with FFmpeg filters. The only filter I used was the volume filter, which I used (blindly) to amplify low-volume audio files. I had even written a FFmpeg tips-and-tricks article in a magazine without knowing any better. Last year, in the midst of the lockdown, I decided to write a follow-up for the article — sort of 'More FFmpeg tips and tricks'. When I went in depth about FFmpeg, I decided to write a book instead. It is now a 152-page full-colour paperback on Amazon - FFMPEG Quick Hacks. In the spirit of open source, I have made its ebook version free. The book has a full tutorial and is not just a collection of hacks. In fact, the tutorial forms much of this book. The hacks part is just one big chapter. I did not realize my mistake until it was too late.

For the book and this article, I used Linux. If you are on Windows, you need to replace the slashes (\) in multiline commands with carets.

FFmpeg Filter Syntax

FFmpeg has lots of filters - more than you can imagine. They are listed and described in detail at this link. Each one of these filters represents a processing phase that can be used individually or daisy-chained with other such filters. One or more multimedia streams (audio, video, subtitles) can be provided as input for the filter. Depending on the kind of processing done by the filter, it might have one or more output multimedia streams. The output from one filter can be used as the input for a subsequent filter.

Here is a simple example of a filter. It increases the volume of the audio stream by a third.

ffmpeg -i low.mp3 -af "volume=1.33" high.mp3

When I wanted to slightly rotate a video, I used this.

ffmpeg -y -i barbara.mp4 \
       -vf rotate=angle=-20*PI/180:fillcolor=brown \
       barb20.mp4

The part in bold type in the above command is the filter definition. Most filter definitions in production may seem undecipherable and confusing but it is not rocket science. FFmpeg filters use a standard syntax:

[input label1][input label2]...[input_labelN]filter=key=value pair1:key=value 
 pair2...key=value pairN[output label1][output label2]...[output labelN]

Use the command switch -filter_complex before defining your filter. It handles all types of filters and eliminates confusing errors.

Multiple filters need to be separated by a semicolon (;). At the end of the string of filters though, there should be no semicolon. A filter may have one or more inputs and/or outputs. (Some special ones have none.) The inputs can be streams in the input files or any ad-hoc output label you specify in the previous filters. The output labels can be used as input for subsequent filters or for map assignments outside the filter definition. If the input(s) of a filter is/are same as the output(s) of the immediately previous filter, then a comma can be used instead of a semi-colon. Let me explain most of this with an example.

ffmpeg -i Tasmania-Nobrainia.mp4 \
       -filter_complex \
          "[0:a:0]atrim=start=60:end=120[trimmed]; 
           [trimmed]volume=volume=1.6[amped]" \
       -map '[amped]' \
       second-mminute-audio-amped.mp3

In this command, two filters are used — atrim and volume. The first filter is for extracting a particular section of its input audio stream. The second is for changing the volume level of its input audio stream. Before atrim, the input audio stream has been specified as [0:a:0]. This refers to the first audio stream in the first input file. The start and end key-value pairs specify the beginning and ending timestamps between which the extraction has to occur — the second minute. The extracted audio is then packaged as the output stream labelled as '[trimmed]'. This label is specified as the input for the next filter volume. The volume filter has a key-value pair, which is also named volume. This filter causes the volume of the input audio stream (labelled '[trimmed]') to be increased by two-thirds. The filter's output is labelled '[amped]'. The filter definition ends thereafter. [Note: There is no semicolon at the end of the filter definition.]

It is important that a filter only refers to existing input file streams or previously created output labels. Otherwise, FFmpeg will start making guesses. If it does not succeed, there will be errors. In the above command, the input audio stream specified as '[0:a:0]' is not really necessary. FFmpeg would have automatically chosen it. Given that the output format is an audio file, the last filter need not have labelled its output for the -map switch. FFmpeg would have automatically chosen it from the outputs of the last filter. [Note: DO NOT leave any outputs unused. The last outputs should match the streams required by the output file container (format).]

ffmpeg -i Tasmania-Nobrainia.mp4 \
       -filter_complex \
          "atrim=start=60:end=120[trimmed]; 
           [trimmed]volume=volume=1.6" \
       second-mminute-audio-amped.mp3

Going further, even the keys are not necessary if the values are in the order specified in the filter documentation.

ffmpeg -i Tasmania-Nobrainia.mp4 \
       -filter_complex \
          "atrim=60:120[trimmed]; 
           [trimmed]volume=1.6" \
       second-mminute-audio-amped.mp3

Most online examples of FFmpeg filters are like this. Without the filter documentation, filter definitions may look cryptic.

More FFmpeg Filter Examples

In some of his movies, Jim Carrey does a live rewind of a scene. Does he sound intelligible if you rewind that footage? Let us use FFmpeg filters and see.

ffmpeg -y -i ace-ventura-reverse.mp4 \
  -filter_complex \
     "[0:v:0]reverse[v]; [0:a:0]areverse[a]" \
  -map '[v]' -map '[a]' \
  ace-ventura-reverse-reversed.mp4

#Place them side-by-side
ffmpeg -y -i ace-ventura-reverse.mp4 -i ace-ventura-reverse-reversed.mp4 \
  -filter_complex  \
    "[0:v:0]pad=1280:360:0:0[frame];
     [frame][1:v:0]overlay=640:0[fullvideo];
     [fullvideo]drawtext=x=30:y=60:fontcolor=yellow:text=
     'Original audio on left speaker':fontsize=30:fontfile=Florentia.ttf[lefttext];
     [lefttext]drawtext=x=670:y=60:fontcolor=yellow:text=
     'Reversed audio on right speaker':fontsize=30:fontfile=Florentia.ttf[v];
     [0:a:0]channelsplit=channel_layout=mono[leftaudio];
     [1:a:0]channelsplit=channel_layout=mono[rightaudio];
     [leftaudio][rightaudio]join=inputs=2:channel_layout=stereo[a]" \
  -map '[v]' -map '[a]' ace-ventura-reversal-truth.mp4

The filters in the first FFmpeg command reverses the video and audio streams. [Note: These filters consume a lot of memory. Use short clips.] The filters in the second FFmpeg places the two videos side-by-side to see if they match up. Specifically, the audio stream from the original video is played on the left speaker while the audio from the reversed video is played on the second speaker. You may have to slide the audio balance of your volume control utility to either side to listen to only one of the audio streams.

Some filters do not need any input. They may use certain built-in filter variables to generate audio or video output. In this command, the testsrc filter generates a video test pattern out of nothing. The aevalsrc generates a random white noise using an algebraic equation. [Note: The -f lavfi switch allows you to use the filter output as a pseudo input file.]

ffmpeg -f lavfi \
  -i "testsrc=size=320x260[out0];aevalsrc=random(0)/20[out1]" \
  -t 0:0:30 -pix_fmt yuv420p \
  test.mp4

In the next example, the built-in filter variables iw and ih are used to resize a series of numbered images and create a slideshow video — the variables representing the actual dimensions of the current image being processed by the filter. [Note: A comma instead of a semicolon has been used to indicate that all outputs of the filter on the left are consumed by the filter on the right.]

ffmpeg -y -framerate 1/3 -i image%02d.jpg \
       -filter_complex "scale=eval=frame:w=640:h=480:force_original_aspect_ratio=decrease, 
              pad=640:480:(640-iw)/2:(480-ih)/2:yellow" \
       -pix_fmt yuv420p -framerate 24 \
       slide.mp4

You will not crack FFmpeg filters on your first attempt. I certainly did not. After I understood how to use them though, a whole new world opened up to me. There is no need to use expensive software to create various effects. FFmpeg filters can do most of them for free.

If you did not understand something in this article, then read my free ebook. It has hundreds of examples and illustrations. Everything is explained in much more detail. After that, you can use the FFmpeg documentation and go nuts with it.

History

• 21^st June, 2021: Initial version
• 22^st June, 2021: Update about the use of comma, instead of a semicolon