HANDY TIP: Using FFprobe for stream analysis

[Rodney Baker]

Use sed/awk to massage the data into a suitable format for graphing, then use gnuplot to produce the output. Once that it settled, tie it all together using a shell script/python/perl or your language of choice. All the tools to do it exist already – why reinvent the wheel?

[Michael Rampe]

Thanks Rodney.

I will investigate.

Michael

“half-way to world domination A.K.A. the belligerent blue bike shed”

[Michael Rampe]

Hi Rodney,

I managed to get gnuplot to show the data after creating the following datafile manually from the data in my previous post.

0 822.000
1 219.000
2 515.000
3 942.000
5 2292.000
4 3896.000
7 5546.000

I have had a quick look into sed and awk. Looks tricky. I will keep investigating.

Michael

“half-way to world domination A.K.A. the belligerent blue bike shed”

[Rodney Baker]

Hi Michael. The syntax for sed and awk can be tricky, and they can both do pretty advanced stuff. I think awk may be the more appropriate tool for this, although it could be done directly using either Python or Perl too (but I’m not a programmer so I’m not the right person to ask about that).

Mind you, I’m no expert on awk or sed either, but I know that you can use either one to achieve what you want. Sed (Stream EDitor) is designed for line-by-line string editing/replacement, whereas awk is more powerful and probably more suited to parsing the data output from ffprobe and rewriting it into the format that you need for gnuplot. That is my understanding, anyway.

[Michael Rampe]

[Rodney Baker] “(but I’m not a programmer so I’m not the right person to ask about that).”

Me either;-)

I did get some good results just then with:
./ffprobe -show_frames two_pass.mp4 | grep ‘size\|coded_picture_number’ > raw.dat && paste -s -d ‘\t\n’ raw.dat > fixed.dat && sed -e ‘s/coded_picture_number=//g’ -e ‘s/size=//g’ fixed.dat > column.dat && gnuplot plot1

1. FFprobe lists all frame details
2. grep strips all lines except size and picture number and writes to a file
3. paste combines lines two by two from previous file to new file
4. sed takes the new file and strips the unnecessary text and creates a data file
5. gnuplot plots the datafile to image output

(the gnuplot “plot1” file contains:
set yrange [0:15000]
set ytics (15000, 10000, 5000)
set xrange [0:200]
set lmargin 9
set rmargin 2
plot ‘column.dat’ using 1:2 notitle with lines)

Not very elegant but it works as a proof of concept.

Thanks for your nudges in the right directions.

Michael

“half-way to world domination A.K.A. the belligerent blue bike shed”

[Michael Rampe]

After a bit of tinkering, I think I have got the result I was looking for. Open source bitrate graphing with frame type identification. The following two images are the results of one pass and two pass x264 encoding with FFmpeg. IPB frames are identified by colour. NOTE: As labelled, the y dimension is measured in Bytes per frame.

For context or interest, this is the test video that I have used for this experiment. (I created it with Blender, another great open source program;-)

I chose/created this video specifically for its inherently variable bitrate structure.

For the unix minded, how I did it?

Let me know if you get stuck;-)

# SOFTWARE NEEDED (all open source)
FFmpeg (>SVN-r23145), FFprobe FFprobe (>SVN-r92, currently being merged into FFmpeg as of time of posting) gnuplot (>4.2), grep, sed.

# COMMAND LINE
./ffprobe -show_frames funnyhq.mp4 | grep ‘size\|coded_picture_number\|pict_type’ > raw.dat && paste -s -d ‘\t\t\n’ raw.dat > fixed.dat && sed -e ‘s/coded_picture_number=//g’ -e ‘s/size=//g’ -e ‘s/pict_type=//g’ -e ‘s/I/167116800/g’ -e ‘s/P/65280/g’ -e ‘s/B/255/g’ fixed.dat > column.dat && gnuplot plot.txt

# GNUPLOT “plot.txt”
set title “1pass”
set xlabel “frame number\n\n./ffmpeg -i funny_bubbles.mov -vcodec libx264 -vpre hq -b 1500k funnyhq.mp4”
set ylabel “Bytes per frame”
set yrange [0:35000]
set ytics (35000, 30000, 25000, 20000, 15000, 10000, 5000)
set xrange [-10:850]
set lmargin 12
set rmargin 2
set grid
set pointsize 2
set label 1 “I frames”
set label 1 at graph .85, .96 tc lt 1
set label 2 “P frames”
set label 2 at graph .85, .92 tc lt 2
set label 3 “B frames”
set label 3 at graph .85, .88 tc lt 3
plot ‘column.dat’ using 2:3:1 notitle with i lc rgb variable

Now, to work out how to write a sh script to make it easier.

Michael

“half-way to world domination A.K.A. the belligerent blue bike shed”

[Michael Rampe]

…and here are the two other rate control methods: -crf and -cqp

seems that constant bitrate might not be possible. makes sense in a way.

Michael

“half-way to world domination A.K.A. the belligerent blue bike shed”

[John Van kemenade]

pretty impressive huh!
Can you point me to a win32 binary of this version of FFprobe.
FFprobe that is packed with the daily builds of FFmpeg does not contain the -show_frames option.

John

[Michael Rampe]

Hi John,

I used svn to get the latest version.

$ svn co https://ffprobe.svn.sourceforge.net/svnroot/ffprobe/trunk ffprobe

This pulled down SVN-R92 of FFprobe.

Not sure about any issues specific to windows but this version definitely does support the -show_frames option.

I am also currently working on a real-time encoding analyser with live graph using the -vstats option in FFmpeg. Worth looking into.

Michael

“half-way to world domination A.K.A. the belligerent blue bike shed”

[Kathija Naseem]

Hi,
Is it possible to use the same tool for files with asf formats. Because i tried with asf on Windows. I was able to parse it but I am not getting the size of each frame.

Kindly help