Introduction
One day, I came across a video that was a dub of an old 8mm film we used to make as kids. The dub was made unprofessionally with no special equipment or even a tripod, so it suffered from a number of problems. The worst was sporadic non-compensated motion of the dubbed screen area. I tried to find software that would allow me to crop the moving areas dynamically but had no luck. There was no free software with such a feature, and all commercial software was way too expensive. Then I came to an idea of writing such software by myself. As a base, I chose VirtualDub because it already provided some static cropping functionality. All I needed to do was to make it dynamic, e.g., with placement of the cropped area defined as a function of time. I decided to make it as flexible as possible, so one could edit, save & restore the dynamic cropping settings, and to design it as an extensible feature.
Design
The "work horse" of the solution is a new IVDDynClippingStorage interface defined in ClippingControl.h:
class VDINTERFACE IVDDynClippingStorage : public vdrefcounted<IVDRefCount> {
public:
virtual void Init(int sourceW, int sourceH) = 0;
virtual int GetWidth() = 0;
virtual int GetHeight() = 0;
virtual VDStringW Load(wchar_t *filename, HWND parent) = 0;
virtual VDStringW Dump(wchar_t *filename, HWND parent) = 0;
virtual void Configure(HWND parent) = 0;
virtual int SetClipBounds(sint64 pos, const vdrect32& r) = 0;
virtual int GetClipBounds(sint64 pos, vdrect32& r) = 0;
virtual sint64 GetClipPos(int idx) = 0;
virtual int GetTotalClipPos() = 0;
virtual void GetCrop(vdrect32& r) = 0;
virtual void DelClipBounds(int idx) = 0;
virtual bool DoCrop(VDPixmap& dst, sint64 pos, vdrect32& crop_area) = 0;
virtual void ChangeTimeline(sint64 pos, sint64 nframes, bool add) = 0;
};
It defines a set of methods to manage the clipping map and to perform the actual dynamic cropping. The VDDynClippingStorage class defined in ClippingControl.cpp is implementing the interface. An instance of the class can be created per FilterInstance (however, a need in having more than one filter with dynamic cropping in the chain of filters is highly doubtful).
The following diagram shows relations between the components of the solution:
+--------------+ +--------------+
| | VDGetDynClippingStorage() | |
| Script.cpp +-----------+ | Job.cpp |
| | | | |
| | +--|---------------------------+ | |
| | | | | | |
| | | | ClippingControl.cpp | | |
| | | V | | |
| | Load() | +--------------------------+ | Dump() | |
| +--------->| IVDDynClippingStorage |<---------| |
| | | | +----------------------+ | | | |
+------------+-+ | | | VDDynClippingStorage | | | +--+-----------+
| | | | | | | |
SetClippingStorage() | | | | | | GetClippingStorage()
| | | | | | | |
| +------>| +----------------------+ | | |
| | | +---^-------^----------^---+ | |
| | | | | | | |
| DoCrop() | Configure() | | | |
| | | | GetClipBounds() | | |
| | | | | SetClipBounds()| |
| | | | | | | |
| | | +---|-------|----------|---+ | |
| | | | |IVDClippingControl| | | |
| | | | +-+-------+----------+-+ | | |
| | | | | VDClippingControl | | | |
| | | | | | | | |
| | | | +----------------------+ | | |
| | | +----^----------------^----+ | |
| | +------|----------------|------+ |
| | | | |
| | GetClippingStorage() SetClippingStorage() |
| | | | |
| | +------+----------------+------+ |
| | | | |
| | | Filtdlg.cpp | |
| | | | |
| | +------+----------------+------+ |
| | | | |
| | SetClippingStorage() GetClippingStorage() |
| | | | |
+--V----+------------V----------------V------------------V--+
| |
| FilterInstance.cpp |
| |
+-----------------------------------------------------------+
There are some changes made to the basic functionality of the cropping control as well. They are not strictly related to dynamic cropping, but are very handy:
- From now on, you will be able not only to stretch the cropped area by dragging its sides and corners with left mouse button, but also to move the complete cropped area using the right mouse button! Try it and you will see how useful it is for dynamic cropping when you have to go frame-by-frame adjusting the cropped area placement.
- A new "lock aspect ratio" checkbox allows the cropped area to have the aspect ratio of the original video. It is suggested to always check it with dynamic cropping enabled.
The following VirtualDub components are affected with the patch:
- Cropping control (ClippingControl.h/.cpp, FiltDlg.cpp, resources)
- Filter instance class (FilterInstance.h/.cpp)
- Saving (Job.cpp) and loading (Script.cpp) of the processing settings
The following project files are modified:
- src\virtualdub\h\clippingcontrol.h
- src\virtualdub\h\filterinstance.h
- src\virtualdub\res\resource.h
- src\virtualdub\res\virtualdub.rc
- src\virtualdub\source\clippingcontrol.cpp
- src\virtualdub\source\filtdlg.cpp
- src\virtualdub\source\filterinstance.cpp
- src\virtualdub\source\job.cpp
- src\virtualdub\source\script.cpp
Building
You should patch the source tree of VirtualDub and re-build it.
There are two ways of patching the source tree. You can either:
This may work well with versions of VirtualDub other than 1.9.9 too.
After that, run the building process as usual.
Running
Load a video, add any video filter, select "Cropping..." and check the "Dynamic cropping" checkbox in the "Filter input cropping" control. Check also "Lock aspect ratio" for better results.
Walk the video frame-by-frame inside of the cropping control and adjust the sizes and position of the cropped area accordingly to your needs. All position and size changes will be saved in the map for rendering. If you rewind the video back to the beginning and go again frame-by-frame, you will notice how the cropped area is moving to follow your adjustments! You can click "Options..." and select the desired settings.
Once done, click "OK" to close "Filter input cropping" and once again "OK" to close "Filters". Run "Preview filtered..." and watch what is happening on the left and right UI panels. Use "Save as AVI..." to save the final rendering.
Here is a result of processing that old video with the software.
Conclusion
In general, this feature can be useful for manual "motion post-compensation" and stabilization in the following cases:
- Old videos made with camcorders with no hardware motion compensation available
- Videos made in complex motion situations (in car, boat, on the run)
- Videos made by kids or impaired people
- "Noisy", dark, low-contrast videos where automatic algorithms usually fail or produce bad results
Also, it can be handy for some special video effects (digital zoom, stretching deformation, etc.).
It is very basic, but it is doing what it is supposed to do. Here are a couple of more things I wanted to share.
Currently only linear interpolation between vertices of the clipping map is implemented. One can try to implement a cubic spline-based interpolation to see if it improves the rendering results and reduces shaking. Also, one can try to implement a real motion-compensation algorithm inside of
IVDDynClippingStorage::DoCrop that will cache and analyze frames.
There is also a "no interpolation" mode available that can be used for videos that contain sets of scenes in which static cropping is required - this avoids cutting / cropping / gluing them piece-by-piece.
With linear interpolation, you usually achieve the best results with two cropping adjustment passes.
The resampling selection has three settings: original frame size, maximal cropping size and custom. In any case, the result will be resampled to the rounded sizes divisible by 16 for general compatibility with compression algorithms. An attempt to force custom sizes to values non-divisible by 16 or bigger than those of the source will result in an error.
There are two resampling algorithms built into VirtualDub - "bilinear" and "nearest". The "bicubic" is not implemented in the core of VirtualDub but instead as a plugin and therefore is not supported.
It appears to be useful to have a line grid optionally appearing on the cropped area when you slide it with the right mouse button (maybe it can appear when you press and hold the right mouse button and then the left one). It would allow to position the cropped area more precisely. I did not implement it, but left it as an idea for others.
The IVDDynClippingStorage::ChangeTimeline method is defined and implemented in VDDynClippingStorage but it is not being used for anything - it can be a future extension, just in case.
All clipping map vertex indexes are 1-based. That is, what index you see on the interface is what you have in the map.
