If you've ever used an application such as Adobe's After Effects, you'll know how much creative potential there is adding and animating filters to video files. If you've worked with Apple's Core Image framework, you may well have added filters to still images or even live video feeds, but working with video files and saving the results back to a device isn't a trivial coding challenge.
Well, my VideoEffects app solves that challenge for you: VideoEffects allows a user to open a video file, apply a Core Image Photo Effects filter and write the filtered movie back to the saved photos album.
VideoEffects Overview
The VideoEffects project consists of four main files:- VideoEffectsView: this is the main user interface component. It contains an image view and a control bar.
- VideoEffectsControlPanel: Contains a scrubber bar, filter selection and play, pause, load and save buttons.
- FilteredVideoVendor: Vends filtered image frames
- FilteredVideoWriter: Writes frames from the vendor to the file system
The first action a user needs to take is to press "load" in the bottom left of the screen. This opens a standard image picker filtered for the movie media type. Once a movie is opened, it's displayed on the screen where the user can either play/pause or use the slider as a scrub bar. If any of the filters are selected, the save button is enabled which will save a filtered version of the video back to the file system.
Let's look at the vendor and writer code in detail.
Filtered Video Vendor
The first job of the vendor class is to actually open a movie from a URL supplied by the "load" button in the control panel: func openMovie(url: NSURL){
player = AVPlayer(URL: url)
guardlet player = player,
currentItem = player.currentItem,
videoTrack = currentItem.asset.tracksWithMediaType(AVMediaTypeVideo).firstelse {
fatalError("** unable to access item **")
}
currentURL = url
failedPixelBufferForItemTimeCount = 0
currentItem.addOutput(videoOutput)
videoTransform = CGAffineTransformInvert(videoTrack.preferredTransform)
player.muted = true
}
There are a few interesting points here: firstly, I reset a variable named failedPixelBufferForItemTimeCount - this is a workaround for what I think is a bug in AVFoundation with videos that would occasionally fail to load with no apparent error. Secondly, to support both landscape and portrait videos, I create an inverted version of the video track's preferred transform.
The vendor contains a CADisplayLink which invokes step(_:):
func step(link: CADisplayLink) {
guardlet player = player,
currentItem = player.currentItemelse {
return
}
let itemTime = videoOutput.itemTimeForHostTime(CACurrentMediaTime())
displayVideoFrame(itemTime)
let normalisedTime = Float(itemTime.seconds / currentItem.asset.duration.seconds)
delegate?.vendorNormalisedTimeUpdated(normalisedTime)
if normalisedTime >= 1.0
{
paused = true
}
}
With the CADisplayLink, I calculate the time for the AVPlayerItem based on CACurrentMediaTime. The normalised time (i.e. between 0 and 1) is calculated by dividing the player item's time by the assets duration, this is used by the UI components to set the scrub bar's position during playback. Creating a CIImage from the movie's frame at itemTime is done in displayVideoFrame(_:):
func displayVideoFrame(time: CMTime) {
guardlet player = player,
currentItem = player.currentItemwhere player.status == .ReadyToPlay&& currentItem.status == .ReadyToPlayelse {
return
}
ifvideoOutput.hasNewPixelBufferForItemTime(time) {
failedPixelBufferForItemTimeCount = 0
var presentationItemTime = kCMTimeZero
guardlet pixelBuffer = videoOutput.copyPixelBufferForItemTime(
time,
itemTimeForDisplay: &presentationItemTime) else {
return
}
unfilteredImage = CIImage(CVImageBuffer: pixelBuffer)
displayFilteredImage()
}
elseiflet currentURL = currentURLwhere !paused {
failedPixelBufferForItemTimeCount += 1
iffailedPixelBufferForItemTimeCount> 12 {
openMovie(currentURL)
}
}
}
Before copying a pixel buffer from the video output, I need to ensure one is available. If that's all good, it's a simple step to create a CIImage from that pixel buffer. However, if hasNewPixelBufferForItemTime(_:) fails too many times (12 seems to work), I assume AVFoundation has silently failed and I reopen the movie.
With the populated CIImage, I apply a filter (if there is one) and return the rendered result back to the delegate (which is the main view) to be displayed:
func displayFilteredImage() {
guardlet unfilteredImage = unfilteredImage,
videoTransform = videoTransformelse {
return
}
let ciImage: CIImage
iflet ciFilter = ciFilter {
ciFilter.setValue(unfilteredImage, forKey: kCIInputImageKey)
ciImage = ciFilter.outputImage!.imageByApplyingTransform(videoTransform)
}
else {
ciImage = unfilteredImage.imageByApplyingTransform(videoTransform)
}
let cgImage = ciContext.createCGImage(
ciImage,
fromRect: ciImage.extent)
delegate?.finalOutputUpdated(UIImage(CGImage: cgImage))
}
The vendor can also jump to a specific normalised time. Here, rather than relying on the CACurrentMediaTime, I create a CMTime and pass that to displayVideoFrame(_:):
func gotoNormalisedTime(normalisedTime: Double) {
guardlet player = playerelse {
return
}
let timeSeconds = player.currentItem!.asset.duration.seconds * normalisedTime
let time = CMTimeMakeWithSeconds(timeSeconds, 600)
player.seekToTime(
time,
toleranceBefore: kCMTimeZero,
toleranceAfter: kCMTimeZero)
displayVideoFrame(time)
}
Filtered Video Writer
Writing the result is not the simplest coding task I've ever done. I'll explain the highlights, the full code is available here.The writer class exposes a function, beginSaving(player:ciFilter:videoTransform:videoOutput) which begins the writing process.
Writing is actually done to a temporary file in the documents directory and given a file name based on the current time:
let urls = NSFileManager
.defaultManager()
.URLsForDirectory(
.DocumentDirectory,
inDomains: .UserDomainMask)
videoOutputURL = documentDirectory
.URLByAppendingPathComponent("Output_\(timeDateFormatter.stringFromDate(NSDate())).mp4")
do {
videoWriter = tryAVAssetWriter(URL: videoOutputURL!, fileType: AVFileTypeMPEG4)
}
catch {
fatalError("** unable to create asset writer **")
}
The next step is to create an asset writer input using H264 and of the correct size:
let outputSettings: [String : AnyObject] = [
AVVideoCodecKey: AVVideoCodecH264,
AVVideoWidthKey: currentItem.presentationSize.width,
AVVideoHeightKey: currentItem.presentationSize.height]
guardvideoWriter!.canApplyOutputSettings(outputSettings, forMediaType: AVMediaTypeVideo) else {
fatalError("** unable to apply video settings ** ")
}
videoWriterInput = AVAssetWriterInput(
mediaType: AVMediaTypeVideo,
outputSettings: outputSettings)
The video writer input is added to an AVAssetWriter:
videoWriterInput = AVAssetWriterInput(
mediaType: AVMediaTypeVideo,
outputSettings: outputSettings)
ifvideoWriter!.canAddInput(videoWriterInput!) {
videoWriter!.addInput(videoWriterInput!)
}
else {
fatalError ("** unable to add input **")
}
The final set up step for initialising is to create a pixel buffer adaptor:
let sourcePixelBufferAttributesDictionary = [
String(kCVPixelBufferPixelFormatTypeKey) : Int(kCVPixelFormatType_32BGRA),
String(kCVPixelBufferWidthKey) : currentItem.presentationSize.width,
String(kCVPixelBufferHeightKey) : currentItem.presentationSize.height,
String(kCVPixelFormatOpenGLESCompatibility) : kCFBooleanTrue
]
assetWriterPixelBufferInput = AVAssetWriterInputPixelBufferAdaptor(
assetWriterInput: videoWriterInput!,
sourcePixelBufferAttributes: sourcePixelBufferAttributesDictionary)
We're now ready to actually start writing. I'll rewind the player to the beginning of the movie and, since that is asynchronous, call writeVideoFrames in the seek completion handler:
player.seekToTime(
CMTimeMakeWithSeconds(0, 600),
toleranceBefore: kCMTimeZero,
toleranceAfter: kCMTimeZero)
{
_inself.writeVideoFrames()
}
writeVideoFrames writes the frames to the temporary file. It's basically a loop over each frame, incrementing the frame with each iteration. The number of frames is calculated as:
let numberOfFrames = Int(duration.seconds * Double(frameRate))
There was an intermittent bug where, again, hasNewPixelBufferForItemTime(_:) failed. This is fixed with a slightly ugly sleep:
NSThread.sleepForTimeInterval(0.05)
In this loop, I do something very similar to the vendor: convert a pixel buffer from the video output to a CIImage, filter and render it. However, I'm not rendering to a CGImage for display, I'm rendering back to a CVPixelBuffer to append to the asset write pixel buffer. The pixel buffer adaptor has a pixel buffer pool I take pixel buffers from which are passed to the Core Image context as a render target:
ciFilter.setValue(transformedImage, forKey: kCIInputImageKey)
var newPixelBuffer: CVPixelBuffer? = nil
CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pixelBufferPool, &newPixelBuffer)
self.ciContext.render(
ciFilter.outputImage!,
toCVPixelBuffer: newPixelBuffer!,
bounds: ciFilter.outputImage!.extent,
colorSpace: nil)
transformedImage is the filtered CIImage rotated based on the original assets preferred transform.
Now that the new pixel buffer contains the rendered filtered image, it's appended to the pixel buffer adaptor:
assetWriterPixelBufferInput.appendPixelBuffer(
newPixelBuffer!,
withPresentationTime: presentationItemTime)
The final part of the loop kernel is to increment the frame:
currentItem.stepByCount(1)
Once I've looped over each frame, the video write input is marked as finished and the video writer's finishWritingWithCompletionHandler(_:) is invoked. In the completion handler, I rewind the player back to the beginning and copy the temporary video into the saved photos album:
videoWriter.finishWritingWithCompletionHandler {
player.seekToTime(
CMTimeMakeWithSeconds(0, 600),
toleranceBefore: kCMTimeZero,
toleranceAfter: kCMTimeZero)
dispatch_async(dispatch_get_main_queue()) {
UISaveVideoAtPathToSavedPhotosAlbum(
videoOutputURL.relativePath!,
self,
#selector(FilteredVideoWriter.video(_:didFinishSavingWithError:contextInfo:)),
nil)
}
...and once the video is copied, I can delete the temporary file:
func video(videoPath: NSString, didFinishSavingWithError error: NSError?, contextInfo info: AnyObject)
{
iflet videoOutputURL = videoOutputURLwhereNSFileManager.defaultManager().isDeletableFileAtPath(videoOutputURL.relativePath!)
{
try! NSFileManager.defaultManager().removeItemAtURL(videoOutputURL)
}
assetWriterPixelBufferInput = nil
videoWriterInput = nil
videoWriter = nil
videoOutputURL = nil
delegate?.saveComplete()
}
Easy!