How I Built a Screen Recorder in Native Swift

The screen recorder in Macsential is about 3,300 lines of Swift across two packages: RecorderKit, which captures, and EditorKit, which turns a capture into a video. Neither has any UI code. This post is about the decisions that shaped them and the AVFoundation errors that cost me the most time.

Screen recording was the first feature on Macsential's plan, ahead of the image, window and cleanup tools. The first recorder commit is dated October 16, 2026, and version 0.1.0 shipped the next day with recording, the editor and export to MP4, MOV and GIF. Most of what follows was settled in the plan before any code was written, so where I give a reason, it comes from that plan or from the code.

Record raw tracks, render the edit later

The first decision was the one everything else follows from: nothing is baked in while recording. No zoom, no background, no camera bubble, and not even the mouse pointer. A recording is a directory with a .msrec extension (Finder shows it as one file):

Recording 2026-11-03 at 10.15.42.msrec/
├── project.json     capture manifest: source, tracks, each track's host start time
├── edit.json        the edit, written by the editor
├── screen.mov       HEVC, native Retina pixels, pointer not drawn
├── camera.mov       H.264, if the camera is on
├── mic.mov          AAC
├── system.mov       AAC
├── events.jsonl     pointer moves, clicks, pointer-image changes
├── cursors/         pointer images, one PNG per distinct image
└── recovery.json    only exists while recording

There were three reasons. Every effect stays editable forever, because the raw tracks are never rewritten; the edit lives in edit.json, apart from project.json. The recording process only has to capture and hand frames to the hardware encoder, so it stays light while you're doing whatever you're recording. And capturing at native Retina resolution leaves enough pixels to zoom in sharply later.

The plan made that an acceptance gate ("Gate A"): low CPU while recording, a playable file after a force quit, and recording that keeps going with the main window closed.

The cost is that the editor has to do all the work at playback time, every frame. More on that below.

One file per track, fragmented every two seconds

ScreenCaptureKit hands me screen frames, system audio and the microphone from one SCStream, and the camera comes from a separate AVCaptureSession. Each source gets its own TrackWriter, which wraps an AVAssetWriter writing its own QuickTime file. The writer is created lazily from the first sample, so its format always matches what the source actually sends:

// TrackWriter.setUp (simplified)
let writer = try AVAssetWriter(outputURL: url, fileType: .mov)
writer.movieFragmentInterval = CMTime(seconds: 2, preferredTimescale: 600)
// ... add a video (HEVC/H.264) or AAC input sized from the first sample's format
writer.startWriting()
writer.startSession(atSourceTime: firstSample.presentationTimeStamp)
onFirstSample?(name, firstSample.presentationTimeStamp.seconds)

movieFragmentInterval is the crash-safety feature. A normal .mov gets its index (the moov atom) written at the end, so if the process dies before finishWriting, the file is unreadable. A fragmented movie writes a fragment every two seconds, so a crash, a force quit or a kernel panic loses at most the last couple of seconds of each track.

The other half is metadata. While recording, the same manifest that becomes project.json is kept in recovery.json. On a clean stop the session writes project.json and deletes recovery.json. On the next launch, any bundle that still has a recovery.json was interrupted, and recovering it is almost nothing:

public static func recover(_ bundle: URL) throws {
    let recovery = bundle.appending(path: RecordingManifest.recoveryFile)
    var manifest = try RecordingManifest.read(from: recovery)
    manifest.finished = true
    manifest.recovered = true
    try manifest.write(to: bundle.appending(path: RecordingManifest.projectFile))
    try FileManager.default.removeItem(at: recovery)
}

The tracks are already playable, so there is nothing to repair. To test it, the debug build has a launch flag, --crash-after 10, that SIGKILLs the app ten seconds into a recording.

This was designed in from the start rather than added after a lost recording. The technical plan, written before the recorder, already called for fragmented files with a two-second interval and a recovery manifest, and the first spike was defined as "force-quit at minute 10: is the file playable?"

A smaller detail in the same writer: ScreenCaptureKit sends no frames while the screen is static, so a screen track could end well before the microphone track. On stop, every writer ends its session at the same host time with endSession(atSourceTime:), holding the last frame, so every track spans the whole recording.

Keeping everything on the host clock

Separate files only work if they can be lined up again. Every track is timed on the host clock (CMClockGetHostTimeClock()), and the first sample's time of each track is stored in the manifest as hostStart. The editor never assumes tracks start together; it asks for the offset:

/// Offset of each track's start from the screen track's start, in seconds.
public func offset(of track: String) -> Double? {
    guard let start = tracks[track]?.hostStart, let zero = tracks["screen"]?.hostStart else { return nil }
    return start - zero
}

ScreenCaptureKit's samples are already on the host clock. The camera's are not: AVCaptureSession timestamps on its own synchronizationClock. So every camera frame is converted before it's written:

// CameraRecorder.captureOutput (simplified)
let hostPTS = CMSyncConvertTime(sample.presentationTimeStamp,
                                from: session.synchronizationClock!, to: CMClockGetHostTimeClock())
var timing = CMSampleTimingInfo(duration: sample.duration, presentationTimeStamp: hostPTS,
                                decodeTimeStamp: .invalid)
CMSampleBufferCreateCopyWithNewTiming(allocator: kCFAllocatorDefault, sampleBuffer: sample,
                                      sampleTimingEntryCount: 1, sampleTimingArray: &timing,
                                      sampleBufferOut: &retimed)
writer.append(retimed)

The pointer log uses the same clock. EventRecorder samples NSEvent.mouseLocation 120 times a second and writes lines like {"t":81234.51,"type":"move","x":812,"y":440}, plus clicks from a global NSEvent monitor (mouse buttons don't need Accessibility permission) and a cursor event whenever the pointer image changes. The image is identified by a hash of its pixels and saved once to cursors/. Disk writes and the fsync each second happen on a background queue so the 120 Hz sampler on the main thread never waits on the disk.

Pause works the same way across all of them. A shared PauseClock drops samples that fall inside a pause and moves later ones back by the total time paused, in every track writer and the event log, so the files have no gaps and stay in sync.

The plan's acceptance target is that the camera and microphone stay within one frame of the screen after a 30-minute recording. msrec-inspect prints each track's offset so that can be checked on any bundle; the 30-minute sync check on real hardware is still listed as open in the plan's status.

The camera's dark first frames

A webcam's first frames are dark while its exposure settles, and a recording that opens the camera when it starts begins with them. The fix opens the camera during the countdown (prepareCamera()), and CameraRecorder treats it as settled once auto exposure and white balance are done, but at least 0.5 s and at most 1.5 s after its first frame. Even then a camera can send pure black frames when it reconfigures, so until the first real picture is written, frames whose sampled luma is all at video black (16) are skipped. The preview layer is attached when the session is created, because attaching it later reconfigures the running camera and produces exactly those black frames.

Camera recording was in the first commits on October 16; this fix landed a week later, on October 23, in a commit whose summary is "camera starts exposed".

One compositor for preview and export

Because nothing is baked in, every frame of the editor's preview is drawn from scratch: background, the screen inset with rounded corners and a shadow, the pointer redrawn from the event log, captions, then the camera. I didn't want two renderers that could drift apart, so there is exactly one, RenderScene.render(screen:camera:sourceTime:outputSize:), built from Core Image filters. It's called from a custom AVVideoCompositing class, EditCompositor, and that compositor is set on the AVVideoComposition used by the preview's AVPlayerItem, by AVAssetExportSession for MP4 and MOV, and by AVAssetImageGenerator for GIF frames. What you see in the editor is what you get in the file.

EditTimeline builds the AVMutableComposition: for each kept segment of the edit, it inserts that range of the screen track, inserts the matching range of the camera and audio tracks shifted by their offsets, and scales the range if the segment is sped up. Each segment becomes one RenderInstruction that knows where the segment starts in source time, so the compositor can map back:

let sourceTime = instruction.sourceStart
    + (request.compositionTime - instruction.timeRange.start).seconds * instruction.speed

Everything that depends on time (zoom, pointer position, clicks, captions) is looked up by source time, which is why cuts and speed changes never shift a zoom off the moment it was placed on.

Zoom is a pure function, ZoomCurve.state(at:zooms:pointer:). Each zoom segment eases in and out over 0.6 s with a critically damped spring curve, and the focus point is clamped so the magnified window never leaves the screen:

static func ease(_ x: Double) -> Double {
    let omega = 7.0
    func spring(_ t: Double) -> Double { 1 - (1 + omega * t) * exp(-omega * t) }
    return spring(min(max(x, 0), 1)) / spring(1)
}

The zoom is applied to the screen and pointer at the source's native resolution before scaling down to the output size, so a 2x zoom on a Retina capture is still sharp. A follow-pointer zoom uses the pointer position averaged over the last half second, so it glides instead of twitching with every small mouse movement.

Making this fast enough to scrub took a dedicated round of work. Decoded frames now arrive in their native 4:2:0 format and Core Image converts them while composing, instead of a separate BGRA conversion of every frame. Frames render through CIRenderDestination without blocking: a second queue waits for the GPU and finishes requests in order, so the next frame is composed while the current one draws. The background, which never changes over time, is rendered once per output size and cached. And cancelAllPendingVideoCompositionRequests finishes queued requests as cancelled, so dragging the playhead doesn't render every frame it passes over.

The same pass took work off the main actor elsewhere. Every edit used to rebuild the scene and re-read all of events.jsonl with one JSONSerialization call per line; now the pointer log and images load once per editor window, off the main actor, decoded as a single JSON array with a line-by-line fallback for a last line cut off by a crash. In the recorder, the pointer image used to be identified by a TIFF encoding ten times a second; now it's a hash of its pixels, skipped entirely while NSCursor returns the same image.

AVFoundation errors that cost me time

-12780: the asset was gone

EditTimeline loads the camera and audio tracks in a helper that returns the AVAssetTrack to insert later. If the helper returns only the track, inserting it into the composition fails with -12780 and no further explanation. An AVAssetTrack references its AVAsset weakly; once the helper returned, nothing held the AVURLAsset, it was deallocated, and the track became useless. The fix is to keep the asset next to the track:

private struct Source {
    let asset: AVURLAsset      // keep alive: an AVAssetTrack only references its asset weakly
    let track: AVAssetTrack
    let offset: Double
    let range: CMTimeRange
    let target: AVMutableCompositionTrack
}

The Source struct, with the asset kept next to the track, has been there since the first EditorKit commit, with a comment on the field saying why it has to stay.

Asset duration is not track duration

Related: clipping an insert to asset.duration sometimes asked for time past the last sample, because the asset duration is rounded. Inserts are clipped to the track's own timeRange instead.

-11838: an empty composition track

Each audio and camera track gets an AVMutableCompositionTrack up front, then segments are inserted into it. If none of the kept segments overlap that track (say, a short microphone track and an edit that keeps only the end), the composition track stays empty, and the export fails with -11838. Empty tracks are removed before the composition is used:

// An empty track makes export fail, e.g. when every segment falls outside a short audio file.
for source in audioSources where source.target.segments.isEmpty {
    composition.removeTrack(source.target)
}

The camera track gets the same check, and its track ID is only passed to the compositor if it survived.

Tools that helped

Two small command-line targets live in the same package. msrec-inspect prints a health report for a bundle: each track's codec, duration, offset from the screen track, dropped samples and errors, plus event counts. The app also writes that report into each bundle as report.txt. msrec-export renders a bundle to MP4, MOV or GIF from the terminal using the same exporter and edit.json. Both make it possible to reproduce a capture or export problem without opening the app.

Because the capture is raw and the edit is separate, some features can be added later without changing the file format. Clicks are already in events.jsonl, so automatic zoom on clicks, which the plan lists as still open, can be computed from recordings made today. The same goes for hiding the camera during chosen ranges: it's a new field in edit.json, and older files decode with defaults.

This recorder ships as Record Screen in Macsential, a Mac utility app I'm building on my own.