An rsync Crash Course

Whether you're a beginner Bash scripter or seasoned sysadmin, rsync is an indispensable tool for efficient file transfer and synchronization. It's easy to use, yet incredibly powerful, and beguiles many a script kiddie and cargo-cult programmer alike, myself included at times. While I do encourage everyone to #ReadTheDocs and open man rsync, it's less of a page and more of a novella, so let's take a bird's-eye view and look at some of the more useful options, and practical examples of their application.

Usage overview

$
rsync [option(s)] [source(s)] [destination]

Like most commands, rsync has a fairly straight-forward signature: Options follow the executable name, then a list of sources to send, with their destination at the end.

Options

  • -a, —archive, same as -rlptgoD, recursively sync source directory while preserving all attributes except hard links

  • -D , same as —devices —specials, syncs system files used to represent connected devices and special files such as shortcuts, and sockets and pipes related to inter-process communication.[1]

  • —delete, delete all files in the destination not present in the source

  • -e, —rsh=COMMAND, specify the remote shell to use

  • -g, —group, preserve group

  • -h, —human-readable, —progress

  • -l, —links, copy symlinks

  • —safe-links, ignore symlinks that point outside the directory from being copied

  • -n, --dry-run

  • -o, —owner, preserve owner

  • -p, —perms, preserve permissions

  • -r, —recursive, recurse into directories in source

  • -t, —times, preserve times

  • -u, —update, skip files with a newer timestamp at the destination

  • -v, —verbose, increase output verbosity

  • -z, —compress, compress file data during the transfer

  • —include=PATTERN, only sync files matching the given pattern

  • —exclude=PATTERN, skip files matching the given pattern

  • —include-from=FILE, read inclusion pattern from given file

  • —exclude-from=FILE, read exclusion pattern from given file

  • —files-from=FILE, read list of source-file names from given file

Sync local directories

$
rsync -avz —delete /src/directory/ /dest/directory

Note that the trailing / on a source directory indicates to copy its contents rather than the directory itself.

Sync remote directories

$
rsync -avz -e ssh /src/directory/ user@host:/dest/directory

macOS

When sending the contents of a volume from its root on macOS, save yourself some cycles and remember to exclude a few volume-specific hidden directories.[2]

$
... —-exclude={ '.fseventsd', '.Spotlight-V100', '.Trashes' }

Permissions and owners and timestamps, oh my!

For most use cases, the --archive flag works perfectly well, but not all. One such case is sending files to a directory with a different owner, for example /var/www on a web server. One probably wouldn't want to preserve ownership or group of the files being sent in that case, and would want to simply use just -r instead of -a.

Conclusion

rsync is a powerful yet easy to use file transfer and synchronization utility that has a spot in every developer's tool belt. While it is just as easy to misuse at times, having an understanding of its most common use cases will help guide you through the documentation and navigate those edge cases where --archive isn't quite the answer.

Additional Resources

References

  1. What is the -D switch in rsync? - Ask Ubuntu

  2. Rsync cheatsheet

❋❋❋

Storing Metadata in Core Data

Persistence frameworks for iOS seem to go from 0 to 100 real quick. Need to store a few booleans and maybe a string? UserDefaults will do just fine! How about a few tens of model instances? Well, you could either serialize it to a plist, à la UserDefaults, or here's an entire object graph and persistence framework backed by an SQL data store and oh by the way, it's one of the most complex Cocoa frameworks and it's called Core Data! Don't get me wrong, I really warmed up to Core Data after I found out how difficult it is to serialize a tree with just Swift's Codable protocol, however defining full-blown entities for certain trivial data makes about as much sense as a nuclear hand grenade, especially when that data is metadata related to our Core Data store, such as the timestamp of the last update or the latest commit ID. Keeping this data in a store alongside the user's preferred theme or units of measurement feels wrong, as it fragments our model layer. Luckily, Core Data comes with a baked-in solution in the form of the metadata dictionary. Each NSPersistentStore object contains a metadata property which can be read from with the NSPersistentStoreCoordinator instance method metadata(for:).

func lastUpdatedOn(_ persistentContainer: NSPersistentContainer) -> Date? {
let coordinator = persistentContainer.persistentStoreCoordinator
guard let store = coordinator.persistentStores.first else {
fatalError("Unable to retrieve persistent store")
}
let metadata = coordinator.metadata(for: store)
guard let lastUpdated: Date = metadata["lastUpdated"] as? Date else {
return nil
}
return lastUpdated
}

Note, if you're using NSPersistentContainer, you can spare yourself the optional chaining and access your persistent store through it instead of through an NSManagedObjectContext instance.

To write metadata, use the setMetadata(_:for:) method on your store's NSPersistentStoreCoordinator.

let coordinator = persistentContainer.persistentStoreCoordinator
guard let store = coordinator.persistentStores.first else {
fatalError("Unable to retrieve persistent store")
}
coordinator.setMetadata(["lastUpdated": lastUpdated], for: store)

After writing metadata, you'll need to save the managed object context referring to the store's coordinator. One possible Good Enough™ solution is to perform the metadata updates and saves in a completion handler after any transactions.

func fetchUpdates(completion: { context in
guard let coordinator = context.persistentStoreCoordinator else {
fatalError("Unable to retrieve persistent store coordinator")
}
guard let store = coordinator.persistentStores.first else {
fatalError("Unable to retrieve persistent store")
}
coordinator.setMetadata(["lastUpdated": Date.now()], for: store)
context.save()
})

An added bonus of using Core Data metadata over other persistence methods is that because it is a property of NSPersistentStore, it follows its associated data wherever the store goes! This is particularly useful if you ever need to seed or move a persistent store.

Additional Resources

❋❋❋

Seeding Core Data

Seeding Core Data with a small amount of data turns out to be a much larger task than one would initially guess. There are several potential avenues for tackling this problem, such as parsing data from an XML or JSON file and saving it to a new persistent store, to procedurally generating the data however, these leave a bad taste in my mouth, as they aren't very DRY. Not only that, but Apple explicitly discourage such practices, as some iOS devices don't have the cycles to spare on such a task. A simpler approach could be to create the database in a simulator, embed it in the app bundle, and copy the resource into place on initial launch.

let modelName = "Model" // Same name as .xcdatamodeld directory
guard let sqliteURL = Bundle.main.url(forResource: modelName, withExtension: "sqlite") else {
fatalError("Unable to find sqlite database")
}
guard let shmURL = Bundle.main.url(forResource: modelName, withExtension: "sqlite-shm") else {
fatalError("Unable to find shared memory file")
}
guard let walURL = Bundle.main.url(forResource: modelName, withExtension: "sqlite-wal") else {
fatalError("Unable to find logging file")
}
let persistentContainerDirectoryURL = PersistentContainer.defaultDirectoryURL()
let persistentContainerSQLiteURL = persistentContainerDirectoryURL.appendingPathComponent("\(modelName).sqlite")
let persistentContainerShmUrl = persistentContainerDirectoryURL.appendingPathComponent("\(modelName).sqlite-shm")
let persistentContainerWalUrl = persistentContainerDirectoryURL.appendingPathComponent("\(modelName).sqlite-wal")
do {
try FileManager.default.copyItem(at: sqliteURL, to: persistentContainerSQLiteURL)
try FileManager.default.copyItem(at: shmURL, to: persistentContainerShmUrl)
try FileManager.default.copyItem(at: walURL, to: persistentContainerWalUrl)
} catch let error {
print("\(error), \((error as NSError).userInfo)")
}

While copyItems(at:to:) will throw if the destination URL isn't empty, it would be cleaner to wrap the above code in a block that only ever executes once. We can use UserDefaults to keep track of its state.

if UserDefaults.standard.bool(forKey: didLaunchBeforeKey) {
...
UserDefaults.standard.set(true, forKey: didLaunchBeforeKey)
}

But wait, you might ask, what about replacePersistentStore(at:​destinationOptions:​withPersistentStoreFrom:​sourceOptions:ofType:)? If your database is too big to be a resource in your app's bundle, you might consider downloading the seed database on initial launch, in which case this would be preferred over manually copying files. Bundle resources, unfortunately, are read-only, which means the NSPersistentStore instantiated from a bundle resource would be read-only as well.

Additional Resources