== NAME == efolderbackup - backup and encrypt data to remote or local destinations == SYNOPSIS == efolderbackup backup [to ...] Start a new backup session. efolderbackup destroyexcluded ... Destroy files previously backed up that have since been excluded. efolderbackup preload ... Preload initial backup to a USB disk. efolderbackup status Get current backup session status. efolderbackup cancel Cancel current backup session. efolderbackup pause Pause current backup session. efolderbackup resume Resume the paused backup session. efolderbackup newconfig ... Generates a default config file. efolderbackup set ... Changes a setting value. efolderbackup get ... Retrieves a setting value. efolderbackup isincluded ... Display if file will be backed up efolderbackup version Display version information. efolderbackup help [topic] Display brief usage information. == DESCRIPTION == efolderbackup is a full backup solution, allowing you to backup and encrypt your data and then securely store it either on a remote server, on a local server, and/or on a locally mounted disk (such as an external USB drive). A full versioning system is provided whereby any number of versions of files may be preserved for any length of time. A smart backup scheme is used whereby the first time a file is backed up a full backup of the file is performed. From then on only incremental changes are sent and stored. However, unlike tape backups, the backed up data set is 'rolled forward' each time, such that the backed up data set always contains a 'full backup' and the incrementals remember how to restore previous versions. This has the advantage that a full backup only need be performed once. efolderbackup is configured by modifying a configuration file. The configuration file specifies server authentication credentials, encryption pass phrase, what data to backup, and versioning and notification settings. == SECURITY == All network communication is performed over a secure TLS/SSL connection. This program will authenticate to the server at the start of each backup sesion using the account username and password as specified in the configuration file. The data is encrypted with the AES-256 bit algorithm using a variant of CTR mode suitable for long term storage. A random initialization- vector (IV) is used for each 2KB block of data. The encryption is keyed by a user-supplied pass phrase. The software itself is owned by the user and group efolderbackup. All processes will run using the permission of the efolderbackup user. You may add users who should be able to control and configure the system to the efolderbackup group. == SETUP == You will need a unique account or sub-account for each computer you plan to backup. You can provision new accounts by using the web portal. Once you have installed this software, setup consists of the following three tasks: 1. Software configuration 2. Scheduling regular backups 3. Initiating your first backup To configure the software, please see the config help topic. ================================================================ | CAUTION: You must know your exact pass phrase that you use | | in the config file or you will not be able to | | restore your data under any circumstances. You | | should print the pass phrase and also save it to a | | *different* computer so you have it at all times. | ================================================================ Once configuration is complete, the following commands can be used in cron jobs or other schedulers to start backup sessions: efolderbackup backup to remote efolderbackup backup to localnetwork efolderbackup backup to localdisk Depending on your configuration, you may need to schedule one, some, or all of these commands. If you are using cron, you can either schedule backups to run using the master crontab file (/etc/crontab) or in the crontab file for the efolderbackup user. The following is an example crontab entry to start a remote backup session every day at 4:30am: 30 4 * * * /usr/local/efolderbackup/efolderbackup backup to remote --cron The --cron option will ensure that nothing is printed to standard out unless the backup cannot be started for whatever reason, which is convenient if you have cron job logs emailed to you or the logs are otherwise monitored. More complex configurations may use the efolderbackup command as part of a larger script. Common scenarios would include scripts that prepare databases for backup or that take volume snapshots for robust open file backup. Finally, use one of the commands listed above to start the intial backup. You should issue a separate command for each backup destination that you have active in your configuration. == CONFIGURATION OVERVIEW == The software is configured by editing the following INI file: /var/efolderbackup/efolderbackup.conf An INI file is a text file containing (name,value) pairs divided into sections. Sections of the file are denoted by the name of the section surrounded by brackets. For example: [section name] Anything after such a line will be in the named section until the start of the next named section. Comments can be added to the file by placing a semicolon (;) at the start of a line. (name,value) pairs are denoted in the following way: name = value The section that a (name,value) pair is placed within is important. The case of both section and value names is not important. The easiest way to configure the system is to start from a default configuration file. A default configuration file can be generated by using the following command: efolderbackup newconfig [-f ] Then open the default configuration file in your favorite text editor (vi, emacs, nano) and change values as appropriate. Alternatively you can use the following command to change individual config values: efolderbackup set [-f ]
= == CONFIGURATION FILE == The configuration file consists of the following sections and values. | [Account] | | ServerHostname = ... ;Network hostname of backup server in datacenter | ServerPort = 443 ;The TCP port the backup server is listening on | Username = ... ;The username of the account to use to authenticate | Password = ... ;The password associated with the username | PassPhrase = ... ;The pass phrase used to encrypt data. In addition to configuring remote backups, you can also configure backups to a local backup server and also a locally mounted volume (such as an external USB disk). To configure backups to a local backup server, specify the following: | [AccountLocalNetwork] | | ServerHostname = ;Hostname or IP of local server | ServerPort = 5470 ;The TCP port the backup server is listening on | EncryptData = <0|1> ;Whether or not to encrypt data stored on disk To configure backups to a locally mounted volume, specify the following: | [Account] | | LocalDiskStoragePath = /path/to/mountpoint ;path to storage location | | [AccountLocalDisk] | | EncryptData = <0|1> ;Whether or not to encrypt data stored on disk == FOLDER CONFIGURATION == The program is told exactly what to back up by specifying both policies and folders. The folders specify locations to scan to look for data to backup. Each folder is a 'root folder' in the sense that the program will scan all files within that directory and all subdirectories in order to know what *can* be backed up. The policies contain rules to indicate what data *should* be backed up. Folders are specified by creating one new INI section for each root folder that you want to backup. The name of the INI section should be: [Folder.NAME] where NAME is a unique identifier for that root folder. IMPORTANT: Once you have backed up data within a root folder with a given name you should not change the name; otherwise it will cause a full backup of that folder to occur. Each folder section should have the following values: | [Folder.NAME] | Path = /path/to/directory | Policy = PolicyNAME If you don't specify a policy then all files and folders within the directory specified by the Path setting will be backed up to that root folder on the server. == POLICY CONFIGURATION == A policy is a list of rules that either includes or excludes files based on certain patterns or conditions. Rules that are specified later take precedence over earlier rules. A policy can also include the rules from another policy. You create a policy be creating a section in the INI file that follows this naming convention: [Policy.NAME] where NAME is the name of the policy you want to create. Each policy section should contain one or more rule values, where each rule value has the name ruleN (where N is the number of the rule, starting with 1). There are currently two types of rules: * A rule that applies a wildcard pattern * A rule that applies another policy Rules that apply another policy have the following format: ruleN=apply policy [-allrules|-includerules|-excluderules] [from] [-invert] This allows you to apply all or some of the rules from another policy as part of this policy. Optionally you can invert the meaning of the rules that are applied (for example, converting include rules to exclude rules). Rules that apply a wildcard pattern have the following format: ruleN= [wildcard] [-absolute] [-files|-folders] [-olderthan ] [-newerthan ] [-largerthan ] [-smallerthan ] [-destroyexcluded] [-nooverride] Note that the rule type, wildcard, is optional. By default the wildcard pattern is relative to the path of the root folder. For example if you had a root folder whose path was /var/mail and your wildcard pattern was /temp/* then it would match any file or directory in the /var/mail/temp directory. If you specify the -absolute option then it will match the pattern against the full path of files and directories. Both the * (many characters) and ? (any one character) wildcards are supported. If the wildcard is part of a directory name in the pattern then it will match directories matching the pattern that are one level deep. For example the pattern queue/tmp*/*.tmp will match the filename queue/tmp-1/1.tmp but not the filename queue/tmp-1/tmp-2/1.tmp because the wildcard tmp* only applies to one level of directories. To match zero or more levels of subdirectories then you can use the pattern .../*/... (where the entire directory name in the pattern is a *). So for example, the pattern queue/tmp*/*/*.tmp will match both queue/tmp-1/1.tmp and queue/tmp-1/subdir/anything/1.tmp. If the * appears at the very beginning of a pattern, then it will match zero or more subdirectories. For example, */*.tmp will match all of the above examples. If the wildcard pattern does not include a / then it will only match on the filename and not the relative or absolute path of the file. So *.tmp will match any file ending with the extension .tmp in any directory. Multiple patterns can be specified by separating each pattern with a semicolon (;). For example, *.gif;*.jpg;*.png will match any file with any of those three extensions. Note that wildcard patterns can contain spaces and you should *not* put quotation marks around the pattern. The parser will automatically remove the options and treat the remaining string as the pattern. You can specify that the pattern should only match files or only matches folders by using the -folders or -files options. The -olderthan and -newerthan options allow you to restrict matching where the modification date of the file is older than or newer than a certain number of days (relative to the date and time when the backup is happening). Both options can be used at the same time (for example, -olderthan 30 -newerthan 60) Similarly the -smallerthan and -newerthan options allow you to match only files if the file is smaller or larger than a certain size. The size is specified using the following format: Where is one of KB,MB,GB. If the unit is not specified then it assumes that the unit is bytes. Note that there should not be any whitespace between the number and the unit (for example, 10MB). These two options can also be used at the same time. It's important to note that the policy must be all on the same line, because values in an INI file cannot be specified across multiple lines of text. == EXAMPLE POLICIES == There are two general stratgies to use when creating a policy: (a) Start by including everything and excluded unwanted files (b) Start by excluding everything and including wanted files The more typical approach is to start by including everything. With this strategy it is easier to ensure that new files you create will be included in the backup. However, there are situations where strategy (b) is appropriate, such as if the root folder is the entire drive and you only want to backup files of a certain type on that drive. Here is an example of a policy that uses strategy (a): | ;Designed to be used with a root folder with path /home | [policy.home] | rule1=include *.* | rule2=exclude *.mp3;*.wmv;*.m4a -largerthan 1MB | rule3=include recordedcalls*/*.m4a This policy includes all files, but then excludes audio files larger than 1MB, except that it will always include .m4a files if they are within a directory whose name starts with recordedcalls. Here is a policy that uses strategy (b): | ;Designed to be used with a root folder with path / | [policy.root] | rule1=exclude *.* | rule2=include *.doc*;*.ppt*;*.xls*;*.txt;*.htm*;*.xml; | rule3=exclude cache/*/*.* This policy excludes all files, except those that are word documents, powerpoint documents, excel documents, text files, web pages, or XML documents. However, it excludes any web pages that are within any directory that is a subdirectory of a directory named cache. IMPORTANT: The first rule of a policy must either be an 'include *.*' rule or a 'exclude *.*' rule; otherwise the meaning is ambiguous. The program will actually treat such a policy as a type (b) policy. == OVERRIDING SETTINGS == You can customize certain settings at both a root folder level and also at the policy level. To customize settings for a root folder, then in the section that contains the root folder configuration use entries in the INI file with the following format: setting.= Each include rule in a policy can also override certain backup settings, such as the number of historical versions to keep. This is done by adding an entry with the following form: ruleN.setting.= Several of the global settings can be overridden with this mechanism. These settings have the same meaning as they do when they are set globally, except that the new setting value only applies to You can override the following settings: NumRetriesPerSessionGeneral NumRetriesPerSessionGeneral NumRetriesPerSession NumResumeRetries SecsBetweenRetries VersionLimitFlag VersionLimitMaxDays VersionLimitMinToKeep VersionLimitMaxToKeep PurgeDeletedMaxDays DestroyExcludedFiles AlwaysCheckFingerprints == GLOBAL SETTINGS == The backup engine is designed to tailor itself to your needs. There are settings for customizing historical versioning, bandwidth consumption, and email notifications. These settings are described in this section by presenting annotated sections of a configuration INI file. Remember that setting names are case insensitive. The easiest way to configure a system is to use the newconfig command to generate a new INI file with all settings set to their default value. This allows you to see all default values and to customize them as needed. Here is the annotated config file that explains the settings: | ; ================== VERSIONING AND DATA DELETION =================== | [sync] | | ;Whether or not to limit the number of historical versions kept. | VersionLimitFlag=<0|1> | | ;Maximum number of days that a historical version should be kept for. | ;Set this to the number of days in the past that you want to be able | ;to roll back to and restore from. A value of -1 indicates infinity. | VersionLimitMaxDays= | | ;Minimum number of versions to keep (including the current version), | ;no matter how old the versions are. Thus even if a version is older | ;than VersionLimitMAXDays days it will still be kept if deleting it | ;would cause there to be fewer versions than required by this setting. | VersionLimitMinToKeep= | | ;The maximum number of versions to keep, including the current version. | ;This setting takes precedence over both VersionLimitMaxDays and also | ;VersionLimitMinToKeep. A value of -1 indicates infinity. | VersionLimitMaxToKeep= | | ;Whether or not to check if there are files that were deleted in | ;the past that should now be permanently destroyed. By default | ;files that are deleted on the client computer are marked as | ;deleted and kept on the server for a period of time. If this | ;is turned off, deleted files will be kept forever. | PurgeDeletedFlag=<0|1> | | ;The maximum number of days that deleted files should be kept | ;on the server before they are permanently destroyed. | ;In general you should always set this to the same value that | ;the VersionLimitMaxDays setting is set to so that you can | ;consistently restore back from any point in time within the | ;last N days. -1 indicates infinity. | PurgeDeletedMaxDays= | | ;Whether or not files that were previously backed up but that | ;are now excluded from the backup should also be destroyed | ;on the server. If this option is on you should add new | ;exclude rules with caution. | DestroyExcludedFiles=<0|1> | | | ; ======================= TIMEOUTS AND RETRIES ====================== | [sync] | | ;Number of retries when a general failure occurs, such as loss of | ;the Internet connection. | NumRetriesPerSessionGeneral= | | ;The number of times to retry when there is an error that is | ;caused by some problem with a specific file. | NumRetriesPerSession= | | ;The number of times to retry when resuming a paused backup. | NumResumeRetries= | | ;The number of seconds to wait between retry attempts. | SecsBetweenRetries= | | | ; ======================= BACKUP BEHAVIOR ====================== | [sync] | | ;The backup will lower its scheduling priority: | ; 0: slightly below normal priority | ; 1: low priority | ; 2: very low priority | ThreadPriority=<0|1|2> | | ;Instructs efolderbackup to scan every block of every file regardless of | ;whether the modification date or filesize has changed. | ;WARNING: Advanced feature. Use with caution. This can affect | ; the performance of your backups in a severe way. | ; This setting should almost always be left to 0. | AlwaysCheckFingerprints=<0|1> | | | ; ==================== NOTIFICATION ACTIONS =================== | [notify] | | ;The name of the person that is receiving the notifications. | EmailName= | | ;The email address to use when sending backup notifications. | ;Separate multiple addresses with a semicolon (;) or comma. | EmailAddress= | | ;What to do when the backup begins: | ; 0: Do nothing. | ; 1: Notify the user interface (NA for cmd line version) | ; 2: Send an email and notify the UI. | BeginBackup=<0|1|2> | | ;What to do when the backup ends with no warnings or errors: | ; 0: Do nothing. | ; 1: Notify the user interface (NA for cmd line version) | ; 2: Send an email and notify the UI. | ; 3: Send an email with the logs and notify the UI. | EndBackupOK=<0|1|2> | | ;What to do when the backup ends with warnings: | ; 0: Do nothing. | ; 1: Notify the user interface (NA for cmd line version) | ; 2: Send an email with the logs and notify the UI. | EndBackupWarn=<0|1|2> | | ;What to do when the backup ends with errors: | ; 0: Do nothing. | ; 1: Notify the user interface (NA for cmd line version) | ; 2: Send an email with the logs and notify the UI. | EndBackupError=<0|1|2> | | | ; ================= EMAIL CONFIGURATION ================ | [email] | | ;Whether or not to try sending via a SMTP relay | AllowUserSMTP=<0|1> | | ;The SMTP server that allows email relays | UserSMTPServer= | | ;Authentication credentials for the relay SMTP server | UserSMTPUsername= | UserSMTPPassword= | | ;Some SMTP relay servers require POP3 authentication | ;before allowing mail to be sent via SMTP. This gives | ;the hostname of the POP3 server and also credentials. | ;If you don't specify POP3 credentials it will reuse | ;the credentials you specified for the SMTP server. | UserPOP3Server= | UserPOP3Username= | UserPOP3Password= | | ;Whether or not to try to send mail directly to the | ;recipient's mail server if sending via the relay | ;SMTP server specified above fails. This normally | ;works, but sometimes the message is marked as SPAM | ;or rejected. | AllowDirectSMTP=<0|1> | | ;Any log file larger than this number of KB will be | ;automatically zipped before attaching it to an email. | ZipMinKB= | | | ; =============== BANDWIDTH THROTTLING ============== | ; Bandwidth can be set to high, medium, low, or unlimited. | ; Different levels can be set for business hours vs | ; off hours. This allows you to run backups during business | ; hours without unduly impacting users while they work. | | [bandwidth] | | ;Customizes the maximum bandwidth for the 'high' setting in kbit/sec. | HighUsage= | | ;Customizes the maximum bandwidth for the 'medium' setting in kbit/sec. | MediumUsage= | | ;Customizes the maximum bandwidth for the 'low' setting in kbit/sec. | LowUsage= | | ;Usage mode during off hours: | ; 0: max (unlimited) | ; 1: high | ; 2: medium | ; 3: low | UsageMode=<0|1|2|3> | | ;Usage mode during business hours. | BHoursUsageMode=<0|1|2|3> | | | ; =============== BUSINESS HOURS SETUP ============== | [bhours] | | ;The first day of the business week: | ; (0=Sunday, 1=Monday, ..., 6=Saturday) | StartDay=<0..6> | | ;The last day of the business week. | EndDay=<0..6> | | ;The time when business hours begin. | ;The format is: HHMMSS | ;Note that you must use 2 digits for | ;the hour, minute, and second. | ;You must use military time (no am/pm) | ;For example 7:30am is: 073000 | ;For example 5:30pm is: 173000 | StartTime= | | ;The time when business hours end. | EndTime= Again, the easiest way to start is to use the newconfig command and then customize to your particular setup. == USAGE SUMMARY == The efolderbackup program can perform many different actions relating to initiation of and control of backup sessions and configuration. The general pattern of usage is as follows: efolderbackup [action options] The next section lists the different actions and their options. Unless an action description below specifically explains what the exit code of a command will be the exit code will have the meaning as explained in the == EXIT CODE == section. == USAGE == efolderbackup backup [to ] [options] This command starts a backup session for the specified destination (or the remote server destination if one is not specified). If a backup for the specified target is already running then nothing will happen. The following options are supported: --cfg-file= Use a configuration file that is not in the default location. Normally you want to edit the configuration file in the default location. --auth-cfg-file= Use a different configuration file for the username, password, and pass phrase. This is useful if you want less restrictive permissions on the general config file and use more strict settings on the file with the authentication and encryption information. --cron Only print messages to stdout if backup fails to start. --only=< [,...] > If this option is specified then only the specified folder names or directories (and their subdirectories) will be backed up. Useful when you want to force an immediate backup of a specific root folder or specific directory. Note that individual filenames are currently not supported. --foreground Do not fork a process to perform the action in the background but rather block until the action completes. If the command runs in the background (the default) then the exit code will have the following meaning: 0 The backup session was started successfully 1 The backup session could not be started If the command runs in the foreground the exit code will mean: 0 The backup session completed successfully 1 The backup session completed with warnings 2 The backup session completed with errors efolderbackup destroyexcluded [to ] [options] This command will destroy any files that are on the specified backup destination that are currently excluded from the backup. This can be useful if certain data was backed up and then this data was excluded from the backup and no longer needs to be stored in the backup. All backup options are supported and have the same meaning. You may want to use the isincluded command after changing the backup configuration to make sure that you have setup your backup policies correctly. efolderbackup preload [options] Some sites with limited bandwidth or large amounts of data may want to save the initial full backup to a locally attached portable disk, and then have this disk shipped to the remote data center. From then on only incremental changes need to be transmitted across the network. All backup options are supported and have the same meaning. You must specify the destination where to store the preloaded backup data. This should point to a mounted volume of an external storage device that can be unmounted and shipped to our remote data center. Note that you can only preload the remote backup server destination. Preloading initial backups for other backup destinations is not currently supported. This action is part of the process to make this happen. To preload the initial full backup, follow these steps: 1. Use the web portal to set the account maintenance flag on the account that is being preloaded. 2. Attach an external USB disk and format and mount it appropriately. If you plan on using a non-standard format you may want to check with our engineers to make sure we can receive it. 3. Use the preload command to preload the data. 4. Unmount the volume with the preloaded data and ship the storage device to the remote data center. Be sure to coordinate with our support team to expedite processing. 5. Leave the account in maintenance mode. Our engineers will receive the preloaded data and put it into place, at which point they will clear the maintenance flag and backups will begin working as you have scheduled them. 6. Your storage device will be shipped back to you and the one time charge will appear on your next bill. efolderbackup status [all|remote|localnetwork|localdisk] This will output a message indicating whether or not the specified backup sessions are running. The default is to check all fully configured backup destinations. Additionally, the exit code of the process will indicate whether or not the backup session(s) were running: 0 One or more sessions are running 1 No sessions that were checked are running efolderbackup cancel [all|remote|localnetwork|localdisk] Cancel the specified backup sessions, if they are running. The default is to cancel all running sessions. The exit code has the following meaning: 0 All backup sessions were canceled (or they were not running) 1 One or more backup sessions are running efolderbackup pause [all|remote|localnetwork|localdisk] Pause the specified backup sessions. Backup sessions can be paused for up to one week. However, the backup session process must continue to run or the backup will be canceled. The default is to pause all running sessions. The exit code has the following meaning: 0 All backup sessions were paused (or they were not running) 1 Some backup sessions could not be paused efolderbackup resume [all|remote|localnetwork|localdisk] Resume the specified backup sessions, if they are paused. The default is to resume any paused sessions. The exit code has the following meaning: 0 All backup sessions were resumed (or they were not running) 1 Some backup sessions could not be resumed efolderbackup newconfig [-f ] [--force] Generates a default configuration file. If the new file exists then the program will fail, except if the --force option is used. If the --force option is specified and the file already exists then it will rename files such that the last several versions of the file will be kept. efolderbackup set [-f ] [--useempty]
= [...] [...] Sets the value of one or more settings across one or more sections to the specified values. Note that if the setting name or value contains spaces you need to be careful to use the appropriate shell quoting mechanisms. If you specify an empty value then the setting will be deleted. Note that the valuename can also contain the section name using the period syntax described by the get action below. The output will print the new value for each setting using the same format as for the get command. The useempty option affects the output in the same way as for the get command. For example: efolderbackup set Account Username=myuser Password=passwd AccountLocalDisk EncryptData=1 efolderbackup get [-f ] [--useempty]
. [...] Retrieves the value of the specified settings. The first valuename must include the section prepended with a period. Values without a section (and thus no period) are considered to be in the section that was last specified. The special valuename 'all' will print all known settings. One line will be printed for each valuename specified. Each line will have the following format:
If the value does not exist in the config file then will be exactly '(not set)' unless the --useempty option is set, in which case the value will be an empty string. For example: efolderbackup get Account.Username Password AccountLocalNetwork.EncryptData could print the following output: Account Username myuser Account Password mypassword AccountLocalNetwork EncryptData (not set) The use of the TAB character as a field separator allows scripts to easily parse the output using commands such as cut or awk. For example, you could reformat the output to have the form
.= as follows: efolderbackup get all | awk '{ print $1 "." $2 "=" $3; }' efolderbackup isincluded [-f ] [--verbose] ... Checks to see if one or more paths are included in the backup set. Note that this doesn't actually check to see if the file has been backed up, only if it should be backed up. Output is normally formatted such that one line is printed for each path specified and will be formatted as follows: The list of root folders will be separated by commas. If the --verbose option is specified then output is formatted such that for each path it will print one line of text for each root folder that could include the path in its policy: If no root folders could possibly include the path then it will output just one line of text for that path: no The exit code has the following meaning: 0 All paths are included in the backup set. 1 Some paths are not included. efolderbackup version Display version information. efolderbackup help [ [subtopic]] Display help information. You can optionally specify a help topic and subtopic. The following topics are available: overview Overview of program. setup Describes how to setup backups. config Describes config file format. usage Overview of program usage. action Usage for a particular action. The subtopic is the action name. == EXIT CODE == Unless the exit code meaning is customized for a specific command as explained above the meaning of the exit code will be as follows: 0 The operation completed successfully. 1 The operation completed with warnings. 2 The operation failed with errors. 50 The requested operation was already running. 100 Failed to load app resources. Contact tech support. 101 The installation is incomplete. Contact tech support. 102 Failed to load the settings from the config file. 103 Failed to initialize IPC. Contact tech support. 104 Failed to initialize the backup operation. 105 The configuration file is incomplete or invalid. 255 The command line options were not valid. == NOTES == * Resource forks, file permissions, and ACLs are not currently backed up, but this will be supported in a future release of the software.