Using Bacula to archive data. – The Technical Templar

I hate tape!

Ever since those early days working as an operator, my least favourite job was changing the daily backup tape. Oh sure, if your company has loads of cash you could shell out for an autochanger, but none of the organisations I worked for were ever prepared to pony up the dosh for one of those when they had a tape-monkey to do it for them!

With the cost per GB of storage dropping, backup up to disk was more of a reality and in looking around for something that could (a) Backup to disk (b) across the network and (c) was automated, I discovered “Bacula”.

So out comes the Ubuntu 10.04 server install disk and away we go…

Assumptions

The neat thing with bacula is that there are a many different scenarios you can cater for. I’m not a Bacula master, nor do I profess to know all the ins-and-outs of its configuration files. No. I started this with a plan, a plan to meet a requirement… so the configuration detailed below is what worked for me, to meet my needs. Its online more as a journal (so I can refer back to it if I ever need to rebuild it) than a claimed definitive HOWTO!!

The “so called” plan is build a Bacula server than will backup remote clients, initially to storage local to the server, but later pointing it to my FreeNAS server.

In this “journal” I’ll follow the following steps/stages:

Stage One– Configuring the Bacula server
- Install and configure Bacula without using “localhost”.
Stage Two– Backup local files to localstorage.
Stage Three– Backup remote files to local storage.
- Install and configure Bacula file daemon on remote machine.
- Backup remote files to local storage.
Stage Four– Backup remote files to remote storage.
- TBA: Install and configure Bacula storage daemon on remote machine.
- TBA: Backup remote files to remote storage.

The Bacula Home Page contains the latest Bacula news and developments, or for more Bacula configuration options refer to the Bacula User’s Manual

Base Install

The base install of Ubuntu server 10.04 took about 30mins, with standard configuration and stopping “network auto-discovery” to declare a static IP. Once the install was completed I checked for any updates…

sudo aptitude update
sudo aptitude safe-upgrade

First we install MySQL which will be the database engine used to store the catalog information.

sudo aptitude install mysql-client mysql-server

During the install you will be prompted for a password for the MySQL “root” user.

Add SSH server to allow telnet remote access.

sudo aptitude install openssh-server

Next we install Apache web server which will be used by “BWeb” for remote administration (see later sections).

sudo aptitude install apache2

Last but not least… Bacula.

sudo aptitude install bacula

“Postfix” will be installed as one of the packages and you will be asked to select a configuration type. I just selected “Local only” and left the server name as the hostname of the machine. Next you will be asked to “Configure database for bacula-director-mysql with dbconfig-common” – select “Yes” then supply credentials for the database administrator followed by a new password for the bacula database user. The “database administrator” will need to have the appropriate rights to create a database.

Stage One: Configuring the Bacula server

Bacula configuration files are formatted based on resources comprising of directives surrounded by “{}” braces. Each Bacula component has an individual file in the /etc/bacula directory.

The various Bacula components must authorize themselves to each other. This is accomplished using the password directive. For example, the Storage resource password in the /etc/bacula/bacula-dir.conf file must match the Director resource password in /etc/bacula/bacula-sd.conf.

The Console can be used to query the Director about jobs, but to use the Console with a non-root user, the user needs to be in the bacula group. To add a user to the bacula group enter the following from a terminal:

sudo adduser username bacula

Log out then back in to make the change effective. As mentioned in the previous section, I’m going to backup my data to local storage so I need to create a location for the “file media“. While we’re at it I’ll also create a directory where files can be restored to.

sudo mkdir -m 770 /var/lib/archive
sudo mkdir -m 770 /var/lib/archive/backup
sudo mkdir -m 770 /var/lib/archive/restore sudo chown -R  bacula:bacula /var/lib/archive

Everything in Bacula is about using IP addresses or resolvable addresses. But despite my hostnames being resolvable by DNS I still had to make sure there was an entry in the hosts file for this machine before it would work. If you’ve defined a static IP address, this will have been done automatically during the OS install.

Configuring the File Daemon (bacula-fd.conf)

sudo vi /etc/bacula/bacula-fd.conf

In the FileDaemon {} block change

FileDaemon {

    FDAddress = hostname.example.com

}

Restart the bacula file daemon for the changes to take effect

sudo /etc/init.d/bacula-fd restart

Configuring the Storage Daemon (bacula-sd.conf)

sudo vi /etc/bacula/bacula-sd.conf

In the Storage {} block change

Storage {

    SDAddress = hostname.example.com

}

In the Device {} block for “FileStorage” change

Device {
    Name = FileStorage

    Archive Device = /var/lib/archive/backup

}

Restart the bacula storage daemon for the changes to take effect

sudo /etc/init.d/bacula-sd restart

Configuring the Director (bacula-dir.conf)

sudo vi /etc/bacula/bacula-dir.conf

In the Director {} block change

Director {

    DirAddress = hostname.example.com

}

In the Client {} block change

Client {
    Name = hostname-fd
    Address = hostname.example.com

}

In the Storage {} block change

Storage {

    Address = hostname.example.com

}

In the RestoreFiles » Job {} block change

Job {
    Name = "RestoreFiles"

    Where = /var/lib/archive/restore
}

In the Catalog {} block add

Catalog {
    Name = MyCatalog
    dbname = bacula; DB Address = "localhost"; dbuser = "bacula"; dbpassword = "baculaPASSWORD"
}

I don’t think this is a “requirement” but I changed it for the sake of “consistancy” and “completeness”.

Restart the bacula director for the changes to take effect

sudo /etc/init.d/bacula-director restart

Using Bconsole

I have to create the media for the backup jobs to use now. For that we use bconsole

bconsole

Connecting to Director hostname.example.com:9101
1000 OK: hostname-dir Version: 5.0.1 (24 February 2010)
Enter a period to cancel a command.
*

Automatically create the media and label it with the following command

* label
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
Automatically selected Storage: File
Enter new Volume name:

Enter a meaningfull name for this “File” based media. I call mine “File-Repositoryxxx” where xxx increments 001,002,003… Select a relevant pool to make this media available in (I select “File“). If the permissions on our /var/lib/archive/ directories and the storage daemon are configured correctly, you should see…

Connecting to Storage daemon File at hostname.example.com:9103 ...
Sending label command for Volume "File-Repository001" Slot 0 ...
3000 OK label. VolBytes=202 DVD=0 Volume="File-Repository001" Device="FileStorage" (/var/lib/archive/backup)
Catalog record for Volume "File-Repository001", Slot 0 successfully created.
Requesting to mount FileStorage ...
3906 File device "FileStorage" (/var/lib/archive/backup) is always mounted.

This has now created a volume that will store a maximum of 50GB, as defined in the Pool {} section for File in the bacula-dir.conf

You can check the status of the various components with the status command. To check the status of the director and defined jobs type

* status director

If you’re unsure of a command, you can list them all by typing

* help

For help on a specific command, type help followed by the command name i.e.

* help status
* help list

Stage Two: Backup local files to local storage

In Bacula, the backup jobs are defined in the bacula-dir.conf file in the Job {} blocks. Normally (and for flexability) each job is defined with reference to a Fileset {}, Schedule {}, and Storage {} subset. Common elements can be included in the JobDefs {} subset and can be referenced in each job to avoid repitition and to save on scripting.

We’ll create a simple job to backup the /etc directory (including sub-directories) but ultimately it’s up to you how you’ll configure your jobs.

sudo vi /etc/bacula/bacula-dir.conf

Check what is already defined for JobDefs {}. It shoudl look something like this:

JobDefs {
  Name = "DefaultJob"
  Type = Backup
  Level = Incremental
  Client = hostname-fd
  FileSet = "Full Set"
  Schedule = "WeeklyCycle"
  Storage = File
  Messages = Standard
  Pool = File
  Priority = 10
  Write Bootstrap = "/var/lib/bacula/%c.bsr"
}

NOTE:

Any entries duplicated in the Job { } section, take precedence over ones already defined in the JobDefs { } section.

Next step is to create a new backup job entry.

Job {
  Name = "LocalConfigBackup"
  JobDefs = "DefaultJob"
  Type = Backup
  Level = Full
  Storage = File
  Client = hostname-fd
  FileSet = "LocalConfigFiles"
  Schedule = "DailyBackup"
}

Notes on the Job {} definition:

Name – defines the name of this particular job.
JobDefs – tells the job to load a particular set of default definitions called “DefaultJob“.
Type – tells the director it is a backup job.
Level – tells the director it should back *everything* up, not just files that have changed.
Storage – tells the director to use the storage defined in the Storage {} section as “File” (this in turn is referenced in the bacula-sd.conf file – in our case the local storage daemon)
Client – tells the director to use the client defined as “hostname-fd”, which in our case is the local machine (or more specifically, the local file daemon).
FileSet – tells the director to backup the files defined in the FileSet {} section named “LocalConfigFiles“.
Schedule – tells the director when to backup the files using values defined in the Schedule {} section named “DailyBackup“.

Now we can create a FileSet {} (referenced in the Job {} above). Filesets are handy and can be used by multiple jobs. An example might be backing up the same files, but off multiple different clients.

FileSet {
  Name = "LocalConfigFiles"
  Include {
    Options {
      signature = MD5
      compression=GZIP
    }
    File = /etc
  }
}

This FileSet will backup the /etc directory. The Options resource directives configure the FileSet to create a MD5 signature for each file backed up, and to compress the files using GZIP.

Next, create a new Schedule for the backup job:

Schedule {
  Name = "DailyBackup"
  Run = Full daily at 06:00
}

The job will run every day at 06:00 or 6:00 am. There are many other scheduling options available. Now lets run bconsole

bconsole

* run

The select the number that corresponds to the job you just created. When prompted with “OK to run?” answer Y and press enter. You should then see…

Job queued. JobId=2
*

It shouldn’t take very long for this job to complete (a few seconds depending on the speed of your machine). If you check the status of the director you should then see your job, with the JobId in the Terminated jobs: section

* status director
hostname-dir Version: 5.0.1 (24 February 2010) x86_64-pc-linux-gnu ubuntu 10.04

Terminated Jobs:
JobId Level Files Bytes Status Finished Name
====================================================================
2 Full 1,284 590.0 K OK 04-Nov-10 13:40 LocalConfigBackup

Congratulations! You have successfully used Bacula to backup files locally!! Now move onto the next section to backup files across the network.

Stage Three: Backup remote files to local storage.

Probably one of the most helpfull images in the Bacula documentation was this one, included below. It was really handy in helping to understand the relationship between the various components, particularly the remote ones!

The File-Daemon on the remote machine

Because of the various “distros” I’ll assume at this point that you’ve installed the bacula-fd daemon on the client you intend to backup. Start by opening up the bacula-fd.conf file on the remote machine and make a few changes.

sudo vi /etc/bacula/bacula-fd.conf

Director {
  Name = hostname-dir   # this is the name of the director
  Password = "aUniquePASSWORDofYourChoiceforFD"
}

 
FileDaemon {                          
  Name = FDName-fd
  FDport = 9102    
  WorkingDirectory = /var/lib/bacula
  Pid Directory = /var/run/bacula
  Maximum Concurrent Jobs = 20
  FDAddress = FDName.example.com
}

Restart the daemon for the changes to become effective. In some cases it may not be started already and you may even need to edit init scripts to ensure it starts on boot (CentOS and FreeBSD I’m looking at you!)

sudo /etc/init.d/bacula-fd restart

Now head back over to the machine running the director and edit…

sudo vi /etc/bacula/bacula-dir.conf

and add an entry for the client you’ve just created.

Client {
  Name = FDName-fd
  Address = FDName.example.com
  FDPort = 9102
  Catalog = MyCatalog
  Password = "aUniquePASSWORDofYourChoiceforFD"
  File Retention = 30 days
  Job Retention = 6 months
  AutoPrune = yes
}

You can also create a Job, Fileset and Schedule as detailed in the previous section and then restart the director for the changes to take effect.

sudo /etc/init.d/bacula-director restart

NOTE:

Be aware of firewalls running on any machines that run the client, as they may block traffic and stop communication between the Director and FileDaemon.

You can check communication between the Director and FileDaemon using bconsole

bconsole

* status client

then select the appropriate client from the list. If you’ve created a Job, Fileset etc… you can perform an estimate on the backup Job…

* estimate

then select the appropriate remote job from the list. You should get a response similar to…

2000 OK estimate files=2678 bytes=109,647,535

Notes on installing bacula-fd on CentOS 5

My VoIP server uses CentOS as its Linux distro, and it would be handy to back the config files and voicemail messages. By default, none of the YUM repositories (package manager for CentOS) have a copy of bacula, so you have to add one that does… EPEL. There was reference online to RPMForge, but once installed I couldn’t find a copy of bacula on it. This site has a great post on how to enable the EPEL repository, otherwise…

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm

will create the relevant repository entries in /etc/yum.repos.d/. Then download and install the bacula-client…

yum install bacula-client

All you have to do is configure the FileDaemon as detailed above. I used chkconfig to get it to start at boot.

chkconfig --level 345 bacula-fd on

Stage Four: Backup remote (or local) files to remote storage

Notes on installing bacula-sd on FreeBSD 7.1

To install the storage daemon add the repository…

setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7-stable/Latest/ pkg_add -r bacula-server

cd /usr/local/lib
ln -s libintl.so.8 libintl.so.9
cp bacula-sd.conf.sample bacula-sd.conf
vi bacula-sd.conf

Start the storage daemon service

/usr/local/sbin/bacula-sd -c /usr/local/etc/bacula-sd.conf

Because this server is actually running FreeNAS server, use the web interface to add the above command into the System|Advanced|Command scripts section.

Installing Bweb

Bweb is a perl based web program that provides a tool to perform basic operations and to get statistics. It obtains its information from your catalog database and the bconsole program. In this section the assumption is that you already have a working Bacula server, Apache server and MySQL.

First, we install some pre-requisite libraries.

sudo apt-get install -y libgdchart-gd2-xpm* make php-pear preload
sudo apt-get install -y libgd-graph-perl libhtml-template-perl libexpect-perl
sudo apt-get install -y libdbd-mysql-perl libdbd-pg-perl libdbi-perl libtime-modules-perl
sudo apt-get install -y libdate-calc-perl
sudo pear install db

You may get a warning at this stage that “DB” is depreciated in favour of “MDB2“. Since MDB2 is newer it’s probably better, but DB works fine for me – as part of the cleanup, if I find there is a major difference, I’ll note it here and alter my notes.

Now we download the bweb which is in the latest Bacula package. At the time of writing this, the latest version was 5.0.3, but you can check here and alter the instructions as required.

wget http://sourceforge.net/projects/bacula/files/bacula/5.0.3/bacula-gui-5.0.3.tar.gz

Now extract the files, compile the source and install some of the files to thier relevant locations

tar -zxvf bacula-gui-5.0.3.tar.gz
cd bacula-gui-5.0.3/bweb
sudo perl Makefile.PL
sudo make install
sudo pear channel-update pear.php.net
sudo mkdir -m 755 /usr/lib/cgi-bin/bweb
sudo install -m 755 -o root -g root cgi/*.pl /usr/lib/cgi-bin/bweb

Next, some authority changes, template copying and installing the HTML files

sudo chown root:bacula /etc/bacula
sudo vi /etc/bacula/bweb.conf

    $VAR1 = { template_dir => "/usr/share/bweb/tpl" };

sudo chown root:www-data /etc/bacula/bweb.conf
sudo mkdir -p /usr/share/bweb/tpl/en
sudo install -m 644 -o root -g root bweb/lang/en/tpl/*.tpl /usr/share/bweb/tpl/en

sudo mkdir /var/www/bweb
sudo install -m 644 -o root -g root bweb/html/*.{js,png,css,gif,ico,html} /var/www/bweb

sudo chmod 666 /etc/bacula/bconsole.conf

The Ext JS JavaScript library is needed for the “Web Restore” functionality to work properly. It’s probably writtern somewhere, but it didn’t leap out at me… it was Firebug that helped me to discover what was missing. The latest version at the time of publication was 3.3.0. First (in case you don’t have it already) install unzip.

sudo aptitide install -y unzip

Then we get the Ext JS package from the Ext JS CDN.

sudo wget http://extjs.cachefly.net/ext-3.3.0.zip

Then install the files in the bweb folder

sudo unzip ext-3.3.0.zip
sudo mkdir -m 755 /var/www/bweb/ext
sudo cp -r ext-3.3.0/. /var/www/bweb/ext

A few more changes to enable the web restore functionality of bweb…

sudo vi /etc/apache2/httpd.conf

    Alias /bweb/fv /var/spool/bweb
    <Directory "/var/spool/bweb">
        Options None
        AllowOverride AuthConfig
        Order allow,deny
        Allow from all
    </Directory>

sudo mkdir /var/spool/bweb
sudo chmod 700 /var/spool/bweb
sudo chown www-data /var/spool/bweb

then there are a number of changes to the database structure to support bweb that need to be made. You can import the commands from the bweb-mysql.sql file located in bacula-gui-5.0.3/bweb/script, but because I wasn’t sure of the results I just “cut-and-pasted” section by section using multiple SSH sessions, one running MySQL the other with the bacula-gui-5.0.3/bweb/script/bweb-mysql.sql file open, checking the results after each command.

To import it directly into the bacula database use

mysql -u root -p bacula

mysql> SOURCE path to files/bacula-gui-5.0.3/bweb/script/bweb-mysql.sql;

Next we must give the www-data (apache) user permission to execute the bconsole.

NOTE:

The locations highlighted in red below may differ on other Linux distributions… BE AWARE – and substitute as required!

cd /etc
sudo visudo

After the line root ALL=(ALL) ALL we add this…

www-data ALL= NOPASSWD: /usr/sbin/bacula-console
www-data ALL = (root) NOPASSWD: /usr/sbin/mtx -f /dev/changer transfer *
www-data ALL = (root) NOPASSWD: /usr/sbin/mtx -f /dev/changer status
www-data ALL = (root) NOPASSWD: /usr/sbin/mtx -f /dev/changer load *
www-data ALL = (root) NOPASSWD: /usr/sbin/mtx -f /dev/changer unload *

You then Ctrl-X to exit, then when it warns you that the buffers have changed, answer “Y” and then change the file name from sudoers.tmp to sudoers. Answer “Y” to overwrite.

Now we go back to the bweb.conf file to make some final changes

sudo vi /etc/bacula/bweb.conf

$VAR1 = bless( {
  'graph_font' => '/usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf',
  'name' => undef,
  'config_file' => '/etc/bacula/bweb.conf',
  'bconsole' => '/usr/sbin/bacula-console -n -c /etc/bacula/bconsole.conf',
  'ach_list' => {
     'S1_L80' => bless( {
         'info' => {
                     'drive' => 0,
                     'io' => 0,
                     'slot' => 0
                   },
         'name' => 'S1_L80',
         'bweb' => undef,
         'device' => '/dev/changer',
         'drive' => [],
         'debug' => 0,
         'label' => {},
         'precmd' => 'sudo',
         'io' => [],
         'mtxcmd' => '/usr/sbin/mtx',
         'drive_name' => [
                           'S1_L80_SDLT0',
                           'S1_L80_SDLT1'
                         ],
         'slot' => []
       }, 'Bweb::Autochanger' )
   },
  'dbi' => 'DBI:mysql:database=bacula',
  'user' => 'bacula',
  'password' => 'baculaPASSWORD',
  'fv_write_path' => '/var/spool/bweb',
  'template_dir' => '/usr/share/bweb/tpl',
  'lang' => 'en',
  'error' => '',
  'debug' => 0,
  'email_media' => 'someone@hostname.example.com'
}, 'Bweb::Config' );

Last of all, restart the server for the changes to take effect…

sudo shutdown -r now

If everything has been configured properly, you should be able to start your web browser and go to http://hostname.example.com/bweb and see the bweb interface with information on your backup jobs, media, etc..

IMPORTANT NOTE:

While BWEB was tested – and worked – with both IE (8) and Firefox (3.6.13), the “Web Restore” component only seemed to function with Firefox. When selected from IE nothing seemed to happen, and no data displayed within the browser window. Since Firefox is my browser of choice I hadn’t put too much effort into investigating why this is, and did… well… nothing! If I do find out, or happen upon a reason I’ll note my findings here.

MILESTONE 24/12/2010:

I had a user delete all the messages from their “Sent” folder and I had to use BWEB to do a restore – It was SO COOL! I had some trepidation over how the restore would work thinking that the Bacula File Daemon would only allow one-way traffic and I would have to restore to a local destination and copy the files to their original location (and all the file permission changes that would ensue).

BUT… Bacula (specifically via BWEB’s “Web Restore” function) allowed a restore to the original location with prompting to overwrite if required!! HOORAY!! I love you Bacula, you’ve made my Christmas…