Bacula tape backup software and LTO tape library tips and troubleshooting

Environment notes: our setup includes a NeoT48 LTO8 tape library with two tape drives and a PostgreSQL-18 catalog for Bacula 15.0.3.

NeoT48 Tape library tips

# find tape library /dev/sg* path. should show up as "mediumx"
lsscsi -g

# get a status listing of tapes in the tape library
mtx -f /dev/sg3 status

# the following commands may not work if bacula is actively using the tape library

# load a tape into tape drive 0 from slot 16
mtx -f /dev/sg3 load 16 0 

# unload a tape from tape drive 0 into slot 16
mtx -f /dev/sg3 unload 16 0

See error codes associated with tapes at library web interface>service>cartridge memory. Code definitions here:

Bacula tips

Job architecture

Although Bacula proports to be able to resume a "incomplete" job, in practice, most jobs that hit errors just fail outright and must be restarted. You can update the catalog to mark a failed job as incomplete and then resume the job, but this appears to resume from the beginning of the job, rather than where it left off. Even if it does resume where it left off, there's no way to know if some of the files were backed up in a bad state due to the error. So it is best to restart the jobs from the beginning.

As such, it is wise to keep job sizes down to a few terabytes at most. Here's an example fileset configuration which separates out a few subdirs into their own filesets, while still keeping an "other" fileset to allow any new subdirs/files in the parent directory to be backed up:

Fileset {
  Name = "ARCHIVE_DATA"
  Description = "The ARCHIVE_DATA directory under /data"
  Include {
    Options {signature=SHA1; onefs=no; }
    File = "/data/ARCHIVE_DATA"
  }
}

Fileset {
  Name = "data1"
  Description = "The data1 directory under /data"
  Include {
    Options {signature=SHA1; onefs=no; }
    File = "/data/data1"
  }
}

Fileset {
  Name = "other"
  Description = "Other dirs and files located in /data"
  Include {
    Options {signature=SHA1; onefs=no; }
    File = "/data"
  }
  Exclude {
    File = "/data/ARCHIVE_DATA"
    File = "/data/data1"
  }
}

Watching Bacula job progress

Pull out the useful information about the job currently accessing the storage device:

# general information about running jobs
watch 'echo "status storage" | bconsole | grep -e 'Files=' -e 'Writing:' -e 'pool='' -e 'FDReadSeqNo'

# More info about current copied file etc. Swap 6 for the number of the bacula client you want to modify
watch '(echo "st client" && echo "6") | bconsole | grep "Running Jobs" -A 7; echo "st s" | bconsole | grep "spooling"'

# running jobs list:
watch 'echo "st dir" | bconsole | grep "Running Jobs" -A 5'

Get Bacula jobs stored on each tape

Bacula catalog PostgreSQL query to get jobs that exist on each tape

select m.mediaid, volumename, array_agg(distinct(j.name)) as jobs_containing, p.name as pool_name from jobmedia jm join media m on jm.mediaid = m.mediaid join job j on jm.jobid=j.jobid join pool p on m.poolid=p.poolid group by p.name, m.mediaid, volumename order by p.name, m.mediaid;

Purging and reusing volumes

Use above query to double check which jobs you're purging, first.

bconsole
> purge volume
> # Select the volume you wish to erase and reuse
> update volume
> # Select status and set to "Recycle"
> # Rather than recycling, you can also relabel a volume here

Reading a Bacula tape label without Bacula

mtx -f /dev/sg3 status  # Make sure only the tape you want to delete is in a drive to avoid confusion
mt -f /dev/nst1 rewind
dd if=/dev/nst1 bs=64512 count=1 | xxd | less  # print off start of data including Bacula label, if it exists
mt -f /dev/nst1 rewind

Deleting the label on a tape (loses data)

# remove label from bacula
bconsole
> delete volume
> 13 # pick the pool the volume is in
> tape1  # enter the tape's name
> yes

mtx -f /dev/sg3 status  # Make sure only the tape you want to delete is in a drive
mt -f /dev/nst1 rewind  # rewind tape to start
dd if=/dev/nst1 bs=64512 count=1 | xxd | less  # Double check that the data/label doesn't look unexpected
mt -f /dev/nst1 rewind  # rewind to start again
mt -f /dev/nst1 weof  # Delete the first block which includes the tape label
mt -f /dev/nst1 rewind  # rewind to start one last time
# now bacula will not be able to read the tape label

Setting up encryption

# Generate master certs
openssl genrsa -out bacula_encryption_master.key 2048
openssl req -new -key bacula_encryption_master.key -x509 -out bacula_encryption_master.cert

# Generate local fd certs
openssl genrsa -out bacula_encryption_fd.key 2048
openssl req -new -key bacula_encryption_fd.key -x509 -out bacula_encryption_fd.cert

# create pem file
cat bacula_encryption_fd.key bacula_encryption_fd.cert > bacula_encryption_fd.pem

In your bacula-fd.conf file, add encryption info to your filedaemon:

FileDaemon {
  ...
  PKI Signatures = Yes  # Enable Data Signing
  PKI Encryption = Yes  # Enable Data Encryption
  PKI Keypair = "/path/to/bacula_encryption_fd.pem"  # Public and Private Keys
  PKI Master Key = "/path/to/bacula_encryption_master.cert"  # ONLY the Public Key
}

Then restart the filedaemon: systemctl restart bacula-fd

More tips

• Force bacula to scan every tape for its contents with update slots scan
• Enter a period (".") to cancel most input prompts in bconsole
• Restart the storage daemon seems to fix some issues: service bacula-sd restart

Resolving Bacula errors

"Could not connect to client"

Make sure ports are open on the client and that the bacula-fd service is running.

firewall-cmd --add-port=9103/tcp --permanent
firewall-cmd --reload

"bacula-fd: Fatal error: failed to connect to storage daemon"

Make sure ports are open on the device which is running the storage daemon. Make sure the storage daemon is running.

firewall-cmd --add-port=9103/tcp --permanent
firewall-cmd --reload

"Waiting to connect to storage daemon"

See above.

"Unable to open device, no medium found"

Manually load a tape into a drive in the tape library using mtx. Then run update slots in Bacula, then try the job again or let it resume.

Job stuck on running when backing up via NFS mount

More reliable to run a file daemon on the remote device, rather than relying on NFS to serve the files for backup.

"Bad storage"

Double check configured autochangers/drives. Make sure the correct devices are used and that the autochangers have the correct /dev/sg numbers.

"Warning: Director wanted Volume 'X'. Current Volume 'Y' not acceptable because: 1998 Volume 'Y' catalog status is Append, not in Pool."

If it loads a drive with a tape but then reports that the wrong tape is loaded (and the error message mentions a tape that is in the other drive), try swapping the DriveIndex numbers in the bacula-sd.conf and restarting bacula-dir and bacula-sd.

"Bconsole cannot connect to director"

Make sure postgresql is running and then restart bacula-dir.

Despite what the status might suggest, the director may not be running. Try executing bacula-dir -d 100 to see if there's any errors on launch. Also double check bacula-fd -d 100. Should say "bacula-fd is already running" if it's working.

Job not automatically finding volume

Double check that the mediatype listed in bacula-dir.conf, bacula-sd.conf, and reported by the media in bconsole (list media) match.

Volume marked "Full" before actually full

Check the tape library's web interface. The NeoT48 may be marking drives with Tape alert flag (TAF) 49 which means "Diminished native capacity" – "The media is either partitioned or shortened and is not configured to use full native capacity."

NeoT48 marks tapes as "Bad Tape"

Stop Bacula jobs and then stop the bacula-dir service. Clean drive, reboot, move each bad tape one-by-one into the drive slot and then back out. Just doing a move operation isn't enough– seems the drive has to be able to load the tape and read it for it to decide the tape is no longer bad. Restart bacula-dir service. Update volstatus to append for any volumes that Bacula marked error due to this problem. Re-run jobs as needed.