Palo Alto Firewalls – Automate Offsite Log Backup
Our organization has over 15,000 users, so our Palo Alto Firewall logs add up fast. On an average day, we may have about 8GB worth of traffic & URL logs. Therefore, the firewall doesn’t keep them for very long, and I wanted to come up with a simple solution to archive logs in the event that we need to lookup something. My solution is simple:
- Export logs from firewall via FTP server.
- Compress all logs to a 7zip archive.
- Upload 7zip archive to Google Drive.
- Delete 7zip file and CSV logs.
I want this to occur nightly, so we have a compressed 7zip archive of each day’s traffic and URL logs. We are a Google Drive shop, so Google Storage isn’t an issue (and our 8GB worth of logs compresses to under 1GB).
Under DEVICE–>Schedule Log Export, I’ve setup exporting our logs to another Windows FTP server:
At 11:59 each night, the firewall will transfer all URL and Traffic logs to our FTP server. I then use these 3 batch files to automate the process:
Compress Logs – using 7zip’s command line interface, this command will zip up all files in the directory with today’s date (“20.03.2017 Firewall Backup.7z”). It will also zip up the batch files and 7zip command line utility, but that’s no big deal. I’m sure you can throw some more commands in there to include only CSV files.
Upload to Google Drive – I use rclone to upload to Google Drive (or to whatever service you want). This uploads any 7zip archive in that folder to our Google Drive – therefore only the 7zip archive it just created.
Delete – This command will delete all CSV and 7zip files, so the directory is clean for the next night’s archive – all while keeping batch files, and 7zip command line exe.
I have these 3 batch files scheduled to run nightly – with some time in between them to allow for compression and uploading. The delete command is done at night, before the next export, just so that I can look at the CSV files during the day if I choose.