Restic 应用实践：线上生产环境日志、数据异机备份

备份规划

与前面讲述的单个备份场景不同，线上生产环境的网络拓扑结构通常会更复杂一些，整个项目也会因为不同中间件的依赖，所需要进行异机备份的内容不同。

下面我们就以一个基于 Nginx、KubeSphere、Nacos、MySQL、Redis、MinIO 的容器化实施的真实的项目部署情况来讲述，我们是如何利用 Restic 实现主机的 系统日志、Nginx 日志、Nacos 日志、MySQL日志、MySQL 数据、Redis 日志、Redis 数据 的异机备份。

整个项目共计有9台机器，主机信息如下：

主机 IP 说明
LSB-Nginx 192.168.39.94 负载均衡、Nacos
K8s-Master1 192.168.39.82 K8s控制节点-1
K8s-Master2 192.168.39.165 K8s控制节点-2
K8s-Master3 192.168.39.233 K8s控制节点-3
K8s-Worker1 192.168.39.107 K8s工作节点-1
K8s-Worker2 192.168.39.100 K8s工作节点-2
K8s-Worker3 192.168.39.217 K8s工作节点-3
RDS-MySQL 192.168.39.87 MySQL 服务、Redis 服务
OSS-MinIO 192.168.39.190 OSS 对象存储

根据主机情况，我们计划把备份工作分为一下5条：

  • LSB-Nginx、K8s-Master1、K8s-Master2、K8s-Master3、K8s-Worker1、K8s-Worker2、K8s-Worker3、RDS-MySQL 的系统日志 备份到 MinIO 对象存储服务
  • LSB-Nginx 的 Nginx 日志、Nacos 日志 备份到 MinIO 对象存储服务
  • RDS-MySQL 的 MySQL 日志、MySQL 数据 备份到 MinIO 对象存储服务
  • RDS-MySQL 的 Redis 日志、Redis 数据 备份到 MinIO 对象存储服务
  • OSS-MinIO 的 系统日志 通过 SFPT 备份到 LSB-Nginx 主机目录下

备份准备

基于 MinIO 服务的备份准备

在备份规划中，规划中的前4条是将日志或数据备份到 MinIO 对象存储服务中，下面我们就开始准备基于 MinIO 服务的备份。

配置 MinIO 服务器的凭据

首先在主机 LSB-Nginx、K8s-Master1、K8s-Master2、K8s-Master3、K8s-Worker1、K8s-Worker2、K8s-Worker3、RDS-MySQL 上设置以下环境变量，指定 Minio 服务器的凭据：

$ echo 'export AWS_ACCESS_KEY_ID=<YOUR-MINIO-ACCESS-KEY-ID>' >>/etc/profile;
$ echo 'export AWS_SECRET_ACCESS_KEY=<YOUR-MINIO-SECRET-ACCESS-KEY>' >>/etc/profile;
$ source /etc/profile;

例如：

echo 'export AWS_ACCESS_KEY_ID=iPpXf8yoWrqHDsj3fPbe' >>/etc/profile;
echo 'export AWS_SECRET_ACCESS_KEY=YlrpuMbHV4IPOwEYpquWIJeAXm00afTGYzskPIih' >>/etc/profile;
source /etc/profile;
免密码

每次备份的时候，都需要输入密码，肯定不适合脚本自动备份，所以我们还需要使用 --password-file 参数来达到自动读取密码的步骤。

# 先将restic仓库密码，比如 test 保存在 /root/resticpasswd 文本中
echo '仓库密码' > /root/resticpasswd
初始化备份仓库

配置完 MinIO 服务器的访问凭据后，就可以为 LSB-Nginx、K8s-Master1、K8s-Master2、K8s-Master3、K8s-Worker1、K8s-Worker2、K8s-Worker3、RDS-MySQL 这些主机分别创建一个备份存储库。

为了方便管理，我们以 backup 作为备份存储桶，以主机名称来作为不同主机的备份目录。

现在可以使用下面的命令轻松初始化 restic 以使用 Minio 服务器作为备份服务端。

# restic -r s3:http://192.168.39.190:9000/backup/<HostName> init
enter password for new repository:
enter password again:
created restic repository 6ad29560f5 at s3:http://localhost:9000/restic1
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is irrecoverably lost.

实际操作如下：

初始化 LSB-Nginx | 192.168.39.94 | 负载均衡、Nacos 的备份仓库

$ restic -r s3:http://192.168.39.190:9000/backup/LSB-Nginx init
enter password for new repository:
enter password again:
created restic repository 8f6b34bb33 at s3:http://192.168.39.190:9000/backup/LSB-Nginx

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

初始化 K8s-Master1 | 192.168.39.82 | K8s控制节点-1 的备份仓库

$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Master1 init
enter password for new repository:
enter password again:
created restic repository 8f6b34bb33 at s3:http://192.168.39.190:9000/backup/K8s-Master1

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

初始化 K8s-Master2 | 192.168.39.165 | K8s控制节点-2 的备份仓库

$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Master2 init
enter password for new repository:
enter password again:
created restic repository 8f6b34bb33 at s3:http://192.168.39.190:9000/backup/K8s-Master2

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

初始化 K8s-Master3 | 192.168.39.233 | K8s控制节点-3 的备份仓库

$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Master3 init
enter password for new repository:
enter password again:
created restic repository 8f6b34bb33 at s3:http://192.168.39.190:9000/backup/K8s-Master3

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

初始化 K8s-Worker1 | 192.168.39.107 | K8s工作节点-1 的备份仓库

$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Worker1 init
enter password for new repository:
enter password again:
created restic repository 8f6b34bb33 at s3:http://192.168.39.190:9000/backup/K8s-Worker1

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

初始化 K8s-Worker2 | 192.168.39.100 | K8s工作节点-2 的备份仓库

$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Worker2 init
enter password for new repository:
enter password again:
created restic repository 8f6b34bb33 at s3:http://192.168.39.190:9000/backup/K8s-Worker2

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

初始化 K8s-Worker3 | 192.168.39.217 | K8s工作节点-3 的备份仓库

$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Worker3 init
enter password for new repository:
enter password again:
created restic repository 8f6b34bb33 at s3:http://192.168.39.190:9000/backup/K8s-Worker3

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

初始化 RDS-MySQL | 192.168.39.87 | MySQL 服务、Redis 服务 的备份仓库

$ restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL init
enter password for new repository:
enter password again:
created restic repository 8f6b34bb33 at s3:http://192.168.39.190:9000/backup/RDS-MySQL

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

创建完备份仓库后，进入 MinIO 服务查看 backup 存储桶下，可见 restic 自动创建出来的备份空间。

基于 SFTP 协议的备份准备

在备份规划中，除 MinIO 服务所在主机外，其他 8台主机的异机备份，均备份到了 MinIO 对象存储服务中，因此 MinIO 对象存储服务所在主机的 系统日志 则通过 SFPT 备份到 LSB-Nginx 主机目录下。

主机间免密钥互信

在主机 OSS-MinIO 上执行下面命令：

# 生成SSH密钥对
ssh-keygen -t rsa;
# 复制公钥到远程服务器
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.39.94;
# 配置SSH连接
vi ~/.ssh/config
    Host XSZHPJ-LSB-Nginx
        HostName 192.168.39.94
        User root
        IdentityFile ~/.ssh/id_rsa
免密码

每次备份的时候，都需要输入密码，肯定不适合脚本自动备份，所以我们还需要使用 --password-file 参数来达到自动读取密码的步骤。

# 先将restic仓库密码，比如 test 保存在 /root/resticpasswd 文本中
echo '仓库密码' > /root/resticpasswd

今天在服务器上使用同样的方法，发现有两点问题：

１．提示“ Permissions 0644 for ‘/root/.ssh/id_rsa.pub’ are too open”

解决方法：使用chmod 0600 /root/.ssh/id_rsa.pub更改将公钥权限改成“600”

２．提示“Enter passphrase for key /root/.ssh/id_rsa.pub”让输入私钥，可不论输与不输都不能直接登录

解决方法：
在本地执行：

eval `ssh-agent`
ssh-add

ssh-agent是用于管理密钥，ssh-add用于将密钥加入到ssh-agent中，SSH可以和ssh-agent通信获取密钥，这样就不需要用户手工输入密码了。
顺序执行以上两条命令后就可以用ssh免密登录远程机器了，但这个配置只对当前会话生效，会话关闭或机器重启后都需要重新执行这两条命令。将命令放到~/.bash_profile中，就可以免去每次输入的麻烦。

初始化备份仓库

初始化 OSS-MinIO | 192.168.39.190 | OSS 对象存储 的备份仓库

$ restic -r sftp:root@192.168.39.94:/backup_192.168.39.190 init
enter password for new repository:
enter password again:
Enter passphrase for key '/root/.ssh/id_rsa':
created restic repository bed32eb48e at sftp:root@192.168.39.94:/backup_192.168.39.190

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

创建完备份仓库后，进入 192.168.39.94 服务器根目录，可见 restic 自动创建出来的备份空间。

备份操作

基于 MinIO 服务的备份操作

常见的备份操作可参考下面的命令：

# 执行数据备份
restic -r s3:http://192.168.39.190:9000/backup backup /var/log
# 查看备份
restic -r s3:http://192.168.39.190:9000/backup snapshots
# 查看备份内容
restic -r s3:http://192.168.39.190:9000/backup ls 875a2a32
# 恢复快照
restic -r s3:http://192.168.39.190:9000/backup restore 875a2a32 -t ./
restic -r s3:http://192.168.39.190:9000/backup restore 875a2a32 --target ./
# 删除备份
restic -r s3:http://192.168.39.190:9000/backup forget 875a2a32

主机 LSB-Nginx | 192.168.39.94 | 负载均衡、Nacos 的备份操作

# 备份系统日志
$ restic -r s3:http://192.168.39.190:9000/backup/LSB-Nginx --password-file /root/resticpasswd backup /var/log
# 备份 Nginx 日志（这里使用了宝塔Linux面板安装的Nginx）
$ restic -r s3:http://192.168.39.190:9000/backup/LSB-Nginx --password-file /root/resticpasswd backup /www/server/nginx/logs
# 备份 Nacos 日志
$ restic -r s3:http://192.168.39.190:9000/backup/LSB-Nginx --password-file /root/resticpasswd backup /usr/local/src/nacos/logs
# 查看备份
$ restic -r s3:http://192.168.39.190:9000/backup/LSB-Nginx --password-file /root/resticpasswd snapshots
# 查看备份内容
$ restic -r s3:http://192.168.39.190:9000/backup/LSB-Nginx --password-file /root/resticpasswd ls f12a2a03

主机 K8s-Master1 | 192.168.39.82 | K8s控制节点-1 的备份操作

# 备份系统日志
$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Master1 --password-file /root/resticpasswd backup /var/log

主机 K8s-Master2 | 192.168.39.165 | K8s控制节点-2 的备份操作

# 备份系统日志
$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Master2 --password-file /root/resticpasswd backup /var/log

主机 K8s-Master3 | 192.168.39.233 | K8s控制节点-3 的备份操作

# 备份系统日志
$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Master3 --password-file /root/resticpasswd backup /var/log

主机 K8s-Worker1 | 192.168.39.107 | K8s工作节点-1 的备份操作

# 备份系统日志
$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Worker1 --password-file /root/resticpasswd backup /var/log

主机 K8s-Worker2 | 192.168.39.100 | K8s工作节点-2 的备份操作

# 备份系统日志
$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Worker2 --password-file /root/resticpasswd backup /var/log

主机 K8s-Worker3 | 192.168.39.217 | K8s工作节点-3 的备份操作

# 备份系统日志
$ restic -r s3:http://192.168.39.190:9000/backup/K8s-Worker3 --password-file /root/resticpasswd backup /var/log

主机 RDS-MySQL | 192.168.39.87 | MySQL 服务、Redis 服务 的备份操作

# 备份系统日志
$ restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /var/log
# 备份 MySQL 日志
$ restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /www/server/data/mysql-slow.log
# 备份 MySQL 数据
$ restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /www/server/data
# 备份 Redis 日志
$ restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /www/server/redis/redis.log
# 备份 Redis 数据
$ restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /www/server/redis/dump.rdb

基于 SFTP 协议的备份操作

主机 OSS-MinIO | 192.168.39.190 | OSS 对象存储 的备份操作

# 执行数据备份
restic -r sftp:root@192.168.39.94:/backup_192.168.39.190 --verbose backup /var/log --password-file /root/resticpasswd
# 查看备份
restic -r sftp:root@192.168.39.94:/backup_192.168.39.190 snapshots
# 查看备份内容
restic -r sftp:root@192.168.39.94:/backup_192.168.39.190 ls 875a2a32
# 恢复快照
restic -r sftp:root@192.168.39.94:/backup_192.168.39.190 restore 875a2a32 -t ./
restic -r sftp:root@192.168.39.94:/backup_192.168.39.190 restore 875a2a32 --target ./
# 删除备份
restic -r sftp:root@192.168.39.94:/backup_192.168.39.190 forget 875a2a32

设置定时备份

Crontab 定时任务

  • 命令行输入 crontab -e 命令， 按下 a 键进入到编辑模式
  • 输入定时备份任务表达式，如：0 */1 * * * /home/work/start-service.sh
  • 同时按下 esc 退出编辑模式，输入 wq 保存退出

    注意：在你的脚本路径或者命令前面加上/usr/bin/env bash或者/bin/sh，这样可以确保无论在哪个环境下，你的脚本都能正确运行。

基于 MinIO 服务的定时备份

主机 LSB-Nginx | 192.168.39.94 | 负载均衡、Nacos 的定时备份

创建或编辑备份脚步：

mkdir -p /home/work
vi /home/work/backup.sh

脚步内容：

export AWS_ACCESS_KEY_ID=iPpXf8yoWrqHDsj3fPbe
export AWS_SECRET_ACCESS_KEY=YlrpuMbHV4IPOwEYpquWIJeAXm00afTGYzskPIih
# 备份系统日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/LSB-Nginx --password-file /root/resticpasswd backup /var/log
# 备份 Nginx 日志（这里使用了宝塔Linux面板安装的Nginx）
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/LSB-Nginx --password-file /root/resticpasswd backup /www/server/nginx/logs
# 备份 Nacos 日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/LSB-Nginx --password-file /root/resticpasswd backup /usr/local/src/nacos/logs

脚步授权执行权限：

cd /home/work
chmod -x backup.sh && chmod -R 755 backup.sh

测试脚步能否正常执行：

$ ./backup.sh
repository 40406e77 opened (version 2, compression level auto)
using parent snapshot f12a2a03
[0:00] 100.00%  3 / 3 index files loaded

Files:           0 new,    14 changed,    84 unmodified
Dirs:            0 new,     3 changed,     5 unmodified
Added to the repository: 42.141 MiB (1.955 MiB stored)

processed 98 files, 232.051 MiB in 0:01
snapshot 991125a8 saved
repository 40406e77 opened (version 2, compression level auto)
using parent snapshot 409b1ffa
[0:00] 100.00%  4 / 4 index files loaded

Files:           0 new,     0 changed,     2 unmodified
Dirs:            0 new,     0 changed,     4 unmodified
Added to the repository: 0 B   (0 B   stored)

processed 2 files, 12.159 KiB in 0:00
snapshot cd70aaa5 saved
repository 40406e77 opened (version 2, compression level auto)
using parent snapshot e81ce055
[0:00] 100.00%  4 / 4 index files loaded

Files:          12 new,    14 changed,    86 unmodified
Dirs:            0 new,     5 changed,     0 unmodified
Added to the repository: 89.632 MiB (3.040 MiB stored)

processed 112 files, 840.087 MiB in 0:01
snapshot 426b9334 saved

crontab -e 增加备份任务表达式

#PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
# 用于测试的表达式，每分钟执行一次
*/1 * * * * /home/work/backup.sh >> /home/work/backup.log 2>&1
# 最终的表达式，每天晚上 00:30 执行一次
30 0 * * * /home/work/backup.sh >> /home/work/backup.log 2>&1

主机 K8s-Master1 | 192.168.39.82 | K8s控制节点-1 的定时备份

这里仅贴出 backup.sh 的内容，其他操作与 LSB-Nginx 主机相同。

脚步内容：

export AWS_ACCESS_KEY_ID=iPpXf8yoWrqHDsj3fPbe
export AWS_SECRET_ACCESS_KEY=YlrpuMbHV4IPOwEYpquWIJeAXm00afTGYzskPIih
# 备份系统日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/K8s-Master1 --password-file /root/resticpasswd backup /var/log

主机 K8s-Master2 | 192.168.39.165 | K8s控制节点-2 的定时备份

这里仅贴出 backup.sh 的内容，其他操作与 LSB-Nginx 主机相同。

脚步内容：

export AWS_ACCESS_KEY_ID=iPpXf8yoWrqHDsj3fPbe
export AWS_SECRET_ACCESS_KEY=YlrpuMbHV4IPOwEYpquWIJeAXm00afTGYzskPIih
# 备份系统日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/K8s-Master2 --password-file /root/resticpasswd backup /var/log

主机 K8s-Master3 | 192.168.39.233 | K8s控制节点-3 的定时备份

这里仅贴出 backup.sh 的内容，其他操作与 LSB-Nginx 主机相同。

脚步内容：

export AWS_ACCESS_KEY_ID=iPpXf8yoWrqHDsj3fPbe
export AWS_SECRET_ACCESS_KEY=YlrpuMbHV4IPOwEYpquWIJeAXm00afTGYzskPIih
# 备份系统日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/K8s-Master3 --password-file /root/resticpasswd backup /var/log

主机 K8s-Worker1 | 192.168.39.107 | K8s工作节点-1 的定时备份

这里仅贴出 backup.sh 的内容，其他操作与 LSB-Nginx 主机相同。

脚步内容：

export AWS_ACCESS_KEY_ID=iPpXf8yoWrqHDsj3fPbe
export AWS_SECRET_ACCESS_KEY=YlrpuMbHV4IPOwEYpquWIJeAXm00afTGYzskPIih
# 备份系统日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/K8s-Worker1 --password-file /root/resticpasswd backup /var/log

主机 K8s-Worker2 | 192.168.39.100 | K8s工作节点-2 的定时备份

这里仅贴出 backup.sh 的内容，其他操作与 LSB-Nginx 主机相同。

脚步内容：

export AWS_ACCESS_KEY_ID=iPpXf8yoWrqHDsj3fPbe
export AWS_SECRET_ACCESS_KEY=YlrpuMbHV4IPOwEYpquWIJeAXm00afTGYzskPIih
# 备份系统日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/K8s-Worker2 --password-file /root/resticpasswd backup /var/log

主机 K8s-Worker3 | 192.168.39.217 | K8s工作节点-3 的定时备份

这里仅贴出 backup.sh 的内容，其他操作与 LSB-Nginx 主机相同。

脚步内容：

export AWS_ACCESS_KEY_ID=iPpXf8yoWrqHDsj3fPbe
export AWS_SECRET_ACCESS_KEY=YlrpuMbHV4IPOwEYpquWIJeAXm00afTGYzskPIih
# 备份系统日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/K8s-Worker3 --password-file /root/resticpasswd backup /var/log

主机 RDS-MySQL | 192.168.39.87 | MySQL 服务、Redis 服务 的定时备份

这里仅贴出 backup.sh 的内容，其他操作与 LSB-Nginx 主机相同。

脚步内容：

export AWS_ACCESS_KEY_ID=iPpXf8yoWrqHDsj3fPbe
export AWS_SECRET_ACCESS_KEY=YlrpuMbHV4IPOwEYpquWIJeAXm00afTGYzskPIih
# 备份系统日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /var/log
# 备份 MySQL 日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /www/server/data/mysql-slow.log
# 备份 MySQL 数据
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /www/server/data
# 备份 Redis 日志
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /www/server/redis/redis.log
# 备份 Redis 数据
/usr/local/bin/restic -r s3:http://192.168.39.190:9000/backup/RDS-MySQL --password-file /root/resticpasswd backup /www/server/redis/dump.rdb

基于 SFTP 协议的定时备份

主机 OSS-MinIO | 192.168.39.190 | OSS 对象存储 的定时备份

这里仅贴出 backup.sh 的内容，其他操作与 LSB-Nginx 主机相同。

脚步内容：

# 执行数据备份
/usr/local/bin/restic -r sftp:root@192.168.39.94:/backup_192.168.39.190 --verbose backup /var/log --password-file /root/resticpasswd
作者：Jeebiz  创建时间：2024-08-19 12:09
最后编辑：Jeebiz  更新时间：2024-11-01 12:20