1.0.0 Release IaaS

This commit is contained in:
il
2026-03-15 04:41:02 +09:00
commit a7365da431
292 changed files with 36059 additions and 0 deletions
+104
View File
@@ -0,0 +1,104 @@
# Operation
## Migration backup
When the migration is decided, the manual backup after shutting all services down is necessary. Run the DB dump backup and kopia backup manually before reset the system.
## Windows
- Following the process step.
- [01-windows.md](./01-windows.md)
## Certificates and wireguard
- Following the process two steps.
- [02-certificates.md](./02-certificates.md)
- [03-wireguard.md](./03-wireguard.md)
## console
### Data restore
Only when kopia repository exists.
```bash
kopia repository connect --override-username="console" --override-hostname="console.ilnmors.internal"
kopia snapshot list --all
# mount volumes
kopia mount $CONSOLE_SNAPSHOT_ID ~/workspace/homelab/volumes/console
kopia mount $INFRA_SNAPSHOT_ID ~/workspace/homelab/volumes/infra
kopia mount $APP_SNAPSHOT_ID ~/workspace/homelab/volumes/app
# Set initiating db dump files
cp ~/workspace/homelab/volumes/infra/cluster/cluster.sql ~/workspace/homelab/config/services/containers/infra/postgresql/init/cluster.sql
# ...
```
### Provisioning
Ansible playbooks should be declarative. This won't contain complex branch logics \(Declarative over imperative\). Playbooks describes what should be there, not how to. The basic rule is manual destroy and auto reprovisioning.
#### vmm and fw
- Following the process step.
- [04-hypervisor.md](./04-hypervisor.md)
```bash
# Sign on vmm's host ssh keys and set libvirt environment # Ignore fingerprint just once
cd ~/workspace/homelab/ansible
# Disconnect all internet connection except LAN between vmm
ansible-playbook playbooks/vmm/site.yaml --tags "init" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
# Create fw
ansible-playbook playbooks/vmm/create_vm.yaml --tags "fw"
# Make it sure disable other NIC (Wifi or etc) before run playbooks, and set gateway
ansible-playbook playbooks/fw/site.yaml --tags "init" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' # When nftables is changed, crowdsec handlers work. However, when it is in initiating, handler is failed. It is normal work.
ansible-playbook playbooks/fw/site.yaml --tags "site"
# Set vmm after air-gap
ansible-playbook playbooks/vmm/site.yaml --tags "site"
```
## External hardware on CLIENT
- Following the process two steps.
- [05-hardwares.md](./05-hardwares.md)
- [06-kopia.md](./06-kopia.md)
## Other vms
```bash
# Kopia mount
## Create Kopia password file
touch /etc/secret/$CONSOLE_UID/kopia_password
### Add kopia.user.console value from secrets.yaml
## Connect repository
KOPIA_PASSWORD="$(cat /etc/secrets/$CONSOLE_UID/kopia_password)" \
/usr/bin/kopia repository connect server \
--url=https://nas.ilnmors.internal:51515 \
--override-username=console \
--override-hostname=console.ilnmors.internal
## mount kopia directories
kopia mount $INFRA_SNAPSHOT_ID ~/workspace/data/volumes/infra
kopia mount $APP_SNAPSHOT_ID ~/workspace/data/volumes/app
## bringing sql dump files to init directory
cp ~/workspace/data/volumes/infra/cluster/$FILE_NAME ~/workspace/config/services/containers/infra/postgresql/init/pg_cluster.sql
## repeate for every service
# Create vms (Possible tags = ["fw", "infra", "auth", "app"])
ansible-playbook playbooks/vmm/create_vm.yaml --tags "$VM_NAME"
# Sign on vms' host ssh keys # Ignore fingerprint just once
ansible-playbook playbooks/$VM_NAME/site.yaml --tags "init" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
ansible-playbook playbooks/$VM_NAME/site.yaml --tags "site"
# app
ansible-playbook playbooks/app/site.yaml --tags "init" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
## Rsync app data
rsync -avz --numeric-ids -e ssh --rsync-path="sudo rsync" ~/workspace/homelab/volumes/kopia/app/ app@app.ilnmors.internal:/home/app/data/
# --numeric-ids: set owner and groups based on uid and gid
# -e ssh --rsync-path="sudo rsync": run rsync as root permission, sudo password needed
# Maybe rsync-path is run on background, it might not print prompt for password. Just enter the password
ansible-playbook playbooks/app/site.yaml --tags "site"
# Create console snapshot
kopia snapshot create ~/workspace
```
+255
View File
@@ -0,0 +1,255 @@
# Windows prerequisite
## Installation
- Language: English (US)
- Region: Korea
- Keyboard: US
- Internet connection
- Account setting: DOS prompt (Shift + F10)
```dos
start ms-cxh:localonly
```
## Initial configuration
### Time and language
- Preferred Languages: Add a Language: Korean
- Don't set additional options
- Typing: Advanced Keyboard setting: Override for default input method: Korean: Microsoft IME
### Set the password
- Settings: Accounts: Sign-in options
- Password
- PIN
- Windows Hello
### Disable unused functions
```PowerShell
# Administrator privileges are required
## Disable Recall
DISM /Online /Disable-Feature /FeatureName:Recall /NoRestart
## Disable Copilot
reg add "HKCU\Software\Policies\Microsoft\Windows\WindowsCopilot" /v TurnOffWindowsCopilot /t REG_DWORD /d 1 /f
reg add "HKLM\Software\Policies\Microsoft\Windows\WindowsCopilot" /v TurnOffWindowsCopilot /t REG_DWORD /d 1 /f
## Delete programs
Get-AppxPackage *bingnews* | Remove-AppxPackage
Get-AppxPackage *bingweather* | Remove-AppxPackage
Get-AppxPackage *gethelp* | Remove-AppxPackage
Get-AppxPackage *tips* | Remove-AppxPackage
Get-AppxPackage *feedbackhub* | Remove-AppxPackage
Get-AppxPackage *solitairecollection* | Remove-AppxPackage
Get-AppxPackage *windowscommunicationsapps* | Remove-AppxPackage
Get-AppxPackage *outlookforwindows* | Remove-AppxPackage
Get-AppxPackage *people* | Remove-AppxPackage
Get-AppxPackage *clipchamp* | Remove-AppxPackage
Get-AppxPackage *todos* | Remove-AppxPackage
Get-AppxPackage *quickassist* | Remove-AppxPackage
Get-AppxPackage *windowsmaps* | Remove-AppxPackage
Get-AppxPackage *xbox* | Remove-AppxPackage
Get-AppxPackage *gamingapp* | Remove-AppxPackage
## Disable telemetry
Stop-Service DiagTrack
Set-Service DiagTrack -StartupType Disable
Stop-Service dmwappushservice
Set-Service dmwappushservice -StartupType Disable
## Compact OS configuration
compact /compactos:always
```
### Delete programs
- Add or remove programs
- Copilot
- Family
- Microsoft 365 (Other languages; except en-us, ko-kr)
- Microsoft 365 copilot
- Microsoft edge game assist
- Microsoft Onedrive
- Microsoft Teams
- Speech pack
### Edge configuration
- Settings: System and performance: System
- Disable All options
### Explorer configuration
- Settings: Privacy
- Disable All options
### Login on Microsoft account
sign in on app only
- surface app
- microsoft office 365
### Install Microsoft app
- Microsoft PC manager
### Firewall configuration
- WindowsDefender Firewall:Inbound Rules:
- File and Printer Sharing (Echo Request - ICMPv4-In) - Profile: Private, Public
- General: \[x\] Enable
- Scope: 192.168.1.0/24, 192.168.10.0/24, 192.168.99.0/24
- File and Printer Sharing (Echo Request - ICMPv6-In) - Profile: Private, Public
- General: \[x\] Enable
- Scope: fd00::/8
- Apply
### NIC VLAN configuration
<!-- There is no physical managed switch, only hub. It is necessary to set so that filter vlan tag from the dummy hub -->
- Device Manager:Network Adapters:Surface network adpaters
- Advanced:Priority & VLAN: Priority & VLAN disable
## Set WSL2
### Create wsl config
- C:\Users\$USERNAME\.wslconfig
```ini
[wsl2]
processors=4
memory=4294967296
swap=0
defaultVhdSize=34359738368
networkingMode=Mirrored
```
### Installation
```PowerShell
# Run with administrator's authority
# Install wsl
wsl --install -d Debian
# Enter new UNIX username: debian
# Enter new password: debian
wsl --shutdown
# --- Uninstall Debian ---
# wsl --unregister Debian
# wsl --uninstall
# ------
```
### WSL Debian configuration
```bash
# PowerShell
wsl -d Debian
# bash
## User create
sudo groupadd -g 2000 svadmins
sudo useradd -u 2999 -g svadmins -G sudo -c "Console Client" -m -d /home/console -s /bin/bash console
sudo passwd console
# New password: [password]
exit
# PowerShell
wsl --shutdown
wsl -d Debian -u console
# bash
## User delete and set wsl.conf
sudo userdel -r debian
echo '[user]' | sudo tee -a /etc/wsl.conf
echo 'default=console' | sudo tee -a /etc/wsl.conf
echo '[network]' | sudo tee -a /etc/wsl.conf
echo 'generateHosts = false' | sudo tee -a /etc/wsl.conf
exit
wsl --shutdown
wsl -d Debian
## package
sudo apt update && sudo apt upgrade
sudo apt install curl jq gnupg git fuse3
sudo apt install ansible-core ansible-lint --no-install-recommends
ansible-galaxy collection install community.libvirt
ansible-galaxy collection install community.general
ansible-galaxy collection install ansible.posix
## SOPS (arm64)
## Check for latest release: Current version: 3.12.1
## x86_64 url: https://github.com/getsops/sops/releases/download/v3.12.1/sops_3.12.1_amd64.deb
curl -L -o sops_3.12.1.deb https://github.com/getsops/sops/releases/download/v3.12.1/sops_3.12.1_arm64.deb
sudo apt install ./sops_3.12.1.deb
rm -rf ./sops_3.12.1.deb
mkdir ~/workspace
```
### VS Code
#### VS Code Download
- Download from https://code.visualstudio.com/Download#
- `User Installer Arm64`
#### VS Code configuration
- WSL extension\(`Ctrl + shift + x`\)
- Install `WSL` by Microsoft
- Remote Explorer:Debian:Connect in Current Windows
- `Ctrl + k` and `Ctrl + o`
- Open folder: `/home/console/workspace`
- `` Ctrl + shift + ` `` for Terminal
- Extensions\(`Ctrl + shift + x`\)
- Install `Ansible` by RedHat
### Playbooks
```bash
# Ansible playbook
# Copy all files "workspace" directory to Debian via VS Code
cd ~/workspace/homelab/ansible
ansible-playbook playbooks/console/site.yaml --tags "init"
```
#### Terminal configuration
- Terminal: Setting: Startup
- Default profile
- Debian
- Profile: Debian:
- Icon
- "ms-appx:///ProfileIcons/{61c54bbd-c2c6-5271-96e7-009a87ff44bf}.png"
- Starting Directory
- ~
## Secret management
### Flow
- Secret data
- `secrets.yaml`
- encrypted by sops with age-key
- age-key
- `age-key.gpg` and `ansible/group_vars/all.yaml`
- encrypted by gpg and ansible vault with master key
- Master key
- The key which has above 40 characters containing upper and lower letters, numbers, and special letters
- managed by physical media \(Mind, MDisc, paper\) as file, string, and QR
- This value is never saved in server or console.
- Root CA \(including ssh CA\) must not be deployed.
- The tasks with root CA must be performed manually. The source of Trust is the most important in security.
- Intermediate CA can be deployed.
- Intermediate CA is operated as a live server.
- Intermediate CA can be revoked by the root CA
- Only encrypted intermediate CA's private key is deployed.
- CA server decrypts intermediate CA's private key in memory at runtime.
+169
View File
@@ -0,0 +1,169 @@
# Certificates
Create and renew certificates are very important, and very barely executed. It is managed manually without ansible.
#### PKI CA signed offline
step-cli is installed by ansible playbook for console.
```bash
# Generate CA key password
openssl rand -base64 32 > /run/user/$UID/root_ca_password
openssl rand -base64 32 > /run/user/$UID/intermediate_ca_password
# Save the values in `secrets.yaml`
# Create CAs \(Key and cert)
# Root CA
step certificate create \
"ilnmors.internal Root CA" /run/user/$UID/root_ca.crt /run/user/$UID/root_ca.key \
--password-file /run/user/$UID/root_ca_password \
--profile root-ca \
--not-after 87600h
# Save the key and crt files content in `secrets.yaml`
# Intermediate CA
step certificate create \
"ilnmors.internal Intermediate CA" /run/user/$UID/intermediate_ca.crt /run/user/$UID/intermediate_ca.key \
--password-file /run/user/$UID/intermediate_ca_password \
--profile intermediate-ca \
--ca /run/user/$UID/root_ca.crt \
--ca-key /run/user/$UID/root_ca.key \
--ca-password-file /run/user/$UID/root_ca_password \
--not-after 43800h
# Save the key and crt files content in `secrets.yaml`
# fw
step certificate create \
"crowdsec.ilnmors.internal" /run/user/$UID/crowdsec.crt /run/user/$UID/crowdsec.key \
--profile leaf \
--san crowdsec.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
step certificate create \
"blocky.ilnmors.internal" /run/user/$UID/blocky.crt /run/user/$UID/blocky.key \
--profile leaf \
--san blocky.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
# infra
step certificate create \
"postgresql.ilnmors.internal" /run/user/$UID/postgresql.crt /run/user/$UID/postgresql.key \
--profile leaf \
--san postgresql.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
step certificate create \
"ldap.ilnmors.internal" /run/user/$UID/ldap.crt /run/user/$UID/ldap.key \
--profile leaf \
--san ldap.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
step certificate create \
"prometheus.ilnmors.internal" /run/user/$UID/prometheus.crt /run/user/$UID/prometheus.key \
--profile leaf \
--san prometheus.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
step certificate create \
"loki.ilnmors.internal" /run/user/$UID/loki.crt /run/user/$UID/loki.key \
--profile leaf \
--san loki.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
# DSM
step certificate create \
"nas.ilnmors.internal" /run/user/$UID/nas.crt /run/user/$UID/nas.key \
--profile leaf \
--san nas.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
## Recreate leaf certificates
## update secrets.yaml
step certificate create \
"crowdsec.ilnmors.internal" /run/user/$UID/crowdsec.crt /run/user/$UID/crowdsec.key \
--profile leaf \
--san crowdsec.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password -f
# print
cat /run/user/$UID/crowdsec.key
cat /run/user/$UID/crowdsec.crt
# Verify
step certificate verify /run/user/$UID/test.crt --roots /run/user/$UID/root_ca.crt
# Inspect
step certificate inspect /run/user/$UID/test.crt
# validate date
sudo step certificate inspect --format json /run/user/$UID/test.crt | jq '.validity.end'
# margin date
echo "$(( ($(date -d 2028-07-17T03:50:10Z +%s) - $(date +%s)) / 60 / 60 / 24 ))"
# Delete temporary files
rm /run/user/$UID/root_ca*
rm /run/user/$UID/intermediate_ca*
rm /run/user/$UID/*.key
rm /run/user/$UID/*.crt
```
#### SSH CA
```bash
# Generate SSH CA
ssh-keygen -t ed25519 -f /run/user/$UID/id_local_ssh_ca -C "LOCAL_SSH_CA" -N ""
# Save the key and crt files content in `secrets.yaml`
echo @cert-authority *.ilnmors.internal "$(cat /run/user/$UID/id_local_ssh_ca.pub)" | sudo tee /etc/ssh/ssh_known_hosts >/dev/null && sudo chmod 644 /etc/ssh/ssh_known_hosts
# Signing HOST SSH crt by SSH CA key
ssh-keygen -s /run/user/$UID/id_local_ssh_ca \
-h \
-I "vmm" \
-n "vmm,vmm_init,vmm.ilnmors.internal,init.vmm.ilnmors.internal" \
/run/user/$UID/id_vmm_ssh_host.pub
# This process is automated by ansible
ssh-keygen -L -f /etc/ssh/ssh_host_ed25519_key-cert.pub
# Create SSH client key
ssh-keygen -t ed25519 -f /etc/secrets/$UID/id_console -C "il@ilnmors.internal" -N ""
# Signing SSH client crt by SSH CA key
ssh-keygen -s /run/user/$UID/id_local_ssh_ca \
-I "console" \
-n "vmm,fw,infra,auth,app" \
/etc/secrets/$UID/id_console.pub
# This process is automated by ansible
```
+23
View File
@@ -0,0 +1,23 @@
# WireGuard key get
```bash
wg genkey | tee /run/user/$UID/server_private.key | wg pubkey | tee /run/user/$UID/server_pub.key
wg genkey | tee /run/user/$UID/client_private.key | wg pubkey | tee /run/user/$UID/client_pub.key
wg genpsk | tee /run/user/$UID/preshared.key
# Save the key files content in `secrets.yaml`
# Client wireguard setting
[Interface]
PrivateKey = client_private.key
Address = 192.168.99.20/32, fd00:99::20/128
DNS = 192.168.10.2, fd00:10::2
[Peer]
PublicKey = server_pub.key
PresharedKey = preshared.key
AllowedIPs = 192.168.0.0/16, fd00::/8
Endpoint = vpn.ilnmors.com:11290
PersistentKeepalive = 25
```
+162
View File
@@ -0,0 +1,162 @@
# Hypervisor \(vmm\)
Initiating hypervisor doesn't use ansible. Hypervisor is working on hardware itself, so there is a lot of possible variables like IOMMU id, MAC addresses, etc.
Hypervisor is initiated manually with the configuration files which are stored in USB or External HDD and, WAN connection is from ISP DHCP. All the files in `~/workspace/homelab/data/vmm_init/`
## Installing Debian on server
- BIOS:
- Check BIOS configuration; IOMMU/VT-d
- General:
- Language: English - English
- Location: Other > Asia > South Korea
- Locale: United State - en_US.UTF-8
- Keymap to use: American English
- Network:
- Auto Configuration: Using DHCP
- Server:
- Hostname: vmm
- Domain: ilnmors.internal
- User:
- Root Password: \[blank\]
- Full name for the new user: vmm
- User Name: bootstrap
- User Password: debian
- Partition setting: manual
- 512MiB - EFI system partition \(Booting flag: on\)
- 1GiB - Ext4 Journaling \(Mount: /boot)
- 800 GiB -LVM
- 64GiB: vmm-root - Ext4 Journaling \(Mount: /\)
- 700GiB: vmm-libvirt - Ext4 \(Mount: /var/lib/libvirt\)
- Debian package manager setting
- Scan extra installation media: no
- Mirror country: South Korea
- Archive mirror: deb.debian.org
- Proxy: \[blank\]
- Popularity-contest: no
- Installing packages setting
- \[\*\] SSH server
- \[\*\] Standard system utilities
### Initial configuration
Hypervisor operates pure L2 switch for fw and it never can access WAN without fw after initial configuration. This means, there is an air-gap which means hypervisor cannot access to WAN for a while \(from end of initial setting to the beginning of fw setting\).
Hypervisor operates on hardware. Hardware information is always uncertain, and it is set only once. Managing this process as IaC is over engineering.
```bash
# Mount USB on server
lsblk -l
# /dev/xxx # USB
sudo mkdir /mnt/usb
sudo mount /dev/xxx /mnt/usb
# Setting user and groups
sudo groupadd svadmins -g 2000
sudo useradd -u 2000 -g svadmins -G sudo -c "Hypervisor" -m -d /home/vmm -s /bin/bash vmm
# Installing packages
sudo apt update && sudo apt upgrade
sudo apt install -y \
acl curl jq crowdsec systemd-resolved \
qemu-system-x86 ksmtuned libvirt-daemon-system virt-top \
python3 python3-apt python3-libvirt python3-lxml
# Deploy ssh ca
sudo cp /mnt/usb/vmm/ssh/local_ssh_ca.pub /etc/ssh/
sudo chmod 644 /etc/ssh/local_ssh_ca.pub
sudo cp /mnt/usb/vmm/ssh/sshd_config.d/*.conf /etc/ssh/sshd_conifg.d/
sudo chmod 644 /etc/ssh/sshd_config.d/
# Deploy networkd configuration files
sudo cp /mnt/usb/vmm/network/* /etc/systemd/network/
sudo chmod 644 /etc/systemd/network/*
sudo cp /mnt/usb/vmm/sysctl.d/bridge.conf /etc/sysctl.d/bridge.conf
sudo chmod 644 /etc/sysctl.d/bridge.conf
# Check physical MAC address and modify .link file
ip addr
sudo nano /etc/systemd/network/eth0.link
sudo nano /etc/systemd/network/eth1.link
sudo systemctl disable networking.service
sudo systemctl enable systemd-networkd.service
# Deploy nftables config files
sudo cp /mnt/usb/vmm/nftables.conf /etc/nftables.conf
sudo chmod 700 /etc/nftables.conf
# Fix grub
sudo cp /mnt/usb/vmm/grub.d/iommu.cfg /etc/default/grub.d/iommu.cfg
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
## Intel CPU: intel_iommu=on, AMD CPU: amd_iommu=on
sudo chmod 644 /etc/default/grub.d/iommu.cfg
# Check PCIE ID and IOMMU group
lspci -nn | grep -i -e "SATA" -e "VGA"
# 00:02.0 VGA compatible controller [0300] ... [8086:46d4]
# PCI address: 00:02.0 / Device ID 8086:46d4
# 04:00.0 SATA controller [0106] ... [1b21:1064]
# PCI address: 04:00.00 / Device ID 1b21:1064
# Check the iommu group of devices
## VGA
readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
# ../../../../kernel/iommu_groups/12
ls /sys/kernel/iommu_groups/12/devices/
# 02:00.00
## SATA Controller
readlink /sys/bus/pci/devices/0000\:04\:00.0/iommu_group
# ../../../../kernel/iommu_groups/14
ls /sys/kernel/iommu_groups/14/devices/
# 04:00.00
# To passthrough device, it has to be only device in its IOMMU group or All devices in the same group should be passthroughed simultanaeously.
# Add modprobe.d/vfio.conf
sudo cp /mnt/usb/vmm/modprobe.d/vfio.conf /etc/modprobe.d/vfio.conf
sudo nano /etc/modprobe.d/vfio.conf
# options vfio-pci ids=8086:46d4,1b21:1064
# softdep i915 pre: vfio-pci
# softdep ahci pre: vfio-pci
sudo chmod 644 /etc/modprobe.d/vfio.conf
# Apply vfio configurations
sudo update-grub
sudo update-initramfs -u
# umount usb
sudo umount -f -l /mnt/usb
# reboot system
sudo reboot
# Connect console and vmm via lan cable, Set console's NIC as 192.168.1.11
# delete default user
# ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vmm_init
# login: vmm
# check vfio-pci
lspci -nnk -d 8086:46d4
lspci -nnk -d 1b21:1064
# Kernel driver in user: vfio-pci
id vmm
# check sudo group
su - bootstrap
sudo passwd vmm
# enter new password
exit
# delete default user
sudo userdel -r bootstrap
id bootstrap
# id: 'bootstrap': no such user
```
+220
View File
@@ -0,0 +1,220 @@
# Hardware
All hardware configuration is set after fw vm. The MAC address of hardware is reservated on kea-dhcp.
## Vlan switch
### Access VLAN switch
- http://switch.ilnmors.internal \(192.168.1.2, KEA-DHCP, Only IPv4 support\)
- before set ipv6, use ip4 address instead of FQDN
- id: admin, password: admin
- new password: switch.password
### Set VLAN
- VLAN:802.1Q VLAN
- \[x\] Enable - Apply
- VLAN client
- id 1
- name default > client
- member \(Untagged\)
- Port 1 \(Trunk, untagged\): Linux bridge is already process untagged packet as id 1
- Port 3
- Port 4
- Port 5
- Port 6
- Port 7
- Port 8
- VLAN server
- id 10
- name server
- member
- Port 1 \(Trunk, tagged\)
- VLAN user
- id 20
- name user
- member
- Port 1 \(Trunk, tagged\)
- Port 2 \(Not a member of client vlan, untagged\)
- VLAN:802.1Q VLAN PVID setting
- Port 2
- PVID 20
### Verify VLAN configuration
- Manually set consol ip as user
- Connect console to Port 2
- Check internet connection
## DSM \(DS124\)
- https://finds.synology.com/# \(192.168.1.11, KEA-DHCP\)
- Install DSM
### Initial configuration
- Device name: ilnmorsNAS
- Administrator account: il
- Password: dsm.il.password
- automatical update
- synology account - skip, skip anyway
- opt disagree
### Storage
- Storage Manager:Storage:Create:btrfs - Create
### IP address
Kea in fw already reserved DSM's IP. However it is necessary to set IP address statically for stable operation.
- Control Panel:Network:Network Interface:LAN
- Edit:IPv4:Use manual configuration
- Edit:IPv6:Use manual configuration
### Certificates
- Control Panel:Security:Certificate
- Replace an existing certificate:synology
- Description: ilnmors.internal
- Private Key
- Certificate
- Intermediate certificate
- Edit: For: Set as default certificate
- Setting \(!CAUTION!\)
- Even though you set the certificate as default, you have to set certificate for each services.
- configure: service: certificate: nas.ilnmors.internal
## Authelia OIDC
- **!CAUTION!** It can be set after authelia is implemented
- Following [here](../../../config/containers/auth/authelia/config/authelia.yaml.j2) for Authelia configuration
- Control Panel:Domain/LDAP:SSO Client
- Login Settings: \[x\] Select SSO by default on the login page
- Services
- \[x\] Enable OpenID Connect SSO service
- OpenID Connect SSO Settings
- Profile: OIDC
- Account type: Domain/LDAP/local
- Name: Authelia
- Well-Known URL: https://authelia.ilnmors.com/.well-known/openid-configuration
- Application ID: dsm \(what you designated\)
- Application Secret: secret value
- Redirect URI: https://nas.ilnmors.internal:5001
- Authorization scope: openid profile groups email
- Username claim: preferred_username
- Match the user name \(ID\) in DSM and lldap id.
### Kopia in DSM
#### Upload Kopia repository to DSM
- Directory
- Control Panel:shared folder: docker
- Create: docker/kopia - permission: everyone rwx - inheretence to sub directories
- Container manager
- Package Center:Conatiner manager:install
- Upload repository directory from console to DSM
- docker/kopia/repository
- Add certificate - DSM reverse proxy cannot deal with gRPC
- /docker/kopia/config/ssl/nas.key
- /docker/kopia/config/ssl/nas.crt \(including intermediate crt\)
- container manager:images:import
- kopia/kopia
- tags: \{\{ version['packages']['kopia'] \}\}
- run
- image: kopia/kopia
- containername: kopia-server
- \[x\] Enable auto restart
- port: 51515:51515
- volume: /docker/kopia/config:/app/config:rw
- volume: /docker/kopia/cache:/app/cache:rw
- volume: /docker/kopia/logs:/app/logs:rw
- volume: /docker/kopia/repository:/repository:rw
- environment: KOPIA_PASSWORD=$KOPIA.REPOSITORY
- command: server start --no-ui --tls-cert-file=/app/config/ssl/nas.crt --tls-key-file=/app/config/ssl/nas.key --address=0.0.0.0:51515 --log-level=info
- action:Terminal:Create
```bash
kopia repository connect filesystem \
--path=/repository \
--override-username="il" \
--override-hostname="nas.ilnmors.internal"
```
- action:restart
- Set firewall nftables
- Remove kopia_tmp dir from console
### Connection from client
#### Structure
Repository directory - encrypted by server KOPIA_PASSWORD as master key of repository
Server manage ACL with user password, user's KOPIA_PASSWORD. When server verify user with their password, server works with its repository password.
Repository - \(Repository key; master key\) - Server - \(User key; access key\) - Client
- Client knows its access password as KOPIA_PASSWORD to access server. It doesn't know master key, server's KOPIA_PASSWORD. server will control repository by its KOPIA_PASSWORD. their name is the same but it is different.
#### Access
```bash
# Console
# you have to use `'` not `"`
KOPIA_PASSWORD='$kopia.user.user_name' \
/usr/bin/kopia repository connect server \
--url=https://nas.ilnmors.internal:51515 \
--override-username=console \
--override-hostname=console.ilnmors.internal
# This makes repository.config on ~/.config/kopia
# verify with kopia server acl list command
# infra or app
## /etc/secrets/$KOPIA_UID/kopia.env
KOPIA_PASSWORD={{ hostvars['console']['kopia']['user'][node['name']] }}
KOPIA_CONFIG_PATH=/etc/kopia/repository.config
KOPIA_CACHE_DIRECTORY=/var/cache/kopia
KOPIA_LOG_DIR=/var/cache/kopia/logs
KOPIA_CHECK_FOR_UPDATES=false
## .service file
BindReadOnlyPaths=/path/to/backup
# In root namescope, %u always bring 0
BindPaths=/etc/kopia
BindPaths=/etc/secrets/{{ kopia_uid }}
BindPaths=/var/cache/kopia
EnvironmentFile=/etc/secrets/{{ kopia_uid }}/kopia.env
ExecStartPre=/usr/bin/kopia repository connect server \
--url=https://{{ infra_uri['kopia']['domain'] }}:{{ infra_uri['kopia']['ports']['https'] }} \
--override-username={{ node['name'] }} \
--override-hostname={{ node['name'] }}.ilnmors.internal
ExecStart=/usr/bin/kopia snapshot create \
/path/to/backup
```
### Check kopia snapshot
```bash
# snapshot id check
kopia snapshot list [--all]
# Snapshot ID check
kopia ls -l [-r: for recursive] $SNAPSHOT_ID
kopia show -l $SNAPSHOT_ID/file/path
# or
kopia show -l $FILE_ID
```
### Restore
```bash
mkdir -p /mnt/kopia
kopia mount [$SNAPSHOT_ID|all] kopia &
```
+248
View File
@@ -0,0 +1,248 @@
# kopia
## Create repository
```bash
# export KOPIA_PASSWORD="your_repository_password"
# It exists by Docker environment, however when it is used by pure linux server this environemnt is needed.
# *! CAUTION !*
# THIS PROCESS CONTAINING SECRET VALUES.
# WHEN YOU TYPE THE COMMAND ON SHELL, YOU MUST USE [BLANK] BEFORE COMMAND
# e.g.
# shell@shell$ command (X)
# shell@shell$ [BLANK]command (O)
# BLANK prevent the command to save on .bash_history
# After finish this process, use `history -c` and `clear` for just in case.
# Kopia CLI Create repository
mkdir -p /home/console/workspace/kopia_tmp/{repository,config,cache}
/usr/bin/kopia repository create filesystem \
--password="kopia.repository" \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--path="/home/console/workspace/kopia_tmp/repository" \
--description="ilnmors.internal kopia repository" \
--create-only
# block hash: BLAKE2B-256-128
# encryption: AES256-GCM-HMAC-SHA256
# key derivation: scrypt-65536-8-1
# splitter: DYNAMIC-4M-BUZHASH
# Kopia CLI Connect repository
/usr/bin/kopia repository connect filesystem \
--password="kopia.repository" \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--path="/home/console/workspace/kopia_tmp/repository" \
--override-username="il" \
--override-hostname="nas.ilnmors.internal" \
--description="ilnmors.internal kopia repository"
# Connected to repository
# Start kopia server on pure linux server
# /usr/bin/kopia server start \
# --password="kopia.repository" \
# --config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
# --cache-directory="/home/console/workspace/kopia_tmp/cache" \
# --no-ui \
# --address=localhost:51515 \
# --insecure \
# --log-level=info &
# All after these processes reset history.
clear
history -c
```
### Set users
- Container manager: container: kopia-server: Action: Open terminal: Create: bash
```bash
# export KOPIA_PASSWORD="your_repository_password"
# It exists by Docker environment, however when it is used by pure linux server this environemnt is needed.
# Create users
## console@console.ilnmors.internal, infra@infra.ilnmors.internal, app@app.ilnmors.internal
## Repository pasword is not required, because it is already connected
/usr/bin/kopia server users add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user-password="$kopia.user.console" \
console@console.ilnmors.internal
/usr/bin/kopia server users add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user-password="$kopia.user.infra" \
infra@infra.ilnmors.internal
/usr/bin/kopia server users add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user-password="$kopia.user.app" \
app@app.ilnmors.internal
## verify
/usr/bin/kopia server users list \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache"
# ACL
## Console - Run these commands on local container
## snapshot, policy, user, acl, content
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=snapshot" \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=policy" \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=user" \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=acl" \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=content" \
--access=FULL
## app and infra - Run these command on console
### Global pull to remove duplicates
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="infra@infra.ilnmors.internal" \
--target="type=content" \
--access=APPEND
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="app@app.ilnmors.internal" \
--target="type=content" \
--access=APPEND
### Snapshot
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="infra@infra.ilnmors.internal" \
--target=type=snapshot,username=infra,hostname=infra.ilnmors.internal \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="app@app.ilnmors.internal" \
--target=type=snapshot,username=app,hostname=app.ilnmors.internal \
--access=FULL
### Policy
kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="infra@infra.ilnmors.internal" \
--target="type=policy" \
--access=READ
kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="infra@infra.ilnmors.internal" \
--target="type=policy,username=infra,hostname=infra.ilnmors.internal" \
--access=FULL
kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="app@app.ilnmors.internal" \
--target="type=policy" \
--access=READ
kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="app@app.ilnmors.internal" \
--target="type=policy,username=app,hostname=app.ilnmors.internal" \
--access=FULL
```
### Set policy
- Container manager: container: kopia-server: Action: Open terminal: Create: bash
```bash
# compression
/usr/bin/kopia policy set \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--global \
--compression=zstd
# infra for db dump sql
kopia policy set infra@infra.ilnmors.internal \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--compression=zstd \
--keep-latest=10 \
--keep-hourly=0 \
--keep-daily=14 \
--keep-weekly=4 \
--keep-monthly=6 \
--keep-annual=1
# app for media, stateful data
kopia policy set app@app.ilnmors.internal \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--compression=none \
--keep-latest=10 \
--keep-hourly=0 \
--keep-daily=14 \
--keep-weekly=4 \
--keep-monthly=6 \
--keep-annual=1
# verify
kopia server acl list \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
kopia policy list \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
```
## Remove repository from Console
```bash
# After uploading repository to DSM
rm -rf /home/console/workspace/kopia_tmp
history -c
```
+71
View File
@@ -0,0 +1,71 @@
# Git configuration
## Local git
```bash
# git config
git config --global user.name "il"
git config --global user.email "il@ilnmors.internal"
git config --global init.defaultBranch main # Set default branch name as main
# Git repository path
# $PROJECT_REPOSITORY/.git
# Remote registration and push git
cd ~/workspace/homelab
# Create .gitignore management for data directory
echo data/bin/ | tee ./.gitignore
echo data/volumes/ | tee -a ./.gitignore
echo data/images/ | tee -a ./.gitignore
echo docs/archives/textfiles/ | tee -a ./.gitignore
# Select files
# When the set repository as first time
git init
# To commit and tag, you should `git add`
git add .
# Check git changes
git status
git commit -m "1.0.0: Release IaaS baseline"
# git commit -m "docs: update 07-git.md to add the way to manage git system"
# Make current documents as snapshot
git tag -a 1.0.0 -m "IaaS baseline"
# Make special changes
# In this homelab, [Infra_structure_change]:[Services_change]:[Documents_and_configuration_change]
# Tagging and commit should be distinguished.
# The change which affects system: tagging
# The change which doesn't affect system: commit
# Commands
git status # What files are changed
git log # The version record
git diff # What is changed after last commit
git show $tag # Tag version and information check.
git checkout $tag # rollback to tag version
git branch $branch_name # Create branch
git switch $branch_name # Switch branch
git branch # list of branch
git switch main # Switch to main branch
git merge $branch_name # run at the main branch, merge.
git stash # temporary save
git stash pop # get temporary save
# After git switch
git switch service
git rebase --ignore-date main # set date as current time on main branch
```
## Add Service with git
```bash
# Example of establish gitea
git branch caddy-app
git switch caddy-app
git commit -m "0.0.1-caddy-app: Start caddy-app branch"
git tag -a 0.0.1-caddy-app -m "caddy-app: Start caddy-app branch"
## After finishing gitea implement
git switch main
git merge caddy-app
```