il/ilnmors-homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

1.0.0 Release IaaS

Browse Source
This commit is contained in:
il
2026-03-15 04:41:02 +09:00
commit a7365da431
292 changed files with 36059 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/001-architecture.md
+72
View File
@@ -0,0 +1,72 @@
# ADR 001 - Architecture
## Date
- Feb/23/2026
- First documentation
- Mar/4/2026
- Refining sentences
## Status
- Accepted
## Context
- Maintaining multi nodes requires a huge amount of resources, including hardware, electricity, even administrative efforts
- All units which responsible for a single role should follow the Principle of Least Privilege \(PoLP\).
- All units should be interchangeable on standard to avoid vendor lock-in.
## Consideration
### Hypervisor
- Proxmox Virutal Environment \(PVE\)
- Based on Debian.
- PVE uses `qm` command which is not a standard to implement the virtual environment.
- VMware ESXi
- Based on UNIX, deveoped by VMware \(Licence is not free\)
- Hyper-V
- Based on Microsoft Windows \(Licence is not free\)
- Debian Stable
- Based on standard linux \(conservative\)
- Standard virtualization technology 'Libvirt, QEMU, KVM'
### Container
- Docker
- Daemon is used to run containers
- Root authority required
- Socket and network problem is complex \(Docker bridge\)
- docker-compose is an orchestration tool
- Rootless Podman
- Daemonless design
- Root authority not required
- Orchestration is integrated into systemd
- PASTA dumps packet via host-gateway
- K8S, K3S
- HA is based on reprovisioning
- Guarantee availability to create and destroy node dynamically
### IaC
- Terraform
- Strength for initiating low-level and dynamic multi node environment
- Ansible
- Declaritive and easy yaml grammar
- SSH is the way to set
## Decisions
- Use Libvirt/KVM/QEMU on pure linux \(Debian stable\).
- Separate all services by VM, and podman rootless containers without K3S.
- Orchestration stack is not needed in single node system
- Services will be defined by Quadelt to integrate into systemd and to manage them declaratively
- IaC will be implemented by Ansible only declaratively
- All VMs and services are isolated logically by VLAN and nftables
## Consequences
- All VMs have independent borderline by VLAN and nftables
- All services have independent namespaces by podman subuid without daemon
- Ansible can manage all configurations of services and VMs declaratively
docs/adr/002-network.md
+63
View File
@@ -0,0 +1,63 @@
# ADR 002 - Network
## Date
- Feb/23/2026
- First documentation
## Status
- Accepted
## Context
- All L3 communications should be contolled by central firewall node.
- Every firewall rule should be managed by code, not clicks.
- Every edge node takes charge of L2 communication rules.
- IPv4 and IPv6 dual stack should be supported for future network environment.
## Consideration
### Firewall
- OPNSense/pfSense
- vendor lock-in
- GUI environment \(WebGUI\) can contain vulnerability
- It is hard to manage configurations by IaC
- iptables
- Previous standard of Linux
- IPv4 and IPv6 configuration is separated \(no inet\)
- nftables
- New standard of Linux
- English grammar friendly
- IPv4 and IPv6 configuration can be set on the same table \(inet\)
### Flat network structure
- LAN only
- L2 communication doesn't need to pass through gateway
- They use MAC address with ARP. Unicast communication is hard to manage.
- It is hard to manage and apply the policy centrally
## Decisions
- Categorize all nodes in 4 roles 'client', 'server', 'user', 'wg0; vpn connections'
- Implement role separation with VLAN tagging on L2 switch (systemd-networkd bridge)
- VLAN 1: client (vmm, console, nas)
- VLAN 10: server (vmm, infra, auth, app)
- VLAN 20: user (DHCP allocated devices)
- wg0: VPN connections
- Manage the rules based on roles fundamentally, furthermore manage them based on ip and ports when it is needed
- All L3 communication which needs to pass gateway should be on control of firewall \(fw\)
- All nodes including firewall uses nftables \(modern standard\) to manage the packets based on zone concept
- IPv6 has two track strategy
- Client and server, wg nodes has static ULA IP, and use NAT66 for permanency
- User nodes has GUA SLAAC IP from ISP for compatibility
## Consequences
- Firewall takes charge of L3 communications
- Each nodes takes charge of L2 communications and communication from FW
- All nodes can communicate under both IPv4 and IPv6
- All policies can be managed by Code
docs/adr/003-pki.md
+57
View File
@@ -0,0 +1,57 @@
# ADR 003 - PKI
## Date
- Feb/23/2026
- First documentation
- Mar/06/2026
- Add expiry date observation way
## Status
- Accepted
## Context
- All communications except loop-back, should be encrypted
- ssh, and TLS communications needs key and certificates
- Public CA never issues for private domain, '.internal'
- Automate issuing and renewing certificates
- Revocation is not needed in this single and small environment.
## Consideration
### Automate protocol
- JWK/JWT provisioner
- It is hard to manage pre-shared secret values than ACME \(Especially nsupdate\)
- authorized_keys
- When the nodes are increased, it is hard to manage authorized_key.
- SSH ca.pub allow all the certificates signed by ca key, so it is not needed to manage authroized_keys from each hosts.
### Revocation
- CRL/OCSP/OCSP-stappling
- All long-term certificates are managed manually
- All short-term certificates are managed by ACME
- When the certificates are leaked, it is easier to change intermediate CA itself
## Decisions
- Operate private CA
- Root CA \(Store on coldstorage\) - 10 years
- Intermediate CA \(Online server as Step-CA\) - 5 years
- SSH CA - No period
- Manage certificates with two track
- ACME with nsupdate \(using private DNS\) for web services via Caddy - 90 days
- Manual issuing and managing leaf certificate for infra services for independency - 2.5 years
- All manual issuing leaf certificate expiry date is observed by x509-exporter on infra vm
- Manage SSH certificates
- *-cert.pub for host \(with -h options\)
- *-cert.pub for client \(without -h options\)
## Consequences
- Private PKI is operated
- Private SSH CA is operated
- All external/internal communication is encrypted as TLS re-encryption. \(E2EE\)
docs/adr/004-dns.md
+52
View File
@@ -0,0 +1,52 @@
# ADR 004 - DNS
## Date
- Feb/23/2026
- First documentation
## Status
- Accepted
## Context
- Private authoritative DNS is required to use private reserved root domain \(.internal\)
- Split horizon DNS needs DNS resolver, because authoritative DNS must not send queries to other DNS.
- Automatical issuing certificates needs private authoritative DNS which supports nsupdate \(RFC 2136\)
## Consideration
### Resolver DNS
- AdGuard Home
- More powerful query routing than blocky
- Web UI dependency
- Extra function which is not useful \(DHCP, etc ..\)
- Unbound DNS
- Cache and forward zone management is powerful
- more complex than blocky
- cache function is not that needed in this environment
- Internal authoritative DNS only takes charge of internal communication
- All security function is delegated to public DNS like cloudflare \(DNSSEC, etc\)
## Decisions
- Operate BIND9 as authoritative DNS
- BIND9 is developed by ISC as de facto standard of authoritative DNS
- It supports nsupdate perfectly
- Use 2 forward zones
- ilnmors.com for split horizon DNS
- ilnmors.internal for internal DNS
- Uses 4 PTR zones
- Client vlan ipv4, v6 PTR zone
- Server vlan ipv4, v6 PTR zone
- Operate Blocky as resolver and cache DNS
- blocky set the configurations with one code file
- It supports query routing based on its domain - Split horizon DNS
## Consequences
- Implementation of split horizon DNS
- ACME is available via nsupdate
- malicious DNS query is blocked in DNS level
docs/adr/005-ids-ips.md
+54
View File
@@ -0,0 +1,54 @@
# ADR 005 - IDS/IPS
## Date
- Feb/23/2026
- First documentation
## Status
- Accepted
## Context
- Automized detection and prevention threats via network
## considerations
### IPS
- Operate suricata as IDS and IPS
- Suricata IPS mode blocks packets by itself, bypassing nftables integration
- Suricata IPS mode overhead is very higher than IDS mode.
- Suricata IPS mode cannot detect and prevent TLS based communication.
- Homelab server resources are not enough to deal with high overhead.
- fail2ban
- Single node only, no centralized decision sharing
- No community-based threat intelligence (CAPI)
- Regex based log parsing is less structured than CrowdSec's parser/scenario model
- Crowdsec
- Community based rules and sinario \(CAPI\)
- Prevention based on local machines and parsers \(LAPI\)
- Bouncers can use nftables to prevent threats
- Parser can detect even L7 attack under TLS
## Decisions
- Operate suricata as IDS
- suricata IDS mode mirror all packets from interfaces
- match the packets based on its rules and writes log as fast.log, and eve.json
- Operate Crowdsec as IPS
- CrowdSec uses two API server, CAPI, LAPI.
- CAPI updates malicious IPs based on community decisions
- LAPI decides malicious attack based on log from its parser and scenario \(Suricata, caddy, etc\)
- When CAPI, and LAPI decides block some IP based on log parsed by parser and scenarios, bouncer block the malicious accesses.
- Crowdsec register blacklist on nftables or iptables.
## Consequences
- All malicious attack from WAN, even from LAN is controlled by CrowdSec and Suricata
- The firewall maintains high network throughput because blocking is performed efficiently at the OS network level (`nftables` sets) rather than through deep inline packet inspection.
docs/adr/006-secrets.md
+60
View File
@@ -0,0 +1,60 @@
# ADR 006 - Secrets
## Date
- Feb/23/2026
- First documentation
## Status
- Accepted
## Context
- Secret values must not uploaded anywhere as plain values.
- Manage secret values as Git without its real values.
## Considerations
### External KMS
- HashiCorp Vault or Infisical
- Very powerful, but introduces significant compute/memory overhead.
- Creates a "Secret Zero" problem for a single-node homelab environment because of dependency \(DB, or etc\).
- It is hard to operate hardware separated key servers.
### Systemd-credential
- VM environment is hard to apply TPM for systemd-credential
- It is hard to guarantee the idempotency of TPM in virtual environment.
### Ansible vault only
- Ansible vault is powerful options but they are not convenient.
- It is necessary to encrypt separately outside of host_vars or group_vars' file.
- It is hard to add or modify secret values in inventory file.
## Decisions
- All secret data which has yaml format is encrypted by sops with age-key in `secret.yaml`.
- age-key is encrypted by gpg and ansible vault with master key \(including upper, lower case, number, special letters) above 40 characters.
- All secret data always decrypt by `edit_secret.sh` script or ansible tasks from secrets.yaml using age-key encrypted by ansible-vault.
- decrypted secret data is always processed on ramfs, they are never saved on disk.
- Master key is never saved on disk, but only cold storage \(USB, M-DISC, operators' memory\)
- The secret data will be saved on each servers specific directory or podman secret.
- OS:
- path: /etc/secrets
owner: root:root
mode: 0711
- path: /etc/secrets/\$UID
owner: \$UID:root
mode: 0500
- Containers:
- podman secret:
path: /run/secret/\$SECRET_NAME
- These data are never backed up by kopia, or uploaded to git.
## Consequences
- Secret values are not located as a plain text in everywhere except where they are needed.
- It is possible to manage encrypted secret data with Git.
docs/adr/007-backup.md
+61
View File
@@ -0,0 +1,61 @@
# ADR 007 - backup
## Date
- Feb/23/2026
- First documentation
- Feb/27/2026
- Status changed from Deffered to Accepted
## Status
- Accepted
## Context
- All configuration file is managed by git \(IaC\)
- All data file should be backed up by kopia
- All backup should follow 3-2-1 backup cycle
## Considerations
### Backup Tool
- Restic / BorgBackup
- Also excellent deduplicating backup tools.
- However, Kopia provides a highly efficient native server mode (API) and cross-platform compatibility, making it easier to integrate with Synology DSM.
### Database Backup Method
- Physical Backup (Raw data folder backup / File system snapshots)
- Backing up the `/var/lib/postgresql` directory directly while the DB is running can lead to severe data corruption and inconsistency.
- Logical dumps (`pg_dump`) are much safer, database-agnostic, and easier to restore in a homelab environment.
## Decisions
- All configuration files are managed by Git
- Configuration files are based on text
- It is necessary to version, history management.
- Local git -> private Gitea -> github private project \(mirrored\)
- This fulfills 3-2-1 backup rules
- Data files are managed by Kopia and DSM
- Local storage - kopia -> DSM's Kopia repository server - CloudSync -> Cloud server such as OneDrive or Google Drive
- This fulfills 3-2-1 backup rules
- Data files which needs backup
- DB data files: dump
- DB data files are located on infra:/home/infra/containers/postgresql/backups/\{cluster,$service\}/
- App data files: Photos, Media, etc ..
- App data files are located on app:/home/app/data/
- Backed up files: kopia
- DSM:/kopia/{infra,app}/
- Kopia over DSM configuration is managed by runbook with equivalent CLI commands due to vendor limitation
- Restore will be processed manually
- DB data files
- From kopia server to console:$HOMELAB_PATH/data/volume/infra/postgresql/\{cluster,data\}
- APP data files
- From kopia server to APP vm after initiating before deploy services
- Automative backup does not guarantee integrity of data system, so before reset the system conduct manual backup after making sure all services are shutdown.
## Consequences
- All files including configuration and data back ups will fulfill 3-2-1 \(3 Copies, 2 different media, 1 offsite\) back up rules
docs/adr/008-passthrough.md
+43
View File
@@ -0,0 +1,43 @@
# ADR 008 - passthrough
## Date
- Feb/23/2026
- First documentation
## Status
- Accepted
## Context
- App VM needs GPU for heavy workloads like Immich \(hardware transcoding and machine learning\)
- App VM needs huge data storage for its own services
## Considerations
### iGPU
- SR-IOV
- SR-IOV is tech to divide PCIe devices for several VMs.
- Current stable linux kernel doesn't support sr-iov
- It is necessary to use DKMS for sr-iov
- Use DKMS is unstable depending on kernel upgrade, and the most important thing in server is stability.
- When passthrough iGPU itself, hypervisor cannot use graphic function.
- All nodes are managed by SSH session, so it is not a problem.
### Storage
- Each HDD
- Aoostar WTR Pro has their own sata controller for HDD.
- It is more effective and advantageous to passthrough SATA controller itself to manage btrfs RAID10, and HDD health check via S.M.A.R.T values.
## Decisions
- Passthrough N150's iGPU to APP VM
- Passthrough SATA controller to APP VM
## Consequences
- Passthrough iGPU itself to APP vm.
- Passthrough SATA controller to APP vm.
docs/adr/009-isolation.md
+49
View File
@@ -0,0 +1,49 @@
# ADR 009 - isolation
## Date
- Mar/06/2026
- First documentation
## Status
- Accepted
## Context
- Distinguish borderline for service unit including hypervisor, vm, container
## Considerations
### Hypervisor
- As a pure hypervisor, it should only operate virtualization for VM.
- Hypervisor just provides resources and dummy hub \(br\)
### VM
- VM should be distinguished based on their logical role.
- Firewall is responsible for networking
- Infra is responsible for infrastructure services such as DB, Monitoring, CA server
- Auth is responsible for authentication and authorization for services
- App is responsible for applications
### Services
- Services should be distinguished based on their needs \(Privilege\)
- Network stack, backup stack needs special privilege for low level ACL or networks.
- application stack doesn't need low level privilege usually
## Decisions
- Hypervisor: Only supply pure virtualization for VM
- VM: isolated by hypervisor from the other vms based on their role
- Services:
- the one which needs previlieges: Run as native on vm. Don't make overhead for virtualization.
- the one which doesn't need previlieges: Isolate as container from host.
## Consequences
- Guarantee scurity integrity
- Simple operational rules
- Optimize the limited resources
docs/adr/010-provisioning.md
+35
View File
@@ -0,0 +1,35 @@
# ADR 010 - provisioning
## Date
- Mar/06/2026
- First documentation
## Status
- Accepted
## Context
- Every sensitive process should be controlled and managed.
## Considerations
### Automate destroying process
- Destroying is not frequent process, there's no reason to make complex logic
- Sensitive process should be double checked by human manually
## Decisions
- Make provisioning process as auto
- Make sensitive process as manual
- Removing
- Formatting
- Destroying
- Certificates and CA \([ADR-003](./003-pki.md)\)
- Etc. what operator decides that is sensitive
## Consequences
- All process can be under control of the operator
docs/adr/011-tls-communication.md
+33
View File
@@ -0,0 +1,33 @@
# ADR 011 - TLS communication
## Date
- Mar/06/2026
- First documentation
## Status
- Accepted
## Context
- To make administrational policy simple
- Set the principle of TLS communication boundry
## Considerations
### Apply mTLS
- implementing mTLS needs both client certificate and server certificate
- Managing a number of certificates makes a huge operational burden \(expiry date, revocation, etc ..\)
## Decisions
- Set TLS for all communication except 'lo' interface
- When it is possible to activate TLS, apply it even in 'lo' interface
## Consequences
- The policy is set simple
- The overhead is increased little
- Exclude the exceptions on operation \(For the administrator\)
docs/adr/012-alerting.md
+45
View File
@@ -0,0 +1,45 @@
# ADR 012 - Alerting
## Date
- Mar/08/2026
- First documentation
## Status
- Accepted
## Context
- The necessity of observability
- Difficulty of realizing present status of services
- Stable restoring process already exists
## Considerations
### Mail based
- MTA is hard to manage even when operator uses this as relay host
- The mail protocol is complex to implement only for internal mail system for single operator
### Chat based
- Using discord, telegram is easy to get announcment automatically
- The dependency of external services
## Decisions
- Do not operate alerting system
- Single node system for small group doesn't need HA
- When single node system is down, the alerting system is also down.
- When the alert system is needed, implement the system on free instance of external IaaS like AWS or Azure
## Consequences
- Simple management and stable restoring
- Check service availability
- Check from grafana
- Access to node via vpn with ssh
- Access to node via physical VLAN
- Reprovisioning the node
- The additional possibility of extension with Cloud services.
docs/archives/2025-06/on-premise.txt
+12151
View File
File diff suppressed because it is too large Load Diff
docs/archives/2025-06/홈 서버 구축 계획.txt
+102
View File
@@ -0,0 +1,102 @@
홈 서버 구축 계획
1. 목표
a) 집에서 실제로 사용할 다목적 홈 서버를 구축하며, 다양한 서비스를 분산형으로(마치 실제 물리서버로 분리된 것같이) 관리하고, 이를 실제로 사용하며 정보보안에 관한 지식을 적용하고자 함.
b) 막연하게 생각되는 기술스택 및 포토폴리오 및 정보보안기사를 구체적으로 준비 하기를 원함 + Git 등의 여러 운영 서비스에 익숙해 지기를 원함.
c) 당장 이 계획을 시작으로 모든 과정을 문서화 하고 정리하여 스택을 쌓기를 원함.
2. 준비된 H/W 및 설정
a) WTR PRO N150 (N150, 2.5gb lan 포트 x2, RAM x1, m.2 nvme x1, sata x4)
b) 16GB DDR4 3200 x1
c) NVME SSD 256GB x1, SATA SSD 250GB(콜드 백업 용)
d) sata 3.5" HDD 500GB x4
e) Domain Name 구매 예정(Cloudflare를 통한 Domain 발급과, api를 활용한 DDNS 자동화, SSL 인증서 발급 및 자동 갱신 고려, 추가로 DDoS 및 자동화된 포트 스캔등을 방지하기 위하여, 공인 ip가 아닌 DNS를 통해 들어오는 요청만 처리할 수 있도록 고려 중)
f) 추가 라우터(iptime a1004v)
g) 현재 홈 네트워크 구조
WAN - easy mesh(192.168.0.x/24)
{
Main Gateway(Main router) - PC1(static IP)
- PC2(static IP)
- Sub router(static IP) - DS124(Main NAS server) (static IP)
- printer (static IP)
- wireless mobile devices(dynamic IP)
}
Main Router 기준: 국가 차단(한국 캐나다 ip만 허용), 열린 포트(443:https)
NAS 기준(SMB, SSH, SFTP 등 모든 well-known 포트 차단, 오직 http/https 기반 접근만 허용)
3. 고려 사항
a) 실제로 서버 구축시 내/외부망 막론하고 사용이 가능한가?
b) 보안 상으로 안전한가?
c) 준비된 H/W 성능 안에서 구현이 가능한가?
d) 이를 통해 나의 기술 및 지식이 성장할 수 있는가?
5. 기본안(3 가지)
a) 각각의 서버 HW 준비하여 전부 구축
b) 하나의 운영체제(리눅스 예정)를 기반으로 각 서비스를 도커의 컨테이너 기술을 기반으로 격리하여 (파일 서버, 웹 서버, WAS, 프록시 서버, 미디어 서버, 사진 서버 및 도커를 통한 여러 기능들) 사용.
c) 하이퍼 바이져를 기반으로 각 서비스를 VM 기반으로 OS별 격리하여 (현재 라우터에서 VLAN 사용이 불가능한 지금, MACVLAN을 이용하여 iptime v1004에 공인 IP를 할당해 주고, 내부에서 다시 LAN 환경을 구축 = 방화벽 등의 기능 실습 가능) 사용.
6. 각 안 별 장단점
a) 기본안 1
불가능- 금전적인 여유 X
b) 기본안 2
1) 장점
A) 관리의 편리함: 모든 서비스는 하나의 OS에서 구동됨. 따라서 각각의 계정을 통한 권한 설정 및 파일 관리의 용이함(각 서버별 파일을 공유할 시, 네트워크를 활용할 필요 없이 그냥 심볼릭링크를 활용하여 도커 서비스에서 마운트하는 등) 이는 보안설정에도 이점을 줌.
B) 이미 구성된 홈 네트워크: 이미 하나의 공인 IP를 기반으로 홈 네트워크가 구성되어 있어, 하나의 OS에 IP 설정만 해두면, 내부적으로 따로 네트워크를 사용할 필요가 없이 서비스가 가능함(도커의 bridge 기능)
C) 하드웨어 사양이 절약됨: 하나의 OS 상에서, OS 보다 가벼운 콘테이너 기반으로 서비스를 운영하기 때문에 하드웨어 사양이 더 높아질 필요가 없음.
D) 데이터 백업의 편리함: 모든 데이터는 OS 상의 /home 안에 저장이 가능함. 각 서버의 설정은 /home/docker 내부에 저장이 가능하며 각 유저의 설정은 /home/usr에서 관리가 가능함. 이는 백업할 데이터가 /home 하나로 일원화 될 수 있음을 이야기함. 심지어 도커의 설정도 단순히 docker-compose.yml을 통해 백업 가능.
2) 단점
A) 성장 가능성: 이미 모든 도커는 사용이 가능한 상태로 올라와 있으며, 이를 통해 내가 무엇인가를 구축할 필요 없이 그냥 있는 것을 설치해서 사용하면 됨. 따라서 이해하는 것이 아닌, 단순히 글을 보고 복사 붙여넣기를 하는 상황에 빠지기 쉬움.
B) 보안 지식 습득의 불리함: 이미 홈 네트워크를 구성하며 외부로 부터 내부 서버를 공격할 때 어떤 부분을 고려하고 어떤 부분을 생각해야 하는지 어느정도 습득한 상태에서, "내부망"으로 부터의 공격에 대한 정보 지식습득에 불리함. 앞서 언급했듯, 각 서버별 통신에 네트워크를 사용하는 것이 아닌, 도커 내부의 브릿지 네트워크를 사용하므로 굳이 내부망 보안을 신경쓸 필요가 없어짐.
c) 기본안 3
1) 장점
A) 실제 서버를 운영할 때 장비를 분리하는 것 처럼 모든 장비를 논리적으로 분리하여 사용할 수 있음. (방화벽, etc...)
B) MACVLAN 기능을 적용하여 새로운 공인 ip를 적용하여 홈 서버가 제대로 구축되고 안정화 될 때 까지 격리된 네트워크를 형성해 기존의 홈 네트워크와 분리된 새로운 홈 네트워크를 사용할 수 있음 (기존의 사용 중인 home network와 격리된 환경에서 실험 가능)
C) kail linux 등 실제 해킹 실습용 OS를 올려 해킹 공격/방어 실습이 가능
D) 방화벽에 대한 심도 깊은 이해 가능
E) 기본안 2의 내용도 동시에 실습 가능: 모든 서버를 전부 각각의 OS로 올리는 것이 아닌, 한 서버에서 여러 가지 역할을 하는 다목적 서버를 구성할 수 있음. (e.g. 미디어 서버 - immich, vaultwarden, jellyfin 등은 하나에 서버에서 Docker를 통해 구축/ 웹 서버 - Docker를 통해 Wordpress 등의 블로그 서비스 혹은 wiki 서비스 등을 구현 가능)
F) 내가 각각 구현하고 올려본 서버들을 도커 컨테이너화 하여 애드온 형식으로 커스텀 서버를 내가 직접 만들어 볼 수도 있음.
2) 단점
A) 하드웨어 사양과 관련된 문제: 현재 PC의 스펙으로 이 모든 것들이 가능할지는 잘 모르겠음. 다만 각 서버의 기반이 될 Linux 등은 전부 CLI로 구성할 것이기에 스펙적으로 크게 모자랄지는 모르겠음.
B) 데이터 백업의 불편함: 각 데이터를 백업하기 위해 모든 데이터를 /home만 백업하는 기본안 2와는 다르게 각각의 운영체제 및 데이터를 전부 따로 백업해야 함. - 데이터 서버를 하나 둠으로, 데이터들은 데이터 대로, 나머지는 각각의 OS단만 백업하는 형식으로 최대한 간단하게 만들 수 있을 것이라고 예상됨.
7. 최종 선택 및 실행 방향
a) 최종 선택: 기본안 3
b) 네트워크 설정: MACVLAN을 통해 새로운 공인 ip를 새로운 공유기에 할당 후, 기존의 네트워크와 별개의 새로운 "홈 서버만을 위한 네트워크 구성"과 고려 사항 확인
1) 하이퍼 바이져 자체가 L2 스위치 기능을 적용할 수 있는지...?
2) 하이퍼 바이져 내부에 있는 격리된 OS들은 각각의 사설 IP를 공유기(라우터)로 부터 할당 받을 수 있는지?
3) 외부망을 통한 공격/방어 실습을 위한 세팅을 어떻게 할 수 있을지? (e.g. 외부에서 실습을 할 기기의 ip에만 모든 포트 개방 후, 칼리 리눅스로 ip로만 포트 포워딩 등.../ vlan 사용 시 더 편리하겠으나 현재 상황에서 vlan 적용 불가)
4) VLAN 적용은 불가능하지만, VLAN의 개념과 원리 그리고 적용에 대하여 어떻게 공부할 수 있는지 확인해보기
5) 기타 문제 사항들은 직접 경험해보며 트러블 슈팅해보기
6) 지금 상황에서 목표 네트워크 구조
WAN - easy mesh(192.168.0.x/24)
{
Main Gateway1(Main router) - PC1(static IP)
- PC2(static IP)
- Sub router(static IP) - DS124(Main NAS server) (static IP)
- printer (static IP)
- wireless mobile devices(dynamic IP)
}
WAN - (Sub Router/ MACVLAN 이용 공인 IP 할당) - Main Gateway2(192.168.0.x/24) - 하이퍼 바이저
- 접속 콘솔
7) DMZ, TwinIP, MACVLAN 차이: DMZ -> 설정되 기기로 모든 패킷 포트 포워딩/ Twin IP -> 라우터가 가지고 있는 공인 IP를 내부 클라이언트가 사용 가능/ MACVLAN -> 라우터가 가지고 있는 공인 IP가 아닌 MAC 주소를 기반으로 새로운 공인 IP를 클라이언트에 할당
c) 하이퍼 바이저 OS 선택 및 세팅
1) 현재 서버에 사양이 그렇게까지 좋지않고 라이센스 비용을 지불하기 어려우므로, 최대한 가볍고, 오픈 소스 프로그램 및 트러블 슈팅이 유리한 OS를 선택
A) TrueNAS: ZFS 지원/이미 상당부분 편리한 GUI가 있음 - 무겁고/ 공부 목적에 맞지 않음
B) Proxmox: 가볍고 오픈 소스 프로그램, 이미 상당부분 커뮤니티가 형성되어 있음.
C) Windows server: GUI 기반, 익숙함 - 무거움, 오픈소스가 아니며, 라이센스가 필요함.
D) 최종 Proxmox 선택
2) 스토리지 구성
A) 각 OS는 용량이 크지 않으므로 그냥 256GB SSD에 용량 할당하여 설치(OS용 SSD)
B) 데이터 서버의 사용 공간을 HDD 4개를 RAID 0+ 혹은 RAID 1+(차이점에 대해 자세히 공부 필요) 혹은 RAID 5로 묶어서 사용해 보며 정리하기(파일 서버에 가용성 중요, 사실 OS 단도 RAID1을 통한 가용성을 확보하고 싶으나 HW상 SSD가 1개만 지원되므로 백업으로 갈음 - 현재는 가용성이 중요하지 않음.)
C) 각 서버들은 필요한 데이터를 데이터 서버를 참조하여 사용(이 경우 webDAV를 통한 마운트나, smb 등의 프로토콜을 통한 읽기/sftp를 통한 업로드를 고려 중 가능한지는 확인해봐야 함)
d) 기타 홈 서버 구축에 필요한 서버: 필요한 기본 서버들(방화벽, 데이터 서버, DB 서버, 웹 서버, 미디어 서버 등등)을 하나씩 순서대로 그리고 중요도 순서대로 선택해서 올려보기
e) 구축된 서버를 도커를 통한 컨테이너화 해서 Git 배포 경험해보기(ex. 내가 구축한 방화벽 설정 도커)
f) 백업
1) 데이터가 빠진 서버 및 OS(VM) 들 각각의 용량은 그렇게 크지 않으므로, 스냅샷 및 외장 SSD를 통한 핫 백업만 진행 (백업은 실습/구축이 이뤄진 후 실제 데이터가 변경되면 변경될 때마다 증분 백업, HDD 부분은 실사용 전까지 예제만 넣을 것이므로 따로 백업 X)
2) 홈 서버가 어느정도 완성되고 안정화 되기 전까지 중요 데이터는 기존의 NAS를 사용
3) 이후 홈 서버가 안정화 되면, NAS를 백업용으로 사용하고 홈 서버를 기존 홈 네트워크에 가지고 들어와 집안의 서버로 활용하기(이후 MACVLAN을 이용한 다른 네트워크에는 실습용 미니 PC를 하나 더 가지고와 공부)
4) 백업에 종류에 대해 알아보기(전체 백업, 증분 백업, 콜드 백업, 핫백업 등등)
g) 문서화
1) 홈 서버에 wiki 및 웹 서버등이 올라가서 그곳에 백업을 하기 전까지는 notion 등을 활용
2) Git 등의 관리/배포 서비스 등에 대하여 아는 것이 없으므로 공부 후 이것 역시 문서화
3) 홈 서버에 wiki 및 웹 서버등이 서비스 되기 시작하면 notion등에 올라간 문서를 홈 서버로 옮기기
docs/archives/2025-12/01_plans/01_01_plans.md
+814
View File
@@ -0,0 +1,814 @@
Tags: #plan, #common
## Hardware information
### Server Hardware
- Server: Aoostar WTR Pro N150
- N150 Processor (4C4T)
- Samsung DDR4 SO-DIM Memory (31GiB)
- Samsung NVMe SSD (1TB)
- SATA HDD (2TB) x 4
### BIOS configuration
- Access BIOS menu with `del`
- BIOS:Advanced:Hardware Monitor:Smart Fan Function
- CPU Fan / Sys Fan1 / Sys Fan2
- Fan Start Temperture: 45
## VM Plans
### Local MAC address
- Private Local MAC address principal
- 0A:49:6E:4D:\[VM\]:\[Ports\]
### Hypervisor
- OS: Debian13
- CPU: pCPU
- Memory: 3GiB
- This value is just margin of hypervisor. The rest of allocation of VMs.
- KSM is activated by ksmtuned
- MAC: C8:FF:BF:05:AA:B0, C8:FF:BF:05:AA:B1
- Disk: 64GiB (`/`), 700 GiB (`/var/lib/libvirt`)
### Firewall
- OS: OPNsense25.7 (FreeBSD14.3)
- CPU: 2vCPU (cputune.shares 2048)
- Memory: 4GiB
- MAC: 0A:49:6E:4D:00:00, 0A:49:6E:4D:00:01
- Disk: 64GiB - qcow2
- Services:
- Firewall
- IPS/IDS (CrowdSec LAPI, Suricata)
- Kea DHCP
- Central ACME client (automation)
> Do not allow web ui access from WAN, and only allow specific console user to access its web ui. Do not open ssh port at all, when you need to access its console use virsh console on the hypervisor. Because this is the center of security in this homelab.
### Network server
- OS: Debian13
- CPU: 1vCPU (cputune.shares 512)
- Memory: 2GiB
- MAC: 0A:49:6E:4D:01:00
- Disk: 32GiB - qcow2
- Services:
- DDNS script
- AdGuard Home (Resolver DNS)
- BIND9 (Authoritative DNS)
### Authorization server
- OS: Debian13
- CPU: 2vCPU (cputune.shares 1024)
- Memory: 4GiB
- MAC: 0A:49:6E:4D:02:00
- Disk: 64GiB - qcow2
- Services:
- Step-CA (File based)
- Caddy-main (Reverse proxy)
- Infrastructure services won't use caddy.
- OPNsense
- CrowdSec
- AdGuard Home
- Step-CA
- Authellia (idP) + LLDAP (PostgreSQL)
### Development server
- OS: Debian13
- CPU: 2vCPU (cputune.shares 1024)
- Memory: 6GiB
- MAC: 0A:49:6E:4D:03:00
- Disk: 256GiB - qcow2
- Services:
- Postgresql
- Prometheus
- OS: node_exporter(and telegraf)
- VM: libvirt-exporter
- HDD: btrfs-exporter
- Grafana
- Uptime kuma (SQLite)
- Loki, Promtail
- Code-server (File based)
- Postfix, Dovecot, mbsync
- These services are only uses in local mail service (@ilnmors.intenral)
- Postfix (Split mail transper, @ilnmors.internal - directly process and @gmail.com - relayhost)
- Dovecot (IMAP/POP3 server, Save the mail itself)
- mbsync (Get external mail from external service to Dovecot with IMAP protocol)
- Diun (File Provider mode + github provider mode)
> Volume:~/data/containers/code-server/workspace/homelab:/path/of/diun:ro
> File Provider activate and read `.container`
> `.container` needs to have label `Label=diun.enable=true`, `Label=diun.watch_repo=true`.
> In case of local image, use this way.
```yaml
# diun.yml
regopts:
- name: "caddy-auth-source"
image: "docker.io/caddy"
```
```ini
# container file
Label=diun.enable=true
Label=diun.watch_repo=true
Label=diun.regopt=caddy-auth-source
```
### Application server
- OS Debian13
- CPU 4vCPU (cputune.shares 2048)
- Memory: 12GiB
- MAC: 0A:49:6E:4D:04:00
- Disk:
- 256GiB - qcow2
- 4TB - RAID10, BTRFS
- Services (OIDC is supported):
- OpenCloud (The fork of OwnCloud; It includes Radical, and LibreOffice. Radical will be used as only CardDAV)
- Vikunja (CalDav and To-Do list server; PostgreSQL)
- Gitea (Git service, and wiki; PostgreSQL)
- Outline (Small memo note server; PostgreSQL)
- Wiki.js (Report and book editor; PostgreSQL)
- Immich (Photo album; PostgreSQL)
- PeerTube (Private UCC platform; PostgreSQL)
- Funkwhale (Music server; PostgreSQL)
- Kavita (Web bookshelf; SQLite)
- Audiobookshelf(SQLite)
- Actual budget (Budget program; SQLite)
- Paperless-ngx (Paper based information collection; OCR; PostgreSQL)
- Miniflux (RSS management; PostgreSQL)
- Linkwarden (Archaiving Website; PostgreSQL)
- Ralph (IT products management; PostgreSQL)
- Conduit (Rust matrix server; Local DB)
- SnappyMail (Web mail service frontend with Dovecot)
- Vaultwarden (Password manager; PostgreSQL)
- n8n (Following goal, automation the flow; PostgreSQL)
- Services (Foward_Auth is needed):
- Kopia (backup)
- Hompage
- Define access control with yaml file via Authelia.
```yaml
- Admin tools:
- group: admin
- OPNsense
- href: "https://opnsnese.iltnmors.internal"
- Services:
- group: ["admin", "user"]
- Gitea:
- href:"https://gitea.iltnmors.com"
```
- Services(Study):
- Kali (Container)
- Alpine (Container)
> These containers will be isolated by podman network (which has no host gateway) and podman volume. The study and practice will be conducted only in container with `podman exec -it kali bash`
### RDBMS and Redis
#### RDBMS
Postgresql and mariaDB will be provide database for various services on `auth`, `dev`, `app` servers. Each app can access RDBMS, Postgresql and mariaDB on `dev` server which is the central DB server with TLS.
#### Redis
Redis is the cache database, it will operate on each server where Redis is needed `dev`, and `app`, and it supports various app as one container with its own id.
## Matrix
### Network matrix
#### LAN
- Subnet: 192.168.1.0/24
- tag: 1 (Native-untagged)
- Static IPs:
- 1: Gateway (opnsense)
- 2-9: Spare IP for APs
- 10: Hypervisor (vmm)
- 11-12: Console
- 20: Backup Server
- 30: Printer
- Dynamic IP pool
- 100-254
#### VLAN10
- Subnet: 192.168.10.0/24
- tag: 10
- Static IPs:
- 1: Gateway (opnsense)
- 10: Hypervisor (vmm)
- 11: Network server (net)
- 12: Authorization server (auth)
- 13: Development server (dev)
- 14: Application server (app)
#### VPN
- Subnet: 10.10.10.0/24, 10.10.1.0/24
- Static IPs:
- 10.10.10.1: Gateway(opnsense)
- 10.10.10.2: console
- 10.10.10.3: phone
- 10.10.10.4: spare
### UID/GID matrix
#### Local UID/GID
- Pool: 2000-2999
- Static UID:
- 2000: Hypervisor (vmm)
- 2001: Network server (net)
- 2002: Authorization server (auth)
- 2003: Development server (dev)
- 2004: Application server (app)
- Static GID: 2000 (svadmins)
#### LDAP reservation
- pool: 3000 - 60000
#### Sub id
- Subuid/Subgid: 100000:65536
## File management
### File name
- Code files have to use `_` as a separator. (`.sh`, `.py`, etc.)
- Normal files have to use `-` as a separator.
### Directory structure
#### Hypervisor
- ~/data/config/{scripts,server,services,vms}
- ~/data/config/vms/{networks,storages,dumps}
- /var/lib/libvirt/images
#### VMs
- ~/data/{config,containers}
- ~/data/config/{containers,scripts,secrets,server,services}
- ~/data/containers/apps/{certs,etc.}
- ~/kopia
- /etc/secrets/$UID
#### Application server
##### SSD
- ~/data/{config,containers}
- ~/data/config/{containers,secrets,scripts,services}
- ~/data/containers/app/{certs,etc.}
- ~/kopia
- /etc/secrets/$UID
##### HDD
- btrfs
- ~/hdd/data/containers
- ~/hdd/backups
- The scrub timer systemd is required for its integrity.
## Certificates management
- CA: Step-CA (private CA)
- DNS: BIND9 (private authoritative DNS)
### ACME client
- ACME client: opnsense's `os-acme-client`
- Automation:
- `Upload certificate via SFTP`
- `Run command via SSH`
### Caddy
- `caddy-dns/rfc2136`
- `hslatman/caddy-crowdsec-bouncer/crowdsec`
- `hslatman/caddy-crowdsec-bouncer/http`
## Secret management
> It is necessary external KMS or secret management server (like Vault, infisical) not to leave plain data on disk. It is to hard to manage in small homelab environment. Especially, even systemd-cred uses TPM or hardware module, this makes harder to use this on rootless and vm environment. This is why compromise with perfect secret management.
### Secret file
- Files:
- ~/data/config/secrets/.secret.yaml
- ~/data/config/secrets/age-key.gpg
- ~/data/config/scripts/edit_secret.sh
- ~/data/config/scripts/extract_secret.sh
- Directories:
- /etc/secrets
- Ownership: root:root
- permission: 511
- /etc/secrets/$UID/file
- Ownership: $UID:root
- Permission: 500(directory), 400(file)
### Sequence
- Create `.secret.yaml`
- Cerate `age-key`
- Encrypt `.secret.yaml` with `sops` by `age` key
- Modify `.secret.yaml` with `edit_secret.sh`
- Create `podman secret` or `/etc/secrets/$UID/file` with `extract_secret.sh`
> Creating podman secret is always manually conducted by `extract_secret.sh`. There is no plain text of secret data in backup target, or git target.
```yaml
# .secret.yaml
# ~/data/config/secrets/.secret.yaml
# Format of .secret.yaml
# app1.env:
1SECRET: '1secret'
2SECRET: '2secret'
app1.file: |
-----TEXT-AREA-----
contents of 3secret
-----END-AREA-----
# app2.env
3SECRET: '3secret'
4SECRET: '4secret'
# ...
```
#### Secret scripts
- File:
- ~/data/config/scripts/secrets/edit_secret.sh
- ~/data/config/scripts/secrets/extract_secret.sh
```bash
#!/bin/bash
# edit_secret.sh /path/of/secret
set -e
KEY_PATH="$HOME/data/config/secrets"
SECRET_FILE="$1"
usage() {
echo "Usage: $0 \"/path/of/secret/file\""
exit 1
}
if [ -z "$SECRET_FILE" -o ! -f "$SECRET_FILE" ]; then
echo "Error: Secret file path is needed"
usage
fi
if [ ! -f "$KEY_PATH/age-key.gpg" ]; then
echo "Error: There is no key file"
exit 1
fi
# Delete password file after script
cleanup() {
if [ -f "/run/user/$UID/age-key" ]; then
rm -f "/run/user/$UID/age-key"
fi
}
trap cleanup EXIT
echo -n "Enter GPG passphrase: "
read -s GPG_PASSPHRASE
echo
echo "$GPG_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
--output "/run/user/$UID/age-key" \
--decrypt "$KEY_PATH/age-key.gpg" && \
chmod 600 "/run/user/$UID/age-key"
if [ -z "/run/user/$UID/age-key" ]; then
echo "Error: Key file does not exist"
exit 1
fi
gpgconf --kill gpg-agent
SOPS_AGE_KEY="$(cat "/run/user/$UID/age-key")"
SOPS_AGE_KEY="$SOPS_AGE_KEY" sops "$SECRET_FILE"
```
```bash
#!/bin/bash
# extract_secret.sh /path/of/secret (-f|-e <value>)
set -e
KEY_PATH="$HOME/data/config/secrets"
SECRET_FILE=$1
# shift the $2 as $1 ($1 < $2)
shift
# usage() function
usage() {
echo "Usage: $0 \"/path/of/secret/file\" (-f|-e \"yaml section name\")" >&2
echo "-f <type name>: Print secret file" >&2
echo "-e <type name>: Print secret env file" >&2
exit 1
}
while getopts "f:e:" opt; do
case $opt in
f)
VALUE="$OPTARG"
TYPE="FILE"
;;
e)
VALUE="$OPTARG"
TYPE="ENV"
;;
\?) # unknown options
echo "Invalid option: -$OPTARG" >&2
usage
;;
:) # parameter required option
echo "Option -$OPTARG requires an argument." >&2
usage
;;
esac
done
# Get option and move to parameters - This has no functional thing, because it only use arguments with parameters
shift $((OPTIND - 1))
# Check necessary options
if [ ! -f "$SECRET_FILE" ]; then
echo "Error: secret file path is required" >&2
usage
fi
if [ -z "$TYPE" ]; then
echo "Error: -f or -e option requires" >&2
usage
fi
if [ ! -f "$KEY_PATH/age-key.gpg" ]; then
echo "Error: There is no key file" >&2
usage
fi
# Delete password file after script
cleanup() {
if [ -f "/run/user/$UID/age-key" ]; then
rm -f "/run/user/$UID/age-key"
fi
}
trap cleanup EXIT
echo -n "Enter GPG passphrase: " >&2
read -s GPG_PASSPHRASE
echo >&2
echo "$GPG_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
--output "/run/user/$UID/age-key" \
--decrypt "$KEY_PATH/age-key.gpg" && \
chmod 600 "/run/user/$UID/age-key"
if [ ! -f "/run/user/$UID/age-key" ]; then
echo "Error: Key file does not exist" >&2
exit 1
fi
gpgconf --kill gpg-agent
SOPS_AGE_KEY="$(cat "/run/user/$UID/age-key")"
if [ "$TYPE" == "FILE" ]; then
if RESULT=$(SOPS_AGE_KEY="$SOPS_AGE_KEY" sops --decrypt --extract "[\"$VALUE\"]" --output-type binary "$SECRET_FILE") ; then
echo -n "$RESULT"
exit 0
else
echo "Error: SOPS extract error" >&2
exit 1
fi
fi
if [ "$TYPE" == "ENV" ]; then
if RESULT=$(SOPS_AGE_KEY="$SOPS_AGE_KEY" sops --decrypt --extract "[\"$VALUE\"]" --output-type dotenv "$SECRET_FILE") ; then
echo -n "$RESULT"
exit 0
else
echo "Error: SOPS extract error" >&2
exit 1
fi
fi
```
##### Secret value management
- Using `extract_secret.sh`
- Inject secret value to `podman secret` or `/etc/secrets/$UID`
```bash
# /etc/secrets/$UID
# Before use sudo tee, make sure sudo doesn't need password.
# i.e. sudo ps -ef command execute before this command.
# Env file
extract_secret.sh ~/data/config/secrets/.secret.yaml -e "$value" > /run/user/$UID/tmp.env \
&& sudo mv /run/user/$UID/tmp.env /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chown $UID:root /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chmod 400 /etc/secrets/$UID/"$FILE_NAME"
# Normal file
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "$value" > /run/user/$UID/tmp.env \
&& sudo mv /run/user/$UID/tmp.env /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chown $UID:root /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chmod 400 /etc/secrets/$UID/"$FILE_NAME"
# Podman secret
# Podman doesn't supports .env file parsing, you have to enroll all values
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "$value" | podman secret create "[$FILE_NAME|$ENV_NAME]" -
```
#### Use podman secret
```ini
# app.container
[Unit]
Description=app
[Service]
StartExecPre=/bin/bash -c "wait-for-it.sh ip:port -t 0"
[Container]
...
Secret=app.env,type=env,target=$ENVRIONMENT_NAME
# or
secret=app_data.file,target=/path/of/secret/file
...
```
> podman secret save the `secret data` as `plain text` in disk. However, it is not necessary to have full security in small homelab (practically, it is hard to realize in small environment without external secret server like infisical or vault). When the root permission or user permission compromised then it can be readable.
#### Change secret
- Edit `.secret.yaml`
- podman container stop (systemctl)
- `podman secret rm $target`
- Use `extract_secret.sh`
- Restart podman container
### After code-server building
Move all secret file on `dev` server's `code-server` container.
- Files:
- .secret.yaml
- age-key.gpg
- edit_secret.sh
- extract_secret.sh
- Path: $HOME/workspace/homelab/data/common/config/{secrets,scripts} (Mapped volume in container)
- Change the KEY_PATH as `$HOME/workspace/homelab/data/common/config/{secrets,scripts}` on scripts
#### Apply secrets from code-server
Use SFTP and SSH (or Ansible playbook), decrypt the secret values and make a file on container's `/run/user/$UID` and upload to target server's `/run/user/$UID`. Then use ssh remote command to add podman secret or mv command at the code-server container.
It can works on the code-server web terminal. However, if there were problems on caddy, which means not to access web console then just use ssh and podman exec.
## Update and upgrade policy
### Hypervisor
- Never update or upgrade hypervisor before the its stability is verified from other VMs
### VMs
- Make a qcow2 snapshot before major update or upgrade, using `virsh snapshot`
- If there were some problems, rollback using snapshot.
### Containers
- Check the version from Diun.
- Read caution and changes.
- Apply the update via container file (Prepare the containerfile to make image). `systemctl --user daemon-reload` and `systemctl --user restart container`
## Backup policy
### Kopia
- ~/kopia: The directory of kopia configuration files.
- ~/hdd/backups: The destination directory of each server's Kopia.
- Don't backup the live data such as live DB data.
- Only configuration files are backed up in hypervisor.
### Configuration file backup
- Save all configuration files in `code-server` container.
- Path: ~/data/containers/code-server/workspace/homelab
- Use `Gitea` container to track and manage files.
- Apply `Ansible` on `code-server` (Following goal)
### opnsense
- `os-sftp-backup` sends its configuration towards `code-server`
### Application data
#### Common data
- `Kopia` backup files to app server using sftp
- Backup target: ~/data
- Path: ~/hdd/backups
#### DB data
Only backup `dump` db data.
##### Schema backup
```bash
# Dev server
podman exec postgresql sh -c 'pg_dumpall --scheme-only' > ~/data/postgresql/backups/postgresql-cluster-\[date\].dump
```
##### DB data backup
```bash
# VM's application data backup
podman exec application sh -c 'pg_dump -U $DB_USER -p $DB_PW' > ~/data/containers/application/backups/application-\[date\].dump
# app's application data backup
podman exec application sh -c 'pg_dump -U $DB_USER -p $DB_PW' > ~/hdd/data/containers/application/backups/application-\[date\].dump
```
##### Container volume
```ini
# app.container
#...
[Container]
# ...
Volume=~/data/containers/application/backups:/backups:rw
```
##### Example of DB backup senario
```ini
# postgres-db-backup.service
[Unit]
Description=PostgreSQL Database Backup
After=postgresql.service
Requires=postgresql.service
[Service]
Type=oneshot
# %% is needed in systemd, because `%` has special meaning in systemd.
ExecStart=/bin/sh -c 'podman exec postgresql sh -c "pg_dumpall --scheme-only" > ~/data/containers/postgresql/backups/postgresql-cluster-$(date +%%Y-%%m-%%d_%%H-%%M-%%S).dump'
Nice=19
IOSchedulingClass=idle
# Management DB dump file
ExecStopPost=/bin/bash -c `find "~/data/containers/postgresql/backups/" -maxdepth 1 -type f -mtime +7 -delete`
```
```ini
# postgres-db-backup.timer
[Unit]
Description=Run PostgreSQL backup daily at 2:30 AM
[Timer]
# everyday 02:30 AM start
OnCalendar=*-*-* 02:30:00
# Random time to postpone the timer
RandomizedDelaySec=15min
Persistent=true
[Install]
WantedBy=timers.target
```
#### External backup
- Use `Kopia` in app server to backup files to external data server.
#### Verify backup
- Restore random directory from backup on dev server's test directory once a month (or week).
- Check its integrity and availability.
- If there were some problems, check the all backup data and conduct full backup immediately.
## Systemd
### `.service` file
- Path: ~/.config/systemd/user
- Example of `.service`
```ini
# ~/data/config/services/opnsense.service
# ~/.config/systemd/user/opnsense.service
[Unit]
Description=opnsense Auto Booting
After=network-online.target
Wants=network-online.target
# Requires=x.services
[Service]
Type=oneshot
# Maintain status as active
RemainAfterExit=yes
# Wait for other dependent services
# ExecStartPre=%h/data/config/scripts/wait-for-it.sh -h [ip] -p [port] -t 0
# Run the service
ExecStart=/usr/bin/virsh -c qemu:///system start opnsense
# Stop the service
ExecStop=/usr/bin/virsh -c qemu:///system shutdown opnsense
[Install]
WantedBy=default.target
```
### Hypervisor
- Adjust booting sequence of VMs via `.service`
- Use `wait-for-it.sh` and `Requires`
- Sequence
- vmm
- opnsense
- net
- auth
- dev
- app
### Containers
#### Quadlet
- Make the `.container` file
- Path: ~/data/config/containers/\[app_name\]
- Symbolic link path: ~/.config/systemd/containers
- `systemctl --user daemon-reload` makes `.service` file automatically
- If pod is needed, then set `.pod` file
```ini
# app.container
[Quadlet]
# Don't make a dependencies
DefaultDependencies=false
[Unit]
# Pod=app.pod
Description=app
After=network-online.target
Wants=network-online.target
Requires=required.service
[Service]
StartExecPre=%h/data/config/scripts/wait-for-it.sh dev.ilnmors.internal:8080 --timeout=0 --strict
[Container]
# pod=app-pod
Image=localhost/app:1.0.0
Name=app
Port=2080:80/tcp
Port=2443:443/tcp
Volume=%h/data/containers/app/etc:/etc/app:rw
Volume=%h/data/containers/app/data:/app:rw
Secret=app.env,type=env
Secret=app.file,type=file,target=/path/of/secret/file
[Install]
WantedBy=default.target
```
```ini
# app.pod
[Quadlet]
# Don't make a dependencies
DefaultDependencies=false
[Pod]
Name=app
PublishPort=2080:80/tcp
```
docs/archives/2025-12/01_plans/01_02_milestone.md
+167
View File
@@ -0,0 +1,167 @@
Tags: #plan, #milestone, #common
## Homelab Project
### Plans
- [x] Build plans
### Organize theory
- [x] Organize DNS
- [x] Organize DHCP
- [x] Organize PKI
- [x] Organize TLS
- [x] Organize SSO
- [x] Organize Email service
### Organize configuration
- [x] Organize Debian installation
- [x] Organize Debian common configuration
- [x] Organize iptables
- [x] Organize podman
- [x] Organize CrowdSec
- [x] Organize BTRFS
### Hypervisor
- [x] Install Debian13
- [x] Set common configuration
- [x] Set network interfaces
- [x] Set QEMU/KVM and Libvirt environment
### opnsense vm
- [x] Generate opnsense template
- [x] Install opnsense
- [x] Set interface configuration
- [x] Set CrowdSec LAPI configuration (without TLS)
- [x] Set KEA DHCPv4 configuration
### net vm
- [x] Generate net vm template
- [x] Install Debian13
- [x] Set common configuration
- [x] Set network interfaces
- [x] Set DDNS script
- [x] Set BIND container
- [x] Set AdGuard Home container
- [x] Fix DHCP and Static IP server's `resolv.conf`, and opnsense dns
### auth vm
- [x] Generate auth vm template
- [x] Install Debian13
- [x] Set common configuration
- [x] Set Step-CA container
### opnsense vm
- [x] Set ACME client in OPNsense
- [x] Set TLS on OPNsense with ACME client
- [x] Set TLS on CrowdSec LAPI with ACME client
- [x] Set ACME automation
### net vm
- [x] Set TLS on AdGuard Home container with ACME client
### dev vm
- [x] Generate dev vm template
- [x] Install Debian13
- [x] Set common configuration
### app vm
- [x] Generate app vm template
- [x] Install Debian13
- [x] Set common configuration
- [x] Set BTRFS on `$HOME/hdd`
### auth vm
- [x] Set Caddy - auth container (Main caddy)
- [x] Caddy TLS certificates
- [x] Caddy bouncer
- [x] Caddy log agent
- [x] Set crowdsec bouncer
- [x] Set collection in LAPI (parser + scenario)
- [x] Set collection in auth vm
- [x] Set acquis.d/caddy-auth.yaml
- [x] Set LLDAP container
- [x] Set Authelia container
- [x] Forward_Auth setting
### dev vm
- [x] Set Postgresql container
- [x] Set TLS on Postgresql with ACME client
- [x] Set Caddy - dev container (sidecar caddy)
- [x] Verify TLS re-encryption
- [x] Veryfiy Forward_Auth from Caddy - auth
- [ ] Set code-server container
- [ ] Generate container file (with Git and Ansible)
- [ ] Apply SSO with Authelia and Forward_Auth
- [ ] SSH setting
- [ ] Upload opnsense backup file via SFTP
- [ ] Get all server's configuration file via from terminal
### app vm
- [ ] Gitea container
- [ ] DB setting
- [ ] OIDC apply with Authelia
- [ ] Code and configuration file Git
- [ ] Vaultwarden container (User secret management)
- [ ] DB setting
- [ ] OIDC apply with Authelia
- [ ] TOTP setting (recovery code will be saved in .secret.yaml)
### dev container
- [ ] Set Diun container
- [ ] Set Prometheus and grafana container
- [ ] Set Loki and promtail container
- [ ] Set Postfix
- [ ] Set Dovecot
- [ ] Set Fetchmail
- [ ] Set Mariadb conatiner (when it needs)
- [ ] Set TLS on Mariadb with ACME client
### app vm
- [ ] Set Caddy - app container (sidecar caddy)
- [ ] Set app service containers
- [ ] Set all server's Kopia and Gitea (with code-server)
- [ ] Conduct backup verification
- [ ] `Git` all code on Gitea
---
## Following goals
- [ ] Ansible
> To manage and automate this project, the tool of automation is necessary. In modern architecture, Ansible is one of most powerful tools to automate configuration. After the project will be finished, Ansible will be adopted to manage server's configurations. It supports idempotency powerfully, so from the basic configuration the dev-ops system will be applied on this project. Idempotence is very important.
- [ ] self inspection or mock audit
> Every architecture has their own vulnerability. It is because always the administrator itself is the weakest chain in the security. So, it is necessary to inspect the system based on external criteria. There's the list of criteria below.
>
> - ISMS-P - Korean standard
> - ISO/IEC 27001 - International standard/Annex A
> - NIST SP 800-53 - NIST CSF
> - CIS Benchmark - checklist of Debian/OPNsense/RDBMS/etc
> - OWASP Top 10
- [ ] documentation deeper
> The system itself can't prove anything. When the document that everyone can understand what it is supports the system, then the system become the most powerful weapon.
>
> - The code and configuration files, and Ansible playbook based on Git (private Gitea)
> - Architecture Report based on bookstack (As-Is)
> - Policy and Norms Report based on bookstack (To-Be)
> - Audit Report based on bookstack (Proof of Compliance)
- [ ] hacking simulation for public licence
> Use podman network and podman volume, create `kali` and `alpin` container to train and study about hacking in `dev` server. These containers won't combined with `systemd` via `Quadlet`
docs/archives/2025-12/02_theory/02_01_dns.md
+101
View File
@@ -0,0 +1,101 @@
Tags: #theory, #network, #protocol
## DNS (Domain Name System)
In the beginning of the internet, there were a few hosts on networks. It was possible to manage all hosts on network via IP address or domain name in `/etc/hosts` file in each servers. However, it is hard for people to match and remember what IP addresses means. When the internet environment became bigger and bigger, the complex of route the target server would be harder. To solve this problem, the DNS emerged as a translator between IP address and domain name. In modern internet environment, DNS has hierarchy structure from root to TLD, TLD to authoritative server for efficiency.
### Structure of DNS
#### Communication
- DNS: 53 tcp/udp
DNS communication basically uses 53/udp port. However, in the modern internet environment; which means complex environment sometimes the size of packet is above 512 bytes. In this case, DNS uses 53/tcp too. The vulnerability of DNS is that all communication is on plain data. Everyone can conduct sniffing attack towards DNS packet.
- DoT (DNS over TLS): 853 tcp
DoT was developed to encrypt DNS query. DoT uses [TLS](./02_04_tls.md) to request query. This protocol uses TLS. Moreover, because of TLS, nobody can do sniffing attack towards DoT. However, it uses specific port 853. If ISP block the 853 or analyze 853 port, the pattern of usage will be analyzed or even you cannot use DoT itself. Additionally, there is also DNS over DTLS which uses 853 udp.
- DoH (DNS over HTTPS): 443 tcp/udp
DoH is very similar with DoT. It uses TLS, and it was developed to encrypt DNS query. there is just one difference. This uses https(443 tcp/udp) instead of 853 tcp. https is standard of web protocol, so it is hard to analyze someone sends DNS request or common web packets. It means, ISP or government cannot block 443 port itself or analyze the pattern of DNS query. Since 2022, there's the new standard DNS over HTTP/3 which uses 443 udp port.
- DNSSEC (DNS SECurity extensions)
Originally, client couldn't verify integrity of the response from DNS server. If malicious attacker could get authority of cache DNS server to change their records, all clients would get affected. (i.e. pharming attack). DNSSEC is a protocol to guarantee integrity of DNS record. DNSSEC protocol adds some records in zone, RRSIG(Resource Record Signature), DNSKEY, DS, NSEC, CDNSKEY, CDS. All resolver DNS verify integrity of their records to authoritative DNS with these records. This process is similar to PKI, the chain of trust.
- ECH (Encrypted Client Hello)
Basically, client hello packet has SNI (Server Name Indication). Even though all communication under TLS is encrypted, but to start session the packet has to contain the SNI to identify server. To encrypt this information, SNI the ESNI(Encrypted SNI) was developed in 2018 based on TLS 1.3. However, ESNI just could encrypt SNI information. Now, since 2020, the new standard ECH was developed to supersede ESNI. ECH not only encrypt SNI but also encrypt all client hello process. ECH is latest protocol, and it has a lot of dependency in DNS server, service server and client. When all of them supports ECH, then user can use ECH. Because when ECH encrypts client hello data client need the target server's public key (certificate), it has to look up from encrypted DNS (DoH or DoT).
#### Zone
DNS server has zones; Forward zone and Reverse zone.
- Forward zone
Forward zone has basically information of the pair of domain and IP address. The role of this zone is change domain name to IP address. The domains are managed by IANA, TLD is already reserved. (i.e. `.com`, `.org`, etc...) For private network, `.home.arpa` or `.internal` are reserved.
- Reverse zone
Reverse zone also has basically information of the pair of IP address and domain. The role of this zone is change IP address to domain name. To change domain to IP address it uses specific domain name. \[reversed_ip_address\].in_addr.arpa (i.e. 1.168.192.in-addr.arpa)
#### Records
Each zone has their record type. If zone were a kind of DB, record would be a data of DB. There is basic records type below.
- SOA type
Information of ZONE management. Every zone has this SOA type record.
- NS type
Designate authoritative name server of domain zone
- A type
Mapping domain to IPv4 address
- AAAA type
Mapping domain to IPv6 address
- PTR type
Mapping IP address to domain
- CNAME type
Mapping domain to domain. CNAME type is kind of alias of domain. It can't have IP address value. The query acts recursively, and it gets IP address at the end.
#### Key
There is the keys to control DNS records or zone, even DNS server itself.
- rndc key
This key is to control DNS server itself. When rndc key set on DNS server, client can control DNS server with this key like, reboot server, load or unload zone. rndc key is basically generated by `rndc-confgen` command and it is defined on `rndc.conf` and `named.conf`.
- tsig key
This key is to guarantee integrity when the server syncronize zone data between other servers (usually master-slave server). It is possible update records via this key depending on the setting. Therefore, tsig key is usually used for DDNS or DNS-01 challenge. The key is generated in the DNS server, and it defined in `named.conf`.
### DNS Server type
DNS server basically separated as authoritative DNS and recursive DNS.
#### Authoritative DNS
Authoritative DNS has literally authority of domain zone. It doesn't ask recursive queries towards other DNS server in case of the query that is in its authoritative zone. It is necessary to use DNS-01 challenge (ACME protocol).
#### Recursive DNS
Recursive DNS oppositely doesn't have authority of the records in its zone. When it gets query request, it ask recursive query towards authoritative DNS. It can store the information of records (cache) and give response towards client with the cache.
### Split Horizon DNS
Split Horizon DNS means getting different IP address depending on where the client exists. For instance, if there were the domain `example.com`. This domain has its own private IP address, simultaneously own public IP address (from NAT). When client request the query `example.com` in the private network, private DNS would respond its private IP address. However, when the client request the query in the WAN network, public DNS would respond its public address. Client can access `example.com` in both case, but the IP address which client respond are different. To use this protocol, the network route will be efficient because the packet doesn't have to go out to the WAN area in private network. Basically, it is implemented internal authoritative DNS and recursive DNS. Recursive DNS decides where to send the query based on domain.
### DDNS (Dynamic DNS)
Public IP address can be changed by ISP at any time. It is hard (or expensive) to get static public IP address by ISP. However, the service (server) always guarantee their availability regardless what is their IP. DDNS is basically the protocol to change A or AAAA (or CNAME) records in DNS as server's current IP. Server keeps checking their current public IP and when it changes the server send the request to change its A or AAAA records to public authoritative DNS server with authentication with API key or tsig key.
docs/archives/2025-12/02_theory/02_02_dhcp.md
+55
View File
@@ -0,0 +1,55 @@
Tags: #theory, #network, #protocol
## DHCP (Dynamic Host Configuration Protocol)
Before DHCP emerged, every client had to set their own static IP or using RARP(Reverse Address Resolution Protocol. They have critical problems.
- Static IP
- Each host has their own IP regardless they run or not. It cause lack of IP address.
- If administrator made configuration mistake, the network itself could stop. For instance, IP conflict, or subnet configuration error, etc.
- RARP
- RARP works on L2, it makes hard to implement on hardware.
- RARP server had to exist on every subnet(L2), it was inefficient.
To solve this problem, BOOTP was developed and it evolved as DHCP. DHCP works on L3, and like its name they lease IP dynamically. It allocates IP to hosts with 3 steps. Lease, renewal, and release.
### Lease
When DHCP server gets request from host to allocate IP, DHCP server choose one IP address from its subnet pool, and it responds IP/subnet and information to the hosts. It follows the process called DORA.
- DHCP Discover
The host which has no IP address broadcast discover packet including its MAC to local network. Only DHCP server responds this packet, the others discard the packets.
- DHCP Offer
When DHCP server gets discover packet, DHCP server broadcast Offer packet including host's MAC of which sends discovery packet. Only host which sends discovery packet, the others discard the packets.
- DHCP Request
When the host gets offer packet, it consider the network has DHCP server and broadcast request packet to the local network. More than one DHCP server can exist in one network, therefore host broadcast request packet.
- DHCP ACK
When DHCP server gets the Request packet, it searches for an IP address which can be allocated in its pool. When DHCP server finds an available IP address, it sends to ACK packet including IP address and subnet, and optional information(DNS, gateway, etc...) with reservation time.
### Renewal
The host allocated IP from DHCP try to renew its IP before it expires. There are two chances to renew reservation time. The first try to renew is when the reservation time remains half of them; T1. In this time, host uses unicast. If the first try failed, then it try one more time when 87.5% of the lease time has passed; T2. In this time, the host uses broadcast. All try fails, client gives up the leased IP and try the lease process again. DHCP server release the IP address from the host.
- DHCP Request
The host sends request packet to DHCP server as unicast.
- DHCP ACK
When DHCP server gets the request, it sends ACK packet to the host as unicast.
### Release
When the host doesn't use IP address anymore, DHCP server makes IP as available IP in its pool. Especially, client can send `DHCPRELEASE` to DHCP server explicitly when it doesn't need IP address.
### DHCP relay
Commonly, DHCP is located in router. Because router is the center of networks, and it takes charge of number of networks. However, DHCP server doesn't have to be located in router because of existence of DHCP relay. When router gets DHCP packets (DORA), router can relay the packets as unicast between host and DHCP server which are in different subnet.
docs/archives/2025-12/02_theory/02_03_pki.md
+103
View File
@@ -0,0 +1,103 @@
Tags: #theory, #network, #security
## PKI(Public Key Infrastructure)
PKI is defined on RFC4949 and RFC5280. PKI is neither one of the protocols nor algorithms, it is a huge infrastructure itself. It has various parts which are CA(Certification Authority), RA(Registration Authority), VA(Validation Authority) and Certificates.
### CA (Certification Authority)
CA is the most important part in PKI. It is the source of certification for the secure communication. CA has various roles, some can be delegated to the RA or VA. However, CA is essential in PKI. RA or VA are optional to support CA. The purposes of CA are as follows:
- Managing certification policy
- Issuing certificates
- Verifying certificates validity
- Managing CRL(Certification Revocation List)
- Cross-certification between root CAs
CAs often operate in a hierarchy. Each CA has their own role, and they are basically PAA(Policy Approving Authority), PCA(Policy Certification Authority), and CA(Certification Authority). Normally, PAA and PCA are usually combined as root CA, and CA is an intermediate CA.
#### root CA
Root CA is simply CA of CA. It is the source of trust chain. Only a root CA can authenticate itself alongside other root CAs(cross certification) or intermediate CAs below it. Theoretically, and precisely root CA can be divided into PAA and PCA, but practically they operate as one authority, root CA.
#### intermediate CA
Intermediate CA is the authority which issues certificates for end entities we can use (i.e. web servers). Their certificate is digitally signed (the hash value is signed by root CA's private key) by root CA, so every user can verify their trust with root CA's certificate (root CA's public key). The reason for using intermediate CA is for security. If CA's private key was violated, all chain of trust will be disrupted. When only root CA exists and its private key was violated, all secure communication will be threatened. To avoid and manage these threats root CA and intermediate CA are divided.
#### Structure of CA
The basic structure of CA is very simple. It only contains private key, certificate (which is designated by X.509), and provisioner. When CA gets a request of sign on services' certificates it operates as shown below.
- Provisioner checks CSR (Certificate Signing Request) is valid on its policy
- When CSR is valid, it checks the content of certificates(x.509) and its policy
- CSR and content of certificates are valid, CA signs on the certificate with its private key (sign on hash value with CA's private key)
After CA's signing, every client can check its validation with CA's certificates (CA's public key)
### RA (Registration Authority)
Due to CAs many roles, all processes above are made to be inefficient when conducted. Therefore, CA delegates its registration role to RA. RA seeks to examine CSR and certificates' content. When the content of certificate and CSR are valid under its policy, RA would send request to CA to sign.
### VA (Validation Authority)
Additionally, just as RA, CA can delegate its validation role to VA. Basically, VA manages certificates' validation based on CRL (Certificate Revocation List). However, in modern internet environment, it is inefficient to manage all invalid certificates based on a list. To solve this problem, OCSP (Online Certificate Status Protocol) was developed. However, OCSP also has its limitation, the advanced way is being suggested.
#### CRL (Certificate Revocation List)
Once the certificate gets signed from CA, it is always a valid cryptograph before it expires. However, in the real world some certificates have to be revoked in case CA's key is violated or the service no longer operates, etc. So, CA needs to manage these kinds of invalid certificates which doesn't expire, such as letting clients know that their certificates are not valid anymore. This is a concept of CRL. VA releases the CRL to clients, the clients download or update the list to judge the certificates' validation.
#### OCSP (Online Certificate Status Protocol)
CRL was very effective to revoke invalid certificates but when internet is growing, a massive of invalid certificates appears it shows its limitation. Because to check certificates validation, clients need to download and update CRL from CA or VA. There are tons of invalid certificates, and CRL's size become bigger and bigger. OCSP's concept is when client request to check certificates validation, OCSP server response it. To use OCSP, clients don't need to download full CRL anymore. It is very effective to solve CRL's problem. However, it has problems too. OCSP is defined on RFC6960. OCSP's problems are below.
- Clients should expose its identity(IP address) to check certificates validation.
- VA can be SPOF(Single Point of Failure).
- It is very hard to match time syncronization
- The more requests come to VA, the more burden is on VA.(DoS, DDoS problem)
#### OCSP stapling
OCSP's problem was that client should request to VA directly. So, when the certificate providers(like web server, db, etc.) request its validation and give the proof to client it, clients doesn't need to check the validation of certificate to VA anymore. It reduces VA's burden and time synchronization problems.
### Certificate
The form and content of certificate are standardized by X.509. X.509 format contains server's public key, domain(SAN), expired date, and sign of CA, and etc. When CA validate services or people who sent CSR(which contains information to create certificate with x.509 form), CA signs on their certificates (sign on hash value with CA's private key). Then, all clients can validate that certificates with CA's public key(CA's certificate) cryptographically. Ultimately, when clients trust CA, then they can trust services that have certificate signed by CA. However, even if the certificate were valid cryptographically, some certificates would not be valid. It is reason why CRL or OCSP is needed.
#### The way to issue certificate
##### ACME
Basically, certificate is formed by X.509. The services which want to get certificate send request(CSR) to CA. When the services were open internet, and it should get the certificate from public CA like let's encrypt on its public domain name. Public CA always sign on public domain which is open internet, and it is the role of public CA. The protocol to automate this process is ACME. Before ACME protocol, getting a certificate was manual. The person in charge should send CSR to CA, and CA checks the CSR and return the certificate ... All process was manual. ACME protocol made this process automatically. Simply, ACME protocol checks services authority on domain with various way, and issues certificate.
- http-01 challenge
The server creates certificate file for CA. CA accesses the service server's specific directory via 80/TCP. When CA checked the file, CA issues certificate.
- DNS-01 challenge
The server sends request to authoritative DNS server to add specific record with TSIG key(API key). When the specific record successfully added on DNS server and CA can check this, CA issues certificate. This challenge can verify full authority of domain including sub domain, so CA can issue wildcard certificate.
- TLS_ALPN-01 challenge
It uses TLS handshake process on 443/TCP. ALPN(Application Layer Protocol Negotiation) is the protocol to decide what protocol server and client will use. In this challenge, CA sends some token to service server. Service server creates temporary TLS certificates using `acme-tls/1` protocol with token from CA. When CA access to server, it asks `acme-tls/1` protocol to server. The server presents the temporary certificate, then CA issues certificate.
##### Based on identity
ACME is powerful protocol to automate issuing certificate. However, it is necessary to check ownership of domain to use ACME challenge (Usually, using http or https). However, It is possible to use TLS protocol without http or https. Originally, the process was also manual way called PKIX(Public-Key Infrastructure). It is so complex and slow, and it is impossible to use ACME. To automate this process, modern CA uses JWK(JSON Web Key) and JWT(JSON Web Token). The process is below.
- The administrator registers system's public key as JWK at CA.
> JWK is a format of specification of key. It shows key and its information as JSON format.
- The system sends the JWT to get certificate signed by its private key to CA.
> JWT is a specification of what client can do after connection or of proof this request is valid. In this process, JWT substitute CSR.
- CA verify JWT by pre-registered JWK. When JWT is valid, it issues X.509 certificate.
##### Usage of X.509 Certificates
Regardless the way to issue certificate, either ACME or the way based on its identity, the certificate which is already issued is always valid before it is revoked. You can use X.509 certificate from the way based on its identity for https, oppositely you can use X.509 certificate from ACME for other TLS protocols (like LDAPS, DB TLS communication, etc).
#### X.500 and LDAP(Lightweight Directory Access Protocol)
In the beginning of PKI, there was plan to make a server for all certificates. It is X.500. However, the protocol to implement X.500, DAP(Directory Access Protocol) was too complex and heavy to use internet environment. To make way easy and light, LDAP(Lightweight Directory Access Protocol) was developed to store certificates. However, there are a lot more efficient way to manage certificates appeared already. LDAP could not realize X.500, but it is utilized as centralized authentication system like [SSO](./02_05_sso.md), or OS account management.
docs/archives/2025-12/02_theory/02_04_tls.md
+92
View File
@@ -0,0 +1,92 @@
Tags: #theory, #network, #security, #protocol
## TLS/SSL
TLS(Transport Layer Security) is the protocol that encrypts communication with certificates under [PKI](./02_03_pki.md). Originally, the communication encryption protocol was suggested as SSL(Secure Socket Layer) in 1995 by Netscape. Before the emergence of TLS/SSL protocol, all communication in web(or database, some more protocols) used plain text, even it contained sensitive data like password. After 1996, the update of SSL protocol stopped at version 3.0, and it has its own vulnerability. The next version of SSL is TLS. Virtually, a lot of people treat SSL as the same as TLS.
### Encryption
Before talking about TLS, it is essential to understand what is encryption. Basically, encryption is to hide plain data using algorithm and key mathematically. Encryption can categorized by various criteria, but in this case focusing on key.
#### Symmetric key encryption
This way needs only one key to encrypt and decrypt. Just because using one key to encrypt and decrypt, it doesn't require complex calculation. This algorithm is fast, and easy to handle huge data. However, the key is only one so for security the key should be protected.
#### Asymmetric key encryption
This way needs two keys to encrypt and decrypt. It means the key for encryption and the key for decryption are different. This uses very complex algorithm to seperate keys. It makes this slow and hard to handle huge data. However, the keys are devided so, one key can be publically shared.
### Principle of TLS
The PKI is necessary to use TLS communication. TLS uses both of algorithm to encrypt the communication. When the communication were encrypted, the key should be existed. For this process(to generate key for encryption), server needs X.509 certificate. The certificate contains data of server's public key, domain, and extra informations. To start communication, the server and the client conduct specific process, called `TLS-handshake`. A client sends the request to start communication. The server accepts this request, and negotiates the protocol what they will use in communication. After negotiation, they start to process to generate symmetric key for communication. In this part, there are two protocol to generate symmetric key: RSA and Diffie-hellman.
#### RSA (Legacy way)
In this process server and client uses server's public key and private key. A client generates metadata (Pre-master secret) to generate symmetric key. Metadata is encrypted with server's public key and sent to the server. The server decrypts the data from a client with its private key. They uses this data to generate symmetric key. When this process ends, they share the same symmetric key to encrypt content of communication.
- Cons of RSA
Security strength of RSA depends on the server's private key. The server's private key was taken from whom have the record of past communication, all communication can be decrypted. Because session key itself (precisely, pre-master secret) is in the communication encrypted by server's public key. So, this way can't guarantee Forward Secrecy which means security of previous communication. Additionally, RSA needs high resource to encrypt and decrypt. Therefore, modern internet environment RSA way is not used frequently.
#### Diffie-Hellman & ECDHE(Modern standard)
On the other hand, TLS can use Diffie-Hellman algorithm. This process doesn't exchange any clue of session key. In this process, the server and clients generate secret values. And they make specific results from calculation with public parameters. The server's private key signs on its results just for prooving they are not altered and authenticated. When they exchange each other's results, both of them generate the same session key from results and their own secret value. All process is publically open, but except the server and client themselves, nobody can calculate the secret values and session key mathematically. After all process is done, finally the client combines all communication value between server. And it makes hash value of this, and encrypt the hash value with the session key. The client sends it to the server as finished message, and server verify this. This key is not permanent, it is temporary. There is no encryption way, so even if hacker could get the secret key of server, he wouldn't know what was the session key.
- Pros of DHE & ECDHE
On the contrary with RSA case, security strength of DHE & ECDHE doesn't depend on some specific key. Because when it generate session key, the server and client don't send any sensitive values (like pre-master secret) in this protocol. What they send to each other is open values. The server's public key is used for only signing. So, every session key can never be existed after each session; PFS (Perfect Forward Secrecy). Additionally, Elliptic curve way can provide the same strength with shorter key comparing RSA.
The detail of `TLS-handshake` process is below.
### Detail of TLS handshake
#### Start
- Client hello
The client sends the request to server with information including SNI (Server Name Indication), TLS version, cipher suite, and client random. SNI is the server's domain information to access. cipher suite is the protocol of encryption, and client random will be used to generate session key. Basically, the content of `client hello` isn't encrypted. It has a problem that ISP(Internet Service Provider) or government can conduct sniffing attack to the Client hello packet. To solve this problem, ECH (Encrypted Client Hello) was developed. However, ECH needs the support of DNS server, browser(host), and server itself. Today, a lot of servers and DNS servers don't support ECH, so it is hard to apply ECH in every environment.
- Server hello
When the server receives clients request, it designate how to communicate from the list of client hello. The server sends to the client with the information of what protocol, TLS version, cipher suite to use, and certificate, and server random. The way of encryption: RSA, DHE, ECDHE are including in the cipher suite.
#### RSA
When the server and client decide to use RSA way, server sends certificate including public key to the client.
- Client key exchange
The client verifies `Server hello` with server's certificate using client's trusted CA list. The certificate is valid then the client generates `Pre-master secret` and encrypts this value with server's public key to send to the server.
- Session key generation
The server recieves `pre-master secret` encrypted by its public key that only server (who knows the private key of server) can decrypt. When the server decrypts this value successfully, the server and client knows three values: `Client random`, `Server random`, and `Pre-master secret`. The server and client generate `Session key` from these values individually.
#### DHE or ECDHE
DHE and ECDHE follows the exactly same principle of Diffie-hellman algorithm. However, they have a difference of basement in mathematic. DHE is using discrete logarithm, but ECDHE is using elliptic curve discrete logrithm. This is a mathametical topic, so this is skipped. Just important thing is ECDHE is more efficient than RSA or DHE.
> Key length to guarantee the same level of security:
>
> - RSA/DHE: above 2048 bit
> - ECDHE: above 256 bit
- Server key exchange
The server generate the pair of temporary key based on its own principal. The server sends public temporary key to the client. The public temporary key is signed by server's private key.
- Client key exchange
The client verify server's temporary public key with server's public key. Simultaneously, client also generate the pair of temporary key based on its own principal. The client sends its public temporary key to the server.
- Session key generation
Both of them have their pair of temporary key, and other's temporary public key. They generate the same `Pre-master secret` from their temporary secret key and other's temporary public key. This process has no communication, they calculate the same `Pre-master secret` from themselves. When they have `Pre-master secret`, they generate session key from pre-master secret and client random and server random.
#### Finish
After all process is done, the server and client both sends the `Finished message` encrypted by session key. When they can decrypt this messages, the session key is generated properly. They can start communication securely with session key.
### TLS 1.3
Current standard TLS 1.3 changed handshake process as 1-RTT(1 Round Trip Time) and there are no longer available to use RSA way (Legacy).
docs/archives/2025-12/02_theory/02_05_sso.md
+131
View File
@@ -0,0 +1,131 @@
Tags: #theory, #network, #security, #protocol
## SSO (Single Sign On)
When someone wants to use some services, usually they have to identify themselves to the services. The service verifies who you are (authentication), and what you can do (authorization). Originally, each service has their own authentication and authorization system. However, in modern internet environment, a lot of services are organically connected to each other. It is common for one provider to operate various services. So, it is very inefficient and complex to operate authentication and authorization system like this way. To solve this problem, the concept of SSO which is to centralize all services' authentication and authorization system at once emerged.
### LDAP (Lightweight Directory Access Protocol)
To centralize authentication and authorization system, it is necessary to kind of central database of user list naturally. Fortunately, there was a very good and suitable protocol already; LDAP. LDAP was originally suggested to replace DAP (Directory Access Protocol) for X.500 in [PKI](./02_03_pki.md). Even though X.500 didn't materialized, the potentiality of LDAP was selected to materialize SSO. Because LDAP's structure which was designed for manage certificates itself allows to manage user and its authorization. In modern internet environment, many services don't usually use LDAP itself directly in SSO anymore, but it is still used as database of user and their authorization.
#### Structrue of LDAP
LLDAP will be used in this homelab. It is the easiest way to understand LDAP is comparing each part of LDAP with filesystem. Because LDAP itself is fundamentally the database which has the tree structure like filesystem.
##### DN (Distinguished Name)
DN is the unique path of some specific entry in tree. It is like an absolute path in filesystem.
- Example:
- `uid=admin,ou=people,dc=ilnmors,dc=internal`
- `uid=authelia,ou=people,dc=ilnmors,dc=internal`
##### Base DN (Base Distinguished Name)
Base DN is the root of the tree of LDAP. it is like root path of filesystem `/`. All the actions such as search or work are started from Base DN.
- LLDAP setting
- Environment.="LLDAP_LDAP_BASE_DN=dc=ilnmors,dc=internal"
- It means, the domain 'ilnmors.internal' will be the Base DN (the root of tree).
##### Components of DN
DN has components what has ordinary order. The order of these components are `cn(or uid)`, `ou`, `dc`. They are special attributes, which makes DN.
- cn (Common name): The name of object. It is like a file itself in filesystem.
- ou (Organizational Unit): The name of the container which contain entries. It is like a folder in filesystem.
- dc (domain component): The domain components, Usually, it devides the full domain `ilnmors.internal` as `dc=ilnmors`, `dc=internal`.
##### Object (or Entry)
Object is the real data item the DN defines, like file or folder itself in filesystem.
##### ObjectClass
This is the template or blueprint of object. ObjectClass defines what is this object; Either a person or a group, or an organization unit.
- What is this object; user (cn or uid) or group (ou).
- What is the attribute object must have, and might have.
- Example:
- `ObjectClass: person`: This must have `sn` and `cn` attributes.
- `ObjectClass: organizationalPerson`: This is inherited the `ObjectClass: person`, and it can have `mail` or `number` attributes.
- LLDAP uses standard ObjectClass: `person`, `groupOfNames`.
##### Attribute
Attribute is the value of object, like the content of file or folder in filesystem. It is saved as the pairs of key-value. This values are following the definition of ObjectClass.
- Example
- `uid=user1` has the value following `objectClass: person`, `objectClass: organizationalPerson`
- It must have `sn`, and `cn`
- It can have `mail`, or `number`
- It can have special attribute `memberOf: cn=admins,ou=admin_group, ...` (The attribute that shows the group the user belongs to)
### IdP (Identity Provider)
Many of modern services, SP (Service provider), use SAML (Security Assertion Markup Language) or OIDC (OAuth 2.0/OpenID Connect) protocol to implement SSO. Originally, each service had to send request of authentication to central server (which can be LDAP or else) individually to implement SSO. It means each service should protect the sensitive data itself, and some services which have vulnerability can threaten all system. This is why many of modern services use and support SAML or OIDC protocols. The IdP is needed to use these protocols. The IdP acts as the agent of all services to substitute authentication process on their behalf. Only IdP can access the real database, and all services trust IdP's authentication in this model.
#### SAML (Security Assertion Markup Language)
SAML protocol was developed in 2001 for SSO. This protocol works on XML (eXtensible Markup Language) format which can meet complex security requirements in enterprise environments. It has been used for a long time, it makes this protocol very stable. However, XML itself is a complex and heavy format. This fact makes the protocol complex and heavy to use in common and small environment. There is the process of SAML below.
- Start
User sends the request to the service. When SP recieves the request, it redirect the request to IdP.
- Authentication
IdP asks the login information to the user and it authenticate user from database (like LDAP). When authentication process succeed, IdP generates SAML assertion including user's identity based on XML with sign to ensure integrity. IdP sends the this assertion to SP via user's browser.
- Finish
SP receives the assertion from IdP via user's browser, it verify the assertion. When it is valid, they allow login.
#### OIDC (OpenID Connect) and OAuth 2.0
OIDC was developed in 2014, it is newer than SAML. This is an authentication layer on OAuth 2.0 protocol. Basically, OAuth 2.0 is for authorization, and OIDC is for authentication. OAuth 2.0 protocol works on JSON/REST format, especially JWT (JSON Web Token) which is lighter and simpler than XML. OIDC is latest standard of SSO, it supports social login, and friendly to API. These features makes this protocol use on small and personal environment easily. There is the process of OIDC below.
- Start
User sends the requset to the service (in OIDC, RP; Relying Party). When RP receives the request, it redirects to IdP (in OIDC, OP; OpenID Provider).
- Authentication
IdP (OP) asks login information, and simultaneously asks permission to provide the information to RP. After getting information and permission, IdP generates two tokens which are for different purposes. One is an ID token (JWT) which contains user identity for authentication, the other is an access token for authorization. IdP sends these token to RP. The differences between SAML are what data format does protocol use, and whether IdP gets permission or not.
- Finish
RP recieves the tokens from IdP, and it verify the tokens. When it is valid, they allow login.
#### Reverse proxy
When application doesn't support SSO, then you can use reverse proxy as the door of SSO using `Forward Authentication`. Usually, every web packet pass through reverse proxy in modern internet environment. Therefore, reverse proxy (i.e. Caddy) can intercepts the packets and force them to SSO from IdP or OP (or LDAP itself) before they reach to the application.
- Header based
When Authelia success to authenticate someone, Caddy sends specific header which contains user information like `X-Forwarded-User: A`. Application gets this header, it automatically allows login to A. However, it is needed the ID in the application manually.
> If hacker can access to application without reverse proxy, it can make X header to fake application. In this homelab, all access towards application will be limited by iptables from reverse proxy or sidecar reverse proxy.
- LDAP based
LLDAP will be a light and simple LDAP server. Authelia supports OIDC OP(idP) based on external LDAP server. LLDAP server will becom the external LDAP server for Authelia
---
## Example of Authelia flow
### Flow
- Define the user and group in LLDAP server
#### OIDC supported app
- At the service, choose login way as Authelia (OP)
- Login at the Authelia.
- Authelia access to LLDAP server to authenticate the user based on LDAP.
- Login succeed, Authelia generate token and Service turst OP's token, and allow login (authentication and authorization)
#### Non-OIDC supported app
- Foward_Auth function with Authelia.
docs/archives/2025-12/02_theory/02_06_email.md
+242
View File
@@ -0,0 +1,242 @@
Tags: #theory, #network, #protocol
## Email service
Email is the mail service online via the internet. ARPANET was developed in 1969, since then there has been many attempts to send messages via the internet. The mail which uses `@` character in 1971 and `SMTP(Simple Mail transfer Protocol)` was developed to standardize various ways to email.
### Component of Email service
#### Address
Basically, Email address has format like this. `local-parts@domain`. `local-parts` is identifier, and `domain` is service provider's domain. Following RFC 5321, `domain` doesn't distinguish it upper or lower case. `local-parts` must distinguish them, but practically they doesn't.
#### MUA (Mail User Agent)
MUA is the client of Email. The user can write Email, or read the Email which they got recieved. For instance, Outlook, Thunderbird, etc.
#### MTA (Mail Transfer Agent)
This is the essential part of Email service. MTA transpers the mail to other MTA or MDA. For instance, Postfix, sendmail, Exim, etc.
#### MDA (Mail Delivery Agent)
MDA recieves the mail from MTA, and it store the mail on receivers' mailbox. Sometimes, it is combined MTA or IMAP/POP3 servers. For instance, Dovecot LDA, Procmail, etc.
#### Flow of Email service
- User writes the mail on MUA.
- User sends the mail from MUA to MTA using SMTP submission protocol.
- MTA checks receiver's domain, and transfer the mail to other MTA which takes charge of that domain.
- MTA recieves the mail and sends receiver's MDA.
- The receiver's MUA access to the MDA such as IMAP or POP3 server. the receiver can check and read the email on their MUA.
## Protocols
### SMTP (Simple Mail Transfer Protocol)
SMTP is standard of email transfer protocol internet defined on RFC 5321. This protocol is used when MUA sends the mail to MTA, and MTA sends the mail other MTAs. This protocol takes charge of all process of transportation of the mails.
#### Detail of SMTP
##### Start
- Connection
The client and server make the connection via SMTP port (25/tcp).
##### Greeting
- `220` code
The server sends `220` code to the client, they are ready.
- `HELO` or `EHLO`
The client sends `HELO` or expand version of `HELO`; `EHLO` command to server to introduce itself.
##### Designate sender and recipient
Use the command below, they designate sender and recient.
- `MAIL FROM:<sender@domain>`
- `RCPT TO:<recipient@domain>`
- If there were various recipients, use this command as much as recipients number.
##### Transper the mail data
- `DATA` and `354`
The client sends `DATA` command to server. After the server responds with `354` code, client sends the data including mail header (From, To, Subject), and content of mail. The end of data is `.`.
##### End
- `QUIT`
The client sends `QUIT` command, the connection is terminated.
##### Ports
- `25/tcp`
Traditional SMTP's standard port. All content using `25/tcp` is not encrypted. Because of security and SPAM problems, a lot of ISP block the `25/tcp` port of common user.
- `587/tcp` (Submission)
The standard port of SMTP for encryption. Generally MUA sends the mail to MTA with this port. It is needed to use encrypted connection via `STARTTLS`
- `465/tcp` (SMTPS)
This port used to be used for TLS/SSL for SMTP. This is not standard, so it is recommended to use `587/tcp` port for TLS/SSL of SMTP. However, even now this is generally and commonly used.
##### Security
SMTP is very old protocol, and this protocol use plain data. It is recommended to use `STARTTLS` or `SMTPS` to encrypt data for security.
- `SMTPS`
It uses TLS/SSL from the beginning of connection via `465/tcp`
- `STARTTLS`
It uses TLS/SSL after beginning of connection via `587/tcp` as plain data, and start encryption with `STARTTLS` command.
##### Authentication
It is necessary to use users' identity like name and password to prevent anyone can sends malicious mail using server. SMTP uses SASL (Simple Authentication and Secuirty Layer) machanism to authenticate its users.
##### Relay
MTA has to send the mail to the other MTA for guarantee the mail can arrive the recipitent. MTA uses `relay` function for this. Make sure to allow this function for authenticated user or trusted network to prevent malicious usage.
### IMAP (Internet Message Access Protocol)
IMAP is the protocol to read and manage the mails from remote MDA (mail server). The difference between POP3 is that IMAP can manage the mail and its mailbox remotely even without download. It is defined on RFC 3501.
#### Detail of IMAP
IMAP is the protocol to have a communication with various commands while the connection is stable. The client sends specific `tag` in front of command, and the server responds with `tag` to process the actions.
##### Authentication
- `LOGIN` or `AUTHETICATE`
IMAP authenticate the user with `LOGIN` command with ID and password or `AUTHENTICATE` command with SASL.
##### Mailbox
- `LIST`
- `SELECT`
- `CREATE`
- `DELETE`
- `RENAME`
##### Mail
- `FETCH`
IMAP can take the mail list, the mail itself, or content of the mail, even the attachment in the mail.
##### Statement
- `STORE`
- `\Seen`
- `\Flagged`
- `\Answered`
- `\Deleted`
IMAP can set the status flag of mail with command flags.
##### Search
- `SEARCH`
IMAP can search the mail with various condition of the mail (Sender, title, contents, date, etc) from server.
##### Ports
IMAP strongly recommend to use TLS/SSL with `STARTTLS`. Even though the beginning of conversation is not encrypted, TLS/SSL is applied with the `STARTTLS` command.
- `143/tcp`
The basic IMAP port. It is mendetory to use `STARTTLS` to use IMAP with this port.
- `993/tcp` (IMAPS)
This port uses TLS/SSL in the beginning of communication. It is not a standard but it is generally and commonly use for security.
##### Synchronization
IMAP basically server's mail and mail list, so wherever you access the mail you can see the same condition and status of mailbox. When one mail is modified on one device it is applied all devices simultanaeously.
- `IDLE`
This command supports to maintain connection between server and client, when the new mail comes or the status is changed the client can get notification immediately.
### POP3 (Post Office Protocol version 3)
POP3 protocol is basically designed to download the mail on local client from remote mail server. It is defined on RFC 1939. The biggest difference between IMAP and POP3 is, POP3 basically delete the mail at the server after downloading.
#### Detail of POP3
##### Authorization
- `USER` and `PASS`
The client connect to server and it conduct authentication with `USER` and `PASS` command.
##### Transcation
- `STAT`
- `LIST`
- `RETR <msg_number>`
- `DELE <msg_number>`
- `RETR`
POP3 uses various commands to download or delete the mail. It checks the number of mail and size with `STAT`, downloads the mail with `RETR`, deletes the mail with `DELE`, and save the mail on client with `RETR`.
##### Update
When the client sends `QUIT` command, then server deletes the mails which have `DELE` marks from server and terminate the connection.
##### Ports
POP3 strongly recommend to use TLS/SSL with `STARTTLS`. Even though the beginning of conversation is not encrypted, TLS/SSL is applied with the `STARTTLS` command.
- `110/tcp`
The basic port of POP3. It is mendetory to use `STARTTLS` to use IMAP with this port.
- `995/tcp` (POP3S)
This port uses TLS/SSL in the beginning of communication. It is not a standard but it is generally and commonly use for security.
##### Simplity and locality
POP3 basically delete the mail from mail server, the mail is only on the local client. However, it doesn't require the complex features like IMAP, it can have simplity.
---
## local mail service in homelab
### SMTP server (MTA)
#### Postfix
Postfix will be used as MTA which takes charge of `@ilnmors.internal` domain. However, Postfix in this homelab will never open towards WAN environment. It works as local private MTA. The internal services (Gitea, OPNsense, Prometheus, etc) will sends the mail via `587/tcp` to Postfix. When it needs to send mail towards WAN, it will use `relayhost` function and external Email services such as Google or Naver, etc. `relayhost` makes postfix as one of a `client` not a `MTA`. It means, administrator never takes care about IP reputation or SPAM problems. WAN area's `MTA` function is delegated to public mail service providers.
### IMAP/POP3 server (MDA)
#### Dovecot
Dovecot will be used as IMAP server of local private MTA; Postfix. The user can use MUA (Thunderbird, Outlook, or mail application, even Roundcube webmail) to access the private mail `
`@ilnmors.internal` via Dovecot. The user will ues `993/tcp` to access Dovecot, and Postfix store the mails on Dovecot.
#### mbsync
mbsync will be used as IMAP client of public MTA; Google or Naver. This will fetch public mail `@external-domain.com` to local Postfix from public mail service provider, and eventually the user can access the mail on Dovecot. However, it is important not to delete the mails from public mail servers with proper configuration.
### MUA
#### SnappyMail web mail
This will be used as MUA server on `app` server to access all mails at the same space.
docs/archives/2025-12/03_common/03_01_debian_configuration.md
+657
View File
@@ -0,0 +1,657 @@
Tags: #common, #configuration, #os
## Installation
### General setting
- Language: English - English
- Location: Other > Asia > South Korea
- Locale: United State - en_US.UTF-8
- Keymap to use: American English
### Network setting
#### Auto Configuration
- Auto Configuration: Using DHCP
#### Manual configuration
> `net` server should be configured manually - It has no private DHCP server and DNS server
- `cancel` auto configuration is needed
- IP address: 192.168.10.11/24
- Gateway: 192.168.10.1
- Name server address: 1.1.1.2
> After AdGuard home and BIND are set, change 192.168.10.11 on `/etc/resolv.conf`
#### Common setting
- Hostname: \[vmm, net, auth, dev, app\]
- Domain: ilnmors.internal
### User setting
- Root Password
- User name: \[vmm, net, auth, dev, app\]
- User account: \[vmm, net, auth, dev, app\]
- User Password
### Partition setting
- Partitioning method: Manual
#### Common
- 512MiB - EFI system partition (Boot flag: on)
- 1GiB - Ext4 Journaling (Mount: `/boot`)
#### Hypervisor
- 780GiB - Physical volume for LVM
- Configure the Logical Volume Manager
- Create volume group - vmm
- Create logical volume - vmm - \[root, swap, data\]
- Finish
- vmm-root: 64GiB - Ext Journaling (Mount: `/`)
- vmm-swap: 16GiB - SWAP area
- vmm-libvirt: 700GiB - Ext4 Journaling (Mount: `/var/lib/libvirt`)
#### VM servers
- 32GiB(net), 64GiB(auth), 256GiB(dev,app) - Ext4 Journaling(mount: /)
> Don't set swap partition, in qcow2 it is not effective but causes over head(CoW)
#### Application server
- Don't set HDD's partition
- Each HDD (2TB)
> btrfs supports RAID itself. After installing, set RAID10. These will be set with `fdisk`
### Debian package manager setting
- Scan extra installation media: no
- Mirror country: South Korea
- Archive mirror: deb.debian.org
- Proxy: \[blank\]
- Popularity-contest: no
### Installing packages setting
- \[\*\] SSH server
- \[\*\] Standard system utilities
## Environment configuration
### Package configuration
#### Common packages
```bash
apt update && apt upgrade
apt install sudo iptables-persistent crowdsec acl
```
- wait-for-it.sh
- Download it [here](https://github.com/vishnubob/wait-for-it/blob/master/wait-for-it.sh)
- ~/data/config/scripts
#### Hypervisor
```bash
apt install qemu-system-x86 ksmtuned libvirt-daemon-system virtinst virt-top ifupdown2 openvswitch-switch
# ksmtuned automatically adjusts ksm following its usage
# `cat /sys/kernel/mm/ksm/pages_sharing` shows the page shared by ksmtuned, result * 4KiB is sharing volume
```
#### VMs
```bash
apt install podman curl jq age
# app
apt install btrfs-progs
# SOPS package
curl -LO https://github.com/getsops/sops/releases/download/v3.11.0/sops-v3.11.0.linux.amd64
mv sops-v3.11.0.linux.amd64 /usr/local/bin/sops
chmod +x /usr/local/bin/sops
# Check update this via Diun
```
### User configuration
```bash
usermod [user_name] -u [uid]
# Change default group as svadmins
groupmod [group_name] -g 2000 -n svadmins
chown -R [user_name]:svadmins /home/[user_name]
chmod 770 /home/[user_name]
usermod -aG sudo [user_name]
# Hypervisor
usermod -aG kvm vmm
usermod -aG libvirt vmm
# After user configuration, proceed all step as user with sudo
```
### Make directory structure
#### Common
- /etc/secrets (root:root 711)
- /etc/secrets/\$UID (\$UID:root 500, files: $UID:root 400)
#### Hypervisor
- (vmm:svadmins 700)
- ~/data/config/{scripts,server,services,vms}
- ~/data/config/vms/{networks,storages,dumps}
- /var/lib/libvirt/images
#### VMs
- (\[vms\]:svadmins 700)
- ~/data/{config,containers}
- ~/data/config/{containers,scripts,secrets,server,services}
- ~/data/containers/apps/{certs,etc.}
- ~/kopia
#### Application server
##### SSD
- (app:svadmins 700)
- ~/data/{config,containers}
- ~/data/config/{containers,scripts,secrets,server,services}
- ~/data/containers/app/{certs,etc.}
- ~/kopia
##### HDD
RAID10 HDDs mount on this directory. Following 08_01_app_vm before you set.
- (app:svadmins 700)
- ~/hdd/data/containers
- (app:svadmins 770)
- ~/hdd/backups
- The scrub timer systemd is required for its integrity.
### SSH configuration
- File:
- /etc/ssh/sshd_config
- ~/.ssh/authorized_keys
```ini
# /etc/ssh/sshd_config
# key generation
ssh-keygen -t ed25519 -f ~/.ssh/key_name -C "comment"
# ...
PermitRootLogin no
# ...
```
```bash
# Reload sshd
sudo systemctl restart sshd
# Deploy publicly key for ssh
mkdir ~/.ssh && chmod 700 ~/.ssh
nano ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
# Paste public key from the client
```
> If the ICMP protocol doesn't work towards Windows, check Windows' firewall inbound `파일 및 프린터 공유(에코 요청 - ICMPv4-In)" and "파일 및 프린터 공유(에코 요청 - ICMPv6-In)` activatation
### Linger configuration
```bash
sudo loginctl enable-linger [user_name]
loginctl show-user $(whoami)
# Linger=yes
```
> Linger is necessary to execute services regardless their login session. If linger option were not activated, all the process would be terminated after logout.
### Network configuration
- File:
- /etc/hosts
- /etc/resolv.conf
- /etc/network/interfaces
#### Hostname
```ini
# /etc/hosts
[ip_address] [FQDN] [hostname]
```
#### DNS
- Only for `hypervisor` and `net` which are set manually.
```ini
# /etc/resolv.conf
# Before setting local DNS
nameserver 1.1.1.2
# After setting local DNS (net server)
# nameserver 192.168.10.11
```
#### Interface
- Only for `hypervisor` and `net` which are set manually
##### Hypervisor
```ini
# /etc/network/interfaces
# ifupdown2 and openvswitch-switch are required
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The openvswitch virtual bridge interfaces
auto ovsbr0
iface ovsbr0 inet manual
ovs_type OVSBridge
ovs_ports enp2s0
auto ovsbr1
iface ovsbr1 inet manual
ovs_type OVSBridge
ovs_ports enp3s0 vlan1 vlan10
# The primary network interfaces
auto enp2s0
iface enp2s0 inet manual
ovs_type OVSPort
ovs_bridge ovsbr0
auto enp3s0
iface enp3s0 inet manual
ovs_type OVSPort
ovs_bridge ovsbr1
ovs_options vlan_mode=native-untagged tag=1 trunks=10
# The vlan interfaces
auto vlan1
iface vlan1 inet static
ovs_type OVSIntPort
ovs_bridge ovsbr1
ovs_options tag=1
address 192.168.1.10/24
auto vlan10
iface vlan10 inet static
ovs_type OVSIntPort
ovs_bridge ovsbr1
ovs_options tag=10
address 192.168.10.10/24
gateway 192.168.10.1
# The primary network interfaces for IPv6
# iface enp2s0 inet6 auto
# iface enp3s0 inet6 auto
# openvswitch vlan options
# ovs_options tag=n : access mode
# ovs_options trunk=n : trunk mode
# ovs_options vlan_mode=native-untagged tag=n trunk=m: native untagged mode
```
> Having two IP addresses in a client can cause some problems. One of them is `Asymmetric routing`. When it gets the packet via vlan10 interface from vlan1 client, the server uses vlan1 interface, because client and server are in the same subnet (L2). It can cause problem, like disconnection usually. There is the solution to solve this.
>
> - Use `mangle` table in iptables, you can solve this, but it is complex to use.
> - Use `IP rule` command, it is easier to set.
##### net
```ini
# /etc/network/interfaces
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto enp1s0
iface enp1s0 inet static
address 192.168.10.11/24
gateway 192.168.10.1
# Don't set dns options. It will be set on /etc/resolv.conf
```
### Hypervisor
#### VFIO configuration
##### IOMMU setting
- File: /etc/default/grub
```ini
# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
# Intel CPU: intel_iommu=on, AMD CPU: amd_iommu=on
```
##### Device setting
- File: /etc/modprobe.d/vfio.conf
```bash
# Check device
lspci -nn | grep -i sata
# 04:00.0 STAT controller [0106] ... [1b21:1064]
# PCI address: 04:00.00 / Device ID 1b21:1064
# IOMMU group checking
readlink /sys/bus/pci/devices/0000\:04\:00.0/iommu_group
# ../../../../kernel/iommu_groups/14
# The group of PCI device which is needed to passthrough is 14
ls /sys/kernel/iommu_groups/devices/14
# 04:00.00 # To passthrough device, it has to be only device in its IOMMU group
```
```ini
# /etc/modprobe.d/vfio.conf
options vfio-pci ids=1b21:1064
softdep ahci pre: vfio-pci
```
##### Apply all configuration
```bash
sudo update-grub
sudo update-initramfs -u
sudo reboot
# Check IOMMU enable message
dmesg | grep -e DMAR -e IOMMU
# Check the driver
lspci -nnk -d 1b21:1064
# Kernel driver in user: vfio-pci
```
### VM servers
#### Serial console setting
```bash
# It is necessary to connect via ssh to set serial setting
sudo systemctl enable getty@ttyS0
sudo systemctl start getty@ttyS0
```
#### Secret management
##### SOPS setting
- File:
- ~/data/config/secrets/age-key.gpg
- ~/data/config/secrets/.sops.yaml
- ~/data/config/secrets/.secret.yaml
- /etc/secrets/\$UID (\$UID:root 500)
```bash
# Generate the key for sops
age-keygen -o ~/data/config/secrets/age-key
# # created: 2025-10-17T13:30:00Z
# # public key: age1ql3z7h0cfscg......
# AGE-SECRET-KEY-1.....
# Public key is printed when key generated
gpg --symmetric age-key && rm age-key
> GPG password: password
nano ~/data/config/secrets/.sops.yaml && chmod 600 ~/data/config/secrets/.sops.yaml
```
```yaml
# ~/data/config/secrets/.sops.yaml
creation_rules:
- path_regex: \.secret\.yaml$
age: [public_key]
```
##### Create Secret file
```yaml
# ~/data/config/secrets/.secret.yaml
# Format of .secret.yaml
# app1.env:
1SECRET: '1secret'
2SECRET: '2secret'
app1.file: |
-----TEXT-AREA-----
contents of 3secret
-----END-AREA-----
# app2.env
3SECRET: '3secret'
4SECRET: '4secret'
# ...
```
```bash
sops --encrypt --in-place ~/data/config/secrets/.secret.yaml
```
##### Secret scripts
- File:
- ~/data/config/scripts/secrets/edit_secret.sh
- ~/data/config/scripts/secrets/extract_secret.sh
```bash
#!/bin/bash
# edit_secret.sh /path/of/secret
set -e
KEY_PATH="$HOME/data/config/secrets"
SECRET_FILE="$1"
usage() {
echo "Usage: $0 \"/path/of/secret/file\""
exit 1
}
if [ -z "$SECRET_FILE" -o ! -f "$SECRET_FILE" ]; then
echo "Error: Secret file path is needed"
usage
fi
if [ ! -f "$KEY_PATH/age-key.gpg" ]; then
echo "Error: There is no key file"
exit 1
fi
# Delete password file after script
cleanup() {
if [ -f "/run/user/$UID/age-key" ]; then
rm -f "/run/user/$UID/age-key"
fi
}
trap cleanup EXIT
echo -n "Enter GPG passphrase: "
read -s GPG_PASSPHRASE
echo
echo "$GPG_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
--output "/run/user/$UID/age-key" \
--decrypt "$KEY_PATH/age-key.gpg" && \
chmod 600 "/run/user/$UID/age-key"
if [ -z "/run/user/$UID/age-key" ]; then
echo "Error: Key file does not exist"
exit 1
fi
gpgconf --kill gpg-agent
SOPS_AGE_KEY="$(cat "/run/user/$UID/age-key")"
SOPS_AGE_KEY="$SOPS_AGE_KEY" sops "$SECRET_FILE"
```
```bash
#!/bin/bash
# extract_secret.sh /path/of/secret (-f|-e <value>)
set -e
KEY_PATH="$HOME/data/config/secrets"
SECRET_FILE=$1
# shift the $2 as $1 ($1 < $2)
shift
# usage() function
usage() {
echo "Usage: $0 \"/path/of/secret/file\" (-f|-e \"yaml section name\")" >&2
echo "-f <type name>: Print secret file" >&2
echo "-e <type name>: Print secret env file" >&2
exit 1
}
while getopts "f:e:" opt; do
case $opt in
f)
VALUE="$OPTARG"
TYPE="FILE"
;;
e)
VALUE="$OPTARG"
TYPE="ENV"
;;
\?) # unknown options
echo "Invalid option: -$OPTARG" >&2
usage
;;
:) # parameter required option
echo "Option -$OPTARG requires an argument." >&2
usage
;;
esac
done
# Get option and move to parameters - This has no functional thing, because it only use arguments with parameters
shift $((OPTIND - 1))
# Check necessary options
if [ ! -f "$SECRET_FILE" ]; then
echo "Error: secret file path is required" >&2
usage
fi
if [ -z "$TYPE" ]; then
echo "Error: -f or -e option requires" >&2
usage
fi
if [ ! -f "$KEY_PATH/age-key.gpg" ]; then
echo "Error: There is no key file" >&2
usage
fi
# Delete password file after script
cleanup() {
if [ -f "/run/user/$UID/age-key" ]; then
rm -f "/run/user/$UID/age-key"
fi
}
trap cleanup EXIT
echo -n "Enter GPG passphrase: " >&2
read -s GPG_PASSPHRASE
echo >&2
echo "$GPG_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
--output "/run/user/$UID/age-key" \
--decrypt "$KEY_PATH/age-key.gpg" && \
chmod 600 "/run/user/$UID/age-key"
if [ ! -f "/run/user/$UID/age-key" ]; then
echo "Error: Key file does not exist" >&2
exit 1
fi
gpgconf --kill gpg-agent
SOPS_AGE_KEY="$(cat "/run/user/$UID/age-key")"
if [ "$TYPE" == "FILE" ]; then
if RESULT=$(SOPS_AGE_KEY="$SOPS_AGE_KEY" sops --decrypt --extract "[\"$VALUE\"]" --output-type binary "$SECRET_FILE") ; then
echo -n "$RESULT"
exit 0
else
echo "Error: SOPS extract error" >&2
exit 1
fi
fi
if [ "$TYPE" == "ENV" ]; then
if RESULT=$(SOPS_AGE_KEY="$SOPS_AGE_KEY" sops --decrypt --extract "[\"$VALUE\"]" --output-type dotenv "$SECRET_FILE") ; then
echo -n "$RESULT"
exit 0
else
echo "Error: SOPS extract error" >&2
exit 1
fi
fi
```
##### Secret value management
- Using `extract_secret.sh`
- Inject secret value to `podman secret` or `/etc/secrets/$UID`
```bash
# /etc/secrets/$UID
# Before use sudo tee, make sure sudo doesn't need password.
# i.e. sudo ps -ef command execute before this command.
# Env file
extract_secret.sh ~/data/config/secrets/.secret.yaml -e "$value" > /run/user/$UID/tmp.env \
&& sudo mv /run/user/$UID/tmp.env /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chown $UID:root /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chmod 400 /etc/secrets/$UID/"$FILE_NAME"
# Normal file
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "$value" > /run/user/$UID/tmp.env \
&& sudo mv /run/user/$UID/tmp.env /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chown $UID:root /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chmod 400 /etc/secrets/$UID/"$FILE_NAME"
# Podman secret
# Podman doesn't supports .env file parsing, you have to enroll all values
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "$value" | podman secret create "[$FILE_NAME|$ENV_NAME]" -
```
> The reason why apply the secret value manually is its complexity. The `yaml` file can hold a lot of values in one section. It is hard to make the script to deal with a lot of exception and senario. The simple way is the best way.
docs/archives/2025-12/03_common/03_02_iptables.md
+273
View File
@@ -0,0 +1,273 @@
Tags: #common, #configuration, #network, #security
## iptables
iptables is the firewall program to manage netfilter which is in the linux kernel. Basically, the iptables' settings are temporary(when you reboot the computer they disappear). So, you can use netfilter-persistent program to make settings permanent. iptables has 3 modes(tables), which are filter, nat, and mangle. And each table has their chain like input, forward or prerouting, etc. iptables' setting is temporary, when the machine reboot, all rules will be reset. After set the rules use `netfilter-persistent save` to make the rules permanently.
### tables
#### filter
filter is the basic and most important table in iptables. Its role is to simply judge, to ACCEPT or DROP the packets. When you use the iptables command without table option (-t), the filter table is default option. There are chians of filter table below.
- INPUT: check the packets in
- OUTPUT: check the packets out
- FORWARD: check the packets which are passing through
#### nat
nat is the table changing packets' address and port without changing contents of packets. There are chians of nat table below.
- PREROUTING: change the destination address or port right after packets arrived
- POSTROUTING: change the source address or port before packets depart
- OUTPUT: change the destination address or port of packets which produced by itself (This doesn't change the source IP; DNAT)
#### mangle
mangle is a special table to mark on the packets. It works on the every chain and it works on special purpose like asymmetric routing.
### grammar
#### Commands
- -A \[--append\]: create the new rules
- -C \[--check\]: check the packets
- -D \[--delete\]: delete the rules
- -F \[--flush\]: delete all rules from the chain
- -I \[--insert\]: Insert the new rules
- -L \[--list\]: print the rules
- -N \[--new\]: create the new chain
- -P \[--policy\]: change the default policy
- -R \[--replace\]: change the rules as a new rule
- -X \[--delete-chain\]: delete chain
- -Z \[--zero\]: reset the packet and byte counter value of all chain
#### match
- -s \[--source\]: designate source ip address or networks
- -d \[--destination\]: designate destination ip address or networks
- -p \[--protocol\]: match protocol(tcp/udp/icmp.. etc.)
- --dport: designate specific protocol number (When the protocol is already defined - tcp or udp)
- --syn: match syn packets. when starting new TCP connection, apply the rule.
- -i \[--in-interface\]: input interface
- -o \[--out-interface\]: output interface
- --comment: comment(max 256byte)
- -f \[--fragment\]
- -t \[--table\]: designate table set(default: filter)
- -j \[--jump\]: designate targets
- -m \[--match\]: match with specific module
- conntrack --ctstate: current linked connection
#### target
- ACCEPT: allow packets
- DROP: deny packets without response (Hide server existence)
- REJECT: deny packets with response (Show server existance)
- LOG: log the packets on syslog
- RETURN: stop current rules, and return to the previous chain
#### Command
```bash
iptables -L -v -n # Print all rules in filter table
iptables -L -v -n -t nat # Print all rules in nat table
```
## netfilter-persistent
### Save the rules
```bash
sudo netfilter-persistent save
```
- File:
- /etc/iptables/rules.v4
- /etc/iptables/rules.v6
### Reload the rules (manual)
```bash
# Edit the file
# Test
sudo bash -c 'iptables-restore --test < /etc/iptables/rules.v4'
# If there were no message, it would have no error
sudo netfilter-persistent start
# or
sudo netfilter-persistent reload
```
### Rule files
- File: /etc/iptables/rules.v4
#### Hypervisor (vmm)
```ini
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "allow established connection" -j ACCEPT
-A INPUT -i lo -m comment --comment "allow local connection" -j ACCEPT
-A INPUT -p icmp -m comment --comment "allow ICMP connection" -j ACCEPT
-A INPUT -s 192.168.1.11/32 -p tcp -m tcp --dport 22 -m comment --comment "allow emergemcy LAN console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.2/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.3/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.4/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.13/32 -p tcp -m tcp --dport 22 -m comment --comment "allow code-server ssh connection" -j ACCEPT
COMMIT
```
#### net
```ini
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "allow established connection" -j ACCEPT
-A INPUT -i lo -m comment --comment "allow local connection" -j ACCEPT
-A INPUT -p icmp -m comment --comment "allow ICMP connection" -j ACCEPT
-A INPUT -s 10.10.10.2/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.3/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.4/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.1/32 -p tcp -m tcp --dport 22 -m comment --comment "allow OPNsense ssh connection for ACME update" -j ACCEPT
-A INPUT -s 192.168.10.10/32 -p tcp -m tcp --dport 22 -m comment --comment "allow hypervisor ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.13/32 -p tcp -m tcp --dport 22 -m comment --comment "allow code-server ssh connection" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2053 -m conntrack --ctorigdstport 53 -m comment --comment "allow tcp DNS connection which is only prerouted from 53" -j ACCEPT
-A INPUT -p udp -m udp --dport 2053 -m conntrack --ctorigdstport 53 -m comment --comment "allow udp DNS connection which is only prerouted from 53" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2443 -m conntrack --ctorigdstport 443 -m comment --comment "allow tcp DoH(https) connection which is only prerouted from 443" -j ACCEPT
-A INPUT -p udp -m udp --dport 2443 -m conntrack --ctorigdstport 443 -m comment --comment "allow udp DoH(https) connection which is only prerouted from 443" -j ACCEPT
-A INPUT -s 192.168.10.1/32 -p tcp -m tcp --dport 2253 -m comment --comment "allow opnsense tcp nsupdate connection" -j ACCEPT
-A INPUT -s 192.168.10.1/32 -p udp -m udp --dport 2253 -m comment --comment "allow opnsense udp nsupdate connection" -j ACCEPT
-A INPUT -s 192.168.10.12/32 -p tcp -m tcp --dport 2253 -m comment --comment "allow auth tcp nsupdate connection" -j ACCEPT
-A INPUT -s 192.168.10.12/32 -p udp -m udp --dport 2253 -m comment --comment "allow auth udp nsupdate connection" -j ACCEPT
-A INPUT -s 192.168.10.13/32 -p tcp -m tcp --dport 2253 -m comment --comment "allow dev tcp nsupdate connection" -j ACCEPT
-A INPUT -s 192.168.10.13/32 -p udp -m udp --dport 2253 -m comment --comment "allow dev udp nsupdate connection" -j ACCEPT
-A INPUT -s 192.168.10.14/32 -p tcp -m tcp --dport 2253 -m comment --comment "allow app tcp nsupdate connection" -j ACCEPT
-A INPUT -s 192.168.10.14/32 -p udp -m udp --dport 2253 -m comment --comment "allow app udp nsupdate connection" -j ACCEPT
COMMIT
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 53 -m comment --comment "allow and preroute tcp DNS connection 53 to 2053" -j REDIRECT --to-ports 2053
-A PREROUTING -p udp -m udp --dport 53 -m comment --comment "allow and preroute udp DNS connection 53 to 2053" -j REDIRECT --to-ports 2053
-A PREROUTING -p tcp -m tcp --dport 443 -m comment --comment "allow and preroute tcp DoH(https) connection 443 to 2443" -j REDIRECT --to-ports 2443
-A PREROUTING -p udp -m udp --dport 443 -m comment --comment "allow and preroute udp DoH(https) connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 53 -m comment --comment "NAT local tcp DNS connection 53 to 2053" -j REDIRECT --to-ports 2053
-A OUTPUT -d 127.0.0.1/32 -p udp -m udp --dport 53 -m comment --comment "NAT local udp DNS connection 53 to 2053" -j REDIRECT --to-ports 2053
-A OUTPUT -d 192.168.10.11/32 -p tcp -m tcp --dport 53 -m comment --comment "NAT local tcp DNS connection 53 to 2053" -j REDIRECT --to-ports 2053
-A OUTPUT -d 192.168.10.11/32 -p udp -m udp --dport 53 -m comment --comment "NAT local udp DNS connection 53 to 2053" -j REDIRECT --to-ports 2053
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 443 -m comment --comment "NAT local tcp DoH(https) connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 127.0.0.1/32 -p udp -m udp --dport 443 -m comment --comment "NAT local udp DoH(https) connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 192.168.10.11/32 -p tcp -m tcp --dport 443 -m comment --comment "NAT local tcp DoH(https) connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 192.168.10.11/32 -p udp -m udp --dport 443 -m comment --comment "NAT local udp DoH(https) connection 443 to 2443" -j REDIRECT --to-ports 2443
COMMIT
```
#### auth
```ini
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [204:15800]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "allow established connection" -j ACCEPT
-A INPUT -i lo -m comment --comment "allow local connection" -j ACCEPT
-A INPUT -p icmp -m comment --comment "allow ICMP connection" -j ACCEPT
-A INPUT -s 10.10.10.2/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.3/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.4/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.1/32 -p tcp -m tcp --dport 22 -m comment --comment "allow OPNsense ssh connection for ACME update" -j ACCEPT
-A INPUT -s 192.168.10.10/32 -p tcp -m tcp --dport 22 -m comment --comment "allow hypervisor ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.13/32 -p tcp -m tcp --dport 22 -m comment --comment "allow code-server ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.1/32 -p tcp -m tcp --dport 9000 -m comment --comment "allow opnsense step-ca connection" -j ACCEPT
-A INPUT -s 192.168.10.13/32 -p tcp -m tcp --dport 9000 -m comment --comment "allow dev step-ca connection" -j ACCEPT
-A INPUT -s 192.168.10.14/32 -p tcp -m tcp --dport 9000 -m comment --comment "allow app step-ca connetcion" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2080 -m conntrack --ctorigdstport 80 -m comment --comment "allow tcp http connection which is only from 80" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2443 -m conntrack --ctorigdstport 443 -m comment --comment "allow tcp https connection which is only from 443" -j ACCEPT
COMMIT
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 80 -m comment --comment "allow and preroute tcp http connection 80 to 2080" -j REDIRECT --to-ports 2080
-A PREROUTING -p tcp -m tcp --dport 443 -m comment --comment "allow and preroute tcp https connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 80 -m comment --comment "NAT local tcp http connection 80 to 2080" -j REDIRECT --to-ports 2080
-A OUTPUT -d 192.168.10.12/32 -p tcp -m tcp --dport 80 -m comment --comment "NAT local tcp http connection 80 to 2080" -j REDIRECT --to-ports 2080
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 443 -m comment --comment "NAT local tcp https connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 192.168.10.12/32 -p tcp -m tcp --dport 443 -m comment --comment "NAT local tcp https connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 636 -m comment --comment "NAT local tcp ldaps connection 636 to 6360" -j REDIRECT --to-ports 6360
-A OUTPUT -d 192.168.10.12/32 -p tcp -m tcp --dport 636 -m comment --comment "NAT local tcp ldaps connection 636 to 6360" -j REDIRECT --to-ports 6360
COMMIT
```
#### dev
```ini
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "allow established connection" -j ACCEPT
-A INPUT -i lo -m comment --comment "allow local connection" -j ACCEPT
-A INPUT -p icmp -m comment --comment "allow ICMP connection" -j ACCEPT
-A INPUT -s 10.10.10.2/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.3/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.4/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.1/32 -p tcp -m tcp --dport 22 -m comment --comment "allow OPNsense ssh connection for ACME update" -j ACCEPT
-A INPUT -s 192.168.10.10/32 -p tcp -m tcp --dport 22 -m comment --comment "allow hypervisor ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.12/32 -p tcp -m tcp --dport 5432 -m comment --comment "allow auth postgresql connection" -j ACCEPT
-A INPUT -s 192.168.10.14/32 -p tcp -m tcp --dport 5432 -m comment --comment "allow app postgresql connection" -j ACCEPT
-A INPUT -s 192.168.10.12/32 -p tcp -m tcp --dport 2080 -m conntrack --ctorigdstport 80 -m comment --comment "allow tcp http connection which is only from 80 and main caddy" -j ACCEPT
-A INPUT -s 192.168.10.12/32 -p tcp -m tcp --dport 2443 -m conntrack --ctorigdstport 443 -m comment --comment "allow tcp https connection which is only from 443 and main caddy" -j ACCEPT
COMMIT
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 80 -m comment --comment "allow and preroute tcp http connection 80 to 2080" -j REDIRECT --to-ports 2080
-A PREROUTING -p tcp -m tcp --dport 443 -m comment --comment "allow and preroute tcp https connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 80 -m comment --comment "NAT local tcp http connection 80 to 2080" -j REDIRECT --to-ports 2080
-A OUTPUT -d 192.168.10.13/32 -p tcp -m tcp --dport 80 -m comment --comment "NAT local tcp http connection 80 to 2080" -j REDIRECT --to-ports 2080
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 443 -m comment --comment "NAT local tcp https connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 192.168.10.13/32 -p tcp -m tcp --dport 443 -m comment --comment "NAT local tcp https connection 443 to 2443" -j REDIRECT --to-ports 2443
COMMIT
```
#### app
```ini
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "allow established connection" -j ACCEPT
-A INPUT -i lo -m comment --comment "allow local connection" -j ACCEPT
-A INPUT -p icmp -m comment --comment "allow ICMP connection" -j ACCEPT
-A INPUT -s 10.10.10.2/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.3/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 10.10.10.4/32 -p tcp -m tcp --dport 22 -m comment --comment "allow vpn console ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.1/32 -p tcp -m tcp --dport 22 -m comment --comment "allow OPNsense ssh connection for ACME update" -j ACCEPT
-A INPUT -s 192.168.10.10/32 -p tcp -m tcp --dport 22 -m comment --comment "allow hypervisor ssh connection" -j ACCEPT
-A INPUT -s 192.168.10.13/32 -p tcp -m tcp --dport 22 -m comment --comment "allow code-server ssh connection" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2080 -m conntrack --ctorigdstport 80 -m comment --comment "allow tcp http connection which is only from 80" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2443 -m conntrack --ctorigdstport 443 -m comment --comment "allow tcp https connection which is only from 443" -j ACCEPT
COMMIT
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 80 -m comment --comment "allow and preroute tcp http connection 80 to 2080" -j REDIRECT --to-ports 2080
-A PREROUTING -p tcp -m tcp --dport 443 -m comment --comment "allow and preroute tcp https connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 80 -m comment --comment "NAT local tcp http connection 80 to 2080" -j REDIRECT --to-ports 2080
-A OUTPUT -d 192.168.10.14/32 -p tcp -m tcp --dport 80 -m comment --comment "NAT local tcp http connection 80 to 2080" -j REDIRECT --to-ports 2080
-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 443 -m comment --comment "NAT local tcp https connection 443 to 2443" -j REDIRECT --to-ports 2443
-A OUTPUT -d 192.168.10.14/32 -p tcp -m tcp --dport 443 -m comment --comment "NAT local tcp https connection 443 to 2443" -j REDIRECT --to-ports 2443
COMMIT
```
docs/archives/2025-12/03_common/03_03_podman.md
+306
View File
@@ -0,0 +1,306 @@
systTags: #common, #configuration, #virtualization, #container, #os
## podman
The container is one of virtual technology to run applications independently regardless of what the host system is. One of them, Docker uses daemon(dockerd) and docker socket(root authority), so all container can access hosts' root authority. It causes danger that hackers can take down root authority via docker containers. Additionally, docker daemon system makes it hard to combine each container to systemd, because all containers are in charge of dockerd.
Podman is a new technology to solve docker's problems. It implements daemonless, and rootless container environment. It reduces the danger of the hacker getting root authority via containers, and it is easy to combine each container to systemd.
### Configuration
- File: /etc/containers/registries.conf
```bash
# Check the linger
loginctl show-user $(whoami)
# Linger=yes
```
```ini
# /etc/containers/registries.conf
unqualified-search-registries = ["docker.io"]
```
### Networking
Podman uses a specific IP address and domain names to communicate with its host system using pasta. Pasta is a new default network mode in podman 4.0 which can allow communication with the host system directly. Therefore, it doesn't need a bridge or network host. `169.254.1.2` is 'link-local address' to communicate with host's system. The pasta conduct the SNAT. Therefore, even though the packet's original source IP were container's IP, when the packet passes the pasta its source IP changes as 127.0.0.1 or its own IP (localhost).
- `/etc/hosts` in container
- 169.254.1.2 host.containers.internal host.docker.internal
When you use the command below, you can add domain to the line.
- --host-name mydomain.internal:host-gateway
Additionally, rootless podman containers cannot bind host's privileged ports number(=<1024). Therefore, if container needed to use these ports, you would have to use iptables' nat table. The example of iptables' usage is [here](./03_02_iptables.md).
#### Bridge mode
Bridge mode create a separated virtual IP network from host's network. This mode supports simple DNS function between containers belonging to the same network. This mode basically uses SNAT(Source NAT) mutually; both inbound and outbond. It is because basically podman runs as rootless. So, the container can't distinguish where the packets come from, except from the container belonging to the same podman network. At the same time, client also can't distinguish where the packets come from. Because every packet seems like come from host server. It makes inspection hard for both of client and containers.
### ID mapping
Podman basically mapped container's root to host's executing user. Despite the root, podman uses subuid, subgid system. They are set on `/etc/subuid`, `/etc/subgid`.
- host(1000:1000) < container(0:0)
- host(100999:100999) < container(1000:1000) : subuid, subgid
When podman runs and executes commands with -u option, --userns=keep-id option, -uid, -gid option, it can adjust mapping.
- -u uid:gid: excute container with hosts's uid (bring the host's UID/GID towards container's `/etc/passwd` directly.)
- --userns=keep-id: mapping all container's file permission to host's file permission.
- --cap-add=DAC_READ_SEARCH option: without root authority, container can access every file regardless permission.
#### Mapping error
Some container doesn't execute a container with root permission. They execute the container with their specific uid (i.e. UID:53 - BIND). In this case, when the container runs with `-u` option or `--userns=keep-id` option can make mapping error very frequently.
- `-u` option
When the container runs with `-u` option, the entrypoint can't work properly in many cases because they were already set that runs entrypoint as specific uid. So, if `-u` option were set, then it would cause permission error.
- `--userns=keep-id` option
This option makes container's directory/file UID as the same as the host's directory/file UID. So, it turns off the UID/GID mapping itself. When some directory which has root authorization mapped with hosts' file it occurs UID mapping error.
#### Permission management
Use ACL packages, to give additional permission of directory. It can give the extra permission to subuid or host's uid.
```bash
# u:[subuid]:rwx
sudo getfacl /path/of/podman_directory
# `-d` option is to set permission for file or directory which are created automatically
# `-R` option is to set permission for file or directory which already existed
sudo setfacl -d -m u:[subuid]:rwx /path/of/podman_directory
sudo setfacl -d -m u:[hostuid]:rwx /path/of/podman_directory
```
### Usage of podman
#### Containerfile and build
Containerfile's format is compatible with dockerfile. Here is the example below. Containerfile can be built as podman image with `podman build` command.
```containerfile
FROM caddy:2.10.2-builder-alpine AS builder
RUN xcaddy build \
--with github.com/caddy-dns/rfc2136 \
--with github.com/hslatman/caddy-crowdsec-bouncer/crowdsec \
--with github.com/hslatman/caddy-crowdsec-bouncer/http \
FROM caddy:2.10.2
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
```
```bash
# Build container file as podman image
podman build -t caddy:2.10.2-main -f /path/of/containterfile-caddy-main . && podman image prune -f
# Delete source images
podman rmi caddy:2.10.2-builder-alpine
podman rmi caddy:2.10.2
```
#### Podman images
- `podman images`: Print list of all local podman images
- `podman image pull [image_name]`: Download podman images from repository
- `podman pull` is the same command
- `podman image prune`: Remove unused and untagged images
- `podman image rm [images]`: Remove local podman image
- `podman rmi` is the same command
#### Run and exec
podman ps \[--all\]: it shows podman container lists
- podman run
- --name: container's name
- --restart: restart mode - unless-stopped
- --add-host: additional host domain name on 169.254.1.2
- --cap-add: add some specific privileges without root authority
- -p host_ports:container_port
- -v host_path:container_path:permission(rw, ro, and when you use SELinux, you can use Z or z)
- -e environment_value
- -d: run background
- image_name
- podman exec -it \[container_name\] \[command\]
#### Pod
Pod makes each container which are in the same pod share some specific resources. The network(IP address), storage volumes. However, each container has their own file system, process, and resource limits. So, this is very useful to use various containers which has close relationship like application and Redis(cache db).
- podman pod create
- --name: pod's name
- -p: host_ports:pod_ports
- podman run
- ...
- --pod pod's name
> Don't use `-p` option. Pod already has `-p` option.
#### Container and file management
#### container
- check pure container
```bash
podman run --rm -it --entrypoint sh [image_name] --args
# or
podman run --rm -it [image_name] sh --args
```
#### file management
- Using `podman exec` to manage file
- Use ACL package `setfacl`
- `--cap-add=DAC_READ_SEARCH` option allows to read all file without permission to backup (for kopia)
### Quadlet and systemd
#### Register the secret on podman secret
- Using `edit_secret.sh` and `extract_secret.sh`
- Inject secret value to `podman secret` or `/etc/secrets/$UID`
```bash
# /etc/secrets/$UID
# Before use sudo tee, make sure sudo doesn't need password.
# i.e. sudo ps -ef command execute before this command.
# Env file
extract_secret.sh ~/data/config/secrets/.secret.yaml -e "$value" > /run/user/$UID/tmp.env \
&& sudo mv /run/user/$UID/tmp.env /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chown $UID:root /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chmod 400 /etc/secrets/$UID/"$FILE_NAME"
# Normal file
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "$value" > /run/user/$UID/tmp.env \
&& sudo mv /run/user/$UID/tmp.env /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chown $UID:root /etc/secrets/$UID/"$FILE_NAME" \
&& sudo chmod 400 /etc/secrets/$UID/"$FILE_NAME"
# Podman secret
# Podman doesn't supports .env file parsing, you have to enroll all values
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "$value" | podman secret create "[$FILE_NAME|$ENV_NAME]" -
```
> ` podman secret inspect --showsecret --format '{{.SecretData}}' $secret_name` shows the content of secret
#### Define quadlet file
- File:
- ~/data/config/containers/app/app.container
- ~/data/config/containers/app/app.pod
Quadlet is to define of specification as `.quadlet` or `.container`. Quadlet uses these file to make `.service` file to combine container to systemd. Here is the example of `.container` file below.
```ini
# app.container
[Quadlet]
# Don't make a dependencies
DefaultDependencies=false
[Unit]
Description=app
After=a.service
Wants=a.service
Requires=a.service
[Service]
ExecStartPre=%h/data/config/scripts/wait-for-it.sh -h 192.168.10.1 -p 8080 -t 20
[Container]
# Pod=app.pod
Image=localhost/app:1.0.0
ContainerName=app
PublishPort=2080:80/tcp
PublishPort=2443:443/tcp
AddHost=app.service.internal:host-gateway
Volume=%h/data/containers/app:/home/app:rw
Environment="ENV1=ENV1"
Secret=ENV_NAME,type=env
Secret=app.file,target=/path/of/secret/file/name
# podman run [options] [image] example --config exconfig
Exec=example --config exconfig
# If you want to change Entrypoint itself, use
Entrypoint=sh -c 'command'
# For Diun
Label=diun.enable=true
# For Diun to track repository new version
Label=diun.watch_repo=true
# For Diun, and it needs `diun.yml` configuration
Label=diun.regopt=container-source
[Install]
WantedBy=default.target
```
```ini
# app.pod
[Quadlet]
# Don't make a dependencies
DefaultDependencies=false
[Pod]
Name=app
PublishPort=2080:80/tcp
```
#### Create systemd `.service` file
```bash
# linger has to be activated
mkdir -p ~/.config/containers/systemd
ln -s ~/data/config/containers/app/app.container ~/.config/containers/systemd/app.container
# This command makes ~/.config/systemd/user/my-app.service
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user enable app.service
systemctl --user start app.service
```
---
## Following goal
### Health check
```ini
# i.e. caddy
# Podman [Container] section
[Container]
# Health check configuration
# Health check command
HealthcheckCommand=curl -f http://localhost/ || exit 1
# Health check interval
HealthcheckInterval=30s
# the time to wait for health check
HealthcheckTimeout=5s
# the number to try to health check
HealthcheckRetries=3
# the time to wait to start first health check
HealthcheckStartPeriod=15s
# override.conf [Service] section
[Service]
# Restart, if it is not healthy
Restart=on-failure
```
docs/archives/2025-12/03_common/03_04_crowdsec.md
+321
View File
@@ -0,0 +1,321 @@
Tags: #common, #configuration, #network, #security
## CrowdSec
CrowdSec is the free, open-source IPS(Intrusion Prevention System). It has a distributed architecture. When an agent detects malicious IPs from log, it reports the information to LAPI. This information is shared to CrowdSec's central server anonymously and is spread across the world. Additionally, bouncer receives a blacklist from LAPI and when they approach the server, the bouncer blocks them.
### Collection
#### Parser
It is how to organize the raw logs to the parsed log that scenario can understand. It is works on agent, and parsed log is transferred to LAPI to decide.
#### Scenario
It is how to analyze the malicious attack from the parsed log from the Agent. The LAPI decides what to do for malicious attack, and transfers the result to Bouncer to block.
### Agent
Agent is the detector in each server. They analyze the logs. When they find malicious approaches, or abnormal and harmful actions, they report the information to LAPI(Local API). It analizes the log following `Parsers`.
### LAPI
LAPI server is a local central collector and reporter of malicious attack information. It decides what to apply for traffic following `Scenarios`. In this homelab it is located in OPNsense, because the center of gateway of home network is Firewall. When agent reports threats to the LAPI server, LAPI decides whether or not to block them, and reports to the central CrowdSec server. This information will be spread to all CrowdSec users in the world.
### Bouncer
When LAPI decides to block some IPs, they create a blacklist and give it to bouncer. Bouncer blocks and bans some IPs depending on LAPI's blacklist. `Caddy-auth (L7)` and `OPNsense (L4)` will be bouncer to ban. The most important thing is LAPI just decide what to ban, and Bouncer conducts ban practically.
## CrowdSec in OPNsense
OPNsense supports CrowdSec with community plugin. This is not a basic function so when you want to use it in OPNsense you should install the community plugin.
### Installation
- System:Firmware:Plugins
- \[\*\] Show community plugins
- `os-crowdsec`
### General configuration
Services:CrowdSec:Settings
- \[\*\] Enable Log Processor (IDS)
- \[\*\] Enable LAPI
- \[\*\] Enable Remediation Component (IPS)
- \[ \] Manual LAPI configuration
- LAPI listen address: \[opnsense IP: 192.168.10.1\]
- LAPI listen port: 8080
- \[\*\] Create blocklist rules
- `Apply`
> Enable Remediation Component (IPS) option means, Bouncer will be integrated with OPNsense's firewall rules
> Set LAPI configuration manually, `Manual LAPI configuration` is needed.
### Machines configuration
#### OPNsense console
```sh
# 8) shell
cscli machines add [server_name] -a -f -
# ---
# Machine 'server_name' successfully added to the local API.
# url: http://192.168.10.1:8080
# login: [server_name]
# password: (API key)
# ---
```
#### Each server
```ini
# /etc/crowdsec/local_api_credentials.yaml
url: http://192.168.10.1:8080
login: [server_name]
password: (API key)
# /etc/crowdsec/acquis.d/sshd.yaml
---
source: journalctl
journalctl_filter:
- "_SYSTEMD_UNIT=ssh.service" labels:
type: sshd
# origin value is syslog
---
```
```bash
sudo systemctl restart crowdsec
```
#### OPNsense web UI
- Services:CrowdSec:Machines
- checking the lists of server
- Main CLI commands of CrowdSec
```sh
# View active decisions(Ban list)
cscli decisions list
# View alerts
cscli alerts list
# Check connected machines(agents)
cscli machines list
```
---
## TLS on crowdsec communication
TLS can be applied in CrowdSec communication when internal PKI are set (BIND, Step-CA, ACME-Client in OPNsense). CrowdSec communication can contain sensitive information such as API key, it is recommended to set TLS.
### General configuration
- Services:CrowdSec:Settings
- \[\*\] Manual LAPI configuration
### DNS setting
### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```text
...
crowdsec IN CNAME opnsense.ilnmors.internal.
...
```
### ACME setting
#### Use opnsense certificates
- Services:ACME Client:Certificates - Certificates
- Alt name: crowdsec.ilnmors.internal
#### Certificate and key file
- opnsense
- file:
- /var/etc/acme-client/cert-home/\[ramdom_string\]/opnsense.ilnmors.internal/
- fullchain.cer
- opnsense.ilnmors.internal.key
> There is `opnsense.ilnmors.internal.cer` file. However, when client verify the certificate, it verify the intermediate CA's certificate and root CA's certificate both. Therefore, in this case, use fullchain.cer
- /usr/local/etc/ssl/cert.pem
### Add TLS setting in LAPI configuration
- opnsense
- file:
- /usr/local/etc/crowdsec/config.yaml
- /usr/local/etc/crowdsec/local_api_credentials.yaml
- /usr/local/etc/crowdsec/bouncer/crowdsec-firewall-bouncer.yaml
```yaml
# config.yaml
api:
client:
# ...client configurations
server:
enable: true
listen_uri: 192.168.10.1:8080 # actual IP address is required. (Do not use FQDN in here, the service listener is binded on network interface)
# ... server configurations
tls:
cert_file: /var/etc/acme-client/cert-home/[random_string]/opnsense.ilnmors.internal/fullchain.cer
key_file: /var/etc/acme-client/cert-home/[ramdom_string]/opnsense.ilnmors.internal/opnsense.ilnmors.internal.key
# random string is generated by opnsense itself. In real environment, check it first.
```
```yaml
# local_api_credentials.yaml
...
url: https://crowdsec.ilnmors.internal:8080/
```
```yaml
# crowdsec-firewall-bouncer.yaml
...
api_url: https://crowdsec.ilnmors.internal:8080/
...
```
```sh
service crowdsec restart
```
#### CrowdSec LAPI restart setting
- opnsense
- file: /usr/local/etc/cron.d/crowdsec
```text
#minute hour mday month wday who command
0 3 * * * root /usr/local/libexec/crowdsec/upgrade-hub
30 3 * * * root /usr/sbin/service crowdsec reload # Add this line to reload every day.
```
### Each server configuration
#### Server's certificate trust
- each server
- file: /usr/local/share/ca-certificates/root_ca.crt
```bash
sudo update-ca-certificates
```
#### CrowdSec Agent setting
```ini
# /etc/crowdsec/local_api_credentials.yaml
url: https://crowdsec.ilnmors.internal:8080
login: [server_name]
password: (API key)
```
```bash
sudo systemctl restart crowdsec
```
---
## Crowdsec in Caddy-auth
Caddy supports bouncer. Also, it can be work as agent via auth server where Caddy-auth is located.
### Bouncer configuration
> Caddy has to contain local CA's root_ca.crt (Step-CA). - containerfile already includes root_ca.crt in container when it was built
#### OPNsense console
```sh
cscli bouncer add caddy-auth
> API key for 'caddy-auth':
> Secret_value
> Please keep this key since you will not be able to retrieve it!
>
>
cscli collections install crowdsecurity/caddy
```
#### Caddyfile
```ini
# ...
# Crowdsec bouncer setting
{
crowdsec {
# CrowdSec LAPI
api_url https://crowdsec.ilnmors.internal:8080
api_key "{env.CADDY_CROWDSEC_KEY}"
}
}
# ...
```
- `podman exec caddy-auth caddy reload --config /etc/caddy/Caddyfile`
### Agent configuration
#### auth sv
- File:
- /etc/crowdsec/acquis.yaml
- ~/data/container/caddy_auth/data/access.log
```yaml
# /etc/crowdsec/acquis.d/caddy-auth.yaml
filenames:
- /var/log/caddy.log
labels:
type: caddy
```
```ini
# Caddyfile
# ...
(crowdsec_log) {
log {
output file /data/access.log {
roll_size 10mb
roll_keep 5
}
}
}
# ...
caddy.ilnmors.com {
import crowdsec_log
route {
crowdsec
root * /usr/share/caddy
file_server
}
}
```
```bash
podman exec caddy-auth caddy reload --config /etc/caddy/Caddyfile
sudo mkdir /etc/crowdsec/acquis.d
sudo nano /etc/crowdsec/acquis.d/caddy-auth.yaml
ln -s /home/auth/data/containers/caddy-auth/data/access.log /var/log/caddy.log
# install collection(senario + parser) crowdsecurity/caddy
sudo cscli collections install crowdsecurity/caddy
sudo systemctl restart crowdsec
sudo cscli metrics
```
---
## PLAN
- distributed bouncer
- [x] Caddy bouncer
- [ ] dash board
docs/archives/2025-12/03_common/03_05_redis.md
+107
View File
@@ -0,0 +1,107 @@
Tags: #os, #configuration, #virtualization, #container, #database, #cache
## Redis
Redis is the cache database. It doesn't use SQL, it is just key-value data storage.
### Redis in Pod
Redis is combined with master services in the pod. It allows master services and Redis can communication in localhost. It doesn't need TLS, password, or even ACL itself.
There is the list of services which use Redis in Pod.
#### Preperation
##### Create directory for container
```bash
mkdir -p ~/data/containers/app/redis
setfacl -m d:g::0 ~/data/containers/app/redis
setfacl -m d:o::0 ~/data/containers/app/redis
setfacl -m u:
$UID:rwx ~/data/containers/app/redis
setfacl -m u:100998:rwx ~/data/containers/app/redis
setfacl -d -m u:$UID:rwx ~/data/containers/app/redis
setfacl -d -m u:100998:rwx ~/data/containers/app/redis
```
>Redis container executes as 999:999(redis:redis) permission in container. It is mapped host's 100998. Therefore, directories have to have ACL via `setfacl`
### Podman Image
```bash
podman pull redis:8.2.2 # Do not use latest version to management
```
### Quadlet
- File:
- ~/data/config/containers/app/app.pod
- ~/data/config/containers/app/app-redis.container
- ~/data/config/containers/app/app.container
```ini
# ~/data/config/containers/app/app.pod
[Quadlet]
DefaultDependencies=false
[Pod]
PodName=app
# web port
PublishPort=9080:9000/tcp
# LDAP port
#PublishPort=[set_port]:3389
# Prometheus Port
#PublishPort=[set_port]:9300
```
```ini
# ~/data/config/containers/app/app-redis.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=app - redis
Before=app.service
[Container]
Pod=app.pod
Image=redis:8.2.2
ContainerName=app-redis
# Port 6379
Volume=%h/data/containers/app/redis:/data:rw
Environment="TZ=Asia/Seoul"
[Install]
WantedBy=default.target
```
```ini
# ~/data/config/containers/app/app.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=app
After=app-redis.service
Requires=app-redis.service
[Container]
Pod=app.pod
Image=app:version
Volume=%h/data/containers/app/app:/data:rw
Environment="TZ=Asia/Seoul"
[Install]
WantedBy=default.target
```
docs/archives/2025-12/03_common/03_06_btrfs.md
+130
View File
@@ -0,0 +1,130 @@
Tags: #common, #configuration, #os, #filesystem
## BTRFS usage
### Setting
```bash
# All hdd needs partition, but it has no filesystem. To make a partition use `fdisk`.
sudo fdisk "$DIVICE_PATH"
> n # create the new parition
> 1 # Partition number
> Default # First Sector
> Default # Last Sectort
> w # write the new partition
# check btrfs-progs package
sudo apt list --installed | grep btrfs-progs
# btrfs-progs/stable,now 6.14-1 amd64 [installed]
sudo mkfs.btrfs -d raid10 -m raid10 -L hdd /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
# These are partition device files
```
### Snapshot
Usually, the Read-Only snapshot is used.
```bash
# Create snapshot_subvolume
sudo btrfs subvolume create /home/app/hdd/data
# Create snapshot_directory
mkdir /home/app/hdd/.snapshot
# Create snapshot
sudo btrfs subvolume snapshot -r /home/app/hdd/data /home/app/hdd/.snapshot/data_[date]
# Rollback (file)
cp /home/app/hdd/.snapshot/data_[date]/file /home/app/hdd/data/
# Roleback (volume)
# Current subvolume move
mv /home/app/hdd/data /home/app/hdd/data_fail
sudo btrfs subvolume snapshot /home/app/hdd/.snapshot/data_[date] /home/app/hdd/data
# If the data successfully recovered
sudo btrfs subvolume delete /home/app/hdd/data_fail
```
### replace HDD to new system
btrfs has its own volume manangement data and filesystem in hdd, as metadata. It has no dependency on specific OS or hardware but it can work on every linux system which supports btrfs.
```bash
# unmount filesystem
sudo umount /home/app/hdd
# turn off the system, and remove all hdd
# add all disk to new hardware(server), and turn on
# scan btrfs
sudo btrfs device scan
sudo btrfs filesystem show
# mount
sudo nano /etc/fstab
# LABEL=hdd /home/app/hdd btrfs defaults,compress=zstd,autodefrag 0 0
sudo mount -a
```
### add extra HDD
```bash
# Add HDD and check the device file
lsblk
# Add hdd to btrfs RAID
sudo btrfs device add /dev/xxx /dev/xxy /home/app/hdd
# Expand the volume
sudo btrfs balance start /home/app/hdd
```
### change HDD
#### btrfs replace
When sata slot is enough to connect new HDD and old HDD, you can use this way.
```bash
# check devid of old HDD
sudo btrfs device stats /home/app/hdd # you can check IO error
sudo btrfs filesystem show /home/app/hdd
# check the new disk's path
lsblk
# /dev/xxx
# Replace
sudo btrfs replace start [old HDD\'s devid] /dev/xxx /home/app/hdd
# check
sudo btrfs replace status /home/app/hdd
```
#### btrfs device add and delete
When sata slot is not enough to connect new HDD and old HDD simultaneously, you can use this way.
```bash
# Check the HDD which will change
sudo btrfs device stats /home/app/hdd # you can check IO error
sudo btrfs filesystem show /home/app/hdd # check disk's devid
# turn off the system and change the broken HDD to the new HDD
# Make sure the hardware supports Hot-Swap, if it didn't support, you would have to turn off your system fully when you are changing the HDD
# if the system couldn't mount automatically, mount it manually
# sudo mount -o degraded /dev/xxx /home/app/hdd # /dev/xxx is one of ordinary HDD
lsblk
# /dev/xxy # new HDD device file path
# Add new HDD
sudo btrfs device add /dev/xxy /home/app/hdd
# Delete broken HDD
sudo btrfs device delete missing /home/app/hdd
# To major balance
sudo btrfs balance start /home/app/hdd
```
docs/archives/2025-12/04_hypervisor/04_01_hypervisor.md
+277
View File
@@ -0,0 +1,277 @@
Tags: #os, #hypervisor, #configuration, #virtualization
## Hypervisor configuration
### Debian installation and configuration
- Following [debian configuration](../03_common/03_01_debian_configuration).
- Following [iptables configuration](../03_common/03_02_iptables.md).
- Following [crowdsec](../03_common/03_04_crowdsec.md) (After OPNsense LAPI configuration).
### QEMU/KVM
#### KVM
KVM is a virtualization engine to share and to allocate physical resource to VM. It works on Linux kernel.
#### QEMU
QEMU is an emulator. It works with KVM frequently, and it emulates hardware or other OS on hypervisor.
### Libvirtd
Libvirtd is a daemon to use libvirt API and command to manage virtualization engines and emulators including QEMU/KVM. It supports not only QEMU/KVM but also XEN, VMWare or etc.
#### Libvirtd configuration
##### LIBVIRT_DEFAULT_URI setting
Basically, KVM and libvirtd require root permission to access low-level device and configuration to implement virtual machines. However, it is possible to use libvirtd command without root permission; `sudo`. When the common user has its group as `kvm`, and `libvirt`, and export environment variable `export LIBVIRT_DEFAULT_URI='qemu:///system'` on `~/.bashrc`.
```bash
# ~/.bashrc
# If the file doesn't exist, execute the commands below
# sudo cp /etc/skel/.bashrc /home/vmm
# sudo cp /etc/skel/.profile /home/vmm
# sudo chown vmm:svadmins .bashrc .profile
# add the line below in .bashrc
echo "export LIBVIRT_DEFAULT_URI='qemu:///system'" >> ~/.bashrc
source ~/.bashrc
```
##### Directory
- ~/data/config/{scripts,server,services,vms}
- ~/data/config/vms/{networks,storages}
#### virsh
##### VM management
- virsh list \[--all\]: Print vm list
- virsh start \[vm_name\]: Start vm
- virsh shutdown \[vm_name\]: Send ACPI signal to vm (shutdown)
- virsh destroy \[vm_name\]: Stop vm forcefully
- virsh reboot \[vm_name\]: Reboot vm
- virsh autostart \[--disable\]: Register vm autostart
- qemu-img create -f \[format\] /path \[volume\]: Create the virtual disk file
##### VM configuration
- virsh edit \[vm_name\]: Open vm template file (xml) and edit
- virsh dumpxml \[vm_name\] > \[file\].xml: Backup vm template file
- virsh define \[file\].xml: Create vm from backup file
- virsh undefine \[vm_name\] \[--nvram\]: Remove vm
> When vm is set as uefi, `--nvram` option is required.
- virsh domrename \[old_vm_name\] \[new_vm_name\]: Rename vm
##### VM access
- virsh console \[vm_name\]: Access vm via serial console
##### Virtual network
- virsh net-define \[file\].xml: Create virtual network from xml file
- virsh net-start \[network_name\]: Start virtual network
- virsh net-autostart \[network_name\]: Register virtual network auto start
##### Snapshot
- virsh snapshot-create-as --domain \[vm_name\] --name \[snapshot_name\] --description \[description\] --disk-only --atomic (for qcow2 format)
- virsh snapshot-list \[vm_name\]
##### VM pool management
- virsh pool-define ~/data/config/vms/storages/vm-images.xml
- virsh pool-start vm-images
- virsh pool-autostart vm-images
- virsh pool-refresh vm-images - After put the images in the pool
- virsh vol-list vm-images
#### virt-install
virt-install command helps to define vm template file with various options.
```bash
virt-install \
[--import] \ # without booting images, boot with disk. [--cdrom|--location] is not available
--boot uefi \ # activate secure booting
--name vm_name \
--os-variant [os] \ # set optimized setting for each OS
--vcpu [num] \
--memory [num] \ # memory size(unit: MiB)
--location [path] \ # or cdrom, add booting images, location is for serial booting
# --disk path=[path],format=[raw|qcow2],discard=unmap \ # discard=unmap: ssd emulation
# --disk vol=vm-images/my-existing-disk.qcow2 \ # To use pool's volume
--disk pool=vms-images,size=[num],format=qcow2,discard=unmap \ # Create qcow2 image with pool configuration
--network network=[network_name],model=virtio,mac=[mac_address] \
--graphics none \
--console pty,target_type=serial \
--extra-args "console=ttyS0,115200" # Define the console configuration
```
#### virsh network configuration
Libvirt's network is defined and saved as a XML. There is setting below. It can be compatible with host's `open vswitch` interfaces.
- Make xml files
- ~/data/config/vms/networks/ovs-wan-net.xml
- ~/data/config/vms/networks/ovs-lan-net.xml
- `chmod 600 ~/data/config/vms/networks/*`
```xml
<!-- WAN network for OVS -->
<!-- ovs-wan-net.xml -->
<network>
<name>ovs-wan-net</name>
<forward mode='bridge'/>
<bridge name='ovsbr0'/>
<virtualport type='openvswitch'/>
</network>
```
```xml
<!-- LAN network for OVS -->
<!-- ovs-lan-net.xml -->
<network>
<name>ovs-lan-net</name>
<forward mode='bridge'/>
<bridge name='ovsbr1'/>
<virtualport type='openvswitch'/>
<portgroup name='vlan-trunk'>
<vlan trunk='yes'>
<tag id='1' nativeMode='untagged'/>
<tag id='10'/>
</vlan>
</portgroup>
<portgroup name='vlan10-access'>
<vlan>
<tag id='10'/>
</vlan>
</portgroup>
</network>
```
- Define XML files
```bash
virsh net-define ~/data/config/vms/networks/ovs-wan-net.xml
virsh net-define ~/data/config/vms/networks/ovs-lan-net.xml
virsh net-start ovs-wan-net
virsh net-start ovs-lan-net
virsh net-autostart ovs-wan-net
virsh net-autostart ovs-lan-net
```
#### virsh storage pool configuration
- Make a storage pool file
- ~/data/config/vms/storages/vm-images.xml
- `chmod 600 ~/data/config/vms/storages/*`
```xml
<!-- Storage pool define -->
<!-- vm-images.xml -->
<pool type='dir'>
<name>vm-images</name>
<target>
<path>/var/lib/libvirt/images</path>
</target>
</pool>
```
- Define XML files
```bash
virsh pool-define ~/data/config/vms/storages/vm-images.xml
virsh pool-start vm-images
virsh pool-autostart vm-images
```
#### ACL setting
```bash
# To manage qcow2 file without non-root user
sudo setfacl -R -m u:vmm:rwx /var/lib/libvirt/images
sudo setfacl -d -m u:vmm:rwx /var/lib/libvirt/images
```
### Backup configuration
```bash
cp /etc/network/interfaces /etc/default/grub /etc/modprobe.d/vfio.conf ~/data/config/server
sudo cp /etc/iptables/rules.v4 ~/data/config/server
```
### Systemd
#### Linger configuration
```bash
# Check linger configuration
loginctl show-user vmm
# Linger=yes
# It is necessary to use the services after session is done
```
- opnsense.service
```ini
# ~/data/config/services/opnsense.service
# ~/.config/systemd/user/opnsense.service
[Unit]
Description=opnsense Auto Booting
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
# Maintain status as active
RemainAfterExit=yes
# Run the service
ExecStart=/usr/bin/virsh -c qemu:///system start opnsense
# Stop the service
ExecStop=/usr/bin/virsh -c qemu:///system shutdown opnsense
[Install]
WantedBy=default.target
```
```bash
mkdir -p ~/.config/systemd/user
chmod -R 700 ~/.config
ln -s ~/data/config/services/opnsense.service ~/.config/systemd/user/opnsense.service
systemctl --user daemon-reload
```
- net.service
```ini
# ~/data/config/services/net.service
# ~/.config/systemd/user/net.service
[Unit]
Description=net Auto Booting
After=opnsense.service
Requires=opnsense.service
[Service]
Type=oneshot
# Maintain status as active
RemainAfterExit=yes
# 8080 is CrowdSec API
ExecStartPre=%h/data/config/scripts/wait-for-it.sh -h 192.168.10.1 -p 8080 -t 20
# Run the service
ExecStart=/usr/bin/virsh -c qemu:///system start net
# Stop the service
ExecStop=/usr/bin/virsh -c qemu:///system shutdown net
[Install]
WantedBy=default.target
```
docs/archives/2025-12/05_firewall/05_01_opnsense_vm.md
+130
View File
@@ -0,0 +1,130 @@
Tags: #os, #firewall, #configuration, #network, #virtualization
## Preparation
### Create VM template
- ~/data/config/scripts/opnsense.sh
```bash
virt-install \
--import \ # Start without CD-ROM or location - This is for Serial installation.
--boot uefi \ # Supports secure booting
--name opnsense \ # VM name
--os-variant freebsd14.2 \ # Choose kind of OS
--vcpus 2 \ # Set number of vcpu (It means core of pcpu)
--memory 4096 \ # Set the memory volume (Default: MiB)
--disk path=/var/lib/libvirt/images/OPNs
ense-25.7-serial-amd64.img,format=raw \ # Installation image file, format raw
--disk pool=vm-images,size=72,format=qcow2,discard=unmap \ # The target disk to install opnsense, it is on SSD; discard=unmap option will emulate SSD.
--network network=ovs-wan-net,model=virtio,mac=0A:49:6E:4D:00:00 \ # Use pre-designated network and specific MAC address
--network network=ovs-lan-net,portgroup=vlan-trunk,model=virtio,mac=0A:49:6E:4D:00:01 \
--graphics none \ # Don't use any graphic
--console pty,target_type=serial # Use console serial
# After enter this command, then the console start automatically
# Remove all annotation before you make the sh file.
```
### OPNsense installation
```sh
# Interface setting
Press any key to start the manual interface assignment: [enter]
LAGGs: N
VLANs: N
Enter the WAN interface name or 'a' for auto-detection: vtnet0
Enter the LAN interface name or 'a' for auto-detection: vtnet1
Enter the Optional interface 1 name or 'a' for auto-detection: [blank]
Do you want to proceed? : Y
# Login as an installer account
login: installer
password: opnsense
# installing
Continue with default keymap
Install (ZFS)
stripe
[*] vtbd1: yes
Complete Install: Halt Now
```
### Modify VM template
- virsh edit
```bash
virsh edit opnsense
```
- Edit template file
```xml
<!-- opnsense -->
...
</vcpu>
<cputune>
<shares>2048</shares>
</cputune>
<!-- cpu priority - 1024: default/2048: high/512: low -->
<!--
<disk>
...
<source file='/var/lib/libvirt/images/opnsense.img'/>
...
</disk>
# Remove booting disk
-->
...
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<!-- Define QEMU-Guest-Agent channel -->
```
- Save template file
```bash
virsh dumpxml opnsense > ~/data/config/vms/dumps/opnsense.xml
```
- opnsense.service
```ini
# ~/data/config/services/opnsense.service
# ~/.config/systemd/user/opnsense.service
[Unit]
Description=opnsense Auto Booting
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
# Maintain status as active
RemainAfterExit=yes
# Run the service
ExecStart=/usr/bin/virsh -c qemu:///system start opnsense
# Stop the service
ExecStop=/usr/bin/virsh -c qemu:///system shutdown opnsense
[Install]
WantedBy=default.target
```
```bash
mkdir -p ~/.config/systemd/user
chmod -R 700 ~/.config
ln -s ~/data/config/services/opnsense.service ~/.config/systemd/user/opnsense.service
systemctl --user daemon-reload
systemctl --user enable opnsense.service
systemctl --user start opnsense.service
```
docs/archives/2025-12/05_firewall/05_02_opnsense_general.md
+91
View File
@@ -0,0 +1,91 @@
Tags: #os, #firewall, #configuration, #network
## Web UI
### Access
- Browser: https://192.168.1.1
- Login with root
### Wizard setting
- Hostname: opnsense
- Domain: ilnmors.internal
- Timezone: Asia/Seoul
> - System:Settings:General - Time Zone
- DNS Server: 1.1.1.2
> Temporary DNS server, when Adguard home and BIND is building, the change it.
> - System:Setting:General - DNS server
- \[ \] Override DNS
- \[ \] Enable Resolver
- WAN
- type dhcp
- \[\*\]Block RFC1918 Private Networks
- \[\*\]Block bogon networks
- LAN
- \[ \] Configure DHCP server
- Change root password
### User setting
- System:Access:Users - \[+\]
- Name: opnsense
- Full name: opnsense management account
- E-mail: opnsense@ilnmors.internal
- Group: admins
> When console access is needed, root account is necessary. Only root account can access console program. However, when using WebUI it is better to use separated account for security.
### system update and pkg
#### Updates
- System:Firmware:Updates
#### Plugins
- System:Firmware:Plugins
- \[\*\] Community plugins
- os-crowdsec
- os-acme-client
- os-qemu-guest-agent
>after install and restart plugin check with CLI `virsh qemu-agent-command opnsense '{"execute":"guest-get-osinfo"}'` on hypervisor
- os-telegraf
- os-sftp-backup
## Disable unused basic function
In this homelab project, Kea DHCP and AdGuard home, BIND will be used as DHCP server and DNS server. Therefore, it is necessary to disable Dnsmasq, ISC DHCP and Unbound which are used as basic function of DHCP and DNS in OPNsense.
- Services:Dnsmasq DNS&DHCP:General
- \[ \] Enable
- `Apply`
- Services:ISC DHCPv4
- \[ \] Enable
- `Apply`
- Services:ISC DHCPv6
- \[ \] Enable
- `Apply`
- Services:Unbound DNS:General
- \[ \] Enable
- `Apply`
## Backup
### ZFS Snapshot
Before major updating, it is important to make a ZFS snapshot.
- System:Snapshots - \[+\]
- Name: \[Date\]
- `Save`
### Setting backup
OPNsense is managed with configuration file. You can download the configuration file as XML format.
- System:Configuration:Backups
- Download: `Download configuration`
- Restore: Select file > `Restore configuration`
- sftp: URL / SSH private key
docs/archives/2025-12/05_firewall/05_03_opnsense_interface.md
+159
View File
@@ -0,0 +1,159 @@
Tags: #os, #firewall, #configuration, #network
## Interface configuration
### VLAN setting
- Interfaces:Devices:VLAN - \[+\]
| Device | Parent | Tag | Description |
| :--------: | :----: | :-: | :---------: |
| vlan0.1.10 | vtnet1 | 10 | Server |
> The device name is reserved in Web UI; Name of VLAN device must start with `vlan0` or `qinq0`. In this project, name of vlan device would be `vlan0.[interface_num].[tag_num]`
- `Apply`
### VPN setting
- VPN:WireGuard
- \[\*\] Enable WireGuard
- VPN:WireGuard:Instance - \[+\]
| Name | Listen Port | Tunnel Address |
| :-------: | :---------: | :------------: |
| WG_SERVER | 11290 | 10.10.10.1/24 |
| WG_USER | 11291 | 10.10.1.1/24 |
- VPN:WireGuard:PeerGenerate
| Instance | End point | Name | Address | Pre-shared Key | Allowed IP | Keepalive interval | DNS |
| :-------: | :---------------: | :-----: | :-----------: | :------------: | :------------: | :----------------: | :-----------: |
| WG_SERVER | ilnmors.com:11290 | console | 10.10.10.2/32 | Generate | 192.168.0.0/16 | 25 | 192.168.10.11 |
| WG_SERVER | ilnmors.com:11290 | phone | 10.10.10.3/32 | Generate | 192.168.0.0/16 | 25 | 192.168.10.11 |
| WG_SERVER | ilnmors.com:11290 | spare | 10.10.10.4/32 | Generate | 192.168.0.0/16 | 25 | 192.168.10.11 |
> Set `PersistentKeepalive = 25` on peer, to avoid NAT timeout.
> Press `Store and generate next` button after generate each row.
> Before building net server (Private DNS), use 1.1.1.2 instead of 192.168.10.11.
```ini
# console
[Interface]
PrivateKey = 2ACJCZV7Zg4fxCHTxfbmAggX/x06Nt05CC6gJvcrokI=
Address = 10.10.10.2/32
DNS = 192.168.10.11
[Peer]
PublicKey = jqDJKe8pZSK8GXwBnrjBJiflYvJDB7GfgogLsSSdnA4=
PresharedKey = OCx3mohOp2Uipxda7ZJs+78Mjh3Lbf0UfvGZB4SULk4=
Endpoint = ilnmors.com:11290
AllowedIPs = 192.168.0.0/16
PersistentKeepalive = 25
# phone
[Interface]
PrivateKey = oIhpQn7yTEU3wH+eOVJuzcgo4t05MBqv+OmD1KC/Z38=
Address = 10.10.10.3/32
DNS = 192.168.10.11
[Peer]
PublicKey = jqDJKe8pZSK8GXwBnrjBJiflYvJDB7GfgogLsSSdnA4=
PresharedKey = pqA5OtJy7lZHD+PzzqVEYa/iMHOOCNagAzCfTLoaDmw=
Endpoint = ilnmors.com:11290
AllowedIPs = 192.168.0.0/16
PersistentKeepalive = 25
# spare
[Interface]
PrivateKey = EIHZRuI5IDG3h8mC3ez4I1duYuVbe5UwgWnVxbg9uFk=
Address = 10.10.10.4/32
DNS = 192.168.10.11
[Peer]
PublicKey = jqDJKe8pZSK8GXwBnrjBJiflYvJDB7GfgogLsSSdnA4=
PresharedKey = Xuf/G0HJ2S/zO0C4zUjjW2rD1b51yUYNhZaByEgfM6I=
Endpoint = ilnmors.com:11290
AllowedIPs = 192.168.0.0/16
PersistentKeepalive = 25
```
- `Apply`
### Interface assignment
- Interface:Assignments
- vlan0.1.10 - VLAN10 - `Add`
- wg0 - WG_SERVER - `Add`
- wg1 - WG_USER - `Add`
- `Save`
### Assigned interface setting
- Interface:WAN
- \[\*\] Enable
- \[\*\] Block private
- \[\*\] Block bogon
- \[\*\] IPv4 DHCP
- `Save`
- Interface:LAN
- \[\*\] Enable
- IPv4 Static (IPv6 Configuration Type: None)
- 192.168.1.1/24
- `Save`
- Interface:VLAN10
- \[\*\] Enable
- IPv4 Static
- 192.168.10.1/24
- Interface:WG_SERVER
- \[\*\] Enable
- `Save`
- Interface:WG_USER
- \[\*\] Enable
- `Save`
- `Apply change`
### Web UI itself settings
- System:Settings:Administration
- Web GUI
- Listen Interfaces: LAN, VLAN10
- `Save`
- Firewall:Settings:Advanced
- \[\*\] Disable anti-lockout
- `Save`
## Aliases and group configuration
### Aliases setting
- Firewall:Aliases - \[+\]
| Name | Type | Content | Description |
| :------------: | :-----: | :----------------------------------------------------------------------------------------------------: | :-----------------------------: |
| ports_vpn | port(s) | 11290,11291 | ports udp vpn |
| ports_web | port(s) | 80,443 | ports tcp web |
| ports_dhcp4 | port(s) | 67,68 | ports udp dhcp4 |
| ports_dns | port(s) | 53,443 | ports tcp/udp dns including DoH |
| ports_crowdsec | port(s) | 8080 | ports tcp crowdsec |
| hosts_console | host(s) | 192.168.1.11,10.10.10.2,10.10.10.3,10.10.10.4 | hosts console |
| hosts_server | host(s) | 192.168.10.10,192.168.10.11,192.168.10.12,192.168.10.13,192.168.10.14,10.10.10.2,10.10.10.3,10.10.10.4 | hosts servers |
| hosts_net | host(s) | 192.168.10.11 | hosts net |
| hosts_auth | host(s) | 192.168.10.12 | hosts auth |
- `Apply`
### Group setting
- Firewall:Groups - \[+\]
| Name | Members | Description |
| :----: | :---------------: | :----------: |
| SERVER | VLAN10, WG_SERVER | Server group |
| USER | LAN, WG_USER | User group |
- `Apply`
docs/archives/2025-12/05_firewall/05_04_opnsense_rules.md
+79
View File
@@ -0,0 +1,79 @@
Tags: #os, #firewall, #configuration, #network, #security
## OPNsense rules
### NAT
- Firewall:NAT:Outbound
- Mode: Automatic outbound NAT rule generation (no manual rules can be used)
- Firewall:NAT:Port Forward - \[+\]
| Interface | TCP/IP version | Proto | Destination | Destination port | Redirect target IP | Redirect target port | Description |
| :-------: | :------------: | :---: | :---------: | :--------------: | :----------------: | :------------------: | :-----------------------------------------------: |
| WAN | IPv4 | TCP | WAN address | ports_web | hosts_auth | ports_web | allow wan clients to access web services from WAN |
> Set after building main reverse proxy, all web packets from WAN are going to the reverse proxy.
> Filter rule association's `Add associated filter rule` option will automatically generate the `Pass` rule on WAN interface for Port Forwarding rule.
### Firewall rules
OPNsense has prioirty in order to rule's squence. The upper rule is prior than below one. Moreover, when `Quick` option is enabled, OPNsense doesn't check the rule below. It means OPNsense just applies the rule what it is first matched (First match). Reversely, when `Quick` option is disabled, OPNsense checks all rules to apply the packet (Last match).
#### WAN interface
- Firewall:Rules:WAN - \[+\]
| Action | Quick | Interface | Direction | TCP/IP version | Proto | Source | Destination | Destination port | Description |
| :----: | :---: | :-------: | :-------: | :------------: | :---: | :----: | :---------: | :--------------: | :------------------------------------------------------------------------: |
| Pass | * | WAN | IN | IPv4 | UDP | * | WAN address | ports_vpn | allow vpn clients from WAN |
| Pass | * | WAN | IN | IPv4 | TCP | * | WAN address | ports_web | allow wan clients to access web services from WAN(Automatically generated) |
- `Apply changes`
#### SERVER interface
- Firewall:Rules:SERVER - \[+\]
| Action | Quick | Interface | Direction | TCP/IP version | Proto | Srouce | Destination | Destination port | Description |
| :----: | :---: | :-------: | :-------: | :------------: | :---: | :-----------: | :-----------: | :--------------: | :--------------------------------------------------------------------: |
| Pass | * | SERVER | IN | IPv4 | UDP | * | * | ports_dhcpv4 | allow server clients to access DHCPv4 server from SERVER net |
| Pass | * | SERVER | IN | IPv4 | ICMP | SERVER net | This Firewall | - | allow server clients to access This Firewall with ICMP from SERVER net |
| Block | * | SERVER | IN | IPv4 | * | !hosts_server | * | * | block undesignated server clients to access any from SERVER net |
| Pass | * | SERVER | IN | IPv4 | UDP | SERVER net | This Firewall | NTP | allow server clients to access NTP from SERVER net |
| Pass | * | SERVER | IN | IPv4 | TCP | SERVER net | This Firewall | ports_crowdsec | allow server clients to access CrowdSec LAPI from SERVER net |
| Pass | * | SERVER | IN | IPv4 | TCP | hosts_console | This Firewall | ports_web | allow console to access This Firewall web GUI from SERVER net |
| Pass | * | SERVER | IN | IPv4 | * | hosts_console | USER net | * | allow console to access USER net from SERVER net |
| Block | * | SERVER | IN | IPv4 | TCP | SERVER net | This Firewall | * | block server clients to access This Firewall from SERVER net |
| Block | * | SERVER | IN | IPv4 | * | SERVER net | USER net | * | block server clients to access USER net from SERVER net |
| Pass | - | SERVER | IN | IPv4 | * | SERVER net | * | * | allow server clients to access WAN from SERVER net |
- `Apply changes`
> When you add the new server client, you must edit alias `hosts_server`
#### USER interface
- Firewall:Rules:USER - \[+\]
| Action | Quick | Interface | Direction | TCP/IP version | Proto | Srouce | Destination | Destination port | Description |
| :----: | :---: | :-------: | :-------: | :------------: | :-----: | :-----------: | :-----------: | :--------------: | :----------------------------------------------------------------: |
| Pass | * | USER | IN | IPv4 | UDP | * | * | ports_dhcpv4 | allow user clients to access DHCPv4 server from USER net |
| Pass | * | USER | IN | IPv4 | ICMP | USER net | This Firewall | - | allow user clients to access This Firewall with ICMP from USER net |
| Pass | * | USER | IN | IPv4 | TCP/UDP | USER net | hosts_net | ports_dns | allow user clients to access DNS server from USER net |
| Pass | * | USER | IN | IPv4 | TCP | USER net | hosts_auth | ports_web | allow user clients to access reverse proxy from USER net |
| Pass | * | USER | IN | IPv4 | TCP | hosts_console | This Firewall | ports_web | allow console to access This Firewall web GUI from USER net |
| Pass | * | USER | IN | IPv4 | * | hosts_console | SERVER net | * | allow console to access SERVER net from USER net |
| Block | * | USER | IN | IPv4 | * | USER net | This Firewall | * | block user clients to access This Firewall from USER net |
| Block | * | USER | IN | IPv4 | * | USER net | SERVER net | * | block user clients to access SERVER net from USER net |
| Pass | - | USER | IN | IPv4 | * | USER net | * | * | allow user clients to access WAN from USER net |
- `Apply changes`
#### LAN interface
OPNsense automatically generates all pass rules on LAN interface. The USER group will be used instead of LAN interface, all rules will be disabled after USER interface rules set.
- Firewall:Rules:LAN
- Disable all auto generated rules
- `Apply changes`
docs/archives/2025-12/05_firewall/05_05_opnsense_suricata.md
+69
View File
@@ -0,0 +1,69 @@
sTags: #os, #firewall, #configuration, #network, #security
## Suricata
Suricata is IPS(Intrusion Prevent System)/IDS(Intrusion Detection System) to supersede snort in 2010 from OISF for NSM(Network Security Monitoring). This program supports multi-thread and check the packet and even the programs (such as exe or shell script) in it based on rules. When the suricata find the packet matched with rule it has, it decides `pass; with alert`, `block` the packet.
### General setting in OPNsense
#### Enable Suricata
- Services:Intrusion Detection:Administration - Settings
- \[\*\] Enabled
- \[\*\] IPS mode
- \[\*\] Promiscuous mode (Scan all traffic even L2 traffics)
- Interfaces: Select ALL
- Pattern matcher: Hyperscan (Intel's opensource regex matching library)
#### Rule set download and update
- Services:Intrusion Detection:Administration - Download
- Select the rule set below
- ET open/botcc: List of bot net C&C server
- ET open/compromised: List of known zombie PC
- ET open/drop: List of certain malicious traffic
- ET open/dshield: List of current activated malicious IP
- ET open/emerging-attack_response: List of response after attack success
- ET open/emerging-coinminer: List of coinminer malicious code
- ET open/emerging-current_event: List of latest attack pattern
- ET open/emerging-dns: List of malicious DNS query
- ET open/emerging-exploit: List of attack towards software vulnerability
- ET open/emerging-exploit_kit: List of automatic hacking tool
- ET open/emerging-ja3: List of malicious programs'(tools, or browsers) finger print
- ET open/emerging-malware: List of malware C&C server
- ET open/emerging-mobile_malware: List of mobile malware C&C server
- ET open/emerging-phishing: List of communication regard phishing
- ET open/emerging-policy: List of non-standard or malicious TLS version or http traffics
- ET open/emerging-scan: List of port scan or network scan traffic
- ET open/emerging-shellcode: List of malicious attack shell script
- ET open/emerging-sql: List of sql injection traffic
- ET open/emerging-user_agent: List of malicious bot user-agent
- ET open/emerging-web_client: List of attack towards web browser
- ET open/emerging-web_server: List of attack towards web server
- ET open/emerging-web_specific_apps: List of attack towards specific web application(like word press)
- ET open/emerging-CS_c2: List of Cobalt Strike hacking tool
- `Enable selected`
- `Download & Update Rules`
> When you want to delete rule set, then select target rule and click `Disable selected` and `Download & Update Rules`
- Services:Intrusion Detection:Schedule
- \[\*\] Enable update
#### Policy of rule set
Suricata rule set in OPNsense has basically `Alert` policy. Therefore, it is necessary to set rules as `Drop` manually. Except `ET open/emerging-ja3` and `ET open/emerging-policy`, it is good for security to drop all rule set. The reason why `ja3` and `policy` remain as `Alert` is they are not extremely dangerous, and they could drop normal packets frequently.
- Services:Intrusion Detection:Policy - Policies - \[+\]
- Ruleset: All ruleset except `ja3` and `policy`
- Action: Alert, Drop
- New action: Drop
- `Save`
- `Apply`
### Check the log
- Services:Intrusion Detection:Administration - Alerts
If suricata made the wrong action like block the normal packets or pass malicious packet, you can tune the ruleset. At the Alerts section, next to log there's Info column. When you click the pencil icon on info column, then you can change that rule's action in specific environment; source IP. It is a suppress.
docs/archives/2025-12/05_firewall/05_06_opnsense_acme.md
+133
View File
@@ -0,0 +1,133 @@
Tags: #os, #firewall, #configuration, #network, #security
## ACME client in OPNsense
ACME client needs private CA(Step-CA) and BIND to issue private certificates for TLS.
### Plugin package
- os-acme-client
### DNS records
- Add new domain in BIND, Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```text
...
opnsense IN A 192.168.10.1
...
```
### Trust setting (CA)
- Add root CA's crt
- System:Settings:Trust:Authorities - \[+\]
- Method: Import an existing Certificate Authority
- Description: step-ca.ilnmors.internal
- Certificate Data: Content of root_ca.crt of Step-CA
- `Save`
### ACME Client setting
#### ACME account and server setting
- Services:ACME Client:Settings
- \[\*\] Enable Plugin
- \[\*\] Auto Renewal
- `Apply`
- Services:ACME Client:Accounts - \[+\]
- Name: acme.ilnmors.internal
> It is only referred to OPNsense itself. It is not the provisioner name. It doesn't support special characters such as `@` or `!`.
- Description: acme
- Custom CA URL: https://step-ca.ilnmors.internal:9000/acme/acme@ilnmors.internal/directory
> `[CA_URL]:[port]/acme/[provisioner_name]/directory`
- `Save`
- `Register account` and check Status column `OK (registered)`
#### ACME challenge
There is private authoritative DNS server in this homelab, therefore DNS-01 challenge will be used.
- Services:ACME Client:Challenge Types - \[+\]
- Name: ilnmors.internal-dns-01-challenge
- Description: step-ca.ilnmors.internal dns-01 challenge
- Challenge Type: DNS-01
- DNS service: nsupdate (RFC 2136)
- DNS Sleep Time: 10
> If this option weren't set, ACME client send query towards public DNS. It is necessary to set this option to use private authoritative DNS server. After set value(second) ACME client send the query towards private authoritative DNS.
- Server (FQDN): bind.ilnmors.internal 2253
> DNS server domain and port. It uses \[space\] as separator.
- Secret Key:
```text
key "acme-key" {
algorithm hmac-sha256;
secret "secret value";
};
```
> About key, following [here](../06_network/06_03_net_bind.md)
- Zone: ilnmors.internal
#### Certificates
- Services:ACME Client:Certificates - \[+\]
- Common name: opnsense.ilnmors.internal
- Description: opnsense
- Alt Names: crowdsec.ilnmors.internal
- ACME account: acme.ilnmors.internal
- Challenge Type: ilnmors.internal-dns-01-challenge
- \[\*\] Auto Renewal
- `Save`
- Click `Issue or renew certificate` and check Last ACME status `OK`
#### Automations
##### Web UI restart automation configuration
- Services:ACME Client:Automations - \[\+\]
- Name: opnsense-auto-restart-web-ui
- Description: restart opnsense web ui
- Run Command: Restart OPNsense web UI
> Web UI account `opnsense` doesn't use `sh`, it makes hard ACME client run the command via sh. Just use crontab to reload the crowdsec service everyday in `/usr/local/etc/cron.d/crowdsec`.
##### Example of remote automations
OPNsense's ACME client can upload certificates to other server or run specific command when the certificates is issued or renewed. Here is the example to upload the certificate towards other server via sftp.
- Services:ACME Client:Automations - \[+\]
- Name: test_acme
- Description: certificate upload test
- Run Command: Upload certificate via SFTP
- SFTP Host: net.ilnmors.internal
- Host Key: \[blank\]
> When it is not set, it automatically registers host key in `known_hosts`.
- Username: net
- Identity Type: ed25519
- Remote Path: /home/net/certificate_test
- `Show Identity` and copy the key value
```text
ssh-ed25519 [key_value] root@opnsense.ilnmors.internal
```
- Add this key on target server's ~/.ssh/authorized_keys
- `Test Connection`
- `Save`
- Services:ACME Client:Certificates - \[Edit Certificates\]
- Automations: test_acme
### OPNsense Web UI certificate setting
- System:Settings:Administration
- SSL certificate: opnsense.ilnmors.internal (ACME Client)
### Crowdsec TLS setting
Following [here](../03_common/03_04_crowdsec.md)
docs/archives/2025-12/05_firewall/05_07_opnsense_kea.md
+40
View File
@@ -0,0 +1,40 @@
Tags: #os, #firewall, #configuration, #network
## Kea DHCP in OPNsense
Kea DHCP is the next generation DHCP server of ISC(Internet Systems Consortium) to supersede ISC DHCP. Kea has more flexibility in configuration, and higher performance, and modern API than ISC DHCP. OPNsense also select Kea as the future DHCP server than ISC DHCP. Basically, Kea supports container(Docker or podman) but it doesn't use docker.io repository but cloudsmith.io repository. Moreover, Kea DHCP is separated into 3 module kea ddns, kea dhcp 4, kea dhcp 6 as containers. This fact makes network environment (especially when the rootless podman network and DHCP relay are considered.) complex. Therefore, even though Kea in OPNsense doesn't support DDNS module, in this home lab Kea DHCP will be located in OPNsense.
### Kea DHCP setting
- Services:Kea DHCP:Kea DHCPv4 - Settings
- \[\*\] Enabled
- Interfaces: LAN, VLAN10
- \[ \] Firewall rules (Manually created)
- Services:Kea DHCP:Kea DHCPv4 - Subnets - \[+\]
- \[ \] Match client-id (To match based on MAC)
- \[ \] Auto collect option data (To designate optional data manually)
| Subnet | Description | Pool | Routers(gateway) | DNS | Domain name |
| :-------------: | :---------: | :---------------------------: | :--------------: | :-----------: | :--------------: |
| 192.168.1.0/24 | LAN | 192.168.1.100-192.168.1.254 | 192.168.1.1 | 192.168.10.11 | ilnmors.internal |
| 192.168.10.0/24 | VLAN10 | 192.168.10.254-192.168.10.254 | 192.168.10.1 | 192.168.10.11 | ilnmors.internal |
> Before building net server (Private DNS), use 1.1.1.2 instead of 192.168.10.11.
> The reason why VLAN10's pool has only 192.168.1.254 is VLAN10 allows only reservation IP.
- Services:Kea DHCP:KeaDHCPv4 - Reservation
- Delete all rows.
- Import csv file: kea_dhcp_v4_reservation.csv
```csv
ip_address,hw_address,hostname,description,option_data,
192.168.1.11,D8:E2:DF:FF:1B:D5,surface,console eth,
192.168.1.30,38:CA:84:94:5E:06,printer,printer,
192.168.10.12,0A:49:6E:4D:02:00,auth,auth,
192.168.10.13,0A:49:6E:4D:03:00,dev,dev,
192.168.10.14,0A:49:6E:4D:04:00,app,app,
```
- `Apply`
> vmm(Hypervisor), opnsense(firewall), net(Net server which has DNS in it) should use static IP instead of DHCP reservation. Because these servers are fundamental servers for building network and DHCP. It is very stable for DNS, hypervisor, firewall themselves to use static IP.
docs/archives/2025-12/06_network/06_01_net_vm.md
+107
View File
@@ -0,0 +1,107 @@
Tags: #os, #configuration, #network, #virtualization
## Preparation
### Create VM template
- ~/data/config/scripts/net.sh
```bash
virt-install \
--boot uefi \
--name net \
--os-variant debian13 \
--vcpus 1 \
--memory 2048 \
--location /var/lib/libvirt/images/debian-13.0.0-amd64-netinst.iso \ # For serial installing, use `--location` instead of `--cdrom`
--disk pool=vm-images,size=34,format=qcow2,discard=unmap \
--network network=ovs-lan-net,portgroup=vlan10-access,model=virtio,mac=0A:49:6E:4D:01:00 \ # Use designated ovs port group
--graphics none \
--console pty,target_type=serial \
--extra-args "console=ttyS0,115200"
# After enter this command, then the console starts automatically
# Remove all annotation before you make the sh file.
```
### Debian installation
- Following [here](../03_common/03_01_debian_configuration.md) to install Debian.
- Debian installer supports serial mode regardless getty@ttyS0 service is enabled or not.
- Following [here](../03_common/03_02_iptables.md) to set iptables.
- Following [here](../03_common/03_04_crowdsec.md) to set CrowdSec
#### Serial console setting
After installation, use `ctrl + ]` to exit console. Before setting getty@ttyS0, you can't use serial console to access VM. Therefore, use IP address set on installation, and connect net server via ssh first, following the step to enable the getty.
### Modify VM template settings
After getty setting, shutdown net vm with `shutdown` in VM or `sudo virsh shutdown net` in hypervisor to turn off vm first.
```bash
virsh edit net
```
```xml
<!-- net -->
...
</vcpu>
<cputune>
<shares>512</shares>
</cputune>
<!-- cpu priority - 1024: default/2048: high/512: low -->
<!--
<disk type='file' device='cdrom'>
...
</disk>
# Remove booting disk
-->
```
```bash
virsh dumpxml net > ~/data/config/vms/dumps/net.xml
# Start net server with console
```
### Common setting
- net.service
```ini
# ~/data/config/services/net.service
# ~/.config/systemd/user/net.service
[Unit]
Description=net Auto Booting
After=network-online.target
Wants=network-online.target
Requires=opnsense.service
[Service]
Type=oneshot
# Maintain status as active
RemainAfterExit=yes
# CrowdSec should be set
ExecStartPre=%h/data/config/scripts/wait-for-it.sh 192.168.10.1:8080 -t 0
ExecStartPre=/bin/bash -c "sleep 15"
# Run the service
ExecStart=/usr/bin/virsh -c qemu:///system start net
# Stop the service
ExecStop=/usr/bin/virsh -c qemu:///system shutdown net
[Install]
WantedBy=default.target
```
```bash
ln -s ~/data/config/services/net.service ~/.config/systemd/user/net.service
systemctl --user daemon-reload
systemctl --user enable net.service
systemctl --user start net.service
```
docs/archives/2025-12/06_network/06_02_net_ddns.md
+387
View File
@@ -0,0 +1,387 @@
Tags: #os, #configuration, #network
## DDNS
### Secret management
- File:
- ~/data/config/secrets/.secret.yaml
- /etc/secrets/2001/ddns.env
- Edit `.secret.yaml` with `edit_secret.sh`
```yaml
# ~/data/config/secrets/.secret.yaml
# DDNS
DDNS:
ZONE_ID: 'encrypted value'
API_KEY: 'encrypted value'
```
```bash
extract_secret.sh ~/data/config/secrets/.secret.yaml -e "DDNS" > /run/user/$UID/tmp.env && sudo mv /run/user/$UID/tmp.env /etc/secrets/$UID/ddns.env && sudo chown $UID:root /etc/secrets/$UID/ddns.env && sudo chmod 400 /etc/secrets/$UID/ddns.env
```
### ddns.sh
- File: ~/data/config/scripts/ddns/ddns.sh
```bash
#!/bin/bash
# ~/data/config/scripts/ddns.sh
# Designate directory
# DIRECTORY="$HOME/data/config/scripts"
# Information
DOMAIN=""
TTL=180
C_TTL=86400
PROXIED="false"
DELETE_FLAG="false"
CURRENT_IP=""
# These will be injected by systemd
# ZONE_ID='.secret'
# API_KEY='.secret'
# usage() function
usage() {
echo "Usage: $0 -d \"domain\" [-t \"ttl\"] [-p] [-r] [-c]"
echo "-d <domain>: Specify the domain to update"
echo "-t <ttl>: Specify the TTL(Time to live)"
echo "-p: Specify the cloudflare proxy to use"
echo "-r: Delete the DNS record"
exit 1
}
# getopts to get arguments
while getopts "d:t:pr" opt; do
case $opt in
d)
DOMAIN="$OPTARG"
;;
t)
TTL="$OPTARG"
;;
p)
PROXIED="true"
;;
r)
DELETE_FLAG="true"
;;
\?) # unknown options
echo "Invalid option: -$OPTARG" >&2
usage
;;
:) # parameter required option
echo "Option -$OPTARG requires an argument." >&2
usage
;;
esac
done
# Get option and move to parameters - This has no functional thing, because it only use arguments with parameters
shift $((OPTIND - 1))
# Check necessary options
if [ -z "$DOMAIN" ]; then
echo "Error: -d option is required" >&2
usage
fi
if ! [[ "$TTL" =~ ^[0-9]+$ ]] || [ "$TTL" -le 0 ]; then
echo "Error: -t option (ttl) requires a number above 0." >&2
usage
fi
# log() function
log()
{
local text="$1"
echo -e "$(date "+%Y-%m-%d %H:%M:%S"): [ddns] $text"
}
# Make log directory
# if [ ! -d "$DIRECTORY/log" ]; then
# mkdir "$DIRECTORY/log"
# fi
# Check and create log file
# LOG_FILE="$DIRECTORY/log/ddns_$(date "+%Y-%m-%d").log"
# if [ ! -f "$LOG_FILE" ]; then
# log "Notice: log file is created"
# fi
# Check package
if ! command -v curl &> /dev/null; then
log "Error: curl package is needed"
exit
fi
if ! command -v jq &> /dev/null; then
log "Error: jq package is needed"
exit
fi
# API options
URL="https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records"
CONTENT_TYPE="Content-Type: application/json"
AUTHORIZATION="Authorization: Bearer $API_KEY"
# Current IP check
CURRENT_IP=$(curl -sf "https://ifconfig.me") ||\
CURRENT_IP=$(curl -sf "https://ifconfig.kr") ||\
CURRENT_IP=$(curl -sf "https://api.ipify.org")
if [ "$CURRENT_IP" == "" ]; then
log "Error: Can't get an IP"
exit
fi
# DNS functions
# get_dns_record() function
get_dns_record()
{
local type="$1"
local name="$2"
local response="$(
curl -s "$URL?type=$type&name=$name"\
-H "$CONTENT_TYPE"\
-H "$AUTHORIZATION")"
if [ "$(echo "$response" | jq -r '.success')" == "false" ]; then
log "Error: Can't get dns record\"Reason: $response"
exit
else
echo "$response"
fi
}
# create_dns_record() function
create_dns_record()
{
local type="$1"
local name="$2"
local ttl="$3"
local comment="$4"
local content="$5"
local response="$(
curl -s "$URL"\
-X POST\
-H "$CONTENT_TYPE"\
-H "$AUTHORIZATION"\
-d "{
\"name\": \"$name\",
\"ttl\": $ttl,
\"type\": \"$type\",
\"comment\": \"$comment\",
\"content\": \"$content\",
\"proxied\": $PROXIED
}")"
if [ "$(echo "$response" | jq -r '.success')" == "false" ]; then
log "Error: Can't create dns record\"Reason: $response"
exit
else
echo "$response"
fi
}
# update_dns_record() function
update_dns_record()
{
local type="$1"
local name="$2"
local ttl="$3"
local comment="$4"
local content="$5"
local id="$6"
local response=$(
curl -s "$URL/$id"\
-X PUT\
-H "$CONTENT_TYPE"\
-H "$AUTHORIZATION"\
-d "{
\"name\": \"$name\",
\"ttl\": $ttl,
\"type\": \"$type\",
\"comment\": \"$comment\",
\"content\": \"$content\",
\"proxied\": $PROXIED
}")
if [ "$(echo "$response" | jq -r '.success')" == "false" ]; then
log "Error: Can't update dns record\"Reason: $response"
exit
else
echo "$response"
fi
}
# delete_dns_record() function
delete_dns_record()
{
local type="$1"
local id="$2"
local response=$(
curl -s "$URL/$id"\
-X DELETE\
-H "$CONTENT_TYPE"\
-H "$AUTHORIZATION"
)
if [ "$(echo "$response" | jq -r '.success')" == "false" ]; then
log "Error: Can't delete dns record\"Reason: $response"
exit
else
echo "$response"
fi
}
# Get DNS A, and CNAME record
A_DNS_RECORD=$(get_dns_record "A" "$DOMAIN")
S_DNS_RECORD=$(get_dns_record "cname" "*.$DOMAIN")
W_DNS_RECORD=$(get_dns_record "cname" "www.$DOMAIN")
# Delete DNS record with Delete flag
if [ "$DELETE_FLAG" == "true" ]; then
FLAG="false"
if [ "$(echo $A_DNS_RECORD | jq -r '.result | length')" -eq 1 ]; then
A_DNS_ID="$(echo $A_DNS_RECORD | jq -r '.result[0].id')"
delete_dns_record "A" "$A_DNS_ID"
log "Delete: root DNS record is deleted"
FLAG="true"
fi
if [ "$(echo $S_DNS_RECORD | jq -r '.result | length')" -eq 1 ]; then
S_DNS_ID="$(echo $S_DNS_RECORD | jq -r '.result[0].id')"
delete_dns_record "cname" "$S_DNS_ID"
log "Delete: sub DNS record is deleted"
FLAG="true"
fi
if [ "$(echo $W_DNS_RECORD | jq -r '.result | length')" -eq 1 ]; then
W_DNS_ID="$(echo $W_DNS_RECORD | jq -r '.result[0].id')"
delete_dns_record "cname" "$W_DNS_ID"
log "Delete: www DNS record is deleted"
FLAG="true"
fi
if [ "$FLAG" == "false" ]; then
log "Notice: Nothing is Deleted. There are no DNS records"
fi
exit
fi
# Create or update DNS A record
if [ "$(echo $A_DNS_RECORD | jq -r '.result | length')" -eq 1 ]; then # root DNS record exist
A_DNS_ID="$(echo $A_DNS_RECORD | jq -r '.result[0].id')"
A_DNS_CONTENT="$(echo $A_DNS_RECORD | jq -r '.result[0].content')"
A_DNS_TTL="$(echo $A_DNS_RECORD | jq -r '.result[0].ttl')"
A_DNS_PROXIED="$(echo $A_DNS_RECORD | jq -r '.result[0].proxied')"
if [ "$A_DNS_CONTENT" != $CURRENT_IP -o "$A_DNS_TTL" != "$TTL" -o "$A_DNS_PROXIED" != "$PROXIED" ]; then
update_dns_record "A" "$DOMAIN" "$TTL" "$(date "+%Y-%m-%d %H:%M:%S"): root domain from ddns.sh" "$CURRENT_IP" "$A_DNS_ID"
log "Update: Root DNS record is successfully changed\nDomain: $DOMAIN\nIP: $A_DNS_CONTENT to $CURRENT_IP\nTTL: $A_DNS_TTL to $TTL\nproxied: $A_DNS_PROXIED to $PROXIED"
else
log "Notice: Root DNS record is not changed\nDomain: $DOMAIN\nIP: $CURRENT_IP\nTTL: $TTL\nproxied: $PROXIED"
fi
else # root DNS record does not exist
create_dns_record "A" "$DOMAIN" "$TTL" "$(date "+%Y-%m-%d %H:%M:%S"): root domain from ddns.sh" "$CURRENT_IP"
log "Create: Root DNS record is successfully created\nDomain: $DOMAIN\nIP: $CURRENT_IP\nTTL: $TTL\nproxied: $PROXIED"
fi
# Create or update DNS CNAME records
if [ "$(echo $S_DNS_RECORD | jq -r '.result | length')" -eq 1 ]; then # sub DNS record exist
S_DNS_ID="$(echo $S_DNS_RECORD | jq -r '.result[0].id')"
S_DNS_CONTENT="$(echo $S_DNS_RECORD | jq -r '.result[0].content')"
S_DNS_TTL="$(echo $S_DNS_RECORD | jq -r '.result[0].ttl')"
S_DNS_PROXIED="$(echo $S_DNS_RECORD | jq -r '.result[0].proxied')"
if [ "$S_DNS_CONTENT" != "$DOMAIN" -o "$S_DNS_TTL" != "$C_TTL" -o "$S_DNS_PROXIED" != "$PROXIED" ]; then
update_dns_record "cname" "*.$DOMAIN" "$C_TTL" "$(date "+%Y-%m-%d %H:%M:%S"): sub domain from ddns.sh" "$DOMAIN" "$S_DNS_ID"
log "Update: Sub DNS record is successfully changed\nDomain: $S_DNS_CONTENT to *.$DOMAIN\ncname: $DOMAIN \nTTL: $S_DNS_TTL to $C_TTL\nproxied: $S_DNS_PROXIED to $PROXIED"
else
log "Notice: Sub DNS record is not changed\nDomain: *.$DOMAIN\ncname: $DOMAIN\nTTL: $C_TTL\nproxied: $PROXIED"
fi
else # sub DNS record does not exist
create_dns_record "cname" "*.$DOMAIN" "$C_TTL" "$(date "+%Y-%m-%d %H:%M:%S"): sub domain from ddns.sh" "$DOMAIN"
log "Create: Sub DNS record is successfully created\nDomain: *.$DOMAIN\ncname: $DOMAIN\nTTL: $C_TTL\nproxied: $PROXIED"
fi
if [ "$(echo $W_DNS_RECORD | jq -r '.result | length')" -eq 1 ]; then # www DNS record exist
W_DNS_ID="$(echo $W_DNS_RECORD | jq -r '.result[0].id')"
W_DNS_CONTENT="$(echo $W_DNS_RECORD | jq -r '.result[0].content')"
W_DNS_TTL="$(echo $W_DNS_RECORD | jq -r '.result[0].ttl')"
W_DNS_PROXIED="$(echo $W_DNS_RECORD | jq -r '.result[0].proxied')"
if [ "$W_DNS_CONTENT" != "$DOMAIN" -o "$W_DNS_TTL" != "$C_TTL" -o "$W_DNS_PROXIED" != "$PROXIED" ]; then
update_dns_record "cname" "www.$DOMAIN" "$C_TTL" "$(date "+%Y-%m-%d %H:%M:%S"): www domain from ddns.sh" "$DOMAIN" "$W_DNS_ID"
log "Update: www DNS record is successfully changed\nDomain: $W_DNS_CONTENT to www.$DOMAIN\ncname: $DOMAIN\nTTL: $W_DNS_TTL to $C_TTL\nproxied: $W_DNS_PROXIED to $PROXIED"
else
log "Notice: www DNS record is not changed\nDomain: www.$DOMAIN\ncname: $DOMAIN\nTTL: $C_TTL\nproxied: $PROXIED"
fi
else # www DNS record does not exist
create_dns_record "cname" "www.$DOMAIN" "$C_TTL" "$(date "+%Y-%m-%d %H:%M:%S"): www domain from ddns.sh" "$DOMAIN"
log "Create: www DNS record is successfully created\nDomain: www.$DOMAIN\ncname: $DOMAIN\nTTL: $C_TTL\nproxied: $PROXIED"
fi
# Remove old backup file (7days before)
# find "$DIRECTORY/log" -maxdepth 1 -type f -mtime +7 -delete
```
### Systemd
- File:
- ~/data/config/services/ddns/ddns.service
- ~/data/config/services/ddns/ddns.timer
- /etc/secrets/2001/ddns.env
```ini
# ~/data/config/services/ddns/ddns.service
# ~/.config/systemd/user/ddns.service
[Unit]
Description=DDNS Update Service
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
# logging
StandardOutput=journal
StandardError=journal
# EnvironmentFile
EnvironmentFile=/etc/secrets/%U/ddns.env
# Run the script
ExecStart=/bin/bash -c '%h/data/config/scripts/ddns/ddns.sh -d "ilnmors.com"'
```
```ini
# ~/data/config/services/ddns/ddns.timer
# ~/.config/systemd/user/ddns.timer
[Unit]
Description=Run DDNS update service every 5 minutes
[Timer]
# Execute service after 1 min on booting
OnBootSec=1min
# Execute service every 5mins
OnUnitActiveSec=5min
# When timer is activated, Service also starts.
Persistent=true
[Install]
WantedBy=timers.target
```
```bash
# Register service
mkdir -p ~/.config/systemd/user && chmod -R 700 ~/.config
ln -s ~/data/config/services/ddns/ddns.service ~/.config/systemd/user/ddns.service
ln -s ~/data/config/services/ddns/ddns.timer ~/.config/systemd/user/ddns.timer
systemctl --user daemon-reload
# Start timer and enable
systemctl --user enable --now ddns.timer
```
docs/archives/2025-12/06_network/06_03_net_bind.md
+338
View File
@@ -0,0 +1,338 @@
Tags: #os, #configuration, #network, #virtualization, #container
## BIND9
BIND9 is an open source authoritative DNS software of ISC. It can work as a personal authoritative in private network environment.
### Secret management
- File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
#### Generate TSIG key as ACME key
```bash
podman run --rm --entrypoint /bin/sh internetsystemsconsortium/bind9:9.20 -c "tsig-keygen -a hmac-sha256 acme-key"
# Paste and add on .secret.yaml
# key "acme-key" {
#     algorithm hmac-sha256;
#     secret "your_base64_tsig_secret_here";
# };'
```
```yaml
# ~/data/config/secrets/.secret.yaml
BIND9_ACME_KEY: |
key "acme-key" {
    algorithm hmac-sha256;
    secret "your_base64_tsig_secret_here";
};
```
```bash
# Copy the secret value from .secret.yaml
# Podman secret
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "BIND9_ACME_KEY" | podman secret create "acme-key" -
```
### Preparation
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/bind
chmod 700 ~/data/containers/bind
sudo setfacl -m d:g::0 ~/data/containers/bind
sudo setfacl -m d:o::0 ~/data/containers/bind
sudo setfacl -m u:net:rwx ~/data/containers/bind
sudo setfacl -m u:100052:rwx ~/data/containers/bind
sudo setfacl -d -m u:net:rwx ~/data/containers/bind
sudo setfacl -d -m u:100052:rwx ~/data/containers/bind
mkdir -p ~/data/containers/bind/{cache,etc,lib,log}
nano ~/data/containers/bind/{etc,lib}/configuration_files
```
> BIND9 container executes as 53:53(bind:bind) permission in container. It is mapped host's 100052. Therefore, directories have to have ACL via `setfacl`
### Podman Image
```bash
podman pull internetsystemsconsortium/bind9:9.20 # Do not use latest version to management
```
### Configuration files
> If `named.conf` file didn't exist in `/etc/bind` or it had error in it, the container would be terminated without any logs. Before starting BIND container, you should makes all configuration file already.
#### named.conf
- file: ~/data/containers/bind/etc/named.conf
```ini
include "/run/secrets/keys/acme-key";
include "/run/secrets/keys/ddns-key";
options {
directory "/var/cache/bind";
listen-on { any; };
listen-on-v6 { ::1; };
// Authoritative DNS setting
allow-recursion { none; };
allow-transfer { none; };
allow-update { none; };
dnssec-validation no;
check-names master warn;
};
zone "ilnmors.internal." {
type primary;
file "/var/lib/bind/db.ilnmors.internal";
notify yes;
// ACME-01 challenge policy. It allows only subdomain's TXT record update.
update-policy {
grant acme-key subdomain ilnmors.internal. TXT;
grant ddns-key subdomain ilnmors.internal. A AAAA DHCID;
};
};
zone "1.168.192.in-addr.arpa" {
type primary;
file "/var/lib/bind/db.1.168.192.in-addr.arpa";
notify yes;
update-policy {
grant ddns-key subdomain 1.168.192.in-addr.arpa PTR DHCID;
};
};
zone "10.168.192.in-addr.arpa" {
type primary;
file "/var/lib/bind/db.10.168.192.in-addr.arpa";
notify yes;
update-policy {
grant ddns-key subdomain 10.168.192.in-addr.arpa PTR DHCID;
};
};
zone "ilnmors.com." {
//split horizon dns
type primary;
file "/var/lib/bind/db.ilnmors.com";
notify yes;
};
logging {
channel default_log {
stderr;
severity info;
};
category default { default_log; };
category config { default_log; };
category queries { default_log; };
};
```
>If ddns function were required, generate tsig key for ddns and give update policy for each zone like below.
>
> - include "/etc/bind/ddns-key";
> - update-policy { grant ddns-key subdomain \[zone_domain\] ANY A AAAA TXT; };
- Verify the named.conf with the command `named-checkconf`
#### Zone files
> When you add the record, you should use `.` at the end of the domain. i.e. mydomain.internal.
- file:
- ~/data/containers/bind/lib/db.ilnmors.internal
```ini
$TTL 86400
@ IN SOA bind.ilnmors.internal. mail.ilnmors.internal. (
2025113001 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
IN NS bind.ilnmors.internal.
bind IN A 192.168.10.11
opnsense IN A 192.168.10.1
vmm IN A 192.168.10.10
net IN A 192.168.10.11
crowdsec IN CNAME opnsense.ilnmors.internal.
adguard IN CNAME net.ilnmors.internal.
step-ca IN CNAME auth.ilnmors.internal.
caddy IN CNAME auth.ilnmors.internal.
ldap IN CNAME auth.ilnmors.internal.
authelia IN CNAME auth.ilnmors.internal.
code-server IN CNAME dev.ilnmors.internal.
postgresql IN CNAME dev.ilnmors.internal.
```
- ~/data/containers/bind/lib/db.1.168.192.in-addr.arpa
```ini
$TTL 86400
@ IN SOA bind.ilnmors.internal. mail.ilnmors.internal. (
yyyymmdd01 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
IN NS bind.ilnmors.internal.
1 IN PTR opnsense.ilnmors.internal.
11 IN PTR console.ilnmors.internal.
30 IN PTR printer.ilnmors.internal.
```
- ~/data/containers/bind/lib/db.10.168.192.in-addr.arpa
```ini
$TTL 86400
@ IN SOA bind.ilnmors.internal. mail.ilnmors.internal. (
yyyymmdd01 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
IN NS bind.ilnmors.internal.
1 IN PTR opnsense.ilnmors.internal.
10 IN PTR vmm.ilnmors.internal.
11 IN PTR net.ilnmors.internal.
12 IN PTR auth.ilnmors.internal.
13 IN PTR dev.ilnmors.internal.
14 IN PTR app.ilnmors.internal.
```
- ~/data/containers/bind/lib/db.ilnmors.com
```ini
$TTL 86400
@ IN SOA bind.ilnmors.internal. mail.ilnmors.internal. (
yyyymmdd01 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
IN NS bind.ilnmors.internal.
* IN A 192.168.10.12
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
### Quadlet
- File:
- ~/data/config/containers/bind/bind.container
```ini
# ~/data/config/containers/bind/bind.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=BIND9 DNS
After=network-online.target
Wants=network-online.target
[Container]
Image=docker.io/internetsystemsconsortium/bind9:9.20
ContainerName=bind
PublishPort=2253:53/tcp
PublishPort=2253:53/udp
Volume=%h/data/containers/bind/etc:/etc/bind:rw
Volume=%h/data/containers/bind/lib:/var/lib/bind:rw
Volume=%h/data/containers/bind/cache:/var/cache/bind:rw
Volume=%h/data/containers/bind/log:/var/log:rw
Environment="TZ=Asia/Seoul"
Secret=acme-key,target=/run/secrets/key/acme-key
Secret=ddns-key,target=/run/secrets/key/ddns-key
Label=diun.enable=true
Label=diun.watch_repo=true
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
# linger has to be activated
mkdir -p ~/.config/containers/systemd
ln -s ~/data/config/containers/bind/bind.container ~/.config/containers/systemd/bind.container
# This command makes bind.service
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start bind.service
```
### nsupdate (RFC 2136) verification
```bash
# Update query at net server
# Before test, create temp file .bind.acme-key
nsupdate -k /run/user/$UID/.bind.acme-key
> server 127.0.0.1 2253
> zone ilnmors.internal
> update add _acme-challenge.ilnmors.internal. 60 TXT "validation-test"
> send
> `ctrl+c`
# Verify
dig @127.0.0.1 -p 2253 _acme-challenge.ilnmors.internal. TXT
# Print
;; QUESTION SECTION:
;_acme-challenge.ilnmors.internal. IN TXT
;; ANSWER SECTION:
_acme-challenge.ilnmors.internal. 60 IN TXT "validation-test"
# Delete query at net server
nsupdate -k /run/user/$UID/.bind.acme-key
> server 127.0.0.1 2253
> zone ilnmors.internal
> update delete _acme-challenge.ilnmors.internal. 60 TXT "validation-test"
> send
> `ctrl+c`
# Verify
dig @127.0.0.1 -p 2253 _acme-challenge.ilnmors.internal. TXT
# Print
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 17572
```
docs/archives/2025-12/06_network/06_04_net_adguard_home.md
+293
View File
@@ -0,0 +1,293 @@
Tags: #os, #configuration, #network, #virtualization, #container
## AdGuard Home
AdGuard Home is one of open sourced recursive DNS resolver which can block malicious domain. It supports powerful DNS query filter based on ruleset and split horizon DNS service, plus recursive query on DoH and DoT towards public or internal authoritative DNS server.
### Secret
Adgaurd Home container doesn't need any secret value.
### Preparation
#### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```ini
# ...
adguard IN CNAME net.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/adguard
chmod 700 ~/data/containers/adguard
sudo setfacl -m d:g::0 ~/data/containers/adguard
sudo setfacl -m d:o::0 ~/data/containers/adguard
mkdir -p ~/data/containers/adguard/{work,config,certs}
```
> AdGuard Home container executes as root permission in container. It is not necessary to give `setfacl`. Because the container's root account is mapped as host's uid.
### Podman Image
```bash
podman pull adguard/adguardhome:v0.107.68 # Do not use latest version to management
```
### Quadlet
- File:
- ~/data/config/containers/adguard/adguard.container
```ini
# ~/data/config/containers/adguard/adguard.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=AdGuard Home DNS
After=bind.service
Requires=bind.service
[Container]
Image=docker.io/adguard/adguardhome:v0.107.68
ContainerName=adguard
AddHost=bind.ilnmors.internal:host-gateway
PublishPort=2053:53/tcp
PublishPort=2053:53/udp
PublishPort=2443:443/tcp
PublishPort=2443:443/udp
PublishPort=3000:3000/tcp
# 3000 is temporary port, After TLS setting, delete it.
Volume=%h/data/containers/adguard/work:/opt/adguardhome/work:rw
Volume=%h/data/containers/adguard/config:/opt/adguardhome/conf:rw
Volume=%h/data/containers/adguard/certs:/etc/ssl/adguard:ro
Environment="TZ=Asia/Seoul"
Label=diun.enable=true
Label=diun.watch_repo=true
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
ln -s ~/data/config/containers/adguard/adguard.container ~/.config/containers/systemd/adguard.container
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start adguard.service
```
### Web UI configuration
#### Access web UI and initial setting
- URL: http://192.168.10.11:3000
> After TLS certificate setting, access to adguard via https `https://192.168.10.11` or `https://adguard.ilnmors.internal`
#### Initial setting wizard
- Administrator interface
- Network interface: all
- Interface port: 3000
- DNS server
- Network interface: all
- Interface port: 53
- Authentication
- User: adguard
- Password: Password
### AdGuard Configuration
#### General setting
- Filter update interval: 1 hour
- \[\*\] Use AdGuard browsing security web service
- \[\*\] Enable log
- Query logs rotation - 90 days
- \[\*\] Enable statistics
- `Save`
#### DNS settings
- Upstream DNS servers
```text
# Cloudflare DNS
tls://security.cloudflare-dns.com
# Internal DNS bind.ilnmors.internal:2253
[/ilnmors.internal/]udp://bind.ilnmors.internal:2253
# Split Horizon DNS
[/*.ilnmors.com/]udp://bind.ilnmors.internal:2253
```
> Internal authoritative DNS will be BIND. BIND will use 2253 port. `bind.ilnmors.internal` is defined on container's `/etc/hosts`, it allows communication with host system.
- \[\*\] Parallel requests
- Private reverse DNS servers
```text
udp://bind.ilnmors.internal:2253
```
> Set this option after BIND prepared
- \[\*\] Use private reverse DNS resolvers
- \[\*\] Enable reverse resolving of client's IP addresses
- `Apply` and `Test upstreams`
- \[ \] Enable cache
- `Save`
- Disallowed domains
```text
...
wpad.ilnmors.internal
_ldap._tcp.dc._msdcs.ilnmors.internal
```
- `Save configuration`
#### Client settings
- Add Client
| Client | Name |
| :----------------------------------------------: | :-------: |
| 169.254.1.2, 192.168.10.11 | localhost |
| 10.10.10.2, 10.10.10.3, 10.10.10.4, 192.168.1.11 | console |
#### Filters - DNS blocklists
- Add blocklist - Choose from the list
- 1Hosts (Lite)
- AdGuard DNS filter
- AdAway DNS Popup Hosts filter
- HaGeZi's Ultimate Blocklist
- OISD Blocklist Big
- KOR: List-KR DNS
#### DNS rewrites
- Filters:DNS rewrites
- bind.ilnmors.internal - 192.168.10.11
#### Clients' DNS setting
##### OPNsense
- System:Settings:General
- DNS server: 192.168.10.11
- Services:KEA DHCP:KEA DHCPv4:Subnets
- DNS server: 192.168.10.11
##### Hypervisor
- /etc/resolv.conf
- nameserver 192.168.10.11
##### net
- /etc/resolv.conf
- nameserver 192.168.10.11
##### VPN client
```ini
[Interface]
DNS = 192.168.10.11
```
---
#### Encryption setting
> It requires `BIND` and `Step-CA`
- \[\*\] Enable Encryption (HTTPS, DNS-over_HTTPS, and DNS-over-TLS)
- \[\*\] Enable plain DNS
- \[\*\] Redirect to HTTPS automatically
- HTTPS port: 443
##### Certificates
> AdGuard home doesn't support ACME protocol by itself. It is necessary to use OPNsense's ACME client function to automate adguard home certificates.
- Add new domain in BIND, Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```text
...
adguard IN CNAME net.ilnmors.internal.
...
```
- ACME setting (OPNsense)
- Services:ACME Client:Certificates - Certificates - \[+\]
- Common Name: adguard.ilnmors.internal
- Description: adguard
- ACME Account: acme.ilnmors.internal
> Even though provisioner's name includes `@`, it has to use as `.`.
>
> i.e. `acme@ilnmors.internal` > `acme.ilnmors.internal`
- Challenge Type: ilnmors.internal-dns-01-challenge
- \[\*\] Auto Renewal
- Automations: adguard-auto-acme, adguard-auto-restart
- Automations (OPNsense)
- Services:ACME Client:Automations - Automation - \[+\]
- Name: adguard-auto-acme / adguard-auto-restart
- Description: adguard acme crt issue / restart adguard after crt is issued
- Run Command: Upload certificate via SFTP / Remote command via SSH
- SFTP Host: adguard.ilnmors.internal
- Username: net
- Identity Type: ed25519
- Remote Path(SFTP): /home/net/data/containers/adguard/certs
- Command(SSH): systemctl --user restart adguard
- `Show Identity`
> Copy Required parameters `ssh-ed25519 ~~~ root@opnsense.ilnmors.internal`
>
> Add parameters in net server's ~/.ssh/authorized_keys
- `Test Connect` and `Save`
- \[\*\] Set a certificates file path: `/etc/ssl/adguard/adguard.ilnmors.internal/fullchain.pem`
- \[\*\] Set a private key file: `/etc/ssl/adguard/adguard.ilnmors.internal/key.pem`
- \[Save configuration\]
##### Modify container file
- Delete `PublishPort=3000:3000/tcp` part
- `systemctl --user daemon-reload`
- `systemctl --user start adguard`
docs/archives/2025-12/06_network/06_05_net_kea.md
+630
View File
@@ -0,0 +1,630 @@
Tags: #os, #configuration, #network, #virtualization, #container
# KEA
#### Generate TSIG key as DDNS key
```bash
podman run --rm --entrypoint /bin/sh internetsystemsconsortium/bind9:9.20 -c "tsig-keygen -a hmac-sha256 acme-key"
# Paste and add on .secret.yaml
# key "acme-key" {
#     algorithm hmac-sha256;
#     secret "your_base64_tsig_secret_here";
# };'
```
### DHCPv4
#### Add user to group
```bash
sudo apt install kea-dhcp4-server kea-dhcp-ddns-server
sudo usermod -aG _kea net
sudo chmod 770 /etc/kea /var/lib/kea
sudo chmod 660 /etc/kea/kea-dhcp4.conf
sudo mkdir /etc/secrets/$(id -u _kea)
sudo chown _kea:root /etc/secrets/$(id -u _kea)
sudo chmod 500 /etc/secrets/$(id -u _kea)
sudo touch /etc/secrets/$(id -u _kea)/ddns-key
sudo nano /etc/secrets/$(id -u _kea)/ddns-key
sudo chown _kea:root /etc/secrets/$(id -u _kea)/ddns-key
sudo chmod 400 /etc/secrets/$(id -u _kea)/ddns-key
```
iptables
```
*mangle
*filter
-A INPUT -p udp -m udp --dport 67 -m comment --comment "allow upd DHCPv4 connection" -j ACCEPT
# check
sudo bash -c 'iptables-restore --test < /etc/iptables/rules.v4'
```
kea-dhcp4.conf
```json
{
"Dhcp4": {
"subnet4": [
{
"subnet": "192.168.10.0/24",
"pools" : [
{
"pool": "192.168.10.254-192.168.10.254"
}
],
"option-data": [
{
"name": "routers",
"data": "192.168.10.1"
},
{
"name": "ntp-servers",
"data": "192.168.10.1"
},
{
"name": "domain-name-servers",
"data": "192.168.10.11"
},
{
"name": "domain-name",
"data": "ilnmors.internal"
}
],
"reservations": [
{
"hw-address": "0a:49:6e:4d:02:00",
"ip-address": "192.168.10.12",
"hostname": "auth"
},
{
"hw-address": "0a:49:6e:4d:03:00",
"ip-address": "192.168.10.13",
"hostname": "dev"
},
{
"hw-address": "0a:49:6e:4d:04:00",
"ip-address": "192.168.10.14",
"hostname": "app"
}
],
"id": 1,
"interface": "enp1s0"
},
{
"subnet": "192.168.1.0/24",
"relay": {
"ip-addresses": [ "192.168.1.1" ]
},
"pools": [
{
"pool": "192.168.1.100-192.168.1.254"
}
],
"option-data": [
{
"name": "routers",
"data": "192.168.1.1"
},
{
"name": "ntp-servers",
"data": "192.168.1.1"
},
{
"name": "domain-name-servers",
"data": "192.168.10.11"
},
{
"name": "domain-name",
"data": "ilnmors.internal"
}
],
"reservations": [
{
"hw-address": "d8:e2:df:ff:1b:d5",
"ip-address": "192.168.1.11",
"hostname": "surface"
},
{
"hw-address": "38:ca:84:94:5e:06",
"ip-address": "192.168.1.30",
"hostname": "printer"
}
],
"id": 2,
"interface": "enp1s0"
}
],
"interfaces-config": {
"interfaces": [
"enp1s0"
],
"dhcp-socket-type": "raw",
"service-sockets-max-retries": 5,
"service-sockets-require-all": true
},
"renew-timer": 1000,
"rebind-timer": 2000,
"valid-lifetime": 4000,
"loggers": [
{
"name": "kea-dhcp4",
"output_options": [
{
"output": "stdout"
}
],
"severity": "INFO"
}
],
"lease-database": {
"type": "memfile",
"persist": true,
"name": "/var/lib/kea/kea-leases4.csv",
"lfc-interval": 3600
},
"dhcp-ddns": {
"enable-updates": false,
"server-ip": "127.0.0.1",
"server-port": 53001
"ncr-protocol": "UDP"
"ncr-format": "JSON"
},
"ddns-send-updates": true,
"ddns-update-on-renew": true,
"ddns-qualifying-suffix": "ilnmors.internal",
"ddns-override-no-update": true,
"ddns-override-client-update": true,
"ddns-replace-client-name": "when-present",
"ddns-generated-prefix": "host"
"hostname-char-set": "[^a-zA-Z0-9.-]",
"hostname-char-replacement": "-"
}
}
// There is one interface for DHCP host. In this case, the subnet which has dhcp server should be `id=1`. Because Kea DHCP allocate IP from the `id=1` subnet when it recieve pacekt has no giaddr. When Kea DHCP is running on Router, then id is not important because DHCP allocate IP based on their interface.
```
#### OPNsense
- Services:Kea DHCP:Kea DHCPv4
- \[ \] Enabled
- Services:DHCP Relay:Configuration
- Destination - \[+\]
- Name: kea-dhcp-v4
- Server: 192.168.10.11
- Relays - \[+\]
- \[\*\] Enabled
- Interface: LAN
- Desitination kea-dhcp-v4
- \[ \]Agent Information
- Status `green box` check
### DDNS
```bash
sudo nano /etc/apparmor.d/local/usr.sbin.kea-dhcp-ddns
# /etc/secrets/** r,
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.kea-dhcp-ddns
sudo systemctl restart kea-dhcp-ddns-server.service
```
/etc/secrets/102/ddns-key
```json
[
{
"name": "ddns-key",
"algorithm": hmac-sha256;
"secret": "secret_value"
}
]
```
/etc/kea/kea-dhcp-ddns.conf
```json
{
"DhcpDdns":
{
"ip-address": "127.0.0.1",
"port": 53001,
"control-socket": {
"socket-type": "unix",
"socket-name": "/run/kea/kea-ddns-ctrl-socket"
},
"tsig-keys": <?include "/etc/secrets/102/ddns-key"?>,
"forward-ddns" : {
"ddns-domains": [
{
"name": "ilnmors.internal.",
"key-name": "ddns-key",
"dns-servers": [
{
"ip-address": "127.0.0.1",
"port": 2053
}
]
}
]
},
"reverse-ddns" : {
"ddns-domains": [
{
"name": "10.168.192.in-addr.arpa.",
"key-name": "ddns-key",
"dns-servers": [
{
"ip-address": "127.0.0.1",
"port": 2053
}
]
},
{
"name": "1.168.192.in-addr.arpa.",
"key-name": "ddns-key",
"dns-servers": [
{
"ip-address": "127.0.0.1",
"port": 2053
}
]
}
]
},
"loggers": [
{
"name": "kea-dhcp-ddns",
"output-options": [
{
"output": "stdout"
}
],
"severity": "INFO",
}
]
}
}
```
kea-dhcp4.conf
``` json
// ...
"dhcp-ddns": {
"enable-updates": true,
// ...
}
```
---
## BIND9
BIND9 is an open source authoritative DNS software of ISC. It can work as a personal authoritative in private network environment.
### Secret management
- File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
#### Generate TSIG key as ACME key
```bash
podman run --rm --entrypoint /bin/sh internetsystemsconsortium/bind9:9.20 -c "tsig-keygen -a hmac-sha256 acme-key"
# Paste and add on .secret.yaml
# key "acme-key" {
#     algorithm hmac-sha256;
#     secret "your_base64_tsig_secret_here";
# };'
```
```yaml
# ~/data/config/secrets/.secret.yaml
BIND9_ACME_KEY: |
key "acme-key" {
    algorithm hmac-sha256;
    secret "your_base64_tsig_secret_here";
};
```
```bash
# Copy the secret value from .secret.yaml
# Podman secret
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "BIND9_ACME_KEY" | podman secret create "acme-key" -
```
### Preparation
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/bind
chmod 700 ~/data/containers/bind
sudo setfacl -m d:g::0 ~/data/containers/bind
sudo setfacl -m d:o::0 ~/data/containers/bind
sudo setfacl -m u:net:rwx ~/data/containers/bind
sudo setfacl -m u:100052:rwx ~/data/containers/bind
sudo setfacl -d -m u:net:rwx ~/data/containers/bind
sudo setfacl -d -m u:100052:rwx ~/data/containers/bind
mkdir -p ~/data/containers/bind/{cache,etc,lib,log}
nano ~/data/containers/bind/{etc,lib}/configuration_files
```
> BIND9 container executes as 53:53(bind:bind) permission in container. It is mapped host's 100052. Therefore, directories have to have ACL via `setfacl`
### Podman Image
```bash
podman pull internetsystemsconsortium/bind9:9.20 # Do not use latest version to management
```
### Configuration files
> If `named.conf` file didn't exist in `/etc/bind` or it had error in it, the container would be terminated without any logs. Before starting BIND container, you should makes all configuration file already.
#### named.conf
- file: ~/data/containers/bind/etc/named.conf
```ini
include "/etc/bind/key/acme-key";
options {
directory "/var/cache/bind";
listen-on { any; };
listen-on-v6 { ::1; };
// Authoritative DNS setting
allow-recursion { none; };
allow-transfer { none; };
allow-update { none; };
dnssec-validation no;
check-names master warn;
};
zone "ilnmors.internal." {
type primary;
file "/var/lib/bind/db.ilnmors.internal";
notify yes;
// ACME-01 challenge policy. It allows only subdomain's TXT record update.
update-policy {
grant acme-key subdomain ilnmors.internal. TXT;
};
};
zone "1.168.192.in-addr.arpa" {
type primary;
file "/var/lib/bind/db.1.168.192.in-addr.arpa";
notify yes;
};
zone "10.168.192.in-addr.arpa" {
type primary;
file "/var/lib/bind/db.10.168.192.in-addr.arpa";
notify yes;
};
zone "ilnmors.com." {
//split horizon dns
type primary;
file "/var/lib/bind/db.ilnmors.com";
notify yes;
};
logging {
channel default_log {
stderr;
severity info;
};
category default { default_log; };
category config { default_log; };
category queries { default_log; };
};
```
>If ddns function were required, generate tsig key for ddns and give update policy for each zone like below.
>
> - include "/etc/bind/ddns-key";
> - update-policy { grant ddns-key subdomain \[zone_domain\] ANY A AAAA TXT; };
- Verify the named.conf with the command `named-checkconf`
#### Zone files
> When you add the record, you should use `.` at the end of the domain. i.e. mydomain.internal.
- file:
- ~/data/containers/bind/lib/db.ilnmors.internal
```ini
$TTL 86400
@ IN SOA bind.ilnmors.internal. mail.ilnmors.internal. (
yyyymmdd01 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
IN NS bind.ilnmors.internal.
bind IN A 192.168.10.11
opnsense IN A 192.168.10.1
vmm IN A 192.168.10.10
net IN A 192.168.10.11
auth IN A 192.168.10.12
dev IN A 192.168.10.13
app IN A 192.168.10.14
console IN A 192.168.1.11
printer IN A 192.168.1.30
crowdsec IN CNAME opnsense.ilnmors.internal.
adguard IN CNAME net.ilnmors.internal.
step-ca IN CNAME auth.ilnmors.internal.
caddy IN CNAME auth.ilnmors.internal.
ldap IN CNAME auth.ilnmors.internal.
postgresql IN CNAME dev.ilnmors.internal.
```
- ~/data/containers/bind/lib/db.1.168.192.in-addr.arpa
```ini
$TTL 86400
@ IN SOA bind.ilnmors.internal. mail.ilnmors.internal. (
yyyymmdd01 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
IN NS bind.ilnmors.internal.
1 IN PTR opnsense.ilnmors.internal.
11 IN PTR console.ilnmors.internal.
30 IN PTR printer.ilnmors.internal.
```
- ~/data/containers/bind/lib/db.10.168.192.in-addr.arpa
```ini
$TTL 86400
@ IN SOA bind.ilnmors.internal. mail.ilnmors.internal. (
yyyymmdd01 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
IN NS bind.ilnmors.internal.
1 IN PTR opnsense.ilnmors.internal.
10 IN PTR vmm.ilnmors.internal.
11 IN PTR net.ilnmors.internal.
12 IN PTR auth.ilnmors.internal.
13 IN PTR dev.ilnmors.internal.
14 IN PTR app.ilnmors.internal.
```
- ~/data/containers/bind/lib/db.ilnmors.com
```ini
$TTL 86400
@ IN SOA bind.ilnmors.internal. mail.ilnmors.internal. (
yyyymmdd01 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
IN NS bind.ilnmors.internal.
* IN A 192.168.10.12
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
### Quadlet
- File:
- ~/data/config/containers/bind/bind.container
```ini
# ~/data/config/containers/bind/bind.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=BIND9 DNS
After=network-online.target
Wants=network-online.target
[Container]
Image=docker.io/internetsystemsconsortium/bind9:9.20
ContainerName=bind
PublishPort=2253:53/tcp
PublishPort=2253:53/udp
Volume=%h/data/containers/bind/etc:/etc/bind:rw
Volume=%h/data/containers/bind/lib:/var/lib/bind:rw
Volume=%h/data/containers/bind/cache:/var/cache/bind:rw
Volume=%h/data/containers/bind/log:/var/log:rw
Environment="TZ=Asia/Seoul"
Secret=acme-key,target=/etc/bind/key/acme-key
Label=diun.enable=true
Label=diun.watch_repo=true
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
# linger has to be activated
mkdir -p ~/.config/containers/systemd
ln -s ~/data/config/containers/bind/bind.container ~/.config/containers/systemd/bind.container
# This command makes bind.service
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start bind.service
```
### nsupdate (RFC 2136) verification
```bash
# Update query at net server
# Before test, create temp file .bind.acme-key
nsupdate -k /run/user/$UID/.bind.acme-key
> server 127.0.0.1 2253
> zone ilnmors.internal
> update add _acme-challenge.ilnmors.internal. 60 TXT "validation-test"
> send
> `ctrl+c`
# Verify
dig @127.0.0.1 -p 2253 _acme-challenge.ilnmors.internal. TXT
# Print
;; QUESTION SECTION:
;_acme-challenge.ilnmors.internal. IN TXT
;; ANSWER SECTION:
_acme-challenge.ilnmors.internal. 60 IN TXT "validation-test"
# Delete query at net server
nsupdate -k /run/user/$UID/.bind.acme-key
> server 127.0.0.1 2253
> zone ilnmors.internal
> update delete _acme-challenge.ilnmors.internal. 60 TXT "validation-test"
> send
> `ctrl+c`
# Verify
dig @127.0.0.1 -p 2253 _acme-challenge.ilnmors.internal. TXT
# Print
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 17572
```
docs/archives/2025-12/07_authorization/07_01_auth_vm.md
+148
View File
@@ -0,0 +1,148 @@
Tags: #os, #configuration, #network, #virtualization, #authorization, #authentication
## Preparation
### Set DHCP reservation and DNS record
#### Set DHCP reservation on KEA DHCP in OPNsense
Following [here](05_07_opnsense_kea.md)
- Services:Kea DHCP:Kea DHCPv4:Reservations - \[+\]
- Subnet: 192.168.10.0/24
- IP address: 192.168.10.12
- MAC address: 0A:49:6E:4D:02:00
- Hostname: auth
- Description: auth
- `save`
#### Set DNS records in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file:
- ~/data/containers/bind/lib/db.ilnmors.internal
- ~/data/containers/bind/lib/db.10.168.192.in-addr.arpa
```ini
# db.ilnmors.internal
# ...
auth IN A 192.168.10.12
# ...
# db.10.168.192.in-addr.arpa
# ...
12 IN PTR auth.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
### Create VM template
- ~/data/config/scripts/auth.sh
```bash
virt-install \
--boot uefi \
--name auth \
--os-variant debian13 \
--vcpus 2 \
--memory 4096 \
--location /var/lib/libvirt/images/debian-13.0.0-amd64-netinst.iso \ # For serial installing, use `--location` instead of `--cdrom`
--disk pool=vm-images,size=66,format=qcow2,discard=unmap \
--network network=ovs-lan-net,portgroup=vlan10-access,model=virtio,mac=0A:49:6E:4D:02:00 \ # Use designated ovs port group
--graphics none \
--console pty,target_type=serial \
--extra-args "console=ttyS0,115200"
# After enter this command, then the console start automatically
# Remove all annotation before you make the sh file.
```
### Debian installing
- Following [here](../03_common/03_01_debian_configuration.md) to install Debian.
- Debian installer supports serial mode regardless getty@ttyS0 service is enabled or not.
- Following [here](../03_common/03_02_iptables.md) to set iptables.
- Following [here](../03_common/03_04_crowdsec.md) to set CrowdSec
#### Serial console setting
After installation, use `ctrl + ]` to exit console. Before setting getty@ttyS0, you can't use serial console to access VM. Therefore, use IP address set on installation, and connect net server via ssh first, following the step to enable the getty.
### Modify VM template settings
After getty setting, shutdown auth vm with `shutdown` in VM or `sudo virsh shutdown auth` in hypervisor to turn off vm first.
```bash
virsh edit auth
```
```xml
<!-- auth -->
...
</vcpu>
<cputune>
<shares>1024</shares>
</cputune>
<!-- cpu priority - 1024: default/2048: high/512: low -->
<!--
<disk type='file' device='cdrom'>
...
</disk>
# Remove booting disk
-->
```
```bash
virsh dumpxml auth > ~/data/config/vms/dumps/auth.xml
virsh start auth && virsh console auth
# Start auth server with console
```
### Common setting
- auth.service
```ini
# ~/data/config/services/auth.service
# ~/.config/systemd/user/auth.service
[Unit]
Description=auth Auto Booting
After=network-online.target
Wants=network-online.target
Requires=opnsense.service
[Service]
Type=oneshot
# Maintain status as active
RemainAfterExit=yes
# CrowdSec should be set
ExecStartPre=%h/data/config/scripts/wait-for-it.sh 192.168.10.1:8080 -t 0
ExecStartPre=%h/data/config/scripts/wait-for-it.sh 192.168.10.11:53 -t 0
ExecStartPre=/bin/bash -c "sleep 15"
# Run the service
ExecStart=/usr/bin/virsh -c qemu:///system start auth
# Stop the service
ExecStop=/usr/bin/virsh -c qemu:///system shutdown auth
[Install]
WantedBy=default.target
```
```bash
ln -s ~/data/config/services/auth.service ~/.config/systemd/user/auth.service
systemctl --user daemon-reload
systemctl --user enable auth.service
systemctl --user start auth.service
```
docs/archives/2025-12/07_authorization/07_02_auth_step-ca.md
+320
View File
@@ -0,0 +1,320 @@
Tags: #os, #configuration, #network, #virtualization, #container, #security
## Step-CA
Step-CA is the modern CA server which can operate in private network environment. It can issue CA and certificates, apply the policy with provisioner. It supports ACME, JWK, etc.
### Secret management
- File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
```yaml
# ~/data/config/secrets/.secret.yaml
# STEP_CA
STEP_CA_PASSWORD: generated_value
```
### Preparation
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/step-ca
chmod 700 ~/data/containers/step-ca
setfacl -m d:g::0 ~/data/containers/step-ca
setfacl -m d:o::0 ~/data/containers/step-ca
setfacl -m u:auth:rwx ~/data/containers/step-ca
setfacl -m u:100999:rwx ~/data/containers/step-ca
setfacl -d -m u:auth:rwx ~/data/containers/step-ca
setfacl -d -m u:100999:rwx ~/data/containers/step-ca
# After generating
sudo find ~/data/containers/step-ca -type f -exec setfacl -m m::rw {} \;
sudo find ~/data/containers/step-ca -type d -exec setfacl -m m::rwx {} \;
```
> Step-CA container executes as 1000:1000(step:step) permission in container. It is mapped host's 100999. Therefore, directories have to have ACL via `setfacl`
#### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: /home/net/data/containers/bind/lib/db.ilnmors.internal
```ini
# ...
step-ca IN CNAME auth.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
### Podman Image
```bash
podman pull smallstep/step-ca:0.28.4 # Do not use latest version to management
```
#### CA generation
```bash
podman run --rm -it \
-v /home/auth/data/containers/step-ca:/home/step:rw \
smallstep/step-ca:0.28.4 step ca init \
--deployment-type standalone \
--name ilnmors.internal \
--dns step-ca.ilnmors.internal \
--address :9000 \
--provisioner step-admin@ilnmors.internal
# Private mode: standalone
# Intermediate CA setting options
# --ra stepCAS \
# --issuer https://step-ca.ilnmors.internal:9000 \
# --issuer-fingerprint ~~~~ \
# --issuer-provisioner jwk-ca@dev.ilnmors.internal
# --confidential-file ~~~
> [leave empty and we\'ll generate one]: [blank]
# Print
---
✔ Password: Generated_value # Copy this value and paste in .secret.yaml file as STEP_CA_PASSWORD=
✔ Root fingerprint: fingerprint
---
```
> Password value encrypts root CA's private key
```bash
# Podman secret
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "STEP_CA_PASSWORD" | podman secret create "STEP_CA_PASSWORD" -
```
### Configuration files
- File: ~/data/containers/step-ca/config/ca.json
#### Provisioner
Provisioner is basically the object of issuing certificates as a RA. They verify CSR from client and when it is valid with its policy they will sign the certificates with CA's private key. Step-CA supports various type of provisioner. In this homelab, only ACME will be used. Because it is easy to manage when you use OPNsense ACME client. Step-CA supports one root CA and one intermediate CA in one container, only one intermediate CA will be operated in this project. However, the way to set multi intermediate CA will be explained, and jwk way in this document.
##### jwk-ca@ilnmors.internal
This provisioner is to issue intermediate CA. It wouldn't be used in this project. The option for CA in X.509 format is optional and defined in as extension option. To define these option in step-ca, the template file is needed.
- file: ~/data/containers/step-ca/templates/ca.tpl
```json
{
"subject": {{ toJson .Subject }},
"keyUsage": ["certSign", "crlSign"],
"basicConstraints": {
"isCA": true,
"maxPathLen": 0
}
}
```
> keyUsage: Designate to manage certificates and CRL
> isCA: Designate the certificate to use CA
> maxPathLen: Designate allowed below CA's number
- Define provisioner
```bash
podman exec -it step-ca \
step ca provisioner add jwk-ca@ilnmors.internal \
--create \ # Generate key pair automatically
--type JWK \
--ca-config /home/step/config/ca.json \ # Sign on certificate with root CA's private key
--x509-template /home/step/template/ca.tpl \ # Use x509 template
--x509-max-dur 87600h \ #
--x509-default-dur 87600h
```
##### jwk@ilnmors.internal
This provisioner is to issue the certificates like DB communication based on its identity (Using JWK and JWT pre-shared). The certificate is issued based on enrolled key in provisioner. However, in this project all crt will be used central ACME client `Opnsens ACME client` and `Caddy`.
- Define provisioner
```bash
podman exec -it step-ca \
step ca provisioner add jwk-crt@ilnmors.internal \
--create \ # Generate key pair automatically
--type JWK \
--x509-default-dur 2160h # To set default expire date as 90 days.
```
##### acme@ilnmors.internal
This provisioner is to issue the certificates for https communication. The certificate is issued based on challenge; the ownership of domain.
- Define provisioner
```bash
podman exec -it step-ca \
step ca provisioner add acme@ilnmors.internal \
--type ACME \
--x509-default-dur 2160h # To set default expire date as 90 days.
```
#### Subject
Step-CA uses subject as a account. It is used to manage Step-CA remotely. To use this, it is necessary to use `--remote-management` option when the step-CA is initially set or fix `ca.json` authority.enableAdmin:true. When subject is enabled, provisioners aren't defined in ca.json but its own DB.
#### Policy
Self-hosted Step-CA server doesn't support to give x509 policy for each provisioner. It only allows public policy. Only `ilnmors.internal` and `*.ilnmors.internal` certificates are required, so designate the policy in `ca.json`
> Policies can be administered using the step CLI application. The commands are part of the step ca policy namespace. In a self-hosted step-ca, policies can be configured on the authority level. Source: [here](https://smallstep.com/docs/step-ca/policies/)
- file: ~/data/containers/step-ca/config/ca.json
```json
...
"authority": {
"policy": {
"x509": {
"allow": {
"dns": [
"ilnmors.internal",
"*.ilnmors.internal"
]
},
"allowWildcardNames": true
}
},
"provisioners": [ ... ]
....
}
...
```
### Quadlet
- File:
- ~/data/config/containers/step-ca/step-ca.container
```ini
# ~/data/config/containers/step-ca/step-ca.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=Step-CA
After=network-online.target
Wants=network-online.target
[Container]
Image=docker.io/smallstep/step-ca:0.28.4
ContainerName=step-ca
PublishPort=9000:9000/tcp
Volume=%h/data/containers/step-ca:/home/step:rw
Environment="TZ=Asia/Seoul"
Environment="PWDPATH=/run/secrets/STEP_CA_PASSWORD"
Secret=STEP_CA_PASSWORD,target=/run/secrets/STEP_CA_PASSWORD
Label=diun.enable=true
Label=diun.watch_repo=true
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
mkdir -p ~/.config/containers/systemd
ln -s ~/data/config/containers/step-ca/step-ca.container ~/.config/containers/systemd/step-ca.container
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start step-ca.service
```
### Verify server
#### Server health check
```bash
curl -k https://step-ca.ilnmors.internal:9000/health
> {"status":"ok"}
```
#### Server policy check
```bash
podman exec -it step-ca step ca certificate test.com test.crt test_key --provisioner acme@ilnmors.internal
> error creating new ACME order: The server will not issue certificates for the identifier
```
---
### Set trust Root CRT
#### Linux
##### Debian/ubuntu
- File: /usr/local/share/ca-certificates/{ca.crt, ca.pem}
- `update-ca-certificates`
##### Cent/RHEL/Fedora
- File: /etc/pki/ca-trust/source/anchors/{ca.crt, ca.pem}
- `update-ca-trust`
#### Windows
- `Windows + R` + `certlm.msc`
- `All Task` - `Import`
#### Firefox
- Setting - Security - certificates - CA - add
---
### intermediate CA setting
It won't be used in this project, however here is the way to set.
#### Example of dev.ilnmors.internal intermediate CA
- init CA
```bash
podman run --rm -it \
-v /home/dev/data/containers/step-ca:/home/step:rw \
smallstep/step-ca:0.28.4 step ca init \
--deployment-type standalone \
--name dev.ilnmors.internal \
--dns step-ca.dev.ilnmors.internal \
--address :9000 \
--provisioner admin@dev.ilnmors.internal \
--ra stepCAS \
--issuer https://step-ca.ilnmors.internal:9000 \
--issuer-fingerprint ~~~~ \ # root CA's fingerprint
--issuer-provisioner jwk-ca@dev.ilnmors.internal
--confidential-file ~~~ # jwk-ca's password file
```
docs/archives/2025-12/07_authorization/07_03_auth_main_caddy.md
+231
View File
@@ -0,0 +1,231 @@
Tags: #os, #configuration, #network, #virtualization, #container, #security, #authentication, #authorization, #sso
## Caddy - auth
Caddy is an open source reverse proxy (web server) which supports automatically to apply TLS certificates via ACME protocol from CA. It supports various module including dns module. However, the most important and fundamental services such as OPNsense, AdGuard Home, Step-CA, Authelia would not use caddy for independency.
### Secret management
- File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
```yaml
# ~/data/config/secrets/.secret.yaml
# CADDY:
CADDY_ACME_KEY: acme-key_key_value (Only secret value)
CADDY_CROWDSEC_KEY: CADDY_LAPI_KEY
```
```bash
# Podman secret
extract_secret.sh .secret.yaml -f CADDY_ACME_KEY | podman secret create CADDY_ACME_KEY -
extract_secret.sh .secret.yaml -f CADDY_CROWDSEC_KEY | podman secret create CADDY_CROWDSEC_KEY -
```
### Preparation
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/caddy-auth/{etc,data}
chmod -R 700 ~/data/containers/caddy-auth
```
> Caddy container executes as 0:0(root:root) permission in container. It is mapped host's UID. Therefore, directories don't have to have ACL via `setfacl`
#### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```ini
# ...
login IN CNAME auth.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
### Podman Image
#### Podman containerfile
Caddy supports various module for it. rfc2136(nsupdate) module, crowdsec will be used in this homelab project.
- file:
- ~/data/config/containers/caddy-auth/containerfile-caddy-2.10.2-auth
- ~/data/config/containers/caddy-auth/root_ca.crt
```containerfile
FROM caddy:2.10.2-builder-alpine AS builder
RUN xcaddy build \
--with github.com/caddy-dns/rfc2136 \
--with github.com/hslatman/caddy-crowdsec-bouncer/crowdsec \
--with github.com/hslatman/caddy-crowdsec-bouncer/http
FROM caddy:2.10.2
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
COPY ./root_ca.crt /usr/local/share/ca-certificates/root_ca.crt
RUN update-ca-certificates
```
#### Podman image build
```bash
podman build -t caddy:2.10.2-auth -f ~/data/config/containers/caddy-auth/containerfile-caddy-2.10.2-auth . && podman image prune -f
# Delete pure caddy and caddy-builder-alpine images after command above manually.
```
### Configuration files
Caddyfile will be updated after Authelia setting
```bash
# fix inconsistencies
podman exec caddy-auth caddy fmt --overwrite /etc/caddy/Caddyfile
# After Caddyfile setting is changed use this command.
podman exec caddy-auth caddy reload --config /etc/caddy/Caddyfile
```
- file:
- ~/data/containers/caddy-auth/etc/Caddyfile
- ~/data/containers/caddy-auth/certs/root_ca.crt
```ini
# Caddyfile
# ~/data/containers/caddy-auth/etc/Caddyfile
# Global option
{
# CrowdSec LAPI connection
crowdsec {
api_url https://crowdsec.ilnmors.internal:8080
api_key "{file./run/secrets/CADDY_CROWDSEC_KEY}"
}
}
# Snippets
# CrowdSec log for parser
(crowdsec_log) {
log {
output file /data/access.log {
mode 0640
roll_size 100MiB
roll_keep 1
}
}
}
# Private TLS ACME with DNS-01-challenge
(private_tls) {
tls {
issuer acme {
dir https://step-ca.ilnmors.internal:9000/acme/acme@ilnmors.internal/directory
dns rfc2136 {
server bind.ilnmors.internal:2253
key_name acme-key
key_alg hmac-sha256
key "{file./run/secrets/CADDY_ACME_KEY}"
}
}
}
}
test.ilnmors.com {
import crowdsec_log
route {
crowdsec
root * /usr/share/caddy
file_server
}
}
caddy.ilnmors.internal {
import private_tls
import crowdsec_log
route {
crowdsec
root * /usr/share/caddy
file_server
}
}
```
### Quadlet
- File:
- ~/data/config/containers/caddy-auth/caddy-auth.container
```ini
# ~/data/config/containers/caddy-auth/caddy-auth.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=Caddy - auth
After=step-ca.service
Requires=step-ca.service
[Container]
Image=localhost/caddy:2.10.2-auth
ContainerName=caddy-auth
# To issue certificate from step-ca
AddHost=step-ca.ilnmors.internal:host-gateway
PublishPort=2080:80/tcp
PublishPort=2443:443/tcp
Volume=%h/data/containers/caddy-auth/etc:/etc/caddy:rw
Volume=%h/data/containers/caddy-auth/data:/data:rw
Environment="TZ=Asia/Seoul"
Secret=CADDY_ACME_KEY,target=/run/secrets/CADDY_ACME_KEY
Secret=CADDY_CROWDSEC_KEY,target=/run/secrets/CADDY_CROWDSEC_KEY
Label=diun.enable=true
Label=diun.watch_repo=true
# This label need configuration on `diun.yml`
Label=diun.regopt=caddy-auth-source
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
# linger has to be activated
ln -s ~/data/config/containers/caddy-auth/caddy-auth.container ~/.config/containers/systemd/caddy-auth.container
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start caddy-auth.service
```
### Crowdsec bouncer and agent
- Following [here](../03_common/03_04_crowdsec.md).
docs/archives/2025-12/07_authorization/07_04_auth_authentik.md
+315
View File
@@ -0,0 +1,315 @@
Tags: #os, #configuration, #network, #virtualization, #container, #security, #authentication, #authorization, #sso
## Authentik
Authentik is one of famous and strong open source idP solutions which support SSO, 2FA, LDAP, RADIUS, etc. It will be combining with Caddy-security module to apply SSO.
### Secret management
File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
```yaml
# ~/data/config/secrets/.secret.yaml
# Authentik:
AUTHENTIK_SECRET_KEY: openssl rand -base64 32 value
AUTHENTIK_POSTGRESQL__PASSWORD: openssl rand -base64 32 value
```
```bash
# Podman secret
extract_secret.sh .secret.yaml -f AUTHENTIK_SECRET_KEY | podman secret create AUTHENTIK_SECRET_KEY -
extract_secret.sh .secret.yaml -f AUTHENTIK_POSTGRESQL__PASSWORD | podman secret create AUTHENTIK_POSTGRESQL__PASSWORD -
```
### Preparation
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/authentik
chmod 700 ~/data/containers/authentik
setfacl -m d:g::0 ~/data/containers/authentik
setfacl -m d:o::0 ~/data/containers/authentik
setfacl -m u:auth:rwx ~/data/containers/authentik
setfacl -m u:100999:rwx ~/data/containers/authentik
setfacl -d -m u:auth:rwx ~/data/containers/authentik
setfacl -d -m u:100999:rwx ~/data/containers/authentik
mkdir -p ~/data/containers/authentik/{backups,certs,media,templates}
# After generating
sudo find ~/data/containers/authentik -type f -exec setfacl -m m::rw {} \;
sudo find ~/data/containers/authentik -type d -exec setfacl -m m::rwx {} \;
```
> Authentik container executes as 1000:1000(authentik:authentik) permission in container. It is mapped host's 100999. Therefore, directories have to have ACL via `setfacl`
#### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```ini
# ...
authentik IN CNAME auth.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
#### Add new database and user
- Following [here](../08_development/08_02_dev_postgresql.md). Authentik uses postgresql.
- dev server
```bash
podman exec -it -u postgres postgresql psql -U postgres
> # Create user and database
> CREATE USER authentik WITH PASSWORD '$AUTHENTIK_POSTGRESQL__PASSWORD_value';
> CREATE DATABASE authentik_db;
> ALTER DATABASE authentik_db OWNER TO authentik;
> \du
> \l
```
#### Add information in caddy-auth
- Following [here](./07_03_auth_main_caddy.md).
- auth server
- File: ~/data/containers/caddy-auth/etc/Caddyfile
```ini
authentik.ilnmors.internal {
import internal_tls
import crowdsec_log
reverse_proxy authentik.ilnmors.internal:9080
}
```
```bash
# After Caddyfile setting is changed use this command.
podman exec caddy-auth caddy reload --config /etc/caddy/Caddyfile
# fix inconsistencies
podman exec caddy-auth caddy fmt --overwrite /etc/caddy/Caddyfile
```
### Podman Image
```bash
podman pull ghcr.io/goauthentik/server:2025.10.0 # Do not use latest version to management
```
### Configuration file
- file:
- ~/data/containers/authentik/certs/root_ca.crt
### Quadlet
- File:
- ~/data/config/containers/authentik/authentik-pod.pod
- ~/data/config/containers/authentik/authentik.container
- ~/data/config/containers/authentik/authentik-worker.container
```ini
# ~/data/config/containers/authentik.pod
[Quadlet]
DefaultDependencies=false
[Pod]
PodName=authentik
# web port
PublishPort=9080:9000/tcp
# LDAP port
#PublishPort=[set_port]:3389
# Prometheus Port
#PublishPort=[set_port]:9300
```
```ini
# ~/data/config/containers/authentik/authentik.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=Authentik - Server
After=caddy.service
Wants=caddy.service
[Service]
ExecStartPre=%h/data/config/scripts/wait-for-it.sh -h postgresql.ilnmors.internal -p 5432 -t 0
ExecStartPre=sleep 5
[Container]
Pod=authentik.pod
Image=ghcr.io/goauthentik/server:2025.10.0
ContainerName=authentik-server
# Change default http port from 9000 to 9080 - in pod
Volume=%h/data/containers/authentik/media:/media:rw
Volume=%h/data/containers/authentik/certs:/certs:ro
Volume=%h/data/containers/authentik/templates:/templates:rw
Volume=%h/data/containers/authentik/backups:/backups:rw
# Default
Environment="TZ=Asia/Seoul"
# Listen
#AUTHENTIK_LISTEN__HTTP=0.0.0.0:9000
# LDAP > 0.0.0.0:3389
# METRICS > 0.0.0.0:9300
# AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS > `127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `fe80::/10`, `::1/128`
# DB connection - This can be changed as hot-reloading, however adding or removing needs restart.
Environment="AUTHENTIK_POSTGRESQL__HOST=postgresql.ilnmors.internal"
Environment="AUTHENTIK_POSTGRESQL__PORT=5432"
# Password will be injected as secret
Environment="AUTHENTIK_POSTGRESQL__USER=authentik"
Environment="AUTHENTIK_POSTGRESQL__NAME=authentik_db"
# SSL DB configuration
Environment="AUTHENTIK_POSTGRESQL__SSLMODE=verify-full"
Environment="AUTHENTIK_POSTGRESQL__SSLROOTCERT=/certs/root_ca.crt"
# This homelab doesn't use mTLS, therefore do not set `AUTHENTIK_POSTGRESQL__SSLCERT` and `AUTHENTIK_POSTGRESQL__SSLKEY`
# Media configuration - 'file' or 's3'
Environment="AUTHENTIK_STORAGE__MEDIA_BACKEND=file"
# Email configuration - after generate local Email services
# AUTHENTIK_EMAIL__HOST=ilnmors.internal
# AUTHENTIK_EMAIL__PORT=25
# AUTHENTIK_EMAIL__USERNAME=authentik
# AUTHENTIK_EMAIL__USE_TLS=true
# AUTHENTIK_EMAIL__FROM=authentik@ilnmors.internal
Secret=AUTHENTIK_SECRET_KEY,type=env
Secret=AUTHENTIK_POSTGRESQL__PASSWORD,type=env
# Start server
Exec=server
Label=diun.enable=true
Label=diun.watch_repo=true
[Install]
WantedBy=default.target
```
```ini
# ~/data/config/containers/authentik/authentik-worker.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=Authentik - worker
After=authentik-server.service
Requires=authentik-server.service
[Container]
Pod=authentik.pod
Image=ghcr.io/goauthentik/server:2025.10.0
ContainerName=authentik-worker
# Change default http port from 9000 to 9080 - in pod
Volume=%h/data/containers/authentik/media:/media:rw
Volume=%h/data/containers/authentik/certs:/certs:ro
Volume=%h/data/containers/authentik/templates:/templates:rw
Volume=%h/data/containers/authentik/backups:/backups:rw
# Default
Environment="TZ=Asia/Seoul"
# Listen
#AUTHENTIK_LISTEN__HTTP=0.0.0.0:9000
# LDAP > 0.0.0.0:3389
# METRICS > 0.0.0.0:9300
# AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS > `127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `fe80::/10`, `::1/128`
# DB connection - This can be changed as hot-reloading, however adding or removing needs restart.
Environment="AUTHENTIK_POSTGRESQL__HOST=postgresql.ilnmors.internal"
Environment="AUTHENTIK_POSTGRESQL__PORT=5432"
# Password will be injected as secret
Environment="AUTHENTIK_POSTGRESQL__USER=authentik"
Environment="AUTHENTIK_POSTGRESQL__NAME=authentik_db"
# SSL DB configuration
Environment="AUTHENTIK_POSTGRESQL__SSLMODE=verify-full"
Environment="AUTHENTIK_POSTGRESQL__SSLROOTCERT=/certs/root_ca.crt"
# This homelab doesn't use mTLS, therefore do not set `AUTHENTIK_POSTGRESQL__SSLCERT` and `AUTHENTIK_POSTGRESQL__SSLKEY`
# Media configuration - 'file' or 's3'
Environment="AUTHENTIK_STORAGE__MEDIA_BACKEND=file"
# Email configuration - after generate local Email services
# AUTHENTIK_EMAIL__HOST=ilnmors.internal
# AUTHENTIK_EMAIL__PORT=25
# AUTHENTIK_EMAIL__USERNAME=authentik
# AUTHENTIK_EMAIL__USE_TLS=true
# AUTHENTIK_EMAIL__FROM=authentik@ilnmors.internal
Secret=AUTHENTIK_SECRET_KEY,type=env
Secret=AUTHENTIK_POSTGRESQL__PASSWORD,type=env
# Start worker
Exec=worker
[Install]
WantedBy=default.target
```
> All configurations, except DB connection information will be saved in postgresql. Do not set Environment varibles. It has the first priority, it will override all configuration from web UI.
#### Create systemd `.service` file
```bash
# linger has to be activated
ln -s ~/data/config/containers/authentik/authentik-pod.pod ~/.config/containers/systemd/authentik-pod.pod
ln -s ~/data/config/containers/authentik/authentik-server.container ~/.config/containers/systemd/authentik.container
ln -s ~/data/config/containers/authentik/authentik-worker.container ~/.config/containers/systemd/authentik-worker.container
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start authentik.service
```
### Web UI configuration
#### Access web UI and initial setting
- URL: https://authentik.ilnmors.internal/if/flow/initial-setup/
> If you can't initialize the authentik at the first time, `DROP DATABASE authentik_db;` and recreate database after authentik container stop.
#### Initial setting wizard
- Email: Admin-email
- thiswork21@gmail.com
- Password: password
docs/archives/2025-12/07_authorization/07_04_auth_lldap.md
+386
View File
@@ -0,0 +1,386 @@
Tags: #os, #configuration, #network, #virtualization, #container, #security, #authentication, #authorization, #sso
## LLDAP (Light LDAP)
LLDAP provides LDAP protocol as very light and modern way. It supports not only main function of LDAP protocol such as Group, or User management but also web UI and RESTFul API. Additionally, it is very simple to set and manage and takes a small amount of resources. Following [here](../02_theory/02_05_sso.md) about the structure of LDAP.
### Secret management
File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
```yaml
# ~/data/config/secrets/.secret.yaml
# LLDAP:
LLDAP_DATABASE_URL: postgres://ldap:$PASSWORD@postgresql.ilnmors.internal/ldap_db?sslmode=verify-full&sslrootcert=/etc/ssl/ldap/root_ca.crt # $PASSWORD=openssl rand -base64 32 value and it should be encoded as URL
LLDAP_LDAP_USER_PASSWORD: $PASSWORD
LLDAP_KEY_SEED: "$(LC_ALL=C tr -dc 'A-Za-z0-9!#%&()*+,-./:;<=>?@[\]^_{|}~' </dev/urandom | head -c 32)"
LLDAP_JWT_SECRET: "$(LC_ALL=C tr -dc 'A-Za-z0-9!#%&()*+,-./:;<=>?@[\]^_{|}~' </dev/urandom | head -c 32)"
```
```bash
# Podman secret
extract_secret.sh .secret.yaml -f LLDAP_DATABASE_URL | podman secret create LLDAP_DATABASE_URL -
extract_secret.sh .secret.yaml -f LLDAP_LDAP_USER_PASSWORD | podman secret create LLDAP_LDAP_USER_PASSWORD -
extract_secret.sh .secret.yaml -f LLDAP_JWT_SECRET | podman secret create LLDAP_JWT_SECRET -
extract_secret.sh .secret.yaml -f LLDAP_KEY_SEED | podman secret create LLDAP_KEY_SEED -
```
### Preparation
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- 636:6360: localhost nat
- LDAPS will be only used localhost (Authelia), so PREROUTING doesn't require in nat table. nly OUTPUT requires in it.
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/ldap
chmod 700 ~/data/containers/ldap
setfacl -m d:g::0 ~/data/containers/ldap
setfacl -m d:o::0 ~/data/containers/ldap
setfacl -m u:auth:rwx ~/data/containers/ldap
setfacl -m u:100999:rwx ~/data/containers/ldap
setfacl -d -m u:auth:rwx ~/data/containers/ldap
setfacl -d -m u:100999:rwx ~/data/containers/ldap
mkdir ~/data/containers/ldap/{certs,data}
# After generating
sudo find ~/data/containers/ldap -type f -exec setfacl -m m::rw {} \;
sudo find ~/data/containers/ldap -type d -exec setfacl -m m::rwx {} \;
```
> lldap container executes as 1000:1000(lldap:lldap) permission in container. It is mapped host's 100999. Therefore, directories have to have ACL via `setfacl`
#### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```ini
# ...
ldap IN CNAME auth.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
#### Add new database and user
- Following [here](../08_development/08_02_dev_postgresql.md). LLDAP can use postgresql (Basic is SQLite).
- dev server
```bash
podman exec -it -u postgres postgresql psql -U postgres
> # Create user and database
> CREATE USER ldap WITH PASSWORD '$POSTGRES_LDAP_PASSWORD';
> CREATE DATABASE ldap_db;
> ALTER DATABASE ldap_db OWNER TO ldap;
> \du
> \l
```
#### Add information in caddy-auth
- Following [here](./07_03_auth_main_caddy.md).
- auth server
- File: ~/data/containers/caddy-auth/etc/Caddyfile
```ini
ldap.ilnmors.internal {
import internal_tls
import crowdsec_log
reverse_proxy host.containers.internal:17170
}
```
```bash
# fix inconsistencies
podman exec caddy-auth caddy fmt --overwrite /etc/caddy/Caddyfile
# After Caddyfile setting is changed use this command.
podman exec caddy-auth caddy reload --config /etc/caddy/Caddyfile
```
#### Certificates configuration
- File: ~/data/containers/certs/root_ca.crt
- ACME setting (OPNsense)
- Services:ACME Client:Certificates - Certificates - \[+\]
- Common Name: ldap.ilnmors.internal
- Description: ldap
- ACME Account: ldap.ilnmors.internal
> Even though provisioner's name includes `@`, it has to use as `.`.
>
> i.e. `acme@ilnmors.internal` > `acme.ilnmors.internal`
- Challenge Type: ilnmors.internal-dns-01-challenge
- \[\*\] Auto Renewal
- Automations: ldap-auto-acme, ldap-auto-restart
- Automations (OPNsense)
- Services:ACME Client:Automations - Automation - \[+\]
- Name: ldap-auto-acme / ldap-auto-reload
- Description: ldap acme crt issue / reload ldap after crt is issued
- Run Command: Upload certificate via SFTP / Remote command via SSH
- SFTP Host: ldap.ilnmors.internal
- Username: auth
- Identity Type: ed25519
- Remote Path(SFTP): /home/auth/data/containers/ldap/certs
- Command(SSH): setfacl -m m::r /home/auth/data/containers/ldap/certs/ldap.ilnmors.internal/* && systemctl --user restart ldap.service
- `Show Identity`
> Copy Required parameters `ssh-ed25519 ~~~ root@opnsense.ilnmors.internal`
>
> Add parameters in net server's ~/.ssh/authorized_keys
- `Test Connect` and `Save`
- SSH command will be applied after postgresql start.
### Podman Image
```bash
podman pull lldap/lldap:v0.6.2 # Do not use latest version to management
```
### Configuration file
- file:
- ~/data/containers/ldap/certs/root_ca.crt
- ~/data/containers/ldap/data/lldap_config.toml
### Initiating
```bash
podman run --rm \
--secret LLDAP_DATABASE_URL,type=env \
--secret LLDAP_KEY_SEED,type=env \
--secret LLDAP_JWT_SECRET,type=env \
--secret LLDAP_LDAP_USER_PASSWORD,type=env \
-e TZ="Asia/Seoul" \
-e LLDAP_LDAP_BASE_DN="dc=ilnmors,dc=internal" \
-v "$HOME"/data/containers/ldap/data:/data:rw \
-v "$HOME"/data/containers/ldap/certs:/etc/ssl/ldap:ro \
lldap/lldap:v0.6.2
# `Ctrl + C` exit
podman secret rm LLDAP_LDAP_USER_PASSWORD
```
### Quadlet
- File:
- ~/data/config/containers/ldap/ldap.container
```ini
# ~/data/config/containers/ldap/ldap.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=LDAP
After=caddy.service
Wants=caddy.service
[Service]
ExecStartPre=%h/data/config/scripts/wait-for-it.sh -h postgresql.ilnmors.internal -p 5432 -t 0
ExecStartPre=sleep 5
[Container]
Image=lldap/lldap:v0.6.2
ContainerName=ldap
# For LDAPS - 636 > 6360 iptables
PublishPort=6360:6360/tcp
# Web UI
PublishPort=17170:17170/tcp
Volume=%h/data/containers/ldap/data:/data:rw
Volume=%h/data/containers/ldap/certs:/etc/ssl/ldap:ro
# Default
Environment="TZ=Asia/Seoul"
# Domain
Environment="LLDAP_LDAP_BASE_DN=dc=ilnmors,dc=internal"
# LDAPS
Environment="LLDAP_LDAPS_OPTIONS__ENABLED=true"
Environment="LLDAP_LDAPS_OPTIONS__CERT_FILE=/etc/ssl/ldap/ldap.ilnmors.internal/fullchain.pem"
Environment="LLDAP_LDAPS_OPTIONS__KEY_FILE=/etc/ssl/ldap/ldap.ilnmors.internal/key.pem"
# SMTP options > you can set all of these at the /data/config.toml instead of Environment
# Only `LLDAP_SMTP_OPTIONS__PASSWORD` will be injected by secret
# LLDAP_SMTP_OPTIONS__ENABLE_PASSWORD_RESET=true
# LLDAP_SMTP_OPTIONS__SERVER=smtp.example.com
# LLDAP_SMTP_OPTIONS__PORT=465
# LLDAP_SMTP_OPTIONS__SMTP_ENCRYPTION=TLS
# LLDAP_SMTP_OPTIONS__USER=no-reply@example.com
# LLDAP_SMTP_OPTIONS__PASSWORD=PasswordGoesHere
# LLDAP_SMTP_OPTIONS__FROM=no-reply <no-reply@example.com>
# LLDAP_SMTP_OPTIONS__TO=admin <admin@example.com>
# Database
Secret=LLDAP_DATABASE_URL,type=env
# Secrets
Secret=LLDAP_KEY_SEED,type=env
Secret=LLDAP_JWT_SECRET,type=env
Label=diun.enable=true
Label=diun.watch_repo=true
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
# linger has to be activated
ln -s ~/data/config/containers/ldap/ldap.container ~/.config/containers/systemd/ldap.container
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start ldap.service
```
### DB backup
- Following [here](../08_development/08_02_dev_postgresql.md).
- dev server
```bash
systemctl --user enable --now postgresql-data-backup@ldap.timer
```
> The data saved in postgresql are all LDAP data including all users, groups and hashed passwords. However, LLDAP_KEY_SEED and LLDAP_JWT_SECRET are not saved in postgresql. It is for container itself and it is used as Environment value in the container.
### Configuration
#### Access web UI and Login
- URL: https://ldap.ilnmors.internal
- ID: admin
- PW: $LLDAP_LDAP_USER_PASSWORD
#### Create the groups
- Groups - \[\+\] Create a group
- Group: admins
- Group: users
#### Create the authelia user
- Users: \[\+\] Create a user
- Username (cn; uid): authelia
- Display name: Authelia
- First Name: Authelia
- Last Name (sn): Service
- Email (mail): authelia@ilnmors.internal
- Password: "$(openssl rand -base64 32)"
- lldap_strict_readonly \[Add to group\]
- This group allow search authority.
> Save the password in .secret.yaml
#### Create the normal user
- Users: \[\+\] Create a user
- Username (cn; uid): user
- First Name: John
- Last Name (sn): Doe
- Email (mail): john_doe@ilnmors.internal
- Password: "$PASSWORD"
- (admins|users) \[Add to group\]
> Custom schema in `User schema`, `Gropu schema` doesn't need to be added. This is for advanced function to add additional value such as `identity number` or `phone number`. Hardcoded schema, which means basic schema the lldap provides is enough to use Authelia.
> After all these steps, now you can integrate the Authelia for SSO.
### Usage of LDAP
#### Service Bind
LDAP call `login` as Bind. When the authelia Bind to the LDAP server, it can get the authority to search in `lldap_strict_readonly` group.
#### Search
authelia account has the authority to search, it can search to send the query.
##### Flow of search
- Client (authelia) sends the query
- `uid=user in dc=ilnmors,dc=ilnmors`
- LDAP server searches the DN of entry
- `uid=user,ou=people,dc=ilnmors,dc=ilnmors`
- LDAP sends the DN to Client (authelia)
### Authelia's work flow
#### First login
##### User login query
User try to login on login page of Authelia.
- id: user
- password: 1234
##### Service Bind (Bind and search)
authelia binds to LLDAP server based on the information in configuration.yml.
- dn: authelia
- password: authelia's password
##### Search
authelia sends the query to LLDAP after bind.
- `uid=user in dc=ilnmors,dc=internal`
##### Request
LLDAP server searches the entry and send the DN information query to authelia.
- `uid=user,ou=people,dc=ilnmors,dc=internal`
#### Verify the user login (Second login)
##### User Bind (Bind only)
authelia tries to bind LLDAP server based on the information that user input.
- dn: requested uid
- password: 1234
##### Verification from LLDAP
LLDAP verify the password from authelia with its hash value saved in LLDAP's database.
##### Request
LLDAP server sends the result as `Success` or `Fail`.
> Search authority is basic authority of user who binds to LDAP server. It is just the way to check success or fail bind is the charge of Authelia.
docs/archives/2025-12/07_authorization/07_05_auth_authelia.md
+651
View File
@@ -0,0 +1,651 @@
Tags: #os, #configuration, #network, #virtualization, #container, #security, #authentication, #authorization, #sso
## Authelia
Authelia is one of the open source authentication and authorization server which can management IAM(Identity and Access Management). It supports SSO and even OpenID Connect 1.0 Provider (idP) on OAuth 2.0. Authelia uses its backend database as the file or LDAP server. LLDAP server is this backend database for authelia in this homelab.
### Secret management
File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
```yaml
# ~/data/config/secrets/.secret.yaml
# Authelia:
AUTHELIA_JWT_SECRET: "$(LC_ALL=C tr -dc 'A-Za-z0-9!#%&()*+,-./:;<=>?@[\]^_{|}~' </dev/urandom | head -c 32)"
AUTHELIA_SESSION_SECRET: "$(LC_ALL=C tr -dc 'A-Za-z0-9!#%&()*+,-./:;<=>?@[\]^_{|}~' </dev/urandom | head -c 32)"
AUTHELIA_STORAGE_SECRET: "$(LC_ALL=C tr -dc 'A-Za-z0-9!#%&()*+,-./:;<=>?@[\]^_{|}~' </dev/urandom | head -c 32)"
AUTHELIA_HMAC_SECRET: "$(openssl rand -base64 32)"
AUTHELIA_JWKS_RS256: |
"$(podman run --rm \
-v ~/data/containers/authelia/certs:/output:rw \
authelia/authelia:4.39.13 \
authelia crypto pair ecdsa \
generate --directory /output/ecdsa)"
AUTHELIA_JWKS_ES256: |
"$(podman run --rm \
-v ~/data/containers/authelia/certs:/output:rw \
authelia/authelia:4.39.13 \
authelia crypto pair rsa \
generate --directory /output/rsa)"
# content of private.pem and rm private.pem/ leave public.pem
# This is set on LLDAP server
AUTHELIA_LDAP_PASSWORD: "$(openssl rand -base64 32)"
POSTGRES_AUTHELIA_PASSWORD: "$(openssl rand -base64 32)"
```
```bash
# Podman secret
extract_secret.sh .secret.yaml -f AUTHELIA_JWT_SECRET | podman secret create AUTHELIA_JWT_SECRET -
extract_secret.sh .secret.yaml -f AUTHELIA_SESSION_SECRET | podman secret create AUTHELIA_SESSION_SECRET -
extract_secret.sh .secret.yaml -f AUTHELIA_STORAGE_SECRET | podman secret create AUTHELIA_STORAGE_SECRET -
extract_secret.sh .secret.yaml -f AUTHELIA_HMAC_SECRET | podman secret create AUTHELIA_HMAC_SECRET -
extract_secret.sh .secret.yaml -f AUTHELIA_LDAP_PASSWORD | podman secret create AUTHELIA_LDAP_PASSWORD -
extract_secret.sh .secret.yaml -f POSTGRES_AUTHELIA_PASSWORD | podman secret create POSTGRES_AUTHELIA_PASSWORD -
extract_secret.sh .secret.yaml -f AUTHELIA_JWKS_RS256 | podman secret create AUTHELIA_JWKS_RS256 -
extract_secret.sh .secret.yaml -f AUTHELIA_JWKS_ES256 | podman secret create AUTHELIA_JWKS_ES256 -
```
### Preparation
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- 636:6360: localhost nat
- LDAPS will be only used localhost (Authelia), so PREROUTING doesn't require in nat table. nly OUTPUT requires in it.
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/authelia/{certs,config}
chmod -R 700 ~/data/containers/authelia
```
> authelia container executes as 0:0(root:root) permission in container. It is mapped host's UID. Therefore, directories don't have to have ACL via `setfacl`
#### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```ini
# ...
authelia IN CNAME auth.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
#### Add new database and user
- Following [here](../08_development/08_02_dev_postgresql.md). LLDAP can use postgresql (Basic is SQLite).
- dev server
```bash
podman exec -it -u postgres postgresql psql -U postgres
> # Create user and database
> CREATE USER authelia WITH PASSWORD '$POSTGRES_AUTHELIA_PASSWORD';
> CREATE DATABASE authelia_db;
> ALTER DATABASE authelia_db OWNER TO authelia;
> \du
> \l
```
#### Add information in caddy-auth
- Following [here](./07_03_auth_main_caddy.md).
- auth server
- File: ~/data/containers/caddy-auth/etc/Caddyfile
```ini
authelia.ilnmors.com {
import crowdsec_log
route {
crowdsec
reverse_proxy host.containers.internal:9091
}
}
authelia.ilnmors.internal {
import internal_tls
import crowdsec_log
route {
crowdsec
reverse_proxy host.containers.internal:9091
}
}
```
```bash
# fix inconsistencies
podman exec caddy-auth caddy fmt --overwrite /etc/caddy/Caddyfile
# After Caddyfile setting is changed use this command.
podman exec caddy-auth caddy reload --config /etc/caddy/Caddyfile
```
### Podman Image
```bash
podman pull authelia/authelia:4.39.13 # Do not use latest version to management
```
### Configuration file
- file:
- ~/data/containers/authelia/certs/root_ca.crt
- ~/data/containers/authelia/config/configuration.yml
#### configuration.yml
```yaml
# authelia configuration.yml
---
# certificates setting
certificates_directory: '/etc/ssl/authelia/'
# them setting - light, dark, grey, auto.
theme: 'auto'
# Server configuration
server:
# TLS will be applied on caddy
address: 'tcp://:9091/'
# Log configuration
log:
level: 'debug'
#file_path: 'path/of/log/file' - without this option, using stdout
# TOTP configuration
totp:
# issure option is for 2FA app. It works as identifier. "My homelab' or 'ilnmors.internal', 'Authelia - ilnmors'
issuer: 'ilnmors.internal'
# Identity validation confituration
identity_validation:
reset_password:
jwt_secret: '' # $AUTHELIA_IDENTITY_VALIDATION_RESET_PASSWORD_JWT_SECRET_FILE
# Authentication backend provider configuration
authentication_backend:
ldap:
# ldaps uses 636 -> NAT automatically change port 636 in output packet -> 6360 which lldap server uses.
address: 'ldaps://ldap.ilnmors.internal'
implementation: 'lldap'
# tls configruation, it uses certificates_directory's /etc/ssl/authelia/root_ca.crt
tls:
server_name: 'ldap.ilnmors.internal'
skip_verify: false
# LLDAP base DN
base_dn: 'dc=ilnmors,dc=internal'
additional_users_dn: 'ou=people'
additional_groups_dn: 'ou=groups'
# LLDAP filters
users_filter: '(&(|({username_attribute}={input})({mail_attribute}={input}))(objectClass=person))'
groups_filter: '(&(member={dn})(objectClass=groupOfNames))'
# LLDAP bind account configuration
user: 'uid=authelia,ou=people,dc=ilnmors,dc=internal'
password: '' # $AUTHELIA_AUTHENTICATION_BACKEND_LDAP_PASSWORD_FILE
# LLDAP schema mapping
attributes:
username: 'uid'
display_name: 'displayName'
mail: 'mail'
group_name: 'cn'
# Access control configuration
access_control:
default_policy: 'deny'
rules:
# authelia portal
- domain: 'authelia.ilnmors.internal'
policy: 'bypass'
- domain: 'authelia.ilnmors.com'
policy: 'bypass'
# Session provider configuration
session:
secret: '' # $AUTHELIA_SESSION_SECRET_FILE
expiration: '24 hours' # Session maintains for 24 hours
inactivity: '2 hours' # Session maintains for 2 hours without actions
cookies:
- name: 'authelia_private_session'
domain: 'ilnmors.internal'
authelia_url: 'https://authelia.ilnmors.internal'
same_site: 'lax'
- name: 'authelia_public_session'
domain: 'ilnmors.com'
authelia_url: 'https://authelia.ilnmors.com'
same_site: 'lax'
# This authelia doesn't use Redis.
# Storage provider configuration
storage:
encryption_key: '' # $AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE
postgres:
address: 'tcp://postgresql.ilnmors.internal:5432'
database: 'authelia_db'
username: 'authelia'
password: '' # $AUTHELIA_STORAGE_POSTGRES_PASSWORD_FILE
tls:
server_name: 'postgresql.ilnmors.internal'
skip_verify: false
# Notification provider
notifier:
filesystem:
filename: '/config/notification.txt'
# Following goal, After `Postfix` and `Dovecot` setting.
#smtp:
#address: 'smtp.ilnmors.internal'
# OIDC preperation
# Identity provisioner configuration
#identity_providers:
# oidc:
# hmac_secret: '' # $AUTHELIA_IDENTITY_PROVIDERS_OIDC_HMAC_SECRET_FILE
# jwks:
# - algorithm: 'RS256'
# use: 'sig'
# key: {{/* {{ secret "/run/secrets/AUTHELIA_JWKS_RS256" | mindent 10 "|" | msquote }} /*}}
# - algorithm: 'ES256'
# use: 'sig'
# key: {{/* {{ secret "/run/secrets/AUTHELIA_JWKS_ES256" | mindent 10 "|" | msquote }} /*}}
# clients:
...
```
### Quadlet
- File:
- ~/data/config/containers/authelia/authelia.container
```ini
# ~/data/config/containers/authelia/authelia.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=Authelia
After=caddy.service
Wants=caddy.service
After=ldap.service
Requires=ldap.service
[Service]
ExecStartPre=%h/data/config/scripts/wait-for-it.sh -h localhost -p 6360 -t 0
ExecStartPre=sleep 5
[Container]
Image=authelia/authelia:4.39.13
ContainerName=authelia
AddHost=ldap.ilnmors.internal:host-gateway
# Web UI
PublishPort=9091:9091/tcp
Volume=%h/data/containers/authelia/config:/config:rw
Volume=%h/data/containers/ldap/certs:/etc/ssl/authelia:ro
# Default
Environment="TZ=Asia/Seoul"
# Enable Go template engine
# !CAUTION!
# If this environment were enabled, you would have to use {{/* ... /*}} for {{ go_filter }} options. Go engine always processes its own grammar first.
Environment="X_AUTHELIA_CONFIG_FILTERS=template"
# Encryption
## JWT
Environment="AUTHELIA_IDENTITY_VALIDATION_RESET_PASSWORD_JWT_SECRET_FILE=/run/secrets/AUTHELIA_JWT_SECRET"
Secret=AUTHELIA_JWT_SECRET,target=/run/secrets/AUTHELIA_JWT_SECRET
## Session
Environment="AUTHELIA_SESSION_SECRET_FILE=/run/secrets/AUTHELIA_SESSION_SECRET"
Secret=AUTHELIA_SESSION_SECRET,target=/run/secrets/AUTHELIA_SESSION_SECRET
## Storage
Environment="AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE=/run/secrets/AUTHELIA_STORAGE_SECRET"
Secret=AUTHELIA_STORAGE_SECRET,target=/run/secrets/AUTHELIA_STORAGE_SECRET
# OIDC (HMAC, JWKS)
# Environment="AUTHELIA_IDENTITY_PROVIDERS_OIDC_HMAC_SECRET_FILE=/run/secrets/AUTHELIA_HMAC_SECRET"
# Secret=AUTHELIA_HMAC_SECRET,target=/run/secrets/AUTHELIA_HMAC_SECRET
# Secret=AUTHELIA_JWKS_RS256,target=/run/secrets/AUTHELIA_JWKS_RS256
# Secret=AUTHELIA_JWKS_ES256,target=/run/secrets/AUTHELIA_JWKS_ES256
# LDAP
Environment="AUTHELIA_AUTHENTICATION_BACKEND_LDAP_PASSWORD_FILE=/run/secrets/AUTHELIA_LDAP_PASSWORD"
Secret=AUTHELIA_LDAP_PASSWORD,target=/run/secrets/AUTHELIA_LDAP_PASSWORD
# Database
Environment="AUTHELIA_STORAGE_POSTGRES_PASSWORD_FILE=/run/secrets/POSTGRES_AUTHELIA_PASSWORD"
Secret=POSTGRES_AUTHELIA_PASSWORD,target=/run/secrets/POSTGRES_AUTHELIA_PASSWORD
Label=diun.enable=true
Label=diun.watch_repo=true
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
# linger has to be activated
ln -s ~/data/config/containers/authelia/authelia.container ~/.config/containers/systemd/authelia.container
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start authelia.service
```
### DB backup
- Following [here](../08_development/08_02_dev_postgresql.md).
- dev server
```bash
systemctl --user enable --now postgresql-data-backup@authelia.timer
```
### Verification
- Web UI:
- https://authelia.ilnmors.internal
- https://authelia.ilnmors.com
- Login with LLDAP's User
- Login: LLDAP User
- Password: LLDAP Password
- Check the session
### Apply Forward_Auth
Basically, OIDC will be used in this homelab for the application which supports OIDC natively. However, some applications don't support OIDC or even login system. Therfore, Forward_Auth function in Caddy is used for authentication of those services.
#### Configuration files
- File:
- ~/data/containers/caddy-auth/etc/Caddyfile
- ~/data/containers/authelia/config/configuration.yml
#### Caddy
```ini
# Caddyfile
# ...
# Foward auth test in local
test-admin.ilnmors.com {
import crowdsec_log
route {
crowdsec
forward_auth host.containers.internal:9091 {
# Authelia Forward Auth endpoint URI
uri /api/authz/forward-auth
copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
}
# back end service
root * /usr/share/caddy
file_server
}
}
# Forward auth test in local
test-default.ilnmors.com {
import crowdsec_log
route {
crowdsec
forward_auth host.containers.internal:9091 {
# Authelia Forward Auth endpoint URI
uri /api/authz/forward-auth
copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
}
# back end service
root * /usr/share/caddy
file_server
}
}
# ...
```
#### Authelia
```yaml
# configuration.yml
# ...
# Access control configuration
access_control:
default_policy: 'deny'
rules:
# authelia portal
- domain: 'authelia.ilnmors.internal'
policy: 'bypass'
- domain: 'authelia.ilnmors.com'
policy: 'bypass'
- domain: 'test-admin.ilnmors.com'
policy: 'one_factor'
# Access control for Forward_Auth
subject:
- 'group:admins'
- domain: 'test-default.ilnmors.com'
policy: 'one_factor'
# Access control for Forward_Auth
subject:
- 'group:admins'
- 'group:users'
session:
secret: '' # $AUTHELIA_SESSION_SECRET_FILE
cookies:
- name: 'authelia_internal_session'
domain: 'ilnmors.internal'
authelia_url: 'https://authelia.ilnmors.internal'
# When login succeed, redirect
# default_redirection_url: 'https://authelia.ilnmors.internal'
same_site: 'lax'
- name: 'authelia_com_session'
domain: 'ilnmors.com'
authelia_url: 'https://authelia.ilnmors.com'
# default_redirection_url: 'https://authelia.ilnmors.com'
same_site: 'lax'
# ...
```
#### Verification
- https://test-admin.ilnmors.com
- user_test (gruop: users): 403 Forbidden
- admin_test (group: admins): File server
- https://test-default.ilnmors.com
- admin_test (Just move): File server
- user_test (Session initiating): File server
---
### OIDC
```bash
openssl rand -base64 32
# Add this value to .secret.yaml
# APP_OIDC_KEY: secret value
# Make the hash value of this secret
# It is needed or not depends on app
extract_secret.sh .secret.yaml -f APP_OIDC_KEY | podman secret create APP_OIDC_KEY -
# Copy this value and paste next to client_secret: in configuration.yml
# Hash value generate
extract_secret.sh .secret.yaml -f CADDY_OIDC_KEY
> secret value
podman run --rm \
authelia/authelia:4.39.13 authelia
rypto hash generate --password secret_value --no-confirm
# Hash value validate
podman run --rm authelia/authelia:4.39.13 authelia crypto hash validate --password secret_value -- 'HASH_VALUE'
> The password matches the digest.
```
##### Authelia
- Remove annotation of OIDC in container file
```ini
# ~/data/config/containers/authelia/authelia.container
# ...
# OIDC (HMAC, JWKS)
Environment=AUTHELIA_IDENTITY_PROVIDERS_OIDC_HMAC_SECRET_FILE=/run/secrets/AUTHELIA_HMAC_SECRET"
Secret=AUTHELIA_HMAC_SECRET,target=/run/secrets/AUTHELIA_HMAC_SECRET
Secret=AUTHELIA_JWKS_RS256,target=/run/secrets/AUTHELIA_JWKS_RS256
Secret=AUTHELIA_JWKS_ES256,target=/run/secrets/AUTHELIA_JWKS_ES256
# Remove annotation
# ...
```
- Fix the configuration.yml
```yaml
---
# certificates setting
# ...
# Identity provisioner configuration
identity_providers:
oidc:
hmac_secret: '' # $AUTHELIA_IDENTITY_PROVIDERS_OIDC_HMAC_SECRET_FILE
jwks:
- algorithm: 'RS256'
use: 'sig'
key: {{ secret "/run/secrets/AUTHELIA_JWKS_RS256" | mindent 10 "|" | msquote }}
- algorithm: 'ES256'
use: 'sig'
key: {{ secret "/run/secrets/AUTHELIA_JWKS_ES256" | mindent 10 "|" | msquote }}
clients:
- client_id: 'app'
client_name: 'app'
# It depends on application
client_secret: 'HASH_VALUE'
# If there were not client secret, public should be `true`
public: \[ true | false \]
response_types:
- 'code'
scopes:
- 'openid'
- 'profile'
- 'email'
- 'groups'
redirect_uris:
- 'https://app.ilnmors.com/oauth2/callback
- 'https://app.ilnmors.com/'
token_endpoint_auth_method: 'client_secret_post | client_secret_basic'
authorization_policy: 'one_factor'
...
```
```bash
# restart the service
systemctl --user daemon-reload
systemctl --user restart authelia
```
#### Caddy
- Add the reverse proxy.
```ini
# Caddyfile
# ~/data/containers/caddy-auth/etc/Caddyfile
# ...
app.ilnmors.com {
import crowdsec_log
route {
crowdsec
# X-Forward-Host Domain doesn't need to appy on DNS
header_up X-Forwarded-Host app.app.ilnmors.internal
reverse_porxy app.ilnmors.internal
}
}
# ...
```
```ini
# app server's sidecar caddy
app.ilnmors.internal
{
import internal_tls
@notes header X-Forwarded-Host app.app.ilnmors.internal
reverse_proxy @notes host.containers.internal:3000
}
```
- Fix Caddyfile format and restart service
```bash
# fix inconsistencies
podman exec caddy-auth caddy fmt --overwrite /etc/caddy/Caddyfile
# After Caddyfile setting is changed use this command.
systemctl --user daemon-reload
systemctl --user restart caddy-auth
```
#### Verification
- Following.
docs/archives/2025-12/08_development/08_01_dev_vm.md
+149
View File
@@ -0,0 +1,149 @@
Tags: #os, #configuration, #development, #virtualization
## Preparation
### Set DHCP reservation and DNS record
#### Set DHCP reservation on KEA DHCP in OPNsense
Following [here](05_07_opnsense_kea.md)
- Services:Kea DHCP:Kea DHCPv4:Reservations - \[+\]
- Subnet: 192.168.10.0/24
- IP address: 192.168.10.13
- MAC address: 0A:49:6E:4D:03:00
- Hostname: dev
- Description: dev
- `save`
#### Set DNS records in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file:
- ~/data/containers/bind/lib/db.ilnmors.internal
- ~/data/containers/bind/lib/db.10.168.192.in-addr.arpa
```ini
# db.ilnmors.internal
# ...
dev IN A 192.168.10.13
# ...
# db.10.168.192.in-addr.arpa
# ...
13 IN PTR dev.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
### Create VM template
- ~/data/config/scripts/dev.sh
```bash
virt-install \
--boot uefi \
--name dev \
--os-variant debian13 \
--vcpus 2 \
--memory 6144 \
--location /var/lib/libvirt/images/debian-13.0.0-amd64-netinst.iso \ # For serial installing, use `--location` instead of `--cdrom`
--disk pool=vm-images,size=258,format=qcow2,discard=unmap \
--network network=ovs-lan-net,portgroup=vlan10-access,model=virtio,mac=0A:49:6E:4D:03:00 \ # Use designated ovs port group
--graphics none \
--console pty,target_type=serial \
--extra-args "console=ttyS0,115200"
# After enter this command, then the console start automatically
# Remove all annotation before you make the sh file.
```
### Debian installing
- Following [here](../03_common/03_01_debian_configuration.md) to install Debian.
- Debian installer supports serial mode regardless getty@ttyS0 service is enabled or not.
- Following [here](../03_common/03_02_iptables.md) to set iptables.
- Following [here](../03_common/03_04_crowdsec.md) to set CrowdSec
#### Serial console setting
After installation, use `ctrl + ]` to exit console. Before setting getty@ttyS0, you can't use serial console to access VM. Therefore, use IP address set on installation, and connect net server via ssh first, following the step to enable the getty.
### Modify VM template settings
After getty setting, shutdown dev vm with `shutdown` in VM or `sudo virsh shutdown dev` in hypervisor to turn off vm first.
```bash
virsh edit dev
```
```xml
<!-- dev -->
...
</vcpu>
<cputune>
<shares>1024</shares>
</cputune>
<!-- cpu priority - 1024: default/2048: high/512: low -->
<!--
<disk type='file' device='cdrom'>
...
</disk>
# Remove booting disk
-->
```
```bash
virsh dumpxml dev > ~/data/config/vms/dumps/dev.xml
virsh start dev && virsh console dev
# Start dev server with console
```
### Common setting
- dev.service
```ini
# ~/data/config/services/dev.service
# ~/.config/systemd/user/dev.service
[Unit]
Description=dev Auto Booting
After=network-online.target
Wants=network-online.target
Requires=opnsense.service
[Service]
Type=oneshot
# Maintain status as active
RemainAfterExit=yes
# CrowdSec should be set
ExecStartPre=%h/data/config/scripts/wait-for-it.sh 192.168.10.1:8080 -t 0
ExecStartPre=%h/data/config/scripts/wait-for-it.sh 192.168.10.11:53 -t 0
ExecStartPre=%h/data/config/scripts/wait-for-it.sh 192.168.10.12:9000 -t 0
ExecStartPre=/bin/bash -c "sleep 15"
# Run the service
ExecStart=/usr/bin/virsh -c qemu:///system start dev
# Stop the service
ExecStop=/usr/bin/virsh -c qemu:///system shutdown dev
[Install]
WantedBy=default.target
```
```bash
ln -s ~/data/config/services/dev.service ~/.config/systemd/user/dev.service
systemctl --user daemon-reload
systemctl --user enable dev.service
systemctl --user start dev.service
```
docs/archives/2025-12/08_development/08_02_dev_postgresql.md
+534
View File
@@ -0,0 +1,534 @@
Tags: #os, #configuration, #virtualization, #container, #database
## postgresql
Postgresql is one of most famous open source RDBMS (Relational Database Management System). RDBMS saves the data as table which is based on row and coloumn. This uses SQL (Structure Quarey Language) to manage a lot of data and guarantee integrity of data.
### Secret management
All transaction of postgresql which needs root permission will be conducted on local container or podman exec (without other Pod) in this porject. However, the environment value `POSTGRES_PASSWORD` should be set when the database is initiated. It makes postgresql programs root permission access way as trust which means only local access is allowed. This infromation will be set on `pg_hba.conf` like below.
- `local all postgres trust`
- `hostssl all all 192.168.10.x/32 scram-sha-256` (The server that uses postgresql)
- `host all all 127.0.0.1/32 scram-sha-256`
#### Secret
- File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
```yaml
# ~/data/config/secrets/.secret.yaml
# postgresql
POSTGRES_PASSWORD: secret
```
### Preparation
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- 5432: auth, app
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/postgresql
chmod 700 ~/data/containers/postgresql
setfacl -m d:g::0 ~/data/containers/postgresql
setfacl -m d:o::0 ~/data/containers/postgresql
setfacl -m u:dev:rwx ~/data/containers/postgresql
setfacl -m u:100998:rwx ~/data/containers/postgresql
setfacl -d -m u:dev:rwx ~/data/containers/postgresql
setfacl -d -m u:100998:rwx ~/data/containers/postgresql
mkdir ~/data/containers/postgresql/{backups,config,certs,data,initdb}
# After generating
c
sudo find ~/data/containers/postgresql -type d -exec setfacl -m m::rwx {} \;
```
> postgresql container executes as 999:999(postgres:postgres) permission in container. It is mapped host's 100998. Therefore, directories have to have ACL via `setfacl`
#### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```ini
# ...
postgresql IN CNAME dev.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
### Podman Image
```bash
podman pull postgres:18.0 # Do not use latest version to management
```
### Config file
- File:
- ~/data/containers/postgresql/config/postgresql.conf
- ~/data/containers/postgresql/config/pg_hba.conf
```bash
# extracting config file
podman run --rm postgres:18.0 cat /usr/share/postgresql/postgresql.conf.sample > ~/data/containers/postgresql/config/postgresql.conf
podman run --rm postgres:18.0 cat /usr/share/postgresql/18/pg_hba.conf.sample > ~/data/containers/postgresql/config/pg_hba.conf
# Generate the password
openssl rand -base64 32
# Podman secret
extract_secret.sh ~/data/config/secrets/.secret.yaml -f "POSTGRES_PASSWORD" | podman secret create "POSTGRES_PASSWORD" -
```
> If there were the schema backup file, then go to the `Restore` section.
```ini
# ~/data/containers/postgresql/config/postgresql.conf
# Add settings for extensions here
# hba_file directory
hba_file = '/config/pg_hba.conf'
# Listen_address
listen_addresses = '*'
# listen_port
port = 5432
# SSL
ssl = on
ssl_ca_file = '/etc/ssl/postgresql/root_ca.crt'
ssl_cert_file = '/etc/ssl/postgresql/postgresql.ilnmors.internal/fullchain.pem'
ssl_key_file = '/etc/ssl/postgresql/postgresql.ilnmors.internal/key.pem'
ssl_ciphers = 'HIGH:!aNULL:!MD5'
ssl_prefer_server_ciphers = on
# log
log_destination = 'stderr'
log_checkpoints = on
log_temp_files = 0
log_min_duration_statement = 500
```
```ini
# ~/data/containers/postgresql/config/pg_hba.conf
# Local host `trust`
local all all trust
# Local connection (container) needs password (127.0.0.1 - container loopback)
host all all 127.0.0.1/32 scram-sha-256
# Local connection (dev) needs password (169.254.1.2 - host-gateway)
# Maybe Grafana(SQLite), Uptime kuma(SQLite), Loki(BoltDB), Dovecot(LDAP or />
#hostssl all all 169.254.1.2/32 scram-sha-256
# auth VM (Authentik, 192.168.10.12)
hostssl all all 192.168.10.12/32 scram-sha-256
# app VM (Applications, 192.168.10.14)
hostssl all all 192.168.10.14/32 scram-sha-256
# explicit deny
host all all 192.168.10.0/24 reject
```
#### Initiating (Restoring)
- File:
- ~/data/config/scripts/postgresql/postgresql_init.sh
- ~/data/containers/postgresql/backups/postgresql-cluster_$(date "+%Y-%m-%d").sql
```bash
#!/bin/bash
# ~/data/config/scripts/postgresql/postgresql_init.sh [FILE_PATH]
set -e
DATA_PATH="$HOME/data/containers/postgresql/data"
FILE_PATH="$1"
VERSION=18
FLAG=""
# Check the PostgreSQL service
if [ $(systemctl --user is-active postgresql) == "active" ]; then
echo "PostgreSQL should be terminated"
exit 1
fi
# Check the podman secret
if [ -z "$(podman secret list | grep "POSTGRES_PASSWORD")" ]; then
echo "POSTGRES_PASSWORD has to be in podman secret"
exit 1
fi
# Check the data path
if [ -n "$(ls -A $DATA_PATH)" ]; then
echo "$DATA_PATH should be empty"
exit 1
fi
# Check the sql file
if [ -z "$FILE_PATH" ]; then
echo "Initiating PostgreSQL"
FLAG="FALSE"
else
if [ ! -f "$FILE_PATH" -o ! -s "$FILE_PATH" -o -z "$(echo $FILE_PATH | grep "\.sql$")" ]; then
echo "Availavble .sql format file is needed"
exit 1
fi
echo "Restoring PostgreSQL"
FLAG="TRUE"
fi
# Initiating
podman run --rm \
--secret POSTGRES_PASSWORD,type=env \
-e TZ="Asia/Seoul" \
-v "$DATA_PATH":/var/lib/postgresql:rw \
postgres:18.0 \
-C "port" || true
# postgresql start
echo "Start postgresql service"
systemctl --user start postgresql
# Restoring
if [ "$FLAG" == "TRUE" ]; then
while [ -z "$(systemctl --user status postgresql | grep "database system is ready to accept connections")" ]; do
sleep 1
done
echo "Start restoring PostgreSQL"
cat "$FILE_PATH" | podman exec -i -u postgres postgresql psql -U postgres
echo "Finish restoring PostgreSQL"
fi
exit 0
```
#### Certificates configuration
- File: ~/data/containers/postgresql/certs/root_ca.crt
- ACME setting (OPNsense)
- Services:ACME Client:Certificates - Certificates - \[+\]
- Common Name: postgresql.ilnmors.internal
- Description: postgresql
- ACME Account: acme.ilnmors.internal
> Even though provisioner's name includes `@`, it has to use as `.`.
>
> i.e. `acme@ilnmors.internal` > `acme.ilnmors.internal`
- Challenge Type: ilnmors.internal-dns-01-challenge
- \[\*\] Auto Renewal
- Automations: postgresql-auto-acme, postgresql-auto-restart
- Automations (OPNsense)
- Services:ACME Client:Automations - Automation - \[+\]
- Name: postgresql-auto-acme / postgresql-auto-reload
- Description: postgresql acme crt issue / reload postgresql after crt is issued
- Run Command: Upload certificate via SFTP / Remote command via SSH
- SFTP Host: postgresql.ilnmors.internal
- Username: dev
- Identity Type: ed25519
- Remote Path(SFTP): /home/dev/data/containers/postgresql/certs
- Command(SSH): setfacl -m m::r /home/dev/data/containers/postgresql/certs/postgresql.ilnmors.internal/* && podman exec -u postgres postgresql pg_ctl reload
- `Show Identity`
> Copy Required parameters `ssh-ed25519 ~~~ root@opnsense.ilnmors.internal`
>
> Add parameters in net server's ~/.ssh/authorized_keys
- `Test Connect` and `Save`
- SSH command will be applied after postgresql start.
#### Quadlet
- File:
- ~/data/config/containers/postgresql/postgresql.container
```ini
# ~/data/config/containers/postgresql/postgresql.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=PostgreSQL
After=network-online.target
Wants=network-online.target
[Container]
Image=postgres:18.0
ContainerName=postgresql
PublishPort=5432:5432/tcp
Volume=%h/data/containers/postgresql/data:/var/lib/postgresql:rw
Volume=%h/data/containers/postgresql/config:/config:ro
Volume=%h/data/containers/postgresql/backups:/backups:rw
Volume=%h/data/containers/postgresql/certs:/etc/ssl/postgresql:ro
Environment="TZ=Asia/Seoul"
Exec=postgres -c 'config_file=/config/postgresql.conf'
Label=diun.enable=true
Label=diun.watch_repo=true
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
mkdir -p ~/.config/containers/systemd
chmod -R 700 ~/.config/containers/systemd
ln -s ~/data/config/containers/postgresql/postgresql.container ~/.config/containers/systemd/postgresql.container
systemctl --user daemon-reload
```
#### Enable and start service
```bash
# Before start service, setfacl for certificates (before register automation)
setfacl -m m::r /home/dev/data/containers/postgresql/certs/postgresql.ilnmors.internal/*
systemctl --user start postgresql.service
```
### Create user
- Login to psql in the container
```bash
podman exec -it -u postgres postgresql psql -U postgres
> # Create user and database
> CREATE USER $USER WITH PASSWORD 'password';
> CREATE DATABASE $DB;
> ALTER DATABASE $DB OWNER TO $USER;
> \du
> \l
# Whenever you modify the schema including user or database structure, conduct postgresql-cluster-backup.service
systemctl --user start postgresql-cluster-backup.service
> # If you want to change the password
> ALTER USER $USER WITH PASSWORD 'password';
> # After this, update the .secret.yaml file and podman secret
```
### Backup
#### Backup service
- File:
- ~/data/config/services/postgresql/postgresql-cluster-backup.service
- ~/data/config/services/postgresql/postgresql-cluster-backup.timer
- ~/data/config/services/postgresql/postgresql-data-backup@.service
- ~/data/config/services/postgresql/postgresql-data-backup@.timer
#### Cluster
```ini
# ~/data/config/services/postgresql/postgresql-cluster-backup.service
# ~/.config/systemd/user/postgresql-cluster-backup.service
[Unit]
Description=PostgreSQL Cluster Backup Service
After=postgresql.service
BindsTo=postgresql.service
[Service]
Type=oneshot
# logging
StandardOutput=journal
StandardError=journal
ExecStartPre=podman exec -u postgres postgresql sh -c "mkdir -p /backups/cluster"
# Run the script
ExecStart=podman exec -u postgres postgresql sh -c "pg_dumpall -U postgres --schema-only | grep -v -E \"ROLE postgres\" > /backups/cluster/postgresql-cluster_$(date "+%%Y-%%m-%%d").sql"
ExecStop=podman exec -u postgres postgresql sh -c "find /backups/cluster -maxdepth 1 -type f -mtime +7 -delete"
```
```ini
# ~/data/config/services/postgresql/postgresql-cluster-backup.timer
# ~/.config/systemd/user/postgresql-cluster-backup.timer
[Unit]
Description=Run PostgreSQL Cluster Backup service every day
[Timer]
# Execute service after 1 min on booting
OnBootSec=1min
# Execute service every day 00:00
OnCalendar=*-*-* 00:00:00
# Random time to postpone the timer
RandomizedDelaySec=15min
Persistent=true
# When timer is activated, Service also starts.
Persistent=true
[Install]
WantedBy=timers.target
```
##### Restore cluster
- File:
- ~/data/config/scripts/postgresql/postgresql_init.sh
- ~/data/containers/postgresql/backups/cluster/postgresql-cluster_$(date "+%Y-%m-%d").sql
- Use `postgresql_init.sh postgresql-cluster_$(date "+%Y-%m-%d").sql` command
#### Data for each app
```ini
# ~/data/config/services/postgresql/postgresql-data-backup@.service
# ~/.config/systemd/user/postgresql-data-backup@.service
[Unit]
Description=PostgreSQL Data %i Backup Service
After=postgresql.service
BindsTo=postgresql.service
[Service]
Type=oneshot
# logging
StandardOutput=journal
StandardError=journal
ExecStartPre=podman exec -u postgres postgresql sh -c "mkdir -p /backups/%i"
# Run the script
ExecStart=podman exec -u postgres postgresql sh -c "pg_dump -U postgres -d %i_db --data-only > /backups/%i/postgresql-%i-data_$(date "+%%Y-%%m-%%d").sql"
ExecStop=podman exec -u postgres postgresql sh -c "find "/backups/%i" -maxdepth 1 -type f -mtime +7 -delete"
```
```ini
# ~/data/config/services/postgresql/postgresql-data-backup@.timer
# ~/data/config/services/postgresql/postgresql-data-backup@.timer
[Unit]
Description=Run %i Data Backup service every day
[Timer]
# Execute service after 1 min on booting
OnBootSec=1min
# Execute service every day 00:00
OnCalendar=*-*-* 00:00:00
# Random time to postpone the timer
RandomizedDelaySec=15min
Persistent=true
# When timer is activated, Service also starts.
Persistent=true
[Install]
WantedBy=timers.target
```
##### Data restore
- File: postgresql-app-data_$(date "+%Y-%m-%d").sql
```bash
# The Schema must be needed. (cluster, DB and user)
# DB owner's session terminate
# server where app is located
systemctl --user stop app.service
# Check session
# dev server
# Print the all session
podman exec -u postgres postgresql psql -U postgres -c "SELECT * from pg_stat_activity;"
> $TARGET_PID
# exit session
podman exec -u postgres postgresql psql -U postgres -c "SELECT pg_terminate_backend($TARGET_PID);"
# Using psql
cat postgresql-app-data_$(date "+%Y-%m-%d").sql | podman exec -i -u postgres postgresql psql -U app
```
#### Register service
```bash
# Register service
mkdir -p ~/.config/systemd/user && chmod 700 ~/.config/systemd/user
ln -s ~/data/config/services/postgresql/postgresql-cluster-backup.service ~/.config/systemd/user/postgresql-cluster-backup.service
ln -s ~/data/config/services/postgresql/postgresql-cluster-backup.timer ~/.config/systemd/user/postgresql-cluster-backup.timer
ln -s ~/data/config/services/postgresql/postgresql-data-backup\@.service ~/.config/systemd/user/postgresql-data-backup\@.service
ln -s ~/data/config/services/postgresql/postgresql-data-backup\@.timer ~/.config/systemd/user/postgresql-data-backup\@.timer
systemctl --user daemon-reload
# Start timer and enable
systemctl --user enable --now postgresql-cluster-backup.timer
systemctl --user enable --now postgresql-data-backup@app.timer
```
### Verification
```bash
# Init database
postgresql_init.sh
# ... Start postgresql service
# Create user and database
podman exec -it -u postgres postgresql psql -U postgres
> CREATE USER test WITH PASSWORD 'abc';
> CREATE DATABASE test_db;
> ALTER DATABASE test_db OWNER TO test;
> \du
> \l
> \q
# Backup service executes
systemctl --user start postgresql-cluster-backup.service
# Stop and remove all data
systemctl --stop postgresql
sudo find "/home/dev/data/containers/postgresql/data" -mindepth 1 -delete
# Restore database
postgresql_init.sh ~/data/containers/backups/filename.sql
# Check restoring
podman exec -it -u postgres postgresql psql -U postgres
> \du
> \l
```
docs/archives/2025-12/08_development/08_03_dev_sidecar_caddy.md
+259
View File
@@ -0,0 +1,259 @@
Tags: #os, #configuration, #network, #virtualization, #container, #security
## Caddy - dev
Caddy is an open source reverse proxy (web server) which supports automatically to apply TLS certificates via ACME protocol from CA. This caddy will work as sidecar caddy, so it only uses private TLS. It can only communication with auth's main caddy. This means it doesn't need any module except RFC2136 module. Because main caddy will conduct WAF function at all.
### Secret management
- File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
```yaml
# ~/data/config/secrets/.secret.yaml
# CADDY:
CADDY_ACME_KEY: acme-key_key_value (Only secret value)
```
```bash
# Podman secret
extract_secret.sh .secret.yaml -f CADDY_ACME_KEY | podman secret create CADDY_ACME_KEY -
```
### Preparation
#### iptables and firewall rules
- Set iptables first, following [here](../03_common/03_02_iptables.md).
- Limit access client as Caddy - auth ( -s 192.168.10.12/32 )
- 443 > 2443 (iptables setting)
- Set firewall rules first, following [here](Latest/05_firewall/05_04_opnsense_rules.md).
#### Create directory for container
```bash
mkdir -p ~/data/containers/caddy-dev/{etc,data}
chmod -R 700 ~/data/containers/caddy-dev
```
> Caddy container executes as 0:0(root:root) permission in container. It is mapped host's UID. Therefore, directories don't have to have ACL via `setfacl`
#### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```ini
# ...
dev IN A 192.168.10.13
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
### Podman Image
#### Podman containerfile
Caddy supports various module for it. rfc2136(nsupdate) module, Sidecar caddy only recieves the request from the main caddy. Therefore it doesn't need anymore.
- file:
- ~/data/config/containers/caddy-dev/containerfile-caddy-2.10.2-dev
- ~/data/config/containers/caddy-dev/root_ca.crt
```containerfile
FROM caddy:2.10.2-builder-alpine AS builder
RUN xcaddy build \
--with github.com/caddy-dns/rfc2136
FROM caddy:2.10.2
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
COPY ./root_ca.crt /usr/local/share/ca-certificates/root_ca.crt
RUN update-ca-certificates
```
#### Podman image build
```bash
podman build -t caddy:2.10.2-dev -f ~/data/config/containers/caddy-dev/containerfile-caddy-2.10.2-dev . && podman image prune -f
# Delete pure caddy and caddy-builder-alpine images after command above manually.
```
### Configuration files
Caddyfile will be updated after Authelia setting
```bash
# fix inconsistencies
podman exec caddy-dev caddy fmt --overwrite /etc/caddy/Caddyfile
# After Caddyfile setting is changed use this command.
podman exec caddy-dev caddy reload --config /etc/caddy/Caddyfile
```
- file:
- ~/data/containers/caddy-auth/etc/Caddyfile
- ~/data/containers/authelia/config/configuration.yml
- ~/data/containers/caddy-dev/etc/Caddyfile
```ini
# Caddyfile
# ~/data/containers/caddy-auth/etc/Caddyfile
# Forward Auth for other vms
(apply_forward_auth) {
forward_auth host.containers.internal:9091 {
uri /api/authz/forward-auth
copy_headers Remote-User Remote-Groups Remote-Email R>
}
reverse_proxy {args[0]} {
header_up Host {http.reverse_proxy.upstream.host}
# X-Forwarded-Host header contains original Host value
}
}
# ...
dev-test.ilnmors.com {
import crowdsec_log
route {
crowdsec
import apply_forward_auth https://dev.ilnmors.internal
}
}
# ...
```
```ini
# Caddyfile
# ~/data/containers/caddy-dev/etc/Caddyfile
{
server {
trusted_proxies static 192.168.10.12/32
trusted_proxies_strict
# To find the real client's IP.
# default = false : left to right, when the Caddy met the untrusted IP first, then it would treat as client IP
# true : right to left > It is easy to find currupted client IP
}
}
(private_tls) {
tls {
issuer acme {
dir https://step-ca.ilnmors.internal:9000/acme/acme@ilnmors.internal/directory
dns rfc2136 {
server bind.ilnmors.internal:2253
key_name acme-key
key_alg hmac-sha256
key "{file./run/secrets/CADDY_ACME_KEY}"
}
}
}
}
dev.ilnmors.internal {
import private_tls
@test header X-Forwarded-Host dev-test.ilnmors.com
route @test {
root * /usr/share/caddy
file_server
}
}
```
```yaml
# configuration.yml
# ...
# Access control configuration
access_control:
default_policy: 'deny'
rules:
# authelia portal
- domain: 'authelia.ilnmors.internal'
policy: 'bypass'
- domain: 'authelia.ilnmors.com'
policy: 'bypass'
- domain: 'dev-test.ilnmors.com'
policy: 'one_factor'
# Access control for Forward_Auth
subject:
- 'group:admins'
# ...
```
### Quadlet
- File:
- ~/data/config/containers/caddy-auth/caddy-dev.container
```ini
# ~/data/config/containers/caddy-dev/caddy-dev.container
# ~/data/config/containers/caddy-dev/caddy-dev.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=Caddy - dev
After=network-online.target
Wants=network-online.target
[Service]
# Main Caddy and Step-CA
ExecStartPre=%h/data/config/scripts/wait-for-it.sh -h 192.168.10.12 -p 443 -t 0
ExecStartPre=%h/data/config/scripts/wait-for-it.sh -h 192.168.10.12 -p 9000 -t 0
ExecStartPre=sleep 5
[Container]
Image=localhost/caddy:2.10.2-dev
ContainerName=caddy-dev
PublishPort=2080:80/tcp
PublishPort=2443:443/tcp
Volume=%h/data/containers/caddy-dev/etc:/etc/caddy:rw
Volume=%h/data/containers/caddy-dev/data:/data:rw
Environment="TZ=Asia/Seoul"
Secret=CADDY_ACME_KEY,target=/run/secrets/CADDY_ACME_KEY
Label=diun.enable=true
Label=diun.watch_repo=true
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
# linger has to be activated
ln -s ~/data/config/containers/caddy-dev/caddy-dev.container ~/.config/containers/systemd/caddy-dev.container
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start caddy-dev.service
```
#### Verification
- https://dev-test.ilnmors.com
- user_test (gruop: users): 403 Forbidden
- admin_test (group: admins): File server
docs/archives/2025-12/08_development/08_04_dev_code-server.md
+341
View File
@@ -0,0 +1,341 @@
Tags: #os, #configuration, #security , #virtualization, #container, #development
## Code-Server
Code-Server is an open source and self-hosted Code editer (or IDE with its plugins) to use on web browser. This supports terminal and code editer, git, and ansible. It will be used as a bastion host in the home-lab. Code-Server doesn't support login system. Therefore, authelia and caddy's Forward-Auth function will be used in this homelab system.
### Secret management
- File:
- ~/data/config/secrets/.secret.yaml
- Edit `.secret.yaml` with `edit_secret.sh`
```yaml
# ~/data/config/secrets/.secret.yaml
# CODE-SERVER:
CODESERVER_SSH_KEY: SSH_KEY_VALUE
```
```bash
ssh-keygen -t ed25519 -f /run/user/$UID/codeserver -C "code-server@ilnmors.internal"
> [enter] # no passphrase
cat /run/user/$UID/codeserver
> Private key value
# add in .secret.yaml as CODESERVER_SSH_KEY
cat /run/user/$UID/codeserver.pub
> Public key value
# add in .secret.yaml as a annotation
rm -rf /run/user/$UID/codeserver*
extract_secret.sh .secret.yaml -f CODESERVER_SSH_KEY | podman secret create CODESERVER_SSH_KEY -
```
### Preparation
#### Create directory for container
```bash
mkdir -p ~/data/containers/code-server
chmod -R 700 ~/data/containers/code-server
setfacl -m d:g::0 ~/data/containers/code-server
setfacl -m d:o::0 ~/data/containers/code-server
setfacl -m u:dev:rwx ~/data/containers/code-server
setfacl -m u:100999:rwx ~/data/containers/code-server
setfacl -d -m u:dev:rwx ~/data/containers/code-server
setfacl -d -m u:100999:rwx ~/data/containers/code-server
mkdir -p ~/data/containers/code-server/{config,local,ssh,workspace}
echo $PUBLIC_SSH_KEY_VALUE > ~/data/containers/code-server/ssh/id_codeserver.pub
```
> Code-Server container executes as 1000:1000(coder:coder) permission in container. It is mapped host's 100999. Therefore, directories have to have ACL via `setfacl`
#### Add new domain in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file: ~/data/containers/bind/lib/db.ilnmors.internal
```ini
# ...
code-server IN CNAME auth.ilnmors.internal.
# ...
```
```bash
# Adguard container has Requires=bind.service. When it restarted, then Adguard also restarted.
systemctl --user restart bind
```
#### Add new rules in Caddy and authelia
##### caddy-auth
- auth server
- file: ~/data/containers/caddy-auth/etc/Caddyfile
```ini
# ...
code-server.ilnmors.internal {
import private_tls
import crowdsec_log
route {
crowdsec
import apply_forward_auth https://dev.ilnmors.internal
}
# ...
```
##### caddy-dev
- dev server
- file: ~/data/containers/caddy-dev/etc/Caddyfile
```ini
# ...
dev.ilnmors.internal {
import private_tls
# ...
@code-server header X-Forwarded-Host code-server.ilnmors.internal
# ...
route @code-server {
reverse_proxy host.containers.internal:8000 {
# Sidecar caddy's Caddyfile should change `Host` header as original `Host` value from `X-Forwarded-Host` to prevent websocket problem.
header_up Host {http.request.header.X-Forwarded-Host}
}
}
}
```
##### authelia
```yaml
# configuration.yml
# ...
# Access control configuration
access_control:
default_policy: 'deny'
rules:
# authelia portal
- domain: 'authelia.ilnmors.internal'
policy: 'bypass'
- domain: 'authelia.ilnmors.com'
policy: 'bypass'
- domain: 'dev-test.ilnmors.com'
policy: 'one_factor'
# Access control for Forward_Auth
subject:
- 'group:admins'
- domain: 'code-server.ilnmors.internal'
policy: 'one_factor'
subject:
- 'group:admins'
# ...
```
#### SSH configuration
##### Each server
- file: ~/.ssh/authorized_keys
```ini
# ...
Contents of code-server's public key value
# ...
```
##### Code-Server container
- file: ~/data/containers/code-server/ssh/config
```ini
Host vmm
HostName 192.168.1.10
User vmm
IdentityFile /run/secrets/CODESERVER_SSH_KEY
Host net
HostName 192.168.10.11
User net
IdentityFile /run/secrets/CODESERVER_SSH_KEY
Host auth
HostName 192.168.10.12
User auth
IdentityFile /run/secrets/CODESERVER_SSH_KEY
# dev is the server where code-server is located. It needs host.containers.internal
Host dev
HostName host.containers.internal
User dev
IdentityFile /run/secrets/CODESERVER_SSH_KEY
Host app
HostName 192.168.10.14
User app
IdentityFile /run/secrets/CODESERVER_SSH_KEY
```
```bash
podman unshare chown 1000:1000 ~/data/containers/code-server/ssh/config
```
### Podman Image
#### Podman containerfile
Code-server supports various module for it. Git and ansible will be used in this project.
- file:
- ~/data/config/containers/code-server/containerfile-code-server-4.105.1
- ~/data/config/containers/code-server/root_ca.crt
```containerfile
FROM codercom/code-server:4.105.1
USER root
RUN export SUDO_FORCE_REMOVE=yes && \
apt-get update && \
apt-get install -y --no-install-recommends git ansible curl jq age gnupg && \
apt-get purge -y --auto-remove sudo && \
apt-get clean
RUN curl -LO https://github.com/getsops/sops/releases/download/v3.11.0/sops-v3.11.0.linux.amd64 && \
mv sops-v3.11.0.linux.amd64 /usr/local/bin/sops && \
chmod +x /usr/local/bin/sops && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
COPY ./root_ca.crt /usr/local/share/ca-certificates/root_ca.crt
RUN update-ca-certificates
USER coder
```
#### Podman image build
```bash
podman build -t code-server:4.105.1 -f ~/data/config/containers/code-server/containerfile-code-server-4.105.1 . && podman image prune -f
# Delete pure code-server
```
### Quadlet
- File:
- ~/data/config/containers/code-server/code-server.container
```ini
# ~/data/config/containers/code-server/code-server.container
# ~/.config/containers/systemd/code-server.container
[Quadlet]
DefaultDependencies=false
[Unit]
Description=Code-Server
After=caddy-dev.service
Wants=caddy-dev.service
[Container]
Image=localhost/code-server:4.105.1
ContainerName=code-server
HostName=code-server
# CrowdSec uses 8080 port
PublishPort=8000:8080/tcp
Volume=%h/data/containers/code-server/workspace:/home/coder/workspace:rw
Volume=%h/data/containers/code-server/config:/home/coder/.config:rw
Volume=%h/data/containers/code-server/local:/home/coder/.local:rw
Volume=%h/data/containers/code-server/ssh:/home/coder/.ssh:rw
Environment="TZ=Asia/Seoul"
# when you needs root permission, you have to access the container via dev server's command 'poman exec -it code-server'
Secret=CODESERVER_SSH_KEY,target=/run/secrets/CODESERVER_SSH_KEY
Label=diun.enable=true
Label=diun.watch_repo=true
# This label need configuration on `diun.yml`
Label=diun.regopt=code-server-source
[Install]
WantedBy=default.target
```
#### Create systemd `.service` file
```bash
# linger has to be activated
ln -s ~/data/config/containers/code-server/code-server.container ~/.config/containers/systemd/code-server.container
systemctl --user daemon-reload
```
#### Enable and start service
```bash
systemctl --user start code-server.service
```
#### Disable password
```bash
nano ~/data/containers/code-server/config/code-server/config.yaml
# bind-addr: 127.0.0.1:8080
# auth: none <- edit this as `none` from `password`
# password: <- remove this part
# cert: false
```
#### Set default workspace
- Setting:Profile:Default:Folders&workspaces
- Add Folder > /home/coder/workspace
- Setting:Settings:Workbench:Settings Editor
- Terminal > Integrated: Gpu Acceleration: off
- Edit in settings.json
```json
{
    "workbench.settings.applyToAllProfiles": [
    ],
    "workbench.colorCustomizations": {
      "terminal.background": "#0C0C0C",
      "terminal.foreground": "#CCCCCC"
    },
    "files.associations": {
        "*.container": "ini",
        "*.service": "ini",
        "*.timer": "ini",
        "containerfile*": "dockerfile"
    }
}
```
#### Verification
docs/archives/2025-12/09_application/09_01_app_vm.md
+253
View File
@@ -0,0 +1,253 @@
Tags: #os, #configuration, #application, #virtualization
## Preparation
### Set DHCP reservation and DNS record
#### Set DHCP reservation on KEA DHCP in OPNsense
Following [here](05_07_opnsense_kea.md)
- Services:Kea DHCP:Kea DHCPv4:Reservations - \[+\]
- Subnet: 192.168.10.0/24
- IP address: 192.168.10.14
- MAC address: 0A:49:6E:4D:04:00
- Hostname: app
- Description: app
- `save`
#### Set DNS records in BIND
Following [here](../06_network/06_03_net_bind.md).
- net server
- file:
- ~/data/containers/bind/lib/db.ilnmors.internal
- ~/data/containers/bind/lib/db.10.168.192.in-addr.arpa
```ini
# db.ilnmors.internal
# ...
app IN A 192.168.10.14
# ...
# db.10.168.192.in-addr.arpa
# ...
14 IN PTR app.ilnmors.internal.
# ...
```
### Create VM template
```bash
virt-install \
--boot uefi \
--name app \
--os-variant debian13 \
--vcpus 4 \
--memory 12288 \
--location /home/vmm/data/vms/images/debian-13.0.0-amd64-netinst.iso \ # For serial installing, use `--location` instead of `--cdrom`
--disk pool=vm-images,size=258,format=qcow2,discard=unmap \
--network network=ovs-lan-net,portgroup=vlan10-access,model=virtio,mac=0A:49:6E:4D:04:00 \ # Use designated ovs port group
--graphics none \
--console pty,target_type=serial \
--extra-args "console=ttyS0,115200"
# After enter this command, then the console start automatically
# Remove all annotation before you make the sh file.
```
### Debian installing
- Following [here](../03_common/03_01_debian_configuration.md) to install Debian.
- Debian installer supports serial mode regardless getty@ttyS0 service is enabled or not.
- Following [here](../03_common/03_02_iptables.mc) to set iptables.
- Following [here](../03_common/03_04_crowdsec.md) to set CrowdSec
#### Serial console setting
After installation, use `ctrl + ]` to exit console. Before setting getty@ttyS0, you can't use serial console to access VM. Therefore, use IP address set on installation, and connect net server via ssh first, following the step to enable the getty.
#### btrfs RAID setting
Following [here](03_06_btrfs.md) how to use btrfs
- directory: /home/app/hdd
```bash
# Make the directory, RAID partition will be mounted
mkdir /home/app/hdd
# check btrfs-progs package
sudo apt list --installed | grep btrfs-progs
# btrfs-progs/stable,now 6.14-1 amd64 [installed]
# Check the disk status
lsblk -o NAME,PTTYPE,FSTYPE,SIZE,MOUNTPOINT
# - /dev/sda: Physical slot 2
# - /dev/sdb: Physical slot 1
# - /dev/sdc: Physical slot 4
# - /dev/sdd: Physical slot 3
# If you want to manage the partition or disk, then use fsdisk.
sudo fdisk "$DIVICE_PATH"
> n # create the new parition
> 1 # Partition number
> Default # First Sector
> Default # Last Sectort
> w # write the new partition
lsblk -o NAME,PTTYPE,FSTYPE,SIZE,MOUNTPOINT
# - /dev/sda1: partition of slot 2
# - /dev/sdb1: partition of slot 1
# - /dev/sdc1: partition of slot 4
# - /dev/sdd1: partition of slot 3
# btrfs RAID10 volume creation
# -d: the way data store, -m: the way metadata store, -L: create label
sudo mkfs.btrfs -d raid10 -m raid10 -L hdd /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
# check the RAID10 volume, it shows label and uuid.
sudo btrfs filesystem show
lsblk -f
# Mount RAID10 volume permanently
sudo nano /etc/fstab
# # btrfs RAID10 storage pool; mount option, compress=zstd: realtime compression. autodefrag: conduct auto defragmentation
# LABEL=hdd /home/app/hdd btrfs defaults,compress=zstd,autodefrag 0 0
# release the fs
sudo systemctl daemon-reload
sudo mount -a
# check the mount
df -h
/dev/sda1 3.7T /home/app/hdd
# Scrubbing btrfs
sudo btrfs scrub start /home/app/hdd
sudo btrfs scrub status /home/app/hdd
```
> btrfs RAID doesn't use fixed hdds pair. It uses flexible chunck unit data management. Therefore user doesn't have to know about which disks are the pair, just change the disk which is broken.
#### btrfs scrub on systemd
File:
- ~/data/config/services/btrfs-scrub/btrfs-scrub.service
- ~/data/config/services/btrfs-scrub/btrfs-scrub.timer
```ini
# ~/data/config/services/btrfs-scrub/btrfs-scrub.service
# /etc/systemd/system
[Unit]
Description=BTRFS Scrub for /home/app/hdd
After=home-app-hdd.mount
Wants=home-app-hdd.mount
[Service]
Type=oneshot
ExecStart=btrfs scrub start /home/app/hdd
Nice=19
IOSchedulingClass=idle
# Nice: CPU priority; -20: highest, 0: default, 19: lowest
# IOSchedulingClass: Disk priority; realtime: highest, best-effort: default, idle: lowest
```
```ini
# ~/data/config/services/btrfs-scrub/btrfs-scrub.timer
# /etc/systemd/system
[Unit]
Description=Run BTRFS scrub for /home/app/hdd monthly
[Timer]
OnCalendar=*-*-01 03:00:00
Persistent=true
# Persistent=true: If the service couldn't run because of some reasons, it execute the service immediately when it is possible
[Install]
WantedBy=timers.target
```
```bash
sudo ln -s ~/data/config/services/btrfs-scrub/btrfs-scrub.service /etc/systemd/system/btrfs-scrub.service
sudo ln -s ~/data/config/services/btrfs-scrub/btrfs-scrub.timer /etc/systemd/system/btrfs-scrub.timer
sudo systemctl daemon-reload
sudo systemctl enable --now btrfs-scrub.timer
```
### Modify VM template settings
After getty setting, shutdown app vm with `shutdown` in VM or `sudo virsh shutdown app` in hypervisor to turn off vm first.
```bash
virsh edit app
```
```xml
<!-- app -->
...
</vcpu>
<cputune>
<shares>2048</shares>
</cputune>
<!-- cpu priority - 1024: default/2048: high/512: low -->
<!--
<disk type='file' device='cdrom'>
...
</disk>
# Remove booting disk
-->
```
```bash
virsh dumpxml app > ~/data/config/vms/dumps/app.xml
virsh start app && virsh console app
# Start app server with console
```
### Common setting
- app.service
```ini
# ~/data/config/services/app.service
# ~/.config/systemd/user/app.service
[Unit]
Description=app Auto Booting
After=network-online.target
Wants=network-online.target
Requires=opnsense.service
[Service]
Type=oneshot
# Maintain status as active
RemainAfterExit=yes
# CrowdSec should be set
ExecStartPre=%h/data/config/scripts/wait-for-it.sh 192.168.10.1:8080 -t 0
ExecStartPre=%h/data/config/scripts/wait-for-it.sh 192.168.10.11:53 -t 0
ExecStartPre=%h/data/config/scripts/wait-for-it.sh 192.168.10.12:9000 -t 0
ExecStartPre=/bin/bash -c "sleep 15"
# Run the service
ExecStart=/usr/bin/virsh -c qemu:///system start app
# Stop the service
ExecStop=/usr/bin/virsh -c qemu:///system shutdown app
[Install]
WantedBy=default.target
```
```bash
ln -s ~/data/config/services/app.service ~/.config/systemd/user/app.service
systemctl --user daemon-reload
systemctl --user enable app.service
systemctl --user start app.service
```
docs/archives/2025-12/console.md
+360
View File
@@ -0,0 +1,360 @@
# Console client
Tags: #os, #windows, #virtualization, #wsl, #vscode
## Preparation
### WSL
#### WSL command
Run the commands in PowerShell or CMD. Installation and uninstallation process needs admin privileges.
```PowerShell
# --- Install and setup ---
# Activate WSL (First time only)
wsl --install
# Install specific OS
wsl --install -d Debian
# Check the list
wsl --list --online
# Check the version
wsl -l -v
# --- Run and manage ---
# Run WSL
wsl -d Debian # -u root # run with root
# Shutdown WSL
# This is needed when the configuration is changed
wsl --shutdown
# Shutdown specific version
wsl --terminate Debian
# --- Backup and restore ---
# Backup WSL
wsl --export Debian C:\backups\wsl.tar
# Import WSL
wsl --import Debian C:\WSL\Debian C:\backups\wsl.tar
# Open the linux directory on windows explorer
# bash
# explorer.exe .
# Windows explorer
# \\wsl$ on the windows explorer
# --- Reset or inactivate ---
# Reset the specific version
wsl --unregister Debian
# Inactive WSL
wsl --uninstall
```
#### WSL configuration
##### Installation
```PowerShell
# Activate WSL and install Debian
wsl --install -d Debian
# Enter new UNIX username: debian
# Enter new password: debian
```
##### Configuration
- `Win`:Windows Linux Subsystem Configuration \(GUI\)
- Processor and memory
- Processor: 4
- Memory: 4096MB
- Swap: 0
- Filesystem
- Basic VHD: 32768MB
- Networking
- Mode: Mirrored
#### WSL Start
```PowerShell
# Start WSL
wsl -d Debian
# User and group configuration
sudo groupadd -g 2000 svadmins
sudo useradd -u 2999 -g svadmins -G sudo -c "Console Client" -m -d /home/console -s /bin/bash console
sudo passwd console
# New password: random string
exit
# PowerShell
wsl --shutdown
wsl -d Debian -u console
# Delete default account
sudo userdel -r debian
# Set default user
sudo nano /etc/wsl.conf
# ...
# [user]
# default=console
exit
# PowerShell
wsl --shutdown
wsl -d Debian
# Check `console` login
# Create the directory for VS Code
mkdir workspace && chmod 700 workspace
```
### VS Code
#### Installation
- Site: https://code.visualstudio.com/
- Download for Windows
- Execute the installation file
#### Configuration
- Extensions\(`Ctrl` + `shift` + `x`\):WSL
- Install WSL by Microsoft
- Remote Explorer:Debian:Connect in Current Windows
- `Ctrl` + `k` and `Ctrl` + `t` for theme
- Dark Modern
- `Ctrl` + `k` and `Ctrl` + `o` for `open folder`
- /home/console/workspace/
- Do you trust the authors of the files in this folder - `Yes, I trust the authors`
- `Ctrl` + `Shift` + `` ` `` for `open terminal`
## Bastion host
### Directory structures
Use `mkdir` to make these directories.
- ~/workspace/homelab/data/
- utils
- common - wait-for-it.sh, sops, etc... 
- \[server_name\]/\[bin_name\] - ddns, init_db, etc ... 
- servers
- os/\[iso or img files for installation\]
- \[server_name\]/\[service_name; iptables, interface, ssh, vfio, etc..\] - rules.v4, sshd_config, etc... 
- services
- \[server_name\]/\[services_name\]
- *.containers or *.service (systemd files)
- config - services configuration (named.conf, etc; !No live data files like DB file or media file. Only configuration files based on text or binary files.) 
- secrets - secret_scripts, secret.yaml (central secret management)
- ~/workspace/homelab/docs 
- library
- archives 
- before_bastion_host/current_documents_and_directories
- references
- techs
- current_common_documents 
- theories
- current_theory_documents 
- images 
 - media 
 - etc. 
- plans 
- plan.md 
- milestone.md 
- infrastructures
- common
- debian_configuration.md (OS, network, uid/gid, packages) 
- deployment.md 
- security_policies.md (iptables, crowdsec) 
- data_polices.md (storage, backup, database)
- \[server_name\]
- \[server_name\].md - virtual hardware, security, services, etc...
- \[services_name\].md
### Packages
- External binary packages are located in here
- ~/workspace/homelab/data/bin/common
```bash
sudo apt update && sudo apt upgrade
# Packages from repository
sudo apt install gnupg acl curl jq age git openssh-client
# Git config
git config --global user.name "il"
git config --global user.email "il@ilnmors.internal"
# Sops
## Sops for amd processor (N150)
curl -LO https://github.com/getsops/sops/releases/download/v3.11.0/sops-v3.11.0.linux.amd64
## Sops for arm processor (Snapdragon Plus)
curl -LO https://github.com/getsops/sops/releases/download/v3.11.0/sops-v3.11.0.linux.arm64
mkdir -p ~/workspace/homelab/data/bin/common && chmod 700 ~/workspace/homelab/data/bin/common
mv sops-v3.11.0.linux.amd64 sops-v3.11.0.linux.arm64 ~/workspace/homelab/data/bin/common/
sudo cp ~/workspace/homelab/data/bin/common/sops-v3.11.0.linux.arm64 /usr/local/bin/sops
sudo chmod +x /usr/local/bin/sops
# wait-for-it.sh
curl -LO https://github.com/vishnubob/wait-for-it/blob/master/wait-for-it.sh
mv wait-for-it.sh ~/workspace/homelab/data/bin/common/
# acme.sh
curl -LO https://github.com/acmesh-official/acme.sh/blob/master/acme.sh
mv acme.sh ~/workspace/homelab/data/bin/common/
```
### Secret management
- Files:
- ~/workspace/homelab/data/secrets/secret.yaml
- ~/workspace/homelab/data/secrets/.sops.yaml
- ~/workspace/homelab/data/secrets/age-key.gpg
- ~/workspace/homelab/data/secrets/edit_secret.sh
- ~/workspace/homelab/data/secrets/extract_secret.sh
#### Apply the secrets
- Server: console
##### Generate and encrypt age key
```bash
# Generate the key for sops
age-keygen -o ~/workspace/homelab/data/secrets/age-key
# # created: 2025-10-17T13:30:00Z
# # public key: age1ql3z7h0cfscg......
# AGE-SECRET-KEY-1.....
# Public key is printed when key generated
gpg --symmetric age-key && rm age-key
> GPG password: password
nano ~/workspace/homelab/data/secrets/.sops.yaml
```
##### Key value setting for sops
```yaml
# ~/workspace/homelab/data/secrets/.sops.yaml
creation_rules:
  - path_regex: secret\.yaml$
    age: [public_key value; age~~~]
```
##### Mnagement secret
```bash
# Create secret
cd ~/workspace/homelab/data/secrets
nano secret.yaml
# Replace the file as secret file
sops --encrypt --in-place secret.yaml
# edit secret.yaml
./edit_secret.sh secret.yaml
# Create secret files in each server
./extract_secret.sh secret.yaml [-n] (-e|-f $ENV) > $TMP_PATH/tmp_secret
# deploy the tmp_secret to server to /run/user/$UID/filename
scp $TMP_PATH/tmp_secret [server]:/run/user/$TARGET_UID/filename
# `<< 'EOF'` sends string itself
# `<< EOF` sends string after interpreting
ssh [server] << 'EOF'
    sudo mv /run/user/$UID/filename /etc/secrets/$UID/secret_file
    rm -rf /run/user/$UID/filename
    sudo chown $UID:root /etc/secrets/$UID/secret_file
    sudo chmod 400 /etc/secrets/$UID/secret_file
EOF
rm -rf $TMP_PATH/tmp_secret
# Podman secret in each server
./extract_secret.sh secret.yaml [-n] -f $ENV | ssh sv "podman secret create $ENV -"
```
#### Usage of podman secret
```container
#...
#...
[Container]
# ..
Secret=env,type=env,target=env
Secret=app,target=/run/secrets/app
```
### ssh configuration
#### ssh key gen
```bash
mkdir -p ~/.ssh && chmod 700 ~/.ssh
ssh-keygen -t ed25519 -f ~/.ssh/id_console -C "il@ilnmors.internal"
# Add private key value to ~/workspace/homelab/data/secret/secret.yaml with sops
## # console ssh public key:
## # ed25519 ~~~~ il@ilnmors.internal
## # console ssh private key
## CONSOLE_SSH_PRIVATE_KEY: |
## ----BEGIN----
## ...
## ----END----
sudo mkdir -p /etc/secrets/2999 # $UID of `console`
sudo chown root:root /etc/secrets && sudo chmod 711 /etc/secrets
sudo chown console:root /etc/secrets/2999 && sudo chmod 500 /etc/secrets/2999
sudo mv ~/.ssh/id_console /etc/secrets/2999/ && sudo chown console:root /etc/secrets/2999/id_console && sudo chmod 400 /etc/secrets/2999/id_console
```
#### ssh key config
```ini
# ~/.ssh/config
Host vmm
HostName [vmm ip from ncpa.cpl's temporary dhcp ip address]
User vmm
IdentityFile /etc/secrets/2999/id_console
# Host vmm
# HostName 192.168.10.10
# User vmm
# IdentityFile /etc/secrets/2999/id_console
# Host net
# HostName 192.168.10.11
# User net
# IdentityFile /etc/secrets/2999/id_console
# Host auth
# HostName 192.168.10.12
# User auth
# IdentityFile /etc/secrets/2999/id_console
# Host dev
# HostName 192.168.10.13
# User dev
# IdentityFile /etc/secrets/2999/id_console
# Host app
# HostName 192.168.10.14
# User app
# IdentityFile /etc/secrets/2999/id_console
```
docs/archives/2025-12/scripts.md
+452
View File
@@ -0,0 +1,452 @@
```bash
#!/bin/bash
# edit_secret.sh /path/of/secret
set -e
KEY_PATH="$HOME/workspace/homelab/data/secrets"
TMP_PATH="/run/user/$UID"
SECRET_FILE="$1"
# Usage function
usage() {
    echo "Usage: $0 \"/path/of/secret/file\"" >&2
    exit 1
}
# log function
log()
{
    local text="$1"
    echo -e "$(date "+%Y-%m-%d %H:%M:%S"): [edit_script] $text" >&2
}
# Secret file check
if [ -z "$SECRET_FILE" -o ! -f "$SECRET_FILE" ]; then
    log "Error: Secret file path is needed"
    usage
fi
# age-key file check
if [ ! -f "$KEY_PATH/age-key.gpg" ]; then
    log "Error: There is no key file"
    exit 1
fi
# Dependency check
if ! command -v sops >/dev/null; then
    log "Error: sops package is needed"
    exit
fi
if ! command -v gpg >/dev/null; then
    log "Error: gnupg package is needed"
    exit
fi
# Delete password file after script certainly
cleanup() {
    if [ -f "$TMP_PATH/age-key" ]; then
        log "Notice: age-key was deleted"
        rm -f "$TMP_PATH/age-key"
    fi
}
trap cleanup EXIT
# Get GPG password from prompt
echo -n "Enter GPG passphrase: " >&2
read -s GPG_PASSPHRASE
echo "" >&2
# Decrypt age-key on memory
echo "$GPG_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
--output "$TMP_PATH/age-key" \
--decrypt "$KEY_PATH/age-key.gpg" && \
chmod 600 "$TMP_PATH/age-key"
unset GPG_PASSPHRASE
# Check the decrypted key on memory
if [ ! -f "$TMP_PATH/age-key" ]; then
        log "Error: Decrypted key file does not exist"
        exit 1
fi
# kill the gpg session
gpgconf --kill gpg-agent
# Open sops editor
SOPS_AGE_KEY_FILE="$TMP_PATH/age-key" sops "$SECRET_FILE"
rm -f "$TMP_PATH/age-key" >&2
exit 0
```
```bash
#!/bin/bash
# extract_secret.sh /path/of/secret [-n] (-f|-e <value>)
set -e
KEY_PATH="$HOME/workspace/homelab/data/secrets"
TMP_PATH="/run/user/$UID"
SECRET_FILE=$1
VALUE=""
TYPE=""
NEWLINE="true"
# Remove $1 and shift $(n-1) < $n
shift
# usage() function
usage() {
        echo "Usage: $0 \"/path/of/secret/file\" [-n] (-f|-e \"yaml section name\")" >&2
        echo "-n: remove the newline" >&2
        echo "-f <type name>: Print secret file" >&2
        echo "-e <type name>: Print secret env file" >&2
        exit 1
}
# log() function
log()
{
    local text="$1"
    echo -e "$(date "+%Y-%m-%d %H:%M:%S"): [extract_script] $text" >&2
}
while getopts "f:e:n" opt; do
    case $opt in
        f)
            VALUE="$OPTARG"
            TYPE="FILE"
            ;;
        e)
            VALUE="$OPTARG"
            TYPE="ENV"
            ;;
        n)
            NEWLINE="false"
            ;;
        \?) # unknown options
            log "Invalid option: -$OPTARG"
            usage
            ;;
        :) # parameter required option
            log "Option -$OPTARG requires an argument."
            usage
            ;;
    esac
done
# Get option and move to parameters
shift $((OPTIND - 1))
# Check necessary options
if [ -z "$SECRET_FILE" -o ! -f "$SECRET_FILE" ]; then
    log "Error: secret file path is required"
    usage
fi
if [ -z "$TYPE" ]; then
        log "Error: -f or -e option requires"
        usage
fi
# age-key file check
if [ ! -f "$KEY_PATH/age-key.gpg" ]; then
    log "Error: There is no key file"
    exit 1
fi
# Dependency check
if ! command -v sops >/dev/null; then
    log "Error: sops package is needed"
    exit
fi
if ! command -v gpg >/dev/null; then
    log "Error: gnupg package is needed"
    exit
fi
# Delete password file after script certainly
cleanup() {
    if [ -f "$TMP_PATH/age-key" ]; then
        log "Notice: age-key was deleted"
        rm -f "$TMP_PATH/age-key"
    fi
}
trap cleanup EXIT
echo -n "Enter GPG passphrase: " >&2
read -s GPG_PASSPHRASE
echo "" >&2
echo "$GPG_PASSPHRASE" | gpg --batch --yes --passphrase-fd 0 \
--output "$TMP_PATH/age-key" \
--decrypt "$KEY_PATH/age-key.gpg" && \
chmod 600 "$TMP_PATH/age-key"
unset GPG_PASSPHRASE
if [ ! -f "$TMP_PATH/age-key" ]; then
        log "Error: Decrypted key file does not exist"
        exit 1
fi
gpgconf --kill gpg-agent
if [ "$TYPE" == "FILE" ]; then
        if RESULT=$(SOPS_AGE_KEY_FILE="$TMP_PATH/age-key" sops --decrypt --extract "[\"$VALUE\"]" --output-type binary "$SECRET_FILE") ; then
                if [ "$NEWLINE" == "true" ]; then
                    echo "$RESULT"
                else
                    echo -n "$RESULT"
                fi
                exit 0
        else
                log "Error: SOPS extract error"
                exit 1
        fi
fi
if [ "$TYPE" == "ENV" ]; then
        if RESULT=$(SOPS_AGE_KEY_FILE="$TMP_PATH/age-key" sops --decrypt --extract "[\"$VALUE\"]" --output-type dotenv "$SECRET_FILE") ; then
                if [ "$NEWLINE" == "true" ]; then
                    echo "$RESULT"
                else
                    echo -n "$RESULT"
                fi
                exit 0
        else
                log "Error: SOPS extract error"
                exit 1
        fi
fi
```
docs/notes/.gitkeep
View File
docs/runbook/00-operate.md
+104
View File
@@ -0,0 +1,104 @@
# Operation
## Migration backup
When the migration is decided, the manual backup after shutting all services down is necessary. Run the DB dump backup and kopia backup manually before reset the system.
## Windows
- Following the process step.
- [01-windows.md](./01-windows.md)
## Certificates and wireguard
- Following the process two steps.
- [02-certificates.md](./02-certificates.md)
- [03-wireguard.md](./03-wireguard.md)
## console
### Data restore
Only when kopia repository exists.
```bash
kopia repository connect --override-username="console" --override-hostname="console.ilnmors.internal"
kopia snapshot list --all
# mount volumes
kopia mount $CONSOLE_SNAPSHOT_ID ~/workspace/homelab/volumes/console
kopia mount $INFRA_SNAPSHOT_ID ~/workspace/homelab/volumes/infra
kopia mount $APP_SNAPSHOT_ID ~/workspace/homelab/volumes/app
# Set initiating db dump files
cp ~/workspace/homelab/volumes/infra/cluster/cluster.sql ~/workspace/homelab/config/services/containers/infra/postgresql/init/cluster.sql
# ...
```
### Provisioning
Ansible playbooks should be declarative. This won't contain complex branch logics \(Declarative over imperative\). Playbooks describes what should be there, not how to. The basic rule is manual destroy and auto reprovisioning.
#### vmm and fw
- Following the process step.
- [04-hypervisor.md](./04-hypervisor.md)
```bash
# Sign on vmm's host ssh keys and set libvirt environment # Ignore fingerprint just once
cd ~/workspace/homelab/ansible
# Disconnect all internet connection except LAN between vmm
ansible-playbook playbooks/vmm/site.yaml --tags "init" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
# Create fw
ansible-playbook playbooks/vmm/create_vm.yaml --tags "fw"
# Make it sure disable other NIC (Wifi or etc) before run playbooks, and set gateway
ansible-playbook playbooks/fw/site.yaml --tags "init" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' # When nftables is changed, crowdsec handlers work. However, when it is in initiating, handler is failed. It is normal work.
ansible-playbook playbooks/fw/site.yaml --tags "site"
# Set vmm after air-gap
ansible-playbook playbooks/vmm/site.yaml --tags "site"
```
## External hardware on CLIENT
- Following the process two steps.
- [05-hardwares.md](./05-hardwares.md)
- [06-kopia.md](./06-kopia.md)
## Other vms
```bash
# Kopia mount
## Create Kopia password file
touch /etc/secret/$CONSOLE_UID/kopia_password
### Add kopia.user.console value from secrets.yaml
## Connect repository
KOPIA_PASSWORD="$(cat /etc/secrets/$CONSOLE_UID/kopia_password)" \
/usr/bin/kopia repository connect server \
--url=https://nas.ilnmors.internal:51515 \
--override-username=console \
--override-hostname=console.ilnmors.internal
## mount kopia directories
kopia mount $INFRA_SNAPSHOT_ID ~/workspace/data/volumes/infra
kopia mount $APP_SNAPSHOT_ID ~/workspace/data/volumes/app
## bringing sql dump files to init directory
cp ~/workspace/data/volumes/infra/cluster/$FILE_NAME ~/workspace/config/services/containers/infra/postgresql/init/pg_cluster.sql
## repeate for every service
# Create vms (Possible tags = ["fw", "infra", "auth", "app"])
ansible-playbook playbooks/vmm/create_vm.yaml --tags "$VM_NAME"
# Sign on vms' host ssh keys # Ignore fingerprint just once
ansible-playbook playbooks/$VM_NAME/site.yaml --tags "init" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
ansible-playbook playbooks/$VM_NAME/site.yaml --tags "site"
# app
ansible-playbook playbooks/app/site.yaml --tags "init" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
## Rsync app data
rsync -avz --numeric-ids -e ssh --rsync-path="sudo rsync" ~/workspace/homelab/volumes/kopia/app/ app@app.ilnmors.internal:/home/app/data/
# --numeric-ids: set owner and groups based on uid and gid
# -e ssh --rsync-path="sudo rsync": run rsync as root permission, sudo password needed
# Maybe rsync-path is run on background, it might not print prompt for password. Just enter the password
ansible-playbook playbooks/app/site.yaml --tags "site"
# Create console snapshot
kopia snapshot create ~/workspace
```
docs/runbook/01-windows.md
+255
View File
@@ -0,0 +1,255 @@
# Windows prerequisite
## Installation
- Language: English (US)
- Region: Korea
- Keyboard: US
- Internet connection
- Account setting: DOS prompt (Shift + F10)
```dos
start ms-cxh:localonly
```
## Initial configuration
### Time and language
- Preferred Languages: Add a Language: Korean
- Don't set additional options
- Typing: Advanced Keyboard setting: Override for default input method: Korean: Microsoft IME
### Set the password
- Settings: Accounts: Sign-in options
- Password
- PIN
- Windows Hello
### Disable unused functions
```PowerShell
# Administrator privileges are required
## Disable Recall
DISM /Online /Disable-Feature /FeatureName:Recall /NoRestart
## Disable Copilot
reg add "HKCU\Software\Policies\Microsoft\Windows\WindowsCopilot" /v TurnOffWindowsCopilot /t REG_DWORD /d 1 /f
reg add "HKLM\Software\Policies\Microsoft\Windows\WindowsCopilot" /v TurnOffWindowsCopilot /t REG_DWORD /d 1 /f
## Delete programs
Get-AppxPackage *bingnews* | Remove-AppxPackage
Get-AppxPackage *bingweather* | Remove-AppxPackage
Get-AppxPackage *gethelp* | Remove-AppxPackage
Get-AppxPackage *tips* | Remove-AppxPackage
Get-AppxPackage *feedbackhub* | Remove-AppxPackage
Get-AppxPackage *solitairecollection* | Remove-AppxPackage
Get-AppxPackage *windowscommunicationsapps* | Remove-AppxPackage
Get-AppxPackage *outlookforwindows* | Remove-AppxPackage
Get-AppxPackage *people* | Remove-AppxPackage
Get-AppxPackage *clipchamp* | Remove-AppxPackage
Get-AppxPackage *todos* | Remove-AppxPackage
Get-AppxPackage *quickassist* | Remove-AppxPackage
Get-AppxPackage *windowsmaps* | Remove-AppxPackage
Get-AppxPackage *xbox* | Remove-AppxPackage
Get-AppxPackage *gamingapp* | Remove-AppxPackage
## Disable telemetry
Stop-Service DiagTrack
Set-Service DiagTrack -StartupType Disable
Stop-Service dmwappushservice
Set-Service dmwappushservice -StartupType Disable
## Compact OS configuration
compact /compactos:always
```
### Delete programs
- Add or remove programs
- Copilot
- Family
- Microsoft 365 (Other languages; except en-us, ko-kr)
- Microsoft 365 copilot
- Microsoft edge game assist
- Microsoft Onedrive
- Microsoft Teams
- Speech pack
### Edge configuration
- Settings: System and performance: System
- Disable All options
### Explorer configuration
- Settings: Privacy
- Disable All options
### Login on Microsoft account
sign in on app only
- surface app
- microsoft office 365
### Install Microsoft app
- Microsoft PC manager
### Firewall configuration
- WindowsDefender Firewall:Inbound Rules:
- File and Printer Sharing (Echo Request - ICMPv4-In) - Profile: Private, Public
- General: \[x\] Enable
- Scope: 192.168.1.0/24, 192.168.10.0/24, 192.168.99.0/24
- File and Printer Sharing (Echo Request - ICMPv6-In) - Profile: Private, Public
- General: \[x\] Enable
- Scope: fd00::/8
- Apply
### NIC VLAN configuration
<!-- There is no physical managed switch, only hub. It is necessary to set so that filter vlan tag from the dummy hub -->
- Device Manager:Network Adapters:Surface network adpaters
- Advanced:Priority & VLAN: Priority & VLAN disable
## Set WSL2
### Create wsl config
- C:\Users\$USERNAME\.wslconfig
```ini
[wsl2]
processors=4
memory=4294967296
swap=0
defaultVhdSize=34359738368
networkingMode=Mirrored
```
### Installation
```PowerShell
# Run with administrator's authority
# Install wsl
wsl --install -d Debian
# Enter new UNIX username: debian
# Enter new password: debian
wsl --shutdown
# --- Uninstall Debian ---
# wsl --unregister Debian
# wsl --uninstall
# ------
```
### WSL Debian configuration
```bash
# PowerShell
wsl -d Debian
# bash
## User create
sudo groupadd -g 2000 svadmins
sudo useradd -u 2999 -g svadmins -G sudo -c "Console Client" -m -d /home/console -s /bin/bash console
sudo passwd console
# New password: [password]
exit
# PowerShell
wsl --shutdown
wsl -d Debian -u console
# bash
## User delete and set wsl.conf
sudo userdel -r debian
echo '[user]' | sudo tee -a /etc/wsl.conf
echo 'default=console' | sudo tee -a /etc/wsl.conf
echo '[network]' | sudo tee -a /etc/wsl.conf
echo 'generateHosts = false' | sudo tee -a /etc/wsl.conf
exit
wsl --shutdown
wsl -d Debian
## package
sudo apt update && sudo apt upgrade
sudo apt install curl jq gnupg git fuse3
sudo apt install ansible-core ansible-lint --no-install-recommends
ansible-galaxy collection install community.libvirt
ansible-galaxy collection install community.general
ansible-galaxy collection install ansible.posix
## SOPS (arm64)
## Check for latest release: Current version: 3.12.1
## x86_64 url: https://github.com/getsops/sops/releases/download/v3.12.1/sops_3.12.1_amd64.deb
curl -L -o sops_3.12.1.deb https://github.com/getsops/sops/releases/download/v3.12.1/sops_3.12.1_arm64.deb
sudo apt install ./sops_3.12.1.deb
rm -rf ./sops_3.12.1.deb
mkdir ~/workspace
```
### VS Code
#### VS Code Download
- Download from https://code.visualstudio.com/Download#
- `User Installer Arm64`
#### VS Code configuration
- WSL extension\(`Ctrl + shift + x`\)
- Install `WSL` by Microsoft
- Remote Explorer:Debian:Connect in Current Windows
- `Ctrl + k` and `Ctrl + o`
- Open folder: `/home/console/workspace`
- `` Ctrl + shift + ` `` for Terminal
- Extensions\(`Ctrl + shift + x`\)
- Install `Ansible` by RedHat
### Playbooks
```bash
# Ansible playbook
# Copy all files "workspace" directory to Debian via VS Code
cd ~/workspace/homelab/ansible
ansible-playbook playbooks/console/site.yaml --tags "init"
```
#### Terminal configuration
- Terminal: Setting: Startup
- Default profile
- Debian
- Profile: Debian:
- Icon
- "ms-appx:///ProfileIcons/{61c54bbd-c2c6-5271-96e7-009a87ff44bf}.png"
- Starting Directory
- ~
## Secret management
### Flow
- Secret data
- `secrets.yaml`
- encrypted by sops with age-key
- age-key
- `age-key.gpg` and `ansible/group_vars/all.yaml`
- encrypted by gpg and ansible vault with master key
- Master key
- The key which has above 40 characters containing upper and lower letters, numbers, and special letters
- managed by physical media \(Mind, MDisc, paper\) as file, string, and QR
- This value is never saved in server or console.
- Root CA \(including ssh CA\) must not be deployed.
- The tasks with root CA must be performed manually. The source of Trust is the most important in security.
- Intermediate CA can be deployed.
- Intermediate CA is operated as a live server.
- Intermediate CA can be revoked by the root CA
- Only encrypted intermediate CA's private key is deployed.
- CA server decrypts intermediate CA's private key in memory at runtime.
docs/runbook/02-certificates.md
+169
View File
@@ -0,0 +1,169 @@
# Certificates
Create and renew certificates are very important, and very barely executed. It is managed manually without ansible.
#### PKI CA signed offline
step-cli is installed by ansible playbook for console.
```bash
# Generate CA key password
openssl rand -base64 32 > /run/user/$UID/root_ca_password
openssl rand -base64 32 > /run/user/$UID/intermediate_ca_password
# Save the values in `secrets.yaml`
# Create CAs \(Key and cert)
# Root CA
step certificate create \
"ilnmors.internal Root CA" /run/user/$UID/root_ca.crt /run/user/$UID/root_ca.key \
--password-file /run/user/$UID/root_ca_password \
--profile root-ca \
--not-after 87600h
# Save the key and crt files content in `secrets.yaml`
# Intermediate CA
step certificate create \
"ilnmors.internal Intermediate CA" /run/user/$UID/intermediate_ca.crt /run/user/$UID/intermediate_ca.key \
--password-file /run/user/$UID/intermediate_ca_password \
--profile intermediate-ca \
--ca /run/user/$UID/root_ca.crt \
--ca-key /run/user/$UID/root_ca.key \
--ca-password-file /run/user/$UID/root_ca_password \
--not-after 43800h
# Save the key and crt files content in `secrets.yaml`
# fw
step certificate create \
"crowdsec.ilnmors.internal" /run/user/$UID/crowdsec.crt /run/user/$UID/crowdsec.key \
--profile leaf \
--san crowdsec.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
step certificate create \
"blocky.ilnmors.internal" /run/user/$UID/blocky.crt /run/user/$UID/blocky.key \
--profile leaf \
--san blocky.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
# infra
step certificate create \
"postgresql.ilnmors.internal" /run/user/$UID/postgresql.crt /run/user/$UID/postgresql.key \
--profile leaf \
--san postgresql.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
step certificate create \
"ldap.ilnmors.internal" /run/user/$UID/ldap.crt /run/user/$UID/ldap.key \
--profile leaf \
--san ldap.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
step certificate create \
"prometheus.ilnmors.internal" /run/user/$UID/prometheus.crt /run/user/$UID/prometheus.key \
--profile leaf \
--san prometheus.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
step certificate create \
"loki.ilnmors.internal" /run/user/$UID/loki.crt /run/user/$UID/loki.key \
--profile leaf \
--san loki.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
# DSM
step certificate create \
"nas.ilnmors.internal" /run/user/$UID/nas.crt /run/user/$UID/nas.key \
--profile leaf \
--san nas.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password
## Recreate leaf certificates
## update secrets.yaml
step certificate create \
"crowdsec.ilnmors.internal" /run/user/$UID/crowdsec.crt /run/user/$UID/crowdsec.key \
--profile leaf \
--san crowdsec.ilnmors.internal \
--ca /run/user/$UID/intermediate_ca.crt \
--ca-key /run/user/$UID/intermediate_ca.key \
--ca-password-file /run/user/$UID/intermediate_ca_password \
--not-after 21900h \
--insecure --no-password -f
# print
cat /run/user/$UID/crowdsec.key
cat /run/user/$UID/crowdsec.crt
# Verify
step certificate verify /run/user/$UID/test.crt --roots /run/user/$UID/root_ca.crt
# Inspect
step certificate inspect /run/user/$UID/test.crt
# validate date
sudo step certificate inspect --format json /run/user/$UID/test.crt | jq '.validity.end'
# margin date
echo "$(( ($(date -d 2028-07-17T03:50:10Z +%s) - $(date +%s)) / 60 / 60 / 24 ))"
# Delete temporary files
rm /run/user/$UID/root_ca*
rm /run/user/$UID/intermediate_ca*
rm /run/user/$UID/*.key
rm /run/user/$UID/*.crt
```
#### SSH CA
```bash
# Generate SSH CA
ssh-keygen -t ed25519 -f /run/user/$UID/id_local_ssh_ca -C "LOCAL_SSH_CA" -N ""
# Save the key and crt files content in `secrets.yaml`
echo @cert-authority *.ilnmors.internal "$(cat /run/user/$UID/id_local_ssh_ca.pub)" | sudo tee /etc/ssh/ssh_known_hosts >/dev/null && sudo chmod 644 /etc/ssh/ssh_known_hosts
# Signing HOST SSH crt by SSH CA key
ssh-keygen -s /run/user/$UID/id_local_ssh_ca \
-h \
-I "vmm" \
-n "vmm,vmm_init,vmm.ilnmors.internal,init.vmm.ilnmors.internal" \
/run/user/$UID/id_vmm_ssh_host.pub
# This process is automated by ansible
ssh-keygen -L -f /etc/ssh/ssh_host_ed25519_key-cert.pub
# Create SSH client key
ssh-keygen -t ed25519 -f /etc/secrets/$UID/id_console -C "il@ilnmors.internal" -N ""
# Signing SSH client crt by SSH CA key
ssh-keygen -s /run/user/$UID/id_local_ssh_ca \
-I "console" \
-n "vmm,fw,infra,auth,app" \
/etc/secrets/$UID/id_console.pub
# This process is automated by ansible
```
docs/runbook/03-wireguard.md
+23
View File
@@ -0,0 +1,23 @@
# WireGuard key get
```bash
wg genkey | tee /run/user/$UID/server_private.key | wg pubkey | tee /run/user/$UID/server_pub.key
wg genkey | tee /run/user/$UID/client_private.key | wg pubkey | tee /run/user/$UID/client_pub.key
wg genpsk | tee /run/user/$UID/preshared.key
# Save the key files content in `secrets.yaml`
# Client wireguard setting
[Interface]
PrivateKey = client_private.key
Address = 192.168.99.20/32, fd00:99::20/128
DNS = 192.168.10.2, fd00:10::2
[Peer]
PublicKey = server_pub.key
PresharedKey = preshared.key
AllowedIPs = 192.168.0.0/16, fd00::/8
Endpoint = vpn.ilnmors.com:11290
PersistentKeepalive = 25
```
docs/runbook/04-hypervisor.md
+162
View File
@@ -0,0 +1,162 @@
# Hypervisor \(vmm\)
Initiating hypervisor doesn't use ansible. Hypervisor is working on hardware itself, so there is a lot of possible variables like IOMMU id, MAC addresses, etc.
Hypervisor is initiated manually with the configuration files which are stored in USB or External HDD and, WAN connection is from ISP DHCP. All the files in `~/workspace/homelab/data/vmm_init/`
## Installing Debian on server
- BIOS:
- Check BIOS configuration; IOMMU/VT-d
- General:
- Language: English - English
- Location: Other > Asia > South Korea
- Locale: United State - en_US.UTF-8
- Keymap to use: American English
- Network:
- Auto Configuration: Using DHCP
- Server:
- Hostname: vmm
- Domain: ilnmors.internal
- User:
- Root Password: \[blank\]
- Full name for the new user: vmm
- User Name: bootstrap
- User Password: debian
- Partition setting: manual
- 512MiB - EFI system partition \(Booting flag: on\)
- 1GiB - Ext4 Journaling \(Mount: /boot)
- 800 GiB -LVM
- 64GiB: vmm-root - Ext4 Journaling \(Mount: /\)
- 700GiB: vmm-libvirt - Ext4 \(Mount: /var/lib/libvirt\)
- Debian package manager setting
- Scan extra installation media: no
- Mirror country: South Korea
- Archive mirror: deb.debian.org
- Proxy: \[blank\]
- Popularity-contest: no
- Installing packages setting
- \[\*\] SSH server
- \[\*\] Standard system utilities
### Initial configuration
Hypervisor operates pure L2 switch for fw and it never can access WAN without fw after initial configuration. This means, there is an air-gap which means hypervisor cannot access to WAN for a while \(from end of initial setting to the beginning of fw setting\).
Hypervisor operates on hardware. Hardware information is always uncertain, and it is set only once. Managing this process as IaC is over engineering.
```bash
# Mount USB on server
lsblk -l
# /dev/xxx # USB
sudo mkdir /mnt/usb
sudo mount /dev/xxx /mnt/usb
# Setting user and groups
sudo groupadd svadmins -g 2000
sudo useradd -u 2000 -g svadmins -G sudo -c "Hypervisor" -m -d /home/vmm -s /bin/bash vmm
# Installing packages
sudo apt update && sudo apt upgrade
sudo apt install -y \
acl curl jq crowdsec systemd-resolved \
qemu-system-x86 ksmtuned libvirt-daemon-system virt-top \
python3 python3-apt python3-libvirt python3-lxml
# Deploy ssh ca
sudo cp /mnt/usb/vmm/ssh/local_ssh_ca.pub /etc/ssh/
sudo chmod 644 /etc/ssh/local_ssh_ca.pub
sudo cp /mnt/usb/vmm/ssh/sshd_config.d/*.conf /etc/ssh/sshd_conifg.d/
sudo chmod 644 /etc/ssh/sshd_config.d/
# Deploy networkd configuration files
sudo cp /mnt/usb/vmm/network/* /etc/systemd/network/
sudo chmod 644 /etc/systemd/network/*
sudo cp /mnt/usb/vmm/sysctl.d/bridge.conf /etc/sysctl.d/bridge.conf
sudo chmod 644 /etc/sysctl.d/bridge.conf
# Check physical MAC address and modify .link file
ip addr
sudo nano /etc/systemd/network/eth0.link
sudo nano /etc/systemd/network/eth1.link
sudo systemctl disable networking.service
sudo systemctl enable systemd-networkd.service
# Deploy nftables config files
sudo cp /mnt/usb/vmm/nftables.conf /etc/nftables.conf
sudo chmod 700 /etc/nftables.conf
# Fix grub
sudo cp /mnt/usb/vmm/grub.d/iommu.cfg /etc/default/grub.d/iommu.cfg
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
## Intel CPU: intel_iommu=on, AMD CPU: amd_iommu=on
sudo chmod 644 /etc/default/grub.d/iommu.cfg
# Check PCIE ID and IOMMU group
lspci -nn | grep -i -e "SATA" -e "VGA"
# 00:02.0 VGA compatible controller [0300] ... [8086:46d4]
# PCI address: 00:02.0 / Device ID 8086:46d4
# 04:00.0 SATA controller [0106] ... [1b21:1064]
# PCI address: 04:00.00 / Device ID 1b21:1064
# Check the iommu group of devices
## VGA
readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
# ../../../../kernel/iommu_groups/12
ls /sys/kernel/iommu_groups/12/devices/
# 02:00.00
## SATA Controller
readlink /sys/bus/pci/devices/0000\:04\:00.0/iommu_group
# ../../../../kernel/iommu_groups/14
ls /sys/kernel/iommu_groups/14/devices/
# 04:00.00
# To passthrough device, it has to be only device in its IOMMU group or All devices in the same group should be passthroughed simultanaeously.
# Add modprobe.d/vfio.conf
sudo cp /mnt/usb/vmm/modprobe.d/vfio.conf /etc/modprobe.d/vfio.conf
sudo nano /etc/modprobe.d/vfio.conf
# options vfio-pci ids=8086:46d4,1b21:1064
# softdep i915 pre: vfio-pci
# softdep ahci pre: vfio-pci
sudo chmod 644 /etc/modprobe.d/vfio.conf
# Apply vfio configurations
sudo update-grub
sudo update-initramfs -u
# umount usb
sudo umount -f -l /mnt/usb
# reboot system
sudo reboot
# Connect console and vmm via lan cable, Set console's NIC as 192.168.1.11
# delete default user
# ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vmm_init
# login: vmm
# check vfio-pci
lspci -nnk -d 8086:46d4
lspci -nnk -d 1b21:1064
# Kernel driver in user: vfio-pci
id vmm
# check sudo group
su - bootstrap
sudo passwd vmm
# enter new password
exit
# delete default user
sudo userdel -r bootstrap
id bootstrap
# id: 'bootstrap': no such user
```
docs/runbook/05-hardwares.md
+220
View File
@@ -0,0 +1,220 @@
# Hardware
All hardware configuration is set after fw vm. The MAC address of hardware is reservated on kea-dhcp.
## Vlan switch
### Access VLAN switch
- http://switch.ilnmors.internal \(192.168.1.2, KEA-DHCP, Only IPv4 support\)
- before set ipv6, use ip4 address instead of FQDN
- id: admin, password: admin
- new password: switch.password
### Set VLAN
- VLAN:802.1Q VLAN
- \[x\] Enable - Apply
- VLAN client
- id 1
- name default > client
- member \(Untagged\)
- Port 1 \(Trunk, untagged\): Linux bridge is already process untagged packet as id 1
- Port 3
- Port 4
- Port 5
- Port 6
- Port 7
- Port 8
- VLAN server
- id 10
- name server
- member
- Port 1 \(Trunk, tagged\)
- VLAN user
- id 20
- name user
- member
- Port 1 \(Trunk, tagged\)
- Port 2 \(Not a member of client vlan, untagged\)
- VLAN:802.1Q VLAN PVID setting
- Port 2
- PVID 20
### Verify VLAN configuration
- Manually set consol ip as user
- Connect console to Port 2
- Check internet connection
## DSM \(DS124\)
- https://finds.synology.com/# \(192.168.1.11, KEA-DHCP\)
- Install DSM
### Initial configuration
- Device name: ilnmorsNAS
- Administrator account: il
- Password: dsm.il.password
- automatical update
- synology account - skip, skip anyway
- opt disagree
### Storage
- Storage Manager:Storage:Create:btrfs - Create
### IP address
Kea in fw already reserved DSM's IP. However it is necessary to set IP address statically for stable operation.
- Control Panel:Network:Network Interface:LAN
- Edit:IPv4:Use manual configuration
- Edit:IPv6:Use manual configuration
### Certificates
- Control Panel:Security:Certificate
- Replace an existing certificate:synology
- Description: ilnmors.internal
- Private Key
- Certificate
- Intermediate certificate
- Edit: For: Set as default certificate
- Setting \(!CAUTION!\)
- Even though you set the certificate as default, you have to set certificate for each services.
- configure: service: certificate: nas.ilnmors.internal
## Authelia OIDC
- **!CAUTION!** It can be set after authelia is implemented
- Following [here](../../../config/containers/auth/authelia/config/authelia.yaml.j2) for Authelia configuration
- Control Panel:Domain/LDAP:SSO Client
- Login Settings: \[x\] Select SSO by default on the login page
- Services
- \[x\] Enable OpenID Connect SSO service
- OpenID Connect SSO Settings
- Profile: OIDC
- Account type: Domain/LDAP/local
- Name: Authelia
- Well-Known URL: https://authelia.ilnmors.com/.well-known/openid-configuration
- Application ID: dsm \(what you designated\)
- Application Secret: secret value
- Redirect URI: https://nas.ilnmors.internal:5001
- Authorization scope: openid profile groups email
- Username claim: preferred_username
- Match the user name \(ID\) in DSM and lldap id.
### Kopia in DSM
#### Upload Kopia repository to DSM
- Directory
- Control Panel:shared folder: docker
- Create: docker/kopia - permission: everyone rwx - inheretence to sub directories
- Container manager
- Package Center:Conatiner manager:install
- Upload repository directory from console to DSM
- docker/kopia/repository
- Add certificate - DSM reverse proxy cannot deal with gRPC
- /docker/kopia/config/ssl/nas.key
- /docker/kopia/config/ssl/nas.crt \(including intermediate crt\)
- container manager:images:import
- kopia/kopia
- tags: \{\{ version['packages']['kopia'] \}\}
- run
- image: kopia/kopia
- containername: kopia-server
- \[x\] Enable auto restart
- port: 51515:51515
- volume: /docker/kopia/config:/app/config:rw
- volume: /docker/kopia/cache:/app/cache:rw
- volume: /docker/kopia/logs:/app/logs:rw
- volume: /docker/kopia/repository:/repository:rw
- environment: KOPIA_PASSWORD=$KOPIA.REPOSITORY
- command: server start --no-ui --tls-cert-file=/app/config/ssl/nas.crt --tls-key-file=/app/config/ssl/nas.key --address=0.0.0.0:51515 --log-level=info
- action:Terminal:Create
```bash
kopia repository connect filesystem \
--path=/repository \
--override-username="il" \
--override-hostname="nas.ilnmors.internal"
```
- action:restart
- Set firewall nftables
- Remove kopia_tmp dir from console
### Connection from client
#### Structure
Repository directory - encrypted by server KOPIA_PASSWORD as master key of repository
Server manage ACL with user password, user's KOPIA_PASSWORD. When server verify user with their password, server works with its repository password.
Repository - \(Repository key; master key\) - Server - \(User key; access key\) - Client
- Client knows its access password as KOPIA_PASSWORD to access server. It doesn't know master key, server's KOPIA_PASSWORD. server will control repository by its KOPIA_PASSWORD. their name is the same but it is different.
#### Access
```bash
# Console
# you have to use `'` not `"`
KOPIA_PASSWORD='$kopia.user.user_name' \
/usr/bin/kopia repository connect server \
--url=https://nas.ilnmors.internal:51515 \
--override-username=console \
--override-hostname=console.ilnmors.internal
# This makes repository.config on ~/.config/kopia
# verify with kopia server acl list command
# infra or app
## /etc/secrets/$KOPIA_UID/kopia.env
KOPIA_PASSWORD={{ hostvars['console']['kopia']['user'][node['name']] }}
KOPIA_CONFIG_PATH=/etc/kopia/repository.config
KOPIA_CACHE_DIRECTORY=/var/cache/kopia
KOPIA_LOG_DIR=/var/cache/kopia/logs
KOPIA_CHECK_FOR_UPDATES=false
## .service file
BindReadOnlyPaths=/path/to/backup
# In root namescope, %u always bring 0
BindPaths=/etc/kopia
BindPaths=/etc/secrets/{{ kopia_uid }}
BindPaths=/var/cache/kopia
EnvironmentFile=/etc/secrets/{{ kopia_uid }}/kopia.env
ExecStartPre=/usr/bin/kopia repository connect server \
--url=https://{{ infra_uri['kopia']['domain'] }}:{{ infra_uri['kopia']['ports']['https'] }} \
--override-username={{ node['name'] }} \
--override-hostname={{ node['name'] }}.ilnmors.internal
ExecStart=/usr/bin/kopia snapshot create \
/path/to/backup
```
### Check kopia snapshot
```bash
# snapshot id check
kopia snapshot list [--all]
# Snapshot ID check
kopia ls -l [-r: for recursive] $SNAPSHOT_ID
kopia show -l $SNAPSHOT_ID/file/path
# or
kopia show -l $FILE_ID
```
### Restore
```bash
mkdir -p /mnt/kopia
kopia mount [$SNAPSHOT_ID|all] kopia &
```
docs/runbook/06-kopia.md
+248
View File
@@ -0,0 +1,248 @@
# kopia
## Create repository
```bash
# export KOPIA_PASSWORD="your_repository_password"
# It exists by Docker environment, however when it is used by pure linux server this environemnt is needed.
# *! CAUTION !*
# THIS PROCESS CONTAINING SECRET VALUES.
# WHEN YOU TYPE THE COMMAND ON SHELL, YOU MUST USE [BLANK] BEFORE COMMAND
# e.g.
# shell@shell$ command (X)
# shell@shell$ [BLANK]command (O)
# BLANK prevent the command to save on .bash_history
# After finish this process, use `history -c` and `clear` for just in case.
# Kopia CLI Create repository
mkdir -p /home/console/workspace/kopia_tmp/{repository,config,cache}
/usr/bin/kopia repository create filesystem \
--password="kopia.repository" \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--path="/home/console/workspace/kopia_tmp/repository" \
--description="ilnmors.internal kopia repository" \
--create-only
# block hash: BLAKE2B-256-128
# encryption: AES256-GCM-HMAC-SHA256
# key derivation: scrypt-65536-8-1
# splitter: DYNAMIC-4M-BUZHASH
# Kopia CLI Connect repository
/usr/bin/kopia repository connect filesystem \
--password="kopia.repository" \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--path="/home/console/workspace/kopia_tmp/repository" \
--override-username="il" \
--override-hostname="nas.ilnmors.internal" \
--description="ilnmors.internal kopia repository"
# Connected to repository
# Start kopia server on pure linux server
# /usr/bin/kopia server start \
# --password="kopia.repository" \
# --config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
# --cache-directory="/home/console/workspace/kopia_tmp/cache" \
# --no-ui \
# --address=localhost:51515 \
# --insecure \
# --log-level=info &
# All after these processes reset history.
clear
history -c
```
### Set users
- Container manager: container: kopia-server: Action: Open terminal: Create: bash
```bash
# export KOPIA_PASSWORD="your_repository_password"
# It exists by Docker environment, however when it is used by pure linux server this environemnt is needed.
# Create users
## console@console.ilnmors.internal, infra@infra.ilnmors.internal, app@app.ilnmors.internal
## Repository pasword is not required, because it is already connected
/usr/bin/kopia server users add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user-password="$kopia.user.console" \
console@console.ilnmors.internal
/usr/bin/kopia server users add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user-password="$kopia.user.infra" \
infra@infra.ilnmors.internal
/usr/bin/kopia server users add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user-password="$kopia.user.app" \
app@app.ilnmors.internal
## verify
/usr/bin/kopia server users list \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache"
# ACL
## Console - Run these commands on local container
## snapshot, policy, user, acl, content
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=snapshot" \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=policy" \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=user" \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=acl" \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="console@console.ilnmors.internal" \
--target="type=content" \
--access=FULL
## app and infra - Run these command on console
### Global pull to remove duplicates
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="infra@infra.ilnmors.internal" \
--target="type=content" \
--access=APPEND
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="app@app.ilnmors.internal" \
--target="type=content" \
--access=APPEND
### Snapshot
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="infra@infra.ilnmors.internal" \
--target=type=snapshot,username=infra,hostname=infra.ilnmors.internal \
--access=FULL
/usr/bin/kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="app@app.ilnmors.internal" \
--target=type=snapshot,username=app,hostname=app.ilnmors.internal \
--access=FULL
### Policy
kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="infra@infra.ilnmors.internal" \
--target="type=policy" \
--access=READ
kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="infra@infra.ilnmors.internal" \
--target="type=policy,username=infra,hostname=infra.ilnmors.internal" \
--access=FULL
kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="app@app.ilnmors.internal" \
--target="type=policy" \
--access=READ
kopia server acl add \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--user="app@app.ilnmors.internal" \
--target="type=policy,username=app,hostname=app.ilnmors.internal" \
--access=FULL
```
### Set policy
- Container manager: container: kopia-server: Action: Open terminal: Create: bash
```bash
# compression
/usr/bin/kopia policy set \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--global \
--compression=zstd
# infra for db dump sql
kopia policy set infra@infra.ilnmors.internal \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--compression=zstd \
--keep-latest=10 \
--keep-hourly=0 \
--keep-daily=14 \
--keep-weekly=4 \
--keep-monthly=6 \
--keep-annual=1
# app for media, stateful data
kopia policy set app@app.ilnmors.internal \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
--compression=none \
--keep-latest=10 \
--keep-hourly=0 \
--keep-daily=14 \
--keep-weekly=4 \
--keep-monthly=6 \
--keep-annual=1
# verify
kopia server acl list \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
kopia policy list \
--config-file="/home/console/workspace/kopia_tmp/config/repository.config" \
--cache-directory="/home/console/workspace/kopia_tmp/cache" \
```
## Remove repository from Console
```bash
# After uploading repository to DSM
rm -rf /home/console/workspace/kopia_tmp
history -c
```
docs/runbook/07-git.md
+71
View File
@@ -0,0 +1,71 @@
# Git configuration
## Local git
```bash
# git config
git config --global user.name "il"
git config --global user.email "il@ilnmors.internal"
git config --global init.defaultBranch main # Set default branch name as main
# Git repository path
# $PROJECT_REPOSITORY/.git
# Remote registration and push git
cd ~/workspace/homelab
# Create .gitignore management for data directory
echo data/bin/ | tee ./.gitignore
echo data/volumes/ | tee -a ./.gitignore
echo data/images/ | tee -a ./.gitignore
echo docs/archives/textfiles/ | tee -a ./.gitignore
# Select files
# When the set repository as first time
git init
# To commit and tag, you should `git add`
git add .
# Check git changes
git status
git commit -m "1.0.0: Release IaaS baseline"
# git commit -m "docs: update 07-git.md to add the way to manage git system"
# Make current documents as snapshot
git tag -a 1.0.0 -m "IaaS baseline"
# Make special changes
# In this homelab, [Infra_structure_change]:[Services_change]:[Documents_and_configuration_change]
# Tagging and commit should be distinguished.
# The change which affects system: tagging
# The change which doesn't affect system: commit
# Commands
git status # What files are changed
git log # The version record
git diff # What is changed after last commit
git show $tag # Tag version and information check.
git checkout $tag # rollback to tag version
git branch $branch_name # Create branch
git switch $branch_name # Switch branch
git branch # list of branch
git switch main # Switch to main branch
git merge $branch_name # run at the main branch, merge.
git stash # temporary save
git stash pop # get temporary save
# After git switch
git switch service
git rebase --ignore-date main # set date as current time on main branch
```
## Add Service with git
```bash
# Example of establish gitea
git branch caddy-app
git switch caddy-app
git commit -m "0.0.1-caddy-app: Start caddy-app branch"
git tag -a 0.0.1-caddy-app -m "caddy-app: Start caddy-app branch"
## After finishing gitea implement
git switch main
git merge caddy-app
```
docs/services/app/igpu_firmware.md
+12
View File
@@ -0,0 +1,12 @@
# Firmware installation
Cloud-init has no firmware in it, so i915 driver is needed to install.
When the app node is initiated by ansible, `firmware-intel-graphics` and `intel-media-va-driver-non-free` are installed. And only when they are installed, `update-initramfs -u` and `reboot` modules run as a handler.
## Verification
After reboot, check the render device.
```bash
ls -l /dev/dri
# crw-rw---- 1 root video 226, 0 ... card0
# crw-rw---- 1 root render 226, 128 ... renderD128
docs/services/common/alloy.md
+35
View File
@@ -0,0 +1,35 @@
# Alloy
## Communication
Alloy runs on systemd \(host\), and postgresql runs as container \(rootless podman\). When host system and container communicate, container recognizes host system as host-gateway \(Link local address\).
## postgresql monitor
### Monitor exporter
```sql
postgres=# CREATE USER alloy WITH PASSWORD 'password';
CREATE ROLE
postgres=# GRANT pg_monitor TO alloy;
GRANT ROLE
postgres=# \drg
List of role grants
Role name | Member of | Options | Grantor
-----------+------------+--------------+----------
alloy | pg_monitor | INHERIT, SET | postgres
(1 row)
```
### pg_hba.conf
```conf
hostssl postgres alloy {{ hostvars['fw']['network4']['infra']['server'] }}/32 trust
hostssl postgres alloy {{ hostvars['fw']['network6']['infra']['server'] }}/128 trust
hostssl postgres alloy {{ hostvars['fw']['network4']['subnet']['lla'] }} trust
hostssl postgres alloy {{ hostvars['fw']['network6']['subnet']['lla'] }} trust
```
### check
```bash
curl http://localhost:12345/metrics
```
docs/services/common/caddy.md
+45
View File
@@ -0,0 +1,45 @@
# Caddy
## TLS re-encryption
This is not a perfect E2EE communication theorogically, however technically it is. The main caddy decrypt as an edge node of WAN side, and it becomes a client of side caddy with private certificate.
### .com public domain
WAN - \(Let's Encrypt certificate\) -> Caddy \(auth\) - \(ilnmors internal certificate\) -> Caddy \(app\) or https services - http -> app's local service
### .internal private domain
client - \(ilnmors internal certificate\) -> Caddy \(Infra\) - http -> local services
### DNS record
*.app.ilnmors.internal - CNAME -> app.ilnmors.internal
## X-Forwarded-Host
When caddy in app conducts TLS re-encryption, it is important to change their Host header as X-Forwarded-Host haeder for session maintainance.
## Example
```ini
# Auth server
test.ilnmors.com
{
import crowdsec_log
route {
crowdsec
reverse_proxy https://test.app.ilnmors.internal
}
}
# App server
test.app.ilnmors.internal
{
import internal_tls
trusted_proxies {{ hostvars['fw']['network4']['auth']['server'] }} {{ hostvars['fw']['network6']['auth']['server'] }}
route {
reverse_proxy host.containers.internal:3000 {
header_up Host {header.X-Forwarded-Host} {Host}
}
}
}
```
docs/services/common/crowdsec.md
+233
View File
@@ -0,0 +1,233 @@
# Crowdsec
## LAPI
### Detecting
Host logs \> CrowdSec Agent\(parser\) > CrowdSec LAPI
### Decision
CrowdSec LAPI \(Decision + Register\)
### Block
CrowdSec LAPI \> CrowdSec Bouncer \(Block\)
## CAPI
CrowdSec CAPI \> crowdsec LAPI \(local\) \> CrowdSec Bouncer \(Block\)
## Ansible Deployment
### Set LAPI (fw/roles/tasks/set_crowdsec_lapi.yaml)
- Deploy fw's config.yaml
- Deploy crowdsec certificates
- Register machines \(Agents\)
- Register bouncers \(Bouncers\)
### Set Bouncer (fw/roles/tasks/set_crowdsec_bouncer.yaml)
- Deploy crowdsec-firewall-bouncer.yaml
- Install suricata collection \(parser\) with cscli
- Set acquis.d for suricata
- set-only: bouncer can't get metrics from the chain and rules count result which it doesn't make. - It means, it is impossible to use prometheus metric with set-only true option.
- chain or rules matched count reasults are able to check on nftables.
- use sudo nft list chain inet filter global to check packet blocked. \(counter command is required\)
### Set Machines; agents (common/tasks/set_crowdsec_agent.yaml)
- Deploy config.yaml except fw \(disable LAPI, online_api_credentials\)
- Deploy local_api_credentials.yaml
### Set caddy host (auth/tasks/set_caddy.yaml)
- Set caddy CrowdSec module
- Set caddy log directory
- Install caddy collection \(parser\) with cscli
- Set acquis.d for caddy
### Set whitelist (/etc/crowdsec/parser/s02-enrich/whitelists.yaml)
- Set only local console IP address
- This can block local VM to the other subnet, but the communication between vms is possible because they are in the same subnet\(L2\) - packets don't pass the fw.
- Crowdsec bouncer only conducts blocks forward chain which pass Firewall, it is blocked by crowdsec bouncer based on lapi
## Test
### Decision test
> Set test decisions and check it
fw@fw:/etc/crowdsec/bouncers$ sudo cscli decisions add --ip 5.5.5.5 --duration 10m --reason "Test"
INFO[12-01-2026 01:50:40] Decision successfully added
fw@fw:/etc/crowdsec/bouncers$ sudo tail -f /var/log/crowdsec-firewall-bouncer.log
time="12-01-2026 01:50:22" level=info msg="backend type : nftables"
time="12-01-2026 01:50:22" level=info msg="nftables initiated"
time="12-01-2026 01:50:22" level=info msg="Using API key auth"
time="12-01-2026 01:50:22" level=info msg="Processing new and deleted decisions . . ."
time="12-01-2026 01:50:22" level=info msg="Serving metrics at 127.0.0.1:60601/metrics"
time="12-01-2026 01:50:22" level=info msg="1320 decisions deleted"
time="12-01-2026 01:50:22" level=info msg="15810 decisions added"
time="12-01-2026 01:50:42" level=info msg="1 decision added"
fw@fw:/etc/crowdsec/bouncers$ sudo nft list ruleset | grep -i 5.5.5.5
5.5.5.5 timeout 9m54s876ms expires 9m22s296ms,
### Parser test
> CrowdSec "crowdsecurity/suricata-evelogs" only parses "event_type: alert". You can test with cscli explain
fw@fw:~$ sudo cscli explain --file /tmp/suri_test.log --type suricata-evelogs --verbose
line: {"timestamp":"2026-01-11T14:43:52.153576+0000","flow_id":972844861874490,"in_iface":"wan","event_type":"alert","src_ip":"197.242.151.53","src_port":42976,"dest_ip":"59.5.196.55","dest_port":38694,"proto":"TCP","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":60,"bytes_toclient":0,"start":"2026-01-11T14:42:51.554188+0000","end":"2026-01-11T14:42:51.554188+0000","age":0,"state":"new","reason":"timeout","alerted":false},"community_id":"1:Ovyuzq7R8yA3YfxM8jEExR5BZMI=","tcp":{"tcp_flags":"02","tcp_flags_ts":"02","tcp_flags_tc":"00","syn":true,"state":"syn_sent","ts_max_regions":1,"tc_max_regions":1}}
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| ├ 🔴 crowdsecurity/apache2-logs
| ├ 🔴 crowdsecurity/nginx-logs
| ├ 🔴 crowdsecurity/sshd-logs
| ├ 🟢 crowdsecurity/suricata-evelogs (+9 ~2)
| ├ update evt.Stage : s01-parse -> s02-enrich
| ├ create evt.Parsed.dest_ip : 59.5.196.55
| ├ create evt.Parsed.dest_port : 38694
| ├ create evt.Parsed.proto : TCP
| ├ create evt.Parsed.time : 2026-01-11T14:43:52.153576
| ├ update evt.StrTime : -> 2026-01-11T14:43:52.153576Z
| ├ create evt.Meta.log_type : suricata_alert
| ├ create evt.Meta.service : suricata
| ├ create evt.Meta.source_ip : 197.242.151.53
| ├ create evt.Meta.sub_log_type : suricata_alert_eve_json
| ├ create evt.Meta.suricata_flow_id : 972844861874490
| └ 🔴 crowdsecurity/suricata-fastlogs
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ create evt.Enriched.MarshaledTime : 2026-01-11T14:43:52.153576Z
| ├ update evt.MarshaledTime : -> 2026-01-11T14:43:52.153576Z
| ├ create evt.Meta.timestamp : 2026-01-11T14:43:52.153576Z
| ├ 🟢 crowdsecurity/geoip-enrich (+13)
| ├ create evt.Enriched.IsInEU : false
| ├ create evt.Enriched.IsoCode : ZA
| ├ create evt.Enriched.ASNumber : 37611
| ├ create evt.Enriched.Latitude : -28.998400
| ├ create evt.Enriched.Longitude : 23.988800
| ├ create evt.Enriched.SourceRange : 197.242.144.0/20
| ├ create evt.Enriched.ASNNumber : 37611
| ├ create evt.Enriched.ASNOrg : Afrihost
| ├ create evt.Meta.ASNNumber : 37611
| ├ create evt.Meta.IsInEU : false
| ├ create evt.Meta.SourceRange : 197.242.144.0/20
| ├ create evt.Meta.ASNOrg : Afrihost
| ├ create evt.Meta.IsoCode : ZA
| ├ 🔴 crowdsecurity/http-logs
| └ 🟢 crowdsecurity/whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
#### Caddy
auth@auth:~/containers/authelia/config$ sudo cscli explain --file /var/log/caddy/access.log --type caddy
line: {"level":"info","ts":1771601235.7503738,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"192.168.99.20","remote_port":"59900","client_ip":"192.168.99.20","proto":"HTTP/2.0","method":"GET","host":"authelia.ilnmors.com","uri":"/static/js/components.TimerIcon.CO1b_Yfm.js","headers":{"Accept-Encoding":["gzip, deflate, br, zstd"],"Referer":["https://authelia.ilnmors.com/settings"],"Te":["trailers"],"Accept":["*/*"],"Sec-Fetch-Dest":["script"],"Priority":["u=1"],"Sec-Fetch-Mode":["cors"],"Accept-Language":["en-US,en;q=0.9"],"Cookie":["REDACTED"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:147.0) Gecko/20100101 Firefox/147.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"authelia.ilnmors.com"}},"bytes_read":0,"user_id":"","duration":0.0077169,"size":10193,"status":200,"resp_headers":{"Via":["1.1 Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Content-Type-Options":["nosniff"],"Content-Security-Policy":["default-src 'none'"],"Date":["Fri, 20 Feb 2026 15:27:15 GMT"],"Etag":["7850315714d1e01e73f4879aa3cb7465b4e879dc"],"Cache-Control":["public, max-age=0, must-revalidate"],"Content-Length":["10193"],"X-Frame-Options":["DENY"],"Content-Type":["text/javascript; charset=utf-8"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Permissions-Policy":["accelerometer=(), autoplay=(), camera=(), display-capture=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), screen-wake-lock=(), sync-xhr=(), xr-spatial-tracking=(), interest-cohort=()"],"X-Dns-Prefetch-Control":["off"]}}
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| ├ 🔴 crowdsecurity/apache2-logs
| └ 🟢 crowdsecurity/caddy-logs (+19 ~2)
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ 🟢 crowdsecurity/http-logs (+7)
| └ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
└-------- parser failure 🔴
## BAN logs case
### LAPI metrics
fw@fw:~$ sudo cscli metrics
Acquisition Metrics:
╭─────────────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├─────────────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ file:/var/log/suricata/eve.json │ 130.25k │ - │ 130.25k │ - │
│ journalctl:journalctl-_SYSTEMD_UNIT=ssh.service │ 6 │ - │ 6 │ - │
╰─────────────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯
Parser Metrics:
╭─────────────────────────────────┬─────────┬─────────┬──────────╮
│ Parsers │ Hits │ Parsed │ Unparsed │
├─────────────────────────────────┼─────────┼─────────┼──────────┤
│ child-crowdsecurity/sshd-logs │ 60 │ - │ 60 │
│ child-crowdsecurity/syslog-logs │ 6 │ 6 │ - │
│ crowdsecurity/non-syslog │ 130.25k │ 130.25k │ - │
│ crowdsecurity/sshd-logs │ 6 │ - │ 6 │
│ crowdsecurity/syslog-logs │ 6 │ 6 │ - │
╰─────────────────────────────────┴─────────┴─────────┴──────────╯
Local Api Metrics:
╭──────────────────────┬────────┬───────╮
│ Route │ Method │ Hits │
├──────────────────────┼────────┼───────┤
│ /v1/alerts │ GET │ 1 │
│ /v1/alerts │ POST │ 6 │
│ /v1/decisions/stream │ GET │ 11337 │
│ /v1/heartbeat │ GET │ 8053 │
│ /v1/watchers/login │ POST │ 145 │
╰──────────────────────┴────────┴───────╯
Local Api Machines Metrics:
╭─────────┬───────────────┬────────┬──────╮
│ Machine │ Route │ Method │ Hits │
├─────────┼───────────────┼────────┼──────┤
│ app │ /v1/heartbeat │ GET │ 1587 │
│ auth │ /v1/alerts │ GET │ 1 │
│ auth │ /v1/alerts │ POST │ 6 │
│ auth │ /v1/heartbeat │ GET │ 1605 │
│ fw │ /v1/heartbeat │ GET │ 1621 │
│ infra │ /v1/heartbeat │ GET │ 1620 │
│ vmm │ /v1/heartbeat │ GET │ 1620 │
╰─────────┴───────────────┴────────┴──────╯
Local Api Bouncers Metrics:
╭───────────────┬──────────────────────┬────────┬──────╮
│ Bouncer │ Route │ Method │ Hits │
├───────────────┼──────────────────────┼────────┼──────┤
│ caddy-bouncer │ /v1/decisions/stream │ GET │ 1608 │
│ fw-bouncer │ /v1/decisions/stream │ GET │ 9729 │
╰───────────────┴──────────────────────┴────────┴──────╯
Local Api Decisions:
╭─────────────────┬────────┬────────┬───────╮
│ Reason │ Origin │ Action │ Count │
├─────────────────┼────────┼────────┼───────┤
│ http:exploit │ CAPI │ ban │ 17803 │
│ http:scan │ CAPI │ ban │ 4583 │
│ ssh:bruteforce │ CAPI │ ban │ 2509 │
│ http:bruteforce │ CAPI │ ban │ 1721 │
│ http:crawl │ CAPI │ ban │ 87 │
│ http:dos │ CAPI │ ban │ 15 │
╰─────────────────┴────────┴────────┴───────╯
Local Api Alerts:
╭───────────────────────────────────┬───────╮
│ Reason │ Count │
├───────────────────────────────────┼───────┤
│ crowdsecurity/http-bad-user-agent │ 2 │
│ crowdsecurity/jira_cve-2021-26086 │ 4 │
╰───────────────────────────────────┴───────╯
### WAF parser alerts
auth@auth:~$ sudo cscli alerts list
╭────┬────────────────────┬───────────────────────────────────┬─────────┬────┬───────────┬─────────────────────────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├────┼────────────────────┼───────────────────────────────────┼─────────┼────┼───────────┼─────────────────────────────────────────┤
│ 25 │ Ip:206.168.34.127 │ crowdsecurity/http-bad-user-agent │ │ │ ban:1 │ 2026-03-07 02:26:58.074029091 +0000 UTC │
│ 23 │ Ip:162.142.125.212 │ crowdsecurity/http-bad-user-agent │ │ │ ban:1 │ 2026-03-07 00:19:08.421713824 +0000 UTC │
│ 12 │ Ip:159.65.144.72 │ crowdsecurity/jira_cve-2021-26086 │ │ │ ban:1 │ 2026-03-06 04:19:04.975124762 +0000 UTC │
│ 11 │ Ip:206.189.95.232 │ crowdsecurity/jira_cve-2021-26086 │ │ │ ban:1 │ 2026-03-06 04:19:01.215582087 +0000 UTC │
│ 10 │ Ip:68.183.9.16 │ crowdsecurity/jira_cve-2021-26086 │ │ │ ban:1 │ 2026-03-06 04:18:22.120468981 +0000 UTC │
│ 9 │ Ip:138.68.144.227 │ crowdsecurity/jira_cve-2021-26086 │ │ │ ban:1 │ 2026-03-06 04:18:18.35776077 +0000 UTC │
╰────┴────────────────────┴───────────────────────────────────┴─────────┴────┴───────────┴─────────────────────────────────────────╯
docs/services/common/kopia.md
+14
View File
@@ -0,0 +1,14 @@
# Kopia
Kopia is one of modern backup solution to support very strong deduplication.
## Repository
Kopia saves all information, even the users and policies on repository. Repository itself is complete. Repository is encrypted by master password.
## User and policy
When kopia is run as a kopia server, client can access to server with user and user password. The clients don't have to know master password. Kopia server decrypt the repository with the master password, and the client just access to the kopia server with their user account.
Repository \<- Master password -\> Kopia server \<- User password -\> Kopia client
docs/services/console/git.md
View File
docs/services/fw/kea.md
+29
View File
@@ -0,0 +1,29 @@
# IP
## IPv4
### Subnet management
- Static subnet \(manage without dhcp\)
- client \(for ipv4, set reservation\)
- server
- Dynamic subnet \(manage with dhcp\)
- user
## IPv6
### Subnet management
- Static subnet \(manage without RA - specific defination\)
- client \(Designated ULA with NAT66\)
- server \(Designated ULA with NAT66\)
- Dynamic subnet \(manage with RA and SLAAC\)
- user \(Autogenerated GUA\)
## Firewall policy for each subnet
### Static subnet
Make polices based on each specific designated IP address for nodes.
### Dynamic subnet
Make polices based on subnet \(or interface itself\)
docs/services/infra/ca.md
+146
View File
@@ -0,0 +1,146 @@
## Operation
Refer to Ansible playbook
## Configuration files
- ca.json
- defaults.json
### Provisioner
Provisioner is basically the object of issuing certificates as a RA. They verify CSR from client and when it is valid with its policy they will sign the certificates with CA's private key. Step-CA supports various type of provisioner. In this homelab, only ACME will be used. Because infrastructure's certificates is issued manually. Step-CA supports one root CA and one intermediate CA in one container, only one intermediate CA will be operated in this project.
#### jwk-ca@ilnmors.internal
This provisioner is to issue intermediate CA. It wouldn't be used in this project. The option for CA in X.509 format is optional and defined in as extension option. To define these option in step-ca, the template file is needed.
- file: ~/data/containers/step-ca/templates/ca.tpl
```json
{
"subject": {{ toJson .Subject }},
"keyUsage": ["certSign", "crlSign"],
"basicConstraints": {
"isCA": true,
"maxPathLen": 0
}
}
```
> keyUsage: Designate to manage certificates and CRL
> isCA: Designate the certificate to use CA
> maxPathLen: Designate allowed below CA's number
- Define provisioner
```bash
podman exec -it step-ca \
step ca provisioner add jwk-ca@ilnmors.internal \
--create \ # Generate key pair automatically
--type JWK \
--ca-config /home/step/config/ca.json \ # Sign on certificate with root CA's private key
--x509-template /home/step/template/ca.tpl \ # Use x509 template
--x509-max-dur 87600h \ #
--x509-default-dur 87600h
```
#### jwk@ilnmors.internal
This provisioner is to issue the certificates like DB communication based on its identity (Using JWK and JWT pre-shared). The certificate is issued based on enrolled key in provisioner. However, in this project all crt will be used central ACME client `Caddy`.
- Define provisioner
```bash
podman exec -it step-ca \
step ca provisioner add jwk@ilnmors.internal \
--create \ # Generate key pair automatically
--type JWK \
--x509-default-dur 2160h # To set default expire date as 90 days.
```
#### acme@ilnmors.internal
This provisioner is to issue the certificates for https communication. The certificate is issued based on challenge; the ownership of domain.
- Define provisioner
```bash
podman exec -it step-ca \
step ca provisioner add acme@ilnmors.internal \
--type ACME \
--x509-default-dur 2160h # To set default expire date as 90 days.
```
### Subject
Step-CA uses subject as a account. It is used to manage Step-CA remotely. To use this, it is necessary to use `--remote-management` option when the step-CA is initially set or fix `ca.json` authority.enableAdmin:true. When subject is enabled, provisioners aren't defined in ca.json but its own DB.
### Policy
Self-hosted Step-CA server doesn't support to give x509 policy for each provisioner. It only allows public policy. Only `ilnmors.internal` and `*.ilnmors.internal` certificates are required, so designate the policy in `ca.json`
> Policies can be administered using the step CLI application. The commands are part of the step ca policy namespace. In a self-hosted step-ca, policies can be configured on the authority level. Source: [here](https://smallstep.com/docs/step-ca/policies/)
- file: ~/data/containers/step-ca/config/ca.json
```json
...
"authority": {
"policy": {
"x509": {
"allow": {
"dns": [
"ilnmors.internal",
"*.ilnmors.internal"
]
},
"allowWildcardNames": true
}
},
"provisioners": [ ... ]
....
}
...
```
## Verify server
### Server health check
```bash
curl -k https://ca.ilnmors.internal:9000/health
> {"status":"ok"}
```
### Server policy check
```bash
podman exec -it ca step ca certificate test.com test.crt test_key --provisioner acme@ilnmors.internal
> error creating new ACME order: The server will not issue certificates for the identifier
```
---
## Set trust Root CRT
### Linux
#### Debian/ubuntu
- File: /usr/local/share/ca-certificates/{ca.crt, ca.pem}
- `update-ca-certificates`
#### Cent/RHEL/Fedora
- File: /etc/pki/ca-trust/source/anchors/{ca.crt, ca.pem}
- `update-ca-trust`
### Windows
- `Windows + R` + `certlm.msc`
- `All Task` - `Import`
### Firefox
- Setting - Security - view certificates - Authority - add
- \[x\] trust this ca to identify website
- \[x\] trust this ca to identify email users
docs/services/infra/grafana.md
+20
View File
@@ -0,0 +1,20 @@
# Grafana
## Operation
Refer to Ansible playbook
\(Postgresql user and DB is needed\)
\(LDAP strict readonly account is needed\)
## Verification
- Check Caddyfile \(without caddy, use 3000 ports\)
- https://grafana.ilnmors.internal
- login with LDAP user
- connection:data sources: \[prometheus|loki\]: provisioned
- https://prometheus.ilnmors.internal:9090
- https://loki.ilnmors.internal:3100
- check drill down:metrics
## Dashboard
- Dashboard isn't saved on local directory. They are saved on DB \(Postgresql\).
docs/services/infra/ldap.md
+154
View File
@@ -0,0 +1,154 @@
## Operation
Refer to Ansible playbook
\(Postgresql user and DB is needed\)
Integrate configuration with various app: https://github.com/lldap/lldap/blob/main/example_configs
## Configuration
### DB URL
Jinja2 `urlencode` module doesn't replace `/` as `%2F`. replace('/', '%2F') is necessary.
ex\) {{ var | urlencode | replace('/', '%2F') }}
### Reset administrator password
```bash
# infra
sudo nano $LDAP_PATH/data/lldap_config.toml
# Add below on file
ldap_user_pass = "REPLACE_WITH_PASSWORD"
force_ldap_user_pass_reset = true
# Restart lldap
systemctl --user restart ldap.service
# Delete added lines from lldap_config.toml
# ldap_user_pass = "REPLACE_WITH_PASSWORD"
# *YOU MUST DELETE PASSWORD PART*
# force_ldap_user_pass_reset = true
```
### Access web UI and Login
- URL: http://ldap.ilnmors.internal:17170 \(This is temporary access way before Caddy, which is reverse proxy, is set)
- ID: admin
- PW: $LLDAP_LDAP_USER_PASSWORD
### Create the groups
- Groups - \[\+\] Create a group
- Group: admins
- Group: users
It is necessary to manage ACL via authelia based on groups.
### Create the authelia user for OCID \(OP\)
- Users: \[\+\] Create a user
- Username (cn; uid): authelia
- Display name: Authelia
- First Name: Authelia
- Last Name (sn): Service
- Email (mail): authelia@ilnmors.internal
- Password: "$(openssl rand -base64 32)"
- Groups:lldap_strict_readonly: \[Add to group\]
- This group allow search authority.
- Users: \[\+\] Create a user
- Username (cn; uid): grafana
- Display name: Grafana
- First Name: Grafana
- Last Name (sn): Service
- Email (mail): grafana@ilnmors.internal
- Password: "$(openssl rand -base64 32)"
- Groups:lldap_strict_readonly: \[Add to group\]
- This group allow search authority.
> Save the password in .secret.yaml
### Create the normal users
- Users: \[\+\] Create a user
- Username (cn; uid): il
- First Name: Il
- Last Name (sn): Lee
- Email (mail): il@ilnmors.internal
- Password: "$PASSWORD"
- Groups:lldap_admin&admins&users: \[Add to group\]
- Users: \[\+\] Create a user
- Username (cn; uid): user
- First Name: John
- Last Name (sn): Doe
- Email (mail): john_doe@ilnmors.internal
- Password: "$PASSWORD"
- Groups:(admins|users): \[Add to group\]
> Custom schema in `User schema`, `Group schema` doesn't need to be added. This is for advanced function to add additional value such as `identity number` or `phone number`. Hardcoded schema, which means basic schema the lldap provides is enough to use Authelia.
> After all these steps, now you can integrate the Authelia for SSO.
## Usage of LDAP
### Service Bind
LDAP call `login` as Bind. When the authelia Bind to the LDAP server, it can get the authority to search in `lldap_strict_readonly` group.
### Search
authelia account has the authority to search, it can search to send the query.
#### Flow of search
- Client (authelia) sends the query
- `uid=user in dc=ilnmors,dc=internal`
- LDAP server searches the DN of entry
- `uid=user,ou=people,dc=ilnmors,dc=internal`
- LDAP sends the DN to Client (authelia)
## Authelia's work flow
### First login
#### User login query
User try to login on login page of Authelia.
- id: user
- password: 1234
#### Service Bind (Bind and search)
authelia binds to LLDAP server based on the information in configuration.yml.
- dn: authelia
- password: authelia's password
#### Search
authelia sends the query to LLDAP after bind.
- `uid=user in dc=ilnmors,dc=internal`
#### Request
LLDAP server searches the entry and send the DN information query to authelia.
- `uid=user,ou=people,dc=ilnmors,dc=internal`
### Verify the user login (Second login)
#### User Bind (Bind only)
authelia tries to bind LLDAP server based on the information that user input.
- dn: requested uid
- password: 1234
#### Verification from LLDAP
LLDAP verify the password from authelia with its hash value saved in LLDAP's database.
#### Request
LLDAP server sends the result as `Success` or `Fail`.
> Search authority is basic authority of user who binds to LDAP server. It is just the way to check success or fail bind is the charge of Authelia.
## verify
- openssl s_client -connect ldap.ilnmors.internal:636 -tls1_3
docs/services/infra/loki.md
+12
View File
@@ -0,0 +1,12 @@
# loki
## Operation
Refer to Ansible playbook
## Verification
- fw@fw:/var/lib/bind$ curl -k https://loki.ilnmors.internal:3100/ready \(Node which is in NET_SERVER except infra itself\)
- ready
- fw@fw:/var/lib/bind$ curl -k https://loki.ilnmors.internal:3100/metrics
- metrics lists
- fw@fw:/var/lib/bind$ curl -k https://loki.ilnmors.internal:3100/loki/api/v1/labels
- no org id
- JSON format labels when alloy is set
docs/services/infra/postgresql.md
+64
View File
@@ -0,0 +1,64 @@
# Postgresql
## Operation
Refer to Ansible playbook
## File management
```bash
# console
## cluster
scp infra@infra:$POSTGRESQL_BACKUP_PATH/pg_cluster.sql $HOMELAB_PATH/data/backups/infra/postgresql/pg_cluster.sql
## data
scp infra@infra:$POSTGRESQL_BACKUP_PATH/pg_backup.sql $HOMELAB_PATH/data/backups/infra/postgresql/pg_backup.sql
## The data is managed by kopia.
```
## Verification
```bash
# ... Start postgresql service
# Create user and database
podman exec -it -u postgres postgresql "psql -U postgres"
> CREATE USER service WITH PASSWORD 'abc';
> CREATE DATABASE service_db;
> ALTER DATABASE service_db OWNER TO service;
> \du
> \l
> \q
# Reset database
> SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'service_db'; # connection reset
> DROP DATABASE service_db;
> CREATE DATABASE service_db;
> ALTER DATABASE service_db OWNER TO service;
> \du
> \l
> \q
# Restor database (manually)
podman exec -u postgres postgresql "psql -U postgres -f $POSTGRESQL_BACKUP_PATH_IN_CONTAINER/script.sql"
# Backup service executes
systemctl --user start postgresql-cluster-backup.service
# Stop and remove all data
systemctl --stop postgresql
sudo find "/home/infra/data/containers/postgresql/data" -mindepth 1 -delete
# Restore database
# Just locate sql files on data_path, and use playbooks
# Check restoring
podman exec -it -u postgres postgresql psql -U postgres
> \du
> \l
# Check extension
postgres=# SHOW shared_preload_libraries;
shared_preload_libraries
--------------------------
vchord.so
(1 row)
```
docs/services/infra/prometheus.md
+12
View File
@@ -0,0 +1,12 @@
# Prometheus
## Operation
Refer to Ansible playbook
## Verification
- Check Caddyfile \(without caddy, use 9090 ports\)
- https://prometheus.ilnmors.internal
- Status:Target Health
- Check `Endpoint localhost:9090 ` with green circle
- Status:command-line flag
- Check `--web.enable-remote-write-receiver: true`
docs/services/systemd/systemd-networkd.md
+35
View File
@@ -0,0 +1,35 @@
# systemd-networkd
- Use `networkctl` and the files in `/etc/systemd/network`
- link file
Link file links hardware interface and kernel while booting
- netdev file
netdev file defines virtual interface \(port, bridge\)
- network file
network file defines network option above interfaces
## commands
- reload
- networkctl reload
- networkctl reconfigure \[interface name\]
## references
- https://manpages.debian.org/testing/systemd/systemd/networkctl.1.en.html
- https://manpages.debian.org/testing/systemd/systemd.link.5.en.html
- https://manpages.debian.org/testing/systemd/systemd.network.5.en.html
- https://manpages.debian.org/testing/systemd/systemd.netdev.5.en.html
## Plans
- Hypervisor's linux bridges work as L2 switch
- br0 is completely L2 switch \(LinkLocalAddressing=no\)
- br1 has ip address for hypervisor itself, but basically works as L2 switch whitch can deal with VLAN tags; id=1,10
- Firewall's port \(wan\) works as Gateway which can conduct NAT
- Firewall's port \(clients\) works as trunk port which can deal with VLAN tags; id=1,10,20
- Firewall's port
- client, id = 1
- server, id = 10
- user, id = 20
- wg0
docs/services/systemd/systemd-quadlet.md
+67
View File
@@ -0,0 +1,67 @@
# systemd-quadlet
Quadlet is for defining container configuration and lifecycle combining systemd and podman.
## Rootless container
Containers should be isolated from host OS. However, docker runs with root permission on daemon \(dockerd\). This means when one docker container has vulnerability and it is taken over, all the host system authority is threatened. Rootless container, podman runs without root permission and daemon so that even if one of containers is taken over, prevent the damage in host's normal user authority.
Rootless container maps UID/GID between host and its own following namespace. Host's user UID/GID is mapped with container's root, and host's subuid/subgid defined on `/etc/subuid`, `/etc/subgid` is mapped with container's user UID/GID by default.
- Default `/etc/subuid` and `/etc/subgid`
- user:100000:65536
- host user 1000 > container root 0
- host subuid 100999 > containers 1000
Rootless services originally depends on session. It is necessary to set `linger` to guarantee the service health regardless the session.
- sudo loginctl enable-linger user
- ls /var/lib/systemd/linger/user
## Quadlet
Quadlet defines specification of container in `.container` file and generates `.service` automatically for systemd. systemd can manage the container like its own service with `systemctl` command.
```ini
# $HOME/.config/containers/systemd/a.container
[Quadlet]
# Don't make a dependencies
DefaultDependencies=false
[Unit]
Description=app
After=network-online.target
Wants=network-online.target
BindsTo=a.service
Requires=a.service
[Service]
ExecStartPre=/bin/sh -c 'echo "Waiting for infra-postgresql..."; until nc -z postgresql.ilnmors.internal 5432; do sleep 1; done;'
[Container]
Image=localhost/app:1.0.0
ContainerName=app
PublishPort=2080:80/tcp
PublishPort=2443:443/tcp
AddHost=app.service.internal:host-gateway
Volume=%h/data/containers/app:/home/app:rw
Environment="ENV1=ENV1"
Secret=ENV_NAME,type=env
Secret=app.file,target=/path/of/secret/file/name
# podman run [options] [image] example --config exconfig
Exec=example --config exconfig
# If you want to change Entrypoint itself, use
Entrypoint=sh -c 'command'
[Install]
# Guarantee auto start
WantedBy=default.target
```
docs/services/vmm/libvirt/cloud-init.md
+125
View File
@@ -0,0 +1,125 @@
# cloud-init and seed.iso
## reference
- https://cloudinit.readthedocs.io/en/latest/reference/examples.html#yaml-examples
## packages
- cloud-image-utils
- genisoimage
## meta-data
- meta-data.yaml
```yaml
instance-id: test-vm-$DATE
local-hostname: test
```
## user-data
- user-data.yaml
```yaml
#cloud-config
# Command which is excuted when systemd boots
bootcmd:
- groupadd -g 2000 svadmins || true
hostname: test
# auto resize partition and filesystem depends on virtual disk image
growpart:
mode: auto
devices: ['/']
ignore_growroot_disabled: false
resize_rootfs: true
# prohibit root login
disable_root: true
users:
- name: test
gecos: test
primary_group: svadmins
groups: sudo
lock_passwd: false
passwd: $(openssl passwd -6 'password')
shell: /bin/bash
ssh_authorized_keys:
- 'ssh-ed25519 KEY_VALUE'
write_files:
# ip_forward option
- path: /etc/sysctl.d/ipforward.conf
content: |
net.ipv4.ip_forward=1
permissions: '0644'
# systemd-networkd files
- path: /etc/systemd/network/00-eth0.link
content: |
[Match]
MACAddress=0a:49:6e:4d:00:00
[Link]
Name=eth0
permissions: '0644'
# - path: /etc/systemd/network/files....
# ssh host files
- path: /etc/ssh/id_test_ssh_host
content: |
-----BEGIN OPENSSH PRIVATE KEY-----
-----END OPENSSH PRIVATE KEY-----
permissions: '0600'
- path: /etc/ssh/id_test_ssh_host.pub
content: |
ssh-ed25519 KEY_VALUE TEST_SSH_HOST
permissions: '0644'
- path: /etc/ssh/id_test_ssh_host-cert.pub
content: |
ssh-ed25519-cert-v01@openssh.com KEY_VALUE TEST_SSH_HOST
permissions: '0644'
# sshd_config
- path: /etc/ssh/sshd_config.d/cert.conf
content: |
HostKey /etc/ssh/id_test_ssh_host
HostCertificate /etc/ssh/id_test_ssh_host-cert.pub
permissions: '0644'
- path: /etc/ssh/sshd_config.d/permit_root_login.conf
content: |
PermitRootLogin no
permissions: '0644'
runcmd:
# systemd-networkd interface loading
- update-initramfs -u
- systemctl disable networking
- systemctl enable systemd-networkd
- systemctl enable getty@ttyS0
- sync
power_state:
delay: "now"
mode: reboot
message: "rebooting after cloud-init configuration"
timeout: 30
```
## network-config
- network-config.yaml
```yaml
version: 2
ethernets: {}
network:
config: disabled
```
## Create seed.iso
```bash
cloud-localds -N network-config test_seed.iso user-data.yaml meta-data
```
docs/services/vmm/libvirt/undefine.md
+18
View File
@@ -0,0 +1,18 @@
# Undefine VM
Undefine VM is critical to whole systme.
## process
```bash
# Shutdown VM
systemctl --user stop "$VM_NAME".service
## virsh stop|destroy "$VM_NAME"
# Undefien VM
virsh undefine "$VM_NAME" --nvram # All vms use uefi, so the option, `--nvram` is needed to remove nvram file
# Delete VM files
sudo rm -r /var/lib/libvirt/images/"$VM_NAME".qcow2
sudo rm -r /var/lib/libvirt/seeds/"$VM_NAME"_seed.iso
```
docs/specifications/environments.md
+154
View File
@@ -0,0 +1,154 @@
# Server and client environments
## Console
- OS: WSL2 \(Debian 13\)
- Processor: 4vCPU
- Memory: 4GiB
- Disk:
- 32GiB for `/` \(VHD file\)
- Services:
- [x] Terminal
- [x] Step-CLI
- [x] Ansible
- Git
- Kopia
- [x] cloud-image-utils
## vmm \(Hypervisor\)
- OS: Debian13
- Processor: pCPU \(N150\)
- Memory: 3GiB \(margin\)
- KSM allows more than 3GiB for vmm
- MAC:
- c8:ff:bf:05:aa:b0
- c8:ff:bf:05:aa:b1
- Disk:
- SSD:
- 64GiB for `/` \(ext4 in LVM\)
- 700GiB for `/var/lib/libvirt` \(ext4 in LVM\)
- Services:
- [x] QEMU/KVM
- [x] libvirtd
- [x] ksmtuned
## fw \(Firewall\)
- OS: Debian13
- Processor: 2vCPU
- cputune.shares 2048
- Memory: 4GiB
- MAC:
- 0a:49:6e:4d:00:00
- 0a:49:6e:4d:00:01
- Disk:
- SSD: 64GiB for `/` \(ext4 in qcow2 file\)
- Services:
- native packages:
- [x] nftables \(firewall based on ZONE\)
- [x] Suricata \(IDS\)
- [x] CrowdSec LAPI \(IPS\)
- [x] Kea DHCP
- [x] Wireguard-tool
- [x] BIND9 \(Local authoritative DNS\)
- [x] Blocky \(Resolver DNS\)
- Scripts:
- [x] ddns.sh
## infra \(Infrastructure\)
- OS: Debian13
- Processor: 2vCPU
- cputune.shares 1024
- Memory: 6GiB
- MAC: 0a:49:6e:4d:01:00
- Disk:
- SSD: 256GiB for `/` \(ext4 in qcow2 file\)
- Services:
- Rootless containers:
- [x] PostgreSQL
- [x] lldap
- [x] Step-CA
- [x] Caddy \(with nsupdate\)
- [x] Prometheus \(alloy - push\)
- [x] Loki \(alloy\)
- [x] Grafana
<!--
Mail service is not needed, especially Diun is not needed.
- Postfix
- Dovecot
- mbsync
- Diun
-->
- Study \(Rootless container\):
- Kali
- Debian
## auth \(Authorization\)
- OS: Debian13
- Processor: 2vCPU
- cputune.shares 512
- Memory: 2GiB
- MAC: 0a:49:6e:4d:02:00
- Disk:
- SSD: 64GiB for `/` \(ext4 in qcow2 file\)
- Services:
- Rootless containers:
- [x] Caddy \(with nsupdate, crowdsec-http, crowdsec-bouncer module\)
- [x] authelia
## app \(Application\)
- OS: Debian13
- Processor: 4vCPU
- cputune.shares 1024
- Memory: 16GiB
- MAC: 0a:49:6e:4d:03:00
- Disk:
- SSD: 256GiB for `/` \(ext4 in qcow2 file\)
- HDD: 4TB for `/home/app/data` \(btrfs\)
- VFIO \(Hardware passthrough):
- Graphic: N150 iGPU
- Disk: SATA Controller
- Services:
- OIDC native services:
- OpenCloud \(with Radicale, Collabora Web Office\)
- Vikunja \(with CalDAV\)
- Gitea
- Outline
- Wiki.js
- WriteFreely
- Immich
- MediaCMS
- Funkwhale
- Kavita
- Audiobookshelf
- we-promise/sure - budget
- Paperless-ngx
- Miniflux
- Linkwarden
- Ralph
- Conduit
- SnappyMail
- Vaultwarden
<!--
- n8n
-->
- Forward_auth
- Homepage
## External Backup server
- OS: DSM \(Synology\)
- Processor: pCPU \(Realtek RTD1619B\)
- Memory: 1GiB
- MAC: 90:09:d0:65:a9:db
- Disk:
- HDD: 4TB
- Services:
- SFTP
- Kopia repository server
- CloudSync \(Upload backup files to Cloud\)
docs/specifications/hardwares.md
+67
View File
@@ -0,0 +1,67 @@
# Hardware specifications
## Servers
### Main server
- Aoostar WTR Pro N150
- Processor: Intel N150 \(4C4T\)
- Graphic: Intel UHD Graphics
- 2.5 Gbps NIC x 2
- M.2 Slot x 2 \(SSD, WiFi\)
- SATA bay x 4
- 279,900 KRW
- Samsung DDR4 SO-DIMM 3200 32G x 1
- 106,900 KRW
- Samsung 980 Pro 1TB TLC x 1
- 276,000 KRW \(Previously owned\)
- 3RAYS glaicer 6 m.2 SSD heatsink x 1
- 7,330 KRW
- HGST Ultrastar 7K4000 2TB HDD x 3
- 99,000 KRW
- HGST Ultrastar 7K2 2TB HDD x 1
- 43,000 KRW
- Total price: 698,030 KRW \(1,460,030 KRW with previously owned ones\)
### Backup server
- Synology DS124
- Processor: Realtek RTD1619B \(4C4T\)
- Memory: DDR4 1GB
- 1 Gbps NIC x 1
- SATA bay x 1
- 242,000 KRW
- TOSHIBA DT02 4TB x 1
- 55,000 KRW
- Total price: 297,000 KRW
### Console \(Laptop\)
- Microsoft surface laptop 7th ZGJ-00021
- Processor: Snapdragon X Plus \(ARM64, 10C10T\)
- Memory: LPDDR5x 16GB
- SSD: 256GB SSD
- OS: Windows11 Home
- 1,290,210 KRW
- Microsoft surface USB-C travel hub x 1
- 157,890 KRW
- Total price: 1,448,100 KRW
### External HDD
- EFM 3.5 External HDD case ipTIME HDD3135 Plus x 1
- 29,400 KRW
- Seagate BARRACUDA HDD 2TB x 1
- 99,000 KRW \(Previously owned\)
- Total price: 128,400 KRW
## Devices
### Switch
- TP-link TL-SG108E x 1
- 1 Gbps NIC x 8
- IEEE 802.2q
- 39,900 KRW
- Total Price: 39,900 KRW
### Monitor
- Samsung S6 LS27F610 x 1
- 277,000 KRW
- Total price: 277,000 KRW
docs/specifications/matrix.md
+117
View File
@@ -0,0 +1,117 @@
# Matrix
## UID/GID Matrix
### Table
|name|uid|gid|comments|
|:-:|:-:|:-:|:-:|
|svadmins|-|2000|server group|
|vmm|2000|2000|hypervisor|
|fw|2001|2000|firewall|
|infra|2002|2000|infrastructure|
|auth|2003|2000|authentication and authorization|
|app|2004|2000|services|
|console|2999|2000|console node\(surface\)|
### subuid and subgid
- user:100000:65536
## Switch ports matrix
### 8 Ports main switch
|port number|node|subnet|id|
|:-:|:-:|:-:|:-:|
|1|WTR Pro N150|Trunk|-|
|2|AP\(Preparation\)|USER|20|
|3|DS124\(NAS\)|CLIENT|1|
|4|Console|CLIENT|1|
|5|Printer|CLIENT|1|
|6|-|-|-|
|7|-|-|-|
|8|-|-|-|
## IP matrix
### Subnet
|name|IPv4|IPv6|id|
|:-:|:-:|:-:|:-:|
|CLIENT|192.168.1.0/24|fd00:1::/64\(ULA\)|1|
|SERVER|192.168.10.0/24|fd00:10::/64\(ULA\)|10|
|USER|192.168.20.0/24|GUA from ISP|20|
|WG0|192.168.99.0/24|fd00:99::/64\(ULA\)|-|
### Host
#### console:
- CLIENT
- 192.168.1.20
- fd00:1::20
- WG0
- 192.168.99.20
- fd00:99::20
#### fw
- CLIENT
- 192.168.1.1
- fd00:1::1
- SERVER
- 192.168.10.1
- fd00:10::1
- USER
- 192.168.20.1
- GUA SLAAC
- WG0
- 192.168.99.1
- fd00:99::1
#### blocky \(fw\)
- SERVER
- 192.168.10.2
- fd00:10::2
#### bind \(fw\)
- SERVER
- 192.168.10.3
- fd00:10::3
#### vmm
- CLIENT
- 192.168.1.10
- fd00:1::10
- SERVER
- 192.168.10.10
- fd00:10::10
#### infra
- SERVER
- 192.168.10.11
- fd00:10::11
#### auth
- SERVER
- 192.168.10.12
- fd00:10::12
#### app
- SERVER
- 192.168.10.13
- fd00:10::13
#### VLAN switch
- CLIENT
- 192.168.1.2
- fd00:1::2
#### ds124
- CLIENT
- 192.168.1.11
- fd00:1::11
#### Printer
- CLIENT
- 192.168.1.101
- fd00:1::101
docs/theories/network/dhcp.md
+53
View File
@@ -0,0 +1,53 @@
# DHCP (Dynamic Host Configuration Protocol)
Before DHCP emerged, every client had to set their own static IP or using RARP\(Reverse Address Resolution Protocol\). They have critical problems.
- Static IP
- Each host has their own IP regardless they run or not. It cause lack of IP address.
- If administrator made configuration mistake, the network itself could stop. For instance, IP conflict, or subnet configuration error, etc.
- RARP
- RARP works on L2, it makes hard to implement on hardware.
- RARP server had to exist on every subnet(L2), it was inefficient.
To solve this problem, BOOTP was developed and it evolved as DHCP. DHCP works on L3, and like its name they lease IP dynamically. It allocates IP to hosts with 3 steps. Lease, renewal, and release.
## Lease
When DHCP server gets request from host to allocate IP, DHCP server choose one IP address from its subnet pool, and it responds IP/subnet and information to the hosts. It follows the process called DORA.
- DHCP Discover
The host which has no IP address broadcast discover packet including its MAC to local network. Only DHCP server responds this packet, the others discard the packets.
- DHCP Offer
When DHCP server gets discover packet, DHCP server broadcast Offer packet including host's MAC of which sends discovery packet. Only host which sends discovery packet, the others discard the packets.
- DHCP Request
When the host gets offer packet, it consider the network has DHCP server and broadcast request packet to the local network. More than one DHCP server can exist in one network, therefore host broadcast request packet.
- DHCP ACK
When DHCP server gets the Request packet, it searches for an IP address which can be allocated in its pool. When DHCP server finds an available IP address, it sends to ACK packet including IP address and subnet, and optional information(DNS, gateway, etc...) with reservation time.
## Renewal
The host allocated IP from DHCP try to renew its IP before it expires. There are two chances to renew reservation time. The first try to renew is when the reservation time remains half of them; T1. In this time, host uses unicast. If the first try failed, then it try one more time when 87.5% of the lease time has passed; T2. In this time, the host uses broadcast. All try fails, client gives up the leased IP and try the lease process again. DHCP server release the IP address from the host.
- DHCP Request
The host sends request packet to DHCP server as unicast.
- DHCP ACK
When DHCP server gets the request, it sends ACK packet to the host as unicast.
## Release
When the host doesn't use IP address anymore, DHCP server makes IP as available IP in its pool. Especially, client can send `DHCPRELEASE` to DHCP server explicitly when it doesn't need IP address.
## DHCP relay
Commonly, DHCP is located in router. Because router is the center of networks, and it takes charge of number of networks. However, DHCP server doesn't have to be located in router because of existence of DHCP relay. When router gets DHCP packets (DORA), router can relay the packets as unicast between host and DHCP server which are in different subnet.
docs/theories/network/dns.md
+99
View File
@@ -0,0 +1,99 @@
# DNS (Domain Name System)
In the beginning of the internet, there were a few hosts on networks. It was possible to manage all hosts on network via IP address or domain name in `/etc/hosts` file in each servers. However, it is hard for people to match and remember what IP addresses means. When the internet environment became bigger and bigger, the complex of route the target server would be harder. To solve this problem, the DNS emerged as a translator between IP address and domain name. In modern internet environment, DNS has hierarchy structure from root to TLD, TLD to authoritative server for efficiency.
## Structure of DNS
### Communication
- DNS: 53 tcp/udp
DNS communication basically uses 53/udp port. However, in the modern internet environment; which means complex environment sometimes the size of packet is above 512 bytes. In this case, DNS uses 53/tcp too. The vulnerability of DNS is that all communication is on plain data. Everyone can conduct sniffing attack towards DNS packet.
- DoT (DNS over TLS): 853 tcp
DoT was developed to encrypt DNS query. DoT uses [TLS](./tls.md) to request query. This protocol uses TLS. Moreover, because of TLS, nobody can do sniffing attack towards DoT. However, it uses specific port 853. If ISP block the 853 or analyze 853 port, the pattern of usage will be analyzed or even you cannot use DoT itself. Additionally, there is also DNS over DTLS which uses 853 udp.
- DoH (DNS over HTTPS): 443 tcp/udp
DoH is very similar with DoT. It uses TLS, and it was developed to encrypt DNS query. there is just one difference. This uses https(443 tcp/udp) instead of 853 tcp. https is standard of web protocol, so it is hard to analyze someone sends DNS request or common web packets. It means, ISP or government cannot block 443 port itself or analyze the pattern of DNS query. Since 2022, there's the new standard DNS over HTTP/3 which uses 443 udp port.
- DNSSEC (DNS SECurity extensions)
Originally, client couldn't verify integrity of the response from DNS server. If malicious attacker could get authority of cache DNS server to change their records, all clients would get affected. (i.e. pharming attack). DNSSEC is a protocol to guarantee integrity of DNS record. DNSSEC protocol adds some records in zone, RRSIG(Resource Record Signature), DNSKEY, DS, NSEC, CDNSKEY, CDS. All resolver DNS verify integrity of their records to authoritative DNS with these records. This process is similar to PKI, the chain of trust.
- ECH (Encrypted Client Hello)
Basically, client hello packet has SNI (Server Name Indication). Even though all communication under TLS is encrypted, but to start session the packet has to contain the SNI to identify server. To encrypt this information, SNI the ESNI(Encrypted SNI) was developed in 2018 based on TLS 1.3. However, ESNI just could encrypt SNI information. Now, since 2020, the new standard ECH was developed to supersede ESNI. ECH not only encrypt SNI but also encrypt all client hello process. ECH is latest protocol, and it has a lot of dependency in DNS server, service server and client. When all of them supports ECH, then user can use ECH. Because when ECH encrypts client hello data client need the target server's public key (certificate), it has to look up from encrypted DNS (DoH or DoT).
### Zone
DNS server has zones; Forward zone and Reverse zone.
- Forward zone
Forward zone has basically information of the pair of domain and IP address. The role of this zone is change domain name to IP address. The domains are managed by IANA, TLD is already reserved. (i.e. `.com`, `.org`, etc...) For private network, `.home.arpa` or `.internal` are reserved.
- Reverse zone
Reverse zone also has basically information of the pair of IP address and domain. The role of this zone is change IP address to domain name. To change domain to IP address it uses specific domain name. \[reversed_ip_address\].in_addr.arpa (i.e. 1.168.192.in-addr.arpa)
### Records
Each zone has their record type. If zone were a kind of DB, record would be a data of DB. There is basic records type below.
- SOA type
Information of ZONE management. Every zone has this SOA type record.
- NS type
Designate authoritative name server of domain zone
- A type
Mapping domain to IPv4 address
- AAAA type
Mapping domain to IPv6 address
- PTR type
Mapping IP address to domain
- CNAME type
Mapping domain to domain. CNAME type is kind of alias of domain. It can't have IP address value. The query acts recursively, and it gets IP address at the end.
### Key
There is the keys to control DNS records or zone, even DNS server itself.
- rndc key
This key is to control DNS server itself. When rndc key set on DNS server, client can control DNS server with this key like, reboot server, load or unload zone. rndc key is basically generated by `rndc-confgen` command and it is defined on `rndc.conf` and `named.conf`.
- tsig key
This key is to guarantee integrity when the server syncronize zone data between other servers (usually master-slave server). It is possible update records via this key depending on the setting. Therefore, tsig key is usually used for DDNS or DNS-01 challenge. The key is generated in the DNS server, and it defined in `named.conf`.
## DNS Server type
DNS server basically separated as authoritative DNS and recursive DNS.
### Authoritative DNS
Authoritative DNS has literally authority of domain zone. It doesn't ask recursive queries towards other DNS server in case of the query that is in its authoritative zone. It is necessary to use DNS-01 challenge (ACME protocol).
### Recursive DNS
Recursive DNS oppositely doesn't have authority of the records in its zone. When it gets query request, it ask recursive query towards authoritative DNS. It can store the information of records (cache) and give response towards client with the cache.
## Split Horizon DNS
Split Horizon DNS means getting different IP address depending on where the client exists. For instance, if there were the domain `example.com`. This domain has its own private IP address, simultaneously own public IP address (from NAT). When client request the query `example.com` in the private network, private DNS would respond its private IP address. However, when the client request the query in the WAN network, public DNS would respond its public address. Client can access `example.com` in both case, but the IP address which client respond are different. To use this protocol, the network route will be efficient because the packet doesn't have to go out to the WAN area in private network. Basically, it is implemented internal authoritative DNS and recursive DNS. Recursive DNS decides where to send the query based on domain.
## DDNS (Dynamic DNS)
Public IP address can be changed by ISP at any time. It is hard (or expensive) to get static public IP address by ISP. However, the service (server) always guarantee their availability regardless what is their IP. DDNS is basically the protocol to change A or AAAA (or CNAME) records in DNS as server's current IP. Server keeps checking their current public IP and when it changes the server send the request to change its A or AAAA records to public authoritative DNS server with authentication with API key or tsig key.
docs/theories/network/email.md
+241
View File
@@ -0,0 +1,241 @@
# Email service
Email is the mail service online via the internet. ARPANET was developed in 1969, since then there has been many attempts to send messages via the internet. The mail which uses `@` character in 1971 and `SMTP(Simple Mail transfer Protocol)` was developed to standardize various ways to email.
## Component of Email service
### Address
Basically, Email address has format like this. `local-parts@domain`. `local-parts` is identifier, and `domain` is service provider's domain. Following RFC 5321, `domain` doesn't distinguish it upper or lower case. `local-parts` must distinguish them, but practically they doesn't.
### MUA (Mail User Agent)
MUA is the client of Email. The user can write Email, or read the Email which they got recieved. For instance, Outlook, Thunderbird, etc.
### MTA (Mail Transfer Agent)
This is the essential part of Email service. MTA transpers the mail to other MTA or MDA. For instance, Postfix, sendmail, Exim, etc.
### MDA (Mail Delivery Agent)
MDA recieves the mail from MTA, and it store the mail on receivers' mailbox. Sometimes, it is combined MTA or IMAP/POP3 servers. For instance, Dovecot LDA, Procmail, etc.
### Flow of Email service
- User writes the mail on MUA.
- User sends the mail from MUA to MTA using SMTP submission protocol.
- MTA checks receiver's domain, and transfer the mail to other MTA which takes charge of that domain.
- MTA recieves the mail and sends receiver's MDA.
- The receiver's MUA access to the MDA such as IMAP or POP3 server. the receiver can check and read the email on their MUA.
## Protocols
### SMTP (Simple Mail Transfer Protocol)
SMTP is standard of email transfer protocol internet defined on RFC 5321. This protocol is used when MUA sends the mail to MTA, and MTA sends the mail other MTAs. This protocol takes charge of all process of transportation of the mails.
#### Detail of SMTP
##### Start
- Connection
The client and server make the connection via SMTP port (25/tcp).
##### Greeting
- `220` code
The server sends `220` code to the client, they are ready.
- `HELO` or `EHLO`
The client sends `HELO` or expand version of `HELO`; `EHLO` command to server to introduce itself.
##### Designate sender and recipient
Use the command below, they designate sender and recient.
- `MAIL FROM:<sender@domain>`
- `RCPT TO:<recipient@domain>`
- If there were various recipients, use this command as much as recipients number.
##### Transper the mail data
- `DATA` and `354`
The client sends `DATA` command to server. After the server responds with `354` code, client sends the data including mail header (From, To, Subject), and content of mail. The end of data is `.`.
##### End
- `QUIT`
The client sends `QUIT` command, the connection is terminated.
##### Ports
- `25/tcp`
Traditional SMTP's standard port. All content using `25/tcp` is not encrypted. Because of security and SPAM problems, a lot of ISP block the `25/tcp` port of common user.
- `587/tcp` (Submission)
The standard port of SMTP for encryption. Generally MUA sends the mail to MTA with this port. It is needed to use encrypted connection via `STARTTLS`
- `465/tcp` (SMTPS)
This port used to be used for TLS/SSL for SMTP. This is not standard, so it is recommended to use `587/tcp` port for TLS/SSL of SMTP. However, even now this is generally and commonly used.
##### Security
SMTP is very old protocol, and this protocol use plain data. It is recommended to use `STARTTLS` or `SMTPS` to encrypt data for security.
- `SMTPS`
It uses TLS/SSL from the beginning of connection via `465/tcp`
- `STARTTLS`
It uses TLS/SSL after beginning of connection via `587/tcp` as plain data, and start encryption with `STARTTLS` command.
##### Authentication
It is necessary to use users' identity like name and password to prevent anyone can sends malicious mail using server. SMTP uses SASL (Simple Authentication and Secuirty Layer) machanism to authenticate its users.
##### Relay
MTA has to send the mail to the other MTA for guarantee the mail can arrive the recipitent. MTA uses `relay` function for this. Make sure to allow this function for authenticated user or trusted network to prevent malicious usage.
### IMAP (Internet Message Access Protocol)
IMAP is the protocol to read and manage the mails from remote MDA (mail server). The difference between POP3 is that IMAP can manage the mail and its mailbox remotely even without download. It is defined on RFC 3501.
#### Detail of IMAP
IMAP is the protocol to have a communication with various commands while the connection is stable. The client sends specific `tag` in front of command, and the server responds with `tag` to process the actions.
##### Authentication
- `LOGIN` or `AUTHETICATE`
IMAP authenticate the user with `LOGIN` command with ID and password or `AUTHENTICATE` command with SASL.
##### Mailbox
- `LIST`
- `SELECT`
- `CREATE`
- `DELETE`
- `RENAME`
##### Mail
- `FETCH`
IMAP can take the mail list, the mail itself, or content of the mail, even the attachment in the mail.
##### Statement
- `STORE`
- `\Seen`
- `\Flagged`
- `\Answered`
- `\Deleted`
IMAP can set the status flag of mail with command flags.
##### Search
- `SEARCH`
IMAP can search the mail with various condition of the mail (Sender, title, contents, date, etc) from server.
##### Ports
IMAP strongly recommend to use TLS/SSL with `STARTTLS`. Even though the beginning of conversation is not encrypted, TLS/SSL is applied with the `STARTTLS` command.
- `143/tcp`
The basic IMAP port. It is mendetory to use `STARTTLS` to use IMAP with this port.
- `993/tcp` (IMAPS)
This port uses TLS/SSL in the beginning of communication. It is not a standard but it is generally and commonly use for security.
##### Synchronization
IMAP basically server's mail and mail list, so wherever you access the mail you can see the same condition and status of mailbox. When one mail is modified on one device it is applied all devices simultanaeously.
- `IDLE`
This command supports to maintain connection between server and client, when the new mail comes or the status is changed the client can get notification immediately.
### POP3 (Post Office Protocol version 3)
POP3 protocol is basically designed to download the mail on local client from remote mail server. It is defined on RFC 1939. The biggest difference between IMAP and POP3 is, POP3 basically delete the mail at the server after downloading.
#### Detail of POP3
##### Authorization
- `USER` and `PASS`
The client connect to server and it conduct authentication with `USER` and `PASS` command.
##### Transcation
- `STAT`
- `LIST`
- `RETR <msg_number>`
- `DELE <msg_number>`
- `RETR`
POP3 uses various commands to download or delete the mail. It checks the number of mail and size with `STAT`, downloads the mail with `RETR`, deletes the mail with `DELE`, and save the mail on client with `RETR`.
##### Update
When the client sends `QUIT` command, then server deletes the mails which have `DELE` marks from server and terminate the connection.
##### Ports
POP3 strongly recommend to use TLS/SSL with `STARTTLS`. Even though the beginning of conversation is not encrypted, TLS/SSL is applied with the `STARTTLS` command.
- `110/tcp`
The basic port of POP3. It is mendetory to use `STARTTLS` to use IMAP with this port.
- `995/tcp` (POP3S)
This port uses TLS/SSL in the beginning of communication. It is not a standard but it is generally and commonly use for security.
##### Simplity and locality
POP3 basically delete the mail from mail server, the mail is only on the local client. However, it doesn't require the complex features like IMAP, it can have simplity.
---
## local mail service in homelab
### SMTP server (MTA)
#### Postfix
Postfix will be used as MTA which takes charge of `@ilnmors.internal` domain. However, Postfix in this homelab will never open towards WAN environment. It works as local private MTA. The internal services (Gitea, OPNsense, Prometheus, etc) will sends the mail via `587/tcp` to Postfix. When it needs to send mail towards WAN, it will use `relayhost` function and external Email services such as Google or Naver, etc. `relayhost` makes postfix as one of a `client` not a `MTA`. It means, administrator never takes care about IP reputation or SPAM problems. WAN area's `MTA` function is delegated to public mail service providers.
### IMAP/POP3 server (MDA)
#### Dovecot
Dovecot will be used as IMAP server of local private MTA; Postfix. The user can use MUA (Thunderbird, Outlook, or mail application, even Roundcube webmail) to access the private mail `
`@ilnmors.internal` via Dovecot. The user will ues `993/tcp` to access Dovecot, and Postfix store the mails on Dovecot.
#### mbsync
mbsync will be used as IMAP client of public MTA; Google or Naver. This will fetch public mail `@external-domain.com` to local Postfix from public mail service provider, and eventually the user can access the mail on Dovecot. However, it is important not to delete the mails from public mail servers with proper configuration.
### MUA
#### SnappyMail web mail
This will be used as MUA server on `app` server to access all mails at the same space.
docs/theories/network/link-local.md
+30
View File
@@ -0,0 +1,30 @@
# Link-local address
link-local address is for reserved subnets for L2 communication.
## IPv4
### APIPA
When the client couldn't get IP address from DHCP, OS automatically allocate IP address subnet 169.254.0.0/16. This address can never pass L3 point \(router\). It is usually used for internal communication in cloud environment, or PASTA network in containers.
### RFC1918
These are originally IP addresses subnet 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 which can communicate beyond L3 point \(Router\). However, it is reserved for LAN \(Local Area Network\) because of the lack of the number of IPv4 address. They can communicate to the other subnets but they cannot be used on WAN environment, which means ISP cannot allocate these IP subnet to their client.
## IPv6
Link-local address is very important in IPv6 unlike IPv4. Basically, every edge node must have GUA in IPv6 protocol, and a node can have number of IPv6 addresses on an interface. So, they communicate with auto-generated linklocal address fe80::/10 in L2 area, and they communicate with ULA or GUA in L3 Area.
## Linklocal in PASTA
### IPv4
- 127.0.0.1: container itself
- 169.254.0.0/16: container and host communication \(linklocal\)
- RFC1918 for private LAN
- WAN
### IPv6
- \[::1\]: container itself
- \[fe80::\]: container and host communication \(linklocal\)
- \[fd00::\]: (ULA) for private LAN
- [ Global IPv6 ]
docs/theories/network/tls.md
+90
View File
@@ -0,0 +1,90 @@
# TLS/SSL
TLS(Transport Layer Security) is the protocol that encrypts communication with certificates under [PKI](../pki/pki.md). Originally, the communication encryption protocol was suggested as SSL(Secure Socket Layer) in 1995 by Netscape. Before the emergence of TLS/SSL protocol, all communication in web(or database, some more protocols) used plain text, even it contained sensitive data like password. After 1996, the update of SSL protocol stopped at version 3.0, and it has its own vulnerability. The next version of SSL is TLS. Virtually, a lot of people treat SSL as the same as TLS.
## Encryption
Before talking about TLS, it is essential to understand what is encryption. Basically, encryption is to hide plain data using algorithm and key mathematically. Encryption can categorized by various criteria, but in this case focusing on key.
### Symmetric key encryption
This way needs only one key to encrypt and decrypt. Just because using one key to encrypt and decrypt, it doesn't require complex calculation. This algorithm is fast, and easy to handle huge data. However, the key is only one so for security the key should be protected.
### Asymmetric key encryption
This way needs two keys to encrypt and decrypt. It means the key for encryption and the key for decryption are different. This uses very complex algorithm to seperate keys. It makes this slow and hard to handle huge data. However, the keys are devided so, one key can be publically shared.
## Principle of TLS
The PKI is necessary to use TLS communication. TLS uses both of algorithm to encrypt the communication. When the communication were encrypted, the key should exist. For this process(to generate key for encryption), server needs X.509 certificate. The certificate contains data of server's public key, domain, and extra informations. To start communication, the server and the client conduct specific process, called `TLS-handshake`. A client sends the request to start communication. The server accepts this request, and negotiates the protocol what they will use in communication. After negotiation, they start to process to generate symmetric key for communication. In this part, there are two protocol to generate symmetric key: RSA and Diffie-hellman.
### RSA (Legacy way)
In this process server and client uses server's public key and private key. A client generates metadata (Pre-master secret) to generate symmetric key. Metadata is encrypted with server's public key and sent to the server. The server decrypts the data from a client with its private key. They uses this data to generate symmetric key. When this process ends, they share the same symmetric key to encrypt content of communication.
- Cons of RSA
Security strength of RSA depends on the server's private key. The server's private key was taken from whom have the record of past communication, all communication can be decrypted. Because session key itself (precisely, pre-master secret) is in the communication encrypted by server's public key. So, this way can't guarantee Forward Secrecy which means security of previous communication. Additionally, RSA needs high resource to encrypt and decrypt. Therefore, modern internet environment RSA way is not used frequently.
### Diffie-Hellman & ECDHE(Modern standard)
On the other hand, TLS can use Diffie-Hellman algorithm. This process doesn't exchange any clue of session key. In this process, the server and clients generate secret values. And they make specific results from calculation with public parameters. The server's private key signs on its results just for prooving they are not altered and authenticated. When they exchange each other's results, both of them generate the same session key from results and their own secret value. All process is publically open, but except the server and client themselves, nobody can calculate the secret values and session key mathematically. After all process is done, finally the client combines all communication value between server. And it makes hash value of this, and encrypt the hash value with the session key. The client sends it to the server as finished message, and server verify this. This key is not permanent, it is temporary. There is no encryption way, so even if hacker could get the secret key of server, he wouldn't know what was the session key.
- Pros of DHE & ECDHE
On the contrary with RSA case, security strength of DHE & ECDHE doesn't depend on some specific key. Because when it generate session key, the server and client don't send any sensitive values (like pre-master secret) in this protocol. What they send to each other is open values. The server's public key is used for only signing. So, every session key can never exist after each session; PFS (Perfect Forward Secrecy). Additionally, Elliptic curve way can provide the same strength with shorter key comparing RSA.
The detail of `TLS-handshake` process is below.
## Detail of TLS handshake
### Start
- Client hello
The client sends the request to server with information including SNI (Server Name Indication), TLS version, cipher suite, and client random. SNI is the server's domain information to access. cipher suite is the protocol of encryption, and client random will be used to generate session key. Basically, the content of `client hello` isn't encrypted. It has a problem that ISP(Internet Service Provider) or government can conduct sniffing attack to the Client hello packet. To solve this problem, ECH (Encrypted Client Hello) was developed. However, ECH needs the support of DNS server, browser(host), and server itself. Today, a lot of servers and DNS servers don't support ECH, so it is hard to apply ECH in every environment.
- Server hello
When the server receives clients request, it designate how to communicate from the list of client hello. The server sends to the client with the information of what protocol, TLS version, cipher suite to use, and certificate, and server random. The way of encryption: RSA, DHE, ECDHE are including in the cipher suite.
### RSA
When the server and client decide to use RSA way, server sends certificate including public key to the client.
- Client key exchange
The client verifies `Server hello` with server's certificate using client's trusted CA list. The certificate is valid then the client generates `Pre-master secret` and encrypts this value with server's public key to send to the server.
- Session key generation
The server recieves `pre-master secret` encrypted by its public key that only server (who knows the private key of server) can decrypt. When the server decrypts this value successfully, the server and client knows three values: `Client random`, `Server random`, and `Pre-master secret`. The server and client generate `Session key` from these values individually.
### DHE or ECDHE
DHE and ECDHE follows the exactly same principle of Diffie-hellman algorithm. However, they have a difference of basement in mathematic. DHE is using discrete logarithm, but ECDHE is using elliptic curve discrete logrithm. This is a mathametical topic, so this is skipped. Just important thing is ECDHE is more efficient than RSA or DHE.
> Key length to guarantee the same level of security:
>
> - RSA/DHE: above 2048 bit
> - ECDHE: above 256 bit
- Server key exchange
The server generate the pair of temporary key based on its own principal. The server sends public temporary key to the client. The public temporary key is signed by server's private key.
- Client key exchange
The client verify server's temporary public key with server's public key. Simultaneously, client also generate the pair of temporary key based on its own principal. The client sends its public temporary key to the server.
- Session key generation
Both of them have their pair of temporary key, and other's temporary public key. They generate the same `Pre-master secret` from their temporary secret key and other's temporary public key. This process has no communication, they calculate the same `Pre-master secret` from themselves. When they have `Pre-master secret`, they generate session key from pre-master secret and client random and server random.
### Finish
After all process is done, the server and client both sends the `Finished message` encrypted by session key. When they can decrypt this messages, the session key is generated properly. They can start communication securely with session key.
## TLS 1.3
Current standard TLS 1.3 changed handshake process as 1-RTT(1 Round Trip Time) and there are no longer available to use RSA way (Legacy).
docs/theories/pki/pki.md
+102
View File
@@ -0,0 +1,102 @@
# PKI(Public Key Infrastructure)
PKI is defined on RFC4949 and RFC5280. PKI is neither one of the protocols nor algorithms, it is a huge infrastructure itself. It has various parts which are CA(Certification Authority), RA(Registration Authority), VA(Validation Authority) and Certificates.
## CA (Certification Authority)
CA is the most important part in PKI. It is the source of certification for the secure communication. CA has various roles, some can be delegated to the RA or VA. However, CA is essential in PKI. RA or VA are optional to support CA. The purposes of CA are as follows:
- Managing certification policy
- Issuing certificates
- Verifying certificates validity
- Managing CRL(Certification Revocation List)
- Cross-certification between root CAs
CAs often operate in a hierarchy. Each CA has their own role, and they are basically PAA(Policy Approving Authority), PCA(Policy Certification Authority), and CA(Certification Authority). Normally, PAA and PCA are usually combined as root CA, and CA is an intermediate CA.
### root CA
Root CA is simply CA of CA. It is the source of trust chain. Only a root CA can authenticate itself alongside other root CAs(cross certification) or intermediate CAs below it. Theoretically, and precisely root CA can be divided into PAA and PCA, but practically they operate as one authority, root CA.
### intermediate CA
Intermediate CA is the authority which issues certificates for end entities we can use (i.e. web servers). Their certificate is digitally signed (the hash value is signed by root CA's private key) by root CA, so every user can verify their trust with root CA's certificate (root CA's public key). The reason for using intermediate CA is for security. If CA's private key was violated, all chain of trust will be disrupted. When only root CA exists and its private key was violated, all secure communication will be threatened. To avoid and manage these threats root CA and intermediate CA are divided.
### Structure of CA
The basic structure of CA is very simple. It only contains private key, certificate (which is designated by X.509), and provisioner. When CA gets a request of sign on services' certificates it operates as shown below.
- Provisioner checks CSR (Certificate Signing Request) is valid on its policy
- When CSR is valid, it checks the content of certificates(x.509) and its policy
- CSR and content of certificates are valid, CA signs on the certificate with its private key (sign on hash value with CA's private key)
After CA's signing, every client can check its validation with CA's certificates (CA's public key)
## RA (Registration Authority)
Due to CAs many roles, all processes above are made to be inefficient when conducted. Therefore, CA delegates its registration role to RA. RA seeks to examine CSR and certificates' content. When the content of certificate and CSR are valid under its policy, RA would send request to CA to sign.
## VA (Validation Authority)
Additionally, just as RA, CA can delegate its validation role to VA. Basically, VA manages certificates' validation based on CRL (Certificate Revocation List). However, in modern internet environment, it is inefficient to manage all invalid certificates based on a list. To solve this problem, OCSP (Online Certificate Status Protocol) was developed. However, OCSP also has its limitation, the advanced way is being suggested.
### CRL (Certificate Revocation List)
Once the certificate gets signed from CA, it is always a valid cryptograph before it expires. However, in the real world some certificates have to be revoked in case CA's key is violated or the service no longer operates, etc. So, CA needs to manage these kinds of invalid certificates which doesn't expire, such as letting clients know that their certificates are not valid anymore. This is a concept of CRL. VA releases the CRL to clients, the clients download or update the list to judge the certificates' validation.
### OCSP (Online Certificate Status Protocol)
CRL was very effective to revoke invalid certificates but when internet is growing, a massive of invalid certificates appears it shows its limitation. Because to check certificates validation, clients need to download and update CRL from CA or VA. There are tons of invalid certificates, and CRL's size become bigger and bigger. OCSP's concept is when client request to check certificates validation, OCSP server response it. To use OCSP, clients don't need to download full CRL anymore. It is very effective to solve CRL's problem. However, it has problems too. OCSP is defined on RFC6960. OCSP's problems are below.
- Clients should expose its identity(IP address) to check certificates validation.
- VA can be SPOF(Single Point of Failure).
- It is very hard to match time syncronization
- The more requests come to VA, the more burden is on VA.(DoS, DDoS problem)
### OCSP stapling
OCSP's problem was that client should request to VA directly. So, when the certificate providers(like web server, db, etc.) request its validation and give the proof to client it, clients doesn't need to check the validation of certificate to VA anymore. It reduces VA's burden and time synchronization problems.
## Certificate
The form and content of certificate are standardized by X.509. X.509 format contains server's public key, domain(SAN), expired date, and sign of CA, and etc. When CA validate services or people who sent CSR(which contains information to create certificate with x.509 form), CA signs on their certificates (sign on hash value with CA's private key). Then, all clients can validate that certificates with CA's public key(CA's certificate) cryptographically. Ultimately, when clients trust CA, then they can trust services that have certificate signed by CA. However, even if the certificate were valid cryptographically, some certificates would not be valid. It is reason why CRL or OCSP is needed.
### The way to issue certificate
#### ACME
Basically, certificate is formed by X.509. The services which want to get certificate send request(CSR) to CA. When the services were open internet, and it should get the certificate from public CA like let's encrypt on its public domain name. Public CA always sign on public domain which is open internet, and it is the role of public CA. The protocol to automate this process is ACME. Before ACME protocol, getting a certificate was manual. The person in charge should send CSR to CA, and CA checks the CSR and return the certificate ... All process was manual. ACME protocol made this process automatically. Simply, ACME protocol checks services authority on domain with various way, and issues certificate.
- http-01 challenge
The server creates certificate file for CA. CA accesses the service server's specific directory via 80/TCP. When CA checked the file, CA issues certificate.
- DNS-01 challenge
The server sends request to authoritative DNS server to add specific record with TSIG key(API key). When the specific record successfully added on DNS server and CA can check this, CA issues certificate. This challenge can verify full authority of domain including sub domain, so CA can issue wildcard certificate.
- TLS_ALPN-01 challenge
It uses TLS handshake process on 443/TCP. ALPN(Application Layer Protocol Negotiation) is the protocol to decide what protocol server and client will use. In this challenge, CA sends some token to service server. Service server creates temporary TLS certificates using `acme-tls/1` protocol with token from CA. When CA access to server, it asks `acme-tls/1` protocol to server. The server presents the temporary certificate, then CA issues certificate.
#### Based on identity
ACME is powerful protocol to automate issuing certificate. However, it is necessary to check ownership of domain to use ACME challenge (Usually, using http or https). However, It is possible to use TLS protocol without http or https. Originally, the process was also manual way called PKIX(Public-Key Infrastructure). It is so complex and slow, and it is impossible to use ACME. To automate this process, modern CA uses JWK(JSON Web Key) and JWT(JSON Web Token). The process is below.
- The administrator registers system's public key as JWK at CA.
> JWK is a format of specification of key. It shows key and its information as JSON format.
- The system sends the JWT to get certificate signed by its private key to CA.
> JWT is a specification of what client can do after connection or of proof this request is valid. In this process, JWT substitute CSR.
- CA verify JWT by pre-registered JWK. When JWT is valid, it issues X.509 certificate.
#### Usage of X.509 Certificates
Regardless the way to issue certificate, either ACME or the way based on its identity, the certificate which is already issued is always valid before it is revoked. You can use X.509 certificate from the way based on its identity for https, oppositely you can use X.509 certificate from ACME for other TLS protocols (like LDAPS, DB TLS communication, etc).
### X.500 and LDAP(Lightweight Directory Access Protocol)
In the beginning of PKI, there was plan to make a server for all certificates. It is X.500. However, the protocol to implement X.500, DAP(Directory Access Protocol) was too complex and heavy to use internet environment. To make way easy and light, LDAP(Lightweight Directory Access Protocol) was developed to store certificates. However, there are a lot more efficient way to manage certificates appeared already. LDAP could not realize X.500, but it is utilized as centralized authentication system like [SSO](./sso.md), or OS account management.
docs/theories/pki/sso.md
+129
View File
@@ -0,0 +1,129 @@
# SSO (Single Sign On)
When someone wants to use some services, usually they have to identify themselves to the services. The service verifies who you are (authentication), and what you can do (authorization). Originally, each service has their own authentication and authorization system. However, in modern internet environment, a lot of services are organically connected to each other. It is common for one provider to operate various services. So, it is very inefficient and complex to operate authentication and authorization system like this way. To solve this problem, the concept of SSO which is to centralize all services' authentication and authorization system at once emerged.
## LDAP (Lightweight Directory Access Protocol)
To centralize authentication and authorization system, it is necessary to kind of central database of user list naturally. Fortunately, there was a very good and suitable protocol already; LDAP. LDAP was originally suggested to replace DAP (Directory Access Protocol) for X.500 in [PKI](./pki.md). Even though X.500 didn't materialized, the potentiality of LDAP was selected to materialize SSO. Because LDAP's structure which was designed for manage certificates itself allows to manage user and its authorization. In modern internet environment, many services don't usually use LDAP itself directly in SSO anymore, but it is still used as database of user and their authorization.
### Structrue of LDAP
LLDAP will be used in this homelab. It is the easiest way to understand LDAP is comparing each part of LDAP with filesystem. Because LDAP itself is fundamentally the database which has the tree structure like filesystem.
#### DN (Distinguished Name)
DN is the unique path of some specific entry in tree. It is like an absolute path in filesystem.
- Example:
- `uid=admin,ou=people,dc=ilnmors,dc=internal`
- `uid=authelia,ou=people,dc=ilnmors,dc=internal`
#### Base DN (Base Distinguished Name)
Base DN is the root of the tree of LDAP. it is like root path of filesystem `/`. All the actions such as search or work are started from Base DN.
- LLDAP setting
- Environment.="LLDAP_LDAP_BASE_DN=dc=ilnmors,dc=internal"
- It means, the domain 'ilnmors.internal' will be the Base DN (the root of tree).
#### Components of DN
DN has components what has ordinary order. The order of these components are `cn(or uid)`, `ou`, `dc`. They are special attributes, which makes DN.
- cn (Common name): The name of object. It is like a file itself in filesystem.
- ou (Organizational Unit): The name of the container which contain entries. It is like a folder in filesystem.
- dc (domain component): The domain components, Usually, it devides the full domain `ilnmors.internal` as `dc=ilnmors`, `dc=internal`.
#### Object (or Entry)
Object is the real data item the DN defines, like file or folder itself in filesystem.
#### ObjectClass
This is the template or blueprint of object. ObjectClass defines what is this object; Either a person or a group, or an organization unit.
- What is this object; user (cn or uid) or group (ou).
- What is the attribute object must have, and might have.
- Example:
- `ObjectClass: person`: This must have `sn` and `cn` attributes.
- `ObjectClass: organizationalPerson`: This is inherited the `ObjectClass: person`, and it can have `mail` or `number` attributes.
- LLDAP uses standard ObjectClass: `person`, `groupOfNames`.
#### Attribute
Attribute is the value of object, like the content of file or folder in filesystem. It is saved as the pairs of key-value. This values are following the definition of ObjectClass.
- Example
- `uid=user1` has the value following `objectClass: person`, `objectClass: organizationalPerson`
- It must have `sn`, and `cn`
- It can have `mail`, or `number`
- It can have special attribute `memberOf: cn=admins,ou=admin_group, ...` (The attribute that shows the group the user belongs to)
## IdP (Identity Provider)
Many of modern services, SP (Service provider), use SAML (Security Assertion Markup Language) or OIDC (OAuth 2.0/OpenID Connect) protocol to implement SSO. Originally, each service had to send request of authentication to central server (which can be LDAP or else) individually to implement SSO. It means each service should protect the sensitive data itself, and some services which have vulnerability can threaten all system. This is why many of modern services use and support SAML or OIDC protocols. The IdP is needed to use these protocols. The IdP acts as the agent of all services to substitute authentication process on their behalf. Only IdP can access the real database, and all services trust IdP's authentication in this model.
### SAML (Security Assertion Markup Language)
SAML protocol was developed in 2001 for SSO. This protocol works on XML (eXtensible Markup Language) format which can meet complex security requirements in enterprise environments. It has been used for a long time, it makes this protocol very stable. However, XML itself is a complex and heavy format. This fact makes the protocol complex and heavy to use in common and small environment. There is the process of SAML below.
- Start
User sends the request to the service. When SP recieves the request, it redirect the request to IdP.
- Authentication
IdP asks the login information to the user and it authenticate user from database (like LDAP). When authentication process succeed, IdP generates SAML assertion including user's identity based on XML with sign to ensure integrity. IdP sends the this assertion to SP via user's browser.
- Finish
SP receives the assertion from IdP via user's browser, it verify the assertion. When it is valid, they allow login.
### OIDC (OpenID Connect) and OAuth 2.0
OIDC was developed in 2014, it is newer than SAML. This is an authentication layer on OAuth 2.0 protocol. Basically, OAuth 2.0 is for authorization, and OIDC is for authentication. OAuth 2.0 protocol works on JSON/REST format, especially JWT (JSON Web Token) which is lighter and simpler than XML. OIDC is latest standard of SSO, it supports social login, and friendly to API. These features makes this protocol use on small and personal environment easily. There is the process of OIDC below.
- Start
User sends the requset to the service (in OIDC, RP; Relying Party). When RP receives the request, it redirects to IdP (in OIDC, OP; OpenID Provider).
- Authentication
IdP (OP) asks login information, and simultaneously asks permission to provide the information to RP. After getting information and permission, IdP generates two tokens which are for different purposes. One is an ID token (JWT) which contains user identity for authentication, the other is an access token for authorization. IdP sends these token to RP. The differences between SAML are what data format does protocol use, and whether IdP gets permission or not.
- Finish
RP recieves the tokens from IdP, and it verify the tokens. When it is valid, they allow login.
### Reverse proxy
When application doesn't support SSO, then you can use reverse proxy as the door of SSO using `Forward Authentication`. Usually, every web packet pass through reverse proxy in modern internet environment. Therefore, reverse proxy (i.e. Caddy) can intercepts the packets and force them to SSO from IdP or OP (or LDAP itself) before they reach to the application.
- Header based
When Authelia success to authenticate someone, Caddy sends specific header which contains user information like `X-Forwarded-User: A`. Application gets this header, it automatically allows login to A. However, it is needed the ID in the application manually.
> If hacker can access to application without reverse proxy, it can make X header to fake application. In this homelab, all access towards application will be limited by iptables from reverse proxy or sidecar reverse proxy.
- LDAP based
LLDAP will be a light and simple LDAP server. Authelia supports OIDC OP(idP) based on external LDAP server. LLDAP server will becom the external LDAP server for Authelia
---
# Example of Authelia flow
## Flow
- Define the user and group in LLDAP server
### OIDC supported app
- At the service, choose login way as Authelia (OP)
- Login at the Authelia.
- Authelia access to LLDAP server to authenticate the user based on LDAP.
- Login succeed, Authelia generate token and Service turst OP's token, and allow login (authentication and authorization)
### Non-OIDC supported app
- Foward_Auth function with Authelia.
docs/theories/virtualization/passthrough.md
+15
View File
@@ -0,0 +1,15 @@
# Hardware passthrough
The concept of hardware passthrough is directly passing the hardware devices to virtual machines, bypassing hypervisor or emulation layers. virtual machines can access to physical hardware directly.
## GRUB
GRUB \(Grand Unified Bootloader\) is bootloader for OS. They runs first when the computer boots, load the kernel on memory, and binds hardwares on kernel with their driver. The configuration of sequence that GRUB decides is stored on initramfs.
## IOMMU
IOMMU is MMU \(Memory Management Unit\) for I/O device. It convert and isolate logical address of IO devices to physical address of memory, so that I/O device's DMA \(Direct Memory Access\). When IOMMU is enabled on GRUB, they can allocate and manage hardwares address on physical memory. This fact allows, this can reserve physical address area for device to allocate virtual machines.
## VFIO
vfio is driver that the devices which is allocated on virtual machines. vfio driver prevents access from hypervisor OS, so that only virtual machine which has that device can access to the hardware.