Embedded Files
all-in-one ----> Troszkę projektów pod ArgoCD. ArgoCD pobiera z github i wdraża na K8s.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: website-db-vault-kaf-redis-arg-kust-kyv-gra-loki-temp-pgui
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/exea-centrum/website-db-vault-kaf-redis-arg-kust-kyv-gra-loki-temp-pgui.git
targetRevision: HEAD
path: manifests/base
destination:
server: https://kubernetes.default.svc
namespace: davtrowebdbvault
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
https://softwaremill.com/kafka-visualisation/
all-in-one ----> Troszkę projektów pod ArgoCD. ArgoCD pobiera z github i wdraża na K8s.
# website-db-vault-kaf-redis-arg-kust-kyv-gra-loki-temp-pgui - Complete Monitoring Stack
## 🚨 Fixed Issues
### 1. ✅ Vault CrashLoopBackOff - FIXED
**Problem**: Vault container was crashing repeatedly
**Solution**:
- Added development mode with proper startup command
- Added health checks (readiness and liveness probes)
### 2. ✅ Kafka Configuration - FIXED
**Problem**: Bitnami Kafka had issues
**Solution**: **Changed to official Apache Kafka 4.1 image**
- Using instead of
- Simplified KRaft configuration
- Proper environment variables for Apache Kafka
### 3. ✅ pgAdmin Email Validation - FIXED
**Problem**: is not a valid email
**Solution**: Changed to
### 4. ✅ Kyverno Policy - FIXED
**Problem**: Policy was too restrictive
**Solution**: Changed to mode for development
## 📊 Architecture Diagram
```
┌─────────────────────────────────────────────────────────────────┐
│ KUBERNETES CLUSTER │
├─────────────────────────────────────────────────────────────────┤
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ INGRESS │ │ ARGOCD │ │ KYVERNO POLICY │ │
│ │ (nginx) │◄───┤ (GitOps) │────│ (Security - Audit) │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ FASTAPI │────│ REDIS │────│ APACHE KAFKA │ │
│ │ (App) │ │ (Queue) │ │ (v4.1 - KRaft) │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
│ │ │ │ │
│ │ │ ▼ │
│ ▼ │ ┌─────────────┐ │
│ ┌─────────────┐ │ │ KAFKA UI │ │
│ │ POSTGRESQL │◄─────────────────────┘ │ (Monitoring)│ │
│ │ (Database) │ └─────────────┘ │
│ └─────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ PGADMIN │ │
│ │ (Admin) │ │
│ └─────────────┘ │
│ │
├─────────────────────────────────────────────────────────────────┤
│ MONITORING STACK │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ PROMETHEUS │◄───│ GRAFANA │ │ LOKI │ │
│ │ (Metrics) │ │ (Dashboards)│ │ (Logging) │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
│ ▲ │ ▲ │
│ │ │ │ │
│ ┌───────┴────────┐ │ ┌───────┴────────┐ │
│ │ Service │ │ │ PROMTAIL │ │
│ │ Discovery │ │ │ (Log Agent) │ │
│ └────────────────┘ │ └────────────────┘ │
│ │ │
│ ┌─────────────┐ │ ┌─────────────────────┐│
│ │ TEMPO │ │ │ APPLICATIONS ││
│ │ (Tracing) │ │ │ (FastAPI, Worker) ││
│ └─────────────┘ │ └─────────────────────┘│
│ ▲ │ │
│ │ │ │
│ ┌───────┴────────┐ │ │
│ │ Distributed │ │ │
│ │ Tracing │ │ │
│ └────────────────┘ │ │
│ │
├─────────────────────────────────────────────────────────────────┤
│ SECURITY (DEV MODE) │
│ │
│ ┌─────────────┐ │
│ │ VAULT │ │
│ │ (Secrets) │──────────────────────────────────────┐ │
│ └─────────────┘ │ │
│ (Dev Mode) ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ Database │ │ Redis │ │ Kafka │ │
│ │ Credentials │ │ Password │ │ Credentials │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
## All Resources Generated:
- ✅ app-deployment
- ✅ postgres-db
- ✅ pgadmin (FIXED email)
- ✅ vault (FIXED CrashLoopBackOff)
- ✅ redis
- ✅ **kafka-kraft (USING APACHE KAFKA 4.1)**
- ✅ kafka-ui
- ✅ prometheus-config
- ✅ prometheus
- ✅ grafana-datasource
- ✅ grafana
- ✅ loki-config
- ✅ loki
- ✅ promtail-config
- ✅ promtail
- ✅ tempo-config
- ✅ tempo
- ✅ ingress
- ✅ kyverno-policy (FIXED to Audit mode)
## 🛠️ Quick Start
```bash
# Generate all files
./unified-stack.sh generate
# Build and push container
docker build -t ghcr.io/exea-centrum/website-db-vault-kaf-redis-arg-kust-kyv-gra-loki-temp-pgui:latest .
docker push ghcr.io/exea-centrum/website-db-vault-kaf-redis-arg-kust-kyv-gra-loki-temp-pgui:latest
# Deploy to Kubernetes
kubectl apply -k manifests/base
# Check status - all pods should be running now
kubectl get pods -n davtrowebdbvault
# Check Kafka specifically
kubectl logs statefulset/kafka -n davtrowebdbvault
```
## 🔧 Kafka Configuration Details
**Using**: Official Apache Kafka 4.1 with KRaft (no Zookeeper)
**Image**:
**Features**:
- Single node KRaft cluster
- PLAINTEXT listeners on port 9092
- Controller on port 9093
- Automatic topic creation enabled
## 🌐 Access Points
| Service | URL | Purpose |
|---------|-----|---------|
| Application | http://app.website-db-vault-kaf-redis-arg-kust-kyv-gra-loki-temp-pgui.local | Main website with survey |
| Grafana | http://grafana-service.davtrowebdbvault.svc.cluster.local | Metrics & logs dashboard |
| Prometheus | http://prometheus-service.davtrowebdbvault.svc.cluster.local | Metrics collection |
| Kafka UI | http://kafka-ui.davtrowebdbvault.svc.cluster.local:8080 | Kafka monitoring |
| pgAdmin | http://pgadmin-service.davtrowebdbvault.svc.cluster.local | Database administration |
| Vault UI | http://vault.davtrowebdbvault.svc.cluster.local:8200 | Secrets management |
## 📝 Notes
- **Vault** is running in development mode (not for production)
- **Kafka** uses official Apache Kafka 4.1 image (KRaft mode)
- **Kyverno** policy is in Audit mode for development
- All components have proper health checks and resource limits
- Survey system should work end-to-end: Web → Redis → Kafka → PostgreSQL
resources:
- app-deployment.yaml
- message-processor.yaml
- postgres-db.yaml
- pgadmin.yaml
- vault.yaml
- vault-secrets.yaml
- vault-job.yaml
- redis.yaml
- kafka-kraft.yaml
- kafka-topic-job.yaml
- kafka-ui.yaml
- kafka-config.yaml
- kafka-exporter.yaml
- fastapi-config.yaml
- prometheus-config.yaml
- postgres-exporter.yaml
- node-exporter.yaml
- service-monitors.yaml
- prometheus.yaml
- grafana-datasource.yaml
- grafana-dashboards.yaml
- grafana.yaml
- loki-config.yaml
- loki.yaml
- promtail-config.yaml
- promtail.yaml
- tempo-config.yaml
- tempo.yaml
- network-policies.yaml
- ingress.yaml
- kyverno-policy.yaml
Google Sites
Report abuse