In this lab you will deploy the OpenTelemetry demo application. This application consists of multiple micro-services and is already instrumented using OpenTelemetry.
Specific information about how OpenTelemetry is deployed in each service can be found here:
First, we need to create a Kubernetes Secret with the URL and secret token of the APM Server. This secret will be used by the OpenTelemetry collector to forward the data to the APM Server running in Elastic Cloud.
In Kibana, open the sidebar and navigate to Integrations (under Management) and select the APM integration.
2. Next scroll all the way down to Configure the agent to find the URL and secret token of the APM Server. It doesn't matter which language you select, we just need these values.
3. On your machine, create environment variables for the URL and secret token. We'll use these environment variables to create the Kubernetes Secret.
Copy the predefined URL that the APM Server listens to, without the HTTPS prefix. Create an environment variable for the URL on your machine
export APM_URL_WITHOUT_PREFIX=<your-url>
⚠️ Make sure to include the port number to your APM endpoint URL, for example "opentelemetry-workshop.apm.eu-west-1.aws.found.io:443".
Next, do the same thing for the secret token
export APM_SECRET_TOKEN=<your-secret-token>
4. Create a secret in Kubernetes using the following command
kubectl create secret generic elastic-secret --from-literal=elastic_apm_endpoint=$APM_URL_WITHOUT_PREFIX --from-literal=elastic_apm_secret_token=$APM_SECRET_TOKEN
We will use Helm to install the official OpenTelemetry demo application. First, we need to add the OpenTelemetry chart repository
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
Next, we will install the demo application. The OpenTelemetry Helm chart also includes the OpenTelemetry Collector. The collector will forward the data to our APM Server in Elastic Cloud. To achieve this, we need some custom configuration. The GitHub repo (the official Elastic fork of the OTel demo) includes the Elastic specific configuration and is already cloned to your machine. You can find the configuration file called values.yaml on your machine. Go to the directory and look at the content of the file, you don't need to change anything.
cd ~/workshop/opentelemetry-demo/kubernetes/elastic-helm
cat values.yaml
Note that the collector is configured to use the Kubernetes secret we created in step 1.
Deploy the demo using helm install and pass in the custom values.yaml file
helm install -f values.yaml my-otel-demo open-telemetry/opentelemetry-demo --version 0.23.0
List all the running pods to make sure that the application is installed successfully, it can take a few minutes for all the pods to become ready
kubectl get pods
You should see all the OpenTelemetry demo pods in addition to an instance of the OpenTelemetry Collector:
NAME READY STATUS RESTARTS AGE
my-otel-demo-accountingservice-bc4446d4b-x646b 1/1 Running 0 4m29s
my-otel-demo-adservice-5f96b995d6-sj289 1/1 Running 0 4m29s
my-otel-demo-cartservice-7fd55475cf-fqmm5 1/1 Running 0 4m30s
my-otel-demo-checkoutservice-56f8467c89-w56z5 1/1 Running 0 4m31s
my-otel-demo-currencyservice-848cd7bbd5-gcmjd 1/1 Running 0 4m31s
my-otel-demo-emailservice-6c449dc86f-k5v6q 1/1 Running 0 4m31s
my-otel-demo-featureflagservice-84bcd65fdc-mj7w6 1/1 Running 0 4m31s
my-otel-demo-ffspostgres-58dd9bdcd4-mpc7h 1/1 Running 0 4m31s
my-otel-demo-frauddetectionservice-7946b5cdb8-m8pkq 1/1 Running 0 4m29s
my-otel-demo-frontend-77d6884f68-lfd68 1/1 Running 0 4m30s
my-otel-demo-frontendproxy-5c84b4f5bc-8zlcq 1/1 Running 0 4m30s
my-otel-demo-kafka-6cf57b7f77-pmrz4 1/1 Running 0 4m31s
my-otel-demo-loadgenerator-6684fc8df-qw2kr 1/1 Running 0 4m31s
my-otel-demo-otelcol-74cdfc7b9b-gb9sx 1/1 Running 0 4m30s
my-otel-demo-paymentservice-5fd857dd66-l6k6m 1/1 Running 0 4m31s
my-otel-demo-productcatalogservice-5c5f66878f-wqbm9 1/1 Running 0 4m31s
my-otel-demo-quoteservice-7dd66b686-gld5t 1/1 Running 0 4m31s
my-otel-demo-recommendationservice-5f47f89c6d-g86kn 1/1 Running 0 4m29s
my-otel-demo-redis-7db54d564c-db8vn 1/1 Running 0 4m30s
my-otel-demo-shippingservice-6c96ff7bb9-574tx 1/1 Running 0 4m31s
To access the frontend of the OpenTelemetry demo application, you need to port-forward the frontend-proxy service. Open a new terminal tab (the command will run in the foreground and will occupy the terminal session) and start the port-forwarding
kubectl port-forward svc/my-otel-demo-frontendproxy 8080:8080 --address=0.0.0.0
There are two ways to access the frontend of the application:
Option 1: In Strigo
Option 2: In your local browser
OPTION 1
Click on the App tab and click "reload this view"
OPTION 2
First, find the public IP address of your machine by clicking on the cogwheel > machine info and copy the public IP (or public DNS).
In your local browser open a new tabblad and go to http://<public ip>:8080.
When the application is up and running, it will send data to the OpenTelemetry Collector which forwards this data to the Elastic APM Server running inside the Integrations Server in your Elastic Cloud deployment. To make sure data is coming in, navigate to Kibana and select Observability > APM > Services. You'll see a list of all the different services. Select Service Map on top to switch to the real-time visual representation of all the instrumented services interacting with each other. This map is generated automatically based on all the APM data coming in.
If you don't see any services, refer to the troubleshooting section below, as you'll likely have a misconfiguration.
If no data is coming in, try the following steps:
Check the logs of the OpenTelemetry Collector:
kubectl logs -l app.kubernetes.io/name=otelcol
If you see errors regarding the connection to the APM Server, double check the values in the Kubernetes secret:
kubectl get secret elastic-secret -o yaml | grep elastic_apm_endpoint: | sed 's/^.*: //' | base64 -d
kubectl get secret elastic-secret -o yaml | grep elastic_apm_secret_token: | sed 's/^.*: //' | base64 -d
If the secret wasn't correct, delete it and go through step 1 again
kubectl delete secret elastic-secret
If you're stuck, feel free to contact the workshop team, they can help you troubleshoot!