OpenShift Service Mesh essentials — Part III — Data Plane
In the previous article, we have seen some of the most important Service Mesh Control Plane aspects, now it’s the time for the Data Plane.
If you want to review the Control Plane part jump to:
Or if you want to start from the beginning:
Or start playing with some examples that show the Service Mesh features:
In this article, we are going to explore the OpenShift Service Mesh Data Plane. The idea here is to learn about the Data Plane by showing how to publish a Service Mesh application but without using the extended Istio features (ie. smart routing, control policies, etc), so we are going to get what we have with standard OpenShift SDN features but using Service Mesh. Istio feature explanation will be done in future articles.
Envoy sidecar container
In this section, we are going to check how the Envoy sidecar is injected into the POD and what implications do you have when you include it (ie. traffic redirection, Multus annotations removal, etc).
Sidecar container injection
If you read the official Istio documentation you can find how there are two ways to inject the Proxy sidecar in the application PODs: manual and automated injection.
In OpenShift Service Mesh we also have both options, but as reviewed in OpenShift Service Mesh essentials — Part II — Control Plane article, there is one main difference between what is proposed by the Istio documentation and how the automatic sidecar injection is done in OpenShift, while the default Istio configuration injects the sidecar into pods in any namespace with the istio-injection=enabled label, in OpenShift Service Mesh that label is not included in the namespace but in an annotation in the Deployment object, giving an additional level of flexibility while including applications inside and outside the Service Mesh in the same namespace and, for example, preventing build or deployment PODs from being injected with the sidecar too.
The automatic injection is great since it not only simplifies the deployment but also dealing with the additional Envoy Proxy container lifecycle. Imagine that there is a new release of the OpenShift Service Mesh Envoy sidecar proxy. If you used automatic injection, the sidecar upgrade will be as easy as restarting your PODs (automatic update strategy in the Service Mesh Operator will upgrade the Control Plane, but it won’t auto-restart PODs in order to upgrade the sidecar assuring that it won’t impact your application), but if you used manual injection you will need to go Deployment definition one-by-one changing the Envoy sidecar container image, that’s why automatic injection is the recommended way to include the sidecar proxy in your PODs.
You might be wondering how does the automatic sidecar injection work? The injection is triggered by an Admission Controller, but… What’s an Admission Controller?… let’s go through that…
An Admission Controller is part of the Kubernetes API server code that makes possible to intercept API request before being enforced in the cluster but after the request is authenticated and authorized, so it’s like the last bastion to cross before the cluster does what you want. There are a number of Admission Controllers built-in Kubernetes, for example, if you have already read my article about how you could configure different secure zones in a single OpenShift Cluster, you will remember that in the last part, when we modified the default namespace creation template including whitelist tolerations by including additional annotations in the Kubernetes Objects, we were actually using the PodTolerationRestriction Admission Controller.
But there is also a way to include additional controllers by using webhooks and you could have two types of these Admission Controllers:
- Validating admission webhook, which permits to enforce custom administration policies during the validation of your requests.
- Mutating admission webhook: Allows you to change requests to enforce custom defaults.
As mentioned here, in order to register admission webhooks, you have to create MutatingWebhookConfiguration or ValidatingWebhookConfiguration API objects.
In our case, what we use to automatically inject the sidecar proxy in the POD is a mutating admission webhook, so we can take a look at the MutatingWebhookconfiguration objects, you will see how there are at least one of them per Control Plane namespace plus an additional one used by the Maistra Operator:
You can open one of the sidecar-injector webhook and take a look at the YAML definition file, you will see a “webhooks” section like this:
When an apiserver receives a request that matches one of the rules, the apiserver sends an admissionReview request to webhook as specified in the clientConfig. In our Admission Controller definition, we force that when we try to create a POD, we invoke the /inject endpoint of the istio-sidecar-injector service from your Control Plane namespace. That will lastly “add” and configure (you can also inject environment variables into the Envoy proxy if you use the sidecar.maistra.io/proxyEnv annotation) the additional proxy sidecar container inside the POD that you are trying to create, if the Deployment contains the sidecar.isito.io/inject="true" annotation.
Let’s inject the sidecar in one test application. First, we are going to deploy the application, I will use this Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
replicas: 1
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: openshift/hello-openshift
ports:
- containerPort: 8080I will create this application in project-a1, which is a namespace included in the OpenShift Service Mesh controlplane-a (remember the ServiceMeshMemberRoll object that we created in the previous article) located in namespace istio-system, so if you check the Kiali interface of that Control Plane, you should see the application running but you will see a message that indicates that such workload does not have the required proxy sidecar:
Note: In addition to Deployment standard Kubernetes objects, OpenShift provides the DeploymentConfig object, which brings some extended capabilities. If you are using DeploymentConfigs instead Deployments, you need to know that the behaviour that I’ve found in Kiali (at in latest versions, such 1.12 that comes in OpenShift 4.5) is that it only shows base Kubernetes objects such as Deployments, Builds, Services, etc, but not OpenShift specific objects (ie. DeploymentConfigs) as Workloads in the UI.
Service Mesh, including sidecard automatic injection, is working independly if you are using either Deployments or DeploymentConfigs the only thing that you will lose (at this moment with this release) is workload Kiali visibility. If you want to check that out you can deploy the same application but with a DeploymentConfig object:
apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
name: helloDC
spec:
replicas: 1
selector:
app: helloDC
template:
metadata:
labels:
app: helloDC
spec:
containers:
- name: helloDC
image: openshift/hello-openshift
ports:
- containerPort: 8080Ok, now let’s inject the sidecar using the sidecar.istio.io/inject: “true” annotation. You should include the annotation in your deployment in this way (check the bold text):
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
replicas: 1
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
annotations:
sidecar.istio.io/inject: "true"
spec:
containers:
- name: hello
image: openshift/hello-openshift
ports:
- containerPort: 8080You can check how after including the annotation, the PODs associated to the Deployment, will have an additional “istio-proxy” container which is the Envoy proxy sidecar:
You can also check that once that’s done, the “Missing Sidecar” message disappears from the Kiali interface.
One more thing about the sidecar, do you remember that in the previous article, when we created the controlplane-a Service Mesh configuration, we included the requests and limits for our sidecars ? (check below)
apiVersion: maistra.io/v1
kind: ServiceMeshControlPlane
metadata:
name: controlplane-A
namespace: istio-system
spec:
version: v1.1
istio:
global:
proxy:
resources:
requests:
cpu: 20m
memory: 200Mi
limits:
cpu: 3000m
memory: 2048Mi
...We can now check that QoS policy has been informed in our container:
Let’s review what happens if we deploy our application in a namespace (project-b1) associated to the other already deployed Service Mesh Control Plane (controlplane-b) where we didn’t setup the QoS values:
As you can see, there is a default setting that has been applied in such a case where there are no limits for the sidecar container.
Automatic POD traffic redirection to Envoy proxy
Now we have our application container(s) running in the POD along with the new sidecar container, How can we be sure that traffic is managed by this new sidecar instead of our own containers? The good news is that everything is setup automatically for you and you don’t have to worry about that, but if you care about how that actually works, you are in the right place because we are going to explore it.
Note: Automatic traffic redirection to the Envoy proxy container is making the proxy sidecar transparent to your application…well, almost transparent since that forced redirection means that until the Envoy proxy is started the POD won’t be able to get connectivity, take this into account since I found some applications that need to connect during the first (sub)second to somewhere else in order to work, and if the application container starts before the Envoy proxy on the POD, the former won’t be able to reach that external resource during that time
In the last article where I described the OpenShift Service Mesh Control Plane, you saw how the Maistra Operator (Service Mesh Operator) deployed in the openshift-operators namespace (since we deployed the Operator for “all namespaces”) a DaemonSet called “istio-node” which deployed an istio-cni POD in each node that monitors if a Service Mesh workload is scheduled in that node, and in that case modifies the node iptables. Now we are going to take a look to those iptable rules.
The istio-cni resource configures the iptables rules inside the network namespace where the POD is running, so it only affects ingress/egress traffic on that POD, not globally in the node. If we want to check what the istio-cni is doing, we need a way to inspect the iptable rules inside a network namespace. I’m coming from the OpenStack world, and there we used to run Linux commands inside specific network namespaces by using the ip netns exec command, but if you used to deal with docker containers you probably have used the nsenter command, because the ip netns exec command is only useful for namespaces listed in /var/run/netns and docker does not put its namespaces there by default.
Luckily, here we can use either ip netns execor nsenter commands and I will show to you how.
The first step is common, we need to know any of our POD container IDs running on our node (in our example we have two containers running in our POD, the app container, and the sidecar container, so there will be two different container IDs bind to this POD). That information is contained in the description of our POD (look for “cri-o”, as CRIO container runtime)
Now we need to know details about that container ID. In order to do that, we need to jump to the node where the POD is running and run this command:
crictl inspect --output json <container ID>That will give us a lot of useful information but we just need to focus on one single thing. Depending on if you want to use ip netns execor nsenter commands you will need either the network namespace ID or the process ID associated with the container, so the best idea here is to grep with “netns” if you plan to use ip netns exec or with “pid” if you are going to use nsenter
Once you have got either the network namespace id or the process id, you can run the command that you want inside the namespace.
With ip netns exec you can run
ip netns exec <namespace id> <my command>With nsenter your command would be (the -nimplies that the command will run in the network namespace that the process is using):
nsenter -t <pid> -n <my command>In our case, we want to check the iptables NAT rules, so “my command” is:
iptables -t nat -SLet’s review an end-to-end example with both commands, starting with ip netns exec command:
and now with nsenter
Both commands give you this output:
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P POSTROUTING ACCEPT
-P OUTPUT ACCEPT
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_INBOUND
-N ISTIO_OUTPUT
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_INBOUND -p tcp -m tcp --dport 8080 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -m owner --uid-owner 1000710001 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1000710001 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECTThose rules make that all ingress to and egress traffic from the application container is redirected to port 15001 of the pod, in this case, our application “hello” is listening in port 8080, but you can see how 8080 is finally redirected to 15001…but what’s listening on that port in the POD?
If you check the POD YAML definition you won’t find any reference to port 15001, that’s because is listening locally to the POD, it’s not “exposed” outside. You can check that port is listening in the POD and the process associated with it by running the following command in the POD network namespace (remember how to use either ip netns execor nsenter commands)
ss -nlp | grep 15001This is the output with the nsenter command
sh-4.4# nsenter -t 1765412 -n ss -nlp | grep 15001tcp LISTEN 0 128 0.0.0.0:15001 0.0.0.0:* users:(("envoy",pid=1765590,fd=63))
You can see how the PID 1765590 is the one listening on that port, let’s see what’s that on the node using ps aux:
sh-4.4# ps aux | grep 17655901000710+ 1765590 0.1 0.2 173012 49724 ? Sl 08:45 0:16 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster hello.project-a1 --service-node sidecar~10.129.2.106~hello-7d55b4456d-sv6lw.project-a1~project-a1.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error --concurrency 2root 2313349 0.0 0.0 9180 1084 ? S+ 11:21 0:00 grep 1765590
If you want to be completely sure that PID is associated with the sidecar container of our POD, we can get the root PID of that container following the procedure mentioned above (but getting the CRIO ID for the istio-proxy container instead of the hello container) and you will see that (in my case it’s 1765517)…oh surprise that's not the same PID (1765590) that’s listening on that 15001 port, that’s a different one.
That’s because the root process of the istio-proxy container, the one that you find using the CRIO ID (1765517), is the one for the “main” process (pilot-agent) but that process starts another child process, our PID (1765590) associated to envoy command. You can easily know that by using the pstree command with the PID that you found listening on port 15001, following this format:
pstree -p -s <envoy process ID listening on 15001>That will give you a view of all parents, where for sure you will find the root PID that you get from the CRIO ID inspect (1765517 in my case). This is my output (forget the processes under the “+”, those are processes with identical branches):
So now you can be completely sure that all your POD traffic is managed by that sidecar container that you injected (if you didn’t trust me when I mentioned it before this test)
Multus and automatic Istio sidecar injection
You might remember from the last article, when I presented the different Control Plane configuration options, that Multus annotations are not preserved by the automatic Envoy sidecar injector by default, but you have a Control Plane configuration variable (sidecarInjectorWebhook) that permits to maintain them. We configured that option in controlplane-a, but not in controlplane-b, so let’s check if that works.
Note: Multus interface won’t be part of the Service Mesh, it will be an additional network access to the application but Service Mesh features cannot be used there.
First of all, we need to configure additional networks in our cluster and test that is working. In one of my previous blog series, specifically in the Security Zones in OpenShift worker nodes — Part III — Network Configuration article, we reviewed different ways when creating network setups in OpenShift. In OpenShift 4.5, the current release, we still don’t have nmstate objects available by default (it’s on the roadmap), so I could install OpenShift Virtualization in my cluster to get access to that API, or use the “standard” method modifying the Network Cluster Operator object. In this case I will use the latter.
In my worker nodes, I have two interfaces, one that is used by the OpenShift SDN and an additional that I will use to provide direct network access to my PODs using Multus. You can check available interfaces in the node using the OpenShift console with clusteradmin privileges. In my case, ens3 is the Cluster Network for the SDN and ens4 will be the one used by Multus (the node already has an IP because DHCP is enabled in that interface):
As already mentioned, the method to enable the additional network in OpenShift will be modifying the Network Cluster Operator.
apiVersion: operator.openshift.io/v1
kind: Network
...
spec:
...
additionalNetworks:
- name: prov-net
namespace: project-a1
rawCNIConfig: '{ "cniVersion": "0.3.1", "type": "macvlan", "capabilities": { "ips": true }, "master": "ens4", "mode": "bridge", "ipam": { "type": "dhcp" } }'
type: Raw
- name: prov-net
namespace: project-b1
rawCNIConfig: '{ "cniVersion": "0.3.1", "type": "macvlan", "capabilities": { "ips": true }, "master": "ens4", "mode": "bridge", "ipam": { "type": "dhcp" } }'
type: Raw
...Right after that, a new NetworkAttachmentDefinition is created automatically in each configured namespace (project-a1 and project-b1 in this case) which can be used to assign additional networks to PODs
In order to test Multus configuration, you could create a POD with an additional NIC and test connectivity (I ping my default network gateway 192.168.100.1):
Note: I use the centos/tools container image here so I can use the POD bash to run some tests (openshift/hello-openshift does not have terminal)
apiVersion: v1
kind: Pod
metadata:
name: multus-test
annotations:
k8s.v1.cni.cncf.io/networks: '[
{
"name": "prov-net"
}
]'
spec:
containers:
- name: example-pod
command: ["/bin/bash", "-c", "sleep 99999999999"]
image: centos/tools
Ok, now we are sure that Multus is working, so let’s create two Deployments that includes both the sidecar injection and multus annotations, one in project-a1 (attached to controlplane-a where the option should preserve Multus annotations) and another one in project-b1. You will see how the additional interface is only configured in the former case, when the Multus annotation is preserved.
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-sidecar-multus
spec:
replicas: 1
selector:
matchLabels:
app: test-sidecar-multus
template:
metadata:
labels:
app: test-sidecar-multus
annotations:
sidecar.istio.io/inject: "true"
k8s.v1.cni.cncf.io/networks: '[
{
"name": "prov-net"
}
]'
spec:
containers:
- name: test-sidecar-multus
command: ["/bin/bash", "-c", "sleep 99999999999"]
image: centos/tools
Accessing the application from outside the cluster
We have the “Hello World” application running on our Service Mesh, and now we want to access it from outside the OpenShift cluster.
Service
Istio creates its own service registry that is automatically filled with the information given by the OpenShift/Kubernetes cluster where it’s installed, specifically with the Service Objects (and the associated endpoints). So in order to let Istio know about a Service, we need to create a standard “Service” Kubernetes object.
apiVersion: v1
kind: Service
metadata:
name: hello
labels:
app: hello
spec:
ports:
- port: 8080
name: http
selector:
app: hello
If you didn’t delete the POD that we created to show Multus with sidecar container injection, you can test that this service is working (remember to use the right service port, 8080 in this case)
Now that we created the OpenShift Service, Istio will know about it too. You can check how Kiali has a dedicated section for it:
Exposing a service in an “Istio namespace” with regular Routes
We still didn’t use any specific Istio object (we used a Kubernetes Service object), if we would like to continue exposing this service with “regular” Objects (taking into account that we would be losing Istio capabilities) we would now create a Route/Ingress to expose that internal Service outside the cluster… let’s try that:
The “regular” Route object does not work, Why? Because you shouldn’t be using that standard object to expose your Service Mesh application. If you remember from the last article, as part of modifications introduced by the Service Mesh Operator, when you include a namespace as part of a specific Service Mesh Control Plane, there are a series of Network Policies that blocks inbound traffic if it’s not coming from the Istio Ingress Gateway, which is the intended entry point into the Mesh and you have to remember that your application is supposed to be using the Service Mesh because it’s located in a namespace attached to one (we configure that using the ServiceMeshMemberRoll object when configuring the Control Plane).
Note: Remeber that the istio-mesh Network Policy only permits traffic in the same Mesh (between applications and applications and Gateways). That includes all namespaces attached to the same Control Plane meaning that you will have, by default if not modified, visibility to all services deployed in the Mesh independently in which namespace your application is running on if they are part of the same Mesh. This Network Policy will also deny not only any external access not going through the Ingress, but also will prevent direct access to internal Kubernetes Services from outside the Mesh
What happens if, as a requirement, I need to run an application inside a namespace attached to a Mesh but I would like to publish it outside that Mesh using regular Route objects (losing some of the main Istio capabilities then)?
You need to allow traffic from the Router to the PODs …. but you don’t need to touch Network Policies since there is one rule already prepared for such a case.
As you can see in the istio-expose-route Network Policy (first in the list above), it permits traffic if it’s coming from a namespace with label network.openshift.io/policy-group=ingress and to PODs with label maistra.io/expose-route: "true"
You can check how the namespace where the Routers are running (openshift-ingress) already contains the label network.openshift.io/policy-group=ingress
Then the only thing remaining is to add the label maistra.io/expose-route: "true" to our PODs. Let’s try it out, I will create a new application adding that label:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-noistio
spec:
replicas: 1
selector:
matchLabels:
app: hello-noistio
template:
metadata:
labels:
app: hello-noistio
maistra.io/expose-route: "true"
annotations:
sidecar.istio.io/inject: "true"
spec:
containers:
- name: hello-noistio
image: openshift/hello-openshift
ports:
- containerPort: 8080I will create an associated Service too:
apiVersion: v1
kind: Service
metadata:
name: hello-noistio
labels:
app: hello-noistio
spec:
ports:
- port: 8080
name: http-noistio
selector:
app: hello-noistioThat should be enough so I will try to expose that new Service with a regular Route:
It still does not work! Why?… if I included the label that the Network Policy is expecting.
Note: It might work for you…that will depend on your underlying environment, continue reading to understand why
If you have read the Security Zones in OpenShift worker nodes — Part III — Network Configuration article you might know the reason… Ingress Controllers/Routers are published in different ways depending on which platform OpenShift is running (endpointPublishingStrategy), for example, if you are running in a Public Cloud the Routers will be using the a LoadBalancerService but if you are using a Baremetal node they will be using the HostNetworkpublishing method.
If you use the HostNetworkpublishing method that has some implications on how the Network Policies affect to your Routers, since the traffic from the Ingress Controller is assigned the netid:0 Virtual Network ID (VNID). That VNID is attached to the default namespace so any namespace labels used in the Network Policies where you want to imply the Routers/Ingress Controllers must be assigned to default namespace.
In summary, if you are using a Baremetal deployment (like me in this example) your Routers will be ”using” the default namespace from the Network Policies point of view, instead of the openshift-ingress namespace.
The following step is only needed if your Ingress Controller is using the hostnetwork endpointPublishingStrategy when deploying on baremetal (as my example), if you deployed OCP on a public cloud, the endpointPublishingStrategy will be loadBalancer and then you won’t need to add any label in the default namespace.
If you double check the labels on the default namespace you won’t find the required label network.openshift.io/policy-group=ingress that we already had in openshift-ingress namespace and that is need by this Network Policy in order to allow the traffic…so the solution here is to add that label to the default namespace
It works! so from now on in this cluster, you will only need to add the label maistra.io/expose-route: "true" to the Deployments if you want to expose with regular Routes even if you deployed your OpenShift cluster using the “Baremetal” approach (since now the default namespace have the “right” label).
Ok, enough of “non-istio stuff”, and let’s jump into the Istio Ingress Gateway Object.
Note: I’ve removed the Routes and the non-istio Service and Deployment that I created in this section, so you won’t see them again in further screencaptures
Exposing the application using the Istio Ingress Gateway
We have already seen how is possible to expose a service in a namespace associated with a Service Mesh using regular OpenShift/Kubernetes Objects, but if you have deployed a Service Mesh is because you want to take the advantage of its features, and in order to get many of the benefits your traffic must use the Istio Ingress Gateway as the entry point.
In the last article, we reviewed how the Ingress Gateways were PODs (actually they are also Envoy Proxies, as the sidecar containers) running on the Service Mesh Control Plane namespace and that is published externally through a standard OpenShift Route. Now we are going to see how you can configure them to expose a Service running in any of the namespaces associated with that Control Plane.
The Ingress Gateways located in the Control Plane namespace are automatically configured when a new “Gateway” object is created in any of the namespaces associated with that Control Plane, let’s create one:
Note: Please include your application subdomain name where it says <domain>, something like apps.ocp.domain.com
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: hello-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "hello.<domain>"Note: In my example test I created this object in the project-a1 namespace, associated to Control Plane running on istio-system namespace
This object configures the Ingress Gateway to accept connections on port 80 with hostname “hello.<domain>”… but it also contains a “selector”, Why?
In this Service Mesh we only deployed an Ingress Gateway. That’s quite common (you have to think that you can deploy multiple different Service Mesh in the same Openshift Cluster as we described in the previous article in order to differentiate/isolate traffic and configuration), but there is also possible to deploy multiple Gateways (either Ingress or Egress) in the same Service Mesh Control Plane and, in that case, you could select which one you want to configure with this Gateway selector..
How would you deploy multiple Gateways in the same Service Mesh? (well, first I would say that think about having multiple Control Planes, depending on your use case… but if you really want to do it…)
In order to deploy multiple Gateways you need to include such configuration in the Service Mesh Control Plane definition, for example, let’s say that I want to configure secondary Ingress and Egress Gateways in the controlplane-a Control Plane that we created in the last article, I should include lines marked in bold text here:
apiVersion: maistra.io/v1
kind: ServiceMeshControlPlane
metadata:
name: controlplane-A
namespace: istio-system
spec:
version: v1.1
istio:
global:
proxy:
resources:
requests:
cpu: 20m
memory: 200Mi
limits:
cpu: 3000m
memory: 2048Mi
gateways:
istio-egressgateway:
enabled: true
autoscaleEnabled: true
autoscaleMin: 1
autoscaleMax: 3
istio-ingressgateway:
enabled: true
autoscaleEnabled: true
autoscaleMin: 2
autoscaleMax: 3
ior_enabled: true
istio-secondary-egressgateway:
enabled: true
autoscaleEnabled: false
labels:
istio: secondary-egressgateway
ports:
- name: http2
protocol: TCP
port: 80
targetPort: 8080
- name: https
protocol: TCP
port: 443
targetPort: 8443
- name: tls
protocol: TCP
port: 15443
targetPort: 15443
istio-secondary-ingressgateway:
enabled: true
autoscaleEnabled: false
ior_enabled: false
resources:
requests:
cpu: 100m
memory: 128Mi
labels:
istio: secondary-ingressgateway
ports:
- name: status-port
protocol: TCP
port: 15020
targetPort: 15020
- name: http2
protocol: TCP
port: 80
targetPort: 8080
- name: https
protocol: TCP
port: 443
targetPort: 8443
- name: tls
protocol: TCP
port: 15443
targetPort: 15443
pilot:
traceSampling: 10.0
sidecarInjectorWebhook:
injectPodRedirectAnnot: true
tracing:
jaeger:
template: production-elasticsearch
elasticsearch:
nodeCount: 1
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "1000m"
memory: "2Gi"You need to add entries under the “gateways” section. As you can see you can customize the configuration for them ( IOR setting, Resources, Autoscaling, etc) but you need to add a new “ports” section inside each one.
That configuration will be used to create the Kubernetes Services associated with the new gateway PODs (I copy-pasted the same ports from our already deployed Ingress and Egress gateways). By default, the Service is are created using “type: ClusterIP” but you can add type: LoadBalancer in the Control Plane configuration (at the same level as ports or labels) and the Operator will configure a service with that type if your OpenShift cluster deployment supports it.
Here you can see the gateway PODs created once the configuration on the Control Plane has been modified as shown above.
Another important setting in this new Gateways is the labels, since those are the ones that we can use in our selector field in the Gateway object to select the right Ingress Gateway. If you check the labels on the new secondary gateways you will see that the istio=secondary-egressgateway label is setup
And then you can use it in your selector setting on you Gateway object:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: hello-secondary-gateway
spec:
selector:
istio: secondary-ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "hello-secondary.<domain>"In this case, the configuration is done in the secondary-ingressgateway PODs and not in the ingressgateway PODs, but why that would be useful?
Aside from having different configurations per Gateway “type”, we can use the labels configured and run the new Istio Gateways POD in dedicated Nodes, so we can split the traffic between the old and the new gateways. If in addition to that we configure dedicated Ingress Controllers with Route Sharding (remember that at the end of the day, we are configuring “regular” OpenShift routes in front of our Istio Gateways, so you should take that into account when splitting traffic too) we can create differentiated “Zones” for our Service Mesh Input/Output.
If you don’t know what I’m talking about I recommend you to take a look at the Security Zones in OpenShift worker nodes blog series, more specifically to the Network configuration article:
Talking about routers configured in front of our Istio Ingress Gateway, if you remember from the previous article, there is a setting on the Control Plane Gateway definition that does a “route synchronization”: IOR.
In controlplane-a we enabled IOR and we keep it disabled in controlplane-b, let’s explore what does exactly means. If you see what happens when I create the Gateway object shown above in one namespace associated with the controlplane-a where IOR is enabled you will notice how in the Control Plane namespace a new route with the configured hostname is created automatically:
With IOR enabled you to configure at once the Istio Ingress Gateway to accept the hostname and the OpenShift route that must be placed in front of it to re-direct the connection to the right Istio Ingress Gateway.
If you take a look at the Route definition you will find that the Route is sending back traffic to a Service called istio-ingressgateway and that service is load balancing PODs with labels app=istio-ingressgateway and istio=ingressgateway that is our (principal, not the secondary that we just configured) Istio Ingress Gateway PODs (there are two because we configured a minimum number of two in our Control Plane definition):
If you create the Gateway in a namespace associated to controlplane-b where IOR is disabled, you will find that you don’t have any route in place in your Control Plane:
You could create a route entry manually pointing to the right Ingress Gateway Service, otherwise, the user requests for that specific hostname that you configured in the Gateway object will be drop at the OpenShift Ingress Controller level, so it does not even reach the Istio Ingress Gateway. This is true in this standard setup where the Istio Ingress Gateways are using Routes to get traffic in, but if you can publish them using other Service designs (not ClusterIP) that makes it possible to access PODs without using the Ingress Controller you can make the traffic reach the Istio Ingress Gateway without the need of an OpenShift route (maybe in that case it makes sense IOR disabled).
We have seen how to expose HTTP services, but What about HTTPS? We will go through that in the “Security Features” section in an upcoming article.
Ok, we discuss multiple things but we didn’t check that our route is working after creating the “Gateway” Istio Object (OpenShift route created by IOR sync in this case):
….and it’s not working…Do not panic, this is because, if you take a look at the Gateway definition, we didn’t include anywhere the Kubernetes Service name object that we created to access our application so the Istio Ingress Gateway is not able to configure an internal route to it.
Ingress gateways are not automagically aware of the services that they must use as we have seen. To make an ingress gateway aware of a Service in the Mesh, a VirtualService must be defined and applied to the gateway object.
Virtual Service
Virtual Service Objects can be used to configure routing, including different advance settings (by default, the Ingress Gateways distribute traffic across each service’s load balancing pool using a round-robin model) that will be reviewed in another article (along with other extra objects as the “Destination Rules”). Here we are focusing on just providing external connectivity to our application in the Mesh with no fancy extra setting.
Let’s take a look at a basic Virtual Service object definition:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: hello-vs
spec:
hosts:
- "hello.<domain>"
gateways:
- hello-gateway
http:
- route:
- destination:
host: hello
port:
number: 8080The Virtual Service object binds the Gatway (hello-gateway) and one of the the hosts (in our example we just configured one, but you can include multiple host definitions there) configured in such Gateway (hello.<domain>) along with a destination Kubernetes Service (hello) and it’s port number. Once you create that binding, you can test that you finally get access to the application from outside the cluster:
Note: If you find that’s not working take a look at Kiali UI, since if for example, you misconfigure the Service name, you won’t get any error while creating the Virtual Service but you will find a message in Kiali:
Egress traffic
The first question here is, how does the Service Mesh handle the outgoing traffic by default?
You might think that you would have the same default behavior as non-Service Mesh outgoing traffic, that’s using the IP of the node where it’s running on, let’s check it:
Note: If you don’t know how to use tcpdump in CoreOS I would suggest to take a look at my Security Zones on OpenShift worker nodes series where, along with other tips, you can see the explanation of using not-preinstalled RPMs on the nodes
As you can see above, by default the POD uses the nodes IP, then, why do we have Egress Gateways? you can use them but you need to configure your Service Mech for such a thing, and that configuration will be shown, along with others, during Service Mesh Routing features review in another article.
Now that we have reviewed both Control and Data planes, in the next article of this series we will start playing with some of the Service Mesh features. I hope that you find it useful…. when it’s finished.
These are the rest of the articles related to the OpenShift Service Mesh series:
