Was this page helpful?
Exposing ScyllaDB cluster¶
This document explains how ScyllaDB Operator exposes ScyllaDB clusters in different network setups. A ScyllaDB cluster can be exposed in various network configurations, independently to clients and nodes.
Expose Options¶
Note
exposeOptions are immutable, they cannot be changed after a ScyllaDB cluster is created.
exposeOptions specifies configuration options for exposing ScyllaCluster’s.
A ScyllaCluster created without any exposeOptions is equivalent to the following:
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: ClusterIP
broadcastOptions:
clients:
type: ServiceClusterIP
nodes:
type: ServiceClusterIP
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: Headless
broadcastOptions:
clients:
type: PodIP
nodes:
type: PodIP
The following sections cover what every field controls and what the configuration options are.
Node Service Template¶
nodeService serves as a template for a node-dedicated Service managed by the Scylla Operator for each node within a ScyllaDB cluster.
The properties of the Services depend on the selected type.
Additionally, there’s an option to define custom annotations, incorporated into each node’s Service,
which might be useful for further tweaking the Service properties or related objects.
Headless Type¶
For Headless type, Scylla Operator creates a Headless Service with a selector pointing to the particular node in the ScyllaDB cluster.
Such Service doesn’t provide any additional IP addresses, and the internal DNS record resolves to the PodIP of a node.
This type of Service is useful when ScyllaDB cluster nodes broadcast PodIPs to clients and other nodes.
Example:
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: Headless
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: Headless
ClusterIP Type¶
For ClusterIP type, Scylla Operator creates a ClusterIP Service backed by a specific node in the ScyllaDB cluster.
These IP addresses are only routable within the same Kubernetes cluster, so it’s a good fit, if you don’t want to expose them to other networks.
Example:
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: ClusterIP
Caution
ScyllaDBCluster is designated for usage across multiple Kubernetes clusters. As ClusterIP Services are conventionally only routable within a single Kubernetes cluster, make sure your CNI allows for external ClusterIP connectivity. Otherwise, switch to a different type of exposure.
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: ClusterIP
LoadBalancer Type¶
For the LoadBalancer type, Scylla Operator generates a LoadBalancer Service that directs traffic to a specific node within the ScyllaCluster.
On platforms with support for external load balancers, this Service provisions one.
The accessibility of this load balancer’s address depends on the platform and any customizations made; in some cases it may be reachable from the internal network or public Internet.
Customizations are usually managed via Service annotations, key-value pairs provided in annotations field are merged into each Service object.
LoadBalancer Services should be configured to pass through entire traffic.
For example, to expose LoadBalancer only to internal network use the following annotations:
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: LoadBalancer
annotations:
networking.gke.io/load-balancer-type: Internal
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: LoadBalancer
annotations:
networking.gke.io/load-balancer-type: Internal
Check platform-specific documentation regarding LoadBalancer configuration to learn more about available options.
LoadBalancer Service is a superset of ClusterIP Service, implying that each LoadBalancer Service also contains an allocated ClusterIP. They can be configured using the following fields, which propagate to every node Service:
externalTrafficPolicy
internalTrafficPolicy
loadBalancerClass
allocateLoadBalancerNodePorts
Check Kubernetes Service documentation to learn more about these options.
Example:
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: LoadBalancer
loadBalancerClass: my-custom-load-balancer-class
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: LoadBalancer
loadBalancerClass: my-custom-load-balancer-class
Broadcast Options¶
Broadcast options control what is the source of the address being broadcasted to clients and nodes. It’s configured independently for clients and nodes because you may want to expose these two types of traffic on different networks. Using different networks can help manage costs, reliability, latency, security policies or other metrics you care about.
PodIP Type¶
Address broadcasted to clients/nodes is taken from Pod.
By default, the address is taken from Pod’s status.PodIP field.
Because a Pod can use multiple address, you may want to provide source options by specifying podIP.source.
Example:
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
broadcastOptions:
clients:
type: PodIP
podIP:
source: Status
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
broadcastOptions:
clients:
type: PodIP
podIP:
source: Status
ServiceClusterIP Type¶
Address broadcasted to clients or nodes is taken from spec.ClusterIP field of a node’s dedicated Service.
In order to configure it, the nodeService template must specify a Service having a ClusterIP assigned.
Example:
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
broadcastOptions:
clients:
type: ServiceClusterIP
Caution
ScyllaDBCluster is designated for usage across multiple Kubernetes clusters. As ClusterIP Services are conventionally only routable within a single Kubernetes cluster, make sure your CNI allows for external ClusterIP connectivity. Otherwise, switch to a different type of exposure.
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
broadcastOptions:
clients:
type: ServiceClusterIP
ServiceLoadBalancerIngress Type¶
Address broadcasted to clients/nodes is taken from the node dedicated Service, from status.ingress[0].ipAddress or status.ingress[0].hostname field.
In order to configure it, the nodeService template must specify the LoadBalancer Service.
Example:
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
broadcastOptions:
clients:
type: ServiceLoadBalancerIngress
podIP:
source: Status
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
broadcastOptions:
clients:
type: ServiceLoadBalancerIngress
podIP:
source: Status
Deployment Examples¶
The following section contains several specific examples of various network scenarios and explains how nodes and clients communicate with one another.
In-cluster only¶
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: ClusterIP
broadcastOptions:
clients:
type: ServiceClusterIP
nodes:
type: ServiceClusterIP
Caution
ScyllaDBCluster is designated for usage across multiple Kubernetes clusters. As ClusterIP Services are conventionally only routable within a single Kubernetes cluster, make sure your CNI allows for external ClusterIP connectivity. Otherwise, switch to a different type of exposure.
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: ClusterIP
broadcastOptions:
clients:
type: ServiceClusterIP
nodes:
type: ServiceClusterIP
Both client and nodes are deployed within the same Kubernetes cluster. They talk through ClusterIP addresses taken from the Service. Because ClusterIP Services are only routable within the same Kubernetes cluster, this cluster won’t be reachable from outside.
In-cluster node-to-node, VPC clients-to-nodes¶
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: ClusterIP
broadcastOptions:
clients:
type: PodIP
nodes:
type: ServiceClusterIP
Caution
ScyllaDBCluster is designated for usage across multiple Kubernetes clusters. As ClusterIP Services are conventionally only routable within a single Kubernetes cluster, make sure your CNI allows for external ClusterIP connectivity. Otherwise, switch to a different type of exposure.
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: ClusterIP
broadcastOptions:
clients:
type: PodIP
nodes:
type: ServiceClusterIP
In this scenario, we assume that the Pod IP subnet is routable within a VPC. Clients within the VPC network can communicate directly with ScyllaCluster nodes using PodIPs. Nodes communicate with each other exclusively within the same Kubernetes cluster.
Multi VPC¶
Both ScyllaDB datacenters use the same exposeOptions, nodes broadcast their Pod IP addresses, enabling them to establish connections with one another.
Check dedicated documentation page to know how to connect two ScyllaClusters into one logical cluster.
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: Headless
broadcastOptions:
clients:
type: PodIP
nodes:
type: PodIP
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: Headless
broadcastOptions:
clients:
type: PodIP
nodes:
type: PodIP
In this scenario, we set up two separate Kubernetes clusters in distinct VPCs. These VPCs are interconnected to facilitate inter-VPC connectivity. We operate on the assumption that the Pod IP subnet is routable within each VPC.
Clients, whether deployed within the same Kubernetes cluster or within a VPC, have the capability to reach nodes using their Pod IPs.
Since there is no requirement for any address other than the Pod IP, the Headless service type is sufficient.
Internet¶
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
spec:
exposeOptions:
nodeService:
type: LoadBalancer
broadcastOptions:
clients:
type: ServiceLoadBalancerIngress
nodes:
type: ClusterIP
Caution
ScyllaDBCluster is designated for usage across multiple Kubernetes clusters. As ClusterIP Services are conventionally only routable within a single Kubernetes cluster, make sure your CNI allows for external ClusterIP connectivity. Otherwise, switch to a different type of exposure.
apiVersion: scylla.scylladb.com/v1alpha1
kind: ScyllaDBCluster
spec:
exposeOptions:
nodeService:
type: LoadBalancer
broadcastOptions:
clients:
type: ServiceLoadBalancerIngress
nodes:
type: ClusterIP
We assume that a Kubernetes cluster has been deployed in a cloud provider environment that supports external load balancers. By specifying the LoadBalancer type in the nodeService template, the Scylla Operator generates a dedicated LB Service for each node. The cloud provider then establishes an external load balancer with an internet-accessible address. ScyllaDB nodes broadcast this external address to clients, enabling drivers to connect and discover other nodes. Since all ScyllaDB nodes reside within the same Kubernetes cluster, there is no need to route traffic through the internet. Consequently, the nodes are configured to communicate via ClusterIP, which is also accessible within LoadBalancer Services.
Other more complex scenarios can be built upon these simple ones.