Welcome to the EnMasse documentation, where you can find information to help you learn about EnMasse and start exploring its features.

From getting started to trying out more advanced installation and configuration, these resources provide what you need to set up and manage EnMasse as a service administrator or a messaging tenant.

1. Overview

EnMasse is an open source project for managed, self-service messaging on Kubernetes and OpenShift. EnMasse can run on your own infrastructure or in the cloud, and simplifies running a messaging infrastructure for your organization.

The service admin can deploy and manage messaging infrastructure, while messaging tenants can request messaging resources, both using both cloud-native APIs and tools.

1.1. Features

  • Built-in authentication and authorization of clients and identity management

  • Runs on Kubernetes and OpenShift: deploy on-premise or in the cloud

  • Different messaging patterns such as request-response, publish-subscribe and events

  • Decouple operation of infrastructure from configuration and use by applications

EnMasse can be used for many purposes, such as moving your messaging infrastructure to the cloud without depending on a specific cloud provider, building a scalable messaging backbone for IoT, or just as a cloud-ready version of a message broker.

EnMasse can provision different types of messaging depending on your use case. A user can request messaging resources by creating an address space.

EnMasse currently supports a standard and a brokered address space type, each with different semantics.

1.1.1. Standard address space

The standard address space type is the default type in EnMasse, and is focused on scaling in the number of connections and the throughput of the system. It supports AMQP and MQTT protocols, with more to come in the future. This address space type is based on other open source projects such as Apache ActiveMQ Artemis and Apache Qpid Dispatch Router and provides elastic scaling of these components. This image illustrates the high-level architecture of the standard address space:

Standard Address Space

1.1.2. Brokered address space

The brokered address space type is the "classic" message broker in the cloud which supports AMQP, CORE, OpenWire, and MQTT protocols. It supports JMS with transactions, message groups, selectors on queues and so on. These features are useful for building complex messaging patterns. This address space is also more lightweight as it features only a single broker and a management console. This image illustrates the high-level architecture of the brokered address space:

Brokered Address Space

2. Getting started

This guide will walk through the process of setting up EnMasse on Kubernetes with clients for sending and receiving messages.

Prerequisite
  • To install EnMasse, you must have Kubernetes installed. You can use minikube if you want to install EnMasse on your laptop.

  • A user on the Kubernetes cluster with cluster-admin permissions is required, in order to set up the required cluster roles and API services.

2.1. Downloading EnMasse

Procedure

2.2. Installing EnMasse using a YAML bundle

The simplest way to install EnMasse is to use the predefined YAML bundles.

Procedure
  1. Create the namespace where you want to deploy EnMasse:

    kubectl create namespace enmasse-infra
    kubectl config set-context $(kubectl config current-context) --namespace=enmasse-infra
  2. Create a certificate to use with the standard authentication service. For testing purposes, you can create a self-signed certificate:

    mkdir -p standard-authservice-cert
    openssl req -new -x509 -batch -nodes -days 11000 -subj "/O=io.enmasse/CN=standard-authservice.enmasse-infra.svc.cluster.local" -out standard-authservice-cert/tls.crt -keyout standard-authservice-cert/tls.key
  3. Create a secret with the standard authentication service certificate:

    kubectl create secret tls standard-authservice-cert --cert=standard-authservice-cert/tls.crt --key=standard-authservice-cert/tls.key
  4. Create a certificate to use with the API server. For testing purposes, you can create a self-signed certificate:

    mkdir -p api-server-cert/
    openssl req -new -x509 -batch -nodes -days 11000 -subj "/O=io.enmasse/CN=api-server.enmasse-infra.svc.cluster.local" -out api-server-cert/tls.crt -keyout api-server-cert/tls.key
  5. Create a secret containing the API server certificate:

    kubectl create secret tls api-server-cert --cert=api-server-cert/tls.crt --key=api-server-cert/tls.key
  6. Deploy using the enmasse-with-standard-authservice bundle:

    kubectl apply -f install/bundles/enmasse-with-standard-authservice

2.3. Creating address spaces using the command line

In EnMasse, you create address spaces using standard command-line tools.

Procedure
  1. Create an address space definition:

    apiVersion: enmasse.io/v1beta1
    kind: AddressSpace
    metadata:
      name: myspace
    spec:
      type: standard
      plan: standard-unlimited
  2. Create the address space:

    kubectl create -f standard-address-space.yaml
  3. Check the status of the address space:

    kubectl get addressspace myspace -o jsonpath={.status.isReady}

    The address space is ready for use when the previous command outputs true.

2.4. Creating addresses using the command line

You can create addresses using the command line.

Procedure
  1. Create an address definition:

    apiVersion: enmasse.io/v1beta1
    kind: Address
    metadata:
        name: myspace.myqueue
    spec:
        address: myqueue
        type: queue
        plan: standard-small-queue
    Note
    Prefixing the name with the address space name is required to ensure addresses from different address spaces do not collide.
  2. Create the address:

    kubectl create -f standard-small-queue.yaml
  3. List the addresses:

    kubectl get addresses -o yaml

2.5. Creating users using the command line

In EnMasse users can be created using standard command-line tools.

Prerequisites
Procedure
  1. Save the user definition to a file:

    apiVersion: user.enmasse.io/v1beta1
    kind: MessagingUser
    metadata:
      name: myspace.user1
    spec:
      username: user1
      authentication:
        type: password
        password: cGFzc3dvcmQ= # Base64 encoded
      authorization:
        - addresses: ["myqueue", "queue1", "queue2", "topic*"]
          operations: ["send", "recv"]
        - addresses: ["anycast1"]
          operations: ["send"]
  2. Create the user and associated user permissions:

    kubectl create -f user-example1.yaml
  3. Confirm that the user was created:

    kubectl get messagingusers

2.6. Sending and receiving messages

Prerequisites
  • Installed Apache Qpid Proton Python bindings.

  • An address space named myspace must be created.

  • An address named myqueue must be created.

  • A user named user1 with password password must be created.

Procedure
  1. Save Python client example to a file:

    from __future__ import print_function, unicode_literals
    import optparse
    from proton import Message
    from proton.handlers import MessagingHandler
    from proton.reactor import Container
    
    class HelloWorld(MessagingHandler):
        def __init__(self, url):
            super(HelloWorld, self).__init__()
            self.url = url
    
        def on_start(self, event):
            event.container.create_receiver(self.url)
            event.container.create_sender(self.url)
    
        def on_sendable(self, event):
            event.sender.send(Message(body="Hello World!"))
            event.sender.close()
    
        def on_message(self, event):
            print("Received: " + event.message.body)
            event.connection.close()
    
    parser = optparse.OptionParser(usage="usage: %prog [options]")
    parser.add_option("-u", "--url", default="amqps://localhost:5672/myqueue",
                      help="url to use for sending and receiving messages")
    opts, args = parser.parse_args()
    
    try:
        Container(HelloWorld(opts.url)).run()
    except KeyboardInterrupt: pass
  2. Retrieve the address space messaging endpoint host name:

    kubectl get addressspace myspace -o jsonpath={.status.endpointStatuses[?(@.name==\'messaging\')].externalHost}

    Use the output as the host name in the following step.

  3. Run the client:

    python client-example1.py -u amqps://user1:password@messaging.example1.com:443/myqueue

We have seen how to set up EnMasse on Kubernetes, and how to communicate with it using an AMQP client.

3. Service admin guide

The service administrator guide provides resources on how to set up and manage EnMasse, infrastructure configuration, and plans.

3.1. Installing EnMasse

These steps follow the manual deployment procedure and work on any platform supporting the kubectl command-line client.

Prerequisite
  • To install EnMasse, you must have Kubernetes installed. You can use minikube if you want to install EnMasse on your laptop.

  • A user on the Kubernetes cluster with cluster-admin permissions is required, in order to set up the required cluster roles and API services.

3.1.1. Downloading EnMasse

Procedure

3.1.2. Installing EnMasse using a YAML bundle

The simplest way to install EnMasse is to use the predefined YAML bundles.

Procedure
  1. Create the namespace where you want to deploy EnMasse:

    kubectl create namespace enmasse-infra
    kubectl config set-context $(kubectl config current-context) --namespace=enmasse-infra
  2. Create a certificate to use with the standard authentication service. For testing purposes, you can create a self-signed certificate:

    mkdir -p standard-authservice-cert
    openssl req -new -x509 -batch -nodes -days 11000 -subj "/O=io.enmasse/CN=standard-authservice.enmasse-infra.svc.cluster.local" -out standard-authservice-cert/tls.crt -keyout standard-authservice-cert/tls.key
  3. Create a secret with the standard authentication service certificate:

    kubectl create secret tls standard-authservice-cert --cert=standard-authservice-cert/tls.crt --key=standard-authservice-cert/tls.key
  4. Create a certificate to use with the API server. For testing purposes, you can create a self-signed certificate:

    mkdir -p api-server-cert/
    openssl req -new -x509 -batch -nodes -days 11000 -subj "/O=io.enmasse/CN=api-server.enmasse-infra.svc.cluster.local" -out api-server-cert/tls.crt -keyout api-server-cert/tls.key
  5. Create a secret containing the API server certificate:

    kubectl create secret tls api-server-cert --cert=api-server-cert/tls.crt --key=api-server-cert/tls.key
  6. Deploy using the enmasse-with-standard-authservice bundle:

    kubectl apply -f install/bundles/enmasse-with-standard-authservice

3.1.3. Installing EnMasse manually

The manual deployment procedure can be performed on any platform supporting the Kubernetes client.

Creating the project for EnMasse
Procedure
  1. Create the enmasse-infra namespace:

    kubectl create namespace enmasse-infra
  2. Set the enmasse-infra namespace as the default namespace:

    kubectl config set-context $(kubectl config current-context) --namespace=enmasse-infra
Deploying authentication services

EnMasse requires at least one authentication service to be deployed:

  • none (allow all),

  • standard (Keycloak), or

  • external (not managed by EnMasse).

Deploying the none authentication service
Procedure
  1. Create a certificate to use with the none authentication service. For testing purposes, you can create a self-signed certificate:

    mkdir -p none-authservice-cert
    openssl req -new -x509 -batch -nodes -days 11000 -subj "/O=io.enmasse/CN=none-authservice.enmasse-infra.svc.cluster.local" -out none-authservice-cert/tls.crt -keyout none-authservice-cert/tls.key
  2. Create a secret with the none authentication service certificate:

    kubectl create secret tls none-authservice-cert --cert=none-authservice-cert/tls.crt --key=none-authservice-cert/tls.key
  3. Create the none authentication service:

    kubectl create -f ./install/components/none-authservice
Deploying the standard authentication service
Procedure
  1. (Optional) If you want to deploy to a namespace other than enmasse-infra you must run the following command:

    sed -i 's/enmasse-infra/my-namespace/' install/components/standard-authservice/*RoleBinding*.yaml
  2. Create a certificate to use with the standard authentication service. For testing purposes, you can create a self-signed certificate:

    mkdir -p standard-authservice-cert
    openssl req -new -x509 -batch -nodes -days 11000 -subj "/O=io.enmasse/CN=standard-authservice.enmasse-infra.svc.cluster.local" -out standard-authservice-cert/tls.crt -keyout standard-authservice-cert/tls.key
  3. Create a secret with the standard authentication service certificate:

    kubectl create secret tls standard-authservice-cert --cert=standard-authservice-cert/tls.crt --key=standard-authservice-cert/tls.key
  4. Create the standard authentication service:

    kubectl create -f ./install/components/standard-authservice
Deploying the Address Space Controller

The Address Space Controller is responsible for creating the infrastructure used by address spaces.

Procedure
  1. (Optional) If you want to deploy to a namespace other than enmasse-infra you must run the following command:

    sed -i 's/enmasse-infra/my-namespace/' install/components/address-space-controller/*.yaml
  2. Deploy the Address Space Controller

    kubectl apply -f install/components/address-space-controller
Deploying the API server

The API server provides a REST API for creating address spaces and addresses. It can also serve as an aggregated API server if it is registered as an API service.

Procedure
  1. (Optional) If you want to deploy to a namespace other than enmasse-infra you must run the following command and substitute enmasse-infra in subsequent steps:

    sed -i 's/enmasse-infra/my-namespace/' install/components/api-service/*.yaml
    sed -i 's/enmasse-infra/my-namespace/' install/components/api-server/*.yaml
  2. Register API service

    kubectl apply -f install/components/api-service
  3. Create a certificate to use with the API server. For testing purposes, you can create a self-signed certificate:

    mkdir -p api-server-cert/
    openssl req -new -x509 -batch -nodes -days 11000 -subj "/O=io.enmasse/CN=api-server.enmasse-infra.svc.cluster.local" -out api-server-cert/tls.crt -keyout api-server-cert/tls.key
  4. Create a secret containing the API server certificate:

    kubectl create secret tls api-server-cert --cert=api-server-cert/tls.crt --key=api-server-cert/tls.key
  5. Deploy the API Server

    kubectl apply -f install/components/api-server/

3.2. Configuring EnMasse

3.2.1. Address space plans

Address space plans are used to configure quotas and control the resources consumed by address spaces. Address space plans are configured by the EnMasse service operator and are selected when creating an address space.

EnMasse includes a default set of plans that are sufficient for most use cases.

Plans are configured as custom resources. The following example shows a plan for the standard address space:

apiVersion: admin.enmasse.io/v1beta1
kind: AddressSpacePlan
metadata:
  name: restrictive-plan
  labels:
    app: enmasse
  annotations:
    enmasse.io/defined-by: default
displayName: Restrictive Plan
displayOrder: 0
shortDescription: A plan with restrictive quotas
longDescription: A plan with restrictive quotas for the standard address space
uuid: 74b9a40e-117e-11e8-b4e1-507b9def37d9
addressSpaceType: standard
addressPlans:
- small-queue
- small-anycast
resources:
- name: router
  max: 2.0
- name: broker
  max: 2.0
- name: aggregate
  max: 2.0

The following fields are required:

  • metadata.name

  • resources

  • addressPlans

  • addressSpaceType

The other fields are used by the EnMasse Console UI. Note the annotation enmasse.io/defined-by, which points to an infrastructure configuration that must exist when an address space using this plan is created. For more information about infrastructure configurations, see Infrastructure configuration.

3.2.2. Creating address space plans

Procedure
  1. Create an address space plan definition:

    apiVersion: admin.enmasse.io/v1beta1
    kind: AddressSpacePlan
    metadata:
      name: restrictive-plan
      labels:
        app: enmasse
      annotations:
        enmasse.io/defined-by: default
    displayName: Restrictive Plan
    displayOrder: 0
    shortDescription: A plan with restrictive quotas
    longDescription: A plan with restrictive quotas for the standard address space
    uuid: 74b9a40e-117e-11e8-b4e1-507b9def37d9
    addressSpaceType: standard
    addressPlans:
    - small-queue
    - small-anycast
    resources:
    - name: router
      max: 2.0
    - name: broker
      max: 2.0
    - name: aggregate
      max: 2.0
  2. Create the address space plan:

    kubectl create -f restrictive-plan.yaml
  3. Verify that schema has been updated and contains the plan:

    kubectl get addressspaceschema standard -o yaml

3.2.3. Address plans

Address plans specify the expected resource usage of a given address. The sum of the resource usage for all resource types determines the amount of infrastructure provisioned for an address space. A single router and broker pod has a maximum usage of one. If a new address requires additional resources and the resource consumption is within the address space plan limits, a new pod will be created automatically to handle the increased load.

Address plans are configured by the EnMasse service operator and are selected when creating an address.

EnMasse includes a default set of address plans that are sufficient for most use cases.

In the Address space plans section, the address space plan references two address plans: small-queue and small-anycast. These address plans are stored as custom resources and are defined as follows:

apiVersion: admin.enmasse.io/v1beta1
kind: AddressPlan
metadata:
  name: small-queue
  labels:
    app: enmasse
displayName: Small queue plan
displayOrder: 0
shortDescription: A plan for small queues
longDescription: A plan for small queues that consume little resources
uuid: 98feabb6-1183-11e8-a769-507b9def37d9
addressType: queue
requiredResources:
- name: router
  credit: 0.2
- name: broker
  credit: 0.3

The following fields are required:

  • metadata.name

  • requiredResources

  • addressType

A single router can support five instances of addresses and broker can support three instances of addresses with this plan. If the number of addresses with this plan increases to four, another broker is created. If it increases further to six, another router is created as well.

Note, however, that although the address space plan allows two routers and two brokers to be deployed, it only allows two pods to be deployed in total. This means that the address space is restricted to three addresses with the small-queue plan.

The small-anycast plan does not consume any broker resources, and can provision two routers at the expense of not being able to create any brokers:

apiVersion: admin.enmasse.io/v1beta1
kind: AddressPlan
metadata:
  name: small-anycast
  labels:
    app: enmasse
displayName: Small anycast plan
displayOrder: 0
shortDescription: A plan for small anycast addresses
longDescription: A plan for small anycast addresses that consume little resources
uuid: cb61f440-1184-11e8-adda-507b9def37d9
addressType: anycast
requiredResources:
- name: router
  credit: 0.2

With this plan, up to 10 addresses can be created.

3.2.4. Creating address plans

Procedure
  1. Create an address plan definition:

    apiVersion: admin.enmasse.io/v1beta1
    kind: AddressPlan
    metadata:
      name: small-anycast
      labels:
        app: enmasse
    displayName: Small anycast plan
    displayOrder: 0
    shortDescription: A plan for small anycast addresses
    longDescription: A plan for small anycast addresses that consume little resources
    uuid: cb61f440-1184-11e8-adda-507b9def37d9
    addressType: anycast
    requiredResources:
    - name: router
      credit: 0.2
  2. Create the address plan:

    kubectl create -f small-anycast-plan.yaml
  3. Verify that schema has been updated and contains the plan:

    kubectl get addressspaceschema standard -o yaml

3.2.5. Infrastructure configuration

EnMasse creates infrastructure components such as routers, brokers, and consoles. These components can be configured while the system is running, and EnMasse automatically updates the components with the new settings. The EnMasse service operator can edit the EnMasse default infrastructure configuration or create new configurations.

Infrastructure configurations can be referred to from one or more address space plans. For more information about address space plans, see Address space plans.

Infrastructure configuration can be managed for both brokered and standard infrastructure using BrokeredInfraConfig and StandardInfraConfig resources.

Brokered infrastructure configuration

BrokeredInfraConfig resources are used to configure infrastructure deployed by brokered address spaces. The brokered infrastructure configuration is referenced by address space plans using a enmasse.io/defined-by annotation. For more information, see Address space plans.

apiVersion: admin.enmasse.io/v1beta1
kind: BrokeredInfraConfig
metadata:
  name: brokered-infra-config-example
spec:
  version: 0.26-SNAPSHOT
  admin:
    resources:
      memory: 256Mi
  broker:
    resources:
      memory: 2Gi
      storage: 100Gi
    addressFullPolicy: PAGE

The version field specifies the EnMasse version used. When upgrading, EnMasse uses this field to determine whether to upgrade the infrastructure to the requested version.

The admin object specifies the settings you can configure for the admin components.

The broker object specifies the settings you can configure for the broker components. Changing the .broker.resources.storage setting does not configure the existing broker storage size.

Standard infrastructure configuration

StandardInfraConfig resources are used to configure infrastructure deployed by standard address spaces. The standard infrastructure configuration is referenced by address space plans using a enmasse.io/defined-by annotation. For more information, see Address space plans.

apiVersion: admin.enmasse.io/v1beta1
kind: StandardInfraConfig
metadata:
  name: myconfig
spec:
  version: 0.26-SNAPSHOT
  admin:
    resources:
      memory: 256Mi
  broker:
    resources:
      memory: 2Gi
      storage: 100Gi
    addressFullPolicy: PAGE
  router:
    resources:
      memory: 256Mi
    linkCapcity: 1000
    minReplicas: 1

The version field specifies the EnMasse version used. When upgrading, EnMasse uses this field to determine whether to upgrade the infrastructure to the requested version.

The admin object specifies the settings you can configure for the admin components.

The broker object specifies the settings you can configure for the broker components. Changing the .broker.resources.storage setting does not configure the existing broker storage size.

The router object specifies the settings you can configure for the router components.

3.2.6. Applying infrastructure configuration

You can edit existing configurations or create new ones.

Prerequisites
Procedure
  1. Create infrastructure configuration:

    apiVersion: admin.enmasse.io/v1beta1
    kind: StandardInfraConfig
    metadata:
      name: myconfig
    spec:
      version: 0.26-SNAPSHOT
      admin:
        resources:
          memory: 256Mi
      broker:
        resources:
          memory: 2Gi
          storage: 100Gi
        addressFullPolicy: PAGE
      router:
        resources:
          memory: 256Mi
        linkCapcity: 1000
        minReplicas: 1
  2. Apply the configuration changes:

    kubectl apply -f standard-infra-config-example.yaml
  3. Monitor the pods while they are restarted:

    kubectl get pods -w

    The configuration changes will be applied within a couple of minutes.

3.3. Upgrading EnMasse

EnMasse supports upgrades between minor versions using cloud native tools and the same mechanism used to apply configuration changes. When upgrading, the updated infrastructure configuration of the new version will trigger the upgrade to start.

Upgrading EnMasse is done by applying the YAML files for the new version.

3.3.1. Upgrading EnMasse using a YAML bundle

Prerequisites
Procedure
  1. Select the namespace where EnMasse is installed:

    kubectl config set-context $(kubectl config current-context) --namespace=enmasse-infra
  2. Apply the new release bundle:

    kubectl apply -f install/bundles/enmasse-with-standard-authservice
  3. Monitor pods while they are restarted:

    kubectl get pods -w

    The upgrade should cause all pods to be restarted within a couple of minutes.

3.4. Monitoring EnMasse

You can monitor EnMasse by deploying built-in monitoring tools or using your pre-existing monitoring infrastructure.

3.4.1. Deploying monitoring

Monitoring services run frequent health checks on EnMasse and send alerts when health checks fail. Health is assessed using Prometheus and kube-state-metrics. Alerting is implemented with Alertmanager. Grafana is also configured to provide a dashboard of the current status of health checks.

Deploying Prometheus
Procedure
  1. Replace the namespace with the namespace EnMasse is currently deployed to:

    sed -i 's/enmasse-infra/_my-namespace_/' install/components/prometheus/*.yaml
  2. Create the Prometheus deployment:

    kubectl apply -f ./install/components/prometheus
Deploying kube-state-metrics
Procedure
  1. Replace the namespace with the namespace EnMasse is currently deployed to:

    sed -i 's/enmasse-infra/_my-namespace_/' install/components/kube-state-metrics/*.yaml
  2. Create the kube-state-metrics deployment:

    kubectl apply -f ./install/components/kube-state-metrics
Deploying Alertmanager
Prerequisites
  • You must have already installed Prometheus and kube-state-metrics.

Procedure
  1. Replace the namespace with the namespace EnMasse is currently deployed to:

    sed -i 's/enmasse-infra/_my-namespace_/' install/components/alertmanager/*.yaml
  2. Create the Alertmanager deployment:

    kubectl apply -f ./install/components/alertmanager
  3. (Optional) Configure Alertmanager to send emails using a custom configmap with the relevant SMTP details:

    kubectl apply -f ./alertmanager-configmap.yaml

    An example Alertmanager config map:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: enmasse
  name: alertmanager-config
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
      smtp_smarthost: localhost
      smtp_from: alerts@localhost
      smtp_auth_username: admin
      smtp_auth_password: password
    route:
      group_by: ['alertname']
      group_wait: 60s
      group_interval: 60s
      repeat_interval: 1h
      receiver: 'sysadmins'
    receivers:
    - name: 'sysadmins'
      email_configs:
      - to: sysadmin@localhost
    inhibit_rules:
      - source_match:
          severity: 'critical'
        target_match:
          severity: 'warning'
        equal: ['alertname']
Deploying Grafana
Prerequisites
  • You must have already installed Prometheus and kube-state-metrics.

Procedure
  1. Replace the namespace with the namespace EnMasse is currently deployed to:

    sed -i 's/enmasse-infra/_my-namespace_/' install/components/grafana/*.yaml
  2. Create the Grafana deployment:

    kubectl apply -f ./install/components/grafana

3.4.2. Monitor EnMasse with existing infrastructure

EnMasse can also be monitored using an existing Prometheus server and optionally kube-state-metrics for the relevant Kubernetes infrastructure.

Exposed metrics

The Address Space Controller and API Server export Prometheus metrics about EnMasse’s health on port 8080. These metrics can be scraped using the services created during EnMasse installation and the following sample Prometheus scrape configuration.

Prerequisites
  • Prometheus' service account must have View privileges to the EnMasse namespace.

Procedure
  1. Add the following job to your Prometheus scrape configuration:

      - job_name: <component-health>
        kubernetes_sd_configs:
          - role: service
            namespaces:
              names:
                - <namespace>
        metrics_path: "/metrics"
        relabel_configs:
        - action: keep
          regex: health.*
          source_labels: [__meta_kubernetes_service_port_name]
Alerts

You can enable alerts regarding the health of EnMasse resources.

Procedure
  1. Add the following rules to your Prometheus Rules configuration:

      - alert: Component_Health
        annotations:
          description: '{{ $labels.summary }}'
          value: "{{ $value }}"
          severity: warning
        expr: health == 1
        for: 60s
      - alert: Address_Space_Health
        annotations:
          description: '{{ $labels.summary }}'
          value: "{{ $value }}"
          severity: warning
        expr: address_spaces_not_ready_total > 0
        for: 300s
      - alert: Address_Health
        annotations:
          description: '{{ $labels.summary }}'
          value: "{{ $value }}"
          severity: warning
        expr: addresses_not_ready_total > 0
        for: 300s

3.4.3. Restarting a component

Procedure
  • Delete the component’s pod:

    kubectl delete pod -l name=component

The component will be automatically restarted in a new pod.

3.4.4. Viewing router logs

Procedure
  1. List all router pods and choose the pod for the relevant address space:

    kubectl get pods -l name=qdrouterd -o go-template --template '{{range .items}}{{.metadata.name}}{{"\t"}}{{.metadata.annotations.addressSpace}}{{"\n"}}{{end}}'
  2. Display the logs for the pod:

    kubectl logs pod -c router

3.4.5. Viewing broker logs

Procedure
  1. List all broker pods and choose the pod for the relevant address space:

    kubectl get pods -l role=broker -o go-template --template '{{range .items}}{{.metadata.name}}{{"\t"}}{{.metadata.annotations.addressSpace}}{{"\n"}}{{end}}'
  2. Display the logs for the pod:

    kubectl logs pod

3.5. Uninstalling EnMasse

Procedure
  1. Delete the rolebinding in the kube-system namespace:

    kubectl delete rolebindings -l app=enmasse -n kube-system
  2. Delete "cluster level" resources:

    kubectl delete clusterrolebindings -l app=enmasse
    kubectl delete crd -l app=enmasse
    kubectl delete clusterroles -l app=enmasse
    kubectl delete apiservices -l app=enmasse
  3. Delete the namespace where EnMasse is deployed:

    kubectl delete namespace enmasse-infra

4. Tenant guide

The tenant guide provides resources on how to manage address spaces, addresses, and users as a messaging tenant.

4.1. Managing address spaces

EnMasse is configured to support managing address spaces using the Kubernetes command-line tools. Address spaces are managed like any other Kubernetes resource using kubectl.

4.1.1. Address space

An address space is a group of addresses that can be accessed through a single connection (per protocol). This means that clients connected to the endpoints of an address space can send messages to or receive messages from any authorized address within that address space. An address space can support multiple protocols, as defined by the address space type.

EnMasse has two types of address spaces:

4.1.2. Standard address space

The standard address space is the default address space in EnMasse. It consists of an AMQP router network in combination with attachable storage units. Clients connect to a message router, which forwards messages to or from one or more message brokers. This address space type is appropriate when you have many connections and addresses. However, the standard address space has the following limitations:

  • No transaction support

  • No message ordering

  • No selectors on queues

  • No browsing on queues

  • No message groups

Clients connect and send and receive messages in this address space using the AMQP or MQTT protocols. Note that MQTT does not support qos2 or retained messages.

Standard address types

The standard address space supports five different address types:

  • queue

  • topic

  • anycast

  • multicast

  • subscription

Queue

The queue address type is a store-and-forward queue. This address type is appropriate for implementing a distributed work queue, handling traffic bursts, and other use cases where you want to decouple the producer and consumer. A queue can be sharded across multiple storage units. Message ordering might be lost for queues in the standard address space.

Topic

The topic address type supports the publish-subscribe messaging pattern where there are 1..N producers and 1..M consumers. Each message published to a topic address is forwarded to all subscribers for that address. A subscriber can also be durable, in which case messages are kept until the subscriber has acknowledged them.

Note
If you create a subscription on a topic, any senders to that topic must specify the topic capability.
Hierarchical topics and wildcards

A client receiving from a topic address can specify a wildcard address with the topic address as the root. The wildcard behavior follows the MQTT syntax:

  • / is a separator

  • + matches one level

  • # matches one or more levels

So, for example:

  • a/#/b matches a/foo/b, a/bar/b, and a/foo/bar/b

  • a/+/b matches a/foo/b and a/bar/b, but would not match a/foo/bar

In the standard address space, the first level must always be a defined topic address; that is, # and + are not valid as the first characters of a subscribing address.

Anycast

The anycast address type is a scalable direct address for sending messages to one consumer. Messages sent to an anycast address are not stored, but are instead forwarded directly to the consumer. This method makes this address type ideal for request-reply (RPC) uses or even work distribution. This is the cheapest address type as it does not require any persistence.

Multicast

The multicast address type is a scalable direct address for sending messages to multiple consumers. Messages sent to a multicast address are forwarded to all consumers receiving messages on that address. Because message acknowledgments from consumers are not propagated to producers, only pre-settled messages can be sent to multicast addresses.

Subscription

The subscription address type allows a subscription to be created for a topic that holds messages published to the topic even if the subscriber is not attached. The subscription is accessed by the consumer using <topic-address>::<subscription-address>. For example, for a subscription mysub on a topic mytopic the consumer consumes from the address mytopic::mysub.

4.1.3. Brokered address space

The brokered address space is designed to support broker-specific features, at the cost of limited scale in terms of the number of connections and addresses. This address space supports JMS transactions, message groups, and selectors on queues and topics.

Clients can connect and send and receive messages in this address space using the AMQP, CORE, OpenWire, and MQTT protocols.

Brokered address types

The brokered address space supports two address types:

  • queue

  • topic

Queue

The queue address type is a store-and-forward queue. This address type is appropriate for implementing a distributed work queue, handling traffic bursts, and other use cases where you want to decouple the producer and consumer. A queue in the brokered address space supports selectors, message groups, transactions, and other JMS features. Message order can be lost with released messages.

Topic

The topic address type supports the publish-subscribe messaging pattern in which there are 1..N producers and 1..M consumers. Each message published to a topic address is forwarded to all subscribers for that address. A subscriber can also be durable, in which case messages are kept until the subscriber has acknowledged them.

Hierarchical topics and wildcards

A client receiving from a topic address can specify a wildcard address with the topic address as the root. The wildcard behavior follows the MQTT syntax:

  • / is a separator

  • + matches one level

  • # matches one or more levels

So, for example:

  • a/#/b matches a/foo/b, a/bar/b, a/foo/bar/b

  • a/+/b matches a/foo/b and a/bar/b, but would not match a/foo/bar

4.1.4. Address space plans

An address space is configured with an address space plan, which describes the allowed resource usage of that address space. The address space plans are configured by the service administrator and can vary between EnMasse installations.

The address space plan can be changed if the address space requires more, or less, resources.

4.1.5. Listing available plans using the command line