Red Hat OpenStack Administration I: Core Operations for Domain Operators

Load Balancing-as-a-Service (LBaaS) distributes incoming requests evenly between instances. Requests are distributed using one of three methods. Round robin rotates requests evenly between all instances. Source IP ensures requests from the same source IP address are always sent to the same instance. Least connections ensures requests are sent to the least busy instance.

Neutron LBaaS v1.0, is the reference architecture for the classroom. Neutron LBaaS v2.0 is available in the OpenStack Liberty release and supports Octavia and HAProxy back ends.

OpenStack images are used to create instances. Images can be preconfigured to include all required packages, configuration, and files to boot a fully operational instance. Each image creates a different type of instance, for example, an application server, a database sever, or a web server. A preconfigured image can speed up instance build time and ensures consistency, but will require patching and keeping licenses up to date. It is possible to create an instance without using an image allowing for greater flexibility, but can increase creation time and result in inconsistencies.

Instances can mount and dismount persistent block storage virtual devices, called Cinder volumes. Cinder volumes can only be attached to one instance at a time. If an instance stops working, a new instance is created and the Cinder volume is attached to the new instance.

Swift is persistent object storage. Objects are stored and retrieved using a RESTFUL API. Unlike a physical device, Swift storage is never mounted to the instance. It is built for scale and optimized for durability and availability across an entire data set.

Neutron is used to create multiple subnets, one for each tier: a presentation subnet, an application subnet, and a data subnet. Neutron routers route traffic between the subnets.

Security groups are used to filter inbound traffic. Multiple rules can be created to allow or deny traffic from specific ports, IP addresses, or ranges. All OpenStack stacks are configured with a default security group. The default security group is assigned to instances that have no other security group defined. Unless modified, the default security group denies all incoming traffic.

Heat uses template files to automate the deployment of complicated cloud applications and environments. Orchestration is more than just standing up instances; it can install software, configure networking, install patches, and set security protocols.

The ability to scale horizontally is one of the greatest advantages of cloud computing. OpenStack can automatically launch additional instances for a specific application as required. Telemetry performs the system resource monitoring and can be configured to send an alarm when thresholds are exceeded. Heat then responds to the alarm according to the configured scale-up or scale-down policy.

In this section the general parameters and restrictions of the stack are configured. The general parameters include, the access authentication mechanism, the flavor of the instances, the image to be used for the creation of the instances, the networks to be assigned and the security groups, for example:

parameters:
  key_name:
    type: string
    description: Name of an existing key pair to use for the instance
    default: example-keypair
    constraints:
      - custom_constraint: nova.keypair

  flavor:
    type: string
    description: Flavor for the instance to be created
    default: default
    constraints:
      - custom_constraint: nova.flavor

  image:
    type: string
    description: Image ID or image name to use for the instance
    default: rhel8
    constraints:
      - custom_constraint: glance.image

  public_network_id:
    type: string
    description: ID of public network for which floating IP addresses will be allocated
    default: ef95203b-7c9f-46c0-b328-e51aa7729798

  private_network_id:
    type: string
    description: ID of private network into which instances get deployed
    default: 8799e43b-2848-4772-8c4c-c6545e54db19

  db_server_name:
    type: string
    description: Name of the database instances
    default: db_server

  app_server_name:
    type: string
    description: Name of the application instances
    default: app_server

  web_server_name:
    type: string
    description: Name of the web instances
    default: app_server

  dns_nameserver:
    type: comma_delimited_list
    description: The IP address of a DNS nameserver in list format
    default: 8.8.8.8,8.8.4.4

In the main HOT template the resources of the stack are defined, making reference to the template from where each one of them will be generated, grouping them for its individual deployment. Besides the network configuration for the different tiers and the launch of the storage or database resources.

resources:
  network_setup:
    type: lib/setup_net_sg.yaml
    properties:
      public_network_id: { get_param: public_network_id }
      dns_nameserver: { get_param: dns_nameserver }

  launch_db_server:
    type: lib/heat_sql_tier.yaml
    properties:
      key_name: { get_param: key_name }
      server_name: { get_param: db_server_name }
      instance_flavor: { get_param: flavor }
      image_id: { get_param: image_id }
      private_network_id: { get_param: private_network_id }
      security_group: { get_param: [network_setup, db_security_group_id] }

  app_server_resource_group:
    type: OS::Heat::ResourceGroup
    properties:
      count: 2
      resource_def: 2
        type: lib/heat_app_tier.yaml
        properties:
          key_name:
            get_param: key_name
          server_name:
            get_param: app_server_name
          instance_flavor:
            get_param: flavor
          image_id:
            get_param: image_id
          private_network_id: {get_attr: [network_setup, private_network_id]}
          security_group: {get_attr: [network_setup, app_security_group_id]}
          pool_name: {get_attr: [network_setup, app_lbass_pool_name]}
          db_server_ip: {get_attr: [launch_db_server, instance_ip]}

  web_server_resource_group:
    type: OS::Heat::ResourceGroup
    properties:
      count: 2
      resource_def: 2
        type: lib/heat_web_tier.yaml
        properties:
          key_name:
            get_param: key_name
          server_name:
            get_param: web_server_name
          instance_flavor:
            get_param: flavor
          image_id:
            get_param: image_id
          private_network_id: {get_attr: [network_setup, private_network_id]}
          app_lbaas_vip: {get_attr: [network_setup, app_lbaas_ip]}
          security_group: {get_attr: [network_setup, web_security_group_id]}
          pool_name: {get_attr: [network_setup, web_lbass_pool_name]}

The output of the main HOT template defines the output parameters of the web application stack after its creation.

outputs:
  web_lbaas_ip:
    descrption: Floating IP assigned to the web loadbalancer
    value: { get_attr: [ network_setup, web_lbaas_ip ] }

  app_lbaas_ip:
    descrption: Floating IP assigned to the app loadbalancer
    value: { get_attr: [ network_setup, app_lbaas_ip ] }

In addition, each of the templates that generate the resources also define different resources and outputs for each of them, using the same required and optional parameters.

This template defines the network configuration to be used in the 3-tier web application. Network segregation, load balancing and security groups are defined here. In this example, the template creates a private network and a security group containing rules for internal communication between the tiers. It also creates a pair of load balancers, one for public access to the web application and and internal one for use by the application tier and the data tier.

resources:
  private_network_subnet:
    type: OS::Neutron::Net
    properties:
      cidr: 192.168.100.0/24
      gateway_ip: 192.168.100.1
      allocation_pools: [{ "start": 192.168.100.10, "end": 192.168.100.100 }]
      name: private_subnet
      dns_nameserver: { get_param: dns_nameserver }
      enable_dhcp: true

  public_router:
    type: OS::Neutron::Router
    properties:
      name: public_router
      external_gateway_info:
        network: { get_param: public_network_id }

  public_router_interface:
    type: OS::Neutron::RouterInterface
    properties:
      router: { get_resource: public_router }
      subnet: { get_resource: private_network_subnet }

  web_tier_loadbalancer:
    type: OS::Neutron::LoadBalancer
    properties:
      protocol_port: 80
      pool_id: { get_resource: web_server_pool }

  web_tier_monitor:
    type: OS::Neutron::HealthMonitor
    properties:
      type: TCP
      delay: 5
      max_retries: 5
      timeout: 5

  web_server_pool:
    type: OS::Neutron::Pool
    properties:
      protocol: HTTP
      monitors: [{ get_resource: web_tier_monitor }]
      subnet: { get_resource: private_network_subnet }
      lb_method: ROUND_ROBIN
      vip:
        protocol_port: 80

  web_vip_port:
    type: OS::Neutron::Port
    properties:
      network: { get_resource: private_network }
      security_groups: [{ get_resource: web_security_group }]
      fixed_ips:
        subnet_id: { get_resource: private_network_subnet }

  web_network_floating_ip:
    type: OS::Neutron::FloatingIP
    properties:
      floating_network: { get_resource: public_network_id }
      port_id: { get_resource: web_vip_port }

  association:
    type: OS::Neutron::FloatingIPAssociation
    properties:
      floating_id: { get_resource: web_network_floating_ip }
      port_id: { get_resource: web_vip_port }

  app_tier_loadbalancer:
    type: OS::Neutron::LoadBalancer
    properties:
      protocol_port: 80
      pool_id: { get_resource: app_server_pool }

  app_tier_monitor:
    type: OS::Neutron::HealthMonitor
    properties:
      type: TCP
      delay: 5
      max_retries: 5
      timeout: 5

  app_server_pool:
    type: OS::Neutron::Pool
    properties:
      protocol: HTTP
      monitors: [{ get_resource: app_tier_monitor }]
      subnet: { get_resource: private_network_subnet }
      lb_method: ROUND_ROBIN
      vip:
        protocol_port: 80

  web_security_group:
    type: OS::Neutron::SecurityGroup
    properties:
      description: A application specific security group that passes ports 22 and 80
      name: workload_web_security_group
      rules:
        - protocol_port: tcp
          port_range_min: 22
          port_range_max: 22
        - protocol_port: tcp
          port_range_min: 80
          port_range_max: 80

  app_security_group:
    type: OS::Neutron::SecurityGroup
    properties:
      description: A application specific security group that passes ports 22, 80 and 3306
      name: workload_app_security_group
      rules:
        - protocol_port: tcp
          port_range_min: 22
          port_range_max: 22
        - protocol_port: tcp
          port_range_min: 80
          port_range_max: 80
        - protocol_port: tcp
          port_range_min: 3306
          port_range_max: 3306

  db_security_group:
    type: OS::Neutron::SecurityGroup
    properties:
      description: A database specific security group that passes ports 22 and 3306
      name: workload_db_security_group
      rules:
        - protocol_port: tcp
          port_range_min: 22
          port_range_max: 22
        - protocol_port: tcp
          port_range_min: 3306
          port_range_max: 3306

output:
  web_security_group:
    description: web security group ID
    value: {get_resource: web_security_group}

  app_security_group:
    description: app security group ID
    value: {get_resource: app_security_group}

  db_security_group:
    description: db security group ID
    value: {get_resource: web_security_group}

  web_lbaas_pool_name:
    description: Name of Web LBaaS Pool
    value: {get_resource: web_server_pool}

  app_lbaas_pool_name:
    description: Name of App LBaaS Pool
    value: {get_resource: app_server_pool}

  web_lbaas_ip:
    description: Public floating IP assigned to web LBaaS
    value: { get_attr: [ Web_Network_Floating_IP, floating_ip_address ] }

The web tier template installs and configures the web server instances to receive requests and consume the services of the following application tier.

resources:
  web_server:
    type: OS::Nova::Server
    properties:
      name: { get_param: server_name }
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - network: { get_param: private_network_id }
      security_groups:
        - { get_param: security_group }
      user_data_format: RAW
      user_data:
        str_replace:
          params:
            $app_lbaas_vip: { get_param: app_lbaas_vip }
          template:
                  #!/bin/bash -v
                  yum install -y httpd
                  yum install -y wget
                  cat >> /etc/httpd/conf/httpd.conf << EOL
                  <VirtualHost *:*>
                         ProxyPreserveHost On
                         ProxyPass / http://$app_lbaas_vip/
                         ProxyPassReverse / http://$app_lbaas_vip/
                         ServerName localhost
                  </VirtualHost>
                  EOL
                  systemctl restart httpd

  Pool_Member:
    type: OS::Neutron::PoolMember
    properties:
      pool_id: { get_param: pool_name }
      address: { get_param: [web_server, first_address]}
      protocol_port: 80

outputs:
  web_private_ip:
    description: Private IP address of the Web Instance
    value: { get_attr: [web_server, first_address] }

  lb_member:
    description: LoadBalancer member details
    value: { get_attr: [Pool_Member, show] }

The app tier template creates and installs everything required by the application, allowing it to be consumed and query the data tier. In the example case, the web-tracker application was developed in node.js.

resources:
  app_server:
    type: OS::Nova::Server
    properties:
      name: { get_param: server_name }
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - network: { get_param: private_network_id }
      security_groups:
        - { get_param: security_group }
      user_data_format: RAW
      user_data:
        str_replace:
          template: |
                  #!/bin/bash -v
                  yum groupinstall -y 'Development Tools'
                  yum module install -y nodejs/development
                  yum install -y wget
                  wget http://materials.example.com/webtracker.tar.gz
                  tar -xz webtracker.tar.gz
                  chmod +x webtracker.js
                  ./webtracker.js &

  Pool_Member:
    type: OS::Neutron::PoolMember
    properties:
      pool_id: { get_param: pool_name }
      address: { get_param: [app_server, first_address]}
      protocol_port: 80

outputs:
  app_private_ip:
    description: Private IP address of the App Instance
    value: { get_attr: [app_server, first_address] }

  lb_member:
    description: LoadBalancer member details
    value: { get_attr: [Pool_Member, show] }

The template for the data tier deployment, requires extra parameters to configure the instance:

parameters:
...output omitted...
  db_name:
    type: string
    description: Database name
    default: webtracker
    constraints:
      - length: { min: 1, max: 64 }
        description: db_name must be between 1 and 64 characters
      - allowed_pattern: '[a-zA-Z][a-zA-Z0-9]*'
        description:
              db_name must begin with a letter and contain only
              alphanumeric characters

  db_username:
    type: string
    description: Database admin account username
    default: webtradmin
    hidden: true

  db_password:
    type: string
    description: Database admin account password
    default: redhat
    hidden: true
    constraints:
      - length: { min: 1, max: 41 }
        description: db_password must be between 1 and 41 characters
      - allowed_pattern: '[a-zA-Z0-9]*'
        description: db_password must contain only alphanumeric characters

  db_root_password:
    type: string
    description: Root password for database
    default: redhat
    hidden: true
    constraints:
      - length: { min: 1, max: 41 }
        description: db_password must be between 1 and 41 characters
      - allowed_pattern: '[a-zA-Z0-9]*'
        description: db_root_password must contain only alphanumeric characters

  db_volume_size:
    type: string
    description: Database volume size (in GB) for database files
    default: 2
    hidden: true

resources:
  db_files_volume:
    type: OS::Cinder::Volume
    properties:
      size: { get_param: db_volume_size }
      name: db_files

  db_volume attachment:
    type: OS::Cinder::VolumeAttachment
    properties:
      volume_id: { get_param: db_files_volume }
      instance_uuid: { get_param: db_instance }

  db_instance:
    type: OS::Nova::Server
    properties:
      name: { get_param: server_name }
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - network: { get_param: private_network_id }
      security_groups:
        - { get_param: security_group }
      user_data_format: RAW
      user_data:
        str_replace:
          template:
                  #!/bin/bash -v
                  mkdir /mnt/db_files
                  chown mysql:mysql /mnt/db_files
                  volume_path="/dev/disk/by-id/virtio-$(echo volume_id | cut -c -20)"
                  echo  ${volume_path}
                  mkfs.ext4 ${volume_path}
                  echo "${volume_path} /mnt/db_files ext4 defaults 1 2" >> /etc/fstab
                  mount /mnt/db_files

                  yum -y install mariadb-server mariadb
                  systemctl start mariadb
                  systemctl stop mariadb
                  chown mysql:mysql /mnt/db_files
                  touch /var/log/mariadb/mariadb.log
                  chown mysql:mysql /var/log/mariadb/mariadb.log
                  mv -f /var/lib/mysql /mnt/db_files/
                  sed -i 's/\/var\/lib\/mysql/\/mnt\/db_files\/mysql/' /etc/my.cnf
                  echo "[client]" >> /etc/my.cnf
                  echo "socket=/mnt/db_files/mysql/mysql.sock" >> /etc/my.cnf
                  systemctl start mariadb
                  systemctl enable mariadb

                  mysqladmin -u root password db_rootpassword

                  cat << EOF | mysql -u root --password=db_rootpassword
                  CREATE DATABASE db_name;
                  CREATE USER 'db_user'@'localhost';
                  SET PASSWORD FOR 'db_user'@'localhost'=PASSWORD("db_password");
                  GRANT ALL PRIVILEGES ON db_name.* TO 'db_user'@'localhost' IDENTIFIED BY 'db_password';
                  CREATE USER 'db_user'@'%';
                  SET PASSWORD FOR 'db_user'@'%'=PASSWORD("db_password");
                  GRANT ALL PRIVILEGES ON db_name.* TO 'db_user'@'%' IDENTIFIED BY 'db_password';
                  FLUSH PRIVILEGES;
                  EOF

          params:
            db_rootpassword: { get_param: db_root_password }
            db_name: { get_param: db_name }
            db_user: { get_param: db_user }
            db_password: { get_param: db_password }
            volume_id: { get_resource: db_files_volume }

outputs:
  completition:
    description:
            Database setup is complete, login username and password are
    value:
      str_replace:
        template:
                Database Name=$dbName, Database Admin Acct=$dbAdmin
        params:
          $dbName: { get_param: db_name }
          $dbAdmin: { get_param: db_username }
  instance_ip:
    description: IP address of the deployed database instance
    value: { get_attr: [db_instance, first_address] }

Users can create many application credentials with the same role assignments. This allows for rotation of credentials without breaking the app. Otherwise changing a service user's password will need downtime until the application is fully updated with the new password.

The process for rotating an application credential: