sap_hana_install: Inventory based Scale-Out and idempotency validations

[marcelmamula]

Premise

Current workflow of sap_hana_install does not execute any tasks on addhosts, be it for new or existing systems, resulting in inconsistent system. All tasks are executed only on managed node, including important configuration tasks like firewall, selinux, etc.

This is re-imagined alternative to open #1122 which uses Inventory instead of delegate_to tasks, which were causing issue when installing packages on delegated hosts.

Changes

This PR introduces split of role workflow into separate runner files for Install and Addhosts along with range of changes:

Conceptual changes

• New FLAG system of variables to easily detect which host task is running on to differentiate between Main host and Addhosts.
  • Main host is defined as Master host for Scale-Out, but remains ansible_hostname for One Host or HA installations.
• Restructure all pre/post tasks into smaller chunks to improve re-usability.

New features

• Range of validations for hosts and inventory validation to fail before it reaches hdblcm.
• Task to create user sapadm on addhosts, because it is required when hdblcm is running in addhosts mode as it cannot use
  root SSH.
• HANA Version validation for new systems. We do not want to idempotently skip install if wrong version is installed!
• Shared filesystem validation for multi host systems to ensure that filesystems are shared.
• SAP HANA 2.0 SPS08 - LSS support:
  • Assert that shared filesystem for /lss/shared is mounted.
  • Add --lss_user_password= to addhosts hdblcm command as it is required now. It does not accept parameter in config file.

Expected new failure states to fail fast

In order of execution:

1. sap_hana_install_addhosts are defined but they do not contain valid hosts or only Master.
2. sap_hana_install_addhosts contains more or less hosts than hosts in play.
3. Roles was executed with using conditional for host, forcing it to subset of hosts in play.
   • IMPORTANT: This will require change in AP4S to remove when: inventory_hostname_short == (groups['hana_primary'] | first) so we need to align releases!
4. HANA is detected on addhosts without correct hosts, which means addhosts was not run and we cannot rejoin 2 different databases.
5. Addhosts was executed but Master is not installed.
6. Detected HANA version is different to version in input manifest file.
7. /hana/shared is not shared filesystem
8. /lss/shared is not shared filesystem for SPS08

Tested

This PR was tested on SLES 15 SPS 6 on AWS with following tests:

1. New Single host installation
2. New Multi host Scale-Up HA
3. New Multi host Scale-Out
4. Add hosts to Multi host Scale-Out

NOTE

Workflow Sanity failure for devel is caused by newly pushed ansible-core 2.21, because we do not have ignore file for it yet. I will fix this issue after this PR.