Day: December 16, 2017

Big thanks to Fabian Herschel and Peter Schinagl from SUSE for proof-reading the blog.

First part of this blog is located here:

* Be Prepared for Using Pacemaker Cluster for SAP HANA – Part 1: Basics

Be Prepared for Using Pacemaker Cluster for SAP HANA – Part 2: Failure of Both Nodes

What happens when both cluster nodes will fail

Let’s start with Pacemaker Cluster running during normal operation. Colors are having following meaning – component is available (green), standby (yellow) or unavailable (red).

Figure 2 – Pacemaker Cluster during normal operation

SAP HANA on server hana43 is primary – this is visible by status PROMOTED, roles string containing “P”, sync state set to PRIM and LPT timestamp set.

SAP HANA on server hana44 is secondary – this is visible by status DEMOTED, roles string containing “S”, sync state set to SOK (meaning replication is healthy) and low “static” LPT value.

Now let’s assume that primary server will crash and is unavailable.

Figure 3 – Pacemaker Cluster after primary server failed

Secondary server will notice that primary server is down and since replication status is SOK it is able to initiate failover. Fencing operation will be executed to ensure hana43 server is offline and failover will be executed.

Figure 4 – Pacemaker Cluster after secondary server failover

When SAP HANA on server hana44 is promoted to new primary all cluster node attributes are updated –status is set to PROMOTED, roles string changed to contain “P” as database failover was executed, sync state was set to PRIM and LPT timestamp was set.

Note that cluster node attributes on primary server remain unchanged – this is because that server is unavailable. These attributes are stored locally on that server.

Now let’s assume that secondary server crashed while primary is still offline.

Figure 5 – Pacemaker Cluster after secondary server failed

Now both servers are offline but SAP HANA databases on both servers are configured to run as primary – this is also confirmed by cluster node attributes set that way.

How to (not) destroy your company data

Now imagine following scenario. You are on-duty System Administrator and you got called in the middle of night that SAP HANA is down and you need fix it. Still half-asleep you are logging to the servers listening to your boss explaining you over the phone how much money your company is losing every minute SAP HANA is down and how important it is to get SAP HANA up and running as fast as possible.

When you are finally there you see that both SAP HANA cluster servers are offline. To make it more obvious – this is what you see.

Figure 6 – Pacemaker Cluster state as System Administrator can see it

Since you are under pressure to fix SAP HANA ASAP you decide to fix primary first – after all secondary can be fixed later once we are back in business.

DANGER!!! Now you should stop and get fully awake. How do we know which server is “last primary”? Unless you got that information from some external monitoring system you have no way of knowing!!!

Let’s look what would happen if you would start wrong server. Let’s assume System Administrator checks the documentation and sees there that hana43 is supposed to be primary and decides to start it up first.

Figure 7 – Pacemaker Cluster after hana43 server was restarted

Once Operating System is rebooted System Administrator starts the Pacemaker Cluster. Since secondary server is still offline Pacemaker Cluster will be unable to retrieve LPT value from cluster node attributes of secondary server. Without LPT values from both server nodes Pacemaker Cluster will not start SAP HANA database and cluster node attribute status will be set to WAITING.

This protection from Pacemaker Cluster is called “restart inhibit”. It is there to ensure that SAP HANA is started only in case that Pacemaker Cluster can clearly determine which server is “last primary”.

DANGER!!! At this point System Administrator should stop and start thinking.

Let’s assume that our System Administrator is still half-sleep and he will be surprised why SAP HANA is still down and will start it manually.

Figure 8 – Pacemaker Cluster after SAP HANA on server hana43 was manually started

Once SAP HANA is started manually Pacemaker Cluster will detect it and will adjust cluster node attribute status to PROMOTED.

From the moment when SAP HANA was started all database updates are stored in SAP HANA database running on server hana43. However SAP HANA database running on server hana43 is not having all the data that was persisted after failover to database on server hana44 – see Figure 4.

By manually starting SAP HANA database our System Administrator caused logical inconsistency that will take incredible effort to fix.

Correct approach

Let’s see what is the correct approach how to deal with situations when both servers are offline.

If you are unable to clearly determine which server was “last primary” then you need to start both cluster nodes. Without access to both servers Pacemaker Cluster is unable to correctly determine which server was running “last primary” SAP HANA database.

Figure 9 – Pacemaker Cluster after both servers are restarted

Once attributes from both cluster nodes are available the Pacemaker Cluster will check which node is having higher LPT value to decide which database was “last primary”. Unfortunately this information is not written to SBD drive or anywhere else outside the local node so both nodes must be available for correct determination of “last primary” SAP HANA database.

Alternative approach can be used only in case that you are 100% sure which SAP HANA was “last primary” – maybe getting this information from some external monitoring system. In such case you can do exactly as described in previous section however you perform described steps on correct server.

First you restart “last primary” server (hana44), start Pacemaker and then manually start SAP HANA that will become primary database. Make sure to execute these steps on correct server.

Later you can restart the other server (hana43), start Pacemaker, register local SAP HANA as new secondary and cleanup resource to start SAP HANA as secondary database.

Please note that there is no protection that will prevent you from manually starting wrong SAP HANA database causing data loss or logical inconsistency. It is responsibility of System Administrator to start both nodes at the same time or correctly determine which database was “last primary”. http://bit.ly/2BjJyD5 #SAP #SAPCloud #AI

Some things are vintage and some things are just old. While we were digging through the archives, we found one of our older videos and decided to dust it off and give it an update. Who says something old can’t be new again? And speaking of new, if you’re starting to learn about streaming analytics and trying to figure out how to create, run, or test a project in studio, this is the perfect video for you.

Ready for more good news? Silly question, I know. That’s like asking if you want the last slice of pizza. Anyways, the good news is that this video is part of a tutorial series. So, after you’ve learned the basics of creating and working with projects in studio, you can then learn how to use cockpit to monitor those projects. The bad news? You’ll have to wait for the end of November for the second video to come out. You know the saying: good things come to those who wait!

On the bright side, this blog does offer a sneak-peek of the second video. Here are the highlights of each video. Happy reading!

Part 1: Studio – Creating, Running, and Testing a Project

In part 1, we:

* create a project that filters data and writes it to HANA,
* compile and execute the project, and
* test the project by manually loading data to the input stream and checking the results.

Here’s a screengrab from the video demo showing us setting a filter:

Only values greater than 20 in the input stream will show in SAP HANA. After setting the filter, we add an SAP HANA Output adapter. Here we are configuring the adapter to write to SAP HANA:

We’re using the ‘hanadb’ service, connecting to the ‘SYSTEM’ schema, and writing to the target table in SAP HANA, ‘TABLE_TEST’.

After running the project, we test it by manually loading two rows into the input stream: the first with a value below 20 (9), and the second with a value above 20 (21):

Because of the filter, only the second row we added shows in SAP HANA:

That’s the gist of the first video tutorial. Check it out for yourself here. Once you’ve completed the studio tutorial, you’re ready to start monitoring streaming in cockpit! (or you will be once we post the video later this month. Stay tuned!)

Part 2: Cockpit – Monitoring Streaming Analytics

In part 2, we monitor:

* streaming analytics and general system behavior, using the Monitoring and Administration section, and
* the streaming project we created in part 1, via the Streaming Analytics

Here’s a screengrab from the video showing the Monitoring and Administration section:

It shows the behavior of the resource you’re connected to, including:

* overall database status,
* number of alerts, and
* memory, CPU, and disk usage.

If you select Show all in the Alerts tile, you can view and configure alerts. In the video, we look at the Inactive Streaming applications alert:

We show you how to:

* set thresholds for prioritized alerting,
* add email recipients, and
* trigger alert checkers.

After monitoring system behavior, next, we monitor the project we created in the previous video, ‘testproject’. We look at the System, Network, Streams, and Adapters tabs. In the Streams tab, you can view QueueDepth and RowsInStore. Here’s a screengrab from the video:

PriceFeed and VWAP have 99 and 26 records in their log store, respectively. If you select a stream, you can see more detailed info, including rows/transaction throughput history.

Finally, in the Adapters tab, you can view the adapters in the project and their status:

And that’s all she wrote!

So, whether you’re a newbie looking to get your feet wet or a wily veteran just looking for a refresher, be sure to check out our video tutorial series on studio and cockpit. Part 1 is already available, and part 2 will be out very soon. Also, if you have any ideas for more videos we could make, let us know in the comments below!

For more on managing projects in studio and monitoring streaming in cockpit, check out:

* the Working with Projects in the Streaming Plugin for SAP HANA Studio section of the Steaming Analytics Developer Guide, and
* the Managing Streaming Analytics with SAP HANA Cockpit section of the Streaming Analytics Configuration and Administration Guide, as well as the SAP HANA Cockpit Administration Guide. http://bit.ly/2Bk1uxy #SAP #SAPCloud #AI

Be Prepared for Using Pacemaker Cluster for SAP HANA – Main Part

Introduction

I am probably stating obvious when saying that every infrastructure deployment option needs to be properly tested before it can be used to host productive workloads. This is even more important for High Availability clusters, as poorly implemented cluster can cause more downtime than decision to not use any clustering at all. Worst situation that must not happen under any circumstances is that cluster would cause impact to the data consistency or data loss.

Typical High Availability cluster testing starts with situations that could happen under normal operation – these include situations like:

* Cluster stability
* Graceful failover
* Crash of primary application or server
* Crash of secondary application or server
* Patching and maintenance, etc.

These scenarios must behave as expected otherwise you better not use the cluster at all.

Next you should focus on testing more advanced scenarios – these typically include multi-level failures. These are happening in very rare cases and goal here is not to ensure that cluster will be able to protect against these failures, but to ensure that cluster will not misbehave and will not cause any damage to database.

Highest level of testing is to consider what human System Administrator will do when he is called in the middle of night to fix failed cluster. Can he accidentally cause data loss and/or inconsistency to SAP HANA database? What are protections that prevent him from inadvertently damaging SAP HANA database?

This blog is about such scenarios where System Administrator must be extremely careful as otherwise he can accidentally cause data loss and inconsistency to SAP HANA database.

Before we jump to the scenarios themselves we need to set the stage and explain some basics.

Big thanks to Fabian Herschel and Peter Schinagl from SUSE for proof-reading the blog.

For better readability whole blog is divided into following parts:

* Be Prepared for Using Pacemaker Cluster for SAP HANA – Main Part (this blog)
* Be Prepared for Using Pacemaker Cluster for SAP HANA – Part 1: Basics (this blog)
* Be Prepared for Using Pacemaker Cluster for SAP HANA – Part 2: Failure of Both Nodes

Be Prepared for Using Pacemaker Cluster for SAP HANA – Part 1: Basics

How Pacemaker Cluster works with SAP HANA System Replication

SUSE developed in collaboration with SAP the SAPHanaSR solution and released it as part of SLES for SAP Applications. This solution is based on Pacemaker Cluster that is automating failovers between two SAP HANA databases that are mirroring each other. This solution was later adopted by RedHat and is now jointly developed by both companies. Therefore, this whole blog is equally applicable to both Operating Systems.

Pacemaker Cluster with SAP HANA System Replication as visualized below is based on two identical servers (VMs) each having one SAP HANA database. Both servers are bundled together by SUSE Pacemaker Cluster.

Figure 1 – Pacemaker Cluster for SAP HANA Architecture

SAP HANA database on primary server is replicating information to SAP HANA database running on secondary server. Replication method is based on Synchronous SAP HANA System Replication – this is to ensure that no data is lost during failover. Both databases are running at the same time, however only primary database can support customer workloads. Secondary database is either completely passive or can be active in read-only mode (since SAP HANA 2.0).

Failure of primary SAP HANA database is automatically detected by Pacemaker Cluster. The cluster will automatically shutdown primary database (if still partially running) and will activate secondary database. It will also relocate virtual IP to ensure that all applications using the database can reconnect to new primary SAP HANA database. Since all the data is already pre-loaded in memory of new primary database this failover is very fast.

More details here:

https://www.suse.com/products/sles-for-sap/resource-library/sap-best-practices

https://access.redhat.com/articles/1466063

Importance of fencing

Under normal operation fencing mechanism is not actively used. Cluster is communicating over network (corosync) and both sides of the cluster are constantly updating each other on the health status of SAP HANA database on given node.

The problem starts when one of nodes stops responding. Let’s assume secondary server is suddenly unable to connect to primary server. In such case the cluster on secondary server is having no way of knowing what happened – generally two options are possible:

* Primary server is not responding because it crashed or is frozen
* Primary server is healthy however due to a network issue it is not reachable

The problem is that in first case cluster should consider executing failover to restore the service while in second case the failover must not happen as otherwise SAP HANA would be active on both servers.

This situation is called split-brain (or dual primary) and is extremely dangerous. It is even more dangerous in this scenario because we are working with two independent SAP HANA databases that can easily be active on both sides.

Business impact would fatal – imagine that you are writing some transactions to database running on primary server and later other transactions to database running on secondary server which is not aware about changes written to the first database.

Now imagine what about other systems in landscape – CRM having records that does not exist in ERP, etc. This would result in logical inconsistency cascading across all systems in the customer landscape. I believe it is now obvious that fixing such situation would be very difficult and would cause huge impact on the business.

It is good to be paranoid when it comes to split-brain situations.

Pacemaker Cluster is addressing this by fencing technique called STONITH (Shoot-the-other-node-in-the-head). This mechanism does exactly what the name suggests. In case that nodes suddenly cannot communicate then one of the nodes will kill the other node to ensure that both nodes are not active at the same time. Surviving node will then serve the customer workloads.

More details here:

http://linux-ha.org/wiki/STONITH

There are multiple techniques how fencing can be implemented. However, following two techniques are most common:

* Node shutdown via IPMI (for most Intel devices), HMC (for Power devices), vCenter or VMware plugins (for VMware VMs)

In case of issue surviving node will power down the other node to ensure that it is not active. Main drawback is that implementation depends on used HW or VMware configuration. In some cases this approach might be considered insecure due to a password being stored in cluster configuration in unencrypted way.

* SBD (Storage-based-death) disk fencing is based on shared disk(s) provided from external source(s) – obviously, multiple SBD disks should not share same single-point-of-failure and if provided over network then not over cluster communication network (corosync).

In case of issue surviving node will write “poison pill” to the disk instructing other node (if active) to commit suicide. Advantage of this approach is that it is generic approach that can be equally applied across different scenarios including bare-metal and virtual solutions.

More details here:

http://clusterlabs.org/doc/crm_fencing.html

http://www.linux-ha.org/wiki/SBD_Fencing

https://www.suse.com/documentation/sle-ha-12/book_sleha/data/sec_ha_fencing_nodes.html

How does cluster know it is safe to failover

At this point we need to deep dive into how Pacemaker Cluster internally works with SAP HANA System Replication.

There are two SAP HANA cluster packages that are automating SAP HANA System Replication failover:

* SAPHanaSR – automating failover for following two SAP HANA single-node scenarios:

* SAP HANA SR performance optimized infrastructure – where secondary node is dedicated to fulfilling High Availability function
* SAP HANA SR cost optimized infrastructure – where secondary node is hosting additional non-productive SAP HANA database

* SAPHanaSR-ScaleOut – automating failover for SAP HANA scale-out scenario (at the time of writing of this blog available only on SUSE Linux Enterprise Server for SAP Applications)

For the sake of simplicity we will focus on single node package (SAPHanaSR) only. This package is designed to monitor and locally record multiple attributes for each node:

* Cluster Resource State (hana__clone_state)

Valid values: PROMOTED, DEMOTED, WAITING, UNDEFINED
This attribute is describing actual status of local SAP HANA cluster resource.

* Remote Node Hostname (hana__remoteHost)

Hostname of remote server (“the other node”).

* SAP HANA Roles (hana__roles)

String describing health status of local SAP HANA database. This includes:

* Return Code from landscapeHostConfiguration.py
* HANA role – primary/secondary
* Nameserver role
* Index server roles

* SAP HANA Site Name (hana__site)

Alias of local SAP HANA database (as registered when replication was configured).

* SAP HANA System Replication mode (hana__srmode)

Valid values: sync, syncmem
Configured SAP HANA replication mode. Replication mode async should not be used in High Availability scenario as it is associated with potential data loss during failover.

* SAP HANA System Replication status (hana__sync_state)

Valid values: PRIM, SOK, SFAIL
Failover to secondary cluster node can happen only in case that replication status on secondary node is SOK as otherwise replication was not operational when primary crashed and data on secondary database is not in sync with primary database. In this case failover will not happen.

* Local Node Hostname (hana__vhost)

Hostname used during SAP HANA installation – this could be either the local hostname any other “virtual” hostname.

* Last Primary Timestamp – LPT value (lpa__lpt)

Value is either timestamp value of SAP HANA database being last seen as primary or low “static” value suggesting that database is not primary.
This attribute is preventing dual primary situation. In case that cluster node attributes on both nodes are showing last state of SAP HANA database as primary (this can happen after multi-level failure – see next part for details) then higher LPT value is used to determine which SAP HANA database was “last primary”. This database is started while the other database is kept down.

* Resource Weight (master-rsc_SAPHana__HDB)

Internal technical cluster attribute used to control failover process. Node with highest weight will be promoted to become primary. It is calculated based on other attributes.

These attributes are updated at regular intervals and stored locally as part cluster node status.

Example of internal cluster states during normal operation (hana43 being primary): Node Attributes: * Node hana43-hb: + hana_hac_clone_state : PROMOTED + hana_hac_remoteHost : hana44 + hana_hac_roles : 4:P:master1:master:worker:master + hana_hac_site : TOR-HAC-00-NODE1 + hana_hac_srmode : sync + hana_hac_sync_state : PRIM + hana_hac_vhost : hana43 + lpa_hac_lpt : 1439227830 + master-rsc_SAPHana_HAC_HDB00 : 150 * Node hana44-hb: + hana_hac_clone_state : DEMOTED + hana_hac_remoteHost : hana43 + hana_hac_roles : 4:S:master1:master:worker:master + hana_hac_site : TOR-HAC-00-NODE2 + hana_hac_srmode : sync + hana_hac_sync_state : SOK + hana_hac_vhost : hana44 + lpa_hac_lpt : 30 + master-rsc_SAPHana_HAC_HDB00 : 100

All these attributes are used to determine actual cluster health and are considered before failover decision is taken. http://bit.ly/2ok6tey #SAP #SAPCloud #AI

The Internet of Things (IoT) offers endless opportunities for enterprises to make their manufacturing, logistics, and services processes smart. So where exactly will they see the benefits of embracing new technologies?

We look at some compelling examples from the SAP environment, where IoT is already an everyday reality in many lines of business and where the latest technology trends are helping companies optimize their packing process, their warehousing, their flow of materials, and much more.

1. Streamlining the material flow in manufacturing and logistics

When we think of Industry 4.0, we think of sensors. Their many uses include helping manufacturers track work pieces in real time and automating the flow of materials from the warehouse. Work pieces are fitted with RFID tags that allow sensors to transmit each component’s position to the relevant SAP software in real time. This allows human operators to refer directly to the SAP system to find out where and when orders are ready for processing ‒ thus making optimal use of time lags in the production process.

2. Enabling autonomous machines

Real-time interaction between machines and backend systems allows efficient make-to-order production. Manufacturers can set up interfaces in an SAP system to enable communication between external systems and almost any machine. Thus, a machine can independently download and execute a “recipe” for mixing individual components or ingredients. While the machine is running, it generates data, which it transmits back into the system. This information provides the foundation for further automation and simplification.

3. Optimizing high-rack warehouses

Operators of high-rack warehouses can create a digital twin to optimize their collaboration with manufacturers and service partners. The digital twin of the warehouse is mapped in the SAP system and shows the condition and position of the assets within it. This makes it possible, for example, to check and monitor the condition of the high-rack aisles. Manufacturers, service providers, and plant operators can all access the relevant digital files via the SAP Asset Intelligence Network portal and optionally store items such as operating manuals and certificates there. When they receive a service request, high-rack storage system operators and service technicians can open a ticket in the system, communicate with one another, and access all the data they need on the go.

4. Establishing new maintenance processes

Service technicians are often responsible for a large number of machines distributed across a wide area. For them, augmented reality smart glasses such as Microsoft’s ”HoloLens” can take service and maintenance processes to a completely new level. Teamed with SAP S/4HANA, SAP Cloud Platform , and a 3D engine, smart glasses can optimize maintenance tasks by ensuring that service technicians have all the information they need ‒ quite literally ‒ right in front of their eyes. They are also empowered to examine the inside of the machine before physically touching it. And, if they have questions, they can communicate with their colleagues in the back office via the HoloLens.

5. Identifying potential hazards early

The cloud-based IoT network SAP Asset Intelligence Network links up manufacturers, operators, and service partners. If a sensor reports an unusually high concentration of dust in a machine room, the machine manufacturer can decide from hundreds of miles away whether or not it needs to send a field technician to the site. Sensors are also the key to predictive maintenance because they transmit information that is stored in the cloud as a basis for detecting possible issues before they cause trouble.

Whether in manufacturing, logistics, or service, IoT-based solutions offer a wealth of opportunities for optimizing processes and making them more efficient. http://bit.ly/2otyEbh #SAP #SAPCloud #AI