Akka.Remote Failure Injection Commands

The Akka.Remote commands supported by Petabridge.Cmd.Host can be added to your Petabridge.Cmd.Host installation by installing the Petabridge.Cmd.Remote.FailureInjection NuGet package:

PM> Install-Package Petabridge.Cmd.Remote.FailureInjection 

Important

You will need to turn on the failure injection adapters inside your HOCON configuration:

akka.remote.dot-netty.tcp.applied-adapters = [trttl, gremlin]

And then inside your Akka.NET application, you can simply register the FailureInjectionCommands command palette with your ActorSystem using the following syntax:

var cmd = PetabridgeCmd.Get(aNode);
cmd.RegisterCommandPalette(FailureInjectionCommands.Instance);
cmd.Start();

Working with Petabridge.Cmd.Remote.FailureInjection

To get the best use of the remote commands you will need to setup the host on the node you would like to monitor. To be able to use these commands your node will need to be connected to other nodes using Akka.Remote or Akka.Cluster.

Available Commands

Command Name	Description
`inject-failure kill-process`	Forcibly kill the process where the ActorSystem is running in.
`inject-failure throttle`	Throttle network transfer rate to another cluster node.
`inject-failure gremlin`	Inject network failure in forms of missing network packages to other cluster nodes.
`inject-failure disassociate`	Inject a disassociation event against the target address.
`inject-failure quarantine`	Quarantine the target address from this node.

inject-failure kill-process

inject-failure kill-process can be used to simulate a crashing ActorSystem process by forcibly killing the process.

inject-failure kill-process doesn't take any command line arguments.

Examples

pbm inject-failure kill-process

inject-failure throttle

inject-failure throttle can be used to simulate network degradation by limiting the network transfer rate between one node to another.

inject-failure throttle takes the following arguments:

Argument Name	Switches	Mandatory?	Allow Multiple?	Is Flag?	Description
target	-t or -T	yes	yes	no	The remote host node address that network transfer is being throttled.
direction	-d or -D	no	no	no	The direction of the failure. Valid values: send, receive, both. Default: both
rate	-r or -R	yes	no	no	The transfer rate the network is being limited to, in megabits per second. Value less than 0.0 will remove the throttle. Value equals to 0.0 will practically disable the network connection between this node and the target node.

 

Examples

Throttle the network transfer rate to 100 kilobits per second

pbm inject-failure throttle -t akka.trttl.gremlin.tcp://[email protected]:2552 -r 0.1

Disconnect network between this node to the target node

pbm inject-failure throttle -t akka.trttl.gremlin.tcp://[email protected]:2552 -r 0

inject-failure gremlin

inject-failure gremlin can be used to simulate network degradation by dropping Akka.NET messages coming into or out of an ActorSystem that is intended for a certain target address.

inject-failure gremlin takes the following arguments:

Argument Name	Switches	Mandatory?	Allow Multiple?	Is Flag?	Description
target	-t or -T	yes	yes	no	The remote host node address where messages are being dropped.
inbound	-i or -I	no	no	no	Set the percent chance of inbound messages being dropped. Valid values: 0.0-1.0 Default: 0.0
outbound	-o or -O	no	no	no	Set the percent chance of outbound messages being dropped. Valid values: 0.0-1.0 Default: 0.0

 

Examples

Simulate network disruption where 50% of all ingoing and outgoing messages are missing

pbm inject-failure gremlin -t akka.trttl.gremlin.tcp://[email protected]:2552 -i 0.5 -o 0.5

inject-failure disassociate

inject-failure disassociate can be used to inject a disassociation of remote addresses event into the ActorSystem

inject-failure disassociate takes the following arguments:

Argument Name	Switches	Mandatory?	Allow Multiple?	Is Flag?	Description
target	-t or -T	yes	yes	no	The remote address that will be disassociated.
reason	-r or -R	no	no	no	Set the disassociation reason. Valid values: unknown, shutdown, quarantined Default: quarantined

 

Examples

Cause the target address to be disassociated

pbm inject-failure disassociate -t akka.trttl.gremlin.tcp://[email protected]:2552

inject-failure quarantine

inject-failure quarantine can be used to cause a remote addresses to be quarantined.

inject-failure quarantine takes the following argument:

Argument Name	Switches	Mandatory?	Allow Multiple?	Is Flag?	Description
target	-t or -T	yes	yes	no	The remote address that will be quarantined.

 

Examples

Cause the target address to be quarantined

pbm inject-failure quarantine -t akka.trttl.gremlin.tcp://[email protected]:2552