Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added chaos monkey setup #177

Merged
merged 3 commits into from
Apr 20, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,8 @@ target/
.idea
*.iml

# Visual Studio Code
.factorypath

# Branch switching
generated/
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,12 @@ The `master` branch uses an Alpine linux with JRE 8 as Docker base. You will f
*NOTE: Under MacOSX or Windows, make sure that the Docker VM has enough memory to run the microservices. The default settings
are usually not enough and make the `docker-compose up` painfully slow.*


## Starting services locally with docker-compose and Java
If you experience issues with running the system via docker-compose you can try running the `./scripts/run_all.sh` script that will start the infrastructure services via docker-compose and all the Java based applications via standard `nohup java -jar ...` command. The logs will be available under `${ROOT}/target/nameoftheapp.log`.

Each of the java based applications is started with the `chaos-monkey` profile in order to interact with Spring Boot Chaos Monkey. You can check out the (README)[scripts/chaos/README.md] for more information about how to use the `./scripts/chaos/call_chaos.sh` helper script to enable assaults.

## Understanding the Spring Petclinic application

[See the presentation of the Spring Petclinic Framework version](http://fr.slideshare.net/AntoineRey/spring-framework-petclinic-sample-application)
Expand Down
6 changes: 6 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@

<spring-boot.version>2.4.2</spring-boot.version>
<spring-cloud.version>2020.0.1</spring-cloud.version>
<chaos-monkey-spring-boot.version>2.3.2</chaos-monkey-spring-boot.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In prevision of next Spring Boot version upgrade, do you know how to handle them the spring boot chaos monkey library?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what they write on their website (https://codecentric.github.io/chaos-monkey-spring-boot/#docs)

  | Chaos Monkey 1.0.1 | Chaos Monkey 1.5.0 | Chaos Monkey 2.0.0 | Chaos Monkey 2.0.1 | Chaos Monkey 2.0.2 | Chaos Monkey 2.1.1 | Chaos Monkey 2.2+
-- | -- | -- | -- | -- | -- | -- | --
Spring Boot 1.5.0 - 1.5.19 | no | yes | no | no | no | no | no
Spring Boot 2.0.0 - 2.0.8 | yes | no | yes | yes | yes | yes | yes
Spring Boot 2.1.0+ | no | no | no | no | yes | yes | yes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok thank you. So there is not one version of Chaos Monkey per version of Spring Boot. It will be easier for upgrades.


<maven-surefire-plugin.version>2.22.0</maven-surefire-plugin.version>

Expand Down Expand Up @@ -62,6 +63,11 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>de.codecentric</groupId>
<artifactId>chaos-monkey-spring-boot</artifactId>
<version>${chaos-monkey-spring-boot.version}</version>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
Expand Down
24 changes: 24 additions & 0 deletions scripts/chaos/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Spring Boot Chaos Monkey Scripts
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I propose to mention this readme and this feature in the root readme file.
Could you also explain how use both shell scripts call_chaos.sh and run_all.sh shell scripts, what they do and how to exploit their results (maybe in grafana ?).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also explain how use both shell scripts call_chaos.sh and run_all.sh shell scripts, what they do and how to exploit their results (maybe in grafana ?).

Sure! Just FYI I had to create run_all.sh which essentially just doesn't use docker to run everything. That's because I wasn't able to run this in docker :) I will update the readme though


You can read more about the possible configuration options [here](https://codecentric.github.io/chaos-monkey-spring-boot/latest/#_properties).

## Scripts

In order to active Spring Boot Chaos Monkey's assault options and component instrumentation, you need to call the project's API. For your convenience we're providing a script that turns on various watchers and attacks. To print out the usage description just call the script without any parameters.

```bash
$ ./scripts/chaos/call_chaos.sh
usage: ./scripts/chaos/call_chaos.sh: <customers|visits|vets> <attacks_enable_exception|attacks_enable_killapplication|attacks_enable_latency|attacks_enable_memory|watcher_enable_component|watcher_enable_controller|watcher_enable_repository|watcher_enable_restcontroller|watcher_enable_service|watcher_disable>
First pick either customers, visits or vets
Then pick what to enable. Order matters!
Example
./scripts/chaos/call_chaos.sh visits attacks_enable_exception watcher_enable_restcontroller
```

The script takes in at minimum 2 parameters. First provides the name of the application for which you want to turn on Chaos Monkey features. The subsequent ones will enable attacks and watchers. The name of the desired feature maps to a json file that gets updated to `http://localhost:${PORT}/actuator/chaosmonkey/assaults` and `http://localhost:${PORT}/actuator/chaosmonkey/watchers` respectively. Example of enabling exception assault via rest controllers for the visits microservice:

```bash
$ ./scripts/chaos/call_chaos.sh visits attacks_enable_exception watcher_enable_restcontroller
```

The default assault configuration is set to fail every 5th request. That means that the first four will work as if Chaos Monkey was be disabled.
6 changes: 6 additions & 0 deletions scripts/chaos/attacks_disable.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"latencyActive": false,
"exceptionsActive": false,
"killApplicationActive": false,
"memoryActive": false
}
4 changes: 4 additions & 0 deletions scripts/chaos/attacks_enable_exception.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"exceptionsActive": true,
"level" : 5
}
3 changes: 3 additions & 0 deletions scripts/chaos/attacks_enable_killapplication.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"killApplicationActive": true
}
5 changes: 5 additions & 0 deletions scripts/chaos/attacks_enable_latency.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"latencyActive": true,
"latencyRangeStart" : 1000,
"latencyRangeEnd" : 10000
}
4 changes: 4 additions & 0 deletions scripts/chaos/attacks_enable_memory.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"memoryActive": true,
"level" : 5
}
49 changes: 49 additions & 0 deletions scripts/chaos/call_chaos.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/usr/bin/env bash

set -o errexit
set -o errtrace
set -o nounset
set -o pipefail

ROOT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

function usage {
echo "usage: $0: <customers|visits|vets> <attacks_enable_exception|attacks_enable_killapplication|attacks_enable_latency|attacks_enable_memory|watcher_enable_component|watcher_enable_controller|watcher_enable_repository|watcher_enable_restcontroller|watcher_enable_service|watcher_disable>"
echo "First pick either customers, visits or vets"
echo "Then pick what to enable. Order matters!"
echo "Example"
echo "./scripts/chaos/call_chaos.sh visits attacks_enable_exception watcher_enable_restcontroller"
exit 1
}

if [[ $# -lt 2 ]]; then
usage
fi

export PORT="${PORT:-}"

while [[ $# > 0 ]]
do
key="$1"
case $1 in
customers)
PORT=8081
;;
visits)
PORT=8082
;;
vets)
PORT=8083
;;
attacks*)
( cd "${ROOT_DIR}" && curl "http://localhost:${PORT}/actuator/chaosmonkey/assaults" -H "Content-Type: application/json" --data @"${1}".json --fail )
;;
watcher*)
( cd "${ROOT_DIR}" && curl "http://localhost:${PORT}/actuator/chaosmonkey/watchers" -H "Content-Type: application/json" --data @"${1}".json --fail )
;;
*)
usage
;;
esac
shift
done
7 changes: 7 additions & 0 deletions scripts/chaos/watcher_disable.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"controller": false,
"restController": false,
"service": false,
"repository": false,
"component": false
}
3 changes: 3 additions & 0 deletions scripts/chaos/watcher_enable_component.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"component": true
}
3 changes: 3 additions & 0 deletions scripts/chaos/watcher_enable_controller.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"controller": true
}
3 changes: 3 additions & 0 deletions scripts/chaos/watcher_enable_repository.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"repository": true
}
3 changes: 3 additions & 0 deletions scripts/chaos/watcher_enable_restcontroller.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"restController": true
}
3 changes: 3 additions & 0 deletions scripts/chaos/watcher_enable_service.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"service": true
}
29 changes: 29 additions & 0 deletions scripts/run_all.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/usr/bin/env bash

set -o errexit
set -o errtrace
set -o nounset
set -o pipefail

pkill -9 -f spring-petclinic || echo "Failed to kill any apps"

docker-compose kill || echo "No docker containers are running"

echo "Running infra"
docker-compose up -d grafana-server prometheus-server tracing-server

echo "Running apps"
mkdir -p target
nohup java -jar spring-petclinic-config-server/target/*.jar --server.port=8888 --spring.profiles.active=chaos-monkey > target/config-server.log 2>&1 &
echo "Waiting for config server to start"
sleep 20
nohup java -jar spring-petclinic-discovery-server/target/*.jar --server.port=8761 --spring.profiles.active=chaos-monkey > target/discovery-server.log 2>&1 &
echo "Waiting for discovery server to start"
sleep 20
nohup java -jar spring-petclinic-customers-service/target/*.jar --server.port=8081 --spring.profiles.active=chaos-monkey > target/customers-service.log 2>&1 &
nohup java -jar spring-petclinic-visits-service/target/*.jar --server.port=8082 --spring.profiles.active=chaos-monkey > target/visits-service.log 2>&1 &
nohup java -jar spring-petclinic-vets-service/target/*.jar --server.port=8083 --spring.profiles.active=chaos-monkey > target/vets-service.log 2>&1 &
nohup java -jar spring-petclinic-api-gateway/target/*.jar --server.port=8080 --spring.profiles.active=chaos-monkey > target/gateway-service.log 2>&1 &
nohup java -jar spring-petclinic-admin-server/target/*.jar --server.port=9090 --spring.profiles.active=chaos-monkey > target/admin-server.log 2>&1 &
echo "Waiting for apps to start"
sleep 60
Loading