Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing micrometer/prometheus metric collection #105

Merged
merged 1 commit into from
Mar 4, 2019

Conversation

kvncrw
Copy link

@kvncrw kvncrw commented Oct 1, 2018

related to #103 - tested using jvm8/10

@@ -72,17 +72,42 @@ ENV SPRING_PROFILES_ACTIVE docker,mysql
In the `mysql section` of the `application.yml` from the [Configuration repository], you have to change
the host and port of your MySQL JDBC connection string.

## Custom metrics monitoring

@todo Add default custom dashboards to grafana
Copy link
Member

@arey arey Oct 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe in a docs/ folder or a gh-pages branch?
I have also to reupload the Spring Petcilinic screenshot of thr readme.md. The link is broken: https://cloud.githubusercontent.com/assets/838318/19653851/61c1986a-9a16-11e6-8b94-03fd7f775bb3.png

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, screenshots of the default dashboard setup in the meantime until the custom metrics dashboard can be built out and included in a custom image build.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For information, docs/ folder has been introduced in the PR #106


@todo Add default custom dashboards to grafana

Grafana and Prometheus are included in the `docker-compose.yml` configuration, and the public facing applications have been instrumented with [MicroMeter](https://micrometer.io) to collect JVM and custom business metrics.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For development, does the application work if Grafana and Prometheus are not started?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no dependency on those services at runtime. This implementation produces an additional endpoint on the actuator which is scraped by prometheus.

@@ -0,0 +1,32 @@
# my global config
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any configuration to commit in the config git repository; https://github.com/spring-petclinic/spring-petclinic-microservices-config ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch! There is. I'll fork that and create a PR for those changes.

@arey
Copy link
Member

arey commented Oct 14, 2018

@mszarlinski did you have time to review this PR?

Copy link
Collaborator

@mszarlinski mszarlinski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your amazing work 👍 please answer my comments


/**
* Create Owner
*/
@PostMapping
@ResponseStatus(HttpStatus.CREATED)
public void createOwner(@Valid @RequestBody Owner owner) {
registry.counter("create.owner").increment();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about using @Timed? If we use it instead of manually incrementing metrics, tests will be alse simplified. IMHO there is little profit in measuring number of requests - Prometheus has metrics for RPS etc.)

https://spring.io/blog/2018/03/16/micrometer-spring-boot-2-s-new-application-metrics-collector

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I'm delivering the workshop today so once that's done I'll make this change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@notsureifkevin do you any time to make the change?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://micrometer.io/docs/concepts#_the_code_timed_code_annotation

It's not clear to me how to use this annotation and I'm constrained for time as-is. You're welcome to use merge this PR as-is, attempt to make these modifications yourself, or drop it completely. I've rebased atop master, so it should be good to go as-is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mszarlinski I propose to merge this PR and create an issue for improvment. I could work on it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've created the #120 in order to merge this PR then improve it.

@arey
Copy link
Member

arey commented Nov 21, 2018

I've tested the notsureifkevin:prometheusby following the readme instruction. All looks like ok. The Grafana dashboard displays information about the Petclinic microservices:
graphana
I don't know how to display custom metrics (i.e. create.owner) in Grafana? Need a custom dashboard and not the default one https://grafana.com/dashboards/4701?
In Promotheus Graph, I could see the Time Series of the update_owner_totalmetric.

@arey
Copy link
Member

arey commented Nov 21, 2018

In order to easier test all metrics dashboard we have (Promotheus/Grafana/OpenTracing/Hystrix Dashboard), I wonder if you could add some load tests with JMeter or Gatlin.

@kvncrw
Copy link
Author

kvncrw commented Nov 21, 2018

I built a simple load generation script with locust/python. I'm happy to share that. In order to test the create functionality there needs to be an additional delete endpoint as to clean up, otherwise performance issues start occurring (especially at high load).

RE: the other changes, I simply haven't had the time and I apologize. I'll make it a point to free up some cycles for this over the next few weeks.

@arey arey changed the title introducing micrometer/prometheus metric collection Introducing micrometer/prometheus metric collection Nov 24, 2018
@arey
Copy link
Member

arey commented Nov 24, 2018

I don't know Locust but why not. It could be great.
For load testing, we could add delete REST endpoints. What are you think about that @mszarlinski? @notsureifkevin I hope you will find some time to continue your contribution.

@mszarlinski
Copy link
Collaborator

mszarlinski commented Nov 24, 2018

What is a point in making performance tests on demo application? I think that simple Gatling simulation to make a couple of requests would be sufficient. By the way, Grafana dashboard looks pretty sweet!

@arey
Copy link
Member

arey commented Nov 25, 2018

Not really for performance tests, but in order to simulate some user traffic and to have pretty dashboard :)

@mszarlinski
Copy link
Collaborator

I would like to keep it simple. Maybe bash script or simple Gatling simulation would be enough to feed metrics?

@arey
Copy link
Member

arey commented Nov 26, 2018

I'm totally agree with you. We have to keep the project simple. Moreover this script. A single script file and a comment line in the readme.md should be enough. @notsureifkevin coulmd be have a look to your locust script? (trough a gist for instance)

@arey
Copy link
Member

arey commented Feb 15, 2019

Hi @notsureifkevin. Could we finish this PR? A least, some comments from @mszarlinski ?

@kvncrw
Copy link
Author

kvncrw commented Feb 16, 2019

Hi @notsureifkevin. Could we finish this PR? A least, some comments from @mszarlinski ?

Hello, I've commented on the outstanding issue, it appears some additional configuration/changes would be required to make use of his suggestion; I feel the current implementation, while not the most pragmatic, is more than adequate for a usage example.

Of course, if someone else is willing to implement/build upon the changes I've made I'd more than welcome the knowledge and experience.

@kvncrw
Copy link
Author

kvncrw commented Feb 16, 2019

I've tested the notsureifkevin:prometheusby following the readme instruction. All looks like ok. The Grafana dashboard displays information about the Petclinic microservices:
graphana
I don't know how to display custom metrics (i.e. create.owner) in Grafana? Need a custom dashboard and not the default one https://grafana.com/dashboards/4701?
In Promotheus Graph, I could see the Time Series of the update_owner_totalmetric.

I have built a dashboard for this application here: https://gist.githubusercontent.com/notsureifkevin/104adca1ccd4b96937738f9bfbb6ba46/raw/1fddbe7eeebcb57dfb25d0407161710928d9b52f/petclinic.json

@kvncrw
Copy link
Author

kvncrw commented Feb 16, 2019

I'm totally agree with you. We have to keep the project simple. Moreover this script. A single script file and a comment line in the readme.md should be enough. @notsureifkevin coulmd be have a look to your locust script? (trough a gist for instance)

https://gitlab.com/kc_wrhse/spring-petclinic-kubernetes/tree/master/scripts/spc-load

@arey arey merged commit b86d8a0 into spring-petclinic:master Mar 4, 2019
@arey
Copy link
Member

arey commented Mar 8, 2019

@notsureifkevin I have a question. In your custom grafana dashboard, what means the spc namespace ?

expr": "max(http_server_requests_seconds_max{namespace=\"spc\", status!~\"5..\"})

I don't see that attribut in data result:

http_server_requests_seconds_max{exception="None",instance="api-gateway:8080",job="api-gateway",method="GET",status="200",uri="/api/gateway/**"}

@kvncrw
Copy link
Author

kvncrw commented Mar 8, 2019

@arey this is an artifact of the work I was doing in a workshop/lab which included multiple namespaces - I believe you could remove that attribute and it would work as intended

talha-01 pushed a commit to talha-01/microservices-ci-cd-pipeline-with-petclinic-app that referenced this pull request Nov 13, 2020
Introducing Micrometer Prometheus metric collection
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants