Introducing micrometer/prometheus metric collection #105

kvncrw · 2018-10-01T16:56:41Z

related to #103 - tested using jvm8/10

arey · 2018-10-03T07:38:46Z

README.md

@@ -72,17 +72,42 @@ ENV SPRING_PROFILES_ACTIVE docker,mysql
 In the `mysql section` of the `application.yml` from the [Configuration repository], you have to change 
 the host and port of your MySQL JDBC connection string. 

+## Custom metrics monitoring
+
+@todo Add default custom dashboards to grafana


maybe in a docs/ folder or a gh-pages branch?
I have also to reupload the Spring Petcilinic screenshot of thr readme.md. The link is broken: https://cloud.githubusercontent.com/assets/838318/19653851/61c1986a-9a16-11e6-8b94-03fd7f775bb3.png

I agree, screenshots of the default dashboard setup in the meantime until the custom metrics dashboard can be built out and included in a custom image build.

For information, docs/ folder has been introduced in the PR #106

arey · 2018-10-03T07:42:58Z

README.md

+
+@todo Add default custom dashboards to grafana
+
+Grafana and Prometheus are included in the `docker-compose.yml` configuration, and the public facing applications have been instrumented with [MicroMeter](https://micrometer.io) to collect JVM and custom business metrics.


For development, does the application work if Grafana and Prometheus are not started?

There is no dependency on those services at runtime. This implementation produces an additional endpoint on the actuator which is scraped by prometheus.

arey · 2018-10-03T07:48:20Z

docker/prometheus/prometheus.yml

@@ -0,0 +1,32 @@
+# my global config


Is there any configuration to commit in the config git repository; https://github.com/spring-petclinic/spring-petclinic-microservices-config ?

Nice catch! There is. I'll fork that and create a PR for those changes.

arey · 2018-10-14T17:59:03Z

@mszarlinski did you have time to review this PR?

mszarlinski

Thanks for your amazing work 👍 please answer my comments

mszarlinski · 2018-10-15T06:28:14Z

...service/src/main/java/org/springframework/samples/petclinic/customers/web/OwnerResource.java


    /**
     * Create Owner
     */
    @PostMapping
    @ResponseStatus(HttpStatus.CREATED)
    public void createOwner(@Valid @RequestBody Owner owner) {
+        registry.counter("create.owner").increment();


What about using @Timed? If we use it instead of manually incrementing metrics, tests will be alse simplified. IMHO there is little profit in measuring number of requests - Prometheus has metrics for RPS etc.)

https://spring.io/blog/2018/03/16/micrometer-spring-boot-2-s-new-application-metrics-collector

Agreed. I'm delivering the workshop today so once that's done I'll make this change.

@notsureifkevin do you any time to make the change?

https://micrometer.io/docs/concepts#_the_code_timed_code_annotation

It's not clear to me how to use this annotation and I'm constrained for time as-is. You're welcome to use merge this PR as-is, attempt to make these modifications yourself, or drop it completely. I've rebased atop master, so it should be good to go as-is.

@mszarlinski I propose to merge this PR and create an issue for improvment. I could work on it.

I've created the #120 in order to merge this PR then improve it.

arey · 2018-11-21T07:24:13Z

I've tested the notsureifkevin:prometheusby following the readme instruction. All looks like ok. The Grafana dashboard displays information about the Petclinic microservices:

I don't know how to display custom metrics (i.e. create.owner) in Grafana? Need a custom dashboard and not the default one https://grafana.com/dashboards/4701?
In Promotheus Graph, I could see the Time Series of the update_owner_totalmetric.

arey · 2018-11-21T07:37:46Z

In order to easier test all metrics dashboard we have (Promotheus/Grafana/OpenTracing/Hystrix Dashboard), I wonder if you could add some load tests with JMeter or Gatlin.

kvncrw · 2018-11-21T13:04:25Z

I built a simple load generation script with locust/python. I'm happy to share that. In order to test the create functionality there needs to be an additional delete endpoint as to clean up, otherwise performance issues start occurring (especially at high load).

RE: the other changes, I simply haven't had the time and I apologize. I'll make it a point to free up some cycles for this over the next few weeks.

arey · 2018-11-24T10:12:55Z

I don't know Locust but why not. It could be great.
For load testing, we could add delete REST endpoints. What are you think about that @mszarlinski? @notsureifkevin I hope you will find some time to continue your contribution.

mszarlinski · 2018-11-24T16:50:46Z

What is a point in making performance tests on demo application? I think that simple Gatling simulation to make a couple of requests would be sufficient. By the way, Grafana dashboard looks pretty sweet!

arey · 2018-11-25T12:38:58Z

Not really for performance tests, but in order to simulate some user traffic and to have pretty dashboard :)

mszarlinski · 2018-11-25T18:03:25Z

I would like to keep it simple. Maybe bash script or simple Gatling simulation would be enough to feed metrics?

arey · 2018-11-26T15:47:50Z

I'm totally agree with you. We have to keep the project simple. Moreover this script. A single script file and a comment line in the readme.md should be enough. @notsureifkevin coulmd be have a look to your locust script? (trough a gist for instance)

arey · 2019-02-15T11:58:48Z

Hi @notsureifkevin. Could we finish this PR? A least, some comments from @mszarlinski ?

kvncrw · 2019-02-16T15:38:46Z

Hi @notsureifkevin. Could we finish this PR? A least, some comments from @mszarlinski ?

Hello, I've commented on the outstanding issue, it appears some additional configuration/changes would be required to make use of his suggestion; I feel the current implementation, while not the most pragmatic, is more than adequate for a usage example.

Of course, if someone else is willing to implement/build upon the changes I've made I'd more than welcome the knowledge and experience.

kvncrw · 2019-02-16T15:44:11Z

I've tested the notsureifkevin:prometheusby following the readme instruction. All looks like ok. The Grafana dashboard displays information about the Petclinic microservices:

I don't know how to display custom metrics (i.e. create.owner) in Grafana? Need a custom dashboard and not the default one https://grafana.com/dashboards/4701?
In Promotheus Graph, I could see the Time Series of the update_owner_totalmetric.

I have built a dashboard for this application here: https://gist.githubusercontent.com/notsureifkevin/104adca1ccd4b96937738f9bfbb6ba46/raw/1fddbe7eeebcb57dfb25d0407161710928d9b52f/petclinic.json

kvncrw · 2019-02-16T15:44:51Z

I'm totally agree with you. We have to keep the project simple. Moreover this script. A single script file and a comment line in the readme.md should be enough. @notsureifkevin coulmd be have a look to your locust script? (trough a gist for instance)

https://gitlab.com/kc_wrhse/spring-petclinic-kubernetes/tree/master/scripts/spc-load

arey · 2019-03-08T07:15:16Z

@notsureifkevin I have a question. In your custom grafana dashboard, what means the spc namespace ?

expr": "max(http_server_requests_seconds_max{namespace=\"spc\", status!~\"5..\"})

I don't see that attribut in data result:

http_server_requests_seconds_max{exception="None",instance="api-gateway:8080",job="api-gateway",method="GET",status="200",uri="/api/gateway/**"}

kvncrw · 2019-03-08T12:41:56Z

@arey this is an artifact of the work I was doing in a workshop/lab which included multiple namespaces - I believe you could remove that attribute and it would work as intended

Introducing Micrometer Prometheus metric collection

arey added the enhancement label Oct 1, 2018

arey reviewed Oct 3, 2018

View reviewed changes

kvncrw mentioned this pull request Oct 8, 2018

added configuration for metrics exporter spring-petclinic/spring-petclinic-microservices-config#6

Merged

mszarlinski requested changes Oct 15, 2018

View reviewed changes

arey changed the title ~~introducing micrometer/prometheus metric collection~~ Introducing micrometer/prometheus metric collection Nov 24, 2018

introducing micrometer/prometheus metric collection

91b6551

kvncrw force-pushed the prometheus branch from 2a403ab to 91b6551 Compare February 16, 2019 15:41

arey mentioned this pull request Mar 4, 2019

Use the Micrometer @Timed annotation #120

Closed

arey merged commit b86d8a0 into spring-petclinic:master Mar 4, 2019

arey mentioned this pull request Mar 20, 2019

Improve Micrometer and Grafana integration #122

Merged

talha-01 pushed a commit to talha-01/microservices-ci-cd-pipeline-with-petclinic-app that referenced this pull request Nov 13, 2020

Merge pull request spring-petclinic#105 from notsureifkevin/prometheus

34391db

Introducing Micrometer Prometheus metric collection

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing micrometer/prometheus metric collection #105

Introducing micrometer/prometheus metric collection #105

kvncrw commented Oct 1, 2018

arey Oct 3, 2018 •

edited

Loading

kvncrw Oct 3, 2018

arey Oct 4, 2018

arey Oct 3, 2018

kvncrw Oct 3, 2018

arey Oct 3, 2018

kvncrw Oct 3, 2018

arey commented Oct 14, 2018

mszarlinski left a comment

mszarlinski Oct 15, 2018

kvncrw Oct 21, 2018

arey Nov 12, 2018

kvncrw Feb 16, 2019

arey Feb 16, 2019

arey Mar 4, 2019

arey commented Nov 21, 2018 •

edited

Loading

arey commented Nov 21, 2018

kvncrw commented Nov 21, 2018

arey commented Nov 24, 2018

mszarlinski commented Nov 24, 2018 •

edited

Loading

arey commented Nov 25, 2018 •

edited

Loading

mszarlinski commented Nov 25, 2018

arey commented Nov 26, 2018

arey commented Feb 15, 2019

kvncrw commented Feb 16, 2019

kvncrw commented Feb 16, 2019

kvncrw commented Feb 16, 2019

arey commented Mar 8, 2019

kvncrw commented Mar 8, 2019


		@todo Add default custom dashboards to grafana

		Grafana and Prometheus are included in the `docker-compose.yml` configuration, and the public facing applications have been instrumented with [MicroMeter](https://micrometer.io) to collect JVM and custom business metrics.

Introducing micrometer/prometheus metric collection #105

Introducing micrometer/prometheus metric collection #105

Conversation

kvncrw commented Oct 1, 2018

arey Oct 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arey commented Oct 14, 2018

mszarlinski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arey commented Nov 21, 2018 • edited Loading

arey commented Nov 21, 2018

kvncrw commented Nov 21, 2018

arey commented Nov 24, 2018

mszarlinski commented Nov 24, 2018 • edited Loading

arey commented Nov 25, 2018 • edited Loading

mszarlinski commented Nov 25, 2018

arey commented Nov 26, 2018

arey commented Feb 15, 2019

kvncrw commented Feb 16, 2019

kvncrw commented Feb 16, 2019

kvncrw commented Feb 16, 2019

arey commented Mar 8, 2019

kvncrw commented Mar 8, 2019

arey Oct 3, 2018 •

edited

Loading

arey commented Nov 21, 2018 •

edited

Loading

mszarlinski commented Nov 24, 2018 •

edited

Loading

arey commented Nov 25, 2018 •

edited

Loading