[jira] [Created] (FLINK-10521) TaskManager metrics are not reported to prometheus after running a job
Florian Schmidt created FLINK-10521:
Summary: TaskManager metrics are not reported to prometheus after running a job
Issue Type: Bug
Affects Versions: 1.6.1
Environment: Flink 1.6.1 cluster with one taskmanager and one jobmanager, prometheus and grafana, all started in a local docker environment.
See sample project at: https://github.com/florianschmidt1994/flink-fault-tolerance-baseline
Reporter: Florian Schmidt
Attachments: Screenshot 2018-10-10 at 11.16.22.png
I'm using prometheus to collect the metrics from Flink, and I noticed that shortly after running a job, metrics from the taskmanager will stop being reported most of the time.
Looking at the prometheus logs I can see that requests to taskmanager:9249/metrics are correct when no job is running, but after starting to run a job those requests will return an empty response with increasing frequency, until at some point most of the requests are not successful anymore. I was able to very this by running `curl localhost:9249/metrics` inside the taskmanager container, where more often that not the response was empty, instead of containing the expected metrics.
In the attached image you can see that occasionally some requests succeed, but there are some big gaps in between. The prometheus scrape interval is set to 1s.
!Screenshot 2018-10-10 at 11.16.22.png!
This message was sent by Atlassian JIRA