git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Setting Flink Monitoring API Port on YARN Cluster


Hi everyone,

I'm running a YARN session on a cluster with one master and one core and would like to use the Monitoring API programmatically to submit jobs. I have found that the configuration variables are read but ignored when starting the session - it seems to choose a random port each run.

Here's a snippet from the startup logs: 

2018-09-06 21:44:38,763 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: env.yarn.conf.dir, /etc/hadoop/conf
2018-09-06 21:44:38,764 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
2018-09-06 21:44:38,765 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 44477
2018-09-06 21:44:38,765 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.web.port, 44477
2018-09-06 21:44:38,765 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.jobmanager.port, 44477
2018-09-06 21:44:38,775 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-hadoop.
2018-09-06 21:44:39,615 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-06 21:44:39,799 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to hadoop (auth:SIMPLE)
2018-09-06 21:44:40,045 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at ip-10-2-3-71.ec2.internal/10.2.3.71:8032
2018-09-06 21:44:40,312 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=4096, numberTaskManagers=1, slotsPerTaskManager=1}
2018-09-06 21:44:43,564 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1536250520330_0007
2018-09-06 21:44:43,802 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1536250520330_0007
2018-09-06 21:44:43,802 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2018-09-06 21:44:43,804 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2018-09-06 21:44:48,326 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
2018-09-06 21:44:48,326 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The Flink YARN client has been started in detached mode. In order to stop Flink on YARN, use the following command or a YARN web interface to stop it:
yarn application -kill application_1536250520330_0007
Please also note that the temporary files of the YARN session in the home directory will not be removed.
2018-09-06 21:44:48,821 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest client endpoint started.
Flink JobManager is now running on ip-10-2-3-25.ec2.internal:38683 with leader id 00000000-0000-0000-0000-000000000000.
JobManager Web Interface: http://ip-10-2-3-25.ec2.internal:38683


I'm setting both the rest.port and jobmanager.web.port, but both are ignored. Has anyone seen this before?

Thanks!