git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Flink Job Cluster Deployment on K8s


Hello,

 

I am trying to experiment with the new Flink job cluster on Kubernetes that is available with the Flink 1.6.x release.

 

I am using the instructions here to create the docker image, which is working great.   This image then gets pushed to our Artifactory.

 

I am able to create the job cluster service using this helm chart.

 

However when I try to deploy the job cluster job using the helm chart below (based on this one):

 

apiVersion: batch/v1
kind: Job
metadata:
  name:
flink-job-cluster
spec:
  template:
    metadata:
      labels:
        app:
flink
       
component: job-cluster
   
spec:
      imagePullSecrets:
     
- name: artifactory-docker-registry
     
restartPolicy: OnFailure
     
containers:
     
- name: flink-job-cluster
       
image: {ImageOnArtifactory}
       
args: ["job-cluster", "--job-classname", "job.jar", "-Djobmanager.rpc.address=flink-job-cluster",
              
"-Dparallelism.default=1", "-Dblob.server.port=6124", "-Dquery.server.ports=6125"]
       
ports:
       
- containerPort: 6123
         
name: rpc
        -
containerPort: 6124
         
name: blob
        -
containerPort: 6125
         
name: query
        -
containerPort: 8081
          
name: ui

 

I get this error on the pod:

 

%d [%thread] %-5level %logger - %msg%n org.apache.flink.util.FlinkException: Could not load the provied entrypoint class.                                                                                               

at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:102) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                       

at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.retrieveJobGraph(StandaloneJobClusterEntryPoint.java:84) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                             

at org.apache.flink.runtime.entrypoint.JobClusterEntrypoint.createDispatcher(JobClusterEntrypoint.java:107) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                   

at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startClusterComponents(ClusterEntrypoint.java:353) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                  

at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:232) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                              

at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:190) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                   

at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                                                             

at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:189) [flink-dist_2.11-1.6.1.jar:1.6.1]                                                                              

at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:170) [flink-dist_2.11-1.6.1.jar:1.6.1]

Caused by: java.lang.ClassNotFoundException: job.jar                                                                                                                                                                    

at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_111-internal]                                                                                                                           

at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_111-internal]                                                                                                                                 

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_111-internal]                                                                                                                         

at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_111-internal]                                                                                                                                 

at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:99) ~[flink-dist_2.11-1.6.1.jar:1.6.1]                                       

... 8 more

 

 

When I launch and attach the docker image, job.jar exists in /opt/flink and there is a symbolic link in /opt/flink/lib.

 

Any ideas as to why job.jar can’t be found?

 

Our flink version is 1.6.1.  We are using the flink-1.6.1-bin-scala_2.11 distribution.

 

Any help would be much appreciated.

 

Thanks,

 

Thad Truman | Software Engineer | Neovest, Inc.

A:

T:

E:

1145 S 800 E, Ste 310 Orem, UT 84097

+1 801 900 2480

ttruman@xxxxxxxxxxx

 

Support Desk: support@xxxxxxxxxxx / +1 800 433 4276

 

Alt logo for white backgrounds (Grey Flat)2

This email is confidential and subject to important disclaimers and conditions including on offers for purchase or sale of securities accuracy and completeness of information viruses confidentiality legal privilege and legal entity disclaimers available at www.neovest.com/disclosures.html