yarnship option

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

yarnship option

Mikhail Pryakhin
Hi all!

I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
Here is the way I run the job:

export HADOOP_USER_NAME=hdfs; \
export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
/opt/flink-1.2.1/bin/flink run \
        -yst \
        -yt /home/user/job-libs  \
        -m yarn-cluster \
        -yn 3 \
        -c com.flink.Test \
        flink-test_2.11-1.0.0-SNAPSHOT.jar

Finally the job fails complaing that it can't find the class:
java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09

I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:

2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner             -  Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>

Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath

----
Mike Pryakhin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: yarnship option

rmetzger0
Hi,

this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
Also, were there any previous Kafka09 related exceptions in the log??

From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695

On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <[hidden email]> wrote:
Hi all!

I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
Here is the way I run the job:

export HADOOP_USER_NAME=hdfs; \
export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
/opt/flink-1.2.1/bin/flink run \
        -yst \
        -yt /home/user/job-libs  \
        -m yarn-cluster \
        -yn 3 \
        -c com.flink.Test \
        flink-test_2.11-1.0.0-SNAPSHOT.jar

Finally the job fails complaing that it can't find the class:
java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09

I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:

2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner             -  Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>

Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath

----
Mike Pryakhin

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: yarnship option

Mikhail Pryakhin
Hi Robert!
Thanks a lot for your reply!

>Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?

The jar does contain the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file is listed below)

> Also, were there any previous Kafka09 related exceptions in the log??    
No, it was the very first exception… You can find log file attached. 


the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:

root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
Archive:  job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2017-04-11 01:59   META-INF/
      443  2017-04-11 01:59   META-INF/MANIFEST.MF
     1451  2017-04-11 01:59   META-INF/DEPENDENCIES
    11358  2017-04-11 01:59   META-INF/LICENSE
      182  2017-04-11 01:59   META-INF/NOTICE
        0  2017-04-11 01:59   org/
        0  2017-04-11 01:59   org/apache/
        0  2017-04-11 01:59   org/apache/flink/
        0  2017-04-11 01:59   org/apache/flink/streaming/
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
    11554  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
     2603  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class
     4463  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class
     2693  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/
     1125  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class
      570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class
    10268  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class
      570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class
     3018  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover.class
    11854  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class
     2182  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
      311  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class
     2329  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class
        0  2017-04-11 01:59   META-INF/maven/
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/
     6039  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml
      131  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/
     3285  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.xml
      114  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.properties
---------                     -------
    76543                     33 files







------
Mike Pryakhin



On 22 May 2017, at 22:14, Robert Metzger <[hidden email]> wrote:

Hi,

this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
Also, were there any previous Kafka09 related exceptions in the log??

From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695

On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <[hidden email]> wrote:
Hi all!

I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
Here is the way I run the job:

export HADOOP_USER_NAME=hdfs; \
export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
/opt/flink-1.2.1/bin/flink run \
        -yst \
        -yt /home/user/job-libs  \
        -m yarn-cluster \
        -yn 3 \
        -c com.flink.Test \
        flink-test_2.11-1.0.0-SNAPSHOT.jar

Finally the job fails complaing that it can't find the class:
java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09

I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:

2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner             -  Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>

Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath

----
Mike Pryakhin



flink-root-client-dmpkit-dev-dn1.log (19K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: yarnship option

Mikhail Pryakhin
Hi Robert!

You were right.. it was my fault, I didn’t copy all required dependencies for the flink-connector-kafka. That made impossible to load the class.

Everything works as expected but only if I put required dependencies inside the flink lib folder, but when dependencies are located in a different folder which is passed in the yarnship command line option then they seem not to be put on the TM's classpath, and the job fails with ClassNotFoundException. 
Is there anything else I need to provide to make libraries in that folder to be on the TM's classpath?

Many thanks,

——

Mike Pryakhin


On 22 May 2017, at 23:06, Mikhail Pryakhin <[hidden email]> wrote:

Hi Robert!
Thanks a lot for your reply!

>Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?

The jar does contain the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file is listed below)

> Also, were there any previous Kafka09 related exceptions in the log??    
No, it was the very first exception… You can find log file attached. 


the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:

root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
Archive:  job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2017-04-11 01:59   META-INF/
      443  2017-04-11 01:59   META-INF/MANIFEST.MF
     1451  2017-04-11 01:59   META-INF/DEPENDENCIES
    11358  2017-04-11 01:59   META-INF/LICENSE
      182  2017-04-11 01:59   META-INF/NOTICE
        0  2017-04-11 01:59   org/
        0  2017-04-11 01:59   org/apache/
        0  2017-04-11 01:59   org/apache/flink/
        0  2017-04-11 01:59   org/apache/flink/streaming/
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
    11554  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
     2603  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class
     4463  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class
     2693  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/
     1125  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class
      570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class
    10268  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class
      570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class
     3018  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover.class
    11854  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class
     2182  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
      311  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class
     2329  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class
        0  2017-04-11 01:59   META-INF/maven/
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/
     6039  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml
      131  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/
     3285  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.xml
      114  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.properties
---------                     -------
    76543                     33 files




<flink-root-client-dmpkit-dev-dn1.log>


------
Mike Pryakhin



On 22 May 2017, at 22:14, Robert Metzger <[hidden email]> wrote:

Hi,

this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
Also, were there any previous Kafka09 related exceptions in the log??

From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695

On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <[hidden email]> wrote:
Hi all!

I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
Here is the way I run the job:

export HADOOP_USER_NAME=hdfs; \
export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
/opt/flink-1.2.1/bin/flink run \
        -yst \
        -yt /home/user/job-libs  \
        -m yarn-cluster \
        -yn 3 \
        -c com.flink.Test \
        flink-test_2.11-1.0.0-SNAPSHOT.jar

Finally the job fails complaing that it can't find the class:
java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09

I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:

2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner             -  Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>

Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath

----
Mike Pryakhin



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: yarnship option

rmetzger0
Hi Mike,

I would recommend you to build a "fat jar" containing your application code and all required dependencies.

On Tue, May 23, 2017 at 10:33 AM, Mikhail Pryakhin <[hidden email]> wrote:
Hi Robert!

You were right.. it was my fault, I didn’t copy all required dependencies for the flink-connector-kafka. That made impossible to load the class.

Everything works as expected but only if I put required dependencies inside the flink lib folder, but when dependencies are located in a different folder which is passed in the yarnship command line option then they seem not to be put on the TM's classpath, and the job fails with ClassNotFoundException. 
Is there anything else I need to provide to make libraries in that folder to be on the TM's classpath?

Many thanks,

——

Mike Pryakhin


On 22 May 2017, at 23:06, Mikhail Pryakhin <[hidden email]> wrote:

Hi Robert!
Thanks a lot for your reply!

>Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?

The jar does contain the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file is listed below)

> Also, were there any previous Kafka09 related exceptions in the log??    
No, it was the very first exception… You can find log file attached. 


the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:

root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
Archive:  job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2017-04-11 01:59   META-INF/
      443  2017-04-11 01:59   META-INF/MANIFEST.MF
     1451  2017-04-11 01:59   META-INF/DEPENDENCIES
    11358  2017-04-11 01:59   META-INF/LICENSE
      182  2017-04-11 01:59   META-INF/NOTICE
        0  2017-04-11 01:59   org/
        0  2017-04-11 01:59   org/apache/
        0  2017-04-11 01:59   org/apache/flink/
        0  2017-04-11 01:59   org/apache/flink/streaming/
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
    11554  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
     2603  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class
     4463  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class
     2693  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/
     1125  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class
      570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class
    10268  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class
      570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class
     3018  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover.class
    11854  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class
     2182  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
      311  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class
     2329  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class
        0  2017-04-11 01:59   META-INF/maven/
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/
     6039  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml
      131  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/
     3285  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.xml
      114  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.properties
---------                     -------
    76543                     33 files




<flink-root-client-dmpkit-dev-dn1.log>


------
Mike Pryakhin



On 22 May 2017, at 22:14, Robert Metzger <[hidden email]> wrote:

Hi,

this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
Also, were there any previous Kafka09 related exceptions in the log??

From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695

On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <[hidden email]> wrote:
Hi all!

I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
Here is the way I run the job:

export HADOOP_USER_NAME=hdfs; \
export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
/opt/flink-1.2.1/bin/flink run \
        -yst \
        -yt /home/user/job-libs  \
        -m yarn-cluster \
        -yn 3 \
        -c com.flink.Test \
        flink-test_2.11-1.0.0-SNAPSHOT.jar

Finally the job fails complaing that it can't find the class:
java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09

I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:

2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner             -  Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>

Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath

----
Mike Pryakhin




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: yarnship option

Mikhail Pryakhin
Hi Robert,

Thanks for your advice but this option doesn’t work for us because our job depends on third party jars with transitive dependencies to the same jar but of different versions and it hardly possible to say which version will participate in the fat jar..

Anyway thanks a lot for your help.

—— 
Mike Pryakhin


On 23 May 2017, at 16:04, Robert Metzger <[hidden email]> wrote:

Hi Mike,

I would recommend you to build a "fat jar" containing your application code and all required dependencies.

On Tue, May 23, 2017 at 10:33 AM, Mikhail Pryakhin <[hidden email]> wrote:
Hi Robert!

You were right.. it was my fault, I didn’t copy all required dependencies for the flink-connector-kafka. That made impossible to load the class.

Everything works as expected but only if I put required dependencies inside the flink lib folder, but when dependencies are located in a different folder which is passed in the yarnship command line option then they seem not to be put on the TM's classpath, and the job fails with ClassNotFoundException. 
Is there anything else I need to provide to make libraries in that folder to be on the TM's classpath?

Many thanks,

——

Mike Pryakhin


On 22 May 2017, at 23:06, Mikhail Pryakhin <[hidden email]> wrote:

Hi Robert!
Thanks a lot for your reply!

>Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?

The jar does contain the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file is listed below)

> Also, were there any previous Kafka09 related exceptions in the log??    
No, it was the very first exception… You can find log file attached. 


the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:

root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
Archive:  job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2017-04-11 01:59   META-INF/
      443  2017-04-11 01:59   META-INF/MANIFEST.MF
     1451  2017-04-11 01:59   META-INF/DEPENDENCIES
    11358  2017-04-11 01:59   META-INF/LICENSE
      182  2017-04-11 01:59   META-INF/NOTICE
        0  2017-04-11 01:59   org/
        0  2017-04-11 01:59   org/apache/
        0  2017-04-11 01:59   org/apache/flink/
        0  2017-04-11 01:59   org/apache/flink/streaming/
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
    11554  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
     2603  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class
     4463  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class
     2693  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class
        0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/
     1125  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class
      570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class
    10268  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class
      570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class
     3018  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover.class
    11854  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class
     2182  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
      311  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class
     2329  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class
        0  2017-04-11 01:59   META-INF/maven/
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/
     6039  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml
      131  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties
        0  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/
     3285  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.xml
      114  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.properties
---------                     -------
    76543                     33 files




<flink-root-client-dmpkit-dev-dn1.log>


------
Mike Pryakhin



On 22 May 2017, at 22:14, Robert Metzger <[hidden email]> wrote:

Hi,

this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
Also, were there any previous Kafka09 related exceptions in the log??

From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695

On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <[hidden email]> wrote:
Hi all!

I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
Here is the way I run the job:

export HADOOP_USER_NAME=hdfs; \
export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
/opt/flink-1.2.1/bin/flink run \
        -yst \
        -yt /home/user/job-libs  \
        -m yarn-cluster \
        -yn 3 \
        -c com.flink.Test \
        flink-test_2.11-1.0.0-SNAPSHOT.jar

Finally the job fails complaing that it can't find the class:
java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09

I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:

2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner             -  Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>

Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath

----
Mike Pryakhin





Loading...