flink 1.2 job fails while trying to use -p option

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

flink 1.2 job fails while trying to use -p option

Sathi Chowdhury

Need quick help on this

Trying to run

 

/bin/flink run -p 6 –c CLASSNAME - /mnt/flink/jarname.jar

 

everything works without –p option ,but runs with parallelization 1, which is what I am trying to get past.

 

java.lang.NoSuchMethodError: org.apache.flink.streaming.api.functions.source.SourceFunction$SourceContext.markAsTemporarilyIdle()V

        at org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.runFetcher(KinesisDataFetcher.java:330)

        at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:227)

        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)

        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)

        at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)

        at java.lang.Thread.run(Thread.java:745)

Thanks in advance

Sathi

=============Notice to Recipient: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain information that is confidential or legally privileged, and intended for the use of the individual or entity named above. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender by telephone or return e-mail and delete the original transmission and its attachments without reading or saving in any manner. Thank you. =============
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: flink 1.2 job fails while trying to use -p option

Tzu-Li (Gordon) Tai
Hi Sathi,

You seem to have built the Kinesis connector on Flink 1.3, but using it against Flink 1.2. Please make sure that the library versions and the Flink version match.

As for the case you’ve bumped in to: it just happens that with parallelism 1, the `markAsTemporarilyIdle` method is not called in the Kinesis connector, hence the exception did not pop up.

Cheers,
Gordon


On 24 May 2017 at 7:27:08 AM, Sathi Chowdhury ([hidden email]) wrote:

Need quick help on this

Trying to run

 

/bin/flink run -p 6 –c CLASSNAME - /mnt/flink/jarname.jar

 

everything works without –p option ,but runs with parallelization 1, which is what I am trying to get past.

 

java.lang.NoSuchMethodError: org.apache.flink.streaming.api.functions.source.SourceFunction$SourceContext.markAsTemporarilyIdle()V

        at org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.runFetcher(KinesisDataFetcher.java:330)

        at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:227)

        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)

        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)

        at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)

        at java.lang.Thread.run(Thread.java:745)

Thanks in advance

Sathi

=============Notice to Recipient: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain information that is confidential or legally privileged, and intended for the use of the individual or entity named above. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender by telephone or return e-mail and delete the original transmission and its attachments without reading or saving in any manner. Thank you. =============
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: flink 1.2 job fails while trying to use -p option

Sathi Chowdhury

Thanks…I was rebuilt the connector. It seems like the jar I was using was not built correctly at the first time. Yes it solved the issue.

 

From: "Tzu-Li (Gordon) Tai" <[hidden email]>
Date: Tuesday, May 23, 2017 at 11:30 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: flink 1.2 job fails while trying to use -p option

 

Hi Sathi,

 

You seem to have built the Kinesis connector on Flink 1.3, but using it against Flink 1.2. Please make sure that the library versions and the Flink version match.

 

As for the case you’ve bumped in to: it just happens that with parallelism 1, the `markAsTemporarilyIdle` method is not called in the Kinesis connector, hence the exception did not pop up.

 

Cheers,

Gordon

 

 

On 24 May 2017 at 7:27:08 AM, Sathi Chowdhury ([hidden email]) wrote:

Need quick help on this

Trying to run

 

/bin/flink run -p 6 –c CLASSNAME - /mnt/flink/jarname.jar

 

everything works without –p option ,but runs with parallelization 1, which is what I am trying to get past.

 

java.lang.NoSuchMethodError: org.apache.flink.streaming.api.functions.source.SourceFunction$SourceContext.markAsTemporarilyIdle()V

        at org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.runFetcher(KinesisDataFetcher.java:330)

        at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:227)

        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)

        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)

        at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)

        at java.lang.Thread.run(Thread.java:745)

Thanks in advance

Sathi

=============Notice to Recipient: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain information that is confidential or legally privileged, and intended for the use of the individual or entity named above. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender by telephone or return e-mail and delete the original transmission and its attachments without reading or saving in any manner. Thank you. =============

=============Notice to Recipient: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain information that is confidential or legally privileged, and intended for the use of the individual or entity named above. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender by telephone or return e-mail and delete the original transmission and its attachments without reading or saving in any manner. Thank you. =============
Loading...