Flink and Akka version incompatibility ..

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink and Akka version incompatibility ..

Debasish Ghosh
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

Reply | Threaded
Open this post in threaded view
|

Re:Flink and Akka version incompatibility ..

Haibo Sun
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.

Reply | Threaded
Open this post in threaded view
|

Re: Flink and Akka version incompatibility ..

Debasish Ghosh
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Akka version incompatibility ..

Zili Chen
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Akka version incompatibility ..

Debasish Ghosh
The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--


--
Reply | Threaded
Open this post in threaded view
|

Re:Re: Flink and Akka version incompatibility ..

Haibo Sun
Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.
I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:
The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--


--
Reply | Threaded
Open this post in threaded view
|

Re: Re: Flink and Akka version incompatibility ..

Jeff Zhang
I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.


Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道:
Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.
I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:
The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--


--


--
Best Regards

Jeff Zhang
Reply | Threaded
Open this post in threaded view
|

Re: Re: Flink and Akka version incompatibility ..

Debasish Ghosh
For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:
I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.


Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道:
Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.
I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:
The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--


--


--
Best Regards

Jeff Zhang
--
Sent from my iPhone
Reply | Threaded
Open this post in threaded view
|

Re:Re: Re: Flink and Akka version incompatibility ..

Haibo Sun
Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,
Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:
For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:
I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.


Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道:
Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.
I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:
The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--


--


--
Best Regards

Jeff Zhang
--
Sent from my iPhone
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: Flink and Akka version incompatibility ..

Debasish Ghosh
Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,
Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:
For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:
I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.


Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道:
Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.
I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:
The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--


--


--
Best Regards

Jeff Zhang
--
Sent from my iPhone


--
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: Flink and Akka version incompatibility ..

Debasish Ghosh
Also wanted to check if anyone has ventured into this exercise of shading Akka in Flink .. 
Is this something that qualifies as one of the roadmap items in Flink ?

regards.

On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh <[hidden email]> wrote:
Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,
Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:
For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:
I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.


Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道:
Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.
I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:
The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--


--


--
Best Regards

Jeff Zhang
--
Sent from my iPhone


--


--
Reply | Threaded
Open this post in threaded view
|

Re:Re: Re: Re: Flink and Akka version incompatibility ..

Haibo Sun

The following JIRA is about the problem you encounter. I think you should be very interested in its comments.There does seem to be a problem with shading Akka, and Flink is considering isolating the classloader that contain Akka and Scala to allow the applications and Flink to use different Akka versions.


Best,
Haibo

At 2019-07-25 00:07:27, "Debasish Ghosh" <[hidden email]> wrote:
Also wanted to check if anyone has ventured into this exercise of shading Akka in Flink .. 
Is this something that qualifies as one of the roadmap items in Flink ?

regards.

On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh <[hidden email]> wrote:
Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,
Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:
For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:
I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.


Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道:
Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.
I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:
The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--


--


--
Best Regards

Jeff Zhang
--
Sent from my iPhone


--


--
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: Re: Flink and Akka version incompatibility ..

Debasish Ghosh
Thanks for the pointer .. yes, we need this ..

On Thu, Jul 25, 2019 at 9:08 AM Haibo Sun <[hidden email]> wrote:

The following JIRA is about the problem you encounter. I think you should be very interested in its comments.There does seem to be a problem with shading Akka, and Flink is considering isolating the classloader that contain Akka and Scala to allow the applications and Flink to use different Akka versions.


Best,
Haibo

At 2019-07-25 00:07:27, "Debasish Ghosh" <[hidden email]> wrote:
Also wanted to check if anyone has ventured into this exercise of shading Akka in Flink .. 
Is this something that qualifies as one of the roadmap items in Flink ?

regards.

On Wed, Jul 24, 2019 at 3:44 PM Debasish Ghosh <[hidden email]> wrote:
Hi Haibo - Thanks for the clarification ..

regards.

On Wed, Jul 24, 2019 at 2:58 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,

I agree that Flink should shade its Akka. 

Maybe you misunderstood me. I mean, in the absence of official shading Akka in Flink, the relatively conservative way is to shade Akka of your application (I concern Flink won't work well after shading its Akka).

Best,
Haibo

At 2019-07-24 16:43:28, "Debasish Ghosh" <[hidden email]> wrote:
For our application users are expected to work with Akka APIs - hence if I shade Akka in my application users will need to work with shaded imports which feels unnatural. With Flink, Akka is an implementation detail and Flink users are not expected to use Akka APIs. Hence shading will not have any user level impact. 

Hence the suggestion to shade Akka in Flink rather than the user application.

regards.

On Wed, 24 Jul 2019 at 2:04 PM, Jeff Zhang <[hidden email]> wrote:
I think it is better to shade all the dependencies of flink so that all the projects that use flink won't hit this kind of issue.


Haibo Sun <[hidden email]> 于2019年7月24日周三 下午4:07写道:
Hi,   Debasish Ghosh

I don't know why not shade Akka, maybe it can be shaded. Chesnay may be able to answer that.
I recommend to shade Akka dependency of your application because it don't be known what's wrong with shading Flink's Akka.

CC  @Chesnay Schepler 

Best,
Haibo

At 2019-07-24 15:48:59, "Debasish Ghosh" <[hidden email]> wrote:
The problem that I am facing is with Akka serialization .. Why not shade the whole of Akka ?

java.lang.AbstractMethodError: akka.remote.RemoteActorRefProvider.serializationInformation()Lakka/serialization/Serialization$Information;
at akka.serialization.Serialization.serializationInformation(Serialization.scala:166)

Akka 2.6 is just around the corner and I don't think Flink will upgrade to Akka 2.6 that soon .. so somehow this problem is bound to recur ..

regards.

On Wed, Jul 24, 2019 at 1:01 PM Zili Chen <[hidden email]> wrote:
I can see that we relocate akka's netty, akka uncommon math but also
be curious why Flink doesn't shaded all of akka dependencies...

Best,
tison.


Debasish Ghosh <[hidden email]> 于2019年7月24日周三 下午3:15写道:
Hello Haibo -

Yes, my application depends on Akka 2.5. 
Just curious, why do you think it's recommended to shade Akka version of my application instead of Flink ?

regards.

On Wed, Jul 24, 2019 at 12:42 PM Haibo Sun <[hidden email]> wrote:
Hi  Debasish Ghosh,
 
Does your application have to depend on Akka 2.5? If not, it's a good idea to always keep the Akka version that the application depend on in line with Flink. 
If you want to try shading Akka dependency, I think that it is more recommended to shade Akka dependency of your application.

Best,
Haibo

At 2019-07-24 14:31:29, "Debasish Ghosh" <[hidden email]> wrote:
Hello -

An application that uses Akka 2.5 and Flink 1.8.0 gives runtime errors because of version mismatch between Akka that we use and the one that Flink uses (which is Akka 2.4). Anyone tried shading Akka dependency with Flink ? 

Or is there any other alternative way to handle this issue ? I know Flink 1.9 has upgraded to Akka 2.5 but this is (I think) going to be a recurring problem down the line with mismatch between the new releases of Akka and Flink.



--


--


--
Best Regards

Jeff Zhang
--
Sent from my iPhone


--


--


--