Flink 1.9, MapR secure cluster, high availability

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink 1.9, MapR secure cluster, high availability

Maxim Parkachov
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.  
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.9, MapR secure cluster, high availability

Stephan Ewen
Hi Maxim!

The change of the MapR dependency should not have an impact on that.
Do you know if the same thing worked in prior Flink versions? Is that a regression in 1.9?

The exception that you report, is that from Flink's HA services trying to connect to ZK, or from the MapR FS client trying to connect to ZK?

Best,
Stephan


On Tue, Aug 27, 2019 at 11:03 AM Maxim Parkachov <[hidden email]> wrote:
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.  
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.9, MapR secure cluster, high availability

Maxim Parkachov
Hi Stephan,

With previous versions, I tried around 1.7, I always had to compile MapR hadoop to get it working.
With 1.9 I took hadoop-less Flink, which worked with MapR FS until I switched on HA.
So it is hard to say if this is regression or not.

The error happens when Flink tries to initialize BLOB storage on MapR FS. Without HA it takes
zookeeper from classpath (MapR org.apache.zookeeper) and with HA it takes shaded one.

After fixing couple of issue with pom, I was able to compile Flink with MapR zookeeper and now
when I start with HA mode it uses shaded zookeeper (which is now MapR) to initialize BLOB and
org.apache.zookeeper (which is as well MapR) for HA recovery.

It works, but, I was expecting it to work without compiling MapR dependencies.

Hope this helps,
Maxim.

On Thu, Aug 29, 2019 at 7:00 PM Stephan Ewen <[hidden email]> wrote:
Hi Maxim!

The change of the MapR dependency should not have an impact on that.
Do you know if the same thing worked in prior Flink versions? Is that a regression in 1.9?

The exception that you report, is that from Flink's HA services trying to connect to ZK, or from the MapR FS client trying to connect to ZK?

Best,
Stephan


On Tue, Aug 27, 2019 at 11:03 AM Maxim Parkachov <[hidden email]> wrote:
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.  
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.9, MapR secure cluster, high availability

Stephan Ewen
Could you share the stack trace where the failure occurs, so we can see why the Flink ZK is used during MapR FS access?

/CC Till and Tison - just FYI

On Fri, Aug 30, 2019 at 9:40 AM Maxim Parkachov <[hidden email]> wrote:
Hi Stephan,

With previous versions, I tried around 1.7, I always had to compile MapR hadoop to get it working.
With 1.9 I took hadoop-less Flink, which worked with MapR FS until I switched on HA.
So it is hard to say if this is regression or not.

The error happens when Flink tries to initialize BLOB storage on MapR FS. Without HA it takes
zookeeper from classpath (MapR org.apache.zookeeper) and with HA it takes shaded one.

After fixing couple of issue with pom, I was able to compile Flink with MapR zookeeper and now
when I start with HA mode it uses shaded zookeeper (which is now MapR) to initialize BLOB and
org.apache.zookeeper (which is as well MapR) for HA recovery.

It works, but, I was expecting it to work without compiling MapR dependencies.

Hope this helps,
Maxim.

On Thu, Aug 29, 2019 at 7:00 PM Stephan Ewen <[hidden email]> wrote:
Hi Maxim!

The change of the MapR dependency should not have an impact on that.
Do you know if the same thing worked in prior Flink versions? Is that a regression in 1.9?

The exception that you report, is that from Flink's HA services trying to connect to ZK, or from the MapR FS client trying to connect to ZK?

Best,
Stephan


On Tue, Aug 27, 2019 at 11:03 AM Maxim Parkachov <[hidden email]> wrote:
Hi everyone,

I'm testing release 1.9 on MapR secure cluster. I took flink binaries from download page and trying to start Yarn session cluster. All MapR specific libraries and configs are added according to documentation.

When I start yarn-session without high availability, it uses zookeeper from MapR distribution (org.apache.zookeeper) and correctly connects to cluster and access to maprfs works as expected.

But if I add zookeeper as high-avalability option, instead of MapR zookeeper it tries to use shaded zookeeper and this one could not connect with mapr credentials:
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient  - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2019-08-27 10:42:45,240 ERROR org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

I tried to use separate zookeeper cluster for HA, but maprfs still doesn't work.

Is this related to removal of MapR specific settings in Release 1.9 ?
Should I still compile custom version of Flink with MapR dependencies ?
(trying to do now, but getting some errors during compilation).

Can I somehow force flink to use MapR zookeeper even with HA mode ?

Thanks in advance,
Maxim.