How can I pass multiple java options in standalone mode ?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

How can I pass multiple java options in standalone mode ?

Vishwas Siravara
Hi guys,
I have this entry in flink-conf.yaml file for jvm options. 
env.java.opts: "-Djava.security.auth.login.config={{ flink_installed_dir }}/kafka-jaas.conf,-Djava.security.krb5.conf={{ flink_installed_dir }}/krb5.conf"

Is this supposed to be a , separated list ? I get a parse exception when the cluster starts. 

Thanks,
Vishwas 
Reply | Threaded
Open this post in threaded view
|

Re: How can I pass multiple java options in standalone mode ?

Zili Chen
Hi Vishwas,

Replace ',' with ' '(space) should work.

Best,
tison.


Vishwas Siravara <[hidden email]> 于2019年8月13日周二 上午6:50写道:
Hi guys,
I have this entry in flink-conf.yaml file for jvm options. 
env.java.opts: "-Djava.security.auth.login.config={{ flink_installed_dir }}/kafka-jaas.conf,-Djava.security.krb5.conf={{ flink_installed_dir }}/krb5.conf"

Is this supposed to be a , separated list ? I get a parse exception when the cluster starts. 

Thanks,
Vishwas 
Reply | Threaded
Open this post in threaded view
|

Re: How can I pass multiple java options in standalone mode ?

Vishwas Siravara
When I add env.java.opts like this
 env.java.opts:"-Dconfig.resource=qa.conf"

I see an error in the log file saying :  - Error while trying to split key and value in configuration file /data/flink-1.7.2/conf/flink-conf.yaml:248: "env.java.opts:"-Dconfig.resource=qa.conf""

This is really confusing and I cant find any document on how I should pass this option. 

Thanks,
Vishwas

On Wed, Aug 14, 2019 at 12:40 PM Vishwas Siravara <[hidden email]> wrote:
Is it possible for me to pass these arguments along with the job when I do flink run and then pass the jvm options. For example if I want to pass this parameter -Dconfig.resource=qa.conf and qa.conf is packaged in the job's fat jar then flink will not find this file if I pass -Dconfig.resource=qa.conf and qa.conf in env.java.opts. 

Thanks,
Vishwas 

On Mon, Aug 12, 2019 at 6:00 PM Zili Chen <[hidden email]> wrote:
Hi Vishwas,

Replace ',' with ' '(space) should work.

Best,
tison.


Vishwas Siravara <[hidden email]> 于2019年8月13日周二 上午6:50写道:
Hi guys,
I have this entry in flink-conf.yaml file for jvm options. 
env.java.opts: "-Djava.security.auth.login.config={{ flink_installed_dir }}/kafka-jaas.conf,-Djava.security.krb5.conf={{ flink_installed_dir }}/krb5.conf"

Is this supposed to be a , separated list ? I get a parse exception when the cluster starts. 

Thanks,
Vishwas 
Reply | Threaded
Open this post in threaded view
|

Re: How can I pass multiple java options in standalone mode ?

Aleksandar Mastilovic
It’s a YAML file, so I think you need to do something like

 env.java.opts: -Dconfig.resource=qa.conf

On Aug 14, 2019, at 11:58 AM, Vishwas Siravara <[hidden email]> wrote:

When I add env.java.opts like this
 env.java.opts:"-Dconfig.resource=qa.conf"

I see an error in the log file saying :  - Error while trying to split key and value in configuration file /data/flink-1.7.2/conf/flink-conf.yaml:248: "env.java.opts:"-Dconfig.resource=qa.conf""

This is really confusing and I cant find any document on how I should pass this option. 

Thanks,
Vishwas

On Wed, Aug 14, 2019 at 12:40 PM Vishwas Siravara <[hidden email]> wrote:
Is it possible for me to pass these arguments along with the job when I do flink run and then pass the jvm options. For example if I want to pass this parameter -Dconfig.resource=qa.conf and qa.conf is packaged in the job's fat jar then flink will not find this file if I pass -Dconfig.resource=qa.conf and qa.conf in env.java.opts. 

Thanks,
Vishwas 

On Mon, Aug 12, 2019 at 6:00 PM Zili Chen <[hidden email]> wrote:
Hi Vishwas,

Replace ',' with ' '(space) should work.

Best,
tison.


Vishwas Siravara <[hidden email]> 于2019年8月13日周二 上午6:50写道:
Hi guys,
I have this entry in flink-conf.yaml file for jvm options. 
env.java.opts: "-Djava.security.auth.login.config={{ flink_installed_dir }}/kafka-jaas.conf,-Djava.security.krb5.conf={{ flink_installed_dir }}/krb5.conf"

Is this supposed to be a , separated list ? I get a parse exception when the cluster starts. 

Thanks,
Vishwas 

Reply | Threaded
Open this post in threaded view
|

Re: How can I pass multiple java options in standalone mode ?

Vishwas Siravara
Thanks a lot, I fixed that, so now this works when I submit my job with the flink UI but when I submit it via flink run(command line) it does not take this env.java.opts: -Dconfig.resource=qa.conf property . How can I pass the jvm property to flink run which is running standalone without resource manager. 

Thanks,
Vishwas 

On Wed, Aug 14, 2019 at 2:35 PM Aleksandar Mastilovic <[hidden email]> wrote:
It’s a YAML file, so I think you need to do something like

 env.java.opts: -Dconfig.resource=qa.conf

On Aug 14, 2019, at 11:58 AM, Vishwas Siravara <[hidden email]> wrote:

When I add env.java.opts like this
 env.java.opts:"-Dconfig.resource=qa.conf"

I see an error in the log file saying :  - Error while trying to split key and value in configuration file /data/flink-1.7.2/conf/flink-conf.yaml:248: "env.java.opts:"-Dconfig.resource=qa.conf""

This is really confusing and I cant find any document on how I should pass this option. 

Thanks,
Vishwas

On Wed, Aug 14, 2019 at 12:40 PM Vishwas Siravara <[hidden email]> wrote:
Is it possible for me to pass these arguments along with the job when I do flink run and then pass the jvm options. For example if I want to pass this parameter -Dconfig.resource=qa.conf and qa.conf is packaged in the job's fat jar then flink will not find this file if I pass -Dconfig.resource=qa.conf and qa.conf in env.java.opts. 

Thanks,
Vishwas 

On Mon, Aug 12, 2019 at 6:00 PM Zili Chen <[hidden email]> wrote:
Hi Vishwas,

Replace ',' with ' '(space) should work.

Best,
tison.


Vishwas Siravara <[hidden email]> 于2019年8月13日周二 上午6:50写道:
Hi guys,
I have this entry in flink-conf.yaml file for jvm options. 
env.java.opts: "-Djava.security.auth.login.config={{ flink_installed_dir }}/kafka-jaas.conf,-Djava.security.krb5.conf={{ flink_installed_dir }}/krb5.conf"

Is this supposed to be a , separated list ? I get a parse exception when the cluster starts. 

Thanks,
Vishwas 

Reply | Threaded
Open this post in threaded view
|

Re: How can I pass multiple java options in standalone mode ?

Yang Wang
Hi Vishwas

The java option is used to start jobmanager and taskmanager jvm process. It should take effect when you set is in the flink-conf.yaml on all nodes. 
It is a flink cluster level configuration, not a job level. So i'm not sure what do you want to do.


Best,
Yang

Vishwas Siravara <[hidden email]> 于2019年8月15日周四 上午3:39写道:
Thanks a lot, I fixed that, so now this works when I submit my job with the flink UI but when I submit it via flink run(command line) it does not take this env.java.opts: -Dconfig.resource=qa.conf property . How can I pass the jvm property to flink run which is running standalone without resource manager. 

Thanks,
Vishwas 

On Wed, Aug 14, 2019 at 2:35 PM Aleksandar Mastilovic <[hidden email]> wrote:
It’s a YAML file, so I think you need to do something like

 env.java.opts: -Dconfig.resource=qa.conf

On Aug 14, 2019, at 11:58 AM, Vishwas Siravara <[hidden email]> wrote:

When I add env.java.opts like this
 env.java.opts:"-Dconfig.resource=qa.conf"

I see an error in the log file saying :  - Error while trying to split key and value in configuration file /data/flink-1.7.2/conf/flink-conf.yaml:248: "env.java.opts:"-Dconfig.resource=qa.conf""

This is really confusing and I cant find any document on how I should pass this option. 

Thanks,
Vishwas

On Wed, Aug 14, 2019 at 12:40 PM Vishwas Siravara <[hidden email]> wrote:
Is it possible for me to pass these arguments along with the job when I do flink run and then pass the jvm options. For example if I want to pass this parameter -Dconfig.resource=qa.conf and qa.conf is packaged in the job's fat jar then flink will not find this file if I pass -Dconfig.resource=qa.conf and qa.conf in env.java.opts. 

Thanks,
Vishwas 

On Mon, Aug 12, 2019 at 6:00 PM Zili Chen <[hidden email]> wrote:
Hi Vishwas,

Replace ',' with ' '(space) should work.

Best,
tison.


Vishwas Siravara <[hidden email]> 于2019年8月13日周二 上午6:50写道:
Hi guys,
I have this entry in flink-conf.yaml file for jvm options. 
env.java.opts: "-Djava.security.auth.login.config={{ flink_installed_dir }}/kafka-jaas.conf,-Djava.security.krb5.conf={{ flink_installed_dir }}/krb5.conf"

Is this supposed to be a , separated list ? I get a parse exception when the cluster starts. 

Thanks,
Vishwas 

Reply | Threaded
Open this post in threaded view
|

Re: How can I pass multiple java options in standalone mode ?

Vishwas Siravara
No this does not work , the config in flink-config.yaml applies only to job manager and task manager for env.java.options. When I say "flink run ..." from command line, there is a new jvm which is created which starts the main method and that does not get the env.java.options which is set in the flink-conf.yaml. This works when I set JVM_ARGS and export it before "flink run ..." 

Thanks,
Vishwas 

On Thu, Aug 15, 2019 at 1:20 AM Yang Wang <[hidden email]> wrote:
Hi Vishwas

The java option is used to start jobmanager and taskmanager jvm process. It should take effect when you set is in the flink-conf.yaml on all nodes. 
It is a flink cluster level configuration, not a job level. So i'm not sure what do you want to do.


Best,
Yang

Vishwas Siravara <[hidden email]> 于2019年8月15日周四 上午3:39写道:
Thanks a lot, I fixed that, so now this works when I submit my job with the flink UI but when I submit it via flink run(command line) it does not take this env.java.opts: -Dconfig.resource=qa.conf property . How can I pass the jvm property to flink run which is running standalone without resource manager. 

Thanks,
Vishwas 

On Wed, Aug 14, 2019 at 2:35 PM Aleksandar Mastilovic <[hidden email]> wrote:
It’s a YAML file, so I think you need to do something like

 env.java.opts: -Dconfig.resource=qa.conf

On Aug 14, 2019, at 11:58 AM, Vishwas Siravara <[hidden email]> wrote:

When I add env.java.opts like this
 env.java.opts:"-Dconfig.resource=qa.conf"

I see an error in the log file saying :  - Error while trying to split key and value in configuration file /data/flink-1.7.2/conf/flink-conf.yaml:248: "env.java.opts:"-Dconfig.resource=qa.conf""

This is really confusing and I cant find any document on how I should pass this option. 

Thanks,
Vishwas

On Wed, Aug 14, 2019 at 12:40 PM Vishwas Siravara <[hidden email]> wrote:
Is it possible for me to pass these arguments along with the job when I do flink run and then pass the jvm options. For example if I want to pass this parameter -Dconfig.resource=qa.conf and qa.conf is packaged in the job's fat jar then flink will not find this file if I pass -Dconfig.resource=qa.conf and qa.conf in env.java.opts. 

Thanks,
Vishwas 

On Mon, Aug 12, 2019 at 6:00 PM Zili Chen <[hidden email]> wrote:
Hi Vishwas,

Replace ',' with ' '(space) should work.

Best,
tison.


Vishwas Siravara <[hidden email]> 于2019年8月13日周二 上午6:50写道:
Hi guys,
I have this entry in flink-conf.yaml file for jvm options. 
env.java.opts: "-Djava.security.auth.login.config={{ flink_installed_dir }}/kafka-jaas.conf,-Djava.security.krb5.conf={{ flink_installed_dir }}/krb5.conf"

Is this supposed to be a , separated list ? I get a parse exception when the cluster starts. 

Thanks,
Vishwas 

Reply | Threaded
Open this post in threaded view
|

Re: How can I pass multiple java options in standalone mode ?

Yang Wang
Hi Vishwas,

I think we are on the same page now.

> When you submit a job through Flink WebUI, your main code will be executed in the job manager jvm process. Since job manager is started with the java opts in flink-conf.yaml, so you could find it works.

> When you submit a job in command line(flink run), your main code will be executed in flink client jvm process. The flink client does not parse the config option(env.java.options) in flink-conf.yaml, so you need to export the JVM_ARGS environment so that the flink script could append it to java command.[1]

Just to make clear that env.java.options only applies to jobmanager and taskmanager jvm processes.




Best,
Yang

Vishwas Siravara <[hidden email]> 于2019年8月15日周四 下午10:38写道:
No this does not work , the config in flink-config.yaml applies only to job manager and task manager for env.java.options. When I say "flink run ..." from command line, there is a new jvm which is created which starts the main method and that does not get the env.java.options which is set in the flink-conf.yaml. This works when I set JVM_ARGS and export it before "flink run ..." 

Thanks,
Vishwas 

On Thu, Aug 15, 2019 at 1:20 AM Yang Wang <[hidden email]> wrote:
Hi Vishwas

The java option is used to start jobmanager and taskmanager jvm process. It should take effect when you set is in the flink-conf.yaml on all nodes. 
It is a flink cluster level configuration, not a job level. So i'm not sure what do you want to do.


Best,
Yang

Vishwas Siravara <[hidden email]> 于2019年8月15日周四 上午3:39写道:
Thanks a lot, I fixed that, so now this works when I submit my job with the flink UI but when I submit it via flink run(command line) it does not take this env.java.opts: -Dconfig.resource=qa.conf property . How can I pass the jvm property to flink run which is running standalone without resource manager. 

Thanks,
Vishwas 

On Wed, Aug 14, 2019 at 2:35 PM Aleksandar Mastilovic <[hidden email]> wrote:
It’s a YAML file, so I think you need to do something like

 env.java.opts: -Dconfig.resource=qa.conf

On Aug 14, 2019, at 11:58 AM, Vishwas Siravara <[hidden email]> wrote:

When I add env.java.opts like this
 env.java.opts:"-Dconfig.resource=qa.conf"

I see an error in the log file saying :  - Error while trying to split key and value in configuration file /data/flink-1.7.2/conf/flink-conf.yaml:248: "env.java.opts:"-Dconfig.resource=qa.conf""

This is really confusing and I cant find any document on how I should pass this option. 

Thanks,
Vishwas

On Wed, Aug 14, 2019 at 12:40 PM Vishwas Siravara <[hidden email]> wrote:
Is it possible for me to pass these arguments along with the job when I do flink run and then pass the jvm options. For example if I want to pass this parameter -Dconfig.resource=qa.conf and qa.conf is packaged in the job's fat jar then flink will not find this file if I pass -Dconfig.resource=qa.conf and qa.conf in env.java.opts. 

Thanks,
Vishwas 

On Mon, Aug 12, 2019 at 6:00 PM Zili Chen <[hidden email]> wrote:
Hi Vishwas,

Replace ',' with ' '(space) should work.

Best,
tison.


Vishwas Siravara <[hidden email]> 于2019年8月13日周二 上午6:50写道:
Hi guys,
I have this entry in flink-conf.yaml file for jvm options. 
env.java.opts: "-Djava.security.auth.login.config={{ flink_installed_dir }}/kafka-jaas.conf,-Djava.security.krb5.conf={{ flink_installed_dir }}/krb5.conf"

Is this supposed to be a , separated list ? I get a parse exception when the cluster starts. 

Thanks,
Vishwas