confusing RocksDBStateBackend parameters

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

confusing RocksDBStateBackend parameters

bowen.li
Hello guys,
      I've been trying to figure out differences among several parameters of RocksDBStateBackend. The confusing parameters are:

      In flink-conf.yaml:
      1. state.backend.fs.checkpointdir
      2. state.backend.rocksdb.checkpointdir
      3. state.checkpoints.dir

      and 
       4. the param 'checkpointDataUri' you pass in to RocksDBStateBackend
constructor in`public RocksDBStateBackend(URI checkpointDataUri)`

        This email thread explained the first three well. But what's the 4th one for? What's its difference from others? I'd appreciate your clarification.

       Thanks very much!
Bowen

        
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: confusing RocksDBStateBackend parameters

mmziyad
Hi,

To set the rocksdb state, you have two options:

1. Set the default state of the flink cluster, using the below parameters in flink-conf.yaml file
state.backend: rocksdb
state.backend.fs.checkpointdir: hdfs://namenode:40010/flink/checkpoints

2. Set a per job state backend (which overrides the default setting)
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStateBackend(new RocksDBStateBackend(parameterTool.getRequired("checkpointDataUri")));
Here you specify the path (for example, an HDFS directory like hdfs://namenode:40010/flink/checkpoints) as the checkpointDataURI.

Hope that helps.

Best
Ziyad

Best Regards
Ziyad Muhammed Mohiyudheen 
407, Internationales Studienzentrum Berlin
Theodor-Heuss-Platz 5
14052 Berlin 
Ph: <a href="tel:%2B49%20176%206587%203343" value="+4917665873343" style="color:rgb(17,85,204)" target="_blank">+49 176 6587 3343
Mail to: [hidden email]

On Fri, Jun 16, 2017 at 8:20 PM, Bowen Li <[hidden email]> wrote:
Hello guys,
      I've been trying to figure out differences among several parameters of RocksDBStateBackend. The confusing parameters are:

      In flink-conf.yaml:
      1. state.backend.fs.checkpointdir
      2. state.backend.rocksdb.checkpointdir
      3. state.checkpoints.dir

      and 
       4. the param 'checkpointDataUri' you pass in to RocksDBStateBackend
constructor in`public RocksDBStateBackend(URI checkpointDataUri)`

        This email thread explained the first three well. But what's the 4th one for? What's its difference from others? I'd appreciate your clarification.

       Thanks very much!
Bowen

        

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: confusing RocksDBStateBackend parameters

bowen.li
Thanks for your clarification, Ziyad! I will try it out.

On Sat, Jun 17, 2017 at 3:45 PM, Ziyad Muhammed <[hidden email]> wrote:
Hi,

To set the rocksdb state, you have two options:

1. Set the default state of the flink cluster, using the below parameters in flink-conf.yaml file
state.backend: rocksdb
state.backend.fs.checkpointdir: hdfs://namenode:40010/flink/checkpoints

2. Set a per job state backend (which overrides the default setting)
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStateBackend(new RocksDBStateBackend(parameterTool.getRequired("checkpointDataUri")));
Here you specify the path (for example, an HDFS directory like hdfs://namenode:40010/flink/checkpoints) as the checkpointDataURI.

Hope that helps.

Best
Ziyad

Best Regards
Ziyad Muhammed Mohiyudheen 
407, Internationales Studienzentrum Berlin
Theodor-Heuss-Platz 5
14052 Berlin 
Ph: <a href="tel:%2B49%20176%206587%203343" value="+4917665873343" style="color:rgb(17,85,204)" target="_blank">+49 176 6587 3343
Mail to: [hidden email]

On Fri, Jun 16, 2017 at 8:20 PM, Bowen Li <[hidden email]> wrote:
Hello guys,
      I've been trying to figure out differences among several parameters of RocksDBStateBackend. The confusing parameters are:

      In flink-conf.yaml:
      1. state.backend.fs.checkpointdir
      2. state.backend.rocksdb.checkpointdir
      3. state.checkpoints.dir

      and 
       4. the param 'checkpointDataUri' you pass in to RocksDBStateBackend
constructor in`public RocksDBStateBackend(URI checkpointDataUri)`

        This email thread explained the first three well. But what's the 4th one for? What's its difference from others? I'd appreciate your clarification.

       Thanks very much!
Bowen

        


Loading...