User self resource file.

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

User self resource file.

yunfan123
For example, I use
./flink run flink_helloworld.jar
the flink_helloworld.jar contains a resource folder in the root dir.
How can I get the resource file in flink.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User self resource file.

yunfan123
Or how can I get the blob store of my jar file.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User self resource file.

yunfan123
It sames not supported by flink?
I think there should a function like getBlobStore in the RuntimeContext interface.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User self resource file.

Aljoscha Krettek
I’m sensing this is related to your other question about adding a method to the RuntimeContext. Would it be possible to extract the resources from the Jar when submitting the program and placing them in the distributed cache? Files can be registered using StreamExecutionEnvironment.registerCachedFile(String filePath, String name) and retrieved using RuntimeContext.getDistributedCache().

Best,
Aljoscha

> On 14. Jun 2017, at 08:47, yunfan123 <[hidden email]> wrote:
>
> It sames not supported by flink?
> I think there should a function like getBlobStore in the RuntimeContext
> interface.
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/User-self-resource-file-tp13693p13720.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User self resource file.

yunfan123
Yes.
My resource file is python or other script reference to each other by relative path.
What I want is all my resource file in one job place in one directory.
And the resource files in different jobs can't place in one directory.
The distributedCache can not guarantee this.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User self resource file.

Aljoscha Krettek
I think the code that submits the job can create an archive of all the files in the “resources”, this making sure that they stay together. This file would then be placed in the distributed cache. When executing the contents of the archive can be extracted again and be used, since they still maintain the same relative folder structure. What do you think?

> On 15. Jun 2017, at 13:53, yunfan123 <[hidden email]> wrote:
>
> Yes.
> My resource file is python or other script reference to each other by
> relative path.
> What I want is all my resource file in one job place in one directory.
> And the resource files in different jobs can't place in one directory.
> The distributedCache can not guarantee this.
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/User-self-resource-file-tp13693p13762.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User self resource file.

yunfan123
So your suggestion is I create an archive of all the file in the resources.
Then I get the distributed cache of this file and extracted it to a path.
Use this path as my resource path?
But in which time I should clear the temp path?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User self resource file.

Aljoscha Krettek
Yes, this is what I’m suggesting. I think you could clear the path when the operator/function shuts down, i.e. in the close() method.

> On 15. Jun 2017, at 14:25, yunfan123 <[hidden email]> wrote:
>
> So your suggestion is I create an archive of all the file in the resources.
> Then I get the distributed cache of this file and extracted it to a path.
> Use this path as my resource path?
> But in which time I should clear the temp path?
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/User-self-resource-file-tp13693p13765.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Loading...