I’m sensing this is related to your other question about adding a method to the RuntimeContext. Would it be possible to extract the resources from the Jar when submitting the program and placing them in the distributed cache? Files can be registered using StreamExecutionEnvironment.registerCachedFile(String filePath, String name) and retrieved using RuntimeContext.getDistributedCache().
My resource file is python or other script reference to each other by relative path.
What I want is all my resource file in one job place in one directory.
And the resource files in different jobs can't place in one directory.
The distributedCache can not guarantee this.
I think the code that submits the job can create an archive of all the files in the “resources”, this making sure that they stay together. This file would then be placed in the distributed cache. When executing the contents of the archive can be extracted again and be used, since they still maintain the same relative folder structure. What do you think?
So your suggestion is I create an archive of all the file in the resources.
Then I get the distributed cache of this file and extracted it to a path.
Use this path as my resource path?
But in which time I should clear the temp path?