Adding Relative Paths to Resource Loading

This thread is meant to discuss the addition of relative path support to the engine, specifically regarding resource loading (and to keep the EASTL thread from getting off topic). The proposed change allows the grouping of resources more easily by map section or character, so that within, for example, “Map/World1/Level1/node.xml” resources can be identified as “./tree1.mdl”, “./tree1leaves.png” without having to write out the full path for every resource. It also makes some situations of getting resources from the code simpler (I want the clothes.png file from the same directory as this character specification file, for example). Present behavior would be preserved for the most part (technically it would change, as relative path directories ../ and ./ are just stripped from the requested paths, but that I hope is not a feature anyone is relying on or using) – requesting ‘Models/Box.mdl’ would still give exactly the same result, but now we could use ‘./normal.png’ from a material rather than having to specify a different directory (which I feel is a much better resource layout for an actual game where textures will largely not be re-used between characters and such, though not so much the samples where the resources are rather heavily re-used so dividing by the type of resource is makes more reasonable).

Existing resource loading will be changed (all of the GetResource type functions), requiring an additional parameter (a base path) or switching the String resourceName (i.e. path) parameters to a separate Path class so that the relative paths can be resolved. The extra parameter will be inserted before the bool argument as it should be grouped with the actual resource path requested rather than whether an event is sent on failure. It will have a default empty string parameter, so only the case of specifically requesting an event not be sent on failure will existing user code need to be changed (and possibly also custom resources will need to be modified).

In terms of a cost-benefit, we gain support for relative paths at the cost of a slight increase in complexity in searching for a resource, which is likely not a bottleneck in anyone’s programs, so this should be fine.

I submitted a Pull Request for this addition two and a half years ago that is still open, so with this thread we can hopefully update and merge that shortly (since then I have let it grow out of date with master, but since there seems to be some interest I’ll go ahead and bring it up to date over the next couple of


There are a few implementation details to discuss, you can see a couple mentioned in the PR comments. One major one that might not have been mentioned is how we should add the base path parameter. I went with just adding a second string, as you may have noticed, but I also favor the addition of an actual Path class in place of just using strings. This Path would either store both the relative and base path itself, or it would just store the final resolved path (not truly an absolute one since we’re still using it to search the resource directories) when constructed from a relative and base path strings. If it were implicitly constructable from a String, I believe it should not require any changes to the code to switch to such a class.

If we went with that, we might want to opt for including this in Urho’s 2.0 release, and we could also go and move a lot of path-related features into the class (see the resource loading code for some of what I mean).

That said, a separate path class seems rather unnecessary to me since strings work perfectly fine for it, and then you have to decide implementation details like do we store the path as a String with the full path or as a StringVector of every component.

2 Likes

We can go another way. The paths to loaded files (base path) user can store in SetVar(). We need only allow users to specify a proxy function (callback), which will modify path before loading.

actually we have AddResourceRouter() I haven’t watched yet if this is suitable

Thanks for the link, I checked it.
I remember now why this PR was not merged back then.

I see the following issues with this approach:

  1. The only way Urho support “prefabs” now is via Scene::Instantiate*.
    However, if you don’t have any mechanism to set base path for node in this case, you will be unable to load prefabs with relative paths. And this is one of important use cases – to instantiate prefabs.

  2. However, if you fix (1) by changing signatures and setting base path, redundant realtime memory consumption will be introduced. It is a lot of wasted strings to keep file name for each node. Nodes in Urho are already fat and there were attempts to make them thinner, not fatter. There are solutions for this type of problem, but I’m not sure if this approach will help here.

  3. Relative paths are very fragile. The only way you can create them is to manually edit text files.
    The moment you save anything via Urho facilities, relative paths are gone. It is especially painful for e.g. materials – the moment you use editor relative paths are ruined.

So far, it doesn’t look as self-consistent feature. It will start falling apart the moment you touch it wrong way.

I don’t have any non-radical ideas now, but I will post here if I come up with anything.

If I remember correctly, Resource Routers are not sufficient at present since it is impossible to determine who is requesting the resource, and beyond that the …/ portions of the path have also been sanitized. This sort of approach doesn’t sound too bad - we add a third parameter to the Resource Cache along the lines of an Object*

For relative paths I have already done it (see point 6). Updating it to work with master is an afternoon’s work probably, but I’m pretty busy with other stuff at the moment so it might take a few weeks to get to.

No, it wasn’t that hard to do. I’m not sure what you mean by manually? I don’t use an external library if that’s what you mean. It’s a pretty simple algorithm - if the path starts with ./ or …/ you add it to the base path directory and then split by /, read from outer to inner directories removing all . and removing the top directory whenever there is a …

For adding spans I’d estimate tentatively a month. How I intend to do it remains to be seen, but since I’m not asking anyone to help with that the method doesn’t matter as only the result will affect anyone else.

No idea what this is, and I couldn’t find it with a quick Google search. It sounds more like an alternate package file, which is not what my proposal was about (but is what a different pull request of mine is about).

This would apply to very large changes, but I would argue that learning if the changes would be welcome should still be done before this step. In any case, I’m not being paid for this, it’s literally just because I want these features and I’m willing to share since others might too. Sometimes it’s less than that, more of I kind of want this feature, and I’m pretty sure others do as well, and I like programming, so I’ll try this, particularly if others do actually want the feature. Also, I don’t really think task allocation fits with a volunteer-based open source project. Who exactly are you allocating tasks to? (To further illustrate my point, and not because I think it will happen: Are you going to tell my that I should work on some other feature like improving networking or PBR because those are considered a ‘higher priority’?)

None of the things I’ve suggested are a wishlist - I wish someone would do the work to make Urho have this feature. They are either things that I intend to do, or they are things that someone else has already done 90% of the work for and has volunteered to finish up if the community is interested.

1 Like

Are relative paths supported in the Scene in your implementation? If so, where do you take the paths from?

Each node stores it’s own file’s path (and one or two other things deriving from Serializable), and when loading resources they pass their own path if they request another resource. I don’t load scenes, I load lots of nodes (for map sections, for characters, etc.) so I believe it would still work perfectly fine with listing scenes but I’ve also not rigorously tested that aspect.

If we want to continue discussing this it may be best to start a new thread or continue in the comments on the PR so we don’t get too off the topic of EASTL (though I understand this is also basically my fault).

1 Like

Do you maybe have prototype code online? I don’t completely understand how it is implemented just from your words and I’m curious about certain aspects of implementation.

I’m not sure what you mean by not a self-consistent feature, and I do think you will always have relative paths fall apart the moment you touch it the wrong way (as soon as you move a file to a different folder it falls apart, for example). The only reason it doesn’t fall apart like that as we have things now is because there is no such feature.

To try to address your issues:

  1. Yes, nodes will have to store paths. This could be done in the User variables or in the NodeImpl class (I went with the latter since it is not a User variable but a library one and this should be slightly more memory efficient than storing it in the HashMap).
  2. This could be addressed by only having the parent node of the prefab store the path, but at the cost of a significantly increased lookup time, or by storing the paths in a global HashMap (HashSet?) and only storing the key in the Node, which adds a global variable but saves memory while not being that great an increase in lookup time.
  3. Firstly, so what if they are fragile. MDL files are also very fragile, once the model has been imported the only way to get it out to a usable format is to export the Drawable to an OBJ. The point is not to make the filenames of the resources shorter when they are referenced (they’re compressed anyways in package files so the difference should be pretty negligible), but to make creating the files referencing the resources easier. While the present architecture prevents us from keeping any sort of ‘this was a relative resource loaded from X’ information with the resource (as that would render the cache less useful since you’d have to keep a new copy of each resource for every possible relative name, as opposed to just every possible full name as it is now), it is possible to generate relative paths just as much as it is to load them (but then you run into issues of should it be ../../LevelResources/Tree.png or just Maps/LevelResources/Tree.png when saving). To the best of my knowledge, any comments in the XML file are also not preserved when saving from the editor, or which values were just left as the default, so I don’t have any problem with this fragility: Use the editor, lose the commented out old diffuse texture. Likewise, use the editor, lose that it was ‘./Skin.png’ as opposed to ‘Characters/Warrior/Skin.png’.

Bad example. MDLs (as Model) does not support save at all.
However, Material does support save. Moreover, it is guaranteed that if you load material you can save it and it will be functionally the same. This guarantee is critical for proper work of tools like material editor and so on.

Firstly, so what if they are fragile

When user try to use material editor on the material from their prefab, save it with or without changes, and get relative paths corrupted, they will report a bug and I wouldn’t argue with them.
Why “corrupted”? Because user will think that everthing is ok until they try to move their folder with relative (as they think) paths. Then things will explode.

I mean amount of places marked as TODO or “doesn’t work”, and general complexity in API changes.
This is a sign that currently this feature doesn’t work together with the rest of the code, but against it.
Do you know this feeling? When you are trying to solve some problem, but every fixed issue greatly increase complexity of changes and you feel like you are sinking deeper and deeper?

I think I just found a chunk of code that is probably screwed due to changed meaning of “name”, and there may be more
https://github.com/urho3d/Urho3D/pull/2070/files#diff-7b9673cfc396212ff55f1c1bee79ab6eR1342

If we claim that relative paths are supported, we need to support them as well as absolute paths.
I.e. if I load scene from file in any way and then instantiate prefab (in any way), I expect relative paths be resolved, if their support is declared.


This part is just a speculation

I believe that most of the complexity here are caused by the fact that relative paths are handled by scene hierarchy, while said hierarchy should not care about relative/absolute paths at all. Paths are responsibility of Resource subsystem.

Generally speaking, Node does not have a file name. Only special types of nodes have it.
And I don’t mean there should be empty strings for the rest of the nodes. There should be no string at all.

In perfect world Scene and Prefab should be Resources and handle all relative-path stuff on their own, leaving Node and Serializable untouched. All paths coming to scene hierarchy should be already converted to absolute.


PS. I’m sorry beforehand if you consider my replies nitpicking for some reason. I’m just reviewing your code with the exactly same attention to details as I review with my own code I merge into master of Urho (or now rbfx), or as I review the code of my colleagues. I just cannot do it in any other way and ignore issues I consider important. Maybe others will find this solution appropriate.

3 Likes

Supporting relative paths for resource dependencies should be possible without modifying function signatures in a way that breaks backwards compatibility. Maybe the ResourceCache and FileSystem could work together to achieve this by keeping track of the master resource location, which would then be used to supplement relative paths. When saving a resource I imagine the option to save relative paths could be passed as a bool which defaults to false. The resource itself would not have to be aware of its path’s relativity.

1 Like

@Eugene you’re probably right about the mdl’s, saving them is not something I’ve ever wanted to do. However, I disagree with your critique regarding the editor, if the resource is simply loaded and saved it will behave exactly the same. It is only if the file is then moved that there would be a difference in behavior, which I don’t see as an issue. (I also don’t really like or use the editor, and wish we would just make the blender exporter slightly better to use it as the default editor, so feel free to take that as you will). If I’m not mistaken (I also haven’t tried it in the editor to see), the texture paths displayed for the material will be the full paths, so there should be no confusion.

Thank you for the clarification about what you meant by self-consistent feature. I see your point, and I mostly agree - at present the feature is half-baked (maybe a bit more than that, but certainly not polished and complete). I don’t see the broken thing in the diff, but I also didn’t scroll all the way through it so you’ll have to be a little more detailed. If it’s using the resource name as a path I disagree, as the name is set when it is loaded based on the path, but I have no idea if that was what you were meaning. If it’s about the JSON support, I’m fine with fixing it, but we have to add a WeakPtr to the JSONFile to every JSONValue just like we can access the XMLFile from an XMLElement (or you may propose an alternate solution).

I half agree, as generally paths are the domain of the resource cache. I half don’t, because Nodes and Components are Serializable, and can thus be deserialized from some source, which generally implies a path (potentially a network update).

Sounds mostly doable to me, if we wanted to. We have a prefab resource that will load a node, we add an optional basePath parameter to Serializable’s Load functions*, and it passes those on to the resource cache whenever is loads something. We add a Prefab component or variable to the node that keeps track of whether the node was created from a prefab so that on Serialization the node gets inserted into the file as a prefab instead of the full heirarchy for it. Saving the prefab is done through the prefab resource and not the node itself, just as the material xml does not get embedded in the scene xml (nor does saving the scene save changes to the material).

*Yes, this does mean the paths aren’t absolute before the node gets them, but it also makes it possible to load a node from an arbitrary base path from code, which seems better to me than forcing the user to create a prefab resource to support any relative paths in the node. But we may also have slightly different ideas of what a prefab is.

Addressing your PS, no, I don’t really consider them nit-picking, just detailed feedback. It’s a good thing that everyone knows in more detail what is being proposed, and also good to work out how we want to address the details I haven’t. I don’t share your polish-then-share philosophy, I finish it until it does what I need it to and then the edges I haven’t gotten to yet (like polishing the JSON aspect) I will finish with feedback from the community (then I don’t have to write a second implementation if they didn’t like the first.

@Modanung it’s entirely possible to do without breaking backwards compatibility, we just use multiple overloads of the functions (unless everyone else insists, I refuse to separate the name (path) and base path parameters in the arguments list since they are closely related. Alternatively, we go with making a separate Path class implicitly constructable from a String (with optional base path). As to “keeping track of the master resource location” I assume you are envisioning some sort of stack that would be queried so resources that request other resources pick the right base path? If so, I don’t think it would be a good idea because 1) that seems to result in Serializable’s like Node not being able to use relative paths and 2) it would make it harder to load a resource from a base path from code.

Also, I like the proposed saving solution.

For the record, I favor the class Path approach, as it would be easier to add features to later, but it is also more work and a bigger change. I’m willing to make those changes if you guys want, but I don’t intend to unless you think it would be accepted, as what I have already meets my needs.

1 Like

In fact, I see no problem in storing ABSOLUTE file path for a loaded resource. Just as a user can save a text document with Save button, the user may want to save the previously loaded scene / node / material. A pointer to a string does not greatly increase size of the class. At the same time, I do not see anything wrong with the fact that when loading a resource, is passed additional information about who requested this resource (pointer to an object as SirNate0 said earlier).

1 Like

Oh, sorry, I always forget that github diff links are wonky in big PRs.


You see, this code assumes that if you pass sanitated resource name into ResourceCache and get back a resource, the name of said resource would be exactly the same as you passed.

Moreover, sanitation kills relative paths, so this code just will not work.

Actually, it may be a sign that you don’t need basePath support in all functions of resource cache. You need it only in SanitateResourceName that will resolve relative paths into absolute. Then you just use this function to get absolute resource path every time you may have relative one.

I have to say that I didn’t read this thread fully and I didn’t realize the problems,yet. (I will take some time after work). But I have another suggestion.
Wouldn’t it be possible to let the resource-cache create sub-resourcecaches as needed. Every single one of them would again be a resource-cache pointing to the relative folder. They would be managed by the parent-resource-cache and would need to be connected in some way to the parent… (if that is needed at all)
Any resource accessing object would have an additional relative resource-cache poiting to folder it was loaded from. On any resource access we would need to check if a relative prefix is present or not and choose between the relative or the global resourcecache.
I’m not sure if that approach is possible at all,maybe there are some oppions and hints from the more experienced users before I start looking deeper into it.

Would (relative) paths not always reside in XML or JSON files? I think relative paths could be fixed as they are read. There is no need for a Path class in that case.

For the sake of consistency, functions that refurbish paths should reside over at the FileSystem. Have a look at these functions in FileSystem.h. Maybe something like String AppendRelativePath(const String& relativePath, const String& mainPath) and String GetRelativePath(const String& path, const String& from) could be added to the list and used when saving and loading resources.

I don’t think adjusting the paths as they are read is a good solution, because that makes the XMLFile and JSONFile very specific to Urho – the code has to determine what is considered a path and what is not, and then selectively changes some of the content accordingly. If you are simply advocating we add another GetInt type method that is a GetPath, and patch it based on that I would be more comfortable, as then it is the code that is using the XML/JSON to save the file and read it and the XML/JSONFile doesn’t have to be aware of what is a path and what is not.

I do think adding such a function is the right idea – I believe I added what would be the AppendRelativePath with a different name already, though I don’t think I did the reverse (creating a relative path from two absolute ones).

TODO: Replies to other posts (I’m too busy right now to get to the rest, sorry guys).

1 Like

Well, they are Urho3D Objects.

I’m not sure what you mean by that. Since the paths would point to resources from inside another resource - to which the path is relative - aren’t the relevant values expected to be paths anyway?

How would you find relative paths in generic XML or JSON without known structure?
It might be material, or node, or cube texture. Or something else, like custom resource.

2 Likes