Runtime-compiled C++ in Urho3D aka "Scripting in C++"

I’ve been working in this a few weeks in my spare time after work. I took the idea of using C++ as a scripting language from http://molecularmusings.wordpress.com/2013/08/27/using-runtime-compiled-c-code-as-a-scripting-language/ and http://runtimecompiledcplusplus.blogspot.com. This is technically called “Runtime-compiled C++”, which I’ll call RCCpp from now on.

Obviously, C++ is not a real scripting language, nor it tends to be. The main goal is to shrink iterations’ time while having a quick preview of what you’re trying to accomplish while using a well-known language such as C++. You have the power of C++ with trusted IDEs, autocompletion, debugger and everything you’d need to work already, so you don’t need either to learn other language nor look for new tools to work with Lua or AngelScript. Of course, you can still shoot yourself in the foot, and have to be very careful about memory leaks, as the idea is to have Urho3DPlayer always running while it loads dynamically a recent compiled version of what you’re trying to do. Ok, enough chit-chat. This is what it looks like:

Enable HD to be able to read the code

To put it simple those are the steps that Urho3D’s RCCpp follows:

  1. If a .cpp is passed as an argument (instead of a .lua or .as), a Makefile is generated on the fly (only GCC/MinGW is supported for now) with all the .cpp’s within that same folder and compiles everything into a shared library. So let’s say we run “Urho3DPlayer RCCpp/Game.cpp”. This will try to compile Game.cpp and all the cpp’s within RCCpp into a shared library called Game.dll (or Game.so if we’re on a Unix system). If instead we pass an already compiled library (.dll or .so), step 2 applies inmediately.

  2. If the compilation succeeds, it will load at runtime the shared library and will execute its “Start” method, the same way it’s done with Lua or AngelScript.

  3. Every time there’s a change in any .cpp being watched (within RCCpp folder, in this case), it will compile again and if it was successful, it will call the “Stop” method of Game, destroy the old Game object, unload the old library, load the new one and call its “Start” method. This has been done this way to avoid freezing the update loop so that the game plays as smoothly as possible. Please note that a total different object is created, so if you would like to have exactly the same state as before, you’ll have to serialize and unserialize it by yourself.

So, right now everything is compiled into a single library. This has pros and cons, but I ended up doing it like that because if I wanted to have a different library for each class (which is desirable in terms of extensibility and speed, so that I don’t have to unload and load all the classes at the same time, just the ones that need to be replaced) I’d have to use a fixed interface (usually Update and Render methods) and have somehow a list of dependencies among classes so that if I change one class, all of its dependant classes are also compiled or at least reloaded. This was quite painful at the end, so having everything in a single library eases things up and avoids some other problems.

I’ve tried to follow the rules that I’ve read in the documentation so that RCCpp is only used if a define is passed via CMake (URHO3D_RCCPP) and so on. I use the FileWatcher to know when a file has changed and needs to be recompiled. I also send useful events when a library/class is going to be unloaded or loaded, when the compilation has started or finished and so. It’s been tested it on Mac and Windows (with MinGW, no Visual Studio support yet) and it should work on Linux too. The only thing you need to make it work is adding your toolchain binary folder in your path so that “make” and “g++” can be called and define an environment variable called URHO3D_HOME pointing to your Urho3D’s main folder so that the include and library paths can be found in order to compile. You could also create a “PreMakefile” that will be included at the very beginning of the Makefile to pass to the compiler/linker your own flags. Take a look into the content of the Makefile generated to find the name of the variables to set.

Lastly, I’d like to hear your thoughts about this. Do you think is useful, do you think is a waste of time, what would you improve, do you have any suggestions? I’ve easily ported all the examples so that the same file and the same code works for the C++ samples as well as for RCCpp using a couple macros. The code is not the cleanest nor the most commented thing right now, but if people are interested, we could clean things up together and polish the bugs that may arise: github.com/pamarcos/Urho3D/tree/RCCpp

1 Like

Interesting idea. Is there any performance boost by scripting this way? Any possible metrics?

Excuse me if I’m being ignorant, but doesn’t this also require the header files to be available? I would probably have to pass on this idea if that was the case, but cool idea none-the-less!

I haven’t run any benchmark myself, but the performance boost would be the same as C++ over AngelScript or Lua. Have a look at codeplea.com/game-scripting-languages. Anyway, a game bottleneck is usually its core (which in this case is already in C++, so no problem here) rather than the game logic. One of the advantages of using C++ directly is that you have the full API 100% available but with RCCpp, you don’t need to worry about large compilation times and closing and opening the game again and again.

Regarding the headers, you don’t really need to pass any header files as that’s taken care of by the Makefile. It adds to the include path every Urho3D folder as well as the same path of the cpp you’re passing. So, you just have to create a class (i.e. Game) that inherits from RCCppMainObject and reimplement its Start and Stop methods. If you wanted to use more classes, create as many as you like that inherit from RCCppObject and then you can use it from your Game class.

I really like the idea of doing this. I build my code around angelscripts hot swapping so doing it in c++ sounds radicool. I don’t have any wisdom to know whether it is a good idea or not. I looked at your code. It looks really good! I will definitely try out your fork sometime this week. Thanks for sharing!!

Super! Going to try.

[after try]
This is awesome!!! Works perfect!!!
Once compiled, you can run the library. Urho3DPlayer.exe Modules/Test.dll -w This should be in the Urho3D!

This is certainly very cool. For inclusion into the official Urho3D repo, I’m not sure. We’d have to think carefully of what application / execution model we want to impose (currently Urho3D doesn’t really impose one for C++), and if reloading the whole “game” is enough, or should we need per-component reloading?

Very nice addition here for direct C++ coders in the engine! I’ve been watching this framework for some time now and its really cool to change C++ code in run-time for development/testing. I would recommend though having this as a build option or “add on” if the core team decides to add this to the repo.

This is extremely cool :slight_smile:

I think it could be extremely helpful for us C++ developers in some situations.

Of course, the ability to use Urho3D as an external library needs to be preserved, but that seems to be the case (?).

So…as long as it is optional, I am definitely in favour of this.

PS:
I am having some trouble getting it to work on Linux, though (though, that might just be me…). It detects the changes properly and compiles successfully and seems to “restart” the application but it doesn’t display the effects of the changes.
I can only see those once I restart Urho3DPlayer itself (even if I pass the library itself as an argument, so it appears to compile correctly).

I believe that there might be an issue with loading the new library.

RCCpp is compiled as a separte module the same way as AngelScript, Lua or some others by defining something during the CMake process. In this case, URHO3D_RCCPP needs to be set.

Yes, in fact if RCCpp is enabled Urho3D is compiled as a shared library so that your game library is linked against it.

[quote=“AGreatFish”]I am having some trouble getting it to work on Linux, though (though, that might just be me…). It detects the changes properly and compiles successfully and seems to “restart” the application but it doesn’t display the effects of the changes.
I can only see those once I restart Urho3DPlayer itself (even if I pass the library itself as an argument, so it appears to compile correctly).[/quote]
Sorry about that, I didn’t have time to test it on Linux (because I didn’t have a Linux partition working at the moment). Have you compiled the debug version to see what RCCpp outputs through stdout? There’s also a Build.log to check that the compilation worked, although a compilation error doesn’t seem to be your issue the way you described it. Take a look at RCCppUnix.cpp and try to write some logs in case dlopen failed. Something like this:

[code]bool RCCppUnix::LoadLib(const String& libraryPath)
{
library_ = dlopen(libraryPath.CString(), RTLD_LAZY);
if (library_ != NULL)
{
String name = GetFileName(libraryPath);
createObject_ = (PCreateRCCppObject)dlsym(library_, String(“create” + name).CString());
destroyObject_ = (PDestroyRCCppObject)dlsym(library_, String(“destroy” + name).CString());
}
else
{
LOGDEBUG("Error opening library " + libraryPath);
}

if (library_ == NULL)
{
    return false;
}
else
{
    return true;
}

}[/code]

Anyway, I’ll try to set a VM with a Linux version to test it out. If you take a look at RCCppWin you’ll notice that I have to do some kind of hack in order to hot-swap the library. The compilation fails to create the library because it’s still being used (which doesn’t happen un Mac). Funny thing is that whereas you cannot change the content of the library itself, you can change the name. So what it’s done is: change the old library name (let’s say from Game.dll to Game.dll.old), compile the new code to Game.dll, unload the old library and load the new one. Once this is done, the .old version is removed. However, this doesn’t seem to be your issue, as if this was the case, the compilation would fail during the linking process.

Thanks everyone for checking this out. There’s still some work to do for a perfect integration with Urho3D, but I think we’re heading in the right direction.

According to the log, the compilation is successful.

I also don’t think that dlopen is the problem. I set debug messages to confirm this and I set a breakpoint and inspected the values of library_.

[quote=“AGreatFish”]According to the log, the compilation is successful.

I also don’t think that dlopen is the problem. I set debug messages to confirm this and I set a breakpoint and inspected the values of library_.[/quote]

At the end Urho3DPlayer wasn’t working well on my VM so I installed Linux in a brand new partition. I’ve had exactly the same problem that you described. It’s been a while until I’ve found the issue. It seems dlclose is not closing properly the library and so, every time dlopen is called it returns exactly the same handler (which is the handler to the very first version of the library compiled) so nothing really changes in the game apart from destroying the main object and creating it again. If you take a look at the documentation at linux.die.net/man/3/dlclose it says:

As far as I understand, it seems that the reference count system is not working as it should or something I can’t understand is happening. In Mac is working well, though (which makes sense, because there’s only one dlopen-dlclose pair). I’m not very proud of what I’ve done to make it work, but it’s working and I think that’s enough for a day. I hope it works for you as well.
Give it a try: github.com/pamarcos/Urho3D/comm … 9ec3c09a5c

i don’t know if this helps… but sdl has stuffs

wiki.libsdl.org/SDL_LoadObject
wiki.libsdl.org/SDL_LoadFunction

I just did a quick test before going to sleep and it works now :slight_smile:

And it seems even cooler after actually having used it !

Will take a closer look at it tomorrow :wink:

Doug here Runtime-Compiled C++.

In our RCC++ implementation we only compile changed files, so load these as new modules with a temp name to avoid name clashing and so we can keep the old modules open (as they may contain code still being used). The solution you’re using for Linux looks the same as the one I’m using.

Let me know if you have any further problems, and I’ll see if I can help - though I’m travelling for work atm, so there may be some delay.

I noticed a funny effect:
Since Urho3D is still running it appears that some things persist after recompiling.
E.g. when I change the position of UI Elements, the elements appear two times. Once in the old position and once in the new position.

[quote=“AGreatFish”]I noticed a funny effect:
Since Urho3D is still running it appears that some things persist after recompiling.
E.g. when I change the position of UI Elements, the elements appear two times. Once in the old position and once in the new position.[/quote]

That’s right. RCCpp only cares about compiling, destroying the object, reloading the new lib and creating the new object. So, you have to take care of everything else by yourself. If you added something to the UI’s root, you’ll have to remove it either in the Stop method or in the destructor of your RCCppMainObject so that when the new one is created, you don’t have duplicates. I thought about cleaning the whole UI after every time your lib is unloaded, but that would clean absolutely everything and there may be cases where that’s not the desired behavior (e.g. in the samples the Urho3D logo would be removed also).

The easiest solution for the UI is adding everything as children of a UiElement that you remove in the Stop method so that the next time the lib is loaded, only the new UI-related stuff is shown. This is what I did in the video with the Urho2DSprite example.

Summarizing, using RCCpp is cool and useful, but there are times where it requires an exta step where you have to make sure that everything you have created/added etc, is removed in your Stop or destructor so that the next time the lib is loaded you don’t have duplicates.

I wonder if instead of firing of the Start event we could create a virtual method OnReload and publish an event that could cancel the reload, the OnReload would just call Start likely but it could allow the logic to grow. I am guessing there are too many things that could happen after initialization for urho to gracefully handle object reloading. We could set up some conventions with the vars/attributes but its hard to know if those would work.

Our RCC++ framework has an object initialization where we pass in a boolean demarking whether the init is being called on a new object or on one being re-loaded. I’d be tempted to go for three separate functions in future, one for first time init, one for reload init (object hasn’t been recreated or had code changed) and one for recreated object.

Right now the Start is always called. Adding an option to avoid reloading everything by RCCpp and leaving you to do it whenever you want should not be difficult. Having said that, there are 6 different events that RCCpp is already firing to give you the possibility to clean things up your way. Those are: E_RCCPP_COMPILATION_STARTED, E_RCCPP_COMPILATION_FINISHED, E_RCCPP_LIBRARY_PRELOADED, E_RCCPP_LIBRARY_POSTLOADED, E_RCCPP_CLASS_PRELOADED, E_RCCPP_CLASS_POSTLOADED. So, you can still set up a listener to any of those methods in order to do something before the library is really reloaded. What you can’t do right now is stopping the process to reload it whenever you want. I’ll give it some thoughts of how that could be done in an easy way.

[quote=“DougBinks”]Doug here Runtime-Compiled C++.

In our RCC++ implementation we only compile changed files, so load these as new modules with a temp name to avoid name clashing and so we can keep the old modules open (as they may contain code still being used). The solution you’re using for Linux looks the same as the one I’m using.

Let me know if you have any further problems, and I’ll see if I can help - though I’m travelling for work atm, so there may be some delay.[/quote]

Good to see you here, Doug :slight_smile:. I was mind-blown the first time I visited your webpage. It’s quite impressive to see C++ compiled and loaded at runtime. I mean, when you think about it you realize it’s technically feasible to do it, but still impressive. I tested the samples and so, but unfortunately I haven’t had much spare time to look at the code to know how everything works.
For the very first version of RCCpp in Urho3D, instead of using a Makefile system that would compile everything into a single library, I thought of having a different shared library for every class. That in terms of modularity is desirable, but the “one-single library” approach won in terms of simplicity. The overhead of loading the whole library for a single class is still there, but I avoided the following issues:

[ul]

  1. Compiling everything the very first time. Makefile does the work compiling only the necessary files, because otherwise I’d have to implement some kind of file date checking to see if the libraries are up-to-date and compile only the files needed.
  2. How do you solve dependencies among libraries? You have to specify one way or another what other libraries you depend on so that when some of that libraries is compiled, you create a new oject of that kind. I thought of having a Class.rcpp file with that kind of information, or embedding some commented code into any class’ cpp file (yeah, that would be specially ugly) specifying that and parsing it from RCCpp. Another option would be to parse every file and recompile everything including Class.h. I didn’t like any of the solutions, though.
  3. By compiling everything, destroying everything and creating it again I ensure that every object is an up-to-date version.
    [/ul]

I get that by creating temporary names for the libraries you can still use the old ones. But why would you want to use an older version of the library/module? Shouldn’t every class reload the new version of the class? Or you simply let the coder choose whether he wants to reload a new version or not? Another question I have is how do you handle exceptions in Mac and Linux where you don’t really have SEH.

Thanks in advance. Cheers.

@Pablo - glad you like the demos! Our objective with the RCC++ project was to change people’s minds about how C and C++ can be used, and it seems we’re getting there. I’ve added Urho3d to a list I keep of projects using RCC++.

You’ll find answers to some questions on our wiki, and there’s a chapter on RCC++ in the book Game AI Pro if you can get hold of it.

  1. Our RCC++ implementation is designed to be used to re-compile code as fast as possible. We avoid needing makefiles etc. as the developer uses their existing infrastructure to compile the code in any way they like, and RCC++ simply handles re-compilation at runtime when a source file is changed.

  2. We solve dependencies by embedding the information in the compiled code. See the wiki on includes, libraries, modifiable headers and source dependencies. Recently people have started to use RCC++ for scripting, where they want to compile new code and I’m looking into ways to make this easier. Personally I prefer using my normal tool chain and just using RCC++ for changes.

  3. We still perform a full data reload (in memory), but keep the compiled and linked code as small as possible. Linkage is often a rate-determining step so this helps considerably. Recently I’m finding the full-reload can be more lengthy that it needs to be so I’m considering adding support for only reloading data for modified files.

  4. We need to keep old libraries around as our implementation compiles changes and dependencies into a single dll per ‘project’ (most users only need one project). So a library may contain code from a.cpp and b.cpp, then when b.cpp changes we get a new library only containing the code from b.cpp, so need to keep the old library for the code from a.cpp. In many cases, the original code is compiled into the executable, so obviously that needs to be kept around.

  5. For error protection outside of windows we use signals. Take a look at the wiki page on error protection and code for RuntimeProtector.h and implementation in RuntimeObjectSystem_PlatformPosix.cpp

Remember that since our RCC++ implementation uses a liberal license, you can borrow as much or as little of the code as you need for Urho3d. I’m travelling atm, so may be a bit slow with replies but please continue asking questions as you need!