Thanks to @buitregool I was looking in to ARToolKit again which is amazingly LGPL.
I was very surprised with the performance of their Natural Feature Tracker.
Additional it can generate markers from jpg images. Which comes in handy for a more streamlined hybrid object designing process.
As you know the Reality Editor builds up on Open Frameworks (OF) and the proprietary Vuforia AR Library.
Currently there is no OF addon for ARToolKit 5. Which would be needed to use ARToolKit within the Reality Editor.
Do you have interest to help building it?
I have added the LGPL ARTookKit library for iOS and Android in a new github project to start somewhere:
I also started a fresh discussion in the OF forum about it:
We should follow the API design from https://github.com/julapy/ofxQCAR
Lets see what we can get started here.