(Discussion) Path moving forward for UI/UX redesign

[josh-audio]

For some background, read the discussion on this PR: #4367. If you have no idea what I'm talking about, see this issue: #1911.

---

I have become very interested in the UI redesign as of late. I would like to see it happen, and I am personally willing to put in the grunt work to make it happen. Things may get more chaotic for me in a few months when school starts back up again, but for the time being I'm in a very stable position with a great work-life balance, and I have plenty of free time to work on projects like this.

As I've thought about it, and as I've talked to various people, I've realized that doing this the right way will involve some major changes to the internals of the software. There are multiple paths forward that we could take, but I believe it is crucial that we take the right one, or at least that we take one that is carefully thought out by all those involved. This is not a project that should be hacked together.

As I see it, the paths we have are as follows. Please add your own ideas if you see more options, and I will add them here.

1. Rewrite the entire software.

This option has its advantages and disadvantages, but as I see it the disadvantages outweigh the advantages. If we had a smaller codebase and a larger team this might be feasible, but a rewrite of a codebase this large would be a monstrous undertaking. In addition, we would need a lot of organization and planning to keep this from making the software worse. I don't believe this is the right way forward, but I'd be happy to be convinced otherwise.

2. Build upon the existing toolkit.

This is the option that I'm currently leaning towards. It means that anyone can start right now with making new widgets based on @budislav's design and make meaningful progress in smaller increments. We will need to put some thought towards some of the core, and we will need to enforce uniformity as well as good design patterns, but that will be very doable. We will also want to put some thought towards things like high DPI displays, as well as how we're going to deal with the core code, but these are all things that can be handled in isolation from one another to some degree.

I think it's crucial that we create a framework for GUI design instead of just a few widgets. We should be able to add various panels by adding a widget instead of forcing each plugin to do the work themselves. This will make compliance to the new UI much easier in the future. I also think we should avoid images as much as possible, but that's a potential debate that I'm not qualified to participate in.

3. Rewrite the UI system.

This is somewhere between options 1 and 2. In the PR I linked above, @fundamental mentioned that his rewrite for Zyn used a custom layout engine and metadata to connect everything together. I don't see an easy way to break this down into smaller projects, but I could be wrong.

---

Please share your thoughts! There are important decisions to be made here which will affect much of the current codebase, and as a consequence, almost all future development. These are decisions that I canont and should not make on my own.

[josh-audio]

Adding on to option 2, here's a comment from @michaelgregorius on this thread. I hadn't read it until now, but it lines up very well with what I was thinking.

The following steps might be feasible to get this one finally going without getting entangled in the existing LMMS code base:

1. Acquire the assets from @budislav.
2. Add a new library to the CMakeLists.txt which only has dependencies to Qt. This library has no connection to LMMS and its functionality.
3. Implement the different user controls that are needed to compose @budislav's design in that library by subclassing existing Qt Widgets and adjusting their style or by implementing completely new widgets. Use @budislav's assets here and keep the library as generic and reusable as possible. Don't add LMMS specific functionality here.
4. Test the GUI components of that library by adding a new GUI prototyping executable to the CMakeLists.txt which only has a dependency to the new GUI library and Qt. This executable acts as an experimentation playground that can be compiled quickly without having to compile the full LMMS code base. At a certain point it will likely look a bit like the GTK3 widget factory in that it showcases all implemented new widgets.
5. Once most of the widgets are implemented add the new GUI library to LMMS and make the existing LMMS widgets subclass from the classes in the library. Add the LMMS specific functionality, e.g. signals and slots, here.
6. Slowly morph the existing LMMS GUI design towards @budislav's design.

This might also be something nice for an LMMS Git Project as it has several characteristics of a project:

• It must be planned well.
• Several people should collaborate on the tasks.
• It will take a longer time.
• It might succeed or fail.

One nice thing with the outlined approach is that the existing LMMS will not be affected in case the project does not make it to step 5. Because a new library and executable is used it is also possible to implement this in master without the need for a separate branch.

Unfortunately I don't have time to manage such a project but perhaps someone else wants to step into this role and use the steps described above as a first project outline.

I am interested to hear your feedback.

I would be willing to take on a project lead role, regardless of what path we choose moving forward. I would also certainly be open to someone more qualified stepping up to the plate.

[fundamental]

This week I'm going to be giving a talk at LAC on the MRuby-Zest framework and several parts of that presentation (and to a lesser extent the details on the rtosc talk also being given at this LAC by myself) are quite relevant for this sort of planning discussion. Rather than just saying "Watch the talk and/or show up at LAC if you're in the area" (which you should do anyhow since LAC is pretty darn cool) I'll provide a preview based upon my current quasi-script I have for myself:

Hello. I'm X and today I'm going to be talking about the MRuby-Zest framework which was used to build the Zyn-Fusion GUI for ZynAddSubFX. This framework does a number of cool things, but before getting to any of the details of the library I want to address one question that's likely on the mind of many of you after hearing it's a new GUI framework.

<change slide>

A new toolkit? Don't we already have a bunch of those?
Well, yes.

<advances-slide>

There's Qt, GTK, fltk, avtk, robtk, JUCE, etc. There's a ton of different options out there.
Then if that's the case, then surely it's better to build off of one of those, no?

Well, that's exactly what I thought before building MRuby-Zest. There's a reason why in the domain of audio applications that you see a number of different toolkits or micro toolkits appear. It's that audio applications pose some interesting and somewhat unique challenges both from a design and an implementation perspective. In an audio application you're generally going to want to present a fairly large amount of fairly complex information to a user in a way that's concise, easy to understand, and easy to manipulate. That's not an easy task, but on top of that the tool needs to present some unique (but self consistent) visual and interactive feel to it.

<changes-slide>

Consider building a simple synthesizer with what tools an existing toolkit provides. You might start out by adding a keyboard to play some test notes, then you might want to see how loud things are by adding a VU meter. Then you would need to control something with a slider, or knob. And to fill out the GUI you might want to show a waveform. At this point you can see that building off tools that a toolkit already has can result in accomplishing each individual goal, but failing the broader goal of having a consistent visual language as the GUI we've just seen built up ends up being a 'Junkyard Front Panel'.

<changes-slides>

So for building a GUI, what are the core features that we need to look for. Well, it needs to be easy to embed, which isn't always the case for popular toolkits like Qt/GTK. It needs to provide some of the basics for audio applications and it needs to be fast to develop in. For ZynAddSubFX this last one was absolutely critical.

When it came to ZynAddSubFX there wasn't much time between other tasks to implement a new GUI. While I was the one who spent most of that time I do have to reveal something. "I'm bad at GUI programming"... and I'm sure I'm not alone in that. What that meant for MRuby-Zest is that it had to help with development otherwise Zyn-Fusion was never going to happen.
In practical terms that meant a good number of things, but let's look at three main characteristic goals.
First the toolkit needed to support the 'do not repeat yourself' principle. If there was a lot of repitition it would drag down the writing of the code and make it take too long.
Second, there had to be a very fast feedback loop within development. As I mentioned earlier, audio GUIs pose unique challenges which means they have a huge design space which needs to be explored and iterated upon. For MRuby-Zest I made sure the library could hot-load code so there was no long change->compile->reload->renavigate->test->loop sort of interaction when development.
Third, the results had to be maintainable. Even if there were short time constraints on the implementation it couldn't be write-only code.

Applying these ideas for Zyn the end goal was to have a new GUI with a from-scratch look and feel which didn't exist in another toolkit and to do so by taking advantage of already accessiable metadata about the synths internals (for more detail on that I'll be giving a talk on rtosc which generates some of that metadata tomorrow).

So with that setup out of the way let's talk about the process which resulted in this toolkit being built. For most of the Zyn's project's life it had a fairly complex multi-window interface in FLTK. You could do stuff within the UI relatively easily, but it did end up feeling rough throughout. It was harder to navigate than it should have been, the widgets were somewhat inconsistent with each other, it didn't scale well. Let's just say that it had it's own myriad of issues. That list of issues resulted in a few people submitting redesign ideas over the years. More often than not the new designed lobotomized the synth. But in 2014 a design was submitted by Buidislav Stepanov. This design got what zyn was about. Much more so than the existing UI. I wanted to see the change happen, but I recognized that it was a huge undertaking.

Eventually I took some time when I saw a local hackaton going on and tried to see "How hard can this really be" (surprise, the answer is pretty hard). I had started with Qt and their new subsystem 'QML' along with using nanovg to render the GUI in OpenGL. Even after implementing parts of a single view for the synth it was clear that Qt and myself had different ideas about how things should be done and a lot of the development time was spent fighting against the framework that should be helping. The project was shelved for a while and eventually I had the idea of seeing if the GUI rewrite could be done in-leu of an internship.

roughly 3 months of development time and 33 different views of the synthesizer to implement. it seemed like it might be possible, but definitely not with pre-existing tools. So before the main sprint I started on building the outline of MRuby-Zest. From the first prototype implementation I had seen that nanovg worked very well, so I kept that. The QML files also worked quite well (I'll get into details in a sec), but QML files relied quite heavily on Qt's javascript engine. For those of you who work with javascript professionally, my condolences. I've never been able to get along with that language, so it had to be replaced.

So, with that in mind some dynamic language was needed to make code-hotloading a reality for the development sprint and it had to be something that a large number of other people use. If it wasn't for that second constraint I would have ended up building the toolkit with common lisp, but as you might be able to guess based upon the title of this talk, that dynamic language ended up being Ruby. In particular it ended up being a new embeddable version of ruby called MRuby. Having a first class scripting language was absolutely crucial for the fast development of the new GUI...

<Talk continues >

So, that's something like the first half of the presentation, I'd recommend watching live to see the rest.
The main points here are that building a new toolkit (even if it's on top of Qt) is pretty much a necessity in order to keep yourself sane. Development will be slow until you have tools to help yourself and those tools will likely need to be written from scratch to take advantage of LMMS specifics and problems that are specific to the audio manipulation domain. Since Zyn-Fusion had a (reasonably successful) development sprint doing the whole GUI in one go was somewhat practical, but for LMMS there's much more of a need to transition in a slower step-by-step manner. For Zyn this was even done for the 2.5.0 release (prior to the 3.0.0 GUI rewrite) which was internally referred to as the "Anti-Broken System" which replaced nearly all of the mechanisms that the GUI communicated to the backend. That "Anti-Broken System" spawned rtosc into existence which helped produce a ton of the metadata needed to keep the new GUI in line with DRY principles.

[budislav]

@SecondFlight I will just observe this because I am not programmer, I don't have any power here, but just want to say keep going with anything you want to make, even if nobody want to contribute please do not be discouraged and do whatever you can, but I hope many of active developers will involve into this. Good luck with this!
@fundamental thanks for sharing your presentation script it's great, and thank you for mentioning me, I am honored!

[josh-audio]

I think for the time being I'll work on the various low-level widgets. Even if none of it is useful based on where this project goes, I'm sure I'll learn a lot and be better equipped to help out.

[tresf]

PR #4367 suggests that option 2 (build upon the existing toolkit) is the accepted approach for now. I'm pervy to the fact that a private rewrite was started but never shared publicly. If that makes it to fruition (or if anyone disagrees with the closing of this thread) we can reopen.