My Experiences with Wayland and Nvidia in 2022

Traditionally, we have been using XOrg, or X11 in its current iteration, as the standard display server in Linux. However, it is an old piece of software that is to a degree optimized for the needs of an older age, which can be considered bloated and has some inherent security flaws like keylogging vulnerabilities. This is where Wayland comes in. Wayland is another display protocol that, from what I understand, aims to be more modern, compact and secure than X11.

Some history

For a long time, Nvidia has been far behind in terms of Wayland compatibility when compared to AMD or Intel. Whereas the open-source drivers for Intel and AMD had implemented support for the GBM API, which is the more standard approach, Nvidia chose to go with the less used EGLStreams API. The community hasn’t been able to implement the functionality either since those drivers are proprietary. There is the Nouveau driver, which is an alternative open-source Nvidia driver, but I’ll discuss that later, too.

The only desktops or compositors properly supporting EGLStreams are GNOME and KDE. There is also an EGLStreams compatible modification of wlroots called wlroots-eglstreams, though, which allowed more compositors (a compositor being the equivalent of a window manager/desktop), but I do not know how well that works.

However, Nvidia has improved its support in recent times. The driver version 470 brought along hardware acceleration for XWayland, which is a compatibility layer of sorts for running X11 software on Wayland. Thanks to this support, XWayland can now provide decent performance.

Driver 495 finally brought along support for the more common GBM API. This is also where I started my experience with Wayland.

My experience with Wayland

My graphics card has been a RTX 2070 Super that I have had since long before I even tried Linux. I first tried Wayland back when I was using Gentoo shortly after the 495 drivers were released into beta in Portage’s repositories as I felt like it might be functional at this point. I tried Sway as that is a very well-known Wayland compositor.

The compositor worked, but it had its caveats. Firstly, I had to ensure that the variable WLR_NO_HARDWARE_CURSORS was equal to 1 or the cursor would be invisible. Secondly, I had to launch Sway with their unsupported GPU launch flag nonsense.

While it ran, it had some issues. Namely, I was unable to run any software under Wayland. I was able to run XWayland software, but nothing native. I did not bother troubleshooting too far as it was a beta driver and I did not feel like putting in the time at that point. Thus I went to Xorg for a few months.

Return to Wayland

Approximately 2 weeks ago I figured I could try Wayland again. I now run NixOS, but that should not affect Wayland much. I decided to begin with Sway. I followed the NixOS Wiki instructions for Sway and got it running with the same extra steps as before.

Yet again it launched fine, but the experience was still not very good. This time software could run, which was a definite improvement. However, there were a whole lot of graphical issues, such as flickering all over the place. I adjusted the output to my screen’s refresh rate (240Hz) in the config and it did not help at all.

I thus gave up on Sway and decided to try GNOME instead as GNOME had been supported for a long time. This might be weird for a standalone WM person like me, but it is possible to strip down GNOME quite a bit, so it was viable in itself.

I had no flickering issues and apps ran, with a caveat: I was never able to run OBS on Wayland with my Nvidia card.

I customized GNOME to be more usable for me. Just a bar, but none of the app menu or activities view stuff.

Here are screenshots of the desktop and the sirula app menu config that I have:

What was good with GNOME

Now that it was configured somewhat properly, it ended up being quite nice, mostly. The UI worked with nice shortcuts and all that. Not quite as nice as a proper WM, but not bad at all.

Using Super and my wasd keys I could position windows on my screen. I could launch kitty (a terminal emulator) with Super and Enter and the app menu with Super and Escape. There are a ton more, but shortcuts worked well.

As mentioned earlier, graphical issues were not present. Screen refresh rate and such also did not give me any problems either.

With software, I did have to play around with stuff like VSCodium as that needs some extra launch flags to run under Wayland in general, but nothing too bad.

What wasn’t so good with GNOME

There were still issues that turned out to be rather major. In specific, Firefox. This thing really does like to cause me problems. I have always had to set things up a lot to get it work even remotely acceptably, but under X11 it was always usable after that initial setting up process. Here, not so much.

Basic web browsing functioned, but a crucial function didn’t: new windows. If I tried to run more than one window of Firefox, it would typically freeze and close after a little bit. Also the previous window would be frozen until the new one closed. On occasions the new window would actually work, though.

The other issue I had was that the extensions that I used for the bar, dash to the panel and just perfection, had some sort of a conflict where the utility icons at the bottom would be placed horizontally whenever I rebuilt my NixOS config and relogged into the DE. This was annoying, so I disabled all of those icons except the volume one as I could not close that one by itself for some reason.

So what to do?

I decided to give other compositors a shot again.

Wayfire

Firstly I tried Wayfire. Guess what? No Firefox new window issues whatsoever. However, the flickering and stuff was still there. Once again no config I could figure out fixed it. I tried all of the options in Wayfire’s config with many different refresh rates from 60 to 240, none of which helped.

So this wasn’t usable either. How about other compositors?

River

I have seen this one pop up on the unixporn reddit occasionally. It’s another tiling compositor. However, it did not run properly. It launched but only gave me a blank blue screen. Had to kill it in another TTY. Configuring the output changed nothing.

Cardboard

This one went beyond a blank screen. It gave me the blank screen, but it also locked my keyboard, so I couldn’t even go to another TTY and thus had to restart.

Hikari

Like Cardboard, but with wallpaper on the screen instead of a blank color.

DWL

Another blank screen, but did not lock up my keyboard. Not much else to say.

KDE

I chose this as the final attempt. It also supports EGLStreams, so it shouldn’t have the graphical issues of wlroots compositors. While it did not have those issues, it had others.

KDE has never been a good experience for me on the Nvidia card. Under X11 it has stuttering issues. But this is nothing compared to the Wayland version. Under Wayland, it would for some reason run at maybe 1/5 of its normal speed. Everything was extremely slow and stuttery to the point of being unusable for anything extended. Disabling animations did not help anything.

Nouveau

The Nouveau is third party open-source driver for Nvidia cards. They are, however, not great with modern Nvidia cards. My card was limited to its lowest clockspeed, so it was much slower than by stock.

While I was fine with the performance, there was another issue. Anything above 60 Hz would cause the top half of my screen to just go black. I could not find any solution.

Conclusions

GNOME wasn’t bad at all. If it didn’t cause issues with Firefox, I would have been perfectly fine.

As far as the GBM and wlroots support goes, my experience is that it is not up to the point of being properly used. Better than before, but not good enough.

Thus I returned to X11.

How it compares to AMD

I have a laptop running an AMD APU (Ryzen 5 4500G) as well, so I have been messing with Wayland on that. I even sold my Nvidia card today since it’s now overkilled for my needs and bought a RX580 instead, so my desktop can have more proper support whenever I get that.

As for the laptop, that’s now running as my only computer since I lack a GPU for my desktop for now. My choice of compositor here was Wayfire. I have also configured it to do mostly what I want. I might change compositors, though, as I only used this one as the first pick due to easy configuration.

Frankly, all of the Nvidia issues are gone. No graphical flickering or anything. I can even run OBS now, although I have not figured out getting screen capture working yet, I should figure that out eventually.