KDE4Daily Virtual Image

Update - 27th January, 2009

KDE4.2 was released today, and the general consensus seems to be that the worst is now behind us :) This being the case, KDE4Daily has probably outlived its usefulness, and will no longer be updated.

Everything under the line is preserved for legacy reasons; downloading KDE4Daily for the first time as of now is not really a good idea :) Note that the IRC channel in the contact section will shortly be closed, but the e-mail address will stay the same. Just for fun, here's how everything looked in the first ever version of KDE4Daily, way back in November 2007:

Many thanks to everyone who contributed bandwidth, bug reports and moral support! :)


Contents

What Is It?
Where is it?
What is Qemu?
What do I Need?
Very Important Warning!
How Do I Install Qemu?
I've Downloaded The File - What Now?
This is Fun, But How Do I Get a More Recent Build of KDE4?
Whoops - Something Crashed!
Are There Any Bugs in KDE4Daily That Do Not Occur in KDE?
How Can I Contact You If Things Go Wrong?
Some More FAQs

What is it?

The KDE4Daily project was created to help people try out and test KDE4 on the run up to its 4.0 release without having to compile from source, or log out to boot from a LiveCD. Further, it aims to be updated nearly every day, so that people always have a mostly up-to-date (but, inevitably, occasionally broken) KDE4 install, and also aims to be able to create backtraces that will be of use to developers. Typical daily updates range from 20-50MB in size and take a few minutes to apply. KDE4Daily accomplishes these goals by using virtualization technology, with Qemu being the VM currently employed.

The current KDE4Daily (v0.0.1) was released on 10th November, 2007. The current latest KDE revision provided for KDE4Daily is r759652, created on 11th January, 2008.

Where is it?

A torrent is available here. If you cannot use Bittorrent, then "sandsmark" has provided a direct download - many thanks, Martin! Additionally, Kiyoshi Aman has provided this mirror - cheers! Additionally, "lufthanza" has recently converted the image to VirtualBox and torrented it here - again Kiyoshi has also mirrored this here :)

What is Qemu?

Qemu is a technology that allows one to use a complete install of an operating system (be it a Linux variant, or Windows, etc) from within the user's "normal" computing system, at a speed approximating the speed the operating system would run at if it were run by itself. It is enormously useful for trying out software such as KDE4 without having to worry about tracking down dependencies or re-booting into a LiveCD - you can even run KDE4 from within Windows!

The drawbacks are that its speed is not quite as fast as running a proper install of KDE4, and that graphical acceleration is not provided, so KWin's new Composite features will not function.

What do I Need?

Very Important Warning!

KDE has not yet been released, and the installs provided by KDE4Daily are not official, nor (due to the rapid rate of release) well-tested. KDE4Daily should absolutely not be used to work with data that is valuable or sensitive. It is entirely possible that a failed update might leave your KDE4 image unusable, or that an update may require you to delete configuration/ data in order to continue working. You have been warned!

How Do I Install Qemu?

Instructions for installing Qemu and the kernel accelerator module for all distros is beyond the scope of these docs, but the Ubuntu Wiki contains detailed instructions for installing under 7.04 and 7.10 here:

https://help.ubuntu.com/community/WindowsXPUnderQemuHowTo

Note that 7.10 (Gutsy) users will have to manually install a replacement .deb as the Qemu install available is broken: see this post for details.

I've Downloaded The File - What Now?

The image is a bzip-compressed Qemu image - first you will need to uncompress it, using the archiving tool of your choice. Assuming you have uncompressed the image and the uncompressed file has the name "kde4daily-0_0_1_r734472-qcow.img", open up a command-prompt at the containing folder and type

qemu kde4daily-0_0_1_r734472-qcow.img -m <amount of RAM to allocate>

256MB of RAM is a good amount to allocate, if you can afford it:

qemu kde4daily-0_0_1_r734472-qcow.img -m 256

but if you are to be submitting backtraces, you will likely want more. To give an idea of how much memory you will need for backtraces - the gdb process alone can easily consume 230+MB of RAM when generating a backtrace.

A window containing your KDE4 image should now appear, and begin the standard Linux (text-mode) boot-up. After a minute or so, you should be presented with the graphical (KDM3) log-in screen. Your username and password are both "kde4daily". Enter your username and password and click "Log-in", and you should end up booting into the KDE4 desktop environment proper. Hooray! All being well, Qemu will have set up a network bridge that enables your KDE4Daily install to access the Internet without you having to lift a finger!

This is Fun, But How Do I Get a More Recent Build of KDE4?

KDE4 is progressing at a rapid rate, and significant changes can often occur overnight. For this reason, the KDE4Daily image contains its own update system that allows you to upgrade your KDE4 install, from within the KDE4Daily image itself while running in Qemu, without having to download a fresh image.

To use the update feature, simply log out of KDE4 or otherwise get back to the log-in screen, click Menu -> Session Type -> KDE4DailyUpdater, and enter your username and password and click Log-in. You should be presented with a black terminal in the centre of the screen and some instructions. Follow the instructions and your KDE4Daily updater scripts will attempt to access the Internet and scan for updates. If none are found, you will be informed and given instructions to return back to the log-in screen. If the attempt fails, you will be told and given a reason why - you can either return to the log-in screen or retry. If it is successful, then, after several pages of scary scrolling text, you will be told the current revision number that you have been updated. Follow the instructions to return to the log-in screen.

After upgrading (or attempting and failing), you can get back to the KDE4 desktop from the log-in screen by choosing Menu -> Session Type -> KDE4Daily and logging in as usual.

Whoops - Something Crashed!

In the truly momentously unlikely event of a KDE app crashing, "DrKonqi" will appear and give you the option of creating a "backtrace". Such backtraces are of great use to the developers, so it is useful to affix them to your bug reports. Please do ensure that you are not using a KDE4 install that is too out of date (use the KDE4DailyUpdater mechanism, described above) as the crash may have already been reported and fixed. Clicking on the "Backtrace" tab will start to generate the backtrace - Warning! Generating the backtrace involves downloading very bulky data from the servers, and consumes a lot of RAM and time. At the moment, there is no progress indicator to let you now what is going on, so please be patient and ensure you have plenty of RAM in your PC and allocated to the KDE4Daily virtual machine. Eventually, "Done" will appear in the backtrace form, ready for inclusion with your bug report.

Are There Any Bugs in KDE4Daily That Do Not Occur in KDE?

Here are a few bugs that appear to be KDE4Daily-specific:

How Can I Contact You If Things Go Wrong?

I can usually be found in #kde4daily on irc.freenode.net, with the nicks "SSJ" or "SSJ_GZ". Note that I'm a Brit (and a particularly lazy one, at that), and thus will likely be sleeping between 11pm and 8am, GMT. I'm particularly interested in hearing bugs relating specifically to KDE4Daily, so swing by if you find one! I can also be reached by e-mail at simon at etotheipiplusone dot com. Please ensure that your queries aren't covered in the FAQ first, though! :)

FAQ

Why Doesn't It Boot?!
Why Qemu, Rather than VMWare or VirtualBox?
Where are Amarok and the Other KDEMultimedia Apps?
Where are Kontact and the Rest of the PIM Suite?
So What Modules are Included?
How Long Will the KDE4Daily Project Provide Updates For?
Is it Possible to get KDE4 Localisations in My Language?
Should I Mention That I'm Using KDE4Daily When I File a Bug?
I Haven't Updated In a While. Will I Have to Download Hundreds of MB to Get Back Up to Speed?
The Updater is a Little Unfriendly. How About a GUI?
Help! I Did An Update, and Now KDE4 Won't Run!
Help! The Updater Crashed While Applying the Updates!
What is the Underlying Distro Used by KDE4Daily?
Seems Pretty Slow. Will the Final KDE4 Be Like This?
What KDE4 SVN Revision am I Currently Using?
What Language are The Updaters and Preparation Scripts Written in?
Can I Provide My Own Updates?
Why Is Backtrace Generation so Darned Slow?

Why Doesn't It Boot?

Most likely, you are using (K)Ubuntu 7.10 (Gutsy). Please see the note about this distro version in this section.

Why Qemu, Rather than VMWare or VirtualBox?

Basically, Qemu is what I know :) But luckily, many people know better than me, and have provided instructions for converting your downloaded image to a VirtualBox-compatible image; see e.g. Liquidat's guide here. Bernd has given some handy hints for VMWare users here.

Many thanks to he and Liquidat and everyone else who provided similar instructions!

Update:Someone has created and uploaded a VirtualBox image torrent - see "Where is it?" for a link!

Where are Amarok and the Other KDEMultimedia Apps?

After consulting with the KDEMultimedia team, it was decided that no virtualization technology could really do justice to the media apps in terms of smoothness of media playback, and so they are not included.

Where are Kontact and the Rest of the PIM Suite?

After consulting with the KDE-PIM team, it was decided that many of the apps are not yet quite at the stage where they can be tested. It's entirely possible that they will be rolled-in to a future update when they are in better shape, though.

So What Modules are Included?

A list of modules checked out and compiled is here.

Is it Possible to get KDE4 Localisations in My Language?

[Note: Original idea and 99% of the work here was done by Toussis Manolis - many thanks! Any errors/ bugs are my own].

Around about r742845, Toussis contacted me with a plan to allow the user to easily install translations from SVN with a view to making life easier for the translation team, and a modified version of his script has since been slipstreamed in. First, ensure you are fully up-to-date using the KDE4Daily update system. Log in to KDE4 and from konsole, do:

bash $KDE4DAILY_UPDATER_DIR/kde4lang.sh <language code>

e.g.

bash $KDE4DAILY_UPDATER_DIR/kde4lang.sh en_GB

and follow the instructions. The script will download and install 30MB of installation tools, gobbling up much of the remaining space on "/" (whoops - my fault for using such a silly partitioning scheme!) and will hopefully pop-up the KDE4 "Language" settings, where you can easily add your new language via a GUI. Re-running the script will fetch and install any translations added for that language since the last run. You may need to log in and out of KDE4 to allow the full set of additional translations to be applied. Note that this feature is very new to KDE4Daily and may have bugs - please report any you find to me!

Should I Mention That I'm Using KDE4Daily When I File a Bug?

Yes, please. There have been a handful of bugs reported that were my own fault rather than KDE's, and mentioning that the bug was found in KDE4Daily has helped to cut down on the time wasted trying to replicate the bug.

I Haven't Updated In a While. Will I Have to Download Hundreds of MB to Get Back Up to Speed?

I will hopefully be creating weekly "bridge" updates, that let you download and apply a week's worth of changes in one go. Experiments have shown that such bridge updates are often not much larger in size than an update for a single day! KDE4Daily always picks the path through updates that results in the smallest download size, so such "bridge" updates will be used automatically.

Update: Bridge updates are now created every week or so :)

The Updater is a Little Unfriendly. How About a GUI?

One is on the TODO list, but I have no idea when I'll get around to it, alas.

Help! I Did An Update, and Now KDE4 Won't Run!

Relax :) It's important to note that, as with all builds drawn directly from SVN on a frequent basis, many will be duff. The KDE4Daily updater has been deliberately designed to have no dependencies on Qt4 or KDE4, so just keep updating every few days and eventually the problem will probably sort itself out. If symptoms persist, call a doctor or, better, try and get in touch with me: see "How Can I Contact You If Things Go Wrong?" for more info.

Do note that a newly-upgraded KDE4 install will occasionally crash on first run, so always try to log in at least once more if KDE4 crashes during start-up. I'm not sure why this happens, but suspect it has something to do with stale libraries being kept in memory.

Help! The Updater Crashed While Applying the Updates!

This really shouldn't happen, and I'd appreciate an e-mail, preferably with a complete log of the entire update session, including errors. Your KDE4 install will likely be broken, but all is not lost: If I can fix the bug, an update to your updater scripts will automatically be applied next time you try to update, and the diff applier is designed to withstand re-running after failures and will (hopefully!) set everything straight.

What is the Underlying Distro Used by KDE4Daily?

KDE4Daily sits on top of Kubuntu 8.10. This is mainly because this is the same distro that my desktop PC, on which builds/ updates are prepared, runs.

Seems Pretty Slow. Will the Final KDE4 Be Like This?

Firstly: There won't really be a "final" KDE4 - many people seem to think that KDE4.0 will be basically the end of development of the KDE4 series, rather than what it really is - the beginning. So when KDE4.0 is released, we can still look forward to years of improvements and optimisations in KDE4 and Qt4!

Secondly: Running in Qemu gives an automatic hit to the speed of computations, so KDE4Daily VM will run more slowly than if it were installed and running on the "bare metal" of your machine, as it were. Also, I don't believe Qemu accelerates the graphics at all - anyone who has experienced the joys of running a Linux desktop with an ATI card with poor drivers will know precisely just how painful that can be! Additionally, the KDE4 build provided is compiled in debug mode, and while the bulky debug data has been extracted from the libraries and executables, the compiler will still (I think?) avoid doing risky optimisations.

What KDE4 SVN Revision am I Currently Using?

Your current SVN revision number is stored in the text file:

$KDE4DAILY_UPDATER_DIR/data/kde4revision.txt
which you obviously shouldn't edit :)

What Language are The Updater and Preparation Scripts Written in?

Virtually everything is written in Ruby. This was my first "proper" experience with the language, so I should probably amend the answer to "Very Bad Ruby" :)

Can I Provide My Own Updates?

This is slightly tricky as I suspect that anyone wishing to provide updates for KDE4Daily will need to be running the same distro as that sitting under the KDE4Daily image - that is, Kubuntu 7.04. I'm not really sure how to approach this problem, to be honest, and would welcome suggestions. The "brute force" approach would be to further abuse virtualization technology and provide an image that mirrors that of the installation that I use to generate the KDE4Daily updates, but I have no idea how slow compilation of KDE4 would be inside a Qemu session. Using distcc on the host computer might be a solution.

Apart from the challenge of providing an environment that will allow one to create updates compatible with KDE4Daily, most other issues are solved: one of the original motivations for the project was to allow the Oxygen style/ windec developers to easily deploy updates to the actual artists (who may not have the technical skills to maintain and update a KDE4 install from SVN), so provision was made from the outset to allow a KDE4Daily to install updates from a source other than KDE's svn. The idea is that the developers and artists switch to a different track (called KDEOXYGENONLY, say), and then the artists' KDE4Daily updates will always be drawn from a source specified by the Oxygen developers and so only receive changes that are created by the Oxygen developers. Currently, though, the only track available is the default one (KDESVNTRUNK), and the updater scripts are hard-coded to use this one.

If there is sufficient interest in creating diffs, and if we can solve the issue of easily creating diffs compatible with KDE4Daily, then I will try and get the preparation scripts into a usable state and make them available for download.

Why Is Backtrace Generation so Darned Slow?

Generating backtraces is, at the best of times, a very intensive procedure: as mentioned, GDB can easily gobble up 230MB of your RAM in the process.

The debugging information used by the KDE4Daily KDE4 install is very large - approximately half a gigabyte, when compressed with maximum bzip compression. Including it would swell the initial download by approximately this size. Worse, the debugging info changes very frequently - rebuilding KDE4 after a days' worth of updates can easily generate hundreds of megabytes of new debugging information. For this reason, including the debugging info in the initial build and providing daily updates to it would be prohibitive in terms of bandwidth requirements.

Luckily, this debugging information is fantastically amenable to utilities such as bsdiff, which enables the "new" debugging info created each day to be expressed as a small patch against the "old" debugging data. This reduces the bulk of the new debugging info to around 8MB a day, rather than 2-300MB. So now, when GDB requests the debugging info for a file (a custom FUSE filesystem daemon, written in Ruby, intercepts these requests), we can download the bulky, original debug info from the server, decompress it, see if todays' build requires newer debugging info than that we have just downloaded and, if so, download the small patch file and apply it, eventually returning up-to-date debugging info for use by GDB.

So, in a nutshell: The process of having to download, decompress and patch the debugging info is what causes the slowdown. On the plus side, KDE4Daily uses caching to keep the debugging info it has downloaded and prepared, so the process of generating a backtrace should be faster after the first time. It also delivers and prepares only the debugging info specifically requested by GDB, and no more. Out of date debug information is automatically cleared from the cache after each update.

Update: There has been a lot of effort recently to try and speed the process up: please see this blog entry for more details!