Fedora 10 -> Fedora 11 upgrade problems

After the last debacle [1, 2] associated with trying a yum upgrade from Fedora 9 to Fedora 10 on my x86_64 Opteron workstation I was sure that this time would be different. I dutifully downloaded and burned a x86_64 installation disk for my Opteron and a i686 for my old home PC.

Neither upgrade worked properly.

I had not run yum update on my home Fedora 10 machine since Fedora 11 was released on June 9 2009. I attempted the DVD upgrade on June 19, left it happily (if slowly) updating the ~1200 packages it wanted to update, only the return to a fatal error message regarding an unhandled exception. On reboot a single Fedora 11 kernel was available which started booting but failed when trying to go to runlevel 5. Reading the Common F11 bugs page indicated that my nvidia AGP card might be the problem, so modifying the kernel boot options as suggested allowed me to boot successfully. A large "yum update" (1.2 GB) was then necessary, but I finally had a working system.

For the Opteron I made sure my Fedora 10 installation was fully updated on the morning of 23rd June, and just before leaving working in the evening I stuck the DVD in the drive, hit the upgrade option when it appeared and left it running overnight. When I returned the upgrade claimed to have completed itself successfully and told me to reboot.

On rebooting grub went straight to a empty command line, without the usual boot menu. Clearly the grub-install that is run every time you update the kernel had not run successfully. As I know /boot is the first partition on my first hard drive I can still find the kernel and boot by doing the following on the grub command line:

root (hd0,0)
kernel /vmlin[hit tab] [select appropriate file] [return]
initrd /initrd[hit tab] [select appropriate file] [return]
boot

I did not use the standard Fedora kernel options "rhgb quiet" as I wanted to watch the services start up to see if there were any problems. Indeed, once the kernel started booting I saw that a variety of services where failing to start, complaining of shared library problems. A classic sign of a incomplete upgrade. Great!

In fact the boot process stalled completely, so I resorted to using the Fedora 11 disk in rescue mode. chrooting into the system once it had booted.

I checked that /boot/grub/grub.conf looked sensible, ran "grub-install /dev/sda" fix the grub problem, and then attempted "yum list updates", only to find that yum wouldn't run as python complained that it could not find the yum module.

"rpm -qa | grep yum" revealed that yum was still a Fedora 10 version, version 3.23. Mounting the Fedora 11 disk (mount -t iso9660 /dev/scd0 /media) revealed that the yum version on the Fedora 11 install disk is 3.22, so clearly the upgrade had failed to update yum because the recently updated Fedora 10 version had a higher version number than the Fedora 11 disk created only a few weeks before! In fact a large number of packages had not been updated to Fedora 11. Luckily rpm had been updated (if rpm was not working I'd have been forced to do a full system restore). Running

rpm -Uvh /media/Packages/yum* --oldpackage

successfully updated yum to the Fedora 11 version, at which point I could run "yum update" and start the process of getting a truly upgrading system underway.

With hindsight my mistake was updating my Fedora 10 system after Fedora 11 had been released. When doing yum upgrades it is a good idea to have your system as closely matching the new version of Fedora as possible, but this appears not to be the case for a DVD-based upgrade.

I'm still surprised that the DVD-based Fedora upgrade doesn't think to correct for version mismatches - you'd think it'd be logical for any f11 package to automatically obsolete a f10 package, even if the version numbers of the f10 package are higher. Or the upgrade could simple run rpm with the "--oldpackage" command line option.

Overall I have little satisfaction with the upgrading process under recent versions of Fedora. Web-based yum upgrades used to work very well, at least between versions ~6 and 8, but with Fedora 9 -> 10 and now with Fedora 10 -> 11 neither web-based or disk based upgrades has worked without significant problems.

Admittedly I've also experienced occasional severe problems with upgrades between different versions of Ubuntu when I was experimenting with it, so this is not a problem unique to Fedora.

The alternative - moving to a long-lifespan distribution such as RHEL or a free clone of it like Scientific Linux - simply replaces upgrade problems with equally severe out-of-date software problems (a friend who relies of the department's sysadmins to maintain his Scientific Linux distribution has had terrible problems with being stuck with old buggy versions of important software that require major version changes to fix, which the standard updates in RHEL/SL don't often do).

The only pro-active solution I can see is to become more involved with testing process on Fedora, and to have a dedicated test machine that I can easily revert to its original state when upgrade tests fail badly.

Missing printers with Firefox 3 and linux.

The Problem: All printing options except "Print to File" have disappeared from the print dialog box in Firefox 3 under Linux. Yet inspection of the print configuration options in about:config reveals no obvious changes and furthermore values related to the expected printers are present.

Crude Fix: Add the following line in /etc/gtk-2.0/gtkrc

gtk-print-backends = "lpr,file"

This will give access to very minimals command line interface to lpr, which means you must know the printer names and options you wish to invoke. Another fine example of of newer Linux software sacrificing core functionality for superficial improvements in other areas.

[From my "Random Unix and Linux tips and tricks" page]

Fedora upgrade problems solved?

Just to follow up from last week's post on my problems upgrading my x86_64 workstation to Fedora 10, I think I now have everything fully functional once again.

The booting/grub problems were definitely related to grub's attempted probing for a floppy drive (/var/log/messages will show kernel I/O errors associated with fd0), even though I've never had a floppy drive in the system and it had never caused a problem before. It appears this is not an uncommon problem, see e.g. here and here, but its not obvious why I'd only experience this now with Fedora 9 an 10.

The command line workaround (install grub to the MBR using the grub shell with "grub --no-floppy," setup (hd0,0), etc, instead of using grub-install) is not ideal as it appears that any kernel update (e.g. when updating the system after a DVD upgrade) will run grub-install itself and fall foul of the problem.

In the end I had to switch off floppy support completely in the BIOS (even though no floppy drive or any associated cabels were present in my system) to solve this problem.

Upgrading from Fedora 8 -> Fedora 9, and then Fedora 9 -> Fedora 10 using the x86_64 DVD install/upgrade disks ultimately worked, in a fashion, but displayed some annoying features that reminded me why I switched to the yum upgrade method before.

The problem is that after "upgrading" and rebooting I found many packages (indeed, the majority) had not been updated to Fedora 9. I presume the reason is that a fully up-to-date Federa 8 system has packages that are more recent, with higher version numbers, than the original Fedora 9 DVD distribution, and that the DVD upgrade process then does not replace the fc8 packages with the fc9 packages.

I had to run "yum update" at least twice (why twice? I don't know why it didn't pick up all the packages that needed updating the first time) before the older Fedora 8 packages were replaced with normal fc9 packages. Even with a fast internet connection updating > 1000 packages takes a full day or so. I then had the same problem with Fedora 9 -> Fedora 10, where the situation was so bad that after the basic DVD upgrade firefox and konqueror simply didn't work at all. Then yet another day waiting for "yum update" to complete.

I did discover some useful yum tips here, in particular how to force yum-fastestmirror to retest the mirror speeds without doing "yum clean all." (Important if you've just spent several hours downloading hundreds of packages, but they haven't yet been installed before the transaction check found a problem, and you need to download more but the once-fast mirror you were downloading from is now incredibly slow.)

The update process was exacerbated by some odd changes the DVD upgrade made to my networking setup that basically messed up /etc/resolv.conf and my DNS configuration. It took me a while to work out that the "Error: Cannot find a valid baseurl for repo: fedora" was basically a DNS/dhclient problem. I don't remember DVD upgrades messing with network settings before, so I'm not sure how or why this happened.

To add insult to injury my rsync backup scripts then no longer worked because the newly installed Gvfs (specificaly ~/.gvfs) breaks rsync. The work-around is to add .gvfs to the rsync excludefile, but what a pain!

All in all a very tiring upgrade process, and all the more annoying because the exact same upgrades for i386/i686 work fine on my other machines. I presume part of the problem, in particular with the DVD upgrade, was leaving the upgrade so late after Fedora 9 and Fedora 10 had come out, so I may upgrade the x86_64 workstation to Fedora 11 relatively soon after it comes out. But on the other hand I don't like to upgrade to a new Fedora version before some of the kinks in the new version are worked out...

Fedora upgrade problems

I've been having a lot of trouble with my main workstation this week, brought on by an attempted yum upgrade from Fedora 8 x86_64 to Fedora 10 x86_64, but probably complicated by hardware problems associated with the floppy drive (or lack of it, to be more precise) that then messed up (and continue to mess up) the grub bootloader.

I had to do a full system restore yesterday from my incremental rsync backups (which worked wonderfully), but I'm still struggling with the system and was forced to do a rpm database rebuild this morning. The DVD-based Fedora 9 upgrade I set running last night, although supposedly successful, in truth failed thanks to grub-install's (which seems to be run whenever you install a new kernel) insistence on probing for a floppy drive even when there isn't one present and even when the device.map correctly excludes fd0 (note that recent versions of grub-install do not have or honor a --no-floppy command line option, despite what people on the Internet seem to think). Attempts to run grub-install from the command line simply hang, and /var/log/messages starts showing kernel I/O errors associated with fd0 (which I'm pretty sure I dont have!).

The messed up Fedora 9 install also seem to have messed up rpm, as on booting I found that even simple rpm queries would hang and /var/lib/rpm/ had the classic __db* files.

Still, looking on the bright side it has been a good test of my linux sysadmin knowledge.

My rsync backup method (based on Mike Rubel's incremental rsync backup with hard links method described here, see also Kevin Korb's page on rsync backups) works well and restore's easily. Note that the method's Rubel and Korb describe don't quite work on my system as rsync can't handle the number of files of my system in one go - I basically run the method by script separately on each of /boot /bin /etc /home /var etc etc).

I still remember how to use the grub command line to find files and boot, even if grub.conf is totally messed up, and I've learnt (the hard way) that grub-install can't quite be trusted and that its better to use the grub shell to install grub to the MBR (Remember to use "grub --no-floppy" to prevent probing for a floppy, even if you dont have on).

The annoying rpm stale locks and corrupted rpm database thing can be fixed ... "rpm -vv -rebuilddb" worked, thankfully.

I'm still surprised that the basic yum upgrade failed, as I've done yum upgrades from Fedora 8 directly to Fedora 10 successfully before, although only with i686 systems and not an x86_64 system. Based on the error messages I experienced it seems that /usr/lib64/libc.so.6 is the problem child in the jump from Fedora 8 to Fedora 10. I really should have done Fedora 8 -> Fedora 9 -> Fedora 10, or burnt myself an upgrade DVD and done a DVD-based upgrade.

That the DVD upgrade from my recovered Fedora 8 system to Fedora 9 also failed (because of the grub/fd0 problem) does suggest that a classic DVD upgrade, and not doing a yum upgrade would not have prevented me from experiencing problems (although it might have saved me a full days work). However I do find it hard to believe two separate problems, one software and one hardware related, would appear simultaneously.

I'm still waiting for the Fedora 9 yum update to finish before I shutdown and go poking inside the machine to see if I can find out what is causing my floppy drive problems. Without solving that issue there is no point attempting to upgrade to Fedora 10...

Anyway, time will tell whether I can get the system safely upgraded to Fedora 10.

pdffonts, updmap, epstopdf and ps2pdf

The occasion of the U.S. National Academy of Sciences "Astro2010" Decadal Survey of Astronomy and Astrophysics has forced me to (re)confront the issue of font embedding in PDF documents produced with LaTeX.

The last time I did this was about five years ago for various NASA ROSES proposals (ATP, the now defunct LTSA, etc) and it was something of a messy process. Which font package to use, what dvips command line options to use to make the postscript file, and then what ps2pdf command line options to use, etc etc.

Nowadays its much easier to produce high quality PDFs from latex source, thanks to pdflatex, and I'd assumed all fonts ended up being embedded properly automatically now. But in writing my Astro2010 White Paper I ended up discovering the very useful pdffonts tool, which in turn informed me that all my fonts where not being embedded. In the process of solving that I got to learn about updmap (not actually the source of the problem, as its correctly set up in Fedora 8 and 10) before realizing the problem was with my figures.

I still produce and include EPS figures, relying on the latex epstopdf package to automatically convert the figures to PDF for pdflatex. Turns out epstopdf (the command line tool used by the epstopdf latex package) does not correctly embed the fonts when coverting the EPS files, so thats where the problem was. The command line ps2pdf tool does embed the fonts, but only when you use the -dPDFX command line option (-dEmbedAllFonts does not work!).

Anyway, all this is now documented on my Unix tips and tricks site.

XFCE4 impresses, Amarok 2 fails.

Still in the process of switching to XFCE4 as the window manager of every Linux machine I own (six machines, counting both work and personal machines), but I'm getting the process of customizing it quickly to my liking down to an art. The more I use it the more I like it, or the more I can make it look and act like KDE 3.5 (only seemingly more responsive).

I also worked out how to get ssh-add to start when the XFCE desktop comes up, slightly different from autostarting processes on KDE.

I only just realized Amarok has changed as well, as we now have Amarok 2 on Fedora 10. As with other KDE4 rewrites of well established KDE tools this is sadly another great step backwards in terms of functionality, despite the standard claims of "significant improvements". Not only has the iTunes-like playlist interface totally disappeared but the customization options are severly restricted compared to the old KDE 3.5 Amarok (just go to Settings -> Configure Amarok and take a look at the barren emptiness). You can right-click to add KDE4's totally awesome Plasmoids, but none of the ones available on Fedora 10's version of Amarok 2 add back the basic playlist/tracklist functionality we've lost.

I do not appear to be the only naysayer either. Add to that the fact that it can't resume playback if you pause playing, and you can appreciate why I'm less than impressed. I may have to switch music players completely.

I hate acting like the Grinch, continually criticizing KDE4 and its developer's ernest belief that their hard effort was producing an improved product. Yet too much of KDE4 has basic functionality missing (despite the fact that they clearly had the time to add new and unnecessary eye-candy) for me to continue with it.

I may have said this before, but I'll say it again. I can understand the motivation for a complete rewrite of some code when the existing code becomes too difficult to extend or add to. There are strong pragmatic reasons to embark upon such a project. But with KDE4, as with Gnome 2 years ago, you end up getting the impression that the developers were more interesting in add fancy new things (Plasmoids) than in recapturing the basic functionality of the older versions. If your new application can't recapture the core of the old application then it'd be better to call it something completely different, rather than adverstise it as "a new and improved" version of the old application.

Its a year since KDE 4 came out, and while KDE 4.1 fixed some of the most annoyingly trivial oversights (like a clock panel application that couldn't shown seconds), many other things I consider to be important (more important than eye candy) still haven't been fixed. I presume they've been redefined as features or expected behavior now.

I get the (possibly mistaken) impression that many of its developers lack a certain attention to detail, or a view of the big picture. Did they sit down before starting to write new code and consider what is essential functionality, what parts of our existing application work best, which part are used most, for example? The KDE4 page on wikipedia makes everything sound semi professional (although with a worrying emphasis on style, icons, multimedia API, desktop effects etc), but something must have gone wrong to end up with a product that radicalizes long term KDE fans (e.g. see here, here or even here).

Of course, KDE4 does have fans too (I suspect they don't use it the same way I do). Ryan Paul at Ars Technica defends KDE4 here. I am not at all convinced by his arguments, and tend to agree with Steven Vaughan-Nichols criticisms:

My real problems with KDE 4.1 is far more fundamental. The developers believe that they have a better way of handling the desktop. For them, I’m sure they do. For users, this user anyway, the new desktop fails at a desktop’s main job: enabling the user to get their work done as easily as possible.

[...]

I could go on, but I’m not going to bother. KDE 4.1 is full of visual improvements that dont’ improve anything. You can see KDE 4’s Plasma interface for yourself. Maybe it will work for you. It certainly doesn’t work for me.

KDE 4 developers, lead by Aaron Seigo, wanted to make a radical change to the desktop. They have. However, in so doing, I don’t think that they have made that classic engineering mistake of making something that’s great for them, but not for users.

Seigo assures me that he can explain what KDE developers are doing to me. I’m sure he sincerely believes that. Unfortunately, in so doing, he’s making my point for me. Desktops shouldn’t need explanations They should just let you do your work. KDE 4.1 gets in the way of my doing work.

Is it because I’m an old foggie? Maybe.

I'm not the only one who has no love for KDE 4. As Linus 'Mr. Linux' Torvalds recently said, "I used to be a KDE user. I thought KDE 4.0 was such a disaster I switched to GNOME. I hate the fact that my right button doesn't do what I want it to do. But the whole 'break everything' model is painful for users and they can choose to use something else."

This last paragraph is from another Vaughan-Nichols article, this time on KDE4.2.

Anyway, enough on KDE. Until the XFCE4 folks do something similar (I'm sure it'll happen eventually) KDE is dead to me. Long live XFCE!

Tomshardware has a Macbook / OSX review from a long time Windows user. All pretty standard stuff given that worldview, but it does inspire me to formalize my impressions of Apples OS from the perspective of a long term *nix power user. Just need to find the time.

OS X, Fedora 10, KDE 4.1.2, XFCE

Still playing around with the new Macbook and OS X 10.5.6, getting used to the odd Mac key and the associated keyboard shortcuts.

Overall I'm quite impressed at the speed and slickness of the basic OS, as much as I try to avoid being impressed with eye-candy. The lack of "focus follows mouse" is a major pain, and copy/paste seems broken or tricky between X11 and non-X11 apps. The Terminal program is OK, less fancy than I was expecting based on word-of-mouth, although nicer than the lowest-common-denominator of the basic X11 xterm.

I was going to say the OS X Terminal is less capable than the KDE Konsole, but then I was rudely reminded last week that the KDE4 Konsole is less capable, and less configurable, than the KDE3 Konsole. Currently I'd say the OS X Terminal and KDE4's Konsole appear pretty comparable in capability and available configuration options at present.

I've complained before about KDE4 on Fedora 9, but had noted some progress and had hoped the version Fedora 10 shipped with would be further improved. Last week I upgraded my work back-up machine to Fedora 10, largely as a test, and experienced many of the same problems...

The Konsole no longer has a login shell "--ls" command line option. Transparency is still not available. Add an application laucher to the Plasma-based panel and dare to change the "Icon Settings" and its icon will be replaced with a question mark. Worse still, the changes you've made to the panel launcher have been propagated back to the main KDE menu, so don't make a mistake! As for trying to move icons around on the panel... it didn't appear to be possible at all (no right click, "Move" option anymore) and its seems other are as confused by this as I am, although here is the non-intuitive way to do it.

And its slowwwww. The price we pay for KDE trying to implement desktop graphical effects similar to OSX, I suppose.

Any, sick of the fuss, I ditched KDE4 on the Fedora 10 machine and switched to XFCE, and managed to quickly configure my desktop into looking something like KDE3.5 with nicely transparent terminals in less time that it took me to get the KDE4 Konsole to use a login shell.

One step forward and one step backwards, I suppose... Ah well.

New Macbook! (and Fallout 3)

My new work laptop arrived yesterday - a 13" 2.4 GHz Macbook. This is my first Mac ever, something of an experiment for me, so lets hope it goes well. So far I'm quite pleased with it, especially now that I've worked out how to right click using the touch pad and found how to add a terminal to the dock.

My old Lenovo Thinkpad X60, dual boot Fedora 8 and Windows XP, was (and still is) a good software development machine but was imperfect at its primary role of conference/travel laptop. On Windows the separate Lenovo and Windows wifi managers would often conflict with one another, although it worked once you'd found and killed one or the other. I'd bought the thing based on the fact that people at ThinkWiki had the wireless hardware working under Linux, but for whatever reason it would only work intermittently and often would disconnect from a wireless network it had managed to connect to. It even did this when I swapped the Intel a/b/g wireless internal card for an Atheros-based one that was supposed to work even better under linux. I spent a lot of time getting into old-school kernel patching and micro-code nonsense, before giving up trying to make wireless work robustly under linux. This was a pain at conferences, where wired ethernet connections are rarely provided.

And I never got sound working under Linux - a failure that rankles me considering I managed to hack sound into working on linux on every laptop I've ever had until the Thinkpad came along two years ago.

I'd originally chosen a dual-boot Windows/Linux combo (as opposed to pure Linux, which is what I did on previous laptops) because conferences had stopped letting you hook your own laptop up to the projector to give your presentation (which used to work fine under linux with OpenOffice for me and still does for departmental Seminars, apart from one memorable incident at a conference in the Canary Islands...).

Now all conferences have adopted a policy of transferring all talks to one of two-conference computers, usually one Windows and one Mac. This initially lead to font problems with talks saved in powerpoint format from OpenOffice Impress, although installing and using Microsoft web fonts under linux solves that problem. I added Windows to the Thinkpad X60 so I can load the ppt in powerpoint to check it. All good so far. This is almost foolproof, except when conference organizers put your powerpoint talk from a XP machine onto a Mac running powerpoint (or vice versa), at which point all hell breaks loose. This has happened to me only a few times, but it happens to some poor soul at every conference I go to.

So, sick of powerpoint's inability to work properly on both XP and Mac, and having to reboot out of a working environment (linux) to get into XP to check the presentation, I have decided to do all future talks in Keynote. Hence the Mac. And once I've unixified the Mac I should be able to do some decent work on the thing at the same time, no rebooting required. Wish me luck.

PS: Fallout 3 did not disappoint. An awesome game, with only minor flaws of an overly-low level cap and a rather abrupt termination to the main quest.

Unix Semaphores - A finite resource!

One of the things about unix-like operating systems is that you never learn them all - there is always something more that you didn't know about that some other unix-user will tell you about that you'd never heard of but is cool and/or useful nonetheless.

I've been using *nixes pretty much full time since 1994, and sys-adminning my own Linux boxes since 1996, and yet yesterday I learned something completely new.I was messing around stress-testing some Message Passing parallel code using MPICH on a shared memory system (couldn't be bother to mess around with going across the network and set up ssh keys for a simple test, OK), only to discover after some weird errors, that the ch_shmem system of MPICH relies on System V-style inter process communication, which in terms uses semaphores, of which each user is only allocated a small number (32 arrays or something silly). Interestingly, nothing else on my system appears to be using semaphores or message arrays.

If MPI crashes while running it fails to clean up the semaphore(s), and if you do this too many times you end up not being able to run anything that relies on semaphores, which causes much confusion if you didn't know about this stuff. Hence you need to run ipcs -s to see your semaphores (hopefully none, if your MPI program has finished running), and ipcrm -s to delete any improperly terminated semaphores. I'd never heard of these commands before, but they solved my problems.

Review: Eschalon Book I

Eschalon Book 1 is a deliberately old-school computer role playing game produced by the indy studio Basilisk games, and available for Windows, Mac OS X and Linux. By old-school I mean turn-based, isometric, fixed camera, with strong textual elements, think of an intermediate between Legend of Zelda and Neverwinter Nights. If you liked those games, then for $19.95 Eschalon Book I might be worth your consideration.

Online reviews of the game can be found at the following sites:
http://www.shamusyoung.com/twentysidedtale/?p=1485
http://jayisgames.com/archives/2008/06/eschalon_book_1.php
http://www.2404.org/reviews/2881/Eschalon:-Book-I-Review
http://www.scorpia.com/?p=812

Overall, I quite enjoyed the game myself, although the first few hours were tough on the Ranger character I generated, as I soon ran out of arrows and the money needed to buy arrows. The strong element of random encounters in the game, while familiar from the pen and paper RPGs I used to play as a teenager, also made life difficult as I'd camp to try to heal wounds only to have camping lead to a random encounter that would cause me more wounds, and exhaust my meagre stock of healing potions and arrows. Reading the reviews shows that others also thought that the random encounters and early game money scarcity needed better play balancing.

Ultimately I got a bit tired of the grind - at one point I had to fight opponents with my fists as I'd broken my only dagger trying to open a sealed barrel (note to self: I should have RTFM before beginning playing, as it tells you to use cleaving or bludgeoning weapons on barrels and chests) and didn't feel like restarting with a swordsman or a magic user.


For the Linux version Basilisk had fixed the game exploits found in the Windows version, so I took matters into my own hands and used a hex editor on the same game file to give myself 45000 gold. After that, able to buy whatever I wanted, and all the arrows I needed, the game was actually a lot more fun.

I was finally able to take the time to notice the excellent audio(*) sound effect work (owls hooting as you walk through the woods, for example), and the prettiness of the level design given the limitations of the isometric engine.

I've previously mentioned I was giving Indy games a try, given my recent dissatisfaction with recent big-budget big-game studio games that given only a few hours game play for $60. Fully playing through Eschalon Book 1 took me about 25 hours (a little longer than the 20 hours some of its reviews mentioned, but for reasons I mentioned above), so at approximately $1 per hour its pretty cheap entertainment. While you won't get as much for that $20 as if bought a big-studio like NWN a few years after it came out, there is something to be said for supporting Indy game studios that do a decent job. Think of it as buying Organic food - maybe its better for the environment.

(*) One annoyance I experienced was occasional interference-like audio buzzing, but I'm not sure if this was intrinsic to the game or really a pulseaudio problem in Fedora 9.

teTex -> TeXlive transition on Fedora 9

Practically every document I have written professionally since 1994 uses LaTeX, either old vanilla latex (tex -> dvi -> ps) or pdflatex (tex -> pdf), often with non-standard cls files (aastex.cls, mn2e.cls).

Ever since I noticed that Fedora 9 was replacing teTeX with TeXlive I've been dreading the transition, and have kept all my work machines on Fedora 8 rather than transition to Fedora 9. (The KDE4 transition also gave me additional reason to wait until the bugs are fixed.)

I finally got time yesterday to back-up all my papers from work to my home Fedora 9 test machine, and just for fun tried running the TeXlive versions of latex and pdflatex on some of my papers and proposals... and they all worked first time, no messing about or reinstalling of the astro cls files required.

My old cls files that were installed in the /usr/share/texmf tree had not been messed with, and the ls-R file had been automatically updated to include them.

I'm pleasantly surprised, given the prevalence of problems a "texlive tetex" google search reveals. Good job Fedora team!

Okteta

Having previously complained about KDE4 on Fedora 9, it is only fair that I also mention something I like about it.

KDE 4.1 comes with Okteta, a gui-based Hex editor, and its pretty easy to use even for someone like me who has no experience with editing raw binary files (*). It turned out to be very useful when I need to alter some Eschalon Book 1 save game files (more on this in a later post).

Still, GUIs are only so useful, and I had to use the command line "cmp" utility to identify the bytes that differed between two different save game files before I could actually do anything useful with Okteta. Another cool command line tool I discovered for the first time: "xxd." Long live the all-powerful command line!

(*) No experience with messing around in binary of unknown format. Binary files with known formats are easy to deal with, you just write a program to process them.)

Do big studio computer games have less bang for buck these days?

Maybe I'm becoming old and jaded, but recently big budget games (PC or console) have been a bit of a disappointment to me.... $60 for <10 hours play time in a First Person Shooter (with maybe 2-3 times more the play time in an CRPG), and frustrating game-play seems the norm these days. I'd even have to buy a new PC to play some of the latest and greatest PC-only titles, as my trusty old Pentium 4 requires the obsolete AGP graphics. With less free time and spare cash (who could have thought that the rampant ideologically-driven deregulation of the financial markets might cause problems!) than I used to have its seems more important to find something decent to play...


After recent disappointments with NWN2 (and its expansion Mask of the Betrayer), Oblivion: Shivering Isles (Oblivion was good, but not as fun as Morrowind) and Star Wars: The Force Unleashed, and giving more money that I can really afford to the Democrats I've been looking for something where the emphasis is on fun and game-play, rather than graphics and movie-quality cut-scene work.

The original title of this post was going to be "Are big studio computer games less fun these days?" But what is fun? Certainly modern computer games, in particular those mentioned above, are graphically outstanding compared to games of even five years ago, let alone 10 or 15 years ago. Yet I remember getting a lot of enjoyment out of such graphically-primitive games such as Legend of Zelda, Civ 2, and X-Com: Terror from the Deep (probably my favorite games of all time). Basically the ingredients and mechanics of game play of different genres of computer games really haven't changed significantly over the same time, and its game play that counts for a lot in terms of enjoyment.

As I also spent vast amount of time playing those games - not only did they take a long time to master and play through, but they kept my attention for the ~100 hours I put into them. If a game is going to be shorter, then I expect to pay less for it. Or to put it another way (combining both fun and value) bang-for-buck is an element in my relative enjoyment of something.

That is not to say pure economics (or buyer's remorse) is the sole determinant of fun (remember, game play counts), but I think it also a crucial part of the recipe. For example, a typical Hollywood big-budget movie (e.g. the latest Indiana Jones movie) might be more fun when rented through Netflix for a net cost of a few bucks, than if you went to the theatre and paid $9 + gas and parking to see it.

Personally, as far as games are concerned I think the paying less than or approximately equal to $1 per hour of first-time game play is great value (e.g. KOTOR, Morrowind, Oblivion), ~a few $ per hour decent (e.g. Far Cry, Half Life 2, Assassin's Creed), and more than $5 per hour (e.g. SW: TFU, Halo 3 single player) poor value for money.

So if my hypothesis is right, what types of games should be fun if I'm not liking the games produced by the current big game studios?

If game play is roughly the same, and graphics really aren't super important, and bang-for-buck is a factor, then games by small independent developers should be fun. They may be short, or lack graphical or play testing finesse, but they should be enjoyable. (The alternative is simply to purchase the big studio games about a year after they first came out, at which their reduction in price makes up for their imperfections.)

So instead of the big games studios, I'm going to give independent developers a chance. In particular developers who support multiple operating systems (I keep a Windows machine to play games on, but if I could play games on other OSs I would happily avoid the use of Windows).

Playgreenhouse.com is perhaps best known as the distributor of Penny Arcade's "On the Rain-Slick Precipice of Darkness", but it now has a few other games available, all for prices of $20 or less. To put my money where my mouth is I've purchased the old-school isometric CRPG Eschalon Book 1 (linux version) for the princely sum of $19.95.

I've spent about 10 hours playing so far, but will hold off on the full review for a while. In the mean time here are some reviews of the game (with screen-shots in some cases) of Eschalon Book 1, should you be interested in wasting a few minutes reading time:
http://www.shamusyoung.com/twentysidedtale/?p=1485
http://jayisgames.com/archives/2008/06/eschalon_book_1.php
http://www.2404.org/reviews/2881/Eschalon:-Book-I-Review
http://www.scorpia.com/?p=812

Who contributes most to the Linux kernel?

The latest Redhat magazine has an interesting aside on who contributes most to the Linux kernel. According to Greg Kroah-Hartman's (*) analysis, and as measured in raw number of patches the general community comes in first, with Redhat coming in second. Canonical, the commercial sponsor of the now popular Ubuntu (a spiffy modification of Debian) comes in 79th.

The overall largest contributions to Linux code come from individuals who have no apparent affiliation with any company, as Kroah-Hartman surmised by looking at their e-mail addresses. Red Hat came in second overall, with 11,846 patches.

By comparison, Canonical, the company behind Ubuntu Linux, is the 79th most active contributor, with 100 patches. Kroah-Hartman said that such behavior on the part of Canonical will be detrimental to the company and the Ubuntu distribution over time [emphasis mine].

“Then there are the distros that base themselves off of other distros, like Ubuntu and [Lance Davis’] CentOS. These distros have yet another layer between them and the original developers. Patches rarely, if ever, flow backwards into an upstream distro, and the developers are very unlikely to push their changes into the upstream packages as they don’t feel the need or don’t realize the issues involved as they rely on the upstream distro so tightly,” said Kroah-Hartman.

This is interesting, and not just as food for a meaningless distro-war. While I personally prefer Fedora to Ubuntu or Gentoo, I can recognize that as a distribution Ubuntu is pretty slick and well structured, and has done a lot to combat the myth that Linux is unfriendly and difficult. Ubuntu has been the most popular distribution according to distrowatch for several years. But somehow the heavy work on Ubuntu has not not flowed back into the kernel, at least as measured in patches. Why?

If forced to speculate without any evidence to support it I'd guess that much of the work on Ubuntu has been in terms of user-interface and applications, and not at the deeper level of the Linux kernel. Yet Redhat clearly feels the need to work very heavily at the kernel level. So this guess doesn't satisfactorily answer the question.

If anyone has a better hypothesis or insight, please let me know.

(*) Greg Kroah-Hartman works for Novell's SUSE labs division.

Linux Journal covers number crunching with GPUs and scientific python

From a number cruncher's perspective the November 2008 issue of Linux Journal is actually quite interesting. In addition to an article on the petaflop Roadrunner supercomputer at LANL, there are two articles on number crunching with GPUs (one by Robert Farber, the other by Michael Wolfe) and one article on numpy/scipy by Joey Bernard.

Farber's article is really an advert for NVIDIA's CUDA, and unfortunately doesn't actually show any examples of doing something (this is something of a trend in LJ, as it seems to drift away from showing you how to actually code things and toward merely describing point-and-click mega-code-projects). Wolfe's article shows snippets of matrix multiplication using both CUDA and Brook, but is really more of a discussion about how to write a compiler that would automatically parallelize for GPGPU work.

Joey Bernard's article on numpy does try to scratch the surface with simple worked examples, including the use of matplotlib for plotting (or "ipython -pylab").

For a moment after reading his article I was terrified that I'd totally messed up using numpy in my python projects, as Bernard states that array multiplication in numpy (e.g. a3=a1*a2) is handled as a matrix multiplication!

However, its pretty easy to verify that unless you specifically created matrix objects (which his example did not do), then a1*a2 is an element-wise array multiplication.

To do matrix-like multiplication on numpy array objects you need to specifically do something like "a4=numpy.dot(a1,a2)" or "a5=numpy.mat(a1)*numpy.mat(a2)."

Anyway, I hope this issue is a sign that Linux Journal may get back to publishing more "hands-on" articles on general programming topics interspersed between all the Web 2.0 and database articles.

Less than impressed with KDE4

My first Linux window manager experience was with fvwm - a step up from twm but hardly fancy and barely, just barely, functional. I forget the names of the window managers we used on the Decs and Suns used back in the mid-90's, but they weren't particularly fancy either. Since then I've used fvwm2, early versions of Enlightenment, Gnome (version 1), KDE2 and KDE3.

And over the years there has generally been progress, and progress by definition is good. Progress until now, that is.

I switched to KDE after experiencing major dissatisfaction with Gnome 2, and now that KDE4 is out I'm contemplating switching window managers again. Only it can't be the latest version of Gnome, as I have that on a spare laptop and don't like that either.

My first experience of KDE4 was on Kubuntu 8.04, an upgrade from 7.10 on an old Dell Latitude C400. That didn't last long enough for me to get a taste of KDE4, as the machine started freezing for no obvious reason after relatively short uptimes. A wipe and fresh install of Fedora 8 with KDE3.5 and the laptop worked fine once again. Was it Ubuntu, or KDE4?

Upgrading my home desktop to Fedora 9 (with KDE4.0) was the first real experience I had with KDE 4 and I was not impressed. Yes - I've read the blogs that discuss the fundamental reasons for chucking out vast amounts of the KDE3 code and rewriting things like the panel from scratch. Maintainable, extensible code is good.

But when the panel clock widget can't even handle displaying seconds you begin to think of KDE4 as a major step backwards. For an application as simple as a clock the KDE4 folks failed to provide the same functionality as KDE3. How the hell did this happen?

As for adding a terminal launcher to the panel - what happened to "Add Application Launcher?" I can add a widget - there seem to be an large number of widgets, most of which are thoroughly useless - but I don't want a widget. It took at least 30 minutes of messing around to work out that items in the launch menu (Kickoff its called, replacing Kmenu), when right clicked on, give an option to add to the panel. But you can't click on the panel itself and add an application launcher from there, even though its (a) the way KDE2 and KDE3 worked and hence the way KDE users expect, and (b) the old was more intuitive.

Oh, and don't edit the launcher's setting because the terminal icon will disappear and be replaced with a question mark. Progress, I tell you.

And transparent Konsoles (terminals) - we used to have those. Except now we can't have that. Sure, we can make the whole window transparent, including the writing, but that makes the writing hard to read.

As for the new panel (Plasma), well, hum, I totally agree that the power to add widgets to the desktop as well as the panel really makes up for the major lack on control over the panel appearance and lack of old panel (kicker features). That's what Linux is all about. Eye candy. No one really uses it because it is (or used to be) more functional, that is just a lie we tell the Windoze fanboyz.

Yes, I'm being sarcastic.

Its enough to make you suspect that major parts of KDE4 were written by 13-year olds whose Ritalin or interest ran out before they'd finished coding even the most basic of applications.

Now I'm a patient kind of guy. KDE4 and Fedora 9 came out ages ago and I refrained from complaining because I expected these "bugs" to be ironed out. "Yes, all very unfortunate to roll out a product clearly not ready for prime time, and that is functionally inferior to its predecessor, but I'm sure it'll get better" I thought in my typical latte-sipping, elitist, East Coast liberal mode of thinking...

So I waited. And lo and behold, the clock widget now does seconds. Hallelujah Brother! Can I get an Amen? Except the damn Konsole launcher in the panel is now failing, and takes a minute or two to tell me "KDEInit: can not launch /usr/bin/konsole". Acting on a hunch I remove the --enable-transparency option and it works again... for now.

So KDE4 remains on probation as far as I'm concerned. I have a lot of experience invested in KDE, which I am loath to lose by switching to Enlightenment or XFCE, but I'm warning you KDE: Mess me around some more and it is over.