Jakoblog ...home of jpetso

Apart from Version Control API, I've also been working on another piece of code, and after a long time of fuzzing around with architecture and internals it's now slowly getting to a state where one can get excited about it. I didn't make it in time to show it off at DrupalCon DC, but now the module can be found on drupal.org: Say hello to Transformations, your favorite new uncut diamond for performing data transformations in Drupal (including, but not limited to import/export).

Experience has shown that starting with the underlying concepts always triggers the question "So what does it actually do?", so let's start with the use cases that lead to the module's inception:

• We [1] want to import CSV data into CCK nodes, mapping CSV columns to node title/body or CCK fields. Ok, that's easy: just take Node import and be done with it. But there's more...
• We want to take XML files conforming to given standardized XML schema, and map some of its fields to the CCK node just like one would do it with CSV fields. That's a bit more difficult, because it needs to deal with nested structures instead of a simple row/column based CSV table.
• Now it's getting fun: We want a single (nested) XML item to import as multiple nodes, for example, into a Pageroute collection.
• Oh right, and exporting all of those nodes into their original formats should be possible as well.
• Why stop with plain import and export? Combining those two, we could build a format bridge that (for example) converts between CSV and XML data, or between two different types of XML structures.
• Also, we've got a site with lots of content types and, consequently, lots of different CCK fields. There were several times when it was necessary to change the structure of these content types, or even split up one type of node into a collection of nodes. (Note the similarity to the Pageroute example above?) It's kinda tedious to write update routines for those "internal migrations" in code - why not map CCK fields to each other in a before/after fashion?
• Lastly, have you thought of the possibility that imported data doesn't come in the required format? For example, a date could be given as timestamp, as ISO date string, or as combination of three XML tags named <date><year>2009</year><month>04</month><day>10</day></date>. Maybe that data is not even the data that we want to import, but only references the actual data. There needs to be some way to convert formats on importing stuff, or use the available data to retrieve the actually desired data.

Now, I hear someone arguing like, "Scope creep! When you pack all that functionality into a single module, it will get bloated, hard to use and unmaintainable." And to a certain extent, I agree with that. As far as I can see, a major reason that Import/Export API failed is that it tried to establish a single common data format that described all possible information, and moreover, every data backend (Drupal nodes, CSV, XML) was required to cope with all that information. In other words, the architecture of Import/Export API might have required too much of a monolithic approach, which lead to the maintainability problems that were at least part of the reason why the project stalled.

Does that mean that generic import/export systems are unfeasible? I don't believe so, and I hope Transformations' architecture takes enough precautions to avoid the fate of Import/Export API.

The key idea to Transformations is that we'll never be able to capture all possible data formats and structures in a single module. What we can do, though, is to decompose that data into bite-size pieces, and define operations that can process those. (For example, operations transforming a Unix timestamp into an ISO date string or into a PHP DateTime object are pretty straightforward.) Building on that, we can assemble the pieces into larger pieces of data, and we can also define operations on those. (Extract the first three dates from a list of dates? Sure, easy enough.) In essence, importing and exporting data is nothing more than decomposing data, processing it according to a given set of rules, and reassembling it in a different form.

The key idea to Transformations is that we don't want to know all those rules upfront, there's just too many ways in which data can be decomposed, processed and reassembled. So instead of trying to cover all use cases, let's provide users with the necessary tools to define their transformations by themselves. Let's not define a single common data format that all data needs to conform to - all we need is a set of data formats, plus the knowledge which operations work on which data, plus a way to wire them up. If you think that wiring up operations sounds like Yahoo! Pipes, you're on the right track - only that Transformations can actually deal with structure information (schemas), and that my current user interface is way more crude than Yahoo's nice JavaScript wizardry.

The key idea to Transformations is that if we want a generic import/export module, it's important not to provide a solution but to provide a framework. Transformations is a framework for creating data transformation pipelines. ETL for Drupal, you might say - a braindead attempt to eliminate a whole class of import/export modules, and a potential basis for a Yahoo! Pipes clone on Drupal. Transformations provides the means for developers and users to build their own stuff.

Oh, and Transformations is incomplete, unpolished and (beware!) object-oriented. Until now, I worked to get the concepts right. Next step: building on the foundations, and extending it to enable more concrete use cases being built upon it. For now, it can be used for CSV import/export in a way that is more complicated than Node Import will ever be. But the flexibility of the framework also promises possibilities that go way beyond nodes and two-dimensional tables.

Still reading? Still interested? Then go and check it out.

[1] "We" as in "my company, Pro.Karriere", or more specifically, my boss and visionary entrepreneur, Klaus.

Since the Summer of Code of Version Control API's inception ended, there had always been one piece of cvs.module functionality that hadn't yet been ported to the new VCS-independent infrastructure. Incidentally, this was also one of the most visible and important features on drupal.org: packaging releases from CVS tags and branches by means of a simple two-step form.

Well, not anymore! Yesterday, I released Version Control API 6.x-1.0-rc1, which sports a number of important improvements under the hood, and versioncontrol_project 6.x-1.0-rc1, which makes use of these new features to provide release node integration and a release tarball packaging script, like the one found on drupal.org. The main difference is that the code is not CVS specific, and even though only the CVS backend supports the required functionality at the moment there is nothing holding up an implementation for SVN or the newly-revamped Git backend.

The latter has pretty much singe-handedly been ported by new contributor CorniI (a.k.a. Cornelius R.) to Drupal 6 and to API changes in Version Control API for 6.x-1.x. He's not yet done making sure that everything works as intended, but most importantly the Git backend runs again and can be played with. Good times coming for DVCS support!

So with Version Control API & friends being "feature-complete", the next step is to push it onto drupal.org and replace cvs.module. For that to happen, the primary task is to complete the migration scripts that pull data from cvs.module to Version Control API, If you want to help out with that, you can get in contact with me, dww and hunmonk - unless new cvs.module functionality is added in the meantime, the data migration is the last missing piece for getting deployed on drupal.org.

In general, the roadmap for Version Control API looks like this:

• Make the 6.x-1.x releases suitable for drupal.org, running on CVS only (like d.o does today). This is important so that improvements get contributed to Version Control API instead of cvs.module, which avoids porting more dozens of newly contributed features.
• Changes that make it possible to run drupal.org on other version control systems will go into a 6.x-2.x branch. That includes user authentication changes to enable SSH key or HTTP(s) authentication, and important non-CVS log parsing performance improvements that can be achieved by making the API object-oriented, and stuff that makes distributed version control systems work more nicely with Version Control API.

Hope to see you contributing soon!

Version Control API is back! There's no stable 5.x-2.0 release yet, and the Drupal 6 version will only be tackled after that happens (i.e. next year), but we're alive and kicking again. And yes, I'm still determined to get it to replace cvs.module on drupal.org. Even if it might take another year until all the odds and ends come together.

Anyways, now is the time for potential contributors to jump in and create recipes on how to set up version control integrated sites, to port the Git and Mercurial backends, create a new one for Bazaar, to make the repository viewer work on previous revisions or to improve the CVS backend so that it's browsable like the SVN backend's development version. If you're ambitious, you might also try porting drupal.org's release scripts to the Version Control API so that automated releases can be made with Subversion and other version control systems.

Whatever the outcome, the era of cumbersome reinventions is over for the time being. Time to get started on features (and the D6 port, soonish) - let's push cvs.module into irrelevance and take version control integration with Drupal to the next level!

If you're passionate about version control systems, here's where you can make a difference!

It all started when I upgraded to the new Kubuntu 8.10, a.k.a. "Intrepid", which did a bunch of nice things to my laptop - most importantly, it resurrected my beloved suspend-to-RAM that went out of order in Hardy. I couldn't be more grateful for that. Intrepid even made the volume keys work... yay!

Of course, every Kubuntu update has its downsides as well. This time, the Wrath of Shuttleworth manifested as a series of interrelated X.org issues: stuttering videos (which played perfectly fine in Hardy), mouse cursor jumps once I keep the mouse quiet for a few seconds, and a curious lack of autoconfiguration for the VGA output - something that had mysteriously worked since the previous release, so I had not questioned it despite knowing that Xrandr 1.2 was not the most enthusiastically hyped area of KDE development known to mankind.

Not possessing or even requiring a projector or external monitor by myself, I was able to live with this for a month or so. But when I had to borrow a collegue's laptop for a university presentation last Friday - after not getting my own machine to spit out proper signals - the Time of Change had come. (And no, I don't mean a certain KDE release with that codename.) On that day, I was torn between no VGA output at all on the one hand, and resolution mismatch problems on the other hand. Neither did I know how this stuff even works, nor that I could have easily solved all of this with a few simple calls to the xrandr command line utility.

I had to get my VGA output working though, because this Tuesday I would throw a movie night at my place, with a nice projector borrowed from my parents, lighting the White Wall of Moving Pictures. I needed to get rid of both the playback stutter and the projector output uglyness. Step by step, I got to the root of all evil.

First step: Figuring out the bad guy. With Google as my weapon of choice, an Ubuntu forum thread about the playback stutter issue was to be found, strengthening my hopes of not suffering alone. The solution of switching to Fluxbox or GNOME is not one of my favorite plans though, but at least it helped determine that KWin is innocent (by making it work in a Fluxbox session, but not with Compiz in a KDE session). Another thread pointed out the amazing correlation between mouse jumps and video stutter.

So while writing all of this up in a bug report, I remembered a blog entry by nixternal pointing out that the "Detecting RANDR (monitor) changes" kded service (to be found in System Settings) was causing problems for people with external monitors. In a stroke of luck, disabling that service was indeed enough to get rid of both the stutter and the jumps. Terrific.

But I still had to get the connection with my projector to work. So I immersed myself in the skillful use of the xrandr command line utility, and went on a search for the 10-second timer that messed up not only my system but that of countless others as well. Turns out that the "Detecting RANDR (monitor) changes" service is not even a part of KDE proper - it's a semi-private branch of krandr by KWin maintainer Lubos Lunak. The thing that causes my video to stutter, my mouse to jump, but also causes other monitors to flicker or recalibrate: all of that is caused by recurring polling of X, in order to notice when external video outputs are connected. (Certain hardware is not able to notify applications in certain scenarios, so that polling would have been ok if it didn't cause havoc on so many systems.)

Second step: getting it all to work! For my movie night, the combination of disabled kded service and xrandr fuzzing was sufficient to get the film nicely rolling. However, it was a slight disappointment that KDE's own display configuration in System Settings ("Size & Orientation", a.k.a. krandr) failed so ungracefully. When it started up, it automatically switched my laptop screen ("LVDS") to some weird resolution (1360x768, I believe), and not for the Love of God would it enable the projector output ("VGA"). All of this while the seemingly trivial xrandr command line tool achieved those tasks rather nicely.

Needless to say, I wasted another day with diving into the krandr code, and fixing one issue after the other. So the net gain for you, dear reader, will be a System Settings screen configuration tool in KDE 4.2 that Mostly Works (TM). Unthinkable joy shall be your reward for reading this crap all the way down! Congrats for your persistence. Dual-monitor setups? Nope, sorry. But reliable mode switching that doesn't make you go OMG WTF BBQ and doesn't mess up your screens if you merely start it. That is what you'll get.

I also hope that Kubuntu will soon update to the 7-week-old latest revision of that notorious "Detecting RANDR (monitor) changes" service, because that version already (already?) has the 10-second fuck-up timer disabled. I only would have hoped that the merging of unreleased hacks into the default setups of every Kubuntu user out there was not taken so lightly, but hey - I still haven't switched to Fedora or OpenSUSE so it can't be that bad. (...right?)

Third step: getting to know the latest trends. Seems like Plasma in 4.2 will feature a new screen management library called kephal, which is going to replace krandr in the near future. Chances are KDE and Plasma get decent multi-head support soon, but that's not quite my area of expertise - I'm just the occasional janitor, whereas Aike or Aaron have a much better grasp what the future holds for the neverending construction site that is screen management. Personally, I'm looking forward to getting my now-working configuration tool not broken until the next Surge of Technology sweeps through the lands of X.org.

Yeah, you might have read about the AutoBrace KTextEditor plugin which I introduced two weeks ago on this blog. You might even have tried it out, but it didn't work for you because it crashed under certain circumstances, it messed up pasted text, it broke undo functionality and it didn't do additional indentation although that would have been useful. You know who to blame for all of these shortcomings! (Right, that would be me.)

But fear not! I found ways to fix all of these, and it now works brilliantly and without a glitch. Unlike last time, I'm now even using the plugin myself. And on top of that, I even created an Oxygen icon (yay!) to display in the "Extensions" listing:

Now this makes you want to use Kate (or KDevelop) with AutoBrace just sooo badly, doesn't it? Heh, I knew it.

Hi Planet KDE!

You might already know me from 2006 or 2007, but currently I'm gone from KDE because of other commitments. So technically you shouldn't hear from me right now; however, my buddy klausi drove me to it.

Surprising high amounts of KDE coders use non-KDE technology like vi or emacs to hack on KDE, and mostly they've got good reasons to do so. The only option to get those people use Kate or KDevelop is to continually improve those. The respective developer teams are doing awesome work in that regard, while my contribution for today is just a small KTextEditor plugin. (Which means you can use it in Kate, KDevelop or whatever KatePart-based editors exist out there.)

The plugin is called AutoBrace and, frankly, an uninspired clone of the best feature that you'll find in Eclipse (apart from the compile-as-you-go automatism and sheer infinite amounts of complexity). Right, it's the automatic insertion of a closing brace ("}") after you ended a line with an opening one ("{"). Kate's standard insertion of parentheses, brackets and braces is simply too obtrusive for me, so even if I tried it several times I always had to disable it again because I can't stand it. So here's how AutoBrace works.

1. You, the developer, are writing a line of code, and want to open a new block delimited by braces. Looks like this:

1. If you press Enter and AutoBrace is enabled, it will insert your favorite new brace where it belongs. Looks like this:

It's no big deal in the code - actually, I should be embarrassed to blog about such a minor thing when other people are continuously rocking KDE with stuff like SVG support in KHTML (new), a whole new power management solution for KDE (relatively new), or a window manager that's just terrific (not new, but still awesome).

But I've got no scruples, so here's your blog post anyways. You can get AutoBrace right now for your current Kate, you just need to compile it beforehand. Here is where I put it. See you some other time!

For a moment, I grew tired of saving the world with complex modules like Version Control API. (Which, btw, after a long time, is slowly nearing the completion of the 5.x-2.x branch. But more on that when it's done.) I also can't seem to stick to my "stealth mode" policy of avoiding the Drupal community for even a short time, so I'm like, well... whatever!

Of course, you're asking yourself how I shall ever become a Drupal rockstar if Version Control API 5.x-2.x is not yet done and my super secret diploma thesis project (yeah, that secret!) is still in the beginnings. And of course, the answer to that legitimate question is to pull an Eaton. Like, putting together a few trivial lines of code to produce a module that everyone wants to use, or stuff. So that's what I did! Ok ok, I'm getting to the point already...

"Do you love RSS and comment threads? Do you hate checking a page dozens of times just to see whether new comments have appeared there, or whether there are comments at all? Have you ever envied those Wordpress users with their blogs showing the current number of comments as a dynamically-generated image? You haven't? Dude, you don't know what you're missing out on."

Anyways, there's a nice blog using a (still) nice theme, so on a quick look everything would indicate that the blog is running on Drupal. It's not, however, because no Drupal site has those "Comments: xyz" images appended to their feeds.

Let's change that. For the sake of our feed-gorging users, every Drupal blog needs this functionality. At least that's what I thought, so I proudly introduce a module bringing no innovation at all - a functionality-wise clone of original Wordpress goodness, but for Drupal! Say hello to Comment Count Image, your favorite new syndication-related module. And here's the screenshot, because you asked for it. (You did, didn't you? ...right!)