Jakoblog ...home of jpetso

Apart from Version Control API, I've also been working on another piece of code, and after a long time of fuzzing around with architecture and internals it's now slowly getting to a state where one can get excited about it. I didn't make it in time to show it off at DrupalCon DC, but now the module can be found on drupal.org: Say hello to Transformations, your favorite new uncut diamond for performing data transformations in Drupal (including, but not limited to import/export).

Experience has shown that starting with the underlying concepts always triggers the question "So what does it actually do?", so let's start with the use cases that lead to the module's inception:

• We [1] want to import CSV data into CCK nodes, mapping CSV columns to node title/body or CCK fields. Ok, that's easy: just take Node import and be done with it. But there's more...
• We want to take XML files conforming to given standardized XML schema, and map some of its fields to the CCK node just like one would do it with CSV fields. That's a bit more difficult, because it needs to deal with nested structures instead of a simple row/column based CSV table.
• Now it's getting fun: We want a single (nested) XML item to import as multiple nodes, for example, into a Pageroute collection.
• Oh right, and exporting all of those nodes into their original formats should be possible as well.
• Why stop with plain import and export? Combining those two, we could build a format bridge that (for example) converts between CSV and XML data, or between two different types of XML structures.
• Also, we've got a site with lots of content types and, consequently, lots of different CCK fields. There were several times when it was necessary to change the structure of these content types, or even split up one type of node into a collection of nodes. (Note the similarity to the Pageroute example above?) It's kinda tedious to write update routines for those "internal migrations" in code - why not map CCK fields to each other in a before/after fashion?
• Lastly, have you thought of the possibility that imported data doesn't come in the required format? For example, a date could be given as timestamp, as ISO date string, or as combination of three XML tags named <date><year>2009</year><month>04</month><day>10</day></date>. Maybe that data is not even the data that we want to import, but only references the actual data. There needs to be some way to convert formats on importing stuff, or use the available data to retrieve the actually desired data.

Now, I hear someone arguing like, "Scope creep! When you pack all that functionality into a single module, it will get bloated, hard to use and unmaintainable." And to a certain extent, I agree with that. As far as I can see, a major reason that Import/Export API failed is that it tried to establish a single common data format that described all possible information, and moreover, every data backend (Drupal nodes, CSV, XML) was required to cope with all that information. In other words, the architecture of Import/Export API might have required too much of a monolithic approach, which lead to the maintainability problems that were at least part of the reason why the project stalled.

Does that mean that generic import/export systems are unfeasible? I don't believe so, and I hope Transformations' architecture takes enough precautions to avoid the fate of Import/Export API.

The key idea to Transformations is that we'll never be able to capture all possible data formats and structures in a single module. What we can do, though, is to decompose that data into bite-size pieces, and define operations that can process those. (For example, operations transforming a Unix timestamp into an ISO date string or into a PHP DateTime object are pretty straightforward.) Building on that, we can assemble the pieces into larger pieces of data, and we can also define operations on those. (Extract the first three dates from a list of dates? Sure, easy enough.) In essence, importing and exporting data is nothing more than decomposing data, processing it according to a given set of rules, and reassembling it in a different form.

The key idea to Transformations is that we don't want to know all those rules upfront, there's just too many ways in which data can be decomposed, processed and reassembled. So instead of trying to cover all use cases, let's provide users with the necessary tools to define their transformations by themselves. Let's not define a single common data format that all data needs to conform to - all we need is a set of data formats, plus the knowledge which operations work on which data, plus a way to wire them up. If you think that wiring up operations sounds like Yahoo! Pipes, you're on the right track - only that Transformations can actually deal with structure information (schemas), and that my current user interface is way more crude than Yahoo's nice JavaScript wizardry.

The key idea to Transformations is that if we want a generic import/export module, it's important not to provide a solution but to provide a framework. Transformations is a framework for creating data transformation pipelines. ETL for Drupal, you might say - a braindead attempt to eliminate a whole class of import/export modules, and a potential basis for a Yahoo! Pipes clone on Drupal. Transformations provides the means for developers and users to build their own stuff.

Oh, and Transformations is incomplete, unpolished and (beware!) object-oriented. Until now, I worked to get the concepts right. Next step: building on the foundations, and extending it to enable more concrete use cases being built upon it. For now, it can be used for CSV import/export in a way that is more complicated than Node Import will ever be. But the flexibility of the framework also promises possibilities that go way beyond nodes and two-dimensional tables.

Still reading? Still interested? Then go and check it out.

[1] "We" as in "my company, Pro.Karriere", or more specifically, my boss and visionary entrepreneur, Klaus.