I will give a brief overview of how it works. All you need is a model with a column that should hold a foreign key to the parent and a column to hold an index number for the position of the element inside its hierarchy. The defaults (convention over configuration!) are “parent_id” and “position”. Suppose our model (the one we want to nest and sort) is named “Item”. You will only need one line of code to implement a data source for it in your controller:

class ItemsController < ApplicationController
	nested_sortable_data_source_for :item
end

By default the data source will be created under an action named “order”, inside your controller. Don’t worry, all the defaults can be overridden (wait for the docs or take a peak at the code).

In your view, it will be pretty easy to add the NestedSortable Widget, as always:

<script>
	jQuery( function($) {
		$('#items_sort').NestedSortableWidget({
 			loadUrl: "/items/order/1.json"
 		});
	});
</script>
<div id="items_sort"></div>

Don’t forget that you will need to include the scripts (jQuery, Interface, etc) and the CSS file for the widget, by putting something like this in the HEAD of your HTML file, in your layout file probably:

<%= stylesheet_link_tag 'ns/nestedsortablewidget' %>
<%= javascript_include_tag "jquery-1.1.4.js" %>
<%= javascript_include_tag "interface-1.2.js" %>
<%= javascript_include_tag "inestedsortable-1.0.1.pack.js" %>
<%= javascript_include_tag "jquery.nestedsortablewidget-1.0.pack.js" %>

Ok, you say. That is neat, but how do I customize what data will be displayed in the columns by my NestedSortable Widget? The default is to display all the columns/attributes your model has (excluding the ID and foreign keys). You can use the :columns option of the nested_sortable_data_source_for to make your data source display pretty much anything you want, in any order you want. Just an example:

nested_sortable_data_source_for(
	:item,
	:columns=>[
		[:title, 'Title of the Item'],
		[:description, 'Description of the Item'],
		[ lambda {|i| i.created_at.to_date.to_formatted_s(:long)}, 'Creation Date'],
		[ lambda {|i| i.attachments.count}, 'Attachments']
	]
)

If you are smart enough you already got it: it will display a table with 4 columns, the title of the item, description, creation date and the number of attachments it has. Note the use of lambdas (functions that are passed as variables). It gives you the power to display whatever you want in the columns of your NestedSortable Widget.

Ok again, you say. The way things are until now will only allow me to have one set of items per database table. In your application, you will usually have a lot of users, and will not want to create one table per user. You want everything in the same table, and your “item” model probably have a “user_id” foreign key. That has been thought of as well. You have the :conditions option, which will give you the flexibility you need. An example:

nested_sortable_data_source_for(
	:item,
	:conditions=> lambda {|controller| {:user_id => controller.params[:id]} }
)

Note the funky lambdas again. Instead of a lambda, you may pass in a symbol, like :conditions_generator with the name of a method in your controller (make it protected, please) that will generate the conditions for you. The conditions are exactly like the ones you may use in your finders in Rails: they may be strings with SQL, they may be arrays like ["a = ?", "b"], and anything else that is allowed in Rails conditions.

In your view, in the part you set up the widget, you would probably use something like:

loadUrl: "/items/order/<%=current_user.id%>.json"

This is insecure: users could easily change the order of the items that belong to other users, with little effort. To implement security, you need to make our condition generator a little bit smarter. It is up to you, though…

Briefly, that is what this plugin for Rails does: it makes it ridiculously easy to implement a NestedSortable Widget in your Rails apps (I bet you could do it in 5 minutes!). Don’t show it to your boss, he will probably fire you and begin doing it himself. Note that the data source you create fully supports pagination: you may use any option the NestedSortable Widget provides and it should work like a charm.

To try the plugin, install it using Rails install command, in your Rails application path:

script/plugin install https://nestedsortables.googlecode.com/svn/trunk/rails/nested_sortable

Get it here: http://nestedsortables.googlecode.com/

jQuery 1.2 was released and it is incompatible with Interface. Some things work, some don’t, and it turns out there is a bug with jQuery 1.2 and the Sortable, in IE only (thanks n3dst4 for reporting). So I am updating my tests to use jQuery 1.1.4, and recommend you guys avoiding  jQuery 1.2 until Interface gets updated (probably never) or the new jQuery user interface plugin, jQuery UI,  is usable enough  to replace it entirely (it is still a little far from it). I wonder whether releasing jQuery 1.2, without making sure it was compatible with Interface,  or waiting until jQuery UI was fully baked, was a good move. It might hurt jQuery popularity, since its current version doesn’t have a decent set of UI plugins (Interface being incompatible and jQuery UI pre-alpha), the first thing people look for when using a JS library.

So, in the not so distant future, I might need to update NestedSortable plugins to use jQuery UI, instead of Interface. It will be a good excuse for a 2.0 version…

After a lot of helpful comments from folks at the WP-Hackers list, I improved a few details in the WordPress page ordering. I also managed to fix all the remaining problems I mentioned in my last post (more about them later on), so I believe it is now ready to be rolled out in the development version. To try it out, you can check my sandbox (user: test / pass: test), download the development version of WordPress with it (up-to-date as of 19-Sep 2007 16-Sep-2007 WP from trunk), or apply it to your own working copy of WordPress from SVN trunk using this patch (you will probably also need this image, which I think can’t be included in the patch). I will submit the patch to the WordPress TRAC. I hope it gets reviewed and accepted.

Besides cosmetic improvements (animations, button position), I changed two configuration parameters for the NestedSortable Widget (can be easily reverted at wp-admin/js/edit-page-order.js), when compared to the first showcase. It is now paginating at 20 items. I also am turning on the “greddy” option, what means the list will be downloaded all at once, even though it will be displayed paginated. Why would I want to do that? Three reasons:

• Each additional item in the list takes about 90 bytes, on average. So, if you had 1000 pages, it would be only 90kB to download. Downloading 90kB at once is way faster than downloading 5kB 20 times, mostly because of server processing.
• Server processing is what takes longer here. When fetching just 50 items out of 1000, the server is actually loading the whole list, then filtering only what we want (yes, this could be optimized, but the issues with hierarchies make it a complex task). So, if we load tiny parts of the list 20 times, it should take 20 times more processing than loading the whole thing at once.
• Paginating the list on the client is still important, because the JavaScript will get heavy on the client’s CPU when you have lists of more than 50 items to sort.

If you still prefer the old fashioned way, all it takes is setting ‘greddy’ to false in wp-admin/js/edit-page-order.js.

What did I have to change in WordPress?

Due to the component nature of the plugins I developed, very little had to be changed into the native WordPress code to add this functionality. The only ‘real’ PHP code created was for the data source that feeds the plugin with JSON and saves the page order, and it was less than 200 lines long. JavaScript was also minimal, since 99% of the logic was in the jQuery plugins I created. The other modifications were minor. Even the edit-pages.php was almost untouched.

I will briefly describe the files that were altered/added:

wp-includes/class-page-order.php – added - this class encapsulates most of the “data-source” functionality, and takes care of both loading the page list, with pagination support, and saving it back to the DB.

wp-includes/class-JSON.php – added - class by Michal Migurski that will convert php data structures to JSON (and also from, but I didn’t need it).

wp-admin/admin-ajax.php – altered - Added two actions to serve as the data source for the widget, one for fetching ( ‘page-order-load’) and other for saving (‘page-order-save’) the order. The actions are small, since they use the Page_Order class.

wp-admin/edit-pages.php – altered - Just a couple of lines to load the JavaScript and CSS that builds the widget, if the user has the permission to order pages.

wp-admin/js/edit-page-order.js – added - This script will add the “Edit Page Order” button, that will build the NestedSortable Widget on the user’s demand. It also holds the configuration of the widget.

wp-admin/css/nestedsortablewidget.css – added - CSS for the plugin.

wp-admin/images/progress_indicator.gif – added - AJAX loading animated gif.

wp-admin/includes/schema.php – altered - Had to change this to add a ‘order-pages’ type of permission to the database. This is likely to break as the DB version number evolves.

wp-admin/includes/upgrade.php – altered - Had to change this to add a ‘order-pages’ type of permission to the database. This is likely to break as the DB version number evolves.

wp-includes/version.php – altered - I had to increment the DB version to make WP upgrade the DB and add the ‘order-pages’ permission.

wp-includes/script-loader.php - altered - added the declaration for the JavaScript files I added, and the text with internationalization.

wp-includes/js/jquery/inestedsortable.js – added - the first of my two jQuery plugins, compressed

wp-includes/js/jquery/jquery.nestedsortablewidget.js – added - the second of my two jQuery plugins, compressed

Will I assemble it as a WordPress plugin?

The sort answer is no, unless there is great demand and strong reasons not to apply this to the core WordPress. Don’t take me wrong, I love plugins. But I don’t think this is good for a plugin, it is too much of a basic functionality. Most people won’t ever even look for a plugin to make sorting their pages faster, only in very special cases; they will just go ahead and sort them the hard way. There is already a plugin with this intention (my page order) and the author admits it doesn’t have that much demand. Making an analogy: car accessories are also great; but you wouldn’t be very happy with a car where the tires and the steering wheel were accessories.

I have set it up to paginate at only 10 items (please add more pages to play with it, I am lazy), to make it easier to see this functionality (the final version should paginate at 50 or more “pages per page”). Besides that, it is pretty much as it should be in a final version. There are probably a lot of minor details in the overall interaction and appearance which can be improved, so I am waiting for suggestions.

Here are a few ideas, questions and dark spots, that I would like people to comment on:

• Is the text used in the buttons/widget ok?
• If the user changes the order of a few pages, doesn’t save, and navigate away, he will lose his changes. Should a “Are you sure?” warning be displayed in this case?
• The position for the “Edit Page Order” button is ok? Perhaps it could it be duplicated or appear at the top of the original page list. Perhaps it could be a link. I think people shouldn’t use it that often, that is why I placed it the way it is now.

There is also a tiny problem that is bothering me a lot. The current version of Interface (1.2), that is in the WordPress SVN, has a bug that prevents the “AutoScroll” (which makes the browser window scroll when you are dragging a page near the edge of the viewport) functionality of the NestedSortable plugin to work. I don’t actually know where in Interface’s code this bug is, but I know it is fixed in its latest SVN version, which is the one I am using in all my examples (I have added it to my WordPress Sandbox and my patch, so you shouldn’t notice it). The problem is that the Interface from SVN is not JS compressed and is not unified in a single file, as is the 1.2 “stable” version. Adding it to WordPress would require doing that, and there are no “compile” scripts in the Interface SVN, as there are for jQuery, for instance. I don’t even know whether Interface is being maintained, so there might never be a 1.3 version out. I could disable it, but without “AutoScroll” it becomes obnoxious to drag the pages when you have long lists , which should be the case if you have more than 10 pages or so. You have to use the scrolling wheel, which some people don’t even have.

There is also another technical detail: I should probably add a “order_pages” permission to WordPress. I can’t use the “edit_page X” permission, as it is applied in an individual basis and there is no way to do that when you are ordering all the pages at once. Is it ok to create that? I will have to figure it out (never played with adding permissions to WordPress before), but I believe it should be no big deal. I also don’t know to what roles I should add it (Editor? Author?).

The widget is feature complete now and working well with the major browsers I have tested it (that is, FF2, IE7, IE6 and Safari 3). It took a little bit of hacking to remove the bugs in IE, but in Safari it worked like a charm the first time I tried it (and Safari is also the browser where it looks and feels faster – they probably have the best JavaScript interpreter out there). I changed a few things in the underlying Nested Sortable plugin, so it should be a lot faster now, in all browsers. The test demo can be seen here as usual and is now up-to-date with my latest SVN updates.

Serialization will, by default, send the data using POST and encoding the data using the “query string standard” , which most server side languages should automatically convert to their equivalent data structures for arrays and hashes. You can also configure the widget to return the data in JSON format (for that you need to include the JSON jQuery pluing) and process it in your server using one of the countless JSON libraries available. Following the REST principles under the HTTP protocol, if the HTTP response has a success code the widget will think data was saved correctly. You can even use the same server path (ie. the same script) to retrieve and save the data, since it will use GET to retrieve and POST to save.

I am aware that the “incremental” feature (showcased in the last example) does have a few problems when you change the item from one “page” to the next (even though you see things in a continuum , what you see is actually the different pages, each inside its own UL with a NestedSortable, being rendered in sequence). I am also aware there is a minor cosmetic issue with the spacing of columns in IE6. I am not much of a CSS browser quircks’ wizard and “incremental” shouldn’t be used in WordPress, I presume, so I am going to leave those issues to be solved after the Widget is integrated in WordPress.

If you find any other bugs or have more suggestions, please post a comment.

I have also added the “incremental” option, which will make the new pages that are loaded be incremented to the existing ones in the user display. This is usable by itself. My intention was to allow the user to display the list inside a scrollable div to go along with this option, but after a few tests, I am seeing the dimensions stored in the Sortables would need to be recalculated every time you scroll the div, and I am afraid making this work seamlessly could take some time and possible changes to Interface’s Sortables itself. It would probably be unbearably slow as well.

I changed the demo to showcase these and also a lot of other options that are available. The demo will now load 5 different instances of the widget.

It can be seen here.

I am posting a demo of what I am calling the Nested Sortable Widget here.

In this example I am only showing a small subset of the possible configuration options it has. The component/widget is basically finished, but there is still a tiny bug in IE when you drag items with more than one hierarchy, which I am going to investigate soon. The “save” function also need a few extra hours of work to be finished, but that doesn’t affect the demo. Reports for any other bugs are welcome.

The main features of the component are:

• Loads data from a JSON data source, with a predefined format (I will explain it once I create the final doc. for the widget, but it is very simple and can be inferred from the demo) . This format is very compact and very easy to generate and manipulate in any server side language, since it only uses standard data structures, like arrays and hashes, unlike XML.
• Support for pagination, both in the server (it will only load smaller chunks of data each time) and in the client (it will display smaller chunks of data at a time).
• Funky way to move items from one page to another, following Mike’s suggestion.
• Can display data with multiple columns, all you have to do is change your data source.
• Allows you to change the nesting direction to right to left, and it will change the order of the columns automatically.
• Lots and lots of easy options to to customize the visual appearance of the widget and most of it other aspects. Don’t worry, I will document them all (or you may dig in the code and figure it out now ).

More details about pagination

In the demo I am showing it being used with pagination, with 5 items per page, on a dataset of about 30 items. I set up a simple PHP page as a demo data source that will spit clusters of a bigger JSON with the whole data, according to the page requested (actually, you request the first item and the total number of items you want, as I will explain). Optionally, you may simplify the data source and always reply with the whole dataset, pagination will still happen in the client side. Or your data source may be “greedy” and reply with a bigger chunk than the one requested. The component will be smart enough not to request any data it already has. Using Firebug in Firefox allows you to see very easily when and what data is being requested/returned.

In most cases I think sending the whole chunk of data could be the simpler and also the faster way to handle pagination, specially in the WordPress page list use case. It would be very unlikely, to have a user with more than, say, 100 pages to order, which are still very fast to load at once (I am guessing it would take less than 5kB for 100 items). If you do that, the client will still see the data paginated, but the server doesn’t need to worry about it at all. The component obviously has support for no pagination as well, you only have to turn a flag off, and in this case the client will see all the data at once.

You will notice that in my demo a page won’t always have 5 items, sometimes it will have more: this is by design, as if I was to enforce only 5 items per page we would have to break hierarchies in half, what would be weird for the user and terrible to program, as well, as I have discussed in earlier posts. The way I figured to solve the inconsistencies and avoid duplicate items among pages is described in the following paragraphs:

First, what does the component displays in each page then? Initially, when it is loaded, it requests the first item of the page being loaded. If loading the third page with 5 elements per page, that would be the element with the zero-based index order of 15. But it will actually display the first element after the one with index 15 that doesn’t have any parents – we will call it a ‘root’ element from now on. And it will display at least 5 items, since it will always show complete hierarchies.

When the component needs some data to display a page, it will make a request with 2 parameters:

• firstIndex (what is the zero-based ordered position of the next item I want to fetch and display. Note the component actually wants to display, as the first item in the page being shown to the user, the first root element after it, but it doesn’t know in advance what it is )
  
• count (how many items, at minimum, should be returned)

The server should reply with the data, in JSON format, that, besides the data itself, has the following parameters:

• requestFirstIndex (what firstIndex was requested)
• count (how many items were actually returned, which should be greater or equal to what was requested)
• firstIndex (what was the index of the first element actually returned; ideally, it should be the first root item after requestFirstIndex , but it could be also be any root element before it, it could even be ’0′, if the server decided to return the whole dataset )
• totalCount (how many items does the dataset has in total)

Note that the server should always return the whole data that the component wanted to display. Eg: if we request firstIndex = 10, count=5, the server could return firstIndex = 5, count=20, but not firstIndex = 5, count=8.

To be brief: The server should always return what the component wants to display right now, but it could also return more, to the left, or to the right. I know it may sound a little confusing, but it is actually not that big deal.

The widget is almost complete, but I am running some issues with pagination, which is a lot more complex than I initially thought, primarily because you can’t show exactly the same number of items per page, due to the nesting (the pagination needs to be “soft”). But I think I am on my way and should be complete in the next 2 or 3 days. Summing up it will be able to fetch a list of nested elements represented in JSON, will display it, allow the user to change the order/nesting and will send it back to the server.

In the meanwhile I created a Google Code project for the NestedSortable plugin and hosted it there.

I hope to be able to write again pretty soon, with good news!

I am showing now the first version of what I call the “Nested Sortable” jQuery plugin. It is based on the same nesting idea shown on the previous prototypes and used on the bbPress forum ordering, but it is totally rewritten and has a very different (and I believe simpler/more efficient) inner working.

A demo of it can be seen here.

The highlights of it are:

• Built on top of Interface ‘s Sortable and maintains compatibility with it. This means you can use the same options available on the Sortable, plus a few extra ones specific to the nesting. You can also mix Sortables and NestedSortables on the same page.
• Just as easy to use (see the code examples on the demo).
• Can be easily configured for left-to-right or right-to-left nesting.
• Can be used with any type of HTML element (that allows nesting), not only ULs and LIs.
• You can define a class for the nesting the draggable is currently hovering on, to make it clearer to the user (see my ugly yellow example – could be made pretty with some css styling).
• All the previous bugs/misbehaviours with the prototype are gone (hopefully without additional ones being introduced).

It was tested under IE7 and FF2. What is still lacking for it to be complete (and useable) is a serialization function, which I should finish in a few days (serialization is now complete, see my comment bellow). As things evolve I should probably add a few callbacks to allow its behavior to be easily configured.

So, the next step for the WordPress page list project is to create a higher level jQuery plugin that will be built on top of this one and will, given a JSON object input, fetched via AJAX, dynamically create the Nested Sortable, with pagination support, using the idea Mike gave on the last comment (the user hovers over a Droppable, which should fetch and display the additional pages). It will be more like a widget.

I also show in it what I think is the best way of handling pagination. The idea is to display an item from the main hierarchy before and another one after the items that belongs to the current page, in the ordering view. If the before/after main hierarchy item has children, all of its children should be displayed as well. In the demo, my before and after main hierarchy items are child less, therefore the “items bellow this” and “items above this” markers show up after one item, but that would vary. I did a lot thinking into this and think this is the only (and simpler) way you can move main items from one page into the other, without having “dark spots” (elements that are “stuck” in a page).

In trying to understand what was making nesting not working in the prototype, I had to dissect the Interface plugin code for the Draggables, Dropables and Sortables. I found the code pretty cluttered and hard to understand, filled up with variables with names such as “oC”, “oD”, “a”. In fact, almost no variables have meaningful, unabridged names. I had to do a lot of debugging, using Firebug. The code is very coupled as well as, for instance, you have calls to the Sortables plugin being made from the Draggable and Dropable (you would expect the Draggables and Dropables, as lower level elements, to be unaware of the Sortables existance). I didn’t like the code, but at least I understand it now.

Meanwhile, I found out the the jQuery team (John Resig, the creator, included) are creating an official, brand new set of UI plugins for jQuery. It is called jQuery UI. I wonder why they decided to skip Interface all together. It was not released yet, but already have working Draggables, Dropables and Sortables. I checked it out from SVN, looked at its code and tested it. The code is a great step foward from what I saw in Interface. Much cleaner, commented, well organized and easy to understand. Impressively, it also looks like they managed to keep it a lot smaller (30kB against 46kB of Interface’s). The coupling problem I mentioned doesn’t exist, as they created a generic callback in the Dragabble (called dropBehaviour), that the Sortable implements. That way, I could, for instance, create a brand new Sortable-like plugin, without having to modify Draggable’s or Dropable’s code. They seem to be tackling the “Sortable with Hierachy” problem as well, and have some partially working demos for it, in their test files.

Regarding the server side implementation, I am leaning towards passing the javascript a JSON representation of the page list, instead of generating the whole UL list in the server side. I think it will make things much more reusable, in both ways (reusing the javacript and the server side script). It will also download a lot faster for the client. I was thinking of using a JSON like this:

{
"columns":["Title(ID)", "Owner", "Updated"],
"items":
[
{
"id":1,
"info":["My Title(1)", "My Owner", "My Date"],
"children":
[
{
"id":2,
"info":["My Title(2)", "My Owner", "My Date"],
},
{
"id":3,
"info":["My Title(3)", "My Owner", "My Date"],
}
]
},
]
}

In this example we have a parent page which has two children.  It doesn’t look so good because I can’t indent the code to display it here. I already implemented it on the prototype, so now, instead of loading the page list from an file with HTML and just displaying it, it loads a file with JSON and processes it to create the HTML on the fly. The JSON file for the prototype can be seen here: list_demo_json.js  The JSON file has 2kB, versus 6kB for the HTML with the exact same data.