Have you missed this presentation?

Have you missed 'Deadlocks: the beginning of the end" at EclipseCon last week?

Here's your chance of having your own personal copy in high definition!

Optimizing concurrency with Freescale’s Deadlock Preventer

The overall concurrency of a program, given it already uses an optimal number of threads, will depend on how much lock contention is occurring.

To avoid both deadlocks and lock contention, synchronization primitives should be the most granular possible given a fixed number of primitives.

For example, in the following snippet:

public synchronized void setProperty(Object newValue) {
    value = newValue;
    notifyPropertyChange();
}

private void notifyPropertyChange() {
    synchronized(listeners) {
        for (IListener listener : listeners)
            listener.changed();
    }
}

The setProperty() method is synchronized in order to protect the “value” data from concurrent data access.  Inadvertently, the synchronized block will cause all the code path executed by the listener.changed() invocation to needlessly have the synchronized(this) lock as a precedent.

This will not only cause the code to have a sub-optimal concurrent execution, but also introduce a lock order dependency between ‘this’, ‘listeners’, and whatever lock is acquired by the listener’s changed() implementation, possibly eventually causing a deadlock (in the Deadlock' Preventer’s terminology, the ‘listeners’ lock is then a ‘follower’ of the ‘this’ lock, and the ‘this’ lock a precedent of the ‘listeners’ lock).

The code could be re-written as follows:

public void setProperty(Object newValue) {
    synchronized(this) {
        value = newValue;
    }
    notifyPropertyChange();
}

By simply reducing the granularity of the ‘this’ lock acquisition, we improve possible concurrent execution, eliminate the lock dependency, and avoid lock ordering acquisition problems between ‘this’ and ‘listeners’, and ‘this’ and all listener’s changed() implementations, without any negative effect whatsoever.

We could also re-write the notifyPropertyChange() implementation as follows:

private void notifyPropertyChange() {
    IListener[] tmp;
    synchronized(listeners) {
        tmp = listeners.clone();
    }
    for (IListener listener : tmp)
        listener.changed();
}

At the cost of the clone() statement (something probably insignificant compared to the code executed in the listeners), the lock dependency is completely removed between ‘this’, ‘listeners’, and all lock acquisitions in the listener’s changed() implementations.

In a java program, the locks with the most number of followers are not only the most likely to cause deadlocks, but also the locks that are likely to reduce overall system concurrency.

It is unfortunately very easy to induce unnecessary dependencies between synchronization primitives, and hard to discover which locks have effectively a large number of followers.

Without a tool such as the Deadlock Preventer, which helps identifying which locks have the most followers, developers would be reduced to half hazardly trying to improve lock granularity without knowing where to start.

This is very similar to performance optimization without using a profiler:  without knowing what takes the most time, developer’s time is easily wasted, and so is concurrency optimization.

Help with the Deadlock Preventer

With the latest release of the Deadlock Preventer, it is now possible to use the Deadlock Preventer view to download lock statistics for a java program.

By right-clicking on the process name in the “Conflicts” tab, and selecting the “Statistics…” menu, the following dialog appears:

The table lists all locks in the program, the number of followers and precedents, and their acquisition locations.  More information can be obtained for each lock by selecting its row and doing copy, which puts the following information in the clipboard (for the first lock in the table above):

lock: java.lang.Object (id=2), precedents(1) followers(1)
  zzz.Main.test(Main.java:14)
  zzz.Main.main(Main.java:6)
  precedents:
    java.lang.Object (id=1), thread id(1 (main))
     zzz.Main.test(Main.java:13)
     zzz.Main.main(Main.java:6)
  followers:
    java.lang.Object (id=1), thread id(13 (Thread-2))
     zzz.Main$1.run(Main.java:22)
     java.lang.Thread.run(Thread.java:619)

The text contains detailed information about each precedents and followers, along with the stack trace in which their relationship to the lock were recorded.

The information for all locks can also be exported to a single text file by using the “Export…” button.

One very useful way to use the statistics generated by the Deadlock Preventer is to sort all the locks by their follower count.

To give an example of what large programs lock statistics look like, the following table shows what the Eclipse IDE synchronization primitives statistics just after opening the workbench window:

Listing the top followers’ count locks out of a total of 21 830 locks.

Running Freescale’s Deadlock Preventer from the command line

One of the best ways to detect potential deadlocks is to integrate the Deadlock Preventer in a release engineering process, so it can run with the java program executing the automated tests, and have an excellent code coverage to analyze the system’s synchronization primitives.

To integrate the Deadlock Preventer in a release engineering process, although, means being able to configure the test scripts to run the Deadlock Preventer.

Adding the Deadlock preventer is as simple as adding the following two arguments to the “java” executable:

-javaagent:path-to\com.freescale.deadlockPreventer.wrapper_1.0.0\com.freescale.deadlockpreventer.jar

-Xbootclasspath/a:path-to\com.freescale.deadlockPreventer.wrapper_1.0.0\com.freescale.deadlockpreventer.jar;path-to\javassist.wrapper_1.0.0\javassist.jar

Where the two plugins “com.freescale.deadlockPreventer.wrapper” and “javassist.wrapper” can be obtained from the following link(in the build/eclipse/plugins sub-folder):

https://github.com/sbeauchamp/Deadlock-Preventer/zipball/master

The following java property can be specified to write the Deadlock Preventer output to a file:

-Dcom.freescale.deadlockpreventer.logToFile=path-to\deadlock_preventer_output.log

Alternatively, the output can be re-directed to the standard output with the following property:

-Dcom.freescale.deadlockpreventer.logToStream=out

or

-Dcom.freescale.deadlockpreventer.logToStream=err

Memory Requirements

An important factor to keep in mind, is that running a java program with the Deadlock Preventer will cause all locks ever acquired to be referenced in memory by the analysis engine, so they will not be garbage collected for the life time of the program.

This can cause some significant memory consumption increase, and may require the –Xmx flag to be passed to java to be increased.

For example, running an eclipse.exe instance with the the Deadlock Preventer can increase its memory usage by several hundred MBs.

Running the Deadlock Preventer interactively

Another alternative to either running the Deadlock Preventer in the Eclipse workbench with the UI or running it in an automated build, is to use the built-in command line server/client.

To start the server, the following command can be used:

java –cp path-to\com.freescale.deadlockPreventer.wrapper_1.0.0\com.freescale.deadlockpreventer.jar com.freescale.deadlockpreventer.ConsoleNetworkServer 43537

The last argument is the port number of the server, which can be omitted, in which case the default port will be used, 43537.

The following output will then appear to the command line:

Server started on port: 43537
services:
  report: report.2
  query: query.1
Type 'help' for a list of commands.

This information can then be used to pass the following two properties to the java instance that is running the Deadlock Preventer (the one that is getting the –javagent flag):

-Dcom.freescale.deadlockpreventer.reportService=localhost:43537:report.2

-Dcom.freescale.deadlockpreventer.queryService=localhost:43537:query.1

Note that if the server is started on a different machine, the “localhost” string must be replaced by the IP address of the server.

When the Deadlock Preventer is connected to the server, any inconsistent locking detected will be displayed on the server’s console interactively, so that the user can decide to continue, or abort the program.  The server console can also be used to query lock statistics with the following commands:

getLockCount

      Returns the number of locks in the program.
getDetails index end

      Return detailed information on the lock(s) from index to end (being numbers between 0 and the number of locks in the program)
getAll

      Return all lock information as output on the standard out stream
writeAll file-path

      Download all lock information and write the complete output in a file

Note that downloading lock statistics can require quite a lot of memory and bandwidth (several hundred MBs for an eclipse workbench, for example).

Freescale's Deadlock Preventer is now released!

To try it out:

1. Be sure to have an Eclipse 3.6.1 SDK layout, and using JDK 1.6 as your default JRE (jdk 1.7 doesn't work yet)
2. Install Git
3. Set your user and email in Git
4. Type in the shell (cygwin on windows):

git clone git@github.com:sbeauchamp/Deadlock-Preventer.git

This will create a directory on your file system containing all the deadlock preventer plugins.

If you simply want to run the deadlock preventer tool, get the 4 plugins under

./Deadlock-Preventer/build/eclipse/plugins/

And copy them in the eclipse/dropins of your eclipse directory:

You will see that a new view is available from "Other/Deadlock Preventer".

The easiest way to try is out is to create a new java project, with the following sample code in Main.java:

public class Main {

public static void main(String[] args) {

new Main().test();

}

Object lock = new Object();

Object lock2 = new Object();

private void test() {

synchronized(lock) {

synchronized(lock2) {

code();

}

}

Thread thread = new Thread(new Runnable() {

public void run() {

synchronized(lock2) {

synchronized(lock) {

code();

}

}

}

});

thread.start();

try {

thread.join();

} catch (InterruptedException e1) {

}

}

private void code() {

System.out.println("test");

}

}

Then create a new launch configuration to run the program. The new configuration will automatically appear in the combo box under the tab "Eclipse JDT Debugging" of the "Deadlock Preventer" view.

Once you click "Debug" from that tab (using the "Run/Debug" menu does not have the same effect), the program runs with the deadlock preventer automatically instrumenting the byte code.

You will then see the following error being reported:

And if double-clicked the full information is displayed:

By setting a breakpoint on "RuntimeException"s and using the 'Interactive' or “throw exception" mode, you are able to break in the debugger at the exact point where the inconsistent locking order is detected.

This should be enough to get your started with using the deadlock preventer, and investigating potential deadlocks in your programs.

In a next post, I will post the different options and runtime settings of the deadlock preventer including those useful to integrate it in a release engineering process.

P.S. I submitted a talk to EclipseCon 2011 on the deadlock preventer and using it to prevent deadlocks in Eclipse plugins. Please drop a comment in the submission if you think it would be an interesting talk.

How to Debug and Detect Deadlocks

Deadlocks in 2 minutes

Deadlocks are software bugs where multiple threads (typically one of them being the main thread) wait indefinitely on common synchronization primitives (locks) because their resolution is mutually interdependent.

Thread A waits on lock L1 to be released, but it held by thread B, who in turns waits on lock L2 to be released while it is held by thread A.

The scenario above is by far the most common source of deadlocks. The fundamental cause is that the locks are acquired in a different order by two different threads, and depending on an unpredictable race condition, can deadlock at runtime.

For example, the following snippet can cause a deadlock when Property.get() and Property.set() are accessed by two different threads:

class Property {
    static final public String DEFAULT_VALUE = "...";
    HashMap map = new HashMap();
    ArrayList listeners = new ArrayList();

    synchronized void set(String key, String value) {
        synchronized (map) {
            map.put(key, value);
        }
        for (Listener listener : listeners)
            listener.notifyChanged();
    }

    String get(String key) {
        synchronized (map) {
            String value = map.get(key);
            if (value == null) {
                value = DEFAULT_VALUE;
                set(key, value);
            }
            return value;
        }
    }
}

When the method Property.set() is called, the lock for "this" and "map" are acquired (using the synchronized keyword), while when Property.get() is called, the locks "map" and "this" are acquired, in the opposite order than Property.set().

This can cause a random deadlock when Property.get() and Property.set() are called from two different threads.

The example illustrate that:

1. It is easy to write code that has incorrect lock acquisition order.
2. It is hard to detect such errors, until a deadlock occurs randomly at runtime, and it might not be consistently reproducible.

Considering large source base where thousand of line of code acquire locks and have a very complex code path, it is often nearly impossible to prevent deadlocks from occurring at runtime, at the developer are faced with reducing the concurrency of the software, leading to poor usability, and sub-optimal performance.

Here is where the deadlock preventer tool comes to help.

The Deadlock Preventer

The deadlock preventer is a tool that dynamically instruments the java bytecode of an application when running in a java virtual machine in order to analyze the lock ordering and detect potential deadlocks without the need to rely on unpredictable and unreliable race conditions.

By running the deadlock preventer while executing the java program (the Eclipse IDE in our case) with a good code coverage (basically covering all features that the customers are susceptible to use), we can ensure that the code does not contain incorrect lock order, and is prone to deadlocks at runtime.

When we run the Property class in the example above with the following junit test code:

@Test
public void testProperty() {
    final Property property = new Property();
    Thread thread = new Thread(new Runnable() {
        @Override
        public void run() {
            property.get("foo");
        }
    });
    thread.start();
    try {
        thread.join();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    try {
        property.set("bar", "value");
        fail("should throw an exception");
    } catch (RuntimeException e) {
        assertTrue(e.getMessage().contains("ERROR"));
    }
}

We obtain a RuntimeException (as validated by the junit test case - something optional for the deadlock preventer to issue), but also the following error report on the console:

***DEADLOCK PREVENTER*** ERROR: Inconsistent locking order while acquiring lock: java.util.HashMap (id=0) in thread: 1 (main)
Property.set(Main.java:812)
Main.testProperty(Main.java:38)
...
(junit stack)
with predecent : Property (id=1)
Property.set(Main.java:811)
Main.testProperty(Main.java:38)
...
(junit stack)
Previously acquired lock: java.util.HashMap (id=0) in thread: 9 (Thread-0)
Property.get(Main.java:818)
Main$1.run(Main.java:28)
java.lang.Thread.run(Thread.java:619)
Previously acquired precedent: Property (id=1)
Property.set(Main.java:811)
Property.get(Main.java:822)
Main$1.run(Main.java:28)
java.lang.Thread.run(Thread.java:619)

The important aspect is that we receive this error report deterministically, each time the code runs, whether or not the code actually deadlocks at runtime given a race condition (something we made impossible with the thread.join() call).

Once the error is detected, it can then be corrected by the developer.

Note also that the deadlock preventer will report the error whether or not the code is actually susceptible to run concurrently or not, which in our case was impossible because we made the main thread block until the second thread finished execution (something that defeats the purpose to have a separate thread to begin with).

The deadlock preventer can also issue warnings when an inverse lock acquisition is detected in the same thread, but never run in different thread - it is always a good programming practice to have consistent lock order.

Using the deadlock preventer agent in a java program does not change any of the application logic and code path, but it does change its performance characteristics. Specifically, acquiring and releasing locks become computationally expensive. Ricks of actual deadlocks are effectively increased too, since the time spent while an incorrect lock acquisition order is progress is longer, especially if the 'Interactive' setting is used (see below).

Therefor, if the java program deadlocks, the last conflict reported by the deadlock preventer agent will be the out that refers to the deadlock obtained.

Using the Deadlock Preventer Agent

The deadlock preventer agent is available in two forms for two different development role.

1. As a set of eclipse plugins for developers to test existing code, and investigate potential deadlocks.
2. As a standalone launcher for testers, to run on a test machine in the background while the Eclipse Workbench is performing automated tests.

The Deadlock Preventer Eclipse integration.

The deadlock preventer Eclipse integration consist of 3 plugins that contribute a new view in eclipse, as shown above (the view is accessible from the menu "Menu / Show View/Other..." under the "Other/Deadlock Preventer" section).

The Deadlock Preventer view allows the developer to select an existing launch configuration (only launch configurations for "Eclipse Application" as listed), configure settings, and click "Debug". The debugged Eclipse session will be automatically instrumented, and report any error in the view, under the "Conflicts" tab.

The best way to use the tool for an Eclipse developer, is first to add a breakpoint on the RuntimeException, then use the 'Interactive' mode, which allows to skip over some errors that are not of interest, and throw an exception for the relevant one.

Important: Note that if the 'throw exception' mode is selected, it will cause the code path to divert from the normal execution path, so it could cause sub-sequent errors, and bugs that would not otherwise happen.

Correcting incorrect lock acquisition order

Generally speaking, when two locks (A, B) are acquired in an inverse order by two different code paths susceptible to be called concurrently (i.e. by two different threads), a deadlock can occur.

For example, if thread 1 acquires the locks A and then B, while thread 2 acquires the locks B and then A, a deadlock is possible.

Thread 1: A -> B

Thread 2: B -> A

There are only 2 ways to avoid the problem:

1. Re-order the lock acquisition, so that both threads acquire the locks in the same order

or

1. Introduce a third lock, C, and make both threads acquire it before acquiring either A or B.

In the last case, the pattern would become:

Thread 1: C-> A –> B

Thread 2: C-> B -> A

The lock "C" would be in effect a "guard", preventing the locks A and B to be acquired concurrently, and avoiding the deadlock. The deadlock preventer agent recognize such patterns, and do not report any errors.

Obviously, the first solution is optimal, since it does not reduce concurrency as the second does, but it is sometimes impractical to re-order lock acquisition order, especially when access or modification of existing code is not possible.

Needless to say, removing (even partially) the locks is probably not a good solution, since the deadlock preventer agent does not report data corruption due to concurrent data access.

Miscellaneous comments

Running a full instrumented Eclipse workbench while the deadlock preventer agent analyzes the lock acquisition order, especially through the debugger, is demanding in computational resources. A recent (preferably 4 core or more) CPU is highly recommended.

Not all causes of deadlocks are detected at the moment by the deadlock preventer engine, such as Semaphore release leaks, or async execution from a background thread to the main thread.

(source code of the deadlock preventer to be posted within a few days)

Use the new Resource Filters in 3.6M7 and JDK 1.7 to simulate snapshots

Suppose you are working in a project in Eclipse, but after updating from your source control server, new files were populated in your workspace that cause some build errors.

How to continue to work in your project without being temporarily distracted by the newly created files?

In Eclipse 3.6, Helios, some new powerful features were added to the Eclipse project management that allows the users to configure exactly what files get to populate a project hierarchy.

By using the new Resource Filter feature (improved in M7), the user can either include only or exclude all files based on a rich set of configurable and extensible filters.

When running on JDK 1.7, another condition is available for filtering, the file creation date.

By creating a resource filter that excludes all files having a creation date newer than a given date, you can prevent those files from even appearing in the workspace, so they not be seen by JDT and generate build errors.

The condition for filtering files and folders are broad, including file name, location, workspace path, last modified, file length, etc... The user can also group those condition in logical prepositions with and, or, and not operators.

3rd party plugins can even contribute new matchers through a core.resources extension point to further extend the filtering ability available to the user (think filtering by CVS status, etc...).

Tip: How to Debug SWT components in Modal Dialogs

SWT and JFace components, and especially layouts can be quite time consuming to debug when some unexpected behavior is observed at runtime.

Some tools (such as the YARI tool set) can be used to debug SWT components and layouts at runtime, but when modal dialogs are displayed, such tools can't be used because the developer can't setup the inspector window and inspect the components in the list while the dialog is up.

To resolve this issue, one can

1. Detach the inspector view so it displays in a floating window outside of the workbench.
2. Put a breakpoint in the modal dialog source code, few lines after the setShellStyle() call, with a breakpoint condition so that it removes the MODAL SWT flag.

For example, for making the WizardDialog non modal, a breakpoint with the following condition:

   setShellStyle(SWT.CLOSE | SWT.MAX | SWT.TITLE | SWT.BORDER       

      | SWT.RESIZE | getDefaultOrientation());    

   return false; 

needs to be added to the "setWizard(newWizard)" line below:

   public WizardDialog(Shell parentShell, IWizard newWizard) {

       super(parentShell);       

       setShellStyle(SWT.CLOSE | SWT.MAX | SWT.TITLE | SWT.BORDER

               | SWT.APPLICATION_MODAL | SWT.RESIZE | getDefaultOrientation());

       setWizard(newWizard); 

This way, you will be able to debug the SWT components at runtime even in modal dialogs:

EclipseCon last week was quite rewarding. The talks were great and finally being able to meet face to face with the people I've been collaborating with for the last year is priceless.

Here's a video I made of the talk I gave on Tuesday about the e4 project file structure improvements:

Note that the format isn't ideal, since it contains screenshots that are hard to read in the youtube format.

The slides (without audio) are available below (best seen in full screen mode):

Project File Structure and Core Resources changes in e4

View more presentations from sergebeauchamp.

Hello to Planet Eclipse

Hello all Planet Eclipse readers :)

My blog has just been included today, so I'd like to say two words to introduce myself.

I've been working for Metrowerks/Motorola now Freescale since the old days of CodeWarrior on the C/C++ IDE, and with Eclipse since we switched to Eclipse in 2005-2006.

I'm interested in a lot of different components in Eclipse that helps it be a great C/C++ IDE for our customers, and have been especially involved lately in the e4 project (where I am a committer) to bring improvements to the Core Resources plugin so that it can now brag to be more powerful and feature complete than the old CW IDE or Visual Studio - at least in terms of project file structure :)

You can see more of my recents posts in the following links:

Latest e4 resource changes

New Resource Filters for the Core Resources

Improving Core Resources in Ganymede

I will give a short talk at EclipseCon 2009, so I hope to see a lot of you there, thanks! :)

Latest e4 resource changes

Here is an overview of the changes in the e4 resources available in the latest e4 build:

1. Editable linked resource locations.
   
   The user can now conveniently change the location of a linked resource by clicking the "Edit..." button in the resource property page.


2. Project Linked Resource Variables
   
   Linked resources that are variable relative can now use a variable list that is defined in the project itself, instead of only relying on the workspace variables. The linked resources can also be specified as relative to the project directory itself, making projects referencing other files outside of the project directory portable across workspace and computers.
   


3. Groups
   Groups are virtual folders that the user can use to create an arbitrarily complex logical project independent of the file system hierarchy.
   
   Groups, as opposed to regular folders, do not exist on the file system and therefor a) do not pickup any additional resources underneath them when a refreshLocal() is performed, and b) can only contain other groups or linked resources as children.
   
   Groups are fully compatible with the existing 3.4 Core Resource API, and all plugins work just as expected. Groups are seen internally by clients as an linked resource folder with an invalid location (that returns null from getLocation()).
   


4. Resource Filters
   Resource filters can be added on any project, folder, linked resource folder or group to automatically build the project structure by limiting what files and folders are visible to the resource workspace layer when a refreshLocal() is performed.
   
   Resource filters is a powerful feature that allows an Eclipse project to include folders that contains thousands of elements, but only including a fraction of them in the project. As opposed to UI view filters, resource filters are applied at the lowest level, and do not cause resources to be wasted for elements excluded from the directory.
   


5. Drag and Drop auto-generation of linked resources
   

   
   (A somewhat better version of the video is available here)

   
   
   When files are dragged and dropped from Windows Explorer on a projet in the Navigator, the user can choose either to
   1. Copy the files under the target directory (the previous operation)
   2. Create a hierarchy of groups and linked resources replicating the file system file and folder hierarchy<
   3. Create only linked resources
   
    Linked resources can also be creating when dragging resources from one project to another, transparently transfering the path variables the linked resource might be referring to.  
   
   Note that this linked-resource drag and drop awareness is currently only available to the Navigator view. 


6. Linked Resource Editor
   
   Under the new Linked Resource project property page, a new Linked Resources tab is available. This tab shows a list of all resources in the project, and group them by their location path, being either invalid (for broken paths), absolute paths (which makes them non-portable) and correct paths.
   
   The user can then easily see which linked resource requires modification, and change it directly in the dialog by choosing the "Edit..." button.
   
   The user can also automatically convert one or many linked resources to absolute or relatives paths by clicking on the Convert Relative/Absolute button. This command will automatically generate linked resources if needed and try to make the path as portable as possible.

Flexible Project Improvements in e4

Several improvements to the Core Resources have been made for e4.

Installing the new build

In order to try them out, go to the following web site:

http://download.eclipse.org/e4/downloads/

Select a recent build, and use the link under the "update" column to use in an Eclipse 3.5M4 installation in the UI available from the menu "Help | Install New Software ...".

Once the "Install" dialog comes up, click on "Add Site..." and enter the update site you selected from the download page ( http://download.eclipse.org/e4/downloads/drops/I20090108-1930/repository for example).

The list will refresh, and you can them select the item "org.eclipse.e4.resources.feature" and click "Next, "Finish", etc...

This will enable all the current e4 Core Resources in the current 3.5M4 layout.

more to come...

Updated Resource Filter UI

Here is a screenshot of the Resource Filter updated UI, as discussed in the previous post.

There is now support for OR, AND and NOT filter groups that allow the user to build arbitrarily complex logical propositions for filtering resources.

Resource Filter UI

An update to the previous post, I now included to the Resource Filter bug report a UI implementation.


The UI consist of a property page available when the user right clicks IContainer objects (basically folders, projects and linked resource folders), and allows him to add, edit, remove and reorder resource filters.

The interface allows the user to choose which resource filter type (currently either a regular expression or a String Matcher), the mode ("Include Only" or "Exclude All"), the target ("Files", "Folders" or "Files and Folders" the filter type arguments, and finally whether it is inheritable to the container's children or not.

Unless I'm mistaken, the result of the filter operation is always the same no matter the order of the filters (i.e. (A & B & !C == A & !C & B), so why allow the user to order them?

The reason why filters can be reordered is mainly for performance reason, since one can imagine filter types that are slower than others (one that check for file content types, for instance), and it is more efficient to put faster filters first so the slower filters have a smaller set to iterate through.

A side note, an interesting test is to put an "Exclude all" filter for files and folders matching "*" on a project itself: Will it cause the ".project" file to be filtered out as well? Thankfully, it doesn't, since the code path to retrieve the ".project" file is different than the regular local refresh. So the project still opens fine, although it is indeed an extreme case.

New Resource Filters for the Eclipse Core Resources

Our customers have to work with large source trees, where creating projects with those trees can be cumbersome.

Often the user wants to deal only with a subset of files in those hierarchies, but each file system object gets created an a resource in the resource tree and causes performance and scalability problems.

To address those limitations, we implemented the "Resource Filter" feature which we provided a patch to the core.resources plugin today (UI to come shortly).

With this feature, the Core Resources plugin now support adding resource filters onto IFolder objects that limits at a low level which file system objects are available to the resource tree so that large source trees can scale easily when filters are used.

How does it work:

• The .project file now contains tags that specify which filters exist on which folder.
  
• A folder can contain a list of Filters (from 0 to n)
  
• Filters can be optionally "inheritable", in which case they apply not only to the folder's children, but also to it's sub-folder children as well.
• Each filter has the following attributes:
  
  
  
  1. The project relative path of the folder it applies.
  
  
  2. The id of the filter type provider (ex: "org.eclipse.core.resources.regexFilterProvider").
  
  
  3. The arguments of the filter. This is filter dependent. (ex: ".*\.c")
  
  
  4. The filter type. This is a integer that can contain any of the following flags:
     
     
     
     • Filters can be set to include only files or folders matching a criteria, or exclude all file and folders matching a criteria. The two following flags choose between the two modes:
       
       public static final int INCLUDE_ONLY = 1;
       public static final int EXCLUDE_ALL = 2;
       
       Note that multiple filters on the same folder can be of a mix of "include_only" and "exclude_all" types.
       
     
     
     • Filters can be applicable to files or folders or both, the following flags indicate the preference:
       
       public static final int FILES = 4;
       public static final int FOLDERS= 8;
       
     
     
     • Filters can be inheritable to sub-folders, the following flag indicate that:
       
       public static final int INHERITABLE = 16;
       
  
  Note that all those flags are declared in the org.eclipse.core.resources.IFilter interface.
  

Something important to realize is that only real files that are located under a folder and appear in the workspace tree by a local refresh can be filtered out. Linked resource explicitly added under a folder still appear no matter what filters are put.

This is by design, since linked resources and resource filters are complementary to each other. Where linked resources allow the user to build the project structure by explicitly including files and folders, resource filters allows the user to build the project structure by explicitly excluding files and folders.

One practical thing to do, though, is to put resource filters on linked resource folders, which work perfectly.

Some interesting behavior:

• If the user creates a new file from the "File | New..." wizard that is actually filtered out by its parent resource filters, it will show up just after creation, but it will disappear from the resource tree when the user manually refreshes the view. I believe this is the proper behavior.


• Deleting folders that contain filtered out children work as expected (hidden children are deleted with their parent), but might cause the user to delete files that he's not aware are present on the file system. This should be addressed by the UI showing a warning in such case.


• Copying folders that have filters is interesting. While the filters get applied to the destination folder, only the visible objects are copied, which may or may not be what the user expects. Again, this issue should be resolved by having a UI giving the user the option (or at least the information) about what will happen


• Moving folders with filters, on the other hand, moves the whole content, including the hidden files invisible from the resource tree. This is because the move operation moves the underlying directory, and then copy the resource tree. The filters are moved as expected as well.

How to try it out:

1. Get the patch from:
   https://bugs.eclipse.org/bugs/show_bug.cgi?id=252996
   
   
2. Apply the patch to the org.eclipse.core.resources project.


3. Import the test project provided in this patch. The project contains two filters:
   
   • One filter on "folder" which "includes only" all files ending with ".c" (and is inheritable)
   • Another filter on "folder/sub_folder" that "excludes all" files matching "output.c".


4. Observe that the following objects are hidden from the resource tree:
   
   
   
   • "folder/foo.h" (the include_only *.c filter on "folder")
   
   
   • "folder/bar.h" (the include_only *.c filter on "folder")
   
   
   • "folder/sub_folder/file.h" (the inherited include_only *.c filter on "folder")
   
   
   • "folder/sub_folder/output.c" (the exclude_all "output.c" on "sub_folder")
   
   

Drag and Drop Improvements in Eclipse

One feature that IDE users are accustomed to is dragging files from Windows Explorer and dropping them on the application window to have them handled by that app.

With the 3.3 Platform, Eclipse added the "Editor area drag and drop" functionality, which aims at implementing this feature.

Unfortunately, there is still some limitations that need to be overcome to handle how users expect a native application to behave.

Handling files that open with external editors

The first thing that the user can notice about Eclipe's handling of dropped files is that for some file types dropped onto the workbench, external editors are launched to open the file type instead of Eclipse itself.

The most obvious example is when the user drags and drops a .bat file in the Eclipse workbench.

Instead of editing the content of the file in a text editor, as the user would expect, Eclipse actually launches the .bat file in a Windows command prompt, mimicking the same behavior as if the user double-clicked it in the navigator pane.

This is very inconsistent, since if the same file is dragged not from Windows Explorer but from an Eclipse project and dropped on the same editor, the .bat file is now properly open in an editor window instead of being launched in a command prompt.

This issue doesn't only affect .bat files, but all files that map to a system editor in Eclipse.

If the user wanted to simply have Windows to handle the .bat file and not Eclipse, he would have simply double-clicked it, and not took the effort to drop it on top of the Eclipse workbench.

Obviously, Eclipse doesn't do what user expect with those kind of files.

Lack of extensibility

Drag and dropping files in an Eclipse workbench demonstrate the general intent of the user to have Eclipse handle that file.

For some files, that may coincide with opening them in the editor, but with other files, Eclipse handling of the file may mean something very different.

For example, dropping an Eclipse project directory on the Eclipse workbench demonstrate that the user doesn't want to edit the folder, but import the project in the current workspace. The same is most likely with .project files as well.

Note that since the user can't currently double-click the .project file in Windows explorer and have Eclipse be open and handle the project, this is the only way he can have to conveniently import projects without having to go through the import wizard.

By dropping a .mcp or .vcproj file on the Eclipse workbench, the user demonstrate his intent of having those CodeWarrior and Visual Studio projects be imported in Eclipse, preferably by having a wizard come up.

Another thing that is typical for C/C++ IDE is dropping an executable on the IDE in order to start a debugging session for that executable.

In all those case, the expected Eclipse behavior by the user is different from what he would expect if he double-clicked the same files in the Eclipse navigator (in which case, he would expect to expand the project directory, to launch CodeWarrior or Visual Studio, and to run the executable respectively).

All those features are impossible to implement with the current Drag and Drop feature which is not extensible.

Our improvements to the Editor area Drag and Drop feature.

In order to overcome the limitations mentioned above, we made the following improvements in the Editor area drag and drop feature:

1. We extended the drop target for the general drag and drop workbench mechanism. Since the Eclipse workbench in general should handle file and folders dropped from Windows Explorer, the user can now use the top part of the Eclipse Workbench window to drop files.
   
   Note that we wanted to include the top most section of the workbench window, but we couldn't find a way to include it without including the whole shell at the same time.
2. We added the org.eclipse.ui.ide.dropHandler extension point to allow a flexible implementation and resolution of the handling of files and folders dropped on the Eclipse workbench (see more details below).
3. We provided default dropHandlers to handle files in a proper way (without resorting to external editors) and to handle project directories and .project files by importing the project in the current workspace.
4. We provided a conflict resolution mechanism that detects when two or more handlers declare supporting the same file and let the user choose between them in a convenient dialog.

Algorithm and Extensibility API

The general workbench Drop Handler algorithm relies solely on the dropHandler extensions to be able to handle dropped files and folders on the Eclipse workbench.

Each handler must have the following attributes:

• A user visible name
• An icon
• A handler class, implementing the IDropHandler interface (as described below)
• A unique string ID.
• A set of rules determining if a given file or folder is supported by the DropHandler.
  

The rules can be any combination of the following:

1. An inspector class, implementing the IDropHandlerInspector
2. Any number of typenames, which consist of the type of the object (file or directory) along with a regex pattern for the file name (such as "\.project").
3. Any number of file extensions, which consist of a pattern matching a file extension, such as "txt".
4. Any number of content types, which consist of the Eclipse content type IDs used in the Platform ContentTypeManager.
5. A lastInspector class implementing the IDropHandlerInspector as well.

When a file or folder is dropped on the Eclipse workbench, the workbench drop feature will iterate through all inspectors for all existing dropHandlers. If one inspector matches the file or folder, its handler class will be used to handle the file or folder and the matching process will end.

If no matching inspectors are found, all typenames will be tested, then all extensions, followed by all content types and finally all lastInspectors.

If two or more dropHandlers are found matching for the same rule level, the Conflict Resolution dialog will be displayed to the user to know which dropHandler to choose, unless a precedence already has been recorded by the user.

The following is an example of a Project Handler extension point:

<extension point="org.eclipse.ui.ide.dropHandler">
   <handler class="org.eclipse.ui.internal.ide.dnd.ProjectDropHandler"
       icon="icons/full/obj16/prj_obj.gif"
       id="org.eclipse.ui.ide.dnd.ProjectDropHandler"
       name="%DropHandlers.ProjectDropHandler">
       <inspector class="org.eclipse.ui.internal.ide.dnd.ProjectDropHandler">
       </inspector>
       <typeName name="\.project"
          type="file">
       </typeName>
   </handler>
</extension>

The IDropHandler interface is defined as follows:

package org.eclipse.ui.ide.dnd;

import org.eclipse.core.runtime.IPath;

/**
* Interface implemented by a dropSupport extension point to
* handle a file or folder dropped on the workbench.
*
*/
public interface IDropHandler {

  /** Handles a dropped file or folder on top of the workbench.
  * @param path  The full absolute path of the file or folder dropped.
  */
  public void handle(IPath path);
}

And the IDropHandlerInspector as follows:

package org.eclipse.ui.ide.dnd;

import org.eclipse.core.runtime.IPath;

/**
* Interface implemented by a dropSupport extension point to
* handle a file or folder dropped on the workbench.
*
*/
public interface IDropHandlerInspector {

  /**Returns true or false whether a handler support the type
  * of file or folder that was dropped on the workbench.
  * @param path  The full absolute path of the file or folder dropped.
  * @return true if the handler support handling the file or folder pointer by path, false otherwise.
  */
  public boolean handles(IPath path);
}

Conclusion

We hope that our improvements can be useful to other Eclipse users, and welcome your comments, questions and criticisms!

Our changes are freely available in the following Eclipse bug report, and we hope to be able to contribute the changes back to the Eclipse standard distribution:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=248462

That contains a patch with the implementation of our drag and drop feature improvement.

Improving Linked Resources in Ganymede - API Changes

Following up on the last post describing the user level changes of our linked resource improvements in Ganymede, I will now go over the underlying API changes.

A important requirement of the implementation of the linked resources improvement was to keep the resource API fully backward compatible: All code previously using the core.resources API should work just fine without any changes.

This is very critical, since so many plugins depend on the core.resources framework that breaking any contract would be unworkable.

Core Resources

The first change is the IResource.setLocation() methods as shown below:

class IResource […]
/**
* Sets the value of the link location for a linked resource.
*
* @param location
*            the new location of the target link resource
* @param updateFlags
*            bit-wise or of update flag constants ({@link #FORCE},
*            {@link #KEEP_HISTORY}, {@link #SHALLOW},
*            {@link #BACKGROUND_REFRESH} and {@link #REPLACE}).
* @param monitor
*            a progress monitor, or <code>null</code> if progress
*            reporting is not desired
* @exception CoreException
*                if isLinked() returns false.
*/
public void setLinkLocation(URI location, int updateFlags, IProgressMonitor monitor) throws CoreException;

/**
* Sets the value of the link location for a linked resource.
*
* @param location
*            the new location of the target link resource
* @param updateFlags
*            bit-wise or of update flag constants ({@link #FORCE},
*            {@link #KEEP_HISTORY}, {@link #SHALLOW},
*            {@link #BACKGROUND_REFRESH} and {@link #REPLACE}).
* @param monitor
*            a progress monitor, or <code>null</code> if progress
*            reporting is not desired
* @exception CoreException
*                if isLinked() returns false.
*/
public void setLinkLocation(IPath location, int updateFlags, IProgressMonitor monitor) throws CoreException;
}

As documented above, the IResource.setLocation() methods allow changing a linked resource location. The notifiers are then fired appropriatly, so that the UI is automatically updated when a location changes.

Second, since each project now has its own IPathVariableManager, its definition has been updated with the following methods:

class IProject […]

/**
* Returns the path variable manager for this project.
*
* @return the path variable manager
* @see IPathVariableManager
*/
public IPathVariableManager getPathVariableManager();

/**
* Returns a variable relative path equivalent to an absolute path for a
* file or folder in the file system, according to the variables defined in
* this project PathVariableManager. The file or folder need not to exist.
* <p>
*
* @param location
*            a path in the local file system
* @return the corresponding variable relative path, or <code>null</code>
*         if no such path is available
*/
public IPath getVariableRelativePathLocation(IPath location);

The method getVariableRelativePathLocation() will converts paths such as "C:\foo\bar.c" into "FOO\bar.c", granted than the path variable FOO exists already in the project PathVariableManager and it points to "C:\foo".

A more powerful method is the one available in the IPathVariableManager:

class IPathVariableManager […]

/** Convert an absolute path to path variable relative path.
*  For example, converts "C:/foo/bar.txt" into "FOO/bar.txt",
*  granted that the path variable "FOO" value is "C:/foo".
*
*  The "force" argument allows intermediate path variable to
*  be created if for a given path can be relative only to a parent
*  of an existing path variable.
*
*  For example, if the path "C:/other/file.txt" is to be converted
*  and no path variables point to "C:/" or "C:/other" but "FOO"
*  points to "C:/foo", an intermediate "OTHER" variable will be
*  created relative to "FOO" containing the value "${PARENT-1-FOO}"
*  so that the final path returned will be "OTHER/file.txt".
*
*  The argument "variableHint" can be used to specify to which
*  path variable the path should be made relative to.
*
* @param path  The absolute path to be converted
* @param force Set to true if intermediate path variables need to be created if the path is relative only to a parent of an existing path variable.
* @param variableHint The name of the variable to which the path should be relative to, or null for the nearest one.
* @return  The converted path
* @exception CoreException if this method fails. Reasons include:
* <ul>
* <li>The variable name is not valid</li>
*/
public IPath convertToRelative(IPath path, boolean force, String variableHint) throws CoreException;
}

Also, a new extension point "variableProviders" is available in the core.resources plugins that allows extending the default path variable list if need be.

Variable providers can be pecified as follows:

<extension

point="org.eclipse.core.resources.variableProviders">

<variableProvider

class="org.myPackage.myPlugin.MyPathVariable"

name="MY_PATH_VARIABLE">

</variableProvider>

</extension>

where "MyPathVariable" must inherit from the "IProjectVariableProvider", defined as follows:

public interface IProjectVariableProvider {

 /**
 * Returns a variable value
 *
 * @param variable
 *            The current variable name.
 * @param project
 *            The project that the variable is being resolved for.
 * @return the variable value.
 */
 public String getValue(String variable, IProject project);

 /**
 * If the variable supports extensions (specified as
 * "${VARNAME-EXTENSIONNAME}"), this method can return the list of possible
 * extensions, or null if none are supported.
 *
 * @param variable
 *            The current variable name.
 * @param project
 *            The project that the variable is being resolved for.
 * @return the possible variable extensions or null if none are supported.
 */
 public Object[] getExtensions(String variable, IProject project);
}

Groups

Groups are seen by clients as a broken folder linked resource. Since groups do not exist on the file system, calling IResource.getLocation() returns null, just as broken folder linked resource do.

All code that handle properly IResource.getLocation() returning null, as they should to support the possibility that a folder linked resource is not resolvable, will work transparently with groups as well.

The groups API changes are as follow:

class IFolder […]
/**
* Creates a new group resource as a member of this handle's parent
* resource. A group is not located anywhere in the file system, and can
* contain only linked files, linked folders and other groups.
* <p>
* This method changes resources; these changes will be reported in a
* subsequent resource change event, including an indication that the folder
* has been added to its parent.
* </p>
* <p>
* This method is long-running; progress and cancellation are provided by
* the given progress monitor.
* </p>
*
* @param updateFlags
*            bit-wise or of update flag constants (currently no flags are
*            relevant here)
* @param monitor
*            a progress monitor, or <code>null</code> if progress
*            reporting is not desired
* @exception CoreException
*                if this method fails. Reasons include:
*                <ul>
*                <li> This resource already exists in the workspace.</li>
*                <li> The workspace contains a resource of a different type
*                at the same path as this resource.</li>
*                <li> The parent of this resource does not exist.</li>
*                <li> The parent of this resource is not an open project</li>
*                <li> The name of this resource is not valid (according to
*                <code>IWorkspace.validateName</code>).</li>
*                <li> Resource changes are disallowed during certain types
*                of resource change event notification. See
*                <code>IResourceChangeEvent</code> for more details.</li>
*                <li>The team provider for the project which contains this
*                folder does not permit groups.</li>
*                <li>This folder's project contains a nature which does
*                not permit groups.</li>
*                </ul>
* @exception OperationCanceledException
*                if the operation is canceled. Cancelation can occur even
*                if no progress monitor is provided.
* @see IResource#isGroup()
*/
public void createGroup(int updateFlags, IProgressMonitor monitor) throws CoreException;

class IResource […]

/**
* Returns whether this resource is a group.
*
* @return <code>true</code> if this resource is a group, and
*         <code>false</code> otherwise
* @see IFolder#createGroup(int, IProgressMonitor)
*/
public boolean isGroup();

Improving Linked Resources in Ganymede

Coming from a C/C++ developer background, moving to the Eclipse CDT tools can sometimes be challenging.

C/C++ developers are used to work with standard IDE such as Microsoft Visual Studio and CodeWarrior, two products that have been designed from the start to be C/C++ IDEs.

One area that is specific with developing in C/C++ is the source file layout. C/C++ source files are divided in headers and sources files, are often shared between projects, and have no rigid layout in the file system.

Some teams group the C/C++ files by software component, some by framework, some by teams, and some by a bit of every division imaginable. C/C++ source trees end up being very complex, and it is of course out of the question to ask developers to structure their sources according to the way an IDE works, the IDE has to adapt itself to the way developers work!

Eclipse's solution for creating arbitrarily complex project structure is using linked resources.

Unfortunately, Eclipse's linked resources have the following limitations:

• Linked resources tend to contain absolute paths, unless the workspace's path variables are used.


• When path variables are used to avoid using absolute paths, the variable list is maintained in the workspace only, and if the user switches workspace, the project becomes broken.


• The linked resource's targets can't be changed by the user in the workbench!


• Linked resources can't be specified as being in a parent directory relative to a path variable (i.e. "PROJECT_LOCATION/../../dir/foo.c").


• Creating linked resources is very time-consuming, the user has to create each single file one by one through the new file wizards.

What we have done for our customers at Freescale is improve Eclipse's support for linked resources while maintaining full compatibility with standard Eclipse IDEs.

We can divide our improvements in 4 different sections:

Project Path Variable Manager

The first improvement consists of creating a IPathVariableManager implementation in each project, analogous to the one at the workspace level.

Then, the path variable list is extensible through an extension point, which allows default variable to be specified (such as PROJECT_LOCATION, ECLIPSE_HOME, etc...) and custom variable to be provided by 3rd party plugins.

For instance, a StarCore tool plugin may want to expose its SDK root as a path variable, so that each project can contain sources from the SDK, while having the source locations always pointing to the correct location.

A dialog has been added to handle drag and drop of files in the project, so the user can easily and quickly create a path variable relative project structure.

Special path variables are provided, such as "PARENT" which allows path variables to be specified as relative to the parent directory of another variable, effectively allowing arbitrarily complex project structure to be specified while keeping the project fully portable across computers, or "ENV" which allows linked resources to be relative to environment variables.

Since each project has its own PathVariableManager, special care had to be taken to be sure that copying linked resources that contain path variables between different projects causes the proper path variables to be copied as well.

Also, the user can now change the linked resource target in the file property page.

Groups

C/C++ developers are used to group source and header files differently than they are structured on the file system. In Visual Studio, the user can create filters in a solution, while with CodeWarrior, groups can be created. In Eclipse, the user has currently no choice but to create a set of empty directories, which is then cumbersome to maintain in a source repository and awkward to use.

For that purpose, we have added support for groups in Eclipse that the user can easily create anywhere in a project. A group works just like a folder, and is fully compatible with the existing core.resources API and all its clients, the only difference between a group and a folder is that:

• Groups are not created on the file system, they are instead recorded in the .project file.


• Only linked resources and groups can be created under a group, no real files or folder can, since a group is only a virtual container.

Internally, groups are implemented as folder linked resources.

Additional features

We have also provided a dialog that allows the user to list all the linked resources in a project and easily manage them.

The user can automatically convert absolute path linked resources to path variable relative, and fix broken linked resources.

We hope that our improvements can be useful to other Eclipse users, and welcome your comments, questions and criticisms!

Our changes are freely available in the following Eclipse bug report, and we hope to be able to contribute the changes back to the Eclipse standard distribution:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=229633