RSS

Category Archives: Software

Basket as a writing tool, SCAN as a collector

Basket has been my favourite notetaking software for a long time, until I had switched to mindmaps. Quite recently I’ve discovered another use for it – a writing aid. Basket in one-column mode allows to rearrange your notes just by dragging them up or down (there’re keyboard shortcuts for that as well). When I’m writing a longer piece, I don’t need to hold a structure of the article in my head. I just collect all the pieces (quotes, blog posts fragments, my own notes, links, tweets etc.) and then rearrange it as much as it’s needed. When the flow of the thoughts is optimal, I start to connect these pieces by writing some text in between :).

I don’t have DevonThink (I don’t have Mac) but for finding similar things in my archive I use SCAN. SCAN can aggregate content from a number of sources (it has plugins to read PDFs, OpenOffice and MSOffice files or even RSS feeds), analyze it, automatically assign tags, extract metadata etc. It has Lucene engine built in and does quite a good job of finding related pieces in the archive. It’s quite buggy, doesn’t read all PDFs (such as encrypted), metadata extraction doesn’t work as expected but overall the tool has a potential (and there’s no similar program available on Linux platform anyway). Its development was recently restarted so there’s hope it’s going to be improved in a near future. Additionally, it has a nice eye-candy – a visual overview of relations between tags.

This strategy is similar to the workflow described by Steven Johnson, but without DevonThink. So far I haven’t found anything better under Linux, but probably I need to check online apps – things do change every month.

1 Comment

Posted by Pawel Szczesny on November 3, 2009 in Research, Software

Tags: Basket, Metadata, RSS, SCAN, Workflow

Dreaming about bio-spreadsheet

19 May

One of the often occuring task in my work is to present results of an analysis in some kind of table. I have used for such purpose quite a number of approaches, starting from generating simple HTML file, through fetching of SQL data into table stored in a wiki, up to using Rails. One of the dreams I have recently is a web-based spreadsheet that would allow me to apply some specific piece of code over every row/column and show resulting table.

A simple mockup is shown above. In this example, a code:

print " &lt;img src="http://www.pdb.org/pdb/images/#{column_1}_bio_r_250.jpg&gt;"

… iterated over first column containing PDB codes, would substitute these codes with an image of a protein from PDB server.

In other words I dream about simple (single file would be the best – I like the approach Sinatra framework is taking) web-based programmable spreadsheet. Something like Resolver One, but simpler. Is there anything like that available?

6 Comments

Posted by Pawel Szczesny on May 19, 2009 in bioinformatics, Software

HMMER3 testing notes – my skills are (finally) becoming obsolete

22 Apr

: Image via Wikipedia

It’s already quite a while since I’ve started to extensively test performance of HMMER3. As many other people noticed before, speed of the search has improved dramatically – I’m really impressed how fast it is. However, it’s only part of the story. The smaller part actually.

As some of readers may know, most of my projects so far were revolving around protein sequence analysis and sequence-structure relationships. Mainly I was doing analysis of sequences that had no clear similarity to anything known, without functional annotation. Usual task was to run sequence comparison software and look at the end of the hit list, trying to make sense from hits beyond any reasonable E-value thresholds (for example I often run BLAST at E-value of 100 or 1000). I use very limited number of tools, because it takes quite a while to understand on which specific patterns a particular software fails.

The high-end tool I use most often is HHpred – HMM-HMM comparison software. It’s slow but very sensitive – my personal benchmarks show that it is able to identify very subtle patterns in sequence formed slightly above level of similar secondary structures (in other words, from the set of equally dissimilar sequences with identical secondary structure order, it correctly identifies the ones with similar tertiary structure).

The most surprising thing about HMMER3 is that in my personal benchmarks it’s almost as sensitive as HHpred. I wasn’t expecting that HMM-sequence comparison can be as good as HMM-HMM. This observation suggests that there’s still a room for improvement for the latter approach, however it has already big implications.

PFAM will soon migrate to HMMER3 (the PFAM team is now resolving overlaps between families that arose due to increased sensitivity) and the moment it is be available, it will make a huge number of publications obsolete, or simply wrong. There are thousands of articles that discuss in detail evolutionary history of some particular domain (many of these will become obsolete) or draw some conclusions from the observation that some domain is not present in analyzed sequence/system (many of these will need to be revised). It will also make my skills quite obsolete, but that is always to be expected, no matter in what branch of science one is working. I also imagine that systems biology people will be very happy to have much better functional annotation of proteins.

I don’t want to call development of HMMER3 a revolution, but it will definitely have similar impact on biology as BLAST and HMMER2 had. Not only because of its speed, but also because it will create a picture of similarities between all proteins comparable to the picture state-of-the-art methods could only calculate for their small subset.

The curse of BLAST (mndoci.com)

3 Comments

Posted by Pawel Szczesny on April 22, 2009 in bioinformatics, Research, Software

Tags: bioinformatics, biology, HMM, HMMER, PFAM

Database query and ranked results

22 Jan

: Image via Wikipedia

Already some time ago I’ve read a piece by Marcelo Calbucci: Is it a database or a search engine?. While it deals with search information within a real estate database, I think his comments are applicable in the many areas of life sciences.

In short, Marcelo points out that people miss a lot of interesting entries while looking for a house, because of inflexibility of the query; number of bedrooms, price, distance from some point – these are all set. However, users are flexible and in such case need rather a search engine that gives them close enough answer or allows to specify weight to each filter.

In life sciences we do search for similarities and analogies all the time. Sometimes it’s direct comparison of sequences, on other occasion is high-level meta-comparison between two systems. And while we have various (statistical) metrics of similarities and they sometimes become a part of a database designs, interfaces of biological databases don’t allow to rank query results according to these metrics. For example I can easily find all human proteins related to disease X or disease Y or disease Z, although I cannot specify that I want proteins related to Z AND Y first on the list. Other example would be searching PubMed – I can look for articles related to “synthetic biology”, but I have no way to specify, that I want papers by James Collins from HHMI AND articles related to these papers to be first on the list. I guess it is possible to obtain such results without going through the whole list, but I doubt the method will be very simple. Filtering still seems to be neglected aspect of database design in life sciences.

My dream biological search engine would have a series of sliders (or ideally, I would like to have a device with series of mechanical knobs attached to the computer) and would allow me to dynamically change weights of various aspects of the query and see immediately how it affects the results. It would be something resembling interactivity of Gapminder World, but on dynamically generated data. Technology and proof of concept seems to be there, but I guess we need to wait quite a few years before this approach will be adopted within life sciences.

4 Comments

Posted by Pawel Szczesny on January 22, 2009 in bioinformatics, Data mining, Software

Tags: bioinformatics, Database, PubMed, Search, Web search engine

Collanos Workplace and scientific collaboration

15 Jan

One of my workspaces in Collanos

For some time already I was looking for a tool that would eliminate a need for sending files back and forth between people collaborating on a the same project. While I’m perfectly aware of various solutions such as wikis, version control systems or online office suites, I didn’t feel like I could convince my collaborators to use any of these. One of the reasons is always a feeling of insecurity when using publicly hosted platform (BTW, this is not that uncommon among scientists – I know at least one scientific institution in Western Europe that explicitly forbids using Google apps, especially Gmail for work-related stuff, because of Google’s privacy policy). The other reason was that such solutions are not the best choice when working on binary files (most of my projects do not involve collaborative programming). When I stumbled across Collanos Workplace, which offers peer-to-peer synchronization (although without revision control), instead of a central-server based, I’ve decided to give it a try. For the last couple of weeks I’ve been using Collanos to collaborate on one relatively simple project and the experience was quite positive.

At first, I thought that Collanos may serve mainly as a tool for secure peer-to-peer files sharing with an information who changed what etc. It turned out that this is a capable project management application, that has a chat and discussion panel, one can post notes, links add tasks and assign them to team members. Files are stored is a separate directory – after one adds a file to Collanos, it should be opened from the application, not from original folder. This seemed a mistake in design at first, but I appreciated it very quickly. Synchronization of project directory would mean sharing all of its contents and that can be sometimes in the range of many GBs. From time to time some bug appeared here or there, but overall it worked as expected. Peer-to-peer sharing means that both people have to be online for synchronization, but so far situation that I switched computer off before a person could download my changes happened only once and it was during a weekend.

As a side note, it’s nice to see that Eclipse becomes an application platform for quite a number of programs. See for example this list of Eclipse-based software.

1 Comment

Posted by Pawel Szczesny on January 15, 2009 in Research skills, Software

Tags: Google apps, Office suite, Peer-to-peer, Project management

Bioinformatics is a visual analytics (sometimes)

18 Dec

Short description of my research interest is “I do proteins” (I took this phrase from my friend Ana). I try to figure out what particular protein, protein family, or set of proteins does in the wider context. Usually I start where automated methods have ended – I have all kinds of annotation so I try to put data together and form some hypothesis. I recently realized that the process is basically visualizing different kind of data – or rather looking at the same issue from many different perspectives.

It starts with alignments. Lots of alignments. And they all end up in different forms of visual representation. Sometimes it’s a conservation with secondary structure prediction (with AlignmentViewer or Jalview):

blog-0005

Sometimes I look for transmembrane beta-barrels (with ProfTMB):

blog-0005

Sometimes I try to find a pattern in hydrophobicity and side-chain size values across the alignment (Aln2Plot):

blog-0005

Afterwards I seek for patterns and interesting correlations in domain organization (PFAM, Smart):

blog-0008

Sometimes I map all these findings onto a structure or a model that I make somewhere in the meantime based on found data (Pymol, VMD, Chimera):

blog-0006

I also try to make sense out of genomic context (works for eukaryotic organisms as well – The SEED):

blog-0005

I investigate how the proteins cluster together according to their similarity (CLANS):

blog-0013

And figure out how the protein or the system I’m studying fits into interaction or metabolic networks (Cytoscape, Medusa, STRING, STITCH):

blog-0007

If there’s some additional numerical information I dump it into analysis software (R, for simpler things DiVisa):

blog-0005

And I make note along the process in the form of a mindmap (Freemind, recently switched to Xmind, because it allows to store attachments and images in the mindmap file, not just link to them like Freemind does): blog-0010

So it turns out that I mainly do visual analytics. I spend considerable amount of time on preparing various representations of biological data and then the rest of the time I look at the pictures. While that’s not something every bioinformatician does, many of my colleagues have their own workflows that also rely heavily on pictures. For some areas it’s more prominent, for others it’s not, but the fact is that pictures are everywhere.

There are two reasons I use manual workflow with lots looking at intermediate results: I work with weak signals (for example, sometimes I need to run BLAST at E-value of 1000) or I need to deeply understand the system I study. Making connections between two seemingly unrelated biological entities requires wrapping one’s brain around the problem and… lots of looking at it.

And here comes the frustration. I counted that I use more than twenty (!) different programs for visualization. And even if I’m enjoying monitor setup 4500 pixels wide which is almost enough to put all that data onto screen, the main issue is that the software isn’t connected. AlignmentViewer cannot adjust its display automatically based on the domain I’m looking at or a network node I’m investigating – I need to do it by myself. Of course I can couple alignments and structure in Jalview, Chimera or VMD but I don’t find such solution to be usable on the long run. To have the best of all worlds, I need to juggle all these applications.

I’ve been longing for some time already for a generic visualization platform that is able to show 2D and 3D data within the single environment, so I follow development of SecondLife visualization environment and Croquet/Cobalt initiatives. While these don’t look very exciting right now, I hope they will provide a common platform for different visualization methods (and of course visual collaboration environment).

But to be realistic, visual analytics in biology is not going to become a mainstream. It’s far more efficient to improve algorithms for multidimensional data analysis than to spend more time looking at pictures. I had already few such situations when I could see some weak signal and in a year or two it became obvious. But I’m still going to enjoy scientific visualization. I came to science for aesthetic reasons after all. 🙂

6 Comments

Posted by Pawel Szczesny on December 18, 2008 in bioinformatics, Proteins, Research, Software, Visualization

Tags: bioinformatics, biology, Chimera, Cytoscape, Online Services, protein, Protein family, Visual analytics, Visualization

Skyrails and STRING

09 Sep

Of course I couldn’t resist not to play a little bit with Skyrails after I saw it at Flowing Data blog. Skyrails is a graph visualization system that was designed with expandability and awesome look in mind. All menus can be programmed in odd-looking, but quite easy to learn language, which helps in writing customized interface to particular data.

My quick attempt was to take some sample data from STRING, feed it into Skyrails and see if that makes any sense. My choice was #1 example from STRING main page, which was trpA protein from E. coli K12. The main graph on the trpA interactions page looks as follows:

The same graph in Skyrails:

Of course Skyrails has a 3D representation, is fully interactive, with a little work one can filter some of the connections out, put images of structures instead of green dots, etc. etc. It doesn’t look as clear as STRING, because it wasn’t optimized for such use – in practice it’s much clearer. The video below shows the basic interactions with this dataset.

Is it useful? At the moment, not really. It has already lots of features that more mature programs lack (completely programmable menus are great idea), but usage is still crude and in some cases the flashy effects are disturbing. However, it’s worth to keep an eye on Skyrails. First, development is pretty much guaranteed, as the author said he starts a PhD on this project. Second, the basic roadmap includes features that again aren’t present anywhere else, like client-server architecture (so you can talk to Skyrails system from external application – dynamic, time-aware visualization?). And third – it’s the most cool-looking visualization system I’ve found so far (will it make into a movie, like Genome Valence from Ben Fry did?).

2 Comments

Posted by Pawel Szczesny on September 9, 2008 in Software, Visualization

Tags: Graph, Information Visualization, Skyrails, STRING, Visualization

Ubiquity – coding something useful in less than 20 minutes

27 Aug

Ubiquity is the new experimental extension to Firefox that will (I’m sure it will) make enormous impact on the way we use the browser. It allows to remix various services and extend functionality of the browser in very easy way (if you don’t get the point of Ubiquity yet, I recommend watching the video that came with official announcement; I needed to see that – description didn’t tell me much about how powerful it can be).

I didn’t have much time to play with it yet, but in spare 20 minutes I attempted to code a command that would show me the image of a structure from PDB given its code and eventually take me to its homepage. Suprisingly it was very easy (and I’m not a JS coder). The source is pasted below.

CmdUtils.CreateCommand({
  name: "pdb",
  description: "Goes to Protein Data Bank given PDB code.",
  icon: "http://www.rcsb.org/favicon.ico",
  help: "You can specify the PDB code and pressing enter will take you to particular structure's homepage." +
    " If you type pdb code and press arrow down, you should see an image from PDB site.",

  takes: {"PDB code": noun_arb_text},

  execute: function( directObj) {
    var pdbcode = directObj.text;
    Utils.openUrlInBrowser("http://www.rcsb.org/pdb/explore/explore.do?structureId="+pdbcode);
  },

  preview: function( pblock, directObj ) {
    var pdbcode = directObj.text;

    pblock.innerHTML = "Preview of the structure:<br/>";
    pblock.innerHTML += "<img src=\"http://www.rcsb.org/pdb/images/" + pdbcode + "_bio_r_250.jpg\" />";

  }
})

It of course could be improved by using also a selected text, or allowing to keyword search the PDB (or basically any other biological database), but its current functionality suits me just fine. Ubiquity is not yet such a stable platform as Greasemonkey (or Chickenfoot), but it’s worth to keep an eye on it. I’m sure we will read sooner or later an article in peer-reviewed journal describing Ubiquity commands for life sciences :).

Mozilla gives the passionates one with Ubiquity

1 Comment

Posted by Pawel Szczesny on August 27, 2008 in Software

Tags: Firefox, Mozilla, Mozilla Labs, Protein Data Bank, protein structure, Ubiquity

Configuring Torque and InterProScan

10 Jul

Image via Wikipedia

If by the chance, you want to use InterProScan with Torque Resource Manager (queueing system based on PBS project) it doesn’t work by default (it’s tested with LSF, configuration files are supplied for original PBS and Sun Grid Engine). Fortunately there are two small changes needed in the InterProScan config files to make it work. First, during iprscan configuration, choose PBS54 as your queueing system. Then, in the file pbs54.conf (IPRSCANHOME/conf) remove “-d” switch from following two lines:

asyncsub=qsub [%optqueue][%optresource] -d -o /dev/null -e /dev/null "[%toolcmd]"
syncsub=qsub [%optqueue][%optresource] -d -o /dev/null -e /dev/null -I "[%toolcmd"]

Assumming that Torque binaries are available in the global PATH (qsub, qdel etc., on my machine they sit under /usr/local/bin), change in default shell in the enviroment file pbs54env.sh – from #!/bin/sh to #!/bin/bash. Also, you can add another directories to the PATH in that file (I didn’t). Voilla. InterProScan jobs are now queued.

3 Comments

Posted by Pawel Szczesny on July 10, 2008 in bioinformatics, Software

Tags: InterProScan, qsub, queueing system, torque

Bug tracking systems in science

18 Apr

I’m not going to describe painful process of correcting entries in biological databases or errors in publications when one is not the author – we all know how difficult and unrewarding it is. All major databases contain wrong entries – I see misannotated (or nonexistent) genes in Genbank, artificial domains in PFAM or poorly solved structures in PDB. It’s even worse in publications, where across the whole spectrum of journals I see errors which in theory shouldn’t slip through peer review (this includes such prominent publishers like NPG).

One of the best idea I heard that addressed this issue was to build a bug tracking system (I would like to give credit to the author, but I cannot find the source; wasn’t that one of biobloggers?). It’s simple and efficient. Something is wrong? Fill a bug report. It would be linking to the original entry, would be available for aggregation (for example to track report’s author activity), and possibly could be closed by somebody else than database maintainers or authors if it’s wrong. Because it would be external to all databases, maybe it could grow to provide “community corrected” versions of these databases?

What do you think? How useful such system could be?

10 Comments

Posted by Pawel Szczesny on April 18, 2008 in Comments, Community, Software

Tags: bioinformatics, bug tracking, NPG, science