Mercurial > repos > peterjc > blast2go
changeset 23:31cb702eb5a8 draft
Uploaded v0.0.9, embed citation, updated README
| author | peterjc |
|---|---|
| date | Thu, 31 Jul 2014 05:40:57 -0400 |
| parents | 8c462f7b2c8d |
| children | 05eef6b222af |
| files | blast2go/README.rst blast2go/blast2go.py blast2go/blast2go.xml blast2go/massage_xml_for_blast2go.py blast2go/repository_dependencies.xml blast2go/tool_dependencies.xml tools/blast2go/README.rst tools/blast2go/blast2go.py tools/blast2go/blast2go.xml tools/blast2go/massage_xml_for_blast2go.py tools/blast2go/repository_dependencies.xml tools/blast2go/tool_dependencies.xml |
| diffstat | 12 files changed, 606 insertions(+), 588 deletions(-) [+] |
line wrap: on
line diff
--- a/blast2go/README.rst Mon Sep 23 05:54:09 2013 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,208 +0,0 @@ -Galaxy wrapper for Blast2GO for pipelines, b2g4pipe -=================================================== - -This wrapper is copyright 2011-2013 by Peter Cock, The James Hutton Institute -(formerly SCRI, Scottish Crop Research Institute), UK. All rights reserved. -See the licence text below (MIT licence). - -This is a wrapper for the command line Java tool b2g4pipe v2.5, -Blast2GO for pipelines. It is available from the Galaxy Tool Shed at: -http://toolshed.g2.bx.psu.edu/view/peterjc/blast2go - - -References -========== - -Peter Cock, Bjoern Gruening, Konrad Paszkiewicz and Leighton Pritchard (2013). -Galaxy tools and workflows for sequence analysis with applications -in molecular plant pathology. PeerJ 1:e167 -http://dx.doi.org/10.7717/peerj.167 - -S. Geotz et al. (2008). -High-throughput functional annotation and data mining with the Blast2GO suite. -Nucleic Acids Res. 36(10):3420-3435. -http://dx.doi.org/10.1093/nar/gkn176 - -A. Conesa and S. Geotz (2008). -Blast2GO: A Comprehensive Suite for Functional Analysis in Plant Genomics. -International Journal of Plant Genomics. 619832. -http://dx.doi.org/10.1155/2008/619832 - -A. Conesa et al. (2005). -Blast2GO: A universal tool for annotation, visualization and analysis in functional genomics research. -Bioinformatics 21:3674-3676. -http://dx.doi.org/10.1093/bioinformatics/bti610 - -See also http://www.blast2go.com/ - - -Automated Installation -====================== - -Installation via the Galaxy Tool Shed should take care of the Galaxy side of -things, including the dependency on 'blast_datatypes' which defines the -'blastxml' file format. However, you will also probably need to configure -the Blast2GO property file(s), for example if you have a local Blast2GO -database (which we recommend for speed). - - -Manual Installation -=================== - -The main dependency is b2g4pipe which must be installed manually. Also we -strongly recommend installing a local Blast2GO database as well (see the -intructions below about the blast2go.loc file). At the time of writing, -the current version is b2g4pipe v2.5 which is available here: - -* http://www.blast2go.com/data/blast2go/b2g4pipe_v2.5.zip - -You can change the path by setting the B2G4PIPE environement variable to -the desired folder, but by default the script looks for the JAR file here:: - - /opt/b2g4pipe_v2.5/blast2go.jar - -To install the wrapper manually, first install 'blast_datatypes', then -copy or move the following files under the Galaxy tools folder, e.g. in a -tools/blast2go/ folder: - -* blast2go.xml (the Galaxy tool definition) -* blast2go.py (the Python wrapper script) -* massage_xml_for_blast2go.py (Python XML reformatting script) -* README.rst (this file) - -For a manual installation of the wrapper you will also need to modify the -tools_conf.xml file to tell Galaxy to offer the tool. We suggest putting -it next to the NCBI BLAST+ wrappers. Just add the line:: - - <tool file="blast2go/blast2go.xml" /> - -If you wish to run the unit tests, also add this to tools_conf.xml.sample -and move/copy the test-data files under Galaxy's test-data folder. Then:: - - $ ./run_functional_tests.sh -id blast2go - - -Configuration -============= - -As part of setting up b2g4pipe you will need to setup one or more Blast2GO -property files which tell the tool which database to use etc. The example -b2gPipe.properties provided with b2g4pipe is often out of date. The current -server IP address and database name may given on the Blast2GO website, or -can be found by running the latest GUI version via Java web-start, and -looking under the tools/options menu. These property files can be anywhere -accessable to the Galaxy Unix user, we put them with the JAR file etc. - -You must tell Galaxy about these Blast2GO property files so that they can be -offered to the user. Copy file blast2go.loc.sample to tool-data/blast2go.loc -under the Galaxy folder and edit this to match your installation. This must -be plain text, tab separated, with three columns: - -1. ID for the setup, e.g. Spain_2012_August -2. Description for the setup, e.g. Database in Spain (August 2012) -3. Properties filename for the setup, e.g. Spain_2012_August.properties - relative to the main JAR file, or with a full path - e.g. /opt/b2g4pipe/Spain_2012_August.properties - -Avoid including "Blast2GO" in the description (column 2) as this text will be -included in the automatically assigned output dataset name. The blast2go.loc -file allows you to customise the database setup. If for example you have a local -Blast2GO server running (which we recommend for speed), and you want this to be -the default setting, include it as the first line in your blast2go.loc file. - -Consult the Blast2GO documentation for details about the property files and -setting up a local MySQL Blast2GO database. - - -History -======= - -======= ====================================================================== -Version Changes -------- ---------------------------------------------------------------------- -v0.0.1 - Initial public release -v0.0.2 - Documentation clarifications, e.g. concatenated BLAST XML is allowed. - - Fixed error handler in wrapper script (for when b2g4pipe fails). - - Reformats the XML to use old NCBI-style concatenated BLAST XML since - b2g4pipe crashes with heap space error on with large files using - current NCBI output. -v0.0.3 - Include sample loc file, tool-data/blast2go.loc.sample -v0.0.4 - Include repository_dependencies.xml file for 'blastxml' format - (previously included in the core Galaxy installation) -v0.0.5 - Quote arguments in case of spaces in filenames (internal change) - - Last release supporting b2g4pipe v2.3.5 -v0.0.6 - Support for b2g4pipe v2.5 instead of v2.3.5 - - - Now invoked with a class path and es.blast2go.prog.B2GAnnotPipe - rather then simply calling the jar file - - Now uses the switch -annot instead of -a (this change breaks - support for b2g4pipe v2.3.5 unfortunately) - - - Catch a few error messages and treat them explicitly as errors. -v0.0.7 - Update output description in XML file (b2g4pipe v2.3.5 included - the sequence description, b2g4pipe v2.5 omits this). -v0.0.8 - Automated installation via the Galaxy Tool Shed. - - Added unit test. - - Explain how to load the tabular file into the Blast2GO GUI. - - Link to Tool Shed added to help text and this documentation. - - Switch to standard MIT licence. - - Use reStructuredText for this README file. - - Updated citation information (Cock et al. 2013). - - Development moved to GitHub, https://github.com/peterjc/galaxy_blast - - Split out massage_xml_for_blast2go.py as a standalone file. -======= ====================================================================== - - -Developers -========== - -This script and related tools were originally developed on the 'tools' branch -of the following BitBucket Mercurial repository: -https://bitbucket.org/peterjc/galaxy-central/ - -As of September 2013, development is continuing on a dedicated GitHub repository: -https://github.com/peterjc/galaxy_blast - -For making the "Galaxy Tool Shed" http://toolshed.g2.bx.psu.edu/ tarball I use -the following command from the Galaxy root folder:: - - $ tar -czf blast2go.tar.gz blast2go/README.rst blast2go/blast2go.xml blast2go/blast2go.py blast2go/massage_xml_for_blast2go.py blast2go/repository_dependencies.xml blast2go/tool_dependencies.xml tool-data/blast2go.loc.sample test-data/blastp_sample.xml test-data/blastp_sample.blast2go.tabular - -Check this worked:: - - $ tar -tzf blast2go.tar.gz - blast2go/README.rst - blast2go/blast2go.xml - blast2go/blast2go.py - blast2go/massage_xml_for_blast2go.py - blast2go/repository_dependencies.xml - blast2go/tool_dependencies.xml - tool-data/blast2go.loc.sample - test-data/blastp_sample.xml - test-data/blastp_sample.blast2go.tabular - - -Licence (MIT) -============= - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. - - -NOTE: This is the licence for the Galaxy Wrapper only. Blast2GO and -associated data files are available and licenced separately.
--- a/blast2go/blast2go.py Mon Sep 23 05:54:09 2013 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,134 +0,0 @@ -#!/usr/bin/env python -"""Galaxy wrapper for Blast2GO for pipelines, b2g4pipe v2.5. - -This script takes exactly three command line arguments: - * Input BLAST XML filename - * Blast2GO properties filename (settings file) - * Output tabular filename - -The properties filename can be a fully qualified path, but if not -this will look next to the blast2go.jar file. - -Sadly b2g4pipe (at least v2.3.5 to v2.5.0) cannot cope with current -style large BLAST XML files (e.g. from BLAST 2.2.25+), so we reformat -these to avoid it crashing with a Java heap space OutOfMemoryError. - -As part of this reformatting, we check for BLASTP or BLASTX output -(otherwise raise an error), and print the query count. - -It then calls the Java command line tool, and moves the output file to -the location Galaxy is expecting, and removes the tempory XML file. - -This script is called from my Galaxy wrapper for Blast2GO for pipelines, -available from the Galaxy Tool Shed here: -http://toolshed.g2.bx.psu.edu/view/peterjc/blast2go - -This script is under version control here: -https://github.com/peterjc/galaxy_blast/tree/master/blast2go -""" -import sys -import os -import subprocess - -#You may need to edit this to match your local setup, -blast2go_dir = os.environ.get("B2G4PIPE", "/opt/b2g4pipe_v2.5/") -blast2go_jar = os.path.join(blast2go_dir, "blast2go.jar") - -def stop_err(msg, error_level=1): - """Print error message to stdout and quit with given error level.""" - sys.stderr.write("%s\n" % msg) - sys.exit(error_level) - -try: - from massage_xml_for_blast2go import prepare_xml -except ImportError: - stop_err("Missing sister file massage_xml_for_blast2go.py") - -if len(sys.argv) != 4: - stop_err("Require three arguments: XML filename, properties filename, output tabular filename") - -xml_file, prop_file, tabular_file = sys.argv[1:] - -#We should have write access here: -tmp_xml_file = tabular_file + ".tmp.xml" - -if not os.path.isfile(blast2go_jar): - stop_err("Blast2GO JAR file not found: %s" % blast2go_jar) - -if not os.path.isfile(xml_file): - stop_err("Input BLAST XML file not found: %s" % xml_file) - -if not os.path.isfile(prop_file): - tmp = os.path.join(os.path.split(blast2go_jar)[0], prop_file) - if os.path.isfile(tmp): - #The properties file seems to have been given relative to the JAR - prop_file = tmp - else: - stop_err("Blast2GO configuration file not found: %s" % prop_file) - del tmp - - -def run(cmd): - #Avoid using shell=True when we call subprocess to ensure if the Python - #script is killed, so too is the child process. - try: - child = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - except Exception, err: - stop_err("Error invoking command:\n%s\n\n%s\n" % (" ".join(cmd), err)) - #Use .communicate as can get deadlocks with .wait(), - stdout, stderr = child.communicate() - return_code = child.returncode - - #keep stdout minimal as shown prominently in Galaxy - #Record it in case a silent error needs diagnosis - if stdout: - sys.stderr.write("Standard out:\n%s\n\n" % stdout) - if stderr: - sys.stderr.write("Standard error:\n%s\n\n" % stderr) - - error_msg = None - if return_code: - cmd_str = " ".join(cmd) - error_msg = "Return code %i from command:\n%s" % (return_code, cmd_str) - elif "Database or network connection (timeout) error" in stdout+stderr: - error_msg = "Database or network connection (timeout) error" - elif "Annotation of 0 seqs with 0 annots finished." in stdout+stderr: - error_msg = "No sequences processed!" - - if error_msg: - print error_msg - stop_err(error_msg) - - -blast2go_classpath = os.path.split(blast2go_jar)[0] -assert os.path.isdir(blast2go_classpath) -blast2go_classpath = "%s/*:%s/ext/*:" % (blast2go_classpath, blast2go_classpath) - -prepare_xml(xml_file, tmp_xml_file) -#print "XML file prepared for Blast2GO" - -#We will have write access wherever the output should be, -#so we'll ask Blast2GO to use that as the stem for its output -#(it will append .annot to the filename) -cmd = ["java", "-cp", blast2go_classpath, "es.blast2go.prog.B2GAnnotPipe", - "-in", tmp_xml_file, - "-prop", prop_file, - "-out", tabular_file, #Used as base name for output files - "-annot", # Generate *.annot tabular file - #NOTE: For v2.3.5 must use -a, for v2.5 must use -annot instead - #"-img", # Generate images, feature not in v2.3.5 - ] -#print " ".join(cmd) -run(cmd) - -#Remove the temp XML file -os.remove(tmp_xml_file) - -out_file = tabular_file + ".annot" -if not os.path.isfile(out_file): - stop_err("ERROR - No output annotation file from Blast2GO") - -#Move the output file where Galaxy expects it to be: -os.rename(out_file, tabular_file) - -print "Done"
--- a/blast2go/blast2go.xml Mon Sep 23 05:54:09 2013 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,118 +0,0 @@ -<tool id="blast2go" name="Blast2GO" version="0.0.8"> - <description>Maps BLAST results to GO annotation terms</description> - <requirements> - <requirement type="package" version="2.5">b2g4pipe</requirement> - </requirements> - <command interpreter="python"> - blast2go.py "${xml}" "${prop.fields.path}" "${tab}" - </command> - <stdio> - <!-- Wrapper ensures anything other than zero is an error --> - <exit_code range="1:" /> - <exit_code range=":-1" /> - </stdio> - <inputs> - <param name="xml" type="data" format="blastxml" label="BLAST XML results" description="You must have run BLAST against a protein database such as the NCBI non-redundant (NR) database. Use BLASTX for nucleotide queries, BLASTP for protein queries." /> - <param name="prop" type="select" label="Blast2GO settings" description="One or more configurations can be setup, such as using the Blast2GO team's server in Spain, or a local database."> - <options from_file="blast2go.loc"> - <column name="value" index="0"/> - <column name="name" index="1"/> - <column name="path" index="2"/> - </options> - </param> - </inputs> - <outputs> - <data name="tab" format="tabular" label="Blast2GO ${prop.fields.name}" /> - </outputs> - <tests> - <test> - <param name="xml" value="blastp_sample.xml" ftype="blastxml"/> - <param name="prop" value="Spain_2011_June"/> - <output name="tab" file="blastp_sample.blast2go.tabular" ftype="tabular"/> - </test> - </tests> - <help> -.. class:: warningmark - -**Note**. Blast2GO may take a substantial amount of time, especially if -running against the public server in Spain. For large input datasets it -is advisable to allow overnight processing, or consider subdividing. - ------ - -**What it does** - -This runs b2g4Pipe v2.5, which is the command line (no GUI) version of -Blast2GO designed for use in pipelines. - -It takes as input BLAST XML results against a protein database, typically -the NCBI non-redundant (NR) database. This tool will accept concatenated -BLAST XML files (although they are technically invalid XML), which is very -useful if you have sub-divided your protein FASTA files and run BLAST on -them in batches. - -The BLAST matches are used to assign Gene Ontology (GO) annotation terms -to each query sequence. - -The output from this tool is a tabular file containing three columns, with -the order taken from query order in the original BLAST XML file: - -====== ==================== -Column Description ------- -------------------- - 1 ID of query sequence - 2 GO term - 3 GO description -====== ==================== - -Note that if no GO terms are assigned to a sequence (e.g. if it had no -BLAST matches), then it will not be present in the output file. - -This tabular file is called an "Annotation File" in the Blast2GO GUI. -If you download the tabular file, and rename it to use the extension -".annot", then it can be opened with the Blast2GO GUI via the "File", -"Load Annotation (.annot)" menu (keyboard shortcut ALT+L). You can -then run some of the interactive analyses offered in the GUI tool. - - -**Advanced Settings** - -Blast2GO has a properties setting file which includes which database -server to connect to (e.g. the public server in Valencia, Spain, or a -local server), as well as more advanced options such as thresholds and -evidence code weights. To change these settings, your Galaxy administrator -must create a new properties file, and add it to the drop down menu above. - - -**References** - -If you use this Galaxy tool in work leading to a scientific publication please -cite the following papers: - -Peter Cock, Bjoern Gruening, Konrad Paszkiewicz and Leighton Pritchard (2013). -Galaxy tools and workflows for sequence analysis with applications -in molecular plant pathology. PeerJ 1:e167 -http://dx.doi.org/10.7717/peerj.167 - -S. Götz et al. (2008). -High-throughput functional annotation and data mining with the Blast2GO suite. -Nucleic Acids Res. 36(10):3420–3435. -http://dx.doi.org/10.1093/nar/gkn176 - -A. Conesa and S. Götz (2008). -Blast2GO: A Comprehensive Suite for Functional Analysis in Plant Genomics. -International Journal of Plant Genomics. 619832. -http://dx.doi.org/10.1155/2008/619832 - -A. Conesa et al. (2005). -Blast2GO: A universal tool for annotation, visualization and analysis in functional genomics research. -Bioinformatics 21:3674-3676. -http://dx.doi.org/10.1093/bioinformatics/bti610 - -See also http://www.blast2go.com/ - -This wrapper is available to install into other Galaxy Instances via the Galaxy -Tool Shed at http://toolshed.g2.bx.psu.edu/view/peterjc/blast2go - - </help> -</tool>
--- a/blast2go/massage_xml_for_blast2go.py Mon Sep 23 05:54:09 2013 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,92 +0,0 @@ -#!/usr/bin/env python -"""Script for reformatting Blast XML to suite Blast2GO. - -This script takes exactly two command line arguments: - * Input BLAST XML filename - * Output BLAST XML filename - -Sadly b2g4pipe (at least v2.3.5 to v2.5.0) cannot cope with current -style large BLAST XML files (e.g. from BLAST 2.2.25+), so we reformat -these to avoid it crashing with a Java heap space OutOfMemoryError. - -As part of this reformatting, we check for BLASTP or BLASTX output -(otherwise raise an error), and print the query count. - -This script is called from my Galaxy wrapper for Blast2GO for pipelines, -available from the Galaxy Tool Shed here: -http://toolshed.g2.bx.psu.edu/view/peterjc/blast2go - -This script is under version control here: -https://github.com/peterjc/galaxy_blast/tree/master/blast2go -""" -import sys -import os -import subprocess - -def stop_err(msg, error_level=1): - """Print error message to stdout and quit with given error level.""" - sys.stderr.write("%s\n" % msg) - sys.exit(error_level) - -def prepare_xml(original_xml, mangled_xml): - """Reformat BLAST XML to suit Blast2GO. - - Blast2GO can't cope with 1000s of <Iteration> tags within a - single <BlastResult> tag, so instead split this into one - full XML record per interation (i.e. per query). This gives - a concatenated XML file mimicing old versions of BLAST. - - This also checks for BLASTP or BLASTX output, and outputs - the number of queries. Galaxy will show this as "info". - """ - in_handle = open(original_xml) - footer = " </BlastOutput_iterations>\n</BlastOutput>\n" - header = "" - while True: - line = in_handle.readline() - if not line: - #No hits? - stop_err("Problem with XML file?") - if line.strip() == "<Iteration>": - break - header += line - - if "<BlastOutput_program>blastx</BlastOutput_program>" in header: - print "BLASTX output identified" - elif "<BlastOutput_program>blastp</BlastOutput_program>" in header: - print "BLASTP output identified" - else: - in_handle.close() - stop_err("Expect BLASTP or BLASTX output") - - out_handle = open(mangled_xml, "w") - out_handle.write(header) - out_handle.write(line) - count = 1 - while True: - line = in_handle.readline() - if not line: - break - elif line.strip() == "<Iteration>": - #Insert footer/header - out_handle.write(footer) - out_handle.write(header) - count += 1 - out_handle.write(line) - - out_handle.close() - in_handle.close() - print "Input has %i queries" % count - - -if __name__ == "__main__": - # Run the conversion... - if len(sys.argv) != 3: - stop_err("Require two arguments: XML input filename, XML output filename") - - xml_file, out_xml_file = sys.argv[1:] - - if not os.path.isfile(xml_file): - stop_err("Input BLAST XML file not found: %s" % xml_file) - - prepare_xml(xml_file, out_xml_file)
--- a/blast2go/repository_dependencies.xml Mon Sep 23 05:54:09 2013 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,4 +0,0 @@ -<?xml version="1.0"?> -<repositories description="Requires BLAST XML and database datatype definitions."> - <repository changeset_revision="a44a7a5456e1" name="blast_datatypes" owner="devteam" toolshed="http://testtoolshed.g2.bx.psu.edu" /> -</repositories>
--- a/blast2go/tool_dependencies.xml Mon Sep 23 05:54:09 2013 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,32 +0,0 @@ -<?xml version="1.0"?> -<tool_dependency> - <package name="b2g4pipe" version="2.5"> - <install version="1.0"> - <actions> - <!-- If used, download_by_url must be the first action --> - <!-- The ZIP file decompresses to give a folder b2g4pipe --> - <action type="download_by_url">http://www.blast2go.com/data/blast2go/b2g4pipe_v2.5.zip</action> - <!-- Galaxy moves into the unzipped folder b2g4pipe --> - <action type="shell_command"> -cp b2gPipe.properties Spain_2012_August.properties && -sed -i "s/Dbacces.dbname=b2g_apr12/Dbacces.dbname=b2g_aug12/g" Spain_2012_August.properties && -sed -i "s/Dbacces.dbhost=10.10.100.203/Dbacces.dbhost=publicdb.blast2go.com/g" Spain_2012_August.properties - </action> - <action type="shell_command"> -cp b2gPipe.properties Spain_2011_June.properties && -sed -i "s/Dbacces.dbname=b2g_apr12/Dbacces.dbname=b2g_jun11/g" Spain_2011_June.properties && -sed -i "s/Dbacces.dbhost=10.10.100.203/Dbacces.dbhost=publicdb.blast2go.com/g" Spain_2011_June.properties - </action> - <action type="move_directory_files"><source_directory>.</source_directory><destination_directory>$INSTALL_DIR/</destination_directory></action> - <!-- Set environment variable $B2G4PIPE so Python script knows where to look --> - <action type="set_environment"> - <environment_variable name="B2G4PIPE" action="set_to">$INSTALL_DIR</environment_variable> - </action> - </actions> - </install> - <readme> -Downloads b2g4pipe v2.5 - </readme> - </package> -</tool_dependency> -
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/blast2go/README.rst Thu Jul 31 05:40:57 2014 -0400 @@ -0,0 +1,220 @@ +Galaxy wrapper for Blast2GO for pipelines, b2g4pipe +=================================================== + +This wrapper is copyright 2011-2014 by Peter Cock, The James Hutton Institute +(formerly SCRI, Scottish Crop Research Institute), UK. All rights reserved. +See the licence text below (MIT licence). + +This is a wrapper for the command line Java tool b2g4pipe v2.5, Blast2GO for +pipelines, currently a free to use download available at: +http://www.blast2go.com/data/blast2go/b2g4pipe_v2.5.zip + +Note that this has been superceded by a non-free "Blast2GO Command Line (CLI)": +http://www.blast2go.com/blast2gocli/ + +This wrapper is freely available from the Galaxy Tool Shed at: +http://toolshed.g2.bx.psu.edu/view/peterjc/blast2go + + +References +========== + +Peter Cock, Bjoern Gruening, Konrad Paszkiewicz and Leighton Pritchard (2013). +Galaxy tools and workflows for sequence analysis with applications +in molecular plant pathology. PeerJ 1:e167 +http://dx.doi.org/10.7717/peerj.167 + +S. Geotz et al. (2008). +High-throughput functional annotation and data mining with the Blast2GO suite. +Nucleic Acids Res. 36(10):3420-3435. +http://dx.doi.org/10.1093/nar/gkn176 + +A. Conesa and S. Geotz (2008). +Blast2GO: A Comprehensive Suite for Functional Analysis in Plant Genomics. +International Journal of Plant Genomics. 619832. +http://dx.doi.org/10.1155/2008/619832 + +A. Conesa et al. (2005). +Blast2GO: A universal tool for annotation, visualization and analysis in functional genomics research. +Bioinformatics 21:3674-3676. +http://dx.doi.org/10.1093/bioinformatics/bti610 + +See also http://www.blast2go.com/ + + +Automated Installation +====================== + +Installation via the Galaxy Tool Shed should take care of the Galaxy side of +things, including the dependency on 'blast_datatypes' which defines the +'blastxml' file format. However, you will also probably need to configure +the Blast2GO property file(s), for example if you have a local Blast2GO +database (which we recommend for speed). + + +Manual Installation +=================== + +The main dependency is b2g4pipe which must be installed manually. Also we +strongly recommend installing a local Blast2GO database as well (see the +intructions below about the ``blast2go.loc`` file). At the time of writing, +the last free version is b2g4pipe v2.5 which is available here: + +* http://www.blast2go.com/data/blast2go/b2g4pipe_v2.5.zip + +You can change the path by setting the B2G4PIPE environement variable to +the desired folder, but by default the script looks for the JAR file here:: + + /opt/b2g4pipe_v2.5/blast2go.jar + +To install the wrapper manually, first install 'blast_datatypes', then +copy or move the following files under the Galaxy tools folder, e.g. in a +tools/blast2go/ folder: + +* blast2go.xml (the Galaxy tool definition) +* blast2go.py (the Python wrapper script) +* massage_xml_for_blast2go.py (Python XML reformatting script) +* README.rst (this file) + +For a manual installation of the wrapper you will also need to modify the +tools_conf.xml file to tell Galaxy to offer the tool. We suggest putting +it next to the NCBI BLAST+ wrappers. Just add the line:: + + <tool file="blast2go/blast2go.xml" /> + +If you wish to run the unit tests, also add this to tools_conf.xml.sample +and move/copy the test-data files under Galaxy's test-data folder. Then:: + + $ ./run_functional_tests.sh -id blast2go + + +Configuration +============= + +As part of setting up b2g4pipe you will need to setup one or more Blast2GO +property files which tell the tool which database to use etc. The example +``b2gPipe.properties`` provided with b2g4pipe is now out of date. The current +server IP address and database name may given on the Blast2GO website, or +can be found by running the latest GUI version via Java web-start, and +looking under the tools/options menu. These property files can be anywhere +accessable to the Galaxy Unix user, we put them with the JAR file etc. + +You must tell Galaxy about these Blast2GO property files so that they can +be offered to the user. Copy file ``blast2go.loc.sample`` to +``tool-data/blast2go.loc`` under the Galaxy folder and edit this to match +your installation. This must be plain text, tab separated, with three columns: + +1. ID for the setup, e.g. ``Spain_2012_August`` +2. Description for the setup, e.g. ``Database in Spain (August 2012)`` +3. Properties filename for the setup, e.g. ``Spain_2012_August.properties`` + relative to the main JAR file, or with a full path + e.g. ``/opt/b2g4pipe/Spain_2012_August.properties`` + +Avoid including "Blast2GO" in the description (column 2) as this text will be +included in the automatically assigned output dataset name. The ``blast2go.loc`` +file allows you to customise the database setup. If for example you have a local +Blast2GO server running (which we recommend for speed), and you want this to be +the default setting, include it as the first line in your ``blast2go.loc`` file. + +Consult the Blast2GO documentation for details about the property files and +setting up a local MySQL Blast2GO database. e.g. +https://www.blast2go.com/b2gsupport/resources/35-localb2gdb + + + +History +======= + +======= ====================================================================== +Version Changes +------- ---------------------------------------------------------------------- +v0.0.1 - Initial public release +v0.0.2 - Documentation clarifications, e.g. concatenated BLAST XML is allowed. + - Fixed error handler in wrapper script (for when b2g4pipe fails). + - Reformats the XML to use old NCBI-style concatenated BLAST XML since + b2g4pipe crashes with heap space error on with large files using + current NCBI output. +v0.0.3 - Include sample loc file, ``tool-data/blast2go.loc.sample`` +v0.0.4 - Include ``repository_dependencies.xml`` file for ``blastxml`` format + (previously included in the core Galaxy installation) +v0.0.5 - Quote arguments in case of spaces in filenames (internal change) + - Last release supporting b2g4pipe v2.3.5 +v0.0.6 - Support for b2g4pipe v2.5 instead of v2.3.5 + + - Now invoked with a class path and es.blast2go.prog.B2GAnnotPipe + rather then simply calling the jar file + - Now uses the switch ``-annot`` instead of ``-a`` (this change + breaks support for b2g4pipe v2.3.5 unfortunately) + + - Catch a few error messages and treat them explicitly as errors. +v0.0.7 - Update output description in XML file (b2g4pipe v2.3.5 included + the sequence description, b2g4pipe v2.5 omits this). +v0.0.8 - Automated installation via the Galaxy Tool Shed. + - Added unit test. + - Explain how to load the tabular file into the Blast2GO GUI. + - Link to Tool Shed added to help text and this documentation. + - Switch to standard MIT licence. + - Use reStructuredText for this README file. + - Updated citation information (Cock et al. 2013). + - Development moved to GitHub, https://github.com/peterjc/galaxy_blast + - Split out ``massage_xml_for_blast2go.py`` as a standalone file. +v0.0.9 - Update README file now that BioBam are selling the latest version + of the Blast2GO command line tool. For now b2g4pipe v2.5 is still + available as a free download. + - Tool definition now embeds citation information. +======= ====================================================================== + + +Developers +========== + +This script and related tools were originally developed on the 'tools' branch +of the following BitBucket Mercurial repository: +https://bitbucket.org/peterjc/galaxy-central/ + +As of September 2013, development is continuing on a dedicated GitHub repository: +https://github.com/peterjc/galaxy_blast + +For making the "Galaxy Tool Shed" http://toolshed.g2.bx.psu.edu/ tarball I use +the following command from the Galaxy root folder:: + + $ tar -czf blast2go.tar.gz tools/blast2go/README.rst tools/blast2go/blast2go.xml tools/blast2go/blast2go.py tools/blast2go/massage_xml_for_blast2go.py tools/blast2go/repository_dependencies.xml tools/blast2go/tool_dependencies.xml tool-data/blast2go.loc.sample test-data/blastp_sample.xml test-data/blastp_sample.blast2go.tabular + +Check this worked:: + + $ tar -tzf blast2go.tar.gz + tools/blast2go/README.rst + tools/blast2go/blast2go.xml + tools/blast2go/blast2go.py + tools/blast2go/massage_xml_for_blast2go.py + tools/blast2go/repository_dependencies.xml + tools/blast2go/tool_dependencies.xml + tool-data/blast2go.loc.sample + test-data/blastp_sample.xml + test-data/blastp_sample.blast2go.tabular + + +Licence (MIT) +============= + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +NOTE: This is the licence for the Galaxy Wrapper only. Blast2GO and +associated data files are available and licenced separately.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/blast2go/blast2go.py Thu Jul 31 05:40:57 2014 -0400 @@ -0,0 +1,134 @@ +#!/usr/bin/env python +"""Galaxy wrapper for Blast2GO for pipelines, b2g4pipe v2.5. + +This script takes exactly three command line arguments: + * Input BLAST XML filename + * Blast2GO properties filename (settings file) + * Output tabular filename + +The properties filename can be a fully qualified path, but if not +this will look next to the blast2go.jar file. + +Sadly b2g4pipe (at least v2.3.5 to v2.5.0) cannot cope with current +style large BLAST XML files (e.g. from BLAST 2.2.25+), so we reformat +these to avoid it crashing with a Java heap space OutOfMemoryError. + +As part of this reformatting, we check for BLASTP or BLASTX output +(otherwise raise an error), and print the query count. + +It then calls the Java command line tool, and moves the output file to +the location Galaxy is expecting, and removes the tempory XML file. + +This script is called from my Galaxy wrapper for Blast2GO for pipelines, +available from the Galaxy Tool Shed here: +http://toolshed.g2.bx.psu.edu/view/peterjc/blast2go + +This script is under version control here: +https://github.com/peterjc/galaxy_blast/tree/master/blast2go +""" +import sys +import os +import subprocess + +#You may need to edit this to match your local setup, +blast2go_dir = os.environ.get("B2G4PIPE", "/opt/b2g4pipe_v2.5/") +blast2go_jar = os.path.join(blast2go_dir, "blast2go.jar") + +def stop_err(msg, error_level=1): + """Print error message to stdout and quit with given error level.""" + sys.stderr.write("%s\n" % msg) + sys.exit(error_level) + +try: + from massage_xml_for_blast2go import prepare_xml +except ImportError: + stop_err("Missing sister file massage_xml_for_blast2go.py") + +if len(sys.argv) != 4: + stop_err("Require three arguments: XML filename, properties filename, output tabular filename") + +xml_file, prop_file, tabular_file = sys.argv[1:] + +#We should have write access here: +tmp_xml_file = tabular_file + ".tmp.xml" + +if not os.path.isfile(blast2go_jar): + stop_err("Blast2GO JAR file not found: %s" % blast2go_jar) + +if not os.path.isfile(xml_file): + stop_err("Input BLAST XML file not found: %s" % xml_file) + +if not os.path.isfile(prop_file): + tmp = os.path.join(os.path.split(blast2go_jar)[0], prop_file) + if os.path.isfile(tmp): + #The properties file seems to have been given relative to the JAR + prop_file = tmp + else: + stop_err("Blast2GO configuration file not found: %s" % prop_file) + del tmp + + +def run(cmd): + #Avoid using shell=True when we call subprocess to ensure if the Python + #script is killed, so too is the child process. + try: + child = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + except Exception, err: + stop_err("Error invoking command:\n%s\n\n%s\n" % (" ".join(cmd), err)) + #Use .communicate as can get deadlocks with .wait(), + stdout, stderr = child.communicate() + return_code = child.returncode + + #keep stdout minimal as shown prominently in Galaxy + #Record it in case a silent error needs diagnosis + if stdout: + sys.stderr.write("Standard out:\n%s\n\n" % stdout) + if stderr: + sys.stderr.write("Standard error:\n%s\n\n" % stderr) + + error_msg = None + if return_code: + cmd_str = " ".join(cmd) + error_msg = "Return code %i from command:\n%s" % (return_code, cmd_str) + elif "Database or network connection (timeout) error" in stdout+stderr: + error_msg = "Database or network connection (timeout) error" + elif "Annotation of 0 seqs with 0 annots finished." in stdout+stderr: + error_msg = "No sequences processed!" + + if error_msg: + print error_msg + stop_err(error_msg) + + +blast2go_classpath = os.path.split(blast2go_jar)[0] +assert os.path.isdir(blast2go_classpath) +blast2go_classpath = "%s/*:%s/ext/*:" % (blast2go_classpath, blast2go_classpath) + +prepare_xml(xml_file, tmp_xml_file) +#print "XML file prepared for Blast2GO" + +#We will have write access wherever the output should be, +#so we'll ask Blast2GO to use that as the stem for its output +#(it will append .annot to the filename) +cmd = ["java", "-cp", blast2go_classpath, "es.blast2go.prog.B2GAnnotPipe", + "-in", tmp_xml_file, + "-prop", prop_file, + "-out", tabular_file, #Used as base name for output files + "-annot", # Generate *.annot tabular file + #NOTE: For v2.3.5 must use -a, for v2.5 must use -annot instead + #"-img", # Generate images, feature not in v2.3.5 + ] +#print " ".join(cmd) +run(cmd) + +#Remove the temp XML file +os.remove(tmp_xml_file) + +out_file = tabular_file + ".annot" +if not os.path.isfile(out_file): + stop_err("ERROR - No output annotation file from Blast2GO") + +#Move the output file where Galaxy expects it to be: +os.rename(out_file, tabular_file) + +print "Done"
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/blast2go/blast2go.xml Thu Jul 31 05:40:57 2014 -0400 @@ -0,0 +1,124 @@ +<tool id="blast2go" name="Blast2GO" version="0.0.9"> + <description>Maps BLAST results to GO annotation terms</description> + <requirements> + <requirement type="package" version="2.5">b2g4pipe</requirement> + </requirements> + <command interpreter="python"> + blast2go.py "${xml}" "${prop.fields.path}" "${tab}" + </command> + <stdio> + <!-- Wrapper ensures anything other than zero is an error --> + <exit_code range="1:" /> + <exit_code range=":-1" /> + </stdio> + <inputs> + <param name="xml" type="data" format="blastxml" label="BLAST XML results" description="You must have run BLAST against a protein database such as the NCBI non-redundant (NR) database. Use BLASTX for nucleotide queries, BLASTP for protein queries." /> + <param name="prop" type="select" label="Blast2GO settings" description="One or more configurations can be setup, such as using the Blast2GO team's server in Spain, or a local database."> + <options from_file="blast2go.loc"> + <column name="value" index="0"/> + <column name="name" index="1"/> + <column name="path" index="2"/> + </options> + </param> + </inputs> + <outputs> + <data name="tab" format="tabular" label="Blast2GO ${prop.fields.name}" /> + </outputs> + <tests> + <test> + <param name="xml" value="blastp_sample.xml" ftype="blastxml"/> + <param name="prop" value="Spain_2011_June"/> + <output name="tab" file="blastp_sample.blast2go.tabular" ftype="tabular"/> + </test> + </tests> + <help> +.. class:: warningmark + +**Note**. Blast2GO may take a substantial amount of time, especially if +running against the public server in Spain. For large input datasets it +is advisable to allow overnight processing, or consider subdividing. + +----- + +**What it does** + +This runs b2g4Pipe v2.5, which is the command line (no GUI) version of +Blast2GO designed for use in pipelines. + +It takes as input BLAST XML results against a protein database, typically +the NCBI non-redundant (NR) database. This tool will accept concatenated +BLAST XML files (although they are technically invalid XML), which is very +useful if you have sub-divided your protein FASTA files and run BLAST on +them in batches. + +The BLAST matches are used to assign Gene Ontology (GO) annotation terms +to each query sequence. + +The output from this tool is a tabular file containing three columns, with +the order taken from query order in the original BLAST XML file: + +====== ==================== +Column Description +------ -------------------- + 1 ID of query sequence + 2 GO term + 3 GO description +====== ==================== + +Note that if no GO terms are assigned to a sequence (e.g. if it had no +BLAST matches), then it will not be present in the output file. + +This tabular file is called an "Annotation File" in the Blast2GO GUI. +If you download the tabular file, and rename it to use the extension +".annot", then it can be opened with the Blast2GO GUI via the "File", +"Load Annotation (.annot)" menu (keyboard shortcut ALT+L). You can +then run some of the interactive analyses offered in the GUI tool. + + +**Advanced Settings** + +Blast2GO has a properties setting file which includes which database +server to connect to (e.g. the public server in Valencia, Spain, or a +local server), as well as more advanced options such as thresholds and +evidence code weights. To change these settings, your Galaxy administrator +must create a new properties file, and add it to the drop down menu above. + + +**References** + +If you use this Galaxy tool in work leading to a scientific publication please +cite the following papers: + +Peter Cock, Bjoern Gruening, Konrad Paszkiewicz and Leighton Pritchard (2013). +Galaxy tools and workflows for sequence analysis with applications +in molecular plant pathology. PeerJ 1:e167 +http://dx.doi.org/10.7717/peerj.167 + +S. Götz et al. (2008). +High-throughput functional annotation and data mining with the Blast2GO suite. +Nucleic Acids Res. 36(10):3420–3435. +http://dx.doi.org/10.1093/nar/gkn176 + +A. Conesa and S. Götz (2008). +Blast2GO: A Comprehensive Suite for Functional Analysis in Plant Genomics. +International Journal of Plant Genomics. 619832. +http://dx.doi.org/10.1155/2008/619832 + +A. Conesa et al. (2005). +Blast2GO: A universal tool for annotation, visualization and analysis in functional genomics research. +Bioinformatics 21:3674-3676. +http://dx.doi.org/10.1093/bioinformatics/bti610 + +See also http://www.blast2go.com/ + +This wrapper is available to install into other Galaxy Instances via the Galaxy +Tool Shed at http://toolshed.g2.bx.psu.edu/view/peterjc/blast2go + + </help> + <citations> + <citation type="doi">10.7717/peerj.167</citation> + <citation type="doi">10.1093/nar/gkn176</citation> + <citation type="doi">10.1155/2008/619832</citation> + <citation type="doi">10.1093/bioinformatics/bti610</citation> + </citations> +</tool>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/blast2go/massage_xml_for_blast2go.py Thu Jul 31 05:40:57 2014 -0400 @@ -0,0 +1,92 @@ +#!/usr/bin/env python +"""Script for reformatting Blast XML to suit Blast2GO. + +This script takes exactly two command line arguments: + * Input BLAST XML filename + * Output BLAST XML filename + +Sadly b2g4pipe (at least v2.3.5 to v2.5.0) cannot cope with current +style large BLAST XML files (e.g. from BLAST 2.2.25+), so we reformat +these to avoid it crashing with a Java heap space OutOfMemoryError. + +As part of this reformatting, we check for BLASTP or BLASTX output +(otherwise raise an error), and print the query count. + +This script is called from my Galaxy wrapper for Blast2GO for pipelines, +available from the Galaxy Tool Shed here: +http://toolshed.g2.bx.psu.edu/view/peterjc/blast2go + +This script is under version control here: +https://github.com/peterjc/galaxy_blast/tree/master/blast2go +""" +import sys +import os +import subprocess + +def stop_err(msg, error_level=1): + """Print error message to stdout and quit with given error level.""" + sys.stderr.write("%s\n" % msg) + sys.exit(error_level) + +def prepare_xml(original_xml, mangled_xml): + """Reformat BLAST XML to suit Blast2GO. + + Blast2GO can't cope with 1000s of <Iteration> tags within a + single <BlastResult> tag, so instead split this into one + full XML record per interation (i.e. per query). This gives + a concatenated XML file mimicing old versions of BLAST. + + This also checks for BLASTP or BLASTX output, and outputs + the number of queries. Galaxy will show this as "info". + """ + in_handle = open(original_xml) + footer = " </BlastOutput_iterations>\n</BlastOutput>\n" + header = "" + while True: + line = in_handle.readline() + if not line: + #No hits? + stop_err("Problem with XML file?") + if line.strip() == "<Iteration>": + break + header += line + + if "<BlastOutput_program>blastx</BlastOutput_program>" in header: + print "BLASTX output identified" + elif "<BlastOutput_program>blastp</BlastOutput_program>" in header: + print "BLASTP output identified" + else: + in_handle.close() + stop_err("Expect BLASTP or BLASTX output") + + out_handle = open(mangled_xml, "w") + out_handle.write(header) + out_handle.write(line) + count = 1 + while True: + line = in_handle.readline() + if not line: + break + elif line.strip() == "<Iteration>": + #Insert footer/header + out_handle.write(footer) + out_handle.write(header) + count += 1 + out_handle.write(line) + + out_handle.close() + in_handle.close() + print "Input has %i queries" % count + + +if __name__ == "__main__": + # Run the conversion... + if len(sys.argv) != 3: + stop_err("Require two arguments: XML input filename, XML output filename") + + xml_file, out_xml_file = sys.argv[1:] + + if not os.path.isfile(xml_file): + stop_err("Input BLAST XML file not found: %s" % xml_file) + + prepare_xml(xml_file, out_xml_file)
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/blast2go/repository_dependencies.xml Thu Jul 31 05:40:57 2014 -0400 @@ -0,0 +1,4 @@ +<?xml version="1.0"?> +<repositories description="Requires BLAST XML and database datatype definitions."> + <repository changeset_revision="939a600f45e9" name="blast_datatypes" owner="devteam" toolshed="https://testtoolshed.g2.bx.psu.edu" /> +</repositories>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/blast2go/tool_dependencies.xml Thu Jul 31 05:40:57 2014 -0400 @@ -0,0 +1,32 @@ +<?xml version="1.0"?> +<tool_dependency> + <package name="b2g4pipe" version="2.5"> + <install version="1.0"> + <actions> + <!-- If used, download_by_url must be the first action --> + <!-- The ZIP file decompresses to give a folder b2g4pipe --> + <action type="download_by_url">http://www.blast2go.com/data/blast2go/b2g4pipe_v2.5.zip</action> + <!-- Galaxy moves into the unzipped folder b2g4pipe --> + <action type="shell_command"> +cp b2gPipe.properties Spain_2012_August.properties && +sed -i "s/Dbacces.dbname=b2g_apr12/Dbacces.dbname=b2g_aug12/g" Spain_2012_August.properties && +sed -i "s/Dbacces.dbhost=10.10.100.203/Dbacces.dbhost=publicdb.blast2go.com/g" Spain_2012_August.properties + </action> + <action type="shell_command"> +cp b2gPipe.properties Spain_2011_June.properties && +sed -i "s/Dbacces.dbname=b2g_apr12/Dbacces.dbname=b2g_jun11/g" Spain_2011_June.properties && +sed -i "s/Dbacces.dbhost=10.10.100.203/Dbacces.dbhost=publicdb.blast2go.com/g" Spain_2011_June.properties + </action> + <action type="move_directory_files"><source_directory>.</source_directory><destination_directory>$INSTALL_DIR/</destination_directory></action> + <!-- Set environment variable $B2G4PIPE so Python script knows where to look --> + <action type="set_environment"> + <environment_variable name="B2G4PIPE" action="set_to">$INSTALL_DIR</environment_variable> + </action> + </actions> + </install> + <readme> +Downloads b2g4pipe v2.5 + </readme> + </package> +</tool_dependency> +