Mercurial > repos > fubar > brokenandnotdeletablebyowneroradmin

changeset 19:a87a262220a4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Adding proof of concept for optional output formats. Updated readme again
author ross lazarus ross.lazarus@gmail.com
date Tue, 05 Jun 2012 21:54:58 +1000 (2012-06-05)
parents 943d4f0ee7c1
children 3f58115c6c5b
files README.txt rgToolFactory.py rgToolFactory.xml
diffstat 3 files changed, 146 insertions(+), 89 deletions(-) [+]
line wrap: on
line diff
--- a/README.txt	Mon Jun 04 21:32:24 2012 +1000
+++ b/README.txt	Tue Jun 05 21:54:58 2012 +1000
@@ -1,63 +1,83 @@
-## WARNING before you start
-## Install on a private Galaxy ONLY
-## Please NEVER on a public or production instance
-# To fully and safely exploit the awesome power of Galaxy with this tool installed
-# you should be a developer installing this tool on a 
-# personal/scratch local instance - ie a private site
-# because then, if you break it, you get to keep all the pieces
-# see https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
+# WARNING before you start
+# Install on a private Galaxy ONLY
+# Please NEVER on a public or production instance
+
+*Read on if* You use a production Galaxy;
+Your users sometimes take data out of Galaxy, process it with ugly
+little perl/awk/sed/R... scripts and put it back;
+They do this when they can't do some transformation in Galaxy (the 90/10 rule);
+You don't have enough developer resources for wrapping dozens of even relatively simple tools;
+Your institution would be better off if those nasty, feral scripts were all tucked safely in
+a local toolshed.
+
+*The good news* If it can be trivially scripted, it can be running safely in your
+local Galaxy via your own local toolshed.
+That's what this tool does. You paste a simple script and the tool returns 
+a new, real Galaxy tool, ready to be installed from the local toolshed to local servers.
+Scripts can be wrapped and online literally within minutes.
+
+*To fully and safely exploit the awesome power* of Galaxy with this tool installed
+you should be a developer installing this tool on a 
+personal/scratch local instance - ie a private site
+because then, if you break it, you get to keep all the pieces
+see https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
+
+*To make the tool work* If not already there, please add:
+<datatype extension="toolshed.gz" type="galaxy.datatypes.binary:Binary" mimetype="multipart/x-gzip" subclass="True" />
+to your local data_types_conf.xml
+Then, PUT some IDs in the list in the XML before you restart Galaxy to load this new tool please
+otherwise, the tool won't run for anybody.
+
+*What it does* This is a tool factory for simple scripts in python, R or whatever ails ye.
+LIMITED to simple scripts that read one input from the history.
+Optionally can write one new history dataset,
+and optionally collect any number of outputs into links on an autogenerated HTML page.
+Generated tools can be edited and enhanced like any Galaxy tool, so start small and build up
+A generated script gets you a serious leg up to a more complex one.
 
-# this is a tool factory for simple scripts in python, R or whatever ails ye.
-# LIMITED to simple scripts that read one input from the history.
-# Optionally can write one new history dataset,
-# and optionally collect any number of outputs into links on an autogenerated HTML page.
-# Generated tools can be edited and enhanced like any Galaxy tool, so start small and build up
-# A generated script gets you a serious leg up to a more complex one.
-#
-# you paste and run your script
-# you fix the syntax errors and eventually it runs
-# that's kind of cool, but
-# now things get even more interesting.
-# Once the script works on some test data, you can
-# optionally generate a toolshed compatible gzip file
-# containing your script neatly wrapped as an ordinary Galaxy script in your 
-# local toolshed forsafe and largely automated installation in a production Galaxy.
-# tests are being worked on - should be done soon
-#
-# installed generated tools are fine if the script is safe.
-# They just run normally and their user cannot do anything unusually insecure
-# but please, practice safe toolshed.
-# Read the fucking code before you install any tool. 
-# Especially this one - it is really scary.
+*What you do* You paste and run your script
+you fix the syntax errors and eventually it runs
+that's pretty good because you can use the redo button
+to rerun it and edit the script as you debug - cool, but
+now the power really kicks in, because
+once the script works on some test data, you can
+generate a toolshed compatible gzip file
+containing your script neatly wrapped and hidden safely inside an ordinary Galaxy script in your 
+local toolshed. That means safe and largely automated installation in any production Galaxy configured
+to use your toolshed or the tool
+Automated build for tests still being worked on - should be done soon
+
+*Generated tool Security* Once you install a generated tool, it's just
+another tool - assuming the script is safe. They just run normally and their user cannot do anything unusually insecure
+but please, practice safe toolshed.
+Read the fucking code before you install any tool. 
+Especially this one - it is really scary.
+
+If you opt for an HTML output, you get all the script outputs arranged
+as a single Html history item - all output files are linked, thumbnails for all the pdfs.
+Ugly but really inexpensive.
 
-# If you opt for an HTML output, you get all the script outputs arranged
-# as a single Html history item - all output files are linked, thumbnails for all the pdfs.
-# Ugly but really inexpensive.
-# 
-# Patches welcome please. 
-#
-#
-# long route to June 2012 product
-# derived from an integrated script model  
-# called rgBaseScriptWrapper.py
-# Note to the unwary:
-#   This tool allows arbitrary scripting on your Galaxy as the Galaxy user
-#   There is nothing stopping a malicious user doing whatever they choose
-#   Extremely dangerous!!
-#   Totally insecure. So, trusted users only
-#
-# a list in the xml source file is searched for users permitted to run this tool.
-# PUT some IDs in the list in the XML before you restart Galaxy to load this new tool please
-# otherwise, the tool won't run for anybody.
-#
+Patches welcome please. 
+
 
-# 
-# copyright ross lazarus (ross stop lazarus at gmail stop com) May 2012
-# 
-# all rights reserved
-# Licensed under the LGPL if you want to improve it, feel free https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
+long route to June 2012 product
+derived from an integrated script model  
+called rgBaseScriptWrapper.py
+Note to the unwary:
+  This tool allows arbitrary scripting on your Galaxy as the Galaxy user
+  There is nothing stopping a malicious user doing whatever they choose
+  Extremely dangerous!!
+  Totally insecure. So, trusted users only
 
-# Material for our more enthusiastic readers continues below
+
+
+
+copyright ross lazarus (ross stop lazarus at gmail stop com) May 2012
+
+all rights reserved
+Licensed under the LGPL if you want to improve it, feel free https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
+
+Material for our more enthusiastic readers continues below
 
 **Motivation** Simple transformation, filtering or reporting scripts get written, run and lost every day in most busy labs 
 - even ours where Galaxy is in use. This 'dark script matter' is pervasive and generally not reproducible.
--- a/rgToolFactory.py	Mon Jun 04 21:32:24 2012 +1000
+++ b/rgToolFactory.py	Tue Jun 05 21:54:58 2012 +1000
@@ -1,27 +1,54 @@
 # rgToolFactory.py
-# derived from 
-# rgBaseScriptWrapper.py
-# but designed to run arbitrary user supplied code 
-# extremely dangerous!!
-# trusted users only - private site only
-# a list in the xml is searched for users permitted to run this tool.
-# DO NOT install on a public or important site - local instance only
-# generated tools are fine as they just run normally and their user cannot do anything unusually insecure
+# https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
+
+# this is a tool factory for simple scripts in python, R or whatever ails ye.
+# you paste and run your script
+# Only works for simple scripts that read one input from the history.
+# Optionally can write one new history dataset,
+# and optionally collect any number of outputs into links on an autogenerated HTML page.
+
+# DO NOT install on a public or important site - please.
+
+# installed generated tools are fine if the script is safe.
+# They just run normally and their user cannot do anything unusually insecure
+# but please, practice safe toolshed.
+# Read the fucking code before you install any tool 
+# especially this one
+
+# After you get the script working on some test data, you can
+# optionally generate a toolshed compatible gzip file
+# containing your script safely wrapped as an ordinary Galaxy script in your local toolshed for
+# safe and largely automated installation in a production Galaxy.
+
+# If you opt for an HTML output, you get all the script outputs arranged
+# as a single Html history item - all output files are linked, thumbnails for all the pdfs.
+# Ugly but really inexpensive.
+# 
+# Patches appreciated please. 
 #
-# copyright ross lazarus (ross.lazarus@gmail.com) May 2012
+#
+# long route to June 2012 product
+# Behold the awesome power of Galaxy and the toolshed with the tool factory binds to bind them
+# derived from an integrated script model  
+# called rgBaseScriptWrapper.py
+# Note to the unwary:
+#   This tool allows arbitrary scripting on your Galaxy as the Galaxy user
+#   There is nothing stopping a malicious user doing whatever they choose
+#   Extremely dangerous!!
+#   Totally insecure. So, trusted users only
+#
+# preferred model is a developer using their throw away workstation instance - ie a private site.
+# no real risk.
+# a list in the xml source file is searched for users permitted to run this tool.
+# PUT some IDs in the list in the XML before you restart Galaxy to load this new tool please
+# otherwise, the tool won't run for anybody.
+#
+
+# 
+# copyright ross lazarus (ross stop lazarus at gmail stop com) May 2012
 # 
 # all rights reserved
-# Licensed under the LGPL for your pleasure
-# Derived from rgDGE.py in May 2012
-# generalized to run required interpreter
-# to make your own tools based on a given script and interpreter such as perl or python
-# clone this and the corresponding xml wrapper
-# replace the parameters/inputs/outputs and the configfile contents with your script
-# Use the $foo syntax to place your parameter values inside the script to assign them - at run time, the script will be used as a template
-# and returned as part of the output to the user - with the right values for all the parameters.
-# Note that this assumes you want all the outputs arranged as a single Html file output 
-# after this generic script runner runs your script with the specified interpreter,
-# it will collect all output files into the specified output_html, making thumbnails for all the pdfs it finds and making links for all the other files.
+# Licensed under the LGPL if you want to improve it, feel free https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
 
 import sys 
 import shutil 
@@ -52,7 +79,7 @@
 
     def __init__(self,opts=None):
         """
-        run the script
+        cleanup inputs, setup some outputs
         
         """
         if opts.output_dir: # simplify for the tool tarball
@@ -93,8 +120,7 @@
     def makeXML(self):
         """
         Create a Galaxy xml tool wrapper for the new script as a string to write out
-        fixme - use templating or something less fugly than this.
-        Here's an example of what we produce
+        fixme - use templating or something less fugly than this example of what we produce
 
         <tool id="reverse" name="reverse" version="0.01">
             <description>a tabular file</description>
@@ -293,7 +319,7 @@
         <body> 
         <div class="document"> 
         """ 
-        galhtmlattr = """<hr/><b><a href="http://rgenetics.org">Galaxy Tool Factory Script Wrapper</a> tool output %s run at %s</b><br/>""" 
+        galhtmlattr = """<hr/><b><a href="https://bitbucket.org/fubar/galaxytoolfactory/overview">Galaxy Tool Factory Script Wrapper</a> tool output %s run at %s</b><br/>""" 
         galhtmlpostfix = """</div></body></html>\n"""
 
         flist = os.listdir(self.opts.output_dir)
@@ -379,7 +405,7 @@
     a('--tool_desc',default=None)
     a('--new_tool',default=None)
     opts, args = op.parse_args()
-    assert not opts.bad_user,'%s is NOT authorized to use this tool. Please ask your friendly admin' % opts.bad_user
+    assert not opts.bad_user,'%s is NOT authorized to use this tool. Please ask your friendly admin to add their ID to %s.xml' % (opts.bad_user,os.path.splitext(self.myname)[0])
     assert opts.tool_name,'## Tool Factory expects a tool name - eg --tool_name=DESeq'
     assert opts.interpreter,'## Tool Factory wrapper expects an interpreter - eg --interpreter=Rscript'
     assert os.path.isfile(opts.script_path),'## Tool Factory wrapper expects a script path - eg --script_path=foo.R'
--- a/rgToolFactory.xml	Mon Jun 04 21:32:24 2012 +1000
+++ b/rgToolFactory.xml	Tue Jun 05 21:54:58 2012 +1000
@@ -1,7 +1,7 @@
 <tool id="rgTF" name="Tool Factory" version="0.05">
   <description>Makes scripts into tools</description>
   <command interpreter="python">
-#if ( $__user_email__ not in ['rlazarus@bakeridi.edu.au','mziemann@bakeridi.edu.au','akaspi@bakeridi.edu.au'] ):
+#if ( $__user_email__ not in ['yourID@email','ross.lazarus@gmail.com'] ):
      rgToolFactory.py --bad_user $__user_email__
   #else:
     rgToolFactory.py --script_path "$runme" --interpreter "$interpreter" 
@@ -27,8 +27,8 @@
 #end if 
   </command>
   <inputs>
-    <param name="input1"  type="data"  label="Select an optional input file from your history" optional="true"
-       help="Your script probably needs an input - but if not, this can be left unassigned. Note that your script MUST be able to parse whatever format you choose!"/>
+    <param name="input1"  type="data"  label="Select an input file from your history" optional="true"
+       help="Most scripts will need an input - your script MUST be ready for whatever format you choose"/>
     <param name="tool_name" type="text" value="My dynamic script" size="80" label="Title for job outputs" help="Supply a meaningful name here to remind you what the outputs contain">
         <sanitizer>
             <valid initial="string.letters,string.digits">
@@ -50,12 +50,18 @@
                     <mapping initial="none"/>
                 </sanitizer>
             </param>
+        <param name="out_format" type="select" label="Galaxy datatype for your tool's output file" help="You may need to edit the xml to extend this list">
+        <option value="tabular" selected="true">Tabular</option>
+        <option value="interval">Interval</option>
+        <option value="gz">gz</option>
+        <option value="text">text</option>
+        </param>
         </when>
     </conditional> 
     <param name="make_HTML" type="select" label="Create an HTML report with links to all output files and thumbnail links to PDF images" 
-         help="Recommended for presenting complex outputs in an accessible manner">
-        <option value="yes">Yes</option>
-        <option value="" selected="true">No</option>
+         help="Recommended for presenting complex outputs in an accessible manner. Turn off for simple tools so they just create one output">
+        <option value="yes" selected="true">Yes</option>
+        <option value="">No</option>
     </param>   
     <param name="make_TAB" type="select" label="Create a new tabular history output" 
          help="This is useful if your script creates a single new tabular file you want to appear in the history after the tool executes">
@@ -77,14 +83,19 @@
      </param>
   </inputs>
   <outputs>
-    <data format="tabular" name="tab_file" label="${tool_name}.xls">
+    <data format="tabular" name="tab_file" label="${tool_name}.${factory.out_format}">
         <filter>make_TAB == "yes"</filter>
+        <change_format>
+            <when input="factory.out_format" value="interval"  format="interval" />
+            <when input="factory.out_format" value="gz" format="gz" />
+            <when input="factory.out_format" value="text"  format="text"  />
+         </change_format>
     </data>
     <data format="html" name="html_file" label="${tool_name}.html">
         <filter>make_HTML == "yes"</filter>
     </data>
     <data format="gz" name="new_tool" label="${tool_name}_tool.gz">
-        <filter>factory['make_Tool'] == "yes"</filter>
+        <filter>make_Tool == "yes"</filter>
     </data>
   </outputs>
 <configfiles>