truststore_browse_testing: TrustStoreGalaxyBrowse.py annotate

annotate TrustStoreGalaxyBrowse.py @ 45:31838e0ecf45 default tip

Fixes.

author	Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au>
date	Fri, 26 Jun 2015 10:24:40 +1000
parents	174df9815ecd
children

rev	line source
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	1 """TrustStore downloaded for Galaxy."""
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	2 from __future__ import division, absolute_import, print_function, unicode_literals
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	3 import sys
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	4 import shutil
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	5 import gzip
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	6 import tempfile
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	7 import os
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	8 import json
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	9 import operator
15 7b3039ccf50f Un-quote url. Catherine Wise <catherine.wise@csiro.au> parents: 14 diff changeset	10 import urlparse
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	11 from py_ts import TrustStoreClient, utils
43 174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	12 # import galaxy.model # need to import model before sniff to resolve a circular import dependency
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	13 from galaxy.datatypes.checkers import util
43 174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	14 # from galaxy.datatypes import sniff
174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	15 # from galaxy.datatypes.registry import Registry
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	16
16 10ce45af14dd Use pyopenssl Catherine Wise <catherine.wise@csiro.au> parents: 15 diff changeset	17 # Tell urllib3 to use pyOpenSSL because we are on old Python stdlib.
24 f4585bf75a8a Give up, try to update Python instead. Catherine Wise <catherine.wise@csiro.au> parents: 16 diff changeset	18 # import urllib3.contrib.pyopenssl
f4585bf75a8a Give up, try to update Python instead. Catherine Wise <catherine.wise@csiro.au> parents: 16 diff changeset	19 # urllib3.contrib.pyopenssl.inject_into_urllib3()
25 18fe446f2d02 Tell requests about the new cert bundle. FFS. Catherine Wise <catherine.wise@csiro.au> parents: 24 diff changeset	20 #
26 463b024b53ff The path was wrong. UUUUGH. Catherine Wise <catherine.wise@csiro.au> parents: 25 diff changeset	21 os.environ["REQUESTS_CA_BUNDLE"] = "/etc/ssl/certs/ca-certificates.crt"
16 10ce45af14dd Use pyopenssl Catherine Wise <catherine.wise@csiro.au> parents: 15 diff changeset	22
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	23 CLIENT_KEY = "desktop"
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	24 CLIENT_SECRET = "cpU92F1PT7VOCANjSknuCDp4DrubmujoBaF6b0miz8OpKNokEbGMHCaSFK5/lISbBmaaGVCgeADI2A39F3Hkeg=="
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	25 CHUNK_SIZE = 2**20 # 1Mb
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	26 SAFE_CHARS = '.-()[]0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ '
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	27
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	28 def print_nice(elem, f, depth):
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	29 """Print the file name."""
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	30 try:
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	31 f.write('\t'*depth + elem.name + " (" + str(len(elem.fragments)) + " parts)\n")
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	32 except AttributeError:
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	33 f.write('\t'*depth + elem.name + "\n")
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	34 for child in elem.children:
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	35 print_nice(child, f, depth+1)
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	36
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	37 def check_gzip(file_path):
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	38 """Check if file is gziped."""
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	39 try:
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	40 temp = open(file_path, "U")
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	41 magic_check = temp.read(2)
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	42 temp.close()
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	43 if magic_check != util.gzip_magic:
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	44 return (False, False)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	45 except Exception:
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	46 return (False, False)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	47 return (True, True)
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	48
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	49 def ungzip(download, outputFile):
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	50 """Uncompress file."""
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	51 is_gzipped, is_valid = check_gzip(download)
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	52
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	53 if is_gzipped and not is_valid:
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	54 print("File is compressed (gzip) but not valid.")
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	55 sys.exit(4)
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	56 elif is_gzipped and is_valid:
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	57 # We need to uncompress the temp_name file, but BAM files must
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	58 # remain compressed in the BGZF format
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	59 file_handle, uncompressed = tempfile.mkstemp(prefix='data_id_upload_gunzip_', dir=os.path.dirname(outputFile), text=False )
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	60 gzipped_file = gzip.GzipFile(download, 'rb')
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	61 while 1:
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	62 try:
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	63 chunk = gzipped_file.read(CHUNK_SIZE)
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	64 except IOError:
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	65 os.close(file_handle)
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	66 os.remove(uncompressed)
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	67 print('Problem decompressing gzipped data %s %s' % (download, outputFile))
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	68 sys.exit(4)
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	69 if not chunk:
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	70 break
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	71 os.write(file_handle, chunk)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	72 os.close(file_handle)
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	73 gzipped_file.close()
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	74
45 31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	75 try:
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	76 shutil.copy(uncompressed, outputFile)
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	77 except shutil.Error:
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	78 pass # If the file is already in the right location, move along.
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	79 try:
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	80 os.remove(uncompressed)
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	81 os.remove(download)
2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	82 except OSError:
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	83 pass
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	84 else:
45 31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	85 try:
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	86 shutil.copy(download, outputFile)
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	87 except shutil.Error:
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	88 pass # If the file is already in the right location, move along.
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	89
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	90 def construct_multi_filename(id, name, file_type):
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	91 """ Implementation of *Number of Output datasets cannot be determined until
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	92 tool run* from documentation_.
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	93 .. _documentation: http://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	94 From https://github.com/mdshw5/galaxy-json-data-source/blob/master/json_data_source.py
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	95 """
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	96 filename = "%s_%s_%s_%s_%s" % ('primary', id, name, 'visible', file_type)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	97 return filename
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	98
45 31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	99 def metadata_to_json(dataset_id, filename, name, extension, ds_type='dataset', primary=False):
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	100 """ Return line separated JSON
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	101 From https://github.com/mdshw5/galaxy-json-data-source/blob/master/json_data_source.py
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	102 """
45 31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	103 ext = extension
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	104 if ext == 'fa':
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	105 ext = 'fasta'
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	106 meta_dict = dict(type=ds_type,
45 31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	107 ext=ext,
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	108 filename=filename,
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	109 name=name,
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	110 metadata={})
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	111 if primary:
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	112 meta_dict['base_dataset_id'] = dataset_id
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	113 else:
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	114 meta_dict['dataset_id'] = dataset_id
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	115 return "%s\n" % json.dumps(meta_dict)
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	116
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	117 def main():
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	118 properties_file = sys.argv[1]
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	119 json_params = None
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	120 metadata_path = None
14 589f9250e30f Fix son reading. Catherine Wise <catherine.wise@csiro.au> parents: 13 diff changeset	121 all_params = None
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	122 with open(properties_file, 'r') as file_:
13 11344d545e88 Debug Catherine Wise <catherine.wise@csiro.au> parents: 10 diff changeset	123 settings = file_.read()
11344d545e88 Debug Catherine Wise <catherine.wise@csiro.au> parents: 10 diff changeset	124 all_params = json.loads(settings)
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	125 json_params = all_params.get("param_dict")
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	126 metadata_path = all_params["job_config"]["TOOL_PROVIDED_JOB_METADATA_FILE"]
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	127
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	128 output_filename = json_params.get('output', None)
14 589f9250e30f Fix son reading. Catherine Wise <catherine.wise@csiro.au> parents: 13 diff changeset	129 output_data = all_params.get('output_data')
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	130 extra_files_path, file_name, ext, out_data_name, hda_id, dataset_id = \
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	131 operator.itemgetter('extra_files_path', 'file_name', 'ext', 'out_data_name', 'hda_id', 'dataset_id')(output_data[0])
40 5ea7590f7395 Try different path for extra files. Catherine Wise <catherine.wise@csiro.au> parents: 39 diff changeset	132 extra_files_path = json_params['__new_file_path__']
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	133
43 174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	134 # datatypes_registry = Registry()
174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	135 # datatypes_registry.load_datatypes(
174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	136 # root_dir=all_params['job_config']['GALAXY_ROOT_DIR'],
174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	137 # config=all_params['job_config']['GALAXY_DATATYPES_CONF_FILE']
174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	138 # )
174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	139
15 7b3039ccf50f Un-quote url. Catherine Wise <catherine.wise@csiro.au> parents: 14 diff changeset	140 url_params = urlparse.unquote(json_params['URL']).split(";")
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	141 if len(url_params) < 3:
15 7b3039ccf50f Un-quote url. Catherine Wise <catherine.wise@csiro.au> parents: 14 diff changeset	142 print("The url we got back is malformed: "+ json_params['URL'])
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	143 sys.exit(5)
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	144 short_url = url_params[0]
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	145 username = url_params[1]
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	146 password = url_params[2]
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	147 if "/short" not in short_url:
15 7b3039ccf50f Un-quote url. Catherine Wise <catherine.wise@csiro.au> parents: 14 diff changeset	148 print("The url we got back is malformed: " + json_params['URL'])
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	149 sys.exit(5)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	150 kms_url = short_url.split("/short")[0]
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	151
29 157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	152 tmp_dir = '/mnt/galaxy/tmp'
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	153 tmp = None
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	154 if os.path.exists(tmp_dir):
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	155 tmp = tmp_dir
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	156
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	157 config = TrustStoreClient.Config(
29 157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	158 None, kms_url, CLIENT_KEY, CLIENT_SECRET, tmpDir=tmp)
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	159 truststore = TrustStoreClient.TrustStoreClient(headless=False, config=config)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	160 try:
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	161 truststore.authenticate(username, password)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	162 except TrustStoreClient.TrustStoreClientAuthenticationException as err:
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	163 print(err)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	164 sys.exit(5)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	165 truststore.getPrivateKey('privkey.pem')
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	166
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	167 path_texts = truststore.lengthenPath(short_url)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	168 if len(path_texts) < 2:
27 901b359bb74a We are getting closer. Catherine Wise <catherine.wise@csiro.au> parents: 26 diff changeset	169 print("The path we got was malformed: " + str(path_texts))
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	170 sys.exit(3)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	171 paths = path_texts[1:]
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	172 store_id = path_texts[0]
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	173
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	174 store = truststore.getStoreByID(store_id)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	175 if store is None:
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	176 print("Coudn't find store with that ID, or don't have access.")
0 2a996fea49a3 Initial. Catherine Wise <catherine.wise@csiro.au> parents: diff changeset	177 sys.exit(2)
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	178 root = truststore.listDirectory(store)
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	179
14 589f9250e30f Fix son reading. Catherine Wise <catherine.wise@csiro.au> parents: 13 diff changeset	180 first = True
589f9250e30f Fix son reading. Catherine Wise <catherine.wise@csiro.au> parents: 13 diff changeset	181
32 20c1d4fa4084 More verbose. Catherine Wise <catherine.wise@csiro.au> parents: 31 diff changeset	182 print("Preparing the following for downloading: " + str(paths))
20c1d4fa4084 More verbose. Catherine Wise <catherine.wise@csiro.au> parents: 31 diff changeset	183
43 174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	184 # Empty parameter file.
174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	185 open(file_name, 'w' ).close()
174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	186
29 157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	187 if root is not None:
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	188 with open(metadata_path, 'wb') as metadata_file:
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	189 for path in paths:
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	190 locations = utils.Navigation.files_at_path(root, path)
33 c404b570013b More verbose 2. Catherine Wise <catherine.wise@csiro.au> parents: 32 diff changeset	191 if not locations or locations == []:
c404b570013b More verbose 2. Catherine Wise <catherine.wise@csiro.au> parents: 32 diff changeset	192 print("Path not found: " + path)
c404b570013b More verbose 2. Catherine Wise <catherine.wise@csiro.au> parents: 32 diff changeset	193 print("In root: " + str(root))
c404b570013b More verbose 2. Catherine Wise <catherine.wise@csiro.au> parents: 32 diff changeset	194 else:
35 d25b82c3276f Print a little less :) mostly working now!!! Catherine Wise <catherine.wise@csiro.au> parents: 33 diff changeset	195 print("Downloading file..." + ", ".join([loc.name for loc in locations]))
29 157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	196 for location in locations:
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	197 filename = "".join(c in SAFE_CHARS and c or '-' for c in location.name)
39 624d8102cb75 Fix json output again. Catherine Wise <catherine.wise@csiro.au> parents: 38 diff changeset	198 extension = os.path.splitext(filename)[1].strip(".")
29 157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	199 name = construct_multi_filename(hda_id, filename, extension)
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	200 target_output_filename = None
39 624d8102cb75 Fix json output again. Catherine Wise <catherine.wise@csiro.au> parents: 38 diff changeset	201 data_type = "new_primary_dataset"
43 174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	202 target_output_filename = os.path.normpath(os.path.join(extra_files_path, name))
29 157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	203 download = truststore.getFile(store, location)
45 31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	204 primary = not first
29 157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	205 if download is None:
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	206 print("File %s not found." % location.name)
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	207 sys.exit(4)
45 31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	208 if first:
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	209 first = False
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	210 target_output_filename = file_name
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	211 data_type = "dataset"
29 157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	212 ungzip(download, target_output_filename)
45 31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	213 metadata_file.write(
31838e0ecf45 Fixes. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 43 diff changeset	214 metadata_to_json(dataset_id, target_output_filename, name, extension, data_type, primary=primary))
43 174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	215 # ext = sniff.handle_uploaded_dataset_file(target_output_filename, datatypes_registry, ext=ext)
174df9815ecd Fix data types. Wise, Catherine (Digital, Acton) <Catherine.Wise@csiro.au> parents: 40 diff changeset	216 # print("Guessed file type: " + ext)
29 157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	217 else:
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	218 print("Store is damaged or we don't have sufficient access.")
157062f5c547 Fix temporary dir. Catherine Wise <catherine.wise@csiro.au> parents: 27 diff changeset	219 sys.exit(4)
9 0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	220
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	221
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	222 if __name__ == '__main__':
0f2a5664c9eb Actually update downloader (probably not working). Catherine Wise <catherine.wise@csiro.au> parents: 0 diff changeset	223 main()

Mercurial > repos > cathywise > truststore_browse_testing

annotate TrustStoreGalaxyBrowse.py @ 45:31838e0ecf45 default tip