Mercurial > repos > tduigou > save_to_db

annotate save_to_db.py @ 3:eca0d710bbe9 draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
author tduigou
date Fri, 23 May 2025 08:51:50 +0000
parents 034686b5bc15
children c7a7520afb4b
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
1 import subprocess
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
2 import time
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
3 import argparse
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
4 import socket
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
5 import os
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
6 import re
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
7 import pandas as pd
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
8 from sqlalchemy import create_engine, inspect
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
9 from sqlalchemy.sql import text
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
10 from sqlalchemy.engine.url import make_url
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
11 from sqlalchemy.exc import OperationalError
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
12
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
13 def fix_db_uri(uri):
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
14 """Replace __at__ with @ in the URI if needed."""
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
15 return uri.replace("__at__", "@")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
16
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
17 def is_port_in_use(port):
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
18 """Check if a TCP port is already in use on localhost."""
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
19 with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
20 return s.connect_ex(('localhost', port)) == 0
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
21
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
22 def extract_db_name(uri):
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
23 """Extract the database name from the SQLAlchemy URI."""
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
24 url = make_url(uri)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
25 return url.database
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
26
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
27 def start_postgres_container(db_name):
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
28 """Start a PostgreSQL container with the given database name as the container name."""
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
29 container_name = db_name
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
30
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
31 # Check if container is already running
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
32 container_running = subprocess.run(
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
33 f"docker ps -q -f name={container_name}", shell=True, capture_output=True, text=True
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
34 )
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
35
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
36 if container_running.stdout.strip():
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
37 print(f"Container '{container_name}' is already running.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
38 return
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
39
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
40 # Check if container exists (stopped)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
41 container_exists = subprocess.run(
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
42 f"docker ps -a -q -f name={container_name}", shell=True, capture_output=True, text=True
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
43 )
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
44
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
45 if container_exists.stdout.strip():
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
46 print(f"Starting existing container '{container_name}'...")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
47 subprocess.run(f"docker start {container_name}", shell=True)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
48 print(f"PostgreSQL Docker container '{container_name}' activated.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
49 return
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
50
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
51 # If container does not exist, create and start a new one
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
52 port = 5432 if not is_port_in_use(5432) else 5433
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
53 postgres_password = os.getenv("POSTGRES_PASSWORD", "RK17")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
54
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
55 start_command = [
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
56 "docker", "run", "--name", container_name,
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
57 "-e", f"POSTGRES_PASSWORD={postgres_password}",
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
58 "-p", f"{port}:5432",
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
59 "-d", "postgres"
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
60 ]
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
61
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
62 try:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
63 subprocess.run(start_command, check=True)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
64 print(f"PostgreSQL Docker container '{container_name}' started on port {port}.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
65 except subprocess.CalledProcessError as e:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
66 print(f"Failed to start Docker container: {e}")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
67
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
68 def wait_for_db(uri, timeout=60):
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
69 """Try connecting to the DB until it works or timeout."""
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
70 engine = create_engine(uri)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
71 start_time = time.time()
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
72 while time.time() - start_time < timeout:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
73 try:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
74 with engine.connect():
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
75 print("Connected to database.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
76 return
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
77 except OperationalError:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
78 print("Database not ready, retrying...")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
79 time.sleep(2)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
80 raise Exception("Database connection failed after timeout.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
81
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
82 def push_gb_annotations(gb_files, sequence_column, annotation_column, db_uri, table_name, fragment_column_name, output, file_name_mapping):
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
83 """Push GenBank file content into the database if the fragment is not already present."""
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
84 db_uri = fix_db_uri(db_uri)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
85 engine = create_engine(db_uri)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
86 inserted_fragments = []
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
87
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
88 try:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
89 # Parse the file_name_mapping string into a dictionary {base_file_name: fragment_name}
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
90 file_name_mapping_dict = {
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
91 os.path.basename(path): os.path.splitext(fragment_name)[0]
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
92 for mapping in file_name_mapping.split(",")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
93 for path, fragment_name in [mapping.split(":")]
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
94 }
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
95
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
96 #print("File name mapping dictionary:")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
97 #print(file_name_mapping_dict) # Debugging: Print the mapping dictionary
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
98
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
99 with engine.begin() as connection:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
100 inspector = inspect(engine)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
101 columns = [col['name'] for col in inspector.get_columns(table_name)]
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
102
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
103 if fragment_column_name not in columns:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
104 raise ValueError(f"Fragment column '{fragment_column_name}' not found in table '{table_name}'.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
105
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
106 # Get existing fragments
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
107 all_rows = connection.execute(text(f"SELECT {fragment_column_name} FROM {table_name}")).fetchall()
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
108 existing_fragments = {row[0] for row in all_rows}
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
109
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
110 insert_rows = []
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
111
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
112 for gb_file in gb_files:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
113 # Extract base file name (just the file name, not the full path)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
114 real_file_name = os.path.basename(gb_file)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
115
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
116 print(f"Processing file: {real_file_name}") # Debugging: Log the current file
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
117
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
118 # Get the corresponding fragment name from the mapping
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
119 fragment_name = file_name_mapping_dict.get(real_file_name)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
120
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
121 if not fragment_name:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
122 raise ValueError(f"Fragment name not found for file '{real_file_name}' in file_name_mapping.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
123
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
124 # If the fragment is already in the DB, raise an error and stop the process
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
125 if fragment_name in existing_fragments:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
126 raise RuntimeError(f"Fatal Error: Fragment '{fragment_name}' already exists in DB. Stopping the process.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
127
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
128 with open(gb_file, "r") as f:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
129 content = f.read()
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
130
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
131 origin_match = re.search(r"^ORIGIN.*$", content, flags=re.MULTILINE)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
132 if not origin_match:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
133 raise ValueError(f"ORIGIN section not found in file: {gb_file}")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
134
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
135 origin_start = origin_match.start()
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
136 annotation_text = content[:origin_start].strip()
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
137 sequence_text = content[origin_start:].strip()
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
138
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
139 values = {}
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
140 values[fragment_column_name] = fragment_name
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
141 values[annotation_column] = annotation_text
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
142 values[sequence_column] = sequence_text
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
143
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
144 insert_rows.append(values)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
145 inserted_fragments.append(fragment_name)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
146
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
147 # Insert the rows into the database
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
148 for values in insert_rows:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
149 col_names = ", ".join(values.keys())
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
150 placeholders = ", ".join([f":{key}" for key in values.keys()])
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
151 insert_stmt = text(f"INSERT INTO {table_name} ({col_names}) VALUES ({placeholders})")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
152
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
153 #print(f"Inserting into DB: {values}") # Debugging print statement
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
154 result = connection.execute(insert_stmt, values)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
155
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
156 #print(f"Insert result: {result.rowcount if hasattr(result, 'rowcount') else 'N/A'}") # Debugging the row count
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
157
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
158 print(f"Inserted {len(insert_rows)} fragments.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
159
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
160 # Write inserted fragment names to a text file
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
161 with open(output, "w") as log_file:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
162 for frag in inserted_fragments:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
163 log_file.write(f"{frag}\n")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
164 print(f"Fragment names written to '{output}'.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
165
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
166 except Exception as e:
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
167 print(f"Error during GB file insertion: {e}")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
168 raise
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
169
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
170 def main():
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
171 parser = argparse.ArgumentParser(description="Fetch annotations from PostgreSQL database and save as JSON.")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
172 parser.add_argument("--input", required=True, help="Input gb files")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
173 parser.add_argument("--sequence_column", required=True, help="DB column contains sequence for ganbank file")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
174 parser.add_argument("--annotation_column", required=True, help="DB column contains head for ganbank file")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
175 parser.add_argument("--db_uri", required=True, help="Database URI connection string")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
176 parser.add_argument("--table", required=True, help="Table name in the database")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
177 parser.add_argument("--fragment_column", required=True, help="Fragment column name in the database")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
178 parser.add_argument("--output", required=True, help="Text report")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
179 parser.add_argument("--file_name_mapping", required=True, help="real fragments names")
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
180 args = parser.parse_args()
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
181
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
182 # Start the Docker container (if not already running)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
183 gb_file_list = [f.strip() for f in args.input.split(",") if f.strip()]
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
184 db_uri = fix_db_uri(args.db_uri)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
185 db_name = extract_db_name(db_uri)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
186 start_postgres_container(db_name)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
187
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
188 # Wait until the database is ready
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
189 wait_for_db(db_uri)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
190
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
191 # Fetch annotations from the database and save as gb
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
192 push_gb_annotations(gb_file_list, args.sequence_column, args.annotation_column, db_uri, args.table, args.fragment_column, args.output, args.file_name_mapping)
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
193
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
194 if __name__ == "__main__":
034686b5bc15 planemo upload for repository https://github.com/brsynth commit 6ae809b563b40bcdb6be2e74fe2a84ddad5484ae
tduigou
parents:
diff changeset
195 main()