Fileglancer Apps

[krokicki]

This PR adds Apps which are CLI tools that can be executed on the cluster from a web form, and monitored within Fileglancer.

There are two new top level pages: Apps and Jobs. The Apps page allows you to add apps via a GitHub URL that conforms to a new "Runnables Protocol" that I created based on inspiration from Nexflow Schema, CWL, IPP, and Fractal. In short, each app repo contains a runnables.yaml manifest that declares one or more entry points, and defines their parameters, resource defaults, and environment variables. A single repo can contain multiple manifests in subdirectories, each discovered and registered as a separate app. Manifests can also reference a separate repo_url for the actual tool code. This will allow us to create a "catalog" repo of external tools in cases their tool's repo can't host the runnables.yaml.

Currently, the prerequisites support Pixi, Maven, and NPM, and only Pixi has been tested. I'd like to expand this to also include Conda and containers. Allowing uncontainerized tools like Pixi is a bit of a risk from a complexity standpoint, and also won't work for users who don't have these prereqs set up. I'd like to keep thinking about the best solution to this, such as offering solutions for automatically installing prereqs, pre-installing all prerequisite platforms in a shared location during Fileglancer server setup, exploring solutions like album, etc.

App URLs are stored in your preferences and the repos are cloned and cached in ~/.fileglancer/apps. When you launch an entry point, the manifest is used to auto-generate a web form with appropriate controls for each parameter type. Parameters can be organized into collapsible sections. File and directory parameters use the file selection dialog.

Jobs are submitted to the cluster via a new py-cluster-api package (inspired by dask-jobqueue and our java-lsf), which initially supports local and LSF executors. We can easily add others (e.g. Slurm) as necessary for non-Janelia users. The cluster executor type and default resources are configured in Fileglancer's config.yaml under a new cluster section. See docs/config.yaml.template for the possible options.

The Jobs page lists all submitted jobs and updates periodically. Clicking on a Job brings you to a Job Detail page that shows four tabs: parameters, the generated script, stdout, and stderr. Jobs can be cancelled, deleted, or relaunched with the same (or modified) parameters.

On server restart, the job monitor automatically reconnects to any active cluster jobs and resumes polling, as long as the job_name_prefix is set (otherwise this is determined randomly each time, similar to the session_secret_key). A reconciliation loop detects zombie jobs and syncs status transitions from the executor to the database.

Services

Services are cluster jobs that are started and stopped by the user as needed. They can be web servers, notebooks, APIs, etc.

• Entry points can declare type: service in runnables.yaml. These run until explicitly stopped rather than running to completion.
• Fileglancer exports SERVICE_URL_PATH as an env var in the job script. Services write their URL to a new file at SERVICE_URL_PATH. Fileglancer polls this file and when the URL is written it gets displayed in the UI for the user to access.

Trying it out

You can paste in these app URLs:

• Demo app: https://github.com/JaneliaSciComp/fileglancer-demo-app
• MRC viewer app: https://github.com/JaneliaSciComp/neuroglancer-mrc
• bioformats2raw: https://github.com/JaneliaSciComp/nf-omezarr (branch fileglancer)
• Cellmap Flow: https://github.com/janelia-cellmap/cellmap-flow (branch fileglancer)

@StephanPreibisch @JaneliaSciComp/fileglancer @dchen116 @mzouink

[allison-truhlar]

I will open a branch to work on some possible UI/styling changes.

General questions:

• Is this meant primarily for usage in the CLI app, or also for deployment to fileglancer.int.janelia.org? If the latter,

  • Do we want to add an option for preconfigured Apps? So that when we launch Fileglancer on production, we can add in a "starter pack" of apps that are commonly used or we want to promote the use of?
  • This is in part a how-does-the-cluster work question - how is it determined what project to charge a job to? By username? Do we want to make it possible for the user to configure what account to charge a job to?

• Should we get rid of the /jobs page (points to Tasks) and the associated convert tab and dialog?

• I tried adding a GitHub repo that didn't have a valid runnables.yaml file - I think the error that was returned could be improved and I was also wondering if we want to remove a cloned repo if it's found not to have a valid runnables.yaml file? Currently it's left on disk.

[allison-truhlar]

I think this description is meant to be on the AppsJobs.tsx page. But it also brings up the greater question for me of - do we even want the original Jobs.tsx page at all anymore (it's called Tasks in the navbar)?

[krokicki]

Nice catch! I think we can keep Tasks for now, they are sort of complementary (manual vs automated), but I agree that we will need to revisit when the automation gets good enough.

[krokicki]

I will open a branch to work on some possible UI/styling changes.

Yes, please! The UI here is just a placeholder. But I would recommend holding off on this for now, until we can discuss this functionality as a team and finalize the requirements.

General questions:

Is this meant primarily for usage in the CLI app, or also for deployment to fileglancer.int.janelia.org? If the latter,

This is intended for deployment on fileglancer.int.janelia.org, where apps will run on the cluster. But you can also use it locally with the local executor.

Do we want to add an option for preconfigured Apps? So that when we launch Fileglancer on production, we can add in a "starter pack" of apps that are commonly used or we want to promote the use of?

This is a fantastic idea. Let's discuss more in person.

This is in part a how-does-the-cluster work question - how is it determined what project to charge a job to? By username? Do we want to make it possible for the user to configure what account to charge a job to?

Yes, and I just added that feature this morning.

Should we get rid of the /jobs page (points to Tasks) and the associated convert tab and dialog?

Not yet, but maybe in the future, I agree we should think closely about this.

I tried adding a GitHub repo that didn't have a valid runnables.yaml file - I think the error that was returned could be improved and I was also wondering if we want to remove a cloned repo if it's found not to have a valid runnables.yaml file? Currently it's left on disk.

Once we nail down the base functionality (and merge the services PR into this one), we'll definitely need to do some passes focused on UX, error handling, etc.