HPC Servers
Because Stage 1 pipelines entail handling of large files and computationally intensive data analysis, in general they are executed on a remote, Linux-based, High Performance Computing (HPC) server cluster. As a result, it is often best to develop pipeline code within such a server environment.
We will also see that MDI web servers can run on a remote HPC server to allow Stage 2 apps to easily access the data files generated by Stage 1 pipelines.
This Basic Training section will get you up to speed working on HPC servers and the Linux command line.