GenePattern provides access to a broad array of computational methods used to analyze genomic data. Its extendable architecture makes it easy for computational biologists to add analysis and visualization modules, which ensures that GenePattern users have access to new computational methods on a regular basis.
If you are new to GenePattern, begin with the basics:
This guide assumes that you are a programmer and familiar with GenePattern. As a programmer, you generally work with GenePattern in one of two ways:
Only the GenePattern team can create, edit, or install modules on the GenePattern public server. Therefore, to create a module, you must have a local GenePattern server installed (see Starting Your Own GenePattern Server).
Creating a GenePattern module is a multi-step process:
Malicious code: By adding a module, a user can execute arbitrary code on the GenePattern server. Because arbitrary code may include malicious code, take precautions to protect your server: for example, employ virus scanner software and restrict access to appropriately privileged (non-root) users. For more information about securing your server, see Securing the Server.
To create a module in GenePattern:
If no errors are found, GenePattern copies the support files to the server and makes the module available to the GenePattern clients.
The in-depth article Creating a GenePattern Module provides a hands-on tutorial that walks you through the process of creating a GenePattern module.
Modules can have end-user license agreements (EULAs) attached to them. In order to run a licensed module (either standalone or within a pipeline), users must view and accept the license terms; GenePattern will display the license and prompt the user to accept it. After a user accepts a licensed module's terms, that user can run the module and GenePattern will not prompt the user to re-accept the license on subsequent runs. However, a user will need to re-accept the license whenever the licensed module is updated to a new version. The GenePattern Server records each acceptance of a licensed module's terms and reports these records to a database application hosted at the Broad Institute. If this module will only be on your GenePattern server and you need to maintain your own database for the terms agreements, contact the GenePattern development team for assistance.
Academic users can agree to the license and use the module. Commercial users, however, need to contact the GenePattern team to work out terms for use, particularly if commercial users are establishing their own GenePattern server on which to host the licensed module.
The EULA terms need to be in a text document (.txt). You can add a EULA to your module in the module integrator by clicking Add License File.
To edit a module:
When you create or edit a module, GenePattern displays its properties in the module integrator. Click the Help icon to display descriptions of each property and its valid values. The help text is provided here for your convenience.
When you create or edit a module, GenePattern displays its properties in the module integrator. Click the Help icon to display the following descriptions of each property and its valid values.
Note: Only the GenePattern team can create, edit or install modules on the GenePattern public server. Therefore, to create a module, you must have a local GenePattern server installed.
Creating a GenePattern module is a multi-step process:
When you save your changes, the module properties that you have entered are validated as follows:
If everything checks out, the uploaded files are saved in the GenePattern module library and the module registered in the module database. The module and its uploaded files are indexed in the background so that they are available for searching. You can run the module immediately and can share it with others.
The following sections describe each module property in detail:
An example for each property is given based on the Consensus Clustering module, which may be found in the module repository.
The name of the module will be used in the drop-down module catalog lists and as a directory name on the server with this name. It should be a short but descriptive name, without spaces or punctuation, and may be mixed upper- and lower-case.
ConsensusClustering example: ConsensusClustering
Each time you update a module, you create a new version of the module. Typically, you want to edit the most recent version of a module. If you want to edit an earlier version, select that version from the drop-down list of versions.
Click Help to display this text.
Click Save to save your changes, creating a new version of the module, and remain in the module integrator.
Click Save and Run to save your changes, creating a new version of the module, exit from the module integrator and run the module.
The Life Science Identifier (LSID) used to uniquely identify a GenePattern module. You cannot create or edit LSIDS. They are created automatically by the GenePattern server when a module is saved.
ConsensusClustering example: urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00030:5
The description is where to explain what your module does, and why someone would want to use it. It can be anywhere from a sentence to a short paragraph in length. The description, sometimes in abridged form, is displayed in the pipeline designer module choice list, in generated code when creating scripts from pipelines, and in the web client. It's a very good way for you to document succinctly why your module exists.
ConsensusClustering example: Resampling-based clustering method
Enter the author's name. If you share this module with others, they will know how to give the author credit and whom to contact with questions, suggestions, or enhancement ideas.
ConsensusClustering example: Stefano Monti
Enter the author's affiliation (company or academic institution). If you share this module with others, they will know how to give the author credit and whom to contact with questions, suggestions, or enhancement ideas.
ConsensusClustering example: Broad Institute
Upload a text file containing the End-User license agreement. Users will be prompted to accept this license before running the module.
Enter a brief description of the changes that you have made to the module. When GenePattern clients display a drop-down list of versions, the comments for each version are visible in the drop-down list.
ConsensusClustering example: Added ability to create heatmap images of clusters
On the GenePattern home page, modules and pipelines are organized by categories. Pipelines are always assigned to the category name pipeline. When you create/update a module, you can choose an existing category name or create a new category name. If your module fits into an existing category, such as Preprocess & Utilities, select that category from the drop-down list; otherwise, click the New button to add a new category. GenePattern creates the drop-down list of categories dynamically based on the categories of the modules installed on your GenePattern server. If you delete the last module in a given category, that category is removed from the drop-down list. ConsensusClustering example: Clustering
Modules may be marked as either public or private. When a module is first created, the default is to mark it private. When a module is first created, the default is to mark it private.
ConsensusClustering example: public
The quality level is a simple three-level classification that lets the user know what level of confidence the author has in the robustness of the module. In increasing order of quality expectations, they are: are "development", "preproduction", and "production". Although these terms have no strict definitions, they are useful for setting user expectations. If you make this module public, set the quality level appropriately.
ConsensusClustering example: production
If your module is compiled for a specific platform (Intel, Alpha, PowerPC, etc.), indicate that here. CPU requirements are enforced when the module is run.
ConsensusClustering example: any
If your module requires a specific operating system (Windows, Linux, MacOS, etc.), indicate that here. Operating system requirements are enforced when the module is run.
ConsensusClustering example: any
There is no specific language support or requirement enforcement at this time. However, by describing the primary language that a module is implemented in, you give some hints to the prospective user about their system requirements.
ConsensusClustering example: Java
If your module requires at least a certain revision of the language runtime environment(eg. 1.3.1_07), indicate that here. This is not currently enforced, but provides useful information to the prospective module user.
ConsensusClustering example: none specified
Each GenePattern job runs in a docker container. Each module must declare a docker image by setting the job.docker.image property in the manifest file. This is passed as the arg to the docker run command. For production modules this should be a tagged version of an image from a public repository.
Docker image specification format: IMAGE[:TAG|@DIGEST]
e.g. genepattern/docker-java17:0.12
When a module is run, the command line will be passed into the container via a "docker run" command. This will be similar to this pattern for running the program on its own (without GenePattern) in docker;
$ docker run [OPTIONS] <job.docker.image> <commandLine>
Note: you shoule NOT include "docker run [OPTIONS]" or the docker image in the module command line. The GenePattern server will format the docker run command and options.
Select the file formats of the output files generated by your module. If your module generates an output file format not included in the list, click New to add that format to the list.
Note on support file obsolescence: This section on Support Files reflects older (pre-docker) versions of GenePattern. This sections behavior is still supported (as of 3.9.11b377) to support older modules, but the preferred method is now to have all support files provided pre-installed in the docker container.
Any files required by your module, such as scripts, libraries, property files, DLLs, executable programs, etc. must be uploaded to the server. These files may be referenced in the command line field using the <libdir>filename nomenclature. There is no upper limit on the number of files which may be uploaded, assuming there is enough space.
Files that have been uploaded appear as links in this section. You may view or download them by clicking appropriately in your browser.
Help Files: Public modules should always include a help file that provides instructions for using the module, a detailed description of each input parameter, a detailed description of each output file (both its format and content), and either an explanation of the algorithm or a reference to the paper, journal or book that explains it.
When a user selects your module, GenePattern displays a form that includes the module parameters and a Help button. When the user clicks the Help button, GenePattern examines the list of support files for the module and displays the first file that has a standard documentation file extension. If no documentation file was provided, GenePattern displays a message indicating that no information is available. (By default, the standard documentation file extensions are html, htm, xhtml, pdf, rtf, and txt. You can modify this list of extensions by editing the files.doc property in the GenePattern /resources/genepattern.properties file.)
ConsensusClustering example: Current files: Acme.jar archiver.jar common_cmdline.jar ConsensusClustering.pdf file_support.jar geneweaver.jar gp-common.jar ineq_0.2-2.tar.gz ineq_0.2-2.tgz jaxb-rt-1.0-ea.jar my.local.install.r RunSomAlg.jar trove.jar version.txt
The crux of adding a module to the GenePattern server is to provide the command line that will be used to launch the module, including substitutions for settings that will be specified differently for each invocation. In the command line field, you will provide a combination of the fixed text and the dynamically-changed text which together constitute the command line for an invocation of the module.
Perhaps the trickiest thing about specifying a command line is making it truly platform-independent. To accomplish this we now use Docker container images to encapsulate the runtime environment for a module. Older versions of GenePattern would dynamically modify the command line for the server they are on.
Parameters: Parameters that require substitution should be enclosed in brackets (ie. <filename>). Every parameter listed in the parameters section must be mentioned in the command line unless its optional field is checked. A default value may be provided and will be used if the user fails to specify a value when invoking the module.
Click the View Argument List button to display a list of the parameters mentioned in the command line. You can change the order of the parameters by dragging them to a new position in the list or by editing the text of the command line.
Substitution properties: In addition to parameter names, you may also use environment variables, Java system properties, and any properties defined in the %GenePatternInstallDir%/resources/genepattern.properties file. In particular, there are predefined values for <java>, <perl>, and <R>, three languages that are used within various modules that may be downloaded from the module catalog at the public GenePattern website. Useful substitution properties include:
| <java> | path to Java, the same one running the GenePattern server. For Docker images java is assumed to be on the path within the container and will substitute as just "java". |
| <perl> | path to Perl, installed with GenePattern server on Windows, otherwise the one already installed on your system. For Docker images perl is assumed to be on the path within the container and will substitute as just "perl". |
| <R> | path to a program that runs R and takes as input a script of R commands. R is installed with GenePatternserver on Windows and MacOS. For Docker images R is assumed to be on the path within the container and will substitute as just "Rscript". |
| <java_flags> | memory size and other Java JVM settings from the GenePattern/resources/genepattern.properties file |
| <libdir> | directory where the module's support files are stored |
| <job_id> | job number |
| <name> | name of the module being run |
| <filename_basename> | for each input file parameter, the filename without directory |
| <filename_extension> | for each input file parameter, the extension without filename or directory |
| <filename_file> | for each input file parameter, the input filename without directory |
| <path.separator> | Java classpath delimiters (: or ;), useful for specifying a classpath for Java-based modules |
| <file.separator> | / or \ for directory delimiter |
| <line.separator> | newline, carriage return, or both for line endings |
| <user.dir> | current directory where the job is executing |
| <user.home> | user's home directory |
Rather than having to customize your module's command line for the exact location of the language runtime on each computer, you can use the substitution properties. For example,
<java> -cp <libdir>mymodule.jar com.foo.MyModule <arg1>
GenePattern will then take care of locating the Java runtime, asking it to begin execution at the MyModule class using code from the uploaded file mymodule.jar.
Standard input/output: If your module is designed to accept a standard input stream and/or write to a standard output stream, you can use redirection syntax when describing the command line. To redirect a file to the input stream, enter the text \< followed by the input file parameter. To redirect the standard output or standard error streams to a named file, enter the text \> or \\>& followed by the name of the output file. In the following example, the LogTransform module reads its input from the standard input stream and writes its output to the standard output stream:
<perl> <libdir>log_transform.pl \< <input.filename> \> <output.file>
ConsensusClustering example (actually all on one line):
<java> <java_flags> -DR_HOME=<R_HOME> -cp <libdir>geneweaver.jar edu.mit.wi.genome.geneweaver.clustering.ConsensusClustering <input.filename> <kmax> <niter> <normalize.type> -N <norm.iter> -S <resample> -t <algo> -L <merge.type> -i <descent.iter> -o <out.stub> -s -d <create.heat.map> -z <heat.map.size> -l1 -v
The input parameters section of the form appears perhaps to be the most daunting. And yet there is little that is required to make a working module declaration. Each parameter in the command line that comes from a user input must have an entry in this section. Otherwise the clients would know nothing about how to prompt the user for input nor could they explain to the user what type of input is expected.
To add one or more parameters, enter the number of parameters to add and click the Add Parameter button.
Each parameter has a name, which can be whatever you like, using letters, numbers, and period as a separator character between "words". It can be of mixed upper- and lower-case. The name is used inside <brackets> within the command line to indicate that the value of that variable should be substituted at that position within the command line. The name is also used as a label within the web client to prompt the user for the value for that field. And the name is used as a way of identifying which parameter is which for the scripting clients.
ConsensusClustering examples: kmax, input.filename
Some parameters are not required on the command line. These parameters, when left blank by the user when the module is invoked, result in nothing being added to the command line for that parameter.
The description field is optional, but is very useful. It allows the module author to provide a more detailed description than the name itself. What is the "kmax" parameter used for? Does it interact with any other parameters? Do you have any advice about what is a reasonable range of settings for it? The description is displayed by the GenePattern clients when they prompt for input for each field.
ConsensusClustering example: Type of clustering algorithm
Some parameters should have a default value which will be supplied on the module's command line if no setting is supplied by the user when invoking the module. This is not the same as the defaults defined in the program invoked by the module. Instead, this allows the module author to create a default, even when none exists in the program being invoked by the module.
The default value may use substitution variables, just like the rest of the command line. So a valid default for an output file might be <input.filename_basename>.foo, meaning that the output file will have the same stem as the input.filename parameter, but will have a .foo extension.
Default values for parameters that have a choice list must be either blank or one of the values from the choice list. Any other setting will result in an error message. If no default for a choice list is provided, the first entry on the list will be the default.
ConsensusClustering examples: NMF, 5, <input.filename_basename>
Some parameters need to have extra text prefixing them on the command line when they are specified. For example, you might need to write "-F filename" to pass in a filename. The prefix text "-F" or "-F " would be specified here. To insert a space between the flag and the parameter, add the space to the prefix text.
example (with space): -F inputfile
example (without space): -Finputfile
Declaration of the type of an input parameter allows the client to make a smarter presentation of the input to the user. (As of GenePattern 1.2, all parameters are being treated as either text or input file types). Parameter type choices are:
When you select a parameter type of input file, a drop-down list of file formats appears in the file format column. Select the valid file format(s) for this parameter. If your module requires an input file format not included in the list, scroll back to the Output Description field and click New to add that format to the list. For this type of parameter, when the user enters the name of the file, the GenePattern clients pass along the entire file rather than just the file name.
Some parameters are best represented as a drop-down list of choices. By constraining input to those from the list, the user is saved typing and cannot make a mistake by choosing an invalid setting (unless there is a dependency on some other parameter). To enter the choices, click the Edit Choices link and enter the choices in the Edit Choice List window.
For each choice enter the value required by the program (Value) and, optionally, a more human-readable value (Display Value). When you exit from the Edit Choice List window, the choices you entered are displayed as a semi-colon delimited set of choices. For example:
hierarchical=Hierarchical clustering;SOM=Self-organizing map;NMF=Non-negative Matrix Factorization;3.14159265=pi
For a simple example of a GenePattern module, please refer to https://github.com/genepattern/ExampleModule.
Creating a GenePattern module is a two-step process:
When writing a program that will be run as a GenePattern analysis module, keep in mind the following:
perl myscript" rather than including "#!/usr/bin/perl" as the first line of the myscript.pl file unless you are certain of the path within the Docker container of your module.Visualization modules are similar to analysis modules. The only difference between analysis and visualization modules is that analysis modules run on the server machine and visualization modules run on the client machine. Each module is launched in a separate process. An applet is used to launch the visualization module.
If you are writing R code to be invoked as a GenePattern module, follow the guidelines in Writing Modules for GenePattern. This section provides additional information for modules written in R:
When you create a module in GenePattern you specify the command line that invokes the program that performs the desired function. Generally the command line includes arguments such as the parameters for the algorithm and the data file to analyze. This is roughly equivalent to the structure of a UNIX command line call. Since GenePattern on its own has no concept of an R session, your command line must create one to execute your code with its arguments. In the R world, the recommended way to accomplish this is the Rscript utility.
GenePattern simply launches Rscript and hands off the parameters, then waits for the results. It does nothing more than an equivalent UNIX command line call, so your R script must be structured to receive and process these parameters as string values and execute any necessary R functions. One way to do this is to declare any functions, constants, etc. up front in the script, with any processing of command line arguments near the end along with the main R function call.
A full tutorial on Rscript is beyond the scope of this Guide. For more details, see the official Rscript documentation at the link above, or - for your particular version - by typing '?Rscript' within R. Beyond that, please see our Tutorial module for example code and usage.
GenePattern provides substitutions to allow you to call Rscript. As an example, the following command line invokes R (version 3.1 in this case) on the myscript.R file, passing input.filename as a single parameter:
commandLine=<R3.1_Rscript> <libdir>myscript.R <input.filename>The GenePattern team maintains a list of R packages to facilitate their use in GenePattern modules. As of GenePattern 3.9.6 you can list these packages in a descriptor file named 'r.package.info' to be included in your module ZIP bundle. When installing your module, GenePattern will automatically attempt to install these packages if they are not already present. Once installed, the packages will be available to any other module using that version of R. To use them simply include normal 'library' calls in your R code as usual. See our Tutorial module for an example.
The r.package.info file is a simple text file in comma-separated value format that specifies the name and (optionally) some other attributes of the package for GenePattern to use to identify, download, and install it.
In general, specifying only the first column (package name) is sufficient for basic use as that is enough for GenePattern to identify the package in both its installed package library and in the supported repositories (CRAN and Bioconductor at present). There are times when it's necessary to pin down a specific package version (due to errors in a CRAN package dependency spec, for example) or even a particular URL (for packages not hosted in one of these two repositories), which is where the other columns come into the picture. These are usually not needed until you are ready to distribute your module to others or run into version issues.
The section below gives a short example along with details for the file's columns and GenePattern's rules to process them. All package dependencies should be listed, not just the top-level ones, as GenePattern requires a complete list of what needs to be installed and managed. The should be listed in the order they are required, with earlier dependencies listed first. Remember to add the header line as it is required.
Here's a short example based on the declaration in our Tutorial module:
package,requested_version,archive_name,src_URL,Mac_URL,Windows_URL
getopt,1.20.0,CRAN
optparse,1.0.2,CRAN
# Comments are added with the '#' character. Blank lines are ignored
limma
edgeR
The columns have the following meanings:
package |
The unadorned package name, e.g. optparse or colorspace, as would be used in a library() or install.packages() call. This is required; the line will be skipped with a non-fatal warning if it's missing. |
requested_version |
The requested version string of the package, e.g. 1.0.2 or 1.2-4. This should exactly match the package’s declared version string as found in CRAN or Bioconductor (e.g. from optparse_1.0.2 or colorspace_1.2-4). This is optional, and its use is not necessary or even recommended except in conjunction with archive_name; see the notes below for details. |
| archive_name | A tag indicating which central package repository hosts this package. You can use CRAN or BIOC to indicate CRAN or Bioconductor respectively, though only the CRAN tag has any special meaning. Any other tag is silently ignored. This is also optional but has special meaning in conjunction with requested_version; see the notes below for details. |
| src_URL Mac_URL Windows_URL |
The URL of a source package to be used on those platforms which install from source (i.e. Linux). This is optional and its use is not recommended unless you need a package not available in CRAN or Bioconductor. It will be ignored on other platforms. The Mac_URL and Windows_URL columns have analogous meanings for the Mac OS X and Windows platforms, though these should point to binary packages for the appropriate platform. NOTE: the Windows platform is no longer officially supported. |
GenePattern will check its installed package library and attempt to install based on the following rules:
Note that the packages are shared across all modules for that version of R, so choosing a different version may not be possible (we are considering support for this in the future). In practice, the GenePattern team will pin CRAN packages to a specific version to prevent after-the-fact version drift and conflicts across packages. If you submit a module for use on one of our public servers, we will likely change the r.package.info file to make sure it plays well with existing modules.
With this method of installing R packages there is no need to bundle them in the module ZIP; nor do you write the extra code to install them when the module is first run. It also means that any package installation errors will be detected during the installation of the module, rather than having it seem to install correctly only to fail at run time. Furthermore, GenePattern also checks the packages ahead of each job run, pre-emptively catching any inadvertent changes in the back-end file system. Last but not least, this allows packages to be installed only once per server and then shared between modules rather than being installed for each individual module.
The easiest way is to use the entries in the r.package.info file from one or more of our existing modules and/or use our Tutorial module as a template for your own. Note that choosing the correct mix of packages is usually dependent on the version of R you are using, so make sure the r.package.info files come from modules matching that version.
For a list of available R libraries from the GenePattern team, please contact us at gp-help AT broadinstitute DOT org. The current list is available at (Review Note: Peter to provide an updated link when ready...)
Note that in order to use this example module, you will need to install R-3.1.3. For information on how to do so please see Using Different Versions of R in GenePattern.
We recommend you develop modules with the latest version of R supported by GenePattern (R-3.1.3 at the time of this writing, April 2016). Note that for purposes of stability this will usually lag the current version available from CRAN. Older versions of R are also available in GenePattern to support existing modules, but it's best to avoid these unless you have particular compatibility needs (e.g. porting old code).
Also be aware that there are known issues running multiple versions of R on the Mac platform that have nothing to do with GenePattern. See our Administrators Guide for details.
The automatic package installer works with Rscript modules for R >= version 2.14. As of this writing (April 2016):
As a note for Administrators: Rscript modules use the built-in run-rscript.sh wrapper script; site-specific customization is defined in the GenePatternServer/resources/custom.properties file.
# example custom.properties entry
R3.1_Rscript=<run-rscript> -v 3.1 --To reverse-engineer for your GP instance, look at the Admin > Server Settings > Custom properties page (which is a view of the GenePatternServer/resources/custom.properties file). Additionally, the wrapper.properties file is a template for manually configuring your server:
GenePatternServer/resources/wrapper_scripts/wrapper.properties (also available on GitHub)GenePatternServer/resources/wrapper_scripts/run-rscript.shCirca the 3.9.7 release, these scripts are in use on Broad hosted production servers.
We target the final point-version for each R major.minor line. This is for stability; it can affect the version of Bioconductor, for example. As of this writing, those versions are:
When designing the program and its command line, use parameter flags (for example, -f input_file) rather than relying on parameter positions. Parameter flags allow users to build command lines with variable numbers of arguments, which makes it easy to omit optional parameters. When writing R code, if you have optional input parameters on the command line, you must use named rather than positional parameters in the command line definition.
For example, to write an R function that takes a filename as input, the main function might be:
myfunction <- function(...)
{
args <- list(...)
for(i in 1:length(args)) {
flag <- substring(args[[i]], 0, 2)
value <- substring(args[[i]], 3, nchar(args[[i]]))
if(flag=='-i')
{
# code to set variables, etc...
}
} There are good alternatives available such as the optparse package. See our Tutorial module for an example.
If you are writing MATLAB code to be invoked as a GenePattern module, follow the guidelines in Writing Modules for GenePattern. In addition, for MATLAB code, you must address licensing and distribution issues, as described in this section:
You can invoke a MATLAB executable from a GenePattern module using one of two approaches: the direct approach or the compiled approach. Following are brief descriptions of each approach, including its advantages and disadvantages:
If you are simply using your M-code on your standalone GenePattern server, the direct approach is simpler; however, if you want to give copies of your M-code to other people or deploy your M-code on a shared GenePattern server, the compiled approach is preferred. The compiled approach may provide slightly better performance for fast running modules since the startup delay will be shorter, but the actual execution time will be approximately the same for either approach.
The instructions in this section are based on the following MATLAB versions:
When you create a module in GenePattern, you specify the command line that invokes the program that performs the desired function. Generally, the command line includes arguments, such as the parameters for the algorithm and the data file to analyze.
Calling script M-code from a command line is possible, but generally not useful because you cannot pass arguments to the script. To pass arguments to your M-code, create a no-return entry function to serve as the top level call into MATLAB. The following example defines a no-return entry function that accepts two parameters:
function analyzeThis ( filename, whatToWrite )
... (your M-code here)Writing Modules for GenePattern provides additional guidelines for writing code that will run as a GenePattern module.
If you do not plan to use the compiled M-code approach, skip this section and continue with Distributing Your MATLAB Code.
Compiling your MATLAB M-code into a standalone executable is described in the MATLAB Compiler Documentation. Please refer to this documentation to understand all of the options available to you. To summarize the simplest case, from within MATLAB, at the MATLAB prompt, execute the following command:
mcc -m analyzeThiswhere analyzeThis is the name of your entry function. This command generates the following files in your $MATLAB_ROOT/work directory:
| analyzeThis (Linux, Mac OS X) or analyzeThis.exe (Windows) |
Executable file |
| analyzeThis.ctf | Component Framework file |
| analyzeThis.c (Linux, Windows) | C language Source Code |
| analyzeThis.h (Linux, Windows | C Language Header file |
| analyzeThis_main.c | C language Source Code |
| analyzeThis_mcc_component_data.c | C language Source Code |
Note: To use the MATLAB compiler on Mac OS X, you must have Xcode 2.2 installed; minimally, the Developer Tools, gcc 4.0, gcc 3.3, Mac OS X SDK, and BSD SDK. These instructions were tested using Xcode 2.2.1.
After writing your MATLAB code, create a GenePattern module that invokes the code that you have written. Creating Modules in GenePattern describes how to create a GenePattern module. This section provides supplemental information for MATLAB:
Creating Modules in GenePattern describes how to create a GenePattern module that invokes the code that you have written. This section provides additional information that applies when you are directly calling the MATLAB executable from the GenePattern module:
On Windows, your GenePattern module definition form can contain a simple command line that calls MATLAB with the -r flag to execute your function; for example:
matlab -nosplash -r "analyzeThis <p1> <p2>"This example invokes MATLAB without the splash screen (-nosplash) and directs it to execute the quoted command, where p1 and p2 are parameters that you specify in the GenePattern module definition form and that are passed to the MATLAB command line as Strings. MATLAB looks for the function analyzeThis on the MATLAB path; therefore, it is not necessary to upload the function as a support file, although it is recommended.
To ensure that the GenePattern server can call the MATLAB executable, you typically add the MATLAB directory to your PATH system environment variable. (Alternatively, you can enter the full path to the MATLAB executable on the command line, but this makes it more difficult to deploy the module on other GenePattern servers.)
To check that MATLAB is on your path:
matlab and press Enter.If the MATLAB application starts, MATLAB is on your path.
If MATLAB is not on your path, add it:
$MATLAB_ROOT/bin directory to the path.Open a new DOS window and check again that MATLAB is on your path.
On platforms other than Windows, the execution of the command line differs slightly due to variations in the Java Virtual Machines (VMs) that GenePattern is running. If you use the simple matlab command, as described for Windows, the Java VMs on these platforms attempt to parse and quote the command line resulting in MATLAB generating errors in its eval function.
On these platforms, you must use a wrapper Java class to launch MATLAB. This wrapper class also works on Windows and does not rely on the PATH variable, which makes it the preferred method for implementing the direct approach on any platform.
To use the wrapper Java class:
runmatlab.jar file as a support file. To request a copy of this file, send e-mail to gp-help (at) broadinstitute.org; alternatively, the java source code for the RunMatlab wrapper class is included here: RunMatlab.java.<java> -cp <libdir>runmatlab.jar RunMatlab <libdir> analyzeThis <p1> <p2>Where analyzeThis is the name of your MATLAB entry function name and <p1> and <p2> are the arguments to the function. The RunMatlab class ensures that the arguments are correctly written out and calls MATLAB with the -nosplash and -nodisplay arguments.
Creating Modules in GenePattern describes how to create a GenePattern module that invokes the code that you have written. This section provides additional information that applies when you are compiling your M-code into a standalone executable and invoking that executable from the GenePattern module:
To run a standalone executable generated by the MATLAB Compiler, the GenePattern server must have the MATLAB Component Runtime (MCR) installed. This is a collection of shared libraries, which contains the runtime code for MATLAB, that is used by the standalone application. If the GenePattern server has MATLAB installed, you do not need to install the MCR; it is already installed.
Full details for installing the MCR can be found in the MATLAB Compiler documentation, in the section titled "Deploying Components to Other Machines". To summarize this documentation, on the GenePattern server machine, you need to run the MCRInstaller:
On Windows, to run the MCRInstaller:
<matlabroot>\toolbox\compiler\deploy\win32\MRCInstaller.exe to the server machine.MCRInstaller.exe.On Linux, to run the MCRInstaller:
buildmcr.<matlabroot>/toolbox/compiler/deploy/MCRInstaller.zip to the server machine.MCRInstaller.zip into a directory (<mcr_root>).setenv LD_LIBRARY_PATH
<mcr_root>/runtime/glnx86:
<mcr_root>/sys/os/glnx86:
<mcr_root>/sys/java/jre/glnx86/jre1.4.2/lib/i386/client:
<mcr_root>/sys/java/jre/glnx86/jre1.4.2/lib/i386:
<mcr_root>/sys/opengl/lib/glnx86:${LD_LIBRARY_PATH}On Mac OS X, to run the MCRInstaller:
buildmcr.<matlabroot>/toolbox/compiler/deploy/MCRInstaller.zip to the server machine.MCRInstaller.zip into a directory (<mcr_root>).setenv DYLD_LIBRARY_PATH
<mcr_root>/<ver>/runtime/mac:
<mcr_root>/<ver>/sys/os/mac:
<mcr_root>/<ver>/bin/mac:
/System/Library/Frameworks/JavaVM.framework/JavaVM:
/System/Library/Frameworks/JavaEmbedding.framework/JavaEmbedding:
/System/Library/Frameworks/JavaVM.framework/Libraries
setenv XAPPLRESDIR <mcr_root>/<ver>/X11/app-defaultsWhen the MATLAB Compiler generates a standalone executable, it also generates a Component Framework (.ctf) file. The .ctf file must be on the path when you run the standalone executable. The easiest way to address this requirement is to create a launcher script (.bat or .sh file) that adds the .ctf file to the PATH or LIBPATH and then runs the standalone executable.
On Windows, for example, to launch the MATLAB executable analyzeThis.exe, create a launcher script, mllaunch.bat, that contains the following lines:
set LIBDIR=%1
set PATH=%LIBDIR%;%PATH%
analyzeThis %2 %3 On Linux, for example, to launch the MATLAB executable analyzeThis.exe, create a launcher script, mllaunch.sh, that contains the following lines:
#!/bin/csh
export MCR_ROOT=<path where you installed the files from MCRInstaller.zip>
export LD_LIBRARY_PATH=$1:$MCR_ROOT/runtime/glnx86:$MCR_ROOT/sys/os/glnx86:\
$MCR_ROOT/sys/java/jre/glnx86/jre1.4.2/lib/i386/client:\
$MCR_ROOT/sys/java/jre/glnx86/jre1.4.2/lib/i386:\
$MCR_ROOT/sys/opengl/lib/glnx86
export PATH=$1:$PATH
chmod a+x $1/analyzeThis
analyzeThis $2 $3 The chmod line sets the executable permission on the executable file; by default, the GenePattern server does not set this permission for uploaded files.
On Mac OS X, for example, to launch the MATLAB executable analyzeThis.exe, create a launcher script, mllaunch.sh, that contains the following lines:
#!/bin/sh
export MCR_ROOT=/Volumes/os9/gpserv
export LD_LIBRARY_PATH=$1:/Volumes/os9/matlab7.2/sys/os/mac:
/Volumes/os9/matlab7.2/bin/mac/
export DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH
export PATH=$1:$PATH
chmod a+x $1/writeToFile
writeToFile $2 "$3" The chmod line sets the executable permission on the executable file; by default, the GenePattern server does not set this permission for uploaded files.
On the GenePattern module definition form, write a command line calls the launcher script, passing the <libdir> parameter as the first argument (so that it can be added to the path).
On Windows, the following command line calls the launcher script, mllaunch.bat:
<libdir>mllaunch.bat <libdir> <param1> <param2>On Linux or Mac OS X, the following command line calls the launcher script, mllaunch.sh:
sh <libdir>mllaunch.sh <libdir> <param1> <param2>In both command lines, the first <libdir> sets the path to the mllaunch script. The second <libdir> is passed as the first argument to the script so that the script can add this directory to the appropriate environment variables. The <param1> and <param2> variables are parameters to the MATLAB application, which you define in the module definition form and specify in the command line as usual.
For the compiled approach, you must specify at least two support files for the MATLAB application: the executable file and .ctf file. If your application requires additional files for its execution, also add those files as support files.
Should you choose to distribute your MATLAB based module to others, you must ensure you are in compliance with the MATLAB licensing agreement:
http://www.mathworks.com/company/aboutus/policies_statements/agreement.pdf
Following are a few key points for GenePattern developers:
Please refer to the MATLAB licensing agreement for exact details. You are responsible for reviewing and complying with the MATLAB software license. The above summary does not exempt you from this responsibility.
This section provides a step-by-step example of deploying a simple M-file application as a GenePattern module on a GenePattern server. Where the instructions are platform specific, the example shows instructions for Windows, Linux, and Mac OS X.
The first step is writing the MATLAB M-file that you want to share. For this example, write a simple application that takes a filename and a String and writes the String out to a file with the given name. This application consists of the following lines:
% write the variable whatToWrite to a file called filename in the current directory
fid = fopen(filename,'w');
fprintf(fid,'#writing to a file\n\n');
fprintf(fid,whatToWrite);
fclose(fid); To call the M-file from the command line and pass it parameters, you must turn this script into a no-return function. To do this, add a function definition line at the start of the M-file and save the file using the name of the function (for example, writeToFile.m).
function writeToFile( filename, whatToWrite)
% write the parameter whatToWrite to a file called filename in the current directory
fid = fopen(filename,'w');
fprintf(fid,'#writing to a file\n\n');
fprintf(fid,whatToWrite);
fclose(fid); Within the MATLAB environment, call the MATLAB Compiler to convert this function into an application:
>> mcc -m writeToFile Within the current working directory, this creates a number of files, including the following:
Note: To use the MATLAB compiler on Mac OS X, you must have Xcode 2.2 installed; minimally, the Developer Tools, gcc 4.0, gcc 3.3, Mac OS X SDK, and BSD SDK. These instructions were tested using Xcode 2.2.1.
Install the MATLAB Component Runtime (MCR) on the GenePattern server, if you have not done so already. If the GenePattern server has MATLAB installed, it also has the MCR installed.
To install the MCR:
<matlabroot>\toolbox\compiler\deploy\win32\MRCInstaller.exe to the GenePattern server machine.MCRInstaller.exeTo install the MCR:
>> buildmcr mcrdirmcrdir, beneath the current working directory and creates a file within that directory called MCRInstaller.zip.matlab, under the GenePattern server directory and install the library files in MCRInstaller.zip into that directory:cd GenePatternServer
mkdir matlab
cd matlab
cp <path to mcrinstaller.zip>MCRInstaller.zip .
unzip MCRInstaller.zipCreate the launcher script that sets the environment variables and then calls the MATLAB application.
Create the launcher script as a batch file that sets the PATH variable for the environment and then calls the MATLAB application. To do so, in a text editor, create the following mllaunch.bat file:
set LIBDIR=%1
set PATH=%LIBDIR%;%PATH%
writeToFile %2 %3Create the launcher script as an .sh file that sets the PATH and LD_LIBRARY_PATH variables for the environment, ensures that the application is executable, and then calls the MATLAB application. To do so, in a text editor, create the following mllaunch.sh file:
#!/bin/csh
export MCRROOT=/home/username/GenePatternServer/matlab/v70
export LD_LIBRARY_PATH=$1:$MCRROOT/runtime/glnx86:$MCRROOT/sys/os/glnx86:$MCRROOT/sys/java/ jre/glnx86/jre1.4.2/lib/i386/client:$MCRROOT/sys/java/jre/glnx86/jre1.4.2/lib/i386:$MCRROOT/ sys/opengl/lib/glnx86
export PATH=$1:$PATH
chmod a+x $1/testTwo
writeToFile $2 $3Note that the MCR_ROOT variable is set to the v70 directory, which you created by unzipping MCRInstaller.zip.
Create the launcher script as an .sh file that sets the LD_LIBRARY_PATH and DYLD_LIBRARY_PATH variables for the environment, ensures that the application is executable, and then calls the MATLAB application. To do so, in a text editor, create the following mllaunch.sh file:
#!/bin/sh
export MCR_ROOT=/Volumes/os9/gpserv
export LD_LIBRARY_PATH=$1:/Volumes/os9/matlab7.2/sys/os/mac:/Volumes/os9/matlab7.2/bin/mac/
export DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH
export PATH=$1:$PATH
chmod a+x $1/writeToFile
writeToFile $2 "$3"Use GenePattern to create a module that executes the launcher script.
sh <libdir>mllaunch.bat <libdir> <fname> <txt>sh <libdir>mllaunch.sh <libdir> <fname> <txt>
Save the module and execute it. The module should create two files:
If the following error appears in the stdout file, you have not correctly set the path to the libraries that you installed from MCRInstaller.zip:
error while loading shared libraries: libmwmclmcrrt.so.7.0: cannot open shared object file: No such file or directory
Double check the path. If it is correct, you may be using a different Unix shell than the one used in this example. Check that the mllaunch.sh file uses the correct command (export in this example) to set PATH and LD_LIBRARY_PATH.
Using Java as a GenePattern client allows you to run GenePattern modules and visualizers from within a Java application. This section describes how you can use the GenePattern Java library to run GenePattern analyses as easily as calling a routine. It contains the following topics:
If you are not familiar with Java, see the http://java.sun.com website, which provides downloadable programs, samples, tutorials, and book suggestions.
The GenePattern Java library allows you to invoke a GenePattern module as if it were a local Java method running on your client and to get back from the module a list of result files. A zip file containing the Java library (and Javadoc that describes the API for accessing the server and running modules) is available on your GenePattern server.
To download the GenePattern Java library to your computer:
This section explores a simple Java application that preprocesses a dataset and displays it using the HeatMapViewer. The included code can be copied and pasted into your Java program so that you can try it out, modify it, and create your own solutions. The full source code of the sample application is available here.
The first statements in the application initialize various settings, which you must do once in every application that accesses GenePattern. You will need to customize the italicized GenePattern server URL and GenePattern user name (typically, your e-mail address) with values appropriate for your GenePattern server.
import org.genepattern.matrix.Dataset; import org.genepattern.client.GPClient; import org.genepattern.webservice.JobResult; import org.genepattern.webservice.Parameter; import org.genepattern.io.IOUtil;
import java.io.File;
public class MyProgram
{
public static void main(String[] args) throws Exception
{
GPClient gpClient = new GPClient("http://localhost:8080", "your email address");
After initializing the required settings, the application runs the PreprocessDataset module to preprocess a dataset. This example references the dataset using a publicly-accessible URL, but a filename would be equally valid. When you invoke the runAnalysis method, the GenePattern library invokes the appropriate module on the server, passing all of the input parameters and input files. Control returns to your application when the module completes. (To run a module asynchronously, invoke the runAnalysisNoWait method or use the runAnalysis method in a separate thread.)
String inputDataset = "ftp://ftp.broadinstitute.org/pub/genepattern/all_aml/all_aml_train.res";
JobResult preprocess = gpClient.runAnalysis("PreprocessDataset",
new Parameter[] {
new Parameter("input.filename", inputDataset)
});
When the module completes, you can query the JobResult object for an array of filenames that are the output from the module. You can download the result files or leave them on the server and refer to them by URL. Referring to result files by URL is especially useful for intermediate results. In this example, the JobResult object named preprocess contains a list of filenames (of length 1, in this case), which the application displays in a heat map:
// view results in a HeatMapViewer visualizer
gpClient.runVisualizer("HeatMapViewer",
new Parameter[] {
new Parameter("dataset", preprocess.getURL(0).toString())
});
The last statements in the application download the preprocessed data and load it into a matrix for further analysis:
String downloadDirName = String.valueOf(preprocess.getJobNumber());
// download result files
File[] outputFiles = preprocess.downloadFiles(downloadDirName);
// load data into matrix for further manipulation
Dataset dataset= IOUtil.readDataset(outputFiles[0].getPath());
}
}
You can combine GenePattern analyses with any capabilities that the Java environment has to offer. Use Java's 2-D and 3-D graphics libraries to create graphic output, or summarize and report on the data using your own code. The basic idea to remember is that GenePattern modules create result files and those files are available to the Java application for processing.
For more information:
Life Science Identifiers (LSIDs) can be used instead of module names to identify modules for GenePattern to run. An LSID may be submitted in place of the module name in the methods runAnalysis and runVisualizer. When an LSID is provided that does not include a version, the latest available version of the module identified by the LSID will be used. If a module name is supplied, the latest version of the module with the nearest authority is selected. The nearest authority is the first match in the sequence: local authority, Broad authority, other authority.
If you are unfamiliar with LSIDs and GenePattern versioning, see Concepts.
Using MATLAB as a GenePattern client allows you to run GenePattern modules and to manipulate and visualize the results in a powerful, commercial technical computing application that works on most major platforms. Using GenePattern allows you to invoke methods written in many other languages without having to worry about how to launch them. This section describes how you can use the GenePattern MATLAB library to run GenePattern analyses:
Resources and documentation for MATLAB are available at http://www.mathworks.com/.
The GenePattern MATLAB library allows you to invoke a GenePattern module as if it were a local MATLAB function running on your client and to get back from the module a list of result files. A zip file containing the MATLAB library is available on your GenePattern server.
To download the GenePattern MATLAB library to your computer:
MATLAB7/toolboxes directory. If you do not have permission to put files in that directory, unzip into any other directory.>>pathtoolGenePatternServer and GenePatternFileSupport directories, with subfolders, to the MATLAB search path.Note: MATLAB 7.0.4 (R14SP2) and later use Java Virtual Machine (JVM) 1.5. If you are using an earlier version of MATLAB, you must change the JVM that MATLAB is using to JVM 1.5. For instructions, see http://www.mathworks.com/support/solutions/data/1-1812J.html?solution=1-1812J.
This section explores a simple MATLAB program that runs a module, displays the resulting output, and loads it into a MATLAB matrix for further analysis. The included code can be copied and pasted into your MATLAB client so that you can try it out, modify it, and create your own solutions.
The first statements in the application initialize various settings, which you must do once in every application that accesses GenePattern. You will need to customize the italicized GenePattern server URL, GenePattern user name (typically, your e-mail address) and password (if required) with values appropriate for your GenePattern server.
% Create a GenePattern server proxy instance
gp = GenePatternServer('http://localhost:8080','my.email@my.domain', 'mypassword');After initializing the required settings, the application runs the TransposeDataset module to transpose a dataset. This example references the dataset using a publicly-accessible URL, but a filename would be equally valid. As shown below, you can call the GenePattern methods directly or by calling the runAnalysis method. When you call a GenePattern method, such as TransposeDataset, the GenePattern library invokes the module on the server, passing all of the input parameters and input files. Control returns to your application when the module completes. (To run a module asynchronously, invoke the method in a separate thread.)
% input dataset for transpose operation
params.output_file_name = 'transposed.out'
params.input_filename='http://www.broadinstitute.org/mpr/publications/projects/Leukemia/
ALL_vs_AML_train_set_38_sorted.res'
% transpose the dataset
transposeResult = gp.TransposeDataset(params)
% alternate call to transpose the dataset
transposeResult = runAnalysis(gp, 'TransposeDataset', params)When the module completes, it returns a MATLAB structure that contains a list of filenames that are the output from the module. In this example, transposeResult is a structure with a list of filenames (of length 1, in this case). The application displays the results in a file viewer window and also loads them into a matrix so that further manipulation can be performed:
% display the transposed results
edit 'transposed.out.gct'
% now read the output into a matrix
% so we can do further manipulation in MATLAB
myData = loadGenePatternExpressionFile('transposed.out.gct')You can combine GenePattern analyses with all of the rich functionality of MATLAB. For example, you can use MATLAB's plotting methods to create graphic output, save modified matrices to files using save, or summarize and report on the data using your own code. The basic idea to remember is that GenePattern modules create result files and those files are available to the MATLAB client for processing.
For a list of the GenePattern modules available on your server, run the listMethods function on your GenePatternServer object. To view the names of the input parameters for a module, use the describeMethod function on your GenePatternServer object, passing it the module name.
% display the available GenePattern modules
listMethods(gp)
% now look at the parameters for the TransposeDataset module
describeMethod(gp, 'TransposeDataset')Alternatively, to get the parameters with their default values filled in, use the getMethodParameters function of the GenePatternServer object. This returns a MATLAB structure with named elements for each parameter, filled in with the default value if one exists. After filling in the missing parameters and overriding defaults if desired, this structure can then be passed on to the runAnalysis method.
% display the available GenePattern modules
params2 = getMethodParameters(gp, 'TransposeDataset')
params2.input_filename='http://www.broadinstitute.org/mpr/publications/projects/Leukemia/ALL_vs_AML_train_set_38_sorted.res'
% transpose the dataset
transposeResult = gp.TransposeDataset(params2)The GenePattern MATLAB library also has convenience methods to read and write GenePattern files (such as res, gct, and odf files). Even if you choose not to look in the library, you can extend the techniques shown above to implement your own analyses.
For more information:
You can use Life Science Identifiers (LSIDs) to identify a module when executing GenePattern code in MATLAB. An LSID may be submitted in place of the module name to getMethodParameters or runAnalysis. When providing an LSID to a method in addition to a module name, the LSID alone is used to determine what module to run. When an LSID is provided that does not include a version, the latest available version of the module identified by the LSID will be used. If you are unfamiliar with LSIDs and GenePattern versioning, see Concepts.
% Example using LSIDs from MATLAB
params = getMethodParameters(gp, 'urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00026:3');
params.output_file_name = 'transposed.out'
params.input_filename='http://www.broadinstitute.org/mpr/publications/projects/Leukemia/ALL_vs_AML_train_set_38_sorted.res'
% transpose the dataset
transposeResult = runAnalysis(gp, 'urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00026:3', params)
Using R as a GenePattern client allows you to run GenePattern modules and to manipulate and visualize the results in a powerful, free statistical desktop package that works on most major platforms. Using GenePattern allows you to invoke methods written in many other languages without having to worry about how to launch them or whether you are passing incorrect parameters. This section describes how you can use the GenePattern R library to run GenePattern analyses:
If you are not familiar with R, see the following resources on the www.r-project.org website:
The GenePattern R package allows you to invoke a GenePattern module as if it were a local R method running on your client and to get back from the module a list of result files. The package requires R version 2.4.1 or greater and the rJava package. The package can be downloaded from your GenePattern server in Windows(.zip), source (.tar.gz), and Mac OS X (.tgz) formats.
To download the GenePattern R package to your computer:
install.packages("full-path-to-GenePattern-R-package", type="source", repos=NULL)Note: If you are using a version of R which you cannot modify (because it is a publicly-shared version and you do not have appropriate privilege), you can have it load the GenePattern library by setting the environment variable R_LIBS=<GenePattern install directory>/R/library in your autoexec.bat, .cshrc, .bashrc or other shell startup file. R will then load from its usual location, but will also search for and find the GenePattern library from your installation.
This section explores a simple R program that runs a module, displays the resulting output, and loads it into an R matrix for further analysis. The included code can be copied and pasted into your R environment so that you can try it out, modify it, and create your own solutions.
The first statements in the application initialize various settings, which you must do once in every application that accesses GenePattern. You will need to customize the italicized GenePattern server URL, GenePattern user name (typically, your e-mail address), and password with values appropriate for your GenePattern server. The gp.login method returns a GPClient object that contains the information required for running modules on a GenePattern server.
# Load GenePattern package
library(GenePattern)
username <- "your email address"
password <- "your password"
servername <- "http://localhost:8080"
# Obtain a GPClient object which references a specific server and user
gp.client <- gp.login(servername, username, password)After initializing the required settings, the application runs the PreprocessDataset module to preprocess a dataset. This example references the dataset using a publicly-accessible URL, but a filename would be equally valid. When you call an R method, such as run.analysis, the GenePattern package invokes the appropriate module on the server, passing all of the input parameters and input files. Control returns to your application when the module completes. (To run a module asynchronously, use the method runAnalysisNoWait.)
# input dataset for preprocess operation
input.ds <- "ftp://ftp.broadinstitute.org/pub/genepattern/all_aml/all_aml_train.res"
# preprocess the dataset
preprocess.jobresult <- run.analysis(gp.client, "PreprocessDataset", input.filename=input.ds)When the module completes, it returns a JobResult object with which you can execute various methods. For example, you can call a method using a JobResult object to get an R list of the filenames that are the output of the module. Afterwards, you can download the files or leave them on the server and refer to them by URL. In this example, we view the results in a heat map:
# Obtain the url location of the result and run the visualizer
preprocess.out.file.url <- job.result.get.url(preprocess.jobresult, 0)
run.visualizer(gp.client, "HeatMapViewer", dataset=preprocess.out.file.url)In this example, the application downloads the result file and displays the results in a file viewer window, then also loads the data into a matrix so that further manipulation can be performed in R:
# download result files
download.directory <- job.result.get.job.number(preprocess.jobresult)
download.directory <- as.character(download.directory)
preprocess.out.files <- job.result.download.files(preprocess.jobresult, download.directory)
# display the preprocessed result
preprocessed.out.file <- as.character(preprocess.out.files[1])
file.show(preprocessed.out.file)
# now read the output into a matrix
# so we can do further manipulation in R
data <- read.dataset(preprocessed.out.file)You can combine GenePattern analyses with all of the rich statistical functionality of R. For example, you can use R's plot and legend methods to create graphic output, output JPEGs of your visualized data using savePlot, save modified matrices to files using save, or summarize and report on the data using your own code. Just remember: GenePattern modules create JobResult objects and those objects are available to the R client for processing.
The GenePattern R package also has methods to read and write GenePattern files (such as res, gct, and cls files), to enable running of multiple modules in parallel, to run modules with input from files that were output from previous modules without moving them from the server, and other utilities. Even if you choose not to look in the library, you can extend the techniques shown above to implement your own analyses.
For more information:
You can use Life Science Identifiers (LSIDs) instead of module names to identify modules for GenePattern to run. For R, this is primarily useful when you want to specify a particular version of a module for GenePattern to run. The easiest way to specify a particular version of a module is to specify the LSID as an argument to an R method such as run.analysis in place of the GenePattern module name. For example, the following statement invokes version 1 rather than the latest version of the PreprocesDataset module:
preprocess.jobresult <- run.analysis(gp.client, "urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00020:5", input.filename=input.ds)If you are unfamiliar with LSIDs and GenePattern versioning, see Concepts.
Using Python as a GenePattern client allows you to run GenePattern modules from a Python script or interactive prompt. This section describes how you can use the GenePattern Python library to run GenePattern analyses as easily as calling a function. It contains the following topics:
If you are not familiar with Python, see the following resources on the https://www.python.org/ website:
The GenePattern Python package allows you to invoke a GenePattern module as if it were a Python method call. The GenePattern Python package supports both Python 2.7 and Python 3.4 or later. The package can be downloaded from your GenePattern server.
To download the GenePattern Python package to your computer:
python setup.py installNote: If you do not have the appropriate permissions to install a python module you may need to run this command as an administrator (Windows) or using the sudo command (Linux, Mac).
The GenePattern Python package can be installed from the Python Package Index (PIP) by running the following command:
pip install genepattern-pythonNote: To install a package you may need administrator permissions, such as running the above command using sudo.
This section explores a simple Python program that connects to a GenePattern Server, runs a module and loads the resulting files for further analysis. The included code can be copied and pasted into your Python script or interactve terminal so that you can try it out, modify it or create your own solutions.
The very first thing you will need to do is to import the GenePattern library into your script. This can be achived by entering the following shown below. All methods provided by the GenePattern library can then be accessed from the gp namespace.
import gpThe next step in using the GenePattern Python library is to connect to an existing GenePattern server. This will require entering the address of your GenePattern, as well as your username and password credentials. The code below shows an example which connects to a GenePattern server running on the same computer as your Python terminal. Note that the address used must end in /gp for the library to successfully connect to GenePattern. Obviously myusername and mypassword should be changed as appropriate.
# Create a GenePattern server proxy instance
gpserver = gp.GPServer('http://localhost:8080/gp','myusername', 'mypassword')If you are not aware of which modules are available on the GenePattern server this can be programmatically explored by running the code shown below. This will return a list of GPTask objects, each representing a different module. These GPTask objects will provide the module name, LSID, a description and the version number
task_list = gpserver.get_task_list()If you already know the name or LSID of the module you want to call, you can obtain a GPTask object for it directly by running the code below. This example code obtains a GPTask object for the PreprocessDataset module.
module = gp.GPTask(gpserver, "PreprocessDataset") # Obtaining GPTask by module name
module = gp.GPTask(gpserver, "urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00020:5") # Obtaining GPTask by LSIDHowever you obtain a GPTask object, before this task can be used to run GenePattern jobs its parameters must first be loaded from the GenePattern server. This can be accomplished by running the code shown below.
module.param_load()After loading the parameters they can be explored by calling the code below. This will return a list of GPParam objects. Each GPParam object contains a parameter name, description, type, whether it's optional and other metadata, as shown below.
params_list = module.get_parameters() # Get the list of GPParam objects
for param in params_list: # Loop through each parameter
print( param.get_name() ) # Print the parameter's name
print( param.get_description() ) # Print the parameter's description
print( param.get_default_value() ) # Print the parameter's default value
print( param.is_optional() ) # Print whether the parameter is optionalIn order to run a GenePattern job from Python you must first obtain a GPJobSpec object from the correct GPTask object and then set the appropriate parameters for the job. Files can be uploaded by calling GPServer.upload_file(). An example for the PreprocessDataset module is shown below.
job_spec = module.make_job_spec() # Create the GPJobSpec
uploaded_file = gpserver.upload_file("file_name", "/path/to/the/file/file_name") # Upload the input file
job_spec.set_parameter("input.filename", uploaded_file.get_url()) # Attach the input file to the correct parameter
for param in module.get_parameters(): # Loop through all the other parameters and set their default values
if param.get_name() != "input.filename" and param.get_default_value() != None: # If it's not the parameter we just set, and if it has a default value
job_spec.set_parameter( param.get_name(), param.get_default_value() ) # Set the default valueOnce the GPJobSpec is ready, it can be used to launch a GenePattern job as shown below. This will return a GPJob object representing the job. By default this will halt code execution until the job has finished running on GenePattern. For long running jobs, however, this may not be desirable. By optionally passing in False as a parameter a GPJob object representing the pending or still processing job will be returned and the execution of the program will continue.
job = gpserver.run_job(job_spec) # This will halt execution until the job is complete
job = gpserver.run_job(job_spec, False) # This will return the job object and continue execution even if the job isn't finishedIf the latter option is used, the status of the job can be quaried programmatically by calling the follwing code:
job.is_finished() # Quaries the server and returns True if the job is complete, False otherwise
job.get_info() # Returns a brief description of the job's current stateFinally, once the job is complete its output files can be obtained by making the call shown below. This will return a list of GPFile objects, each containing methods to download or read the contents of the file.
output_list = job.get_output_files() # Get a list of output files
for file in output_list: # Loop through each output file
print( file.get_url() ) # Print the URL to the file
data = file.read() # Read the data in the fileYou can use Life Science Identifiers (LSIDs) instead of module names to identify modules for GenePattern to run. For Python, this is primarily useful when you want to specify a particular version of a module. The easiest way to specify a particular version is to specify the LSID as an argument to a Python method - such as the GPJobSpec constructor - in place of the GenePattern module name. For example, the following statement invokes version 1 rather than the latest version of the PreprocesDataset module:
job_spec = gp.GPJobSpec(gpserver, "urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00020:5")If you are unfamiliar with LSIDs and GenePattern versioning, see Concepts.
A tutorial for using the GenePattern Python library, as well as a tutorial for using GenePattern in conjunction with common Python libraries for scientific computing and plotting, are available in the GenePattern Notebook Environment. These notebook tutorials can be downloaded from the Example Notebooks page.
This section is currently under construction.
We have documented our current method for download a filtered set of jobs from GenePattern in our Google forum, and will continue to build out this section.
Please contact us with any questions.