Transition to EF Portal
If you’re using EnginFrame, it can be very beneficial transition to EF Portal, the next evolution of the powerful HPC and VDI portal platform. Built on the same code base as EnginFrame, EF Portal offers a familiar installation process, consistent user experience, and exciting new features designed to enhance productivity and performance. As part of NI SP’s commitment to supporting EnginFrame customers, we offer free migration options and ongoing support to ensure a seamless upgrade experience.

What’s New in EF Portal 2025.0

(PDF Version: What’s New in EF Portal 2025.0)

EF Portal REST API

• JSON Output for Easy Consumption: Simplified data exchange for seamless integration into scripts, command-line tools, and third-party applications.
• Programmatic Job Submission: Automate job submissions directly from scripts or the command line.
• Enhanced Automation and Batch Operations: Effortlessly manage repetitive tasks with robust automation capabilities.
• Comprehensive Job Management: Query job information, control job states (Cancel, Suspend, Resume), and monitor progress in real-time.
• Cluster Node Insights: Gather detailed information about cluster nodes for improved resource management.
• File Management Capabilities:
  • Upload files directly to EF Portal from scripts or the command line.
  • Monitor and download output files from HPC jobs efficiently.
• User Monitoring: Monitor logged-in users for better oversight and security and license usage analysis

The EF Portal REST API is built on OpenAPI/Swagger standards, offering unparalleled ease of integration into workflows. It supports:

• Cross-scheduler and cross-platform compatibility for diverse environments.
• Real-time monitoring of jobs and resources.
• Seamless interaction with third-party tools based on industry-standard protocols.

With the REST API you can extend automation a step further and push the boundaries towards an end-to-end HPC automated workflow: from job submit to monitoring, to downloading results, to post-processing.

Developer-Friendly Features

With support for popular programming languages like Python, PHP, Node.js, Go, Rust, and more, developers can quickly generate EF Portal REST clients tailored to their needs. For simpler use cases, direct CURL access is also available.

EF Portal 2025.0 empowers users with simplified integration processes, enhanced automation capabilities, and real-time monitoring—all designed to optimize your HPC workflows and improve operational efficiency.

The new EF Portal client – efpclient – is implemented in Python and offers easy consumption of the REST API via an auto-completing client supporting all REST API features.

Below is an overview of EF Portal REST API endpoints and features.

API Structure

The EF Portal API is organized into three distinct categories for easy navigation and use:

1. HPC Schedulers: Manage hosts, jobs and queues.
2. Monitor: Access user information and manage licenses (admin-only operations).
3. System: Work with services, spoolers, and file operations.

Key Features

Grid Operations

• Host and queue management
  • List hosts
  • Get details of a host
  • Get details of a cluster in a scheduler 
• Job control
  • List all jobs
  • List jobs belonging to a user
  • List jobs running on a specific host
  • Get details of a job
  • Delete a job
  • Resume a job 
  • Suspend a job
  • Find a job related to a spooler

System Operations

• Create an REST API Access Token
• Service management
  • Submit a Service including file upload
  • List options of a service
• Spooler control and file management
  • List all Spoolers
  • Remove a Spooler
  • List files in a Spooler
  • Upload file into a spooler

Monitor Operations (Admin Only)

• User session management (List users logged in, logout a user)
• License control and usage monitoring

The REST API Documentation including the OpenAPI/Swagger openapi.json definition can be found at docs/rest/ in the EF Portal installation, e.g. https://efp:8443/enginframe/docs/rest/.

EF Portal Client – efpclient

Here you can find the documentation of the new EFPClient implemented in Python in PyPi and then download and install the efpclient: https://pypi.org/project/efpclient.The efpclient is also available as a downloadable package. 

Here is an overview of typical EF Portal Client use cases: 

• Create a New Authentication Token
• Get all Jobs on the default HPC Scheduler or VDI Session Manager
• Get Info for Job with id 142
• List all Services available for the user in the HPC Applications Portal
• Describe a Service available in the HPC Applications Portal
• Submit the job.submission Service Using local file $HOME/file.log and Compression Level 4
• List Files in Spooler after a Service Submission
• Download a File from the Spooler
• Upload a File into the Spooler
• List Spoolers

The EFPClient supports the same configuration file like the efclient (EnginFrame WS Client) in the past for backward compatibility and a new configuration file: $HOME/.efpclient.conf.

Here we show the commands of a typical use case in the shell or a script:

# Install the EFP Client 
python3 -m pip install efpclient
# Start the client command
> efpclient
Commands:
  admin     Administration related commands group.
  clusters  Clusters related commands group.
  hosts     Hosts related commands group.
  jobs      Jobs related commands group.
  queues    Queues related commands group.
  services  Services related commands group.
  spoolers  Spoolers related commands group.
  token     Authentication token commands group.

# Get the REST API token
> efpclient token create --token-only
[https://demo.ni-sp.com:8448/enginframe] Enter the username to authenticate: username
[https://demo.ni-sp.com:8448/enginframe] Enter the password to authenticate:
20d00e78fd87cc6179e16e7b02fe1d1427ce340d
# Or in one line with the credentials in the config file only readable for the user: 
# export EFP_CLIENT_TOKEN=$(efpclient token create --token-only)
> efpclient jobs all
[
    {
        "manager": "slurm",
        "id": "85",
        "name": "Job_Compress_Image_078.png",
        "owner": "efadmin",
        "queue": "test",
        "total_cpu_usage": "0:01",
        "memory_usage": "0",
        "swap_usage": "0",
        "execution_hosts": "demo",
        "submission_time": "2025-04-01T11:13:53",
        "execution_time": "2025-04-01T11:13:53",
        "execution_directory": "/opt/nisp/enginframe/spoolers/efadmin/tmp3093980369702684817.ef",
        "nice": "0",
        "reasons": [
            {
                "value": "None"
            }
        ],
        "status": {
            "ef": "Done",
            "grid": "COMPLETED",
            "value": "DONE"
        }
    }]
> efpclient hosts
Commands:
  all   List all available hosts.
  info  Get host info for a given host name.
  jobs  List jobs by host name, same as 'jobs host' command.
# Enable autocompletion
eval "$(_EFPCLIENT_COMPLETE=bash_source efpclient)"
efpclient hosts <tab>
all   info  jobs   # output from the auto completion

# Configure the HPC Applications SDF in $HOME/.efpclient.conf:
# sdf = https://demo.ni-sp.com/enginframe/applications/applications.xml
# List the services available
> efpclient services list
[
    {
        "id": "batch_builtin_jupyter_notebook.published",
        "name": "Jupyter Notebook",
        "uri": "//applications/batch_builtin_jupyter_notebook.published"
    },
    {
        "id": "batch_builtin_sample_compress_job.published",
        "name": "Sample Compress Job",
        "uri": "//applications/batch_builtin_sample_compress_job.published"
    }
]
> efpclient services describe -s  //applications/batch_builtin_sample_compress_job.published
{
    "id": "batch_builtin_sample_compress_job.published",
    "uri": "//applications/batch_builtin_sample_compress_job.published",
    "name": "Sample Compress Job",
    "options": [
        {
            "id": "file",
            "label": "File to compress:",
            "type": "sfu",
            "option": "--opt-file FILENAME",
            "help": "File to compress:. Specify a single local file to upload."
        },
        {
            "id": "level",
            "label": "Compression level:",
            "type": "list",
            "value": "9",
            "choices": [
                {
                    "value": "9",
                    "label": "maximum"
                },
                {
                    "value": "4",
                    "label": "medium"
                },
                {
                    "value": "1",
                    "label": "minimum"
                }
            ],
            "option": "--opt-level VALUE_1|VALUE_2|...",
            "help": "Compression level:. Default value is: 9. Valid values to choose from are: 9, 4, 1"
        },
        {
            "id": "cluster",
            "label": "Execution cluster:",
            "type": "list",
            "value": "cluster:slurm",
            "choices": [
                {
                    "value": "cluster:slurm",
                    "label": "cluster"
                }
            ],
            "option": "--opt-cluster VALUE_1|VALUE_2|...",
            "help": "Execution cluster:. Default value is: cluster:slurm. Valid values to choose from are: cluster:slurm"
        }
    ],
    "actions": [
        {
            "id": "submit",
            "result": "text/xml"
        }
    ]
}
# Submit the compress service with file results-112.out
> efpclient services submit -s  //applications/batch_builtin_sample_compress_job.published --opt-file results-112.out --opt-level 4
{
    "uri": "spooler:///opt/nisp/enginframe/spoolers/efadmin/tmp9559103907370519188.ef",
    "output": "/enginframe/rest/system/services?_uri=//com.enginframe.system/show.spooler&_spooler=spooler%3A%2F%2F%2Fopt%2Fnisp%2Fenginframe%2Fspoolers%2Fefadmin%2Ftmp9559103907370519188.ef"
}
# Use the Spooler URI to show the files in the Spooler
> efpclient spoolers files -u spooler:///opt/nisp/enginframe/spoolers/efadmin/tmp9559103907370519188.ef
[
    {
        "path": "/",
        "vroot": "8b45963b96801790f315dff51718ca9cb23db834",
        "name": "results-112.out.gz",
        "type": "file",
        "modified": "2025-04-08T11:31:00+00:00",
        "url": "https://demo.ni-sp.com/enginframe/download?file=/8b45963b96801790f315dff51718ca9cb23db834//results-112.out.gz&_spooler=spooler:///opt/nisp/enginframe/spoolers/efadmin/tmp9559103907370519188.ef&_size=51&_plugin=fm",
        "size": {
            "value": 51,
            "unit": "bytes"
        }
    },
    {
        "path": "/",
        "vroot": "8b45963b96801790f315dff51718ca9cb23db834",
        "name": "results-87.out",
        "type": "file",
        "modified": "2025-04-08T11:31:00+00:00",
        "url": "https://demo.ni-sp.com/enginframe/download?file=/8b45963b96801790f315dff51718ca9cb23db834//results-87.out&_spooler=spooler:///opt/nisp/enginframe/spoolers/efadmin/tmp9559103907370519188.ef&_size=0&_plugin=fm",
        "size": {
            "unit": "bytes"
        }
    }
]
# Download a file from the Spooler
> efpclient spoolers download -u spooler:///opt/nisp/enginframe/spoolers/efadmin/tmp9559103907370519188.ef -f results-112.out.gz
Content saved to results-112.out.gz

The EF Portal REST API offers a powerful and comprehensive control of EF Portal via own API clients or the Python-based efpclient installable from PyPi or as download.

EF Portal Modern Look and Feel 

The EF Portal look and feel has been updated to a more modern design. Here are some examples: 

The former nice-jump theme is still available in case of interest and can easily be configured.

Configurable Quick Commands for Host Information

To access information on Linux cluster hosts quickly EF Portal 2025.0 introduces quick commands available in the upper right of the respective host information. This is supported for Linux schedulers and access to the hosts can be configured via ssh or a scheduler command.

By default ps, top, free, w, df are configured: 

The output of the “top” command e.g. looks like this:  

Using either one of the following conf file:

• $EF_ROOT/plugins/hydrogen/conf/host.commands.conf (preinstalled)
• $EF_CONF_ROOT/plugins/hydrogen/host.commands.conf

admins can define one or more global actions to run custom commands on the host the user is viewing, and display the command output in a dialog window. Quick Commands are available to admins only by default.

Administrators can also allow regular users to use and also override admin defined commands, setting the configuration parameter HY_HOST_INFO_COMMANDS_ADMIN_ONLY to “false” in either $EF_ROOT/plugins/hydrogen/conf/host.commands.conf or $EF_CONF_ROOT/plugins/hydrogen/host.commands.conf. In this case, users can define their actions using the file $HOME/.ef/host.commands.conf. 

New commands can be added with the syntax “description|command|question”. A question can be configured for further input to the command which is stored in %INPUT% and can then be used in the command replacing the value. Pipes are allowed in the command as well escaped with \|. 

The remote command to execute can be configured. The default is ssh with examples for SLURM and LSF.  

Every admin user – and standard users as well in case allowed – can configure their own commands overwriting the system-wide configured commands in the file ~/.ef/host.commands.conf. 

Example configuration in host.commands.conf:

# System processes
PS|ps -ef
W|w
# Performance monitoring
Top|top -b -n 1 -w 120 --sort %CPU
Free|free -h
# Storage information
DF|df -h
DF Grep|df -h \| grep %INPUT%|Grep for
# SLURM
#sinfo|sinfo -l
#show hosts|scontrol show nodes
#show node|scontrol show node=%INPUT%|Which host to show?
# Hardware information
#lspci|lspci
#lsusb|lsusb

In ui.hydrogen.conf ($EF_ROOT/plugins/hydrogen/conf/ui.hydrogen.conf or $EF_CONF_ROOT/plugins/hydrogen/ui.hydrogen.conf) the Host Info functionality can be configured: 
# Host Info
HY_HOST_INFO_SERVICE_URI="//ui.hydrogen/grid.host.info.data"
HY_HOST_INFO_COMMANDS_CONF="host.commands.conf"
HY_HOST_INFO_COMMANDS_ADMIN_ONLY="true"
HY_HOST_INFO_COMMANDS_REMOTE_CMD="ssh"         # ssh from EF server as logged in user
# HY_HOST_INFO_COMMANDS_REMOTE_CMD="srun -w"   # SLURM
# HY_HOST_INFO_COMMANDS_REMOTE_CMD="lsrun -m"  # LSF

Prevent File Downloads by Optional Allowlist

To prevent download, preview or streaming of files via EF Portal and allow only specific users in case to download files a new feature and configuration files have been introduced: 

• $EF_ROOT/plugins/fm/conf/fm.download.allowed.conf
• $EF_CONF_ROOT/plugins/fm/fm.download.allowed.conf

# Every line is a user or a regex matching a set of user
# Comments with # are supported as part of the configuration
#efadmin
#ef.*
#root

Limiting the Hosts Displayed

In certain cases the host list should only show a limited set of hosts. This can be implemented with the new configuration introduced via 

• $EF_ROOT/plugins/hydrogen/conf/list.hosts.custom.filter.conf
• $EF_CONF_ROOT/plugins/hydrogen/conf/list.hosts.custom.filter.conf

With this configuration only hosts matching certain regexps are shown in the Custom View available in the Filters on the right side: 

# Here you can configure a custom filter for List Hosts view
# to show only a selected group of hosts based on their names.
# Every line not commented is a host name or a regex matching a set of host names.
# Examples:
#   node-1
#   node-2
#   node-[3-4]
#   node-[1-2,6-7].*
demo
demo[4-5]

So the standard display of hosts changes from: 

To the filtered host list when selecting “Custom View”:

The custom view can be selected as default view with a configuration in ui.hydrogen.conf.

Jupyter Notebook Integration

EF Portal now offers Jupyter Notebook integration with a new service included in the EF Portal services distribution. Using the Jupyter Notebook service the user can upload a notebook or select a notebook on the server side via Remote File Browser: 

The Notebook service supports specifying a default port which will be used or the service will use the next free portal found. The NAT translation via the NAT configuration file configurable in the Configuration of the service allows to configure the nat-translated URL at which the Notebook can be reached.

When the service has started a notebook.url file is created which allows to directly access the Notebook via the Preview function of the EF Portal File Manager: 

The screenshot shows the selected notebook, the output of the Job starting the Jupyter Notebook and the Preview URL.

Other updates

Other Features and Improvements

• A new Star-CCM+ service template has been added.
• File Manager: Display of clickable URLs for files ending with .url
• File Manager: added more file types to preview as text files
• VDI: Connection Quick Access bar in session preview can be hidden via configuration

Security / Upgrades

• Upgraded Apache Tomcat to latest version 9.0.102
• Upgraded MyBatis to 3.5.6
• Upgraded Apache Commons IO to 2.18.0
• Upgraded Google Guava to 32.0.0-jre
• Upgraded Xalan-J to 2.7.3
• Upgraded Xalan-J Serializer to 2.7.3
• Upgraded Xerces2-J to 2.12.2