logback config

The log output for the morpheus-ui service is configured in the logback.xml file. Log output levels can be updated when more or less log output is desired.

Setting Log Levels

To change a log level, edit the logback configuration file in /opt/morpheus/conf/logback.xml and save. The changes will be reflected within the configured scanPeriod, 30 seconds by default.

Levels:

OFF (no log output)
ERROR (includes error logs)
WARN (includes warn and error logs)
INFO (includes info, warn and error logs)
DEBUG (includes info, warn, error and debug logs)
TRACE (includes info, warn, error, debug and trace logs)

Ansible Troubleshooting

When a workflow is executed manually, the Ansible run output is available in the Instance History tab. Select the i bubble next to the Ansible task to see the output. You can also see the run output in UI logs at /var/log/morpheus/morpheus-ui/current. These can be tailed by running morpheus-ctl tail morpheus-ui.

Verify Ansible is installed on the Morpheus Appliance

Ansible should be automatically installed but certain OS or network conditions can prevent the automated install. You can confirm installation by running ansible --version in the Morpheus appliance, or by viewing the Ansible integration details page (Administration > Integrations > Select Ansible Integration). We also see it in the Ansible tab of a Group or Cloud scoped to Ansible, just run --version as ansible is already included in the command.

Validate Git repo authorization and the configured paths

The public and private SSH keys need to be added to the Morpheus appliance via Infrastructure > Keys & Certs and the public key needs to be added to the Git repo via user settings. If both are set up correctly, you will see the playbooks and roles populate in the Ansible Integration details page.

The Git Ref field on playbook tasks is to specify a different git branch than default. It can be left to use the default branch. If your playbooks are in a different branch you can add the brach name in the Git Ref field.

When running a playbook that is in a workflow, the additional playbooks fields do not need to be populated, they are for running a different playbook than the one set in the Ansible task in the Workflow, or using a different Git Ref.

Attaching Logs to Case

When submitting a case it is critical to attach the relevant logs. The logs can be found at /var/log/morpheus/morpheus-ui/current. Logs can be attached to the case at anytime.

When submitting logs please reproduce the error right before capturing and sending the log file. This will ensure the activity that took place and resulted in an error is contained in the logs.

Log rotation takes the current file each night or after it’s a certain size and compresses them. The *.s files in the current directory are rotated and zipped logs that can be sent as is.

The logs can also be captured from the Morpheus UI. Under Administration > Health > Morpheus Logs. Please copy relevant logs and add to case as an attachment.

../_images/Morpheus-Health-Logs.png

Cannot Login

Forgot password

If a user forgets their password, they can use the FORGOT PASSWORD? link on the login page. They can then enter their username or email address to send a reset password email to the email address defined on the user.

If the default or user added SMTP server is not functioning or blocked, a System Admin user can impersonate that user and update their password.

If the System Admin user password needs to be reset and the default or user added SMTP server is not functioning or blocked, please contact Morpheus support for assistance.

Sub-Tenant user cannot login after 3.4.0 upgrade

Morpheus v3.4.0 added support for all subtenant users to login via the main tenant url using subtenant id or subdomain prefix, ie tenantId\username or subdomain\username.

The login requirements were added in v3.4.0 to allow subtenant users with identity source integration generated user accounts to be able to login to the master tenant, gain API and CLI access, and remove the requirement for usernames to be unique across all tenants.

Previously subtenant users that had local/morpheus generated user accounts could login to their tenant via the master tenant url, while subtenant users that had identity source integration generated user accounts had to use the subtenant specific login url.

In v3.4.0+ all subtenant users can login via the master tenant url by specifying their tenant id or subdomain prefix, \, then username. Subtenants can still use the tenant specific login url as well.

Active Directory user suddenly cannot Login

In Morpheus v3.4.0 and prior, OU changes in Active Directory can disable logins for AD users who had previously authenticated/have existing user accounts in Morpheus. If an Active Directory user cannot login to Morpheus after their OU was changed in AD, please contact Morpheus support for a resolution. The OU association for the user(s) can also be manually updated in the database. This issue is resolved in Morpheus versions 3.4.1 and higher.

How to un-manage an Instance/VM/Host

Description

A managed VM (and associated Instance) needs to be unmanaged and returned to Discovered type.

Solution

Delete the record from the Infrastructure > Compute (! not from Provisioning - Instances) selection with the following configuration in the Delete modal:

Remove Infrastructure UNCHECKED
Remove Associated Instances Must be checked if the server has an associated Instance, as deleting the VM but not the Instance would result in an abandoned Instance thus not allowed.
Force Delete UNCHECKED

The most important items to be aware of when “un-managing” an Instance/VM/Host are:

The “Remove from Infrastructure” flag when deleting a VM or Host in Morpheus determines if the actual VM is deleted from the target Infrastructure.
- Checking “Remove Infrastructure” means you WANT TO DELETE THE ACTUAL VM. Typing “DELETE” in the confirmation field is required when “Remove From Infrastructure” is enabled.
- Unchecking “Remove Infrastructure” means you only want to delete the record in Morpheus but leave the actual VM untouched.
Deleting an Instance will always remove Infrastructure.
After removing the record from Morpheus, the VM must be in a Cloud with Inventory enabled to automatically be re-discovered.

Process

Steps to delete a managed VM from Morpheus and, when necessary, remove the associated Instance:

Navigate to the VM (not Instance) detail page at Infrastructure > Compute - VMs
Select DELETE
Configure the DELETE HOST modal with the following settings:
- Remove Infrastructure UNCHECKED
- Remove Associated Instances Must be checked if the server has an associated Instance, as deleting the VM but not the Instance would result in an abandoned Instance thus not allowed.
- Force Delete UNCHECKED.
Select DELETE
The VM and associated Insatnce will be removed from Morpheus but the actual VM will remain.
Wait up to 5 min or click REFRESH on the associated Clouds details page to force a cloud sync.
The VM is now back in Morpheus as discovered/unmanaged. To managed and create a new Instance from the VM, select ACTIONS : Convert To Managed.

MySQL Too many connections error

If you see the following error in the Morpheus UI logs:

SqlExceptionHelper - Data source rejected establishment of connection,  message from server: "Too many connections"

it means the number connections between Morpheus application and mysql have reached the max_connections limit set in mysql (default is 151), or the max_active setting, which limits the number of connections on the Morpheus end (default is 150), and the limit needs to be raised, either in Morpheus or mysql, or both depending on the number of connections and configuration.

Morpheus Agent Install Troubleshooting

When provisioning an Instance, there are network and configuration requirements to consider in order to successfully install the Morpheus Agent. Typically, when a VM Instance is still in the provisioning phase long after the VM is up, the Instance is unable to reach Morpheus. Depending on the Agent install mode, it could also mean Morpheus is unable to reach the Instance.

The most common reason an Agent install fails is the provisioned Instance cannot reach the Morpheus Appliance via the Appliance URL set in Administration > Settings over port 443. When an Instance is provisioned from Morpheus, it must be able to reach the Morpheus appliance via the Appliance URL or the Agent will not be installed.

../_images/agent-7c9a2.png

Agent Install Methods

Morpheus Agent installation supports multiple install methods.

SSH/WinRM
VM Tools
Cloud-Init & Cloudbase-Init
Windows Unattended
Manual

SSH

Port 22 is open for Linux images, and SSH is enabled
Credentials set on the image if using a custom or synced image. Credentials can be entered on images in the Library > Virtual Images section

WinRM

Port 5985 must be open and WinRM enabled for Windows images
Credentials have been entered on the image if using a custom or synced image. Credentials can be entered on images in the Library > Virtual Images section.

Windows Unattended

Windows Administrator Password defined in Administration > Settings > Provisioning section
VMware: Force Guest Customizations set to forced on Virtual Image config when using DHCP (Static Assignment will already force Guest Customizations)
Nutanix & SCVMM: Virtual Image is sysprepped and shutdown, Sysprep Enabled flagged on Virtual Image config

Manual

Agent Install scripts can be downloaded from Morpheus by selecting Actions > Download Agent Script from an Instance detail page, then run manually on the target host when required for a given managed resource. Please note the script will be unique per managed resource and should not be saved to run as needed on any arbitrary resources in the future.

Linux

On the target host, run sudo morpheus-node-ctl restart morphd and the Morpheus agent will restart. morpheus-node-ctl status will also show the agent status.

Windows

The Morpheus Windows Agent service can be restarted in Administrative Tools > Services.

CentOS/RHEL 7 Images

For custom CentOS 7 images we highly recommend setting up Cloud-Init and fixing the network device names. More information for custom CentOS images can be found in the CentOS 7 image guide.

Morpheus UI not loading after upgrade or reconfigure

Problem:

The Morpheus ui does not load after performing an upgrade.

Common Causes:

The morpheus-ui has not finished loading
The morpheus-ui was not fully stopped before reconfigure, or not started after reconfigure
Morpheus was forced to restart or shut down while the database schema was being migrated during an upgrade.

Remote Console

Morpheus has a built in Remote Console for Instances, Hosts, Virtual Machines and Bare Metal. The following information reviews the Roles Settings, Protocols, and Requirements necessary to configure and troubleshoot Remote Console access.

Role Settings

User Role settings determine if the Console tab or Open Console Action appear for a user, and if a login prompt is presented or the user is automatically logged in when using the Console.

Remote Console (None, Provisioned, Full)

None

The user will not have access to remote console.

Provisioned

The user will only have remote console access for Instances they provisioned.

Full

The user will have remote console access for all instances they have access to.
Remote Console: Auto Login (No, Yes)
No

A login prompt will be present in the console for Linux platforms, and the main login screen will present for Windows platforms.

Yes
Morpheus will automatically login to the remote console using the credentials defined on the VM or Host. For provisioned Instances, the credentials are defined either from the credentials defined on the Virtual Image used, added via cloud-init or VMware Tools using the global cloud-init settings (Administration - Provisioning) or the Linux or Windows settings defined in User Settings. For Instances created when converting a VM or Host to managed, the credentials are entered when converting to managed. These credentials can be changed by editing the underlying VM or Host of the Instance.

Protocols

Platform Type and Cloud Settings determines the protocol and port used for Remote Console connections.

SSH

The SSH protocol will be used for Linux and OSX platform types, and 22 is the default port used.

RDP

The RDP (Remote Desktop) protocol will be used for Windows platform types over port 3389 by default.

VNC

The VNC protocol will be used for all platform types in Clouds with the Hypervisor Console option enabled in cloud settings. VNC connection are made directly to the Hypervisor Host over port 443.

SSH

For all Linux and OSX platform types, Morpheus will use the SSH protocol via port 22 by default for Remote Console connections, unless the Hypervisor Console` option is enabled for VMware type clouds.

Morpheus will SSH using the username, password, RPC Host IP address and Port defined in the VM or Host record.

Default Requirements for SSH Connectivity

SSH Enabled on the target VM or Host

Port 22 incoming open on the target VM or Host firewalls and security groups from the Morpheus Appliance (not from the users IP address)

An IP address defined on the VM or Host record that is routable from the Morpheus Appliance.

Valid credentials defined on the VM or Host record in the RPC host field.

Remote Console Role Permissions set to Provisioned or Full if the User provisioned the instance, or Full if the user did not provision the instance.

RDP

For all Windows platform types, Morpheus will use the RDP protocol via port 3389 by default for Remote Console connections, unless the Hypervisor Console` option is enabled for VMware type clouds.

Morpheus will RDP using the username, password, RPC Host IP address and Port defined in the VM or Host record.

Default Requirements for RDP Connectivity

Remote Access enabled on the target VM or Host and Remote Desktop enabled in the Windows Firewall settings. If the VM or Host is on a different network than the Morpheus appliance, public access for Remote Desktop must be enabled in the Firewall settings.

Port 3389 incoming open on the target VM or Host firewalls and security groups from the Morpheus Appliance (not from the users IP address)

An IP address defined on the VM or Host record that is routable from the Morpheus Appliance.

Valid credentials defined on the VM or Host record in the RPC host field.

Remote Console Role Permissions set to Provisioned or Full if the User provisioned the instance, or Full if the user did not provision the instance.

VNC (VMware Hypervisor Console)

When the Hypervisor Console option is enabled in cloud settings, the VNC protocol will be used for all platform types that Cloud.

When using VNC Hypervisor Console, the Morpheus Appliance connects directly to the host the VM is on, not directly to the VM.

Morpheus features Remote Console support directly to hypervisors. To enable this feature a few prerequisites must be met:

The Morpheus Appliance must have network access to the host the VM is on over 443.

The Morpheus Appliance must be able to resolve the hypervisor hostnames.

Guacamole

Overview

Morpheus uses Apache Guacamole, a clientless remote console. Guacamole is installed on the Morpheus Appliance during the initial reconfigure. In Morpheus versions 3.2.0 and higher, Guacamole 0.9.14 is automatically installed. On Morpheus versions older than 3.2.0, 0.9.9 is installed. The 0.9.14 version is required for VNC Hypervisor Console functionality on ESXi v6.5 and later.

The Guacamole proxy daemon, guacd, is used for all Remote Console connections and must be running for Remote Console functionality.

Troubleshooting guacd

If all console connections are not functioning, the Guacamole proxy daemon (guacd) process may not be running or have a stuck process preventing console connections. This is evident when only the header appears in the console tab/window, and no console window appears below the header and no connection status is show in the console header. The following commands can be used on the Morpheus Appliance to restore console functionality.

SSL Self-signed Certificate Regeneration

When Morpheus is deployed it generates a 10 year self-signed non-trusted SSL certificate. Below details the process to regenerate this certificate and key.

Replacing both the certificate and private key

Delete the certificate and key files in /etc/morpheus/ssl/ that end in .crt and .key
Run Reconfigure morpheus-ctl reconfigure
Restart NGINX morpheus-ctl restart nginx

Replacing only the certificate

Delete the certificate file in /etc/morpheus/ssl/ it ends in .crt
Run Reconfigure morpheus-ctl reconfigure
Restart NGINX morpheus-ctl restart nginx

Unable to Delete Tenant

Problem

When trying to delete a tenant, a message stating manage resources must be removed or other error occurs and the tenant is not deleted. The tenant may be stuck in a deleting status or return to OK status after delete attempt.

Cause

All managed resources must be removed from a tenant in order for that tenant to be deleted. This includes instances and their underlying managed vm’s

Solution

Login or impersonate that an Admin user inside the tenant
Navigate to Infrastructure > Hosts
Under Hosts and VM’s, delete any managed resources
- Uncheck remove infrastructure when deleting a VM to only remove it from Morpheus but not from the underlying hypervisor/cloud
- You must check remove associated instances if the VM has an associated instance
- If the VM no longer exists but there is still a record in Morpheus, uncheck remove infrastructure and check force delete
Once all managed resources are removed from the tenant, the tenant can then be deleted
In certain situations other components may prevent a tenant from being deleted. If you have removed all managed resources from a tenant and the tenant still cannot be deleted, please contact Morpheus support.

Unable to Provision a Custom Image

Prior to provisioning an custom image, the image must be configured in the Library > Virtual Images section by selecting Edit on the Actions dropdown of the Virtual Image.

In the Edit Virtual Image pane:

Select “Cloud Init Enabled?” only if the Virtual Image is a linux image with cloud init installed.
Enter the username and password that are set on the Virtual Image.

Variables

A vast number of variables are available for use in Tasks, Scripts, Templates, Resource Names, Cloud-Init User Data and Option List configs.

Pre-Provision Vars

A subset of variables are available for Instance, Host Name and Hostnames. These can be passed inside ${ } blocks during provisioning or in relevant policy configs. Groovy syntax can be resolved to allow for dynamic name generation as shown in some of the examples below.

Instance Naming Policy example: ${userInitials}-${cloudCode}-${platform == 'windows' ? 'W' : 'L'}-${sequence}

Syntax Examples

PowerShell Example: $app_id = "<%= instance.metadata.app_id %>"

Bash Example: HOSTNAME="<%= container.server.hostname %>"

Python Example: hostname = morpheus['server']['hostname']

HTTP Body Example: {"name": "<%= instance.createdByUsername %>"}

../_images/Metadata-Enviornment-Variable-Spot ../_images/Tags-Variable-Spot