Evil_TTL> show | s

ACS Troubleshooting

Category:Cisco Systems -> Security

ACS GUI not responding

Application process boot up review:

# show application status acs

ACS rolePRIMARY

Process 
'database'                  running
Process 
'management'                running (HTTP is nonresponsive)
Process 'runtime'                   running
Process 
'ntpd'                      running
Process 
'view-database'             Restarting
Process 
'view-jobmanager'           not monitored
Process 
'view-alertmanager'         not monitored
Process 
'view-collector'            initializing
Process 
'view-logprocessor'         not monitored

# show application status acs

ACS rolePRIMARY

Process 
'database'                  running
Process 
'management'                running (HTTP is nonresponsive)
Process 'runtime'                   running
Process 
'ntpd'                      running
Process 
'view-database'             Restarting
Process 
'view-jobmanager'           not monitored
Process 
'view-alertmanager'         not monitored
Process 
'view-collector'            initializing
Process 
'view-logprocessor'         initializing


# show application status acs

ACS rolePRIMARY

Process 
'database'                  running
Process 
'management'                running (HTTP is nonresponsive)
Process 'runtime'                   running
Process 
'ntpd'                      running
Process 
'view-database'             Restarting
Process 
'view-jobmanager'           initializing
Process 
'view-alertmanager'         initializing
Process 
'view-collector'            initializing
Process 
'view-logprocessor'         initializing

# show application status acs

ACS rolePRIMARY

Process 
'database'                  running
Process 
'management'                running (HTTP is nonresponsive)
Process 'runtime'                   running
Process 
'ntpd'                      running
Process 
'view-database'             Restarting
Process 
'view-jobmanager'           not monitored
Process 
'view-alertmanager'         not monitored
Process 
'view-collector'            initializing
Process 
'view-logprocessor'         not monitored

# show application status acs

ACS rolePRIMARY

Process 
'database'                  running
Process 
'management'                running (HTTP is nonresponsive)
Process 'runtime'                   running
Process 
'ntpd'                      running
Process 
'view-database'             not monitored
Process 
'view-jobmanager'           not monitored
Process 
'view-alertmanager'         not monitored
Process 
'view-collector'            running
Process 
'view-logprocessor'         not monitored 

In the long run HTTP became responsive but the database process remained “non monitored”. The ACS server was unmanageable but responded well to authentication requests.

How to start individual processes

# sh application status acs

ACS rolePRIMARY

Process 
'database'                  running
Process 
'management'                running
Process 
'runtime'                   running
Process 
'ntpd'                      running
Process 
'view-database'             not monitored
Process 
'view-jobmanager'           not monitored
Process 
'view-alertmanager'         not monitored
Process 
'view-collector'            running
Process 
'view-logprocessor'         not monitored

# acs start ?
  
adclient           Start adclient
  database           Start database
  management         Start management
  runtime            Start runtime
  view
-alertmanager  Start view-alertmanager
  view
-collector     Start view-collector
  view
-database      Start view-database
  view
-jobmanager    Start view-jobmanager
  view
-logprocessor  Start view-logprocessor
  
<cr>               Carriage return.

# acs start view-logprocessor
Starting view-logprocessor
# sh application status acs

ACS rolePRIMARY

Process 
'database'                  running
Process 
'management'                running
Process 
'runtime'                   running
Process 
'ntpd'                      running
Process 
'view-database'             not monitored
Process 
'view-jobmanager'           not monitored
Process 
'view-alertmanager'         not monitored
Process 
'view-collector'            running
Process 
'view-logprocessor'         running 

I was not able to start the database process this way. It was stuck in “not monitored” state.

Checked the logs:

#sh logging application MonitoringAndReportingDatabase.log | b ERROR
E08/27 03:27:25. *** ERROR *** Assertion failed201129 (12.0.1.3798)[tempdb]
File is shorter than expected 

Let’s see what Google says about this error:

“File is shorter than expected” can happen when the database file’s file-system metadata has not been flushed to disk correctly and power is lost to the drive.

In my case it was a routine HA test process without powercycling. The ACS server was restared from CLI and never returned to normal state.

ACS Version : 5.5.0.46

One of the possible solutions is to either restore it from a backup copy or re-image entirely.

By privilege15