Friday 24 November 2017

OIM 12c Server unable to start and throws error "Issue seen during keystores generation.. Keystore was tampered with, or password was incorrect"

Environment: Windows, OIM 12.2.1.3.0

1. Topology Details :-

TopologyStatus
Clusterfalse
SSLfalse
Upgrade Setupfalse

2. System Level Details :-

SysLevelDetailPath
JAVA_HOMED:\Java\jdk1.8.0_151
OIM_HOMED:/idm12c/idm/server
JDK Version1.8.0_151
MIDDLEWARE_HOMED:/idm12c
DB VersionOracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
DOMAIN_HOMED:/idm12c_config/iam_domain

Issue: OIM managed server not able to start

Error in logs: 

(self-tuning)'] [userId: <anonymous>] [ecid: 5c15e6e0-8cf5-4cc9-b3ae-76813d10f682-0000000b,0] [APP: oim] [partition-name: DOMAIN] [tenant-name: GLOBAL] [[
Issue seen during keystores generation..
Keystore was tampered with, or password was incorrect
]]
[2017-11-22T16:30:39.237+05:30] [oim_server1] [ERROR] [] [oracle.iam.OIMPostConfigManager] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <anonymous>] [ecid: 5c15e6e0-8cf5-4cc9-b3ae-76813d10f682-0000000b,0] [APP: oim] [partition-name: DOMAIN] [tenant-name: GLOBAL] [[
Issue seen during keystores generation..
]]
[2017-11-22T16:30:39.239+05:30] [oim_server1] [ERROR] [] [oracle.iam.OIMPostConfigManager] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <anonymous>] [ecid: 5c15e6e0-8cf5-4cc9-b3ae-76813d10f682-0000000b,0] [APP: oim] [partition-name: DOMAIN] [tenant-name: GLOBAL] Exception occurred during generation of keystores.[[
java.lang.Exception: Exception occurred during generation of keystores.
                at oracle.iam.OIMPostConfigManager.config.util.EncryptConfigurationAndDB.encryptConfiguration(EncryptConfigurationAndDB.java:161)
                at oracle.iam.OIMPostConfigManager.config.OIMConfigManager.encryptConfiguration(OIMConfigManager.java:5319)
                at oracle.iam.OIMPostConfigManager.config.OIMConfigManager.executeAndRegisterTask(OIMConfigManager.java:1634)
                at oracle.iam.OIMPostConfigManager.config.OIMConfigManager.configureOIM(OIMConfigManager.java:1556)
                at oracle.iam.OIMPostConfigManager.config.OIMConfigManager.doExecute(OIMConfigManager.java:1175)
                at oracle.iam.OIMPostConfigManager.appListener.BootStrapListener.preStart(BootStrapListener.java:128)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at weblogic.application.internal.flow.BaseLifecycleFlow$LifecycleListenerAction.run(BaseLifecycleFlow.java:240)
                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326)
                at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
                at weblogic.application.internal.ApplicationContextImpl$WLSSecurityProvider.invokePrivilegedAction(ApplicationContextImpl.java:1594)
                at weblogic.application.internal.flow.BaseLifecycleFlow$LifecycleListenerAction.invoke(BaseLifecycleFlow.java:201)
                at weblogic.application.internal.flow.HeadLifecycleFlow.prepare(HeadLifecycleFlow.java:245)
                at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:729)
                at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45)
                at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:241)
                at weblogic.application.internal.EarDeployment.prepare(EarDeployment.java:66)
                at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:158)
                at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:65)
                at weblogic.deploy.internal.targetserver.AppDeployment.prepare(AppDeployment.java:166)
                at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:41)
                at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:193)
                at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:31)
                at weblogic.management.deploy.internal.ConfiguredDeployments$2.doItem(ConfiguredDeployments.java:741)
                at weblogic.management.deploy.internal.parallel.BucketInvoker.invoke(BucketInvoker.java:138)
                at weblogic.management.deploy.internal.ConfiguredDeployments.transitionAppsParallel(ConfiguredDeployments.java:749)
                at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:363)
                at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:232)
                at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:219)
                at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:133)
                at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:76)
                at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1287)
                at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:333)
                at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
                at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
                at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
                at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
                at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
                at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
                at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:698)
                at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:78)
                at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:212)
                at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:235)
                at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:358)
                at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
                at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
                at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
                at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
                at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
                at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
                at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1237)
                at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1168)
                at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:786)
                at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:670)
                at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
                at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
                at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
                at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
                at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
                at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)

]]
[2017-11-22T16:30:39.334+05:30] [oim_server1] [NOTIFICATION] [] [oracle.iam.OIMPostConfigManager] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <anonymous>] [ecid: 5c15e6e0-8cf5-4cc9-b3ae-76813d10f682-0000000b,0] [APP: oim] [partition-name: DOMAIN] [tenant-name: GLOBAL] [[
[OIM_CONFIG] Configuration Encryption failed.
]]
[2017-11-22T16:30:39.335+05:30] [oim_server1] [ERROR] [] [oracle.iam.OIMPostConfigManager] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <anonymous>] [ecid: 5c15e6e0-8cf5-4cc9-b3ae-76813d10f682-0000000b,0] [APP: oim] [partition-name: DOMAIN] [tenant-name: GLOBAL] Exception[[
java.lang.Exception: Exception occured while encrypting the configuration
                at oracle.iam.OIMPostConfigManager.config.util.EncryptConfigurationAndDB.encryptConfiguration(EncryptConfigurationAndDB.java:181)
                at oracle.iam.OIMPostConfigManager.config.OIMConfigManager.encryptConfiguration(OIMConfigManager.java:5319)
                at oracle.iam.OIMPostConfigManager.config.OIMConfigManager.executeAndRegisterTask(OIMConfigManager.java:1634)
                at oracle.iam.OIMPostConfigManager.config.OIMConfigManager.configureOIM(OIMConfigManager.java:1556)
                at oracle.iam.OIMPostConfigManager.config.OIMConfigManager.doExecute(OIMConfigManager.java:1175)
                at oracle.iam.OIMPostConfigManager.appListener.BootStrapListener.preStart(BootStrapListener.java:128)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at weblogic.application.internal.flow.BaseLifecycleFlow$LifecycleListenerAction.run(BaseLifecycleFlow.java:240)
                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326)
                at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
                at weblogic.application.internal.ApplicationContextImpl$WLSSecurityProvider.invokePrivilegedAction(ApplicationContextImpl.java:1594)
                at weblogic.application.internal.flow.BaseLifecycleFlow$LifecycleListenerAction.invoke(BaseLifecycleFlow.java:201)
                at weblogic.application.internal.flow.HeadLifecycleFlow.prepare(HeadLifecycleFlow.java:245)
                at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:729)
                at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45)
                at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:241)
                at weblogic.application.internal.EarDeployment.prepare(EarDeployment.java:66)
                at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:158)
                at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:65)
                at weblogic.deploy.internal.targetserver.AppDeployment.prepare(AppDeployment.java:166)
                at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:41)
                at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:193)
                at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:31)
                at weblogic.management.deploy.internal.ConfiguredDeployments$2.doItem(ConfiguredDeployments.java:741)
                at weblogic.management.deploy.internal.parallel.BucketInvoker.invoke(BucketInvoker.java:138)
                at weblogic.management.deploy.internal.ConfiguredDeployments.transitionAppsParallel(ConfiguredDeployments.java:749)
                at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:363)
                at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:232)
                at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:219)
                at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:133)
                at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:76)
                at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1287)
                at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:333)
                at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
                at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
                at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
                at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
                at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
                at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
                at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:698)
                at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:78)
                at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:212)
                at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:235)
                at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:358)
                at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
                at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
                at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
                at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
                at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
                at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
                at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1237)
                at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1168)
                at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:786)
                at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:670)
                at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
                at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
                at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
                at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
                at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
                at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
Caused by: java.lang.Exception: Exception occurred during generation of keystores.
                at oracle.iam.OIMPostConfigManager.config.util.EncryptConfiguratio
nAndDB.encryptConfiguration(EncryptConfigurationAndDB.java:161)

                ... 66 more

Cause:        Keystore password entered during oim configuration not working.




Workaround: 


Step 2: Update keystore credential in enterprise manager retrieved from Step 1

Step 3: Restart OIM managed server.

Step 4: Validate Bootstrap Repot in OIM logs folder 
%DOMAIN HOME%/servers/oim_server1/logs/BootStrapReportPreStart.html

BOOTSTRAP REPORT


VALIDATION DETAILS

1. Connectivity Check :-

Connection Validation DetailStatus
SOA ConnectivityUp
Admin ConnectivityUp
DB ConnectivityUp

EXECUTION DETAILS :-

TaskNameStatusExecution Time(in ms)
SOAPRESTARTCOMPLETE12390
CREATEPOLICYSETCOMPLETE1479
COPYMBEANSCOMPLETE3
OIMCONFIGXMLUPDATECOMPLETE3074
IMPORT_KEYSTORESCOMPLETE1022
ENCRYPT_CONFIGURATIONCOMPLETE868
LOADXMLMETADATACOMPLETE1338
SEEDADAPTERSCOMPLETE46705
SEEDANDCONFIGURESCHEDULERCOMPLETE17744
ENCRYPT_DBCOMPLETE7283
NOTIFICATIONTEMPLATECOMPLETE3237
UPDATEWEBLOGICADMINCOMPLETE0