EM12c Metric Evaluation Error – Supplied date is in the future

After the  Daylight Saving Time we are getting below metric collection errors from some host targets.

Metric Evaluation Error Details:
Host=oraprod
Target type=Database Instance
Target name=PROD
Categories=Error
Message= Row(1): Supplied date is in the future : now = Wed Nov 10 10:39:01 EST 2015 supplied value = Wed Nov 10 11:38:07 EST 2015
Severity=Critical
Event reported time=Nov 10, 2015 10:39:01 AM EST
Contact=Sivaram
Location=Banalore
Comment=DB archive
Department=CRM
Operating System=Linux
Platform=x86_64
Event Type=Metric Evaluation Error
Event name=wait_bottlenecks
Metric Group=Wait Bottlenecks
Collection Name=wait_bottlenecks_10i
Rule Name=CRM Metric Evaluation Error Any Target
Rule Owner=SYSMAN
Update Details:
Row(1): Supplied date is in the future : now = Wed Nov 10 10:39:01 EST 2015 supplied value = Wed Nov 10 11:38:07 EST 2015

Solution:
***********

Clearing and restarting agents on these hosts will fix the problem and it will disappear.
$ ps -ef|grep pmon
oracle   15676     1  0 Jul28 ?        00:15:26 ora_pmon_prod
oracle   20606 20239  0 04:07 pts/0    00:00:00 grep pmon
$ . oraenv
ORACLE_SID = [oracle] ? prod
The Oracle base has been set to /u01/app/oracle

$ ps -ef|grep emagent

oracle    3390 20239  0 04:31 pts/0    00:00:00 grep emagent
oracle    9521     1  0 Jul15 ?        00:02:12 /u01/app/oracle/product/12.1.0.2/core/12.1.0.4.0/perl/bin/perl

$ cd /u01/app/oracle/product/12.1.0.2/core/12.1.0.4.0/bin
$ ./emctl stop agent
$ ./emctl stop agent
$ ./emctl clearstate agent
$ ./emctl start agent
$ ./emctl upload agent

Happy Learning :)

How to disable the mount point monitoring from the 12c OEM

This post will describe how to disable the mount point monitoring in the 12c.

1. Go to the target and click on the host and select the option as below.

2. Go to the file system section and click on the Pencil symbol which is on right hand side.

3. Select the radio button on which file system you need to disable the monitor. Here i am disabling the monitoring for the /var filesystem.

If the required file system is not shown then click on the add button and select the required file system on which you need to disable the monitoring.

4. Then change the values of warning and critical as null. And then click on continue button.

5. It will ask for the confirmation click on ok button.
6. Again click on ok button so that you changed the threshold values.

Happy Learning :)

Upload timed out before completion. Agent upload status failed

bash-4.1$  ./emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: Upload Recent Sending Statistics
________________________________

Avg send time=0.0 milliseconds
Backoff Event List
________________________________
Upload Failure List
_______________________________
Connection Event List
_____________________________
Upload timed out before completion.
Number of files to upload before the uploadNow call: 4903, total size (MB): 3.049202
Remaining number of files to upload: 4903, total size (MB): 3.049202 (TIMEOUT)

When you got the above error we need to do the as the below.

-bash-4.1$ emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.
-bash-4.1$ pwd
/u01/app/oracle/product/agent12c/agent_inst/sysman/emd

you need to move the upload,colletction and state folders as stated the below.

-bash-4.1$ ls -ltrh
total 72K
drwxr-----  2 oracle dba 4.0K Sep 16  2014 dumps
drwxr-----  4 oracle dba 4.0K Sep 16  2014 upload
drwxr-----  2 oracle dba 4.0K Sep 16  2014 metricExtLib
drwxr-----  2 oracle dba 4.0K Sep 16  2014 attachmentDB
drwxr-----  2 oracle dba 4.0K Sep 16  2014 metricCache
drwxr-----  2 oracle dba 4.0K Sep 16  2014 temp
drwxr----- 12 oracle dba 4.0K Dec 10  2014 state
-rw-r-----  1 oracle dba 5.9K Apr  8 15:32 targets.xml.bak
-rw-r-----  1 oracle dba  910 Jun 26 22:10 blackouts.xml.bak
-rw-r-----  1 oracle dba   13 Jun 27 00:46 blackouts.xml
-rw-r-----  1 oracle dba 3.1K Jun 30 09:06 emagent_storage.config
-rw-r-----  1 oracle dba 5.9K Jul  1 03:48 targets.xml
-rw-r-----  1 oracle dba    0 Jul  1 03:48 blackouts.lck
-rw-r-----  1 oracle dba  282 Jul  1 04:01 agntstmp.txt.bak
drwxr-----  2 oracle dba 4.0K Jul  1 04:03 collection
-rw-r-----  1 oracle dba   23 Jul  1 04:03 ag_current_status.msg
-rw-r-----  1 oracle dba  321 Jul  1 04:03 agntstmp.txt
-bash-4.1$ cd upload
-bash-4.1$ ls -ltrh
total 16K
drwxr----- 10 oracle dba 4.0K Sep 16  2014 upload
drwxr-----  2 oracle dba  12K Jul  1 04:03 pending
-bash-4.1$ mkdir old
-bash-4.1$ pwd
/u01/app/oracle/product/agent12c/agent_inst/sysman/emd/upload
-bash-4.1$ mv upload /u01/app/oracle/product/agent12c/agent_inst/sysman/emd/upload/old
-bash-4.1$ mv pending /u01/app/oracle/product/agent12c/agent_inst/sysman/emd/upload/old
-bash-4.1$ ls -ltrh
total 4.0K
drwxr-xr-x 4 oracle dba 4.0K Jul  1 04:05 old
-bash-4.1$ cd ..
-bash-4.1$ ls -ltrh
total 72K
drwxr-----  2 oracle dba 4.0K Sep 16  2014 dumps
drwxr-----  2 oracle dba 4.0K Sep 16  2014 metricExtLib
drwxr-----  2 oracle dba 4.0K Sep 16  2014 attachmentDB
drwxr-----  2 oracle dba 4.0K Sep 16  2014 metricCache
drwxr-----  2 oracle dba 4.0K Sep 16  2014 temp
drwxr----- 12 oracle dba 4.0K Dec 10  2014 state
-rw-r-----  1 oracle dba 5.9K Apr  8 15:32 targets.xml.bak
-rw-r-----  1 oracle dba  910 Jun 26 22:10 blackouts.xml.bak
-rw-r-----  1 oracle dba   13 Jun 27 00:46 blackouts.xml
-rw-r-----  1 oracle dba 3.1K Jun 30 09:06 emagent_storage.config
-rw-r-----  1 oracle dba 5.9K Jul  1 03:48 targets.xml
-rw-r-----  1 oracle dba    0 Jul  1 03:48 blackouts.lck
-rw-r-----  1 oracle dba  282 Jul  1 04:01 agntstmp.txt.bak
drwxr-----  2 oracle dba 4.0K Jul  1 04:03 collection
-rw-r-----  1 oracle dba   23 Jul  1 04:03 ag_current_status.msg
-rw-r-----  1 oracle dba  321 Jul  1 04:03 agntstmp.txt
drwxr-----  3 oracle dba 4.0K Jul  1 04:05 upload
-bash-4.1$ cd collection

-bash-4.1$ mkdir old
-bash-4.1$ mv * old
mv: cannot move `old' to a subdirectory of itself, `old/old'
-bash-4.1$ ls -ltrh
total 4.0K
drwxr-xr-x 2 oracle dba 4.0K Jul  1 04:05 old
-bash-4.1$ cd ..
-bash-4.1$ ls -ltrh
total 72K
drwxr-----  2 oracle dba 4.0K Sep 16  2014 dumps
drwxr-----  2 oracle dba 4.0K Sep 16  2014 metricExtLib
drwxr-----  2 oracle dba 4.0K Sep 16  2014 attachmentDB
drwxr-----  2 oracle dba 4.0K Sep 16  2014 metricCache
drwxr-----  2 oracle dba 4.0K Sep 16  2014 temp
drwxr----- 12 oracle dba 4.0K Dec 10  2014 state
-rw-r-----  1 oracle dba 5.9K Apr  8 15:32 targets.xml.bak
-rw-r-----  1 oracle dba  910 Jun 26 22:10 blackouts.xml.bak
-rw-r-----  1 oracle dba   13 Jun 27 00:46 blackouts.xml
-rw-r-----  1 oracle dba 3.1K Jun 30 09:06 emagent_storage.config
-rw-r-----  1 oracle dba 5.9K Jul  1 03:48 targets.xml
-rw-r-----  1 oracle dba    0 Jul  1 03:48 blackouts.lck
-rw-r-----  1 oracle dba  282 Jul  1 04:01 agntstmp.txt.bak
-rw-r-----  1 oracle dba   23 Jul  1 04:03 ag_current_status.msg
-rw-r-----  1 oracle dba  321 Jul  1 04:03 agntstmp.txt
drwxr-----  3 oracle dba 4.0K Jul  1 04:05 upload
drwxr-----  3 oracle dba 4.0K Jul  1 04:05 collection
-bash-4.1$ cd state

-bash-4.1$ mkdir old
-bash-4.1$ mv * old
mv: cannot move `old' to a subdirectory of itself, `old/old'
-bash-4.1$ ls -ltrh
total 4.0K
drwxr-xr-x 12 oracle dba 4.0K Jul  1 04:06 old
-bash-4.1$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running
-bash-4.1$ emctl clearstate agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
EMD clearstate completed successfully
-bash-4.1$ df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper         99G   63G   32G  67% /u01

-bash-4.1$ emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent ......... started.
-bash-4.1$ emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

How to Kill a session in 10g OEM

To kill a session in OEM level,

Login to the OEM > Go to the Targets> Select the database > Click on Performance > select Search Sessions (in Additional Monitoring Links) > In the Search filter give the Sid if you know otherwise press the Go button it will list out all the sessions then select the required sid > Click on kill session button > it will ask for the Confirmation, Select the Kill Immediate button then Press Yes >  Then you will get
ORA-00031: Session marked for kill error.
Then the session is no more exist in the database.


Happy Learning :)

Remove or Delete or Un-schedule Oracle OEM scheduled backup jobs

To permanently delete a scheduled backup job from OEM 10g

Login to the Grid > go to jobs tab >  

Here you can see the Job Activity page. Give the details:

Status: All
Name: Name of the database the job is scheduled
View: Executions

By choosing the above options it will list out the jobs under the database.

Use the “Select” radio buttons on the left side of the list to select the job you want to delete.

Then click on the 'Delete Run' button >

OEM will give you two options:

Do you want to delete just this run or 
Delete all runs of job

Select the Delete all runs of job option if you want to remove this job permanently from the OEM.

If you select first option on the above it simply deletes the job for this occurrence. After that OEM will submit this job automatically for the next occurrence.

The Delete all runs of job option will delete the job from the OEM permanently. 

If you want to re run a failed backup at OEM level then:

Login to Grid > go to jobs tab >

Here you can see the Job Activity page. Give the details:

Status: All
Name: Name of the database the job is scheduled
View: Executions

By choosing the above options it will list out the jobs under the database.

Use the “Select” radio buttons on the left side of the list to select the job you want to run it again.

Then click on the try again, it will ask for the confirmation just click on Yes button.

If you want to show the status of the job then set the options as above on the Job Activity Page.