“Kevin Closson
May 17
to me
…***hold it*** I messed up… was mixing half and full rack.
OK, always start with datasheet which is 50,000 WIOPS full rack
Exadata only supports ASM and ASM partner writes are not accounted for.
Your application write payload max would therefore be 25,000/full rack normal redund.”

regarding the imbalance… mmm I can argue that Exadata is still a strong platform because you are just running databases on it and nothing else. And that alone is a big difference. Let me put some real world examples:

>> First. I’ve diagnosed a performance problem on a newly migrated database from an old to new SAN environment (TIER1). And apparently the reads are a lot slower because the SAN storage processor is already saturated because they’ve got a mix of MS Exchange Server workload, databases, and other uses for that SAN (NFS, CIFS, etc.). Simply put, the requirements are exceeding the capacity causing the storage processor to be affected which leads to bad IO times. Since this database is mission critical, they’ve migrated it to TIER2 storage (supposedly a slower array) which apparently is way faster than their TIER1 and that solved their database IO issues.

>> Second. I’ve seen a case where an M5000 box (64 CMT cores) is backed by 800+ spindles Symmetrix storage which is using the EMC “FAST” feature. But even though it’s M5000 they are not using it only for databases, it’s a multi-tenant Solaris environment making intensive use of zones. In one M5000 box they’ve got 7 zones dedicated for app servers and only 2 zones for databases. The 1st database zone is holding the EMGC, the other is holding 7databases. The end result is a mixed workload (file IO and database IO) that doesn’t really like each other and they’re not really enforcing the resource management features of Solaris so it’s a big playground for all the zones and when all of them reach their peak periods then there comes the latency issues and high SYS CPU. Simply put, even with a Ferrari SAN if you are not doing the right thing.. you’ll not get the right results.

I’ve got some screenshots here http://goo.gl/MYKxK
see
>> average latency issue correlation of SAN, datafiles, session IO
>> IO issue – SAN performance validation – saturated storage processor

-Karl

-Karl

Yury

Thanks for dropping by Mark!

-Karl

As for collectl, it’s always fun to see people using it in new and different ways and I guess what I’m wondering is why you’re redirecting collectl’s output to a text file. Do you realize that means you can only look at what you’ve decided to record at that point in time. Perhaps you didn’t realize that you could run: collectl –all -P -f/dirname and it will record everything to a raw file.

Then when you’re tests are done, simply ^C collectl and then you can run any of your above commands against that data as often and as many different ways as you like. In other words, in your text files you also have disk data in detail format and cpu data in verbose format. But what if you want to look at individual CPUs or networks? You can’t because they’re not in your txt files. On the other hand, if you recoded the data in raw files you can play it back in as many different formats as you like.

Also, how did you get the data in a format suitable for plotting? If you recorded data in raw files you can convert it to space-separated files suitable for plotting directly with gnuplot or better yet, colplot.

In other words, my advise is to ALWAYS record collectl data in raw files as you never know how you want to drill down into it. Some times you want brief more, sometimes verbose mode and sometimes detail mode. Sometimes you want to see it on your terminal and sometimes you want to plot it. WIth raw files you can do it all.

The other thing I couldn’t tell is whether you’re running these commands on a single server or multiple ones. If you’re running on multiple servers you really should look into colmux at that will make your life much easier. And speaking of Kevin Closson, I’ve been after him for months to check out colmux and he hasn’t either.

-mark

Great job and documentation.

I am having the same intermittent poor database performance and applications are complaining. This happens once or twice a week and it is getting to be a concerned situation.

I read about your gc block lost problem and how your resolved it. I believe it is same problem that we are having. When I query gv$sysstat view, I see a very high value of 26000 for RAC node 1 and 7000 for RAC node 2. This tells me there is problem, but I want to gather all the numbers to make presentation to management, Unix SA and Network team.

Can you send me a copy of the Tanel Poder’s Perfsheet file. This file does not exist anymore on the website. I want to create a PowerPoint presentation with convincing numbers and graphs.

I will appreciate if you can assist.

Thank you
Grace

Really powerfull tool.. thanks a lot.. I will dig more !!

I did not go as far as R2 linear regression ;(

http://www.gmanetwork.com/news/story/249404/scitech/technology/10-000-students-to-benefit-from-oracle-dost-program

It was really nice meeting you. You are a very lucky man to win an ipad2 and have your name picked up from another sponsor for another ipad2! :p

I’ll see you again next RMOUG!

-Karl

It was great meeting you at RMOUG lunch on Wednesday! I didn’t attend your session ’cause I’m not a DBA and it looked way over my head. Mainly I support the data warehouse (Cognos) at my company.

Anyway, I wish you the best with your career and life in general.

Jeff

-Karl

(1) From AWR I get that snap_id (6784) starts at 10:00:35

SQL> select output from table(dbms_workload_repository.AWR_REPORT_TEXT(2461100887,1,6784,6785));
WORKLOAD REPOSITORY report for

DB Name DB Id Instance Inst Num Startup Time Release RAC
————- ——————- ———— ———— ——————– ———– —–
ECCDV3 2461100887 eccdv3 1 22-Dec-11 23:11 11.2.0.2.0 NO

Host Name Platform CPUs Cores Sockets Memory(GB)
————————– ————————————— ——- ——— ———— —————-
VECC-ORACLE2-DE Microsoft Windows x86 64-bit 8 8 8 10.00

Snap Id Snap Time Sessions Curs/Sess
——— ————————- ———- ————–
Begin Snap: 6784 02-Feb-12 10:00:35 331 5.6
End Snap: 6785 02-Feb-12 11:00:25 349 4.2
Elapsed: 59.83 (mins)
DB Time: 70.59 (mins)

(2) But if I check snaps timing intervals from dba_hist_active_sess_history, then I get that snap_id 6784 starts at 09:00.

select SNAP_ID, min(SAMPLE_TIME) as mins, max(SAMPLE_TIME) as maxs
from dba_hist_active_sess_history h
where h.sample_time BETWEEN TO_DATE(’20120202070000′,’YYYYMMDDHH24MISS’) AND TO_DATE(’20120202140000′,’YYYYMMDDHH24MISS’)
group by snap_id
order by 1,2
/
SNAP_ID MINS MAXS
—————- ———————————— ———————————–
6782 02-FEB-12 07.00.26.797 AM 02-FEB-12 08.00.07.092 AM
6783 02-FEB-12 08.00.27.092 AM 02-FEB-12 09.00.17.436 AM
6784 02-FEB-12 09.00.47.484 AM 02-FEB-12 09.59.48.821 AM
6785 02-FEB-12 10.00.38.825 AM 02-FEB-12 11.00.10.118 AM
6786 02-FEB-12 11.00.20.118 AM 02-FEB-12 12.00.22.882 PM
6787 02-FEB-12 12.00.32.885 PM 02-FEB-12 01.00.04.662 PM
6788 02-FEB-12 01.00.14.662 PM 02-FEB-12 01.59.59.035 PM

(3) Even from dba_hist_snaphot I get which would be the correct snap_id interval

select snap_id, BEGIN_INTERVAL_TIME, end_interval_time from dba_hist_snapshot where snap_id in (6782, 6783, 6784, 6785,6786, 6787, 6788) order by 1;

SNAP_ID BEGIN_INTERVAL_TIME END_INTERVAL_TIME
———- ————————- ————————-
6782 02-FEB-12 07.00.19.643 AM 02-FEB-12 08.00.25.016 AM
6783 02-FEB-12 08.00.25.016 AM 02-FEB-12 09.00.37.499 AM
6784 02-FEB-12 09.00.37.499 AM 02-FEB-12 10.00.35.529 AM
6785 02-FEB-12 10.00.35.529 AM 02-FEB-12 11.00.25.308 AM
6786 02-FEB-12 11.00.25.308 AM 02-FEB-12 12.00.33.729 PM
6787 02-FEB-12 12.00.33.729 PM 02-FEB-12 01.00.25.383 PM
6788 02-FEB-12 01.00.25.383 PM 02-FEB-12 02.00.38.022 PM

- Why awr report is saying that snap_id (6784) starts at 10:00 ?

Best Regards
LuisGomez

I was trying to download your scripts, which says not availlable… can you please help me to download them

Thanks,
Raghav

-Karl

When we generate AWR report in oracle 10g then there is sections called SQL Statistics and under which following sub sections are there:

SQL ordered by Elapsed Time
SQL ordered by CPU Time
SQL ordered by Gets
SQL ordered by Reads
SQL ordered by Executions
SQL ordered by Parse Calls
SQL ordered by Sharable Memory
SQL ordered by Version Count
Complete List of SQL Text

I can not generate AWR report as per license issue also Statspack is free but it is not giving enough information.

Is there any SQL script which can present similar data from statspack tables as per AWR.

Please provide such oracle script.

Thanking You

Excellent script. I have used for one of performance issues comparision workload for given databases and its rocked ….

Thanks for the good work.. saves a lot of time.

Keep going…

Surya

Maybe you don’t remember me. Im currently diagnosing a ASH report from a Oracle 11g RAC. I encountered a blocking sessions, with the wait event enq: WF – contention. Unfortunately the User name is not found. What can caused an “enq: WF – contention”? I only notice this because a used DBA_HIST_ACTIVE_SESS_HISTORY base on SID and INST_ID to generate a snap id. This might help my investigation.

Thanks

Yes I think I had a working version of that before and suddenly I got busy with the DBA_HIST views.. but creating a statspack version will still use the logic and formulas of awr_genwl only that statspack has slightly different columns which lead me to join it to additional tables to derive some of the values making it perform poorly.. but I already have an idea to get around it I just don’t have the time to work on it I will let you know once I’m done with it! Glad to know awr_genwl is useful

Richard.

Simple Question …how to find out from AWR reports whats the CPU usage of the Whole BOX and how much is Oracle Using ?

Thanks.