原文链接： http://www.dbaleet.org/overview_of_exadata_storage_architecture/
本文旨在简单介绍Exadata存储的架构，通过阅读本文能对Exadata Storage有一个初步的了解。
Exadata的磁盘层次结构非常清晰， 自第向上依次是 Physicaldisk=>LUN=>celldisk=>griddisk=>ASM disk
注1：其中上图表示操作系统所在的磁盘（一般为前两块磁盘）的架构；下图表示非操作系统盘的存储架构。（剩余没有进行分区的十块盘）。
注2： griddisk在RDBMS层面对应的是ASM disk， griddisk和ASM disk实际是同一个东西，但是是分别站在Exadata Stroage和RDBMS的角度来看的。
注3： 当前Celldisk与Griddisk的对应关系为1：m，即一对多。但是Kevin Closson提到准确的应该是多对多的关系，但是为了理解上的简单，可以认为是celldisk和griddisk是一对多的关系。
we (Exadata development) considered supporting celldisk creation from HW RAID volumes
but opted for a 1:1 relationship instead for many reasons. Griddisks are the virtualization of celldisks (the presentation form to ASM).
To that end there is a M:M relationship between celldisks and griddisks.
下面通过分自底向上的方式分别介绍各层的情况。
首先是physicaldisk:
[root@dm01cel01 ~]#cellcli -e list physicaldisk
dm01cel01: 20:0 RETS0D normal
dm01cel01: 20:1 REXHAD normal
dm01cel01: 20:2 RE5VTD normal
dm01cel01: 20:3 RE5SYD normal
dm01cel01: 20:4 RDDTYD normal
dm01cel01: 20:5 RETB5D normal
dm01cel01: 20:6 RDDS0D normal
dm01cel01: 20:7 RDDULD normal
dm01cel01: 20:8 RDDPZD normal
dm01cel01: 20:9 REXS8D normal
dm01cel01: 20:10 RDDTBD normal
dm01cel01: 20:11 RDDT9D normal
dm01cel01: FLASH_1_0 1202M0CPA5 normal
dm01cel01: FLASH_1_1 1202M0CPA7 normal
dm01cel01: FLASH_1_2 1202M0CQKE normal
dm01cel01: FLASH_1_3 1202M0CPA6 normal
dm01cel01: FLASH_2_0 1202M0CQE6 normal
dm01cel01: FLASH_2_1 1202M0CQE0 normal
dm01cel01: FLASH_2_2 1202M0CQA3 normal
dm01cel01: FLASH_2_3 1202M0CQAL normal
dm01cel01: FLASH_4_0 1202M0CP0E normal
dm01cel01: FLASH_4_1 1202M0CP0D normal
dm01cel01: FLASH_4_2 1202M0CNXH normal
dm01cel01: FLASH_4_3 1202M0CP0A normal
dm01cel01: FLASH_5_0 1202M0CQAE normal
dm01cel01: FLASH_5_1 1202M0CQ9V normal
dm01cel01: FLASH_5_2 1202M0CQA0 normal
dm01cel01: FLASH_5_3 1202M0CQAD normal
从上面可以看到：
以20:*表示的是这个cell节点上的12块物理硬盘；
以FLASH_*-*的表示FLASH卡，每块FLASH卡有4个FMOD，一共有4块FLASH卡，所以能看到16块闪盘。
我们在操作系统层面使用操作系统工具df/fdisk来查看当前cell的操作系统，磁盘/闪盘信息。
[root@dm01cel01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md6 9.9G 3.6G 5.9G 38% /
tmpfs 12G 0 12G 0% /dev/shm
/dev/md8 2.0G 647M 1.3G 34% /opt/oracle
/dev/md4 116M 60M 50M 55% /boot
/dev/md11 2.3G 130M 2.1G 6% /var/log/oracle
操作系统所用的分区为：md6，md8， md4， md11 。
[root@dm01cel01 ~]# mdadm -Q -D /dev/md6
/dev/md6:
Version : 0.90
Creation Time : ......
Raid Level : raid1
Array Size : 10482304 (10.00 GiB 10.73 GB)
Used Dev Size : 10482304 (10.00 GiB 10.73 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 6
Persistence : Superblock is persistent
Update Time : ......
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
......
Number Major Minor RaidDevice State
0 8 6 0 active sync /dev/sda6
1 8 22 1 active sync /dev/sdb6
可以看到/dev/md6实际上是由/dev/sda6与/dev/sdb6组成的soft RAID。同样md8， md4， md11 分别是由/dev/sda8与/dev/sdb8，/dev/sda4与/dev/sdb4, /dev/sda11与/dev/sdb11组成的RAID。限于篇幅，这里不一一列出来。
[root@dm01cel01 ~]# fdisk -l
Disk /dev/sda: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 15 120456 fd Linux raid autodetect
/dev/sda2 16 16 8032+ 83 Linux
/dev/sda3 17 69039 554427247+ 83 Linux
/dev/sda4 69040 72824 30403012+ f W95 Ext'd (LBA)
/dev/sda5 69040 70344 10482381 fd Linux raid autodetect
/dev/sda6 70345 71649 10482381 fd Linux raid autodetect
/dev/sda7 71650 71910 2096451 fd Linux raid autodetect
/dev/sda8 71911 72171 2096451 fd Linux raid autodetect
/dev/sda9 72172 72432 2096451 fd Linux raid autodetect
/dev/sda10 72433 72521 714861 fd Linux raid autodetect
/dev/sda11 72522 72824 2433816 fd Linux raid autodetect
Disk /dev/sdb: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 15 120456 fd Linux raid autodetect
/dev/sdb2 16 16 8032+ 83 Linux
/dev/sdb3 17 69039 554427247+ 83 Linux
/dev/sdb4 69040 72824 30403012+ f W95 Ext'd (LBA)
/dev/sdb5 69040 70344 10482381 fd Linux raid autodetect
/dev/sdb6 70345 71649 10482381 fd Linux raid autodetect
/dev/sdb7 71650 71910 2096451 fd Linux raid autodetect
/dev/sdb8 71911 72171 2096451 fd Linux raid autodetect
/dev/sdb9 72172 72432 2096451 fd Linux raid autodetect
/dev/sdb10 72433 72521 714861 fd Linux raid autodetect
/dev/sdb11 72522 72824 2433816 fd Linux raid autodetect
Disk /dev/sdc: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdd doesn't contain a valid partition table
Disk /dev/sde: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sde doesn't contain a valid partition table
Disk /dev/sdf: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdf doesn't contain a valid partition table
Disk /dev/sdg: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdg doesn't contain a valid partition table
Disk /dev/sdh: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdh doesn't contain a valid partition table
Disk /dev/sdi: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdi doesn't contain a valid partition table
Disk /dev/sdj: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdj doesn't contain a valid partition table
Disk /dev/sdk: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdk doesn't contain a valid partition table
Disk /dev/sdl: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdl doesn't contain a valid partition table
Disk /dev/sdm: 4009 MB, 4009754624 bytes
126 heads, 22 sectors/track, 2825 cylinders
Units = cylinders of 2772 * 512 = 1419264 bytes
Device Boot Start End Blocks Id System
/dev/sdm1 1 2824 3914053 83 Linux
Disk /dev/md1: 731 MB, 731906048 bytes
2 heads, 4 sectors/track, 178688 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md1 doesn't contain a valid partition table
Disk /dev/md11: 2492 MB, 2492137472 bytes
2 heads, 4 sectors/track, 608432 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md11 doesn't contain a valid partition table
Disk /dev/md2: 2146 MB, 2146697216 bytes
2 heads, 4 sectors/track, 524096 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md2 doesn't contain a valid partition table
Disk /dev/md8: 2146 MB, 2146697216 bytes
2 heads, 4 sectors/track, 524096 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md8 doesn't contain a valid partition table
Disk /dev/md7: 2146 MB, 2146697216 bytes
2 heads, 4 sectors/track, 524096 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md7 doesn't contain a valid partition table
Disk /dev/md6: 10.7 GB, 10733879296 bytes
2 heads, 4 sectors/track, 2620576 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md6 doesn't contain a valid partition table
Disk /dev/md5: 10.7 GB, 10733879296 bytes
2 heads, 4 sectors/track, 2620576 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md5 doesn't contain a valid partition table
Disk /dev/md4: 123 MB, 123273216 bytes
2 heads, 4 sectors/track, 30096 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md4 doesn't contain a valid partition table
Disk /dev/sdn: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdn doesn't contain a valid partition table
Disk /dev/sdo: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdo doesn't contain a valid partition table
Disk /dev/sdp: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdp doesn't contain a valid partition table
Disk /dev/sdq: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdq doesn't contain a valid partition table
Disk /dev/sdr: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdr doesn't contain a valid partition table
Disk /dev/sds: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sds doesn't contain a valid partition table
Disk /dev/sdt: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdt doesn't contain a valid partition table
Disk /dev/sdu: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdu doesn't contain a valid partition table
Disk /dev/sdv: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdv doesn't contain a valid partition table
Disk /dev/sdw: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdw doesn't contain a valid partition table
Disk /dev/sdx: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdx doesn't contain a valid partition table
Disk /dev/sdy: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdy doesn't contain a valid partition table
Disk /dev/sdz: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdz doesn't contain a valid partition table
Disk /dev/sdaa: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdaa doesn't contain a valid partition table
Disk /dev/sdab: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdab doesn't contain a valid partition table
Disk /dev/sdac: 24.5 GB, 24575868928 bytes
255 heads, 63 sectors/track, 2987 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdac doesn't contain a valid partition table
fdisk的输出至少包含如下信息：
- 此cell为HP（high performance）的硬盘，单个磁盘的大小为600G；
- 设备号sda和sdb为操作系统所在的磁盘， 这两个磁盘分别划分了11个partition；
- 12块硬盘的对应设备编号为：/dev/sda到/dev/sdk；
- 16个FDOM对应的设备编号为：/dev/sdn到/dev/sdac。
继续查看LUN的信息：
[root@dm01cel01 ~]#cellcli -e list LUN
dm01cel01: 0_0 0_0 normal
dm01cel01: 0_1 0_1 normal
dm01cel01: 0_2 0_2 normal
dm01cel01: 0_3 0_3 normal
dm01cel01: 0_4 0_4 normal
dm01cel01: 0_5 0_5 normal
dm01cel01: 0_6 0_6 normal
dm01cel01: 0_7 0_7 normal
dm01cel01: 0_8 0_8 normal
dm01cel01: 0_9 0_9 normal
dm01cel01: 0_10 0_10 normal
dm01cel01: 0_11 0_11 normal
dm01cel01: 1_0 1_0 normal
dm01cel01: 1_1 1_1 normal
dm01cel01: 1_2 1_2 normal
dm01cel01: 1_3 1_3 normal
dm01cel01: 2_0 2_0 normal
dm01cel01: 2_1 2_1 normal
dm01cel01: 2_2 2_2 normal
dm01cel01: 2_3 2_3 normal
dm01cel01: 4_0 4_0 normal
dm01cel01: 4_1 4_1 normal
dm01cel01: 4_2 4_2 normal
dm01cel01: 4_3 4_3 normal
dm01cel01: 5_0 5_0 normal
dm01cel01: 5_1 5_1 normal
dm01cel01: 5_2 5_2 normal
dm01cel01: 5_3 5_3 normal
其中：
0_0到0_11为12块磁盘的LUN号
1_0到1_3为第一块flash卡的LUN号，依次类推，flash卡的LUN号有时并不连续，是因为更换过flash卡导致的。
接下来再来看celldisk的信息：
[root@dm01cel01 ~]#cellcli -e list celldisk
dm01cel01: CD_00_dm01cel01 normal
dm01cel01: CD_01_dm01cel01 normal
dm01cel01: CD_02_dm01cel01 normal
dm01cel01: CD_03_dm01cel01 normal
dm01cel01: CD_04_dm01cel01 normal
dm01cel01: CD_05_dm01cel01 normal
dm01cel01: CD_06_dm01cel01 normal
dm01cel01: CD_07_dm01cel01 normal
dm01cel01: CD_08_dm01cel01 normal
dm01cel01: CD_09_dm01cel01 normal
dm01cel01: CD_10_dm01cel01 normal
dm01cel01: CD_11_dm01cel01 normal
dm01cel01: FD_00_dm01cel01 normal
dm01cel01: FD_01_dm01cel01 normal
dm01cel01: FD_02_dm01cel01 normal
dm01cel01: FD_03_dm01cel01 normal
dm01cel01: FD_04_dm01cel01 normal
dm01cel01: FD_05_dm01cel01 normal
dm01cel01: FD_06_dm01cel01 normal
dm01cel01: FD_07_dm01cel01 normal
dm01cel01: FD_08_dm01cel01 normal
dm01cel01: FD_09_dm01cel01 normal
dm01cel01: FD_10_dm01cel01 normal
dm01cel01: FD_11_dm01cel01 normal
dm01cel01: FD_12_dm01cel01 normal
dm01cel01: FD_13_dm01cel01 normal
dm01cel01: FD_14_dm01cel01 normal
dm01cel01: FD_15_dm01cel01 normal
可以看到在celldisk中磁盘与闪盘分别以前缀CD和FD代替了，并且从上面的信息来看：没一块磁盘对应一块celldisk，每一flash卡的FDOM模块也对应一块celldisk。
最后来看一下griddisk：
[root@dm01cel01 ~]#cellcli -e list griddisk
dm01cel01: DATA_CD_00_dm01cel01 active
dm01cel01: DATA_CD_01_dm01cel01 active
dm01cel01: DATA_CD_02_dm01cel01 active
dm01cel01: DATA_CD_03_dm01cel01 active
dm01cel01: DATA_CD_04_dm01cel01 active
dm01cel01: DATA_CD_05_dm01cel01 active
dm01cel01: DATA_CD_06_dm01cel01 active
dm01cel01: DATA_CD_07_dm01cel01 active
dm01cel01: DATA_CD_08_dm01cel01 active
dm01cel01: DATA_CD_09_dm01cel01 active
dm01cel01: DATA_CD_10_dm01cel01 active
dm01cel01: DATA_CD_11_dm01cel01 active
dm01cel01: DBFS_DG_CD_02_dm01cel01 active
dm01cel01: DBFS_DG_CD_03_dm01cel01 active
dm01cel01: DBFS_DG_CD_04_dm01cel01 active
dm01cel01: DBFS_DG_CD_05_dm01cel01 active
dm01cel01: DBFS_DG_CD_06_dm01cel01 active
dm01cel01: DBFS_DG_CD_07_dm01cel01 active
dm01cel01: DBFS_DG_CD_08_dm01cel01 active
dm01cel01: DBFS_DG_CD_09_dm01cel01 active
dm01cel01: DBFS_DG_CD_10_dm01cel01 active
dm01cel01: DBFS_DG_CD_11_dm01cel01 active
dm01cel01: RECO_CD_00_dm01cel01 active
dm01cel01: RECO_CD_01_dm01cel01 active
dm01cel01: RECO_CD_02_dm01cel01 active
dm01cel01: RECO_CD_03_dm01cel01 active
dm01cel01: RECO_CD_04_dm01cel01 active
dm01cel01: RECO_CD_05_dm01cel01 active
dm01cel01: RECO_CD_06_dm01cel01 active
dm01cel01: RECO_CD_07_dm01cel01 active
dm01cel01: RECO_CD_08_dm01cel01 active
dm01cel01: RECO_CD_09_dm01cel01 active
dm01cel01: RECO_CD_10_dm01cel01 active
dm01cel01: RECO_CD_11_dm01cel01 active
从上面的信息来看每一块celldisk被分为3块griddisk。分别对应于ASM的diskgroup中的三个磁盘组DATA, DBFS_DG, RECO。也就是说DATA 这个diskgroup只使用含有前缀DATA的griddisk，依此类推。在默认没有使用interleaving的情况下， DATA前缀的griddisk使用的是磁盘的最外侧磁道（最先创建，偏移量最小），所以速度最快，RECO前缀的griddisk使用的是磁盘的最内测磁道，所以速度最慢。(update： 如果有DBFS， DBFS在最内圈，速度比RECO慢)
Comment