Ilja Coolen \ ICSS

We recently were implementing a new TSM farm, based on TSM 6.1.
Since we are  also using a NAS system, we were also testing the NDMP functions of TSM6 against our NAS device. We previously were using TSM5.5 which worked fine so we expected it to work just as fine in TSM6.1
Unfortunately it  didn’t. We opened a PMR with IBM to investigate this problem, but they quickly pointed to the NAS vendor. IBM/TSM support said the NDMP code was not changed from TSM5.5 to TSM6.1, and TSM uses an NDMP API to perform the NDMP functions agains a NAS device. So we also opened a ticket with the NAS vendor support dept. We did some traces on TSM5.5, TSM6.1, and the NAS device. All commands and responses were like expected, and there seemed to be nothing wrong. We send the ticket back to IBM/TSM because the TSM version was the only change made. The NDMP backup was still working perfectly with TSM5.5, and not a single change was made on the NAS device. So the obvious cause had to be TSM.

After some more investigation IBM/TSM support came out with a probable cause (not confirmed yet) being memory management changes in TSM6.1. In TSM5, memory is still allocated if required bytes is zero, while in TSM6 no memory is allocated and null is returned when the required bytes is zero. The “show nasd” command will not work if zero is the value of SCSI info in TSM6. All commands in TSM to define paths to the tape drives the NAS device sees, will fail as well.

This problem applies to all TSM6 releases currently out there. An official APAR is not yet opened, because we will be running some additional tests with a special purpose built TSM server to further diagnose with greater detail. I will update this post when these tests are done and the APAR is filed.

I do have to note though that although IBM and the NAS (HDS)  vendor are competitors in the NAS arena, they did work together nicely to resolve this issue for the customer. Although this should always be the case in the interest of the customer, I have had a case or two when the vendors wouldn’t cooperate and the customer was left in the dark. So to me, IBM and HDS  deserve a compliment for this.

Just recently I was working on a new TSM farm with some co-workers.

We came across some confusing tape limitation documentation in regards to TSM on Linux hosts.
We had to figure out what device drivers are used in what configurations and what limitations would apply then.

We were triggered by some Linux administrator who came in and said Linux only supports 128 tape drives and we should reconfigure our complete TSM infrastructure.

After some checking back and forth between Linux and TSM support we finally had a clear picture. I don’t know if we were the only ones confused by all the kinds of tape device drivers IBM/TSM uses on different platforms, but just in case you are confused as well, I thought I make a summary of all the device drivers, limitations of a TSM server on a Linux host.

If the comments on this article indicate there’s a demand for more, I would love to complete the list with the matrices for more host platforms.

TSM 6.1 on RedHat Linux (RHEL 5.4)

On Linux we have a couple of default SCSI tape device drivers available.

st (SCSI tape) driver.

Resource:
This is a Linux default kernel module driver. in RHEL 5.4 it supports 128 tape drives.
When the st driver is loaded, it detects all sequential media devices (unless they are in the reject_list) if you have a large environment, with a bunch of tape drives (eg a Virtual Tape Library) you might quite easily exceed the maximum number of drives for this driver.  Your system log will show a message like ” st:Too many tape devices (max. 128)” Your solution is to either put the devices in the reject_list or blacklist the module to prevent it from loading, provided you are going to use a vendor provided device driver.

sg (SCSI generic) driver.

Resource:
All scsi devices in Linux can be accessed by this driver. SCSI attached devices like CDROM drives, sequential media devices or hard drives all can make use of this device driver.
Kernel  2.6 and above support at least 1024 devices used by the sg driver.

• lin_tape               (IBM devices) supports 1024 devices
• tsm tape              (non-IBM devices) limited to 256 devices on Linux according to IBM support matrices

You will all have already seen IBM’s announcement last week on it’s Long Term File System (LTFS). In short, it’s a software feature that enables you to mount a sequential media (only LTO gen 5 or newer) as if it were a local file system. In itself this is not a real innovation. Seagate had this for years, although it never kicked off. In my opinion there has never been a real use case for this type op I/O.

IBM obviously sees a market for this, or they wouldn’t have put in the effort of developing this feature. For now, it’s only supported on Redhat Enterprise Linux 5.4 in combination with IBM’s tape drivers. So what would the use case be for a feature like this.

In their announcement they mention you can use it to store unused archive-able data. A customer case mentions saving a lot of money by moving old digital media to the LTFS, which I guess would make most sense. Media files are usually quite large. I wonder how this would be made a workable solution without radically disturbing the current operating procedures within a file serving environment. You’d have to create procedures on how to (manually) move the old file data to the tape tier (lets call it tier3 for now). At the moment, I know of no automated mechanism that moves file data into another tier if it is not virtualized by a NAS like device.

Next, the LTO media cartridge would remain in the drive for as long as the filesystem has to be mounted, eating up precious resources in case you are using this drive for backup purposes as well. Even if you were to buy a LTO5 drive for this purpose alone, this would cost quite a some of money. For the price of a single LTO5 drive you would be able to by a bunch of 2TB hard disks. Even then, you would be filling up a tape with about 1.5TB to 2.5TB (IBM numbers) depending on the compression rate. That’s not a large enough amount in your file sharing environment to make it worth your while I guess. It would make a lot more sense if you were able to attach an autochanger, like a tape library, to your RHEL server and have RHEL automatically load and mount the desired tape cartridge ( a lot like any decent HSM appliance/application would).

Even with an autochanger or libray solution, you would still need a mechanism to protect this archival data from a tape media failure, or you could be in a heap of trouble if that archived data would get lost.

What about filesystem checks. I imagine you can turn of filesystem checks, by in case the host goes down unplanned while writing data to the tape device, how can you make sure the data on the tape is not corrupted?

So what would work? Well, I had some ideas on this, and one would be to attach these LTO5 drives and library to the back of (in IBM’s case) a N-Series (NetApp) filer and load the LTFS code in there. That’s an issue now, as you would not be able to load FUSE modules into the ONTAP code. But, given IBM’s (and maybe even NetApp’s) resources, they should be able to code this into ONTAP or any other NAS head operating system. So later, if you can load the LTFS code into a NAS head you could be able to automatically migrate older data from high-end drives to midrange or even low-range drives, or if desired directly to tape. The end-users wouldn’t notice that, meaning they would not need to change their way of managing their files.

Since IBM already has an enterprise level file virtualization appliance with SONAS, I would think it would make sense to inject the LTFS code into SONAS and have one of their tape libraries attached to the back of SONAS with a bunch of drives. You would need to keep an index of all files in memory to present to the end users, and mount the required media in case an end user would eventually request an archived file. But again, this is nothing new in HSM land, so it shouldn’t be much trouble to incorporate this.

One thing that could pose a big issue to enterprise customers with millions of files is the amount of memory required to index these files. IBM documents show that you need 1GB or RAM for every 1 million files on the tape media.

My opinion on this LTFS feature is that it is quite nice and adds perfectly to the virtualization layers already in use throughout the datacenters, but not in its current offering. Stick it in SONAS or N-Series, and I think it would or could make a valuable addition into the quest to increase savings in environmental costs.

Please feel free to leave your remarks in the comments, or on twitter @iCoolen

For a while i’ve been tracking the development of FujistuSiemens’ CentricStor Virtual Tape solution. For what I’ve seen so far is that it is an outstanding solution for enterprise datacenters. What I do ask myself is; If their product works as fine as I think it does, why isn’t there a storageblogger out there that does as much as mention it?

While all well-known enterprise (virtual) tape vendors do their best to offer consolidation, virtualization,  dedupe and all that, FujistuSiemens(FS) remains unmentioned in the blogs I follow (I might be missing the blogs that do mention the CentricStor).

• They do offer consolidation by enabling almost all open platforms to connect, and also perform very well in the z/OS arena, combined in one installation. Not many vendors attempt to consolidate open and z/OS in one solution. There must be a reason for this.
• They virtualize tape, like most other vendors do. They transparently allow about any tape library and drive to connect to the back-end and do a form of inline virtualization of physical tape.
• They can write data to multiple tapes (even of different brands/sizes/types) simultaneously and thereby automatically enabling an instant remote copy.
• What they still can’t (or do not want to) do is dedup. FS states they still have no faith in currently used techniques to dedupe, especially when considering the need to recover vast amounts of data in case of a disaster. To some extent I can relate to this reasoning.
• Although  many (virtual tape)  vendors try to convince you that physical tape is dead, I personally think tape will not be dead for a long tim, as does FS.

I imagine they have a small install base and are missing marketing presence, but that has never stopped you guys from noticing interesting players before, so please comment to this post and enlighten me.

Today,I’ve come across some problems when configuring a Commvault Galaxy server to use the Diligent’s Virtual Tape Facility including ProtecTIER. I thought you’d be interested.

This applies to the Commvault Galaxy Media Agents for Windows for as far as I know.
Another criteria, is that you have chosen to configure drives to be spread across two or more fibrechannel connections.

When doing a standard detection in Galaxy, the library device and tape units are properly detected, and the units can be automatically configured into the corresponding drive slots in the library. While this is the preferred method according to Windows/Galaxy users, (clickity click ) this does not result into the desired configuration though.

The symptoms are as follows:
When doing tape IO in Galaxy, Galaxy expects carts in certain tape units, but they are mounted in another unit, resulting in corresponding SCSI errors, like LOAD_OR_UNLOAD_FAILURES. In some cases, the exhaustive detection and validations also fails.

The reason:
When defining a new virtual library in the ProtecTIER GUI, and have chosen to spread the drives across two or more channels, Diligent has decided to use a naming scheme that uses add drive number assignments on one path, and even drive letter assignments on the other path. I have not been able to check the naming scheme when using more than two paths. But my best guess is that the problems will be similar.
The LUN id’s are numbered in consecutive order, from 0 (or 1) to n, on one path, and 0 (or 1) to n on the other path. Whether or not the LUN numbers start at zero is dependent on which port you are configuring the library controller device on. The library device apparently is always LUN ID zero.

For the example, lets make 6 drives, spread across two paths (controllers).

Port 0, holds :

• Library Device, LUN 0
• Drive 0, LUN 1
• Drive 2, LUN 2
• Drive 4, LUN 3

Port 1, holds :

• Drive 1, LUN 0
• Drive 3, LUN 1
• Drive 5, LUN 2

You could say this is a nice config, because most of the load will be spread across the two paths. Our experience tells us, this is indeed working well. I need to point out the current experience was based on TSM on AIX.  TSM on AIX automatically detects in which library  drive slot the tape drive actually sits. Galaxy messes this up though.

In the above screenshot of the drives, the element address of the drive slot (in the GUI it’s called “Address” ) is listed in the last column.
When taking a close look at this column, you should see that this list is consecutive also, but always starts at Address 16. You should also notice, that this list is hopping paths. When I sort on the ” Port”  column, you will see, that all the even numbered Addresses are listed together, as well as the odd numbered Addresses. This tells me that the programmers at Diligent were at least consequent in their design and naming schemes.

This is also what apparently imposes the  problem on Galaxy.
When doing your scsi discovery, normal operations scan one bus at a time, resulting in a list in tape drives (LUN’s) that are consecutive by port. During the discovery of the library device, inventory on that library also tells us, or Galaxy in this case, that the library holds n tape slots, as well as n drive slots, n drives and such.

The drive slots are consecutive according to  Galaxy, because the info Galaxy goes by is provided by the library device, and not by scsi inquiry commands. When using the Galaxy auto config features, Galaxy will match the drives to the drive slots, in the order they are detected.

Thus,

• \.\tape0 (LUN 1 port 0) is matched to Drive Slot 1 (Address 16)
• \.\tape1 (LUN 2 port 0) is matched to Drive Slot 2 (Address 17)
• \.\tape2 (LUN 3 port 0) is matched to Drive Slot 3 (Address 18)
• \.\tape3 (LUN 0 port 1) is matched to Drive Slot 4 (Address 19)
• \.\tape4 (LUN 1 port 1) is matched to Drive Slot 5 (Address 20)
• \.\tape5 (LUN 2 port 1) is matched to Drive Slot 6 (Address 21)

But, according to ProtecTIER, the drives and slots (Addresses) are matched hopping ports.
So, this is what it should look like.

• \.\tape0 (LUN 1 port 0) is matched to Drive Slot 1 (Address 16)
• \.\tape1 (LUN 2 port 0) is matched to Drive Slot 3 (Address 18)
• \.\tape2 (LUN 3 port 0) is matched to Drive Slot 5 (Address 20)
• \.\tape3 (LUN 0 port 1) is matched to Drive Slot 2 (Address 17)
• \.\tape4 (LUN 1 port 1) is matched to Drive Slot 4 (Address 19)
• \.\tape5 (LUN 2 port 1) is matched to Drive Slot 6 (Address 21)

Below is a Galaxy screenshot of a partially configured tape library. Here you can also notice the odd/even numbered features. In this screenshot I renamed the Drive aliasses to match the ProtecTIER naming of Drives. Elm is short for element, and indicated the Address column in the ProtecTIER GUI.

So, you need to do a manual matching (moving in the Galaxy Library and Drive configuration tool) between the drives and drive slots.
Use the ProtecTIER GUI (Drives tree within the Library) to track back the port number, LUN number and Drive Slot (Address).
Once you’ve tackled this problem, everything should work like a charm.

This post isn’t intended to be used as a manual, but merely as a reference for possible problems you might run into when trying to configure or troubleshoot the Galaxy configuration in combination with Diligent’s  VTF/ProtecTIER. As most virtual libraries use the most common devices and methods, this post could possible apply to more virtual libraries in combination with Galaxy.

I hope is has some use for you.

After a couple of months of negotiation’s and specification checking, we finally were able to place a new storage order for replacing our current IBM Sharks and DS4300 (fiber) disk systems. We decided to go for 1 kind of box, even though we could save money on some midrange storage systems. We have had our share of nasty experiences with the mid-range boxes, so we decided never to make any concessions on reliability again.

For the next few years, we will be running our centralized data on IBM’s DS8300 boxes. Two to be precise. One in each of the two locations, serving 90TB net capacity each. We will be mixing z/OS and OpenSystems, like we currently do on the Sharks.In front of the DS8300 we will keep our San Volume Controllers active for OpenSystems servers, even though one could questions the necessity for this.  We are very pleased with the SVC’s due to it’s flexibility and possibilities.

We are now preparing the migration to the new environment.
We would normally have no work on this, because the SVC does all that in the background. We have those old nasty AIX4 servers biting us in the butt though. The AIX admins are forcing upgrading to AIX5 to our customers, but this still isn’t going as fast as we’d like. The AIX admins are not working with me on upgrading their fixlevels and drivers for SVC, so we cannot upgrade the SVC to it’s current level 4.1. We need the current level though for the new hardware on which the SVC software runs. We cannot stay behind, just because of some unsupported operating systems, so we decided to put them in a death-bed on the current older SVC storage. We will be migrating all supported platforms to the new SVC hardware running the new code, using the somewhat traditional methods.  AIX supports the migration using the migratepv commands, and Windows will be done using the TOPIO software. So we have some nice tasks ahead of us.

Our IBM mainframe will have the least amount of work, using FDRPASS for the migration. We’ve done this a couple of times before, and this works perfectly. As it seems, AS400 systems can also do online migration by telling the OS to start removing volumes from its memory pool. We are investigating this. Otherwise, it will be a traditional backup/restore migration.

After the migration, we will be implementing full synchronous replication between both sites for all data based upon IBM’s Metro-Mirror. This will include Windows, AIX, Solaris, and VMware ESX data.

While going through all necessary preparations, it seems that everyone (not involved in storage) knows best, and is trying to force us into using their insights when designing the replication. When we would give into this, we would be replicating tons of applications using all application specific methods. What a nightmare that would be.
I preemptively agree on comments you will probably post, stating that several situations would be best solved by using application specific solutions. As the setup progresses, we are sure to encounter some situations where this applies. But sometimes you have to choose the lesser option, in order not to make stuff to complex or not to have consistancy issues.

The storage deal on disk replacement is over. A decision has been made.
Are we happy? Sure, why not. Did we get what we want? Well, technically we had a better option, but financially, we couldn’t refuse this final offer. The financial gap between the winner and the runner up, was just to big. The technological differences aren’t enough to justify the extra investments.
So we didn’t get the best money could buy, but we did get a good deal, and we do get good storage for these euros. Besides, the big-iron players’ high-end range storage subsystems are all good stuff and they do not differ too much in capabilities.
Another plus for my company is, that we do not need to go through a massive migration, don’t have to rewrite tons of scripts, don’t have to change multipath drivers on hundreds of servers. That’s worth something too, right? We were willing to go through all this changes, but in the end, like Meja said in track 2, it’s all ’bout the money.
My company is happy, because we made sure that the next four years, we have a good price per gig. Which in turn will make our customers happy as well. And this is why we have the job.

A downside to this is, that we don’t get experience with new equipment, new vendors, new methods, and more. Well, better luck in about 4 years.
Later.

PS: Merry Christmas to you all.

It has been a while, but here’s something I am trying to figure out for some time now.

We’ve bought a Diligent VTF with ProtecTIER (data deduplication on virtual tapes) in the first quarter this year. The main reason was that nice feature where we could store up to 25 times the amount of actual disk space we had available for virtual tape.
Since then, almost all virtual tape vendors have some form of data deduplication. I guess this is a flaming hot feature. The big-irons are lagging behind a bit, but no doubt they will soon step up, and try to dominate this field too.

The one thing I wonder about is, since storage is this hot, why the disk vendors aren’t incorperating data deduplication in their subsystems?

• Is it  the fact that they will loose revenue on the disk sales? (my best guess).
• Is it because it is to hard to write the code? I don’t think this is a problem though.

Computer room cooling, energy costs, floor space should all be factors to take into consideration. These factors would justify data deduplication on disk storage from an end-user point of view. Vendors always find ways to increase their profits, even though the hardware prices are dropping constantly. I see no obstacle here.

Up untill 2003 we were using IBM’s RVA (StorageTek OEM)  on mainframe systems. The logstructured volumes were perfect. Compression was done in de RVA, without the host knowing it. We were able to store about three times the amount of data than was actually present in the box. Need more volumes. No problem. We’d carve out another model 9 volume (9GB), and no actual physical disk space was consumed, untill some data was stored on it.
When StorageTek came out with the successor of the RVA, called the V2X, we were eager to use it because it also supported Open systems lun’s.

Unfortunately the design of the V2X had some flaws. Especially in the microcode. The V2X wasn’t stable enough for us to run open systems volumes on it. The compression and snapshot mechanisms worked fine, but paths from the host to the V2X kept on dropping connections. We decided to send it back to STK, and we continued our storage services on the IBM Sharks.

So doing compression and deduplication in the storage box is just a heartbeat away. Now we just need to wait for the first vendor to pick up the gauntlet and start shipping the storage box including dedupe code. The rest will soon follow.

Need someone to test for you? Just give me a call

I just read this post on SANgod.com.
I must admit, you have to have some balls to call yourself a god , but if it works for him…..

But i had a deja-vu when reading it. We don’t use EMC, but the issues we have with M$ VSS and FlashCopy are about the same. I did some integration of VSS to the IBM San Volume Controller, just to try it out. We use IBM’s Tivoli Storage Manager for backup, but in fact, that doesn’t even matter.

Consider a M$ Windows Server 2003 host, booting of a SAN volume.
When we issue a backup of the M$ OS, including the Registry, System State, System Services and all that crap M$ automatically uses VSS to create snapshots of the Registry, System State en System Services files. When booting from SAN and enabling software connection from VSS to the storage, VSS actually goes into the storage system and issues the FlashCopy mechanism to create the copy of those files. But, instead of only cloning the files associated to the M$ services, it goes in and takes a complete copy of the entire LUN. Because the System State en the System Services are not cloned simultaneously, but serially, the boot LUN is actually copied several times.
One could argue that this is not a M$ issue, but more an IBM FlashCopy issue. I will probably agree on this. But still. Why clone a complete LUN, when you only need a couple of files. Even worse… The VSS service waits for the background flashcopy process to completely finish, before actually giving control back to the backup tool. This way, a normal (no VSS connection into the storage) backup of the OS completes in about 5 to 10 minutes. When using the VSS support to  perform FlashCopy tasks in the storage box, that same backup takes up 45 to 60 minutes. This setup doesn’t “do” it for me.

When the backup completes, the copy-LUNS are disconnected and no longer available for other purposes.
This is a shame. One could put these volumes to good use for other scenarios.

It’s Friday afternoon where i live, and i took the afternoon off, to be with my wife. Surprise i said when i got home. I surprised only my dog. The kids are in school, and the wife is obviously gone somewhere. So i sit here, with my Senseo coffee, thinking about the discussion i had earlier this morning at work.

I wanted to design some flash-copy scripts against our IBM San Volume Controller, in order to perform some flash-copy tasks for the Windows servers. In the forum i already started a new topic about this. So i talked to my partner about how i intended to do it, and what my obstacles were. I wanted to know how the disk (read volume) was mounted on the Windows host. But i had no method in matching the unique ID’s i got from the SVC tot the disks in the host. Windows doesn’t keep track of the LUN’s serial numbers without installing third-party tools. How could I possibly know where to put the target disks. I want them to be easily recognized by the path they are mounted in.

An example.
The source host (called HOST01) has 2 disks, one is mounted as E: and the other is mounted as F:\Subdir\Mountpoint. Let’s not make it to complex, and have both volumes comprised out of only one disk, so no spanning here. My target host is another server called HOST02.
I want the targets to be recognized by the path on HOST02. All FC targets will be mounted somewhere under the drive letter F: which tells me it are FC volumes. Something psychological i guess. That would then look something like this:
HOST02 -> F: \ HOST01 \ E
HOST02 -> F: \ HOST01 \ F \ Subdir \ Mountpoint

All the information on the paths and drive letters have to be dynamically gathered. Quite the task i tell you. But given my know how, i should be able to pull it off. By doing this, i tried to solve my flash-copy problems on Windows, and simultaneously make a completely new backup design. My partner nods his head. Don’t make it this complex he said. Just try to pull off the flash-copy first, without considering all the variables. We are in the middle of redesigning our infrastructure anyway. We want to ban the current methods of doing our DR backups, and recoveries. So we are planning on doing a full flash-copy to a remote location of all our data anyway. So, just have the SVC do it to all the LUNs it knows about. Hmm, i think. But i want to make the file level backups easier as well i say.
Even though it’s hard to admit, he’s got a point there. Why change the complete file level backup methods, when you can have all the data flash-copied with minimal amount of effort? After having accomplished the easy part, you still start on working on the way we process file level backups he said, but there’s no pressure there. And you know what? He’s right. So i will probably propose the new insights to my management.
Guess those mainframe storage admins have an advantage on the OpenSystems guys when it comes to simplicity. Why do the OpenSystems admins tend to choose to do stuff the hard way?