BNET Business Network:
BNET
TechRepublic
ZDNet

August 21st, 2008

Open Source, Dtrace, and tuning

Posted by Paul Murphy @ 12:15 am

Categories: General, Applications, Development, Enterprise Policy, Database Management, What users care about, Wintel vs Lintel

Tags: Performance, MySQL, Memory, Lock, Performance Management, Open Source, Databases, Human Resources, Workforce Management, Enterprise Software

In the good old days being an Oracle performance tuning expert paid pretty good money - and it was easy: because most problems were consequences of tuning expertise, the right answer was usually to set things to their defaults, switch to raw devices, and grab as much memory as possible before running out the door with another job well done. Unfortunately for me, memory got to be cheaper than expertise - and now I see people running a few million rows in 32GB of RAM.

On the other hand you do still see people whose applications max out their computers and who’re therefore facing a choice between committing some expertise to buying time by making the thing run better on the existing hardware or going back to the bosses for emergency hardware - and licensing- upgrades.

If you’re in that position - and almost every production systems manager gets into it sooner or later - there’s an important lesson to be learned from a Sun guy’s blogging on work he and his team did to find and remediate a locking problem with MySQL.

The entire story is on Neelakanth Nadgir’s blog - Here’s his introduction followed by his action summary:

While comparing sysbench runs using two different versions of MySQL 5.1, I noticed a big difference in the system utilization. One version had much more idle time than the other. This difference was much more apparent with high thread counts on machines with lots of CPU. A look at the system calls showed a large number of lwp_park system calls. That is, threads were being put to sleep.

[lots of detail on what they did and how]

So the mysql server is requesting ha_innobase::info() to not hold a lock, and it is being ignored by ha_innobase::info()!.

I compared against MySQL 5.0 and saw that this particular lock was not held when ha_innobase::info() was called. Searching through the commit logs I found that this was introduced by Bug#32440.

Quickly hacking the code to revert to the old behavior gave a big boost in performance. Hence I filed Bug #38185 ha_innobase::info can hold locks even when called with HA_STATUS_NO_LOCK. Luckily its a very small change and a fix is already in progress.

Moral of the story? A simple contended lock can bring down your performance by quite a lot.

It’s his story, but he gets the moral of the thing completely wrong: this isn’t about how a lock affected performance, it’s about how open source made it possible to identify and remediate a problem that in a proprietary context would probably have been best addressed by buying more hardware and/or more expensive licensing.

And there’s a bottom line lesson too: open source coupled with expertise opens the door to increased performance and flexibility (customization) in meeting unusual requirements today in much the same way that cheaper memory did in the nineties.

August 20th, 2008

The opportunities in search

Posted by Paul Murphy @ 12:15 am

Categories: Unanswered questions

Tags: Google Inc., Advertisement, Corporate Communications, Marketing, Paul Murphy

A couple of weeks ago I got the cuil.com press release touting their company as offering the latest and greatest in search.

After a few tries I didn’t think their front page cool and didn’t think their search better than google’s either - and that’s too bad because google is the most over priced company in history and a little real competition would be good for them.

When you look closely at google’s search product it turns out to be basically just 1950s boolean text search ideas (cf COLEX (1958), SDC Dialog (1967) and BRS Search on BSD (1984)) implemented in a grid file framework with a simple front end and an automated data collection backend. Cool, but a breakthrough in business and presentation, not technology. And Cuil? looks like less of the same to me - at least so far.

Three things about this situation bother me:

the technology needed is well understood. It takes money, not genius, to go after google’s market using a comparable technology - and yet Yahoo suicides, Microsoft dawdles, and Cuil, isn’t. search seems to have lots of room for genius - for someone to improve it to the point that the improvements can replace money in the drive to business success. The person with the idea exists somewhere - so why haven’t we heard about it? the technology underlying the current financial model for search is extremely weak. Google makes most of its money selling eyeballs on ads, but an effective ad should provide value to the viewer, and google’s ad matching almost never does that.

Back in the mid nineties when Illustra got spun out of the UCB Postgres group it came with text and image data blades and one of the options was a video indexing and retrieval blade (plugin) out of Stanford that let you highlight an image and then search a video for occurances of that image - something you’d think You Tube would kill for.

It was neither efficient nor 100% accurate, but it was a mid nineties out of the box solution for the two kinds of search google does best and one it doesn’t currently do at all: image matching. More subtly, the datablade technology it came with would make it fairly easy to incorporate much more information about the user into ad selection and placement - thus offering greater value to both users and advertisers.

The main problem with image matching is that you have to convert the input images to a form that’s independent of both perspective and scale while not trusting color as primary discriminator - something you can do by computing the minimal set of normal vectors for every contiguous color surface, because that produces a unique description of the object’s shape whose determinant happily functions as your primary hash key. Since both Cell and T2 are now fast enough to do this in real time for video input, it would now be possible to resurrect that technology, improve it a little, and enable someone like me to upload a single image for network search.

A couple of weeks ago, for example, I bought a little statuette at a garage sale for $5.00.

It’s signed, apparently as “Alfred”, appears to be a thin, somewhat patina-ed, bronze coating over some light ceramic like terra-cotta, and is numbered X18 - where the “X” is unreadable but could be almost anything from a dash to an A, F, or 8.

What I’d like to do is upload that image to a search engine and have it return as hits sites that have photos of very similar things - and therefore might have the information I need to see whether I overpaid for junk with spray on patina or got something others see as valuable.

But I can’t: Google’s image search is based on text labels, cuil doesn’t seem to have gotten there at all -and neither Microsoft nor Yahoo are in the game.

And that, I think, is the bottom line on Cuil and the state of search: to the extent that it’s about money, it’s very 1950ish - and to the extent that it should be about making use of technology it’s a whole bunch of technical and financial niches waiting for someone to fill them.

August 19th, 2008

Excel and Basic Accounting Error

Posted by Paul Murphy @ 12:15 am

Categories: General, Applications, Enterprise Policy, Government, Unanswered questions, Wintel vs Lintel

Tags: Revenue, Accounting, Restatement, Compliance, Microsoft Excel, Spreadsheet, Operational Accounting, Revenue Recognition, Productivity, Financial Services

From CA Magazine:

Do spreadsheets lead to compliance failures?

A full 92% of all US public companies use spreadsheets for critical accounting activities in their revenue reporting processes, according to a recent survey of financial executives. And that increases the likelihood of compliance failures and financial restatements. The research, which involved 685 companies, was conducted by www.RevenueRecognition.com and IDC and sponsored by Softrax Corp.

Revenue spreadsheets: the compliance killers

The reason for widespread spreadsheet use, says the survey, is that key revenue recognition and reporting tasks are still not automated in financial/ERP systems. Only 8% of all responding companies say they are able to complete their revenue reporting process without having to take data offline and into spreadsheets. The rest of the surveyed companies use spreadsheets, which are prone to errors, lack audit capabilities and resist internal controls.

According to the survey, more than half of all companies use spreadsheets to create their accounting entries for revenue. Other spreadsheet-based tasks include revenue scheduling, allocation and redistribution based on accounting guidelines. Surprisingly, public companies with more than $200 million in revenue are substantially more reliant than the overall sample on spreadsheets for revenue accounting entries.

From Revenue Recognition.com:

Revenue Recognition Restatements Increase 42% as Basic Accounting Errors Plague Reporting

Revenue recognition restatements increased 42% from 2002 to 2006 leading to a lot of speculation about the underlying causes. With all the hype about Sarbanes-Oxley, increased auditor scrutiny, and complex guidelines it is surprising to learn that mundane internal errors were the leading cause of restatements from 2003 to 2006. That’s the conclusion of a new report entitled “An Analysis of the Underlying Causes of Restatements” by Marlene Plumlee, University of Utah and Teri Lombardi Yohn from Indiana University.

The authors analyzed 3,744 disclosures related to each restatement to identify and categorize the underlying causes. Of restatements caused by revenue recognition errors, approximately: 57% were due to a basic internal error, 28% were due to some characteristic of the accounting standards, 13% were due to intentional manipulation, and 2% were due to transaction complexity. That would make over half of all revenue recognition restatements avoidable if companies had better procedures for performing, monitoring, and controlling their revenue processes.

Bottom line?

User spreadsheet abuse leads to compliance failure - and can be largely attributed to IT’s failure to gain top management support for the provision and use of adequate integrated ERP/SCM and related functionality in central systems.

Speculation: the symptoms, frequent restatement requirements, are heavily correlated with large client-server installs. And why do I think that? because if you implement something like Oracle’s combined ERP/SCM suite on Sun with tools like Hyperion for real time, cross functional, reporting you won’t need Excel or any other PC client tools to get your reports out - but if you implement that same suite using Windows client-server you not only invite “power users” into the hen house, but I don’t think you can actually get the reports out without them.

August 18th, 2008

Compliance and IT

Posted by Paul Murphy @ 12:15 am

Categories: General, Enterprise Policy, Strategy, Government

Tags: Information Technology, Compliance, Strategy, Management, Paul Murphy

Some common wisdom:

Legal defense costs aren’t IT costs. For example, a discovery requirement initiated as part of a lawsuit over a non IT related matter can create huge staff and legal costs but can’t be attributed to IT. A regulatory, court, or government mandated product recall that has nothing to do with IT isn’t chargeable to IT. For example a fourteen million pound hamburger recall imitated because it can’t be shown that no material from a diseased animal was ever allowed into or near the production process is not attributable to IT. personal costs incurred by employees because a third party lost a laptop full of personnel data aren’t attributable to IT. For example, the internal IT people can’t be held responsible if the treasury department hired a name brand accounting firm to review its pensions management options and one of that firm’s professionals lost a laptop loaded with employee payroll data.

In all three cases the common wisdom is dead wrong - these are all IT failures. More subtly, these all resulted from failures by top management to tell their IT people to do the right things to avoid these problems - and arguably, therefore, these are the fault of the IT people who didn’t successfully sell top management on the need to authorize and fund appropriate pre-emptive measures.

The first one’s easy: every document, every record from the phone switch to board minutes, should go on a write once device, be duplicated once, with both copies stored separately on removable media tracked using standard chain of evidence methods. For Intel the costs wouldn’t have amounted to a million a year - and for the average company with three or four sites and a few thousand employees it’s typically in the hundred thousand a year range.

The second one depends on what’s installed at the packing plants. Fundamentally it’s not hard to track most cuts from the animal to the retailer, but things get rather more difficult on standardized, higher volume, composite products like hamburger and sausages where the right answer involves breaking production into batches separated by environmental and machine testing. That’s practical with modern automated gear but impractical with older stuff - so if you’ve got older gear and manual processes remediation starts with plant floor change, but all of it gets driven from IT abilities to limit the costs of compliance.

The third one is the most directly IT related - and correspondingly easy to deal with: a matter of getting top level management to accept and enforce sensible policies on data access.

Notice that all three examples, (and as many more as you may want to come up with) require top management to either take, or agree to and enforce, IT action. To get them to do it, focus on the cost of litigation and related insurance, and go from there to whatever intangible costs - like loss of market credibility for them as well as the company - apply in your business.

Now as far as I know - which isn’t very far given that I’m distant from these kinds of discussions - no major insurer currently focuses on positive IT action in terms of risk reduction and loss prevention, but all of the majors have people who provide risk reviews and offer to help customers understand and mitigate risk. So talk to your own senior managers first, then get your insurer involved - because the bottom line is simple: it can’t hurt to do your homework and you could end with some additional budget and a lot more credibility in the executive suite.

August 16th, 2008

Exploits, vulnerabilities, and questions

Posted by Paul Murphy @ 12:15 am

Categories: General, Linux, Enterprise Policy, Unanswered questions, Wintel vs Lintel

Tags: Paul Murphy, Vulnerability, Security, Viruses And Worms

Some time ago Jeffrey Jones, “a Security Strategy Director in Microsoft’s Trustworthy Computing group”, of PC style security vulnerability patches issued during the first quarter of 2008 for two Windows variants, two Linux variants, and two MacOS X releases.

Here’s what he says about what he includes:

Note that I will not be counting every vulnerability that affects the hundreds of optional application components that ship with the Linux distributions. Instead, for both Red Hat and the Ubuntu products, I intall using the desktop installation defaults (which excludes most of the optional packages) and additionally:

excluding ‘Office’ packages (e.g. OpenOffice, Evolution, Thunderbird), since Microsoft Office is not included with the Windows client operating systems excluding ‘Graphics’ packages (e.g. Gimp, ImageMagick), since Microsoft Expression products are not included with Windows client operating systems Note that this process means that Apache, MySQL and all of those optional ’server’ components are not installed either. After installation, I use the appropriate package management tool (ie, rpm or dpkg) to list out the actual packages installed and use that to filter on affected components.

After extensive work he arrived at the following numbers:

Client OS Vulnerabilities fixed Security advisories Patch events
Windows Vista 9 6 2
Windows XP 12 8 2
Red Hat RHEL 5 (reduced) 60 19 12
Red Hat RHEL 4 (reduced) 75 18 14
Ubuntu 6.06LTS (reduced) 54 15 13
Mac OS X 10.5 Leopard 83 6 5
MacOS X 10.4 Tiger 81 5 5

Since numbers showing an overwhelming security advantage for Windows over both Linux and MacOS X are, at least for me, somewhat counter-intuitive, the obvious question is whether or not he’s cooking the books here.

The big problem in doing this kind of this is, as he says in this report, that no two different OSes are directly comparable in terms of the content of default installs - and I’ll add that the absence of effective standards on defining and counting either vulnerabilities or patches makes things even harder. Thus one group’s “critical” can look unimportant to another, root cause patches affecting many vulnerabilities can be counted as one or many, and technologies considered part of the base OS in one community may have no counterparts in the other.

There is, for example, really no such thing as a Linux “client”: Linux is Linux and works the same way whether you put it on your desktop or in the server room - meaning that any comparison between Windows and Linux “clients” is flawed from the gitgo.

What’s needed to fix this is, of course, some measure of realized risk - the expected costs of loss and remediation for each exploit.

(Note too that only exploits count because a vulnerability indicates that a risk exists, but that risk cannot become a cost without an exploit - something that’s so trivial in the x86 world that the terms have become virtually synonymous there. Outside x86, however, a vulnerability by itself is about as useful as Viagra without a girlfriend - this is why Apple has been in panic patch mode on their x86 products since their first release, but only updates the same MacOS X for the iPhone’s firmware every three to four months.)

We don’t have such a measure, and Jones doesn’t provide a list of the patches and vulnerabilities he counted, but if you follow the references he provides to the source listings at Microsoft, Red Hat, Ubantu, and Apple you can get some hints about what’s going on.

For example, I think he could reasonably have counted some version of RHSA-2008:0164-6 (covering CVE-2008-0062, CVE-2008-0063, and CVE-2008-0947) against at least three of the five Unix variants - here’s part of Red Hat’s writeup explaining it:

Updated krb5 packages that resolve several issues and fix multiple bugs are now available for Red Hat Enterprise Linux 5.

This update has been rated as having critical security impact by the Red Hat Security Response Team.

Kerberos is a network authentication system which allows clients and servers to authenticate to each other through use of symmetric encryption and a trusted third party, the KDC.

A flaw was found in the way the MIT Kerberos Authentication Service and Key Distribution Center server (krb5kdc) handled Kerberos v4 protocol packets. An unauthenticated remote attacker could use this flaw to crash the krb5kdc daemon, disclose portions of its memory, or possibly execute arbitrary code using malformed or truncated Kerberos v4 protocol requests. (CVE-2008-0062, CVE-2008-0063)

This issue only affected krb5kdc with Kerberos v4 protocol compatibility enabled, which is the default setting on Red Hat Enterprise Linux 4. Kerberos v4 protocol support can be disabled by adding “v4_mode=none” (without the quotes) to the “[kdcdefaults]” section of /var/kerberos/krb5kdc/kdc.conf.

Similarly, I think he’d reasonably have counted Microsoft’s vulnerability MS08-008 (Vulnerability in OLE Automation Could Allow Remote Code Execution) against both XP and Vista. Here’s part of their write-up:

This critical security update resolves a privately reported vulnerability. This vulnerability could allow remote code execution if a user viewed a specially crafted Web page. The vulnerability could be exploited through attacks on Object Linking and Embedding (OLE) Automation. Users whose accounts are configured to have fewer user rights on the system could be less impacted than users who operate with administrative user rights.

If so he could have counted the Unix vulnerability noted above at least nine times even though it’s a minor bug affecting few users; and the Windows one only twice, even though it directly threatened users on every supported Windows OS product.

If so, you’d expect the combination of some new code with lots of migrated XP code to produce more bugs in Vista than in XP - but that’s not what he reports.

To me this means either that he’s cooking the books by counting vulnerabilities as CVEs for Linux and MacOS X, but only once, and only against the originating OS, for Windows; or, that Microsoft’s work on Vista including finding and fixing XP bugs that they choose not to report or fix in XP.

To find out which it is I sent him an email asking for the list of CVEs counted for each category and promptly got a very nice response:

I am not in the office right now, but you can generate the list yourself by examining all of the security advisories released by each vendor. It shouldn’t take that long to validate the Mac OS X Leopard totals, for example, to give you confidence of my numbers The link is - http://support.apple.com/kb/HT1222, which contains only 16 security advisories in Q1.

If you click on the last one in June (http://support.apple.com/kb/HT2163), you can identify 25 vulnerabilities by CVE id, but you will only want to count the ones where they say it applies to Mac OS X v10.5 (some do not).

It is straightforward.

Unfortunately that just duplicates information in the report and therefore doesn’t help, so my provisional bottom line on this is that either I’m missing something important (?), he’s cooking the books to favor Vista, and/or work on Vista revealed bugs in XP Microsoft choose not to fix until they were found and made public by third parties.

August 15th, 2008

From Chapter one: Data Processing and the IBM Mainframe

Posted by Paul Murphy @ 12:15 am

Categories: Defenestration

Tags: IBM Mainframe, PC, Mainframe, Data-processing, IBM Corp., Data Management, Help Desk, Data Centers, Service Level Management, Call Centers

This is the 13th excerpt from the second book in the Defen series: BIT: Business Information Technology: Foundations, Infrastructure, and Culture

Note that the section this is taken from, on the evolution of the data processing culture, includes numerous illustrations and note tables omitted here.

Roots (Part Four: sample System 360 best practices)

Clearly Defined Line Management Structure with rigid role separation

At a minimum there should be:

An operations unit responsible for day to day execution of scheduled processor tasks; A control group responsible for data collection and other user input; A systems development group responsible for development work; A help desk manager responsible for PC operations; A capacity and utilization management unit; An end user support manager responsible for business applications operations; A data management unit containing a data architect, data manager, and one or more database administrators; A technical support group responsible for managing and updating system software; A license and documentation management unit responsible for tracking PC licensing and application documentation; A contracts manager; A security and related (personnel) policies administrator; and, A PC systems administration unit responsible for PC hardware and software.

SLA includes annually budgeted operations

The service level agreement is the contract between the data center and the user community. This is the peace treaty in the battle for resources and control between user groups and the data center. As such it governs expectations and is renegotiated annually as part of the budget process.



The SLA should be integrated with the overall systems governance process and be administered by a systems steering committee including members of the senior executive.

Clearly documented SDLC standards

Data centers that run only packaged applications tend to stagnate. The growth and service potential is in new development, new deployments, and the discharge of ever increasing corporate responsibilities.

Early System 360 adopters generally underestimated development complexities and limitations, and therefore tended to over promise. As most projects failed while a few succeeded the critical success factors for developers soon became clear and, high among these, was the use of clearly enunciated and strongly enforced systems development lifecycle methodology or SDLC.

Developers who obtained user sign-off at each stage of a project’s lifetime and then incorporated the resulting expectations into service level agreements generally found that users who had been co-opted during project design accepted weaker results as successes and were less likely to rebel at budget increases.

The typical SDLC is defined in terms of steps leading to deliverables and sign-offs rather than working code or reviewable systems documentation. Many of these steps are inherently technical but the focus is on the signoffs and processes rather than the contents of each deliverable, thus decoupling the systems development management process from systems development and testing.

“Lights out” 24 x 7 operation

Automated, or “lights out” operation is normally presented as a means of saving costs - not having to run a night shift means not paying those salaries. But, in reality, people assigned operational functions during these shifts tend to be low cost, so savings are usually neligible on the scale of the overall data center budget.

The management value of lights out operation as a best practice derives from something else entirely: the fact that it is functionally impossible to achieve this without first implementing a series of related practices ranging from proper management of job scheduling, to accurate capacity planning, effective abend minimization, and automated report distribution.

Use of Automated Tape Library

Use of an automated tape library coupled with vaulted third party off-site storage for backups is a common best practice mainly because it reduces both data loss and tape mount errors.

Disaster Recovery or Business Continuity Plan

A documented disaster recovery plan must exist.

The traditional first step in a mainframe disaster recovery planning effort is the classification of systems (meaning applications groups) according to the severity of the impacts associated with processing failure. Thus most plans are ultimately predicated on the time frames within which processing is to resume for each of a set of jobs grouped according to headers like Critical, Vital, Sensitive, or Non Critical.

The more common recovery strategies are built around:

Hot site agreements with commercial service organizations under which the company regularly transfers tapes to the hot site operator and the site operator assures the company of access to physical and processing facilities for the duration of any emergency.

Hot site agreements come in multiple “temperatures” with a cold site, for example offering little more than space and a physical facility without having any of the company’s code preloaded or communications links pre-tested.

Internal systems duplication in which the company maintains two or more independent data centers and uses each as backup for the other.



Disasters are extremely rare. When they do occur weaknesses in the recovery plan are usually found in one or more of three main places:

The materials needed to resume processing - including things like network information, back-up applications, libraries, and data, licensing, or report distribution maps - turn out not to have been updated sufficiently recently to allow full functionality to resume without significant and unexpected recovery effort; The organizational effort to re-route manpower and re-assign personnel to the interim facility often turns out to be much greater than expected and an initial period of apparent chaos ensues as roles, assignments, and authorities are worked out; and, Third party access to, or from, the interim facility often fails; resulting in missed file transfers which, in turn affect scheduled batches and thus the applications in which those batch runs figure start to “go off the rails” - ultimately requiring database rollback and imposing extensive re-processing on users.

As a result it is common in real processing disasters to find the data center director reporting full functionality at the interim site several days before users can resume normal operations.


Some notes:

These excerpts don’t (usually) include footnotes and most illustrations have been dropped as simply too hard to insert correctly. (The wordpress html “editor” as used here enables a limited html subset and is implemented to force frustrations like the CPM line delimiters from MS-DOS). The feedback I’m looking for is what you guys do best: call me on mistakes, add thoughts/corrections on stuff I’ve missed or gotten wrong, and generally help make the thing better.

Notice that getting the facts right is particularly important for BIT - and that the length of the thing plus the complexity of the terminology and ideas introduced suggest that any explanatory anecdotes anyone may want to contribute could be valuable.

When I make changes suggested in the comments, I make those changes only in the original, not in the excerpts reproduced here.

August 14th, 2008

Development environments: Microsoft vs. Open Source

Posted by Paul Murphy @ 12:15 am

Categories: General, Development, Enterprise Policy, Wintel vs Lintel

Tags: Hiring, Wintel, Data Center, Open Source, Information Technology, Environment, Sun Microsystems Inc., Microsoft Corp., Sun Ray, Development Environment

As we saw yesterday the much vaunted Unix skills premium over Windows is pretty small - 15% or so in an overheated market and less than that elsewhere.

Notice, however, that this information pertains only to larger organizations: people who can pay their neighborhood pretend-a-geek $25 per hour to futz with their Windows machines are generally convinced that’s a lot cheaper than paying a formal IT labor retailer $180 and up for insurable skills.

Underlying that belief is an assumption: that the services have comparable value, and while I doubt that, this is another area where the data needed to know for sure simply doesn’t exist. For larger businesses, however, we have lots of numbers - and their consequences are mind boggling.

Read the rest of this entry »

August 13th, 2008

The Unix sysadmin salary premium

Posted by Paul Murphy @ 12:15 am

Categories: General, Enterprise Policy, Government, Wintel vs Lintel

Tags: System Administrator, Wintel, Data-processing, Salary, Payscale, Outliers, Sun Solaris, UNIX, Operating Systems, Linux

According to payscale.com the “typical Salary for a Sun Solaris System Administrator in United States is $56,082 - $85,226.”

Payscale’s software and data don’t provide directly comparable Linux and Windows systems administration numbers for the whole country, but approximations suggest that the Wintel range is from $48,000 to $75,000 and that for Linux about $51,000 to $76,000.

Outliers are more interesting. In the somewhat over heated Fairfax county market Payscale’s colleagues at simplyhired report a Solaris sysadmin average of $102,000, a Linux average of $95,000, and a Windows average of $87,000.

Those numbers, and lots of others I looked at from major U.S. market players, suggest that the “Unix Premium” runs no more than about 15% nationally and the salaries overlap for better than 90% of the range - meaning that the people who argue that lots of big companies prefer Windows over Unix because Wintel sysadmin staff are a lot cheaper are simply wrong.

In the course of an afternoon wasted reviewing job ads and numbers I came across, however, something worth futher exploration. Specifically, almost every Unix ad written by someone with a clue required additional skills, usually with respect to Wintel and/or RDBMS administration - and comparable Wintel ads did not.

As a result I ended up convinced that the so called “Unix premium” does not reflect the additional cost of hiring Unix skills at all but, instead, reflects the typical Unix sysadmin’s ability to provide both the time and skills required to cover a much broader range of responsibilities.

If, for example, you have three qualified Solaris sysadmins each of whom can also act as your Oracle DBA, then the additional 15% per year each of the three Solaris guys costs you still amounts to less than half what that Oracle guy you would have cost you in a Wintel server environment - and because the skills are spread broadly, you get 24 x 7 and vacation coverage on Oracle at no additional cost.

And that, I think, is really the bottom line: not only is the Unix salary premium mostly a myth, but on net the Unix guys cost less than Windows people because broader skills and higher productivity mean that many fewer are needed to achieve the same or higher system wide service levels.


Some notes:
Some Unix ads have, presumably unintentional, comic aspects. Dice, for example, has this “8-9-2008″ listing:

Title: UNIX, LINUX, SOLARIS Systems Administrator 116528

Skills:

Education Requirements: Bachelor’s degree in Computer Science/Engineering preferred. Excellent knowledge of Windows 2000, Windows XP, Windows Server 2000/2003, TCP/IP networking technologies. MCP\MCSE preferred but not required.

Description:

MUST HAVE EXPERIENCE WITH UNIX, LINUX, SOLARIS, ADMINISTRATION.

Job Description: 3+ year’s experience with IT systems technology, hardware installation and configuration of desktop and peripheral technology preferred.

Good understanding of IT hardware and installation model concepts.

Candidates should have a strong knowledge of LAN, WAN, internetworking technologies, TCP/IP, and DNS.

Skills/Experience: Key Responsibilities include, but are not limited to: Installation of hardware and software in current lab systems.

Support standard hardware platforms and systems installed throughout QCT engineering labs.

Trouble shooting and resolution of business application and system problems.

Trouble shooting and resolution of engineering lab Systems related to RF test equipment.

Installation of MS security patches and Virus protection software.

Familiar with HP, IBM and Dell hardware. Certifications a plus.

I would have used Computerworld’s 2007 Salary data for this blog except that their input surveys continue the data processing tradition under which systems administrators are treated as clerks. That was right for data processing’s machine operations in the 1920s, but wrong for Unix - and it reflects one reason almost all data processing attempts to work with Solaris fail. Payscale needs some help with their own systems -the stuttering grammer suggests a simple minded $i style application, while changing the search criterion from Solaris to Linux produces this:

The typical Hourly Rate for an Office Administrator in Washington-Arlington-Alexandria, District of Columbia / Virginia / Maryland / West Virginia Metropolitan Area is $13.22 - $19.03.

On the other hand Adobe FrameMaker’s spell checker changes “Word Perfect” to “notepad” and “Microsoft” to “Microstate” -so who knows how clever somebody at payscale might really be?

Those of you who enjoy the GNU naming model might note the strength of character it took not to structure some sentence about the inter-disciplinary skills the typical Unix sysadmin brings an employer around the phrase “contain multitudes”.

August 12th, 2008

When equivalence is the wrong question

Posted by Paul Murphy @ 12:15 am

Categories: General, Applications, Development, Linux, Enterprise Policy, Apple, Wintel vs Lintel

Tags: Microsoft Corp., Tool, Desktop Publishing, Productivity, Microsoft Word, Word Processors, Open Source, Software, Microsoft Office, Office Suites

Probably the single most common mistake people make when assessing open source tool choices against the Microsoft toolset is to seek direct functional equivalence.

You see a popular variation of this among people who decry open source word processors like Writer because these don’t allow the kind of near desktop publishing Microsoft’s Word can be stretched to accommodate.

What’s really going on, however, is that the Microsoft tool is the wrong tool for the job - and when these people demand equivelance from the open source community they’re really asking us to legitimize and perpetuate their mistakes.

It’s easy, of course, to see how this evolves: early Word users got assignments requiring them to stretch its appropriate use just a little further - and a little further - and pretty soon lots of users had learning and ego investments in pushing the envelope, quality compromises got institutionalized, Microsoft responded by stacking another floor on tottering foundations - and the communications gap between Word users pretending to desktop publishing and the people who know what the results should be, and use the right tools to get them, just got larger and larger.

At some point what you get out of that process is business applications written in Access; statistical applications written for Excel; thundering MCSE denunciations of guys like me who think this almost criminally absurd; and mounting corporate loses in accountability, compliance, data security, and organizational productivity.

Using Microsoft Word in desktop publishing is a classic example of the co-evolution of a product, its markets, human barriers to change, and a learning curve gone wrong: it’s easy to get into but impossible to do well -and the rejection of open source tools because they get stuff like this right ultimately just an almost inconsequential component of the overall productivity loss this imposes on its victims.

In the general case it all comes down to this: a house made of straw has to have a design that’s appropriate to straw - and if you change to brick you can build a better, stronger, house that will last longer - but you can’t just replace a few bales with bricks, you can’t use the same design, and you better re-assess the foundations before starting work.

August 11th, 2008

Second guessing at 3AM

Posted by Paul Murphy @ 12:15 am

Categories: General, Linux, Enterprise Policy, Strategy, Sun

Tags: risk, business ethics, strategy, professional development, security, leadership, management, career, paul murphy

An imaginary, but very real, diary entry:

It’s 2:52 AM - and I’ve been wide awake all night; tossing, turning, sweating. Why? because the decision I’m planning to announce tomorrow is the right one for the company and one we’ll make work - and work well. But - it goes against popular wisdom, lots of senior people are going to be sure it’s wrong, some of my own staff will express doubts when I’m not there, and when I leave the company, they’ll hire someone who buys executive support by sneering at our work and getting funding to reverse it.

I’ve been in the diarist’s position - and so has every IT manager who’s decided not to “buy from IBM” - or, more recently, Microsoft.

At the immediate heart of this is a fundamental conflict: a decision made for the objective benefit of the employer can carry serious personal risks - and every decison maker has to balance risks to the employer against risks to himself.

At a deeper level this is really about balancing situational ethics against the real thing - and in that context you should remember that had a moral absolutist like Churchill or Truman been in place instead of either Chamberlain or Roosevelt in the 1930s, the holocaust would probably not have happened - and neither would seven million deaths in the Ukraine.

A few thousand years ago -more or less- Horatius (along with unheralded colleagues Laritus and Herminius) saved the city of Lars Porsena from sack and ruin by blocking a bridge needed by an army of 50,000 Tuscan looters long enough for it to be destroyed - but when it comes to your career in IT management which looks more attractive: playing Horatius every single day of your working life, or hanging around with the Tuscan quarter masters and camp followers?

The bottom line is that the fear of being seen as out of step with a majority is a very important decision influencer - because, at three AM, what really counts isn’t whether a million blondes can be wrong, but how much you care.

Paul Murphy (a pseudonym) is an IT consultant specializing in Unix and related technologies. See his full profile and disclosure of his industry affiliations.

advertisement
[ http://bwp.zdnet.com/search?dw-siteid=24

Recent Entries

Most Popular Posts

Archives

ZDNet Blogs

Sports and Technology Major League Baseball pitches new app to iPhone users At Apple's Worldwide Developers Conference in San Francisco, Jeremy Schoenherr of MLB.com demos At-Bat, a new iPhone app from Major League Baseball.
View the ZDNet video to learn more
The SF Giants' new hi-tech ballpark SF Giants CIO Bill Schlough discusses new technology upgrades at AT&T Park and outlines his dual role- managing technology operations at the backend while using hi-tech to improve player performance on the field.
View the ZDNet CIO Vision Series video
From our Sponsors Fantasy Football cbs sports 3 Great Ways To Play Fantasy Football Play for free, play to win cash prizes- up to $3500, or customize your own league.
Learn More »
cbs sports
advertisement
Click Here


You are viewing a mobilized version of this site...
View original page here

Mobilized by Mowser Mowser