Drizzle: Now with the InnoDB Plugin

The InnoDB plugin is now merged into the Drizzle mainline. There is still one weird outstanding bug that I know of. Why is this exciting? The main things found in the plugin that I’m excited about are:

• New compressed table format.
• Adding or dropping indexes can now bypass the table copy
• Information Schema plugins with InnoDB internals data

Not to mention, something tells me that the plugin will be getting more dev attention, and should be easier to track moving forward.

I’ve been doing my best to make the changes in such a way that they could be potentially merged back into the plugin upstream, should Oracle care to. (which I hope they will)

Help pay for drizzle.org

It just goes to show that code isn’t the only way to contribute. Mike Shadle recently did some leg work the Drizzle world by not only fronting the domain cost for drizzle.org, but also in actually negotiating down the guy who owned it already. That being done, the time has come (if you’re so inclined) to chip in and help defray Mike’s out-of-pocket.

The goal is to raise roughly $1000 USD to cover the domain + Escrow costs. I’ve already said I would contribute a chunk of that. Please include in the PayPal description your full name/company/whatever identifying information you’d like and if you’d like it recorded, and I will record it and if the Drizzle guys wish, we can post your info on the [not established yet] website as a Drizzle supporter.

If you’ve got an extra few buck lying around, please feel free to Help Establish a permanent address for the Drizzle

s/FAIL/ewwwww/

Antony points out:

Unfortunately, I had to work with the limitations of the C programming language and its preprocessor so to achieve a solution which behaved as if it had the strong typing like generics but working with simple C;

Yeah, I started out ranting about that whole system too, but decided against it since I knew the real culprit was, as is usual, requirements. I do not envy the task of implementing C++ generics in preprocessor macros. (I do look forward to going and reading the worklog though)

As Antony suggests, I am speaking in the context of Drizzle, which means I’m speaking from the vantage point of having change the underlying assumptions. We use C++ fully in Drizzle. So whereas in MySQL this code is, as Antony says, performing properly (quite amazingly well, actually) in Drizzle it’s rather FAIL. (Funny how changing design assumptions can do that). Since it’s hard and slightly unfair to reverse judge something old in the context of new assumptions, I revoke my “FAIL” and will just leave it at “ewwwwww”.

Antony’s right: for us this is ripe for replacement by some nice templates – as is sys_var, et al. Actually, I’m starting at sys_var and hope to have it sensible enough that plugins can just use it directly rather than having to have a separate system.

For Internal Use Only – FAIL

The following comment:

/* the following declarations are for internal use only */

is about 35% of the way down drizzled/plugin.h. How about:

If it’s for internal use only – DONT PUT IT IN A HEADER FILE!!!!!!!!! At least not a header file that’s used by things.

Free Software Understanding Follow-up

I would like to add two follow up thoughts to yesterday’s post about Free Software.

First, I was speaking with giuseppe on IRC who said:

however, the economy of free software goes beyond passion. You can make money with services, as you have done, personally, for years!

Giuseppe is, as usual, right on. I hope I did not imply that I thought it was impossible to make money or to make a living around Free Software, or that to endeavor to do such a thing was folly. What I was trying to get at is that pursuing the writing of software with the goal of becoming rich was a bad idea. It might happen, but if it does it will likely be due to luck.

I am currently paid by Sun Microsystems to write Free Software. I am thrilled about that, and think Sun are wonderful for supporting me. I would also love to have some chunk of Free Software that I write make me rich – but neither my employment by Sun nor my desire to have countless large estates worldwide have much to do with my desire to write Free Software. The two are largely orthogonal, although they do intersect at the point where one is beneficial to the other.

Second, I received a blog comment from thpi:

Don’t you think that you need professionals for the sciences to evolve?
The bare definition of a professional is related to getting paid.
I wonder if movies would’ve became what they have, if there was no money behind actors (directors, producers, etc).
Do you make a living writing code? Would you be as good as you are if you had to live on something else?
Maybe there’s a healthy middle between everything for a penny and all for free.

I would like to agree with this, then answer a question. I would like to end by agruing with the semantics of a word choice and rejecting a premise. (All in reverse order)

First of all. I agree. There is certainly a healthy middle between everything for a penny and all for free. As above, I think making money from writing software is fine, and I’m glad that I do. I’m more on about motivation – but we’ve covered that. Also, we have the ever-present problem in English of the overloading of the word Free. It’s been beaten to death, but here I’m talking about Liberty, not the eradication of Capitalism.

Secondly, yes, I do make a living writing code. Although it is quite difficult to prove in any sensible way, I don’t think that I would necessarily be worse at it if I did not make a living doing it. The more operative thought here might be that if I had a non-computer related day job, then I might not be programming as much. However, I currently only rarely design lighting for a show, perhaps only once or twice a year. I still get paid to do it when I do, although I do not light shows so that I will get paid. And I’m at least “skilled” to fit the other implied definition of the word professional above.

Which brings me to professional vs. amateur. Now, I know I have as much chance in reverting changes to English on this as I do on convincing the pop-culture folks that the word “hacker” does not mean someone who breaks into computers. But bear with me. To me, the word professional means someone who does something because they are paid to, and the word amateur means someone who does something by choice. Modern connotations there are that professionals are skilled and amateurs are unskilled. So there are two different possible meanings implied by the sentence “Don’t you think that you need professionals for the sciences to evolve?” Of course, I think that skilled individuals are necessary for the sciences to evolve. As I think I’ve already made it clear, I do not think that paid individuals are necessary. So I think that the words professional and amateur are unclear and potentially clouding in this discussion, and I reject the premise of attempting to entangle skill and employment status merely because we use the same word in English to describe both things. It’s the same problem we have with the entanglement of Freedom and Without Charge.

Of the 49 definitions of the word “free” that are listed in the Random House Dictionary, only 5 of them have anything to do with money or cost. I’d cite the OED, but their services are not without charge.

I won’t deny that monetary injections can remove obstacles and make things easier. All I’m really getting after is that I have gotten together with a group of fellow artists and put on a production of a three-act Sam Shepard play in a borrowed warehouse for no pay where we charged nothing at the door. We rehearsed 6 hours a day in an un-air conditioned warehouse in West Texas in May. It was dirty. It was hot. I was a lot of work. Why did we do it? Because we wanted to.

Will Free Software go away once the economy collapses? No. We may have to find friends who will let us borrow their warehouse for a little while. We may have to get day jobs farming or fishing or fending off looters – but we will keep doing it and our products will continue to be better than those produced the other way, because it’s what we want to do.

But please, by all means, make a living!

Free Software still isn’t Understood

I’d say I normally don’t respond to idiots… but I think we all know that’s a lie. Our fine friends at Slashdot pointed out an “interesting” article which opines that the downturning Economy to Give Open-Source a Good Thumping. There’s a bit in the middle that points out that people still don’t get it.

The hungry and cold unemployed masses aren’t going to continue giving away their intellectual labor on the Internet in the speculative hope that they might get some “back end” revenue. “Free” doesn’t fill anyone’s belly; it doesn’t warm anyone up.

The disconnect seems to continue to be an idea that we’re doing the Free Software in hopes that it might turn a profit one day. The people who share this disconnect seem to think Free Software is a “gimmick” of some sort.

I’ve got news for them… we do it for many reasons, none of them are “we think we’re going to get rich.” Of course, there isn’t actually a “we” that I can speak for, but I can speak for me.

I write Free Software because I believe that Software should be Free.

I write Free Software because I enjoy writing software.

I write Free Software because someone needs to, and although there aren’t that many things I can do that will actually make the world a better place, this is one of them.

It so happens that I grew up and went to school in another field that shares a similar characteristic of individual drive – Theatre.

If you are going to study theatre with the intention of pursuing it as a career, the first thing to learn is that it is likely you will never actually make a living at it. Here are some stats (1):

• There are more than 100k professional actors in the US. Fewer than half of them make an income higher than the national poverty level for a single person in any given year.
• On average, only 5 or 6% of them make a normal middle class income in a given year.
• In a country of more than a million lawyers, 4.5 million mechanics and nearly 8 million machine operators, there are only about 2000 to 3000 people who make a living as an actor for at least ten years in a row.
• Add to that, there are nearly 200 graduate actor-training programs and over a thousand university theatre programs churning out new actors every year.

If you want to make it as an actor, you might spend your entire life working your ass off to try, and you have to be in the 98th percentile to make an average middle class existence. Oh – and if you make it to the big time on Broadway, you get to work 8 shows a week.

Doesn’t sound like a smart way to make a gazillion dollars does it? You’re right – it’s not. It boils down to the fact that even if you are trying to make money from theatre, you are essentially just stupid. Even though Julia Roberts and Robert DeNiro and Al Pacino and Johnny Depp all make millions, you won’t. Statistically.

Guess what? Just because Jeff Bezos and Larry Page and Sergey Brin and Monty Widenus are now quite rich, doesn’t mean I am. I’m not. And you won’t be either. Not by programming. Not ever. Statistically.

Now, do you see any of the actors stopping? I’ve worked on several shows while here in Seattle. A couple of them have been at respectable professional theatres. A couple have been with fringe groups. I have myself produced and directed a couple of shows funded solely with the ever popular “sweat equity.” Do I do it for money? No. I had a day job – and the “stipend” that I get paid even at the pro theatres when amortized over time spent would be far less than minimum wage. Do I do it because I was hoping that a travelling reviewer from New York would happen in on my production, review me and put my name in front of a big-time producer? No, that would be assinine.

I do it because it’s what I do.

I won’t stop if the economy takes a downturn. In fact, if I’m out of work, that means I’ll have more time to spend doing what I do. If EVERYONE is out of work, we’ll ALL have more time to work on things.

I truly feel sorry Andrew Keen and his ilk. If the economy truly bottoms, I’ll still have something to do, because what I do isn’t driven by the chase of wealth. I will still know who I am when I wake up in the morning.

I will still do Theatre.

I will still write Free Software.

I hope Mr. Keen has a hobby.

————–
(1) From Acting Professionally by Robert Cohen

Drizzle Virtual Columns

A while back, Giuseppe pointed out Andrey Zhakov’s implementation of Virtual Columns for MySQL.

Since then, Andrey has ported that code to Drizzle and sent us a message about the Launchpad Tree.

First, the bad news. I waited long enough to actually look at it that the merge was no longer clean. The Drizzle codebase moves pretty quickly, you know. Andrey was wonderful about helping fix those issues, I got it all merged in and tested, and pushed.

Drizzle now has Virtual Columns. w00t.

As a quick aside, I’d like to mention a few things Andrey did very, very well:

• The code is commented.
• Variables are named sensibly.
• An entire suite of test cases was provided.

I don’t think we’re quite yet able to do what the bazaar developers do, where we will not accept patches without a test case, but gosh it’s nice and I can’t wait until we can impose that rule.

32-bit? Really?

Is anyone out there actually still using 32-bit systems for new deployments? On purpose?

I know I occasionally see people who have 64-bit systems and have installed 32-bit OS on them. They are one of two things: people who don’t know what they are doing, or why their server is then having memory problems, or people who have 32-bit Linux installed on their laptops because there is no good 64-bit Flash Player plugin for Linux. (/me shoots Adobe in the Face… it’s called re-compile it and release, please)

The 32-bit laptop people I don’t care about – they are not yet hosting websites on their laptops while browsing YouTube. Yet.

The others just need the learning.

Which brings me back to… should we start to consider 32-bit a dinosaur sort of like AIX 4.1?

(I should be clear here… I am honestly asking… not just trolling. I’m also not advocating bad code – see previous post.)

ulong FAIL: more Drizzle cleaning

I haven’t been doing any new features or exciting things like Query Logging on Drizzle, although I have to say that I’m very much looking forward to logging to syslog. What I have been up to is continuing the seemingly never ending battle to clean up things in the code. Some of the changes are essentially style issues, but some of them additionally help to uncover potential problems. Take my most recent nemesis, ulong.

There are multiple problems with ulong. The first is that it’s non-standard. If you want your code to be portable at all, you have to do something like this in a header file.
#if !defined(HAVE_ULONG) && !defined(__USE_MISC)
typedef unsigned long ulong; /* Short for unsigned long */
#endif
Now, of course, there’s nothing fundamentally wrong with that. #ifdef and typedef are core parts of the language. But in this case, we’re doing at least three things.

• We’re adding a global dependency throughout the source tree for something as silly as an alias.
• We’re adding one more way to write this, in addition to unsigned long and unsigned long int to the code base.
• We’re adding to the amount of stuff an incoming developer has to “get up to speed on” regarding the code base. Should ulong be used exclusively? If so, why is unsigned long used in places? Is ulong meant to convey any information that writing it one of the other way doesn’t?

These are all largely the domain of the pedant in me, but they essentially boil down to unnecessarily sloppy style. The more a person has to work on parsing a thousand stylistic idiosyncrasies in the code, the less of their brain they have to lend to the actual matter at hand.

But the real damage of this whole ulong thing is that ulong doesn’t actually mean anything useful. From the Wikipedia:

In computer science, a long integer is a data type that can represent a whole number which may have a larger range, while taking more storage, compared with a standard integer on the same machine.

In C, it is denoted by long. A long integer is often double the size of a standard integer, but this relation may not be correct. It is often a 32-bit integer, although it could be larger, for example, 64 bits. A conforming program can assume that it can safely store values between −(2³¹-1) and 2³¹-1, but it may not assume that the range isn’t larger.

It might be 32 bits. It might be 64 bits. It might be double the size of int, but it might be the same size. Please note that the Wikipedia article is Int64, and then contains “It is often a 32-bit integer”.

Sigh.

One of the reasons I’m quite pleased that we’re using the C99 types. uint32_t is quite clear as to how long it is, and it doesn’t matter which platform it is on. The real fun, though, is in going through the code base instance by instance and trying to determine if a given ulong is meant to be a uint32_t or a uint64_t. Global search and replace doesn’t work. Believe me. I tried.

I’d also like to point out that the MySQL Cluster devs figured out that this was a problem quite some time ago and use Uint32 and Uint64 in their code. Of course, that breaks my rule above for using self-defined types when there are system types that do the job… but the Cluster guys and MySQL guys in general have to support compiling on many more platforms than I do, some of whom may not have C99 headers.

I’d also like to reiterate what Brian has said a few times already: there are bugs in the code currently that are nothing more than use of a long and assuming it is the wrong number of bits. (Try bitwise operations intended for 32-bits on a long on a 64-bit platform. Good luck.)

When it rains…

It may be rather obvious to some folks that I’ve been doing a lot of work on Drizzle since Brian released it. The great news is that my employer thinks this is a good thing, and is actually now paying me to work on Drizzle as a day job.

So as of yesterday I am no longer a member of the MySQL Professional Services team, and instead will be sitting on my couch all day annoying my wife and hacking on Drizzle.

The last three years I’ve been a consultant for MySQL have been wonderful. I’ve gotten to work with a whole load of bright people and many of the big exciting shops and large projects out there. I’ve also done some work with companies not much bigger than 2 or 3 people. I can truthfully say there is an almost never ending supply of interesting things people are trying.

At the same time, I can’t tell you about most of them, which is the downside to being a consultant. There are many upsides, including travel to all sorts of exotic locations… so feel free to apply! For me, though, at this point, the prospect of working on an open public project (hosted on Launchpad, IRC on freenode … no private mailing lists) is quite welcome after all the confidentiality that working for clients brings.

Moving forward, I’m very excited to be able to work full-time on the Drizzle codebase. There are many, many tasks that need work, and tons of fun challenges ahead of us. I hope we can push forward with something that will forge new ground, but without forging so much new ground that it’s unusable.

It’s been a great three years, and I hope I see many of you at the various conferences (or bars) around the world!