ARPACK situation

Jordi

An open letter to all ARPACK users and developers

I’m writing this email to discuss the future of ARPACK. The problem is this: it’s a widely-used library, but it seems abandoned upstream (and upstream, to whom this is addressed, can confirm or deny). This has resulted in the problem of many mini-forks as each organisation distributes ARPACK patches its own way, and very often, for the same bugs. These are the ones I could find:

Additionally, the Mathworks (they make Matlab) probably also has their own version of ARPACK, but I wasn’t able to find a public version of it, nor an email to send them questions to. If someone could contact them, it would be nice to let them know.

These all seem to have modified ARPACK in some way, with minor or major bugfixes, and as far as I can tell, have mostly done so independently. To me, this seems like unnecessary work, if we’re all patching the library again and again and making our own private forks. What I therefore propose is to have some sort of central location for it and we all pool our efforts on this one location. I think it would be easiest to use Andreas Klöckner’s existing fork on github, since this requires the least maintenance and work from anyone. All that it requires for now is for each of the people above to see what patches they have made and transplant them to the git repo.

It would be helpful if upstream could confirm that they are happy with ARPACK development continuing on github and mention this on the ARPACK webpage, so that new people who are interested on ARPACK can be redirected.

Thanks,
– Jordi G. H.
GNU Octave developer

Number Theory 2 available soon!

Jordi

This is pretty exciting.

I first read the first one of these about 10 years ago. The second one has finally been translated, and looks like publication is imminent. I hope the 3rd one doesn’t take as long!

What I like about these books is how poetic they are. They really expose the poetry of number theory. They don’t always give you the full details, often delegating them to 3rd party references, but they do give you that lovely beauty of the subject. Very motivating texts. I’m currently enjoying the first one.

The Japanese have an exquisite sense of aesthetics.

We do what we can because we must

Jordi

A recent conversation with a friend about Twitter vs identi.ca prompted the following response, slightly edited for formatting and anonymity:

<my friend>
man.. that’s my primary annoyance with the open source movement

<me>
What is?

<my friend>
the belief that everything has to be duplicated especially for open source, or it’s invalid

<my friend>
and a version of something that already exists with a smaller community around it and a worse user experience is somehow better

[snip]

<my friend>
someone creates something useful, and then a bunch of people jump on it and do a piss poor job trying to replicate it because they lack the ability to actually design (in the architectural as well as UX sense) and come up with a shit product that they think everyone should use because of some subjective moral high ground

While we were having this conversation, I was sitting in a cafe, it was a lovely sunny afternoon, the sun was beginning to set, and I was feeling very peaceful. I briefly tried a sort of rebuttal but couldn’t conjure up the energy to continue when the argument started to get heated. I replied instead that my surroundings were making me feel mellow. I wanted time instead to respond, which I will attempt to do now.

Let me first try to address the concern at hand. Twitter vs identi.ca. If Twitter exists, why should identi.ca exist? The only reason I started to curiously poke around identi.ca is that I noticed Debian people were using it, and I’ve found that I’m very like-minded with other Debian users and developers. So I trusted their judgement about the usefulness and safety of identi.ca, and I started using it. I’m not particularly defensive about either website; I still think microblogging is kinda dumb. I just wanted to see for myself it would be as captivating as everyone seems to think it is, and I was glad I found a microblogging service that my friends seemed to like.

And yes, it’s true that the fact that I have the source for the website, (it’s under the AGPLv3, which is like the GPL but activates its copyleft clauses by network interaction, e.g. a website!) gives me warm fuzzies. I like having source, and I’m a little sad that the other big websites that I use frequently for their services, bitbucket and gmail, don’t give me source. I briefly tried out gitorious because I wish I could at least submit patches for bitbucket, but then that’s git, and I still prefer hg.

At any rate, if I have two websites of approximately equal functionality, one gives me source, another one doesn’t, then I will disregard popularity and choose the one with source. It’s just what I prefer. I think this is what the attack about “high moral ground” was about, that I think a website should be used because it has source.

I would like to defend against the accusation of high moral ground or holier-than-thou attitude. I now try to never personally appear or act that way, except in very obvious jest (e.g. by speaking religiously about Emacs). I have learned that telling people, don’t use that software, don’t buy that product are extremely counterproductive, so I’m not trying to impose upon anyone any sort of rules of what software I think they can or can’t use. I wish everyone was using identi.ca instead of Twitter and that identi.ca would stay free with that burst in popularity, but the way to that goal isn’t by telling people what they should do; at best you might suggest why they should value source for its own sake, because it enables other kinds of interaction, but it’s pointless to pontificate. So if I or any of my peers in the free software movement have pontificated to you, I apologise for myself and I apologise on their behalf. Empty rhetoric is not what we want you to get out of our work.

Which brings me to the rest of the accusation, what in my friend’s words is a “shit product”… which I don’t think identi.ca is, the interface seems very polished to me (I probably shouldn’t have started off showing the interface by showing off my own identi.ca dark theme; dark themes seem to provoke very intense negative reactions in people). If my wish of everyone using identi.ca or (Octave instead of Matlab!) won’t happen with preaching, then how will it happen? By building a non-shit product, one with a good user experience, one that is architecturally sound.

Clearly being free software isn’t an impediment to being of good quality. We can find software both free and not of good and bad quality. But if there is software that already exists of good quality and it’s not free, then why bother creating a free alternative? And here I come again to Octave and why I felt a poignant need to write this blog post. I am, after all, working on a free project with which I hope to give Matlab users and other people with numerical computing needs better ways to freely collaborate. Octave does frequently receive accusations of having a bad interface (but Jacob Dawid’s Quint project in our private Octave clone may very soon fix this!), of not having enough toolboxes, of being slow, of not having a JIT compiler… all of these are complaints of varying validity. Matlab users have come to expect them, and if we are to offer Octave as an alternative that respects their freedom, we also need to address user complaints.

Here I come to what I felt was the crux of my friend’s complaint. Why bother? If there is something that already works, why recreate it freely and offer something that is a worse user experience? Well, I find this defeatist. If you had the option of software of equal functionality and quality, wouldn’t you prefer it to be free? I think you, yes, you deserve free software, so I’m trying to build it for you, with you. It may not be all that you want right now, but we can always try to make it better. Perhaps you may even help me! Or not, you’re free to do what you want, just improve it privately without even giving me your improvements. That’s fine too.

Will we ever win? Will Octave ever be the de-facto numerical computing environment instead of Matlab like our brothers-in-GNU with the R project are for statistical computing? Will identi.ca receive Twitter refugees when the company takes an irredeemably evil turn in the future? Who knows. But we must not lose heart. We have the capacity to build something better, and because we all deserve something better, we must keep trying. It’s our duty to ourselves, as a community. It’s not a holier-than-thou attitude; it’s thou-art-just-as-holy-as-everyone-else attitude. These are worthwhile windmills to tilt at, we have the steed, we have the lance, so tilt we must.

Knitting for geeks

Jordi

Margarita Manterola from Debian Women asks:

So, after the whole Wikip. thing, I ended up following lots of links
about knitting, and as everytime I come in contact with this, I feel
like I should learn to knit (or maybe I should say, re-learn, I was
taught during primary school, but forgot all about it afterwards).

Is there a “knitting for geeks” tutorial around? :)

There’s tons of videos in YouTube to get you started, although I don’t think many of them are “for geeks” specifically… or at least different kind of geeks. Knitters geek out over types of yarn, needles, complicated patterns, unorthodox knitting, knaughty knitting (e.g. knitting a bra or a thong)… they do end up speaking a special kind of language and getting very passionate about it. What worked for me was when my mom sat down with me to teach me how to cast on, the basic knit and purl stitch, and from then on I was able to bootstrap my first scarf. Perhaps you can learn from videos instead. We didn’t have YouTube when I learned how to knit, heh.

I personally do appreciate a certain mathematical aspect of knitting. I got started because I saw a friend in algebra class in university (y’know, the Galois theory kind of algebra) do complicated lace patterns with needles while she was listening to the lecture, so I was intrigued (and later in love, long story). My kind of mathematical knitting is usually limited to things like knitting a Möbius scarf (non-orientable knitting!) or a hyperbolic plane (negative curvature knitting!). It’s rather remarkable how with just a few basic stitches you can build very complicated things. It does feel a little like Turing machines!

If you need a book, I rather like the Vogue knitting series myself. I hear Stitch’n’Bitch is popular. And most knitting magazines also have introductory instructions in every issue. I liked Knit 1, another Vogue publication, or you can read Knitty online. Knit 1 seems to have ceased publication, but you can probably find back issues in your LYS[1].You might also want to find people near you to knit with, in which case language can be a slight hurdle at first. I really enjoy social knitting. I actually get a little lonely when I can’t find people to share my knitting with.

Incidentally, it’s kinda interesting how knitting patterns are a lot like source code, with similar kind of politics. People share them, remix them, some jealously guard them and try to sell them, others want to ensure as many people as possible can compile your knitting instructions… Interesting world, definite parallels with the free software world.

[1] Local yarn store… too bad most knitting distros don’t ship with the wtf(1) binary. ;-)

How to write a patch for Octave

Jordi

Often people will find ways to improve Octave’s source and provide a patch, but in a way that is slightly inconvenient to analyse and apply. I will now present instructions for creating a patch for Octave in a manner that is easiest for core Octave developers to apply. In what follows, I assume that you have already managed to install Mercurial, that you’re using some sort of shell (probably even Windows), and that hg is already in your shell’s PATH.

First, you should work with an hg clone of the repository:

hg clone http://www.octave.org/hg/octave
cd octave

Once you’re in there, you probably would want to build Octave so you can test your patch. On a Unix-like system, assuming you’ve installed all the dependencies (and that’s a big assumption, because Octave dependencies can be difficult to track down), you should run autogen.sh in the hg clone before you follow the general build instructions.

Then you can proceed to modify Octave sources however you see fit. You can run the hg st command to see which files you’ve modified and hg diff to get a detailed diff of your changes which will eventually become your patch. I personally find the hg colour extension to be a very nice visual aid for these two commands. You should also try to build Octave again with your changes as a minimum sanity check that your patch is good.

A word on building when patching Octave: the deeper in the build dependencies you patch, the longer you’ll spend rebuilding Octave. So, for example, liboctave depends on libcruft, and liboctinterp (most of the stuff under src/) depends on liboctave, so if you touch a file in libcruft, it’s very likely that you’ll have to rebuild liboctave and liboctinterp. The rough depth of things that can be patched by their directory locations is this: libcruft/ → liboctave/ → src → src/DLD-FUNCTIONS → scripts/, in fact, for scripts, you almost never need to rebuild Octave when patching stuff there (unless you want to rebuild the documentation); but sometimes you do need to restart Octave when testing.

Once you think your patch is in good shape, and you’ve written some tests for it (look for assert commands near the bottom of source files to see how tests are written) you should commit it so that it becomes a (semi-)permanent part of your local hg clone. If it’s your first time using Mercurial, you should create an .hgrc on a Unix-like system or Mercurial.ini in Windows and put in there lines to identify you, like so:

[ui]
username = Your Real Name <some@email.com>

Now do

hg ci
hg export . -o my_fix.patch

to record your change. When you type the first command, hg should open an editor where you will now have to write a commit message. Now your change has become part of your local repository, and the second command has exported your patch to a file named my_fix.patch in a format that is easy for others to apply. Now just send that patch to the Octave patch tracker or attach it to an open bug report, and we’ll take care of it!

Making posting easier

Jordi

Dear lazyweb,

How can I make blogging as painless as possible? I’m an Emacs + WordPress user. I wish I could just easily post from Emacs. Right now, I still use the WordPress admin website for posting, and although of course it’s not hard to do it this way, it’s awkward to have to log in and do a few administrative tasks before I can post. I tried org2blog, but I don’t like it too much because it was almost as much as work, plus I had a hard time getting org-mode to not try to be more clever than me and no write its own HTML. Right now, the way I feel most comfortable writing in Emacs is in html-mode. I like writing my own HTML. It’s easy enough that I don’t need any pretty interface to it; plain HTML is simple enough.

I really need to lower the activation energy to blogging to almost zero. Of course, that alone might not make me blog more often, but any amount of nontrivial activation energy will slow me down. I may end up writing my own Emacs solution if I don’t find one, but perhaps there’s an easy method out there I’ve missed.

Earning my wings

Jordi

I forgot when it happened, but it was sometime in late January or February that I was finally given push access to the Octave repositories, plus being a manager for Octave items at the Savannah website where it’s hosted. This means I have the privilege to commit my changes to Octave code on my own without needing to ask someone to push them for me, plus I can also handle bug reports to Octave on my own. In addition to that (bring it on, spammers!) I also now have an @octave.org mailing address.

This is a pretty big deal for me. I had been dreaming of being a formal member of the Octave dev team, and I’m really glad it’s finally happened.

At the same time, I wanted to get this distinction because I had obviously earned it… but what actually happened is that I obliquely requested it and head honcho jwe responded to it.

I’m a junior Octave dev, and I expect to be one for a while until I get to feel more comfortable with the code base. This means that I still need to be very careful with what I push, and I should consult publicly on the mailing list if my patches are acceptable before I push them.

So what have I done so far with my shiny new Octave badge? So far as I write this, not much only 8 changesets, of which 6 are documentation fixes, one was a minor m-script fix for imshow, and the last fix as I write this that looks like a minor thing, but took I would say around 30 hours of debugging to find. The Nikolai Tesla fable comes to mind.

I have done other things that don’t show up in the hg log. I try to help as much as I can in the mailing lists and in the #octave channel in Freenode. I’ve been trying to help triaging bug reports in Savannah. I’ve revived work on Agora a little. A friend has lent me hosting for it! Snippets are now fully functional as far as I’m concerned. I even created a better Octave syntax highlighter for Pygments, although I’m still waiting for the official Pygments maintainer to pull my patches. It really does work, although I gotta fiddle around with the setup in my shiny new webhost to make it work.

I’m going to keep working in the immediate future on the sparse matrix bugs I’ve been looking at. Squashing #32747 was a lot of fun, and it forced me to finally use a good gdb setup, plus learn more about gdb itself. Wow. What an awesome debugger.

And as I keep working on Octave, perhaps I’ll feel more justified for the Octave badge that has been handed to me, feel like I actually earned it, not that I just asked for it and jwe is just a generous guy.

I don’t hate git as much anymore…

Jordi

So a partial retraction to my previous post…

It turns out that git has something called a reflog, which seems to be purely documented in the git-reflog(1) manpage, and to my taste, rather tersely. It appears that git does keep at least a partial log of recent operations so it’s possible, with varying degrees of success, to undo past operations. Or at least this was the theory I read scattered across several bits of oral tradition like personal webpages and blogs and what I could glean from the manpage.

I didn’t see any commands to actually recover data from the reflog, only to read what’s in it. Regardless, at least I was able to recover a few of the branches that git deleted remotely and from reading the commit messages, which I hadn’t noticed before, I was able to guess where the remaining accidentally deleted branches were.

So, anyways, git does have a recovery mechanism… somewhat… It’s about the same way that it’s still possible with some luck do an undelete on an ext3 filesystem. In other aspects, git is still too complicated and dangerous for my taste. It’s a little funny because I’m actually quite comfortable with complicated and dangerous tools elsewhere, like good ol’ rm, cp, mv and other dangerous Unix tools where a typo can mean instant data loss. I recognise that perhaps there should be better ways to do this, but I don’t know of one I like. In the case of VCSes, I know of a better, simpler, safer tool than git, and it’s hg. Perhaps it’s because I have an alternative and feel proficient with it that I prefer it to git.

I will use git again, very cautiously. It’s not a VCS where I can experiment widely without fear, because git erases data remotely. It’s however unavoidable due to its popularity, and I will need to acquire some fluency with it if I’m to keep collaborating with people who use git. For my personal use, I don’t see myself moving away from hg for quite some time to come.

I hate git

Jordi

… or at least I hate it for now.

I’m a Mercurial kinda guy (hereafter hg). Mercurial is the version control system (VCS) that Octave uses, so that’s mostly the reason why I started using it too. I started reading about it, and learning it, and liking it a lot. It makes a lot of sense to me. It’s simple when it needs to be simple and flexible when it needs to be complex.

The other big contender for a VCS is usually git. In fact, it’s quite a large contender. Just going by comparing github and bitbucket, the two large commercial hosts for git and hg respectively (don’t let that .org domain fool you, bitbucket is definitely a commercial venture), github is way larger. It is by far easier to find people praising git in the blogs, the discussion forums, and the mailing lists than it is to find people cheering for hg. I have dabbled with git in the past, and I always found it difficult to understand. I always chalked up this difficulty to just being more familiar with hg, and being nothing more than a personal preference. However, I have recently seen that I am not alone in thinking that git is complicated. Regardless, seeing how immensely popular, vastly more popular than hg it is, I decided to try git again today.

I decided to make a conscious effort again today to learn and use git. I had a practical reason too, to fix a Debian RC bug (perhaps a little late, I hope the release managers let the package back into testing after this). Also, I wanted to streamline the flow for hacking on Debian packages. The Debian packaging of Octave used to be under svn, and later turned into git packages. One thing that makes a lot of sense under svn, since it tracks individual directories, not whole projects, is to only put the debian/ directory under source control. This practice got inherited with the git repositories, and it’s really awkward. Setting yourself up to hack a Debian Octave package goes something like this:

apt-get source octave-foo
rm -r octave-foo-$version/debian
cd octave-foo
git clone git+ssh://git.debian.org/git/pkg-octave/octave-foo.git

    # ...
    # hack hack hack
    # ...

dpkg-buildpackage

which is really awkward. My goal was to get to this:

git clone git+ssh://git.debian.org/git/pkg-octave/octave-foo.git

    # ...
    # hack hack hack
    # ...

git-buildpackage

So I set out to do that. With somewhat unfortunate results.

Let me talk a little more about where I’m coming from: hg. In hg, there are some guiding philosophical principles that have become second nature to me when working with source. One of the core hg principles is that it’s really hard to destroy data, in particular history. There are certain destructive operations with hg, but they almost all create backups, and are disabled by default. The way to enable them is to turn on extensions. In particular, hg makes it virtually impossible to destroy any data remotely unless the person who controls that remote repository somehow enables it with hooks. That is, the person would have to write a script that when you manipulate their repository remotely (and the only commands to do this are pull and push), that script would delete some data.

This vibes really well with me. One of the things that git users praise the most is how easy it is to edit history, to undo mistakes, to rebase changes… hg doesn’t make these tasks impossible, merely difficult or disabled by default; and I tend to side with that point of view. It’s safer. Mercurial takes care of my data when I want it to, and when I need it to do dangerous things, I first have to remove its muzzle, and the muzzle snaps back into place when hg is done doing its dastardly deed. It’s a bit like using “sudo” to perform just one dangerous operation instead of “su”, and then staying in the root shell, while performing several operations, none of which really needs the extra permissions.

So on to what happened: during my work of trying to make it easier to work with Debian, I had created several git branches (which are nothing like hg branches, but whatever, that’s not a big deal). When it looked like my work was in good shape, I pushed it to the Debian git repo. Oh, oops, that only pushes one branch. That’s quite unlike hg which pushes all of the work here that doesn’t exist there. Well, not a big deal, that’s a bit like git’s staging area I thought. Just one more step to get what I want. But I had like three different branches here that weren’t there, so I figured there must be some command to get them all there at once. I asked around in IRC, and someone naïvely suggested using the --mirror command, and I naïvely trusted them without checking what that option would do. I thought it would just copy all of my branches from here to there, mirroring all of them.

And so it did. However, it also checked that there were some branches there that I didn’t have in my local clone, and it erased them. I blinked. Wait. Did git just remotely remove some branches? Oh, well, I’m sure it’s just some metadata that got moved around. Where’s the undo button? Rollback? Restore? I went to #git in IRC to ask.

“… you do have backups, don’t you?”

I blinked again.

Whiskey.

Tango.

Foxtrot.

You’re telling me… that a VCS… one of the most popular ones out there… allows me to delete data remotely? With a command that isn’t called even called “delete” or “wipe” or “force” but innocuously called “mirror”?

I was aghast.

My conversational partners in #git gave me the usual spiel about backups, about how it’s great to be able to shoot yourself in the foot, about how it was my fault, about how I should have read the manual… but I was unable to accept any of this. I just couldn’t conceive that a tool that is supposed to keep my history … to be a little “bit-hoarder” … to never lose data … not only lets me lose data locally, which is ok, but furthermore lets me delete data remotely.

Now, granted, this wasn’t terribly important data. Nothing of great value was lost. Since branches in git are more like tags (but not what git calls tags), it’s just metadata that was lost. The functional part is all there. At the same time, a user’s most valuable possession, data, was harmed by the very tool that’s supposed to protect it. I hate the idea of having to tiptoe around my VCS, which should be a tool that lets me experiment wildly with my source, to try out crazy ideas, and at the same time keep my source safe, multiply backed up, fully mirrored in every clone of the source whether local or remote. Mercurial, for example, doesn’t let me delete data remotely. The worst I can do is add a lot of useless data remotely, but that’s much better than being able to delete it.

Reeling, I did the only thing that could be done and emailed the Debian Octave Group mailing list, asking if someone had a clone of the repository with the missing branches. I hoped that I could recover the lost data by copying it from them. If not, it won’t be a great loss, just an awkward inconvenience. The whole experience, though, has given me a great distaste for git. I still find it much more complicated than hg, even despite my best attempts to understand it. And it’s shown me that I can’t treat it carelessly, that I have to read its gargantuan manpages and thoroughly understand each and every command and option before I use them, lest I provoke damage.

Next time, I’m using the hg-git extension, and I don’t think I’ll be touching git again for a while until I recover from this nasty experience.

La Red Social

Jordi

I just watched the Facebook movie, The Social Network, or whatever, La Red Social en español. I started brewing in my head a post about it, but I think I will omit most of it except to mention that it’s nice to finally see realistic depictions of computers in mainstream media. I will reserve the rest of my thoughts of the matter to myself. I got other more important things to blog about right now.