Nerdcore 4 Life

So in my last post I mentioned that I want to start doing mobile apps and I chose to work with Android(I’ll explain why later). And Android has proven to be great, but developing in it has forced me to use Eclipse, which is…less than great.

Now when I say forced it’s not like anyone from Google came to my house and threatened me but all the learning material I’ve come across, from the free stuff on the web to published books to what Google puts out themselves, assumes that you are using it. There are some explanations of how to do things without Eclipse but when you’re learning something you should always try as best you can to mirror the environment of who ever’s teaching. It helps narrow down the origin of errors.

And when I say it’s less than great I don’t mean it’s bad. I just don’t like IDEs, and for 2 main reasons. They hide things I should know from me and force me to do things their way. As an example of the first one, Eclipse has this nice feature that allows you to hit CTRL-SHIFT-o and it will automatically import any packages you need for a java file. Very convient. Except now I don’t know what packages corrolate with which objects. Even the most dynamic of languages force you to import foreign code expliciatly, that tells me that this is information the programmer should know.

As an example of the second one, it’s a pain in the ass to edit files in Vim. Every other text editor is notepad to me. I tried googling around and there are a few ways. The first if viPluggin, which seems good but it’s not free and to be honest I’m going to try and ditch Eclipse as soon as possible. Theres vimpluggin which I installed but it doesn’t work right/well. I couldn’t get arguments to Vim and I kept having to right-click and open-as to switch between Vim and the native text editor. And cosidering I sent a question to the support list and was promptly offered a deal on pills to make it ‘Thicker, longer, better.’ I’m going to assume it’s not supported anymore. I didn’t try Eclim but that seemed to be going backwards, adding Eclipse features to Vim. I need to make sure my developement environment is as close as possible to standard.

So what am I really looking for? I’d like to be able to edit files with full Vim functionality(including using my .vimrc file) and use the full feature set of Eclipse at the same time. And I’d like to be able to jump from one to the other seemlessly. That I did not accomplish(hence the title) but I have a hybrid approach that comes pretty close.

HowTo

So my two pronged approach involves installing VRapper so I can quickly switch to a Vim like navigation scheme within Eclipse. 4 keyboard strokes and I’m there. But it doens’t have the full functionality I need or take rc files(I can’t live without the elflord) so I used Eclipses built in ability to use an external editor. It’s a bit slower to start and switch back and forth but it works adequatly.

Setting up VRapper

VRapper is an open source project that is constantly evolving and adding new features, but currently missing quite a few. Still, being able to navigate with the home keys and yank lines makes installing this worthwhile.

First, you have to add the repo path to your software updates. Click on Help->Software Updates:

You should the software window. Click on the tab Available Software and the button Add Site:

You’ll get a popup window that let’s you…add a sight. Go ahead and put in http://vrapper.sourceforge.net/eclipse-update-site

Then click OK and you’ll have this repo added to your list. Select Vrapper and then press Install

Then just click through until it starts installing and restart. You should now notice a new button on your toolbar:

Pressing the button toggles Vim edditing mode on and off. Congratulations, you now have some vim functionality available to you. However, it’s still kind of a pain to use. You have to close whatever you’re working on, click the button then open it again. I could open it in Vim for that much effort.

So what we’re going to do is set up a hot key to toggle Vim mode, then we can switch back and forth from VRapper to native Eclipse editing with a few keystrokes.

First we need to open up the preferences. Go to Window->Preferences

Then expand General, select Keys and type vrapper into the filter.

Then type any key combo you want into the binding box(I chose VV) and done. Now typing those keys toggles Vim mode on and off. You still have to close and reopen the file, but I can do entirely from the keyboard(CTRL-W closes, F3 opens a file). So I type CTRL-W SHIFT-v SHIFT-v F3 and I can jump from one mode to the other. There’s probobly a way to macro that all into one keypress for Eclipse but that’s fast enough for my needs.

However, VRapper is still missing a lot of Vim functionality. No macros, no rc files, etc. So I also want to be able to open files from Eclipse directly in gVim.

Setting up gVim

This is actually pretty easy using Eclipse’s ability to call external editors. If I could find a way to do this with keystrokes instead of using the mouse I wouldn’t even bother with VRapper.

optional

I don’t call gVim directly from Eclipse because I want it to behave slightly differently when I’m editing files from Eclipse. This means I set up a little bash script in my home directory that looks like this:

/usr/bin/gvim -U /home/whaledawg/.vimeclipserc -- $1

That calls gVim and gives it an rc file I use just for Eclipse. The rc file looks like this:

set noswapfile
set nobackup
set noswapfile
set nobackup

Note: if you do this then the bash script must be named gvim. gVim checks the name of whatever calls it and it must be gvim or it won’t work.

/optional

OK so again we need to go to Window->Preferences, expand Gernal but now we’re going to choose File Associations. If you click add you can add new file associations here(you may need to do this for *.java files) and if you select a file type you can see all the associated editors in the bottom window:

Clicking Add(next to the associated editors) allows us to add a new editor to that file type(these are the editors that appear when you right click a file in Eclipse and choose “Open with”) From the pop up window select External Programs and then click Browse. If you created my bash script above, then you would browse to tha script(make sure it’s permissions are executable) or if not you will browse to wherever gvim is on your system.

Now if you right click on a file of that type and mouse to Open With you’ll see gvim as an option. Hurray.

There’s still a minor problem though. If you have a file open in Eclipse and open it again with gVim it will either not update the changes to Eclipse or will give you an annoying prompt. Let’s fix that.

Again under Window->Preferences->General, we’ll find Workspace. You don’t need to expand Workspace, just select it and you’ll see some options. Check Refresh Automically and it will automatically update your open files in Eclipse if you edit them with an external editor.

Click OK and that’s it.

Now you have a quick way to get most of the functionality in Vim and a slightly more cumbersome way to get the full functionality. While far from perfect, this approach has worked for me so far, at least until I’m able to dump Eclipse completely(which I hope to do in the near future).

As much as I love Vim, it has some issues that I’ve pointed out before. Chief among them are the little ways that it proves to be inconsistent. The other day I found another one. Highlighting.
Vim has a very robust system for highlighting and coloring various categories of text, that’s one of the things that makes it a great editor. But finding out how to do it yourself was monstrously difficult for a couple of reasons:

• There are examples, but they are kept in 2 different files. The color scheme file(usually under $VIMRUNTIME/colors) holds some of the info and the syntax(automatically loaded from a syntax file in $VIMRUNTIME/syntax) holds the rest. And you’ll be looking for examples because
• The :help files are confusing. The same command is used 2 different ways to set up a highlighting scheme, so unless you read the entirety of the help files before you start then the example commands you are looking at are nothing like the description of those commands in :help. And the help files are huge. This is because there is so much you can do with syntax in Vim, but it makes it impossible to get started with a little tweak you want to make.

So with that in mind here is my goal. I want to present a quick, simple tutorial on how to alter the coloring and highlighting in Vim short of making your own syntax files. If you want to get that deep into you you’ll just have to wade through the whole country mile of :help pages.

First the theory. There are 2 commands you need to be concerned with, :hi and :syn. :syn is the syntax command. It has three ways to define a syntax group, and we’ll take a look at each in turn.

:hi is the highlight command. It allows you to set the coloring(foreground and background) of a syntax group. It’s simple really, but nothing in the :help explains this info :/

So why don’t we start our tutorial with a scenario:
Let’s say we have a weird text file that configures an obscure application. We only need to look at it every 6 months or so but it would be helpful if it was easier to read. It looks something like this:

  %% These files hold database information %%
  llv require("File1")
  llv require("File2")
  llv require("File3")
  %% This file holds user names %%
  llv require("File4")
  %% Output is written %%
  ${DUMP}("File5")

I’d advise you copy/paste this into Vim while you follow along

Basic Matching

It would be much easier to read if we could do a little coloring. Lets start with comments. You first need to set up a syntax group for comments using the syntax command. The simplest way of doing this is with something like this

  :syn match myComment "^%%.*%%$"

So we’re using the match type of syntax command. That will match a regular expression. Then we create a syntax group called myComment and we list, inside quotes, the regular expression we want to use to define myComment. In plain English it says to match if the line starts with 2 percent symbols, then has anything, then ends with 2 percent symbols.

And if you type all that in you get..nothing.
That’s because you haven’t told Vim to highlight that syntax group yet. If you type in

  :hi myComment term=bold ctermfg=DarkCyan guifg=#80a0ff

Then(assuming you’re on gVim) it should look like this

  %% These files hold database information %%
  llv require(”File1″)
  llv require(”File2″)
  llv require(”File3″)
  %% This file holds user names %%
  llv require(”File4″)
  %% Output is written %%
  ${DUMP}(”File5″)

This highlight command is pretty simple to break down. The first argument is the syntax group you want it applied to, in this case the myComment group we just created. The following arguments are for each of the Vim environments

guifg=#80a0ff

Says that if this is being displayed in gVim, set the foreground color of this group to the hex value 80a0ff(Dark Cyan).

  ctermfg=DarkCyan

Says that if this is being displayed by Vim running inside a color terminal, set the foreground color of this group to the named color DarkCyan.

  term=bold

And if this is Vim running inside a black and white terminal, set this syntax group to bold.

Region

Using the match argument with :syn is very powerful because of the flexibility with regular expressions. But sometimes we don’t want to add that much complexity but we still want to be able to identify whatever is between 2 markers. For that we can use :syn region.
We probably want to clearly identify the strings inside this file for easy reading, so we want to create a syntax group that concerns the region between quotes. We can use

:syn region myString start=+"+ end=+"+
:hi myString term=underline ctermfg=LightRed guifg=LightRed

And it should look like this

  %% These files hold database information %%
  llv require(“File1″)
  llv require(“File2″)
  llv require(“File3″)
  %% This file holds user names %%
  llv require(“File4″)
  %% Output is written %%
  ${DUMP}(“File5″)

So a region is similar to a match except we’re defining a start and end pattern each delimited by plus signs. Anything in between(and including) will be part of this syntax group. By the way you don’t have to use plus signs, you can use any single character at the begging and end, but you should avoid characters that are within the pattern(in this case double quotes). You could have also typed

:syn region myString start=|"| end=|"|

Keywords

There’s one other common syntax situation, the keyword. Luckily, :syn has the keyword option just for this(and it works pretty much as you’d expect).

  :syn keyword myKeywords llv
  :hi myKeywords ctermfg=Blue guifg=Blue

And your text should look like this

  %% These files hold database information %%
  llv require(“File1″)
  llv require(“File2″)
  llv require(“File3″)
  %% This file holds user names %%
  llv require(“File4″)
  %% Output is written %%
  ${DUMP}(“File5″)

The list of keywords being space separated.

Linking

Now there’s one more thing, let’s assume that ${DUMP} is a macro of some kind. Now let’s also assume that we always want it to look the same as myKeywords. Defining the syntax group is easy

  :syn region myMacro start=+${+ end=+}+

And we could highlight it with the following

  :hi myMacro ctermfg=Blue guifg=Blue

But what if we were on a computer where that color didn’t look good? Or if some other syntax group made that one hard to pick out? We would have to change myKeywords and myMacro separately. And if we have tens of syntax groups that we want to look alike this can get complicated.
An easier solution is to link them together like so

  :hi link myMacro myKeywords

And now the highlighting of the syntax group myMacro is linked to the highlighting of myKeywords.

  %% These files hold database information %%
  llv require(“File1″)
  llv require(“File2″)
  llv require(“File3″)
  %% This file holds user names %%
  llv require(“File4″)
  %% Output is written %%
  ${DUMP}(“File5″)

So that’s about everything you need to start tweaking your text colors in Vim.  Knowing this will make it a lot easier to go through and read the actual :help files now.

Remember that you can store a bunch of common commands into a file and load it using :source if that’s more convenient. Also remember that Vim commands without arguments give you a list of all current options. So if you type

  :syn

You’ll get

  myComment      xxx match /^%%.*%%/
  myString       xxx start=/”/ end=/”/
  myKeywords     xxx llv
  myMacro        xxx start=/${/ end=/}/
                     links to myKeywords

Which is very useful for debugging your highlighting.

I’ve been getting more and more heavily involved in Vim recently(in no small part because I’m avoid doing actual work) and I’ve been ruminating on what’s wrong with it. It has 2 seemingly obvious flaws, it’s not graphical and it’s not intuitive. But I’ve come to realize those aren’t so much flaws as a consequence of design. You can’t fix those without breaking Vim. The real problem, however, is something eminently fixable if someone sits down and doe s the work.

The Red Herrings:

Firstly, it is true that Vim is not very graphical. Even Vim’s well to do cousin gVim makes using the mouse regularly a painful experience. And the feedback is entirely textual, with coloring of syntax being as much as you’re gonna get. But even though that’s a response to the limitations of terminals that Vim originally had to run within, this system has some advantages.

Coloring of syntax is about as graphical as you want to get. Typesetting and fonts give useful information to clue readers into how to interpret their text, but just get in the way of editors. They force the editor to make choices about what should be emphasized or stand out. These are not things you should be focused on while coding.

And while using a mouse as a primary input device seems to be the way to go these days, it has some real drawbacks. It slows down advanced users to the point of tears, and in a text editor it creates lots of ‘switches‘, places where I have to move from 2 hands on the keyboard to one hand on the mouse, one on the keyboard. That kills flow.

Secondly, while Vim is unintuitive due to it’s modal way of doing business that’s not a bad thing. People aren’t used to modal tools but if you told someone that they had to design a program that kept a users hands on the keyboard at all times(preventing ’switching’) there’s only 2 solutions you would come up with. Modal(like Vim) or meta keys(like Emacs). I doubt people would call holding Ctrl-Alt-Shift-X to move around intuitive.  Nothing intuitive can keep both your hands on the keyboard because users are used to mouse based, graphical interfaces.

The Real Issue:

So what is Vim’s big flaw in attracting new users? Discoverability. This is a concept that’s explored quite a bit in The Art of Unix Programming (one of my favorite books of all time). Good systems should be discoverable, the user should be able to learn new ways of using them within the system. If a program is hard to use and unintuitive but still discoverable, the user will learn it(as long as the tool is better than what their using, which Vim probably is).

To me discoverability entails 3 things:

1. Help is available, thorough, contextual and easy to use. Vim’s help is available and thorough, but falls flat on other accounts. There’s nothing built in to give you help on whatever it is your currently working on(like hitting <F12> on the command line if you start typing :%s/ ). And it is far from easy to use. It’s set up as a series of related text files that tie into each other, but finding a help term that gets you near what you’re looking for is nearly impossible. For example, what if you find ‘:set <M-x> XXX’ in one of your scripts? There is no way to find out what’s going on there.
2. The system is easy to experiment with, with no penalties. Now here Vim does quite well, except for finding what you want to experiment with. Which leads to
3. The system is consistent and builds upon itself in a meaningful way. Here’s where Vim really craps it’s pants. Within certain sub-sections it’s consistent, but across the board it’s not.What shows me all the current variables? ‘:let’ Oh, except for the Vim options, they’re under set. So I should type ‘:set’, right? Actually it doesn’t list all of them, for that I need to use ‘:set all’. Oh, except for termcap options which are under ‘:set termcap’ or any other option sets they decided to make separate which I may not know about. It’s difficult to discover information about my system when each way of querying is different.

Now I know why Vim is this way. It’s a program that’s been ‘grown’ over 20 years and things get added one piece at a time. But I think that it’s about time Vim was refactored with the goal of having a better user experience overall. I suggest(in no particular order):

1. A unified, searchable help database built into Vim(EDIT:turns out this is on the Vim ToDo list) and preferably crowd sourced. I’m tired of struggling to do something for hours, poping into #vim on irc.freenode.net and someone just says :help XXX and I find the solution right away. Vim’s helpfiles are great, I just need a way to get to the ones I want.
   And building in crowdsourcing would be a huge boon. if someone searches the help files, a way to indicate that a particular page answered their question could be massively useful data. Again this should(ideally) be built into Vim.
2. An easy way to source the current buffer so I don’t have to save it to a file every time I make a change. This would only speed up script writing, but anything that tightens the read-eval-print loop is welcome.
3. All variables, functions, options, etc stored within one dictionary. Currently this is how global, local and script variables work(for example g: is a dictionary you can search or step through). As a caveat to that I’d like options to go from being some special variable to just another variable set(so I could use ‘:let o:sethidden = 0′ instead of ‘:set unhidden’ which is weird and inconsistent.
4. Examples. This is the one place that the current helpfiles need more content. Every command/function should have examples for every option. The help could even have a howto.txt set up similar to a faq, making it easier to find the help the user needs within the tool itself.
5. Get rid of the function/command distinction. Either pick one or do the perl thing where the parens are optional. Having to write a command to call a function, having some functions called by their placement and others need to have the :call placed in front, all of this makes VimL less consistent.

Whoa, this turned out longer than I thought. I’ll stop here since I have no power to make these changes anyways, but I think it’s a good start toward fixing a lot of Vims real usability flaws and getting programmers on board with the greatest editor ever.

I recently asked this question on Stack Overflow and man, I am now seriously tempted to see what it’s like to build a byte code compiler. As I mentioned there, I have built a compiler before(for school). It was a trivial example and I remember that taking an obscene amount of time.

Fleshing out the VimL language with a byte coder is insane. It’s years of work when I have other things I want to do and other things I need to do. Most people don’t write VimL code long enough to get an advantage from it to begin with.
But still, the thought haunts me.

I just bumped into a very old article on Vim vs. Emacs and it had a few interesting points. The fact that Vi was designed to run over a 300 baud modem was something I never knew. The gist seems to be that Vi was built to be as powerful an editor as possible with a very light footprint. It didn’t take up much memory or use a lot of bandwidth in a remote terminal. Emacs, on the other hand, was made by people with much greater computing resources. That’s why Emacs is built on top of the OS to do a million different things and Vi was built to use the OS(for things like ‘ls’ or ‘awk’).

So why then is Vi still around? Why is Vim still being developed? It was created during the computing equivalent of the paleozoic, how can a thin program still be useful? After all, Joel On Software tells us that features are what people want. Shouldn’t the more feature rich Emacs be the choice for programmers? It has a built in email client for God’s sake!

Well, no. Features are not what people want, useful features is. An email client doesn’t help me edit text files in the slightest. And the same goes with extensibility and options, useful is good and useless is bad. While Emacs has proven itself more extensible than Vi(or Vim) it isn’t more useful than Vim. That’s why an editor designed to be used over TTY is still prominent.

A less philosophical reason for Vim’s persistence, chorded keys suck. The Control and Alt keys are in the least convient places possible on a keyboard and constantly having to press them is a pain the ass. While command mode/insert mode is unintuitive, it works well by allowing me to avoid having to press 4 keys at once.

And for those who say that Vim has chorded keys too, that’s right. But I map them away. So instead of <Cntrl-[> to enter command mode I press ‘ii’(how many words have 2 i’s in a row?) See, Vim’s customizable too.