My understanding of this phenomenon is incomplete.
I think it goes like this:

• Spam robot crawls my website
• Spammer links to my website
• {insert marketing-speak involving black-hat SEO here}
• Massive profit for spammer

I've gone around in circles trying to decide how to handle this issue. Visits from the spam crawlers aren't costing much data transfer at the moment, I'm nowhere near my limit over at Linode, but they are swamping any real traffic in my analytics reports.
I was finally bothered enough to research my options.

My initial thought was to add iptables firewall rules since that would block the requests even before they pass into the webserver (nginx). I didn't want to filter on IP address since these crawlers are likely coming from multiple IP's and I don't want to spend time poring over my server logs to track them down. That left me with blocking by domain, and I found this nice article covering the topic. After looking into the some of the syntax used in the post, I started to get the feeling that iptables might not be the right tool to handle this sort of fuzzy string matching. Also, iptables rule sytax makes me cross-eyed and even though I only have a handful of domains to block now, I don't want to be writing new firewall rules for every spam domain.

I briefly wondered if I could do something creative with fail2ban, which is already installed on the box. But, fail2ban apparently translates it's block list into iptables firewall rules, so it would be added complexity on top of a tool I was hoping to avoid.

I was trying to get closer to the metal than the web server, but nginx is quite good at string matching, so I decided it was the better choice. After much internet searching, I finally came across a nice gist showing an nginx configuration file:

I liked the syntax, but there's no setup context and I wasn't sure if this extra configuration file should be referenced in the main nginx.conf file or had to be included in each of the separate server configuration files. I tried to set it up both ways but got errors when restarting nginx. Apparently the map directive can't be used in either of those ways.

The technique used here also looked promising (and it didn't throw errors when restarting nginx).

In the end, I went with the basic pattern as described by the second link, but took a few pointers from the first method. I created a new nginx configuration directory and file, /etc/nginx/global-server-conf/bad_referer.conf, which contains:

if (
    $http_referer ~ "(
        semalt\.com
        |buttons-for-website\com
        |blackhatworth\.com
        |hulfingtonpost\.com
        |darodar\.com
        |cenoval\.ru
        |priceg\.com
    )"
) {
    set $bad_referer "1";
}

if ($bad_referer) {
    return 444;
}

I feel like the 444 status code was the right choice for the return value:

Then within each site's server configuration file, I added the line:

include /etc/nginx/global-server-conf/*;

In my bad_referer.conf file, I listed all of the sketchy domains, except one. I skipped bestwebsitesawards.com for now since it wasn't generating much traffic, and I wanted to have something to use as a "control group".

In a month or so, I'll update this post with another screen shot to prove that I've kept the spam robots at bay, or show that I've failed miserably.

2015-03-11 update:
I still have some more work to do.

Due to the still large Russian numbers, I've added o-o-6-o-o.com and humanorightswatch.org to my block list.

Maybe this traffic will never quite go away...

I've been standing in front of my computers for a few years now, and while my feet might feel tired at the end of the day, my back and hips are less stiff.
This is anecdotal, I'm not going to tell you that you're going to get bowel cancer if you sit in a chair.
Obligatory xkcd link on standing desks.
I just want to put up a quick post with some (hopefully) inspiring pics of standing desks in the real world that don't cost thousands of dollars or involve weird Ikea kung-fu.

For my desk at work, I spent about $15 at the hardware store on a cheap, plastic bookshelf.
I first tried the supports at their original length and then cut them so I could get the tops of my monitors level with my eyes, and my keyboard resting where I had a 90-degree bend in my elbows.
Here's where I ended up:

I liked standing enough that I commissioned a friend to build me a bespoke standing desk for my home office (pardon my mess, but do check out his more glamorous projects over at the Priceless Projects on Facebook):

While I'm at it, a sysadmin friend sent along his setup.
He started with a trial version using boxes found around his building (notice the fish-eye mirror for the truly security conscious; no chance of shoulder-surfers stealing passwords):

And then he made it a little snazzier:

With no more chair, I think PEBKAC has to become PEBCAF.

Note, this occurred on a Windows 7 system, but I'm unsure what updates may have been installed at the time because now I can't duplicate the bug anymore. Being presently unable to reproduce this behavior also means this post won't have any screen shots (my apologies).

However, it was odd enough that I thought it deserved a quick write-up...

The basic steps were:
- Open Windows file explorer to a folder containing lots of files, in my case they were '.zip' files - Use the search box in the upper right to find a subset of the files in the folder, probably by searching on some part of the filename - Select one or more files from the search results - Drag selected files into another folder in the folder tree view or onto another folder's explorer window.

Finally, marvel at how your files have been silently renamed using all lower-case characters. Note, this all applies to network files and I don't have any details on the setup of the file server.

Note, I originally started writing this sometime around 8-15-2012...

I'm finally learning how to use VIM. It's quite possibly the most geekily arcane thing I've ever done. During my first few days of using vim regularly, I'm finding it hard to go from using an email client or some other non-modal app and get back into that 'VIM mindset'. Once I got my head around 'normal' mode and the motion keys, things got a lot better. Understanding the difference between buffers, windows, and tabs in gVIM was also a hurdle.

I was initially confused by gVIM occasionally splitting it's main window into multiple windows, but apparently it splits automatically when there are unwritten changes to the current buffer/file. Due to this confusion, I tried briefly using tabs in gVIM to handle multiple files. This did not go well and I decided to try and use gVIM without tabs since it seems more in the spirit of the 'VIM-way'. Those graphical tabs seem to be a bolt-on anyway, plus I'm not yet to the point where I would have any extensive projects open in VIM so switching buffers is not unwieldy yet. End of week update...multiple/split windows can be handy to reference one file while editing another (a bit like having two windows open on dual monitors). Also, ctrl-w commands to jump from one window another are really the way forward here.

I have my graphical cheat sheet perched just above my keyboard and some sticky notes with handy commands not covered by the sheet. After a few weeks (or has it been months already?) all of this vim weirdness is starting to get it into my fingers...I'm seeing how fast text wrangling can get.

Update (unknown date),

I'm in a strange transition period where VIM will occasionally surprise me with its behavior, however I really miss the modal functionality whenever I'm in Visual Studio or SQL Server Management Studio. And what is the deal with yanked text from VIM not making it onto the system pasteboard? Ah, that's because I hadn't yet discovered the named registers. Specifically, that I should yank text to the *-register and it will then show up on the system clipboard (should work on windows and linux). I also didn't understand how the double-quote key is the signifier to name a register (to yank to or paste from).

Update 10-15-2012,

I found a good article by vim creator. He advises learning a few basics and focusing on trying to become more efficient at small common tasks. I have to agree, this is where my best progress has been made, so hopefully I'm on the right track with regards to becoming more profiecient with vim.

Update 11-21-2012,

I installed spf13 vim-distro on my home system running Linux Mint. The spf13 color scheme conflicts with the Linux Mint terminal color scheme, which made vim unusable. I couldn't see the cursor and I couldn't distinguish most text from the background. I was hoping to take a shortcut to a tricked-out vim setup where I didn't have to do any configurations or plug-in installation. I'm glad I didn't spend much time with spf13...it was really the wrong direction for me to take. I ended up with a bunch of stuff that I didn't know how to use and no context for me to know how tweak it. Opening the .vimrc configuration file that spf13 created was the last straw. It was a strange landscape full of fancy plugins and settings. Instead of trying to familiarize myself with all that complexity, I realized (again) that it's much better to go from the ground up.

After using 'vanilla' vim on linux systems for a while, it bothers me that the cursor doesn't change shape under INSERT mode. I really like the visual cue in gVim or Sublime-Vintage when the cursor changes from a block to a thin vertical line. I know plain-old vim shows -- INSERT -- at the bottom of the screen, but it's nice to have that little bit of fancy UI for the cursor...it's where your eyes are looking anyway.

I discovered and completed the vimtutor training built into the terminal/vim on linux (just type vimtutor into terminal!). This was surprisingly good practice...especially with delete/put and yank/put.
I'm finally starting to understand how that grey-beard programmer down the hall can be actually be productive even though they're a 2-finger typist.

Update 03/27/2013,

Good perspective (if a little depressing)...
Just Use Sublime Text

And a Stack Overflow question on vim with passionate responses with an overwhelming number of tips. The first answer is great, if rather long. That answer points out that some of the vi conventions go all the way back to before computers had electronic displays. Now, if we're going to stick with the same techniques needed when the user couldn't see a live copy of the text they were editing, they better be saving us lots of keystrokes. I'm not saying it's not worth it, but it might explain some of vim's opacity and difficult learning curve.

Update 06/20/2013,

Just had my IT-OPS guy install the VsVim Visual Studio plugin for me (yes, I work in a shop where dev's aren't allowed to install their own software), and I'm having fun in Visual Studio for the first time in years! A co-worker had previous experience with VsVim and told me it was flaky and not worth running, but this was maybe a year ago. However, I recently found VsVim on github and saw that it has been under active development, so I decided to give it a try. I wish I'd installed this sooner, it'll make everyday work better in VS...especially now that I'm using VS almost exclusively for tSQL coding.

Peer reviewed publications came about as a way to disseminate and review knowledge. That's great but their time may have passed.
I think a decent web platform can do all of that and more for science. Maybe the process of science can even be sped up at the same time. Does Moore's law apply to science?

If the goal is to publish peer reviewed and repeatable results, why is that delayed until the end of the process?
If there were a wiki, a blog, a data API...really just a basic web publication platform, then the whole world can peer review and recreate results even as the science happens. Really, no one cares, but the cream would rise and like minded collaborators could more easily find each other.

Would it really be so bad for the masses to witness the real pursuit of knowledge (should they care to tune in)? It might benefit a few more people than those who can afford a subscription to the dead tree edition of New England Journal of Medicine.

What does the IT infrastructure look like in a typical lab? Are there any easy wins for a standard/secure web interface over whatever on-site IT that's typical in a lab? Where to start?...a few choice research projects
would probably go a long way toward more general extrapolation...after all, this is pretty much how some of the open-source CMS platforms were born out of newsrooms. What does the LHC use in the way of data management and
collaboration?
The web was invented by some guy at CERN who was hoping for a better way to share scientific documents.

The "file drawer effect" came up on a recent episode of my very own podcast, At Least You're Trying.
I hadn't heard this term before, so my co-host had to explain to me that it's the pattern where non-splashy/non-sexy research is relegated to the file drawer instead going through the publication process.
I would much rather take the file drawer effect and turn it on it's head; the entire world needs access to once giant internet file drawer.

Here's a step in the right direction:
Academic Torrents

Now we just need a snappy name...
WWFD?
WWF?...No, name collisions with wresting and wildlife.
Global file drawer?
I'll workshop it.

I wanted a lighter and simpler system...hopefully one built using hipper technologies that I was curious to learn about. I've been using markdown for a few years, so static blog generators like Jekyll were intriguing. The static site generators also fit in well with the minimal Linode VPS I was already using to run nginx to serve the At Least You're Trying Podcast feed. I didn't last long with any of the Ruby-based options since RVM would never cleanly install on my local PC (Linux Mint 13).

Since I could (still) use some JavaScript experience, I started looking into node.js-based static blog generators (poet, wintersmith, blacksmith). These were all too bleeding-edge and not very n00b-friendly, though. They all also suffer from the, "how would I explain this workflow if I were setting up a blog for someone who's never used the terminal or version control?" problem.

There was also another possibility based on node.js...ghost had a successful kickstarter campaign and released a public beta sometime around October-2013. I was able to get ghost installed and running on my local system and I liked what I saw. Ghost makes for a less arcane workflow for non-programmers, while still being lightweight.

I was excited until I began reading tutorials on getting ghost running on my server. Did I mention that my server setup and admin experience are minimal? Well, I managed to render my precious VPS unbootable while trying to add some (probably over-kill) security changes prior to installing ghost. Linode support was quickly able to save my from my own stupidity. At that point, I gave up since it seemed out of my reach to be able to get all the node.js, ghost, and nginx configuration correct and stable.

The container ship comes in

A few months go by and I start to hear and learn about Docker. Nifty, lots of the benefits of virtual machines, but without all the performance overhead...something like BSD jails on Linux. The mental hamster wheel started creaking away once I saw what was possible with Docker manifest files and the Docker index. Of course some neckbeard-angel had already created a dockerfile to reliably install node.js...some other wonderful individual even created one with a ghost installation.

Ok, at that point I was itching to snap all of these pieces together and things went well at first. I saw that Linode had added support for Docker, so I installed new kernels on my VPS and on my local PC. I got Docker installed and pulled my first Docker image. I built a ghost blog image from the files posted to the Docker index.

Then it was on to testing my new ghost container. When starting a container, Docker allows for a folder on the host file system to be mounted in the container. The ghost container uses this so all ghost-related files (content, images, themes) can be stored separate from the container. Here was my main source of frustration. I wanted to be sure that all my content would survive if the container crashed, but stopping and running a new ghost container during testing would fail to bring back any uploaded images. The ghost container would also not recognize any changes to the ghost config.js file.

I fought with it and cursed at it and nearly gave up again, but at some point I'd gained enough bash scripting experience to see the problems with the setup bash file used by the ghost container. This was a great reason to play around on github a little, so I made my own branch of the ghost container repository and set about fixing things.

Update Feb-2014, similar fixes to my own have been made to the central dockerfile/ghost repository. This is a great sanity check for myself, but it probably also means I missed out on getting my first accepted pull request.

This has proven to be a stable and performant setup. I'm using it for the At Least You're Trying podcast website and this very blog. Running two node.js/ghost containers on my VPS with 1GB of RAM hasn't been a drama. The server usually shows about 30% RAM usage and there wasn't any notable performance hit for running the second container. It's doubtful any of my efforts will generate enough traffic to really load test this setup (not planning on being internet famous any time soon). Using one Docker container for each ghost blog has nicely enforced storing the content separately. Since ghost only currently supports a single admin log-in, separate container instances of ghost make even more sense.

I'm currently hoping to fully crash-proof the running containers using upstart, but at the moment the containers don't come back up after a full system reboot. As far as I can tell, I correctly followed the Docker upstart tutorial. If you have any ideas as to what I've done wrong, shoot me an answer on unix.stackexchange.com.

• hsivonen.fi/eme/
• ameliaandersdotter.eu/2013/10/13/drmeme-html5-american-thing
• brendaneich.com/2013/10/the-bridge-of-khazad-drm

I think some of the outrage could have been avoided with a different sales pitch...

Wouldn't it be nice to have a web-standard, secure content pipe between you and your users?
Content producers could narrow-cast directly to audience members. Content middle-men would be cut out of the loop. You wouldn't need to depend on YouTube, Vimeo, or iTunes (for podcasts). Comments below your video would come from interested, paying customers instead of internet trolls.
Of course, those hosted content options still exist for people who can't be bothered to host their own media or don't want to pay for the bandwidth. This seems like an opportunity for independent info-products and content producers.

Okay, maybe it's not a good thing for this spec to pushed by "big content" movie studios, but they're going to implement DRM somehow anyway. They already have Flash and Silverlight "solutions". I can only hope that a browser-implemented DRM scheme would cause fewer software crashes. The security patching cycle would probably also improve if it didn't fall on a single company to update a giant, crufty code-base. Does Microsoft even still officially support Silverlight?

Similarly, people are afraid of UEFI/secure-boot. There is much confusion about the secure-boot process and Microsoft's implementation (and de-facto role as gatekeeper). People should be worried about the potential loss of the ability to install another operating system on their hardware (the FSF calls this "restricted boot"), but being able to sign and validate all code loaded during the boot process benefits all platforms.

Just like most powerful and flexible technologies, it can be used for good or evil; privacy or oppression. We shouldn't blame the tech.

Now, is everyone still angry?

01/28/2014 update,
Hopefully this article with good perspective still applies. According to Big Web Show, episode 109, Brendan Eich working on watermarking tech that might be more useful than DRM. Maybe we'll end up with a more enlightened technical implementation after all. It would be fun to see the entrenched content distributors cling to Silverlight while all the web angels migrate to something simpler.

Hungarian notation for variable names in my database code.

Sure, I've written many lines of solid and readable code using Hungarian notation. VB6, VBA, and tSQL...it was previously company-standard for all DB code (and the data type prefixes were helpful in the days before IntelliSense arrived in SSMS).

However, I was happy to leave this convention behind in favor of more descriptive variable names. So I tried to argue against it, but I just come across as someone who's being obstinate for no good reason.

Oh well, dttmToday+1 is just another dttmDay.

I was then asked what programming languages I used at my day job...the reaction when I said 'VB.Net' was astounding.
His reply was, 'I thought you were a unix guy!'.
Oh well, it's nice to masquerade as a proper neck beard, even if I'm still stuck with VB.

"Why would you need 3 versions of Visual Studio?", I hear you asking. Well, bear with me...

• Visual Studio 2008, needed for SQL-Server Integration Services (SSIS) development on SQLServer-2008-R2. Full stop. Nothing else can be used to work on SSIS packages for that version of SQL-Server. In fact, Visual Studio 2008 masquerading as Business Intelligence Development Studio (BIDS) installs with SQL-Server. I don't want to get started on the naming inconsistencies of these tools, that's a topic for another post. Also, Visual Studio 2008 needs a service pack and plug-in's so it can play nice with TFS-2012...another layer of complexity.

• Visual Studio 2010, company-standard IDE for the foreseeable future.

• Visual Studio 2012, the IDE shell had to be installed because VS-2010 couldn't delete or roll-back a file in TFS. The full version was installed later, and I tried to switch to it, but there was no syntax highlighting for tSQL (deal-breaker). At least I was able to import my VS-2010 settings, although I had to re-do any custom keyboard mappings.

Still, it's not as bad as the dark-old days when VB6 would conflict with any later versions of Visual Studio, or you couldn't remove a beta install of VS without basically re-installing the OS. It's impressive (in a way) that 3
versions of this weighty IDE can live together in harmony on one PC.
It's also rather silly that all of this is necessary.

As fiddly and arcane as vim can be, it can be a real boon when all your configuration and tweaking is encapsulated in one easily portable text file.

...Oh, hey, look! tSQL syntax highlighting has gotten fixed in VS-2012 at some point over the last few weeks. Maybe i can whittle this lot down to only two versions of Visual Studio.

09.05.2013 update, never mind...it's still 3 versions (at least for the rest of 2013).
Apparently there are conflicts when checking in changes to .sln or .proj files among different versions of Visual Studio, so i'm stuck back on VS-2012 for a while.

This post involved simple firewall rule addition using ufw and a bit of configuration with squid proxy. It was all quite easy. I was familiar enough with Ubuntu, the command line, and vim to cruise through it and it was exhilirating.

This time around, I plan to keep the VPS running...I even paid for a year in advance. Apart from occasional BBC shenanigans, I want to use the server to host a personal blog, my silly podcast, and a few random websites and projects. I followed Linode's security documentation more closely this time, which included the addition of some iptables rules. Now, when I tried to give the BBC setup another try, I got a connection timeout...d'oh.

I realized that the previous iptables rules were somehow trumping the ufw settings, even though ufw is built on top of iptables. So at 8pm on a weeknight, halfway through a burrito, I'm faced with trying to unpack the counter-intuitive syntax of iptables rules. It's certainly deeper into administration than I was planning to delve on my second day with the server.

Oh well, I'm now able to say that I have experience constructing basic iptables rules, so maybe there's hope for me yet.