Your browser (Internet Explorer 6) is out of date. It has known security flaws and may not display all features of this and other websites. Learn how to update your browser.
X
Aside

EdgeOS Passwordless SSH

Having purchased the Ubiquiti EdgeRouter ERPOE-5 router awhile back, I have upgraded my home network’s capabilities as a whole.

In addition to a Web Admin UI to configure the router, there is a console port for a physical access to the router and a SSH management access via the ethernet interfaces.

In an effort to secure my router, I elected to use a 15 character password to login to the router. However, being the lazy me, I want to avoid entering the full 15 character password whenever I want to access the router.

On the Web Admin UI side, it is quite simple, as I can simply just use a password manager to manage and remember the password.
On the SSH management console though, I’ll have to copy the password from the password manager whenever I want to access through that route, which can be quite troublesome

This is where passwordless SSH comes in.

Generate the authentication key pair, using ssh-keygen on *nix or PuTTY Key Generator on Windows, and store the private key in a secure location. The public key is what will be used to store on the router.

Normally for a *nix machine, the public key usually goes into the .ssh/authorized_keys file in the user’s home directory.
On the EdgeOS though, the top of the file contains some warning about losing changes
edgeos-authorized_keys

It seems like the setting will not persist across reboots and sure enough, after the next EdgeOS software update and reboot (I love the uptime of the router!), the key was gone from the file!

A bit of googling brought me to this post in the Ubiquiti Forum: ssh authorized_keys

Now the solution only showed the use of the loadkey command, which requires uploading the public key file to the router.
Another way to do it is to use the commandline in configure mode

[email protected]:~$ configure
[email protected]# set system login user ubnt authentication public-keys <key-name> type ssh-rsa
[email protected]# set system login user ubnt authentication public-keys <key-name> key <base64 encoded public key>
[email protected]# commit
[email protected]# save

<key-name>: Arbitrary name for the key
<base64 encoded public key>: The base64 public key from PuTTY Key Generator or id_rsa.pub

Viola, now the public key will persist across reboots and will also be exported out when you backup the config.

Aside

MariaDB Memory Footprint

Recently, I’ve upgraded the database of this blog from MySQL to MariaDB.

For those of you who are not aware, MariaDB is a fork from MySQL by the original founder of MySQL.
MariaDB 5.5 is fully compatible with MySQL 5.5 and can be replaced by “dropping in” directly over MySQL.

What I noticed after upgrading to MariaDB was that the memory usage shot up to close to my VPS limit.
As my VPS has limited memory, this sudden memory increase might create problems down the road.

After investigating the possible causes, it was traced to the new database engine introduced in MariaDB, the Aria Storage Engine.

According to the Knowledge Base, the default buffer size is 128M, a pretty big number for a VPS with only 512M of available memory.
Since I’m still using the ISAM engine and have no need to use the Aria engine, it should be turned off to reduce the memory usage.
However, as the storage engine is an integral part of the DBMS, it couldn’t be turned off, so the next best thing would be to tune the config in my.cnf

[mariadb]
aria_pagecache_buffer_size = 512k
aria_sort_buffer_size = 16k

There, after tuning the buffer size to 512K and 16K, the memory usage is back to normal.

Aside

Embedding Certificates into OpenVPN Config

I found out a very cool configuration trick for OpenVPN while doing some read-up on OpenVPN encryption key size.

In the middle of the thread, one of the user, “300000”, posted his/her configuration settings.
The part that caught my eye was the chunk of Base64 encoded certs.

I never knew you could embed the certs directly into the config file!

All these while I’ve been using the respective keywords to define the path to the individual cert files. This have made the distribution of configuration to each user quite a pain, since in addition to the config file, I have to send them the cert and key files and also to instruct them on where to put the individual files.

Now, I can just pass them a single .ovpn file and tell them where to place it and they are good to go. No more additional steps like telling them to download the cert files and placing them in a specific directory.

To embed the certs, simply place the Base64 encoded cert text into the respective <ca> </ca>, <cert> </cert> and <key> </key> tags in your .ovpn config file and comment out the “ca”, “cert” and “key” keywords.

client
remote my-server 1194
proto udp
dev tun
persist-key
persist-tun
resolv-retry infinite
nobind
#ca ca.crt
#cert client.crt
#key client.key
comp-lzo
verb 3
<ca>
-----BEGIN CERTIFICATE-----
***Paste CA Cert Text Here***

-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----
***Paste Your Cert Text Here***

-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
***Paste Your Cert Private Key Here***

-----END PRIVATE KEY-----
</key>

There, simple.

Aside

Why we can’t have nice things

Brainfart.SG was taken down for almost a month since last April, due to my VPS being exploited. The reason I believe is due to a misconfiguration of the webserver.

Somebody managed to install a backdoor in the VPS and then installed a script to launch a DoS attack.
My VPS was only suspended at first, but after recovering my VPS, it was again compromised to launch DoS attack. Which resulted in me being banned from BuyVM.

Since then, I have moved to Virpus and have spent a considerable effort to harden the VPS.
I’ve installed APF, rkhunter, ZB Block, among other things. And not to mention closing the security hole for nginx + PHP.
Looks like I’ll have to be on an active lookout for vulnerabilities and also solutions…

It seems like in a perfect world, you can leave your doors unlock at night and you also need not worry about your webserver much. But since we live in a imperfect world, we’ll have to lock up our doors, harden our webservers, deploy SSL for our web connections, etc.

And this is why we can’t have nice things.

Aside

Android Customisation

The awesome thing about Android phones, is the amount of customisation you can do to it, so much so that it is almost entirely a different phone from the original one you got.

HTC Desire Z with Sense UI
HTC Desire Z with Sense UI

After spending hours installing a custom ROM and fiddling with the widgets:

My Customised Homescreen
My Customised Homescreen

The only downside to this is, you need to know what’s going on and know how to fiddle with the widgets. And of course, some time to do the layout and configuration. Definitely not for the faint hearted.

In case you are wondering what are the widgets I used:

Homescreen Breakdown
Homescreen Breakdown
Aside

Email Sender Authentication

Came across a very interesting service today while trying to setup Sender Policy Framework (SPF) for my work email domain.

Port25 Solutions*, an email infrastructure software provider, has a service that helps check the sender authenticity of your email service (ie, the SPF settings, DomainKeys, etc) and also the spam rating of your email according to SpamAssassin’s settings.

Basically, it works by sending an email to this address:
[email protected]
The content of the email can be empty, or if you want to test your marketing email or newsletter for “spamminess” against a SpamAssassin filter, you can include it as the email subject and content.

After sending the email, the service will reply with a report detailing your authentication settings and the spam rating of the email that you have sent.

The is a truncated report of my domain before implementing SPF:

This message is an automatic response from Port25's authentication verifier
service at verifier.port25.com.  The service allows email senders to perform
a simple check of various sender authentication mechanisms.  It is provided
free of charge, in the hope that it is useful to the email community.  While
it is not officially supported, we welcome any feedback you may have at
<<a href="mailto:[email protected]">[email protected]</a>>.

Thank you for using the verifier,

The Port25 Solutions, Inc. team

==========================================================
Summary of Results
==========================================================
SPF check:          neutral
DomainKeys check:   neutral
DKIM check:         neutral
Sender-ID check:    neutral
SpamAssassin check: ham

==========================================================
Details:
==========================================================

HELO hostname:  mail.brainfart.sg
Source IP:      209.141.57.235
mail-from:      <a href="mailto:[email protected]">[email protected]</a>

----------------------------------------------------------
SPF check details:
----------------------------------------------------------
Result:         neutral (SPF-Result: None)
ID(s) verified: <a href="mailto:[email protected]">[email protected]</a>
DNS record(s):
    brainfart.sg. SPF (no records)
    brainfart.sg. TXT (no records)

----------------------------------------------------------
DomainKeys check details:
----------------------------------------------------------
Result:         neutral (message not signed)
ID(s) verified: <a href="mailto:[email protected]">[email protected]</a>
DNS record(s):

----------------------------------------------------------
DKIM check details:
----------------------------------------------------------
Result:         neutral (message not signed)
ID(s) verified:

NOTE: DKIM checking has been performed based on the latest DKIM specs
(RFC 4871 or draft-ietf-dkim-base-10) and verification may fail for
older versions.  If you are using Port25's PowerMTA, you need to use
version 3.2r11 or later to get a compatible version of DKIM.

----------------------------------------------------------
Sender-ID check details:
----------------------------------------------------------
Result:         neutral (SPF-Result: None)
ID(s) verified: <a href="mailto:[email protected]">[email protected]</a>
DNS record(s):
    brainfart.sg. SPF (no records)
    brainfart.sg. TXT (no records)

----------------------------------------------------------
SpamAssassin check details:
----------------------------------------------------------
SpamAssassin v3.3.1 (2010-03-16)

Result:         ham  (-0.0 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
-0.0 T_RP_MATCHES_RCVD      Envelope sender domain matches handover relay
                            domain
-0.0 BAYES_40               BODY: Bayes spam probability is 20 to 40%
                            [score: 0.3966]
 0.0 HTML_MESSAGE           BODY: HTML included in message

And this is after implementing SPF:

<pre>This message is an automatic response from Port25's authentication verifier
service at verifier.port25.com.  The service allows email senders to perform
a simple check of various sender authentication mechanisms.  It is provided
free of charge, in the hope that it is useful to the email community.  While
it is not officially supported, we welcome any feedback you may have at
<<a href="mailto:[email protected]">[email protected]</a>>.

Thank you for using the verifier,

The Port25 Solutions, Inc. team

==========================================================
Summary of Results
==========================================================
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         neutral
Sender-ID check:    pass
SpamAssassin check: ham

==========================================================
Details:
==========================================================

HELO hostname:  mail.brainfart.sg
Source IP:      209.141.57.235
mail-from:      <a href="mailto:[email protected]">[email protected]</a>

----------------------------------------------------------
SPF check details:
----------------------------------------------------------
Result:         pass
ID(s) verified: <a href="mailto:[email protected]">[email protected]</a>
DNS record(s):
    brainfart.sg. SPF (no records)
    brainfart.sg. 86400 IN TXT "v=spf1 mx -all"
    brainfart.sg. 86400 IN MX 10 mail.brainfart.sg.
    mail.brainfart.sg. 85429 IN A 209.141.57.235

----------------------------------------------------------
DomainKeys check details:
----------------------------------------------------------
Result:         neutral (message not signed)
ID(s) verified: <a href="mailto:[email protected]">[email protected]</a>
DNS record(s):

----------------------------------------------------------
DKIM check details:
----------------------------------------------------------
Result:         neutral (message not signed)
ID(s) verified:

NOTE: DKIM checking has been performed based on the latest DKIM specs
(RFC 4871 or draft-ietf-dkim-base-10) and verification may fail for
older versions.  If you are using Port25's PowerMTA, you need to use
version 3.2r11 or later to get a compatible version of DKIM.

----------------------------------------------------------
Sender-ID check details:
----------------------------------------------------------
Result:         pass
ID(s) verified: <a href="mailto:[email protected]">[email protected]</a>
DNS record(s):
    brainfart.sg. SPF (no records)
    brainfart.sg. 86400 IN TXT "v=spf1 mx -all"
    brainfart.sg. 86400 IN MX 10 mail.brainfart.sg.
    mail.brainfart.sg. 85429 IN A 209.141.57.235

----------------------------------------------------------
SpamAssassin check details:
----------------------------------------------------------
SpamAssassin v3.3.1 (2010-03-16)

Result:         ham  (-1.9 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
-0.0 T_RP_MATCHES_RCVD      Envelope sender domain matches handover relay
                            domain
-1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
                            [score: 0.0000]
 0.0 HTML_MESSAGE           BODY: HTML included in message

Pretty nifty eh?

And there is another use for this service that I can foresee, testing end-to-end email connectivity.
After setting up a new email server, you can use this automated service for mail flow testing, sending and receiving of email. And at the same time test your SPF settings.
Cool, ain’t it!

*Note: I am in no way affiliated with Port25 Solutions, I just found this service useful and thought I should share it

Aside

Apache Load Balancing and Session Stickiness with Tomcat

Me and my colleague were trying to set up an Apache load balancer yesterday to balance the connections to a pair of Tomcat servers. As we were balancing connections to application servers, one of the main requirements was to maintain session stickiness when routing the connections.

Most articles on the internet are dealing with load balancers serving static or session-insensitive pages. So it took us quite a while to figure out how to setup what we wanted. We went through countless articles and tutorials teaching us to set up simple load balancing functions on the Apache web server. And we followed those instructions to the word and still can’t get session stickiness to work…

That is, until we stumble across this goldmine!

Near the bottom of the tutorial was the magic that made everything work, the configuration to the Tomcat server. Seems like not many people on the internet have to deal with Apache load balancing to Tomcat with session stickiness as none of them mentioned anything about configuring the Tomcat. (either that, or we are pretty stupid and clueless about Apache and Tomcat… we are n00bs afterall) The articles and tutorials just casually mention the “route” parameter and then move on to the rest of balancing configurations.

So, to configure an Apache load balancer with session stickiness to Tomcat, you first have to configure the Apache webserver:

LoadModules proxy_module mod_proxy.so
LoadModules proxy_http_module mod_proxy_http.so
LoadModules proxy_balancer_module mod_proxy_balancer.so

ProxyRequests Off
ProxyPreserveHost On

apacheLB>
BalancerMember http://server1:8080 route=http1
BalancerMember http://server2:8080 route=http2
</Proxy>

ProxyPass /examples balancer://apacheLB/examples stickysession=JSESSIONID
ProxyPassReverse  /examples balancer://apacheLB/examples

The first 3 lines will load the necessary modules for a Proxy HTTP Load Balancer in Apache.

“ProxyRequests Off” will turn off the forward proxy function of Apache as we are only interested in the reverse proxy function.
“ProxyPreserveHost On” will preserve the host header to the proxied host.

The Proxy stanza is where you define the members of the load balancer pool and the name of the balancer url. Note value of the “route” parameter,  you can set it to whatever name you want as long as it is unique, you will need this in the Tomcat configuration. You can also set some load balancing parameters like the lbmethods, the retry timeout, the lb weightage and so on. You can find out more in the mod_proxy documentation.

You define the load balancer like any other Apache Reverse Proxy, the only difference is in the mapping. In a regular reverse proxy, you’ll map the application context to a specific server. But in the case of a Load Balancer, you map the application context to a balancer url, as defined in the Proxy stanza. Note the “stickysession” parameter, this is to define the session name. In the case of a Tomcat server, we’ll use “JSESSIONID”.

Next, you have to configure the Tomcat servers, in the server.xml configuration file:

# In Server 1 server.xml
<Engine name="Catalina" defaultHost="server1" jvmRoute="http1">

# In Server 2 server.xml
<Engine name="Catalina" defaultHost="server2" jvmRoute="http2">

Locate the “Engine” tag in the server.xml file of each Tomcat server or instance and add in a “jvmRoute” parameter. Make sure the value of the “jvmRoute” parameter is the same as the one defined in the Apache configuration file for each server.

And, you are done! A simple Apache load balancer with session stickiness to Tomcat servers.

Aside

WordPress Upload Problem

I’ve finished setting up the LNMP stack on my VPS, and got WordPress running on it.

I will be sharing the steps I’ve done to setup the LNMP stack on a later date, but first, I’d like share an issue I’ve encountered after getting WordPress up and running.

After running the install.php script and doing up the settings of my WordPress site, I tried to upload a free theme pack from WooThemes. (Typebased, if you are wondering which one)

The first problem I hit was the server giving me a “Connection Reset” everytime I tried to upload the theme. In the Nginx error log, this was recorded:

2011/09/24 07:34:58 [error] 30097#0: *70 client intended to send too large body: 2617076 bytes, client: 202.156.8.11, server: www.brainfart.sg, request: &quot;POST /wp-admin/update.php?action=upload-theme HTTP/1.1&quot;, host: &quot;www.brainfart.sg&quot;, referrer: &quot;http://www.brainfart.sg/wp-admin/theme-install.php?tab=upload&quot;

The “client intended to send too large body” tells us that Nginx was rejecting the theme zip file as it was too big (it’s 2.49MB in size). A check on the Nginx Wiki shows that the default maximum size accepted by Nginx is 1MB, and Nginx resets the connection whenever a browser tries to send or request a file greater than 1MB. So we’ll have to update the nginx.conf file with the “client_max_body_size” directive to get Nginx to send and receive a bigger file. I’ve changed mine to a value of 3MB and you may adjust yours accordingly. Note that the directive is placed in the “http” section.

...
http {
...
client_max_body_size 3M;
...
}

I tried to upload the theme file again, and this time the file was accepted by Nginx without a problem. However, WordPress threw an error, “The uploaded file could not be move to …/wp-content/uploads.” Not a very intuitive error, seems like it is a permission issue with WordPress trying to write to the uploads directory… The logs weren’t showing any error and a Google search resulted in quite a few people having the same problem.

Most of the resolution are to do with setting the correct permission for the uploads directory. A check with my uploads directory shows that the permissions are correct  and I even tried to give full 777 permission to the directory! But error still persist… It took me quite a while, trying to figure out what’s wrong and then it suddenly hit me! It could be related to the Nginx size error, but at the PHP layer. A quick Google shows that PHP has a “upload_max_filesize” and “post_max_size” directive, so I updated my php.ini to include the two directives:

...
upload_max_filesize = 3M
post_max_size = 3M
...

Bingo!  The theme file went through and was installed in WordPress.

Aside

VPS – Setting up

I've logged in to my VPS and started setting it up.

Basically, the VPS has the following specs:

  • BuyVM-256MB
  • 256MB RAM (Burstable to 512MB)
  • 2 Core CPU
  • 30GB of HDD
  • 1000GB of data transfer
  • 1 US based IP

The VPS was first setup with BuyVM’s Centos 6 template (selected during the VPS ordering page). However, the memory usage  seems to be a little bit on the high side, roughly 30mb idling after I cleaned up the unnecessary services and unused packages. No good… Still too much memory for running nothing…

So the next thing I tried was to reinstall to their “centos-5-i386-minimal”. After reinstalling, I can’t even ssh or console into the VPS… Resintalled again, still the same… Seems like that template is broken… No go.

Reinstalled to “CentOS 5 32bit”, the idling memory shows ~14mb.
Sweet! We are good to go! More…