Combined Intelligence – Taking on Google takes more than just a few crufty brain pans!

[rant]

Sometimes I like to do a little reading on the web. In many cases it is a quick and easy way to kill a few minutes, and I don’t have to keep track of a hardbound book. I was inspired to write this article by what I read, hoping perhaps you will enjoy this reading as well. Anyway, I’m reading through the web in one window, and tailing apache logs in another. Suddenly I notice some systems displaying tar-pit like messages:

TCP: Treason uncloaked! Peer 38.108.180.95:31794/80 shrinks window 344798477:344803373. Repaired.
TCP: Treason uncloaked! Peer 38.108.180.95:31794/80 shrinks window 344798477:344803373. Repaired.

This is typically a mis-configured interface, or perhaps a mis-behaving switch. However, in totally clean cluster environments like the one I was looking at, I usually give this some follow through as I should not be getting window scale corruption. A quick look at the ns records refers me to “name = h-180-95.scoutjet.com”. Curious about what they are doing, I opened up the browser and read the little spider page.

http://www.scoutjet.com

I decided to take a closer look, after all its every day that some foreign cracker will spoof IP addresses of idiotic tech companies just to get double desert. A quick look and what do you know!? The offending IP is not spoofed, and does in fact reside within their netblock:

%rwhois V-1.5:0010b0:00 rwhois.cogentco.com
38.108.180.95
network:ID:NET-266CB40018
network:Network-Name:NET-266CB40018
network:IP-Network:38.108.180.0/24
network:Postal-Code:94065
network:State:CA
network:City:Redwood City
network:Street-Address:100 Marine Parkway
network:Org-Name:Blekko
network:Tech-Contact:ZC108-ARIN
network:Updated:2008-09-17 10:42:07
network:Updated-by:John Knowles

First thing that crossed my mind was why does such an innocent looking spider page come up while investigating a potential TCP datagram corruption situation?

So, reading on a little I see these fellows want to take on Google! This altered my thinking rather quickly. If you are going to take on Google saying you are building a search engine that will outperform theirs (and say that Google is “going down”), you should probably at least have a very public debut of your web search, some rallied public support, and perhaps a few hard metrics! Otherwise, it gives the tech savvy reader much the same impression I got: “This sounds like a bunch of geeks who hatched some plan at a pot party, and are now scouring the web for attention”.

Moving forward, I see I was most likely correct: http://www.techcrunch.com/2008/01/02/the-next-google-search-challenger-blekko/

Now, I normally won’t rag on people, especially not publicly, but here’s the part where I start getting annoyed with society today. At what point do people become so naive that they will launch a publicly printed article based on so much freeze dried crap? The part that really stuck in my craw was this -> http://www.crunchbase.com/person/rich-skrenta {}

I’m sorry Rich, been involved with computers since the early 1970’s, and despite the clarity of your ego you did NOT invent the first “self-spreading” virus. The FIRST ever computer virus was named Creeper, and it was created back in the early 1970s by a fellow named Bob Thomas. At the time the Internet didn’t exist as we know it today, it was referred to as ARPA-NET which was one of the many extra-nets used by the medical, military, and scientific communities to rapidly share everything from medical prescriptions to scientific data sets. Though Bob’s virus was done as a test, and only affected VAX type computers, it none-the-less was created while Rich was still just a wee tot.

The first “wild” Trojan was called “Animal” (named for the pervasive way it would copy to any connected equipment, including disk drives) came out shortly thereafter in the mid 70’s.

Now credit where credit is due, Mr. Skrenta did create the Elk Cloner virus as he claimed in 1982, however lets reveal the truth behind the myth of this particular virus (of which I still have a copy on a library of intentionally infected disks, even though I don’t have a living Apple II to run it on). Creating a boot virus for the Apple II is hardly something I’d try to stake a claim to fame on…

A virus is nothing more than a malicious computer program which typically harbors some ability to copy itself to other components of the system. So, creating a simple computer virus? Not a daunting task. Its (once again) QUITE simply a MALICIOUS COMPUTER PROGRAM. Considering people have been programming with computer languages since Bell Labs in the 1950’s, this hardly seems like a feat.

That being said Elk Cloner was not really so much of a feat at all. In fact, other similar mal-ware was created, just never unleashed on the public (which seems a lot like a high school prank). The Apple II was widely known by experts of the time to be very susceptible to intrusion, particularly with respects to the disk drives. Since Apple II was reliant on floppy disks to run it seemed like a no brainer that copying malicious data on the source disks would certainly allow for operating system corruption on subsequent systems that booted from that same floppy.

Secondarily, the Apple II was also widely known for its lack of boundaries with boot strapping. A “Boot strap” simply refers to a small set of functions coded into the disk startup that allow user driven executable code to be launched into initial memory on a system. While the straps typically contain only logical volume data to be used in sizing up the geometry metrics of the installed hard disk, and launching the disk’s start-up programming, the Apple II boot routines were widely known to be easily extensible through a lack of byte counts on the strap. In English, this type of malicious coding will not work on any computer that’s less than 25 years old, and really just knowing you can modify the boot sector or write to unallocated memory of the boot areas is more than half the battle.

To prove my case, my good friend Donald Judd and I wrote a simple virus while we were at Arizona Tech that was intended as “proof of concept” that Windows operating systems of the day lacked any sanity when it came to intrusion of their operating environment. The really cool part, is we wrote it using nothing more than Debug.exe (a system included debugging/memory editing utility), and did it in less than 10 lines! In fact the entire virus was only 8 bytes in size. For those of you who are familiar with the environment, here’s the concept:

Simple put: Read the A register in memory. Move the AX register to value 0, call interrupt 19 (hard reboot), read the difference between the registers (8 bytes), write the file.

Now, this is just proof of concept stuff. Interrupt 19 was a hardware interrupt which signaled the system to restart (regardless of circumstances). While this code could not copy itself to a boot sector, it does demonstrate that writing a malicious computer program is simply that: Writing a malicious program. In fact interrupt 19 went away when the inception of software based interrupts within Windows in the late 90’s, and running it on a modern system will result in nothing more than a “protected memory” error (if anything at all…you’d have to run it from %comspec% to get the error):

In other words lots of people out there started exploiting Microsoft’s inability to control protected memory in a sane way, and MS learned from it. Anyway, the point is this code was eventually discovered and exploited by the people who wrote virii like “Melissa” and “Love”. In fact we experimented creating DOS boot type virii which would do non-destructive things like cascade ASCII smiley faces at you while your computer boots.

This article seems a lot like a rant, and well, I guess it is. After reading self edited wiki entries about Mr. Rich I decided it was high time that someone stepped up, and told the rest of the world that are NOT computer savvy, DON’T BE IMPRESSED with tomfoolery! Computing is a combined intelligence, and not one single stand alone person in the world possesses the knowledge to completely rebuild a modern computer in the event all the technology somehow went away. In other words, if there was a global holocaust, and all the computers and supporting archives were destroyed, there’s not a SINGLE person in the world that could recreate what we think of as a computer by themselves! Computing is a COMBINED INTELLIGENCE, and as such, there will always be someone smarter out there.

Don’t let someone take fame from a high school prank that isn’t really all that original, and use it to rally esteem which would cause one to think such a person can take on Google. If you think I’m kidding, look at what Google represents. The literal definition of the math phrase the search engine took their name from is:

“A googol is the large number 10100, that is, the digit 1 followed by one hundred zeros (in decimal representation). The term was coined in 1938 by Milton Sirotta (1929–1980), nephew of American mathematician Edward Kasner.”

To take them on you’ll need a lot more than a few geeks with a funny paper bag on their head, or pictures of some geek brainstorming session coupled with someone who self-proclaims to the world they are superior because they created a malicious computer program. You’ll need someone smarter than them. Smarter than me, perhaps even smarter than Kevin David Mitnick. You’ll need Steven Hawkings on acid! You’ll need….a googleplex of brains. In short, you’ll need COMBINED INTELLIGENCE!

[/rant]

Enabling WebDav for CDN [Building Coda file system support] using CentOS 5.2 Linux

Many people have asked me about setting up WebDav to directly address a CDN repository as a /local/path directory. This requires Coda file system (Advanced NFS) support. This particular tutorial is directed at people who wish to use WebDav support for a CDN such as BitGravity (as in this example) on a Linux system. The particular flavor of Linux used in this tutorial is CentOS 5.2, so if you are using a different form of Linux the procedure is similar but will vary from distro to distro.

Install the Necessary Packages:

To access the necessary packages in Yum, the RPM repository needs to be up to date. I used the following commands to clean up Yum:

rpm -e yum-plugin-fastestmirror
yum clean all
yum install yum-metadata-parser yum-updatesd

yum install yum-plugin-fastestmirror

Once added, run the following installations:

yum install davfs2
yum install dkms
yum install dkms-fuse
yum install fuse
yum install fuse-davfs2

Verify that the Coda File System is Supported in the kernel:

For the WebDAV share to mount correctly the module for the Coda file system needs to be supported by the kernel. To test if it is supported run the following command command:

dmesg | grep -i coda

If the system returns an output, then everything should be fine, and the configuration of WebDAV can continue. However, if it responds with nothing then the kernel does not support coda. To overcome this obstacle the kernel source tree will need to be built so that the necessary coda files can be extracted.

Build the Kernel Source Tree and Add Coda Support

By building the kernel source tree the necessary files for coda file system support can be manually compiled, and then copied into the current kernel’s library. In this way you don’t have to rebuild your running kernel.

During the most of the build procedure it is critical that it be performed as a non-root user to maintain the necessary permissions.

>> As root user or su -  <<

yum install kernel-devel
yum install kernel-headers
yum install rpm-build
yum install redhat-rpm-config
yum install unifdef

>> Exit su – (or sudo to a non-root user) and continue with the following steps! <<

After installing the packages log into the non-root user and navigate to ~ (the $HOME directory), and create the directory for the source:

mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
echo ‘%_topdir %(echo $HOME)/rpmbuild’ > .rpmmacros

Type “uname -ar” to display the kernel version and then install the respective kernel source. For example:

rpm -i http://mirror.centos.org/centos/5/updates/SRPMS/kernel-2.6.18-92.1.18.el5.src.rpm 2> /dev/null

Now spec the kernel module version for this build with:

rpmbuild -bp –target=`uname -m` kernel-2.6.spec 2> prep-err.log | tee prep-out.log

After installing the kernel source tree navigate to the install location and start the build process:

cd ~/rpmbuild/BUILD/kernel-2.6.18/2.6.18.x86_64/
make oldconfig

Accept the default values [especially if you aren't sure].

make menuconfig

Browse the source tree menus to ‘File Systems’ –> ‘Network File Systems’ –> ‘Coda File System’. Type “m” for the value, which specifies modular support for coda.

Continue the build process with these commands:

make prepare
make modules_prepare
make M=fs/coda

With the build steps completed the fruits of labor can be plucked from the kernel source tree:
>> As the root user, or su – <<

cp ~/rpmbuild/BUILD/kernel-2.6.18/2.6.18.x86_64/fs/coda/coda.ko /lib/modules/2.6.18-53.1.6.el5/updates/

And at last, coda module installation can be initiated:

depmod -a

modprobe coda

Check that the module started correctly with :

lsmod | grep coda

Now we want to be sure the Coda module starts at boot time so add this line to /etc/modules.conf

probe coda

Configure WebDAV

Once all packages are installed navigate to /etc/davfs2 and edit two configuration files. The configuration for BitGravity’s WebDAV will be used as an example.

By default all the options are commented out on /etc/davfs2/davfs2.conf. For WebDAV to work properly for Deca’s setup the following lines need to be un-commented, and edited as such:

# General Options

# —————

dav_user        davfs2            # system wide config file only
dav_group     users             # system wide config file only

kernel_fs     coda

# WebDAV Related Options # ———————-

use_locks 0

# Debugging Options # —————–

debug most           # possible values: config, kernel, cache, http, xml,
#      httpauth, locks, ssl, httpbody, secrets, most

And add the following line to the Credential Line section of the secrets file:
[Adjust as per your own particular CDN. This line was for BitGravity but check with your CDN provider to be sure what your credentials are before proceeding.]

http://webdav.bitgravity.com/   userid@yourdomain.com  password

Next, make a directory for the WebDAV share to mount:

mkdir /davtest

Create dav2fs user:

useradd dav2fs [make sure you choose nologin, and no shell]

Check /etc/passwd to be sure the user dav2fs has no permissions on the system [/sbin/nologin should be the shell interpreter]:

davfs2:x:101:102:davfs2:/var/cache/davfs2:/sbin/nologin

Give the new user ownership of the WebDAV mount

chown -R dav2fs.users /davtest

The final file to edit is /etc/fstab, in which you will add a line like this:

http://webdav.bitgravity.com/       /davtest     davfs   uid=dav2fs,gid=users,dir_mode=755    0 0

To initiate the mount run:

mount -a

Test that you have everything correct with:

touch /davtest/test.txt
ls -l /davtest/test.txt

You can manually mount the share for troubleshooting purposes with:

mount.davfs -o dir_mode=755,uid=dav2fs,gid=users http://webdav.bitgravity.com/ /davtest

If there is a problem check /var/log/messages!

Linux and WiFi

Configuring a wireless networking card under Linux can sometimes be a hassle. Most hardware vendors are not yet advanced enough to offer multi-platform drivers, and the need for wireless networking in Linux has grown substantially as the OS gains favor in the home user environment.

While many distributions of Linux support wireless networking cards, as well as extended USB support, some installations can be a little counter-intuitive when it comes to getting the hardware installed and working. For this reason a good recommendation is Mandriva Linux (2008.1 Spring / One). Mandriva has done even more than before to expand their already robust hardware support, and a Linksys USB N type wireless network adapter installed with little or no trouble at all.

It is important to point out that Mandriva configured the wireless card through the USB core, and NOT through NDISWrapper. This is important because it is a native driver base, and not an emulator. The one caveat to the installation was getting the network connection manager to see the previously installed adapter. There are two easy ways to accomplish this.

Plug in the Linksys N card to the USB port.

Insert your Linksys driver disk in the cdrom, and mount it, or note if it is auto-mounted in /media.

If you are running a desktop, such as KDE or Gnome, then you will go through the “System”…”Applications” menu to “Configure your computer” (Control panel). From there, you will click on “Add new network connection”, and select “Wireless” as the type. When prompted for the driver, point the wizard to the Linksys factory Windows XP driver. In this case, the path was /media/WUSB300N/Drivers/XP/*.inf

At this point if your system as correctly installed initially, it should recognize the USB core driven devices such as the Linksys N card, and should prompt you with the message “The device you are attempting to configure is already configured using the USB core. Do you really want to use NDISWrapper?”

Click NO/Cancel

At this point a “service network restart” from a terminal (command line), or starting the control panel “Configure network connections” applet should allow you to see the working adapter. If it did not immediately refresh, you might consider a reboot if you are a novice, so that the system should init the wireless WLAN0 interface on start-up. If you are an experienced user you should be able to simply refresh the networks list in the wireless control interface in the “Start Bar” icons.

Enhancing Program Performance on Windows

Unix system administrators noticed a long time ago, that if you installed multiple partitions sized carefully, you could split your operating system data from your program and application data by partitioned drives. By doing this, you could prevent your application data from being intermingled with operating system data on the disk’s physical geometry, and since Unix or Linux won’t store data in a fragmented state, you could be sure that the OS data was (for the most part) “One continuous read” for the hard disk. The same could be said for the application data.

It also allows you to format and re-install your primary disk/OS without completely losing your application or program data, and visa – versa.

Perhaps one of the greatest benefits of this configuration is that if you inadvertently wipe out your partitions you might lose your OS, OR your applications/programs, but usually not both.

Long ago I started doing this on my windows systems, and I’ve had tremendous success. I’ve seen cases where my small outdated desktop at home ran much faster than my all powerful office desktop, mainly because of configuration customizations like this.

If you are installing a Windows XP or similar type system I recommend you do the following with regards to disk partitioning.

Primary partition 1 – 7.5 GB Size – Windows Installation and OS disk.

Partition 2 – 2.5 X the amount of RAM in your system.

Partition 3, 4, 5… Allocated as you wish, with at least one large NTFS partition to support your program file installations and applications data. I recommend not less than 50% of the drive, or more if you choose. If the system is not going to be multi-boot, then you might as well allocate at least 15GB blocks. You needn’t use the entire disk if you aren’t sure what you will need for a breakdown, you can always modify those later.

Install Windows and Critical Drivers

Format your drives as NTFS. You can format them all, or just the system disk (partition 1) for now, and format the others later. It might be smart to do it now to avoid getting lost later in the article. Install Windows to the C: drive (Partition 1). Install ONLY THE DRIVERS and critical device software needed for the system. Connect to update.microsoft.com and follow instructions to get your service packs (and OS patches) up to date as you desire. The service packs are important because SP3 for XP (for example) is over 250 MB, and is more of an OS “re-installation” than a service pack.

Move the system page file (swap file)

Right click on “My Computer”, and choose “Properties” (or simultaneously press the Windows Key + Pause Break) to reach the “System Properties” window.

Click on the “Advanced” tab, and then on “Performance” click “Settings”.

Click on the “Advanced” tab within the “Settings” window, and under the “Virtual Memory” settings click the “Change” button.

Disable the system swap file on drive C:.

Enable the system swap file on drive D (second partition). Set your swap file MIN and MAX to NO less than 1.4 X the amount of RAM in your system, and not more than 2.5 X your system RAM. I recommend using 2X the size of my system’s RAM, or 2048 MB, whichever is lowest. Some experts recommend up to 2.5 X the amount of your RAM be allocate for swap, so I have set that as the MAX for purposes of this article. The important part is to make the MIN and MAX the same, which will alleviate window’s propensity to continuously map swap space.

Click OK to save, and exit the properties dialogs.

Defragment the Windows Drive

Install Programs and Application Data

On your third partition, which should be at least 15 GB, install all your favorite programs. Make sure as you go through each program or application installation routine that you choose “Custom” installation type, and change the installation directory to be “E:\Program Files\programname” (or whatever drive letter you choose past the swap disk of D:).

All future software an add-ons should be installed here if possible.

Defragment the Applications Drive

That’s it! You are done! Now if you will be able to reap the benefits from the performance metrics I hit on at the start of this article.

Good luck, and happy computing!