How to process AOL’s search logs with PHP

AOL, for some reason, released 2,000,000 search records taken from the last 3 months. With a tiny bit of programming, you can use PHP to search through these records. This post describes how.

But first, let’s cut to the chase:

First, you need to download the files, which you’ll have to find on your own, as I can’t remember where I found them. The file is ~450MB.


wget http://whereverthosefilesare.com/AOL-data.tgz

Next, uncompress this file into its multiple parts:


tar zxvf AOL-data.tgz

Then, un-gzip the compressed files:


gzip -f user*.gz

This will leave you with a bunch of text files containing the actual log data. These files will be very large. So large in fact that your web server might not be able to deal with them in PHP. So, you should split them into multiple smaller files. For each .txt file, run a command such as:


csplit -f aol user*01.txt 1000 {1000} &

That code will split the files into multiple files of 1000 lines each, naming each them aol100, aol101 and so on. In these smaller chunks, PHP will be able to deal with them more easily. When I ran this command, I ended up with 2000 files of 1000 lines each. For the code below to work, each of these files must have the letters “aol” in their names.

Next, you can create a PHP page that will search the AOL log files for a certain string, and then return matching lines. The string you are looking for will be passed on the URL, as in http://example.com/search.php?search=sex

Here is the code for the PHP page.

Save the file as search.php or whatever, and then access it at its url, e.g., http://yourwebsite.com/search.php?search=thewordyouarelookingfor

Change the URL to run another search. You can take the user id from one search and put it on the URL, thereby finding all the searches by that user.

Brief thoughts on the iPhone

I don’t usually do opinion pieces here, but today is sort of a notable day. Here are my general thoughts on the iPhone.

  • it is small and light
  • the virtual keyboard thing is a pain in the ass. i’m sure it gets better the more you get used to it, and the auto-correction works well, but it never types the letter you are aiming for. it really slows you down. the keys need to be wider. also, the virtual keyboard gets in the way. for instance, when accessing my gmail, i had to type in a captcha code to sign in, but the keyboard obscured so much screen real estate, i couldn’t see the captcha text and the text field at the same time. it also sets the visual focus wherever you are typing, so you can’t look at the captcha while typing blindly. i had to memorize the sequence, then scroll down and type it in.
  • interface is not as dead simple as you would expect. for instance, in the web browser, i couldn’t figure out how to go to a new page! turns out the address bar is pinned to the top of the page, instead of the top of the browser window. SO YOU HAVE TO SCROLL UP on a PAGE to find the address bar. Really poor decision there, and very uncharacteristic of Apple. the apple store employee who i asked about this was puzzled by it and couldn’t immediately figure out how it worked.
  • it’s a little slower than you would hope. for example, when you pan-and-scan a pdf document, the screen redraw takes a second or two. that said, the ability to open up and zoom in and out of a pdf or other attachment is nice.
  • the interface across applications is not consistent. forward/back work differently in safari, the notepad, and itunes.

finally: because it is still a closed system, you sit there using it for 15 minutes or so, fooling around with safari, the camera, email, and itunes, and then you realize that you have done everything you can do with it. the words that went through my head were, “Well, I guess that’s it.” In the end, it’s just a cool phone.

i don’t think i am going to get one. it seems to me that any Nokia Series 60 (aka “N Series”) phone does nearly everything that the iPhone does, with the exception of the touch-screen interface, The Nokias have these advantages that are valuable to me:

  • GPS (N95 only)
  • unlocked; no contract
  • multimedia messaging, which was inexplicably left off the iPhone
  • 3G data
  • open architecture, so you can install third-party applications
  • way better camera (zeiss lens 5MP vs. run-of-the-mill 2MP)
  • video recording (inexplicably left off the iphone; N95 does 480p)

sure, the Nokia interface isn’t as slick, but it is still very well designed, and probably more efficient for the most common operations (calling people, texting, sending someone a picture).

the one caveat for me here is that the iPhone OS X certainly has 1000x more potential than the Series 60/Symbian OS, so if/when Apple opens it up, it will blow away the Nokias and everything else.

How to use iTunes “currently playing” info as your email signature in Outlook

outlook with itunes screenshot

Ok, here’s one for the masses.

This post describes how to add info about the song currently playing in iTunes to emails you compose in Outlook. Anyone can do it!

First, create the macro. To do so:

  1. Open Outlook.
  2. From the Tools menu, select Macro, then Macros…
  3. In the Macro field, enter a name for your macro. “AddiTunesSig” is probably appropriate. [screenshot]
  4. Click “Create”.
  5. A “code window” will open. This is where, in a minute, you will type the code that makes up the macro.
  6. But first, you need to tell Outlook how to talk to iTunes. To do so, go to the Tools menu and select References. Scroll down until you find “iTunes Type Library” or something similar. Select its checkbox and click OK. [screenshot]
  7. Back in the code window, you will see two lines of code: Sub AddiTunesSig() and End Sub. In between these lines, paste in the following code:
    
    
    ' purpose: opens a new mail message, 
    ' appends info re: currently playing track in iTunes
    
    Dim itunes As iTunesApp
    Dim strArtist As String
    Dim strTitle As String
    Dim strOutput As String
    Dim mail As Outlook.MailItem
    
    Set itunes = New iTunesApp
    
    'load the track info into easy-to-read variables
    strArtist = itunes.CurrentTrack.Artist
    strTitle = itunes.CurrentTrack.Name
    strAlbum = itunes.CurrentTrack.Album
    
    'assemble the signature
    strOutput = vbNewLine & "------------------------------"
    strOutput = strOutput & vbNewLine & "Now playing: " & vbCrLf _
         & strTitle & vbCrLf & strArtist & vbCrLf & strAlbum
    
    'create a new email message
    Set mail = Outlook.CreateItem(olMailItem)
    
    'put the signature in the body
    mail.Body = vbNewLine & vbNewLine & strOutput
    
    'show the message to the user
    mail.Display
    
    'clear up the memory
    Set itunes = Nothing
    
    

    See how easy programming is! [screenshot]

  8. To finish up: Click the Save button on the toolbar and close the code window.

Now, you need to add a toolbar button to your Outlook toolbar that will run this macro. To do so:

  1. Go to the Tools menu, and select Customize…
  2. In the dialog box that pops up, scroll down the left side, and click Macros. [screenshot]
  3. You will see your macro on the right (“Project1.AddiTunesSig”). Drag it up to a convenient location on your toolbar. (Next to the “New” button is probably a good place.)
  4. To rename the button, right-click the button and type the new name where it says “Name:” in the pop-up menu. “New (with music)” is probably a good name. [screenshot]
  5. Close the Customize window.

That’s it! To use it, open iTunes, play a song, go back to Outlook, and click your new button. A new email will open with info about the currently playing song in the signature area.