I have recently started using MongoDB for a very demanding task. Storing all Forex pair ticks (e.g. every change in bid/ask prices). I know there are frameworks designed for this task (e.g. kdb+), but since I wanted to avoid the learning curve. Besides I already use Spring Data in my project and it works with a minimal number of changes for Mongo.

In Mongo I have a collection with more than 3.5 billion records (and growing) and I want to find the latest date for each pair. I tried using the aggregation framework of Mongo, but it doesn’t seem to use the indexes and takes ages (didn’t finish after one day).

Relational Structure

In relational DB the table structure  would look something like:

idpairdateTimebidask
1EUR/USD2015-04-03 21:32:31.4561.141411.14142
............

Then you would have to run the following query:

 

MongoDB Aggregation Framework

In MongoDB the document structure is the same. I am a very very novice user of Mongo, but I gather we could use the aggregation framework for this query:

However, this takes ages, even though I have used a composite index on pair and dateTime.

 

Very Fast Result Using MongoShell

I tried using a sort of iterative approach using MongoShell:

This approach seems to use the composite index on pair and dateTime I defined and the results are lightning fast (for 3.5 billion records).

Maybe there are other ways, but after some digging around I couldn’t find any other method that would use indexes.

I have a Thomson TG585v8 router and I connect to a Cisco VPN using my Macbook running Mavericks. When I connect to the VPN all other devices in the house lose connectivity to the internet! I checked the router logs and I got quite a few errors like the following when connecting to the VPN:

UPnP action ‘AddPortMapping’ from ip (Undocumented UPNP error)

It is clearly and error related to the uPnP. So I disabled UPnp on the Thomson router and now everything works perfectly when I connect to the VPN.

I installed a fresh Ubuntu desktop guest on VirtualBox 4.3.14. Trying to install VBox additions I was getting an error message:

The headers for the current running kernel were not found. If the following
module compilation fails then this could be the reason.

 

Apparently all I had to do is reinstall the headers using the following two commands:

 

Trying to reinstall Linux Additions for VirtualBox works like a charm!

I am using VirtualBox 4.3.2. I have recently upgraded to OSX Mavericks and after I close VirtualBox it doesn’t seem to close properly and it’s still running. Doing force quit doesn’t seem to close VirtualBox.

The worse part is that I can’t seem to be able to shutdown my laptop! When try to shutdown after having run VirtualBox the shutdown process gets stuck and the only way to shutdown properly is by pressing the “Power On/Off” button on the laptop for some time.

The solution that works for me, is to go to the “Login Window” screen (by clicking on my account name on the top right corner, next to the clock -> Login Window…”. Then I click on “Shutdown” from that screen and it works.

I am using Logback to do most of my logging. I am using a third party library for some computation. Luckily the library uses slf4j, meaning it’s agnostic of the logging implementation.

The library caches two collections of items and depending on the circumstances the item I am looking for may be in any of the two collections. This is not important. What is important, is that the library for some reason logs an error if an item cannot be found in the first collection.

I wanted to get rid of this message because it was cluttering my logs. But I didn’t want to disable all logging from this specific class, because it may print some useful logs. I just wanted to get rid of this specific log. There is a way to do it Continue reading

Since Spring 3.2 it should be possible to use a qualifier in the “Async” annotation of a method, to indicate which specific executor to use. For example, I have the following class, that is supposed to collect the HTML from a website asynchronously:

HTMLFetcher Interface

TestHTMLFetcher Class

And the following excerpt from the context.xml file:

This didn’t work! I submitted 10 tasks to the executor and 10 thread, instead of 1 as I had instructed it had been created. I digged into the Spring code a bit and found the AnnotationAsyncExecutionInterceptor class that was doing tbe job of assigning tasks from methods annotated as async to executors. Putting  a breakpoint on its getExecutorQualifier method it became evident why it doesn’t work.

You need to annotate the interface method rather than the class with the:

annotation. In my example that is the getHTML method of the HTMLFetcher interface. It now works. I am not sure if it was done on purpose (i.e. if it’s part of the specification). I don’t have time to read the related documentation or search the Spring Jira. However, I would assume that the right place to put the annotation is the implementation. I may want to have two implementations of the same method, one decorated with “Async” that will be asynchronously executed and another without any annotation that will be synchronously executed.

By default on Windows Eclipse autocomplete shortcut is: Ctrl+Space. In my Mac laptop I use this combination for the operating system’s spotlight quick search. As a result, I wanted to change the default autocomplete shortcut. What you need to do is:

1. Go to Window->Preferences (Eclipse -> Preferences on Mac).

2. In the new window that pops up, go to “General -> Keys”

3. Find the “Content Assist” command and change it to whatever you like (in my case I chose Cmd+Space).

4. Don’t forget to click on “Save” on this pop up window!

I wanted to add a new device to my wireless network, but I had forgotten the wireless key. My network uses WEP encryption. So, I used aircrack to recover the key. Basically what I had to do was:

    • use airodump to save a large number of  transmissions between the wireless router and a device that is already connected to the network.
    •  use aircrack to analyse the file produced by airodump and find the password

In particular, I managed to recover the key in the following simple steps:

  1. Download BackTrack Linux Distribution and burn it on a DVD.
  2. Boot my laptop using the live DVD
  3. On command prompt type:
    ifconfig to see the available network interfaces in Linux. Doing this I was able to find my laptop’s wireless interface.
  4. Type: airodump-ng -write afile.cap wlan0 , where afile.txt is the file that airodump will save all communication and wlan0 is the wireless network interface of my laptop as discovered in step 3. Let it run and collect packets for quite some time. It may a few hours (in my case it took 6 hours) for this step to collect enough packets. The time it will require depends on the traffic of the network. The more traffic the better. Once enough packets have been collected press Ctr+c to kill the process.
  5. Type: ivstools -convert afile.cap afile.ivs to convert the captured packets to ivs format compatible with aircrack
  6. Use aircrack-ng afile.ivs. Aircrack will pop up a menu to ask you which network you want to crack. Select the SSID of your network and if enough packets have been collected in step 3, you will have the key of your wireless network in no time!