Simple script to extract final GAMESS geometry

Am dabbling with QM codes again, so I needed this quick script without much baggage of other dependencies, so wrote a quick one in Python. You can get this from Github: https://github.com/tovganesh/myrepo/blob/master/extractConvergedGeometry.py

I will call these scripts - quick and useful scrips (QUS) - hence forth and post others when I feel the need :)

Count number of lines for each PDF in a folder

This is just a note about a script which may be useful to you. This one calculates the number of lines per PDF and prints the final count.

import sys

import fnmatch

import os

matches = []

for root, dirnames, filenames in os.walk(sys.argv[1]):

   for filename in fnmatch.filter(filenames, '*.pdf'):

       matches.append(os.path.join(root, filename))

count = 0

for mat in matches:

   if not mat.lower().endswith("pdf"): continue

   cmd = "pdftk " + mat +  " dump_data | grep NumberOfPages > pn.log"

   os.system(cmd)

   try:

     f = open("pn.log")

     l = f.read().strip().split(":")[1].strip()

     f.close()

     print(mat + "," + l)

     count = int(l) + count

   except:

     continue

print(count)

Have a great weekend ! :)

Running mobihf on the iPhone

mobihf (https://sites.google.com/site/tovganesh/s60) is a code that I wrote almost a decade ago on my first phone: the Nokia 6600. mobihf was meant to be an educational tool on how to write a basic Hartree-Fock Quantum Chemistry code from scratch, and to me it was a way to experiment on how much can I push the mobile device to do scientific number crunching. At that time the processing power of my mobile was just about 100MHz with just about 10MB of RAM! And now I have this two year old iPhone 5s that has a 1.3GHz 64 bit processor, with about 1GB of RAM.

To test out how things have panned out for my purely Python mobihf (not the C++ integral code that I wrote later), I tried a number of Python interpreters available on the Apple App store.  While Python for iOS (https://itunes.apple.com/us/app/python-2.7-for-ios/id485729872?mt=8) seemed like a good choice, its editor was never able to even load the mobihf script to do anything meaningful. A mail to the developer did not get me any response (as of this writing).

Next up was pythoni (https://itunes.apple.com/in/app/pythoni-run-code-autocomplete/id493505744?mt=8). This is surprisingly well written app with custom keyboard shortcuts and well made editor for editing Python scripts. It is worth buying the pro features in this app if you do a lot of Python programming on the move. So, finally I was able to run the pure Python mobihf code (available from GitHub: https://github.com/tovganesh/myrepo/blob/master/mobihf.py). This code is heavy on numeric computations (evaluations of exponential functions, matrix and vector algebra etc.). Without going into the details of what the code does, I typically run two test cases to see how things are going. These are for two very simple molecules: H2 (hydrogen molecule) and H2O (water molecule) at a basis level called STO-3G. All very basic to the one who knows Quantum Chemistry, and pretty much a jargon to the rest ;-) Since I am using iOS9, I decided to run the job in two modes: once with battery saver on and another without.

As can be observed above, even with battery saver on, there is 50 to 65 times improvements in the timings over the same pure Python. And, to top it the timings also handsomely beat the C++ compiled code on the old phone.

With the battery saver turned off, the performance improves as expected. What is surprising however is that in comparison to when the battery saver is on, the performance improvement is about 60%. That is quite a hit to keep the phone going for longer. I wish we have some radical improvements in the battery technology so that we can keep enjoying our devices at full speed.

The next app was Sketch Python (https://itunes.apple.com/in/app/sketch-python/id984990674?mt=8). There are a number of different Python implementations and each can give you a different performance numbers. So far I have found Sketch Python to be the fastest.

With the battery saver on, Sketch Python performs almost as fast as pythoni with battery saver off. That says a lot about the performance of Sketch Python interpreter, especially in handling Python code with heavy numeric computation.

The performance of Sketch Python to run mobihf increases substantially when battery saver is off, and is probably the best timing that I can get on the A7 processor of iPhone 5s. If you do the math, this is 140-166 times faster than the pure Python code that ran on my Nokia 6600 a decade ago. In comparison to the C++ (native code) running on Nokia 6600, the gains are an order of magnitude times less, but still impressive. I am sure if I port this code over to native, we can do pretty complicated Quantum chemical calculations on the phone. The newer A9 processor in iPhone 6s or the Qualcomm processors (8-core 810 or higher), would be interesting processors to see the current state. But I can say that these mobile processors have reached the performance levels of the most desktop processors for all practical purposes. And this is exactly that I had in mind when I took a look at the ability to run scientific codes on mobile processors (https://sites.google.com/site/tovganesh/s60).

On a different note, the ARM architecture also makes an interesting case for computational codes because of the low power requirement, a topic I briefly touched, but now studied much deeper by Kristopher et.al. (http://pubs.acs.org/doi/abs/10.1021/acs.jctc.5b00713). Alistair Rendell, my postdoc guide, is a co-author in this paper, so it must be real good :)

Note: The experiments above were run 5 times, and the lowest time was taken.

PS: This post has been since mentioned on MacInChem.org (http://www.macinchem.org/blog/files/0c7f90a37910d2ed90402dcddb6cf4e2-1814.php)

Notes on "sorting a hash table" in Python

Well, by definition you can not sort a hash table as "order" is not really important in that data structure. However, there are situations where data stored in a hashtable needs to be displayed in some ordered fashion. In such a case you can obtain a "sorted representation" of the hash table.
For instance, I had the following hashtable, which I wanted to display as tabular data:

td = {1: {"mykey1":[4,5,6, -2, 5, 6,7], "mykey2":[6,7,8]}, 2: {"mykey1":[5,7,8,9], "mykey2":[0, 9, 7, 6, 8]}, 3:{"mykey1":[5,7,8,9], "mykey2":[0, 9, 7, 6, 8,9]}}

My intention is to display:

	mykey1	mykey2
1	7	3
3	4	6
2	4	5

Essentially, the data is displayed based on the number of items in mykey1 and then sorted on mykey2. You could write code to flatten the hashtable, but you can write this more elegantly as follows:

def hashCountSort(h, k, r=False):

    def len_cmp(x, y):
      ln = 0
      for ck in k:
          ln = (len(x[1][ck]) - len(y[1][ck]))
          if (ln != 0): return ln
      return ln

    return sorted(h.iteritems(), cmp=len_cmp, reverse=r)

And use:

print hashCountSort(td, ["mykey1", "mykey2"], True)

Above, I use nested function in Python (essentially a closure) to define a len_cmp function, that iterates through a list of keys. Forward key comparisons are only made if the earlier key compare returned an equality. Also note that the iteritems() converts individual key, value pairs of the outer hashtable to an iteratable tuple list, with each tuple containing (key, value) items.

More generally one may write:

def hashSort(h, k, cmpf, r=False):

    def hash_cmp(x, y):
      res = 0
      for ck in k:
          res = cmpf(x[1][ck], y[1][ck])
          if (res != 0): return res
      return res

    return sorted(h.iteritems(), cmp=hash_cmp, reverse=r)

And use:

def cmp(x, y):
   return (len(x) - len(y))

print hashSort(td, ["mykey1", "mykey2"], cmp, True)

In the above example, I am using an external user provided function that may be defined by the user indicating how exactly the comparison function should be made for the purpose of sorting. One should note that the cost of such functions is high, and one may need to optimize if you have very large dataset.

Hope someone finds this useful ;)
Have a great weekend!

Calling external Python function from a C/C++ routine

An aricle by Jum Du at CodeProject (see: http://www.codeproject.com/Articles/11805/Embedding-Python-in-C-C-Part-I) gives a very detailed overview of how to call a Python function from C/C++ routines in an embedded interpreter using CPython interface. While this quite useful, however, one needs to keep the .py file at the same place where the C/C++ executable resides.

I needed a solution where the .py file could reside anywhere on the local file system. Turns out that the modification is quite simple, you just need to make sure that sys.path is appended with the correct path at runtime where the .py file can be found. The follwing is the pseudo(C++)-code of how I do this:

PyObject* runFunction(std::string scriptFilePath, std::string funcName, PyObject *arglist)
{
    // this code is based on http://www.codeproject.com/Articles/11805/Embedding-Python-in-C-C-Part-I
    try {
        PyObject *pName, *pModule, *pDict, *pFunc, *pValue = Py_None;
        std::string thePath = // .. your code to extract the path, for '/home/ganeshv/pyfiles/my.py' : path is '/home/ganeshv/pyfiles/'
        std::string theModule =  // .. your code to extract the module, for '/home/ganeshv/pyfiles/my.py' : module is 'my'
        // printf("importing [%s] from [%s]\n", theModule.c_str(), thePath.c_str());
        // first extract the file name and file path
        std::string code = "sys.path.append(\"" + thePath + "\")\n";
        // add the path
        PyRun_SimpleString(code.c_str());
        // Build the name object
        pName = PyString_FromString(theModule.c_str());
        if (pName == Py_None) return Py_None;
        // Load the module object
        pModule = PyImport_Import(pName);
        if (pModule == Py_None) {
            Py_DECREF(pName);
            return Py_None;
        }
        // pDict is a borrowed reference
        pDict = PyModule_GetDict(pModule);
        if (pDict == Py_None) {
            Py_DECREF(pModule);
            Py_DECREF(pName);
            return Py_None;
        }
        // pFunc is also a borrowed reference
        pFunc = PyDict_GetItemString(pDict, funcName.c_str());
        if (PyCallable_Check(pFunc)) {
            pValue = PyObject_CallObject(pFunc, arglist);
        } else {
            PyErr_Print();
        } // end if
        // Clean up
        Py_DECREF(pModule);
        Py_DECREF(pName);
        return pValue;
    } catch(...) {
        return Py_None;
    }
}

Another reason to fool around with MeTA Studio: Python support

Well yes! Many had asked for it so now here is it!

First a bit of history. When I started developing MeTA Studio, I did evaluate Jython (Python implementation for JVM) along with BeanShell and many others. But then finally decided that BeanShell would be the first class scripting interface for MeTA Studio with provision for adding any other scripting language that is targeted for JVM. There were couple for reasons for this decision:

- Jython is too heavy. At about 3 mb (with no debug info). My modified BeanShell is 800kb while if u compile it with no debug info it boils down to a mere 200kb!!
- Prototype-to-production. Python code, though easy to write, doesn't translate easily to Java code. It is very easy to prototype in BeanShell and put it into Java.
- Speed. While BeanShell can run at native speed of Java (i.e. after you compile the code for a target JVM), Jython is purely interpreted, and it was definitely orders of magnitude slower than BeanShell. When I tried Jython the first time many years ago it was about 40-50 times slower than BeanShell, enough to keep me away form making it the primary scripting interface for MeTA Studio. In recent times though Jython has improved substantially. In many cases faster, but not as flexible as BeanShell in terms of the earlier point(s).
- License. It is not LGPL/BSD... So I will only distribute it as a separate addon package (maintainers are needed here ;) !)

and why it is included now:
- IBM supports Jython in their enterprise apps like websphere, I want to impress them ;) With rumours of them taking over Sun.. (Glee .. Well now wondering on that as Oracle plans to buy them instead!)
- I want more developers and users for my platform!!
- Proof that other languages can be easily supported, with out changing anything within the core IDE!
- I want to have fun ;)

Well the complete Jython support for MeTA Studio was added with about 4 days of work, which largely involved writing appropriate wrapper functions akin to BeanShell. And thanks to ways to add external library and widgets framework in MeTA Studio, there were no changes made to the core IDE specifically incorporate Jython support.

So you wanna give a try? Here is how:
The installation instructions are same as previous, but again ..
0) Ensure u have v 2.0.01042009 or a higher version installed. Project URL: [http://code.google.com/p/metastudio/]
1) Unzip metajython.zip (download from: Jython support file on Skydrive) in meta/lib/ext
2) Shut down MeTA Studio if it is already running
3) from command line:
meta/bin >$ java -jar MeTA.jar --addlibs jython
(you can skip this step, if you have already done so with the previous builds)
4) Start MeTA Studio, open the code editor. Then open jythonWidget.bsh from meta/lib/ext/meta-jython directory
5) Click on Make Widget to get the Jython widget on the the widget panel

Now you can either use the shell (like the usual python shell) or use the editor to write code in python as you would normally do for BeanShell scripts.

I will be putting up user guides on how to use the Jython support in MeTA Studio at: http://code.google.com/p/metastudio/wiki/PythonSupportForMeTAStudio. You are also strongly encouraged to contribute to the wiki, just drop me in a mail if you want to!

Now that Jython support is out, would like to state few things about future of this Jython support of MeTA Studio:
- Firstly, BeanShell will always be the primary scripting interface in MeTA Studio. Meaning that all new releases of MeTA Studio will come installed by default with BeanShell as has always been the case.
- Jython will never be distributed along with binary distribution of MeTA Studio. This support will always be available as an external download as is provided above.
- Since this project is getting big day-by-day, it would be really nice if some one can take up the responsibility of maintaining the Jython support in near future. Though I would maintain it with a bit less priority till I can get a maintainer for this port.

Fool around ;-)