Stop Flickering in the Visual Studio 2010 Beta 2 Editor

The Visual Studio 2010 beta 2 release is awesome. One issue for me is a strange flickering of huge white blocks. I use Vibrank Ink2 theme which has a black background which might be a cause of the problem. The cause seems to be when I hit one of the outline hot regions it must be selecting the corresponding region and applying a white highlight. Quick solution is just to turn off outlining – which I never anyway.

Right click in the text editor –> outlining –> stop outlining

Openfst Drawing for Japanese (UTF-8)

The string “重み 付き 有限状態 トランスデューサ” is represented as automata in text format (jpn.wfst)

0 1 重み
1 2 付き
2 3 有限状態
3 4 トランスデューサ
4


A symbols file (jpn.syms) required is also required to compile the machine



0 1 重み
重み 1
付き 2
有限状態 3
トランスデューサ 4


The machine can be compiled and drawn with graphviz using the following commands:



fstcompile -acceptor=true -isymbols=jpn.syms jpn.wfsa | fstdraw -acceptor=true -isymbols=jpn.syms  -portrait=true | dot -Tpdf > jpn.pdf


If all the text files are in UTF-8 the pdf should display the Japnese strings correctly (as long as the viewer supports Japanese)

OpenFst Example Operator Overloading

Quick snippet to drop into programs using OpenFst that overloads the * and + operators to perform the Times and Plus. I find this more readable when adding/multiplying many times.

inline LogArc::Weight operator+(const LogArc::Weight& a, const LogArc::Weight& b)

{

    return Plus(a,b);

}; 

 

inline LogArc::Weight operator*(const LogArc::Weight& a, const LogArc::Weight& b)

{

    return Times(a,b);

}  

OpenFst Example: Using User Defined Flags

In this OpenFst sample I will show you how to create a tool in the style of the library tools and add a custom command option (user define flag) that does not require the compilation of the whole library.

The below testfst.cpp file calls parses the command options and calls a main function that essentially does nothing. The DEFINE_double macros is used to define our flag names test_flag in this case.

 

#include "test-main.h"

 

namespace fst

{

  REGISTER_FST_MAIN(TestMain, StdArc); 

  REGISTER_FST_MAIN(TestMain, LogArc);

 

DEFINE_double(myflag, 1234, "My Flag");

using fst::CallFstMain;

int main(int argc, char **argv)

{

  string usage = "Test flags \n\n  Usage: ";

  usage += argv[0];

  usage += "  Flags: myflag\n";

  std::set_new_handler(FailedNewHandler);

  SetFlags(usage.c_str(), &argc, &argv, true);

  if (argc !=1)

  {

    ShowUsage();

    return 1;

  }

  return CallFstMain("TestMain", argc, argv, FLAGS_arc_type);

}

The corresponding test-main.h files is the actual main function that just echo value of myflag a corresponding DECLAR_double macro.

#ifndef TEST_MAIN_H__

#define TEST_MAIN_H__

 

#include <fst/main.h>

DECLARE_double(myflag);

DECLARE_string(arc_type);

 

namespace fst

{

 

  template <class Arc>

  int TestMain(int argc, char **argv, istream&, const FstReadOptions&)

  {

 

    cout << "myflag=" << FLAGS_myflag << endl;

      return 0;

  }

#endif 

Next compile with:

g++ fsttest.cpp -lfstmain -lfst -ldl -o fsttest

Running the help option we see our new flag in the help options.

fsttest –help

..

..

--delta: type = double, default = 0.000976562
      Comparison/quantization delta
    --height: type = double, default = 11
      Set height
    --myflag: type = double, default = 1234
      My Flag
    --nodesep: type = double, default = 0.25
      Set minimum separation between nodes (see dot documentation)
    --ranksep: type = double, default = 0.4

..

And double check for a few values

$/fsttest --myflag=10
myflag=10

$ ./fsttest --myflag=-2.2134234
myflag=-2.21342

My two feature request are the shorthand versions, for example –i for –isymbols and that a tool only displays it relevant options.

Fst types avaliable in OpenFst

In OpenFst the command fstconvert is used to convert the fst type, either based on a stl vector container or const type. In version 1.1 a potentially very cool compact fst type was introduced.

Displaying the help by fstconvert –help displays the

Converts an FST to another type.

  Usage: fstconvert [in.fst [out.fst]]
  Flags: fst_type

  ……

  --fst_type: type = string, default = "vector"
      Output FST type

This wasn’t so helpful, so I added a quick Function to the FstRegister in include/fst/register.h

  void PrintFstNames()

  {

    for(typename map<string, Entry>::const_iterator it = fst_table_.begin(); it!=fst_table_.end(); ++it)

    {

      cout << it->first << endl;

    }

 

  }

The created a quick app to create the FstRegister  and print out the types.

 

#include <iostream>

#include <fst/main.h>

 

using namespace std;

using namespace fst;

 

 

int main()

{

        FstRegister<StdArc>* r = FstRegister<StdArc>::GetRegister();

        r->PrintFstNames();

        return 0;

}

Here are the list of the fst types for the StdArc (tropical semiring) should be the same for LogArc types. Unfortunately, no compact transducer though. The acceptor types might be useful for large language models.

compact16_acceptor
compact16_string
compact16_unweighted
compact16_unweighted_acceptor
compact16_weighted_string
compact64_acceptor
compact64_string
compact64_unweighted
compact64_unweighted_acceptor
compact64_weighted_string
compact8_acceptor
compact8_string
compact8_unweighted
compact8_unweighted_acceptor
compact8_weighted_string
compact_acceptor
compact_string
compact_unweighted
compact_unweighted_acceptor
compact_weighted_string
const
const16
const64
const8
vector

OpenFst.net Update

I’m making good progress in creating a set of .net and Python wrappers for OpenFst.

Following on from my last post I have managed to create a single .net wrapper using the mergebin utility. It took a bit of hacking of the mergebin code and experimentation but I’m glad I’ve it working.

Now I want to take the integration one step further. For example what if the dll consumer wants to then merge it with their exe’s or additional libraries both managed or unmanaged.

Here is a quick C# snippet that will load an fst and iterate the states in outer loop and the arcs in the inner to print out the fst console. I’ve tried to use the the iterator support and foreach statement to make a clean looking syntax.

VectorFst fst = VectorFst.Read(@"t.ofst");

      if (fst != null)

      {

        foreach (var s in fst.States)

        {

          foreach (var a in fst.Arcs(s))

          {

            Console.WriteLine("{0}\t{1}\t{2}\t{3}\t{4}", s, a.nextstate, a.ilabel, a.olabel, a.weight);

          }

        }

     }

DSLs seem to be the current hotness and I’m seeing T4 mentioned a lot. I haven’t looked at T4 properly yet, but I’m thinking about if a text templating system might be good approach for generating parts of wrapper, such as methods common to all FST types or for generating semiring specialized methods.

OpenFst for .net/C#

Recently I have been writing .net wrapper around OpenFst. I want to be able to OpenFst in .net and managed languages and I often prototype in C# because not only is the less typing but the compile are way faster. The heavy templated in OpenFst can sometime lead to long compilations time.

My approach is create C wrapper function then PInvoke the wrapper in C# to expose to. This have the advantage of managed c++ in that I can use the wrapper with other languages and platforms for example Python and Unix. One disadvantage is the OpenFst.net library will be distributed as two dlls, the native library and the managed wrapper. I’ve found two solutions to enable the library to distributed as a single file. 1) embed the un-managed dll as resource and then at run-time pull out the dll to a temp directory and load the library manually as describe here here. 2) Use mergebin a tool that is included in sqlite.

XBox 360 HDMI –> DVI

Recently we got an XBox 360, mostly as media player for my wife. We had a spare 20” widescreen panel and got a HDMI to DVI cable. Reading the Microsoft site it wasn’t clear if the HDMI->DVI wasn restricted to the HTML resolutions such as720p, 1080i etc and the display would have to scale a lower resolution signal.

The good news is the cable can support the native 1680x1050 (1680 1050) resolution of the panel. The below image shows all the resolutions that are offered when connect via HDMI->DVI.

IMG_0690

I knew the monitor wouldn’t support Hi-def playback because of it’s lack off HDCP, this wasn’t a problem because HD-DVD is dead and there isn’t any netflix in Japan. However, no the lame part came when I tried to play the DVD I got “HDCP secure link” error. Now apparently the XBox 360 needs a HDCP compliant display to play up scaled DVDs.

The lameness continues the HDMI socket is right below the standard AV connector and can't both inserted at the same time, I presume so you have to buy XBox HDMI cables, my solution was to just pop the bottom half of the case of the standard xbox connector.

Generalized Callbacks in C++ using TR1

Recently, I had situation where a class (say called B) aggregated a class (say A). Class A would fire messages during a computation and I wanted to be able to subscribe to these messages in class B by passing a pointer to member function . The syntax of pointer to member functions is a little strange and seen often. I also wanted arbitrary global functions or other classes to be valid subscribers.

TR1 provides two useful classes to achieve this std::tr1::function which allows us to create to create a call wrapper . And std::tr1:bind which allows for us to bind to an object.

The below code implements the scenario described in the first paragraph.

#include <iostream>

#include <functional>

#include <vector>

 

using namespace std;

using namespace std::tr1;

 

void foo(int i)

{

  cout << "foo " << i << endl;

}

 

 

class A

public:

  typedef function< void(int) > Delegate;

  void addSubscriber(Delegate func)

  {

    _subscribers.push_back(func);

  }

 

  void action()

  {

    for(vector<Delegate>::iterator it = _subscribers.begin(); it!=_subscribers.end(); ++it)

    {

      (*it)(10);

    }

  }

 

private:

  vector< Delegate > _subscribers;

};

 

 

class B

{

  A  a;

  void callback(int i)

  {

    cout << "B::callback() " << i << endl;

  }

 

public:

  B(int i)

  {

  }

 

  void connect()

  {     

    A::Delegate memfunc = std::tr1::bind(&B::callback,*this,std::tr1::placeholders::_1);

    a.addSubscriber(memfunc);

 

    A::Delegate globalfunc = &foo;

    a.addSubscriber(globalfunc);   

  }

 

  void action()

  {

    a.action();

  }

 

private:

  B()

  {

  }

 

};

 

 

int main()

{

  B b(10);

  b.connect();

  b.action();

  return 0;

}

 

Tested on Window using Visual Studio 2008, GCC should no be a problem (might just need the includes changing).

Windows 7 Sendto and some different shortcuts

I wanted to add notepad2 to the sendto menu so I can quickly view source files and files with unix style newlines.

Access is denied to c:\users\username\sendto

Fortunately this blog post explains how to customize and the shift+sendto feature (which I don’t think I will use). After opening shell:sendto I removed all the defaults, but the sento menu still shows the drives by default.

Two shortcut annoying changes that have come from Vista, are F6 to select the address bar has changed to ALT+D and backspace to move up a directory has changed to ALT+UPARROW.

Some C# snippets I Use a Lot

Quick and dirty code with no error checking

Often I want to count occurrences or have a set like function to list unique labels, for this task I use a simple wrapper around the dictionary.

  class Bag<T> : Dictionary<T, int>

  {

    public void Add(T t)

    {

      if (!this.ContainsKey(t))

      {

        this.Add(t, 1);

      }

      this[t]++;

    }

  }

Splitting on whitespaces whilst removing empty fields is another frequent task,  I include this extension method in my projects.

public static class StringEx

  {

    public static String[] SplitOnWhiteSpace(this String s)

    {

      return s.Split(new String[] { "\t", " " },  StringSplitOptions.RemoveEmptyEntries);

    }

  }

I formatted the code using the CopySourceAsHtml Pluing

HAS_ITERATOR_DEBUGGING Caution

When using Visual studio setting HAS_ITERATOR_DEBUGGING=0 can lead to quite a large speedup.

However, it is essential to set macro value the same for all projects. I had library and executable that were crashing in Release mode only. The library was compiled with HAS_ITERATOR_DEBUGGING=0 whilst the exe was not. The program would immediately crash. Setting HAS_ITERATOR_DEBUGGING=0 on the exe fixed the problem

Using OpenFst on Linux

The new version of OpenFst is awesome (reading the source code is highly recommended). The *nix installer makes it very simple to add use the library in your own code and building new tools in the style of the existing fst tools is also straightforward.

Using OpenFst Library

Download and install openfst-1.1: OpenFst-1.1 is essential as there was small bug in the openfst-1.0 installer

Make sure the libraries can be found: Assuming the default installation as root, the OpenFst libraries will be put in /usr/local/lib. To add these to the LD_LIBRARY_PATH either run the below command from the shell or in my case I add it to my .bash_profile.export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH

Build a simple application:

Lets use the code from the Windows walkthrough describe in this post. Create a file for example named openfstest.cpp and paste in the following code:

#include <fst/vector-fst.h>
using namespace fst;
int main()
{
  typedef StdVectorFst::StateId StateId;
  typedef StdVectorFst::Weight Weight;
  StdVectorFst ofst;
  StateId q = ofst.AddState();
  ofst.SetStart(q);
  for(int i=0; i!=10;++i)
  {
      StateId n = ofst.AddState();
      ofst.AddArc(q, StdArc(i+1, i+1, Weight::One(), n));
      q = n;
  }
  ofst.SetFinal(q,Weight::One());
  ofst.Write("test.ofst");
  return 0;
}


Next compile with the following command:



g++ openfsttest.cpp -o openfsttest -lfst



To use openfst we just need to add a reference to libfst by using the -lfst option.



Test the application:



If everything went correctly g++ should execute without any warnings or errors. Next invoke the application



./openfsttest



There should be a file named test.ofst which contains a simple machine. We can view the machine in text format using



fstprint test.ofst



Which should display the following:



0  1  1  1
1  2  2  2
2  3  3  3
3  4  4  4
4  5  5  5
5  6  6  6
6  7  7  7
7  8  8  8
8  9  9  9
9  10  10  10
10


The machine information can be displayed using


fstinfo test.ofst


Which should give the following:


fst type                                          vector
arc type                                          standard
input symbol table                                none
output symbol table                               none
# of states                                       11
# of arcs                                         10
initial state                                     0
# of final states                                 1
# of input/output epsilons                        0
# of input epsilons                               0
# of output epsilons                              0
# of accessible states                            11
# of coaccessible states                          11
# of connected states                             11
# of strongly conn components                     11
expanded                                          y
mutable                                           y
acceptor                                          y
input deterministic                               y
output deterministic                              y
input/output epsilons                             n
input epsilons                                    n
output epsilons                                   n
input label sorted                                y
output label sorted                               y
weighted                                          n
cyclic                                            n
cyclic at initial state                           n
top sorted                                        y
accessible                                        y
coaccessible                                      y
string                                            y


Speeding up Compile Times with Linux PS3

The PS3 doesn't have a huge amount of memory. Recently, when compiling a template heavy C++ library the memory swapping was causing incredibly slow build times.

I managed to get speed-up by creating a swap on Flash Card, as Root:

/sbin/mkswap /dev/sdc1
/sbin/swapoff -a
/sbin/swapon /dev/sdc1

This might be useful for a quick speedup, however, because there limited read/write cycles on flash memory devices, a better solution would be to setup the toolchain on another machine

OpenFst For Windows Getting Started

Requirements

This version is known to work under 32bit versions of Microsoft Windows XP or Vista. The Compiler requirements are Visual Studio 2008 (SP1) this includes free C++ express edition available here.

Obtaining

From the contrib section of the OpenFst site download the openfstwin.zip file.

Build

Unzip openfstwin.zip in the desired location (for example C:\code ). The files should now be in c:\code\openfstwin\ and from there open the solution file openfstwin.sln in Visual Studio. Select either the Debug or Release build and from the build menu run the Build Solution.

Using The Command Line Tools

The command lines tools and library should now be located in C:\code\openfstwin\Release or C:\code\openfstwin\debug.

 

Creating a Simple Application

Open Visual Studio and create a new C++ Project. From the New Project Dialog select Project type Win32 and template type Win32 Console Application.

newproject

On the next dialog. Uncheck Precompiled header and check Empty Project and Application Type: Console Application.

wizard

Add a new C++ file to the project.

newitem

Add some code, for example just paste the below

#include <fst/vector-fst.h>
using namespace fst;
int main()
{
  typedef StdVectorFst::StateId StateId;
  typedef StdVectorFst::Weight Weight;
  StdVectorFst ofst;
  StateId q = ofst.AddState();   
  ofst.SetStart(q);    
        
  for(int i=0; i!=10;++i)
  {
      StateId n = ofst.AddState();        
      ofst.AddArc(q, StdArc(i+1, i+1, Weight::One(), n));
      q = n;
  }
  ofst.SetFinal(q,Weight::One());  
  ofst.Write("test.ofst");
  return 0;
}


Add include and library paths to the project.



Either From the main menu go to Properties -> ProjectName(openfsttest) in this example. Or right click the project node in the solution explorer.



Go to Configuration Properties -> C/C++ -> General and add an additional include directory to the openfst src include directory. Be sure to set Configuraion -> All Configurations in the top left combo box



includeproj



Next: go to Linker input and add additional dependency to openfstwinlib.lib again set this for all configurations



libname



Finally, Linker->General add Additional Library Directory. Set the correct paths for the Release and Debug Configurations.



libpath 



Now compile either the Release or Debug from the build menu. It everything went smoothly there should be openfsttest.exe in the Release and or Debug directory. After invoking openfsttest.exe a file names test.ofst should be written to the same directly.



Running the fstprint.exe from the Openfst tools should give the following output:



0       1       1       1

1       2       2       2


2       3       3       3


3       4       4       4


4       5       5       5


5       6       6       6


6       7       7       7


7       8       8       8


8       9       9       9


9       10      10      10


10

今日の練習

何時もより寝坊して(僕にしたは)5:30目が覚めて、急いで朝ごはんを食べたり、準備したりで6:00発。

一人で和田峠行ってきました。走者時間4時間半、往復距離は約120KM。(ちょっと短い)

以上

Route Calculation Error

I use a Garmin Edge 705 for training and exploring new rides and climbs.
One of the benefits of the 605/705 devices is they mount as a mass storage device allowing the GPX routes to be manullay copied over to the device without needing any other software. However, when I would try to navigate a manually copied route created with Marengo GPS Route Planner I would get the following error message:

Route Calculation Error: No roads near starting point

Routes transferred via Mapsource would work fine. One difference between the original GPX files and the route transferred viaa Mapsource is each of the route points < rtept > included extra xml child nodes

<extensions>
<gpxx:routepointextension>
<gpxx:subclass>000000000000ffffffffffffffffffffffff</gpxx:subclass>
</gpxx:routepointextension>
</extensions>

After adding the extension nodes to each route point in my Marengo created GPX files I can manually copy the GPX and navigate them without receiving an error message.