Nerd Blog by Bob Wold

Bing Maps and Google Maps are off in the Bahamas

2011-05-27T10:37:00.000-07:00

I was pretty blown away when I started debugging an issue for one of our users.

He reported that he would mark a way point on the web (using Google Maps), and when it went over to the phone (using Bing Maps), it showed up in the wrong place. Now this is a pretty serious claim since that's pretty much the one thing that we should never mess up.

I started by asking our Q.A. team to confirm the issue. They reported back that they couldn't reproduce the issue (which was some relief), but the user seemed pretty sure so I hit him up for some more specific details. He provided a latitude and longitude, and when I retested with that, I was able to reproduce.

The point was in the Bahamas, so I started trying to brainstorm the things that could be different about the map there:

The Map Projection (Nope)
Maybe there is some funny rule for the area because Cylindrical Mercator projection doesn't work well there? That couldn't be because that projection is the basis of the tile system that everyone uses.

Google:

Google Maps uses a variant of the Mercator projection for its map images

http://en.wikipedia.org/wiki/Mercator_projection

(Google) Maps uses Mercator because it preserves angles

http://www.google.com/support/forum/p/maps/thread?tid=075eb10962e00cc5&hl=en

Bing:

We chose to use the Mercator projection

http://msdn.microsoft.com/en-us/library/bb259689.aspx

The Datum (Nope)
Maybe there is a special datum that people use here? --Maybe. Most of the world uses WGS84/NAD83, and our apps support that plus NAD27, but there are plenty of other datums out there.

Google:

Google Maps are the GPS coordinates based on the WGS 84 datum

http://en.wikipedia.org/wiki/Google_Maps

Bing:

The latitude and longitude are assumed to be on the WGS 84 datum

http://msdn.microsoft.com/en-us/library/bb259689.aspx

The Map Data

This can't be wrong right? It's Bing and Google... No way either of these guys get this wrong... they can't afford to get this stuff wrong... Right?

After looking at it pretty closely, it looks like it's just a pretty simple problem: The Google Maps and Bing Maps are offset from eachother. I don't know who is more right, but they just don't match, and all other things seem to match up, so I'm sold. I think one of them is just plain wrong. ~~I tried a number of other sources, and it looks most likely that BING is wrong, but since I'm not sitting in the Bhamas with a GPS unit, I'm not sure :-)~~

UPDATE: The guys on the bing forums looked into this and found compelling evidence that Bing Maps are right and Google Maps are wrong. The best way to see this is by checking out this map:http://maps.google.com/maps?t=k&ie=UTF8&ll=25.228858,-76.272143&spn=0.00757,0.013937&z=17
Here you can see that the tiles just stop stitching correctly on Google's Maps. When you turn on the Street layer you can see that even the vector data doesn't line up correctly.
Thanks for helping identify the guilty part

25.166766, -76.157151 on Google

25.166766, -76.157151 on Bing

So far this is the only place I've seen bad data from bing, but I'll keep my eye out for it now :-)

Thanks to our LizardLeg for spotting the issue.

Convert.ToInt32(DateTime) is a Lie

2011-05-25T13:02:00.000-07:00

Warning, this is a bit of a rant, I'm just blown away.

I'm building some new features into our C# REST API that all of our phone apps talk to and I wanted to send down a Date in the form of an integer. I noticed that there was a surprising helper method in the Convert class called ToInt32(DateTime). That sounded perfect for what I needed, but I really wan't sure what the format would be since you can't really squeeze a DateTime object into 32 bits. I figured they probably had some standard though, maybe the number of seconds since 1/1/1970 (like Unix/POSIX) or something like that. So I opened up the documentation and low and behold I find:

Convert.ToInt32 Method (DateTime)
.NET Framework 4
Other Versions

Calling this method always throws InvalidCastException.

http://msdn.microsoft.com/en-us/library/ch56d3w9.aspx

Using this doesn't result in a compiler error. It doesn't even result in any warnings at compile time. It just results in a runtime exception if you were naive enough to assume it would do something good. What are we developing in VB?

In my mind, the biggest benefit to strongly typed languages like c# is that they should expose the fact that you did something stupid at compile time so you don't need to wait for it to crash on your when it's released... (OK: there are probably a million different reasons why strongly type languages make sense, but that's one of them ;-) )

Sorry for the Rant-- Back to being productive.

Windows 7: SMTP server running on localhost

2011-05-24T09:33:00.000-07:00

I recently migrated my development machine to Windows 7 64-bit, and I didn't run into too many issues that weren't already on Google :-)

I did hit one that I didn't find a good solution for on Google though: How to install the windows SMTP component.

I figured they must have removed that component, but It turned out that they didn't remove it, they just stopped listing it under IIS in the add/remove windows components section of the control panel.

To add it you just have to (unintuitively) select the following:

Control Panel > Programs > Turn Windows feature On/Off > Internet Information Systems > World Wide Web Services > Application Development features > Turn On ASP.

It's not intuitive, but it works.

You'll now see SMTP in your IIS Manager console:

Nokia 6650 Certificate error.

2009-06-30T10:59:00.001-07:00

One of the guys on the team was doing some work on the Nokia 6650 for AT&T and we kept getting a certificate error when we would try to load the build onto the device. It would say:

Certificate error. Contact the application supplier.

We had signed it with the Certificate required by AT&T, we examined the root certificates on the device, we even swapped out the device and tried another, but the problem still remained. We started pulling out MIDlet-Permissions, nothing.

Finally I made a test application that did nothing and had basically no jad attributes. I signed that app, loaded it, and it worked. I started moving in changes from the real app into the test app, and quickly found that the error was in a place I would never expect. It was in the "MIDlet-Info-URL". If you include this simple jad attribute, it will not load onto the device.

I checked all of the documentation on Nokia's developer site, and I found full documentation for the MIDlet-Info-URL jad attribute, and it seemed like it should be fully supported, but it will result in that certificate error :-/

I thought it was worth a post since it wasted hours of my life, and still doesn't seem to be documented on the internet.

Good Luck!

Building Projects and Solutions without Visual Studio

2008-05-30T11:00:00.001-07:00

In the 2.0 version of the .net framework, Microsoft included a little gem called MSBuild.exe. MSBuild provides a command line interface for compiling entire projects and solutions.

Normally we would maintain NANT build scripts for our projects, and then we would run them on a build server, using CruiseControl .net, as our source control changed. However with this cool tool we can now build the solution directly from the sln file; removing our need to maintain build files.

I did run into a few snags:

J# support

J# is not entirely supported in the 2.0 framework, until you install the j# redistributable (3.7 mb) http://www.microsoft.com/downloads/details.aspx?FamilyId=E9D87F37-2ADC-4C32-95B3-B5E3A21BAB2C&displaylang=en

Web Applications

Web Applications aren't supported in the 2.0 framework. They came as an addition after the initial release of Visual Studio 2005. All of the information about them came with Microsoft® Visual Studio® 2005 Team Suite Service Pack 1. There was really no change required to the framework for these types of projects, it's really only affected Visual Studio, so there is no service pack for the framework.

I tried to download and install the service pack, but it's huge (431 mb) and it won't install:

So when all else fails, start over :)

The error message was:

error MSB4019: The imported project "C:\Program Files\MSBuild\Microsoft\VisualStudio\v8.0\WebApplications\Microsoft.WebApplication.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

so I just copied the \WebApplication\ folder from a PC with Visual studio installed, and added it into to the directory where it was expecting to find it (which I had to create)

Now everything works great, and life is easier than it was before. Every time I commit a change it is auto deployed to the our dev site and life is good.

Bluetooth GPS Getting Smaller

2007-07-18T09:10:00.001-07:00

Check this out! It's a bluetooth GPS receiver (nothing new there) but its so much smaller than I've seen them before. It's roughly the size of a thumb drive, and it alleges to have 8 hours of battery life, which is very impressive indeed.

I did notice that the initial aquisition time seemed a little slower than it's older bigger brothers'. But I suppose that is to be expected. The quality of the GPS output did not seem any worse to the naked eye.

In general, I would probably prefer the bigger one, just because it really doesn't need to be this small, and the bigger ones do seem to get the initial fix faster, which is the most important fix, but I'll continue to play with it and let you know if I change my mind.

Challenges of Street Maps

2007-06-15T10:03:00.000-07:00

So I was talking to my brother Al today, who is one of the smarter people I have met, and he clearly thought I was too harsh on openstreetmap.org.

I really had no intention to sound too negative. I think that they are one the forefront of inovation. They are applying wiki ideas to geospacial data... that is huge. My point was only that the data they collect is very sensitive, and its very important that it all be right, not only geographically, but also in every minuet detail. If a contributer forgets to mention that a street is one way, or that it doesn't have a sidewalk, etc, a program would not be able to properly use the data, and it would get a bum rep.

I think that the consumer of trail data will initially just be humans. And humans are really flexible in their interpretation of the data they are given. Also, trail data is not as complex, nor is it as important to have 100% accurate. If it's a little off, it won't make a big deal.

Thanks for your input Al, the more I have to think about the idea, and the more we talk about it, the more likely I will be to follow through with it.

alwold: so i started looking at open street map
alwold: and i think it's really cool
doctor wold: yeah... it's real cool
doctor wold: did you read the nerd blog I posted?
alwold: yeah
doctor wold: ahh
alwold: you were talking shit about it
doctor wold: cool
alwold: so thats why i looked at it
doctor wold: nah, not shit
doctor wold: it's just that routable data is so hard
doctor wold: like having people contribute it correctly would be incredibly difficult
doctor wold: and trail data isn't so complex
alwold: right thats why wikipedia is so inaccurate
alwold: cause everyone is too dumb
doctor wold: no
doctor wold: words are easy
doctor wold: people know the right information
doctor wold: but its easy to put it in words
doctor wold: it's harder to turn knowledge of roads into data that is accurate enough to actually use in a program
doctor wold: I think that for prited maps, open street map is great
doctor wold: and could be used very soon
doctor wold: but for routable data, it's very sensitive
alwold: so like you're saying that this would be easy to write
alwold: http://en.wikipedia.org/wiki/Antibody
doctor wold: if you know the information then yes
doctor wold: and the information is being used by humans, which are very flexible
doctor wold: with map data it is being consumed by programs
doctor wold: which are not so smart, and flexible with input
doctor wold: anyway... you're opinion is good
doctor wold: and I think that it will be useful at somepoint soon
doctor wold: if they would have started with the tiger data, it would have maybe been useful from the start too
doctor wold: idk

in HTML

2007-06-14T10:06:00.000-07:00

The world is a funny place, full of funny people, who do funny things.

I remember at one point in my programming career, I used to worry more about how to get a task done, given what I knew at that point. Now I worry more about knowing the correct way to complete a task.

Here is a funny example of someone trying to get a task done given what they currently know, instead of figuring out the right way to do it:

samsonasu:
<td valign="top">    
      
      
  </td>
samsonasu: thats such shitty code

Though one " " is often good in HTML, using them to achieve your desired spacing is HORRIBLY GHETTO :-p. If you do this, and want to know a better alternative, try either using a spacer image (a 1x1 transparent gif), with a width and height, or better yet using the padding-left or margin-left css attributes.

Thanks for this one Samson :-p And Happy HTML'n to the rest of you.

OpenTrailMap.org

2007-06-13T19:29:00.000-07:00

In one of my more recent impulses, I purchased a domain, opentrailmap.org/com, with the intent to start a project where people can contribute GIS data, and try to produce a map of the trails around the world.

I'm hoping that I will be able to leverage off of the technologies used by openstreetmap.org, and that it will mostly be a matter of figuring out what to leverage off of, and not so much how to implement it myself.

My theory is that openstreetmap.org, though very cool, is years away from being useful, and even farther away from being better than the data that people pay for. You may ask, "If you think it's so far out, why do the same thing for trails"? There are many compelling reasons:

No Good Alternative. There is no good alternative for trail maps. Topo maps are from the 80's and trails are always shifting because of fires/floods/development/etc.
Passion. Outdoor enthusiasts have passion. They care about the trails, and they are very into it.
Hikers have GPS. Hikers are already equipped with the GPS necessary to collect and store the trail data need, and they are already accustom to using it, in case they get lost.
Looser Requirements. Unlike street maps, there isn't going to be a turn by turn direction application that is trying to find you the best route based off this data. This allows the data to be more sparse, and less detailed (no one way steets, no HOV lane, etc)

These factors make this a project that can start off useful. From day 1 it would be better than what exists today right from. There is no need to collect all the data for it to work either. The idea would be to start local, and spread out. Work with local groups that need the data, and see if it takes off.

In addition to the collection of data, the site would have to have a way to view and print the maps, and I would probably want to partner with a Topo map printer, and allow users to print the trails they want over a high quality topo map too.

As far as the data is concerned, I would not want to hoard this data. I'd be interested in providing it to anyone who wanted it, at least for their own use. I'm not sure yet how I would fund this concept, but one initial idea is to find funding through people like the city of phoenix parks. I know that they make maps, and I wonder how much work it is. Having online maps and having users keeping those maps up to date may be very appealing to them, or may not be :).

As I come up with more on this idea, I will continue to post under the label : opentrailmap.org. Comment if you have any ideas/input.

m500 Crashes Loading Large Images

2007-06-13T15:44:00.000-07:00

the Samsung m500 throws a java.lang.IllegalArgumentException when you try to load a large image using the Image.createImage method.

In the program I'm using, it tries to download an Image that is twice the width, and twice the height of screen. It downloads it through a socket, and tries to load it via a byte array. If you reduce the size to like 1.5 x the width and height, it loads... This issue is killing me because this exception just doesn't make any sense :)

Has anyone else seen this issue?

Image.createImage throws an IOException on Sprint CDMA Phones

2006-10-06T17:38:00.000-07:00

Carrier: Sprint
Network: CDMA
Manufacturer: Sanyo
Programming Language: J2ME

I ran into an issue trying to download jpg data to a J2ME application recently, and I figured it warranted a blog post.

The data the comes over the air is a bunch of meta data about the image, followed by all the bytes that are the contents of the image. When the request comes in I find where the Image starts and try to load it into an Image object. The Image object has a method:

Image.createImage(byte[] buffer, int start, int length)

which is perfect. I figured I could just pass in the whole response with the right start index and length, and it would work perfect. When I tested it, it worked great on IDEN phones, and some sprint phones, but on a number of them, Sanyo phones in particular, it would only work a small fraction of the time. Most of the time it would return a IOException with a null message.

After debugging this for a few hours, I found that the data being sent from the server was being received right by the phone, and that the phone was using the correct start index and length. Then I found that if the same contents were sent to two different phones, one would work and one would fail. I couldn't figure out why it would fail, and it really didn't make any sense.

Fortunately I was able to come up with a workaround that I haven't seen on the Internet yet, hence this blog...

Instead of passing in the whole buffer, with start and length values, I just the image bytes into a new array and pass in 0 for start and the length of the array for the length.

This solved the problem, but blew my mind. :)

Original Code that crashed:

                     Image img = null;
                     try {
                        img = Image.createImage(output, imageStartIdx, length)
                     } catch(Exception e) {
                        throw new Exception("The map returned was not valid.");
                     }

After The Fix:

                     byte[] imageBytes = new byte[length];
                     System.arraycopy(output, imageStartIdx, imageBytes, 0, length);

                     Image img = null;
                     try {
                        img = Image.createImage(imageBytes, 0, imageBytes.length);
                        // img = Image.createImage(output, imageStartIdx, length)
                     } catch(Exception e) {
                        throw new Exception("The map returned was not valid.");
                     }

This is nothing too impressive, but I figured someone might not figure it out on their own, so there it is...

Happy Coding!

.mobi Domain Names

2006-09-28T09:52:00.000-07:00

.mobi has been introduced as the defacto for mobile websites, and though it's pretty intuitive, (mobi->mobile), imagine typing it on your phone. M and O are on the same key, so you would have to type:

6
wait about 5 seconds
6
6
6
2
2
4
4
4

What a pain...

The Nerd community, like me, knows that web content for phones has clasically been refered to as "WAP pages", WAP being Wireless Application Protocol, and though I'm not a huge fan of the term, it's always been particulary convinient, because W-A-P is not only on 3 different keys on a phone, but each letter is the first letter for it's key, so instead of the above key sequence a .wap extension would be entered with the following:
9
2
7

That saves 7 key strokes, and a ton of time... I'm not going to talk to much smack, but it seems like there was not nearly enough thought on this one. I suppose one day WAP browsers will assume that the extension is .mobi if no other extension is specified, so it won't matter anyway, but for all of us with old phones, it will always be a pain. Such is life I suppose.

The Motorola i580 is NOT my phone of choice

2006-08-16T13:52:00.000-07:00

Nextel now sells the i580. It looks nice at first... it's very rugged, and comes equiped with a 1.3 MP camera. After playing with it for a while, I came to the conclussion that the Motorola iden group is heading in the right direction, but still isn’t anywhere near where they should be.

...Major Points:

The Record Store Is Much Faster.

In older iden phones (even ones with higher model numbers) always had a very sluggish record store. Any time you had to write data to the memory store, especially when you store lots of data, it would always run super slow. CDMA phone always had iden phones beat. The i580 is much faster, and is finally competitive on Record Store performace.

The Up stream network IO is HORRIBLE!

I loaded up an app onto the i580 and the i870, and I found that the network upload speeds on the i580 were about 4 times slower, and that’s a generous estimate. The download speeds seem to be competitive (at least with other iden phones). In addition to the slow speeds, you can not usually make it through a large upload without being interrupted by a random socket error, so really we shouldn’t even worry about the fact that it’s slow, instead we should just try to get it to work period.

The i580 does have widen, iden’s high(er) speed data, support… but I haven’t tried it yet, that may help getting around this horrible performance.

There are a lot of new bells and whistles on the i580, but the network IO is enough to make me say that I would advise against buying this phone until it is resolved.

SQL Server Index Fragmentation

2006-07-02T12:28:00.000-07:00

SQL Server, but like most nice tools, it doesn't come without it's short comings. Among the list of short comings is Index Fragmentation. You add an index to a table to make a certain column a good candidate to look up data with. Behind the scenes, this index is really basically just a sorted table with a lot less data than the big table so that you can lookup the record in the sorted table, and then use the key to get record from the big table. This makes things nice and fast... But over time the indexes slowly get fragmented, and selecting data gets slower and slower. Most people think that it gets slow because there is just too much data, but computers are FAST! And they are good at handling lots of data... That's why we made them :) the problem is that these indexes become so fragmented that they are basically useless.

How do you fix it?
Well before you fix it you need to know what's broken. You can use the "DBCC SHOWCONTIG" command (find details on how to use this and all commands at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/tsqlref/ts_tsqlcon_6lyk.asp) this command will tell you all you need to know (and a lot that you don't need to know) about the indexes you ran it on. You can run it across the whole database, a single table, or even a single index. If you are only interested in the fragmentation, you probably want to look at the "Extent Scan Fragmentation" field that it outputs... The closer to 0, the better.

Next, if you determine you have a lot of fragmentation, you have a number of options to resolve the issue. The Naive approach is to drop the index and recreate it, and actually this will give you the best final results, but in the mean time anyone using your DB will be running without an index, and that will cause horrible performance for them. For this reason, SQL Server provides a number of other options.

My favorite is the "DBCC DBREINDEX" you just give this command an index, and it churn for a while and end up giving you a new fast index. There is also a "DBCC INDEXDEFRAG" which keeps the index, but shifts around all the info in it to try to defrag it. I don't use it, but I've heard that it has decent results, and doesn't cause as much data locking as all other solutions. The last option is recreating the index with the DROP_EXISTING clause. I'm no expert but I believe this syntax is supported by most databases, and the DBCC commands are SQL Server specific. But recreating the index, even if you use the DROP_EXISTING clause has most of the same disadvantages of dropping and recreating.

Moral of the story? You have to change the oil in your car from time to time... We don't like to do it, but we have to do it. Indexes are not much different. They require maintenance. :-p

RecordStream issue with LG 535 (Fusic)

2006-06-21T13:17:00.000-07:00

Areas of interest: J2ME, Sprint/Nextel, CDMA, LG Fusic (535), RecordControl.setRecordStream

I’m working with the J2ME for the Fusic, an awesome new CDMA cell phone from LG. I’ve noticed that the AudioRecording doesn’t seem to work right.

When I try to set the output stream of the RecordControl via the RecordControl.setRecordStream() method, it throws an IO Exception (which it’s not documented to throw). The exception is as follows:

java.io.IOException: Could not create recording at given location

It’s almost like its trying to use the RecordLocation instead of the stream. The RecordLocation is a locator string that specifies a URL to save the media to.

I’m only making note of it, I have no solution at this issue. I plan to disable audio recording for now on this phone in the mean time, but since I couldn’t find any mention of this on google, I figured I’d mention it here so you know you’re not the only one seeing this problem. If anyone does happen to solve the problem, feel free to post additional details here. I suspect that you could work around it by using the setRecordLocation and actually recording that way, then going back and pulling it out from that location.

Good Luck.

View-Based Refresh Queries

2006-06-16T15:17:00.000-07:00

Google Earth is a super cool application to dork out on... I've done a ton of work with KML, the Keyhole Markup Language. But no matter how much I do with it, it still continues to blow my mind.

Right now I'm working on a project with google to provide content via View-Based Refresh Queries. These are queries where you provide a url and they post a GET request to you and you feed back KML. Their request contains the corner lat/lon points so you just decide what you think should show up in Google Earth, then generate the KML and return it in your response. The concept is simple, but the actual work involoved is not as trivial as it sounds. When google Earth loads, they are zoomed way out, and basically you can see the whole earth... so what do you return? data for the whole earth? probably not... that would kill your database, and the client would be stuck downloading everything you had. and if you only return a subset of the data, how do you choose what to return? You want to spead it out so that the data that you show is scattered around the world, then when they zoom in, you want to show more places in the zoomed in level. And what kind of data do you show? At a high level, you want a complex collection of data to be represented as a single point probably, but as they get closer you want to show more and more data about the complex thing. All of this logic, which I've only started to dip into here is what takes the time. And with the masses that google can provide, these questions are very important.

My Manager estimated that this project could be done in a matter of days, and *something* can be done in a matter of days, but the solution should be load tested and usability tested, and really, even the server architecture should be reconsidered. I really am worried that a million people will all start hitting our servers all at once, and I don't think we're ready for that :)

So how do I make this situation work?
-first, direct all traffic to some subdomain like googleearth.yourdomain.com and have your main server host that subdomain... this way when the traffic kills you you can issolate it and move it somewhere else, or just turn it off.
-second, write it without strong consideration for the consequences. you can get it done now, then prove it needs more time by load testing it and showing that it is not scallable.
-third, work harder

The saga continues

2006-06-16T09:58:00.000-07:00

I've been keeping a few blogs on a drupal installation on an old solaris box I had around, but I've been trying to change my nerdy habits... it's cool to be a nerd, but why waste all your time doing something that the rest of the world can do better using free stuff like this? for respect? who cares! :)

What is this blog for?

I'll keep another personal blog, this one will be reserved for nerdier things than your average person wants to hear about.

Why would I read this blog?

because your a nerd too.

What kind of stuff will it contain?

This blog will only rant about things that I either work with, or have worked with. I work for one of the leading GPS compaines in the world, working with various trechnologies... I'm always working with KML, GoogleMaps, Geodedic Math, C#, Java, XML, XSL, all the dorky stuff that people like you and me are all about.

Should I trust this Blog?

No. The internet makes you stupid. I'll do exactly what every other nerd does, pretend I am awesome and that I know everything... oddly enough I don't. You can always read, and you can choose to accept or reject whatever you want, but I am not anything special. and itf you blindly trust me, you are a sucker! :-p

What good does Blogging do?

I will not promote this... If I have anything good to say, Googles web crawler will promote it for me, and if it doesn't then I obveously didn't have anything good to say. If you're reading this, then I guess I must have said something worth reading:) weird!

I don't plan to move over my old blogs, but I'll leave them up where they were... If you find them, feel free to indulge