04.09.07

Valid UTF-8 data (hex:) followed by invalid UTF-8 sequence

Posted in Uncategorized at 5:26 am by ryan

OK, this one is a bit geeked out again, but it’s relevant to China. If you’re an american, you could probably go your entire life without ever bumping into codepages, but if you’re life crosses paths with asia, you almost certainly will…

As we’re developing a new website,doing our subversion (version control system) check-in, I started bumping into a very unusual error.

ryan@116843:/spike/public/news/app/webroot/redv1.0/img/menu$ sudo svn up
svn: Valid UTF-8 data
(hex:)
followed by invalid UTF-8 sequence
(hex: b8 b4 bc fe)

Unfortunately, google didn’t come up with much. The best hit was a Oct 10th post on the subversion users mailing list. Basically, the answer is that there’s no answer.

Well, I did an svn up in each child directory of the one causing the problem and eventually tracked the error down through my project’s directory tree. It looks like one of the guys using a windows system copied a JPEG with a Chinese GBK encoded filename onto the server. Everything is best kept in UTF-8.

Once finding the right file, you have to figure out how to delete a file with a name that can’t be typed…

ryan@116843:/spike/public/news/app/webroot/redv1.0/img/menu$ ls
logo02.jpg       ???? logo.jpg  menu_acc_down.jpg      menu_home_down.jpg  menu_work_down.jpg
logo03.jpg       logo.jpg       menu_acc.jpg           menu_home.jpg       menu_work.jpg
logo04.jpg       logo_top1.jpg  menu_cameras_down.jpg  menu_len_down.jpg
logo05.jpg       logo_top2.jpg  menu_cameras.jpg       menu_len.jpg
logo06.jpg       logo_top3.jpg  menu_gall_down.jpg     menu_tech_down.jpg
logo_bottom.jpg  logo_top.jpg   menu_gall.jpg          menu_tech.jpg

In this case, I just used: rm *\ logo.jpg since there was only one file matching this pattern… Next, I could commit again!

ryan@116843:/spike$ sudo svn up
D public/.htaccess
Updated to revision 38.

4 Comments »

  1. Barry Hunter said,

    May 4, 2007 at 1:24 am

    Thanks, that was just my problem!

    (no other results suggested filename issues :( )

    I wrote a tiny script to enumerate though the directory, outputing the path then running ’svn status’ on each one to find the culprit. (as found no way to get svn to output which folder it was about to try before doing it - so it would show before the ‘helpful’ error message)

  2. Carl-Erik said,

    May 4, 2007 at 2:59 pm

    Thanks! I also ran into this problem, and could not see anyone coming up with a solution. Actually thought the problem laid *within* the files - thus deleting the ones making trouble would fix the problem. Good thing I spotted your blog first :-)

  3. Tom L said,

    May 9, 2007 at 2:27 pm

    Thanks for this post. You saved me quite a bit of time.

  4. Owen said,

    June 29, 2008 at 10:50 am

    Thanks, you saved me a bit of time.
    You can run into problems importing invalid utf too, I was importing a Wordpress sitemap plugin that gave me problems. The last filename before the error was the folder of files that were invalid.

    Hope this helps some one.

Leave a Comment