BSD - Remove non-ascii characters from all files in a directory recursively...

I'm trying to migrate a bunch (300GB+) of files from a FAT32 drive to my freeNas ZFS filesystem but every command I throw at it (tar,pax,mv,cp) throws an 'invalid argument' when it encounters a non-ASCII filename - it's usually something that's been created under Windows and it reads something along the lines of "foo?s bar.mp3..." where the ? may have been an apostrophe or such.

Can anyone help with a few lines of code to recursively go through the directory tree and rename files to remove the offending characters.

Much appreciated.

From serverfault Dan

• Rename can do this..

  try something like

  find dir -depth -exec rename -n 's/[^[:ascii:]]/_/g' {} \; | cat -v
  

  you may need the cat -v to properly display any weird characters without your terminal getting screwed.

  if that prints acceptable substitutions change the -n to -v.

  That said, it sounds like the charset on your filesystem is wrong(mount -o utf8 ?), since this sort of thing should really work...

  Dan : Thank you for the reply-- I have read that I should be able to mount my filesystem as something different but the web seemed to indicate that it doesn't apply to FAT32 partitions? I'd like to be corrected though if this isn't the case? FreeNAS auto-mounts the drive when it starts but I do believe there's the option to override/re-mount etc.

  Dan : Hmm... I don't seem to have the rename command on this box? I've tried man rename with no luck
  From Justin

• Try mounting the filesystem with the iocharset option set to the encoding it uses.

  From man mount under the "Mount options for fat" section:

     iocharset=value
            Character set to use for converting between 8 bit characters and
            16 bit Unicode characters. The default is iso8859-1.  Long file‐
            names are stored on disk in Unicode format.
  

  See also under the "Mount options for vfat" section:

     uni_xlate
            Translate  unhandled  Unicode  characters  to  special   escaped
            sequences.   This lets you backup and restore filenames that are
            created with any Unicode characters. Without this option, a  '?'
            is used when no translation is possible. The escape character is
            ':' because it is otherwise illegal on the vfat filesystem.  The
            escape  sequence  that gets used, where u is the unicode charac‐
            ter, is: ':', (u & 0x3f), ((u>>6) & 0x3f), (u>>12).
  

  and

     utf8   UTF8  is  the  filesystem safe 8-bit encoding of Unicode that is
            used by the console. It can be be  enabled  for  the  filesystem
            with this option or disabled with utf8=0, utf8=no or utf8=false.
            If `uni_xlate' gets set, UTF8 gets disabled.
  

  Edit:

  I'm sorry, that was Linux, this is for BSD (from man mount_msdosfs:

   -L locale
       Specify locale name used for file name conversions for DOS and
       Win'95 names.  By default ISO 8859-1 assumed as local character
       set.
  
   -D DOS_codepage
       Specify the MS-DOS code page (aka IBM/OEM code page) name used
       for file name conversions for DOS names.
  

  Dan : Thanks for the reply, here's what I tried:

  oracle:/mnt# mount -t msdos -o iocharset=utf8 /dev/ad6s1 /mnt/Elements
  

  But this failed with:

  mount: Using "-t msdosfs", since "-t msdos" is deprecated.
  mount_msdosfs: /dev/ad6s1: mount option  is unknown: Invalid argument
                                      

  Dennis Williamson :

   doesn't work in comments, use backticks instead. 
                                      

  Dan : Okay I tried with `mount_msdosfs` but I'm unlear what to specify for the L or D switches as the docs don't go into any specific detail. I figured the drive meant I needed the `large` option and here's what I tried: `mount_msdosfs -o large /dev/ad6s1 /mnt/Elements`. Unfortunately this didn't work either, I'm still getting pax (and others) choking on certain filenames containing 'special' characters.

  Dennis Williamson : What is the origin of the disk? Was it from a Windows system? What language version? Likely values for -L or -D would include CP437 or IBM437, CP1252, ASCII, ISO-646, other ISO-8859-* or variations of those names or others. On my Ubuntu system there's a directory at `/usr/share/i18n/charmaps/` with files of character maps (the filenames and the header text in them) can be informative. See also: http://en.wikipedia.org/wiki/Character_encoding

  Dan : It was a Western Digital "Elements" External USB HDD, of which the external case failed, which I removed it from and popped it into a spare SATA port in my NAS box. OS X showed it as a FAT drive so made the assumption of it being FAT32. I've used it with UK versions of Windows XP and OS X.

  Dennis Williamson : If you look at WD's tech support pages, they have warnings about sharing their drives between OS X and Windows. My opinion is that it's just CYA, but you might look into it. However, are the problematic files exclusively of OS X origin? Does the problem have anything to do with resource forks?

  Dan : CYA? The files are a mix but mainly of Windows origin

  Dennis Williamson : CYA=Cover Their Posterior (approximately)

  Dan : Haha, I'll note that for future use. So back home , I tried your advice: `oracle:/mnt/Elements# mount_msdosfs -L ISO8859-1 -D CP1252 /dev/ad6s1 /mnt/Elements/ mount_msdosfs: ISO8859-1: No such file or directory` So I tried without the 'L' switch thusly: `racle:/mnt/Elements# mount_msdosfs -D CP1252 /dev/ad6s1 /mnt/Elements/mount_msdosfs: cannot find or load "msdosfs_iconv" kernel module mount_msdosfs: msdosfs_iconv: No such file or directory` Bugger! I seem to have stalled on this one again. It does appear that there are some issues with this switch though `http://bit.ly/6KsTGW`
  From Dennis Williamson

• Use convmv to convert the file names if they are really incorrectly encoded. You should prefer mounting the filesystem with the correct encoding in the first place.

  From joschi