Skip to main content

Encoding for Atlas' schema files

Question​

What encoding is used by Atlas for schema files? How do you avoid encoding issues with PowerShell on Windows?

Answer​

Atlas uses UTF-8 to encode schema files and migration files generated by Atlas. UTF-8 is the default encoding on nearly all system terminals, but on Windows, PowerShell uses UTF-16 by default. This can cause an encoding issue when Atlas loads the schema files generated by the inspect command.

For example, running the following command in PowerShell will create the schema.sql file with UTF-16 encoding:

atlas schema inspect -u <URL> > schema.sql

When loading this file for another command, like atlas migrate apply, this will lead to an error because Atlas expects UTF-8 encoding:

Error: schema.sql:1: Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1

The same problem happens with the external_schema datasource:

data "external_schema" "gorm" {
  program = [
    "go",
    "run",
    "-mod=mod",
    "ariga.io/atlas-provider-gorm",
    "load",
    "--path", "./path/to/models",
    "--dialect", "postgres", // | mysql | sqlite | sqlserver
  ]
}

You will get the following error because the output from program is in UTF-16 encoding:

Error: data.external_schema.gorm: running program C:\\Program Files\\Go\\bin\\go.exe: 13:49: illegal character U+0073 's' in escape sequence (and 6 more errors)
exit status 1

To prevent these errors, we recommend changing the default encoding of PowerShell on Windows to UTF-8 by creating a default profile.

Create or edit your PowerShell profile to set the default encoding to UTF-8. For most users, this is done at the $PROFILE location (typically $HOME\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1). This does not require administrator privileges and only affects your user account.

If you want to set the encoding for all users (requires administrator privileges), you can create or edit the profile at the $PSHOME location (typically found at C:\Windows\System32\WindowsPowerShell\v1.0). See Microsoft's documentation for more details.

Add the following content to your profile file:

$PSDefaultParameterValues['Out-File:Encoding'] = 'utf8'
$PSDefaultParameterValues['*:Encoding'] = 'utf8'

After creating the profile, restart all Terminal apps by closing and reopening them. Then, run $PSDefaultParameterValues['Out-File:Encoding'] to confirm that the default encoding has been updated:

$ $PSDefaultParameterValues['Out-File:Encoding']
utf8
$ $PSDefaultParameterValues['*:Encoding']
utf8