Post Syndicated from James Culveyhouse original https://blog.cloudflare.com/internationalizing-the-cloudflare-dashboard/

Cloudflare’s dashboard now supports four new languages (and multiple locales): Spanish (with country-specific locales: Chile, Ecuador, Mexico, Peru, and Spain), Brazilian Portuguese, Korean, and Traditional Chinese. Our customers are global and diverse, so in helping build a better Internet for everyone, it is imperative that we bring our products and services to customers in their native language.

Since last year Cloudflare has been hard at work internationalizing our dashboard. At the end of 2019, we launched our first language other than US English: German. At the end of March 2020, we released three additional languages: French, Japanese, and Simplified Chinese. If you want to start using the dashboard in any of these languages, you can change your language preference in the top right of the Cloudflare dashboard. The preference selected will be saved and used across all sessions.

In this blog post, I want to help those unfamiliar with internationalization and localization to better understand how it works. I also would like to tell the story of how we made internationalizing and localizing our application a standard and repeatable process along with sharing a few tips that may help you as you do the same.

Beginning the journey

The first step in internationalization is externalizing all the strings in your application. In concrete terms this means taking any text that could be read by a user and extracting it from your application code into separate, stand-alone files. This needs to be done for a few reasons:

• It enables translation teams to work on translating these strings without needing to view or change any application code.
• Most translators typically use Translation Management applications which automate aspects of the workflow and provide them with useful utilities (like translation memory, change tracking, and a number of useful parsing and formatting tools). These applications expect standardized text formats (such as json, xml, md, or csv files).

From an engineering perspective, separating application code from translations allows for making changes to strings without re-compiling and/or re-deploying code. In our React based application, externalizing most of our strings boiled down to changing blocks of code like this:

<Button>Cancel</Button>
<Button>Next</Button>

Into this:

<Button><Trans id="signup.cancel" /></Button>
<Button><Trans id="signup.next" /></Button>
 
// And in a separate catalog.json file for en_US:
{
 "signup.cancel": "Cancel",
 "signup.next": "Next",
 // ...many more keys
}

The <Trans> component shown above is the fundamental i18n building block in our application. In this scheme, translated strings are kept in large dictionaries keyed by a translation id. We call these dictionaries “translation catalogs”, and there are a set of translation catalogs for each language that we support.

At runtime, the <Trans> component looks up the translation in the correct catalog for the provided key and then inserts this translation into the page (via the DOM). All of an application’s static text can be externalized with simple transformations like these.

However, when dynamic data needs to be intermixed with static text, the solution becomes a little more complicated. Consider the following seemingly straightforward example which is riddled with i18n landmines:

<span>You've selected { totalSelected } Page Rules.</span>

It may be tempting to externalize this sentence by chopping it up into a few parts, like so:

<span>
 <Trans id="selected.prefix" /> {totalSelected } <Trans id="pageRules" />
</span>
 
// English catalog.json
{
 "selected.prefix": "You've selected",
 "pageRules": "Page Rules",
 // ...
}
 
// Japanese catalog.json
{
 "selected.prefix": "選択しました",
 "pageRules": "ページ ルール",
 // ...
}
 
// German catalog.json
{
 "selected.prefix": "Sie haben ausgewählt",
 "pageRules": "Page Rules",
 // ...
}
 
// Portuguese (Brazil) catalog.json
{
 "selected.prefix": "Você selecionou",
 "pageRules": "Page Rules",
 // ...
}

This gets the job done and may even seem like an elegant solution. After all, both the selected.prefix and pageRules.suffix strings seem like they are destined to be reused. Unfortunately, chopping sentences up and then concatenating translated bits back together like this turns out to be the single largest pitfall when externalizing strings for internationalization.

The problem is that when translated, the various words that make up a sentence can be morphed in different ways based on context (singular vs plural contexts, due to word gender, subject/verb agreement, etc). This varies significantly from language to language, as does word order. For example in English, the sentence “We like them” follows a subject-verb-object order, while other languages might follow subject-object-verb (We them like), verb-subject-object (Like we them), or even other orderings. Because of these nuanced differences between languages, concatenating translated phrases into a sentence will almost always lead to localization errors.

The code example above contains actual translations we got back from our translation teams when we supplied them with “You’ve selected” and “Page Rules” as separate strings. Here’s how this sentence would look when rendered in the different languages:

To compare, we also gave them the sentence as a single string using a placeholder for the variable, and here’s the result:

As you can see, the translations differ for Japanese and German. We’ve got a localization bug on our hands.

So, In order to guarantee that translators will be able to convey the true meaning of your text with fidelity, it’s important to keep each sentence intact as a single externalized string. Our <Trans> component allows for easy injection of values into template strings which allows us to do exactly that:

<span>
  <Trans id="pageRules.selectedForDeletion" values={{ count: totalSelected }} />
</span>

// English catalog.json
{
  "pageRules.selected": "You've selected %{ count } Page Rules.",
  // ...
}

// Japanese catalog.json
{
  "pageRules.selected": "%{ count } 件のページ ルールを選択しました。",
  // ...
}

// German catalog.json
{
  "pageRules.selected": "Sie haben %{ count } Page Rules ausgewählt.",
  // ...
}

// Portuguese(Brazil) catalog.json
{
  "pageRules.selected": "Você selecionou %{ count } Page Rules.",
  // ...
}

This allows translators to have the full context of the sentence, ensuring that all words will be translated with the correct inflection.

You may have noticed another potential issue. What happens in this example when totalSelected is just 1? With the above code, the user would see “You’ve selected 1 Page Rules for deletion”. We need to conditionally pluralize the sentence based on the value of our dynamic data. This turns out to be a fairly common use case, and our <Trans> component handles this automatically via the smart_count feature:

<span>
  <Trans id="pageRules.selectedForDeletion" values={{ smart_count: totalSelected }} />
</span>

// English catalog.json
{
  "pageRules.selected": "You've selected %{ smart_count } Page Rule. |||| You've selected %{ smart_count } Page Rules.",
}

// Japanese catalog.json
{
  "pageRules.selected": "%{ smart_count } 件のページ ルールを選択しました。 |||| %{ smart_count } 件のページ ルールを選択しました。",
}

// German catalog.json
{
  "pageRules.selected": "Sie haben %{ smart_count } Page Rule ausgewählt. |||| Sie haben %{ smart_count } Page Rules ausgewählt.",
}

// Portuguese (Brazil) catalog.json
{
  "pageRules.selected": "Você selecionou %{ smart_count } Page Rule. |||| Você selecionou %{ smart_count } Page Rules.",
}

Here, the singular and plural versions are delimited by ||||. <Trans> will automatically select the right translation to use depending on the value of the passed in totalSelected variable.

Yet another stumbling block occurs when markup is mixed in with a block of text we’d like to externalize as a single string. For example, what if you need some phrase in your sentence to be a link to another page?

<VerificationReminder>
  Don't forget to <Link>verify your email address.</Link>
</VerificationReminder>

To solve for this use case, the <Trans> component allows for arbitrary elements to be injected into placeholders in a translation string, like so:

<VerificationReminder>
  <Trans id="notification.email_verification" Components={[Link]} componentProps={[{ to: '/profile' }]} />
</VerificationReminder>

// catalog.json
{
  "notification.email_verification": "Don't forget to <0>verify your email address.</0>",
  // ...
}

In this example, the <Trans> component will replace placeholder elements (<0>,<1>, etc.) with instances of the component type located at that index in the Components array. It also passes along any data specified in componentProps to that instance. The example above would boil down to the following in React:

// en-US
<VerificationReminder>
  Don't forget to <Link to="/profile">verify your email address.</Link>
</VerificationReminder>

// es-ES
<VerificationReminder>
  No olvide <Link to="/profile">verificar la dirección de correo electrónico.</Link>
</VerificationReminder>

Safety third!

The functionality outlined above was enough for us to externalize our strings. However, it did at times result in bulky, repetitive code that was easy to mess up. A couple of pitfalls quickly became apparent.

The first was that small hardcoded strings were now easier to hide in plain sight, and because they weren’t glaringly obvious to a developer until the rest of the page had been translated, the feedback loop in finding these was often days or weeks. A common solution to surfacing these issues is introducing a pseudolocalization mode into your application during development which will transform all properly internationalized strings by replacing each character with a similar looking unicode character.

For example You've selected 3 Page Rules. might be transformed to Ýôú'Ʋè ƨèℓèçƭèδ 3 Þáϱè Rúℓèƨ.

Another handy feature at your disposal in a pseudolocalization mode is the ability to shrink or lengthen all strings by a fixed amount in order to plan for content width differences. Here’s the same pseudolocalized sentence increased in length by 50%: Ýôú'Ʋè ƨèℓèçƭèδ 3 Þáϱè Rúℓèƨ. ℓôřè₥ ïƥƨú₥ δô. This is useful in helping both engineers as well as designers spot places where content length could potentially be an issue. We first recognized this problem when rolling out support for German, which at times tends to have somewhat longer words than English.

This meant that in a lot of places the text in page elements would overflow, such as in this “Add” button:

There aren’t a lot of easy fixes for these types of problems that don’t compromise the user experience.

For best results, variable content width needs to be baked into the design itself. Since fixing these bugs often means sending it back upstream to request a new design, the process tends to be time consuming. If you haven’t given much thought to content design in general, an internationalization effort can be a good time to start. Having standards and consistency around the copy used for various elements in your app can not only cut down on the number of words that need translating, but also eliminate the need to think through the content length pitfalls of using a novel phrase.

The other pitfall we ran into was that the translation ids — especially long and repetitive ones — are highly susceptible to typos.

Pop quiz, which of these translation keys will break our app: traffic.load_balancing.analytics.filters.origin_health_title or traffic.load_balancing.analytics.filters.origin_heath_title?

Nestled among hundreds of other lines of changes, these are hard to spot in code review. Most apps have a fallback so missing translations don’t result in a page breaking error. As a result a bug like this might go unnoticed entirely if it’s hidden well enough (in say, a help text flyout).

Fortunately, with a growing percentage of our codebase in TypeScript, we were able to leverage the type-checker to give developers feedback as they wrote the code. Here’s an example where our code editor is helpfully showing us a red underline to indicate that the id property is invalid (due to the missing “l”):

Not only did it make the problems more obvious, but it also meant that violations would cause builds to fail, preventing bad code from entering the codebase.

Scaling locale files

In the beginning, you’ll probably start out with one translation file per locale that you support. In addition, the naming scheme you use for your keys can remain somewhat simple. As your app scales, your translation file will grow too large and need to be broken up into separate files. Files that are too large will overwhelm Translation Management applications, or if left unchecked, your code editor. All of our translation strings (not including keys), when lumped together into a single file, is around 50,000 words. For comparison, that’s roughly the same size as a copy of “The Hitchhiker’s Guide to the Galaxy” or “Slaughterhouse Five”.

We break up our translations into a number of “catalog” files roughly corresponding to feature verticals (like Firewall or Cloudflare Workers). This works out well for our developers since it provides a predictable place to find strings, and keeps the line count of a translation catalog down to a manageable length. It also works out well for the outside translation teams since a single feature vertical is a good unit of work for a translator (or small team).

In addition to per-feature catalogs, we have a common catalog file to hold strings that are re-used throughout the application. It allows us to keep ids short ( common.delete vs some_page.some_tab.some_feature.thing.delete ) and lowers the likelihood of duplication since developers habitually check the common catalog before adding new strings.

Libraries

So far we’ve talked at length about our <Trans> component and what it can do. Now, let’s talk about how it’s built.

Perhaps unsurprisingly, we didn’t want to reinvent the wheel and come up with a base i18n library from scratch. Due to prior efforts to internationalize the legacy parts of our application written in Backbone, we were already using Airbnb’s Polyglot library, a “tiny I18n helper library written in JavaScript” which, among other things, “provides a simple solution for interpolation and pluralization, based off of Airbnb’s experience adding I18n functionality to its Backbone.js and Node apps”.

We took a look at a few of the most popular libraries that had been purpose-built for internationalizing React applications, but ultimately decided to stick with Polyglot. We created our <Trans> component to bridge the gap to React. We chose this direction for a few reasons:

• We didn’t want to re-internationalize the legacy code in our application in order to migrate to a new i18n support library.
• We also didn’t want the combined overhead of supporting 2 different i18n schemes for new vs legacy code.
• Writing our own trans component gave us the flexibility to write the interface we wanted. Since Trans is used just about everywhere, we wanted to make sure it was as ergonomic as possible to developers.

If you’re just getting started with i18n in a new React based web-app, react-intl and i18n-next are 2 popular libraries that supply a component similar to <Trans> described above.

The biggest pain point of the <Trans> component as outlined is that strings have to be kept in a separate file from your source code. Switching between multiple files as you author new code or modify existing features is just plain annoying. It’s even more annoying if the translation files are kept far away in the directory structure, as they often need to be.

There are some new i18n libraries such as jslingui that obviate this problem by taking an extraction based approach to handling translation catalogs. In this scheme, you still use a <Trans>component, but you keep your strings in the component itself, not a separate catalog:

<span>
  <Trans>Hmm... We couldn't find any matching websites.</Trans>
</span>

A tool that you run at build time then does the work of finding all of these strings and extracting then into catalogs for you. For example, the above would result in the following generated catalogs:

// locales/en_US.json
{
  "Hmm... We couldn't find any matching websites.": "Hmm... We couldn't find any matching websites.",
}

// locales/de_DE.json
{
  "Hmm... We couldn't find any matching websites.": "Hmm... Wir konnten keine übereinstimmenden Websites finden."
}

The obvious advantage to this approach is that we no longer have separate files! The other advantage is that there’s no longer any need for type checking ids since typos can’t happen anymore.

However, at least for our use case, there were a few downsides.

First, human translators sometimes appreciate the context of the translation keys. It helps with organization, and it gives some clues about the string’s purpose.

And although we no longer have to worry about typos in translation ids, we’re just as susceptible to slight copy mismatches (ex. “Verify your email” vs “Verify your e-mail”). This is almost worse, since in this case it would introduce a near duplication which would be hard to detect. We’d also have to pay for it.

Whichever tech stack you’re working with, there are likely a few i18n libraries that can help you out. Which one to pick is highly dependent on technical constraints of your application and the context of your team’s goals and culture.

Numbers, Dates, and Times

Earlier when we talked about injecting data translated strings, we glossed over a major issue: the data we’re injecting may also need to be formatted to conform to the user’s local customs. This is true for dates, times, numbers, currencies and some other types of data.

Let’s take our simple example from earlier:

<span>You've selected { totalSelected } Page Rules.</span>

Without proper formatting, this will appear correct for small numbers, but as soon as things get into the thousands, localization problems will arise, since the way that digits are grouped and separated with symbols varies by culture. Here’s how three-hundred thousand and three hundredths is formatted in a few different locales:

The way that dates are formatted varies significantly from country to country. If you’ve developed your UI mainly with a US audience in mind, you’re probably displaying dates in a way that will feel foreign and perhaps un-intuitive to users from just about any other place in the world. Among other things, date formatting can vary in terms of separator choice, whether single digits are zero padded, and in the way that the day, month, and year portions are ordered. Here’s how the March 4th of the current year is formatted in a few different locales:

Time format varies significantly as well. Here’s how time is formatted in a few selected locales:

Libraries for Handling Numbers, Dates, and Times

Ensuring the correct format for all these types of data for all supported locales is no easy task. Fortunately, there are a number of mature, battle-tested libraries that can help you out.

When we kicked off our project, we were using the Moment.js library extensively for date and time formatting. This handy library abstracts away the details of formatting dates to different lengths (“Jul 9th 20”, “July 9th 2020”, vs “Thursday”), displaying relative dates (“2 days ago”), amongst many other things. Since almost all of our dates were already being formatted via Moment.js for readability, and since Moment.js already has i18n support for a large number of locales, it meant that we were able to flip a couple of switches and have properly localized dates with very little effort.

There are some strong criticisms of Moment.js (mainly bloat), but ultimately the benefits realized from switching to a lower footprint alternative when compared to the cost it would take to redo every date and time didn’t add up.

Numbers were a very different story. We had, as you might imagine, thousands of raw, unformatted numbers being displayed throughout the dashboard. Hunting them down was a laborious and often manual process.

To handle the actual formatting of numbers, we used the Intl API (the Internationalization library defined by the ECMAScript standard):

var number = 300000.03;
var formatted = number.toLocaleString('hi-IN'); // 3,00,000.03
// This probably works in the browser you're using right now!

Fortunately, browser support for Intl has come quite a long way in recent years, with all modern browsers having full support.

Some modern JavaScript engines like V8 have even moved away from self-hosted JavaScript implementations of these libraries in favor of C++ based builtins, resulting in significant speedup.

Support for older browsers can be somewhat lacking however. Here’s a simple demo site ( source code) that’s built with Cloudflare Workers that shows how dates, times, and numbers are rendered in a hand-full of locales.

Some combinations of old browsers and OS’s will yield less than ideal results. For example, here’s how the same dates and times from above are rendered on Windows 8 with IE 10:

If you need to support older browsers, this can be solved with a polyfill.

Translating

With all strings externalized, and all injected data being carefully formatted to locale specific standards, the bulk of the engineering work is complete. At this point, we can now claim that we’ve internationalized our application, since we’ve adapted it in a way that makes it easy to localize.

Next comes the process of localization where we actually create varying content based on the user’s language and cultural norms.

This is no small feat. Like we mentioned before, the strings in our application added together are the size of a small novel. It takes a significant amount of coordination and human expertise to create a translated copy that both captures the information with fidelity and speaks to the user in a familiar way.

There are many ways to handle the translation work: leveraging multi-lingual staff members, contracting the work out to individual translators, agencies, or even going all in and hiring teams of in-house translators. Whatever the case may be, there needs to be a smooth process for both workflow signalling and moving assets between the translation and development teams.

A healthy i18n program will provide developers with black-box interface with the process — they put new strings in a translation catalog file and commit the change, and without any more effort on their part, the feature code they wrote is available in production for all supported locales a few days later. Similarly, in a well run process translators will remain blissfully unaware of the particulars of the development process and application architecture. They receive files that easily load in their tools and clearly indicate what translation work needs to be done.

So, how does it actually work in practice?

We have a set of automated scripts that can be run on-demand by the localization team to package up a snapshot of our localization catalogs for all supported languages. During this process, a few things happen:

• JSON files are generated from catalog files authored in TypeScript
• If any new catalog files were added in English, placeholder copies are created for all other supported languages.
• Placeholder strings are added for all languages when new strings are added to our base catalog

From there, the translation catalogs are uploaded to the Translation Management system via the UI or automated calls to the API. Before handing it off to translators, the files are pre-processed by comparing each new string against a Translation Memory (a cache of previously translated strings and substrings). If a match is found, the existing translation is used. Not only does this save cost by not re-translating strings, but it improves quality by ensuring that previously reviewed and approved translations are used when possible.

Suppose your locale files end up looking something like this:

{
 "verify.button": "Verify Email",
 "other.verify.button": "Verify Email",
 "verify.proceed.link": "Verify Email to proceed",
 // ...
}

Here, we have strings that are duplicated verbatim, as well as sub-strings that are copied. Translation services are billed by the word — you don’t want to pay for something twice and run the risk of a consistency issue arising. To this end, having a well-maintained Translation Memory will ensure that these strings are taken care of in the pre-translation steps before translators even see the file.

Once the translation job is marked as ready, it can take translation teams anywhere from hours to weeks to complete return translated copies depending on a number of factors such as the size of the job, the availability of translators, and the contract terms. The concerns of this phase could constitute another blog article of similar length: sourcing the right translation team, controlling costs, ensuring quality and consistency, making sure the company’s brand is properly conveyed, etc. Since the focus of this article is largely technical, we’ll gloss over the details here, but make no mistake — getting this part wrong will tank your entire effort, even if you’ve achieved your technical objectives.

After translation teams signal that new files are ready for pickup, the assets are pulled from the server and unpacked into their correct locations in the application code. We then run a suite of automated checks to make sure that all files are valid and free of any formatting issues.

An optional (but highly recommended) step takes place at this stage — in-context review. A team of translation reviewers then look at the translated output in context to make sure everything looks perfect in its finalized state. Having support staff that are both highly proficient with the product and fluent in the target language are especially useful in this effort. Shoutout to all our team members from around the company that have taken the time and effort to do this. To make this possible for outside contractors, we prepare special preview versions of our app that allow them to test with development mode locales enabled.

And there you have it, everything it takes to deliver a localized version of your application to your users all around the world.

Continual Localization

It would be great to stop here, but what we’ve discussed up until this point is the effort required to do it once. As we all know, code changes. New strings will be gradually added, modified, and deleted over the course of ti me as new features are launched and tweaked.

Since translation is a highly human process that often involves effort from people in different corners of the world, there is a lower bound to the timeframe in which turnover is possible. Since our release cadence (daily) is often faster than this turnover rate (2-5 days), it means that developers making changes to features have to make a choice: slow down to match this cadence, or ship slightly ahead of the localization schedule without full coverage.

In order to ensure that features shipping ahead of translations don’t cause application-breaking errors, we fallback to our base locale (en_US) if a string doesn’t exist for the configured language.

Some applications have a slightly different fallback behavior: displaying raw translation keys (perhaps you’ve seen some.funny.dot.delimited.string in an app you’re using). There’s a tradeoff between velocity and correctness here, and we chose to optimize for velocity and minimal overhead. In some apps correctness is important enough to slow down cadence for i18n. In our case it wasn’t.

Finishing Touches

There are a few more things we can do to optimize the user experience in our newly localized application.

First, we want to make sure there isn’t any performance degradation. If our application made the user fetch all of its translated strings before rendering the page, this would surely happen. So, in order to keep everything running smoothly, the translation catalogs are fetched asynchronously and only as the application needs them to render some content on the page. This is easy to accomplish nowadays with the code splitting features available in module bundlers that support dynamic import statements such as Parcel or Webpack.

We also want to eliminate any friction the user might experience with needing to constantly select their desired language when visiting different Cloudflare properties. To this end, we made sure that any language preference a user selects on our marketing site or our support site persists as they navigate to and from our dashboard (all links are in French to belabor the point).

What’s next?

It’s been an exciting journey, and we’ve learned a lot from the process. It’s difficult (perhaps impossible) to call an i18n project truly complete.  Expanding into new languages will surface slippery bugs and expose new challenges. Budget pressure will challenge you to find ways of cutting costs and increasing efficiency. In addition, you will discover ways in which you can enhance the localized experience even more for users.

There’s a long list of things we’d like to improve upon, but here are some of the highlights:

• Collation. String comparison is language sensitive, and as such, the code you’ve written to lexicographically sort lists and tables of data in your app is probably doing the wrong thing for some of your users. This is especially apparent in languages that use logographic writing systems (such as Chinese or Japanese) as opposed to languages that use alphabets (like English or Spanish).
• Support for right-to-left languages like Arabic and Hebrew.
• Localizing API responses is harder than localizing static copy in your user interface, as it takes a coordinated effort between teams. In the age of microservices, finding a solution that works well across the myriad of tech stacks that power each service can be very challenging.
• Localizing maps. We’ll be working on making sure all content in our map-based visualizations is translated.
• Machine translation has come a long way in recent years, but not far enough to churn our translations unsupervised. We would however like to experiment more with using machine translation as a first pass that translation reviewers then edit for correctness and tone.

I hope you have enjoyed this overview of how Cloudflare internationalized and localized our dashboard.  Check out our careers page for more information on full-time positions and internship roles across the globe.