02 August 2020 Reading time: 15 minutes

Lazy translations loading from Angular

Kirill Shushkovskii

Kirill Shushkovskii

FE Developer

ISPSystem

If you have ever been involved in developing a large angular project with localization support, this article is just for you. If not, you may be wondering how we solved the problem of downloading large translation files when the application is started: in our case ~2300 lines and ~200 KB for each language.

Brief context overview

I am a FE Developer in VMmanager team. It is a large frontend project. At present, it is based on angular version 9. Localization support is provided through the ngx-translate library. The translations within the projects are stored in json files. POEditor is used to interact with translators.

ЧWhat is the problem with large translation files?

Firstly, the need to download a large json file at the first use of the application.

Yes, this file can be cached, but the product update is released approximately every 2 weeks, and after each release translation must be downloaded again.

In addition, the project uses an access sharing system, which means that a user with the minimum level of access will only see a small fraction of all translations (it is hard to be precise, but the difference may be a few, maybe dozens of times), and everything else will be a burden for that user.

Secondly, navigation in a large json file is simply inconvenient.

Of course, we do not write code in Notepad. Still, finding a certain key in a certain namespace becomes a difficult task. For example, you need to findTITLE, which is inside HOME(HOME.....TITLE), when the file has another hundred of TITLE, while the object inside HOME also contains a hundred different keys.

What can be done with these problems?

As the name of the article suggests: cut the translations into chunks, spread them into different files and download them as needed.

This concept falls well within the modular architecture of angular. We will indicate in the angular module what chunk of translation we need for it.

We may also need the same chunk of translation in different parts (modules) of the application, so we can put it in a separate file for this purpose. Of course, we can put this chunk in the main file, from where it will be available in the whole application, but that is not our purpose here. Again, this chunk can be a part of the admin interface only and less privileged users will never need it.

Also, some of translations that can be stored separately may be a part of other "separate" translations (for a smaller split).

Based on the listed wishes, we get approximately the following file structure:


/i18n/
  ru.json
  en.json
  HOME/
    ru.json
    en.json
  HOME.COMMON/
    ru.json
    en.json
  ADMIN/
    ru.json
    en.json

Here json files in the root are the main files, they will be downloaded always (the necessary file, depending on the selected language). The files in HOME are the translations only the regular user will need. ADMIN — files required only for the admin. HOME.COMMON — files required only for the admin.

Every json file must have a internal structure that matches its namespace:

  • root files simply contain {...}
  • files inside ADMIN contain { "ADMIN": {...} }
  • files insideHOME.COMMON contain { "HOME": { "COMMON": {...} } }
  • etc.

So far, it may look like my personal quirk, but I will justify this approach below.

We will depart from this structure. You may have a slightly different one, which will change some parts of the code below.

ngx-translate is not capable of all this out-of-the-box, but provides enough functionality to implement it with your own resources:

  • you can process the missing translation — we use this to download the missing required translations on the fly;
  • you can implement your own translation file loader — which we use to download several translation files required by the current module.

Implementation

Translation loader: TranslateLoader

To create your own translation loader, you need to create a class that implements a single method abstract getTranslation(lang: string): Observable<any>. For semantics, it can be inherited from the abstract class TranslateLoader (can imported from ngx-translate), which we will later use for providing.

Since our class will not merely download translations, but will somehow have to merge them into one object, there will be slightly more code than one method:

See code 

export class MyTranslationLoader extends TranslateLoader implements OnDestroy {
  /** Global cache with flags of downloaded translation files (to avoid downloading them repeatedly for different modules) */
  private static TRANSLATES_LOADED: { [lang: string]: { [scope: string]: boolean } } = {};

  /** Here we sort keys by ascending length (small chunks will be merged into bigger ones) */
  private sortedScopes = typeof this.scopes === 'string' ? [this.scopes] : this.scopes.slice().sort((a, b) => a.length - b.length);

  private getURL(lang: string scope: string): string {
    // this string will depend on where and how you put the translation files
    // in our case, they are in the project root directory of i18n
    return `i18n/${scope ? scope + '/' : ''}${lang}.json`;
  }

  /** Here we download the translations and make note of the fact that they have been already downloaded */
  private loadScope(lang: string, scope: string): Observable {
    return this.httpClient.get(this.getURL(lang, scope)).pipe(
      tap(() => {
        if (!MyTranslationLoader.TRANSLATES_LOADED[lang]) {
          MyTranslationLoader.TRANSLATES_LOADED[lang] = {};
        }
        MyTranslationLoader.TRANSLATES_LOADED[lang][scope] = true;
      })
    );
  }

  /** 
   * All downloaded translations need to be merged into a single object 
   * since we know that translation files do not coincide in keys, 
   * then instead of complicated logic of deep merge, we can simply overlay the objects,
   * but it needs to be done in the right order, that is precisely why we have sorted our scope by length,
   * to overlay HOME.COMMON over HOME, and not vice versa.
   */
  private merge(scope: string, source: object, target: object): object {
    // here we process an empty string for the root module
    if (!scope) {
      return { ...target };
    }

    const parts = scope.split('.');
    const scopeKey = parts.pop();
    const result = { ...source };
    // we recursively get the link to the object to which a chunk of translations needs to be added
    const sourceObj = parts.reduce(
      (acc, key) => (acc[key] = typeof acc[key] === 'object' ? { ...acc[key] } : {}),
      result
    );
        // also recursively we pull the required chunk of translations and assign
    sourceObj[scopeKey] = parts.reduce((res, key) => res[key] || {}, target)?.[scopeKey] || {};

    return result;
  }

  constructor(private httpClient: HttpClient, private scopes: string | string[]) {
    super();
  }

  ngOnDestroy(): void {
    // clear the cache so that translations are re-downloaded during hot reload
    MyTranslationLoader.TRANSLATES_LOADED = {};
  }

  getTranslation(lang: string): Observable {
    // we only take the scope, which are not yet downloaded
    const loadScopes = this.sortedScopes.filter(s => !MyTranslationLoader.TRANSLATES_LOADED?.[lang]?.[s]);

    if (!loadScopes.length) {
      return of({});
    }

    // download them all and merge them into one object
    return zip(...loadScopes.map(s => this.loadScope(lang, s))).pipe(
      map(translates => translates.reduce((acc, t, i) => this.merge(loadScopes[i], acc, t), {}))
    );
  }
}





As you can see, scope here is used both as part of the url to download the file and as a key to access the necessary part of json, which is why the directory and structure in the file must match.


The description of how it should be used is provided below.


Missing translation loader: MissingTranslationHandler


To implement this logic, we need to create a class with handle method. The easiest way is to inherit the class from  MissingTranslationHandler, , which can be imported from ngx-translate. The description of the method in the  ngx-translate repository appears as follows:






export declare abstract class MissingTranslationHandler {
  /**
   * A function that handles missing translations.
   *
   * @param params context for resolving a missing translation
   * @returns a value or an observable
   * If it returns a value, then this value is used.
   * If it return an observable, the value returned by this observable will be used (except if the method was "instant").
   * If it doesn't return then the key will be used as a value
   */
  abstract handle(params: MissingTranslationHandlerParams): any;
}





We are interested in the second option: returning Observable to downloading the required chunk of translation




 See code






export class MyMissingTranslationHandler extends MissingTranslationHandler {
  // we cache Observable with the translation, because when entering the page which does not have translations yet
  // each translate pipe will call the handle method
  private translatesLoading: { [lang: string]: Observable } = {};

  handle(params: MissingTranslationHandlerParams) {
    const service = params.translateService;
    const lang = service.currentLang || service.defaultLang;

    if (!this.translatesLoading[lang]) {
      // we request translations loading via loader (the very same, which was implemented above)
      this.translatesLoading[lang] = service.currentLoader.getTranslation(lang).pipe(
        // add translations to the common ngx-translate storage
        // the true flag indicates that the objects need to be merged
        tap(t => service.setTranslation(lang, t, true)),
        map(() => service.translations[lang]),
        shareReplay(1),
        take(1)
      );
    }

    return this.translatesLoading[lang].pipe(
      // we pull the necessary translation by its key and insert parameters in it
      map(t => service.parser.interpolate(service.parser.getValue(t, params.key), params.interpolateParams)),
      // in case of error we emulate standard behavior, in case of no translation - we return the key
      catchError(() => of(params.key))
    );
  }
}





In the project, we always use only string keys (HOME.TITLE), however, ngx-translate also supports keys in the form of an array of strings (['HOME', 'TITLE']). 
If you use these, then you should add a check similar to of(typeof params.key === 'string' ? params.key : params.key.join('.')) to catchError processing.


Using all of the above


To use our classes, you need to specify them when importingTranslateModule:




See code






export function loaderFactory(scopes: string | string[]): (http: HttpClient) => TranslateLoader {
  return (http: HttpClient) => new MyTranslationLoader(http, scopes);
}

// ...

// app.module.ts
TranslateModule.forRoot({
  useDefaultLang: false,
  loader: {
    provide: TranslateLoader,
    useFactory: loaderFactory(''),
    deps: [HttpClient],
  },
})

// home.module.ts
TranslateModule.forChild({
  useDefaultLang: false,
  extend: true,
  loader: {
    provide: TranslateLoader,
    useFactory: loaderFactory(['HOME', 'HOME.COMMON']),
    deps: [HttpClient],
  },
  missingTranslationHandler: {
    provide: MissingTranslationHandler,
    useClass: MyMissingTranslationHandler,
  },
})

// admin.module.ts
TranslateModule.forChild({
  useDefaultLang: false,
  extend: true,
  loader: {
    provide: TranslateLoader,
    useFactory: loaderFactory(['ADMIN', 'HOME.COMMON']),
    deps: [HttpClient],
  },
  missingTranslationHandler: {/*...*/},
})









Flag useDefaultLang: falseis required for correct operation of missingTranslationHandler.


Flag extend: true(was added in version ngx-translate@12.0.0) is necessary to enable subsidiary modules to work with parent module translations.


In addition, to get rid of copying such imports for each module with separate translations, you can write a couple of utilities:




 See code






export function translateConfig(scopes: string | string[]): TranslateModuleConfig {
  return {
    useDefaultLang: false,
    loader: {
      provide: TranslateLoader,
      useFactory: httpLoaderFactory(scopes),
      deps: [HttpClient],
    },
  };
}

@NgModule()
export class MyTranslateModule {
  static forRoot(scopes: string | string[] = [], config?: TranslateModuleConfig): ModuleWithProviders {
    return TranslateModule.forRoot({
      ...translateConfig([''].concat(scopes)),
      ...config,
    });
  }

  static forChild(scopes: string | string[], config?: TranslateModuleConfig): ModuleWithProviders {
    return TranslateModule.forChild({
      ...translateConfig(scopes),
      extend: true,
      missingTranslationHandler: {
        provide: MissingTranslationHandler,
        useClass: MyMissingTranslationHandler,
      },
      ...config,
    });
  }
}









Such imports should only exist in the root modules of the individual parts of the application. Then (to use the translate pipe or directive) you just need to import TranslateModule.


At the moment (with version ngx-translate@12.1.2) you can notice that when you switch the language while the translation is being downloaded thetranslate pipe outputs  [object Object]. This is an internal pipe error.


POEditor


As I mentioned above, we upload our translations to POEditor so that a translator can work with them there. For this purpose, the service has an appropriate API:



  



Both of these handlers work with complete translation files, but in our case, all translations are stored in different files. It means that before uploading, all translations need to be merged into a single file, but when downloading, they all need to be distributed into different files.


We have implemented this logic in a python3 script.
In general terms, it uses the same principle of translations consolidation as MyTranslateLoader. Separation follows the same scheme, only it extracts chunks from the large file.


Several commands are implemented in the script:



  
    
    
  • split — accepts the file and the directory where you have the translation structure prepared for your translations and places the translations according to this structure (in our example, it is the directory i18n);
    • 
    
  • join — performs the reverse action: accepts the path to the translation directory and places the joined json either in stdout or in the specified file;
    • 
    
  • download  — downloads translations from POEditor, then either places them into files in the transferred directory, or places them into one file transferred into arguments;
    • 
    
  • upload — respectively uploads translations to POEditor either from the transferred directory or from the transferred file;
    • 
    
  • hash — calculates the md5 sum of all translations from the transferred directory. This is useful if you are add hash into the options for downloading translations so that they are not cached in the browser when you change them.
    • 
  




In addition, the argparse package is used there, which makes working with arguments convenient and generates the --help command.


The source script will not be considered within this article because it is quite large, but you can find it in the repository following the link below.


The repository also contains a demo project, where everything described in the article has been implemented. Also, this project has been uploaded on the stackblitz platform, where you can check everything out for yourself.


GitHub repository

Stackblitz demo


What we have come to


This approach is currently already used in VMmanager. Of course, we have not split all our translations at once, because there are quite a few. We gradually separate them from the main file, and try to implement the new functionality with the separation of translations already in place.


At the moment, lazy translation upload has been implemented in the panel user management, panel settings, and recently implemented backups.


What about you? How do you solve the problem of large localization files? Or why have you decided not to seek a solution?