Samuel Glauser
ERNI Switzerland

According to The Open Group Architecture Framework (TOGAF), data architecture describes the structure of an organisation’s logical and physical data assets and data management resources.

Thus, data architecture defines how data is acquired, stored, processed, distributed and consumed. It lays the groundwork for an environment where data scientists, data engineers and data analysts operate, making decisions around data architecture inherently strategic.

This is because successful data architectures need to deal with a company’s present challenges as well as accommodate future developments. For example, which data is stored now and in a couple of years, and what is the best way to ensure persistence for which data? To what extent should consistency be enforced across data sets & projects? What about data governance: who is allowed access to what data?

Answering these questions is all about balancing trade-offs, such as:

 

ETL vs. ELT:

  • Transform and then Load decreases the number of times data must be moved to reduce cost, increase data freshness and optimise enterprise agility
  • Load and then Transform keeps raw data (e.g. JSON files) close, so the extraction process can be adapted easily if needed – requires more space, but offers more agility

 

Autonomy in Projects vs. Common Vocabulary Across Projects:

  • Data modelling should happen autonomously within projects as their requirements are known, giving users maximum flexibility to develop tailor-made solutions, but risking data sets that are no longer consistent across the organisation
  • The same entities should have consistent naming schemes across data sets & projects, making it easier to navigate the data architecture, but providing less flexibility since any development needs to be coordinated with other projects

 

Easy Data Access vs. Data Governance:

  • Data scientists need easy access to data sets for the purpose of exploratory data analysis, testing & experimentation, but the data produced by an ETL process may have undergone too much processing to be useful
  • Each Transformation step makes it easier for the data to comply with data governance demands by tightly controlling which information is present in the data at which stage and to whom, making e.g. sure that there is no personally identifiable information (PII) in the data

Using a microservices architecture is one way of addressing the trade-offs mentioned above and allows for more customised configuration possibilities. This architecture type structures a process as a collection of services that are maintainable, loosely coupled and independently deployable.

When processing data, microservices can be configured more flexibly to first store the data after import from the respective data source and then transform it in another microservice for the specific use case such as a machine learning model in production.

Thus, microservices perform fine-grained, lightweight & clearly defined tasks that can, e.g. provide the raw data for data scientists but with PII removed to comply with data governance standards.

 

Do you want to know more about data and how your company can generate more digital value? Our sales team will happily answer all your questions; feel free to contact Viktoria Mauz.

News from ERNI

In our newsroom, you find all our articles, blogs and series entries in one place.

  • 27.09.2023.
    Newsroom

    Unveiling the power of data: Part III – Navigating challenges and harnessing insights in data-driven projects

    Transforming an idea into a successful machine learning (ML)-based product involves navigating various challenges. In this final part of our series, we delve into two crucial aspects: ensuring 24/7 operation of the product and prioritising user experience (UX).

  • 13.09.2023.
    Newsroom

    Exploring Language Models: An overview of LLMs and their practical implementation

    Generative AI models have recently amazed with unprecedented outputs, such as hyper-realistic images, diverse music, coherent texts, and synthetic videos, sparking excitement. Despite this progress, addressing ethical and societal concerns is crucial for responsible and beneficial utilization, guarding against issues like misinformation and manipulation in this AI-powered creative era.

  • 01.09.2023.
    Newsroom

    Peter Zuber becomes the new Managing Director of ERNI Switzerland

    ERNI is setting an agenda for growth and innovation with the appointment of Peter Zuber as Managing Director of the Swiss business unit. With his previous experience and expertise, he will further expand the positioning of ERNI Switzerland, as a leading consulting firm for software development and digital innovation.

  • data230.08.2023.
    Newsroom

    Unveiling the power of data: Part II – Navigating challenges and harnessing insights in data-driven projects

    The second article from the series on data-driven projects, explores common challenges that arise during their execution. To illustrate these concepts, we will focus on one of ERNI’s latest project called GeoML. This second article focuses on the second part of the GeoML project: Idea2Proof.

  • 16.08.2023.
    Newsroom

    Unveiling the power of data: Part I – Navigating challenges and harnessing insights in data-driven projects

    In this series of articles (three in total), we look at data-driven projects and explore seven common challenges that arise during their execution. To illustrate these concepts, we will focus on one of ERNI’s latest project – GeoML, dealing with the development of a machine learning algorithm capable of assessing road accident risks more accurately than an individual relying solely on their years of personal experience as a road user, despite limited resources and data availability.

     

  • 09.08.2023.
    Newsroom

    Collaborative robots revolutionising the future of work

    The future of work involves collaboration between robots and humans. After many years of integrating technology into work dynamics, the arrival of collaborative robots, or cobots, is a reality, boosting not only safety in the workplace but also productivity and efficiency in companies.

  • 19.07.2023.
    Newsroom

    When the lid doesn’t fit the container: User Experience Design as risk minimisation

    Struggling with a difficult software application is like forcing a lid onto a poorly fitting container. This article explores the significance of user experience (UX) in software development. Discover how prioritising UX improves efficiency and customer satisfaction and reduces risks and costs. Join us as we uncover the key to successful software applications through user-centric design.

  • 21.06.2023.
    Newsroom

    How does application security impact your business?

    With the rise of cyber threats and the growing dependence on technology, businesses must recognize the significance of application security as a fundamental pillar for protecting sensitive information and preserving operational resilience.