What is ETL Testing and How it Works

In this post we define ETL and discuss what does it mean exactly. We will then define ETL testing, the process, how it works, and why it’s important. Moreover, we will discuss the relevance of ETL today, common users and use cases, and pros and cons. Last, we will provide some best practices to be followed for optimal ETL testing.

 

What is ETL?

ETL is a process in data warehousing that is responsible for pulling data out of the source systems and placing it into a data warehouse. This process includes the performance of three tasks: Extracting the data, transforming the data, and then loading the data. Data is extracted from an OLTP database, transformed to match the data warehouse schema and loaded into the data warehouse database.  Many data warehouses also incorporate data from non-OLTP systems such as text files, legacy systems and spreadsheets.

For more info on ETL, check my other blog post regarding this

ETL Testing

ETL testing is done to ensure that the data that has been loaded from a source to the destination after business transformation is accurate. It also involves the verification of data at various middle stages that are being used between source and destination.

 

ETL testing is performed in five stages. First one is identifying the data sources and requirements for the system in use.Second is performing an efficient and timely data acquisition. Followed by implementing business logics and dimensional Modelling. Then, building and populating general and targeted data. Finally, generating and analyzing reports that reflect the performance.

 

ETL testing is about discovering problems in the data stream and correcting them. In the past, this was done in a waterfall approach by identifying problems in the data or the ETL process, building a system to resolve those problems, testing that everything works, and rolling into production. But it is expected that the future of this field will be more of an agile process in which data issues are fixed on the fly, and largely automatically, with no interruption to data ingestion.

 

The general methodology of ETL testing is usually to use SQL scripting, which can time-consuming, error-prone and seldom provide complete test coverage. To accelerate, improve coverage, reduce costs in production and development environments, ETL can be automated completely using various tools.

 

Why is it relevant?

Data Warehouse testing assures that information is not just loaded correctly, but that it is appropriately aggregated, catalogued and verified so that it is useful and accurate for analysis and decision-making. It is worth noting that the ETL process is particularly vulnerable in this regard.

 

An example of its necessity with regards to performance and scalability related issues, a hypothetical successful client who opens new branches in other cities and has not tested the scalability of their data warehouse. As queries to the BI system increase, the system bogs down and the response time slows down. This will cause customers to contact a competitor who is able to handle the increased business.

 

While it seems obvious it is also worth noting that testing should go hand-in-hand with any application changes or new releases. This is regression testing, in fact, automated regression testing is typically performed weekly or even daily, depending upon users’ or the system’s requirements.

 

For more info, check this detailed webpage.

 

Common Challenges

ETL Testing is different from application testing because of the fact that it operates on a data centric testing approach. Many challenges are faced sometimes when dealing with ETL, some of which are: ETL Testing involves comparing large volumes of data, that is usually in the millions of records, the data that needs to be tested is in heterogeneous data sources (eg. databases, flat files), data is often transformed which might require complex SQL queries for comparing the data, ETL testing is very much dependent on the availability of test data with different test scenarios.

 

Best Practices

Many users are capable of increasing the efficiency of their tools notably by simply getting to know how to use it well. First advice always is to research, ask, question and wonder around, there are many resources with different features for different purposes.

 

Achieving proficiency in using ETL Testing can be achieved generally by working on multiple points, the user, for example, should make sure that the data is transformed correctly without any data loss and that the truncation projected data is loaded into the data warehouse. The user should also ensure that the ETL application appropriately rejects and replaces with default values and reports any dysfunctions or errors. You need to also ensure that the data loaded in the data warehouse is within the prescribed and expected time frames to confirm scalability and performance. All the methods should have appropriate unit tests,that should use appropriate coverage techniques to measure their effectiveness. Finally, the user should create unit tests that target outliers, exceptions and unusual cases that might be a sign of a a specific error or malfunction.

 

Closing Thoughts

Building a data warehouse is a major undertaking that’s expected to yield substantial business benefits in order to justify the cost and effort. To ensure that your ETL data warehouse project lives up to expectations, ETL testing must be front and center, particularly in the all-important early stages of the project being worked on.

 

We have seen the definition of ETL and ETL Testing. After that, we saw different reasons that make relevant for lots of users. Later on, we elaborated on some best practices and a checklist to follow when working on an ETL tool, finishing with those closing thoughts.

 

Reference

ETL Testing the Future is Here

https://www.alooma.com/blog/etl-testing-the-future-is-here

ETL Testing or Data Warehouse testing tutorial

https://www.guru99.com/utlimate-guide-etl-datawarehouse-testing.html

Basics of ETL Testing

http://www.datagaps.com/concepts/etl-testing

Why is ETL Testing so Important?

https://www.coherentsolutions.com/blog/why-is-etl-testing-so-important/

Bug Reporting, Error Handling, Error Monitoring – Definition and Best Practices

Overview

With the advancement of coding and its use in everyone’s life, and with the abundance of its presence in the world comes the fact that it is written by humans, mostly at least, at the end of the day. This logically means that error will occur and won’t be foreseen in many cases. Some errors will be detected quickly and some will need intense processing to be spotted or realized.

 

In this article, we will discuss and explain what bug reporting, error handling and error monitoring mean. We will also see some of their use cases and examples of each, before viewing the importance of tracking bugs and errors. Then, we will see the similarities and differences between these concepts and some of the best practices while dealing with them.

 

What do these terms mean?

Bug reporting

Bug reporting is an important aspect of software testing. An effective bug report communicates well with the development team and avoids confusion or miscommunication.Defect writing and reporting is one of the most important areas in testing life cycle and is one of the most neglected areas.

 

Error Handling

Error handling refers to the anticipation, detection, and resolution of programming, application, and communications errors. Specialized programs, called error handlers, are available for some applications. They should forestall errors if possible, recover from them when they occur without terminating the application, or, if all else fails, gracefully terminate an affected application and save the error information to a log file.

 

Error Monitoring

When errors happen in a production environment, they can be detrimental to application performance. Errors could be exceptions not handled by the code in the context of a business transaction, caught exception in a database call, and HTTP 404 errors.

Errors are tracked with data about how many of them occurred in a period of time, the frequency with which they occur , and error rate of the total process.

 

Use Cases & Examples

Bug Reporting can be done by any individual using any sort of platform, and it would always help the developer, when used properly.

This page explains in detail the methods to follow when writing an effective Bug Report.

Error Handling is a very essential stage in every program’s lifecycle, it is handled more on the developers’ level than the direct user, and is managed by powerful software.

Error Monitoring is also a production-level stage, but can extend, and should, to the execution level where the user is interacting with the code. It can be monitored using various methods, one of which is this effective tool by Bugsnag or Rollbar.

 

Why is it important to track bugs and errors, and what can go wrong if you don’t?

As long as humans will be writing codes, there will be error by nature. This is not the problem, as this is a natural process that every program or platform will go under to develop and advance. But problems occur when those errors go unreported and unnoticed. Bugs and errors need to be identified, tracked, monitored, analyzed and fixed at later stages, and the more accurate and useful data a company has, the more control they have over the process of enhancing their product.

 

What are the similarities and differences between them?

An error is a mistake, misconception, or misunderstanding on the part of a software developer. In the category of developer we include software engineers, programmers, analysts, and testers. For example, a developer may misunderstand a design notation, or a programmer might type a variable name incorrectly – leads to an error. It is the one which is generated because of wrong login, loop or due to syntax. Error normally arises in software; it leads to change the functionality of the program.

 

On the other hand, a bug is the result of a coding error. An error found in the development environment before the product is shipped to the customer. A programming error that causes a program to work poorly, produce incorrect results or crash. An error in software or hardware that causes a program to malfunction. Bug is terminology of Tester.

 

For more info, check this informative page.

Best practices for handling bugs and errors

Bug reporting is an important aspect of software testing. A good bug report should be clear and concise without missing key points. Any lack of clarity leads to misunderstanding and slows down the development process. Defect writing and reporting is one of the most important areas in testing life cycle and is one of the most neglected areas. Reports should include all, or as much as possible, of those things:

Bug ID, Bug Title, priority, platform, description, steps to reproduce, expected result (or actual if known), screenshot (if it has a visual effect).

Errors on the other hand should be dealt with the same level of professionalism, as they need to be under supervision and filled with a near live-time into a log that would help the developer identify patterns and future converges, and this can be only achieved when using a powerful tool that would track the errors with the greatest details, supplying the environment with valuable data that would place the coder ahead of the game.

 

Closing Thoughts

We have seen the definition of bug reporting, error handling and error monitoring. After that, we saw different use cases and examples of use for the three of them. Later on, I explained why is it important to perform some of those tasks, and the problematic complications that would occur if not enough attention was paid.

Furthermore, we have seen the differences and the similarities between errors and bugs and how do they differ by definition. We wen then over the importance of following appropriate practices while dealing with bugs and errors, finished with brief closing thoughts.

 

References

How to Write a Good Bug Report? Tips and Tricks

How to Write A Good Bug Report? Tips and Tricks

What is Error Handling

http://searchsoftwarequality.techtarget.com/definition/error-handling

Difference between Defect, Error, Bug, Failure and Fault!

Difference between Defect, Error, Bug, Failure and Fault!

Error Monitoring

https://www.appdynamics.com/opscentral/error-monitoring/

What is AWS EC2?

What is AWS EC2?

Amazon Elastic Compute Cloud (Amazon EC2) is an Amazon platform that provides  secure, resizable compute capacity, all in the cloud. It claims to make web-scale cloud computing easier and more accessible for developers. Its interface is designed for the user to be able to obtain and configure capacity with minimal friction (see the official documentation).

 

This post will be presenting the definition of AWS EC2 with some of its characteristics and reviewing some of its common use cases and tools used to access it. We will then review some concerns and possible security issues. Finally, we’ll discuss some best practices to serve as guidance for working with EC2.

 

Use Cases & Related Tools

This platform of Amazon cloud has lots of uses, with lots of purchase options and API, instances and networking. It offers flexibility and choices in the choice of the processor, memory, storage options, accelerated graphics and performance.

 

Using AWS’s high-performance-computing capabilities, Autodesk can scale the use of generative design to run hundreds of simultaneous simulations with complex parameters. Autodesk develops software for the engineering, design, and entertainment industries.

 

An example of its use can be also seen in this web application use. MediaWiki installed on Apache with 140 pages of content. They received efficient and timely memory, disk, network and CPU stats.

 

Fore more info, check the Amazon’s official description.

 

Security & Performance

Security

Cloud security at AWS was a priority for the company, as their job requires high security standards due to the sensitivity some of the data the platform might be handling. An AWS customer can benefit from a data center and network architecture built to meet the requirements of the most security-sensitive organizations. Amazon EC2 works in conjunction with Amazon VPC to provide security and robust networking functionality for the user’s compute resources.

Reliability

Amazon EC2 offers also a highly reliable environment where replacement instances can be rapidly and predictably commissioned. The service runs within Amazon’s proven network infrastructure and data centers. The company has shown a history of strict and secure systems throughout its path. However, liability should never be assumed to be on the other party, and best practices should always be followed to avoid any obstacles or unexpected errors.

Performance

It provides the user with complete control of the computing resources and allows them to run on Amazon’s computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing the user to quickly scale capacity, both up and down, as the computing requirements change. Amazon EC2 changes the economics of computing by allowing the user to pay only for capacity that they actually use, instead of leaving partially filled spaces of storage on the cloud. It also provides developers the tools to build failure resilient applications and isolate them from common failure scenarios.

 

Best Practices

Amazon EC2 (Elastic Compute Cloud) provides scalable computing capacity in the cloud. When using this service, it is highly recommended to monitor it for intrusion attempts or other unauthorized actions performed against your cloud infrastructure.

 

Trying to leverage traditional, non-cloud native solutions in order to backup AWS resources may be costly and ineffective. Traditional backup software and methods are very centralized by nature, holding disadvantages such as creating single points of failure as well as the high cost of software licenses and required dedicated hardware resources.

 

You can also follow this brief checklist:

Security and Network

Managing and monitoring the access to the AWS resources, APIs and storage platforms using identity federation, IAM users, and IAM roles.

Establishing an automated system for managing the process of distributing the credentials. The user should implement the least permissive rules for their security and privacy.

Regularly patch, update, and secure the operating system and applications on the instance used.

Storage

Understand the implications of the root device type for data persistence, backup, and recovery.

Use the instance store available for your instance to store temporary data.

 

For more info, check the official guidelines.

Closing Thoughts

In conclusion, AWS EC2 simple and practical when it comes to scale of the various and multi-layered options it paves the way for. Using traditional forms of hosting seems too old to use with the existence of this powerful cloud. This tool represents a solution for many companies that need this availability, speed, security, and flexibility in the platform they are using.

 

References

Amazon EC2

https://aws.amazon.com/ec2/?sc_channel=PS&sc_campaign=acquisition_US&sc_publisher=google&sc_medium=ec2_b&sc_content=ec2_e&sc_detail=amazon%20ec2&sc_category=ec2&sc_segment=175055296277&sc_matchtype=e&sc_country=US&s_kwcid=AL!4422!3!175055296277!e!!g!!amazon%20ec2&ef_id=WnPKRgAAAFPlIVHa:20180216220855:s

Choosing the Right EC2 instance and Applicable use case

How to Automate your EC2 instance backup step by step

How to Automate EC2 Instance Backup Step-by-Step

AWS Customer Success

https://aws.amazon.com/solutions/case-studies/

EC2 Use Cases

https://documentation.wazuh.com/3.x/amazon/use-cases/ec2.html

Marketing Tools for Developers

Overview

Sales and Marketing have changed in form and structure so much in the last 100 years, probably the biggest change in its history. Companies are using advanced technologies nowadays by making the process increasingly automated, and focusing more on the customers themselves, and personalizing their purchasing experience as much as possible.

In this article, we will discuss various marketing tools and technologies, for developers. Moreover, we will see why developers need those tools.

Finally, we will go over some best practices to get familiar with marketing tools while avoiding the need to go through the internet’s endlessness. To get started, here is a wiki by Clearbit which maps out various types of marketing tools and includes definitions and resources about each of them.

What are Marketing Tools and Technologies?

There are countless marketing tools in the internet, available for various purposes and different users.

I will present some of those tools, put into categories that are divided based on the purpose of the tool.

Idea Generators: Some of the good tools are Xmind, Scapple, Mindmup and Google Trends.

Ad Networks: iAd for Apple, Chartboost for mobile games, AdColony for Video Ads, are some of the famous tools for Ads.s

Market Research: Google Trends, Google Keywords Tools and Priori Data for Mobile Apps.

InApp Analytics: Google Analytics, Apple App Analytics, and Flurry Analytics (free one).

Push Notifications: Puch.io, Xtify, and Appscend are some of the most used.

And much more, for different purposes and/or more customization.

Do Developers Need Marketing Tools? Why?

Nowadays, the internet is full of information about marketing strategies, methods and tools that can help your company, and most companies have a good toolkit for dealing with that. However, even the best marketing experts in the world need an advice or a warning every while and another. There is an increasing push towards consumer attention. In fact, there are few main fields that modern tools can help you with.

Goals
Most companies that have weak marketing strategies fail to define a clear goal for those strategies. This can stem from many things, like not using enough resources in the plan, or the lack of understanding the true demand of the customers and what do they want from the manufacturer. Moreover, not tracking and monitoring the progress of the campaign, for example, would definitely mean that the company would have less control on its market and product.

Improving Strategies

Marketing tools can help companies not just with the overall big picture, but also dive deep into specific parts of it, to the details. There are many specific priorities for many businesses, conversion optimization for example, that can be tackled using different Tools that can increase the conversion rate with touchable results, by analyzing the existing structure, of a website for example, and make suggestions for improvements accordingly.

 

Ease of Work

After all, a company can decide to use marketing tools fully, and apply a full marketing automation. This would include, analytics, tracking, web content personalization, campaign management, and more.

Generally, companies that use marketing automation to generate leads have higher conversion rates that those who don’t. Because they offer a personalized support that would assist the company in reaching their milestones.

How to get acquainted with Marketing Technologies without getting lost in all of the Information online

There are different answers for this question, especially with regards to the person asking. But generally, people can acquire lots of info and skills from attending courses that touch on various fields within this sphere.

There are several courses you can attend, they would give a detailed content with various useful tips and a professional guidance, the one you should sign for at least!

There are many online courses for marketing for developers, this is a good example of one of them.

On the other hand, scanning the internet for useful sources isn’t always futile. There are many resources that can be used to further one’s knowledge about the topic, acquire guidance for which tools or/and methods to use, and much more.

Two examples of useful resources: The Developer Marketing Guide and this list by apptamin.

Closing Thoughts

After we have seen how important marketing tools are for developers and any successful company, we have seen the reasons that present the need for and the importance of the new Tools available in the stock in today’s Market, as analyzing and responding to customers reviews and opinions is a fundamental process in Sales. I discussed the meaning of the term, its relevance, and importance to people, especially marketing managers, and presented several common platforms for different use cases. Moreover, we saw the way into knowing more about marketing tools by following best practices and using the right resources.

Whether a company is trying to enhance its performance and reach more customers and increase its knowledge of its long lists of past customers, or if it is a startup that is initiating its very first campaign to the Market, both companies need marketing tools to succeed. This is due to the importance of reaching out to new customers and personalizing their experiences.

References

The Developer Marketing Guide

https://www.devmarketingguide.com/

The Top 60+ App Marketing Tools for Developers

https://www.linkedin.com/pulse/top-60-app-marketing-tools-developers-jemmy-patel/

App Marketing Tools for Developers

App Marketing Tools For Developers

Why Companies Need Marketing Tools in 2017