Your steaming cup of coffee is warming your hand. You slowly sit down to start your workday and then suddenly your neighbour’s piercing car alarm goes off, again and again. Your good mood, concentration and the heat of your coffee seem to evaporate.

Why does this throw you off while the sound of classical (input preference) music soothes you? Noise and music are both sound waves. What is the difference?

Music is ordered sound where noise is disordered sound. The same applies to data and is the reason you are likely hearing the terms ETL and ELT with increasing frequency.

Today’s problem is that there is too much data to handle and make sense of – noise. Data with insight is competitive edge and customer satisfaction and to achieve this, data centralisation is needed – A single source of truth or bringing together data to create harmony – music over noise.

Suddenly, the whole business is singing from the same source sheet and that is a competitive edge many companies are lacking.

I sat down with Archana Arakkal, Lead Machine Learning Engineer at Synthesis Software Technologies, to get a deeper understanding of ETL and ELT. Within five minutes, I had a comprehensive crash course and so can you.

Quick summary:

This is a process is where data is extracted from different sources, transformed into usable resources and loaded onto a single system. ETL tools breakdown data silos and make it easier for data scientists to access, analyse data and turn it into business intelligence

Now, let’s get technical. What is ETL?

Loosely defined ETL stands for Extract, Transform and Load.

ETL consists of the following lifecycles before data can be consumed by its end user. This could be a data analyst, data scientist or business owner – basically, anyone who needs to understand the data in a safe and secure manner:

1. The first step “Extract” is the process of aggregating data from its original source. This original source could range from hard copy documents, excel spreadsheets all the way to a database cluster.
2. The second step “Transform” consists of processing the data from its original source into a format that can be consumed for further analysis. This process could entail the standard data cleaning techniques all the way to creating combinatory datasets that will achieve a specific use case with the data at hand.
3. The last step “Load” requires the data to be stored into a database that is in a consumable infrastructure layer – this means the database will adhere to all necessary security standards that will ensure the data is used with correct policies. The transformed/pre-processed data needs to be migrated or moved into this target database.

What is ELT?

The letters in ELT stand for the same as ETL however ELT swaps the “Load” and “Transform.” The reason is that the data is loaded into the target source, and only then will it be transformed.

The purpose behind this approach is that in most to some cases data sets can’t necessarily be standardised with one transformation and often decision makers need to have a look at the raw data sources prior to understanding what transformations are required for their exact dataset.

This allows for more flexibility in loosely defined use cases that needs further exploratory data analyses before the final use case is defined.

When to use one over the other?

A key indicator between and ETL and ELT process is that often in an ELT process the central datastore would be a data lake, reservoir or a data stream to ensure the rawest form of data is transported near real time from the data sources.

Why use them?

ETLs should be used in situations where there is a data source or multiple data sources that need to be centrally manipulated for the final end user to consume.

This would typically be in team dynamics that perhaps have business units that are not interested in managing data and would prefer to consume the data in a clean manner for reporting purposes.

The central data team would be responsible for transporting data of high quality with all the necessary data manipulations prior to the final consumption.

Should every company be preparing to use ETL or ELT?

Yes, they should. ETLs or ELTs are in any situation that requires data analyses, data storage, data migration, data engineering, data strategy, data consolidation etc.

How hard is it to implement?

The complexity would differ based on the organisations existing standards and systems. In the event that no data strategy has been implemented and no central data team exists this could be a lengthy process.

If a team already exists with an adequate data strategy and direction it would depend on the quality of the data as well as the complexity that of data combinations that are required.

What does ETL or ELT solve for?

All data projects that require some form of data analyses requires an ETL process that can further prepare the data.

The key points that an ETL process solves for is having the ability to work with data that will be in a reliable data store or warehouse that unifies data policies and data standards have been adhered to – this further improves the quality of data inferences and data extrapolation by ensuring that data consolidation is uniform across the organisation.

Another way of representing ETL processes is the refinery process that is used across multiple data sources to consolidate all the data into a single centralised location that can be accessible to personal with ease.

While ETL processes may provide a clean overview of data from the very start, ELT processes provide data consumers with the opportunity to consciously decide which parts of their data sets are useful or not.

How do ETLs and ETLs improve data information and data within an organisation?

ETLs improve data information across the organisation with the following means:

• Centralised store that allows for easier access to further consolidate data and preform even more richer analyses of data across multiple sources (with the comfort that all data is of a high standard).
• Data security and data integrity is maintained due to ETL processes cleansing data that will be consumed by end users.
• All ETL processes are automated, this means that time is saved.
• ETL processes further simplify transforming complex data sets by ensuring that data formats conform to standards that are agreed to by the organisation.
• ETL processes ensure a drastic reduction in human error since ETL process can be used as a means to validate data prior to the end user consuming data for analytics purposes.
• Data quality can be attributed to higher standards since automating processes and reducing errors significantly improves the output of data.

Bottom line, what value will this provide?

An example of how a ETL process could improve the overall efficiency of the business is the following:

Suppose there are close to a 1000 different sources that originate from different locations in different formats and each format has multiple permutations of naming conventions for product types that need to be further implemented.

Utilising an ETL process will ensure that the “Transform” will standardise the names for all the product types in an automated manner further saving the data analyst down the line from having to consolidate a 1000 sources manually – the benefit of having a faster turnaround as well as a higher accuracy on having these ETL processes is that decisions can be made at a faster pace to an ever-changing market that requires businesses to act fast.

The post Making Music from Your Data with ETL appeared first on Synthesis.

If you are not measuring the right data, you are likely rewarding the wrong behaviour. Or you are not rewarding the right behaviour leading to poor business outcomes.

This is what Adam Grant refers to as obvious insights. It seems obvious yet over the years I have come across numerous businesses who are making these same mistakes.

Why? Because measuring the right data is not simple.

You can go to the doctor with chest pain, have an ECG and be told that the data shows you have nothing to be concerned about. A concerned doctor sends you home with a 24-hour monitor and stress tests you during an ECG by making you run on a treadmill and suddenly the data tells a different story.

The first set of data was not lying – it was just siloed.

The aim is to reveal the truth and the truth requires a full picture. Siloes or examining information disparately are the enemy of truth, especially customer moments of truth where we feel our businesses are creating excellent customer service, but our customers are actually experiencing something very different.

The antidote

The antidote – ETL. This stands for extract, transform and load – bring data into one place.

This process is where data is extracted from different sources, transformed into a usable resource, and loaded onto a single system.

“ETL tools break down data silos and make it easier for your data scientists to access, analyse data and turn it into business intelligence. It also enables senior management to have access to accurate, real-time information allowing them to make accurate business decisions based on all the data, not siloes, ” explains Deon Schwabsky, Customer Success Manager at Synthesis Software Technologies.

Let’s make this relatable: “Every single company in this world has exorbitant amounts of data, and, in fact, they probably have more data that they can’t actually utilise. But if that data is transformed where the labels across systems mean the same thing (customer name is customer name, all the customers’ addresses are in the same format and phone numbers are in the same format) and then allocated into a single system – suddenly you have business clarity because every system speaks the same language.”

The stories below illustrate how easy it is to make the above-mentioned mistakes and how a single source of truth with ETL prevents my opening statement.

Story 1: Rewarding the minute maker

Over a decade ago, a Customer Service expert gets called into a call centre to make improvements. The company explains that they define success as solving the call and wrapping up the problem or request in a certain time frame. Let’s call it a minute. As a starting point, our expert decides to secretly listen in on some of the calls beginning with the company’s top performer. This person has managed to wrap up every call in a minute.

He jumps on the call eager to learn what makes the “minute maker” tick. As he listens in and the clock rapidly ticks closer to a minute, he realises that the minute maker is about to lose their track record.

The call is nowhere near being wrapped up. Suddenly the line goes dead. This happened time and time again. The “best performer” was being rewarded for putting the phone down in the designated time, not solving the problem and servicing the customer. The call centre was confusing closing the call with call closure.

Applying ETL

What happens to companies that cannot listen to every call to detect this? They need integrated systems.

The systems that measured time of calls could have been integrated with the system that measures repeat calls – the customers would be calling back to complain of a dropped line. A dashboard with consolidated data would indicate that a significant number of customers allocated to the minute maker call back. This consolidated data could then trigger an investigation.

Story 2: Welcome to the park-in

A fast-food drive-in wanted to get customers in and out as quickly as possible – who doesn’t want fast fries?

They decided that employees would be penalised if customers were parked too long and not out of the drive-in in a certain number of minutes. According to the data, this was working beautifully.

However, the reality was different. Picture the poor employee who gets a person rolling up to the window requesting a veggie burger – no lettuce, extra mayonnaise, double cheese, added onion and no tomato as the driver is allergic.

The employee starts to sweat. They are going to get penalised. What do they do?

They get the idea to tell the driver to park their car in the parking lot and they will come when it is ready. The cars keep moving. Problem solved. Employees catch on that complicated customers just need to simply park elsewhere and wait.

Applying ETL

The drive-in was measuring the wrong metric (a siloed metric) – how long a car is parked for.

They could have been measuring how long the car was parked for as well as order time.

The system could have shown that when those two numbers do not correlate – something has gone wrong. Instead, they were rewarding employees for encouraging a park-in and not creating solutions for complex orders.

Story 3: Covid and the overlooked star performers

When Covid hit a business’ call centre was faced with a problem – they could not access enough 3G cards to keep their over 500 agents connected remotely. How would they keep the centre up and running?

They decided to give 80 of the top salespeople 3G cards and laptops. They would have to hold down the fort or rather hold up the business.

What they did not expect is how well these 80 people would do. These 80 people made 80% of the monthly sales.

Suddenly the Pareto Affect or the 80-20 rule was real with 20% of the workforce making 80% of the sales. This allowed the business to see that it was not rewarding its star performers as well as they should and that other employees needed attention to determine why they were starkly underperforming.

Applying ETL

We understand what certain areas of the brain do because unfortunate individuals over the years had damage to those areas or had to have them removed – this illustrated not only their function was but the function of other areas.

This is not the ideal way to find out a function and neither is the above. In this case, star performers were not being rewarded as needed and there was no retention strategy.

Under performers were not being identified, nor the reason for the problem. In this case, if the data was transformed and brought into a single system, there would have been visibility of which salespeople were generating the most signed deals and under performers could have been managed.

Closing thoughts

Customers experience moments of truth every day – an instance where they come into contact with a company and it leaves an impression, often a lasting one.

However, numerous businesses are missing their moments of truth because their data is siloed, leaving them to measure the wrong data, reward the wrong behaviours and miss problems that need solving. That may be the harshest truth of all but it does not need to be the reality.

The post ETL Stories of Siloed Metrics appeared first on Synthesis.

“We’re actually just getting started.” – Adam Selipsky

This year kicked off the 10-year anniversary edition of the AWS re:Invent conference and also marked the return of it being hosted in-person in Las Vegas after a virtual-only event in 2020. Another significant change was the delivery of the first keynote by AWS’s new CEO, Adam Selipsky, replacing Andy Jassy who led the company since its inception in 2003 (Andy Jassy replaced Jeff Bezos as CEO of Amazon in July this year). Cloud technology has been improving at a rapid pace since the first re:Invent conference in 2012 and as Adam so aptly put it in his keynote address: “we’re actually just getting started”.

As per usual, the event was jammed-packed with announcements and major launches across the wide categories of services that AWS offers.

Adam’s began his keynote by reiterating how AWS has maintained leadership in the cloud space for the 11^th consecutive year according to Gartner’s Magic Quadrant reports and acknowledging cloud innovators including Netflix who has been a significant AWS user since its beginning, and NASA who used AWS services to stream the landing of its Curiosity rover mission to Mars. He then went on to talk about Amazon’s virtual server, the EC2 instance, and how it has kept on evolving over the years (they now have 475 instance types!).

This year, the latest generation of AWS’s home-grown processor was announced, the Graviton3 which is reportedly 25% faster on average than the previous Graviton2, as well as being three times faster for general machine learning workloads and uses 60% less energy than the previous generation chip.

This new generation processor can initially be found in the new EC2 C7g instance type and is a significant step forward considering Graviton2’s already impressive price-to-performance ratio.

Another Arm-based CPU announcement was later made by AWS CTO, Dr Werner Wogels. Werner arrived on stage with a grand entrance wearing a t-shirt of UK rock band, The Stanglers, and soon after introduced new EC2 M1 Mac instances which come after the first batch of EC2 Intel-based Mac instances were announced at re:Invent 2020.

An additional exciting announcement was the Mainframe Modernisation service which may just be the push needed for enterprises such as banks and insurers to migrate and modernize their legacy and proprietary technology by automating conversion of mainframe code (goodbye COBOL, hello Java).  This, combined with the announcement of offline tape storage transfer to the cloud using AWS Snowball Edge, means that enterprises have little excuses left to let go of this aging hardware.

Speaking of hardware, in the era of the fourth industrial revolution, providing strong interconnectivity between people and devices is key and AWS is seeking to capitalize this with the announcement of AWS Private 5G which is a new managed pay-as-you-go service that helps enterprises set up and scale their own private 5G mobile networks in their facilities in days instead of months.  It’s still early days, but more and more industries are looking to utilise this kind of next generation network such as Porche within their production facilities to enable wireless robotics communication at high speed and low latency.

Continuing the Industry 4.0 trend, AWS also announced AWS Cloud WAN for building a global office network and AWS IoT TwinMaker which allows developers to create digital twins or virtual representations of real-world systems such as buildings, factories, industrial equipment, and production lines.  These can then be regularly updated with real-world data to mimic the structure, state, and behaviour of the systems they represent.

On the machine learning front, Swami Sivasubramanian, AWS’s vice president of machine learning (ML) announced several new enhancements to the platform. His first announcement was Amazon DevOps Guru for RDS – a new machine learning-based capability for Amazon relational database service (RDS) that can automatically detect and diagnose database performance and operational issues. There were also numerous enhancements to Amazon’s ML platform, SageMaker, to help create datasets, improve training and inference performance, and cost effectiveness.  For training of ML models, AWS introduced new hardware in the form of EC2 Trn1 instances powered by Amazon’s own Tranium chip.

But what if you don’t have any ML or coding experience and want to get started? This year’s re:Invent has you covered with new product announcements to open up development and machine learning to a broader user base such as business analysts with little or no coding required.  These included: Amazon SageMaker Canvas – a drag-and-drop way to generate ML predictions, and Amazon Kenda Experience Builder – a  customisable way to deploy intelligent search applications in a few clicks.  To help people begin their ML learning journey, AWS launched a free machine learning development environment called Amazon SageMaker Studio Lab which provides compute, storage, and security at no cost for anyone to learn and experiment with ML technologies. For general cloud development there’s also the new AWS Amplify Studio which helps developers build cloud-hosted applications in “hours instead of weeks.”

Some other smaller announcements at the event included enhancements to existing services such as improved security and transactions for AWS’s data lake service, Lake Formation;  price reductions for S3 Glacier storage (now known as S3 Glacier Flexible Retrieval), a S3 Intelligent tier that can automatically move data to the new Glacier instant retrieval storage class; and serverless version of data analytics services:  Redshift, MSK, EMR and Kinesis.  Developers can also get excited with new AWS SDKs for Swift, Kotlin and Rust as well as the general release of AWS Cloud Development Kit (AWS CDK) v2.

Sustainability has been a key theme throughout re:Invent 2021, and rightly so since AWS data centres consume huge amounts of energy.  That said, AWS has ambitious goals of achieving net-zero carbon by 2040 and powering operations with 100% renewable energy by 2025. Just earlier this year, they launched a 10-megawatt solar project in the Northern Cape which will supply renewable energy to AWS data centres in South Africa.

At the conference, AWS introduced the Shared Sustainability Model (similar to the security-based Shared Responsibility Model), where customers are also responsible for sustainability in the cloud, for e.g. moving towards serverless services which minimizes energy wastage. To assist in tracking sustainability data, AWS introduced the AWS Customer Carbon Footprint tool to give customers a view of exactly what they are using and also added a new Sustainability Pillar to the AWS Well-Architected Framework.  As Werner put it: “Every resource you are not using is the greenest resource you can think of”.

One bit of exciting news for South Africa is that AWS is launching what they call a Local Zone in what appears to be Johannesburg in 2022.  Local Zones are an extension of an AWS Region that places select services closer to a large population, industry, and IT centres, to lower latency significantly. This comes after the launch of the AWS Cape Town region in April 2020.

Overall, this year’s re:Invent contained plenty of new announcements (with tons more not mentioned here), but also contained a fair amount of nostalgia as to how far we’ve come and it doesn’t seem like the pace is slowing down any time soon.

See AWS’s blog post here for a more complete list of new launches.

The post 10 Years of re:invent appeared first on Synthesis.

The word of the year in 1999 was “Y2K.”  In 2007 it was “subprime.”

And then in 2016 it was “Brexit.” 2017 saw the emergence of “Fake news,” and in 2019 it was “Climate emergency.”

And then the world changed. In 2020 the word of the year was “Pandemic” and this year, 2021, according to the Oxford English dictionary the word of the year is “Vax.”

The Collin English Dictionary took a slightly different angle. Their word of the year was NFT. The abbreviation for non-fungible tokens, or as Collins defines it: “unique digital certificate, registered in a blockchain, that is used to record ownership of an asset such as an artwork or a collectible, shot to prominence as the market for NFT’s soared, aided by high-profile celebrities stirring up the hype.

According to Collins managing director Alex Beecroft “It’s unusual for an abbreviation to experience such a meteoric rise in usage, but the data we have from the Collins Corpus reflects the remarkable ascendancy of the NFT in 2021.”

Other words that made the shortlist are also somewhat unsurprising. “Metaverse,” “Crypto,” whilst “Hybrid Working” and “double-vaxed” also made the shortlist.

Our use of specific words reflects not only what dominates our language but reflects our thoughts and perhaps obsessions. It can also denote our mental and emotional state as well as the desired outputs we would like progress. This is indicated by “pandemic” being last year’s word and then “vax” being the word of the year for 2021.

What is worth noting is that almost all the words of the year concerned one of two things: either the Covid-19 pandemic and technological developments. The world, it would seem, is myopically focused on these two areas.

That said, I believe that there is a missing word in 2021. And that word is “gratitude”. There is so much that we have to be grateful for. With the vaccine, the world is moving back to health, economies are starting to recover and if I look at our company, we have not only “survived” the last few years, but we have thankfully thrived.

Commercially we have done so because of a number of factors that are in our favour.  As a technology company, we are extremely fortunate to be able to continue unimpeded with our work, and we operate in an area experiencing high and growing demand. Our customers have “doubled down” on Cloud, Digital, and innovation in general – and we have been the beneficiaries of this increased activity.

Globally enterprises are acclimatised to and comfortable with remote development. It means a whole range of international opportunities are accessible to South African companies.

With remote working, companies also need to acknowledge the partners and families of their employees. Whereas in the past we might have been grateful to our team and our people, the shift to distanced working has made partners very much part of the team. Managers need to acknowledge and recognise partners and family who have endured an untold number of MS Teams or Zoom meetings and who through no choice of theirs became part of their partners’ work environment. Children, pets and vacuum cleaners as well as Hadadas might also have made countless appearances, but I am not sure that gratitude needs to extend to them.

Our company strategy was to do as much as possible to assist our employees. We arranged weekly Masterclasses with wellness practitioners and industry experts, virtual board game and poker evenings, online team builds and coffees. We ran competitions to keep people engaged and created a Covid Task team responsible for wellness. We kept them informed with a weekly Covid podcast, that landed up reaching many hundreds of thousands of viewers and was nominated for a number of awards. In short, we focused on our team so that they could focus on customers.

We have no idea what the next year will bring. But I am greatly looking forward to 2022. It is time to bid farewell to a year that challenged us and welcome in the new. We use words over and over. The Oxford English Dictionary, Collins and others reflect what we speak about as humans across the globe. My wish to us all is that in 2022 the word of the year will be “fulfillment” as we all strive and reach our potential.

The post Words Of The Year appeared first on Synthesis.

On 5 March 2020, over 140 Synthesis developers, code enthusiasts and solution seekers came together to develop for a cause in the annual Synthesis Hackathon. Ten teams were tasked to create technology to tackle gender-based violence.

The stage was set with just under 24 hours, hundreds of cups of coffee to caffeinate creativity, music, midnight snacks and one exceptional initiative – use tech for good.

These teams had 24 hours to create their tech and present it to a panel of judges, including 1^st for Women representatives, Jolene Chait and Tula Yalpe, ChaiFM’s Kathy Kaler, Space Generation Advisory Council, Ani Vermeulen, My Broadband’s Jan Ani Vermeulen and Synthesis’ Michael Shapiro.

To get as many ideas as possible about the topic, Synthesis hosted a competition with radio station, ChaiFM, calling for concepts to tackle gender-based violence. The winner, Gavin Noik, submitted an idea that tackles domestic (repeated and once-off) violence through an app that is silently triggered with a keyword. The phone then begins recording and sends bursts to contacts and authorities so that as many messages can be sent through with location before the phone is potentially damaged. Gavin joined the hackathon as team lead and helped develop his concept.

Another idea that was created during the hackathon was “AImee” (pronounced AI-me,). This is a voice-activated personal safety location and tracking tool, where users can keep loved ones informed of their physical movements. Where the user veers off route or does not arrive at their planned destination, AImee attempts to establish contact with the user, failing which will send the necessary SOS to the preselected alerts and emergency contacts through integration into existing “panic button” applications.

The winning idea came from a team that created Mamela. This technology provides a safe and convenient way to log a police case from anywhere ensuring privacy and sensitivity to victims. It’s done digitally and guides the users on what information is needed for their case type. If implemented, this would aid in fast reporting and allow reports to be tracked to ensure action is taken and data is collected. It was also designed to provide access to instant emergency services using a panic feature.

“These are the kind of solutions we need, solutions that are technology driven but also take into account South Africa’s context such as high data costs. We hope to see some of these Solutions being made available to the general population in the near future,” says Zandile Mkwanazi, Chairwoman of GirlCode.

The winning technology was an app that empowers individuals in abusive relationships to track evidence, seek assistance and prepare a checklist to safely exit their relationship and lay charges. The use of this technology would allow the person to progress through three distinct stages of the journey. The first allows the person to realise what the signs of abuse are by answering questionnaires and then track evidence of this abuse without divulging their identity. The second stage allows them to reach out to people near them that they can trust or to organisations that are trained to deal with situations like these, also without compromising their identity. This phase is also geared towards helping the person prepare for the last and final stage. Finally, the person allows the system to know who they and the abuser are, and the system assists them in preparing all the evidence to take to the relevant authorities to lay a charge. The system is geared around helping mobilise the support network of people that were prepared in stage 2 in order to get through the very traumatic experience for the person. This idea is centered around the end-to-end journey of the person from realisation (of the abuse), connection (to other people) and finally action (to escape the abuse).

Jolene Chait from 1st for Women Insurance commented on the event, “we were delighted to partake in the Synthesis hackathon. Women abuse in South Africa is a burgeoning and complex issue that requires a multifaceted approach to address the underlying attitudes, beliefs, practices and systems that condone, justify or excuse gender inequality. In just 24-hours, the teams created viable solutions to assist victims and survivors and we were thoroughly impressed by the creativity, collaboration, innovation and enthusiasm of all participants.”

The hackathon was an experience like no other. The Synthesis building was full of energy, enthusiasm and a shared commitment to developing technology to help others that was not diminished, even without sleep. It shows that when people come together to create something positive, so much can be achieved, in the case of the hackathon 10 technologies to make a difference.

The post Deving For Good 2 appeared first on Synthesis.