The post Using Confluent Cloud and Kafka tooling on your laptop for solution prototyping and development. appeared first on Synthesis.

A Tale of Efficiency and Limits

Confluent Cloud, positioned as a managed solution, has garnered attention for its ability to simplify even the most complex data streaming tasks. Our foray into this realm led us to a series of revelations – ones that illuminated the path ahead while occasionally presenting us with formidable challenges.

Bandwidth Boundaries:

KSQL’s pull queries hold great promise in facilitating seamless data retrieval. Yet, we found ourselves ensnared in bandwidth limits that acted as gatekeepers to data accessibility. Confluent implemented an hourly bandwidth on all pull queries, once this limit is reached all queries being executed would timeout. This caused all our applications to timeout when retrieving data. Confluent’s response was to raise these limits, however even with the raised limits we still ran into them often. The monitoring capabilities also limited us from pinpointing the application and query that was eating up all the bandwidth. The solution to this problem was to host KSQL locally and continue to use Confluent Cloud to host Kafka. With KSQL running locally the bandwidth issues became a thing of the past.

Persistence Predicaments:

Within the logistics realm, the ability to perform complex queries in a timely manner is paramount. However, Confluent Cloud’s limit of 20 persistence queries proved restrictive for our purposes. Collaborating with a client unveiled that this threshold barely scratched the surface of their requirements. This compelled us to seek a resolution by liaising with Confluent once more, driving a collaborative process of tweaking the architecture to align with these ever-evolving needs. This firsthand encounter highlighted the dynamic nature of the logistics industry and the corresponding need for tools that can seamlessly adapt.

Navigating Complex Lookup Landscapes

In an era where logistics have transcended mere transportation to encompass intricate supply chain ecosystems, the ability to instantaneously access relevant data has emerged as a cornerstone of success. While Kafka and KSQL provide immense value, our journey taught us that the road gets rougher when complex search scenarios come into play.

Complex Lookups, Diminishing Returns:

KSQL’s knack for key-value lookups shines brightly, but we encountered rough seas when dealing with intricate search patterns. Our logistics use case demanded not just key-based queries, but also complex WHERE clauses to sift through the data for precise information. However, as the data corpus expanded, performance dwindled, rendering results increasingly impractical. This prompted a pivot towards MongoDB, a database solution that allowed the introduction of indexes, thereby reviving the performance and offering a practical solution to complex queries – something KSQL’s architecture currently lacks.

Character Constraints:

Logistics often entails grappling with massive and intricate datasets. Our encounter with KSQL queries presented a challenge – a limitation of 2000 characters. This restriction forced us to adapt, dissecting larger queries into smaller components and subsequently resorting to joins. While this workaround kept us afloat, it also resulted in a heightened consumption of persistence queries. This experience reemphasized the necessity for query adaptability, particularly in scenarios where data is voluminous and intricate.

Weaving Success through Insight

The trajectory we charted within the realm of Confluent Cloud and KSQL speaks volumes about their potential, while also unearthing their limitations within the dynamic logistics industry. The architectures that were initially put forward were tweaked constantly to deliver the best most efficient solution to our client. The addition of MongoDB and moving KSQL to a locally hosted solution allowed us to still use KSQL and get its benefits. The addition of MongoDB and linking it to Kafka via Kafka connectors also enabled us to index and perform complex queries thus getting the best of both worlds.

Charting the Course Ahead

In an industry where time is a currency, and efficiency holds the key to maintaining the supply chain’s delicate balance, the convergence of Confluent and Kafka offers promise. At Synthesis, our journey through the intricacies of Confluent Cloud and KSQL has laid bare their strengths and limitations, underscoring the need for adaptable solutions that accommodate the diverse and complex logistics landscape. As we reflect on our experiences, we recognize the boundless potential for Confluent and Kafka to reshape the logistics industry. They provide innovative solutions to the multifaceted challenges that define our era.

About Aleshan Maistry, Digital Principal at Synthesis.  

Ends

For more information on the innovative work Synthesis has done for its clients, contact us on 087 654 3300

About Synthesis

Synthesis uses innovative technology solutions to provide businesses with a competitive edge today. Synthesis focuses on banking and financial institutions, retail, healthcare and telecommunications sectors in South Africa and other emerging markets.

In 2017 Capital Appreciation Limited, a JSE-listed Fintech company, acquired 100 percent of Synthesis. Following the acquisition, Synthesis remains an independent operating entity within the Capital Appreciation Group providing Cloud, Digital, Payments and RegTech services as well as corporate learning solutions through the Synthesis Academy.

The post Exploring the Realities of Confluent Cloud and KSQL in Transforming the Logistics Landscape appeared first on Synthesis.

To explore the value of Data Streaming, we took a deep dive into some of South Africa’s Top Financial Services, Insurance, and Retail companies. Earlier this month, Synthesis hosted Confluent Round Tables in both Johannesburg and Cape Town, moderated by Colin Iles. We invited our clients; Standard Bank, ABSA, FNB, Osiris Trading, Discovery Vitality, Old Mutual, RMB, Momentum Metropolitan, Sanlam, and Woolworths, where we had an engaging discussion on data journeys, use cases, and challenges.

A common theme across the companies is at the use case level. Understanding how Data streaming can really add value to business processes and benefit the company as a whole.

Some of the Use Cases for Confluent Data Streaming that were identified include the following:

• Instant Gratification

Discovery, Vitality dominates as the most mature platform using Confluent Cloud. The company needed speed of data and insight into individual experiences. Confluent Data Streaming enabled instant gratification where the data gathered from incentives and insights are used for decision-making.

“Individuals receive points for a smoothie while they are on a run – instant reward, instant gratification,” says a representative from Discovery, Vitality.

• Fraud Prevention

At Standard Bank, Confluent Data Streaming is used for fraud detection and prevention purposes. It enables real-time prompts which assists the bank in detecting fraud as it takes place.

“The bank was looking for ways to detect fraud as quickly as it happens, and Confluent Data Streaming assisted in solving the challenge,” says a representative from Standard Bank.

Choosing how to interact with consumers reduces anxiety and allows you to remodel your business and software.  At Sanlam, data streaming is used to leverage insights by conducting background checks, such as confirming someone’s ID and using data streaming to leverage the insights from that data.

“By being an effective bouncer, you are able to get as many logs as possible to shine a light on fraud. Solve real-time fraud. To, not only, detect but prevent” says Darren Bak, Synthesis Head of Intelligent Data.

• Call Centre Events

Customers will believe you when you tell them, “Your call is important to us”. The reality is that an average call center receives thousands of calls per month and, companies don’t have visibility from the start of the call to the end. Using event streaming, you can make it a reusable system where you gain insight at every step of the process. 

“With Kafka, we are about to tap into event statuses of claims in an call center environment. Kafka enables you to tap into an event status and respond quickly” says a representative from Discovery, Vitality.

• The “Netflix” experience

Confluent Data Streaming can be used for many use cases.

A typical scenario:

Jack downloads an app but doesn’t use it.

Just from that alone, we know who Jack is, his GPS location, and other insightful information in real time. It allows for personalised messaging to Jack that can generate potential revenue just from the insights of data generated.  

“Tracking events and users of an event using analytical data to build persistent storage in real-time. Rapid transactions are the value,” says a representative at Old Mutual.

Can you use Confluent to improve existing infrastructure and enable rules in the data stream?

Confluent Data Streaming enables rules along the data stream. Mainframe events become real-time, and companies can tap into that for various use cases. In terms of legacy, learning from the system and understanding how the mainframe operates allows you to summarise and add rules to data processes.  Platforms need to be modernised and Confluent Data Streaming enables the idea of having a central hub that collects data as a Data App.

“Data is a new currency. Upload data on the data app for permanent storage and find analytics from data. There is no need to build multiple data streaming platforms” says a representative from FNB.

Synthesis has been a Confluent partner since 2018, with advanced experience and skills. Confluent Data Streaming is not an emerging technology and has been implemented successfully globally with a number of success stories. It’s all about business value and data quality. Data Streaming is a reactive application that can future-proof your organisation.

“We need to start thinking of data as ongoing value and data management = data quality. Data is never 100% accurate, therefore data quality and foundational data management practices need to be applied. In order for us to say that data is a currency, we need to make sure we value it,” says Darren Bak.

How can Confluent Data Streaming be used for Data Quality?

“Remember what they say, “garbage in, garbage out”. Data quality is crucial and good data quality will give you great insights,” says a representative from Standard Bank.

Confluent Cloud has stream governance and design principles to improve data quality. By charging your data, you need to make sure that it’s worth paying for. Data quality is a human thing. It is everyone’s responsibility.

“Companies need to adapt to fast data. There is more demand for real-time data. People want instant gratification and instant access to data. There is an entire reality that you didn’t know exists. Discovery, Vitality is ahead of the curve. It’s less about complexity and more about having a bouncer”, says a representative from Discovery, Vitality.

How do you start small without over-investing, and how do you get your business buy-in and commitment?  

Understand that it is a process. Getting your systems to speak to each other on a single platform is important. If a single team buys in, there is no value. There needs to be buy-in across the organisation.

“You are going to die nine times before you win, but when you win, you will see the value,” says a representative from Sanlam.

Start small to prove value. A small proof of concept that shows value assists with organisation buy-in. Selling to the business at a segment level and then the board will make the journey better and easier.

“The common challenge is that we want data more often. The solution you’re going to hear is “so just run a batch more frequently” but this limits you. If it is real-time, there is no waiting period”, says a representative from Old Mutual.

Having state-of-the-art tech will assist you in staying relevant and competitive. The tech is not the issue, it’s the people and culture that are important, and understanding the problem to not only try and solve it but to communicate the value of solving it effectively.

“It’s about organisational competitiveness, if you are not upgrading you might be falling behind. People are becoming comfortable with instant gratification so use storytelling to sell the value” say Nick Walker, Confluent.

Culture is progressive. Companies want to be better and don’t want to have a lot of constraints. Having effective change management will also enable a proactive culture and assist you in the process.

Some of the companies’ challenges with their first implementation include:

ABSA: “Bleeding fear of being the first one to go cloud”.  

Osiris Trading: “Having good security expertise and skills.”  

Standard Bank: “Change management and communication of what it all means for the organisation as a whole.”

70% of Fortune 500 companies buy into the concept but not in South Africa – How are other companies motivating value?

The answer is three-fold, explains Jack Bingham, Confluent:

1.Integration:

Organisations are siloed with multiple integrations. They treat data at rest instead of a stream where they are not getting any value. They need to move data from a silo to a stream.  

2.Event Streaming Apps:

By treating data as a stream, they will be able to walk that journey and define the processes and benefits. It’s not just about “real-time” data, it’s about the RIGHT data. The right data is invaluable.  

3.Potential Use Cases:

What does the business not have insight into that can be a game changer if they did? Businesses need to understand and identify the value and potential use cases to cross-sell, hyper-sell, and create business value.  

How is it growing, saving, and protecting the business?

Problem question:

If you knew the state of every event in your business and could reason on top of that data in real-time, what problems would you solve, or opportunities would you capitalise on?

Solution:

The Data Streaming platform will provide the right event data, in the right format, at the right time, for consumption anywhere and at any scale.

This needs to speak to non-tech people who are not necessarily going to understand the tech. Recognise that so many mechanisms are used to understand data and what companies do if they try adding on. Why not use one mechanism as a data stream that can be used for different channels? Reusable events that empower IT with innovative ideas.

Some advice for new investors in Confluent Data Streaming:

Don’t walk the journey alone. Focus on what it all means and what the objective is – Data Quality. Everyone is considering Rands and Cents. By crystalising the opportunity for the rest of the organisation, you will get more buy-in.

Ends

For more information on the innovative work Synthesis has done for its clients, contact us on 087 654 3300

About Synthesis

Synthesis uses innovative technology solutions to provide businesses with a competitive edge today. Synthesis focuses on banking and financial institutions, retail, healthcare and telecommunications sectors in South Africa and other emerging markets.

In 2017 Capital Appreciation Limited, a JSE-listed Fintech company, acquired 100 percent of Synthesis. Following the acquisition, Synthesis remains an independent operating entity within the Capital Appreciation Group providing Cloud, Digital, Payments and RegTech services as well as corporate learning solutions through the Synthesis Academy.

The post We asked the Top Financial Services, Insurance, and Retail companies in South Africa – Is Data the New Currency? appeared first on Synthesis.

The days of waiting for the next episode to be aired or standing in line to get tickets at the movie theatre are behind us. We have a variety of options to choose from, and genres of movies and series are recommended to us based on what we have already watched. Our Netflix feed is personalised according to our likes and dislikes mapping out our personalities on the screen.

How is this possible? With data.

Data represents who we are.

As is common knowledge in the Data world, Data is often stuck and siloed in business areas and not leveraged the way it should be. When data is siloed, businesses lose the benefits of leveraging deep insights on their customers at the point in time when it is most valuable; “the moment of value”. This is often referred to as contextual value and can only be extracted when the data is processed “in motion” as opposed to “at rest”. Through data and event streaming, your data is set into motion to understand your customers better and create a personalised experience. Netflix, LinkedIn and Walmart, to name a few, can only do what they do by enabling Event streaming and setting data into motion. 

Synthesis teamed up with Confluent as an early adopter of Apache Kafka to map out events within businesses to help create personalised journeys for customers. If you knew the state of every event in your bank, and could reason on top of that data, what problems would you solve, or opportunities would you capitalise on?

“A central nervous system for your organisation will provide the right event data, in the right format, at the right time, for consumption anywhere and at any scale.”, says Darren Bak, Synthesis – Head of Intelligent Data.

The entire journey of the event is mapped out with insights from the data generated at every step. Apache Kafka correlates and joins events together in real-time to process multiple concurrent sources of data, where businesses not only gain insight through the journey but process huge streams of data and build real time AI and redundancy.

“Our partners are essential in supporting Confluent’s mission to bring the power of Kafka and Data in Motion to companies of all sizes across the globe. We are excited to collaborate with the Synthesis team to provide further value for our customers in the Netherlands.”, says Dick Lans, Confluent, Regional Director, Netherlands.

Since 2018, Synthesis has been a Consulting and System Integration Focus Partner to Confluent and has built a team of Confluent Certified Kafka Engineers that have deployed Confluent in self-managed scenarios including Kubernetes as well as a Service in the Cloud. Synthesis also provide technical training in Apache Kafka and supports Kafka environments and solutions on-premise and as an AWS cloud service, including containerised architectures.

“Synthesis is Confluent’s trusted partner, having successfully established a data streaming platform that sets Data in Motion across organisations” – Jack Bingham, Confluent, Enterprise Account Executive, UKI & Africa.

This year Synthesis has been afforded the opportunity to present at the Kafka Summit 2023 in London, where some of its Data Streaming experts will be sharing their experience of building and deploying Kafka based solutions across a global client base.

Synthesis is making its mark in Europe and has setup it’s European base in the Netherlands, to further extend its global footprint.

“Our European base in the Netherlands, enables us to further build in-country capability and skills to deliver to a European & UK client base. This further extends our global capabilities and brings our clients access to a global talent pool, whilst also providing highly competitive pricing options.”, says Keith Milner, Synthesis – Executive Head International Expansion & Strategy.

Synthesis already has a team of Confluent trained and certified engineers based in Netherlands, as well as Synthesis Labs, the research & innovation hub that drives the emerging technology and product incubation of the company.

“We’re standing on the threshold of a new era where data becomes hyper-personalised and hyper-contextual. We must shift our view from thinking about big chunks of data to zooming in on individual events as they occur. It’s about People & Moments. Relevant information needs to be processed right then and there; we cannot have data sitting in external systems because we cannot afford to wait for them to catch up. It must be in working memory. It’s like having a conversation versus sending an email – completely different experiences, and both increasingly necessary to build relationships with your customers.”, says Tom, Synthesis – Chief Disruption Officer.

Why choose Synthesis as your data partner?

As a global partner for Confluent, we have a proven record of accelerating time to value in our clients’ businesses. We now have extensive experience in building systems that are highly scalable and reliable that unlock the value in data within a business.  

Projects that include Kafka and Event Streaming based solutions include Vitality Group, Discovery Group, Vodacom, Absa Bank, Capitec Bank, Sanlam Insurance, Bankserv, The Foschini Group and Easy Equities (Purple Group). 

Synthesis has had a successful long-term partnership with the Vitality Group where Synthesis manages the Group’s services. Vitality Group, a global company that spans across several countries, needed assistance moving onto a new event streaming platform. The Group receives myriads of messages per second through smart devices when its customers exercise globally. The event streaming capability is a cheaper, scalable and more reliable platform. Synthesis Event Streaming experts were tagged in, and the team helped replace this platform within budgets and timelines. Kafka, Event streaming allowed Vitality Group to improve reliability due to built-in error handling, scalability and retry functionality. It also has smart message deduplication functionality which allows Vitality Group to ingest more customer data in a shorter period and eliminate message lag. Kafka is also enabling their shared value promise through gained fitness using IOT fitness data.

“As the event streaming capability was a new technology being introduced into my team, my existing developers had very little practical exposure to the product.​ Through our arrangement with Synthesis, we were able to very quickly onboard effective engineers who were able to assist us with defining the solution, rolling it out, and optimising the throughput of the solution while still being able to deliver the project on time.”, says Nic Mouyis, Vitality Group Senior Development Manager.

Synthesis Intelligent Data offers end-to-end solutions across the data journey, incorporating; Data engineering and streaming, Cloud Deployments, Data Analytics, AI and Machine Learning. Confluent makes it easy to connect your apps, systems, and organisation with real-time data flow and processing.

As an international Confluent partner and subscription reseller, we can help you set your data in motion.

Start your journey today and have a conversation with us.

Ends

For more information on the innovative work Synthesis has done for its clients, contact us on 087 654 3300

About Synthesis

Synthesis uses innovative technology solutions to provide businesses with a competitive edge today. Synthesis focuses on banking and financial institutions, retail, healthcare and telecommunications sectors in South Africa and other emerging markets.

In 2017 Capital Appreciation Limited, a JSE-listed Fintech company, acquired 100 percent of Synthesis. Following the acquisition, Synthesis remains an independent operating entity within the Capital Appreciation Group providing Cloud, Digital, Payments and RegTech services as well as corporate learning solutions through the Synthesis Academy.

The post Event streaming of data – why is it a goldmine for businesses? appeared first on Synthesis.