Streaming pipelines have become simpler and onboarding new data sources is now much fasterOne of the best features Upsolver offers is the automatic schema evolution. Another good feature is SQL-based streaming transformations. Complex streaming transformations such as cleansing, deduplication, and enrichment were implemented using SQL and drastically reduced the need for custom Spark code. My experience with the SQL-based streaming transformations in Upsolver is that it had a significant positive impact on the overall data engineering workflow. By replacing custom Spark streaming jobs with declarative SQL logic, I simplified development, review, and deployment processes. Data transformations such as parsing, filtering, enrichment, and deduplication could be implemented and modified quickly without rebuilding or redeploying complex code-based pipelines. Upsolver has impacted my organization positively because it brings many benefits. The first one is faster onboarding of new data sources. Another one is more reliable streaming pipelines. Another one is near-real-time data availability, which is very important for us. It also reduced operational effort for data engineering teams. A specific outcome that highlights these benefits is that the time to onboard new sources is reduced from weeks to days. Custom Spark code reduction reached 50 to 40 percent. Pipeline failures are reduced by 70 to 80 percent. Data latency is improved from hours to minutes.
