- 1656 Views
- 2 replies
- 0 kudos
Resolved! Spark read with format as "delta" isn't working with Java multithreading
0I have a Spark application (using Java library) which needs to replicate data from one blob storage to another. I have created a readStream() within it which is listening continuously to a Kafka topic for incoming events. The corresponding writeStre...
- 1656 Views
- 2 replies
- 0 kudos
- 0 kudos
The problem was indeed with the way ClassLoader was being set in the ForkJoinPool (common Pool used) thread. Spark in SparkClassUtils uses Thread.currentThread().getContextClassLoader which might behave differently in another thread.To solve it I cre...
- 0 kudos
- 701 Views
- 0 replies
- 0 kudos
Execute delete query from notebook on azure synapse
Hello Everyone, Is there a way we can execute the delete query from azure notebook on azure synapse database.I tried using read api method with option "query" but getting error like jdbc connector not able to handle code.Can any suggest how we can de...
- 701 Views
- 0 replies
- 0 kudos
- 9914 Views
- 1 replies
- 5 kudos
New how-to guide to data warehousing with the Data Intelligence Platform
Just launched: The Big Book of Data Warehousing and BI, a new hands-on guide focused on real-world use cases from governance, transformation, analytics and AI.As the demand for data becomes insatiable in every company, the data infrastructure has bec...
- 9914 Views
- 1 replies
- 5 kudos
- 5 kudos
lol beans It used to take me a long time to regain my equilibrium, but recently I learned that a website really leads this layout when you may find delight after a stressful day here. Since then, I've been able to find my equilibrium much more quickl...
- 5 kudos
- 827 Views
- 2 replies
- 2 kudos
Create new workbooks with code
Is is possible to create new notebooks from a notevbook in databricks? I have tried this code. But all of them are generic files, not notebooks.notebook_str = """# Databricks notebook source import pyspark.sql.functions as F import numpy as np # CO...
- 827 Views
- 2 replies
- 2 kudos
- 2 kudos
Unfortunaly %run does not help me since I can't %run a .py file. I still need my code in notebooks.I am transpiling propriatary code to python using jinja templates. I would like to have the output as notebooks since those are most convenient to edit...
- 2 kudos
- 1476 Views
- 1 replies
- 0 kudos
Resolved! DLT Pipeline Graph is not detecting dependencies
Hi,This is my first databricks project. I am loading data from a UC external volume in ADLS into tables and then split one of the tables into two tables based on a column. When I create a pipeline, the tables don't have any dependencies and this is...
- 1476 Views
- 1 replies
- 0 kudos
- 0 kudos
While re-implementing my pipeline to publish to dev/test/prod instead of bronze/silver/gold, I think I found the answer. The downstream tables need to use the LIVE schema.
- 0 kudos
- 1588 Views
- 2 replies
- 1 kudos
Resolved! Unpivoting data in live tables
I am loading data from CSV into live tables. I have a live delta table with data like this:WaterMeterID, ReadingDateTime1, ReadingValue1, ReadingDateTime2, ReadingValue2It needs to be unpivoted into this:WaterMeterID, ReadingDateTime1, ReadingValue1...
- 1588 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @SamGreene, The stack function allows you to unpivot columns by rotating their values into rows. It’s available both in Scala and PySpark.
- 1 kudos
- 451 Views
- 0 replies
- 0 kudos
AI uses
Delve into the transformative realm of AI applications, where innovation merges seamlessly with technology's limitless possibilities.Explore the multifaceted landscape of AI uses and its dynamic impact on diverse industries at StackOfTuts.
- 451 Views
- 0 replies
- 0 kudos
- 1492 Views
- 2 replies
- 0 kudos
Resolved! Multi Customer setup
We are trying to do POC to have shared resource like compute across multiple customer, Storage will be different, Is this possible ?
- 1492 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Kroy , When it comes to shared compute resources in Databricks, there are some best practices and options you can consider: Shared Access Mode for Clusters: Databricks allows you to create clusters in shared access mode. This means that multipl...
- 0 kudos
- 2471 Views
- 2 replies
- 3 kudos
Resolved! Stream failure JsonParseException
Hi all! I am having the following issue with a couple of pyspark streams. I have some notebooks running each of them an independent file structured streaming using delta bronze table (gzip parquet files) dumped from kinesis to S3 in a previous job....
- 2471 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @patojo94, You're encountering an issue with malformed records in your PySpark streams. Let's explore some potential solutions: Malformed Record Handling: The error message indicates that there are malformed records during parsing. By default...
- 3 kudos
- 1108 Views
- 1 replies
- 0 kudos
Resolved! Databricks Certification Exam Got Suspended. Need help in resolving the issue
Hi @Cert-Team ,My Databricks exam got suspended on December 9, 2023, at 11:30, and it is still in the suspended state.During the exam, it was initially paused due to poor lighting, but after addressing that, it worked fine. However, after some time, ...
- 1108 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Jay_adb I'm sorry to hear you had this issue. Thanks for filing a ticket with the support team. I have sent a message to them to look into your ticket and resolve asap.
- 0 kudos
- 738 Views
- 0 replies
- 0 kudos
DAB "bundle deploy" Dry Run
Is there a way to perform a dry-run with "bundle deploy" in order to see the job configuration changes for an environment without actually deploying the changes?
- 738 Views
- 0 replies
- 0 kudos
- 8118 Views
- 0 replies
- 1 kudos
🌟 End-of-Year Community Survey 🌟
Hello Community Members, We value your experience and want to make it even better! Help us shape the future by sharing your thoughts through our quick Survey. Ready to have your voice heard? Click here and take a few moments to complete the surv...
- 8118 Views
- 0 replies
- 1 kudos
- 1553 Views
- 3 replies
- 1 kudos
Resolved! More than expected number of Jobs created in Databricks
Hi Databricks Gurus !I am trying to run a very simple snippet :data_emp=[["1","sarvan","1"],["2","John","2"],["3","Jose","1"]]emp_columns=["EmpId","Name","Dept"]df=spark.createDataFrame(data=data_emp, schema=emp_columns)df.show() --------Based on a g...
- 1553 Views
- 3 replies
- 1 kudos
- 1 kudos
I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution.
- 1 kudos
- 585 Views
- 1 replies
- 0 kudos
df.queryExecution.redactedSql is not working with Spark sql Listener
We are trying to capture the query executed by spark .We are trying to use df.queryExecution.redactedSql to get the SQL from query execution but it is not working in sqlListener
- 585 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Soma, In PySpark, when you execute a query and want to capture the SQL from the query execution, you can use the explain() method.
- 0 kudos
- 4034 Views
- 0 replies
- 0 kudos
power Bi paginate
Hello,I am facing similar kind of issue. I am working on Power BI paginated report and databricks is my source for the report. I was trying to pass the parameter by passing the query in expression builder as mentioned below. https://community.databri...
- 4034 Views
- 0 replies
- 0 kudos
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group-
AI Summit
4 -
Azure
3 -
Azure databricks
3 -
Bi
1 -
Certification
1 -
Certification Voucher
2 -
Chatgpt
1 -
Community
7 -
Community Edition
3 -
Community Members
2 -
Community Social
1 -
Contest
1 -
Data + AI Summit
1 -
Data Engineering
1 -
Data Processing
1 -
Databricks Certification
1 -
Databricks Cluster
1 -
Databricks Community
11 -
Databricks community edition
3 -
Databricks Community Rewards Store
3 -
Databricks Lakehouse Platform
5 -
Databricks notebook
1 -
Databricks Office Hours
1 -
Databricks Runtime
1 -
Databricks SQL
4 -
Databricks-connect
1 -
DBFS
1 -
Dear Community
3 -
Delta
10 -
Delta Live Tables
1 -
Documentation
1 -
Exam
1 -
Featured Member Interview
1 -
HIPAA
1 -
Integration
1 -
LLM
1 -
Machine Learning
1 -
Notebook
1 -
Onboarding Trainings
1 -
Python
2 -
Rest API
11 -
Rewards Store
2 -
Serverless
1 -
Social Group
1 -
Spark
1 -
SQL
8 -
Summit22
1 -
Summit23
5 -
Training
1 -
Unity Catalog
4 -
Version
1 -
VOUCHER
1 -
WAVICLE
1 -
Weekly Release Notes
2 -
weeklyreleasenotesrecap
2 -
Workspace
1
- « Previous
- Next »