- 13678 Views
- 9 replies
- 2 kudos
Kafka timout
Hello, I am trying to read topics from a kafaka stream but I am getting the time out error below.java.util.concurrent.ExecutionException: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: describeT...
- 13678 Views
- 9 replies
- 2 kudos
- 2 kudos
try this :.option('kafka.session.timeout.ms', 200000).option('group.max.session.timeout.ms', 7200000) kafka.session.timeout.ms: Specifies the timeout for detecting consumer failures.group.max.session.timeout.ms: Sets the maximum allowed session timeo...
- 2 kudos
- 306 Views
- 2 replies
- 0 kudos
FeatureEngineeringClient and Databricks Connect
Hello everyone, I want to use Databricks Connect to connect externally to my clusters and run code, and while Databricks connect works without any issue, like this: ```from databricks.sdk.core import Config config = Config(cluster_id="XXXX")spark = S...
- 306 Views
- 2 replies
- 0 kudos
- 13767 Views
- 12 replies
- 2 kudos
Resolved! Dbfs init script migration
I need help with migrating from dbfs on databricks to workspace. I am new to databricks and am struggling with what is on the links provided.My workspace.yml also has dbfs hard-coded. Included is a full deployment with great expectations.This was don...
- 13767 Views
- 12 replies
- 2 kudos
- 3000 Views
- 5 replies
- 0 kudos
Default maximum spark streaming chunk size in delta files in each batch?
working with delta files spark structure streaming , what is the maximum default chunk size in each batch?How do identify this type of spark configuration in databricks?#[Databricks SQL]​ #[Spark streaming]​ #[Spark structured streaming]​ #Spark​
- 3000 Views
- 5 replies
- 0 kudos
- 0 kudos
doc - https://docs.databricks.com/en/structured-streaming/delta-lake.html Also, what is the challenge while using foreachbatch?
- 0 kudos
- 398 Views
- 4 replies
- 1 kudos
workflow fails when ran using a job cluster but not in shared cluster
Hi,We have a workflow which calls 3 notebooks and when we run this workflow using shared cluster it runs fine, but when ran with job cluster , one of the notebooks fail.This notebook uses sql function Lpad and looks like it errors because of it. Has ...
- 398 Views
- 4 replies
- 1 kudos
- 674 Views
- 5 replies
- 0 kudos
Delta live type 2 scd Liquid clustering on Start and end dates
I've created a DLT pipeline that creates type 2 SCDs and often the __Start_at and __end_at columns are beyond the first 32 columns for stat collection.I'd like to add these columns to liquid clustering without increasing the number of columns in the ...
- 674 Views
- 5 replies
- 0 kudos
- 0 kudos
Are these responses generated by chatgpt? They don't answer the question and very much have the tone of generative AI
- 0 kudos
- 394 Views
- 2 replies
- 0 kudos
Error connecting to MySQL from Databricks: (2003, "Can't connect to MySQL server")
Hello Community,I'm facing an issue connecting to a MySQL database hosted on AWS RDS from within a Data bricks notebook. My Python script to connect to MySQL works fine locally, but when I run it in Data bricks, I receive this error:Error connecting ...
- 394 Views
- 2 replies
- 0 kudos
- 727 Views
- 2 replies
- 1 kudos
Resolved! Asset Bundles pause_status Across Different Environments
HiQuestion probably around best practices, but curious if someone else has dealt with a similar situation. I have 2 Databricks workspaces - one for Dev and one for Prod. Had to be two workspaces because Azure Landing Zones had to be air gapped from e...
- 727 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @RangaSarangan ,We have faced same issue and solved using databricks workflow API and json file for job metadata that consist job and thier respective status for each env.You can create azure devops that run after your cicd pipeline and change the...
- 1 kudos
- 233 Views
- 1 replies
- 0 kudos
CREATED WIDGET - SQL syntax - how do I specify a label?
What is the syntax in SQL for creating a widget in a notebook with a label?This documentation says "The last argument is label, an optional value for the label shown over the widget text box or dropdown."The one example provided on that page doesn't ...
- 233 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @eriodega I do not know how it works with sql but a possibility is to use a python cell in your SQL notebook and create it like this:%python dbutils.widgets.text("some_name", "a great default", "some label/description")All the best
- 0 kudos
- 10507 Views
- 3 replies
- 1 kudos
Schedule a Notebook Dashboard
Hey all,I have a workflow that updates a delta table, and then runs a notebook that generates a dashboard. I was hoping that by adding this second step that the dashboard would get updated to show the most current data, instead of the user needing to...
- 10507 Views
- 3 replies
- 1 kudos
- 1 kudos
@ehpogue at the end of the day I just used chatgpt to rewrite a bunch of python code into SQL and mash together all of the temporary views into one giant query. hacky but it did work
- 1 kudos
- 715 Views
- 1 replies
- 0 kudos
Managed Table
Hi Experts,I have a workspace created and associated a metastore with it, the metastore points to a storage location USDATA and then I create two catalogs in the workspace and one is using default meta store as the external storage location and other...
- 715 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mani2105,if i create a table in the sales catalog without specifiying any external location, will the tables created be managed and will go to the Sales storage account Yes, if you create a table in the sales catalog without specifying any exter...
- 0 kudos
- 2453 Views
- 2 replies
- 1 kudos
Resolved! Unity Catalog Metastore Details
hi,I would like to seek response to my following questions regarding Unity Catalog Metastore's path.While configuring metastore, designating a metastore storage account (in case of Azure, it's ADLS Gen2) seems to be an optional thing. In case I confi...
- 2453 Views
- 2 replies
- 1 kudos
- 1 kudos
The storage container you configure for the metastore will contain the files of managed tables and volumes. The metadata is stored in a database of the Databricks control plane.
- 1 kudos
- 3796 Views
- 1 replies
- 0 kudos
DLT full refresh
Running a task with full refresh in delta live tables removes existing data and reloads it from scratch. We are ingesting data from an event hub topic and from files. The event hub topic stores messages for seven days after arrival. If we would run a...
- 3796 Views
- 1 replies
- 0 kudos
- 0 kudos
I know it's a bit after the fact, but in case you didn't solve it, I came across this article in the Databricks documentation. You can set pipelines.reset.allowed to false on a table to prevent a full refresh of a table. Ref: https://docs.databrick...
- 0 kudos
- 467 Views
- 1 replies
- 0 kudos
Resolved! can't create cluster
Hi,I am trying to learn databricks and created a student account In Azure.currently I have 100$ available with me.But when i try to create a basic cluster with 1DBU/hour I am getting warning and not able to create.This account may not have enough CP...
- 467 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi This is because new accounts on azure have lower quotas: https://learn.microsoft.com/en-us/azure/quotas/quotas-overviewTry to choos another Node-Type like Standard_4Fs, which has less CPU. And you can increase th quota: https://learn.microsoft.com...
- 0 kudos
- 1416 Views
- 7 replies
- 2 kudos
Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' :
Hi there,My cluster version is 15.4 LTS, and the workspace has UC enabled. When I used the initialization script to install ODBC Driver 17 for SQL Server, there were no errors and the cluster started successfully. But when I use ODBC Driver 17 for SQ...
- 1416 Views
- 7 replies
- 2 kudos
- 2 kudos
Also is INIT script route is the only route? or is there any other option available?the reason I am asking, some time back we had a call with DBK and then they mentioned like usage of INIT script is not the right way or so.. cant recall exact explana...
- 2 kudos