- 19288 Views
- 10 replies
- 2 kudos
Kafka timout
Hello, I am trying to read topics from a kafaka stream but I am getting the time out error below.java.util.concurrent.ExecutionException: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: describeT...
- 19288 Views
- 10 replies
- 2 kudos
- 2 kudos
What's your Kafka Broker version and which Kafka client is in use (spark's, python-kafka, kafka-confluent,...) ?
- 2 kudos
- 6219 Views
- 3 replies
- 0 kudos
Clarification Needed: Ensuring Correct Pagination with Offset and Limit in PySpark
Hi community,I hope you're all doing well. I'm currently engaged in a PySpark project where I'm implementing pagination-like functionality using the offset and limit functions. My aim is to retrieve data between a specified starting_index and ending_...
- 6219 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, did you find answer to this question? I am having similar problems and a slow solution, which I need to improve upon. Thanks in advance
- 0 kudos
- 13127 Views
- 11 replies
- 6 kudos
Resolved! How can search in a specific folder in Databricks?
There is a keyword search option in Databricks that searches for a command or word in the entire workspace. How can search for a command in a specific folder or repository?
- 13127 Views
- 11 replies
- 6 kudos
- 6 kudos
@AtanuI agree with nelsoncardenas, the problem is not solved, and the answer currently only provides us with saying we need to raise a feature request.Would it be possible to at least link the feature requested by nelsoncardenas to this post/answer? ...
- 6 kudos
- 1678 Views
- 0 replies
- 0 kudos
Implementing ADB Autoloader with Managed File Notification Mode for UC Ext Location (public preview)
Hi everyone,I'm planning to implement Azure Databricks Auto Loader using the Databricks-managed file notification mode for an external location registered in Unity Catalog. I understand this feature is currently in public preview, and I’d love to hea...
- 1678 Views
- 0 replies
- 0 kudos
- 687 Views
- 3 replies
- 0 kudos
Installing Maven in UC enabled Standard mode cluster.
Curios if anyone face the issue of installing Maven packages in UC enabled cluster. Traditionally we use to install maven packages from artifactory repo. I am trying to install the same package from a UC enabled cluster (Standard mode). It worked whe...
- 687 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @nayan_wylde Yes, this is a common challenge when transitioning to Unity Catalog (UC) enabled clusters.The installation of Maven packages from Artifactory repositories does work differently in UC environments,but there are several approaches you c...
- 0 kudos
- 1357 Views
- 1 replies
- 0 kudos
Resolved! How to add permissions to a Databricks Workflow deployed via Asset Bundle YAML?
Hey! I was deploying a new Databricks Workflow into my workspace via Databricks Asset Bundles. Currently, I have a very simple workflow, defined in a YAML file like this: resources: jobs: example_job: name: example_job schedule: ...
- 1357 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @PedroFaria2135, this can be done using the permission key in the YAML file. Please refer to this document: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/reference#permissions permissions: - level: CAN_VIEW group_name: te...
- 0 kudos
- 2474 Views
- 5 replies
- 2 kudos
Resolved! Unable to see All purpose compute
In the workspace, I can only see SQL warehouse, and apps, I've attached a screenshot. I don't see an option to create all purpose compute. Can you please tell me if there is a way to create one? Under user entitlements page look Identity and access >...
- 2474 Views
- 5 replies
- 2 kudos
- 696 Views
- 1 replies
- 1 kudos
Resolved! How to log the errors?
We have a notebook with some generic framework that we created to run for multiple tables everyday. We wanted to log the error/success/exceptions any such errors needs to be recorded in a log table so that we can troubleshoot based on the error log f...
- 696 Views
- 1 replies
- 1 kudos
- 1 kudos
You can basically create some custom functions to log the events and write it to a data lake and then use structured streaming to read the data from data lake to a delta table.%scala// Functionsdef set_local_variables() = { // get the variables ...
- 1 kudos
- 1507 Views
- 6 replies
- 1 kudos
Liquid clustering performance issue
Hello,I have a table with approximately 300 million records. It weighs 3.4 GB and consists of 305 files.I wanted to create liquid clustering for it and chose a date column as the key for clustering. When I created a new table with the above details b...
- 1507 Views
- 6 replies
- 1 kudos
- 1 kudos
Hey @OODDATAEng To create a new table in Databricks using the schema and data from an existing table, you can use the CREATE TABLE AS SELECT command. This command allows you to define a new table based on the results of a SELECT query executed on the...
- 1 kudos
- 5532 Views
- 2 replies
- 1 kudos
Resolved! WorkspaceClient authentication fails when running on a Docker cluster
from databricks.sdk import WorkspaceClientw = WorkspaceClient()ValueError: default auth: cannot configure default credentials ...I'm trying to instantiate a WorkspaceClient in a notebook on a cluster running a Docker image, but authentication fails.T...
- 5532 Views
- 2 replies
- 1 kudos
- 570 Views
- 2 replies
- 0 kudos
Resolved! Git cerdentials for serivce principal running jobs
Hello, I have a permission issue when trying to access Azure DevOps and run a job using a Service Principal.I’ve read about the whole credentials topic, and indeed, when I create a PAT (Personal Access Token) through my personal user account, I can s...
- 570 Views
- 2 replies
- 0 kudos
- 0 kudos
Using a PAT is how you authenticate as a user, so that you can configure your Service Principal (SP) - if you follow this link, there's a guide to the next steps (you're on step 3 now)Thie article explains a bit more on how to setup up the SP in Azur...
- 0 kudos
- 1744 Views
- 0 replies
- 0 kudos
Executors getting killed while Scaling Spark jobs on GPU using RAPIDS(NVIDIA)
Hi Team , I want to take advantage of Spark Distribution over GPU clusters using RAPID(NVIDIA) , everything is setup 1. The Jar is loaded correctly via Init script , the jar is downloaded and uploaded on volume (workspace is unity enabled) and via In...
- 1744 Views
- 0 replies
- 0 kudos
- 1321 Views
- 4 replies
- 1 kudos
Simba JDBC Exception When Querying Tables via BigQuery Databricks Connection
Hello, I have a federated connection to BigQuery that has GA events tables for each of our projects. I'm trying to query each daily table which contains about 400,000 each day, and load into another table, but I keep seeig this Simba JDBC exception. ...
- 1321 Views
- 4 replies
- 1 kudos
- 1 kudos
I also have this issue, and I resolved it by cast all the records columns in bigquery to string before I dump the data.I first create a view likecreate view xxx as select string_1, string_2, string_3, to_json_string(record_1) as record_1, to_json_s...
- 1 kudos
- 1668 Views
- 3 replies
- 1 kudos
Resolved! How Increase REPL time to prevent timeout error
Hi everyone, I've tried setting the Spark configuration spark.databricks.repl.timeout to 300, but I’m still getting a REPL timeout error saying it took longer than 60 seconds. It seems like the configuration might be incorrect. Can someone guide me o...
- 1668 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Saritha_S ,Yes! I've configured spark config that you said. I'll observe for few days and let you know.Thanks! For your Help.
- 1 kudos
- 23931 Views
- 8 replies
- 2 kudos
Connect to Databricks from PowerApps
Hi All,Currently I trying to connect databricks Unity Catalog from Powerapps Dataflow by using spark connector specifying http url and using databricks personal access token as specified in below screenshot: I am able to connect but the issue is when...
- 23931 Views
- 8 replies
- 2 kudos
- 2 kudos
If you are an Azure Databricks customer, there is now a connector for Power Platform (Power Apps, Copilot Studio, and Power Automate) in Public Preview.BlogDocumentation
- 2 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 455 | |
| 349 | |
| 287 |
