- 2127 Views
- 1 replies
- 0 kudos
Facing Issue in Importing Delta Live Tables - Databricks Runtime 14.3
I am facing issues while Importing dlt library in Databricks Runtime 14.3. Previously while using the Runtime 13.1 The `import dlt` was working fine but now when updating the Runtime it is giving me error.This is the Cluster's Configuration Also ...
- 2127 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod Do you have any solution for this above problem ? I saw you reply in this link https://community.databricks.com/t5/data-engineering/no-module-named-dlt/td-p/21105. so i just ask you. Thank you !
- 0 kudos
- 2478 Views
- 4 replies
- 1 kudos
XML file Load to Delta table with different fields list
I there,I have a scenario where the source XML files may have all the fields or may be 80% of fields in next run. How to we load the files in Delta tables which should handle the XML files with all field lists and also with few fields only. In smalle...
- 2478 Views
- 4 replies
- 1 kudos
- 1 kudos
Auto Loader is not acceptable solution in my case. I tried to make an empty table using XSD file and then load the data frame. Some how it worked to meet the objective.
- 1 kudos
- 8422 Views
- 0 replies
- 3 kudos
How I Tuned Databricks Query Performance from Power BI Desktop
As someone who frequently works with large datasets in Power BI, I’ve had my fair share of frustrations with slow query performance, especially when pulling data from Databricks. After countless hours of tweaking and experimenting, I’ve finally found...
- 8422 Views
- 0 replies
- 3 kudos
- 2556 Views
- 3 replies
- 2 kudos
Resolved! Not able to get a SQL warehouse in Databricks AWS
Hey folks, I have been trying to set up an SQL warehouse for Databricks in AWS (on a new account) but I keep getting this:Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you.This kept h...
- 2556 Views
- 3 replies
- 2 kudos
- 2 kudos
Hey Brahma,Thanks for the hints. I actually tried a lot of thinks back and forth and the only thing that finally worked was to create a new workspace. Since this was a new account, that was easy. I suppose something would have gone wrong with the con...
- 2 kudos
- 1645 Views
- 2 replies
- 0 kudos
What are the use cases of AI for data engineers?
I am a data engineer and want to start with AI and ML, how and where to start with?
- 1645 Views
- 2 replies
- 0 kudos
- 0 kudos
@JayanthDE refer this .https://github.com/amueller/introduction_to_ml_with_python/blob/main/03-unsupervised-learning.ipynb
- 0 kudos
- 1904 Views
- 1 replies
- 0 kudos
Clone/copy python file
We are using Databricks extensively in the company. We found out that we can’t clone/copy *.py file using UI. We can clone notebooks but not python file. If we clone folder, we only clone notebooks inside folder but not python file.
- 1904 Views
- 1 replies
- 0 kudos
- 0 kudos
@shiva1212 When working with Databricks and managing Python files, it's true that the UI limitations can sometimes be restrictive. you can use Databricks CLI and REST API for file management and copying.
- 0 kudos
- 4319 Views
- 1 replies
- 0 kudos
How to pass parameters to different notebooks in a workflow?
I have three notebooks: Notebook-1, Notebook-2, and Notebook-3, with a workflow dependency sequence of 1 -> 2 -> 3.Notebook-1 dynamically receives parameters, such as entity-1 and entity-2.Since these parameters change with each run, how can I pass t...
- 4319 Views
- 1 replies
- 0 kudos
- 0 kudos
As long as the parameter names are the same in all Notebooks, whatever value you trigger the workflow with will automatically be sent to all the notebooks. For example, if all three notebooks in the workflow have parameters entity-1 and entity-2, and...
- 0 kudos
- 2774 Views
- 2 replies
- 1 kudos
Databricks Cluster Start and Stop
I want to send out an email if the cluster failed to start, i used to start the cluster using Databricks cli and then terminate them. In some cases ,my cluster is not starting at all and there are some errors.My use case is to send an email using dat...
- 2774 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @mailsathesh ,You can write a script that will use databricks cli to start cluster. You can you --timeout flag to set maximum amount of time to reach running state. If this amoun is exceeded or if there is any error you can then send an email with...
- 1 kudos
- 11331 Views
- 3 replies
- 2 kudos
Resolved! Why Spark Streaming from S3 is returning thousands of files when there are only 9?
I am attempting to stream JSON endpoint responses from an s3 bucket into a spark DLT. I have been very successful in this practice previously, but the difference this time is that I am storing the responses from multiple endpoints in the same s3 buck...
- 11331 Views
- 3 replies
- 2 kudos
- 2 kudos
@Carter Mooring​ Thank you SO MUCH for coming back to provide a solution to your thread! Happy you were able to figure this out so quickly. And I am sure that this will help someone in the future with the same issue.
- 2 kudos
- 2656 Views
- 2 replies
- 0 kudos
Converting databricks query output in JSON and creating end point
Hello I am very new to DB and building an UI where I need to show data from databricks table. Unfortunately , I am getting access delta sharing feature by administrator. Planning to develop own API and expose endpoint with JSON output. I am sure th...
- 2656 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Mangeysh,You could achieve this using Databricks SQL Statement Execution API. I would recommend going through the docs and looking at the functionality and limitations and see if it serves your need before planning to develop your own APIs.
- 0 kudos
- 9238 Views
- 12 replies
- 22 kudos
Resolved! Environment magic command?
Noticed something strange in the Databricks version control- I'm getting hidden commands with %environment that I can only see in the UI for version control. No idea what it's from, just a minor nuisance I'm curious if anyone can shed light on.+ %env...
- 9238 Views
- 12 replies
- 22 kudos
- 22 kudos
Hello all, this is Justin from the PM team at Databricks. We are sorry about the friction this caused you. This was a feature related to the new way that we are doing dependency management in our serverless offerings - Environments. We are going to r...
- 22 kudos
- 6832 Views
- 6 replies
- 6 kudos
Why is writing to MSSQL Server 12.0 so slow directly from spark but nearly instant when I write to a csv and read it back
I have a dataframe that inexplicably takes forever to write to an MS SQL Server even though other dataframes, even much larger ones, write nearly instantly. I'm using this code:my_dataframe.write.format("jdbc") .option("url",sqlsUrl) .optio...
- 6832 Views
- 6 replies
- 6 kudos
- 6 kudos
Had a similar issue. I can do 1-4 million rows in 1 minute via SSIS ETL on SQL server. Table is 15 fields long. Looking at your code it seems you have many fields but nothing like 300-400 fields which can affect performance. You can check SQL Server ...
- 6 kudos
- 8511 Views
- 5 replies
- 1 kudos
Unity Catalog is set up for multi region
Hello Team, I need some clarification on the below diagram . According to the documentation, the Unity Catalog is set up for each region. If we are using multiple clouds, the diagram shows only one Unity Catalog across regions. Shouldn't there be two...
- 8511 Views
- 5 replies
- 1 kudos
- 1 kudos
Yes, the reason why they are grouped into a single rectangle is probably to show that they both are Unity Catalog enabled. It can indeed be confusing, represented like that.If you want to let them connect to each other, delta sharing or metastore fed...
- 1 kudos
- 5464 Views
- 4 replies
- 1 kudos
Resolved! How to execute stored procedures on synapse sql pool from databricks
In the current version of databricks, previous methods to execute stored procedures seem to fail. spark.sparkContext._gateway.jvm.java.sql.DriverManager/spark._sc._gateway.jvm.java.sql.DriverManager returns that it is JVM dependent and will not work....
- 5464 Views
- 4 replies
- 1 kudos
- 1 kudos
can you create a connection to external data in unity catalog, and then:use <connectiondb>;exec <sp>
- 1 kudos
- 1297 Views
- 1 replies
- 0 kudos
playcasinosnj.com
One of the best platforms where you can find games like online slots is https://playcasinosnj.com/. It is in my opinion the safest site to find the games that suit you. I also had the chance to try out the different games and they all worked for me. ...
- 1297 Views
- 1 replies
- 0 kudos
- 0 kudos
Cool, I love slots. Thanks for the tip. I like that they are very diverse and easy to learn, so I often enjoy spending a couple of hours on these games in the evenings after work. Also, I recently found some betting apps not on Gamstop that turned ou...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
