- 7167 Views
- 6 replies
- 1 kudos
Resolved! Unable to add column comment in Materialized View (MV)
The following doc suggests the ability to add column comments during MV creation via the `column list` parameter.Thus, the SQL code below is expected to generate a table where the columns `col_1` and `col_2` are commented; however, this is not the ca...
- 7167 Views
- 6 replies
- 1 kudos
- 1 kudos
@leungi you've shared the python language reference. This is the SQL Reference from where I've based my example.
- 1 kudos
- 1699 Views
- 1 replies
- 0 kudos
Making transform on pyspark.sql.Column object outside DataFrame.withColumn method
Hello,I made some transform on pyspark.sql.Column object: file_path_splitted=f.split(df[filepath_col_name],'/') # return Column object file_name = file_path_splitted[f.size(file_path_splitted) - 1] # return Column object Next I used variable "file_na...
- 1699 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Marcin_U , Thank you for reaching out. The transformation you apply within or outside the `withColumn` method will ultimately result in the same Spark plan. The answer is no, it's not possible to have rows mismatch if you're referring to the s...
- 0 kudos
- 3840 Views
- 1 replies
- 0 kudos
migration Azure to AWS
Hello, I need to migrate from Databricks Azure to AWS, using tool-databricks-migration generates many errors, if I do it manually using databeicks-cli, what would be the best practice?Any tips, for example:-first migrate notebooks-second jobs-third u...
- 3840 Views
- 1 replies
- 0 kudos
- 4051 Views
- 3 replies
- 1 kudos
How to speed-up Azure Databricks processing
Hi Team,My team has designed Azure Databricks solution and we are looking for solution to speed-up process.Below are details of project:1- Data is copied from SAP to ADLS-Gen-2 based External location.2- Project follows medallion architecture i.e. we...
- 4051 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod , @raphaelblg , would you like to throw some light on this issue.
- 1 kudos
- 6209 Views
- 4 replies
- 0 kudos
Resolved! Job fails on cluster with runtime version 14.3 with library installation failure error
Library installation failed for library due to user error for jar: \"dbfs:////<<PATH>>/jackson-annotations-2.16.1.jar\"\n Error messages:\nLibrary installation attempted on the driver node of cluster <<clusterId>> and failed. Please refer to the foll...
- 6209 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Edouard_JH Adding more details on this issue.We faced this issue with several other jars in databricks 14.3, adding the error stacktrace for the same, seems like the error comes from changes made under https://issues.apache.org/jira/browse/SPARK-...
- 0 kudos
- 81015 Views
- 11 replies
- 2 kudos
Resolved! Using current_timestamp as a default value in a delta table
I want to add a column to an existing delta table with a timestamp for when the data was inserted. I know I can do this by including current_timestamp with my SQL statement that inserts into the table. Is it possible to add a column to an existing de...
- 81015 Views
- 11 replies
- 2 kudos
- 2 kudos
Can you please provide information on the additional expenses related to using this feature compared to not utilizing it at all?
- 2 kudos
- 13516 Views
- 5 replies
- 1 kudos
Resolved! Can someone help me understand how compute pricing works.
Im looking at using Databricks internally for some Data Science projects. I am however very confused to how the pricing works and would like to obviously avoid high spending right now. Internal documentation and within Databricks All-Purpose Compute...
- 13516 Views
- 5 replies
- 1 kudos
- 1 kudos
Hello,I was able to get a very precise cost of Azure Databricks Clusters and Computers jobs, using the Microsoft API and Databricks APIThen I wrote a simple tool to extract and manipulate the API results and generate detailed cost reports that can be...
- 1 kudos
- 7243 Views
- 2 replies
- 1 kudos
Setting Date Range Picker to some default date
I have a date range picker filter in Databricks Lakeview Dashboard, so when i open dashboard there is no date selected and i want to set a default date. Is that possible with lakeview dashboard filters?
- 7243 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi there - We're working on default filter values actively and that will help here. For now, when you change filter values you'll notice the URL changes. You can always bookmark the URL or share that modified one with other and when they open it, t...
- 1 kudos
- 16226 Views
- 4 replies
- 1 kudos
Resolved! How to import a Lakeview Dashboard programmatically (API or CLI)?
I'm trying to import a Lakeview Dashboard that I've originally exported through the CLI (version 0.213.0). The exported file has extension .lvdash.json and is a single line json file.I can't get it to work, I tried this command: databricks workspace ...
- 16226 Views
- 4 replies
- 1 kudos
- 15960 Views
- 5 replies
- 6 kudos
Resolved! Call a Stored Procedure in Azure Synapse with input and output Params
driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager connection = driver_manager.getConnection(mssql_url, mssql_user, mssql_pass) connection.prepareCall("EXEC sys.sp_tables").execute() connection.close()The above code works fine but however...
- 15960 Views
- 5 replies
- 6 kudos
- 6 kudos
This blog helped me with the output of the stored procedure: https://medium.com/@judy3.yang/how-to-run-sql-procedure-in-databricks-notebook-e28023555565
- 6 kudos
- 1853 Views
- 0 replies
- 2 kudos
Unable to see / download course materials in Advanced Data Engineering with Databricks ID: E-VDG8QV
I am trying to download course materials(.dbc file and the presentation slides) in Advanced Data Engineering with Databricks ID: E-VDG8QV. However I do not see those materials even when I scroll all the way down in that page. I have tried multiple br...
- 1853 Views
- 0 replies
- 2 kudos
- 82726 Views
- 13 replies
- 4 kudos
How can I export a result of a SQL query from a databricks notebook?
The "Download CSV" button in the notebook seems to work only for results <=1000 entries. How can I export larger result-sets as CSV?
- 82726 Views
- 13 replies
- 4 kudos
- 4 kudos
If you have a large dataset, you might want to export it to a bucket in parquet format from your notebook:%python df = spark.sql("select * from your_table_name") df.write.parquet(your_s3_path)
- 4 kudos
- 5650 Views
- 4 replies
- 3 kudos
Sending email alerts to non-databricks user
I am trying to send email alerts to a non databricks user. I am using Alerts feature available in SQL. Can someone help me with the steps.Do I first need to first add Notification Destination through Admin settings and then use this newly added desti...
- 5650 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Mitali Lad​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 3 kudos
- 1827 Views
- 1 replies
- 0 kudos
integrating Azure Databricks with AAD
Hi Team, Could you please provide the details/process for integrating Azure Databricks - Unity Catalog and AAD? Regards,Phani
- 1827 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Phani1 ,These doc pages might be useful for you: Set up and manage Unity CatalogSync users and groups from Microsoft Entra ID
- 0 kudos
- 2197 Views
- 1 replies
- 1 kudos
Upsert into a Delta Lake table with merge when using row masking function
I'm using databricks rls functions on my tables, and i need to make some merges into, but tables with rls functions does not support merge operations (https://docs.databricks.com/en/data-governance/unity-catalog/row-and-column-filters.html#limitation...
- 2197 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ismaelhenzel, if you want to use the "MERGE INTO" sql command, you must turn-off rls. This is by design.
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
