can create custom session variable for login user authentication in databricks .Like HANA session Variables, we have scenarios like today’s spotfire where we use a single generic user to connect to HANA ( we don’t have single sign on enabled ) in th...
Hi @vinay kumar​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...
Hi @prasad95 , To prevent automatic line wrapping in Databricks Notebooks, follow these steps:
Run Selected Text: If you want to execute specific code lines without wrapping, select the desired text and either:
Click Run > Run selected text from ...
HiI am using DLT with Autoloader.DLT pipeline is running in Continuous mode.Autoloader is in Directory Listing mode (Default)Question.I want to move files that has been processed by the DLT to another folder (archived) and planning to have another no...
Hi @Gilg, Deploying an AutoML pipeline in production while using a shared cluster in Databricks can be a bit tricky due to compatibility constraints.
Let’s explore some potential workarounds:
Shared Cluster with AutoML Compatibility:
As you ment...
Hi, When using the MERGE statement, if merge key is not unique on both source and target, it will throw error. If merge key is unique in source but not unique in target, WHEN MATCHED THEN DELETE/UPDATE should work or not? For example merge key is id....
Cool, this is what I tested out. Great to get confirmed. Thanks. BTW, https://medium.com/@ritik20023/delta-lake-upserting-without-primary-key-f4a931576b0 has a workaround which can fix the merge with duplicate merge key on both source and target.
I have a SQL query that generates a table. I created a visualization from that table with the UI. I then have a widget that updates a value used in the query and re-runs the SQL, but then the visualization shows nothing, that there is "1 row," but if...
Hi @Erik_L , It seems like you’re encountering an issue with your visualization in Databricks.
Let’s troubleshoot this!
Here are a few common reasons why visualizations might not display as expected:
Data Issues:
Ensure that your SQL query is cor...
I want to run Databricks ETLs on on-prem Unix, Azure and on AWS (in future). I am trying to find suitable JDBC drivers but couldn't find anything except CDATA which is very costly.Can someone please help me? Also, what could be other viable solutions...
Hi @aurora, Let’s explore your options for running Databricks ETLs and connecting to Microsoft Dataverse.
1. JDBC Drivers for Databricks:
Databricks provides its own JDBC drivers that allow you to connect to various data sources, including Microso...
While writing the data to a pre-existing table in the unity catalog using JDBC. it just writes the Delta of the data. Driver used: com.databricks:databricks-jdbc:2.6.36Lets say I have the table has rows:+-+-+
|a|b|
+-+-+
|1|2|
|3|4| and I am appendi...
Hi @DumbBeaver, When writing data to a pre-existing table in the Unity Catalog using JDBC, it’s essential to understand how the .union operation and the .overwrite mode work.
Union Operation:
When you use .union to append rows to an existing Data...
Hi community,I hope you're all doing well. I'm currently engaged in a PySpark project where I'm implementing pagination-like functionality using the offset and limit functions. My aim is to retrieve data between a specified starting_index and ending_...
Hi @himanshu_k, Let’s delve into your questions regarding pagination using the offset and limit functions in PySpark, especially when dealing with partitioned data frames.
Consistency of offset and limit Functions:
The offset and limit functions ...
Hi, Do you have any experience of rotating Tokens in Delta Sharing automatically?There is an option to do that using CLI (Create and manage data recipients for Delta Sharing | Databricks on AWS). But what to do next? Sending new link to the token via...
Hi @Leszek, Rotating tokens in Delta Sharing is a crucial security practice.
Let’s break down the steps:
Token Rotation:
First, you’ve already taken the right step by using the CLI to create and manage data recipients for Delta Sharing. When you...
Hi,Has anyone successfully configure azure apim to access databricks rest api ? If yes, appreciate he can provide the setup guide for me as I am stuck at this point. Thanks.
Hi @Check, Configuring Azure API Management (APIM) to access Databricks REST API can be a bit tricky, but I’ll guide you through some potential approaches:
Using Environment Variables and cURL:
To execute Databricks API via a curl request, you ne...
I want to install my own Python wheel package on a cluster but can't get it working. I tried two ways: I followed these steps: https://docs.databricks.com/en/workflows/jobs/how-to/use-python-wheels-in-workflows.html#:~:text=March%2025%2C%202024,code%...
@397973 - Once you uploaded the .whl file, did you had a chance to list the file manually in the notebook?
Also, did you had a chance to move the files to /Volumes .whl file?
Hi,I have a delta live table workflow with storage enabled for cloud storage to a blob store.Syntax of bronze table in notebook===@dlt.table(spark_conf = {"spark.databricks.delta.schema.autoMerge.enabled": "true"},table_properties = {"quality": "bron...
Hi Kaniz,Thanks for replying back.I am using python for delta live table creation, so how can I set these configurations?When creating the table, add the IF NOT EXISTS clause to tolerate pre-existing objects.consider using the OR REFRESH clause Answe...
I am facing an issue when using databricks, when I set a specific type in my schema and read a json, its values are fine, but after saving my df and loading again, the value is gone.I have this sample code that shows this issue: from pyspark.sql.typ...
I have a requirement to read and parse JSON files using autoloader where incoming JSON file has multiple sub entities. Each sub entity needs to go into its own delta table. Alternatively we can write each entity data to individual files. We can use D...
I think using DLT's medallion architecture should be helpful in this scenario. You can write all the incoming data to one bronze table and one silver table. And you can have multiple gold tables based on the value of the sub-entities.
Hi,I'm trying to do the vacuum on a Delta table within a unity catalog. The default retention is 7 days. Though I vacuum the table, I'm able to see the history beyond 7 days. Tried restarting the cluster but still not working. What would be the fix ?...
No, that's wrong. VACUUM removes all files from the table directory that are not managed by Delta, as well as data files that are no longer in the latest state of the transaction log for the table and are older than a retention threshold.VACUUM - Azu...
