Hi Everyone.I am trying to connect and read data from the Databricks table using SQL Warehouse and return it using Azure API.However, the non-English characters, for example, 'Ä', are present in the response as following: ��.I am using the databricks...
Hi @crankerkor,
JDBC Driver Configuration:
Ensure that you are using the correct JDBC driver. You mentioned using the databricks-jdbc driver. Make sure it’s the latest version and compatible with your Databricks cluster.The Simba Spark JDBC driv...
Hi team.Everyday once we overwrite the last X month data in tables. So it generate a every day a larger amount of history. We don't use time travel so we don't need it.What we done:SET spark.databricks.delta.retentionDurationCheck.enabled = false ALT...
Hi @Spenyo,
Consider increasing the retention duration if you need to retain historical data for longer periods.If you’re not using time travel, you can set a retention interval of at least 7 days to strike a balance between history retention and st...
HI All,We are ingesting 1000 files in json format and different sizes per minute. DLT is in continuous mode. Unity Catalog is enabled workspace. We are using the default setting of Autoloader (Directory Listing) and Silver has CDC as well.We aim to ...
Hi @Gilg, Achieving near real-time processing for your data ingestion and processing pipeline is crucial.
Here are some best practices to consider:
Plan your data isolation model: When using a data platform like Azure Databricks, consider setting up...
Hi there! Appreciate this reply is 3 years later than it was originally asked, but people might be coming across it still. A few things:
Koalas was deprecated in spark 3.2 (runtime 10.4). Instead, the recommendation is to use pandas on spark with `im...
I am able to ues expectation feature in delta live table using by creating the expectations as below checks = {}checks["validate circuitId col for null values"] = "(circuitId IS not NULL)"checks["validate name col for null values"] = "(name IS not ...
Hi @manish1987c, To dynamically generate expectations based on different conditions, you can create the dlt table inside an if condition.
If you encounter any further issues, feel free to ask for additional assistance!
We are using unity catalog.Is there a way to set up relations in unity catalog tables like key column relations, 1 to many, many to 1.Can we also generate ER diagrams if we are able to set up these relations.
Hi @NarenderKumar,
Unity Catalog allows you to define relationships between tables using key columns. Here are the common types of relationships you can set up:
One-to-Many (1:N): In this relationship, one record in the primary table corresponds to ...
Hi team, I installed Databricks CLI, and run "databricks auth login --profile xxx" successfully. I can also connect from vscode to Databricks. "databricks clusters list -p xxx" also works. But when I tried to rundatabricks bundle validateI got"Error:...
Hi @Brad,
Ensure that your Databricks CLI configuration is correctly set up with the right access token. Verify that the token is mentioned in both the password field and the Extra field.The Extra field should be configured with a JSON string like t...
Hi AllI just wanted to create PAT for Databricks Service Principle but getting below code while hitting API or using CLI - Please help me to create PAT for the same.#dataengineering #databricks
Hi Team,We have a job it completes in 3 minutes in one Databricks cluster, if we run the same job in another databricks cluster it is taking 3 hours to complete.I am quite new to Databricks and need your guidance on how to find out where databricks s...
can create custom session variable for login user authentication in databricks .Like HANA session Variables, we have scenarios like today’s spotfire where we use a single generic user to connect to HANA ( we don’t have single sign on enabled ) in th...
Hi @vinay kumar​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...
Hi @prasad95 , To prevent automatic line wrapping in Databricks Notebooks, follow these steps:
Run Selected Text: If you want to execute specific code lines without wrapping, select the desired text and either:
Click Run > Run selected text from ...
HiI am using DLT with Autoloader.DLT pipeline is running in Continuous mode.Autoloader is in Directory Listing mode (Default)Question.I want to move files that has been processed by the DLT to another folder (archived) and planning to have another no...
Hi @Gilg, Deploying an AutoML pipeline in production while using a shared cluster in Databricks can be a bit tricky due to compatibility constraints.
Let’s explore some potential workarounds:
Shared Cluster with AutoML Compatibility:
As you ment...
Hi, When using the MERGE statement, if merge key is not unique on both source and target, it will throw error. If merge key is unique in source but not unique in target, WHEN MATCHED THEN DELETE/UPDATE should work or not? For example merge key is id....
Cool, this is what I tested out. Great to get confirmed. Thanks. BTW, https://medium.com/@ritik20023/delta-lake-upserting-without-primary-key-f4a931576b0 has a workaround which can fix the merge with duplicate merge key on both source and target.
I have a SQL query that generates a table. I created a visualization from that table with the UI. I then have a widget that updates a value used in the query and re-runs the SQL, but then the visualization shows nothing, that there is "1 row," but if...
Hi @Erik_L , It seems like you’re encountering an issue with your visualization in Databricks.
Let’s troubleshoot this!
Here are a few common reasons why visualizations might not display as expected:
Data Issues:
Ensure that your SQL query is cor...
I want to run Databricks ETLs on on-prem Unix, Azure and on AWS (in future). I am trying to find suitable JDBC drivers but couldn't find anything except CDATA which is very costly.Can someone please help me? Also, what could be other viable solutions...
Hi @aurora, Let’s explore your options for running Databricks ETLs and connecting to Microsoft Dataverse.
1. JDBC Drivers for Databricks:
Databricks provides its own JDBC drivers that allow you to connect to various data sources, including Microso...
