cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

giladba
by New Contributor III
  • 666 Views
  • 4 replies
  • 0 kudos

Databricks API - Create Connection

Hi,Is it possible to use the Databricks API to create a connection to a different Azure Databricks workspace?Thanks

  • 666 Views
  • 4 replies
  • 0 kudos
Latest Reply
giladba
New Contributor III
  • 0 kudos

My apologies for not being cleared in the first post.Terraform can handle it. The API is probably behind and can't handle the connection to Databricks yet. The best solution would probably be to just move to Unity Catalog...Thanks for your quick repl...

  • 0 kudos
3 More Replies
Chvyaken
by New Contributor III
  • 1168 Views
  • 5 replies
  • 7 kudos

Resolved! Change Runtime version from 10.4 to 11.3 in ADF

Hello! At the moment, I am considering replacing runtime 10.4 with runtime 11.3 in my ADF. I would like to know how big the differences are between them, and if I can just change the version without worrying about breaking anything? 

Chvyaken_0-1720363212242.png
  • 1168 Views
  • 5 replies
  • 7 kudos
Latest Reply
Witold
Contributor III
  • 7 kudos

On thing worth mentioning is the DBR Migration Guide. There you'll find all potential changes between the runtimes.

  • 7 kudos
4 More Replies
guangyi
by Contributor III
  • 440 Views
  • 1 replies
  • 0 kudos

OutputMode “complete” unable to replace the entire table

According to the document https://docs.databricks.com/en/structured-streaming/delta-lake.html#complete-mode, the “complete” option seems to “replace the entire table with every batch”. However, it is not working in my case.Here is how I reproduce the...

  • 440 Views
  • 1 replies
  • 0 kudos
Latest Reply
guangyi
Contributor III
  • 0 kudos

I figure out it already. I cannot find the delete button. Please ignore this post

  • 0 kudos
ibrahim21124
by New Contributor III
  • 2480 Views
  • 1 replies
  • 1 kudos

Resolved! Databricks Job Timeout after 20 minutes

Hello,I have created a job with no timeout-seconds provided. But I am getting Error: Timed out within 20 minutes. I am running the below commands using Bash@3 task in ADO Pipeline yaml file. The code for the same is given belowtask: Bash@3  timeoutIn...

  • 2480 Views
  • 1 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
Hello,I have created a job with no timeout-seconds provided. But I am getting Error: Timed out within 20 minutes. I am running the below commands using Bash@3 task in ADO Pipeline yaml file. The code for the same is given belowtask: Bash@3  timeoutIn...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
acegerace
by New Contributor II
  • 618 Views
  • 1 replies
  • 1 kudos

Resolved! Deep Clone

Will a deep clone of a table also clone the table's tags?

  • 618 Views
  • 1 replies
  • 1 kudos
Latest Reply
brockb
Databricks Employee
  • 1 kudos

Hi @acegerace , By default, table Tags are not included in the cloned metadata. This can be observed by cloning the table and confirming in the `information_schema.table_tags` system table. If this is not the desired behavior, the tags could be set o...

  • 1 kudos
maoutir
by New Contributor
  • 1027 Views
  • 1 replies
  • 1 kudos

structured streaming hangs when writing or sometimes reading depends on SINGLE USER or shared mode

Hi Guys,I'm new to this community, I am beginning a new project with Azure Databricks and a Python script on my Mac that manipulates data (reading from delta share tables and inserting into local Postgres database ) coming from a remote databricks cl...

Data Engineering
Azure Databricks
Change Data Feed
deltaSharing
python
spark
  • 1027 Views
  • 1 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

Hi,I reckon I've seen this problem earlier and if I remember correctly there was an issue related to Databricks Connect and single user mode.I think your code will work if you use it directly in notebook, but will fail with Databricks Connect.About t...

  • 1 kudos
fk5
by New Contributor II
  • 792 Views
  • 3 replies
  • 1 kudos

SQL Server Connection on LTS 14.3

Hi, what is the best way to connect to a SQL Server on LTS 14.3?I've been trying to setup a connection to an SQL Server as referenced here. However both using both "sqlserver" and "jdbc" as format has resulted in an exception when using display as sp...

  • 792 Views
  • 3 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

Bit weird, I've tested option with format('sqlserver') and I didn't notice any problems on LTS 14.3. Did you check what happens if you use databricks specific display command, like so:display(df) 

  • 1 kudos
2 More Replies
AyushModi038
by New Contributor III
  • 6586 Views
  • 7 replies
  • 7 kudos

Library installation in cluster taking a long time

I am trying to install "pycaret" libraray in cluster using whl file.But it is creating conflict in the dependency sometimes (not always, sometimes it works too.) ​My questions are -1 - How to install libraries in cluster only single time (Maybe from ...

  • 6586 Views
  • 7 replies
  • 7 kudos
Latest Reply
Spencer_Kent
New Contributor III
  • 7 kudos

@Retired_modWhat about question #1, which is what subsequent comments to this thread have been referring to? To recap the question: is it possible for "cluster-installed" libraries to be cached in such a way that they aren't completely reinstalled ev...

  • 7 kudos
6 More Replies
christian_chong
by New Contributor III
  • 932 Views
  • 2 replies
  • 0 kudos

databricks structured streaming external table unity catalog

Hi everbody,I am facing a issue with spark structured steaming, with databricks on gcp. I use external table type but schema are managed schema. The code below in my notebook works very wellBut if i add column masking to target table, and rerun the n...

  • 932 Views
  • 2 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi everbody,I am facing a issue with spark structured steaming, with databricks on gcp. I use external table type but schema are managed schema. The code below in my notebook works very wellBut if i add column masking to target table, and rerun the n...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
cpelazza
by New Contributor III
  • 2092 Views
  • 2 replies
  • 2 kudos

Resolved! Create a SQL (Python) UDF in a Serverless SQL Warehouse using an external library

Hi There,Scenario at work is as described in the subject line:Is it possible to author a SQL (Python) scalar UDF IN A SQL SERVERLESS WAREHOUSE, which involves a library NOT included in any Databricks Runtime? And how would one go about it? (Documenta...

  • 2092 Views
  • 2 replies
  • 2 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 2 kudos

This widget could not be displayed.
Hi There,Scenario at work is as described in the subject line:Is it possible to author a SQL (Python) scalar UDF IN A SQL SERVERLESS WAREHOUSE, which involves a library NOT included in any Databricks Runtime? And how would one go about it? (Documenta...

This widget could not be displayed.
  • 2 kudos
This widget could not be displayed.
1 More Replies
grazie
by Contributor
  • 4875 Views
  • 5 replies
  • 0 kudos

Resolved! Is LEGACY_SINGLE_USER_STANDARD no longer supported?

Based on [an answer in this community](https://community.databricks.com/s/question/0D58Y00009jfKM1SAM/error-no-metastore-assigned-for-the-current-workspace), we set "data security mode" to LEGACY_SINGLE_USER_STANDARD in a cluster policy which needed...

  • 4875 Views
  • 5 replies
  • 0 kudos
Latest Reply
grazie
Contributor
  • 0 kudos

...and it works again - no changes made by us

  • 0 kudos
4 More Replies
lovinchanglei
by New Contributor II
  • 379 Views
  • 0 replies
  • 0 kudos

Failed to signup community version

I've been trying to create Community Edition account, but keep getting: "An error has occurred. Please try again later" message. I searched the other posts, there are some people running into the same issue as well, one post talked about VPN and succ...

lovinchanglei_0-1720155596817.png
Data Engineering
community edition
community version
signup
  • 379 Views
  • 0 replies
  • 0 kudos
Nino2024
by New Contributor II
  • 563 Views
  • 1 replies
  • 1 kudos

Migration Azure to GCP

is there a place I could find the best practices to migrate Databricks from Azure to GCP?

  • 563 Views
  • 1 replies
  • 1 kudos
Latest Reply
amr
Databricks Employee
  • 1 kudos

If you want to migrate Databricks to Databricks, you should use Databricks Teraform Exporter for that job

  • 1 kudos
RushabhNovelis
by New Contributor
  • 652 Views
  • 1 replies
  • 0 kudos

Is it possible to use a finetuned model with genie spaces feature?

I was trying to explore Genie room, I observed that I may need to finetune the model. Can I finetuned exisiting model with the same UI?

  • 652 Views
  • 1 replies
  • 0 kudos
Latest Reply
amr
Databricks Employee
  • 0 kudos

No. Genie spaces uses a fine tuned specialised text-2-sql model. fine-tuning text to 2 SQL model is not an easy job, so I would not recommend you doing yourself unless your business requires it.

  • 0 kudos
mayank_gupta
by New Contributor II
  • 720 Views
  • 2 replies
  • 0 kudos

Trying to create external table in Hive Metastore

Receiving this error: KeyProviderException: Failure to initialize configuration for storage account adlspersonal.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyI used hive meta store to save my table %python spark....

  • 720 Views
  • 2 replies
  • 0 kudos
Latest Reply
amr
Databricks Employee
  • 0 kudos

not sure about that error, but try to do it using SQL, see if it will workcreate table hive_metastore.annual_enterprise_survey as select * from catalog.defaul.table

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels